Metadata-Version: 2.4
Name: ic-python-db
Version: 0.15.0
Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
Home-page: https://github.com/smart-social-contracts/ic-python-db
Author: Smart Social Contracts
Author-email: Smart Social Contracts <contact@smartsocialcontracts.org>
License: MIT License
        
        Copyright (c) 2025 Smart Social Contracts
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/smart-social-contracts/ic-python-db
Project-URL: Repository, https://github.com/smart-social-contracts/ic-python-db.git
Project-URL: Issues, https://github.com/smart-social-contracts/ic-python-db/issues
Keywords: database,key-value,entity,relationship,audit
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# IC Python DB

A lightweight key-value database with entity relationships and audit logging capabilities, intended for small to medium-sized applications running on the Internet Computer. Forked from [kybra-simple-db](https://github.com/smart-social-contracts/kybra-simple-db).

[![Test on IC](https://github.com/smart-social-contracts/ic-python-db/actions/workflows/test_ic.yml/badge.svg)](https://github.com/smart-social-contracts/ic-python-db/actions)
[![Test](https://github.com/smart-social-contracts/ic-python-db/actions/workflows/test.yml/badge.svg)](https://github.com/smart-social-contracts/ic-python-db/actions)
[![PyPI version](https://badge.fury.io/py/ic-python-db.svg)](https://badge.fury.io/py/ic-python-db)
[![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3107/)
[![License](https://img.shields.io/github/license/smart-social-contracts/ic-python-db.svg)](https://github.com/smart-social-contracts/ic-python-db/blob/main/LICENSE)

## Features

- **Persistent Storage**: Works with StableBTreeMap stable structure for persistent storage on your canister's stable memory so your data persists automatically across canister upgrades.
- **Range-based paging**: `page`, `load_some` and `rebuild_field_index` read a page of entities with a single ordered `range` call on the stable map (Basilisk `StableBTreeMap.range`), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends without `range` (older CDK builds, plain dict test doubles) transparently fall back to one read per id. `instances()` and `find()` load a whole type: they are deprecated and raise `TooLargeError` past 1,000 entities (use `count()`, `find_by()` or `page()`).
- **Entity-Relational Database**: Create, read and write entities with OneToOne, OneToMany, ManyToOne, and ManyToMany relationships.
- **Schema Versioning & Upgrade Safety**: Automatic schema introspection, compatibility checking, and auto-migration for safe changes. Breaking changes without a `migrate()` method are rejected, preventing data corruption on canister upgrades.
- **Entity Hooks**: Intercept and control entity lifecycle events (create, modify, delete) with `on_event` hooks.
- **Access Control**: Thread-safe context management for user identity tracking and ownership-based permissions.
- **Namespaces**: Organize entities into namespaces to avoid type conflicts when you have multiple entities with the same class name.
- **Audit Logging**: Track all changes to your data with created/updated timestamps and who created and updated each entity.


## Installation

```bash
pip install ic-python-db
```

## Quick Start

The database storage must be initialized before using IC Python DB. Here's an example of how to do it:

```python
from basilisk import StableBTreeMap
from ic_python_db import Database

# Initialize storage and database
storage = StableBTreeMap[str, str](memory_id=1, max_key_size=100, max_value_size=1000)  # Use a unique memory ID for each storage instance
Database.init(db_storage=storage)
```

Read [Basilisk's documentation](https://github.com/smart-social-contracts/basilisk) for more information regarding StableBTreeMap and memory IDs.

Next, define your entities:

```python
from ic_python_db import (
    Database, Entity, String, Integer,
    OneToOne, OneToMany, ManyToOne, ManyToMany, TimestampedMixin
)

class Person(Entity, TimestampedMixin):
    __alias__ = "name"  # Use `name` as the alias field for lookup
    name = String(min_length=2, max_length=50)
    age = Integer(min_value=0, max_value=120)
    friends = ManyToMany("Person", "friends")
    mother = ManyToOne("Person", "children")
    children = OneToMany("Person", "mother")
    spouse = OneToOne("Person", "spouse")
```

### Entity Lookup

Entities can be retrieved using `Entity[key]` syntax with three different lookup modes:

```python
# Create an entity
john = Person(name="John", age=30)

# Lookup by ID
Person[1]                  # Returns john (by _id)
Person["1"]                # Also works with string ID

# Lookup by alias (defined via __alias__)
Person["John"]             # Tries ID first, then alias field "name"

# Lookup by specific field (tuple syntax)
Person["name", "John"]     # Lookup by field "name" only
```

| Syntax | Behavior |
|--------|----------|
| `Person[1]` | Lookup by `_id` |
| `Person["John"]` | Try `_id` first, then `__alias__` field |
| `Person["name", "John"]` | Lookup by specific field `name` only |

Then use the defined entities to store objects:

```python
    # Create and save an object
    john = Person(name="John", age=30)

    # Update an object's property
    john.age = 33  # Type checking and validation happens automatically

    # use the `_id` property to load an entity with the [] operator
    Person(name="Peter")
    peter = Person["Peter"]

    # Delete an object
    peter.delete()

    # Create relationships
    alice = Person(name="Alice")
    eva = Person(name="Eva")
    john.mother = alice
    assert john in alice.children
    eva.friends = [alice]
    assert alice in eva.friends
    assert eva in alice.friends

    print(alice.serialize())  # Prints the dictionary representation of an object
    # Prints: {'timestamp_created': '2025-09-12 22:15:35.882', 'timestamp_updated': '2025-09-12 22:15:35.883', 'creator': 'system', 'updater': 'system', 'owner': 'system', '_type': 'Person', '_id': '3', 'name': 'Alice', 'age': None, 'children': '1', 'friends': '4'}

    assert Person.count() == 3
    assert Person.max_id() == 4

    # Cursor-based pagination: (page, next_from_id), None on the last page
    page, cursor = Person.page(1, 2)
    assert page == [john, alice]
    page, cursor = Person.page(cursor, 2)
    assert page == [eva] and cursor is None

    # Retrieve database contents in JSON format
    print(Database.get_instance().dump_json(pretty=True))

    # Audit log
    audit_records = Database.get_instance().get_audit(id_from=0, id_to=5)
    pprint(audit_records['0'])
    ''' Prints:

    ['save',
    1744138342934,
    'Person@1',
    {'_id': '1',
    '_type': 'Person',
    'age': 30,
    'creator': 'system',
    'name': 'John',
    'owner': 'system',
    'timestamp_created': '2025-04-08 20:52:22.934',
    'timestamp_updated': '2025-04-08 20:52:22.934',
    'updater': 'system'}]

    '''
```

For more usage examples, see the [tests](tests/src/tests).

## Indexed Fields

Declare `indexed=True` on a scalar property to maintain a persistent secondary index, enabling equality lookups without scanning all entities:

```python
class Proposal(Entity):
    title = String()
    status = String(max_length=32, indexed=True)
    org_scope = String(max_length=128, indexed=True)

Proposal(title="A", status="voting", org_scope="Justice")
Proposal(title="B", status="executed", org_scope="Justice")

# Paginated equality lookup — cost proportional to matches, not max_id
entities, next_from_id = Proposal.find_by("status", "voting", from_id=1, count=50)

# Count without loading entities
Proposal.count_by("org_scope", "Justice")  # 2
```

The index is maintained automatically on create, update, and delete. `None` values are not indexed. Querying a field that is not declared `indexed=True` raises `ValueError`.

### Backfilling existing data

Entities created before a field became indexed are missing from its index. Rebuild in bounded batches (resumable, safe for IC instruction limits — spread calls across multiple update calls or timers):

```python
cursor = 1
while cursor is not None:
    cursor = Proposal.rebuild_field_index("status", from_id=cursor, batch=200)
```

The rebuild is idempotent. The `indexed` flag is not part of the schema descriptor, so toggling it never triggers upgrade-compatibility errors.

Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.

### Values that hold `|`

An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.

**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:

```python
db = Database.get_instance()
cursor = ""
while cursor is not None:
    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
```

`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.

When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.

Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).

**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.

## Sorted Fields

Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:

```python
class Listing(Entity):
    name = String()
    installs = Integer(sorted=True)
    likes = Integer(sorted=True)
    updated_at = Float(sorted=True)
    __sorted_indexes__ = [("installs", "likes")]   # installs, then likes

top, cursor = Listing.top_by("installs", count=20)              # most installed
top, cursor = Listing.top_by(("installs", "likes"), count=20)   # ties broken by likes
recent, cursor = Listing.top_by("updated_at", count=20)
oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
more, cursor = Listing.top_by("installs", count=20, cursor=cursor)   # next page; None at the end
```

The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.

`where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.

Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.

## Large Relations

`OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:

```python
if section.stands: ...                       # one read
len(section.stands)                          # one read
stand in section.stands                      # two reads
stands, cursor = section.stands.page(1, 50)  # one range read + 50 loads
group.members.add(user)                      # index writes only
```

Iterating, indexing and comparing still work, but they load every member, so past `constants.MAX_UNPAGED` (1,000) they raise `TooLargeError` instead of running into the IC instruction limit. Walk a large relation with `page()`, across calls if needed:

```python
cursor = 1
while cursor is not None:
    stands, cursor = section.stands.page(cursor, 50)
```

`list(view)` gives a plain list. Deleting a parent is O(1): its index entries are purged in bounded steps by later writes, or by `Database.get_instance().purge_step()` from a timer.

## Namespaces

Organize entities with the `__namespace__` attribute to avoid type conflicts when you have the same class name in different modules:

```python
# In app/models.py
class User(Entity):
    __namespace__ = "app"
    name = String()
    role = String()
```

```python
# In admin/models.py  
class User(Entity):
    __namespace__ = "admin"
    name = String()
    permissions = String()
```

```python
from app.models import User as AppUser
from admin.models import User as AdminUser

app_user = AppUser(name="Alice", role="developer")      # Stored as "app::User"
admin_user = AdminUser(name="Bob", permissions="all")     # Stored as "admin::User"

# Each namespace has isolated ID sequences and storage
assert app_user._id == "1"
assert admin_user._id == "1"
```

## Entity Hooks

Intercept and control entity changes with the `on_event` hook:

```python
from ic_python_db import Entity, String, ACTION_MODIFY

class User(Entity):
    name = String()
    email = String()
    
    @staticmethod
    def on_event(entity, field_name, old_value, new_value, action):
        # Validate email format
        if field_name == "email" and "@" not in new_value:
            return False, None  # Reject invalid email
        
        # Auto-capitalize names
        if field_name == "name":
            return True, new_value.upper()
        
        return True, new_value

user = User(name="alice", email="alice@example.com")
assert user.name == "ALICE"  # Auto-capitalized
```

See [docs/HOOKS.md](docs/HOOKS.md) for more patterns.

## Access Control

User context management with `as_user()`:

```python
from ic_python_db import Database, Entity, String, ACTION_MODIFY, ACTION_DELETE
from ic_python_db.mixins import TimestampedMixin
from ic_python_db.context import get_caller_id

class Document(Entity, TimestampedMixin):
    title = String()
    
    @staticmethod
    def on_event(entity, field_name, old_value, new_value, action):
        caller = get_caller_id()
        
        # Only owner can modify or delete
        if action in (ACTION_MODIFY, ACTION_DELETE):
            if entity._owner != caller:
                return False, None
        
        return True, new_value

db = Database.get_instance()

# Alice creates a document
with db.as_user("alice"):
    doc = Document(title="My Doc")  # Owner: alice

# Bob cannot modify Alice's document
with db.as_user("bob"):
    doc.title = "Hacked"  # Raises ValueError
```

See [docs/ACCESS_CONTROL.md](docs/ACCESS_CONTROL.md) and [examples/simple_access_control.py](examples/simple_access_control.py).

## Type Hints

The library is fully typed (PEP 561 compliant). Type checkers and IDEs automatically infer property types:

```python
class User(Entity):
    name = String()      # Inferred as str
    age = Integer()      # Inferred as int
    active = Boolean()   # Inferred as bool

user = User(name="Alice", age=30, active=True)
user.name   # IDE knows this is str
user.age    # IDE knows this is int
```

For stricter typing, you can add explicit annotations:

```python
from typing import Optional

class User(Entity):
    name: str = String()
    age: int = Integer()
    profile: Optional["Profile"] = OneToOne("Profile", "user")
```

## Schema Versioning & Upgrade Safety

ic-python-db includes built-in upgrade compatibility checking. The system introspects your Entity class definitions to detect schema changes and ensure safe upgrades.

### Auto-migration for safe changes

Adding a field with a default value requires no migration code — the system handles it automatically:

```python
# v1
class Product(Entity):
    __version__ = 1
    name = String()

# v2 — just add the field with a default, no migrate() needed
class Product(Entity):
    __version__ = 2
    name = String()
    price = Float(default=0.0)    # auto-injected for existing entities
    active = Boolean(default=True) # auto-injected for existing entities
```

### Custom migration for breaking changes

For type changes, field renames, or data transformations, override `migrate()`:

```python
class Product(Entity):
    __version__ = 2
    name = String()
    price_dollars = Float()  # was price_cents: Integer in v1

    @classmethod
    def migrate(cls, obj, from_version, to_version):
        if from_version == 1:
            obj["price_dollars"] = obj.pop("price_cents") / 100.0
        return obj
```

### Upgrade compatibility enforcement

Call `check_upgrade_compatibility()` from your canister's `post_upgrade` to reject incompatible upgrades before they corrupt data:

```python
from basilisk import post_upgrade
from ic_python_db import Database

@post_upgrade
def on_post_upgrade():
    db = Database.get_instance()
    db.check_upgrade_compatibility()
    # If a breaking change lacks migrate(), this raises SchemaIncompatibleError
    # which traps post_upgrade, causing the IC to roll back the upgrade
```

The system classifies schema changes as:

| Change | Safety | Action |
|--------|--------|--------|
| Add field with `default=` | Safe | Auto-migrated |
| Remove field | Safe | Old data ignored |
| Add new Entity type | Safe | No migration needed |
| Change field type | Breaking | Requires `migrate()` |
| Add field without default | Breaking | Requires `migrate()` |
| Change relationship type | Breaking | Requires `migrate()` |

### Schema introspection

You can inspect and compare schemas programmatically:

```python
from ic_python_db import Database, build_schema, diff_schemas

db = Database.get_instance()

# Build current schema from Entity definitions
schema = db.build_schema_from_entities()

# Compare two schemas
changes = diff_schemas(old_schema, new_schema)
for change in changes:
    print(f"{change.entity_type}.{change.field}: {change.reason}")
```

See [docs/SCHEMA_VERSIONING.md](docs/SCHEMA_VERSIONING.md) for the full reference.

## API Reference

- **Core**: `Database`, `Entity`
- **Properties**: `String`, `Integer`, `Float`, `Boolean`
- **Relationships**: `OneToOne`, `OneToMany`, `ManyToOne`, `ManyToMany`
- **Mixins**: `TimestampedMixin` (timestamps and ownership tracking)
- **Hooks**: `ACTION_CREATE`, `ACTION_MODIFY`, `ACTION_DELETE`
- **Context**: `get_caller_id()`, `set_caller_id()`, `Database.as_user()`
- **Schema**: `build_schema`, `diff_schemas`, `schema_hash`, `SchemaIncompatibleError`

## Development

### Setup Development Environment

```bash
# Clone the repository
git clone https://github.com/smart-social-contracts/ic-python-db.git
cd ic-python-db

# Recommended setup
pyenv install 3.10.7
pyenv local 3.10.7
python -m venv venv
source venv/bin/activate

# Install development dependencies
pip install -r requirements-dev.txt

# Running tests
./run_linters.sh && (cd tests && ./run_test.sh && ./run_test_ic.sh)
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

[MIT](LICENSE).
