Metadata-Version: 2.5
Name: promptev-context-engine
Version: 0.0.4
Summary: Promptev Context Engine — language-agnostic ingestion + hybrid retrieval
Project-URL: Homepage, https://promptev.ai/documentation/context-engine/
Project-URL: Documentation, https://promptev.ai/documentation/context-engine/
Project-URL: Repository, https://github.com/promptev/context-engine
Project-URL: Issues, https://github.com/promptev/context-engine/issues
Author: Promptev Inc
License: Apache-2.0
License-File: LICENSE.md
License-File: NOTICE
Keywords: embeddings,graph-rag,hybrid-search,llm,mcp,pii-redaction,rag,retrieval,vector-search
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: alembic>=1.13
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: cryptography>=42
Requires-Dist: httpx>=0.27
Requires-Dist: json-repair>=0.60
Requires-Dist: lingua-language-detector>=2.1
Requires-Dist: openai>=1.40
Requires-Dist: openpyxl>=3.1
Requires-Dist: pgvector>=0.4
Requires-Dist: pydantic-settings>=2.4
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=5.0
Requires-Dist: pypdfium2>=4.30
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: compute
Requires-Dist: pandas>=2.0; extra == 'compute'
Provides-Extra: django
Requires-Dist: django>=4.2; extra == 'django'
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.115; extra == 'fastapi'
Requires-Dist: python-multipart>=0.0.32; extra == 'fastapi'
Provides-Extra: flask
Requires-Dist: flask[async]>=3.0; extra == 'flask'
Provides-Extra: gemini
Requires-Dist: google-genai>=1.24; extra == 'gemini'
Provides-Extra: graph
Requires-Dist: neo4j>=5.20; extra == 'graph'
Requires-Dist: networkx>=3.2; extra == 'graph'
Provides-Extra: mcp
Requires-Dist: aiohttp>=3.9; extra == 'mcp'
Requires-Dist: mcp>=1.2; extra == 'mcp'
Provides-Extra: ocr
Requires-Dist: pillow>=10; extra == 'ocr'
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: pdf
Requires-Dist: pdf-inspector>=0.2.6; extra == 'pdf'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
Provides-Extra: reranker
Provides-Extra: vision
Requires-Dist: pillow>=10; extra == 'vision'
Description-Content-Type: text/markdown

<!-- Absolute URL, not a relative path: this README is also the PyPI
     long-description and PyPI does not resolve relative image paths.
     A copy of the mark lives at assets/promptev-logo.png. -->
<p align="center">
  <img src="https://promptev.ai/favicon.png" alt="Promptev" width="88" height="88">
</p>

<h1 align="center">context-engine</h1>

<p align="center">
  <b>Retrieval that enforces access control and PII redaction inside the query,<br>
  as a library, on your own Postgres.</b>
</p>

<p align="center">
  <a href="https://pypi.org/project/promptev-context-engine/"><img src="https://img.shields.io/pypi/v/promptev-context-engine?label=PyPI&color=3775A9" alt="PyPI"></a>
  <a href="https://www.npmjs.com/package/@promptev/context-engine"><img src="https://img.shields.io/npm/v/@promptev/context-engine?label=npm&color=CB3837" alt="npm"></a>
  <a href="https://github.com/promptev/context-engine/actions/workflows/publish.yml"><img src="https://github.com/promptev/context-engine/actions/workflows/publish.yml/badge.svg" alt="CI"></a>
  <img src="https://img.shields.io/badge/python-3.12%2B-3776AB" alt="Python 3.12+">
  <a href="LICENSE.md"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="Apache-2.0"></a>
</p>

> **Built by [Promptev](https://promptev.ai?utm_source=github&utm_medium=readme) — the governed AI agent platform.**
> This library is the retrieval engine behind Promptev's Context Packs,
> open-sourced (Apache-2.0). Use it standalone, or
> [see what the platform adds](#when-you-outgrow-the-library).

Most retrieval stacks make permissions your problem: you filter results after
the fact, in application code, and hope the ordering is right. This one pushes
the ACL into the SQL predicate every search leg shares, so a document a caller
may not see is never fetched, never ranked, and never reaches a reranker. PII
redaction sits at the same depth — masked *before* hits are POSTed to a hosted
reranker and *before* the tool-call audit row is written.

It is a **library, not a server**. No UI, no orchestration, no workflow engine,
no opinion about your framework — `pip install`, point it at a Postgres you
already run, call `engine.search()`. **BYO everything: your database, your
embedding provider, your LLM.** It never calls a hosted service you didn't
configure, and never implements auth, billing, or credential storage — it
exposes hooks (`on_usage`, `on_error`, `on_progress`) and injection points (`auth`,
`principals`) for your app to wire up.

On top of that it does the ordinary work well: structure-aware ingestion for
PDF/DOCX/PPTX/XLSX/HTML/images, hybrid retrieval (full-text + trigram + vector,
RRF-fused) with an optional graph leg, and a governed tool-execution path with
approval gates and audit.

> Not overclaiming: it does not prevent hallucination or guarantee grounding,
> and its `compute()` action (LLM-generated Python) is **off by default** —
> only enable it behind real OS-level isolation.

📖 **Full documentation:** https://promptev.ai/documentation/context-engine/

In this repo: **[access control](docs/access-control.md)** (the trichotomy,
enforcement, and the vector-index interaction) ·
**[the ACL-recall benchmark](benchmarks/acl_recall/)** ·
**[cookbook](cookbook/)** ·
**[TypeScript client](js/)** (`@promptev/context-engine`)

---

## Why you'd choose this

**1. Access control that survives the vector index.** Filtering an approximate
index with a `WHERE` clause is a *post-filter* — the index picks candidates
first, your ACL throws them away second, and the query returns short. We
measured it on 200k public documents at production embedding width, and found
something the literature misses: **because real permissions are topically
clustered rather than random, the loss is about twice what a conventional
benchmark reports** — recall 0.473 against 0.980 at identical visibility, a
quarter of queries empty, no error raised. Tuning does not fully close it, so
this library decides between an exact and an approximate scan per query
instead. [How access control works](docs/access-control.md) ·
[benchmark and method](benchmarks/acl_recall/).

Suspect your current stack? Prove it on your own data — one read-only
command, no embedding provider called:

```bash
context-engine check-acl-exposure --database-url postgresql://...
```

**2. Redaction that runs in the right place.** Not more PII detectors than
everyone else — better *placement*. Hits are masked **before** the reranker
POSTs them to a hosted provider and **before** the audit row is written, so
the masked value is the only one that ever leaves the process. Wiring a
redaction library into a pipeline by hand is exactly where that ordering goes
wrong. (Bring [Presidio](https://github.com/microsoft/presidio) for breadth —
its entities become ordinary rules.)

**3. It's a library, so the data never moves.** No server to run, no corpus to
hand over, no orchestration framework to adopt. If you're subject to rules
where *you* stay accountable for what your AI did — the EU AI Act's Article 12
logging duties fall on the deployer, not the vendor — then "runs in my
process, on my Postgres" is the answer you need, and a hosted service can't
give it to you.

**4. Governed tool execution in the same place as retrieval.** ACL-scoped
tools, encrypted credentials, a fail-closed approval gate with single-use
atomic resolution, and an audit row per call. Most stacks buy this separately
from a gateway.

<details>
<summary><b>How this compares to the alternatives</b></summary>

|  | Shape | Permissions |
|---|---|---|
| **context-engine** | Library, your Postgres | Enforced in the SQL predicate of every retrieval leg |
| pgai | Library, your Postgres | None — bring your own |
| R2R, RAGFlow | Server you deploy | Own user/collection model |
| Onyx, Glean | End-user search app | Synced from source-system ACLs, at query time |
| Azure AI Search | Managed cloud service | Query-time, tied to Entra |
| LlamaIndex, LangChain | Framework you build with | Your application's problem |

The closest thing to this package's *shape* is Timescale's `pgai` — same "a
library over the Postgres you already have" posture, without the governance.
The closest on *permissions* are Onyx and Glean, which are applications rather
than something you build on.

</details>

---

## When you outgrow the library

The library is the engine. [Promptev](https://promptev.ai?utm_source=github&utm_medium=readme)
is the platform it was extracted from. It is the same engine either way; what
differs is how much of the surrounding plumbing you run yourself.

**Your database, your model keys and your connectors stay your choice either
way** — run them yourself, or have Promptev run them. That is a deployment
decision, not a difference between the two:

| | Self-managed | Promptev-managed |
|---|---|---|
| Database | Your Postgres — your backups, tuning and upgrades | Hosted for you |
| Embeddings & LLM | Your keys, in your environment | Your keys still (BYOK), stored encrypted at rest and shared across agents |
| Connectors | You write the sync | Managed by Promptev, re-indexed on change |

What the platform adds on top of the engine:

| | The library | Promptev |
|---|---|---|
| Building agents over the data | A Python API | No-code builder, playground, deploys to web/Slack/WhatsApp |
| Approvals a human can operate | The state machine — the UI is yours | Approval gates, traces and usage dashboards included |
| RBAC and compliance posture | Hooks (`on_usage`, `on_error`, `on_progress`) | Included; SOC 2 Type II |

Self-hosting the engine is a first-class path and stays one — it is
Apache-2.0 and never calls home. The platform is for when the plumbing around
the engine becomes the job.

---

## Install

```bash
pip install "promptev-context-engine[postgres]"
# or with uv:
uv add "promptev-context-engine[postgres]"

# before it's on PyPI, install from git:
pip install "promptev-context-engine @ git+https://github.com/promptev/context-engine"
uv add "promptev-context-engine @ git+https://github.com/promptev/context-engine"
```

TypeScript lives in [`js/`](js/) as `@promptev/context-engine` — same Postgres
schema, so a corpus ingested by either client is searchable by the other:

```bash
npm install @promptev/context-engine pg
```

Needs **Node 22+**.

Needs **Python 3.12+** and **Postgres** with the `vector`, `pg_trgm`, and
`unaccent` extensions (auto-created by `context-engine migrate` if the role has
`CREATE EXTENSION`).

**The core engine is web-framework-agnostic** — it depends on no web framework.
Pick the router adapter that matches YOUR project (FastAPI, Flask, or Django)
via an extra, or use none and call `engine.*` directly.

Optional extras — install only what you use:

| Extra | Needed for |
|---|---|
| `postgres` | **the database driver — you almost certainly want this.** Separate from core because psycopg2 is LGPL and everything else is MIT/BSD/Apache, so a bare install pulls no copyleft. Any SQLAlchemy Postgres driver can be used instead, but this is the one that is tested (pg8000, measured: runs migrations and the vector and full-text legs, then fails the trigram leg — its paramstyle cannot express pg_trgm's `%` operator). |
| `fastapi` | `create_router()` — mount in a FastAPI app |
| `flask` | `create_flask_blueprint()` — register in a Flask app |
| `django` | `create_django_urlpatterns()` — include in Django `urls.py` |
| `graph` | `mode="graph"` (entity + community graph, Neo4j) |
| `vision` | scanned pages & images — transcribed by **whichever LLM you configure** |
| `gemini` | only if that LLM (or your embedder) is Gemini — covers both the `gemini` and `vertex_ai` providers |
| `ocr` | Tesseract OCR fallback |
| `mcp` | `create_mcp_app()` / `context-engine mcp` |
| `compute` | `engine.compute()` dataframe support |
| `presidio` | Microsoft Presidio entities as redaction detectors |
| `pdf` | structure-preserving PDF text extraction (tables stay tables) |

```bash
pip install "promptev-context-engine[fastapi,graph]"   # or [flask] / [django]
uv add "promptev-context-engine[fastapi,graph]"
```

### Teach your coding agent the access-control rules

Most integration code is now written with an agent's help, and the highest-cost
mistake in this library is one an agent makes readily: `principals=None` means
*trusted caller, no filtering*, so treating it as "nobody is logged in" returns
the entire corpus.

That spelling is being retired. `None` is falsy, so it is what every accessor
degrades to — `user.groups or None`, a missing key, `if user else None` — which
made the unauthenticated path the maximum-privilege path. Say it explicitly
instead:

```python
from context_engine import TRUSTED

await engine.search(q, principals=TRUSTED)      # skip ACL, deliberately
await engine.search(q, principals=[])           # anonymous
await engine.search(q, principals=user.groups)  # scoped
```

`None` still works and warns; it raises in 1.0. The package also ships a skill
stating this and the other invariants worth not guessing.

```bash
context-engine install-skill            # -> ./.claude/skills/context-engine/
context-engine install-skill --global   # -> ~/.claude/skills/
context-engine install-skill --print    # stdout, for agents that read other formats
```

It is an explicit command, never an install hook — the file becomes instructions
inside your agent, so it should be your decision. An edited skill is never
overwritten without `--force`.

## Configure

`ContextEngineConfig` is the one settings object (or build it from `CE_`-prefixed
env vars, nested with `__`, e.g. `CE_EMBEDDING__PROVIDER`).

```python
from context_engine import ContextEngineConfig, EmbeddingConfig, LLMConfig

config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", api_key="sk-..."),
    # llm is optional — only for structured extraction, compute, and the graph:
    llm=LLMConfig(provider="openai", model="gpt-4o-mini", api_key="sk-..."),
)
```

Embeddings: `openai` · `azure_openai` · `gemini` · `vertex_ai` · `voyage` ·
`cohere` · `custom` (any OpenAI-compatible `/embeddings` endpoint via
`base_url`). LLMs add `anthropic` · `bedrock`. (Anthropic has no embeddings
API — pick another provider for `EmbeddingConfig`.)

`vertex_ai` is the same Gemini models served from Vertex AI, and the only
difference is how you authenticate: Application Default Credentials against a
GCP project instead of an API key.

```python
llm=LLMConfig(provider="vertex_ai", model="gemini-2.5-pro",
              project="my-project", location="us-central1"),
```

`project` and `location` are optional — leave them out and the Google SDK
resolves them from `GOOGLE_CLOUD_PROJECT` / `GOOGLE_CLOUD_LOCATION`, so an
existing GCP deployment needs no new configuration. The Python SDK will also
discover the project from ADC when neither is set; **the TypeScript client's
SDK will not** — measured, not assumed — so for `@promptev/context-engine`
name the project in config or in the environment.

Credentials themselves come from ADC in every form:
`GOOGLE_APPLICATION_CREDENTIALS`, workload identity, or
`gcloud auth application-default login`. There is no credentials field to set.
`api_key` is ignored when a project is present (the SDK rejects the pair
outright); on its own it selects Vertex express mode.

`StorageConfig` holds the deployment knobs: `ann_exact_threshold` (the vector
leg's exact/approximate crossover — hardware- and dimension-specific, see
[docs/access-control.md](docs/access-control.md)) and the connection pool,
passed through to SQLAlchemy — `pool_size` (5), `max_overflow` (10),
`pool_timeout` (30s), `pool_recycle` (1800s, keep it under the idle timeout of
any pooler in front of the database) and `pool_pre_ping` (on: without it every
connection left stale by a database restart or failover fails one query
first). All reachable as env vars, e.g. `CE_STORAGE__POOL_SIZE`.

## Provision the database

```bash
context-engine migrate --database-url postgresql://user:pass@localhost:5432/mydb --dim 1536
# add --graph to also create entity/relationship/community tables
```

`--dim` must match your embedding model's width (1536 for
`text-embedding-3-small`) and is locked in on first migrate.

## Ingest

```python
from context_engine import ContextEngine

engine = ContextEngine(config)

# Plain text
report = await engine.ingest(
    text="Annual leave accrues at two days per month for every band-three employee.",
    name="leave-policy",
    source_id="hr-handbook",       # a namespace to scope search/stats
    acl=["group:hr"],              # omit for an unrestricted document (visible to everyone)
)

# A file — PDF/DOCX/PPTX/XLSX/HTML/EML/images/plain text, auto-detected
with open("policy.pdf", "rb") as f:
    report = await engine.ingest(file=f, name="policy.pdf", source_id="hr-handbook")

print(report.totals)               # {"files": 1, "failed": 0, "units": 3, ...}
print(report.documents[0].status)  # "completed" | "failed" | "batch_pending"
```

Construction is cheap and lazy — no connection is opened until the first call
that needs one. Release the pool (and the embedding client, and the Neo4j
driver) with `await engine.aclose()`, or use the engine as an async context
manager; a process-lifetime singleton can skip it, anything building engines
per job or per test should not.

- `mode="hybrid"` (default) or `mode="graph"` (needs the `[graph]` extra).
- Scanned pages / images are transcribed to **structured Markdown** (tables and
  headings preserved) when `vision_llm` is configured — otherwise embedded text,
  with Tesseract as a fallback if `[ocr]` is installed. **Vision is
  provider-agnostic**: it goes through the same `call_llm` seam as everything
  else, so Claude, GPT-4o, Gemini and Bedrock all work — whatever your
  `LLMConfig` names. The `[vision]` extra only adds page rasterization.
- Born-digital PDFs keep their structure too, with the `[pdf]` extra: the text
  layer is converted to Markdown locally — no OCR, no LLM, no network — so a
  table survives as a table into the chunker instead of being flattened into a
  run of words. Without the extra you get today's plain-text extraction.
- `ingest()` never raises on a bad document — check `report.documents[i].status`.

## Search

```python
result = await engine.search(
    "how much annual leave do I get",
    source_ids=["hr-handbook"],
    document_ids=None,         # optionally pin to specific documents WITHIN the sources
    principals=["group:hr"],   # TRUSTED = trusted/internal (skips ACL); [] = anonymous
    top_k=10,
)

for hit in result.hits:
    print(hit.document_name, hit.score, hit.chunk_text[:80])
```

Fuses full-text + trigram + vector search with Reciprocal Rank Fusion;
`mode="graph"` adds a graph leg. `compress_to_tokens=N` trims weak hits to fit a
prompt budget.

## Compute over tables

`engine.compute()` turns the in-scope CSV/XLSX documents into dataframes and has
the LLM write code against them. `compute_over_frames()` is the same path for
frames you already hold — an uploaded workbook, a connector's sheet, a query
result — with no document to point at:

```python
from context_engine import compute_over_frames

out = await compute_over_frames(
    {"expenses": df},              # {sheet name: DataFrame}
    "total the amount column",
    config=engine.config,
)
out["result"]
```

Both are **off by default** (`enable_code_execution=True` turns them on — only
behind real OS-level isolation) and both apply `config.redaction` before the LLM
sees anything: every string cell *and* every column header is masked, and the
returned dict — generated code included — is swept on the way out.

## Serve

The same document/search/stats surface ships as a ready adapter for **FastAPI,
Flask, or Django** — pick the one matching your project. All three expose
identical routes (`POST/GET/DELETE/PATCH /documents`, `POST /search`,
`GET /stats`) with identical behavior (they share one framework-neutral core).
`auth` and `principals` are **required** arguments — the package implements no
auth, and resolving who the caller is has no safe default. To run a mount
open on purpose, say so: `principals=lambda: []` (anonymous — sees only
documents with no ACL). Returning `TRUSTED` (import it from `context_engine`)
means a fully trusted caller and skips ACL filtering entirely; returning
`None` is rejected with a 500 so a missing resolver can never open a mount by
accident.

A caller may only file a document under an ACL it already holds; `POST
/documents` and `PATCH /documents/{id}` answer `403` otherwise, so an
untrusted request can't push content into another group's retrieval scope.

**FastAPI** (`[fastapi]` extra):

```python
from context_engine import create_router

router = create_router(engine, auth=my_auth, principals=my_principals)
app.include_router(router, prefix="/context")
```

**Flask** (`[flask]` extra) — `auth(request)` / `principals(request)`, async views:

```python
from context_engine import create_flask_blueprint

bp = create_flask_blueprint(engine, auth=my_auth, principals=my_principals)
app.register_blueprint(bp, url_prefix="/context")
```

**Django** (`[django]` extra) — include the urlpatterns in your `urls.py`:

```python
from django.urls import include, path
from context_engine import create_django_urlpatterns

urlpatterns = [
    path("context/", include(create_django_urlpatterns(engine, auth=my_auth, principals=my_principals))),
]
```

Not on any of them? Just call `engine.ingest()` / `engine.search()` directly
from your own views — the engine needs no web framework at all.

Or serve the same engine as an **MCP** tool surface (`[mcp]` extra):

```bash
context-engine mcp --database-url postgresql://... --port 8080
```
```python
from context_engine import create_mcp_app, UNSCOPED
app.mount("/mcp", create_mcp_app(engine, principals=my_principals, scope=UNSCOPED))
```

`scope` is the ceiling of source ids this mounted tool may ever reach, and is
REQUIRED. It is resolved per call like `principals` and is never a tool
argument: a model can ask for any source id it likes, and a tool that believed
it would let one caller read another's documents. Map your own word onto it —
a project, a matter, a customer — and pass `scope=lambda: ["src-a", "src-b"]`,
or `Scope(source_ids=(...), document_ids=(...))` to hand an agent specific
documents. A host with one shared corpus writes `scope=UNSCOPED` on purpose.
The caller can narrow within that ceiling and never widen past it.

Registers ONE knowledge tool, `search_knowledge_base`, whose `action` picks
what it does:

| Action | What it is for |
|---|---|
| `discover` | what documents there are and what is inside each one — sheets and columns, sections and pages, or JSON keys — plus the extracted fields by document type, a census of the corpus, and what each action can do. Call it first |
| `search` | passages by meaning or keywords |
| `get_doc` / `get_docs` | one whole document by id, or several at once |
| `get_chunks` | walk one long document in order, a piece at a time |
| `list` | browse the documents without searching |
| `query_meta` | filter documents by their structured fields |
| `compute` | a figure derived from the spreadsheets, over every row |
| `map_reduce` | the same question asked of every document in scope |
| `get_neighbors` / `traverse` | what is one step, or a few steps, from a named thing |
| `find_related` | connections of a kind across the corpus |
| `community_summary` | the themes the corpus groups into |

Combining the tools into sub-actions of one is better for the model that calls
them: one description carries the decision rules once, and there is one name
to route through.

`discover` answers with the SCHEMA, not an inventory, so one call is enough to
write one correct `compute` or `query_meta` call instead of three exploratory
ones. Every listed document carries `document_type`, `mode`, and a `structure`
whose shape follows the type — `sheets` (name, column headers, row count) for
a workbook, `sections` and `last_page` for a document with headings, `keys`
for JSON, and `chunks` as the floor for anything else. Beside them,
`fields_by_type` lists the extracted structured field names with each field's
data type and how many in-scope documents carry it, and `document_types` is a
census of the corpus. Both are keyed by `(kind, type)`: `kind` comes from the
mime and is always known, while `document_type` is free text an LLM wrote and
exists only where `extract_structured` was enabled — so a PDF invoice and a
spreadsheet of invoice rows stay separate rather than merging under one label.

Lists inside a `structure` are capped, with the remainder reported as
`more_columns` / `more_sections` / `more_keys` / `more_sheets`, because
`discover` is the first call of a conversation and an unbounded list lands in
the model's context before it has asked anything. **The cap is for the call
that did not name its documents:** `discover` and `list` take `document_ids`,
and a `discover` scoped to specific documents answers their structure whole
and narrows the field grouping and the census to them too. While the remainder
is small enough to be worth fetching, the truncated structure carries that
call already filled in; past that it says what to do instead, because a sheet
with hundreds of columns is one to compute over rather than read back.

Omit `mode` on a search and it is worked out from the documents in scope — the
graph leg is added when the deployment has a graph and something in scope was
ingested that way, and it runs beside the other legs rather than ahead of
them. Naming `"hybrid"` or `"graph"` forces one.

`query_meta` and `map_reduce` need an LLM, `compute` needs code execution
explicitly enabled, and the graph actions need a configured graph — `discover`
marks them off otherwise, and an off action called anyway answers a plain
`{"success": false, "error": …}` result instead of hiding. `discover` and
`list` page (`has_more` + `next_cursor`, sent back as `cursor`), and a
truncated answer carries the `next_page` call that reads the rest. The
connector-tool gateway (`search_tools` + `execute_tool`) is registered beside
it.

`knowledge_tool_definition()` exports the tool as data — name, description and
input schema as JSON Schema — so a host wiring it into its own agent loop never
hand-writes the schema. `create_mcp_app` also takes `redaction` (a policy
applied per call), and `compute` / `map_reduce` callables that REPLACE the
built-in ones, which is where a host applies its own rules for permission,
billing and approval.

The same tool is a plain method, so an application driving its own agent loop
needs no MCP at all — and cannot get a different answer, because both doors
call one implementation:

```python
answer = await engine.search_knowledge_base(
    action="search", query="annual leave", principals=caller, scope=["hr"]
)
```

## Tools

Let an LLM/agent call `http` / `db` / `mcp` (external server) and `function`
(a Python callable) tools through one governed path — ACL-checked, config
AES-256-GCM-encrypted at rest, every call audited.

```python
from context_engine import ToolConfig

# HTTP tool — the LLM fills `city` at call time
await engine.register_tool(ToolConfig(
    name="get_weather", kind="http",
    description="Fetch the current weather for a city",
    config={
        "method": "GET",
        "url": "https://api.example.com/weather",
        "headers": {"Authorization": "Bearer super-secret-token"},
        "llmQueryParameters": {"properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
))  # -> call_name "http_get_weather"

# Function tool — no config, no persistence
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b
engine.register_function_tool(add)  # -> call_name "fn_add"

out = await engine.execute_tool("http_get_weather", {"city": "Lahore"})
# -> {"result": {"status_code": 200, "data": {...}}, "usage": {"kind": "tool", "units": 1}}
```

**Approval gate** — a tool with `requires_approval=True` returns an
`approval_required` payload instead of running, until resolved:

```python
from context_engine import resolve_approval

out = await engine.execute_tool(
    "http_get_weather", {"city": "Lahore"},
    principals=["user:a"],
    approval_scope="run-42",   # opaque, chosen by you — a run id is the usual shape
)
# -> {"approval_required": {"approval_id": "...", "expires_at": "...", ...}}   (not executed yet)
await resolve_approval(engine, approval_id, "approved", approver="boss")
# the same call, same approval_scope, runs for real — exactly once
```

`approval_scope` is the claim scope: an approved record is consumed only by
the same call in the same scope (and, when the record carries principals, by a
caller who overlaps them). Within a scope, identical pending requests share one
record, so a retrying agent gets the same `approval_id` back. Resolve the scope
server-side — on the routers and the MCP gateway it is a dependency beside
`principals`, never a request field. Calling without it still works this
release, unscoped and with a `DeprecationWarning`; the next release refuses a
gated call with no scope.

Getting the request in front of a human (Slack, in-app) is the consumer's job —
the engine owns only the single-use, atomic approval state machine. Park until
`expires_at`; after that the record can no longer be approved.

## Redaction

Span-level masking of PII (or anything you can describe with a regex or a
function) as content is ingested, retrieved, or returned. It **composes with
ACL** (ACL = *who* may see a document; redaction = *what's masked* inside it)
and is a **pure transform** — no model, no network. The default policy is a
guaranteed no-op.

```python
from context_engine import ContextEngineConfig, RedactionPolicy, RedactionRule

config = ContextEngineConfig(
    database_url="postgresql://...",
    embedding=...,
    redaction=RedactionPolicy(rules=[
        RedactionRule(name="emails", detector="email"),                  # -> [EMAILS]
        RedactionRule(name="order_id", pattern=r"\bORD-\d{6}\b"),         # -> [ORDER_ID]
    ]),
)
```

Built-in detectors: `email` · `phone` · `ssn` · `credit_card` · `iban` ·
`api_key`. Actions: `mask` · `hash` (keyed, joinable pseudonym — needs
`secret_key`) · `remove`. `apply_at`: `output` (default, masks on every read,
can be principal-conditional via `unless`) · `ingest` (scrub before storage) ·
`both`.

**The detectors are deliberately a small set — the placement is the point.**
Redaction runs *inside* retrieval: hits are masked before `_maybe_rerank` POSTs
their text to Cohere/Voyage/Jina, and tool results are masked before the audit
row is written, so the masked value is the only one that leaves the process or
lands in retention. That ordering is what usually goes wrong when a redaction
library is bolted onto a pipeline by hand.

For broader detection, bring [Microsoft Presidio](https://github.com/microsoft/presidio)
(`[presidio]` extra) — its entities become ordinary rules and inherit the same
placement guarantees:

```python
from context_engine import presidio_detectors

RedactionPolicy(
    rules=[RedactionRule(name="person", detector="PERSON")],
    custom_detectors=presidio_detectors(["PERSON", "LOCATION"]),
)
```

Any `Callable[[str], list[tuple[int, int]]]` works as a detector, so an
in-house classifier drops in the same way.

---

## Benchmarks

**Does ACL filtering cost you retrieval quality?** Yes — badly — unless the
vector leg is tuned for it, and the failure is silent.

Any system that applies an access-control predicate as a `WHERE` clause on top
of an approximate vector index is doing a *post-filter*: the index walk is
ordered by distance to the query, rows the caller may not see are discarded as
they are met, and the walk stops when its candidate budget runs out. If the
caller can see only a slice of the corpus, that budget can be spent entirely
on rows they aren't allowed to have.

And the size of the visible slice is not the variable that matters most.
**Every ACL benchmark we know of assigns permissions at random. Real
permissions are clustered** — `group:hr` covers HR documents, and those are
semantically adjacent by construction, so a group's slice is a *region* of the
embedding space rather than a uniform sample.

200k documents from BEIR `nq`, real 1536-dim embeddings, exact brute-force
oracle over eligible rows. Same corpus, same index, same queries, same number
of visible rows — only the arrangement differs:

| permissions | visible | post-filter recall | empty | with iterative scan |
|---|---:|---:|---:|---:|
| random | 10% | 0.980 | 0.0% | 0.980 |
| **clustered** | 10% | **0.473** | **25.0%** | 0.928 |
| random | 1% | 0.175 | 20.8% | 0.966 |
| **clustered** | 1% | **0.087** | **75.0%** | 0.883 |

Clustering roughly doubles the recall loss. HNSW descends greedily from an
entry point toward the query: scattered eligible rows are met along the walk,
concentrated ones are not, so a query pointing away from the visible region
spends its whole budget on rows the caller may not see.

**Iterative scan does not fully close it** — 0.883 clustered against 0.966
random, same setting. That residual is why this library *decides* between an
exact and an approximate scan per query rather than tuning a parameter and
hoping.

**You may not be able to reproduce the failure, and that is also a result.**
With a GIN index on `acl`, all twelve default-planner rows chose an exact
`bitmap+sort` instead of the vector index — recall 1.000, no failure, at
44–107 ms p50. Which plan Postgres picks is a cost estimate that moves with
dimension, corpus size and statistics freshness, and `&&` selectivity
estimation on GIN arrays is unreliable enough to flip it without warning. The
table above therefore forces the index. Whether *you* are exposed is decided
per query, by a cost model — which is the actual argument for deciding it
yourself.

**Measure it on your own corpus** rather than trusting ours — offline,
read-only, no embedding provider called:

```bash
context-engine check-acl-exposure --database-url postgresql://...
```

It reports, for the ACL values that actually occur in your data, what a bare
scoped query returns versus what this library returns, both scored against an
exact oracle computed inside Postgres.

Written up in full: **[Your ACL Benchmark Is Measuring the Easy Case](https://promptev.ai/acl-benchmarks-measure-the-easy-case/)**.
Method, how `clustered` is constructed, the limitations — including a metric
that behaved as a control rather than a result, and a retracted earlier table
— are in [`benchmarks/acl_recall/`](benchmarks/acl_recall/), along with the
harness and raw CSVs. Requires pgvector ≥ 0.8.

### Checking your database

Managed Postgres varies in ways that matter here. To check a provider before
committing to it:

```bash
CE_COMPAT_DATABASE_URL='postgresql+psycopg2://user:pass@host/db' \
    uv run pytest tests/test_managed_postgres_compat.py -v -s
```

Point it at a **scratch database** — it runs the real migrations. It verifies
pgvector is new enough for iterative scan, that `pg_trgm`/`unaccent` are
installable, and — the one that catches people — that their functions are
actually reachable on the role's `search_path`. Supabase installs extensions
into an `extensions` schema, so `CREATE EXTENSION` can succeed while
`similarity()` and `%` stay unresolvable and the trigram leg breaks at query
time rather than at migrate time.

## Deep dives

This README covers the basics. Full documentation is at
**https://promptev.ai/documentation/context-engine/**. The exhaustive design
notes — redaction apply points, `hash` joinability, overlapping-rule merge
semantics, ACL edge cases, graph internals, cost model, and known limitations —
also live in the module and config docstrings (`config.py`, `redaction.py`,
`engine.py`).
