Metadata-Version: 2.5
Name: promptev-context-engine
Version: 1.0.1
Summary: Promptev Context Engine: language-agnostic ingestion + hybrid retrieval
Project-URL: Homepage, https://promptev.ai/documentation/context-engine/
Project-URL: Documentation, https://promptev.ai/documentation/context-engine/
Project-URL: Repository, https://github.com/promptev/context-engine
Project-URL: Issues, https://github.com/promptev/context-engine/issues
Author: Promptev Inc
License: Apache-2.0
License-File: LICENSE.md
License-File: NOTICE
Keywords: embeddings,graph-rag,hybrid-search,llm,mcp,pii-redaction,rag,retrieval,vector-search
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: alembic>=1.13
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: cryptography>=42
Requires-Dist: httpx>=0.27
Requires-Dist: json-repair>=0.60
Requires-Dist: jsonschema>=4.20
Requires-Dist: lingua-language-detector>=2.1
Requires-Dist: openpyxl>=3.1
Requires-Dist: pgvector>=0.4
Requires-Dist: pydantic-settings>=2.4
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=5.0
Requires-Dist: pypdfium2>=4.30
Requires-Dist: python-docx>=1.1
Requires-Dist: python-pptx>=1.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tiktoken>=0.7
Provides-Extra: compute
Requires-Dist: pandas>=2.0; extra == 'compute'
Provides-Extra: django
Requires-Dist: django>=4.2; extra == 'django'
Provides-Extra: fastapi
Requires-Dist: fastapi>=0.115; extra == 'fastapi'
Requires-Dist: python-multipart>=0.0.32; extra == 'fastapi'
Provides-Extra: flask
Requires-Dist: flask[async]>=3.0; extra == 'flask'
Provides-Extra: graph
Requires-Dist: neo4j>=5.20; extra == 'graph'
Requires-Dist: networkx>=3.2; extra == 'graph'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.2; extra == 'mcp'
Provides-Extra: ocr
Requires-Dist: pillow>=10; extra == 'ocr'
Requires-Dist: pytesseract>=0.3; extra == 'ocr'
Provides-Extra: pdf
Requires-Dist: pdf-inspector>=0.2.6; extra == 'pdf'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgres'
Provides-Extra: presidio
Requires-Dist: presidio-analyzer>=2.2; extra == 'presidio'
Provides-Extra: reranker
Provides-Extra: vision
Requires-Dist: pillow>=10; extra == 'vision'
Description-Content-Type: text/markdown

<!-- Absolute URL, not a relative path: this README is also the PyPI
     long-description and PyPI does not resolve relative image paths.
     A copy of the mark lives at assets/promptev-logo.png. -->
<p align="center">
  <img src="https://promptev.ai/favicon.png" alt="Promptev" width="88" height="88">
</p>

<h1 align="center">context-engine</h1>

<p align="center">
  <b>Retrieval that enforces access control and PII redaction inside the query,<br>
  as a library, on your own Postgres, with your own models.</b>
</p>

<p align="center">
  <a href="https://pypi.org/project/promptev-context-engine/"><img src="https://img.shields.io/pypi/v/promptev-context-engine?label=PyPI&color=3775A9" alt="PyPI"></a>
  <a href="https://www.npmjs.com/package/@promptev/context-engine"><img src="https://img.shields.io/npm/v/@promptev/context-engine?label=npm&color=CB3837" alt="npm"></a>
  <a href="https://github.com/promptev/context-engine/actions/workflows/publish.yml"><img src="https://github.com/promptev/context-engine/actions/workflows/publish.yml/badge.svg" alt="CI"></a>
  <img src="https://img.shields.io/badge/python-3.12%2B-3776AB" alt="Python 3.12+">
  <a href="https://github.com/promptev/context-engine/blob/main/LICENSE.md"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="Apache-2.0"></a>
</p>

> Built by [Promptev](https://promptev.ai?utm_source=github&utm_medium=readme).
> Apache-2.0. A TypeScript twin, [`@promptev/context-engine`](https://github.com/promptev/context-engine/tree/main/js), shares the
> same Postgres schema.
>
> Self-host it, or try it in the cloud without hosting anything: bring your own
> database and model keys and [explore promptev cloud](https://app.promptev.ai?utm_source=github&utm_medium=readme&utm_campaign=cloud).

`context-engine` is a Python library for applications that answer questions
over documents and act through tools, and that have to say who may see what.
It ingests files into the Postgres you already run, retrieves with full-text,
trigram and vector legs fused together (a graph leg optionally), and puts two
things where other stacks leave them to application code: the access-control
list goes into the SQL predicate every retrieval leg shares, so a document a
caller may not see is never fetched, never ranked and never reaches a
reranker; and PII redaction runs before a hit is sent to a reranker, before a
tool result is written to its audit row, and (if you ask) before a chunk is
embedded.

The models are yours. Hand the engine one function and every model call it
makes (extraction, vision transcription, graph building, map-reduce, compute)
goes through your own harness, with your keys, retries, limits and cost
accounting. A small OpenAI-compatible client is built in for a quick start.

It is a library, not a server: no UI, no orchestration, no opinion about your
framework. It never calls a service you did not configure and implements no
auth, billing or credential storage of its own; it exposes hooks (`on_usage`,
`on_error`, `on_progress`) and injection points (`auth`, `principals`,
`scope`) for your application to wire up. It does not prevent hallucination
or guarantee grounding, and its `compute()` action (model-written Python) is
off by default.

Who it is for: a team building a governed assistant, a search or extraction
service, or an agent with tools, on their own database, subject to rules
under which they stay accountable for what the system did.

**Full documentation:** https://promptev.ai/documentation/context-engine/ ·
in this repo: [access control](https://github.com/promptev/context-engine/blob/main/docs/access-control.md) ·
[the ACL-recall benchmark](https://github.com/promptev/context-engine/tree/main/benchmarks/acl_recall) · [the retrieval benchmark](https://github.com/promptev/context-engine/tree/main/benchmarks/retrieval) · [cookbook](https://github.com/promptev/context-engine/tree/main/cookbook) ·
[model adapters](https://github.com/promptev/context-engine/tree/main/examples/model_adapters) · [TypeScript](https://github.com/promptev/context-engine/tree/main/js)

---

## Install

```bash
pip install "promptev-context-engine[postgres]"
# or with uv:
uv add "promptev-context-engine[postgres]"
```

Needs **Python 3.12+** and **Postgres** with the `vector` (pgvector 0.8+),
`pg_trgm` and `unaccent` extensions, created by `context-engine migrate` when
the role may `CREATE EXTENSION`. No provider SDK is installed: the built-in
client is plain HTTP, and a host model brings its own.

The core engine depends on no web framework. Install only the extras you use:

| Extra | Needed for |
|---|---|
| `postgres` | **the database driver; you almost certainly want this.** Separate from core because psycopg2 is LGPL and everything else is MIT/BSD/Apache, so a bare install pulls no copyleft. It is the tested driver (pg8000, measured: runs migrations and the vector and full-text legs, then fails the trigram leg, since its paramstyle cannot express pg_trgm's `%` operator). |
| `fastapi` | `create_router()`, mounted in a FastAPI app |
| `flask` | `create_flask_blueprint()`, registered in a Flask app |
| `django` | `create_django_urlpatterns()`, included in Django `urls.py` |
| `graph` | `mode="graph"` (entity and community graph; Neo4j optional) |
| `vision` | page rasterization for scanned pages and images, transcribed by whichever model you bring |
| `ocr` | Tesseract OCR fallback |
| `mcp` | `create_mcp_app()` / `context-engine mcp` |
| `compute` | `engine.compute()` dataframe support |
| `presidio` | Microsoft Presidio entities as redaction detectors |
| `pdf` | structure-preserving PDF text extraction (tables stay tables) |

```bash
pip install "promptev-context-engine[postgres,fastapi,graph]"   # or [flask] / [django]
```

Upgrading an existing deployment: [MIGRATING.md](https://github.com/promptev/context-engine/blob/main/MIGRATING.md).

## Quick start

**Migrate the database first.** The engine needs a Postgres with pgvector and
its own tables. To try it locally, start one and migrate it:

```bash
docker run -d --name context-engine-pg -p 127.0.0.1:5432:5432 \
  -e POSTGRES_USER=user -e POSTGRES_PASSWORD=pass -e POSTGRES_DB=mydb \
  pgvector/pgvector:pg16
until docker exec context-engine-pg pg_isready -h localhost -U user -d mydb; do sleep 1; done
context-engine migrate --database-url postgresql://user:pass@localhost:5432/mydb --dim 1536
```

The second line waits until Postgres accepts connections: `docker run -d`
returns before it does. The port is published on loopback only. If port
5432 is already taken, publish another one (`-p 127.0.0.1:5433:5432`) and
use it in the URL (`localhost:5433`).

`migrate` creates the extensions and the tables, and is safe to run again
(see [Provision the database](#provision-the-database)). On a database that
was never migrated, the first call raises `DatabaseNotMigrated`, and its
message is the command to run:

```text
context_engine.errors.DatabaseNotMigrated: This database has no Context Engine tables, or this role cannot see them. Run the migration once, then try again: context-engine migrate --database-url <your database URL> --dim 1536
```

`DatabaseNotMigrated` is a `RuntimeError`; the HTTP routers answer it with 503
and the MCP tools with one fixed sentence.

A plain `postgresql://` or `postgres://` URL uses psycopg2, the driver the
`[postgres]` extra installs, on any SQLAlchemy 2.x. Name another driver in the
URL (`postgresql+psycopg://`) to use it.

### Bring your own model (recommended)

Run your own model stack (keys, routing, rate limits, tracing, cost) and hand
the engine three functions. Every model call, every embedding request and
every rerank then goes through them. This is complete and runs as written;
the three stubs are what you replace with calls on your own provider (or copy
a ready adapter from [`examples/model_adapters/`](https://github.com/promptev/context-engine/tree/main/examples/model_adapters)):

```python
import asyncio

from context_engine import ContextEngine, ContextEngineConfig, EmbeddingConfig, ModelReply, ModelRequest


async def my_model(request: ModelRequest) -> ModelReply:
    # request.purpose is one of structured_extraction, entity_extraction,
    # community_summaries, vision_pages, vision_image, map_reduce, compute.
    # request.system / request.user are the prompt; request.images are PNG
    # bytes for the vision purposes; request.response_schema, when set, is a
    # strict JSON Schema you can hand to a provider that enforces one; and
    # request.max_tokens / temperature / thinking_budget are the engine's asks.
    # Stub: "{}" does not satisfy request.response_schema, so the structured
    # purposes fail until this returns your provider's real reply.
    text = "{}" if request.response_schema else ""
    return ModelReply(
        text=text,
        tokens={"input": 0, "output": 0},   # what your provider billed
        mode="prompt",                       # "native" only when it enforced request.response_schema
    )


async def my_embedder(texts: list[str], *, kind: str) -> tuple[list[list[float]], int]:
    # kind is "document" or "query". One vector per text, config.embedding.dim
    # wide, and the tokens billed. Stub: a unit vector, never all zeros
    # (cosine distance to a zero vector is undefined and breaks search).
    return [[1.0] + [0.0] * 1535 for _ in texts], 0


async def my_reranker(query: str, docs: list[str]) -> list[int]:
    return list(range(len(docs)))   # indices into docs, most relevant first


config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    # Required with a host embedder too: the identity the database records
    # (model name, dimension) and the batch limits. No api_key needed.
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", dim=1536),
)
engine = ContextEngine(config, model=my_model, embedder=my_embedder, reranker=my_reranker)


async def main() -> None:
    report = await engine.ingest(
        text="Annual leave accrues at two days per month.",
        name="leave-policy",
        source_id="hr-handbook",
        acl=["group:hr"],
    )
    print(report.documents[0].status)          # "completed"
    result = await engine.search("how much annual leave", principals=["group:hr"])
    print([hit.chunk_text for hit in result.hits])
    await engine.aclose()


asyncio.run(main())
```

The engine keeps the prompts, the JSON Schemas, reply validation with one
repair retry, redaction before anything leaves, and metering. `mode="native"`
promises your provider enforced the schema; leave it at the default `"prompt"`
when it did not, and the engine validates and repairs. With a `model`,
`config.llm`, `config.vision_llm` and `graph.extraction_llm` are optional, and
vision is on. A model that cannot read images should raise for `vision_*`;
those pages end as `vision_failed`. A refusal goes in `ModelReply.refusal`,
never in `text`, so it is never stored as a document's data.

`embedder` is a function `(texts, *, kind) -> (vectors, tokens)`, or any
object with an `embed` method of that shape; the dimension check runs on every
reply. Vectors and indices may be numpy arrays (anything with `tolist()`), as a
local model returns them. `reranker` is `(query, docs) -> list[int]`; it runs on the widened
candidate window (`config.reranker.candidates`, 50), and the engine drops
out-of-range and duplicate indices and appends any it omitted, so it can
reorder hits and never drop them. Each of the three may be sync or async: a
sync one runs in a worker thread.

**Your harness owns the call.** The engine builds each request (prompt,
schema, images, redaction) and checks the reply. Everything about making the
call is yours: retries, timeouts, rate limits, concurrency, caching, cost. The
engine does not retry a host call (it repairs a reply that breaks the schema
once, and re-asks for pages a vision reply omitted) and does not time it out;
whatever the host raises is final for that call site. So you can size your
limits: the engine sends up to 8 vision calls per document, 5 `map_reduce`
calls by default (20 at most) and 4 embedding batches at once, and concurrent
ingests add up. Which model writes `compute` code, most specific first: a
per-call `model=` on `compute()` / `compute_over_frames()`, then a per-call
`model_cfg=`, then the engine's `model`, then `config.llm`. A call you hand
to a provider batch API can raise `ModelDeferred` instead of blocking; see
[Batch mode](#batch-mode-deferred-model-calls).

### Or use the built-in default

Pass no `model` or `embedder` and the engine calls the endpoints named by
`config.llm` and `config.embedding` itself, through one OpenAI-compatible
client over plain HTTP (`POST {base_url}/chat/completions` and
`POST {base_url}/embeddings`):

```python
from context_engine import ContextEngine, ContextEngineConfig, EmbeddingConfig, LLMConfig

config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", api_key="sk-..."),
    # The model for structured extraction, compute, map_reduce and query_meta.
    # Optional, and unused when the engine is given a host model:
    llm=LLMConfig(provider="openai", model="gpt-4o-mini", api_key="sk-..."),
    # Optional: the model that transcribes scanned pages and images.
    vision_llm=LLMConfig(provider="openai", model="gpt-4o-mini", api_key="sk-..."),
)
engine = ContextEngine(config)
```

`provider` is one of three:

| provider | endpoint | auth |
|---|---|---|
| `openai` | `https://api.openai.com/v1` unless `base_url` says otherwise | `Authorization: Bearer` |
| `azure_openai` | `base_url` required: the Azure v1 path, `https://<resource>.openai.azure.com/openai/v1/` | `api-key` header |
| `custom` | `base_url` required: any OpenAI-compatible root | `Authorization: Bearer` when `api_key` is set |

`provider` names the wire format, not the vendor: `custom` with the endpoint
root as `base_url` reaches Ollama, vLLM, Groq, Mistral, a LiteLLM proxy and
anything else that speaks the OpenAI chat and embeddings format. The
per-vendor table is under [Use any model provider](#use-any-model-provider).

The client retries 408, 409, 429 and 5xx with full-jitter backoff up to
`max_retries` (2), honouring `Retry-After`; any other 4xx is final. On a
search query's embedding the wait is capped at
`search.query_embed_retry_after_cap_s` (5 seconds) with
`search.query_embed_max_retries` (1), so a rate-limited key cannot hold a
search for minutes. A successful reply that carries no choice, or an
embeddings reply that does not hold exactly one vector per input with each
index used once, raises `OpenAICompatReplyError` rather than passing as an
empty answer. It never
follows a redirect: a 3xx from a provider endpoint would carry the key and the
prompt wherever `Location` points, so it fails the call with that status. It
claims an enforced (`native`) JSON schema only on OpenAI proper and Azure;
every other endpoint is reported as `prompt` and validated, because accepting
`response_format` is not enforcing it. Set `schema_enforced=True` on an
`LLMConfig` for an endpoint that does enforce it (vLLM with guided decoding,
say), or `False` to validate everywhere. `thinking_budget` is ignored by this
client, since thinking control is provider-specific; a host model sees it.

Both paths can run on a transport you supply instead of one the engine
builds: `ContextEngine(config, http_client_factory=lambda purpose: ...,
mcp_resolver=...)`. The factory is asked for an `httpx.AsyncClient` per
purpose (`llm`, `embeddings`, `reranker`, `tool_http`, `mcp_oauth`;
`context_engine.HttpClientFactory` / `HttpClientPurpose`); returning `None`
keeps the engine's. It must be long-lived and host-owned, since the engine
never closes it. A host `model`, `embedder` or `reranker` never calls it.

## Use any model provider

The engine works with any model provider. There are two ways in, and one rule:

- **`model=` / `embedder=` / `reranker=`** (recommended): any SDK, any vendor,
  any local model, your own gateway. You keep the provider's native schema
  enforcement, prompt caching, thinking control, and your own retries, routing
  and tracing. The engine never sees a key.
- **The built-in OpenAI-compatible client**: `provider="openai"` and
  `"azure_openai"` only fill in the URL and the auth header; `provider="custom"`
  plus `base_url` reaches anything that speaks the OpenAI chat/embeddings
  format.

The rule: `provider` names the wire format, not the vendor. `provider="anthropic"`,
`"gemini"`, `"vertex_ai"` or `"bedrock"` is refused at config time, on purpose,
with a message pointing at the two ways above; a vendor URL table inside the
engine would go stale, and `model=` gives you the vendor's real SDK instead.

| Vendor or runtime | Recommended path | Built-in alternative (`custom` + `base_url`) |
|---|---|---|
| Anthropic | `model=` with [`anthropic_native.py`](https://github.com/promptev/context-engine/blob/main/examples/model_adapters/anthropic_native.py): native structured output and thinking control | `https://api.anthropic.com/v1/`, for trying it out only: Anthropic does not support it for production and it ignores `response_format`. No embeddings endpoint; pair it with another `EmbeddingConfig` |
| OpenAI | `provider="openai"` (`https://api.openai.com/v1`, `native` schemas), or `model=` with the OpenAI SDK | same |
| Azure OpenAI | `provider="azure_openai"`, `base_url="https://<resource>.openai.azure.com/openai/v1/"`, `native` schemas | same |
| Google Gemini | `model=` with [`gemini_native.py`](https://github.com/promptev/context-engine/blob/main/examples/model_adapters/gemini_native.py) | `https://generativelanguage.googleapis.com/v1beta/openai/` (beta per Google; JSON-schema output, images and embeddings work) |
| Vertex AI | `model=` with [`gemini_native.py`](https://github.com/promptev/context-engine/blob/main/examples/model_adapters/gemini_native.py) (Application Default Credentials) | none listed here; see Google's docs on calling Vertex AI with OpenAI libraries |
| AWS Bedrock | `model=` with [`bedrock.py`](https://github.com/promptev/context-engine/blob/main/examples/model_adapters/bedrock.py) (Converse, every Bedrock model) | `https://bedrock-runtime.<region>.amazonaws.com/openai/v1` with a Bedrock API key, for models whose card lists Chat Completions; or a LiteLLM proxy |
| Mistral | `custom`, or `model=` with its SDK | `https://api.mistral.ai/v1` |
| Groq | `custom` | `https://api.groq.com/openai/v1` |
| DeepSeek | `custom` | `https://api.deepseek.com` |
| Together | `custom` | `https://api.together.ai/v1` |
| Fireworks | `custom` | `https://api.fireworks.ai/inference/v1` |
| OpenRouter | `custom` | `https://openrouter.ai/api/v1` |
| Ollama | `custom` (chat and embeddings) | `http://localhost:11434/v1` |
| vLLM | `custom`; `schema_enforced=True` with guided decoding on | `http://localhost:8000/v1` |
| LM Studio | `custom` (chat and embeddings) | `http://localhost:1234/v1` |
| LiteLLM proxy | `custom`, or `model=` with [`litellm_adapter.py`](https://github.com/promptev/context-engine/blob/main/examples/model_adapters/litellm_adapter.py) (the SDK, no proxy) | `http://localhost:4000` (the proxy's default port) |
| Any other gateway | `custom` | its OpenAI-compatible root |

**Schema enforcement.** Only OpenAI and Azure are claimed `native` by the
built-in client. Set `LLMConfig(schema_enforced=True)` only for an endpoint you
know enforces `response_format` json_schema; every other endpoint is reported
`prompt`, validated and repaired once. A host `model` makes the same promise
through `ModelReply.mode`.

**Embeddings.** Any vendor through `embedder=`: a SentenceTransformer, a
hosted API, a numpy reply (anything with `tolist()`). Or through `custom` for
an OpenAI-format embeddings endpoint (Ollama, vLLM, LM Studio, Gemini's
compatible URL, a gateway). `config.embedding.model` and `dim` are the
database's identity: the lock compares the model, so a database built through
one provider keeps working under `custom` or a host embedder as long as the
model string and the dimension match.

**Reranking.** Any reranker through `reranker=`, a hosted rerank API or a
local cross-encoder alike:

```python
import numpy as np
from sentence_transformers import CrossEncoder

cross_encoder = CrossEncoder("<cross-encoder model>")


def my_reranker(query: str, docs: list[str]):
    scores = cross_encoder.predict([(query, doc) for doc in docs])
    return np.argsort(scores)[::-1]   # numpy indices are accepted
```

**Per-purpose routing.** `ModelRequest.purpose` lets one function send
`map_reduce` to a cheap model and `vision_*` to a vision model:

```python
async def my_model(request: ModelRequest) -> ModelReply:
    if request.purpose == "map_reduce":
        return await cheap_model(request)
    if request.purpose.startswith("vision_"):
        return await vision_model(request)
    return await default_model(request)
```

## Batch mode: deferred model calls

A provider batch API answers within a day at about half the price. The engine
awaits every model call as it goes, so instead of blocking for hours your
`model=` or `embedder=` raises `ModelDeferred`: the request is accepted and
its answer comes later. During ingest the document parks as
`status="waiting_model"` with everything stored so far kept, and
`engine.resume_documents()` asks the same requests again, under the same
keys, once you have the answers. Batching stays yours: the engine never keeps
a half-finished conversation, your store is the memory.

```python
from context_engine import ModelDeferred, ModelReply, ModelRequest

async def my_model(request: ModelRequest) -> ModelReply:
    answer = await store.get(request.key)                  # your store, keyed by request.key
    if answer is not None:
        return ModelReply(text=answer.text, tokens=answer.tokens)   # the original counts
    await batch.enqueue(request.key, request)              # your provider's batch, same key
    raise ModelDeferred(retry_after=3600)                  # seconds until a resume is useful

# A cron on any worker, as often as you like: parked documents whose retry time
# has passed are claimed one at a time, so two workers never ask you twice.
report = await engine.resume_documents(content_source=lambda doc: blobs.get(doc.document_id))
for doc in report.documents:
    print(doc.document_id, doc.status)                     # "completed" | "failed" | "waiting_model"
for park in report.superseded:
    await batch.drop(park.request_keys)                    # nobody asks for these again
```

`ModelDeferred(retry_after=None)` takes seconds; a negative or non-finite
value is refused at construction. A stage never stops at its first deferral:
it asks every request in the stage, collects the deferrals, and parks once.
The stages after chunking (chunk embedding, structured extraction with its
registry embedding, entity extraction) run in the same pass, so a PDF takes at
most two rounds, page images then everything after chunking. With a 24-hour
batch window that is up to about 48 hours.

**What parks and what does not.**

| Call | On `ModelDeferred` |
|---|---|
| `vision_pages`, `vision_image` | every batch is still asked; the document parks at stage `extract`, storing no partial text, so no page is ever dropped |
| chunk embedding (`kind="document"`) | a deferred batch leaves its rows without a vector; parks at stage `enrich` |
| `structured_extraction` and its registry embedding | no repair re-ask on a deferral (the repair prompt would change the key); parks at stage `enrich` |
| `entity_extraction` | every batch is asked; if any deferred, nothing from the pass is stored, so a resume never writes a batch twice; parks at stage `enrich` |
| `community_summaries` and the community embedding (only with `graph.community_summaries` on) | the document does not park: it completes, the source's communities are stored without summaries, and the next `resume_documents` fills them per source |
| the query embedding (`kind="query"`) | treated as an embedder outage: the vector leg is dropped, the text legs answer, `on_error` fires. Answer it at once |
| a `reranker` | the fused order stands and `on_error` fires (stage `rerank`), as on any reranker error |
| `map_reduce`, `compute` | refused: `the host model deferred a {purpose} call. Deferral is only for ingest; a query needs its answer now.` A caller is waiting, so there is nothing to park |

While a document is parked nothing is billed and nothing is reported: no
usage event, no `on_error`, and the parked stage sends no `done` progress
event. Its stored chunks are still found by the text legs. `get_document`
and every listing show `status="waiting_model"` and a `parked` block beside
`meta_data`: `stage`, `parked_at`, `retry_at`, `rounds`, `request_keys`,
`last_skip_reason` and `last_skip_at`. The knowledge tool's document rows
carry the same two fields. `resume_embeddings` leaves a parked document alone
and reports `skipped` with `skip_reason="waiting_model"`.

**Resuming.** `resume_documents(document_ids=None, *, content_source=None)`
returns a `ResumeReport`. With no ids it sweeps every parked document whose
`retry_at` has passed (the database clock; `retry_at` is the park time plus
the smallest `retry_after` any deferral gave, or the park time when none
did), oldest first, in pages of 100. Naming ids resumes those whether due or
not; one that is not parked is skipped without a word, so a resume is
idempotent. Each document is claimed by one compare-and-set, so a cron on
several workers never asks you twice for the same request. A document that
defers again parks again with `rounds` incremented; one that completes gets
the same finalize as an ingest, with one usage event.

- `content_source` is for a document parked at stage `extract`: called with
  a `ParkedDocument` (`document_id`, `source_id`, `external_id`, `name`,
  `content_sha256`) it returns the original bytes, sync or async, or `None`.
  Without it, with `None`, or when it raises (`on_error` fires once with
  stage `content_source`), the document stays parked with reason
  `no_content_source`. Bytes whose sha256 is not the recorded one leave it
  parked with `content_mismatch`; two files are never mixed. A parked stage
  `enrich` is rebuilt from the database and needs no bytes.
- `ResumeReport.documents` is one `DocumentReport` per document worked on
  (`completed`, `failed` or `waiting_model` again); `skipped` is
  `{"document_id", "reason"}` per document left parked on this call, reason
  `no_content_source`, `content_mismatch` or `redaction_mismatch`, listed on
  every call until a resume succeeds and shown by `get_document` as
  `parked.last_skip_reason` / `last_skip_at`; `superseded` is one
  `SupersededPark` (`document_id`, `request_keys`, `superseded_at`,
  `reason`) per set of request keys nobody will ask for again, reported
  once: `reason="superseded"` for a park a re-ingest replaced,
  `reason="failed"` for the keys a pass deferred before it failed outright
  (a real embedding failure after a deferral, say) or an expired park
  held; that document's own ingest or resume report carries them too, as
  `abandoned_request_keys`.
  A document mid-ingest reports nothing until its row settles.
  `communities_resumed` is the source ids whose community summaries this
  call filled, in whole or in part.
- **A re-ingest of a parked document wins.** The old request keys are
  reported once in `superseded`, so you can drop those batch requests. An
  unchanged re-offer (the same bytes or text under the same arguments, an
  hourly folder sync say) is not a re-ingest: it leaves the park as it is,
  asks nothing, and returns the `waiting_model` report. The permissions,
  name, description and metadata it declares are still applied to the row
  and its stored chunks, as on any unchanged re-ingest: the latest
  declaration wins, also over an `update_document` made in between. A changed one that
  parks again asks the unchanged pages and chunks under their old keys, and
  those keys are left out of `superseded`. A key that is reported can still
  be owed by another parked document that shares the request (the same page
  image, the same chunk text); dropping it costs that document one more
  round. An anonymous ingest (no `external_id`, no `name`) that parks at the
  page reading stage is not deduplicated against a second anonymous upload
  of the same bytes; give documents an identity if that matters.
- **An edit made while a document is parked is kept.** `update_document`
  works on a parked document as on any other, and the row is what a resume
  reads: permissions, `name`, `description` and `meta_data` are the ones the
  row holds when the resume runs, never the ones the ingest call declared
  when the document parked. A principal removed while a document waits for
  a batch stays removed when it completes. A parked document shows the
  declared `meta_data` from the moment it parks, and a document deleted
  while parked stays deleted.
- `ingest.max_park_seconds` (default `None`, wait for ever) bounds the
  current round: a document whose latest park is older than that is
  finalized `failed` with `failure_reason="model_deferred_expired"` on the
  next resume, and `on_error` fires then.

**The request key.** `ModelRequest.key` (also `request_key(request)`) is the
sha256 hex of the canonical JSON of `{"v": 1, purpose, system, user, images,
response_schema, schema_name, json_mode, max_tokens, temperature,
thinking_budget}`, where `images` is the sha256 of each image's bytes in
order. It is the same in Python and TypeScript and on every resume: a schema
repair or a vision re-ask is deterministic given the same earlier answers, so
a follow-up request gets the same key each time. The key covers exactly what
your model receives. The key covers the prompt text, so an engine upgrade
that changes a prompt changes the key of every request made with it: resume
your parked documents before you upgrade, or expect the upgraded engine to
ask for them again under new keys. Embeddings are keyed PER TEXT, not per batch:
`embedding_request_key(text, kind, model)` with `model` your
`config.embedding.model`, because a resume re-embeds only the chunks still
missing a vector and may pack them into different batches. Store one vector
per text key; answer a batch only when every text in it is answered, and
otherwise queue the missing ones and defer the whole batch. Namespace your
answers by model on your side.

**Tokens.** A resume re-asks every model call of the parked stage, so return
a stored answer with its ORIGINAL token counts: model-call tokens are billed
once, in the completing pass. Work a parked round already stored is never
asked again and is billed once at completion: chunk vectors, registry
vectors, and the page text of a round that parked after chunking.

**Redaction, and what your store holds.** For every text purpose the key is
computed from post-redaction text, the text your model saw. Vision is the
exception: a `vision_pages` or `vision_image` request, and its key, is built
from the original page images, which redaction cannot reach, so the batch
requests you store for vision hold unredacted document content. Delete stored
batch requests and answers once the document completes. Resume from the same
engine, or the same `with_redaction` view: a different policy or key would
send different text and change every key, so such a resume leaves the
document parked with `redaction_mismatch`. The same goes for the other port:
Python and TypeScript do not compute the same fingerprint for the same
policy, so a document parked under a redaction policy is resumed by the port
that parked it.

**Community summaries** (only with `graph.community_summaries` on; with it
off this pass is skipped) are asked again per source, not per document.
Every `resume_documents` call, ids named or not, re-runs the summaries for
each source that has communities without one, under the same per-source lock
a graph ingest takes, lists the sources it completed in
`communities_resumed`, and bills each as one usage event (`detail.operation`
`resume_communities`). A summaries call that failed outright at ingest is
retried the same way; a resume that fails or defers again is neither billed
nor listed and is retried next call. A reply that summarises only some of
the communities it was asked about keeps the ones that came back: the source
is listed and billed for those, stays pending, and the next call asks for
the rest alone, in a smaller request with its own key. Superseded community keys are not
reported: answer by key, and ignore a request that is never asked again. A
round whose summaries were answered but whose community embedding deferred
asks the summaries key again. The source's lock is held on a session that
sits idle while your model answers, so a database
`idle_in_transaction_session_timeout` shorter than that answer ends that
source's resume with `on_error`, and the source stays pending.

## Configure

`ContextEngineConfig` is the one settings object, built directly or from
`CE_`-prefixed environment variables (nested fields use `__`, for example
`CE_EMBEDDING__PROVIDER`). `database_url` and `embedding` are required; with a
host `model` nothing else is.

```python
from context_engine import (
    ContextEngineConfig, EmbeddingConfig, GraphConfig, LLMConfig,
    RerankerConfig, SearchConfig, StorageConfig,
)
from context_engine.config import IngestConfig

config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", dim=1536),
    graph=GraphConfig(enabled=True),                  # Postgres-only graph mode: no neo4j_uri
    reranker=RerankerConfig(candidates=50),           # the window a host reranker sees
    search=SearchConfig(vector_floor="adaptive"),     # opt in, see Search
    ingest=IngestConfig(max_rows=500_000),
    storage=StorageConfig(pool_size=10, max_overflow=20, pool_recycle=300),
    secret_key="...",                                 # base64url, 32 bytes: tool-config encryption
)
```

`EmbeddingConfig` carries the identity the database records (`provider`,
`model`, `dim`; `dim` is detected on the first embed when unset) and the
request packing: `max_batch_items` (256), `max_batch_tokens` (250,000) and
`max_input_tokens` (8,192) default to the OpenAI-compatible row and are
overridable for an endpoint whose limits differ; `tokenizer` (`"auto"`: cl100k
through `tiktoken` only if its encoding file is already cached, never
downloading; `"exact"` may fetch; `"estimate"` never imports it);
`token_bytes_ratio` (3.0) and `safety_margin` (0.85) for the estimator; and
`max_concurrency` (4) embedding requests in flight per document.

`LLMConfig` and `EmbeddingConfig` both take `max_retries` (2; `0` disables;
`CE_LLM__MAX_RETRIES`, `CE_EMBEDDING__MAX_RETRIES`, `CE_VISION_LLM__MAX_RETRIES`,
`CE_GRAPH__EXTRACTION_LLM__MAX_RETRIES`), applied by the built-in client only.
The call timeout is per attempt, so one call's ceiling against a hung endpoint
is `(max_retries + 1) x timeout + backoff`, about 12 minutes at the defaults
(240 s x 3); structured extraction re-asks once after an unusable reply
(`max_reasks`, 1), and vision's batch attempts multiply again. Set
`max_retries=0` on a latency-sensitive path. Unknown fields on either config
are rejected.

`StorageConfig` holds the deployment knobs: `ann_exact_threshold` (50,000; the
vector leg's exact/approximate crossover, hardware- and dimension-specific,
see [docs/access-control.md](https://github.com/promptev/context-engine/blob/main/docs/access-control.md)), `write_batch_rows` (32; the
most chunk rows one write statement carries) and the SQLAlchemy pool.
A document's chunks and vectors are written in statements of at most
`write_batch_rows` rows, and each embedding statement commits on its own.
Writing a vector also writes its index entry, so on a small database a
statement carrying a few hundred vectors can run past a `statement_timeout`.
Lower the setting there, or for wide embeddings; raise it on a large instance
to save round trips; `None` puts no cap on a statement. A statement that is
cancelled loses only its own rows: the document ends `failed` with every
earlier vector stored, and ingesting the same content again embeds only the
chunks still missing one.
A caller who may see that many chunks or fewer gets an exact sort of them,
which returns the true top results and costs time in proportion to the rows
sorted: just under the threshold (48,378 visible chunks of 384 dimensions,
on a laptop) the vector leg took 337 ms at the median and 646 ms at the 95th
percentile, against 18 ms for the index scan on the same caller at recall
0.980. Lower the threshold to trade that last recall for speed, raise it to
keep the exact route for more callers, and measure at your own dimension
([benchmarks/acl_recall/](https://github.com/promptev/context-engine/blob/main/benchmarks/acl_recall/README.md#more-than-50000-visible-chunks)).
The pool settings:
`pool_size` (5), `max_overflow` (10), `pool_timeout` (30 s), `pool_recycle`
(1800 s; keep it under the idle timeout of any pooler in front of the
database) and `pool_pre_ping` (on). All reachable as env vars, for example
`CE_STORAGE__POOL_SIZE`.

`ExtractionConfig`: `max_render_px` (2000, the long edge of rendered pages and
sent images), `vision_max_output_tokens` (16,000 per vision batch),
`pages_per_vision_batch` (5), `vision_missing_page_retry`
(`single_after_cut`), `ocr_language` (detected when unset).

`vision_missing_page_retry` decides how the pages a vision batch did not
answer are asked for again. `single_after_cut`: after a reply that reached
`vision_max_output_tokens`, each missing page gets its own call, so a page the
model cannot stop writing about loses only itself. A reply that is merely
short is asked again in one call. `single`: one page per call after any
incomplete reply. `batch`: the missing pages always go back together, which is
the fewest calls. Raising `vision_max_output_tokens` does not rescue a runaway
page, it only makes it longer.
`IngestConfig`: the intake caps and resume fence, under [Ingest](#ingest).

The fields that hold a secret (`database_url`, `secret_key`,
`redaction_secret_key`, every `api_key`, `neo4j_password`) are pydantic
`SecretStr`. Pass plain strings, in code or in the environment; read one back
with `.get_secret_value()`. `repr`, `str`, `model_dump()` and
`model_dump_json()` show `'**********'` in their place, so a host that
serialises a config to rebuild it later must dump the secrets itself. A
validation error never prints its input either; when you log
`ValidationError.errors()`, log `err.errors(include_input=False)`.

## Provision the database

```bash
context-engine migrate --database-url postgresql://user:pass@localhost:5432/mydb --dim 1536
# add --graph to also create entity/relationship/community tables
```

`--dim` must match your embedding model's width (1536 for
`text-embedding-3-small`) and is locked in on first migrate. Run it again on
every deploy: it is idempotent, and the migration history ships inside the
package.

`migrate` does not write-lock your tables: every index after the initial
schema is built with `CREATE INDEX CONCURRENTLY`, so ingest keeps writing
while the build runs; it prints one line to stderr per index it builds, and
`SELECT * FROM pg_stat_progress_create_index` shows how far along it is. A
failed `migrate` is safe to re-run, and re-running is the repair: every
statement is idempotent, and an interrupted concurrent build is dropped and
rebuilt by the next run. A `lock_timeout` or `statement_timeout` on the
connection applies to these builds; `migrate` will not override it. Also from
code: `context_engine.cli.run_migrate(database_url, dim=1536, graph=False)`.

Managed Postgres varies in ways that matter. Point the compatibility test at a
**scratch database** (it runs the real migrations) before committing to a
provider; it checks pgvector for iterative scan, that `pg_trgm` and `unaccent`
are installable, and that their functions are reachable on the role's
`search_path` (Supabase installs extensions into an `extensions` schema, so
`CREATE EXTENSION` succeeds while `similarity()` and `%` stay unresolvable):

```bash
CE_COMPAT_DATABASE_URL='postgresql+psycopg2://user:pass@host/db' \
    uv run pytest tests/test_managed_postgres_compat.py -v -s
```

## Ingest

```python
# Plain text
report = await engine.ingest(
    text="Annual leave accrues at two days per month for every band-three employee.",
    name="leave-policy",
    source_id="hr-handbook",       # a namespace to scope search and stats
    external_id="leave-policy",    # your own id; with source_id it identifies the document on re-ingest
    acl=["group:hr"],              # omit for an unrestricted document (visible to everyone)
)

# A file: PDF/DOCX/PPTX/XLSX/CSV/HTML/EML/images/plain text, auto-detected
with open("policy.pdf", "rb") as f:
    report = await engine.ingest(file=f, name="policy.pdf", source_id="hr-handbook")

print(report.totals)               # {"files": 1, "failed": 0, "units": 3, ...}
print(report.documents[0].status)  # "completed" | "failed" | "skipped" | "embedding" | "waiting_model"
```

Exactly one of `file=` (a path or an open binary file), `content=` +
`filename=`, or `text=` (skips extraction). `ingest()` never raises on a bad
document; check `report.documents[i].status`. Construction is cheap and lazy,
so nothing connects until the first call; release the pool (and the embedding
client and the Neo4j driver) with `await engine.aclose()` or use the engine as
an async context manager. A process-lifetime singleton can skip it; anything
building engines per job or per test should not.

- **What a document is** is decided in this order: `mime_type="text/plain"`
  (or any other mime type) when you pass it, then the filename's extension
  (`.txt` and `.md` included; the `name` stands in for it on a `text=`
  ingest), then the content. Only the last is a guess. Text is read as CSV
  only on strong evidence: three or more rows, the same number of commas (two
  or more) on at least four lines in five, and cells that are values, not
  sentences. Prose with commas in it stays plain text. Pass `mime_type` when
  you know better: a tabular type keeps a document out of graph mode.
  The content check is narrow on purpose. It does not recognise a two-line
  CSV or a header on its own, a two-column file, tab- or semicolon-separated
  text, or a CSV whose last column is free-text sentences. Give those a
  `.csv` name or `mime_type="text/csv"`. `mime_type` is accepted by
  `engine.ingest` and by the HTTP ingest route, as the `mime_type` field of
  the JSON body or of the multipart form.
- `mode="hybrid"` (default) or `mode="graph"` (needs the `[graph]` extra and
  `graph.enabled`). `extract_structured=True` runs a structured-extraction pass
  (`document_type`, `structured_data`) through the host model or `config.llm`;
  `field_hints=[...]` makes it return exactly the hinted keys.
- **Scanned pages and images are transcribed to structured Markdown** (tables
  and headings preserved) by the host `model` or `vision_llm`, through the
  same seam as every other call, so any model that reads images works; with
  neither, embedded text is used, with Tesseract as a fallback if `[ocr]` is
  installed. The `[vision]` extra only adds page rasterization. Born-digital
  PDFs keep their structure too with the `[pdf]` extra: the text layer is
  converted to Markdown locally, no model, no network.
- **A re-ingest only pays for what changed.** Identical content is
  deduplicated by hash and skipped; when the content did change, every chunk
  whose text did not keeps the vector the previous version paid for, copied
  inside the database, and only the changed chunks go to the embedder (matched
  on a hash of the whitespace-normalised text, and only for the same embedding
  provider/model/dimension). Two callers racing the same document are settled
  by the database: the loser reports `skipped` with
  `skip_reason="in_progress"` and writes nothing, ACL updates included, so
  re-offer it once the winner is done.
- **A document with unreadable pages is read again.** A document that
  completed with `unreadable_pages > 0` is not treated as finished: ingesting
  the same bytes again extracts the whole file again, so a better model, a
  changed setting or a second try can recover the lost pages. It is chunked
  and embedded again only when the text came back different. A second read
  that lost more pages than the first is discarded and the stored text is
  kept. The read itself is paid for every time, so set
  `ingest.reread_unreadable_pages=False` when a scheduled sync keeps offering
  files with a page no model can read.
- **Chunks are budgeted to `min(2000, embedding.max_input_tokens)`**, because a
  chunk is one embedding input. Changing provider or that cap re-chunks each
  document on its next ingest.
- **Two intake caps, both refusing rather than trimming.** `ingest.max_file_bytes`
  (256 MiB) is answered without reading the document (`os.stat` for a path,
  `Content-Length` for an upload). `ingest.max_rows` (unset) counts `.csv`,
  `.tsv` and `.xlsx` only, since the extension is what routes a file to a
  tabular reader; the streaming reader's count is the authoritative one. Over
  either raises `IngestTooLarge`, which the routers map to 413 naming the knob.
  `None` lifts either; nothing is ever sampled or truncated.
  `routing_core.upload_limit_bytes(engine)` hands the byte cap to your own
  upload middleware.
- **Chunks are stored before they are embedded**, and requests are packed by
  tokens as well as by count against `EmbeddingConfig`'s limits. A single
  chunk over `max_input_tokens` is never split: the document fails with
  `failure_reason="input_too_large"` (the error behind it is `InputTooLarge`,
  which is what `embed_chunks` raises when you call it yourself). With the
  built-in client, a size rejection halves the cap for that run, and a
  one-item batch still refused fails the document the same way
  (`EmbeddingBatchRejected`); a host embedder is never retried or halved.
- **A failure keeps every batch already paid for.** The document sits at the
  non-terminal status `embedding` with `meta_data.embedding_progress =
  {done, total, tokens}`, and `await engine.resume_embeddings(document_id)`
  embeds only the rows still missing a vector. Called with no id it sweeps
  every resumable document, fenced by `ingest.resume_stale_seconds` (600) so a
  cron job cannot take over a run that is still moving; naming an id is never
  fenced. A partly embedded document is still findable on the text legs.
  `EmbedBatchFailed` carries how far the run got.
- **A host model that answers later parks the document.** A `model=` or
  `embedder=` that raises `ModelDeferred` leaves the document at the
  non-terminal status `waiting_model`, and `await engine.resume_documents()`
  asks again under the same request keys; see
  [Batch mode](#batch-mode-deferred-model-calls).
- **Progress is visible.** `on_progress` fires `started`/`done` per stage and,
  on the `embed` stage, one `"progress"` event per completed batch carrying
  `{done, total, tokens}` (`ingest.progress_batches=False` turns those off).
  Switch on the state and ignore one you do not know.
- A file that could not be read raises `ExtractionFailed` with the library's
  error as `__cause__`; `ImportError` (a missing extra, with an install hint)
  and `IngestTooLarge` pass through unwrapped.

### Why a document failed

A failed document carries two answers. `error` is the raw `str(exc)`, an
operator's copy: a provider body has carried an API key, a model name and a
quoted fragment of the document, so `engine.get_document`,
`engine.list_documents` and `DocumentReport.error` keep it and the
model-facing doors never return it. `failure_reason` is one code from a fixed
vocabulary, classified from the exception rather than its text
(`input_too_large`, `provider_rejected`, `provider_unavailable`,
`extraction_failed`, `chunk_set_changed`, `model_deferred_expired`,
`embedding_dimension_mismatch`, `unknown`), and `failure_message` is
the one fixed sentence for that code. `embedding_dimension_mismatch` is an
embedder (yours or the built-in client's) returning vectors of a different
width than `embedding.dim` or than this database's vector columns: check the
model and the dimension the database was migrated with. The knowledge tool's `get_doc`,
`get_docs` and `list`, the MCP tool behind them and `GET /documents/{id}` on
every router return the code and the sentence; a router mounted with
`principals=lambda: TRUSTED` still gets `error`. A later success clears both.

### Spreadsheets and CSVs

CSV/XLSX documents are indexed in hybrid mode, even in a graph-mode corpus
(`report.documents[i].mode_used == "hybrid"` with a fixed `mode_reason`): rows
are emitted as RFC 4180 CSV, streamed rather than materialised, and each sheet
gets a leading chunk with `meta.chunk_kind == "schema"` naming the columns,
their inferred types and a few sample values, pointing the model at `compute`
for arithmetic. Leave `ingest.max_document_text_chars` unset unless you never
compute: `compute` rebuilds its dataframes from the stored text, so a cut body
is a spreadsheet that quietly answers with fewer rows than it has. Setting it
is recorded (`meta_data.text_truncated`), and `compute` and `discover` refuse
a truncated document by name.

## Search and access control

```python
result = await engine.search(
    "how much annual leave do I get",
    source_ids=["hr-handbook"],
    document_ids=None,         # optionally pin to specific documents WITHIN the sources
    principals=["group:hr"],   # TRUSTED = trusted/internal (skips ACL); [] = anonymous
    top_k=10,
    compress_to_tokens=2000,   # optional: trim weak hits to fit a prompt budget
)

for hit in result.hits:
    print(hit.document_name, hit.idx, hit.score, hit.chunk_text[:80])
```

Full-text, trigram and vector legs, fused with reciprocal rank fusion
(`FusionConfig`: `k` 20, per-leg `weights`); `mode="graph"` adds the graph
leg. `document_ids` intersects `source_ids` inside the same SQL predicate
every leg shares, so a foreign id can never widen a search. Every hit carries
`idx`, its position inside the document (the knowledge tool reports it as
`chunk_idx`, which `get_chunks` takes as `start`).

**Long questions and the trigram leg.** The trigram leg compares the query's
trigrams with every chunk's, so its cost grows with the length of the query.
Measured on 100,646 chunks with questions of 16 words at the median, a hybrid
search at the defaults took about 2.6 s at the median (2,563 ms for a caller
who sees 10% of the corpus), nearly all of it in the trigram leg.
`search.trgm_max_query_words` skips that leg for a query of more words than
the setting. At `8` the median search fell to about 0.11 s (113 ms), because
95% of those questions then skip the leg. At `16` it barely moved (2,160 ms):
53% of the questions are 16 words or fewer and still run it. The results
change with the setting: at `8`, 92% to 95% of the default's results are
still in the top ten. The setting is unset by default, so the trigram leg
always runs and nothing changes until you set it. It is your choice: set it
when your queries are whole sentences and search time matters, and leave it
unset when they are names, codes and identifiers, which are short and are
what the leg is for. The times are one laptop, one query at a time, with the
query embedding not counted, and other work was running on that machine
during these runs: they show the size of the effect and are not a latency
benchmark. Conditions and every row:
[benchmarks/acl_recall/](https://github.com/promptev/context-engine/blob/main/benchmarks/acl_recall/README.md#search-time-and-the-trigram-leg-on-long-questions).

**`principals` is a trichotomy, and the dangerous value is the one that looks
like an absence.** `TRUSTED` (import it) disables filtering on purpose. `[]`
is an anonymous caller and matches only documents with no ACL. A list matches
unrestricted documents plus any whose ACL overlaps it. `None` is accepted for
compatibility and means `TRUSTED` too, with a `DeprecationWarning`; a later
release refuses it, because it is what every accessor degrades to
(`user.groups or None`, a missing key, `if user else None`), which makes the
unauthenticated path the maximum-privilege path. Never pass `None` because a
request had no authenticated user:

```python
from context_engine import TRUSTED

await engine.search(q, principals=TRUSTED)      # skip ACL, deliberately
await engine.search(q, principals=[])           # anonymous
await engine.search(q, principals=user.groups)  # scoped
```

On remote surfaces principals are injection-only: derive them server-side
from the session, never from a request body. Filing a document under an ACL
the caller does not hold is a 403 on ingest and on PATCH alike. Do not add a
`WHERE` of your own on top, and do not cache a `SearchResult` across users.

**Access control that survives the vector index.** Filtering an approximate
index with a `WHERE` clause is a post-filter: the index picks candidates
first, your ACL throws them away second, and the query returns short. So this
library puts the permission check inside every leg's query, and for the vector
leg it counts the rows a caller may see and sorts them exactly when there are
50,000 or fewer (`storage.ann_exact_threshold`). Measured on a
100,000-document corpus with synthetic team-shaped permissions: a caller who
may see 10% or 1% of the documents gets from every leg exactly the top ten
that a corpus holding only their documents returns, while the same search
filtered afterwards returns nothing for 64% of hybrid searches at 10% and 99%
at 1%, and recovers 0.42 and 0.02 of the results when it fetches ten times
more first. Above the threshold the vector leg is an approximate index scan:
it scored 0.979 to 0.988 with most of the corpus visible, and 0.652 for a
caller who sees 1% when that path was forced at this size. It needs pgvector
0.8 or newer; below that, ACL-filtered vector search silently loses recall.
Method, conditions and data:
[benchmarks/acl_recall/](https://github.com/promptev/context-engine/tree/main/benchmarks/acl_recall). Why the layout of
permissions matters to an index scan is written up in
[Your ACL Benchmark Is Measuring the Easy Case](https://promptev.ai/acl-benchmarks-measure-the-easy-case/).
Measure your own corpus, offline, read-only, no embedder called:

```bash
context-engine check-acl-exposure --database-url postgresql://...
```

### Thresholds follow the query's shape

`pg_trgm`'s similarity threshold moves with the query
(`context_engine.query_shape`, the same rules in both ports): 0.45 for a
two-character query down to 0.15 for a long sentence, measured in characters
for spaceless scripts (Chinese, Japanese, Thai, Khmer, Burmese, Lao, Tibetan)
and in words for everything else. `search.trgm_limit` pins it to one value.

The vector leg has a matching floor, **off by default**: `vector_floor="adaptive"`
drops candidates under a minimum cosine similarity chosen from the query (0.45
for one or two words, 0.40 for a code or an acronym, 0.25 for a sentence)
before fusion. Rows under it leave that leg only, can still arrive through
full-text or trigram, and the leg can return fewer than `limit` with nothing
backfilled. Measure before turning it on: those numbers came from one
embedding model, and cosine bands differ enough between models that an
absolute floor can empty the vector leg for short queries on another.
`SearchConfig` validates assignment, so `config.search.vector_floor = "adaptative"`
raises instead of reading as off.

### Keyword search settings

The full-text leg matches with Postgres text search over the `simple`
configuration, which keeps every word, stopwords included. Seven settings on
`SearchConfig` shape it; the defaults are the first value of each.

| Setting | Values | What it trades |
|---|---|---|
| `lexical_match` | `"all"`, `"any"` | `"all"` parses the query with `websearch_to_tsquery`, so a chunk must contain every word: precise for keyword queries, but a question such as "what is the notice period for contractors" usually matches nothing, because no chunk contains "what" and "is" and every other word. `"any"` matches a chunk containing any of the query's words after dropping stopwords, and leaves the ordering to the rank function. It finds more candidates, so it suits question-shaped queries and leans harder on ranking and fusion. |
| `lexical_rank` | `"ts_rank_cd"`, `"ts_rank"`, `"bm25"` | `ts_rank_cd` (cover density) rewards query words that occur close together. `ts_rank` counts how often they occur, wherever they are in the chunk. `bm25` (experimental, opt-in with `context-engine bm25-enable`) is Okapi BM25 computed in SQL: rare words count for more, repeating a word saturates, and long chunks are normalised. See [BM25 ranking](#bm25-ranking-experimental). |
| `bm25` | `Bm25Config(k1=1.2, b=0.75, idf_scope="source")` | BM25's parameters, read only when `lexical_rank="bm25"`. |
| `lexical_normalization` | `0` to `63` | The rank function's normalization bitmask, passed to Postgres unchanged. `0` ignores chunk length. `1` divides by 1 + log(length), `2` by length, `4` by the mean distance between matches (`ts_rank_cd` only), `8` by unique words, `16` by 1 + log(unique words), `32` maps the rank into 0 to 1. Length normalization stops a long chunk outranking a short one just by repeating a word. |
| `lexical_stopwords` | `None` or a list | The words `"any"` mode drops from the query. `None` uses the built-in English list, `context_engine.LEXICAL_STOPWORDS`; `[]` keeps every word. They are parsed by the same text-search configuration as the query, so case does not matter. A query made only of stopwords, with no phrase or `-term`, keeps its words. Ignored in `"all"` mode. |
| `trgm_max_query_words` | `None` or a whole number of 1 or more | Skips the trigram leg for a query of more than this many words, exactly as a fusion weight of 0 would for that query: it is not sent or fused, and it is absent from `usage["leg_hits"]` and `timings`. A word is a run of characters that are not Unicode White_Space, so punctuation stays part of its word and a query in a script written without spaces (Chinese, Japanese, Thai) counts as one word. The trigram leg is the slowest on long queries and helps most on short ones (names, codes, misspellings). `None` runs it for every query. |
| `lexical_candidates` | `None` or a whole number of 1 or more | How many candidates the full-text and trigram legs each hand to fusion. `None` uses the depth every leg uses: ten times the requested result count, between 50 and 500. A larger value is capped at 500. The vector and graph legs are unaffected. |

```python
from context_engine import ContextEngineConfig, SearchConfig

config = ContextEngineConfig(
    ...,
    search=SearchConfig(lexical_match="any", lexical_rank="ts_rank_cd", lexical_normalization=1),
)
```

#### BM25 ranking (experimental)

`lexical_rank="bm25"` orders the keyword leg's matches with Okapi BM25, in
plain SQL inside your Postgres. It matches exactly the chunks the other ranks
match (`lexical_match` still decides that, `"all"` or `"any"`), applies the
same permission check to every row before it is scored, and only changes the
order.

```python
from context_engine import Bm25Config, ContextEngineConfig, SearchConfig

config = ContextEngineConfig(
    ...,
    search=SearchConfig(lexical_rank="bm25", bm25=Bm25Config(k1=1.2, b=0.75, idf_scope="source")),
)
```

| `Bm25Config` field | Default | What it does |
|---|---|---|
| `k1` | `1.2` | Term-frequency saturation, `0` or more. `0` counts only whether a word is present; larger values let repetition matter longer. |
| `b` | `0.75` | Length normalisation, `0` to `1`. `1` fully penalises a chunk for being longer than average, `0` ignores length. |
| `idf_scope` | `"source"` | Where the corpus statistics (how rare a word is, the average chunk length) come from. `"source"`: the chunk's own source, so one source's documents never change another source's scores. `"global"`: every source together; other sources' (other tenants') documents shift the scores. `"document"`: the chunks of the chunk's own document, counted at query time. |

Each query word in a chunk adds
`ln(1 + (N - df + 0.5) / (df + 0.5)) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl / avgdl))`,
where `N` is the number of chunks in the statistics scope, `df` how many of them
contain the word, `tf` how often it occurs in the chunk, `dl` the chunk's length
and `avgdl` the average length. Lengths and counts come from the stored
`tsvector` (Postgres keeps at most 256 positions per word per chunk). Negated
terms (`-term`) never add to a score. `lexical_normalization` is ignored; `b`
does that job. The scores are computed by SQL functions the migration installs,
so the Python and TypeScript packages return the same numbers on the same
database.

- **Opt-in, per database.** Nothing changes for a database that does not use
  BM25: the migration only adds two small tables and some SQL functions, and
  never touches the chunk table. To use it, run once:

  ```bash
  context-engine bm25-enable --database-url postgresql://...
  ```

  or `await engine.enable_bm25()`. It installs four triggers on the chunk table (taking its
  lock briefly, with a 2-second timeout and retries) and counts the chunks
  already stored, one source at a time, without blocking ingest. Running it
  again recounts. Until it is enabled and has finished counting, a `lexical_rank="bm25"`
  search ranks with `ts_rank_cd`, reports a `LexicalRankFallback` to
  `on_error`, and says why in `usage["lexical"]` (`{"rank": "bm25", "rank_used": "ts_rank_cd", "reason": ...}`):
  `bm25_not_enabled` before `bm25-enable` has run, `bm25_stats_pending`
  while it is still counting (or if one of its triggers has since been
  dropped; running it again repairs that), and `bm25_schema_missing` before
  `migrate`. It never ranks with incomplete numbers. `idf_scope="document"` counts live
  and needs no enabling. `bm25-disable` (`engine.disable_bm25()`) removes the triggers and
  the statistics.
- **Statistics, and nobody waits.** How many chunks contain a word is counted
  at query time from the full-text index, for the query's words and the
  sources being scored only (`idf_scope="global"` counts across the whole corpus and pays
  for it). The chunk count and total length per source are kept by the
  triggers in an insert-only table, one small row per chunk write, in the same
  transaction. No row is shared between writers and no lock is taken, so two
  ingests into the same source never wait on each other, and an unchanged
  re-ingest nets to zero.
- **Compaction runs on its own.** After a write commits, a source holding more
  than `search.bm25.compact_after_deltas` rows (default 1000) is folded into one row. It never makes the
  write wait: when another maintenance call holds the source, it is skipped
  until a later write. `None` turns it off; then run
  `context-engine bm25-compact` (`engine.compact_bm25_stats()`) now and then. Recounting
  (`bm25-enable`) and compacting take a per-source lock that only maintenance
  uses, so any number of them may overlap each other and ingest (several
  replicas enabling at startup, a scheduled compaction) and the numbers stay
  exact. Chunks deleted outside this library's own methods (for example a
  cascade from your own SQL) are counted at once, but their source is only
  compacted at its next write through the library. While BM25 is not enabled
  the write path never looks for compaction work: it checks the enabled flag
  at most once every 5 seconds.
- **Statistics include documents the caller may not see.** How rare a word is
  is a corpus number, as in every search engine. No content or id is exposed
  and a hidden chunk is never scored or returned, but its words move other
  chunks' scores. `idf_scope="document"` limits that to the chunk's own document.
- **Measured: not a default, and not an improvement to default search.** On
  the five public datasets above, `lexical_rank="bm25"` with `lexical_match="any"` raised hybrid nDCG@10 on
  MuSiQue (+0.039), 2WikiMultihopQA (+0.016) and MultiHop-RAG (+0.062), made no
  measurable difference on ArguAna (+0.004) and lowered it on CQADupStack
  (-0.043). With the default `"all"` matching it changes almost nothing on
  question-shaped queries. The tables are in
  `benchmarks/retrieval/results/bm25.md`. Measure on your own queries before
  turning it on.
- **Cost.** Measured on 1,000-chunk writes: with BM25 not enabled, ingest is
  unchanged; enabled, an insert took about 10 ms more (184 against 172 ms)
  and a delete about 7 ms more. A `bm25` query scores every matching chunk and
  counts each query word in the index, so its cost grows with how common the
  words are; it was measured only on corpora of 7,000 to 12,000 chunks.
- **Another BM25 engine.** To rank with an external BM25 service or another
  Postgres extension instead, plug it in as a custom retrieval leg (see
  [Plug in your own retrieval](#plug-in-your-own-retrieval)); its results are
  re-filtered through the same permission check before fusion.

Quoted phrases and `-term` keep their meaning in both modes. Take the query
`"notice period" contractor -draft`:

- in `"all"` mode it matches a chunk that contains the exact phrase "notice
  period" AND the word "contractor", and does not contain the word "draft";
- in `"any"` mode it matches a chunk that contains the phrase OR the word
  "contractor" (or both), and does not contain the word "draft".

**What the defaults rest on.** The keyword and trigram defaults were measured
with the retrieval benchmark in this repository's `benchmarks/retrieval/`, on
five public datasets (MuSiQue, 2WikiMultihopQA, MultiHop-RAG, ArguAna and the
CQADupStack English forum), 300 queries each, with the `BAAI/bge-small-en-v1.5`
embedding model; the tables are in
`benchmarks/retrieval/results/lexical-defaults.md` and
`benchmarks/retrieval/results/lexical-defaults-2.md`.

- The fusion constant `fusion.k` is `20`. Against `60`, `20` raised hybrid
  nDCG@10 by 0.039 to 0.049 on four of five datasets, with no measurable
  change on the fifth; `100` lowered it on four.
- The trigram leg's weight in `fusion.weights` is `0.4`. `0.4` raised
  nDCG@10 over `0.8` by 0.04 to 0.07 on four of five datasets, with no
  measurable change on the fifth. Turning the leg off
  (`0`) did better still on those four but lost 0.027 on MultiHop-RAG, a news
  corpus full of names, so it stays on. If your corpus has no names, codes or
  misspellings to match, measure `0` on your own queries.
- `lexical_match="any"` raised nDCG@10 on MultiHop-RAG (+0.027 with the trigram leg at
  `0.4`) and lowered it on ArguAna (-0.122) and CQADupStack (-0.053), so
  `"all"` stays the default. Try `"any"` on your own queries before switching.
- `trgm_max_query_words` of 8 or 16 raised nDCG@10 on three datasets, but
  every value tried (8, 16, 32) lost about 0.025 on MultiHop-RAG, whose
  questions are long and full of names, so it stays off. On search time it
  is the largest setting there is for sentence-length questions: about 2.6 s
  at the median unset, 2,160 ms at `16` and about 0.11 s at `8` on 100,646
  chunks (see "Long questions and the trigram leg" above). Consider it when
  your queries are long sentences and search time matters more than name
  matching.
- `lexical_candidates` of 50 or 100 changed nothing measurable except on one dataset,
  so it stays unset.
- Lowering the full-text weight lowered nDCG@10 on two datasets, and
  `lexical_rank="ts_rank"` changed nothing measurable, so both stay as they were.

The `simple` configuration does not stem, so `-draft` excludes "draft" but not
"drafts" or "drafted", and `contractor` does not match "contractors". The query
is split into words, phrases and `-terms` on ASCII whitespace (space, tab, line
breaks). Query text is always bound as data and parsed by Postgres, never pasted
into SQL, and every variant applies the same scope predicate as the other legs.
The settings are also environment variables:
`CE_SEARCH__LEXICAL_MATCH=any`, `CE_SEARCH__LEXICAL_RANK=bm25`,
`CE_SEARCH__BM25__K1=0.9`, `CE_SEARCH__BM25__IDF_SCOPE=document`.

`fusion.weights` sets each leg's weight in fusion. A weight of `0` turns that
leg off, and its query is not sent:

- `fts` or `trgm`: that leg's SQL is skipped.
- `ann`: the vector leg is skipped, and the query is not embedded either unless
  `compress_to_tokens` needs the vector. The same holds, at any weight, for a
  storage backend that has no vector leg (`supports_ann` false): there is
  nothing to send the vector to, so the query is not embedded.
- `graph`: the graph stage is never started, even with `mode="graph"`. The
  search returns the hybrid result, is billed as hybrid, and reports
  `usage["graph"]["reason"] == "graph_weight_zero"`.

A leg missing from `weights` keeps the neutral weight of 1.0.

### The graph leg is chosen by the query

Leaving `mode` out runs the graph leg when the query itself resolves to an
entity the graph knows and this caller may see; one graph document in scope is
not a reason to re-rank every search by its vocabulary. `mode="graph"`
bypasses that gate but not the entity set: a query naming nothing gets the
hybrid answer. The result says which happened:

```python
result.usage["mode"]       # "graph": what you asked for
result.usage["mode_used"]  # "hybrid": what actually ran
result.usage["graph"]      # {"applied": False, "reason": "no_entities_in_query", "entities": []}
```

`reason` is one string from `context_engine.GRAPH_REASONS` (`not_requested`,
`graph_disabled`, `graph_unreachable`, `no_graph_documents`,
`no_entities_in_query`, `graph_detection_failed`, `no_visible_graph_chunks`,
`auto`, `requested`, `graph_weight_zero`). A mode-less search degrades to the
hybrid legs when the decision itself fails and reports
`graph_detection_failed`; `mode="graph"`
raises instead, so a caller that named the graph is never handed a hybrid
answer that looks like an empty corpus. A missing `[graph]` extra raises on
both paths. `entities` lists the entities the query named, in their normalised
stored form.

`graph_unreachable` means the configured Neo4j did not answer: the connection
was refused, the credentials were refused, or no reply came in time. A search
checks with one real round trip, bounded by 2 seconds or
`search.plugin_timeout_s` if that is shorter, and remembers the answer for 30
seconds either way, so a server that is down costs one probe and the searches
that follow answer from the other legs at once. The graph leg's own call to
Neo4j is bounded by `search.plugin_timeout_s` (10 seconds when unset): a
server that stops answering mid-search is cut off there, reported through
`on_error` (stage `graph_expand`, `"unreachable": True`), and skipped for the
next 30 seconds. Postgres-only graph mode has nothing to probe and never
reports it.

### Plug in your own retrieval

Every retrieval stage takes a plugin: extra legs beside the built-in ones (your
own keyword index, an Elasticsearch or OpenSearch query, a learned sparse
retriever), your own fusion function, and your own graph backend. They are
constructor arguments, not config.

```python
from context_engine import ContextEngine

async def es_leg(query, *, limit, scope):
    # scope.source_ids, scope.document_ids and scope.principals are read-only
    # tuples (None means not narrowed; principals None means a trusted caller).
    return await my_index.search(query, size=limit)  # chunk ids, best first

def my_fusion(ranked, weights, k):
    # ranked = {"fts": [...], "trgm": [...], "ann": [...], "es": [...]}
    scores = {}
    for leg, ids in ranked.items():
        for rank, cid in enumerate(ids, start=1):
            scores[cid] = scores.get(cid, 0.0) + weights[leg] / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

config.fusion.weights["es"] = 2.0  # adds a key; the built-in legs keep theirs
engine = ContextEngine(config, retrieval_legs={"es": es_leg}, fusion=my_fusion)
```

Set a leg's weight by adding its key, as above. Assigning a whole dict
(`config.fusion.weights = {"es": 2.0}`) replaces the defaults, so every leg
you leave out falls back to 1.0, `trgm` included.

Whatever a plugin returns is checked against the caller's permissions before
it is used. A plugin can rank chunks, but it can never widen what a caller sees:

- **Legs.** `retrieval_legs={"name": fn}`. `fn(query, *, limit, scope)` returns
  chunk ids, best first, and may be sync or async. Every leg runs concurrently
  with the built-in legs. Its ids are re-filtered by `source_ids`,
  `document_ids` and `principals` before fusion, so an id the caller may not
  see never reaches the caller and never takes a slot. The leg is then cut to
  `limit`. Its weight is `fusion.weights["name"]`: 1.0 when absent, and `0`
  switches it off so it is never called. A leg that raises is reported through
  `on_error` with `{"stage": "search_leg", "leg": name}` and the search
  continues with the other legs. If every leg failed and a custom leg is
  among the failed legs, the search raises `RetrievalFailed("every retrieval
  leg failed")`, without the plugin's own message; `POST /search` answers it
  with a 503. A name that reuses `fts`, `trgm`, `ann` or `graph` raises
  `ValueError` at construction.
- **Fusion.** `fusion=fn` replaces reciprocal rank fusion. `fn(ranked, weights,
  k)` gets each leg's filtered ids, each leg's weight and `fusion.k`, and
  returns ids, best first, sync or async. Its answer is de-duplicated and
  restricted to ids some leg produced, so it cannot add an id. Each hit keeps
  its RRF score as `score`; the list order is yours. A fusion function that
  raises falls back to RRF and is reported with `{"stage": "fusion"}`.
- **Graph backend.** `graph_backend=obj` supplies the raw graph; the engine
  does the rest. `obj` has two methods, sync or async (the
  `context_engine.GraphBackend` protocol):
  - `expand_candidates(seeds, query_entities, *, max_depth, scope)` returns
    `{"related": {entity: [ids]}, "sequential": [ids]}`: the chunks reached
    through each query entity a seed mentions, and the seeds' neighbours.
  - `chunk_degrees(ids, *, scope)` returns `{id: {"mentions": n, "next":
    [ids], "prev": [ids]}}`.

  Return the raw graph, uncapped. The engine always finishes through the same
  steps as the built-in backends: the caller's scope applies first, then the
  seeds come first, at most 100 chunks per entity and 500 in all, and a link
  counts toward a degree only when the chunk at its other end is visible. A
  backend with no entity structure tags its ids under one pseudo-entity,
  which is capped at 100 like any other. The built-ins, `PostgresGraphBackend`
  and `Neo4jGraphBackend` in `context_engine.graph.pg_expand`, implement the
  same protocol.

**Timeouts.** Every plugin call is bounded by `search.plugin_timeout_s` (10
seconds when unset; `CE_SEARCH__PLUGIN_TIMEOUT_S`). A leg that times out is a
leg outage, reported with `"timeout": True` in the `on_error` context; a
fusion function falls back to RRF; a graph backend leaves the graph leg with
its vector seeds. The built-in Neo4j backend is bounded by the same setting.
Sync plugins run on a small thread pool of their own, so a
plugin that hangs cannot hold up the threads the built-in legs use.

The knowledge tool and the MCP server search through the engine, so the
plugins apply there too. `usage["leg_hits"]` counts each leg's ids after the
filter. With a plugin configured, `usage["fusion"]` says which fusion ordered
the result (`"custom"`, or `"rrf"` when there is no custom function or it
fell back). `SearchResult.timings` holds each leg's wall time in
milliseconds, the graph leg included. Timings never appear in `usage`, in
the knowledge tool's output or over MCP.

## Redaction

Span-level masking of PII, or anything you can describe with a regex or a
function, as content is ingested, retrieved or returned. It composes with ACL
(ACL is *who* may see a document; redaction is *what is masked* inside it)
and is a pure transform: no model, no network. The default policy is a
guaranteed no-op.

```python
from context_engine import ContextEngineConfig, RedactionPolicy, RedactionRule

config = ContextEngineConfig(
    database_url="postgresql://...",
    embedding=...,
    redaction=RedactionPolicy(rules=[
        RedactionRule(name="emails", detector="email"),                    # -> [EMAILS]
        RedactionRule(name="order_id", pattern=r"\bORD-\d{6}\b"),           # -> [ORDER_ID]
        RedactionRule(name="acct", pattern=r"Account:\s*(?P<value>\d+)"),   # -> "Account: [ACCT]"
        RedactionRule(name="ssn", detector="ssn", action="hash",            # keyed, joinable pseudonym
                      apply_at="both", unless=["group:compliance"]),
    ]),
)
```

Built-in detectors: `email`, `phone`, `ssn`, `credit_card`, `iban`, `api_key`.
Actions: `mask` (default), `hash` (a keyed pseudonym; needs
`redaction_secret_key`, which falls back to `secret_key`), `remove`.
`apply_at`: `output` (default; masks on every read; `unless=[...]` lets named
principals see through) · `ingest` (scrub before storage, so masked text is
what reaches the embedder) · `both`. A `pattern` rule whose regex names a
group `value` masks only that group's span.

`phone` has two forms. Without a country code it needs a `-` or a `.` between
the groups, the same one in both gaps, or the `(415) 555-0123` form, optionally
behind a `1`/`+1`; `415 555 0123` and a bare `4155550123` do not match. Behind a
leading `+` the rule is looser: `+14155550123` and `+1 415 555 0123` match.
`hash` normalises per detector before HMACing (digits only for `phone`,
`ssn` and `credit_card`, a 10-digit phone hashed as if it carried `+1`; `email`
trimmed and lower-cased; `iban` upper-cased without whitespace), so one value
written two ways gives one token.

**The placement is the point.** Hits are masked before a reranker sees their
text, and tool results are masked before the audit row is written, so the
masked value is the only one that leaves the process or lands in retention.
For broader detection bring [Microsoft Presidio](https://github.com/microsoft/presidio)
(`[presidio]` extra); its entities become ordinary rules with the same
placement, and any `Callable[[str], list[tuple[int, int]]]` works as a
detector:

```python
from context_engine import presidio_detectors

RedactionPolicy(
    rules=[RedactionRule(name="person", detector="PERSON")],
    custom_detectors=presidio_detectors(["PERSON", "LOCATION"]),
)
```

One engine serving many tenants gives each its own policy and hash key
without a second connection pool:

```python
tenant_engine = engine.with_redaction(tenant_policy, secret_key=tenant_hash_key)
```

The view shares the pool, providers and tools, applies `tenant_policy` on
every method (ingest rules included) and hashes with `tenant_hash_key`, so
tokens do not join across tenants; closing it closes nothing. A `hash` rule
with no effective key is refused when the view is built; a view built with no
key while `redaction_secret_key` is unset falls back to `secret_key` and warns.
`redaction=` / `secret_key=` are also per-call arguments on every read path
(`search`, `get_document`, `get_documents`, `get_chunks`, `list_documents`,
`map_reduce`, `compute`, `search_knowledge_base`, `execute_tool` and the
rest), and `create_mcp_app` takes the pair per call.

## The knowledge tool and MCP

One tool, `search_knowledge_base`, whose `action` picks what it does. It is a
plain method, so an application driving its own agent loop needs no MCP, and
`create_mcp_app` serves the same function, so both doors give one answer:

```python
answer = await engine.search_knowledge_base(
    action="search", query="annual leave", principals=caller, scope=["hr"]
)
```

| Action | What it is for |
|---|---|
| `discover` | what documents there are and what is inside each one (sheets and columns, sections and pages, or JSON keys), plus the extracted fields by document type, a census of the corpus, and what each action can do. Call it first |
| `search` | passages by meaning or keywords |
| `get_doc` / `get_docs` | one whole document by id, or several at once |
| `get_chunks` | walk one long document in order, a piece at a time. Start at a search hit's `chunk_idx` to read around that passage |
| `list` | browse the documents without searching |
| `query_meta` | the structured fields a question names (dates, amounts, parties) and each document's value; it does not filter or compare by value |
| `compute` | a figure derived from the spreadsheets, over every row |
| `map_reduce` | the same question asked of every document in scope |
| `get_neighbors` / `traverse` | what is one step, or a few steps, from a named thing |
| `find_related` | connections of a kind across the corpus |
| `community_summary` | the themes each source groups into. Needs `graph.community_summaries` on, and is advertised as off without it. Refused when `scope` names documents, since a summary covers a whole source |

`scope` is required: the ceiling of source ids (or a `Scope(source_ids=...,
document_ids=...)`) the tool may ever reach, host-supplied and resolved per
call like `principals`, never a tool argument, because a model can name any
source id. Map your own word onto it (a project, a matter, a customer); a
caller narrows within it and never past it; an id outside is dropped; nothing
in scope returns nothing. A host with one shared corpus writes
`scope=UNSCOPED` on purpose.

`discover` answers with the schema, not an inventory: every listed document
carries `document_type`, `mode` and a `structure` whose shape follows the type
(`sheets` with column headers, row counts and the `frame_key` that `compute`
will call that sheet's dataframe; `sections` and `last_page`; `keys` for JSON;
`chunks` otherwise), and beside them `fields_by_type` and `document_types`,
both keyed by `(kind, type)`. Lists inside a `structure` are capped
(`more_columns`, `more_sections`, ...) because `discover` is the first call of
a conversation; `discover` and `list` take `document_ids`, and a call scoped
to named documents answers their structure whole. An action this deployment
cannot service is advertised as off, and called anyway answers a plain
`{"success": false, "error": ...}`; a truncated answer carries the
`next_page` call that reads the rest.

An **aggregate** question (a total, an average, a ranking) is refused when
every document in scope is a spreadsheet: `search` and `query_meta` answer
`{"success": false, "error": ..., "next_action": {"action": "compute",
"query": ..., "suggestion": {"frame_key": ..., "measure": ..., "group_by": ...}}}`
instead of rows, because search returns excerpts and a total over excerpts is
a sample reported as the figure. `map_reduce` is refused the same way for any
instruction over an all-spreadsheet target set. A mixed scope still answers and
carries a `hint`. `redirect_aggregates_to_compute=False` turns it off. A
question that **names a file** ("what is in INV-EG.pdf") searches document
names and answers one row per document with `matched_by: "filename"`;
nothing matching falls through to the ordinary search. `filename_search=False`
turns it off.

`knowledge_tool_definition()` exports the tool as data (name, description,
JSON Schema) for your own agent loop. `create_mcp_app` also takes `redaction`
(applied per call) and `compute` / `map_reduce` callables that replace the
built-in ones, which is where a host applies its own permission, billing and
approval rules. Serve it (`[mcp]` extra):

```bash
context-engine mcp --database-url postgresql://... --port 8080   # loopback, anonymous, for trying it out
```
```python
from context_engine import UNSCOPED, create_mcp_app

app.mount("/mcp", create_mcp_app(engine, principals=my_principals, scope=UNSCOPED))
```

The app is stateless (MCP spec 2026-07-28; no `Mcp-Session-Id`), so any
replica answers any request. It checks no `Host` header by default, like the
REST routers; `allowed_hosts=["api.example.com"]` turns on DNS-rebinding
protection for exactly those hosts (values are exact; an empty list or `*` is
refused), and `context-engine mcp` bound to loopback passes every loopback
name at its port. A refusal the engine means (a bad id, something the caller
cannot see) reaches the client with its reason; any other failure reaches it
as a generic tool error, so its text stays on the server. The connector-tool
gateway (`search_tools` + `execute_tool`) is registered beside the knowledge
tool. When the gateway is mounted over your own wrapper and its
`execute_tool` returns an MCP result object (`CallToolResult` or
`InputRequiredResult`), the gateway sends it as it is, so structured content
and a question for the person reach the client.

## Tools and governance

Let a model call `http`, `db`, `mcp` (an external server) and `function` (a
Python callable) tools through one governed path: ACL-checked, config
AES-256-GCM-encrypted at rest, every call audited.

```python
from context_engine import ToolConfig

await engine.register_tool(ToolConfig(
    name="get_weather", kind="http",
    description="Fetch the current weather for a city",
    config={
        "method": "GET",
        "url": "https://api.example.com/weather",
        "headers": {"Authorization": "Bearer super-secret-token"},
        "llmQueryParameters": {"properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
    acl=["group:ops"],
))  # -> call_name "http_get_weather"

def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b
engine.register_function_tool(add)  # -> call_name "fn_add"; not persisted

out = await engine.execute_tool("http_get_weather", {"city": "Lahore"}, principals=["group:ops"])
# -> {"result": {"status_code": 200, "data": {...}}, "usage": {"kind": "tool", "units": 1}}
```

A caller may only register or relabel a tool under principals it holds (403
otherwise), and on `POST /tools` and `PATCH /tools/{id}` an omitted `acl` is
403 for a non-trusted caller, since a tool with no ACL is callable by every
principal. Nothing makes a tool's name unique: every tool `list_tools` and
`search_tools` return carries its `tool_id`, and a name that matches more than
one tool the caller can see raises `AmbiguousToolError` (409 on the route)
unless `tool_id=` says which.

**Discovery.** `search_tools(query, principals=..., kind=..., name_contains=...,
requires_approval=..., limit=..., cursor=...)` ranks on the query and filters
on the rest (AND). An empty query with filters lists the filtered set; a
non-empty query also excludes tools none of its terms match. The page carries
`next_cursor` when more remain; send it back with the same query and filters.
`list_tools` never hides a tool.

**Approval gate.** A tool with `requires_approval=True` returns an
`approval_required` payload instead of running, until resolved:

```python
from context_engine import resolve_approval

out = await engine.execute_tool(
    "http_get_weather", {"city": "Lahore"},
    principals=["user:a"],
    approval_scope="run-42",   # opaque, chosen by you; a run id is the usual shape
)
# -> {"approval_required": {"approval_id": "...", "expires_at": "...", ...}}   (not executed yet)
await resolve_approval(engine, approval_id, "approved", approver="boss")
# the same call, same approval_scope, runs for real, exactly once
```

A conditional gate is `approval_policy={"condition": "amount > 100000"}`:
one `field op value` comparison (`>`, `>=`, `<`, `<=`, `==`, `!=`) against
the call's arguments, never evaluated as code. It fails closed: a condition
that does not parse, an argument that is missing, and an ordering that
cannot be made all require approval. An ordering is made only between two
numbers or two strings, so `"9,999,999"`, `None`, a boolean, a list or NaN
against a number requires approval. Both packages answer the same for every
operator and argument type.

`approval_scope` is the claim scope: an approved record is consumed only by
the same call in the same scope (and, when the record carries principals, by
a caller who overlaps them); within a scope, identical pending requests share
one record, so a retrying agent gets the same `approval_id` back. Resolve it
server-side; on the routers and the MCP gateway it is a dependency beside
`principals`, never a request field. A call without it runs unscoped with a
`DeprecationWarning`; a later release refuses a gated call with no scope. An
approval is for one tool, not a name: the record carries the `tool_id` it
opened for. Getting the request in front of a human is your job; the engine
owns the single-use, atomic state machine. Park until `expires_at`.

**Result budget.** `result` is cut to 8,000 characters and 100 rows unless you
say otherwise (`result_max_chars=`, `result_max_rows=`; `None` lifts a limit;
a db tool's own `max_rows`, 1000, still applies first). `response_mode="tsv"`
works for any tool: every array of objects in the result becomes a TSV string
under the same key (`\N` is NULL, nested values are JSON cells, tab, newline
and backslash are escaped), and the budget cuts at whole rows, reporting
`_result_shaping`. An http tool can set `"response_mode": "tsv"` once in its
config, for every method (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `QUERY`);
a caller's explicit `response_mode=` wins. The audit row keeps the original
under its own 50 KB cap. `format_result(result, "tsv")` and `rows_to_tsv(rows)`
do the conversion for any JSON you hold.

**One call's values over the saved ones.** `overrides=` runs one call with
your own values in place of what the tool was saved with: a signed-in user's
own token instead of the shared one, a tenant in the query, a field in the
body.

```python
out = await engine.execute_tool(
    "http_get_weather", {"city": "Lahore"},
    principals=["group:ops"],
    overrides={
        "headers": {"Authorization": f"Bearer {users_own_token}"},
        "query": {"units": "metric"},
    },
)
```

`headers` (http and mcp tools) are merged over the saved headers, names
compared without case; on an mcp tool an `Authorization` replaces the saved
sign-in for this call, and the next call signs in as saved. `query` (http) is
merged over the saved fixed query parameters, and `body` (http) over the saved
body's top-level keys; a saved body that is text and not a JSON object refuses
the call. Any name is accepted in each place. A key other than those three, a
value that is not an object, a header value that is not printable ASCII text,
or a place the tool's kind does not have (`query` or `body` on an mcp tool,
anything on a `function` or `db` tool) raises `InvalidInputError`, which names
the place and never a value. An override wins over a value the model would
fill; the model's value for that name is not sent. Otherwise the call is the
one a tool saved with those values would make: a body is sent only by a method
that sends one, and a redirect to another origin drops an overridden header
exactly as it drops a saved one.

The option is the host's. It exists on `engine.execute_tool` only: the HTTP
routes and the MCP gateway never read it from a request, and it is not among
the model's arguments. The engine applies exactly what you pass and restricts
nothing by name, so deciding who may override what is yours. The values are
used for that call and kept nowhere: not in the stored config, the audit row,
the usage event, an approval record or a log line, and a failed call's error
text has them blanked. What the tool's endpoint answers is the result as
ever, so an endpoint that repeats a header in its response puts it there.
`test_tool` takes no overrides. An approval stays bound to the tool and the
model's arguments. Overrides are not part of the claim, so you are
responsible for what you override on an approved call.

**Database access modes.** A `db` tool runs under `access_mode` `readonly`
(the default), `readwrite` or `full`. Read-only mode runs exactly one
statement, and that statement must be a read (`SELECT`, `WITH`, `SHOW`,
`EXPLAIN` without `ANALYZE`, `DESCRIBE`). A query that holds more than one
statement is refused before anything reaches the database, so a leading
`SELECT` followed by `COMMIT` and a write can no longer end the read-only
transaction and let the write land. `SELECT ... INTO`, `EXPLAIN ANALYZE`, and
statements that open with transaction control or a session change (`COMMIT`,
`BEGIN`, `SET`, `COPY`, and the like) are refused in read-only mode as well. A
read-write tool refuses `COPY`, which can run a shell command on the server.
A trailing semicolon is fine, and a semicolon inside a string, a comment, a
quoted identifier or a dollar-quoted body is not a statement boundary.

**Egress.** An `http` or `mcp` tool cannot reach private, loopback, link-local
or metadata addresses (`169.254.169.254` included), on the URL and on every
redirect hop; for `mcp` that covers the server URL, every redirect it is
allowed to follow and the OAuth endpoints it advertises. The check is
enforced at the connect: the name is resolved once, private answers dropped,
and the socket opens to an address that passed, so a name that rebinds between
check and connect is refused with the same `EgressDenied`. A redirect to
another origin drops every header the tool was configured with, saved or
overridden for the call, so a credential issued for the first host (an
`Authorization`, a `Cookie` or a custom `X-API-Key`) never reaches the host it
redirects to; a same-origin hop keeps them. Engine-built clients ignore
`HTTP_PROXY`/`HTTPS_PROXY` and netrc; to use a proxy, hand the engine your own
client through `http_client_factory`. Set `allow_private_egress=True`
deliberately to reach an internal service.

**A failed call's error text is stripped, then masked**, and the caller, the
audit row (`context_engine_tool_calls.error_message`) and the `on_tool_call`
hook all get the same reduced text: a db driver's message loses the echoed
statement, the `DETAIL`/`HINT`/`CONTEXT` lines and every single-quoted
literal, and what survives goes through the call's redaction policy. A quoted
run is kept only when its text appears verbatim in the SQL the query ran (a
column or table name the model already wrote, so it can repair its query), and
a value that came from a row stays masked. If the masking itself fails, all
three get `error text withheld: the redaction policy could not be applied`. An
MCP tool that answers with a result the server marked `isError` is a failed
call too: its text (not a row value) comes back the same way, no usage unit is
billed, and an approval-gated tool that failed keeps its approval.

**Every call whose tool was started leaves one audit row.** `execute_tool`
first puts the arguments in one storable form. A NUL character (U+0000) is
removed before the tool receives them. Arguments that cannot be stored are
refused with `InvalidInputError` before the tool runs and before any approval
is opened: broken Unicode (a lone surrogate), a non-finite number (`NaN`,
`Infinity`), nesting deeper than 64 levels, or a value that is not a JSON
value (anything but a dict, list, tuple, str, int, float, bool or `None`: a
`datetime`, a `Decimal`, a `UUID`, bytes, a set, your own class). A tuple is
sent as a list; a set has no order to store, so pass a list. The message
names the argument's path and never repeats its value, and the
routers answer it with 400. A refused call ran nothing, so it has no audit
row.

A result or an error is never refused, since the tool has already run. A
result keeps its JSON form on the router, over MCP and in its audit row, and
all three spell a value the same way: a `datetime`, `date` or `time` is ISO
8601 text (`"2026-10-02T12:15:19.667254Z"`), a `timedelta` is an ISO 8601
duration (`"P1D"`), a pydantic model or a dataclass is an object, an `Enum`
is its value, a `UUID` and a `Decimal` are text. Binary (`bytes`,
`bytearray`, a `memoryview`, so a `bytea` column of a db tool) is returned as
text when it is valid UTF-8 without a NUL byte, otherwise as base64 text, and
the value alone does not say which.

A result that is a one-shot iterator (a generator, `map`, `reversed`, at the
top or nested in a dict or a list) is read once, right after the tool ran,
and returned as a list, to an in-process caller too. An iterator that yields
more than 50,000 items is a failed call. A db tool's `text` preview, a TSV
cell and the text of a truncated result render a timestamp in Python's text
form (`2026-10-02 12:15:19.667254+00:00`), while `rows` carry the ISO form.

Only a value JSON cannot carry is changed: a lone surrogate (broken Unicode)
becomes U+FFFD, a non-finite number becomes the text `"NaN"`, `"Infinity"` or
`"-Infinity"`, U+0000 is removed, and a value nested deeper than 64 levels is
cut there (`output_truncated` on the row). So a call that ran and was
recorded never comes back as a server error. `engine.execute_tool` called in
process returns the tool's own objects unchanged, an iterator apart.

A tool or an approval deleted while the call ran costs the row only its
reference. The row is written with its real arguments and result, `tool_id`
and `approval_id` NULL, and a note in `error_message` that begins `the audit
row was stored without its references` and names both ids. So filter audit
rows on `success`, not on `error_message`: a successful row can carry a note.

If the full row cannot be written for any other reason, a last-resort row is
written in its place: who called which tool, whether it succeeded and how
long it took, with `input_args` set to `{"_unstorable": true}`, no result,
`output_truncated` true and an error note naming what failed. It depends on
no other row: its `tool_id` and `approval_id` are NULL and the ids are in the
note.

A call that is cancelled after its tool was started (your own timeout, an
MCP client that gave up) is recorded as a failed call with the error text
`cancelled`, and the cancellation then propagates as usual. A tool that ends
with any other `BaseException` is recorded as `interrupted by <type>`. The
row is written before the cancellation is raised, even when the cancellation
arrives during the write. A call cancelled during the write of a tool that
succeeded is recorded as a success and metered: `on_usage` fires once, in
step with `on_tool_call`.

A cancelled call waits for that write for at most
`tools.audit_cancel_wait_s` seconds (default `10`; `0` does not wait). The
write is the full row and, if that fails, the fallback rows. When the
database does not answer in that time, `on_error` is called with
`stage="tool_audit"` and a `TimeoutError`, the cancellation is raised, and
the write is left to finish in the background, so the row may still arrive.
A call that is not cancelled is not bounded by this setting. What no library
can record is a process that dies between the tool running and the insert,
or a coroutine that is closed without being run to its end.

**When no audit row can be written at all** (the database is down), the tool
has already run. `on_error` is called with `stage="tool_audit"`, and what the
call does next is the setting `tools.audit_on_failure`:

- `"fail"` (the default): the call raises `ToolAuditFailed`, so nothing runs
  unrecorded. The error carries `tool_ran = True`, the tool's `result` shaped
  and redacted as the caller would have received it, `truncated`, and
  `tool_error` when the tool itself failed. The router answers 502 with
  `{"detail": ..., "tool_ran": true, "result": ..., "truncated": ...,
  "tool_error": ...}`. The MCP surface answers an error whose text says the
  tool ran and ends with the result, never a normal result with a flag. The
  tool ran, so the call is metered: `on_usage` fires for it as it does for
  any successful call, in step with `on_tool_call`.
- `"warn"`: the call returns the tool's result as usual. Choose it when
  staying available matters more to you than a complete audit trail.

Do not retry a call whose error says the tool ran: a retry runs the tool a
second time. Use the result on the error. A refusal made before the tool runs
is a different error, `InvalidInputError`, whose `tool_ran` is `False`. A
tool behind an approval never runs when its approval record cannot be read
or written, under either value.

```python
config = ContextEngineConfig(..., tools={"audit_on_failure": "warn"})

try:
    out = await engine.execute_tool("http_send_invoice", args, principals=principals)
except ToolAuditFailed as exc:
    out = {"result": exc.result, "truncated": exc.truncated}  # it ran: do not send it again
```

**An approval matches its arguments exactly.** They are compared in the
canonical JSON form PostgreSQL writes: key order does not matter, and `1`,
`1.0`, `true` and `"1"` are four different arguments, so an approval given
for one is never claimed by another. Two spellings of one number with the
same decimal places are the same argument: `1e16` and `10000000000000000`,
`0.0` and `-0.0`.

**Connecting an `mcp` tool that needs OAuth** ends in a popup on your origin
that `postMessage`s the result to `window.opener`. On failure the message is
`{type: "mcp-oauth-error", code, message, correlation_id}`; branch on `code`
(`idp_denied`, `malformed_callback`, `state_invalid`,
`token_endpoint_unreachable`, `token_exchange_failed`, `internal_error`),
since `message` is one of six fixed sentences for the person reading the
popup and never the provider's error text. `correlation_id` is
`mcpoauth_<32hex>`, shown as `Reference: ...` and logged server-side beside the
detail that is deliberately not rendered; the success payload carries it too.
The callback (`GET /mcp/oauth/callback`) is the provider's bare browser
redirect, so it is the one route every router answers without the `auth`
dependency, under any mount prefix. Every other route runs behind `auth`.

### Connecting MCP servers

**1. What it is.** An `mcp` tool is a third-party MCP server the engine calls
as a governed tool: the same ACL check, encrypted config, approval gate, audit
row and usage event as any other tool. The engine speaks to the server through
the official MCP Python SDK client (the `mcp` package, versions `>=2.2,<3`),
installed by the `[mcp]` extra:

```bash
pip install 'promptev-context-engine[mcp]'
```

The SDK owns the protocol (version negotiation, the handshake, sessions,
resumption). The engine owns what runs around it: the egress checks, the
redirect policy, the time bounds and the hook that asks your user.

**2. Supported servers.** Every MCP protocol version the SDK supports:
`2024-11-05`, `2025-03-26`, `2025-06-18` and `2025-11-25` (the `initialize`
handshake), and `2026-07-28` (the stateless revision). A newer protocol
version arrives by upgrading the SDK; the engine has no version code of its
own.

The transport is Streamable HTTP, and only that. A server that offers only
HTTP+SSE or WebSocket is refused before anything is dialled. That covers a
URL that starts with `sse+http://`, `sse+https://`, `ws://` or `wss://`, and
a URL whose path ends in `/sse` or `/events`. The error is
`UnsupportedMCPTransport` (a `ValueError`) with this text:

```text
MCP over SSE and WebSocket is no longer supported. Use the server's Streamable HTTP URL.
```

The fix is to register the server's Streamable HTTP URL (usually the one that
ends in `/mcp`). `test_tool` reports the same refusal as
`{"ok": False, "error": "unsupported_transport"}`, and
`POST /mcp/connect-and-list` answers 400 with the text.

A URL that carries a user name or password (`https://user:pw@host/mcp`) is
refused too, when the tool is registered or updated and again when it is
called, with a message that never repeats it:

```text
MCP server URL must not carry a user name or password. Send credentials in the tool's headers.
```

**3. Settings.** Each setting has a global value in the engine config and may
be set again on one connection, in the tool's own `config` beside `url` and
`headers`.

| Setting | Global key | Per-connection key | Default | Allowed values |
|---|---|---|---|---|
| Protocol | `tools.mcp.protocol` | `protocol` | `"auto"` | `"auto"`: the SDK asks the server which version it speaks. When the server answers that it does not know the question (a JSON-RPC error, or a 4xx such as 404 or 405), the SDK falls back to the `initialize` handshake. No answer, a connection error or any 5xx fails the connect at once, and so does a 401 or 403. `"modern"`: the 2026-07-28 revision only, with no handshake. `"legacy"`: the `initialize` handshake only. |
| Redirects | `tools.mcp.redirects` | `redirects` | `"refuse"` | `"refuse"`: a redirect fails the request. `"same_origin"`: follow a redirect that keeps the scheme, host and port. `"any"`: also follow one to another origin, without credentials. |
| URL questions | `tools.mcp.elicitation_url` | `elicitation_url` (or `elicitationUrl`) | `False` | `True` passes a server's "open this page" question to your hook. `False` declines it without asking you. |
| Hook timeout | `tools.mcp.elicitation_timeout_s` | `elicitation_timeout_s` (or `elicitationTimeoutS`) | `300` | Seconds, above 0. How long `on_elicitation` may take for one question. |
| Log redaction | `tools.mcp.redact_logs` | none (global only) | `True` | `True`: no log line from an MCP connection holds a URL query string, userinfo, a session id, a header value or a traceback, and a URL is logged as the server identity (scheme, host, port, path). `False`: the MCP SDK's and the HTTP library's lines pass as they are, for debugging, except the SDK's whole-message lines (`Sending client message:`, `SSE message:`, `client callback for`), which stay dropped because they can repeat what a user answered. |
| Response size cap | `tools.mcp.max_response_bytes` | none (global only) | `10_000_000` | The most decoded bytes one MCP response may carry, counted as it is read, so a compressed body that unpacks to more is stopped too. Past it the request fails with an error naming this setting. `None` lifts it. |

Connecting to a server and listing its tools, which happens before every
call, is bounded as a whole at 60 seconds, on top of each request's own bound,
so a server that answers slowly cannot hold a call. The listing is not
optional: the SDK checks a tool's result against the schema the listing
returned.

The connection's value wins when it is set; otherwise the global value
applies. An invalid value is refused when the tool is registered or updated
(`ConfigTemplateError`, 400 on the routes) and again when the tool is called.
It never falls back to a default. The global keys are also environment
variables: `CE_TOOLS__MCP__PROTOCOL`, `CE_TOOLS__MCP__REDIRECTS`,
`CE_TOOLS__MCP__ELICITATION_URL`, `CE_TOOLS__MCP__ELICITATION_TIMEOUT_S`,
`CE_TOOLS__MCP__REDACT_LOGS`. `POST /mcp/connect-and-list` takes `protocol`
and `redirects` in its body too, so you can list a server's tools with the
settings it needs before you register it.

When `"auto"` cannot get an answer to its version question, the error ends
with the way out:

```text
The server did not answer the protocol version question. Set protocol to "legacy" on this connection if it only speaks the initialize handshake.
```

That advice is given only under `"auto"`, the one mode that asks the
question. A 401 or 403 there fails the connect without it, as
`MCPHttpStatusError`, so the OAuth path can take over. Setting `"legacy"` on
an older server also saves it one request on every call.

**4. Examples.** A global config:

```python
from context_engine import ContextEngine, ContextEngineConfig, McpClientConfig, ToolsConfig

config = ContextEngineConfig(
    ...,
    tools=ToolsConfig(mcp=McpClientConfig(
        protocol="auto",              # let the SDK negotiate
        redirects="refuse",           # a redirect fails the request
        elicitation_url=False,        # decline "open this page" questions
        elicitation_timeout_s=120,    # two minutes per question
    )),
)
```

Two tools that override it for their own connection:

```python
from context_engine import ToolConfig

# An old server that only speaks the handshake: skip the negotiation.
await engine.register_tool(ToolConfig(
    name="legacy_crm", kind="mcp",
    config={"url": "https://crm.example.com/mcp", "protocol": "legacy",
            "headers": {"Authorization": "Bearer ..."}},
    acl=["group:sales"],
))

# A server you trust that redirects /mcp to /mcp/ on its own origin.
await engine.register_tool(ToolConfig(
    name="docs_search", kind="mcp",
    config={"url": "https://docs.example.com/mcp", "redirects": "same_origin"},
    acl=["group:everyone"],
))
```

A hook that answers the server's questions:

```python
from context_engine import ContextEngine, ElicitationRequest, ElicitationResponse

async def ask_user(req: ElicitationRequest) -> ElicitationResponse:
    # req.server, req.tool, req.tool_id, req.principals: who is asking, for whom
    # req.mode == "form": req.message, req.requested_schema (a flat JSON schema)
    # req.mode == "url":  req.message, req.url (show it, never open it for them)
    choice = await my_ui.prompt(req)                 # your UI, queue or policy
    if choice.dismissed:
        return ElicitationResponse(action="cancel")  # closed without choosing
    if not choice.agreed:
        return ElicitationResponse(action="decline") # an explicit no
    return ElicitationResponse(action="accept", content=choice.values)

engine = ContextEngine(config, on_elicitation=ask_user)   # a sync function works too
```

**5. Redirects.** The engine's server dials the URL you registered, so a
redirect is a second destination that the MCP server chose. It could point at
an internal address, or carry your credentials to another host. That is why
the default refuses every redirect, with this error:

```text
MCP server redirected; register the final URL
```

- `"refuse"` fails any 3xx. Register the URL the server redirects to.
- `"same_origin"` follows a redirect that stays on the same scheme, host and
  port, with its headers. A redirect to another origin is refused
  (`MCP server redirected to another origin; register the final URL`).
- `"any"` also follows a redirect to another origin. From that hop on, the
  request goes without `Authorization`, without cookies, without the MCP
  session id and resumption token, and without the headers you configured
  for the tool. They stay off for the rest of the chain, even on a hop back
  to the first origin.

In both following modes every hop passes the egress check and is opened by
the guarded transport, at most 3 hops are followed for one request, and a
redirect from https to http is never taken. A 301, 302 or 303 answer to a
POST is refused too, because following it would drop the message, and so is
a redirect whose target carries `user:password@`. Links inside a server's
answers are never followed by the engine: a tool result is data, returned to
the caller as it is.

**6. Questions from the server (elicitation).** A third-party MCP server can
ask the user something during a call: a confirmation, a missing field, a link
to open.

- **You own how the user is asked.** The engine passes the question to your
  `on_elicitation` hook and sends back what you return.
- **The engine never accepts on its own.** With no hook the client tells the
  server nothing about elicitation. A server that asks anyway gets a
  protocol error instead of `decline`. On a 2026-07-28 connection that fails
  the call. On a handshake connection the server receives the error and
  decides what happens next. An `accept` is checked
  against the server's `requested_schema` (types, `required`, `enum`,
  bounds). Content that does not conform is not sent: the server gets
  `decline` and `on_error` says which field failed.
- **At most 8 questions per call.** Later ones are declined and reported
  through `on_error`. A call may also take at most 8 input-required rounds.
- **`sensitive_hint`** is true when a form field looks like a credential
  (named like password, secret, token or api_key, or `format: password`).
  Servers may not ask for those by form, so treat such a question with
  suspicion.
- **The server's text is untrusted.** `message` and `requested_schema` are the
  server's words. Show `req.server` beside them so the person knows who is
  asking.
- **URL mode stays off unless you enable it** (`elicitation_url`). With it on
  the URL goes to your hook and nowhere else. The engine never opens or
  fetches it, and declines one that is not https. With
  `allow_private_egress` on, http is accepted as well; any other scheme is
  still declined.
- **The hook has a time limit** (`elicitation_timeout_s`). A timeout or an
  exception sends `cancel` and reports through `on_error`. Time spent in the
  hook does not count against the tool call's own timeout. A sync hook that
  times out keeps running: the server gets `cancel` on time, but a blocking
  function cannot be stopped, so make long waits async.
- Questions are put to your hook one at a time. An accepted form with nothing
  filled in is sent as `{}`, and its required fields still apply.
- A failed call that asked questions reports them (actions and identities,
  never the content) through `on_error`, stage `mcp_tool_call`. A successful
  one lists them on its usage event, in `detail["elicitations"]`: one
  `{server, tool, tool_id, mode, action}` per question.

**7. Security guarantees.**

- Egress is checked on every connection: the registered URL, every redirect
  hop, and the address each socket actually dials.
- Proxy variables in the environment are ignored.
- A call is never sent again after a failure: the client sends `tools/call`
  once and never re-sends it after a timeout, an error or a dropped
  connection. The server may already have run the tool, so whether to retry
  is your decision.
- Every request is bounded. A tool call ends at its own timeout (120 seconds
  by default, with the time your hook takes left out). Every other request
  must send its response headers within 30 seconds and finish within 60. A
  request that gets no answer fails with its name:
  `MCP server did not answer 'initialize' within 30s`. Closing a connection
  waits at most 5 seconds for the server.
- Elicited content is never logged, audited or metered, whatever
  `tools.mcp.redact_logs` says: the MCP SDK's whole-message lines
  (`Sending client message:`, `SSE message:`, `client callback for`) are
  always dropped for the engine's connections, because a reply, the next
  call and a result can all repeat what the user answered. With the setting
  on (the default) no log line from an MCP connection holds a URL query
  string, userinfo, a session id or a header value either, and an exception
  is logged by its type name, without the traceback that would quote the
  URL or the server's body.
- Only the engine's own MCP connections are filtered. If your application
  uses the MCP SDK or its HTTP library itself, in the same process, its log
  lines are left exactly as it wrote them, and two engines with different
  settings each get their own.
- An HTTP error keeps its status: `MCPHttpStatusError` carries `status`
  (401, 404, 429, 503 and so on), so a rate limit, a missing endpoint and a
  broken server can be told apart.

A response stream that ends before the call's response is a failure
(`ConnectionError`), never an empty success, so it is audited as failed and
not billed. Your own name resolution goes in `ContextEngine(mcp_resolver=...)`:
an async `(host, port)` function that returns addresses. Its answers are
checked like any other.

**8. Upgrading from earlier releases.** See `MIGRATING.md` for the details.

- HTTP+SSE and WebSocket MCP servers are no longer supported. Register the
  server's Streamable HTTP URL.
- `mcp_resolver` is now an async `(host, port)` function that returns a list
  of addresses (any callable that returns an awaitable will do). It used to
  be an aiohttp resolver object, which is refused with a `TypeError`. So is
  a function that returns its addresses without being awaited, when it is
  first called.
- The `[mcp]` extra no longer installs `aiohttp`. If your own code imports
  it, add it to your own requirements.
- A redirect from an MCP server is still refused by default. It can now be
  allowed per connection with `redirects`.
- With no `on_elicitation` hook, a question a server sends anyway now gets a
  protocol error instead of `decline`: the call fails on a 2026-07-28
  connection, and on a handshake connection the server decides the outcome.
- A server must answer in the shape the protocol defines: `initialize` needs
  `serverInfo`, and every tool needs an `inputSchema`.
- In `"auto"` mode a version question that gets no answer, a connection
  error, a 5xx, a 401 or a 403 now fails the connect at once. Set `protocol`
  to `"legacy"` on a connection to an older server that cannot answer it.
- A response on a call's own event stream is taken as that call's response,
  whatever id it carries. This is the SDK's behaviour.
- The mcp tool config no longer requires `transport`.
- An MCP URL with a user name or password in it is refused. Move the
  credentials to the tool's `headers`.

## Graph mode

`graph.enabled` turns on entity and relationship extraction on
`ingest(mode="graph")`, a Postgres mirror of the graph, Louvain community
detection (with model-written summaries when you opt in), the four
navigation actions (`get_neighbors`,
`traverse`, `find_related`, `community_summary`) and the graph leg of search.
It needs a host `model` or `graph.extraction_llm` (the engine raises at
construction with neither), the `[graph]` extra for `networkx`, and
`migrate --graph`.

**Two graph backends: Postgres by default, Neo4j as an alternative.** The
graph leg widens the vector seeds before it ranks them, and both backends do
it the same way:

- **Expansion.** From the seeds to every chunk that mentions an entity the
  query names and a seed also mentions, plus each seed's neighbouring chunks
  in its own document (two positions either side). The union is capped at
  500 candidates.
- **Connectivity.** Each candidate's entity links plus its links to the
  previous and next chunk of its document, divided by the largest count in the
  candidate set. It is one of the five ranking signals (`graph_connectivity`).

Leave `neo4j_uri` unset and both steps run in Postgres, on the graph the
engine already stores there: one indexed query per hop, with the caller's
permissions applied on every hop. Set `neo4j_uri` and Neo4j, which the engine
keeps in step with every graph ingest, supplies the graph instead. Either way
the same final step runs after the caller's permissions are applied:

- at most 100 chunks per entity, the first by chunk id, taken in turns across
  the query's entities, so an entity mentioned in thousands of chunks cannot
  fill the candidate set by itself;
- the order (seeds, then entity matches, then neighbours) and the 500 cap;
- connectivity counts a link only when the chunk at its other end is visible
  to the caller.

So the two backends return the same results for every caller, and a chunk the
caller cannot see never takes a candidate or result slot, or adds to a visible
chunk's connectivity. Two cases can still differ. Neo4j holds only what graph
ingest synced to it, so a chunk from a document ingested without graph mode,
or whose Neo4j sync failed, has no neighbours or connectivity there. And
Neo4j's raw output is bounded at 20,000 ids per entity, and for neighbours, as
a safety backstop, so for an entity mentioned by more than 20,000 chunks that
bound applies before permissions do. Which chunks a hub contributes is decided
by chunk id, not by relevance to the query.

Choose Postgres unless you have a reason not to: it is one database to run,
back up and secure, the permissions are the same SQL predicate every other
search leg uses, and it needs no extra service. Choose Neo4j when you already
operate one, or want the same graph available to your own Cypher queries and
tools. `GraphConfig.neo4j_configured` is the predicate. Neo4j is
all-or-nothing: a URI without a password, or a password without a URI, is an
error, since the latter is indistinguishable from a dropped URI silently
moving a deployment to the other backend. The driver is imported only when a
URI is set.

**Graph ingests, re-ingests and deletes can run side by side.** Entities are
shared across the database by name, so documents ingested together write the
same rows. Everything that locks entity rows takes them in one order, by
name: an ingest's write (which creates new entities with an upsert), a
re-ingest's repair of the entities its old text mentioned, and a delete's
cleanup. So they wait for each other instead of deadlocking, and concurrent
work that shares entities completes, with one extraction call per batch. If
Postgres still ends a write, the engine repeats the write, never the
extraction: up to five attempts with a short random wait. Only then does the
document fail, and its `error` says so and asks for that one document to be
ingested again.

**Near-duplicate names are merged best when documents arrive one at a time.**
A name close to one already stored ("Apple Inc" beside "Apple Inc.") joins
that entity. The merge needs the other row to exist: documents that introduce
such variants at the same moment each create their own, so a corpus ingested
concurrently can end with a few more near-duplicate entities than the same
corpus ingested one document at a time. Nothing fails and every name stays
searchable; it is a difference in quality, not an error. Two names that are the
same but for their numbers are never merged ("invoice 10234" and "invoice
10235", "room 101" and "room 102" stay separate), whatever the similarity. The
merge thresholds are settings on `GraphConfig`: `entity_merge_threshold` (0.75,
the strict any-type step) and `entity_merge_fuzzy_threshold` (0.45, the
same-type step with a shared token).

A name that is not an exact or normalised match joins a stored entity only by
`entity_fuzzy_merge`: `"words"` (the default) also requires every word of one
name to appear in the other, ignoring punctuation, so "Acme Corp" joins "Acme
Corp." and "International Business Machines" joins "International Business
Machines Corp", while "Northgate Logistics" never joins "Northwind
Logistics"; `"off"` merges exact or normalised names only; `"similarity"`
merges on the scores alone. A fuzzy match is taken only from the document's
own source unless `entity_fuzzy_merge_across_sources=True`; an exact match
links across sources either way.

**Neo4j is kept in step one document at a time, and says when it is not.**
Postgres is written first and is always complete; the Neo4j write follows. A
document's sync writes only what that document brings: its chunk nodes, the
entities its chunks mention and the two ends of the relationships they attest,
those relationships, and its mention edges. Its cost follows the document, not
the corpus, and documents ingested together write few of the same nodes. A
re-ingest and a `delete_document` take the document's old chunk nodes out and
repair what they touched from Postgres: an edge takes the evidence of the
chunk that now attests it first, or is deleted when none does, and an entity
nothing mentions any more is deleted.

Each statement is retried on a transient error such as a deadlock between two
ingests. If the sync still fails, the document completes (search keeps working
from Postgres) and the gap is recorded three ways:
`report.documents[i].graph_backend_stale` is `True`, `on_error` is called with
`stage="graph_sync"` and the `document_id`, and the document's `meta_data`
carries `graph_sync_pending`. No later document's sync fills that gap.
`await engine.resync_graph()` does: it replays every recorded document from
Postgres, each with its own entities and relationships, retries every delete
Neo4j missed, and returns `{"documents", "synced", "failed"}`; pass `document_id=` or `source_id=` to
replay one document or every graph document of a source (an id that names no
document raises `KeyError`; a document that was not ingested in graph mode has
nothing to replay). It is idempotent, calls no model and bills nothing. Like
`rebuild_communities` it is a maintenance call for the host: it takes no
`principals`, and no router, tool or MCP action reaches it.

**An edge keeps the first evidence, whatever order documents arrive in.**
An edge in Neo4j carries the evidence of the chunk that attested it first, as
Postgres orders them. Each document sends its own evidence with that order,
and Neo4j keeps the earliest it has been shown, compared under the edge's
lock. So documents ingested
together end with the same evidence as documents ingested one at a time, and
a document replayed after an outage takes back an edge that a later document
wrote in the meantime.

**A delete that Neo4j missed is retried.** When `delete_document`'s Neo4j step fails
(`on_error`, `stage="delete_document_graph"`), the removal is recorded in Postgres
(`context_engine_graph_removals`, created by `context-engine migrate --graph`), and the next
`resync_graph()` retries it and reports `"removals": {"done", "failed"}`. Until then Neo4j still holds that
document's chunk nodes, with their short previews of the deleted text; no
search returns them, because every id is checked against Postgres. The
record keeps the entities and relationships the document had in Postgres, so
the retry repairs them even when Neo4j never received the document's chunks.
On a database without that table the engine logs one warning and attempts
each removal without a record. It looks for the table once per process and
remembers the answer, so run `context-engine migrate --graph` before starting
workers: a worker started before the migration keeps skipping the record
until it restarts.

**An ingest outside graph mode reads Neo4j too.** A graph document ingested
again outside graph mode (as `hybrid`, or as a spreadsheet), or completed that
way by `resume_embeddings` or `resume_documents`, is taken out of Neo4j the
same way. To find out, every ingest outside graph mode in a deployment with
`neo4j_uri` set makes one indexed read in Neo4j. It finds nothing for a
document that never had a graph, and when Neo4j is slow or down it waits as a
graph ingest does. The ingest still completes: a failure is reported through
`on_error` with `stage="graph_sync"` and retried by `resync_graph()`.

One case is left to the full replay. A graph document whose sync failed, so
that Neo4j never received its chunks, can still be handed an edge when the
document that first attested the edge is deleted. If that document is then
ingested again outside graph mode before any replay, nothing in Neo4j and
nothing in that ingest names the edge, so it stays, with its two entities if
nothing else mentions them. Only `await engine.resync_graph(full=True)` removes it.

**The full replay.** `await engine.resync_graph(full=True)` replays the whole
corpus (every entity, every graph document, every relationship), then removes
from Neo4j what Postgres no longer has, and adds
`"removed": {"chunks", "relationships", "entities"}` to its result. Its cost
follows the corpus, read a page at a time. It is the one path for what a
document's own sync does not carry:

- a relationship stored before evidence chunks were recorded belongs to no
  chunk, so no document's sync writes it. Run one full replay after upgrading;
- a longer spelling of an entity reaches Neo4j with the document that gave
  it, provided that document's sync succeeds or it mentions the entity in its
  text. Otherwise it arrives with the next document that mentions the entity,
  or with a full replay.

If a step that belongs to no single document fails (the entities, the
relationships, the final removal), the replay stops, reports it once through
`on_error` with `scope="corpus"`, and returns it as `"error"`; only documents whose
own write failed are marked for a re-sync.

Entity nodes carry `name` and `type` only. `frequency` and every other count
stay in Postgres, where search reads them, so nothing derived can drift.
Neo4j 5.1 or later is supported (tested on 5.1 and 5.26); from 5.23 the server logs a deprecation
notice for the `CALL { WITH ... }` subqueries the engine sends, which changes
no result.

**An entity is named from what the caller can read.** One entity row serves
every document that names it, across sources and access groups, so one
source's documents can influence how another's are ranked; no text crosses
that way. Each mention records the spelling found in its chunk, and every
graph answer (neighbours, walks, relationships, query entities and the names
a community summary is written from) names an entity by the longest spelling
among the mentions the caller may read. A restricted document's spelling
never reaches a caller who cannot read it. For a graph built before 1.0.0 the
stored name is shown only when every mention is readable or a readable chunk
contains it, otherwise the lower-cased key. Run
`await engine.resync_graph(full=True)` once after upgrading: it records every
mention's spelling from its own chunk's text (pure SQL, no model call, with or
without Neo4j) and reports how many as `"spellings"`. An extracted entity
whose type is outside the seven the extraction schema allows is kept, as
`CONCEPT`, rather than costing its passage.

**Communities belong to one source.** Entities are shared across the database
by name, but a community is detected, summarised and read inside one source,
and a graph ingest rebuilds (and bills for) its own source only. For a source
that gets no new upload, `await engine.rebuild_communities(source_id)`
(`None` for documents ingested without a source) rebuilds once, metered to
`on_usage` as graph units with `detail={"operation": "rebuild_communities"}`.

**Community summaries are opt-in.** `GraphConfig.community_summaries`
(default `False`) asks the model for a short summary of each detected
community and embeds it. Detection itself (Louvain, no model call) always
runs. Summaries cost one model call plus one embedding batch per source on a
graph ingest or `rebuild_communities` that finds communities without one, and
they help questions about the whole corpus ("what are the main themes
here?"), not questions about a named thing. With the flag on:

- a community keeps the summary it already has only while nothing in its
  input changed: the same entities with the same types and mention counts,
  and the same relationships, exactly as the summaries prompt would carry
  them. A later graph ingest into its source, `rebuild_communities` and
  `resume_documents` ask only for communities that are new or whose input
  changed. A kept summary whose vector is missing (its embedding failed
  earlier) is embedded again by the source's next graph ingest or
  `rebuild_communities`, not by `resume_documents`, which only visits a source
  with a community still waiting for its summary; that costs an embedding
  and bills no community unit;
- a reply that summarises only some of the asked communities keeps the ones
  that came back, and the rest stay pending for the next `resume_documents`;
- community units are billed for the communities that got a summary in that
  pass, at ingest, `rebuild_communities` and `resume_documents` alike: a
  summaries call that fails bills no community unit (its tokens, when the
  provider reported any, are still in `provider_tokens`), and the pass that
  later fills those communities bills them once.

With the flag off:

- communities are stored without a summary or an embedding, and no model is
  asked for them at ingest, at `rebuild_communities` or at `resume_documents`;
- `resume_documents` skips its community pass, so those rows are never
  pending;
- the graph leg's `community_match` signal contributes nothing, and the
  `community_summary` action is advertised as off: `discover` marks it
  `(OFF)` and never suggests it, and a call is refused with an error naming
  `graph.community_summaries`;
- no community unit is billed: a graph ingest bills its entity batches
  alone, and `rebuild_communities` bills 0 units;
- the next graph ingest or `rebuild_communities` of a source drops that
  source's existing summaries, since its communities are stored again
  without one.

Turn it on later and the next `resume_documents` summarises every source
whose communities have no summary, or call `rebuild_communities(source_id)`
for one source. Community units are billed only for communities actually
summarised, so turning the flag off and on again never bills a source
twice.

**Deleting a document repairs the graph it contributed to.** After the
document and its chunks are gone, `delete_document` recomputes `frequency`
(the number of distinct chunks that mention an entity) for the entities it
mentioned, deletes the ones nothing mentions any more, and deletes the
communities that touched them in the document's own source; the next graph
ingest or `rebuild_communities` rebuilds them. Neo4j follows after the
Postgres commit: the document's chunk nodes go, an edge only it attested is
deleted, and an entity nothing mentions any more is deleted. A Neo4j failure
never fails the delete: it reaches `on_error` with
`stage="delete_document_graph"`.

Four knobs shape the leg, all on `GraphConfig`: `rerank_weights`
(`vector_score` 0.3, `entity_match` 0.3, `relationship_relevance` 0.2,
`community_match` 0.1, `graph_connectivity` 0.1; a partial dict overrides only
the keys it names, an unknown key raises), `max_query_entities` (30),
`entity_match_threshold` (0.6; the fuzzy fallback's floor, `None` switches it
off) and `entity_max_name_words` / `entity_min_name_chars` (5 / 3). Every
graph answer carries the entity `id` beside the name and `entity=` accepts
one: the handle that survives an output redaction policy, since a masked name
matches nothing. Ids are stable while the entity exists; rebuilding the graph
mints new ones.

## Compute over tables

`engine.compute()` turns the in-scope CSV/XLSX documents into dataframes and
has the model write code against them; `compute_over_frames()` is the same
path for frames you already hold, with no document to point at:

```python
from context_engine import compute_over_frames

out = await engine.compute("total the amount column", source_ids=["finance"], principals=caller)
out = await compute_over_frames({"expenses": df}, "total the amount column", config=engine.config)
out["result"]            # plain data: numbers, strings, lists of row dicts
out["attempts"]          # 1, or 2 when the model got its one rewrite
out["provider_tokens"]   # summed over both calls
```

Both are **off by default** (`enable_code_execution=True` turns them on) and
both apply `config.redaction` before the model sees anything: every string
cell and every column header is masked, and the returned dict, generated code
included, is swept on the way out. The prompt states what the sandbox refuses,
and code the static check refuses buys one rewrite; a frame key that does not
exist buys the same one rewrite (`no frame named 'revenue.xlsx'; available:
[...]; did you mean 'revenue.xlsx:Q1'?`), and no other runtime error is
retried. `on_error` fires once per failed attempt with `stage="compute_codegen"`
and the attempt number.

**What the frames are called.** Over one document in scope the key is the
sheet name; over several, a one-sheet document is keyed by its document name
(`dfs['revenue.xlsx']`) and a multi-sheet one by `<name>:<sheet>`, with `#2`,
`#3` appended on a collision. The keys depend on the document set, never its
order, so `discover` reports the same `frame_key` that `compute` builds.

The in-process sandbox behind `enable_code_execution` is defence in depth, not
a security boundary. To run the generated code somewhere actually isolated (a
subprocess, a container, gVisor, WASM) pass `code_runner=` to `ContextEngine`
or to either call: a callable that runs the same `(code, context, timeout)`
shape out of process (`context_engine.CodeRunner` names it). That hook is the
real isolation boundary; its result is trusted only as far as JSON. In-process,
`out["result"]` is converted to plain data (a DataFrame to row dicts, with
`{"columns", "data"}` when column labels repeat; a Series to a dict; numpy
scalars to numbers; dates to ISO strings) inside the runtime guard, so nothing
the generated code left attached to the result can run afterwards.

## Schema-enforced replies

Every place the engine asks a model for JSON (community summaries, entity
extraction, vision page transcription, structured extraction) sends a JSON
Schema in the **portable subset**, the intersection of every provider's strict
mode: objects, arrays, strings, numbers, integers, booleans, nulls and `$ref`
into a root `$defs`; nullable as a type array; every object with
`additionalProperties: false` and all properties required; enums of scalars;
no `minItems`/`maxItems`, `pattern`/`format`, `anyOf`/`oneOf`/`allOf` or
recursion. `strict_portable()` (`context_engine.llm_schemas`) is the linter
and raises `SchemaError` before any request goes out. A host model receives
it as `request.response_schema`, ready for a provider that enforces one.

**`mode` is the guarantee, and it is on every reply.** `native` and `tool`
mean the provider enforced the schema, so a reply that still breaks it is
refused rather than re-asked. `prompt` means the shape is only asked for in
words: the reply is validated locally, and one failure buys exactly one retry
carrying the validation error and the model's own answer back. A second
failure is loud. `map_reduce` is the one JSON call that sends no schema, since
its shape is the caller's. `structured_repair_count()` says how many
enforced-mode replies needed the prose helper this process; it should stay at
zero.

Ask for a shape of your own with `structured_call`:

```python
from context_engine.providers.structured import structured_call

result = await structured_call(
    config.llm,                       # or client=context_engine.model.model_client(my_model, "map_reduce")
    system="Extract the invoice fields.",
    user=text,
    schema={"type": "object", "properties": {"total": {"type": ["number", "null"]}}},
    schema_name="invoice",
)
result.value       # validated against the same bytes sent to the provider
result.mode        # "native" | "tool" | "prompt"
result.attempts    # 2 means the prompt path needed its one retry
result.tokens      # summed over every attempt
```

It returns a `StructuredResult` or raises `StructuredCallError`
(`StructuredRefusal` when the provider refused in the field it reserves for
saying so), both carrying `mode`, `attempts`, `tokens` and `raw`, the full
final reply; read `raw` deliberately and never dump the exception into a log.
For a shape the subset cannot describe (keys that come from the document) pass
`enforce=False`: JSON mode goes on the wire and the reply is validated against
the schema as written. `on_error` reports `structured_call_attempt` (a failed
attempt, a retry follows) and `structured_call` (terminal) from the wrapper,
plus `entity_extraction` and `structured_extraction` at the call sites, so a
host sees attempts and outcomes as different events.

## Serving over HTTP

The same document/search/stats surface ships as a ready adapter for FastAPI,
Flask and Django, with identical routes (`POST/GET/DELETE/PATCH /documents`,
`POST /search`, `GET /stats`) over one framework-neutral core. `auth` and
`principals` are required: the package implements no auth, and resolving who
the caller is has no safe default. To run a mount open on purpose, say so with
`principals=lambda: []` (anonymous); returning `TRUSTED` skips filtering
entirely; returning `None` is rejected with a 500 so a missing resolver can
never open a mount by accident. `GET /stats` counts only what the caller may
see.

```python
from context_engine import create_router                                    # [fastapi]
app.include_router(create_router(engine, auth=my_auth, principals=my_principals), prefix="/context")

from context_engine import create_flask_blueprint                           # [flask]: auth(request) / principals(request)
app.register_blueprint(create_flask_blueprint(engine, auth=my_auth, principals=my_principals), url_prefix="/context")

from context_engine import create_django_urlpatterns                        # [django]
urlpatterns = [path("context/", include(create_django_urlpatterns(engine, auth=my_auth, principals=my_principals)))]
```

`create_tools_router` (FastAPI) serves the tool plane the same way, with an
`approval_scope` resolver beside `principals`. Not on any of them? Call
`engine.ingest()` / `engine.search()` from your own views; the engine needs no
web framework at all.

## Usage metering and hooks

```python
from context_engine import ContextEngine, UsageEvent

def on_usage(event: UsageEvent) -> None:
    # event.kind: "ingest" | "search" | "tool" | "map_reduce" | "compute"; event.units; event.detail;
    # event.provider_tokens: {"embedding_tokens", "llm_input", "llm_output"}
    meter.record(event)

engine = ContextEngine(
    config,
    on_usage=on_usage,                                  # the metering seam; the package never touches billing
    on_error=lambda exc, ctx: log.error(ctx["stage"], exc_info=exc),
    on_progress=lambda event: ui.update(event["document_id"], event["stage"], event["state"]),
    on_elicitation=ask_user,                            # a third-party MCP server's question (Tools)
)
engine.hooks.on_tool_call = lambda event: audit.append(event)   # the audit row of each tool call
```

A hook that raises is caught and logged, never raised into the engine. Every
`UsageEvent(kind="ingest")` carries in its `detail` the `mode` the document
is processed at (a spreadsheet asked for `graph` bills as `hybrid` with
`mode_reason="tabular documents stay hybrid"`) and, for a DOCX, `pages` from
the file's own page count with `pages_source` saying where it came from. A
host model's reported `tokens` reach `provider_tokens` unchanged; the engine
never counts for you. Search bills one unit per scope; graph units count only
when the leg contributed. A graph ingest bills `ceil(chunks / 8)` entity units
plus one unit per primary community it summarised in that pass, so a
failed summaries call, a community that kept an earlier summary, and
`graph.community_summaries` off all bill no community unit. `map_reduce` and
`compute` (`compute_over_frames` included) emit one event per run, whichever
door called: `units` is the number of model calls the run made (one per
document for `map_reduce`, one or two for `compute`) and `provider_tokens` is
the result's own.

## The CLI

```bash
context-engine migrate --database-url URL --dim 1536 [--graph]           # create or upgrade the schema
context-engine mcp --database-url URL [--host 127.0.0.1] [--port 8080] [--graph]   # serve the MCP surface, anonymous, loopback
context-engine check-acl-exposure --database-url URL [--samples 100] [--top-k 10] [--max-scopes 8]
context-engine bm25-enable --database-url URL    # opt in to BM25 statistics (bm25-disable, bm25-compact)
context-engine install-skill [--global] [--dir DIR] [--print] [--force]  # teach your coding agent the rules
```

`mcp` builds its config from the same `CE_*` variables and is unauthenticated,
answering as an anonymous caller over the whole corpus; it binds loopback for
that reason and warns when widened. `check-acl-exposure` reports, for the ACL
values that occur in your corpus, what a bare scoped query returns versus what
this library returns, both scored against an exact oracle computed inside
Postgres.

### Teach your coding agent the access-control rules

Most integration code is written with an agent's help, and the costliest
mistake in this library is one an agent makes readily: reading `principals`
of `None` as "nobody is logged in" and returning the entire corpus. The
package ships a skill stating that and the other invariants worth not
guessing (principals are injection-only on remote surfaces, `auth` and
`principals` are required on every router factory, `scope` is required on
the knowledge tool, `acl=None` unrestricts on update, pgvector 0.8 is a hard
floor, code execution is off by default):

```bash
context-engine install-skill            # -> ./.claude/skills/context-engine/
context-engine install-skill --global   # -> ~/.claude/skills/
context-engine install-skill --print    # stdout, for agents that read other formats
```

It is an explicit command, never an install hook: the file becomes
instructions inside your agent, so it should be your decision. An edited skill
is never overwritten without `--force`.

## Reference

### `ContextEngine(config, **seams)`

| argument | type | what it does |
|---|---|---|
| `config` | `ContextEngineConfig` | required; every setting, see [Configure](#configure) |
| `model` | `(ModelRequest) -> ModelReply \| str`, sync or async | every model call the engine makes; `request.purpose` says which. With it, `llm`, `vision_llm` and `graph.extraction_llm` are optional |
| `embedder` | `(texts, *, kind) -> (vectors, tokens)`, or an object with that `embed` | every embedding request; `config.embedding` stays the identity the database checks |
| `reranker` | `(query, docs) -> list[int]` | reranks the widened candidate window on every search |
| `retrieval_legs` | `{name: (query, *, limit, scope) -> list[str]}`, sync or async | extra retrieval legs, re-filtered by permissions before fusion; see [Plug in your own retrieval](#plug-in-your-own-retrieval) |
| `fusion` | `(ranked, weights, k) -> list[str]`, sync or async | replaces reciprocal rank fusion; can only order ids a leg produced |
| `graph_backend` | `GraphBackend` | supplies the raw graph (`expand_candidates`, `chunk_degrees`); the engine applies scope and caps |
| `on_usage` | `(UsageEvent) -> None` | one event per metered unit of work |
| `on_error` | `(Exception, dict) -> None` | errors the engine handled or degraded around; `ctx["stage"]` names where |
| `on_progress` | `(dict) -> None` | ingest stage boundaries, plus per-batch embedding progress |
| `code_runner` | `(code, context, timeout) -> dict`, sync or async | where `compute` runs generated code; `None` is the in-process sandbox |
| `http_client_factory` | `(purpose) -> httpx.AsyncClient \| None` | your own long-lived client per outbound purpose (`llm`, `embeddings`, `reranker`, `tool_http`, `mcp_oauth`) |
| `mcp_resolver` | `async (host, port) -> list[str]` | your own name resolution for MCP connections; answers are still address-checked |

The engine also exposes `config`, `hooks` (set `on_tool_call` there),
`backend`, `session_factory`, `embedder` (built on first use), `model`,
`reranker`, `retrieval_legs`, `fusion` and `graph_backend`, plus
`with_redaction(policy, secret_key=...)`,
`resume_documents(document_ids=None, *, content_source=None)`,
`execute_tool(call_name, args, *, principals, source_id=None, actor=None,
source="api", approval_scope=None, result_max_chars=8000,
result_max_rows=100, response_mode=None, redaction=None, secret_key=None,
tool_id=None, overrides=None)` and `aclose()`.

### `ContextEngineConfig`

| field | type | default | what it does |
|---|---|---|---|
| `database_url` | `SecretStr` | required | Postgres URL, handed to SQLAlchemy; it carries the password, so it is masked in `repr` and dumps |
| `embedding` | `EmbeddingConfig` | required | `provider` (`openai` / `azure_openai` / `custom`), `model`, `dim`, `api_key`, `base_url`, `max_retries`, the packing knobs |
| `llm` | `LLMConfig \| None` | `None` | the built-in model for extraction, `compute`, `map_reduce` and `query_meta`. Unused with a host `model` |
| `vision_llm` | `LLMConfig \| None` | `None` | the built-in model that transcribes scanned pages and images. Unused with a host `model` |
| `default_mode` | `"hybrid" \| "graph"` | `"hybrid"` | the mode `ingest()` uses when a call names none |
| `graph` | `GraphConfig` | disabled | `enabled`, `neo4j_uri`/`neo4j_user`/`neo4j_password`/`neo4j_database`, `neo4j_connection_timeout_s` (10), `neo4j_connection_acquisition_timeout_s` (15), `extraction_llm`, `rerank_weights`, `max_query_entities`, `entity_match_threshold`, `entity_max_name_words`, `entity_min_name_chars`, `entity_merge_threshold`, `entity_merge_fuzzy_threshold`, `entity_fuzzy_merge` (`"words"`), `entity_fuzzy_merge_across_sources` (`False`) |
| `tools` | `ToolsConfig` | defaults | `mcp` (connection settings), `db.query_timeout_s` (30; `None` lifts it), `audit_on_failure`, `audit_cancel_wait_s` |
| `reranker` | `RerankerConfig` | `candidates=50` | the widened candidate window a host reranker sees |
| `fusion` | `FusionConfig` | RRF, `k=20` | `weights` per leg: `fts` 1.0, `trgm` 0.4, `ann` 1.0, `graph` 1.0; `0` skips the leg |
| `search` | `SearchConfig` | floor off, keyword leg `"all"` / `ts_rank_cd` | `vector_floor` (`"adaptive"`), `trgm_limit`, `lexical_match` (`"all"`, `"any"`), `lexical_rank` (`"ts_rank_cd"`, `"ts_rank"`, `"bm25"`), `bm25` (`k1` 1.2, `b` 0.75, `idf_scope` `"source"`), `lexical_normalization` (0 to 63), `lexical_stopwords`, `trgm_max_query_words`, `lexical_candidates`, `plugin_timeout_s` (10 seconds when unset), `query_embed_retry_after_cap_s` (5), `query_embed_max_retries` (1) |
| `extraction` | `ExtractionConfig` | defaults | `max_render_px`, `vision_max_output_tokens`, `pages_per_vision_batch`, `vision_missing_page_retry`, `ocr_language` |
| `ingest` | `IngestConfig` | defaults | `max_file_bytes`, `max_rows`, `max_document_text_chars`, `progress_batches`, `reread_unreadable_pages`, `resume_stale_seconds`, `max_park_seconds` |
| `storage` | `StorageConfig` | defaults | `ann_exact_threshold`, `write_batch_rows`, `pool_size`, `max_overflow`, `pool_timeout`, `pool_recycle`, `pool_pre_ping` |
| `enable_code_execution` | `bool` | `False` | lets `compute()` run model-written code; turn on only inside OS-level isolation |
| `redirect_aggregates_to_compute` | `bool` | `True` | the knowledge tool refuses aggregate questions over spreadsheets and points at `compute` |
| `filename_search` | `bool` | `True` | a question naming a file searches document names first |
| `secret_key` | `SecretStr \| None` | `None` | base64url 32-byte AES key for tool configs holding secrets |
| `redaction_secret_key` | `SecretStr \| None` | `None` | the HMAC key `hash` rules use; falls back to `secret_key` |
| `allow_private_egress` | `bool` | `False` | lets http/mcp tools reach private, loopback and metadata addresses |
| `redaction` | `RedactionPolicy` | empty (no-op) | span-level masking at ingest and/or output |

Every field is reachable as an environment variable with the `CE_` prefix and
`__` for nesting (`CE_GRAPH__EXTRACTION_LLM__MODEL`).

## Contributing

Issues and discussions are welcome. Pull requests from forks are closed
automatically for now, so to propose a change, open an
[issue](https://github.com/promptev/context-engine/issues/new/choose) and paste
the patch there if you have one; see
[CONTRIBUTING.md](https://github.com/promptev/context-engine/blob/main/CONTRIBUTING.md).

## Deep dives

The full guide is at **https://promptev.ai/documentation/context-engine/**.
The exhaustive design notes (redaction apply points, `hash` joinability,
overlapping-rule merge semantics, ACL edge cases, graph internals, cost model,
known limitations) live in the module and config docstrings (`config.py`,
`redaction.py`, `engine.py`, `model.py`).
