Metadata-Version: 2.4
Name: osp-provider-runtime
Version: 0.16.1
Summary: Thin runtime harness for OSP providers (RabbitMQ transport + contract execution).
Author: OSP Team
License: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.13
Requires-Dist: loguru<1,>=0.7
Requires-Dist: osp-provider-contracts==0.18.0
Requires-Dist: pika<2,>=1.3
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: hatch<2,>=1.14; extra == 'dev'
Requires-Dist: pytest<10,>=9.0.3; extra == 'dev'
Requires-Dist: ruff>=0.15; extra == 'dev'
Requires-Dist: twine<7,>=6; extra == 'dev'
Requires-Dist: ty>=0.0.18; extra == 'dev'
Requires-Dist: uv<0.14,>=0.13.4; extra == 'dev'
Description-Content-Type: text/markdown

# osp-provider-runtime

Thin, boring runtime harness for OSP providers.

This package handles RabbitMQ message plumbing so provider implementations can
focus on business logic.

## What it does

- Parses a versioned request envelope.
- Builds provider `RequestContext`/`ProviderRequest` and calls `execute(...)`.
- Dispatches optional read-only provider assessment RPCs without task execution.
- Emits canonical asynchronous task updates and synchronous control-RPC replies.
- Applies explicit ack/requeue/dead-letter decisions.
- Emits structured logs for delivery decisions.
- Supports explicit runtime knobs for prefetch/concurrency/retries/transport timeouts/DLQ.
- Accepts wire-version-9 request envelopes with trusted family Role, sudo, task identity
  and cleared Gate fingerprints separated from provider action payload.
- Provides `ProviderIdentity` so providers derive the same RabbitMQ
  queue/binding, routing prefix, and lane name from `PROVIDER`,
  `PROVIDER_INSTANCE`, and optional `PROVIDER_INSTANCE_ID`.

## Provider identity

Use a provider family for permissions and API requests, then add an instance
only when you need a separate runtime lane:

```python
from osp_provider_runtime import ProviderIdentity

identity = ProviderIdentity(
    provider="nrec",
    instance="pr",
    instance_id=37,
)

identity.routing_prefix   # "nrec.pr.37"
identity.request_queue    # "provider.nrec.pr.37"
identity.request_binding  # "nrec.pr.37"
```

The common case stays small: `ProviderIdentity("vmware", "dev")` gives the
`vmware.dev` routing prefix. Instance IDs refine routing within a lane.

## Result payload conventions

Provider results should keep `ProviderResult.data` focused on the **resolved**
values for the task. The runtime builds the full update payload envelope:

- `success`: provider outcome flag; `false` still represents a completed no-op
  or degraded outcome
- `resolved`: your `ProviderResult.data` (after runtime normalization)
- `provenance`: optional; extracted from `ProviderResult.data["provenance"]`
- `produced_resources`: typed provider-native references produced by the operation

Raise a `ProviderError` when the task should enter a failed lifecycle state.
`ProviderResult(success=False)` does not turn a completed outcome into a task
failure.

Special keys in `ProviderResult.data`:

- `progress_events` (list): lifted into the update payload and removed from
  `resolved` to avoid duplicate TaskEvent rows.
- `provenance` (dict): lifted into the top-level `provenance` field.

Providers may opt into live updates by accepting a keyword-only `progress`
argument on `execute`. Use `ProgressReporter` so the same action also works
without live delivery:

```python
from osp_provider_contracts import ProducedResource, ProviderResult
from osp_provider_runtime import ProgressReporter


def execute(self, request, context, *, progress=None):
    reporter = ProgressReporter(progress)
    reporter.report("Creating virtual machine", {"stage": "openstack_create"})
    vm = create_virtual_machine(request)
    return ProviderResult(
        success=True,
        message="Virtual machine created",
        produced_resources=(ProducedResource(kind="vm", target_ref=vm.id),),
        data={"progress_events": reporter.events},
    )
```

With a callback, the event is published live and `reporter.events` stays
empty. Without one, the event is returned through `progress_events` and the
runtime expands it from the terminal update. This keeps each event on one
delivery path. The callback is best-effort and transport-owned. `request.action`
is the sole action source.

## Cancellation

Cancellation RPCs (`.cancellation.rpc`) use their own broker-named exclusive
queue, channel and one-worker executor per connection, so a
slow resource listing or console read never delays a cancel.

- **Not started:** a task delivered to this runtime but still waiting for a
  task worker is revoked by the runtime itself; no provider hook is needed.
  The RPC answers `supported=true` with
  `cancellation={"cancellable": true, "requested": true}`. When a worker picks
  the delivery up it does not call `execute()`: it publishes the normal
  terminal cancelled update (`TaskReporter.STATUS_CANCELLED`) and ACKs the
  delivery.
- **Running:** stopping live work needs provider support. The runtime forwards
  to the optional `task_cancellation(task_ref)` / `cancel_task(task_ref)`
  hooks, which recheck backend state and stop the work (cooperative
  checkpoints or a native backend cancel). Providers without the hooks report
  unsupported cancellation.
- **Unknown to this runtime** (still in the broker, or held by another
  replica): treated like a running task.

`requested=true` only records the stop request. The task's terminal update is
the confirmation: a revoked task's update follows once a task worker frees up,
and a running task must finish with `TaskCancelled(ProviderResult(...))` after
confirmed termination, retaining any surviving resource references.

Providers can reuse `osp_provider_runtime.cancellation.CancellationController`
for cooperative execution. Keep one controller in the provider's running-task
registry, enter `cancellation_scope(controller)` around execution, and delegate
status/request hooks to it. A request remains accepted until execution ends,
independently of native task handles. `task_step()` checks the controller before
starting an operation; polling loops call `cancellation_checkpoint()` or use
`cancellation_wait(seconds)`. The checkpoint raises `TaskCancelled`; the owning
workflow adds compensation results and surviving resource references to its
`ProviderResult`. Run compensation with `cancellation_scope(None)` so a pending
request cannot interrupt cleanup. Context propagates through `asyncio.to_thread`.
A native adapter must settle an in-flight operation before allowing a checkpoint
to unwind its accepted resource identity.

Revocations live in memory. A revoked result enters the idempotency cache, so
a redelivery of the same dispatch replays the cancellation. After a restart,
unacknowledged deliveries return and run normally; the orchestrator still
waits for a terminal update and a repeated cancel is safe.

## Consumer readiness

`osp-provider-runtime-health` exits 0 only after task, control and cancellation
consumers have all started and the connection event loop has refreshed its
receipt within 30 seconds. It exits 1 before setup, during control recovery,
while disconnected or after shutdown. Container health checks should use this
command; provider `doctor` commands validate configuration prerequisites.

The receipt defaults to `/tmp/osp-provider-runtime-health.json` inside the
container. Local processes sharing a filesystem must select distinct paths
with `OSP_RUNTIME_HEALTH_FILE`; keep receipts off persistent provider scratch.
Consumer setup failures escape to the process supervisor. Established-session
failures retain the existing reconnect and in-flight task handling.

Control and cancellation queues disappear with their connection, so requests
sent while no provider is listening are dropped. Cancellation remains
best-effort and can be requested again; only a terminal task update confirms
termination. See the [transport decision](docs/adrs/ADR-connection-owned-control.md)
for routing, rollout and replica consequences.

### Remove obsolete control queues after rollout

After each concrete DEV, PR or PROD lane is healthy on runtime 0.11.1, use
its own vhost-scoped management API to delete both legacy queues:

```text
DELETE /api/queues/{encoded-vhost}/{encoded-request_queue.control}
DELETE /api/queues/{encoded-vhost}/{encoded-request_queue.cancellation}
```

For example, a PR lane's request queue is `provider.nrec.pr.37`; delete only
`provider.nrec.pr.37.control` and `provider.nrec.pr.37.cancellation` in its PR
vhost. Repeat for every deployed lane. Confirm the new version and readiness
before cleanup. These obsolete queues retain their old bindings and would
otherwise receive duplicate RPC/cancel traffic without consumers. The runtime
must never access them during setup: a queue on an unavailable node can block
an unbind or delete just as it blocks declaration.

Log each deletion result, including an already-absent 404 or a broker timeout;
those cleanup outcomes do not invalidate the new consumer session. Escalate
unresponsive queues to Intark. Do not delete durable task or update queues.
Runtime 0.11.1 startup no longer depends on the old cancellation queue, so its
unavailability alone does not block a coordinated PROD provider restart.

## Task logging

The runtime binds the task context once around `execute()` and emits one
`task.summary` event when the delivery finishes. Provider log sinks must use
JSON serialization so these fields are retained:

- `task_ref`: stable provider execution UUID
- `dispatch_id`: one delivery attempt
- `provider` and `task_type`
- `step`, `outcome`, and `duration_ms`
- `error.type`, `error.message`, and `error.code` when applicable

Use `task_step("external.operation")` around meaningful external operations.
It records the step in the summary trail and logs a full traceback before
re-raising failures. Task code must not bind correlation fields itself.

```python
from osp_provider_runtime import task_step

with task_step("openstack.create"):
    server = openstack.create_server(...)
```

Provider data is preserved inside `result.resolved`; its keys cannot override
the shared result fields. Runtime extracts the existing `progress_events` and
`provenance` metadata into their dedicated result fields.

The orchestrator already owns the original task request. Provider updates do
not echo it; HTTP and CLI read models compose request and result when needed.

## What it does not do

- No provider framework.
- No plugin system.
- No workflow orchestration.

## Install

```bash
pip install osp-provider-runtime
```

## Development

```bash
env -u VIRTUAL_ENV uv sync --extra dev
hatch shell
hatch run dev:check
hatch run dev:build
hatch run dev:verify
```

## Runtime Knobs

Set these via `RuntimeConfig` in your provider `runtime_app.py`:

- `prefetch_count` (default `1`)
- `concurrency` (default `1`)
- `max_attempts` (default `5`)
- `idempotency_cache_max_entries` (default `1024`)
- `snapshot_cache_ttl_seconds` (default `3600`)
- `dead_letter_exchange` (optional)
- `dead_letter_routing_key` (optional)
- `heartbeat_seconds` (default `60`)
- `blocked_connection_timeout_seconds` (default `30`)

## Update Emission

Use `TaskReporter` for provider task lifecycle updates. The runtime keeps
transport details compatible with orchestrator consumers.

Docs:
- `docs/runtime-contract.md`
- `docs/provider-updates.md`
- `docs/migration-task-reporter.md`
- `docs/runtime-upgrade-checklist.md`
- `docs/release-notes-task-reporter.md`

Tag and push:

```bash
git tag v0.2.0
git push origin v0.2.0
```

### Execution build evidence

The runtime adds `payload.execution` to live and terminal task updates. Each
actual invocation gets a fresh `attempt_id`; cached duplicate deliveries replay
the original update and identity. `dispatch_id` links the invocation to its
orchestrator dispatch. Approval resumes and broker retries can run another build.

Deployment supplies `OSP_IMAGE_DIGEST` (registry manifest digest),
`OSP_GIT_COMMIT` (the image's OCI revision label), and optional
`OSP_RELEASE_VERSION` (a production release tag). Missing values are omitted,
not inferred from moving tags or a package version. The progress callback captures
the attempt evidence before it crosses worker-thread boundaries.

Providers with external data wrap execution in `data_snapshot(path)` and use
`pinned_data_dir()` in their data path resolver. The deployment publisher writes
`.osp-revision` into a retained immutable directory and switches its current
symlink atomically. The runtime records that revision only when such a snapshot
is used. Plain local working copies remain unversioned. This identifies code and
provider data, not secrets, live infrastructure, inventory, or promotion state.

`RuntimeConfig.control_concurrency` sizes the independent control-RPC worker
pool and its broker prefetch (default 1). Providers expose this as
`CONTROL_CONCURRENCY`; 4 allows overlapping reads without increasing task
execution concurrency. Delivery logs include worker wait, handler and
finalization timings, correlated with the orchestrator by `rpc_id`.

## Unexpected execution failures

`unexpected_provider_error(exc, produced_resources=...)` is the public conversion
for adapter boundaries that must attach surviving resources before re-raising.
It uses the runtime's task-step trail for `failed_step`, marks state uncertain,
and forbids automatic retries. The transport catch-all uses the same function.

## Request-scoped policy overrides

Wire 9 carries the trusted effective provider-family `role` and strict `sudo`
boolean into `RequestContext`; assessment retains both facts. Runtime checks
the manifest kind/action minimum Role before execution or assessment, including
for standalone providers. Resource list, exact-reference, console and history
RPCs carry trusted family Role in `ResourceAccessContext` and enforce kind
floors independently of ownership filters. Shared contract ordering owns the cumulative Role
comparison. Sudo remains separate from Role and cannot bypass a declared floor.
The orchestrator has already authorized its existing scoped Admin sudo session.
A provider must never recover elevation from editable action data.

Construct one `Policy(context, progress=progress)` at each provider boundary
and pass it explicitly to planners/helpers. `policy.allows(rule, satisfied,
evidence=...)` is the shared override decision: ordinary policy must pass,
while trusted sudo records the judgement and permits the attempt. Use small,
public JSON evidence; secrets and raw backend responses do not belong here.
Evidence larger than 2 KiB is represented by its digest and byte count.
`filter_plan(plan, action)` applies it to Gates and `policy_violation` entries,
preserving `invalid_value`, `missing_required` and incomplete execution plans.
Backend calls and their errors never go through `allows`.

Return `policy.result(result)` or call `policy.error(error)` before propagating
a `ProviderError`. Collected `policy_overrides` travel in result data or error
`extra`; rejected/gated update payloads retain their `extra` diagnostics too.
The orchestrator can persist audit evidence even when best-effort progress was
lost. This hook does not authorize a new sudo session, change CHECK to APPLY,
or create a durable journal across a provider crash. Standalone providers
must use this same hook; changing the SDK alone cannot update their policy.

## Reloadable provider configuration

Providers with external configuration may implement `request_snapshot()` as a
context manager. The runtime enters it before binding Role checks and keeps it
through execution, assessment or resource reads. Static providers need no hook.
The existing `capabilities(payload={"projection": "action_inputs"})` contract
supplies deterministic permissions without optional remote hint reads.

`SnapshotStore` owns atomic last-good adoption and request-local pinning. The
provider supplies `SnapshotSource(root, revision)` and a complete semantic
loader; runtime never guesses file formats or fetches configuration. Published
revisions must identify immutable complete bundles. Local fingerprints may be
used for reload detection but never claim Git execution provenance. Expected
validation failures use `SnapshotValidationError(code, location, line=...)`;
raw parser excerpts do not enter shared status or first-start tracebacks.

The runtime is the transported Role guard for native providers. Native CLI
commands call their operator pipelines directly; they are trusted operator
entrypoints, not an alternate family-authenticated API. The optional SDK also
enforces floors at its documented direct Provider methods, independently of
AMQP. See the SDK's policy-reload documentation for authoring and delivery.

### Ownership enforcement (contracts 0.17.0 / wire 11)

Before execution or assessment, the runtime checks the cumulative manifest Role
floor, then reads each existing target through `list_resources` with an exact
reference in the requested kind and `freshness="live"`. The reference must match;
`ownership` must contain the current provider facts. The canonical contracts
fingerprint must match the envelope's per-target grant. Changed facts fail with
`ConflictError`; unavailable facts fail with `DependencyError`. Ordinary callers
must share a trusted membership with each target's resolved owning netgroups.
Admin bypasses ownership and sudo overrides policy; neither bypasses freshness
or a missing resource. Identity swaps require both targets. Providers implement
no ownership guard and never resolve siteadmin authority.

The manifest action target determines enforcement: `none` skips existing-resource
checks; `resource` requires them regardless of verb spelling. The Role floor
applies to both. Every birth is contact-owned.
For VMs, `context.primary_owner_netgroup`
is an optional orchestrator-selected folder fact for placement only, never authority.
Provider code owns placement requirements.
Verb spelling grants no exemption, including an existing-resource
action named `create`. Store the selected birth team as ownership and report
current facts on reads. Trusted caller memberships are available as `context.netgroups` for provider narrowing
rules. Inventory access contexts contain only subject and Role; ownership-based
“mine” filtering belongs to the orchestrator. Console reads retain their kind
Role floor and authenticated read context.

Run the broker boundary suite against a local or disposable broker with
`OSP_TEST_RABBITMQ_URL=... uv run pytest tests/integration`. It creates unique
queues/exchanges and removes them on completion.
