Metadata-Version: 2.4
Name: annotations-orchestration
Version: 0.2.0
Summary: Generic LLM evaluation orchestration: resumable parallel runs, manifest-driven experiments, span metrics via annotations-evaluate
Keywords: llm,evaluation,orchestration,ner,nlp,benchmark
Author: Nicole Dresselhaus
Author-email: Nicole Dresselhaus <nicole.dresselhaus@hu-berlin.de>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: annotations-evaluate>=0.1,<1
Requires-Dist: pyyaml>=6.0.2
Requires-Dist: rich>=13.9.0
Requires-Python: >=3.11, <4.0
Project-URL: Repository, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration
Project-URL: Issues, https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/issues
Description-Content-Type: text/markdown

# annotations-orchestration

Generic Python library for orchestrating LLM evaluation runs: a **resumable
parallel scheduler** over experiment manifests, plus **span metrics** for
precision/recall/F1 against gold annotations — computed by
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate),
the family's spec-driven metrics library.

Part of the `annotations-*` family: the annotation pipeline lives in
[`annotations4all`](https://scm.cms.hu-berlin.de/annotations4all/annotations4all),
this library provides the evaluation/experiment infrastructure, and concrete run
projects combine both.

## Features

- **Library-agnostic**: the scheduler speaks a subprocess CLI contract
  (RunSpec → flags), not annotations4all types. Reusable for any LLM
  evaluation workflow.
- **Conservative concurrency**: documents within a run are processed
  linearly; `remote` runs share a pool (`execution.remote_parallelism`,
  default 3); `self_hosted` runs get one sequential lane per host.
- **Resumable**: run state is persisted; completed runs are skipped,
  interrupted/failed runs re-execute on restart.
- **Manifest-driven and validated**: experiment YAMLs + optional controller
  manifests (`defaults`/`execution`/`order`). Run fields (`model`, `host`,
  `port`, `context`, `shots`, `data_dir`, `net`, `api_url`, `api_key_env`,
  `reasoning`, `tagger`, `nested`) are checked for type and allowed values, and
  **unknown keys are rejected** instead of being ignored silently — with file,
  key and the list of allowed keys (plus a typo suggestion); errors are one line
  (`error: …`, exit code 2) instead of a traceback.
- **Metrics**: span evaluation with precision/recall/F1, provided by
  `annotations-evaluate` — the SemEval-2013 compatible flat schemes (strict,
  entity type, partial, exact) used for the run reports, and that library's
  nested standard report (levels, structure, decode rate) for nested
  annotations.

## Installation

```bash
python -m pip install annotations-orchestration
```

Requires Python 3.11+.

The metrics library of the family,
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate),
is installed as a regular dependency from PyPI.

## Quickstart

```bash
# Validate an experiments directory (dry run, no execution)
annotations-orchestrate --experiments-dir experiments/ --dry-run

# Run the experiments
annotations-orchestrate --experiments-dir experiments/
```

### Python API

```python
from pathlib import Path

from annotations_orchestration import calculate_metrics, evaluate_classifier

# One record in the harness format: gold spans per annotation tool, predicted spans.
records = [
    {
        "ground_truth": {"prodigy": [{"start": 0, "end": 3, "label": "PER"}]},
        "predictions": [{"start": 0, "end": 3, "label": "PER"}],
    }
]

# Pass records in memory, or omit "predictions" to read a run directory.
overall, per_tag = evaluate_classifier(
    data_dir=Path("data"),
    eval_dir=Path("results"),
    predictions=records,
)
f1 = calculate_metrics(overall, "ent_type")["f1"]  # 1.0
```

## CLI

`annotations-orchestrate` options: `--manifest`, `--experiments-dir`,
`--state`, `--results-dir`, `--dry-run`, `--active-self-hosted`,
`--graceful-shutdown-seconds`.

## Development

```bash
uv sync
uv run ruff check .
uv run ruff format --check .
uv run pytest
```

## Citation

If you use this software in academic work, please cite it as follows:

> Dresselhaus, Nicole. (2026). *annotations-orchestration* (Version 0.2.0) [Software]. Humboldt-Universität zu Berlin. <https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration>

DOI: `10.5281/zenodo.22797759`

Machine-readable metadata is available in [`CITATION.cff`](https://scm.cms.hu-berlin.de/annotations4all/annotations-orchestration/-/blob/main/CITATION.cff).

## License

MIT.

The metrics are computed by
[`annotations-evaluate`](https://scm.cms.hu-berlin.de/annotations4all/annotations-evaluate)
(MIT, same family) — the earlier vendored snapshot of
[MantisAI/nervaluate](https://github.com/MantisAI/nervaluate) was removed in
GitLab issue #3.
