Metadata-Version: 2.3
Name: evaluators
Version: 3.3.1
Summary: Various scene understanding and perception evaluation metrics.
Author: Kurt Stolle
Author-email: Kurt Stolle <kurt@computer.org>
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Requires-Dist: torch>2.7
Requires-Dist: torchvision
Requires-Dist: torchmetrics
Requires-Dist: tensordict>=0.11.0
Requires-Dist: numpy>=2.0.0
Requires-Dist: matplotlib>=3.9.0
Requires-Dist: rich>=13.9.4
Requires-Dist: pydantic
Requires-Dist: opencv-python-headless>=4.13.0.92
Requires-Dist: omegaconf>=2.3.0
Requires-Dist: hydra-zen>=0.16.0
Requires-Dist: scipy>=1.14.0
Requires-Dist: shapely>=2.0.0
Requires-Dist: torchmatch>=1.0.4
Requires-Dist: blosc2>=4.1.2 ; extra == 'blosc2'
Requires-Dist: lightning>=2.6.1 ; extra == 'lightning'
Requires-Dist: lmdb>=2.2.0 ; extra == 'lmdb'
Requires-Dist: imagecodecs>=2026.3.6 ; extra == 'tiff'
Requires-Dist: tifffile>=2026.3.3 ; extra == 'tiff'
Requires-Python: >=3.13, <3.14
Provides-Extra: blosc2
Provides-Extra: lightning
Provides-Extra: lmdb
Provides-Extra: tiff
Description-Content-Type: text/markdown

# Scalable Distributed Evaluation for Computer Vision

[![PyPI](https://img.shields.io/pypi/v/evaluators)](https://pypi.org/project/evaluators/)
[![Python Version](https://img.shields.io/pypi/pyversions/evaluators)](https://pypi.org/project/evaluators/)
[![License](https://img.shields.io/pypi/l/evaluators)](https://github.com/khwstolle/evaluators/blob/main/LICENSE)
[![Build Status](https://img.shields.io/github/actions/workflow/status/khwstolle/evaluators/ci.yml)](https://github.com/khwstolle/evaluators/actions)

Metrics like panoptic quality over video (VPQ, DVPQ), depth-aware segmentation-and-tracking quality (DSTQ), and multi-object tracking metrics (HOTA, CLEAR, IDF1) are computed over whole sequences, not independent per-frame samples: window/tube matching across consecutive frames, or track-identity consistency across an entire video. A validation loop doesn't hand a metric a whole sequence at once. It hands it one batch at a time, and a given sequence's frames can land in different steps, different dataloader workers, and different distributed ranks depending on how the loader shards them. Standard online metric aggregation, `update()` per step, reduce across ranks at `compute()`, works when each sample's contribution reduces independently of order. It breaks down when a metric needs contiguous, ordered access to every frame of a sequence before it can produce a result.

The usual workaround is to write every prediction to disk during the validation epoch and run the official reference script by hand afterward, outside the training loop. Evaluators automates that instead. Predictions still stream to a storage backend (Memory Map or LMDB) during inference, with no metric computation in the hot path. Once the epoch or run ends, the pipeline reassembles complete, correctly-ordered sequences, regardless of how their frames were scattered across steps and ranks, and computes the metric against the whole sequence automatically. Dropping `DistributedEvaluationCallback` into a PyTorch Lightning `Trainer` is the common case: whole-sequence metrics get logged like any other validation metric, with no manual step in between.

Because predictions are persisted as a side effect, the same pipeline also supports deferred, offline re-evaluation: save the config during inference, then compute later, on a different machine, without re-running the model. That capability is real and useful, particularly for reproducibility, but it follows from the design; it isn't the reason the library exists.

The key features are:

- **Whole-sequence metrics, computed automatically.**
  Handles metrics that standard per-step online aggregation cannot: window/tube-based video panoptic quality (VPQ, DVPQ, DSTQ) and identity-consistency tracking metrics (HOTA, CLEAR, IDF1), reassembled and computed correctly regardless of how a sequence's frames were split across steps and ranks.

- **Drops into a training loop.**
  `DistributedEvaluationCallback` binds to a PyTorch Lightning `Trainer` and handles the write, reassembly, and compute stages automatically. Predictions are also written to disk, which supports deferred, offline re-evaluation on stored results.

- **Distributed by design.**  
  Synchronizes metric computation across multiple nodes and GPUs using `torchmetrics`, `torchmatch` (Hungarian assignment for the tracking metrics), and custom scheduling logic.

- **Explicit Memory Schemas.**
  Uses Pydantic-based schemas to define data formats and encodings (PNG, TIFF, Raw) up front, ensuring type safety and storage efficiency.

- **Lazy loading.**  
  Supports referencing ground truth data from disk rather than duplicating it in memory caches, enabling evaluation of terabyte-scale datasets.

- **Multi-domain.**
  Includes verified implementations for:
  - Segmentation: Panoptic quality (PQ), semantic mIoU.
  - DVPS: Video panoptic quality (VPQ), depth-aware video panoptic quality (DVPQ).
  - Depth: Eigen et al. metrics (AbsRel, RMSE).
  - Tracking: HOTA, CLEAR (MOTA/MOTP), identity F1 (IDF1).
  - STEP: Segmentation and tracking quality (STQ), depth-aware STQ (DSTQ).
  - Occupancy: per-class IoU/mIoU, occupied-vs-free precision/recall/F1.
  - 3D detection: per-class/per-threshold AP and mAP (Shapely-based 3D IoU or center-distance matching), optional TP-error decomposition and AOS.
  - Vector maps: per-class/per-threshold AP and mAP (Chamfer or Fréchet matching), optional topology accuracy.
  - Sensor simulation: paired image/point-cloud/radar quality, plus single-stream self-consistency (temporal, cross-modal).

- **CLI.**
  A command-line interface to inspect, index, and query saved inference results.

---

## Installation

```shell
pip install evaluators
```

## Quick start

### In the loop (PyTorch Lightning)

This is the common case: a whole-sequence metric (here `SemanticMetric`; see
`examples/03_video_dvps.py` for the DVPQ case this pipeline exists for) is
declared on the `LightningModule`, and `DistributedEvaluationCallback` handles
writing, sequence reassembly, and computation automatically at the end of
validation.

```python
import lightning as L
from tensordict import TensorDict
from evaluators import MemorySchema, MetricStream, TensorField
from evaluators.integrations.lightning import DistributedEvaluationCallback
from evaluators.metrics.domain.segmentation import SemanticKeyMap, SemanticMetric


class SegmentationModel(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = ...  # the actual model

        keys: SemanticKeyMap = {
            "pred": "pred_sem",
            "target": "target_sem",
            "sequence_id": "sequence_id",
            "frame_index": "frame_index",
        }
        self.metric = SemanticMetric(key_map=keys, num_classes=19, ignore_index=255)

        # Both prediction and ground truth are written to storage here; see
        # examples/02_lazy_loading.py for fetching ground truth from the
        # dataset instead, via `target_source`.
        schema = MemorySchema(fields={
            "pred_sem": TensorField(dtype="int64", shape=(1024, 2048)),
            "target_sem": TensorField(dtype="int64", shape=(1024, 2048)),
            "sequence_id": TensorField(dtype="int64", shape=()),
            "frame_index": TensorField(dtype="int64", shape=()),
        })
        self.eval_stream = MetricStream(metrics=[self.metric], name="semantic", schema=schema)

    def validation_step(self, batch, batch_idx):
        # `batch` already provides "target_sem", "sequence_id", and
        # "frame_index" from the dataset.
        preds = self.layer(batch["image"]).argmax(dim=1)
        batch["pred_sem"] = preds
        # A leading time dimension is required even for single-frame batches.
        self.eval_stream.update(
            TensorDict(batch, batch_size=[preds.shape[0]]).unsqueeze(1)
        )


trainer = L.Trainer(callbacks=[DistributedEvaluationCallback(output_dir="./eval_cache")])
trainer.fit(SegmentationModel(), datamodule)
```

Results are logged to the trainer's loggers like any other validation metric.
See `examples/01_getting_started.py` for a complete, runnable version.

### Manual (without Lightning)

This is what `DistributedEvaluationCallback` does under the hood. Use it
directly outside a Lightning loop, or to re-evaluate predictions that were
already written to disk without re-running the model.

The core abstractions are `MetricStream` (for writing) and `run_offline_evaluation` (for computing).

#### Step 1: Inference (online)

```python
import torch
from tensordict import TensorDict
from evaluators import MetricStream, MemorySchema, TensorField, MemmapTemporalWriter
from evaluators.metrics.domain.segmentation import SemanticKeyMap, SemanticMetric
```

##### 1. Configure metrics and schema

```python
### Define the source for ground truth (lazy loading)

dataset = CityscapesDataset(...)
keys: SemanticKeyMap = {
    "pred": "pred_sem",
    "target": "target_sem",
    "sequence_id": "sequence_id",
    "frame_index": "frame_index",
}
metric = SemanticMetric(
    key_map=keys,
    num_classes=19,
    ignore_index=255,
    target_source=dataset,
)

### Define the explicit memory schema
### Only the prediction and metadata are written; ground truth is fetched
### from `dataset` during the Compute stage.

schema = MemorySchema(fields={
    "pred_sem": TensorField(dtype="int64", shape=(1024, 2048)),
    "sequence_id": TensorField(dtype="int64", shape=()),
    "frame_index": TensorField(dtype="int64", shape=()),
})
```

##### 2. Initialize stream and writer

```python
# Create a writer (backend)
writer = MemmapTemporalWriter(output_dir="./inference_cache/semantic", schema=schema)

# Create a stream and bind the writer
stream = MetricStream(
    metrics=[metric],
    name="semantic",
    schema=schema
)
stream.bind(writer)
```

##### 3. Run inference loop

```python
for batch in dataloader: # Model forward pass
    preds = model(batch["image"])
    batch_size = preds.shape[0]

    # Push to stream (non-blocking). A TensorDict is required, with a
    # leading time dimension even for single-frame batches.
    stream.update(
        TensorDict(
            {
                "pred_sem": preds,
                "sequence_id": batch["sequence_id"],
                "frame_index": batch["frame_index"],
            },
            batch_size=[batch_size],
        ).unsqueeze(1)
    )

# Finalize
writer.close()
```

    > Note: `sequence_id` and `frame_index` must be `torch.int64`.

##### 4. Finalize and compute

```python
from evaluators import run_offline_evaluation

# Syncs workers, builds catalog, and runs metrics
results = run_offline_evaluation(
    metrics=[metric],
    artifact_dir="./inference_cache/semantic"
)
print(results["SemanticMetric"]["mIoU"])
```

#### Step 2: Re-evaluation (offline)

Because predictions are persisted, metrics can be re-calculated or added without re-running the model.

```python
# Run evaluation on existing artifacts
results = run_offline_evaluation(
    metrics=[new_metric],
    artifact_dir="./inference_cache/semantic"
)
```

### CLI tools

The library includes a CLI for managing the inference cache.

_List stored sequences._

```shell
evaluators memory ls ./inference_cache
```

_Inspect specific tensor shapes._

```shell
evaluators memory inspect ./inference_cache frankfurt_000001
```

_Export to standard PyTorch file._

```shell
evaluators memory export ./inference_cache frankfurt_000001 --out my_video.pt
```

---

## Supported metrics

### Depth estimation

Implements standard error metrics (AbsRel, SqRel, RMSE, RMSElog) and threshold accuracies ($\delta < 1.25^n$).

### Segmentation

- _Semantic:_ Mean intersection over union (mIoU).
- _Panoptic:_ Panoptic quality (PQ), segmentation quality (SQ), recognition quality (RQ). Supports "Thing" and "Stuff" splits.

### Depth-aware video panoptic segmentation (DVPS)

Implements VPQ (video panoptic quality) and its depth-aware extension, DVPQ. Both evaluate spatio-temporal consistency using sliding window tubes over consecutive frames; DVPQ additionally gates matches by pixel-wise depth accuracy against one or more relative-error thresholds.

### Tracking

Multi-object tracking metrics: HOTA, CLEAR (MOTA/MOTP), and identity F1 (IDF1). Assignment (frame-level detection-to-track matching) is solved by `torchmatch` rather than `scipy.optimize.linear_sum_assignment`; see [OFFLINE_EVALUATION.md](OFFLINE_EVALUATION.md) for the migration note.

### STEP

Segmentation and tracking quality (STQ), and its depth-aware variant (DSTQ).

### Occupancy

Per-class IoU/mIoU and occupied-vs-free precision/recall/F1 over voxel-grid occupancy predictions, reusing the same statistics as `SemanticMetric`. `OccupancyMetric` (`evaluators.metrics.domain.occupancy`).

### 3D detection

Per-class, per-threshold average precision (AP) and mAP over 3D bounding boxes. Matching is a pluggable strategy (`IoU3DSimilarity`, backed by Shapely polygon intersection, or `CenterDistanceSimilarity`, the nuScenes convention), and curve integration is a pluggable rule (COCO's 101-point, KITTI's 40-point, or nuScenes' clipped-recall convention). Optional true-positive error decomposition (mATE, mAOE) and an orientation-aware AOS aggregate. `Box3DAveragePrecisionMetric` (`evaluators.metrics.domain.box3d`).

### Vector maps

Per-class, per-threshold AP and mAP over polylines (lane/vector-map elements), matched by a pluggable similarity strategy (`ChamferSimilarity` or `FrechetSimilarity`) and integrated by the same curve-integration rules as 3D detection. Optional topology accuracy over the matched polyline graph. `PolylineAveragePrecisionMetric` (`evaluators.metrics.domain.vectormap`).

### Sensor simulation

Paired reference-vs-simulated quality for images, point clouds, and radar (`SensorImageQualityMetric`, `PointCloudQualityMetric`, `RadarQualityMetric`), plus single-stream self-consistency metrics that need no paired reference (`TemporalConsistencyMetric`, `CrossModalAlignmentMetric`). All five live under `evaluators.metrics.domain.sensorsim`.

> As with `SemanticMetric`/`PanopticQualityMetric`, none of the classes above are re-exported from the top-level `evaluators` package; import them from their domain submodule as shown.

---

## Architecture

Reassembling whole sequences out of frames scattered across steps and ranks
takes three stages, run automatically by `DistributedEvaluationCallback`
during validation, or manually via the CLI.

1. **Write.**  
   Each GPU writes predictions to locally sharded files (e.g. `.memmap` or `.lmdb`). No communication occurs.
2. **Schedule.**  
   A synchronization barrier is reached. The main process aggregates metadata manifests from all shards to build a Global Catalog. It partitions the workload (videos, not individual frames) among workers using a greedy strategy to balance duration, so each worker gets complete, ordered sequences.
3. **Compute.**
   Workers iterate through their assigned Virtual Sequences. Data is streamed from disk, processed by torchmetrics, and reduced globally.

See [OFFLINE_EVALUATION.md](OFFLINE_EVALUATION.md) for driving this manually via the CLI (`compute`, plus `results`, `memory`, and `visualize` for reading, inspecting, and rendering stored predictions), and detailed design principles.

---

## Performance

`evaluators` is built for high-throughput I/O. The following benchmarks were conducted using the **Comprehensive Memory Evaluation Suite (CMES)** on a mobile workstation (i7-12700H, NVMe SSD).

### Throughput (FPS)

| Backend | Codec | Resolution | Write FPS | Read FPS | Compression |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Memmap** | Raw | 512x1024 | 250.8 | 501.6 | 1.0x |
| **Memmap** | Blosc | 512x1024 | 59.7 | 119.3 | 0.85x |
| **LMDB** | Raw | 512x1024 | 13.3 | 26.7 | 1.0x |
| **LMDB** | PNG | 512x1024 | 27.2 | 13.6 | 0.62x |
| **Filesystem**| TIFF | 512x1024 | 124.9 | 249.7 | 0.89x |

### Insights

- **Memmap is king for raw throughput:** The `MemmapTemporalWriter` achieves >500 FPS for read operations on mid-resolution video frames, making it ideal for fast metric computation.
- **Blosc provides the best balance:** Using `BloscCodec` with `Memmap` offers significant storage savings with minimal CPU overhead compared to PNG/TIFF.
- **LMDB for stability:** While slower for sequential video access, LMDB provides robust ACID compliance and is preferred for random-access metadata or small feature vectors.

> Full benchmark reports, including memory usage (Peak RSS) and CPU overhead plots, are available in `docs-assets/benchmarks/` (published at `/benchmarks/` on the docs site).

---

## Development

This project uses modern Python tooling for dependency management and quality assurance.

### Setup

We use `uv` for fast dependency management.

```shell
# Install dependencies
uv sync --all-extras
```

### Testing

Tests are managed by `pytest`.

```shell
# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=evaluators
```

### Linting & Formatting

We use `ruff` for all linting and formatting needs.

```shell
# Check code style
uv run ruff check .

# Format code
uv run ruff format .
```

---

## Contributing

Contributions are welcome. Please ensure that:

1. New features are covered by tests.
2. Code passes all static analysis checks (`ruff`).
3. Architecture changes are discussed in an issue first.

## Acknowledgements

This work was developed at the Mobile Perception Systems (MPS) lab at Eindhoven University of Technology.

## License

This project is licensed under the MIT License.
