Metadata-Version: 2.4
Name: evalmetry
Version: 1.0.0
Summary: A research toolkit for measuring and analyzing model behavior.
License-Expression: MIT
Project-URL: Homepage, https://github.com/minguinho26/evalmetry
Project-URL: Repository, https://github.com/minguinho26/evalmetry
Project-URL: Issues, https://github.com/minguinho26/evalmetry/issues
Project-URL: Releases, https://github.com/minguinho26/evalmetry/releases
Keywords: lm-eval,language-models,evaluation,interpretability
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lm-eval[hf]==0.4.13
Requires-Dist: torch>=2.1
Requires-Dist: transformers==5.16.1
Requires-Dist: pyarrow>=14
Requires-Dist: pandas>=2.0
Requires-Dist: safetensors>=0.4
Requires-Dist: matplotlib>=3.7
Requires-Dist: sentencepiece
Requires-Dist: protobuf
Requires-Dist: pyyaml
Requires-Dist: tqdm
Provides-Extra: pinned-latest
Requires-Dist: lm-eval==0.4.13; extra == "pinned-latest"
Requires-Dist: transformers==5.16.1; extra == "pinned-latest"
Requires-Dist: accelerate; extra == "pinned-latest"
Requires-Dist: pyarrow==25.0.1; extra == "pinned-latest"
Requires-Dist: pandas==3.0.5; extra == "pinned-latest"
Requires-Dist: safetensors==0.8.0; extra == "pinned-latest"
Requires-Dist: matplotlib==3.11.1; extra == "pinned-latest"
Requires-Dist: sentencepiece; extra == "pinned-latest"
Requires-Dist: protobuf; extra == "pinned-latest"
Dynamic: license-file

# Evalmetry

A research toolkit for measuring and analyzing model behavior.

Evalmetry evaluates Hugging Face language models with lm-eval and collects internal signals from the same forward passes used for scoring.

## Features

- Benchmark scores with per-layer logit lens and layer similarity.
- Optional attention, hidden states, custom hooks and module statistics.
- Custom benchmarks, LLM judge scoring and modified-model adapters.
- Resumable runs, saved-data readers and comparison reports.

Model evaluation supports one process on one CUDA GPU. CPU execution is for test fixtures and verification tools. Reports and saved-data readers do not require a GPU.

## Install

Install from PyPI in your Python environment:

```bash
pip install evalmetry
```

Evalmetry requires Python >=3.10. Use a compatible CUDA build of PyTorch. The Python import and CLI are both `evalmetry`.

To install this release explicitly, use `pip install evalmetry==1.0.0`.
For development, clone this repository and run `python -m pip install -e .`.

## Quick start

Use `run` to evaluate eight documents and save the default signals:

```bash
evalmetry run --model-args pretrained=Qwen/Qwen3-0.6B,dtype=bfloat16,device=cuda \
    --tasks arc_easy --num-fewshot 0 --limit 8 --batch-size 1 \
    --output results/quickstart
```

Use `report` to generate a report from that run:

```bash
evalmetry report results/quickstart --output report/quickstart
```

The first run downloads the model and dataset if needed. Reusing a run directory resumes its recorded configuration; use a different directory for a different experiment.

Other commands: `collect-research-data` adds optional tensors to a completed run, `debug` reads a saved module trace, and `module-stats` reads saved statistics. Traces and statistics must be enabled during collection. Use `evalmetry <command> --help` for options.

## Reading results

Evaluation results, manifests and collected signals are written under the selected output directory. Read saved signals with `evalmetry.load_signals(run_dir)`; `evalmetry.describe_schema()` describes the columns.

## License

Evalmetry is released under the MIT License.
