Metadata-Version: 2.5
Name: interp-engine
Version: 1.2.8
Summary: A fast, standardized interpretability engine that supports most modern models and architectures. Powers Neuronpedia.
Project-URL: Homepage, https://github.com/decoderesearch/interp-engine
Project-URL: Repository, https://github.com/decoderesearch/interp-engine
Project-URL: Issues, https://github.com/decoderesearch/interp-engine/issues
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: <3.14,>=3.11
Requires-Dist: einops
Requires-Dist: numpy>=1.24
Requires-Dist: torch>=1.10
Requires-Dist: transformers>=4.57.1
Provides-Extra: awq
Requires-Dist: accelerate>=1.0; extra == 'awq'
Requires-Dist: gptqmodel>=5.0; extra == 'awq'
Provides-Extra: dev
Requires-Dist: pyright<1.2,>=1.1.411; extra == 'dev'
Requires-Dist: pytest<9,>=8.3.1; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.16.2; extra == 'dev'
Provides-Extra: parity
Requires-Dist: transformer-lens>=3.0; extra == 'parity'
Provides-Extra: quant
Requires-Dist: accelerate>=1.0; extra == 'quant'
Requires-Dist: kernels<0.17.0,>=0.15.2; extra == 'quant'
Provides-Extra: vllm
Requires-Dist: vllm>=0.27.1; (sys_platform == 'linux') and extra == 'vllm'
Description-Content-Type: text/markdown

# interp-engine



🔗 **[interp-engine.org](https://interp-engine.org)**



`interp-engine` is an interpretability engine that is fast (>40x tok/s vs HF eager), standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of [Neuronpedia](https://neuronpedia.org)'s inference and is checked for accuracy against HF Transformers and other engines.





This repo contains:

1. `[validator/](validator/)`, which compares/validates it against TransformerLens, and nnsight/nnterp on real architectures.
2. `[visualizer-web/](visualizer-web/)`, a "cheat sheet" hosted at [interp-engine.org](https://interp-engine.org) of each 'point' (eg `resid_post.16`), standardized across architectures.

## Installation

```bash
pip install 'interp-engine[vllm]'  # preferred install: includes vLLM support (CUDA required)
pip install interp-engine          # eager backend only
```

## Simple Usage

```python
from interp_engine import Address, load_model, run_with_cache

# VLLM MODE (default): low VRAM, medium speed
model = load_model("Qwen/Qwen3-8B") 

# VLLM-FREEZE MODE: high VRAM, high speed, only frozen points (default resid_post)
# model = load_model("Qwen/Qwen3-8B", freeze_points="auto")

# EAGER MODE: low VRAM, low speed
# model = load_model("Qwen/Qwen3-8B", backend="eager")

point = Address("resid_post", 10)      # or string: "resid_post.10"
cache = run_with_cache(model, model.to_tokens("Hello, world"), [point])
cache[point]  # [batch, pos, ...]
```

### AI Agents

Add "use interp-engine" to your prompt and let your agent figure it out - everything is fully documented in this repo and open source.

## Supported Points ("Addresses")

`interp-engine` supports 34 standardized points ("Addresses") across architectures: every one of them on the eager backend, 28 of them on vLLM. Check [interp-engine.org](https://interp-engine.org) for the "cheat sheet", or [SUPPORTED_POINTS.md](docs/SUPPORTED_POINTS.md) for a markdown version with the per-backend detail.

## Performance / Speed

vLLM gives `interp-engine` high throughput via concurrency, and **graph freeze** adds CUDA-graph replay on top of that *without giving up capture or steering*. Every column below is capture-capable.

<!-- THROUGHPUT:START -->

<!-- Generated by `python -m benchmarks.report_bench`. Do not edit: rerun the sweep. -->

Measured on NVIDIA B200, bf16, 512-token prompt, 128 new tokens.

One stream (tok/s):

| model                    | eager | vLLM       | vLLM + graph freeze |
| ------------------------ | ----- | ---------- | ------------------- |
| `gemma-2-2b`             | 31    | 31 (1.0x)  | **214 (6.9x)**      |
| `qwen3-4b`               | 24    | 47 (2.0x)  | **296 (12.3x)**     |
| `llama-3.1-8b`           | 33    | 57 (1.7x)  | **256 (7.9x)**      |
| `qwen3.8-27b`            | 9.9   | 12 (1.2x)  | **63 (6.4x)**       |
| `deepseek-v4-flash-0731` | 3.3   | 2.9 (0.9x) | **119 (36x)**       |

8 concurrent requests (aggregate tok/s):

| model                    | eager | vLLM        | vLLM + graph freeze |
| ------------------------ | ----- | ----------- | ------------------- |
| `gemma-2-2b`             | 30    | 226 (7.5x)  | **1,238 (41x)**     |
| `qwen3-4b`               | 24    | 333 (14.0x) | **1,018 (43x)**     |
| `llama-3.1-8b`           | 32    | 419 (13.0x) | **1,536 (48x)**     |
| `qwen3.8-27b`            | 9.5   | 87 (9.2x)   | **386 (41x)**       |
| `deepseek-v4-flash-0731` | 3.2   | 23 (7.2x)   | **402 (127x)**      |

<!-- THROUGHPUT:END -->

**Graph freeze** is opt-in via `freeze_points`, and a frozen engine serves only the set it froze. [PERFORMANCE.md](docs/PERFORMANCE.md) has how it works and what it trades; `[benchmarks/results-latest.md](benchmarks/results-latest.md)` has every figure at full precision, including capture, steering and lens latencies; `[benchmarks/README.md](benchmarks/README.md)` has how the tables above are rounded.

## Correctness

We verify correctness in two main ways:

1. A test suite that checks results across several models - what each check is designed to catch is in [INTERNALS.md](docs/INTERNALS.md#correctness).
2. A full `validator` comparison engine that checks most hook points across 50+ models, at early, middle and late layers - fully reproducible, with detailed results saved in the git repo at `[validator/](validator/)`.

## Why use an Interpretability Engine, instead of just having my AI code whatever it needs on the fly?

Software is easy to make in 2026, here's why an engine makes sense:

1. **Speed**: Get performance without sacrificing correctness.
2. **Standardization + Verification**: Eliminate ambiguity when referring to points, plus a full test suite included.
3. **Faster Dev / Fewer Tokens Used**: You could spend ten million tokens and have your AI write, test, and make production-ready an interpretability engine. Or you could just `pip install interp-engine[vllm]`.

## Development / Contributing

Activate the shared git hooks once per clone — they format staged Python, rebuild the generated files, and run CI's static checks before a push. Details in [CONTRIBUTING.md](CONTRIBUTING.md).

```bash
make hooks    # or: git config core.hooksPath .githooks
```

## Contact

Bugs and feature requests belong in [issues](https://github.com/decoderesearch/interp-engine/issues). For anything else: [johnny@neuronpedia.org](mailto:johnny@neuronpedia.org).

## License

Apache 2.0