Metadata-Version: 2.4
Name: consequence-ledger
Version: 0.3.0
Summary: Measure what AI coding decisions actually cause: randomized routing, delayed outcome labels, causal estimates.
License-Expression: MIT
Project-URL: Homepage, https://github.com/sujan-mitta/consequence-ledger
Project-URL: Source, https://github.com/sujan-mitta/consequence-ledger
Project-URL: Changelog, https://github.com/sujan-mitta/consequence-ledger/blob/main/CHANGELOG.md
Keywords: ai,coding-agents,causal-inference,off-policy-evaluation,szz,git,software-engineering
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Version Control :: Git
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Consequence Ledger

**Which AI coding tool actually *causes* fewer reverted and bug-fixed changes on your code?**

AI tools are judged on instant signals: a suggestion accepted, tests passing, a developer saying it felt faster. What matters arrives weeks later: the change gets reverted, or someone ships a fix for it. Comparing tools on those later outcomes is misleading unless you account for *which tasks each tool was given*. This prototype records that properly and turns it into cause-and-effect estimates.

It is the 90-day MVP from the "AI elephant in the room" research: decision logging with propensities, commit attribution, revert and SZZ outcome labels, and IPW / doubly-robust estimates with confidence intervals. It is pure Python standard library, with no dependencies beyond `git`.

## Install

Python 3.9+ and git 2.28+. There are no dependencies.

```bash
git clone https://github.com/sujan-mitta/consequence-ledger.git
cd consequence-ledger
pip install .          # gives you the `ledger` command; or skip this and use `python -m ledger` from the folder
ledger --version
```

## Quick start

```bash
python -m ledger demo
```

This builds a synthetic repository with about 3,700 commits and a known true effect for each agent, then runs the whole pipeline and writes an HTML report. You'll see the naive comparison blame the wrong agent and the causal estimate recover the truth. A copy of the output is in [examples/demo-report.html](examples/demo-report.html).

```bash
python -m ledger audit --repo path/to/repo --html audit.html
```

This gives a day-one baseline from git history alone: revert and fix rates per tool, detected from commit trailers and bot authors (Claude Code, Copilot, Cursor, Devin, Jules, Codex, Aider, OpenHands). It is **correlational**, and the output says so. See [examples/manifold-audit.html](examples/manifold-audit.html) for a run on a real open-source repo.

## Using it on a real repo

```bash
python -m ledger init                                   # create .ledger/ and install the commit hook
python -m ledger decide --arms claude-code,codex,cursor \
    --scores 1.0,0.6,0.4 --epsilon 0.4 \
    --context '{"stratum": "bugfix"}'                   # prints which tool to use for this task
# ... do the task with that tool; commits on this branch are tagged "Ledger-Decision-Id: <id>" automatically
python -m ledger done                                   # stop tagging on this branch (--all: every branch)
python -m ledger attach --decision <id> --commit <sha>  # if a squash merge dropped the trailer

# optional: production incidents and deploys exported from Sentry, PagerDuty, incident.io …
python -m ledger import incidents --file incidents.csv  # id, started_at, title, commits, files
python -m ledger import deploys --file deploys.csv      # sha, deployed_at

# or pull incidents live (read-only key in an environment variable, never an argument)
python -m ledger sync sentry --org your-org-slug --days 90      # needs SENTRY_AUTH_TOKEN
python -m ledger sync pagerduty --days 90                       # needs PAGERDUTY_API_KEY

# before you start, and any time you wonder if something is stuck
python -m ledger plan --base-rate 0.12 --effect -0.03           # how many finished decisions you need
python -m ledger status                                         # gaps that would mislead an estimate

# a week in: an early readout; weeks later: the real one
python -m ledger label --fix-style keywords             # reverts, fixes and incidents within 30 days
python -m ledger estimate                               # causal estimates with 95% CIs
python -m ledger report                                 # .ledger/report.html

# once you have a few hundred finished decisions
python -m ledger breakdown --stratum difficulty         # which tool suits which kind of task
python -m ledger recommend --stratum difficulty         # and what routing by that is honestly worth
```

| Command | What it does |
|---|---|
| `init` | Creates `.ledger/` (self-ignored by git) and installs a `prepare-commit-msg` hook. It won't overwrite a hook you already have, and re-running it upgrades an older ledger hook. |
| `decide` | Chooses an arm at random with `p = (1-ε)·softmax(score/τ) + ε/k` and logs every arm's probability. The decision is active on the branch you are on, so several agents can work on several branches or git worktrees at once without sharing a decision. All worktrees and subfolders use one ledger at the repository root. |
| `sync sentry` / `sync pagerduty` | Pull error-level Sentry issues or high-urgency PagerDuty incidents as incidents. Keys come only from `SENTRY_AUTH_TOKEN` / `PAGERDUTY_API_KEY` and are never printed, even in errors. Neither API names the commit that caused an incident, so these carry timing only: link them with an export that names commits, or `--incident-linking`. Re-syncing never erases links from an earlier export. |
| `plan` | Simulates your routing and tells you how many finished decisions give 80% power, both for one look and for checking repeatedly. |
| `status` | Flags what would mislead an estimate: decisions with no commit, stale labels, near-zero routing chances, one-arm ledgers, incidents that can't be linked. |
| `import` | Loads incident or deploy exports (CSV or JSON, common field names accepted). |
| `label` | Maps commits to decisions and finds reverts, fixes (SZZ via `git blame`) and linked incidents. A decision counts only once its whole window has passed; a separate early label (`--early-days 7`) feeds the early readout. |
| `estimate` | Naive, IPW, self-normalised IPW, direct method and doubly robust estimates per arm. Bootstrap and influence-function CIs, effective sample size, minimum propensity, and a check that unmerged work is balanced across arms. `--cluster developer` resamples whole clusters when outcomes are correlated. |
| `breakdown` | Which tool is better for which kind of task (`--stratum difficulty`). Raw and shrunk effects per stratum with Bonferroni-adjusted p-values, so a lucky stratum is not mistaken for a pattern. |
| `recommend` | What routing each task type to its best tool would be worth. The tool is chosen on one half of your decisions and judged on the other, so the gain is not inflated; it also shows the gain from simply using the best tool. |
| `report` | A self-contained HTML report (light and dark). |
| `audit` | Per-tool baseline from history, with Wilson CIs and size-adjusted rates. |
| `demo` | Synthetic end-to-end run that checks labels against an oracle. |

Useful options:
- `--fix-style keywords` for repos that write "Fix crash in …" rather than `fix: …`
- `--fix-regex` for anything else
- `--ignore-revs .git-blame-ignore-revs` to see through formatting commits
- `--horizon 30`, `--as-of 2026-06-30`, `--baseline <arm>`, `--clip 20`
- `--cluster developer` (any context key) for cluster-robust intervals
- `--stratum difficulty,language` to model outcomes per combination of context keys
- `--incident-linking latest|window` to also link incidents that name no suspect commit (off by default; see below)
- `--planned-n 4000` sets the sample size the anytime-valid interval is tuned for. Choose it before you start.
- `--folds 5` cross-fits the outcome model (0 turns it off). It matters when strata are many and small: in
  simulation the interval covered the truth 87.7% of the time without it and 95.3% with it.

**Checking every week?** Ordinary 95% intervals don't hold up to repeated looks: in simulation, 16 weekly checks under a true null "found" a winner 30% of the time. Each verdict therefore also says whether it survives the anytime-valid confidence sequence (Waudby-Smith et al., *Annals of Statistics* 2024), which kept that rate at 3%.

The report warns when:
- one arm's work goes unmerged more often (a chi-square test), since that biases the merged-only comparison
- an arm had a near-zero chance of being picked for some tasks
- an arm's effective sample size is small

## How it decides "bad"

A change is bad if, within the horizon after its first commit:
- it is **reverted**: git's `This reverts commit <sha>`, or a squash-merged GitHub `Revert "<subject>"` PR, or
- a **fix commit** changes lines it introduced (SZZ), or names its PR ("Bug introduced in #1507", "regression from #88"), or
- a **production incident** names it as a suspect commit.

Incidents that name only files are left unlinked by default. In testing, blaming "the latest deploy that touched those files" found the true culprit only 14% of the time, because files change often. A missed incident can only shrink a difference between tools; a wrongly blamed one adds noise. Export suspect commits where your tooling has them (Sentry does).

**Early readout.** `estimate` also reports differences on a 7-day label, which includes recent decisions the main estimate must exclude. Next to it, the report shows how 7-day and 30-day differences compared on decisions old enough to have both. In the demo only 20% of bad outcomes surfaced within a week, and one tool's early difference pointed the wrong way. Treat the early readout as a hint until that comparison says otherwise.

The SZZ step includes the standard refinements against false positives. Each one mattered on real data:
- ignore whitespace-only edits (`-w`), and blank or comment-only lines
- ignore lines that were moved rather than changed
- skip tests, docs, editor and CI config, and lockfiles
- never treat a merge commit as a fix, even one titled "Merge pull request … from me/fix-crash"
- ignore a fix that blames more than 10 commits (`--max-blamed`, 0 to keep all): the research found such fixes are outliers, and cutting them raised precision
- optionally look through listed formatting commits

**Squash merges.** GitHub's squash merge drops the `Ledger-Decision-Id` trailer, which would make merged work look abandoned. `label` recovers these automatically: it fingerprints the branch's combined diff (`git patch-id --stable`) and looks for a commit on your branch with the same fingerprint. It only accepts a unique match that is newer than the branch's first commit, never guesses, and reports `squash_recovered`. It needs the branch's commits to still exist locally (a branch, a remote-tracking ref, or the reflog); `ledger attach` covers anything else.

Fix analysis is remembered in the ledger, so re-running `label` is near-instant: the same repository went from 46 seconds to under a tenth of a second.

## What we learned

Full details are in [research/FINDINGS.md](research/FINDINGS.md):
- Doubly-robust estimates are unbiased (< 0.1 pp) with 93.5–95% bootstrap CI coverage. Naive comparisons were off by up to 7 pp and picked the wrong winner.
- Re-checking weekly with ordinary CIs gives a **30% false-positive rate**. The anytime-valid sequence gives 3%.
- How much data you need depends on your routing. A 3 pp difference needed about 8,000 mature decisions when routing was lopsided (the demo, ε = 0.1) but about 4,000 with near 50/50 routing, or about 8,000 if you also want it safe to check repeatedly. Run `ledger plan` with your own numbers.
- Label noise inflates levels but mostly leaves differences between arms alone. Missed bugs only shrink effects. Report differences.
- On elalish/manifold, Claude Code commits looked worse raw (11.3% vs 7.5%) and about the same size-adjusted (7.1% vs 8.1%). The CIs overlap, which is the case for randomizing.

## Documentation

- [docs/METHODOLOGY.md](docs/METHODOLOGY.md): exactly what is computed, the assumptions it needs, and what it cannot tell you
- [research/FINDINGS.md](research/FINDINGS.md): simulation studies and real-repository results
- [CHANGELOG.md](CHANGELOG.md), [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md)

## Layout

```
ledger/          the package: policy (routing), label + squash (outcomes), estimate, audit, plan, status,
                 incidents + sentry + pagerduty, report, demo, db, gitio, cli
tests/           230 unittest cases: real git repos in temp dirs, local fake servers, estimator Monte Carlo checks
tools/           coverage_lite.py, dependency-free line coverage
docs/            METHODOLOGY.md
research/        simulate.py (coverage, power, noise, design, peeking studies), results/, FINDINGS.md
examples/        generated demo report and the manifold audit
```

Run the tests:

```bash
python -m unittest discover -s tests
```

## Limits (honest list)

- Sentry and PagerDuty can be synced live, but the syncs were only tested against fake servers and (for Sentry) an empty account; others need exports. Customer impact (tickets, churn) isn't linked.
- The outcome model for DR is a per-stratum mean. Many context keys at once need a real model with cross-fitting.
- The early readout is a plain shorter window, not a calibrated surrogate index.
- Audit mode is observational. It describes; it does not prove.
- Python 3.9+ and git 2.28+.
