Metadata-Version: 2.5
Name: interrater
Version: 0.0.1.dev0
Summary: Inter-rater agreement for systematic reviews: auditable cleaning, bootstrap intervals, and simulated ratings with known truth
Project-URL: Homepage, https://github.com/djarenas/Inter-Rater
Project-URL: Repository, https://github.com/djarenas/Inter-Rater
Project-URL: Issues, https://github.com/djarenas/Inter-Rater/issues
Project-URL: Prior version (arXiv), https://arxiv.org/abs/1809.05731
Project-URL: Prior version (Zenodo), https://doi.org/10.5281/zenodo.1227660
Author: Daniel J. Arenas
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: bootstrap,cohens-kappa,fleiss-kappa,inter-rater-reliability,interrater-agreement,krippendorff-alpha,systematic-review
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# interrater

Inter-rater agreement for systematic reviews and other studies where not every
rater sees every subject.

**This release reserves the package name. It is not yet usable.** Version 2 is
under active development; follow
[the repository](https://github.com/djarenas/Inter-Rater) for progress.

## Background

`interrater` is the successor to
[Inter-Rater](https://github.com/djarenas/Inter-Rater), described in
[arXiv:1809.05731](https://arxiv.org/abs/1809.05731) (2018) and archived at
[Zenodo](https://doi.org/10.5281/zenodo.1227660). Version 1 has been used in
published systematic reviews to compute Fleiss' kappa and pairwise Cohen's
kappa across multiple reviewers.

Version 2 is a complete rewrite with a different API. Version 1 remains
available at git tag `v1.5`.

## What version 2 adds

- **Bootstrap confidence intervals.** Version 1 reported an analytic standard
  error for the average of pairwise kappas, which treats those kappas as
  independent. They are not: two pairwise coefficients sharing a rater are
  correlated, and the resulting interval is too narrow. Resampling subjects
  preserves that correlation without having to model it.

- **An auditable cleaning layer.** Which strings meant nothing, which were
  repaired and into what, how thin the data was allowed to get: declared in a
  versioned config file and reported afterwards, rather than reconstructed from
  memory when a reviewer asks.

- **Simulated ratings with known truth.** Generate data from an explicit model
  of rater accuracy, category prevalence and subject assignment, so an
  estimator can be checked against an answer known in advance and a confidence
  interval can be checked for actually covering it.

- **Support for unbalanced designs.** Where each subject is rated by a subset
  of the reviewer pool rather than all of them, which is what most reviews can
  actually afford.

## Requirements

Python 3.10 or later.

## License

GNU General Public License v3 or later. Same license as version 1.
