Metadata-Version: 2.5
Name: rlshiftlab
Version: 0.3.0
Summary: Stress-test frozen RL policies under controlled shifts and get degradation curves with confidence intervals.
License: MIT
License-File: LICENSE
Requires-Python: >=3.9
Requires-Dist: gymnasium<2,>=0.29
Requires-Dist: matplotlib>=3.6
Requires-Dist: numpy>=1.23
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# RLShiftLab

RLShiftLab 0.2.0 evaluates frozen policies under controlled observation and
action shifts. A single-policy run produces a degradation curve: shift
strength on the x axis, mean episode return with a pointwise 95% confidence
interval on the y axis. Version 0.2 also evaluates groups of already trained
policies, reports the observed spread across their members, compares policies
directly on paired evaluation episodes, and records custom per-episode
metrics. Each shift is varied separately. Reports describe measured
differences and do not establish their causes.

## Install locally

Requires Python 3.9 or newer. From a checkout of this repository:

```sh
python -m pip install -e .
```

For tests and lint checks, use `python -m pip install -e ".[dev]"`, then
`python -m pytest -q` and `python -m ruff check .`.

## Quickstart

Save this as `quickstart.py` and run `python quickstart.py` from a writable
directory after installing the package:

```python
import gymnasium as gym
from rlshiftlab import ActionSlip, ObservationNoise, Suite

def policy(obs):
    return int(obs[2] + 0.5 * obs[3] > 0)

suite = Suite(
    env_fn=lambda: gym.make("CartPole-v1"),
    shifts=[ObservationNoise(levels=[0.01, 0.05]),
            ActionSlip(levels=[0.05, 0.1])],
    n_episodes=20,
    seed=0,
    success_fn=lambda ep: ep.truncated,
)
results = suite.evaluate({"angle+velocity": policy})

print(results.summary())
results.to_csv("results.csv")
results.plot("degradation.png")
results.to_html("report.html")
```

`policy` is a callable from observation to action. RLShiftLab does not train
or alter it. Use a deterministic policy for paired comparisons; if a policy
has a `reset()` method, RLShiftLab calls it at the start of each episode.
The suite gives every policy and condition the same sequence of environment
reset seeds. Shift randomness is also seeded per episode, and reused across
levels and policies for the same shift. Repeated runs with the same inputs and
seed produce identical CSV rows.

The nominal row has no shift. For shifted rows, **drop** is the nominal episode
return minus the shifted return, paired by episode seed; a positive value means
lower return under the shift. Mean-return and drop intervals use percentile
bootstrap over episodes. Optional success-rate intervals use the Wilson
method. The CSV includes mean episode length and, when `success_fn` is given,
success rate. A policy's behavior can still be stateful or stochastic if its
callable is written that way; the suite cannot guarantee reproducibility for
randomness that the policy manages independently. This original
`Suite.evaluate()` interface and its 14-column `to_csv()` export remain
compatible with v0.1.

## Shifts and outputs

- `ObservationNoise`: Gaussian noise for Box observations; random replacement
  for Discrete observations.
- `ActionSlip`: random replacement of an action before it reaches the environment.
- `StickyActions`: repeat the previously executed action with a given probability.

Discrete random replacement may select the original value. At replacement
probability `p` with `n` equally likely values, the observed mismatch rate is
`p * (1 - 1/n)`. The level is the replacement probability, not a guaranteed
mismatch rate.

`Results.to_csv()` writes tidy rows; `Results.plot()` returns a Matplotlib
figure and optionally writes a PNG; `Results.to_html()` writes a self-contained
report with the plot, setup, summary, and full results table. Plotting uses the
headless Agg canvas.

## v0.2: groups of frozen policies

Supply a mapping from group name to member ID to an **already trained**
callable policy. Each member ID should identify a separately trained frozen
policy, such as a run with its own training seed. Multiple checkpoints from
one training run are not independent training members. RLShiftLab evaluates
the supplied members; it does not train them or verify their independence.
The runnable multi-seed CliffWalking example in `examples/` shows how to
train the members outside the package and pass their frozen policies to
`evaluate_groups()`.

For example, after creating four frozen CliffWalking callables in your own
training code, configure `suite` with that CliffWalking environment and its
evaluation shifts. The earlier CartPole quickstart suite is for CartPole
policies and cannot evaluate these CliffWalking tables:

```python
groups = {
    "Q-learning": {"train-0": q_policy_0, "train-1": q_policy_1},
    "SARSA": {"train-0": sarsa_policy_0, "train-1": sarsa_policy_1},
}
group_results = suite.evaluate_groups(groups)
group_results.to_csv("groups.csv")
group_results.to_member_csv("members.csv")
group_results.to_episode_csv("episodes.csv")
group_results.plot("group_degradation.png", band="ci")
group_results.plot("member_spread.png", band="std")

contrast = group_results.compare("Q-learning", "SARSA", metric="return")
contrast.to_csv("comparison.csv")
contrast.plot("comparison.png")
group_results.to_html("group_report.html", comparisons=[contrast])
```

The same `Suite` configuration works for both single policies and groups.
At least two members are required per group, but groups may have different
member counts. Each member receives the same ordered evaluation reset seeds
at every condition. The group mean first averages episodes within a member,
then gives each member equal weight. The reported `member_std`, minimum, and
maximum describe the observed spread of member means. That spread includes
finite evaluation noise and is neither a confidence interval nor a pure
estimate of training variance.

Group confidence intervals use a crossed bootstrap that resamples members
and evaluation episode keys. They are approximate **pointwise** 95%
intervals, not simultaneous bands or multiple-comparison-adjusted claims.
Intervals from only a few members may be unstable. Single-policy success
rates retain Wilson intervals; group success rates use the crossed bootstrap.
The [pigeonhole bootstrap paper](https://arxiv.org/abs/0712.1111) informs the
two-axis resampling design, and [Agarwal et al.'s RL evaluation
paper](https://arxiv.org/abs/2108.13264) motivates careful reporting across
trained runs. Neither source guarantees exact coverage for a particular
RLShiftLab experiment.

## Direct comparisons and custom metrics

Call `Results.compare(left, right, metric="return")` for two individual
policies, or `GroupResults.compare(...)` for two groups. Every nominal and
shifted comparison uses the same evaluation episode keys on both sides;
missing or duplicate keys are errors. The estimate is **left minus right**.
A positive value means the left side has a larger measured metric, which is
not always better (for example, more falls or longer episodes). Single-policy
comparisons bootstrap the paired episode differences. Group comparisons
resample members independently on each side while sharing evaluation episode
keys; equal member ID strings do not assert matched training runs. Each
contrast has its own pointwise 95% interval and can be saved as CSV, a
zero-reference plot, or standalone HTML. Pass a list to `to_html(...,
comparisons=[contrast])` to include chosen comparisons in an evaluation
report.

Pass `step_metrics` and `episode_metrics` to `Suite` to collect scalar metrics
in both single and group runs. Each step callback contributes a finite
number; RLShiftLab sums it over the episode. For the configured CliffWalking
environment, a reward of -100 marks a fall, so this metric counts falls:

```python
suite = Suite(
    env_fn=make_cliffwalking_env,
    shifts=[ActionSlip([0.1, 0.2])],
    n_episodes=100,
    seed=1000,
    step_metrics={"cliff_falls": lambda t: float(t.reward == -100.0)},
    episode_metrics={"return_per_step": lambda ep: ep.episode_return / ep.length},
)
```

Here `make_cliffwalking_env` is your environment factory. The -100 rule is
specific to that environment, not a generic fall detector. Step callbacks
receive a `Transition` with policy-visible observations, requested action,
next observation, reward, termination/truncation flags, step index, episode
seed, and step `info`. Its requested action may differ from the action
actually executed after action slip or sticky actions. Episode callbacks
receive a copy of the final `EpisodeResult`, including step metric totals,
length, return, flags, and final step `info`. Each callback must return one
finite numeric scalar; no full trajectory is retained. The resulting
`metric_rows` and `to_metrics_csv()` summarize each metric, while
`to_episode_csv()` includes scalar values under `metric:<name>` columns.

The raw episode CSV records policy/group, member, shift, level, episode index,
reset seed, return, length, flags, optional success, and custom scalars. It
does not serialize arbitrary observations or `info` objects. These records
make pairing and reported aggregates auditable.

Repeated runs with the same inputs, software versions, suite seed, and pure
deterministic or consistently resettable policy and metric callbacks produce
the same CSV bytes. Suite seeds control Gymnasium resets and shift randomness;
they do not control randomness inside arbitrary user policies or callbacks.

| Result object | Main export | Additional exports |
| --- | --- | --- |
| `Results` | `to_csv(path)` keeps the v0.1 14 columns | `to_metrics_csv(path)`, `to_episode_csv(path)`, `plot(metric=...)`, `to_html(path, comparisons=...)` |
| `GroupResults` | `to_csv(path)` writes aggregate rows | `to_member_csv(path)`, `to_metrics_csv(path)`, `to_episode_csv(path)`, `plot(metric=..., band="ci" or "std")`, `to_html(path, comparisons=...)` |
| `ComparisonResults` | `to_csv(path)` writes left-minus-right rows | `plot(path)`, `to_html(path)`, `summary()` |

## v0.2 examples

Run either example from the repository root:

```sh
python examples/cliffwalking_multiseed.py
python examples/custom_metrics.py
```

The [multi-seed CliffWalking example](examples/cliffwalking_multiseed.py)
trains its Q-learning and SARSA tables in the example code, then evaluates
them as frozen members. It writes separate group, member, metric, episode,
and paired-comparison files, plus plots and HTML, under
[`artifacts/v0.2/cliffwalking/`](artifacts/v0.2/cliffwalking/).
The [small custom-metrics example](examples/custom_metrics.py) uses fixed toy
policies to demonstrate step and episode metric callbacks and comparison
exports under [`artifacts/v0.2/custom_metrics/`](artifacts/v0.2/custom_metrics/).
Its outputs illustrate the API, not a research benchmark. The package does
not perform the training in either case.

### Measured multi-seed CliffWalking run

The generated [v0.2 group results](artifacts/v0.2/cliffwalking/group.csv)
used deterministic CliffWalking-v1, ten Q-learning training seeds (0–9) and
ten SARSA training seeds (100–109), with 500 training episodes per member.
Each frozen member was evaluated on the same 100 reset seeds per condition
for one nominal condition and eight shift levels (18,000 raw episodes).
[run.json](artifacts/v0.2/cliffwalking/run.json) records the configuration,
software versions, and source/output hashes.

Nominal mean return across members was -13.000 for Q-learning (observed
member SD 0.000) and -35.300 for SARSA (SD 57.870). In the
[member results](artifacts/v0.2/cliffwalking/member.csv), SARSA training seed
100 had nominal return -200, length 200, and no success; each of the other
nine SARSA seeds had return -17, length 17, and success. These member scores
underlie the reported nominal spread; they do not identify why that one
training outcome occurred.

At action slip 0.3, group mean returns were -149.928 for Q-learning and
-41.324 for SARSA. The direct Q-learning-minus-SARSA return difference was
-108.604, with a pointwise 95% CI of [-138.084, -79.740]. The corresponding
[cliff-fall difference](artifacts/v0.2/cliffwalking/falls_comparison.csv) was
+1.066 falls per episode [0.808, 1.329], so Q-learning had more recorded
falls under that shift in this configured run. These are measured differences
for these trained tables, environment settings, and evaluation seeds; they
are not causal explanations or a general ranking of the algorithms.

![v0.2 multi-seed CliffWalking group return curves](artifacts/v0.2/cliffwalking/mean_return_ci.png)

The [return comparison CSV](artifacts/v0.2/cliffwalking/return_comparison.csv),
[episode audit CSV](artifacts/v0.2/cliffwalking/episodes.csv), and
[standalone report](artifacts/v0.2/cliffwalking/report.html) provide the
condition-level and per-episode evidence. A repeated execution with the
same configuration produced byte-identical hashes for all 13 generated
outputs. That repeatability applies to this deterministic example and
software setup, not to arbitrary user-supplied stochastic callbacks.

The [toy custom-metrics outputs](artifacts/v0.2/custom_metrics/) use two
fixed policies and 20 paired evaluation episodes per condition. At action
slip 0.5, the measured always-zero-minus-always-one return difference was
0.85 [0.5, 1.2], while its unsafe-event difference was -0.5 [-0.7, -0.3].
This demonstrates the metric and comparison APIs; it is not a research
benchmark.

## Historical v0.1 examples and measured results

The original v0.1 repository contains a CliffWalking Q-learning versus SARSA example and a
CartPole comparison of two hand-coded policies in `examples/`. Training code
for CliffWalking stays in the example; the RLShiftLab package evaluates only
the resulting frozen policies. The CliffWalking example uses the registered
`CliffWalking-v0` when available, or `CliffWalking-v1` with
`is_slippery=False` on newer Gymnasium versions. See the [Gymnasium
CliffWalking documentation](https://gymnasium.farama.org/v1.3.0/environments/toy_text/cliff_walking/)
for the v1 environment and its optional slippery mode.

Run the examples from the repository root:

```sh
python examples/cliffwalking_q_vs_sarsa.py
python examples/cartpole_heuristic.py
```

The CliffWalking run writes [results.csv](results.csv),
[report.html](report.html), and the figure below. The CartPole run writes
[results.csv](artifacts/cartpole/results.csv),
[report.html](artifacts/cartpole/report.html), and
[degradation.png](artifacts/cartpole/degradation.png) under
`artifacts/cartpole/`.

![Measured CliffWalking degradation curves](docs/cliffwalking_degradation.png)

In the generated CliffWalking results (200 evaluation episodes per condition),
nominal mean return was -13.000 for Q-learning and -19.000 for SARSA. At action
slip 0.3, mean return was -198.250 for Q-learning and -48.815 for SARSA. The
paired drops from nominal were 185.250 (95% CI [154.945, 220.342]) and 29.815
(95% CI [23.120, 37.440]), respectively. This run measured a larger drop for
Q-learning at that shift level. The example trained one table for each
algorithm with 500 episodes before evaluation; it does not measure variation
across training seeds.

In the generated CartPole results (100 evaluation episodes per condition),
nominal mean return was 500.000 for the angle+velocity heuristic and 41.040
for the angle-only heuristic. At observation noise 0.2, their mean returns were
267.400 and 38.310, respectively. These values describe the two particular
frozen policies and the configured evaluation seeds.

## What v0.2 does not do

Version 0.2 does not train or adapt policies inside the package, measure
recovery, apply reward shifts or delays, change environment dynamics, support
vectorized environments or a CLI, or accept Dict, MultiDiscrete, or Tuple
observation spaces. A group member ID is caller-provided provenance, not
proof that training runs were independent. Reports do not provide causal
explanations or multiplicity-adjusted confidence intervals.

## Roadmap after v0.2

Future work may add learners and adaptation with recovery metrics, reward
shifts and delays, environment-specific dynamics hooks, and Dict and
MultiDiscrete spaces.

## License

MIT; see [LICENSE](LICENSE).
