Metadata-Version: 2.4
Name: mostlyright-data
Version: 0.25.5
Summary: Mostly Right hosted CLI for reviewed datasets
Project-URL: Homepage, https://mostlyright.md/
Project-URL: Documentation, https://mostlyright.md/docs/guides/cli/
Requires-Python: >=3.11
Provides-Extra: build
Requires-Dist: hatchling==1.28.0; extra == 'build'
Requires-Dist: uv-build==0.11.3; extra == 'build'
Description-Content-Type: text/markdown

# mostlyright-data

Browse datasets built by our community or create your own with agents that find the sources and
configure the pipeline. Mostly Right treats a successful run as one readable dataset version;
publishing that version does not by itself prove that catch-up or scheduled refresh is running.

`mostlyright-data` is the command-line client. It probes sources, registers one recipe document,
starts and follows hosted dataset runs, reads what they delivered, and downloads the datasets after
they finish.

## Requirements

Use CPython 3.11 or newer.

## Install

```bash
python -m pip install mostlyright-data
```

That is the whole install. There is no second profile and no lane to choose: every `mr-data`
command sends its work to the backend, and it works on Linux, macOS, and Windows.

It also carries the `mr-data-build` agent skill, which Claude Code and Codex read to author a
recipe without rediscovering the contract a run at a time. The first `mr-data` command you run
places it in your agent's skill directory; there is nothing to install by hand. A skill you have
edited yourself is never replaced, and `MOSTLYRIGHT_SKILL_AUTOINSTALL=0` switches the placement
off.

## Sign in

```bash
mr-data login
mr-data whoami
```

`login` stores a device key on the current machine. `whoami` checks that key.

New macOS/Linux logins use `~/.mostlyright/credentials.secure`, an unencrypted file restricted to
your OS account (0600 inside a 0700 directory). Processes running as you and backups can read it.
Older file-based logins migrate to this file without requesting Keychain access. Windows keeps
DPAPI encryption as its default.
Existing Keychain, DPAPI, or Secret Service logins keep their selected store; an unavailable store
never causes an automatic switch to file storage.

If an existing macOS login requests Keychain access, it is requesting the `md.mostlyright.cli`
device credential. To switch that login to file storage, restore access to its Keychain item,
run `mr-data auth logout` successfully, then run `mr-data login`. The `--credential-store secure-file`
option does not move an existing native-store login.

## Check the installation

```bash
mr-data --version
mr-data --help
```

## Build a dataset

Create the dataset, choose its primary category from the fixed vocabulary, then register a recipe document with the sources, table schema, transformations and checks. Use the created dataset ID in the recipe:

```bash
mr-data dataset create --name "Dataset title" --json
mr-data dataset categories --json
mr-data dataset set DATASET_ID --category climate-environment
mr-data recipe recipe.json --json
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --full
mr-data status RUN_ID
mr-data checks RUN_ID
mr-data download RUN_ID --output ./out
```

Full builds are progressive by default: one acquisition exposes an inspection checkpoint after
about five minutes and continues toward the finished table. Inspect it with
`mr-data status RUN_ID --inspection --json`; the checkpoint does not pause for approval or start
another full acquisition. Any required spend confirmation happens before acquisition.
`--progressive` remains a compatibility alias. Use `--sample` only for an explicit standalone
bounded inspection; row ceilings do not impose a five-minute acquisition limit.

Run submission is idempotent: sending identical recipe, mode and bounds returns the same run,
including a previous failure. The CLI reports a non-queued result as `run_returned` and exits
with code 2 when that run failed. Read its failure before retrying; `mr-data run --retry RUN_ID`
can reissue supported platform execution failures without changing the recipe.

Use the identifiers returned by registration and run submission. `mr-data dataset create`
creates a dataset page before a recipe is ready. `mr-data watch RUN_ID` follows a submitted run.
A scheduled refresh runs only Studio's persisted bounded source-action plan; it never silently
falls back to a mutable whole-source fetch. `mr-data table resync TABLE_ID --request-id UUID`
explicitly requests a full source reread when that is needed. Choose and retain the UUID before
submitting: if the response is lost, repeat that exact command with the same UUID rather than
starting another reread. A persisted spend hold is still an accepted resync: its receipt names
the run, projection and exact `mr-data run --confirm-held RUN_ID` action. New full resyncs use
progressive acquisition too. Existing legacy sample-first pairs can still finish through their
original confirmation and approval actions. `mr-data recipe readiness` reads Studio's paginated,
immutable-revision inventory before a production-wide schedule sweep: predecessor,
incremental-materialization and differential-proof readiness, together with each source's next
action. It reports Studio's persisted facts; it does not authorize a schedule.

To request an offline replay of retained inputs with a registered revision:

```bash
mr-data run --recipe RECIPE_ID --digest RECIPE_DIGEST --mode replay --sources-from RUN_ID
```

Studio must have replay enabled and the named successful run must belong to the same table with
its raw inputs still retained. Replay compares against that run and never becomes the live version.
Studio returns a typed refusal when replay is unavailable; the CLI does not fetch sources locally.

The hosted engine executes sample, full and refresh runs. Studio's action classifier can name
closed-source reuse, request-window or collection acquisition, and recorded stream input, but
normal refresh admits only two families of plan: every source a recorded stream, and at least one
direct partition request window with a compatible predecessor, with `closed` reuse permitted
beside it. Once such a plan holds more than one source, every window in it must agree on one
`merge.materialization`, `merge.partition.column` and `merge.partition.key`, and that column must
be one the table declares. Closed-only plans, any plan carrying a collection, a window beside a
recorded stream, a mutable snapshot and any other unwindowed source return `RESYNC_REQUIRED`
before acquisition. Nothing is revalidated or re-acquired whole as an implicit fallback. Use
`mr-data table resync TABLE_ID --request-id UUID` when a deliberate full source reread is
intended, retaining that UUID through an uncertain response. URL windows may use declared query
parameters or path placeholders; a fixed date URL does not advance automatically.
Configure a correction lookback where the publisher can revise earlier observations.

A source exposing only its current snapshot cannot supply a historical delta. It is supported only
through an explicit table resync; refresh does not turn its whole current result into an
incremental update. Recorded streams use their registered capture and continuation semantics.

See [Recipe documents](docs/RECIPE-DOCUMENT.md) for source-window and bootstrap contracts.
A table's first succeeded run goes live on its own, whatever mode it was; `mr-data promote
TABLE_ID` records how often it refreshes and why, and puts back a table that was withdrawn. Claim
a table is live and current only when Studio returns that evidence.

## There is no local execution

There used to be. A second product lane built datasets on your own machine, and `mr-data` carried
about fifty commands for making, running, serving, indexing, checking and signing a build here.
That lane is gone: the engine it ran now ships only inside the backend's worker images.

`mr-data --help` lists every command this product has. The execution engine and its dependency
closure live in Mostly Right Studio's `apps/worker`; this repository ships no engine extras,
worker executables or image publisher.

## Documentation

- [Install and sign in](https://mostlyright.md/docs/start/install/)
- [Build your first dataset](https://mostlyright.md/docs/start/first-dataset/)
- [CLI reference](https://mostlyright.md/docs/reference/cli/)
- [Recipe reference](https://mostlyright.md/docs/reference/recipe/)
- [Recipe examples](https://mostlyright.md/docs/recipes/)
- [Join market settlements](https://mostlyright.md/docs/recipes/market-settlement-join/)
- [Compare a forecast with observations](https://mostlyright.md/docs/recipes/forecast-vs-observation/)
- [Model a snapshot source](https://mostlyright.md/docs/recipes/snapshot-window/) — normal refresh
  is unavailable until its bounded materializer ships; use explicit table resync today
- [Union many weather stations](https://mostlyright.md/docs/recipes/many-station-weather/)
- [Aggregate a stream into bars](https://mostlyright.md/docs/recipes/stream-to-bars/)
- [Use a public dataset](https://mostlyright.md/docs/guides/use-public-datasets/)
- [Certified document extraction](docs/DOCUMENT-EXTRACTION.md)
- [Use a table to drive a stream's market roster](docs/TABLE-DRIVEN-DISCOVERY.md)

Use `mr-data --help` for the full command list and options. Commands that support
`--json` write one JSON object.

`--output` may use a relative path.

## Behavior and limits

- Weather Reader versions 1 and 2 accept one extracted GRIB2 record and refuse a multi-record GFS
  or HRRR file. Version 3 reads a bounded collection and selects exactly one sealed record.
- A single-address credential-free public HTTPS source uses the reviewed hosted route. Bounded
  collections have no route at all; the retired local lane was their only route.
- A recipe supports up to 256 declared sources. Source count is separate from bounded request concurrency and resource limits. Existing source adapters and Readers retain their own format, pagination, byte, row and credential contracts; a URL collection is not an implicit permission to crawl arbitrary links.
- A keyed stream venue is named by a credential reference the backend holds, never by a value. No command flag takes a secret as an argument. Venues reached over an actual WebSocket are covered; a venue whose credential comes from an interactive human login, or that is not WebSocket at all, is not.

## Verification

```bash
make test
```

`make test` runs the thin client suite while editing. `make verify` is required
before review or merge.

## Development

```bash
uv sync --locked
```

There is no `dev` extra to ask for -- `uv run --extra dev` answers
``error: Extra `dev` is not defined in the project's `optional-dependencies` table``. The test
tooling is the default dependency group, which `uv sync` installs.

`make lint` is the repository's one lint set (`uv run ruff check .` plus `uv lock --check`); it is
the first thing `make verify` runs and the same target the Linux leg of `thin-client-smoke.yml`
runs, so the two cannot disagree about what is linted.

The test suite checks client behavior, contract compatibility and thin package ownership. Backend
execution and worker-boundary tests run in Studio. The client supports Linux, macOS and Windows.

To update the public Studio contract pin, run `make repin STUDIO_GIT=/path/to/studio STUDIO_REF=<commit>`. It reads committed OpenAPI and JSON Schema bytes, records their hashes and source commit in `vendor/studio-contracts/client-v4-pin.json`, and updates the thin client. Run the cross-repository tests against that Studio revision before release. Worker/client generation and backend job-contract repins now belong to Studio.
