Metadata-Version: 2.4
Name: caeli-data
Version: 0.8.0
Summary: Python client of the Caeli data repository API (data.caeli.space): load tables, publish rasters, read the catalogue
Author-email: "Caeli B.V." <tim@caeli.nl>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://data.caeli.space
Project-URL: Catalogue, https://catalog.caeli.space
Keywords: caeli,air quality,STAC,Iceberg,data repository
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.27
Requires-Dist: pyarrow>=14
Provides-Extra: aws
Requires-Dist: boto3>=1.26; extra == "aws"
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"

# caeli-data

The Python client of the Caeli data repository API (https://data.caeli.space). One door for every producer and reader
of datasets: the inventory platform, pipelines and colleagues all use this library; nothing writes to the tables or the
buckets directly.

    pip install caeli-data            # caeli-data[aws] also brings boto3, for Repository.from_secret

    from caeli_data import Repository, TableRejected, IngestFailed

    repo = Repository()               # CAELI_API_URL (default https://data.caeli.space) and CAELI_API_KEY
    repo.whoami()                     # {"name": "tim", "role": "admin", ...}

    # a pandas DataFrame with utc_time, latitude, longitude and the value columns: the platform derives the rest
    # (pt_region, pt_tile, row_hash; the coordinates rounded to 5 decimals), splits per region and month, loads, waits
    jobs = repo.load_dataframe(df, "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"}, source_ref="download 4711")
    print(jobs[0]["message"])         # "200000 rows appended in cdse_sentinel_5p_l2_no2, ... item N_EU__2026-09__v1 now ..."

    # a source with its own row identity (a station id, an orbit with scanline and pixel) names it
    jobs = repo.load_dataframe(df, "openaq-stations-no2", row_key="location_id")

    # a prepared Parquet file with the six standard columns already in it, any regions and months
    jobs = repo.load_file("prepared.parquet", "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"})

A complete month delivered again is a new version of that month: `mode="replace", reason="re-import of 2026-09"`.
A service takes its key from Secrets Manager: `Repository.from_secret("caeli/data-platform/api-key-inventory-platform")`.

The rules a table must meet are in Caeli_data_repository_conventions (section 6); the library checks them before
uploading and says which rule is broken (`TableRejected`; for an extra column the message carries the schema change to
make). `IngestFailed` means the platform failed; the same data sent again is the same job, never a double load.
Python 3.9 and later.

A raster (a GeoTIFF on a registered grid) is one item of a raster dataset, area and period:

    from caeli_data import PublishRejected

    job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", software={"caeli-maps": "0.1.0"})
    print(job["message"])             # "NL__2024-05__v1 published as s3://caeli-file-store/datasets/.../v1/... (0.1 MB, 60x38 px ...)"
    job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", mode="revise", reason="recipe v2")

The worker refuses a raster that is not on the area's grid, has no CRS or nodata, or is not a Cloud-Optimized GeoTIFF;
with rasterio and rio-cogeo installed the library checks the same rules before uploading (`PublishRejected`) and
`convert=True` makes the COG for you. `publish_file` does the same for any file or model; `register_grid` (admin) and
`grids()` manage the grids.

A collection is one recipe: register a raster dataset with the configuration that makes it
(`register_dataset(..., recipe=recipe_dict)`, which records it as `caeli:recipe` with its `fingerprint()`), send the
fingerprint with every publication (`publish_raster(..., recipe_fingerprint=fingerprint(recipe_dict))`), and the
platform refuses a map made with another recipe — that is a different dataset.

Reading goes through the same door (0.6.0, API 0.9.x). One call gives the rows of a table dataset as one file or
one DataFrame, at any size:

    repo.download("cdse-sentinel-5p-l2-no2", "no2_nl_2026-06.csv", area="NL", period="2026-06",
                  columns=["product__nitrogendioxide_tropospheric_column", "product__qa_value"],
                  filters=[("product__qa_value", ">=", 0.75)])
    df = repo.read_table("ads-cams-forecast", regions=["N_EU"], start="2026-06-01", end="2026-06-08", columns=["pm2p5", "lead_hours"])
    repo.plan("cdse-sentinel-5p-l2-no2", regions=["N_EU"], period="2026-06", columns=["product__nitrogendioxide_tropospheric_column"])

The API decides beforehand, from the tables' metadata, whether a request is small enough; a request that is too large
comes back as consecutive windows in time, and `download` and `read_table` fetch them one after another and join
them into one file (`split=False` refuses instead, with `ReadTooLarge` carrying the plan). Every download is pinned to
one table snapshot (`version=` takes an item key or `snapshot:<id>`), filters are typed against the schema
(`("qa_value", ">=", 0.75)`, `("orbit", "in", [1, 2])`, `("no2", "notnull")`), and the three index columns
`idx_utc`, `idx_lat`, `idx_lon` are always part of a download. Without Python: `caeli-data plan ...`,
`caeli-data download ... --to file.csv`, `caeli-data reads`.

Maps and other published files are read by their item (0.6.1, API 0.10.0). An item is one area and period of a
dataset (`N_EU__2026-06` is the current version of that month, `N_EU__2026-06__v2` one version), and its file is
fetched through a link the API hands out for a quarter of an hour, straight from the file store, checked against
the record's size and sha256; what is already there is not fetched again:

    repo.items("caeli-iasi-nh3-tc-monthly-wgs001", start="2025-01", end="2025-12")            # the months, by area and period
    repo.download_item("caeli-iasi-nh3-tc-monthly-wgs001", "N_EU__2026-06", "maps")          # one map into the folder maps/
    repo.download_items("caeli-iasi-nh3-tc-monthly-wgs001", "maps", start="2025-01", end="2025-12")   # a year of maps

Without Python: `caeli-data items <dataset> --from 2025-01 --to 2025-12`, `caeli-data fetch <dataset> --from 2025-01
--to 2025-12 --into maps`, `caeli-data fetch <dataset> --item N_EU__2026-06 --into maps`. Admins make keys with
`repo.create_key(name, role, owner_email)` or `caeli-data key create --name piet --role reader --owner piet@caeli.nl`;
the key is shown once and goes to its owner through Secrets Manager (the platform's vault), never through a chat or a mail.

A map series is bound to the recipe that makes it, and its months are made by the maps service on request (0.6.2,
API 0.11.0). The versions of a recipe are numbered by the API and kept in the record and in the library of the file
store (`recipes/<name>/v<N>.json`); a changed recipe becomes the next version (registrar), the same map-making fields
again change nothing. A map job is one period of a series: the API records it, queues it for the maps service and,
from maps 0.3.9 on, follows it from queued to done, exists, empty, failed or rejected:

    repo.recipe("caeli-tropomi-no2-trop-monthly-wgs001")                                   # the current version, recipe included
    repo.bind_recipe_file("caeli-tropomi-no2-trop-monthly-wgs001", "recipes/tropomi-no2-trop-monthly-wgs001.json", notes="why")
    repo.create_map_jobs("caeli-tropomi-no2-trop-monthly-wgs001", start="2019-01", end="2026-10")   # one job per month (publisher)
    repo.map_jobs(dataset="caeli-tropomi-no2-trop-monthly-wgs001", status="queued")
    repo.cancel_map_job(job_id)

Without Python: `caeli-data recipes`, `caeli-data recipe show <dataset>`, `caeli-data recipe bind <dataset> --file
<recipe.json> --notes "why"`, `caeli-data map-jobs create <dataset> --from 2019-01 --to 2026-10`, `caeli-data map-jobs
list --dataset <dataset>`, `caeli-data map-job show <id>`, `caeli-data map-job cancel <id>`.

The lab (0.7.0, API 0.12.0, maps 0.4.0) tries a recipe on a small box and one period without touching any dataset:
the lab tasks of the maps service run it on the box, snapped to whole cells of the recipe's grid, and write `map.tif`,
`preview.png`, `bounds.json` and `result.json` under `_lab/<user>/<job id>/` in the file store, kept seven days (a
reader may ask for 20 a day, a publisher 100). A draft is the recipe a person is editing, kept in the library next to
the bound versions (`recipes/<name>/draft.json`) and never read by the platform itself; the style of a series is the
colour map and value range every month is shown with:

    repo.lab_job(recipe=recipe_dict, period="2026-06", box=[1, 48.5, 8, 53.5])      # or dataset="...", version="current"
    repo.wait_lab_job(job_id); repo.lab_files(job_id); repo.download_lab(job_id, "lab/2026-06")
    repo.draft("tropomi-no2-trop-monthly-wgs001"); repo.save_draft(name, recipe_dict); repo.delete_draft(name)
    repo.set_style("caeli-tropomi-no2-trop-monthly-wgs001", colormap="reds", rescale=[0, 1000])      # registrar

Without Python: `caeli-data lab run --file <recipe.json> --period 2026-06 --box 1,48.5,8,53.5 --wait --into lab`,
`caeli-data lab list`, `caeli-data lab show <id>`, `caeli-data lab fetch <id> --into lab`, `caeli-data draft show | save
| delete`, `caeli-data style set <dataset> --colormap reds --rescale 0,1000`. The laptop mirror of the bound recipes
(`caeli-maps-recipes/recipes`) is `python scripts/mirror_recipes.py --into <folder> --apply`.

Deleting a dataset (0.8.0, API 0.13.0, schema v1.12) can be taken back for thirty days. A delete marks the dataset in
the record (when, by whom, why, and the day of its purge); it leaves the API, the catalogue and the tiles at once, its
queued map jobs are cancelled, but its rows, its table and its files stay until the API's sweep purges it after that
day. A real dataset is deleted by an admin who repeats its key and gives a reason; a `zz-test-*` dataset needs the
reason only, or goes at once with `purge=True` (the tests' clean start). The impact is read first:

    repo.impact("caeli-tropomi-no2-trop-monthly-wgs001")                                    # what a delete would touch, and whether it is blocked
    repo.delete_dataset("caeli-tropomi-no2-trop-monthly-wgs001", reason="why, ten characters or more", confirm=True)   # admin
    repo.deleted_datasets()                                                                  # what can still be restored, with the marks
    repo.restore_dataset("caeli-tropomi-no2-trop-monthly-wgs001")                           # back whole, catalogue included
    repo.delete_dataset("zz-test-load", purge=True)                                          # a test dataset, gone at once

Without Python: `caeli-data dataset impact <dataset>`, `caeli-data dataset delete <dataset> --confirm <dataset> --reason
"why"`, `caeli-data dataset list --deleted`, `caeli-data dataset restore <dataset>`, `caeli-data dataset delete
zz-test-load --purge`.
