Metadata-Version: 2.4
Name: fiboa-cli
Version: 0.30.0
Summary: CLI tools such as validation and file format conversion for fiboa.
Author: Matthias Mohr, Ivor Bosloper
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/fiboa/cli
Project-URL: Bug Reports, https://github.com/fiboa/cli/issues
Project-URL: Source, https://github.com/fiboa/cli
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: vecorel-cli==0.4.2
Requires-Dist: beautifulsoup4>=4.12
Requires-Dist: spdx-license-list==3.27.0
Dynamic: license-file

# fiboa CLI

A command-line interface (CLI) for working with fiboa files.

- [Getting Started](#getting-started)
- [Documentation / Commands](#commands)
- [Development](#development)

## Getting Started

In order to make working with fiboa easier we have developed command-line interface (CLI) tools such as
inspection, validation and file format conversions.

### Installation

#### Using Pixi (Recommended)

This project uses [Pixi](https://pixi.sh/) for dependency management. Install Pixi first, then:

```bash
# Clone the repository and navigate to it
git clone https://github.com/fiboa/cli.git
cd cli

# Install all dependencies
pixi install

# Run the CLI
pixi run fiboa
```

#### Using pip

Alternatively, you can install from PyPI with **Python 3.11** or any later version:

```bash
pip install fiboa-cli
```

### Execute a command

After the installation you should be able to run the following command: `fiboa` (or `pixi run fiboa` if using Pixi)

You should see usage instructions and [available commands](#commands) for the CLI.

fiboa CLI supports various commands to work with the files:

- [fiboa CLI](#fiboa-cli)
  - [Getting Started](#getting-started)
    - [Installation](#installation)
      - [Using Pixi (Recommended)](#using-pixi-recommended)
      - [Using pip](#using-pip)
    - [Execute a command](#execute-a-command)
  - [Commands](#commands)
    - [Validation](#validation)
    - [Create fiboa GeoParquet from GeoJSON](#create-fiboa-geoparquet-from-geojson)
    - [Create fiboa GeoJSON from GeoParquet](#create-fiboa-geojson-from-geoparquet)
    - [Inspect fiboa GeoParquet file](#inspect-fiboa-geoparquet-file)
    - [Merge fiboa GeoParquet files](#merge-fiboa-geoparquet-files)
    - [Create JSON Schema from fiboa Schema](#create-json-schema-from-fiboa-schema)
    - [Validate a Vecorel Schema](#validate-a-vecorel-schema)
    - [Improve a fiboa Parquet file](#improve-a-fiboa-parquet-file)
    - [Update an extension template with new names](#update-an-extension-template-with-new-names)
    - [Converter for existing datasets](#converter-for-existing-datasets)
    - [Publishing with Portolan](#publishing-with-portolan)
  - [Development](#development)
    - [Implement a converter](#implement-a-converter)
  - [Run in Docker](#run-in-docker)

## Commands

### Validation

To validate a fiboa GeoParquet or GeoJSON file, you can for example run:

- GeoJSON: `fiboa validate example.json --collection collection.json`
- GeoParquet: `fiboa validate example.parquet --data`

Check `fiboa validate --help` for more details.

The validator also supports remote files.

- `http://` or `https://`: no further configuration is needed.
- `s3://`: With Pixi, run `pixi install -e s3` or with pip, run `pip install fiboa-cli[s3]` and you may need to set environment variables.
  Refer to the [s3fs credentials documentation](https://s3fs.readthedocs.io/en/latest/#credentials) for how to define credentials.
- `gs://`: With Pixi, run `pixi install -e gcs` or with pip, run `pip install fiboa-cli[gcs]`.
  By default, `gcsfs` will attempt to use your default gcloud credentials or, attempt to get credentials from the google metadata service, or fall back to anonymous access.

### Create fiboa GeoParquet from GeoJSON

To create a fiboa-compliant GeoParquet for a fiboa-compliant set of GeoJSON files containing Features or FeatureCollections,
you can for example run:

- `fiboa create-geoparquet geojson/example.json -o example.parquet -c geojson/collection.json`

Check `fiboa create-geoparquet --help` for more details.

### Create fiboa GeoJSON from GeoParquet

To create one or multiple fiboa-compliant GeoJSON file(s) for a fiboa-compliant GeoParquet file,
you can for example run:

- GeoJSON FeatureCollection:
  `fiboa create-geojson example.parquet -o dest-folder`
- GeoJSON Features (with indentation and max. 100 features):
  `fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f`

  `fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f`

Check `fiboa create-geojson --help` for more details.

### Inspect fiboa GeoParquet file

To look into a fiboa GeoParquet file to get a rough understanding of the content, the following can be executed:

- `fiboa describe example.parquet`

Check `fiboa describe --help` for more details.

### Merge fiboa GeoParquet files

Merges multiple fiboa datasets to a combined fiboa dataset:

- `fiboa merge ec_ee.parquet ec_lv.parquet -o merged.parquet -e https://fiboa.org/hcat-extension/v0.1.0/schema.yaml -i ec:hcat_name -i ec:hcat_code -i ec:translated_name`

Check `fiboa merge --help` for more details.

### Create JSON Schema from fiboa Schema

To create a JSON Schema for a fiboa Schema YAML file, you can for example run:

- `fiboa jsonschema example.json --id=https://vecorel.org/specification/v0.3.0/geojson/schema.json -o schema.json`

Check `fiboa jsonschema --help` for more details.

### Validate a Vecorel Schema

To validate a Vecorel Schema YAML file, you can for example run:

- `fiboa validate-schema schema/schema.yaml`

Check `fiboa validate-schema --help` for more details.

### Improve a fiboa Parquet file

Various "improvements" can be applied to a fiboa GeoParquet file.
The commands allows to

- detect fiboa-0.2 files and convert them to fiboa-0.3
- change the CRS (`--crs`)
- change the GeoParquet version (`-gp1`) and compression (`-pc`)
- add/fill missing perimeter/area values (`-sz`)
- fix invalid geometries (`-g`)
- rename columns (`-r`)
- add HCAT columns (`--hcat=mapping.csv`) based on the input crop:code (from crop-extension) and a hcat-mapping csv file

Example:

- `fiboa improve file.parquet -o file2.parquet -g -sz -r old=new -pc zstd`

Check `fiboa improve --help` for more details.

### Update an extension template with new names

Once you've created and git cloned a new extension, you can use the CLI
to update all template placeholders with proper names.

For example, if your extension is meant to have

- the title "Administrative Division Extension",
- the prefix `admin` (e.g. field `admin:country_code` or `admin:subdivision_code`),
- is hosted at `https://github.io/vecorel/administrative-division-extension`
  (organization: `vecorel`, repository `/administrative-division-extension`),
- and you run Vecorel in the folder of the extension.

Then the following command could be used:

- `fiboa rename-extension . -t "Administrative Division" -p admin -s administrative-division-extension -o vecorel`

Check `fiboa rename-extension --help` for more details.

### Converter for existing datasets

The CLI ships various converters for existing datasets.

To get a list of available converters/datasets with title, license, etc. run:

- `fiboa converters`

Use any of the IDs from the list to convert an existing dataset to fiboa:

- `fiboa convert de_nrw`

See [Implement a converter](#implement-a-converter) for details about how to

### Publishing with Portolan

fiboa datasets are published as a [Portolan](https://github.com/portolan-sdi/portolan-cli) catalog. The fiboa CLI
converts and validates; Portolan writes the STAC metadata, the PMTiles, the checksums and the README, and uploads.
The steps below are written so that an agent can follow them; the
[Portolan skills](https://github.com/portolan-sdi/portolan-skills) cover the Portolan side in more depth.

Inside a Portolan catalog (`portolan init`), for dataset `<id>` and edition `<variant>`:

1. Convert and validate. Name the file after the edition, so each edition is its own asset:
   ```bash
   fiboa convert <id> --variant <variant> -c <cache> -o <id>/<id>-<variant>.parquet
   fiboa validate <id>/<id>-<variant>.parquet
   ```
2. Seed the collection metadata from the converter: title, description, providers, license, fiboa version, and the
   columns with a description for every fiboa property. The columns move to the collection; the assets go, as
   Portolan adds its own. Only for a new collection; Portolan keeps these fields afterwards.
   ```bash
   fiboa create-stac-collection <id>/<id>-<variant>.parquet -o <id>/stac.json
   jq '."table:columns" = .assets.data."table:columns" | del(.assets)' <id>/stac.json > <id>/collection.json
   rm <id>/stac.json
   ```
3. Add the file with its campaign date, and generate the PMTiles (requires
   [tippecanoe](https://github.com/felt/tippecanoe)):
   ```bash
   portolan add <id> --datetime <variant>-01-01 --pmtiles
   ```
4. Describe the dataset from its [data survey](https://github.com/fiboa/data-survey/tree/main/data): usually
   `<ID>.md` with the id upper-cased and `_` as `-` (`de_nrw` is `DE-NRW.md`), otherwise the country's file
   (`nl_block` is in `NL.md`).
   - Run `portolan metadata init <id>` and fill `<id>/.portolan/metadata.yaml`:

     | data survey | metadata.yaml |
     |---|---|
     | Data Provider (Legal Entity) | `providers`, role `producer` (and `licensor`) |
     | Homepage, Data URL | `source_url` |
     | License | `license`, `license_url` |
     | Overview and the dataset's section | `description` |
     | Caveats in the text (coverage, preliminary editions) | `known_issues` |
     | the fiboa project, publishing this copy | `contact`, and `providers` with role `host` |

   - Describe the dataset's own columns in `<id>/collection.json` (`table:columns[].description`) from the
     survey's Properties table; the converter's `columns` show which source column each one comes from.
   - Run `portolan readme <id>`.
5. Complete what else Portolan asks for until `portolan check` passes, such as a thumbnail (skill
   `portolan-thumbnails`).
6. Upload: `portolan push <remote> --collection <id>` (skill `sourcecoop` for Source Cooperative).

## Development

This project uses [Pixi](https://pixi.sh/) for dependency management and development workflows.

```bash
# Install all dependencies including development tools
pixi install -e dev

# Install the package in editable mode
pixi run install-dev

# Run tests
pixi run test

# Format and lint code
pixi run format
pixi run lint

# Run all checks (lint, format, test)
pixi run check

# Install and run pre-commit
pixi run pre-commit-install
pixi run pre-commit-run
```

### Implement a converter

The following high-level description gives an idea how to implement a converter in fiboa CLI:

1. Create a new file in `fiboa_cli/datasets` based on the `template.py`
2. Fill in the required variables / test it / run it
3. Add missing dependencies into the appropriate feature group in `pixi.toml` (or `setup.py` for pip users)
4. Add the converter to the list above
5. Create a PR to submit your converter for review

## Run in Docker

There's a Dockerfile based on a modern Ubuntu+GDAL image. With the Dockerfile, you can run the CLI in a container.
This can be handy if you lack the required dependencies on you local machine (e.g. a modern GDAL capable of 
reading GeoParquet files).

Example usage (mounting /tmp/fiboa as a volume)

```
docker build . -t fiboa
docker run -it --rm -v /tmp/fiboa:/fiboa fiboa bash

fiboa convert de_nrw -o /fiboa/de_nrw.parquet
```

If you want to temporarily work on a specific branch of fiboa-cli, you can use the following in the container. 
This works because the fiboa-cli is pip-installed with `-e` (in-place):

```
cd fiboa && git checkout <branch>
```
