Metadata-Version: 2.4
Name: datahub-yaml-source
Version: 0.1.0
Summary: DataHub ingestion source that reads declarative YAML metadata files.
Home-page: https://github.com/davidouagne/datahub-yaml-source
Author: David Ouagne
Author-email: david.ouagne@aphp.fr
License: Apache-2.0
Project-URL: Homepage, https://github.com/davidouagne/datahub-yaml-source
Project-URL: Issues, https://github.com/davidouagne/datahub-yaml-source/issues
Project-URL: Changelog, https://github.com/davidouagne/datahub-yaml-source/releases
Keywords: datahub,metadata,ingestion,yaml,metadata-as-code,data-catalog
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: acryl-datahub<1.8,>=1.7.0.9
Requires-Dist: pyyaml>=6.0
Requires-Dist: pydantic<3.0.0,>=2.4.0
Requires-Dist: requests<3,>=2.28.0
Provides-Extra: git
Requires-Dist: GitPython<4,>=3.1.37; extra == "git"
Provides-Extra: s3
Requires-Dist: boto3<2,>=1.35.0; extra == "s3"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: deepdiff>=6.0.0; extra == "dev"
Requires-Dist: jsonschema>=4.0.0; extra == "dev"
Requires-Dist: GitPython<4,>=3.1.37; extra == "dev"
Requires-Dist: boto3<2,>=1.35.0; extra == "dev"
Requires-Dist: ruff<0.17,>=0.12; extra == "dev"
Requires-Dist: mypy>=1.17; extra == "dev"
Requires-Dist: types-PyYAML>=6.0; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# datahub-yaml-source

[![CI](https://github.com/davidouagne/datahub-yaml-source/actions/workflows/ci.yml/badge.svg)](https://github.com/davidouagne/datahub-yaml-source/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/datahub-yaml-source)](https://pypi.org/project/datahub-yaml-source/)
[![Python versions](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)](https://pypi.org/project/datahub-yaml-source/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://github.com/davidouagne/datahub-yaml-source/blob/main/LICENSE)

A standalone [DataHub](https://datahubproject.io/) ingestion source plugin that
reads a directory tree of declarative YAML "metadata as code" files and emits
the DataHub entities they describe (platforms, tags, glossary, domains,
containers, datasets with schema/lineage, data products, pipelines, pipeline
run history, and data quality assertions).

See [docs/sources/yaml/yaml.md](https://github.com/davidouagne/datahub-yaml-source/blob/main/docs/sources/yaml/yaml.md) for a narrative
introduction, [docs/sources/yaml/reference.md](https://github.com/davidouagne/datahub-yaml-source/blob/main/docs/sources/yaml/reference.md)
for a generated field-by-field reference of every `kind`, and
[docs/sources/yaml/yaml_recipe.yml](https://github.com/davidouagne/datahub-yaml-source/blob/main/docs/sources/yaml/yaml_recipe.yml) for an
example recipe. See [_PLANNING.md](https://github.com/davidouagne/datahub-yaml-source/blob/main/_PLANNING.md) for the architecture
decisions behind this connector.

A [JSON Schema](https://github.com/davidouagne/datahub-yaml-source/blob/main/docs/sources/yaml/schema/yaml-metadata.schema.json) for the
document format (autocomplete/validation in VS Code, IntelliJ, ...) is also
generated from the Pydantic models -- see the "Editor autocomplete and
validation" section in `yaml.md`.

Both `reference.md` and the JSON Schema are generated from
`src/datahub_yaml_source/models.py`; regenerate them after changing a model:

```bash
python scripts/generate_json_schema.py
python scripts/generate_markdown_docs.py
```

## Installation

```bash
pip install -e .
```

This registers the `yaml` source type with `acryl-datahub` via a
`datahub.ingestion.source.plugins` entry point. Verify it's picked up with:

```bash
datahub check plugins
```

## Usage

```bash
datahub ingest -c docs/sources/yaml/yaml_recipe.yml
```

## Development

```bash
pip install -e ".[dev]"
pytest tests/unit                 # unit tests
pytest tests/integration          # integration test against a curated fixture,
                                   # golden-file checked
```

To regenerate the integration golden file after an intentional output change:

```bash
pytest tests/integration/yaml_source/test_yaml_source_golden.py --update-golden-files
```
