Metadata-Version: 2.4
Name: al-data-utils
Version: 0.3.2
Summary: Reusable utilities for data engineering (Airflow, Azure, Snowflake, SQL, config).
Author-email: AL Data Labs <pypi@aldatalabs.com>
License-Expression: MIT
Project-URL: Homepage, https://aldatalabs.com
Keywords: airflow,data-engineering,azure,snowflake,sql
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml
Dynamic: license-file

# al-data-utils

Reusable Python utilities for data and automation workflows.

This repository contains independent utility modules for sync operations, git automation, SQL work, logging helpers, and MSSQL schema snapshot generation.

## Install

```bash
pip install al-data-utils
```

Python 3.10+. Runtime dependencies (`pyyaml`, `sqlalchemy`, `pandas`, `requests`, `msal`, ...) are expected from the host environment, typically an Airflow image.

## Modules

- `airflow_utils` — Airflow Variable / connection helpers
- `azure_as_utils` — Azure Analysis Services model refresh (REST + MSAL)
- `backup_utils` — MSSQL schema snapshot to filesystem
- `config_utils` — YAML config loading
- `files_sync_utils` — folder inventory, sync plan, copy, retention
- `geo_utils` — geo helpers
- `git_utils` — git automation wrappers
- `logging_utils` — unified logging
- `ones_api_utils` — 1C/ОneS API client
- `sql_utils` — `UnifiedSQLClient` (MSSQL / PostgreSQL / Snowflake via SQLAlchemy)
- `tg_utils` — Telegram helpers

## Examples

Runnable examples are available in the `examples` folder of the repository for the main utility groups.
Use them as reference implementations for local scripts and automation jobs.

## MSSQL Schema Snapshot

The snapshot tool exports SQL objects from MSSQL into a filesystem layout and syncs staging output to destination folders.

Current behavior highlights:

- Supports single DB and multi-DB runs in one execution.
- Always writes outputs into per-DB folders.
- Uses staging first, then syncs to final output.
- Supports cleanup of staging folders.
- Supports sync log retention cleanup.
- Supports include and exclude filters per object type.
- Supports DataGrip template override files.

Output shape:

```text
<MSSQL_OUT_DIR>/<DatabaseName>/schemas/<schema>/
	tables/
	views/
	routines/
	sequences/
	synonyms/
```

### Snapshot Environment Variables

Core connection and output:

- MSSQL_OUT_DIR
- MSSQL_SERVER
- MSSQL_DATABASE
- MSSQL_DATABASES
- MSSQL_AUTH_MODE
- MSSQL_USER
- MSSQL_PASSWORD
- MSSQL_DRIVER

Snapshot behavior:

- MSSQL_ENABLE_TABLE_DEFINITIONS
- MSSQL_ENABLE_INDEXES
- MSSQL_ENABLE_FOREIGN_KEYS
- MSSQL_ENABLE_VIEWS
- MSSQL_ENABLE_STORED_PROCEDURES
- MSSQL_ENABLE_FUNCTIONS
- MSSQL_ENABLE_SEQUENCES
- MSSQL_ENABLE_SYNONYMS
- MSSQL_CLEANUP_STAGING
- MSSQL_SYNC_LOG_RETENTION_DAYS

Filters:

- MSSQL_INCLUDE_SCHEMAS
- MSSQL_EXCLUDE_SCHEMAS
- MSSQL_INCLUDE_TABLES
- MSSQL_EXCLUDE_TABLES
- MSSQL_INCLUDE_VIEWS
- MSSQL_EXCLUDE_VIEWS
- MSSQL_INCLUDE_STORED_PROCEDURES
- MSSQL_EXCLUDE_STORED_PROCEDURES
- MSSQL_INCLUDE_FUNCTIONS
- MSSQL_EXCLUDE_FUNCTIONS
- MSSQL_INCLUDE_SEQUENCES
- MSSQL_EXCLUDE_SEQUENCES
- MSSQL_INCLUDE_SYNONYMS
- MSSQL_EXCLUDE_SYNONYMS

Notes:

- If MSSQL_DATABASES is set, it is treated as a comma-separated list.
- If MSSQL_DATABASES is empty, MSSQL_DATABASE is used.
- If MSSQL_DATABASE contains commas, it is also treated as a list for backward compatibility.

## File Sync Utilities

`files_sync_utils` provides:

- Inventory building
- Plan generation
- Copy and update execution
- Extra destination cleanup rules
- Activity CSV logging
- Log retention cleanup

## Git Utilities

`git_utils` wraps common git automation operations with consistent behavior and error handling.

## Changelog

See `CHANGELOG.md` in the repository for release notes.
