# epftoolbox2

> Python library for electricity price forecasting research. It downloads market data (ENTSO-E, Open-Meteo, calendars, CSV), builds leak-free features, runs rolling-window forecasts with one model per period of day, horizon and run day, and evaluates them. `DataPipeline` produces one DataFrame, `ModelPipeline` fits models and returns an `EvaluationReport`, and `Workflow` runs both from YAML. Install with `pip install epftoolbox2` (Python 3.10 or newer).

This file is an index. Read the rules below, then open only the pages you need. Links are repo-relative paths to the page sources. The same pages are published at `https://dawidlinek.github.io/epftoolbox2/<slug>/`, where `<slug>` is the path under `docs/src/content/docs/` without its extension, for example `data-pipeline/overview/`.

Rules to know before using the library:

- **Time zones.** Every source returns a UTC index. Set the market zone once with `DataPipeline(timezone="Europe/Warsaw")`. `TimezoneTransformer`, `CalendarSource` and `OpenMeteoSource` inherit it when their own zone is unset, and the dates and keywords (`"today"`, `"today_d+7"`) passed to `run()` then mean local calendar days.
- **DST.** A local day has 23, 24 or 25 hours. Nothing invents the missing 02:00 hour. Day lags shift by local calendar days, not by 24 hours. `DSTNormalizeTransformer` gives every day 24 slots when a model needs them.
- **Column names** are relative to the row's local date D. `x_d-k` is `x` on D-k and `x_d+h` is `x` on D+h as known on D. Targets are `{target}_d+{h}`. Predictors may contain `{horizon}`, for example `"temperature_2m_d+{horizon}"`, or be callables `(horizon)` or `(horizon, hour)`.
- **Transformer order:** `TimezoneTransformer`, `ResampleTransformer`, `ImputeTransformer`, feature transformers (`DailyProfileTransformer`, `DailyStatsTransformer`, `LambdaTransformer`, `DSTNormalizeTransformer`), then `LagTransformer`. Lagging a UTC index shifts by UTC days and is wrong around every DST switch.
- **Information set.** A forecast issued on day D may only use values known at issue time. `ModelPipeline(issue_time="11:45", issue_timezone="Europe/Brussels")` checks every predictor and the newest training target before any fit and raises `InformationSetError`. Without `issue_time` or `information_set` the check is skipped. `InformationSet()` defaults to 11:45 Europe/Brussels, 15 minutes before the 12:00 CET gate closure.
- **Leak-free inputs.** Realised values such as `load_actual` and `generation_*` are known only as `_d-1` or older. `load_forecast` is usable unshifted or as `_d+1`, and `_d+2` and later are look-ahead. The source pages list the safe shifts for every column.
- **Open-Meteo.** `{variable}_d+i` is the forecast for the same local hour on D+i, taken from runs initialised at or before D at `run`:00 UTC (`run` defaults to `"00"` and is a UTC hour, not a local issue time). `api="previous_runs"` (default) has the long archive and caps `horizon` at 6 (`jma_seamless`, `gfs_seamless`, `ecmwf_ifs025`, `ecmwf_ifs`) or 5 (`icon_seamless` and other models), so `d+7` needs `api="single_runs"` (horizon 1 to 14, shorter archive). `horizon=None` picks the largest value without look-ahead. A zone is required, directly or from the pipeline.
- **Data cache.** `pipeline.run(start, end, cache=True)` keeps one cache per source under `.cache/sources/`, keyed by its `get_cache_config()`. The last two days before now are never recorded as cached, so a daily run sees newly published data. A cached range is not revised, so clear the cache to pick up provider revisions. `cache="file.parquet"` stores the whole source output in one file instead.
- **Resumable training.** `ModelPipeline.run(..., save_dir="results")` writes one JSONL file and one manifest per model and computes only the missing tasks on a rerun. Results made with a different configuration raise `ManifestMismatchError`.
- **Training boundary.** For a run day D the training targets end at D-1. Horizons are 1 to 7 days ahead. Each (period of day, horizon, run day) is one task in a process pool, so on Windows (which spawns workers) scripts that call `run()` at top level need an `if __name__ == "__main__":` guard.
- **Missing values.** A NaN predictor skips that forecast. Skips are counted, never dropped silently. `report.forecast_counts()` shows planned, produced and missing forecasts per model. Compare models only on identical forecast keys and report how many forecasts each produced.
- **YAML.** `DataPipeline`, `ModelPipeline` and `Workflow` have `save(path)` and `load(path)`. `save()` writes the ENTSO-E key as `${ENTSOE_API_KEY}` and `load()` expands it from the environment. Pass `include_secrets=True` only for files you never commit. Callable predictors cannot be saved, so use `{horizon}` templates. An unregistered class name raises on load.
- **Custom components** must be registered in `COMPONENT_REGISTRY` to load from YAML, see the Extending page.
- **Examples.** Runnable scripts are in `examples/` and need `ENTSOE_API_KEY` in the environment. Each has a `main()` that takes its date ranges and, if it writes files, an output directory. Worked snippets are on the two Examples pages.

## Getting started

- [Introduction](docs/src/content/docs/getting-started/introduction.mdx): what the library does, its architecture and key features
- [Installation](docs/src/content/docs/getting-started/installation.mdx): requirements, free-threaded Python setup, process environment variables and platform notes
- [Quick Start](docs/src/content/docs/getting-started/quickstart.mdx): a first data pipeline and a first forecast
- [CLI Reference](docs/src/content/docs/getting-started/cli.mdx): the `epf` commands, flags and interactive wizards

## Data pipeline

- [Data Pipeline Overview](docs/src/content/docs/data-pipeline/overview.mdx): sources, transformers, validators, date formats and local-day semantics of `DataPipeline.run()`
- [Pipeline Examples](docs/src/content/docs/data-pipeline/examples.md): from a minimal ENTSO-E pipeline to a full Polish one with weather and validators
- [Data Caching](docs/src/content/docs/data-pipeline/caching.mdx): per-source cache, single-file cache, what is recorded as cached, cache keys and clearing
- [Serialization](docs/src/content/docs/data-pipeline/serialization.md): YAML format of a `DataPipeline`, secrets, relative paths and the timezone key

## Data sources

- [EntsoeSource](docs/src/content/docs/data-sources/entsoe.mdx): ENTSO-E load, price, generation, transfer and installed capacity, with output columns and safe shifts
- [OpenMeteoSource](docs/src/content/docs/data-sources/openmeteo.mdx): archived weather forecasts by cutoff run, models, horizon limits and several locations
- [CalendarSource](docs/src/content/docs/data-sources/calendar.mdx): holidays, school holidays, weekday, hour, month and daylight features
- [CsvSource](docs/src/content/docs/data-sources/csv.md): time series from a CSV file
- [EpftoolboxBenchmarkSource](docs/src/content/docs/data-sources/epftoolbox-benchmark.mdx): the five benchmark datasets (NP, PJM, FR, BE, DE) of Lago et al. (2021)

## Transformers

- [TimezoneTransformer](docs/src/content/docs/transformers/timezone.mdx): UTC index to local time and daily values onto local midnight
- [ResampleTransformer](docs/src/content/docs/transformers/resample.mdx): one regular frequency without inventing data
- [ImputeTransformer](docs/src/content/docs/transformers/impute.mdx): gap filling with leak-safe methods only
- [DSTNormalizeTransformer](docs/src/content/docs/transformers/dst-normalize.mdx): exactly 24 (or 96) slots per local day
- [DailyProfileTransformer](docs/src/content/docs/transformers/daily-profile.mdx): a local day's hourly values copied onto every row of that day
- [DailyStatsTransformer](docs/src/content/docs/transformers/daily-stats.mdx): per local day minimum, maximum, mean and last value
- [LambdaTransformer](docs/src/content/docs/transformers/lambda-transformer.mdx): custom column arithmetic or a full custom transform without a new class
- [LagTransformer](docs/src/content/docs/transformers/lag.mdx): `_d-k` and `_d+k` lag columns by local calendar day

## Validators

- [Validators overview](docs/src/content/docs/validators/overview.md): which validator catches what and how to read the results
- [DSTStructureValidator](docs/src/content/docs/validators/dst-structure.md): rows per local day (23, 24 or 25 hours)
- [CoverageValidator](docs/src/content/docs/validators/coverage.md): index checks and per-column coverage and missing runs
- [PlausibilityValidator](docs/src/content/docs/validators/plausibility.md): value ranges such as SDAC price limits and frozen constant runs
- [EdaValidator](docs/src/content/docs/validators/eda.md): exploratory statistics shown in the terminal

## Model pipeline

- [Model Pipeline Overview](docs/src/content/docs/model-pipeline/overview.mdx): `ModelPipeline.run()`, predictors, pre-fit data checks and parallelism
- [Model Pipeline Examples](docs/src/content/docs/model-pipeline/examples.md): complete `ModelPipeline` usage examples
- [Working with Results](docs/src/content/docs/model-pipeline/results.md): `EvaluationReport` breakdowns, forecast counts and raw predictions
- [Incremental Training](docs/src/content/docs/model-pipeline/caching.mdx): `save_dir` results, resuming and the manifest
- [Serialization](docs/src/content/docs/model-pipeline/serialization.mdx): YAML format of a `ModelPipeline`
- [Forecast-Only Mode](docs/src/content/docs/model-pipeline/forecasting.mdx): operational forecasts for the next days without back-testing
- [Information Set](docs/src/content/docs/model-pipeline/information-set.mdx): availability rules, the issue-time check and the leakage probe
- [Predictor Presets](docs/src/content/docs/model-pipeline/presets.mdx): LEAR and expert-model inputs in this library's column naming

## Workflow

- [Workflow](docs/src/content/docs/workflow/overview.mdx): YAML experiment orchestration, environment settings, overrides and SLURM array jobs

## Models

- [OLSModel](docs/src/content/docs/models/ols.mdx): ordinary least squares
- [LassoCVModel](docs/src/content/docs/models/lasso.mdx): Lasso with cross-validated regularization
- [Scalers and VSTs](docs/src/content/docs/models/scalers.mdx): standard scaling and variance-stabilising transformations (asinh, N-PIT) for spiky prices

## Evaluators

- [MAEEvaluator](docs/src/content/docs/evaluators/mae.md): mean absolute error
- [RMSEEvaluator](docs/src/content/docs/evaluators/rmse.md): root mean squared error
- [rMAEEvaluator](docs/src/content/docs/evaluators/rmae.md): relative mean absolute error

## Exporters

- [TerminalExporter](docs/src/content/docs/exporters/terminal.md): results as tables in the terminal
- [ExcelExporter](docs/src/content/docs/exporters/excel.md): formatted Excel workbook
- [CsvExporter](docs/src/content/docs/exporters/csv.md): wide-format CSV

## Reference

- [API Reference](docs/src/content/docs/reference/api.md): constructor and method signatures of the pipelines, `Workflow`, `EvaluationReport` and every shipped component
- [Extending the Library](docs/src/content/docs/reference/extending.md): custom sources, transformers, validators, models, evaluators and exporters

## Optional

- [examples/](examples/): runnable scripts for a data pipeline, a model pipeline, a full workflow, a YAML workflow, forecast-only mode and 15-minute forecasting
- [experiment.yaml](experiment.yaml): a `Workflow` file that points at the two pipeline files below
- [data_pipeline.yaml](data_pipeline.yaml): a saved `DataPipeline` with ENTSO-E, Open-Meteo and calendar sources
- [model_pipeline.yaml](model_pipeline.yaml): a saved `ModelPipeline` with models and evaluators
- [README.md](README.md): feature list and quick start
