# Golden-Capture Runbook

Run this on a **Windows host** after the sprint merges to populate `golden/*.parquet` and `manifest.json` for each corpus package.  The resulting files are committed to the repository so the differential validation layer can run in CI.

---

## Prerequisites

### 1. Windows with SSIS / dtexec

Install SQL Server Developer Edition (free) or the standalone SSIS runtime.  After installation, `dtexec.exe` must be on `PATH`:

```powershell
dtexec /?   # must print usage without error
```

If `dtexec` is not on PATH, pass its location via `--dtexec-path` (see Step 6).

### 2. Microsoft ODBC Driver 18 for SQL Server

Download from https://learn.microsoft.com/sql/connect/odbc/download-odbc-driver-for-sql-server and install.  Verify:

```powershell
Get-OdbcDriver -Name "*SQL Server*"   # must list "ODBC Driver 18 for SQL Server"
```

### 3. Python 3.11+ (3.14 recommended; pinned via .python-version)

From the repo root:

```powershell
uv sync
```

Verify:

```powershell
uv run python -c "import validation.capture.capture; print('OK')"
```

### 4. Remote SQL Server access

The same SQL Server instance used by CI.  Network connectivity and credentials are required.

---

## Environment setup

The framework reads two SQL Server **DSNs** straight from the process
environment (it does **not** load a `.env` file). Each is a full ODBC
connection string passed to `pyodbc.connect`:

```
# the target — where converted T-SQL is provisioned, seeded, and run
export MSSQL_TARGET_DSN='DRIVER={ODBC Driver 18 for SQL Server};SERVER=target-host,1433;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes;'
# the source — the original server this capture step reads for golden data
export MSSQL_SOURCE_DSN='DRIVER={ODBC Driver 18 for SQL Server};SERVER=source-host,1433;Trusted_Connection=yes;Encrypt=yes;TrustServerCertificate=yes;'
```

These are the exact variable names `data_dp_ssis2sp/dual_connection.py` reads
(`MSSQL_TARGET_DSN` / `MSSQL_SOURCE_DSN`). `capture.py` connects to the
**source**; the test fixtures and differential run use the **target**. Because
the DSN is a complete connection string, authentication and encryption are your
choice (a named `DSN=MyDsn;` reference works too). The test suite and the
`data_dp_ssis2sp` CLI rely on environment variables only. For standalone
`capture.py` runs, operators must export those DSNs into the environment:

```powershell
$env:MSSQL_SOURCE_DSN = "DRIVER={ODBC Driver 18 for SQL Server};SERVER=source.local;DATABASE=master;Trusted_Connection=yes;TrustServerCertificate=yes;"
$env:MSSQL_TARGET_DSN = "DRIVER={ODBC Driver 18 for SQL Server};SERVER=target.local;DATABASE=master;Trusted_Connection=yes;TrustServerCertificate=yes;"
```

---

## Capturing golden fixtures for a single package

Run from the repo root on the Windows capture host:

### Step 1 — Verify the package transpiles

```powershell
uv run python -c "
from data_dp_ssis2sp import convert_file, ConvertOptions
from pathlib import Path
r = convert_file(Path('validation/corpus/passthrough_basic/package.dtsx'), ConvertOptions())
print('SQL length:', len(r.sql)); print('Warnings:', r.warnings)
"
```

### Step 2 — Run the capture

```powershell
.\validation\capture\capture.ps1 -PackageDir validation\corpus\passthrough_basic
```

Or invoke Python directly:

```powershell
uv run python -m validation.capture.capture --package-dir validation\corpus\passthrough_basic
```

The harness will:

1. Connect to the remote SQL Server and create/reset the package database.
2. Apply `schema.sql` DDL.
3. Seed source tables from `seed/*.csv`.
4. Run `dtexec /FILE package.dtsx` (exit code != 0 aborts with an error).
5. Read back each `dst_*` table.
6. Write `validation/corpus/passthrough_basic/golden/<table>.parquet` for each destination.
7. Write `validation/corpus/passthrough_basic/golden/manifest.json`.

### Step 3 — Verify outputs

```powershell
# Parquet files and manifest must exist
ls validation\corpus\passthrough_basic\golden\

# Spot-check the manifest
Get-Content validation\corpus\passthrough_basic\golden\manifest.json

# Spot-check a Parquet file (requires pandas/pyarrow)
uv run python -c "
import pandas as pd
df = pd.read_parquet('validation/corpus/passthrough_basic/golden/dst_items.parquet')
print(df)
"
```

### Step 4 — Commit the golden fixtures

```powershell
git add validation/corpus/passthrough_basic/golden/
git commit -m "feat: golden fixtures for passthrough_basic"
```

---

## Capturing all packages at once

```powershell
.\validation\capture\capture.ps1 -AllPackages
```

Failed packages are reported at the end; exit code is non-zero if any fail.

After all captures complete:

```powershell
git add validation/corpus/
git commit -m "feat: golden fixtures for all corpus packages"
```

---

## Specifying a non-PATH dtexec

```powershell
.\validation\capture\capture.ps1 `
    -PackageDir validation\corpus\passthrough_basic `
    -DtexecPath "C:\Program Files\Microsoft SQL Server\160\DTS\Binn\dtexec.exe"
```

---

## Output locations

| File | Description |
|------|-------------|
| `validation/corpus/<pkg>/golden/<dst>.parquet` | Golden fixture for one destination table |
| `validation/corpus/<pkg>/golden/manifest.json` | Seed checksum, row counts, column types |

The `.gitkeep` placeholder in each `golden/` directory ensures the directory exists before capture.  The actual `.parquet` files and `manifest.json` are committed after capture.

---

## Integrity gate

The validation CI re-computes the seed checksum from the live `seed/*.csv` files and asserts it matches `manifest.json["seed_checksum"]`.  If seed CSVs change after capture, re-run the capture and re-commit the golden files.

Error message on mismatch: `"golden fixture is stale — seed_checksum does not match; re-run validation/capture/RUNBOOK.md"`.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `dtexec` not found | Not on PATH | Install SSIS or pass `--dtexec-path` |
| `pyodbc.Error: [28000]` login failure | Wrong auth in the DSN | Check the credentials/`Trusted_Connection` in `MSSQL_SOURCE_DSN` / `MSSQL_TARGET_DSN` |
| `pyodbc.Error: [08001]` connection failure | Server unreachable | Check network / the `SERVER=` clause in your DSN |
| `SQL Server not configured or unreachable` | DSN env var unset | Export `MSSQL_TARGET_DSN` (and `MSSQL_SOURCE_DSN` for capture) |
| Exit code non-zero from dtexec | Package execution error | Check `dtexec` stdout; verify ODBC connection strings in the package |
| `manifest.json` seed_checksum mismatch in CI | Seed CSVs changed after capture | Re-run capture and re-commit golden files |
| `ODBC Driver 18 for SQL Server` not found | Driver not installed | Install from Microsoft download link above |
