# Batch-warning reference

This is the reference a reviewer uses when reading a `convert-tree` run's
`_batch_warnings.log`. Every line is one of three things:

- **RESOLVED-BUG** — a defect the converter itself has fixed; the line should no
  longer appear. Treat any recurrence as a regression.
- **OPTIMISATION-MISS** — a proc that shipped **correct but un-flattened**. The SQL
  is safe; the converter left a set-based collapse on the table because flattening
  it could not be proven safe offline.
- **ADVISORY-BY-DESIGN** — the converter reporting a semantic decision it made
  (row order not preserved, a union column NULL-filled, a component with no
  translation). Not a defect; not "resolved" by code, only by triage.

Counts below cite `docs/plans/plan-batch-warning-remediation.md` §1 — 801 lines
from a Windows `convert-tree` run against the `data-dp-datafactory` tree,
normalised (component names, bracketed columns and temp-table names collapsed)
and counted.

## How to read `_batch_warnings.log`

The file lives at `<output-root>/_batch_warnings.log` and is **always** written,
even for a clean run (zero warnings → a zero-byte file).

- **Format:** `rel/path.sql[:line]: <message>` — the path is the emitted `.sql`
  relative to the output root, and the optional `:line` is the 1-based line within
  that file when the emitter knows it (the fluff gate lints the written proc
  verbatim, so its line is the line deploy will run).
- **One sorted line per warning, written once.** `batch.py` accumulates every
  warning into a single `batch_warnings` list and flushes it with one truncating
  `write_text("\n".join(sorted(...)))` at end of run (`batch.py:578-582`). The
  file is therefore sorted and **byte-identical across two runs of the same
  input** — an invariant pinned by `test_batch.py`
  (`test_successful_output_and_log_unchanged_by_a_failed_sibling`).
- **loguru is a separate channel.** GAP6 gate events are also emitted through the
  observability `logger` for the operator (stderr), but that list stays the *sole*
  writer of this file (plan decision 3) — loguru is never pointed at it, so the
  format never drifts.

**Deferred (out of scope this pass).** Plan decision 4's optional severity tier
(`blocking` / `advisory` / `info`) and the `--quiet-advisories` filter are **not**
implemented here. Every line is currently plain text with no tier prefix, and no
flag hides advisory lines; a reviewer separates signal from noise using the triage
table below.

## Triage table

| Category (normalised message) | §1 count | Class | Source |
| --- | --- | --- | --- |
| `fluff gate: sqlfluff lint failed: Error loading config: Extra config path '…'` | 339 | **RESOLVED-BUG** | `fluff_gate.py`, `batch.py:554-557` |
| `flattening skipped (OptimizeError); returned unchanged` | 85 | **OPTIMISATION-MISS** | `flatten.py:136-137` |
| `flattening left an unresolved table alias; statement returned unchanged` | 18 | **OPTIMISATION-MISS** | `flatten.py:142-143` |
| `flattening skipped (ambiguous table qualification); statement returned unchanged` | 5 | **OPTIMISATION-MISS** | `flatten.py:128-130` |
| `lookup/oledb_source '…': a trailing ORDER BY was removed …` | 50 + 42 | ADVISORY-BY-DESIGN | `transforms/base.py:685` (and `:802`) |
| `component '…' (script/oledb_command/scd) has no behaviour-preserving T-SQL translation …` | 46 + 4 + 1 | ADVISORY-BY-DESIGN | `transforms/flow.py:92` |
| `union '…': branch '…' has no column [X] - filled with NULL` | 44 | ADVISORY-BY-DESIGN | `transforms/set_ops.py:55` |
| `oledb_source '…': source query has CTE(s) / bare-table self-reference … materialised via SELECT … INTO #…_src` | 42 + 18 | ADVISORY-BY-DESIGN | `transforms/base.py:817` |
| `sort '…': intermediate row order is not preserved through a CTE …` | 22 | ADVISORY-BY-DESIGN | `transforms/grouping.py:117` |
| `lookup '…': retrieved column(s) […] have no evidenced no-match default …` | 15 | ADVISORY-BY-DESIGN (GAP4) | `transforms/lookup.py:278` |
| `data flow '…' has no destination - emitted a SELECT preview …` | 14 | ADVISORY-BY-DESIGN | `generator.py:412` |
| `merge join '…': emitted as INNER JOIN / LEFT OUTER JOIN` | 7 + 4 | ADVISORY-BY-DESIGN | `transforms/set_ops.py:85` |
| `merge '…': emitted as UNION ALL` | 2 | ADVISORY-BY-DESIGN | `transforms/set_ops.py:45` |
| misc single-digit advisories (precedence edge dropped, row-count drop, not-fully-connected skips, EXEC capture, aggregate GROUP BY, error-output ignored, synthesised orchestrator, unresolvable source column) | ~30 | ADVISORY-BY-DESIGN | various control-flow / transform builders (e.g. precedence edge: `control_graph.py:116,119`) |

**Headline.** The 339 `Error loading config` lines were a single broken-gate bug
(42% of the log), not 339 findings. The 108 flattening bail-outs (85 + 18 + 5) are
un-flattened but correct — the flattening section below records a verdict for each
cluster. The remaining ~354 lines are correct-by-design advisories.

### The 339 fluff-gate failures — RESOLVED-BUG

Previously the gate computed its sqlfluff config path relative to its install
location (`Path(__file__).parent.parent / ".sqlfluff"`). On an installed wheel
that path did not exist, so every proc failed with
`Error loading config: Extra config path '…'`, and the failure was mis-modelled as
a per-proc **blocking** lint violation — 339 phantom blocks and a non-zero
`convert-tree` exit that had nothing to do with the SQL.

Two fixes close it:

- **W1 — package the config as a resource.** `fluff_gate.py` now loads the shared
  house config via `importlib.resources` from the in-tree `data_dp_ssis2sp.resources`
  package (`sqlfluff.cfg`, materialised with `as_file`), so it resolves identically
  from a source checkout and an installed wheel.
- **W2 — a gate outage is a single non-blocking advisory.** A sqlfluff
  tool/config/runtime failure now raises `GateUnavailableError` (a gate-DOWN
  signal) rather than returning a violation-shaped tuple. `batch.py` probes the
  gate **once** up front (`fluff_gate.probe()`, `batch.py:554`); on an outage it
  logs one operator warning and appends **one** `fluff gate unavailable: …` line to
  `batch_warnings` (`batch.py:555-557`), skips the per-proc lint loop, and never
  increments `lint_failures` — so a broken toolchain fails safe instead of failing
  the whole conversion. Genuine `PRS` / `RF01` / `ST03` hits still block exactly as
  before.

### Why each advisory is correct by design

Each line below is the converter telling the DBA which behaviour-preserving
translation decision it made. None is a defect.

- **ORDER BY removed** (`transforms/base.py:685`, `:802`) — a trailing `ORDER BY`
  is invalid inside a derived table and set-based SQL does not preserve row order,
  so it is dropped and the removal announced.
- **Source query materialised via `SELECT … INTO #…_src`** (`transforms/base.py:817`)
  — a source query that carries CTE(s), or reads a bare table matching its own
  source-CTE name (which would become a self-referential CTE), cannot be lifted
  into a derived table, so it is run verbatim into a temp table and read from
  there.
- **Union branch NULL-fill** (`transforms/set_ops.py:55`) — an SSIS Union All
  branch missing an output column is filled with `NULL` to keep every branch's
  projection union-compatible, exactly as SSIS does.
- **Merge join emitted as INNER / LEFT OUTER JOIN** (`transforms/set_ops.py:85`) —
  the join type is chosen from the component's declared join kind
  (`0: FULL OUTER, 1: LEFT OUTER, 2: INNER`) and stated so a reviewer can verify it
  against the SSIS component.
- **Merge emitted as UNION ALL** (`transforms/set_ops.py:45`) — an SSIS Merge is
  emitted as `UNION ALL`; the interleaved sort order of a Merge is not preserved by
  set-based SQL, and the note says so.
- **Sort order not preserved through a CTE** (`transforms/grouping.py:117`) — an
  intermediate Sort's `ORDER BY` only survives if the Sort feeds a destination
  directly; through a CTE the order is not guaranteed, so the note flags where the
  `ORDER BY` was and was not applied.
- **Lookup with no evidenced no-match default** (`transforms/lookup.py:278`, GAP4)
  — when a lookup's retrieved columns have no evidence of a no-match default,
  unmatched rows keep `NULL`; the note asks the reviewer to confirm the original
  did not apply a default.
- **Data flow with no destination → SELECT preview** (`generator.py:412`) — a data
  flow with no destination component has nothing to insert into, so the converter
  emits a `SELECT` preview of the final component rather than fabricating a target.
- **Script / OLE-DB-command / SCD no translation** (`transforms/flow.py:92`) —
  these components have no behaviour-preserving T-SQL equivalent, so they are
  emitted as a pass-through with an explicit "manual rework required" note rather
  than a silently wrong translation.
- **Precedence edge dropped** (`control_graph.py:116,119`) — a control-flow
  precedence edge whose endpoint is not among the `execute_package_tasks` is
  skipped and named, because it cannot be wired into the orchestrator ordering.
- **Misc single-digit advisories** (row-count drop, not-fully-connected skips,
  EXEC capture, aggregate GROUP BY, error-output ignored, synthesised orchestrator,
  unresolvable source column) — each is the converter naming a semantic
  simplification it made; they are informational, emitted by their respective
  control-flow / transform builders.

## Flattening bail-outs

`flatten.py` collapses single-consumer CTE chains into one set-based
`INSERT … SELECT` (see its module docstring). When a collapse cannot be proven
safe offline it returns the statement **unchanged** with an advisory note: the
proc ships correct but un-flattened. The three clusters in the log
(85 `OptimizeError`, 18 dangling alias, 5 ambiguous qualification) were each
reproduced against a representative shape. **All three bail as designed — every
verdict is *expected-documented*, and no flattener code was changed.**

### Double-attempt semantics

`flatten_statement` (`flatten.py:105-111`) makes up to two attempts. It first runs
the schema-backed arm (`_flatten_once(sql, schema)`); if that bails **and** a
schema was supplied, it re-runs the heuristic, air-gapped arm
(`_flatten_once(sql, None)`) and returns *its* result. A note therefore only
reaches `_batch_warnings.log` when **both** arms bail, and then both reasons are
surfaced — the heuristic's first, deduplicated — so a log line never blames the
snapshot for a statement that does not flatten air-gapped either. A supplied
schema can only ever improve on the heuristic, so enabling a snapshot never costs
a collapse. One consequence: the *ambiguous table qualification* note is emitted
only by the schema arm, so it can appear in the log **only alongside** the
heuristic arm's own bail reason.

### Cluster verdicts

**C1 — ambiguous table qualification** (`_NESTING_NOTE`, `flatten.py:128-130`,
schema arm only). **Verdict: expected-documented.**
Repro: `WITH A AS (SELECT o.Id FROM dbo.Src AS o), B AS (SELECT s.Id FROM sales.Src AS s), C AS (SELECT A.Id FROM A UNION ALL SELECT B.Id FROM B) INSERT INTO dbo.T (Id) SELECT Id FROM C`
against the snapshot `{cat.dbo.src, cat.sales.src}` gives
`_flatten_once(sql, schema).notes == ['flattening skipped (ambiguous table qualification); statement returned unchanged']`.
The physical name `src` recurs under `(cat, dbo)` and `(cat, sales)`, so
`_normalize_nesting` returns `None`: the name-keyed `_restore_nesting` cannot
reverse one name to two qualifications (`merge_subqueries` rebuilds table nodes,
destroying object identity), and guessing would risk restoring `sales.Src` as
`dbo.Src` — wrong SQL. No collapse is lost, because
`flatten_statement(sql, schema=…).notes == []` here: the heuristic fallback
flattens the same statement, so the note only ever surfaces when the heuristic
also bails. Zero benefit, real risk → the bail is correct; a fix would need a
merge-stable identity key, a redesign of the normalise/restore pair rather than a
surgical change. Covered by `test_schema_bailout_falls_back_to_heuristic_flatten`
and `test_double_bailout_surfaces_both_reasons`.

**C2 — `OptimizeError` (broad except around `qualify()` + `merge_subqueries()`)**
(`flatten.py:136-137`). **Verdict: expected-documented.**
Repro: `WITH A AS (SELECT 1 AS x), B AS (SELECT x FROM A) SELECT x FROM A, B`
gives `notes == ['flattening skipped (OptimizeError); returned unchanged']` — the
column `x` is ambiguous across the two sources of `FROM A, B`, so `qualify`
raises. A mixed-nesting sub-case (a 3-part and a degraded 2-part sole-source
table, no caller schema) raises `SchemaError` through the same arm →
`['flattening skipped (SchemaError); returned unchanged']`. These statements are
genuinely unresolvable to `qualify`; any flatten would have to guess
column-to-source binding and risk emitting SQL bound to the wrong source. The
`except` is intentionally broad because sqlglot's optimizer is best-effort and
raises several error types, and it upholds the module docstring's contract that no
exception escapes. There is no safe flatten of an unresolvable statement. Covered
by `test_optimizer_error_returns_unchanged_with_note` and
`test_mixed_nesting_bails_with_schema_error_note_without_schema`.

**C3 — dangling table alias** (`_DANGLING_NOTE`, `flatten.py:142-143`).
**Verdict: expected-documented.**
Repro: `WITH [Src] AS (SELECT [Id] FROM (SELECT * FROM dbo.T) AS _src) INSERT INTO [dbo].[Target] ([Id]) SELECT [Id] FROM [Src]`
gives `_flatten_once(sql, None).notes == ['flattening left an unresolved table alias; statement returned unchanged']`.
The `SELECT *` inner source exposes no explicit column names, so `_synth_schema`
maps nothing for `dbo.T` and `merge_subqueries` inlines leaving `[Id]` qualified
by an alias no surviving source defines. This is exactly the safety net the module
docstring describes: it stops the emitter shipping SQL that SQL Server would reject
("the multi-part identifier could not be bound"). Resolving it requires expanding
the `*` — impossible offline without a schema — and the dangling reference proves
the merge already produced invalid structure, so bailing is the safe outcome.
Covered by `test_star_source_incomplete_schema_bails_out_unchanged`.

## Out of scope / deferred

- The severity tier and `--quiet-advisories` filter (plan decision 4) are **not**
  in this pass — every warning line is emitted plain, with no tier and no filter.
- The ~354 correct-by-design advisories are **not** silenced. They are the
  converter reporting its semantic translation decisions and are resolved only by
  the triage above, never by suppression.
- No flattener code was changed: all three bail-out clusters are conservative,
  correct bails; each is documented as expected rather than fixed.
