# asmdiff field-test report

**Version tested:** working tree at 0.1.3 (post response-file/object-label/walk-up/verbose fixes), run via `uvx --from <repo>` so the packaged entry point, not the checkout, is what executed.
**Environment:** S3-Amysynth - a real ESP32-S3 / ESP-IDF v6 firmware (FreeRTOS, TinyUSB, vendored AMY synth engine), `build/compile_commands.json` with 1024 entries, xtensa-esp-elf-gcc 15.2.0 (picolibc via specs in a `@response` file), plus host gcc/clang and g++ for contrast.

## Scenario matrix

| # | Scenario | Result |
|---|----------|--------|
| A1 | Small harness, `old_*`/`new_*` auto-pairing, default gcc+clang fallback | works - both pairs found, fold-vs-libcall visible immediately |
| A2 | Same harness through a config target: glob `cc`, `compile_commands = true`, harness not in db | works - glob resolution announced, soft-miss note printed, xtensa FPU asm side-by-side |
| A3 | Two-target matrix (`--target s3 --target s3-size`) | works - O2 vs Os counts differ as expected |
| B1 | Whole-file summary of a real DSP file (`oscillators.c`, 58 functions) via db flags | works - and immediately useful: softfloat `__muldf3`/`__extendsfdf2` calls stand out in oscillator init paths |
| B2 | `--pair render_lut:render_lut_fb` inside a real file | works |
| C1 | `--across render_lut` between two config profiles | works - zero-overhead loop spans (`.L*_LEND`) detected on both sides |
| C2 | `--across` real file vs modified copy outside the tree | fails cleanly: soft-miss note + trimmed "amy.h: No such file" error, `--verbose` hint |
| C2b | `--across` real file vs copy **next to** the original | **compiles but silently misleads** - see Trap 1 |
| C3 | Two real files (both in db), per-file summary blocks | works - each file gets its own recorded flags |
| E1-E3 | Error paths: bogus target, string `flags`, explicit db + absent source | all produce specific, actionable messages (bogus target lists available targets) |
| E4 | C++ file via `g++` | compiles, functions extracted - but see Trap 3 |
| F1 | Giant function (`play_delta`, ~1000 insns) across O2/Os | works; see note on span semantics |
| F2 | Bare `--` extra flags (README's own example) | works |
| F3 | Source with no functions | `(no functions found)`, no crash |
| F4 | `--pair` with two files | correct argparse guard pointing at `--across` |

## Traps and gaps (ranked)

### 1. Same-directory copy: identical source, different assembly, one stderr note
The natural before/after move - copy `oscillators.c` to `osc_tweak.c` in the same
directory and `--across` them - *succeeds*, because quote-includes resolve next to
the source. But the copy has no db entry, so it compiles without the recorded
`-D`/`-I` set: in this project that means no `ESP_PLATFORM`, a different
`AMY_SAMPLE_RATE`, the portable code paths. The two columns show different
register allocation and frame sizes **for byte-identical code**. The only
warning is the one-line soft-miss note, whose implication ("the flag sets now
differ") is easy to miss. A user would read real codegen differences into a
header-config artifact.

**Recommendation:** when the two sides of an `--across`/two-file run received
*different* borrowed-flag sets (one hit, one soft miss), say so loudly - e.g.
tag the column header `[no db entry]` and print a warning that the comparison
crosses header configurations.

### 2. Before/after on a project source has no supported path
The tool's pitch ("what does the compiler emit before and after my rewrite")
cannot currently be executed for a source that needs db flags, except by
editing in place and diffing two separate runs by eye, or by maintaining two
built checkouts with two databases. C2/C2b are both dead ends by design.

**Recommendation (feature):** an opt-in fallback such as "borrow flags from
the same-basename entry" (with a warning), or an explicit
`--compile-commands-entry PATH` to name which db entry a given source should
borrow from. Either would make the copy workflow first-class and would also
neutralize Trap 1.

### 3. C++ defeats auto-pairing silently
`_Z9old_scalef` does not start with `old_`, so a C++ harness falls back to
whole-file summary with no hint about why. Workable via `extern "C"` or
explicit `--pair` with mangled names, but nothing tells the user that.
Acceptable for a C-focused tool; a doc note (or a "mangled names detected,
auto-pairing skipped" hint) would close it.

### 4. Loop spans are coarse on large control flow
`play_delta` (1001 insns) reports spans like `.L513:676` - two-thirds of the
function - because any backward branch defines a span, and big switch/dispatch
functions branch backwards for control flow that is not a loop. The README's
"the listing and your source know, not the tool" caveat covers this honestly,
but on 1000-instruction functions the span column stops being informative.

### Minor observations
- The glob-resolution notice (`cc pattern matched 2 toolchains ...`) prints on
  every run; correct behavior (never silent), slightly noisy in scripted use.
- Two-file summary blocks are rendered as independent tables, so their columns
  do not align with each other - cosmetic.
- `-ffast-math` after `--` reached the compiler as documented (no behavior
  change for the test case, which was already folded).

## What clearly works now (regression-verified this session)
- ESP-IDF v6 out of the box: `@response-file` expansion delivers
  `-specs=picolibc.specs`; `main.c` and every AMY source compiled with only
  `-O2 -mlongcalls` supplied by the user.
- Function extraction is clean: 72/72 real functions on `amy.c`, zero data
  labels, counts match `.type @function` ground truth.
- Walk-up discovery: project root, `main/`, and `components/amy/src/` all find
  `build/compile_commands.json`; the `.git` boundary prevents escaping the repo.
- Failure UX: 4-line trimmed error with a `--verbose` escape hatch instead of a
  13 KB flag dump.
- Config system end-to-end: CWD discovery, `default`, multi-target matrices,
  glob `cc`, env expansion, precise config error messages.

## Postscript: guards added after this report

Traps 1-3 were addressed in the working tree the same day, and re-verified
against the scenarios above:

- **Trap 1:** a two-file comparison with borrowed flags on only one side now
  tags the column `[no db entry]` and warns that differences may be header
  configuration, not codegen. The C2b rerun makes the case vividly: identical
  source showed 51 vs 14 instructions (partial outlining under the portable
  header config) — now labeled instead of silent.
- **Trap 2:** new `--flags-like PATH` lets a db-less source borrow a named
  entry's flags. Rerunning C2 and C2b with it produced identical 51/51
  columns for identical source, in-tree and out-of-tree. The absent-source
  error and the soft-miss note both suggest the flag when a same-basename
  entry exists.
- **Trap 3:** falling back to summary when only mangled `_Z*old_*` symbols
  exist now prints a note naming `extern "C"` and `--pair` as the outs.

Trap 4 (span coarseness on giant control flow) is documented behavior and
was left as-is.

## Evaluation
For its stated job - harness-based codegen comparison and whole-file/function
inspection of real embedded sources - the tool now works credibly in a real
ESP-IDF v6 project, with fast (<2 s) turnarounds and output that surfaced a
genuine finding within minutes (software-double promotion in AMY filter/
oscillator setup paths via `__extendsfdf2`/`__muldf3` in the calls column).
Error paths are uniformly specific and short. The remaining weaknesses cluster
around one theme: comparing *modified copies* of db-backed sources, where the
current behavior ranges from a clean failure (C2) to a quietly misleading
success (C2b). That workflow is close enough to the tool's core promise that
Trap 1's warning and Trap 2's fallback are the highest-value next changes;
everything else found is polish.
