TL Agent Deterministic Fallback

[tsrikris]

Deterministic (no-LLM) fallback for graph-collapsed traces

Some inference traces are graph-under-recorded: a GPU graph replay or a
compiled region collapses the whole workload behind a single launch, so the
profiler records device kernels but not the per-op decomposition (Python / aten
op, shapes, launcher path) the analysis pipeline needs. On these traces per-op
decomposition fails and the LLM analysis path cannot run.

This change adds a deterministic fallback that still emits a parser-compatible
analysis.md. It recovers the only three things a graph-collapsed trace carries
(kernel name, time, and %E2E) and marks every other field as unrecoverable
instead of guessing at it.

Architecture

                    Step 1: perf report ──> perf_report_csvs/
                                                   │
                                                   ▼
                    Step 2: check_graph_replay_coverage()  ── graph_replay_fraction
                                                   │
                        ┌──────────────────────────┴──────────────────────────┐
              frac ≤ MAX │ (healthy)                          frac > MAX │ (graph-collapsed)
                         ▼                                                ▼
              Steps 3-6: prepare category data          render_fallback_report()
              Steps 7-8: analysis sub-agents                        │
              Step 12: analysis.md (full)          ┌────────────────┴──────────────┐
                         │                          │ _load_surviving_rows          │
                         │                          │  (drop plumbing wrappers)     │
                         │                          │ rank device kernels by %E2E   │
                         │                          │ apply %E2E floor + count cap  │
                         │                          │ unrecoverable cells = "—"     │
                         │                          │ Kernel Name  = raw symbol     │
                         │                          │ P-item heading = normalized   │
                         │                          └────────────────┬──────────────┘
                         ▼                                           ▼
                 analysis.md (full)                       analysis.md (degraded)
                         └────────────────────┬──────────────────────┘
                                              ▼
                        same contract: #### P{rank}: / reasoning-candidate /
                        impact-begin kind=p_item / 9-column **Data:** table
                                              │
                                              ▼
                               downstream analysis.md consumer

The fallback branch is a drop-in: it emits the same analysis.md marker and
table contract as the full analysis path, so a downstream consumer that parses
analysis.md needs no structural change.

Design

• Trace-quality gate (Step 2). After perf-report generation,
  check_graph_replay_coverage() reads the already-written perf_report_csvs/,
  computes the fraction of device time hidden behind graph-replay / compiled-region
  launches, and trips when it exceeds GRAPH_REPLAY_FRACTION_MAX. Benign
  eager-launch and memcpy plumbing wrappers (present in healthy traces too) are
  excluded, so the signal isolates the pathology.
• Deterministic report writer. On a bad verdict, render_fallback_report()
  builds P-items from the surviving device-kernel rows, ranks them by %E2E, and
  writes the contract the pipeline already emits: #### P{rank}: headings, one
  reasoning-candidate marker and one impact-begin kind=p_item marker per P-item,
  and the 9-column **Data:** table. No LLM, no GPU, fully deterministic.
• Honest unrecoverable cells. Fields that a graph-collapsed trace never captured
  (Operation, Args, Kernel Path, Count, FLOPS/Byte, Efficiency, Bound) render a
  literal — instead of a fabricated or duplicated value. The raw device symbol is
  preserved verbatim in the Kernel Name cell so a downstream consumer can key on it,
  and the P-item heading shows a display-shortened form of that symbol.
• P-item count control. A %E2E floor (MIN_PITEM_PERCENT_E2E) plus a defensive
  count cap (MAX_PITEM_COUNT) collapse noisy traces to the significant few. The
  percentage denominator is built over all surviving rows before filtering, so
  dropping the tail never inflates the survivors. Drops are reported in the banner
  (never silent) when non-zero.
• Display-only name normalization. _normalize_kernel_name() shortens the P-item
  heading to a fixed length with a truncation marker. It never demangles and never
  strips qualifiers, so the symbol in Kernel Name and the impact-grouping key both
  stay raw. Two long GEMM variants that differ only past the truncation boundary
  collapse to the same heading but remain distinct P-items in the arithmetic.
• Degraded banner. A visible banner states the graph-replay fraction, what is
  recoverable (kernel, time, %E2E), and what was never captured (shapes, launcher
  path, quant operands, category, efficiency), so the report can never be mistaken for
  a full analysis.
• CSV field-size hardening. The gate reads every trace's CSVs, and traces with
  very large cells exceeded csv's default field limit and aborted the gate before it
  could pass. A module-level csv.field_size_limit(...) is now raised once with a
  platform-safe clamp (start high, halve on OverflowError).
• Pipeline step renumber. Inserting the gate as Step 2 shifted the rest of the
  pipeline by one, so the orchestrator prep, the validation utility, and the
  orchestrator skill / reference / template prose were renumbered in lockstep. This is
  prose and print-label only, with no logic change.

Validation

• Unit tests (CI): tests/test_analysis_agent_deterministic.py covers the gate
  (fraction math, plumbing-wrapper exclusion, field-size case), the writer contract
  (markers 1:1 with P-items, unrecoverable cells em-dashed, raw kernel name preserved,
  normalized heading), the normalizer matrix (identity / truncate / mangled-stays-raw
  / length boundary / never-empty), the floor/cap boundaries and drop reporting, and
  the consumer-side substitution of the kernel name when Operation is an em dash. No
  GPU, no LLM.
• End-to-end: exercised through the full orchestrator on several graph-collapsed
  inference traces spanning distinct workload shapes (dense and MoE decode, diffusion
  image generation, speculative decode). Each tripped the Step 2 gate and routed to
  the fallback, and every report preserved the marker contract
  (reasoning-candidate = impact-begin = data-tables = P-item count) with the floor
  and cap applied.
• Coverage vs the full analysis path: on traces where a non-fallback baseline
  exists, the fallback recovers the complete compute-heavy tail (every dominant GEMM,
  attention, quantization, sort, and norm kernel), and surfaces additional
  kernel-granular items the category-grouped path folds together. The only kernels it
  cannot recover are ones that live inside a fused region and are visible only when
  capture-enabled decomposition splits them apart. That is a genuine limit of a
  graph-collapsed trace, and the — cells reflect it honestly.
• Healthy no-op: on healthy traces the gate returns false and the writer never
  fires.

Downstream / contract impact

The change is additive and parser-compatible. The 9-column table header, the
#### P{rank}: / reasoning-candidate / impact-begin markers, and their attribute
names are all unchanged. The one behavioral note for a downstream consumer is the —
Operation cell: a consumer that today drops rows with an empty Operation should
substitute the Kernel Name symbol for those rows (a backward-compatible reader
relaxation) so that fallback reports parse to the intended candidates.

Test plan

• pytest tests/test_analysis_agent_deterministic.py
• black --check on all changed Python (clean)
• Copyright headers valid on all changed files
• End-to-end fallback on multiple graph-collapsed inference traces
• Coverage compared against the full analysis path
• Field-size fix verified on a large-cell trace
• Healthy no-op returns false at the gate

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[gabeweisz]

I feel like this function should not be necessary. But I guess this is a test so maybe not a big deal

[gabeweisz]

It looks like the code that generates the rows has to generate 3 inputs - so isn't missing the third columns an error?