Row: BACKEND-ROCM-BF16-MOE
The production token gate of #3096 differs at six generated positions across three length-33, concurrency-2 repeats. Disposition analysis (recorded with the BACKEND-ROCM-RESIDUAL-NORM evidence, verdict b) shows the six positions are three repeats of the same two-token chain of request 0: decode step 6 is an exact bf16 tie in the pinned primary itself, and the differences that feed it are upstream of the residual sites this row's residual repair covers.
Documented differences in scope of this issue:
- Attention-output arithmetic at rows >= 1: the first L33/C2/R0 attention output carries 2902 differing words while row 0 is exact (
rocm-residual-norm.md:307).
- The Q/K preamble FP32-vs-BF16 RoPE boundary (
rocm-residual-norm.md:309-316).
- vLLM's batch-2 kernel arithmetic: the oracle answers 63 at its own cc1 execution and 118 at cc2 for the identical prompt and step, separating an exact tie by one BF16 ulp; native is bit-identical across cc1 and cc2 and matches the primary's cc1 answer.
- Expert-output ulp differences on identical captured inputs, <= 6.1e-5 (
rocm-bf16-moe.md:517-521).
Evidence: operator logs and captures in /home/vikash/.cache/residual-norm-repair1/green-cc9d4f565/ (fusion-1/fusion-0 logs, production-fusion-1.json), the oracle diagnostics in the moe build evidence oracle-selection-6/ and oracle-diagnostic-2/, and the disposition findings recorded with the residual row. Native-vs-oracle logit noise is RMS ~0.0005 with max <= 0.002 per logit at every matched step, identical to the 210 passing positions.
Scope note: the residual row spec (rocm-residual-norm.md ## Owed) requires this issue to be assigned before implementation. A second issue owns the BF16 LM-head output boundary; hidden-state parity at ~1 BF16 ulp is additionally required there because even a perfect head narrowing turns the run's 63/118 margin into an exact tie rather than into the cc2 answer.
Row:
BACKEND-ROCM-BF16-MOEThe production token gate of #3096 differs at six generated positions across three length-33, concurrency-2 repeats. Disposition analysis (recorded with the
BACKEND-ROCM-RESIDUAL-NORMevidence, verdict b) shows the six positions are three repeats of the same two-token chain of request 0: decode step 6 is an exact bf16 tie in the pinned primary itself, and the differences that feed it are upstream of the residual sites this row's residual repair covers.Documented differences in scope of this issue:
rocm-residual-norm.md:307).rocm-residual-norm.md:309-316).rocm-bf16-moe.md:517-521).Evidence: operator logs and captures in
/home/vikash/.cache/residual-norm-repair1/green-cc9d4f565/(fusion-1/fusion-0 logs,production-fusion-1.json), the oracle diagnostics in the moe build evidenceoracle-selection-6/andoracle-diagnostic-2/, and the disposition findings recorded with the residual row. Native-vs-oracle logit noise is RMS ~0.0005 with max <= 0.002 per logit at every matched step, identical to the 210 passing positions.Scope note: the residual row spec (
rocm-residual-norm.md## Owed) requires this issue to be assigned before implementation. A second issue owns the BF16 LM-head output boundary; hidden-state parity at ~1 BF16 ulp is additionally required there because even a perfect head narrowing turns the run's 63/118 margin into an exact tie rather than into the cc2 answer.