Row: BACKEND-ROCM
tests/vt/test_backend_cross_device.cpp:2304, TEST_CASE("MoeRouterTopK matches the CPU oracle (f32 and bf16 logits)"), failed once during work on a fresh implementer's row (row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1, unrelated to MoE routing), on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3):
tests/vt/test_backend_cross_device.cpp:2351: ERROR: CHECK( Nmse(ref_w, dw.Download()) <= kNmseTol ) is NOT correct!
values: CHECK( 0.214825 <= 0.0005 )
plus mismatched expert indices in the same case.
Reproduction rate is low and I want to record it honestly rather than overstate it. The implementer who found it saw 1 failure in 12 full-test_backend_cross_device-binary runs with their (unrelated) change applied, and 1 in 12 with that change stashed at the pristine base commit c6e1092dffc5f3484d9c8d24ad3b4e3f7b7531cc — ruling their own diff out as the cause. I (operator) then ran the isolated case 10/10 clean and the full binary 6/6 clean, so 0 reproductions in 16 of my own runs. Net: 2 failures observed in 28 total runs across two sessions (~7%), never in isolation from either session — it may require the full binary's accumulated device/allocator state to trigger, which is why isolated single-case runs may under-sample it.
Confirmed unrelated to any in-flight PR: the test case and vt::MoeRouterTopK's kernel source are both absent from row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's diff, and the test case is confirmed present, unmodified, at that branch's pre-reconciliation head 22800690b (predates all work on that row).
Distinct from #3119 (causal conv1d, a different test case, much higher ~37% isolated reproduction rate) and from #1954 (MoeSiluMul). Given this project's position that no device-side ASan exists for RDNA/gfx12, and that this session alone has found two independent classes of adjacent-but-different intermittent ROCm numerical flakes, this may be worth checking whether all three share a root cause (a common allocator/queue-state issue) rather than being investigated as three unrelated kernels — but that is a hypothesis, not a finding, and needs someone to actually look.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Row:
BACKEND-ROCMtests/vt/test_backend_cross_device.cpp:2304,TEST_CASE("MoeRouterTopK matches the CPU oracle (f32 and bf16 logits)"), failed once during work on a fresh implementer's row (row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1, unrelated to MoE routing), onisravale(RX 9060 XT, gfx1200, ROCm 7.2.3):plus mismatched expert indices in the same case.
Reproduction rate is low and I want to record it honestly rather than overstate it. The implementer who found it saw 1 failure in 12 full-
test_backend_cross_device-binary runs with their (unrelated) change applied, and 1 in 12 with that change stashed at the pristine base commitc6e1092dffc5f3484d9c8d24ad3b4e3f7b7531cc— ruling their own diff out as the cause. I (operator) then ran the isolated case 10/10 clean and the full binary 6/6 clean, so 0 reproductions in 16 of my own runs. Net: 2 failures observed in 28 total runs across two sessions (~7%), never in isolation from either session — it may require the full binary's accumulated device/allocator state to trigger, which is why isolated single-case runs may under-sample it.Confirmed unrelated to any in-flight PR: the test case and
vt::MoeRouterTopK's kernel source are both absent fromrow/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's diff, and the test case is confirmed present, unmodified, at that branch's pre-reconciliation head22800690b(predates all work on that row).Distinct from #3119 (causal conv1d, a different test case, much higher ~37% isolated reproduction rate) and from #1954 (MoeSiluMul). Given this project's position that no device-side ASan exists for RDNA/gfx12, and that this session alone has found two independent classes of adjacent-but-different intermittent ROCm numerical flakes, this may be worth checking whether all three share a root cause (a common allocator/queue-state issue) rather than being investigated as three unrelated kernels — but that is a hypothesis, not a finding, and needs someone to actually look.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]