Row: BACKEND-ROCM
test_backend_cross_device.cpp:1459, TEST_CASE("causal conv1d fwd/update match the CPU oracle"), fails intermittently on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3) with a real numerical mismatch, not a crash:
tests/vt/test_backend_cross_device.cpp:1534: ERROR: CHECK( Nmse(ref_out, dout.Download()) <= kNmseTol ) is NOT correct!
values: CHECK( 0.0858971 <= 0.0005 )
logged: DeviceName(dt) := 1
Reproduction: run the isolated case (--test-case="causal conv1d fwd/update match the CPU oracle") repeatedly under the GPU file mutex. Observed 3 failures in 8 consecutive runs (~37.5%), same NMSE value (0.0858971) both times it was captured. Not a timeout, not a launch failure — the kernel runs and returns a wrong number some fraction of the time.
Confirmed unrelated to any in-flight PR: the test case and the GDN conv1d kernel source are both untouched by row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's reconciliation work (verified present, unmodified, at that branch's pre-reconciliation head 22800690b). Not previously tracked under any issue search for conv1d/ROCm/gfx1200.
Given this project's stated position that no device-side ASan exists for RDNA/gfx12 and that green tests have twice this session alone hidden a real defect in adjacent ROCm kernel work (races, OOB reads), this deserves the same address-arithmetic-level scrutiny rather than being waved off as environment noise — but it is out of scope for the PR that found it, which does not touch this kernel.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Row:
BACKEND-ROCMtest_backend_cross_device.cpp:1459,TEST_CASE("causal conv1d fwd/update match the CPU oracle"), fails intermittently onisravale(RX 9060 XT, gfx1200, ROCm 7.2.3) with a real numerical mismatch, not a crash:Reproduction: run the isolated case (
--test-case="causal conv1d fwd/update match the CPU oracle") repeatedly under the GPU file mutex. Observed 3 failures in 8 consecutive runs (~37.5%), same NMSE value (0.0858971) both times it was captured. Not a timeout, not a launch failure — the kernel runs and returns a wrong number some fraction of the time.Confirmed unrelated to any in-flight PR: the test case and the GDN conv1d kernel source are both untouched by
row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's reconciliation work (verified present, unmodified, at that branch's pre-reconciliation head22800690b). Not previously tracked under any issue search for conv1d/ROCm/gfx1200.Given this project's stated position that no device-side ASan exists for RDNA/gfx12 and that green tests have twice this session alone hidden a real defect in adjacent ROCm kernel work (races, OOB reads), this deserves the same address-arithmetic-level scrutiny rather than being waved off as environment noise — but it is out of scope for the PR that found it, which does not touch this kernel.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]