From 5a31fb73774a0b27c7df33c870f223805ca0a93c Mon Sep 17 00:00:00 2001 From: Brian Nguyen Date: Wed, 12 Aug 2026 14:58:34 -0700 Subject: [PATCH 1/2] [https://nvbugs/6599150][fix] Initialize dt_bias in KDA verify-parity test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The test builds KimiKDARuntime with random weights and no checkpoint, but dt_bias is constructed with torch.empty and only filled by load_weights(), so it held uninitialized host memory. When that memory contained NaN/Inf bit patterns, both the decode and verify paths produced identical NaN outputs and assert_close (equal_nan=False by default) reported a 100% mismatch — the intermittent DGX_B200 CI failure. Finite garbage preserved parity because both paths share the parameter, which is why the failure was nondeterministic and platform-dependent. Initialize dt_bias explicitly after construction, as the sibling KDA parity tests already do. Verified: 200/200 fresh-process runs of the [2-1]/[2-3] parametrizations on an x86_64 B200 node (previously ~64% of processes failed), plus a full-file run of all four parametrizations. Signed-off-by: Brian Nguyen --- tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py b/tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py index 0b20cfc34bd2..f23ba5983313 100644 --- a/tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py +++ b/tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py @@ -59,6 +59,10 @@ def test_kda_verify_matches_sequential_decode(batch, t_steps): w = lin["short_conv_kernel_size"] runtime = KimiKDARuntime(cfg, layer_idx=0).to(device) + # dt_bias is torch.empty at construction and only filled by + # load_weights(); with random weights it holds heap garbage, and a + # NaN/Inf bit pattern poisons both paths identically (nvbug 6599150). + torch.nn.init.normal_(runtime.mixer.dt_bias, std=0.1) slots = batch + 2 # non-trivial slot mapping cache = _LayerCache(slots, 3 * dim, w, h, lin["head_dim"], lin["head_dim"], t_steps, device) slot_indices = torch.arange(2, 2 + batch, device=device, dtype=torch.long) From 944c4c068fa39e8ad188aeffa14e71e447c78649 Mon Sep 17 00:00:00 2001 From: Brian Nguyen Date: Wed, 12 Aug 2026 16:04:54 -0700 Subject: [PATCH 2/2] [https://nvbugs/6599150][fix] Unwaive KDA verify-parity test The flake is fixed by initializing dt_bias in the test; remove the DGX_B200 waive for test_kda_verify_matches_sequential_decode[2-1]. Verified after rebasing onto current main: 40/40 fresh-process runs of the [2-1]/[2-3] parametrizations on an x86_64 B200 node, plus all four parametrizations passing in a single-process run. Signed-off-by: Brian Nguyen --- tests/integration/test_lists/waives.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index abf8522a9f40..5704fa50adc4 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -322,7 +322,6 @@ unittest/_torch/attention/sparse/dsa/test_req_idx_per_token.py::test_on_update_k unittest/_torch/attention/test_attention_backends.py::test_attention_backend[deepseekv3_mla-gen-bf16-HND-p32-v1] SKIP (https://nvbugs/6507109) unittest/_torch/executor/test_overlap_scheduler.py::test_overlap_scheduler_consistency[no_reuse-cpp_scheduler-TorchSampler] SKIP (https://nvbugs/6561559) unittest/_torch/misc/test_autotuner.py::test_cutedsl_nvfp4_heuristic_matches_full_sweep SKIP (https://nvbugs/6490028) -unittest/_torch/modeling/test_kimi_kda_verify_parity.py::test_kda_verify_matches_sequential_decode[2-1] SKIP (https://nvbugs/6599150) unittest/_torch/modeling/test_modeling_qwen_moe.py::TestQwenMoe::test_qwen_moe_allclose_to_hf[backend:trtllm-use_cuda_graph:False] SKIP (https://nvbugs/6566765) unittest/_torch/modeling/test_modeling_qwen_moe.py::TestQwenMoe::test_qwen_moe_allclose_to_hf[backend:trtllm-use_cuda_graph:True] SKIP (https://nvbugs/6575012) unittest/_torch/modeling/test_modeling_qwen_moe.py::TestQwenMoe::test_qwen_moe_allclose_to_hf[backend:vanilla-use_cuda_graph:False] SKIP (https://nvbugs/6566765)