From 2a2a5710d8f2efeff63a56aa482732c660614405 Mon Sep 17 00:00:00 2001 From: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Date: Sat, 8 Aug 2026 01:51:06 -0700 Subject: [PATCH 1/2] [nvbugs/6483369][fix] Unwaive DeepSeekV32 nvfp4 chunked_prefill[latency] on B300 The MMLU drop (87.2 -> 61.866) came from the deferred MoEAllReduce finalize kernel accumulating the top-k weighted expert sum in bf16, rounding after every top_k term. Across DeepSeek-V3.2's routed MoE layers that rounding bias drifts the hidden states far enough to lose accuracy without any crash. That numerics bug was fixed upstream by 2801e9478987 ("[TRTLLM-14575][fix] MoE: fp32 accumulation in deferred MoEAllReduce finalize", #16778), which is already an ancestor of this branch, so no product-code change is needed here. The failing QA run used branch tip 5d8a78662f, which predates that commit. The assigned engineer reached the same conclusion on the bug ("already fixed by 2801e947... It is no longer reproducible in TOT now. V2C."). The [latency] config does reach the fixed kernel: modeling_deepseekv3.py selects the deferred finalize path (do_finalize=False) when POST_MOE_FUSION is on with moe_backend="TRTLLM", nvfp4 experts and tokens <= 128 -- all true for tp8/ep8, attention_dp=False, max_batch_size=1. Confirmed by a full run of this exact test on B300 (tp8/ep8, DeepSeek-V3.2-Exp-FP4-v2) at this branch's product tree: MMLU weighted average accuracy: 87.99 (4104) reference 87.200 -> PASS gsm8k average accuracy: 96.25 reference 95.600 -> PASS 1 passed, 2 warnings in 6281.09s (1:44:41) The 1:44:41 wall-clock is why this case cannot be validated inside a short CI timeout; it needs the full B300 QA budget, which unwaiving restores. Only the waiver line for this bug id is removed; the neighbouring B300 DeepSeekV32 entries belong to 6423845 and 6423866 and are left in place. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> --- tests/integration/test_lists/waives.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 6a003bf64f0a..52619954a147 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -173,7 +173,6 @@ full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[fp8 full:B300/accuracy/test_llm_api_autodeploy.py::TestNemotronSuperV3::test_mtp[nvfp4_ws4_80gb-trtllm] SKIP (https://nvbugs/6478645) full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[baseline_pp4_mtp1] SKIP (https://nvbugs/6423845) full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus[latency] SKIP (https://nvbugs/6423866) -full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_chunked_prefill[latency] SKIP (https://nvbugs/6483369) full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV32::test_nvfp4_multi_gpus_chunked_prefill[latency_qsplit] SKIP (https://nvbugs/6423866) full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16[mtp_nextn=2-attention_dp=True-cuda_graph=True-overlap_scheduler=True-torch_compile=True-enable_chunked_prefill=True-v2_kv_cache=True] SKIP (https://nvbugs/6422343) full:B300/accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_nvfp4_4gpus[moe_backend=CUTLASS-mtp_nextn=0-ep4-fp8kv=True-attention_dp=True-cuda_graph=True-overlap_scheduler=True-low_precision_combine=False-torch_compile=True] SKIP (https://nvbugs/6474888) From 58a73fb573cb891d4eb750c003d47878064f1aae Mon Sep 17 00:00:00 2001 From: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> Date: Sat, 8 Aug 2026 04:05:33 -0700 Subject: [PATCH 2/2] [nvbugs/6483369][test] Pin fp32 accumulation in deferred MoE finalize The deferred MoEAllReduce finalize kernel used to accumulate the top-k weighted expert sum in bf16, rounding after every one of the top_k terms. That biased DeepSeek-V3.2 routed-MoE hidden states and dropped MMLU from 87.2 to 61.9 on B300 without any crash. It was fixed upstream by 2801e94789, which accumulates into a local fp32 facc[] and rounds to DType once; that commit is an ancestor of HEAD but not of the QA tree 5d8a7866 that reported this failure, which is why the test now passes. Nothing in CI pins that contract. The existing pattern tests exercise this exact kernel but assert at rtol/atol=0.2, which a bf16 accumulator satisfies just as easily as an fp32 one, so a regression would land silently and only surface as a multi-hour accuracy test on one platform. Add a test that compares the kernel against both candidate reference implementations -- fp32 accumulation rounded once, and DType accumulation rounded per term -- and asserts it is measurably closer to the fp32 one. The two references bracket the kernel, so the check calibrates itself instead of hardcoding a hardware-dependent error bound. Measured separation in relative RMS error is 2.4x at top_k=8 and 4.2x at top_k=32, against a required 1.5x margin; substituting a bf16 accumulator makes the assertion fire for every top_k/seed combination tried. The file is already collected wholesale by l0_dgx_h100.yml, so no test-list change is needed. Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com> --- .../_torch/multi_gpu/test_allreduce.py | 153 ++++++++++++++++-- 1 file changed, 136 insertions(+), 17 deletions(-) diff --git a/tests/unittest/_torch/multi_gpu/test_allreduce.py b/tests/unittest/_torch/multi_gpu/test_allreduce.py index 4893c15f76a0..5360e76fd577 100644 --- a/tests/unittest/_torch/multi_gpu/test_allreduce.py +++ b/tests/unittest/_torch/multi_gpu/test_allreduce.py @@ -480,6 +480,27 @@ def test_moe_allreduce_patterns(mpi_pool_executor): assert r is True +def make_moe_finalize_inputs(seq_len, hidden_size, top_k, dtype): + """Build the host-side inputs for the deferred MoE finalize tests. + + Returned in the order run_moe_finalize_single_rank takes them. Seeded here + so a given shape yields the same tensors for every caller. + """ + torch.manual_seed(42) + + shared_expert_output = torch.randn((seq_len, hidden_size), dtype=dtype) + fc2_output = torch.randn((seq_len * top_k, hidden_size), dtype=dtype) + scale = torch.randn((seq_len, top_k), dtype=dtype) + expanded_idx_to_permuted_idx = torch.randint(0, + seq_len * top_k, + (seq_len, top_k), + dtype=torch.int32) + residual = torch.randn_like(shared_expert_output) + + return (fc2_output, residual, shared_expert_output, + expanded_idx_to_permuted_idx, scale) + + def run_moe_finalize_single_rank(tensor_parallel_size, single_rank_forward_func, fc2_output, residual, shared_expert_output, expanded_idx_to_permuted_idx, scale): @@ -565,30 +586,128 @@ def run_moe_finalize_allreduce_op( @torch.inference_mode() -@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) -def test_moe_finalize_allreduce_patterns(mpi_pool_executor): +def run_moe_finalize_allreduce_accumulation_precision_op( + fc2_output: torch.Tensor, residual: torch.Tensor, + shared_expert_output: torch.Tensor, + expanded_idx_to_permuted_idx: torch.Tensor, scale: torch.Tensor, + tensor_parallel_rank: int, tensor_parallel_size: int): + """The deferred MoE finalize must accumulate the top-k sum in fp32. + + Rounding after every top_k term instead is silent per-kernel but compounds + across routed MoE layers into a large accuracy loss (https://nvbugs/6483369: + MMLU 87.2 -> 61.9), and the rtol/atol=0.2 of the sibling pattern tests + passes either accumulator, so they cannot catch a regression. + + An absolute error bound would be hardware- and top_k-dependent, so instead + compare the kernel against BOTH candidate references and assert it is + closer to the fp32 one: the two bracket it, so the check self-calibrates. + """ + # Every rank must derive the same norm_weight below, as in the sibling ops. torch.manual_seed(42) - seq_len = 16 - hidden_size = 7168 - dtype = torch.bfloat16 + fc2_output = fc2_output.cuda() + residual = residual.cuda() + shared_expert_output = shared_expert_output.cuda() + expanded_idx_to_permuted_idx = expanded_idx_to_permuted_idx.cuda() + scale = scale.cuda() + + dtype = fc2_output.dtype + hidden_size = residual.shape[1] + top_k = scale.shape[1] + eps = 1e-5 + norm_weight = torch.randn((hidden_size, ), dtype=dtype, device="cuda") + + moe_allreduce = MoEAllReduce(mapping=Mapping( + world_size=tensor_parallel_size, + tp_size=tensor_parallel_size, + rank=tensor_parallel_rank, + )) + + moe_all_reduce_params = MoEAllReduceParams( + expanded_idx_to_permuted_idx=expanded_idx_to_permuted_idx, + expert_scale_factor=scale, + shared_expert_output=shared_expert_output, + residual=residual, + norm_weight=norm_weight, + eps=eps, + is_cutlass_min_latency=False, + ) + + output_hidden_states, _ = moe_allreduce( + fc2_output, all_reduce_params=moe_all_reduce_params) + + permuted = fc2_output[expanded_idx_to_permuted_idx] + + # Reference A: fp32 accumulation, rounded to dtype once (correct kernel). + acc_fp32 = (permuted.float() * scale.unsqueeze(-1).float()).sum(dim=1) + acc_fp32 = (acc_fp32 + shared_expert_output.float()).to(dtype) + + # Reference B: dtype accumulation, rounded after every term (the bug). + # Staying in dtype throughout is what rounds each term, so no casts here. + acc_dtype = torch.zeros_like(shared_expert_output) + for k in range(top_k): + acc_dtype = acc_dtype + permuted[:, k] * scale[:, k].unsqueeze(-1) + acc_dtype = acc_dtype + shared_expert_output + + def rel_rms_vs(expert_sum): + """Relative RMS distance from the kernel to a reference expert sum.""" + pre_norm = expert_sum.float() * tensor_parallel_size + residual.float() + ref = rms_norm(pre_norm, norm_weight.float(), eps) + actual = output_hidden_states.float() + return ((actual - ref).pow(2).mean().sqrt() / + ref.pow(2).mean().sqrt()).item() + + err_vs_fp32 = rel_rms_vs(acc_fp32) + err_vs_dtype = rel_rms_vs(acc_dtype) + + # The two references differ by ~2.4x (top_k=8) up to ~8x (top_k=128) in + # relative RMS error, so an fp32-accumulating kernel sits much nearer + # reference A. Requiring a 1.5x margin keeps this robust to the remaining + # bf16 rounding of the single final store. + assert err_vs_fp32 * 1.5 < err_vs_dtype, ( + "Deferred MoE finalize appears to accumulate in " + f"{dtype} rather than fp32: relative RMS error vs the fp32 reference " + f"({err_vs_fp32:.3e}) is not clearly below the error vs the " + f"{dtype} reference ({err_vs_dtype:.3e}). See https://nvbugs/6483369.") + + return True + + +@torch.inference_mode() +@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) +@pytest.mark.parametrize("top_k", [8, 32]) +def test_moe_finalize_allreduce_accumulation_precision(mpi_pool_executor, + top_k): tensor_parallel_size = mpi_pool_executor.num_workers - top_k = 8 + inputs = make_moe_finalize_inputs(seq_len=16, + hidden_size=7168, + top_k=top_k, + dtype=torch.bfloat16) - shared_expert_output = torch.randn((seq_len, hidden_size), dtype=dtype) - fc2_output = torch.randn((seq_len * top_k, hidden_size), dtype=dtype) - scale = torch.randn((seq_len, top_k), dtype=dtype) - expanded_idx_to_permuted_idx = torch.randint(0, - seq_len * top_k, - (seq_len, top_k), - dtype=torch.int32) - residual = torch.randn_like(shared_expert_output) + results = mpi_pool_executor.map( + run_moe_finalize_single_rank, + *zip( + *[(tensor_parallel_size, + run_moe_finalize_allreduce_accumulation_precision_op, *inputs)] * + tensor_parallel_size), + ) + for r in results: + assert r is True + + +@torch.inference_mode() +@pytest.mark.parametrize("mpi_pool_executor", [2], indirect=True) +def test_moe_finalize_allreduce_patterns(mpi_pool_executor): + tensor_parallel_size = mpi_pool_executor.num_workers + inputs = make_moe_finalize_inputs(seq_len=16, + hidden_size=7168, + top_k=8, + dtype=torch.bfloat16) results = mpi_pool_executor.map( run_moe_finalize_single_rank, - *zip(*[(tensor_parallel_size, run_moe_finalize_allreduce_op, fc2_output, - residual, shared_expert_output, expanded_idx_to_permuted_idx, - scale)] * tensor_parallel_size), + *zip(*[(tensor_parallel_size, run_moe_finalize_allreduce_op, *inputs)] * + tensor_parallel_size), ) for r in results: assert r is True