Skip to content

[https://nvbugs/6544407][fix] unwaive glm5 stress test #17276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chuangz0 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[https://nvbugs/6544407][fix] unwaive glm5 stress test #17276

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion tests/integration/test_lists/waives.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,6 @@ full:B200/accuracy/test_llm_api_pytorch.py::TestNemotronV3Ultra::test_nvfp4_4gpu
full:B200/accuracy/test_llm_api_pytorch.py::TestNemotronV3Ultra::test_nvfp4_4gpus_block_reuse[TEP4] SKIP (https://nvbugs/6474894)
full:B200/accuracy/test_llm_api_pytorch.py::TestQwen3_30B_A3B::test_dummy_load_format SKIP (https://nvbugs/6525059)
full:B200/disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-deepseek_r1_v2_fp4_stress] SKIP (https://nvbugs/6472256)
full:B200/disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-glm5_nvfp4_tp4_ep4_dp_stress] SKIP (https://nvbugs/6544407)

@xinhe-nv xinhe-nv Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The case is still hang on B200, https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/blue/rest/organizations/jenkins/pipelines/DEBUG_LLM_FUNCTION_CLUSTER_TEST/runs/1968/nodes/141/steps/156/log/?start=0

full:B200/disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-qwen3_32b_fp8_stress] SKIP (https://nvbugs/6472256)

@brnguyen2 brnguyen2 Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth noting for whoever reviews the QA side: the two sibling stress waivers here (deepseek_r1_v2_fp4_stress and qwen3_32b_fp8_stress, both https://nvbugs/6472256) stay in place, so this unwaive is genuinely scoped to the GLM5 case — good. But since none of these run in pre-merge CI, the first signal that the GLM5 unwaive was premature will be a red QA stress campaign. If you're not confident the underlying issue is fixed (vs. just not reproducing), consider holding this until a couple of clean QA stress runs land rather than using the campaign as the experiment.

@crazydemo crazydemo Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MAJOR] Unwaive not verifiable by CI; intermittent stress test may regress

The deleted line unwaived test_disaggregated_stress_test[...glm5_nvfp4_tp4_ep4_dp_stress], but this test appears only in the QA stress list (tests/integration/test_lists/qa/llm_function_stress.txt), not in any test-db/l0_*.yml. Concretely: run /bot run on this PR and the test never executes, so CI cannot confirm the unwaive — the only evidence is 5 manual runs on a b200 node for a workload (8xB200, 35k requests, conc512, 0.90 accuracy gate, 10% cancellation) whose known failure mode is intermittent. Trigger for regression: the next scheduled QA stress run flakes exactly as before, now un-waived, and fails the suite with no CI trail pointing back here. Fix: before merging, (1) link at least one passing run and state the run count, and (2) resolve nvbugs/6544407 explicitly (close / close-not-repro) so the waiver and bug do not drift apart. If a specific commit fixed the root cause, name it in the description.

full:B200/llmapi/test_llm_api_pytorch_moe_lora.py::test_qwen_moe_routed_expert_multi_lora_varying_ranks[cudagraph] SKIP (https://nvbugs/6475623)
full:B200/llmapi/test_llm_api_pytorch_moe_lora.py::test_qwen_moe_routed_expert_multi_lora_varying_ranks[eager] SKIP (https://nvbugs/6475621)
Expand Down
Loading