Skip to content

[None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases #17183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruodil merged 5 commits into from
Aug 6, 2026
Merged

[None][test] Fix perf-core timeouts and 96G-part coverage for large-checkpoint cases #17183

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0f07dfe
[None][test] Raise perf-core TIMEOUT for large-checkpoint cases on sl…
ruodil Aug 3, 2026
272e405
[None][test] Drop 645G DeepSeek FP8 perf cases from 96G-per-GPU parts
ruodil Aug 3, 2026
17187c0
Merge branch 'main' into user/ruodil/perf-core-timeout-slow-platforms
ruodil Aug 3, 2026
874f3da
Merge branch 'main' into user/ruodil/perf-core-timeout-slow-platforms
ruodil Aug 6, 2026
0ff0b09
Merge branch 'main' into user/ruodil/perf-core-timeout-slow-platforms
yufeiwu-nv Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 26 additions & 5 deletions tests/integration/test_lists/qa/llm_perf_core.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ llm_perf_core:
# 8: B300 test cases
# 9: H100, H20, B200, B300, RTX6000-Server test cases
# 10: H20, B200, B300, RTX6000-Server test cases
# 10b: H20, B200, B300 test cases (>120G per GPU; excludes 96G RTX6000-Server)
# 11: RTX-6000 Server test cases
# ===============================================================================

Expand Down Expand Up @@ -304,7 +305,9 @@ llm_perf_core:
lte: 12.0
tests:
#qwen3.5_397b_a17b_fp8 (MoE FP8 380G, 8-GPU ep=8)
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp8-bench-pytorch-float8-input_output_len:128,128-ep:8-tp:8-gpus:8]
# 128,128 is a trivial workload but the 380G checkpoint load alone exceeds the
# 60min default on the slower platforms in this condition.
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp8-bench-pytorch-float8-input_output_len:128,128-ep:8-tp:8-gpus:8] TIMEOUT(120)
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp8-bench-pytorch-float8-input_output_len:500,2000-ep:8-tp:8-gpus:8]
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp8-bench-pytorch-float8-input_output_len:2000,500-ep:8-tp:8-gpus:8]
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp8-bench-pytorch-float8-input_output_len:8000,1000-ep:8-tp:8-gpus:8]
Expand All @@ -323,9 +326,6 @@ llm_perf_core:
gpu_memory:
gt: 90000
tests:
# deepseek_r1_0528
- perf/test_perf.py::test_perf[deepseek_r1_0528_fp8-bench-pytorch-float8-input_output_len:8000,1000-reqs:20000-ep:8-tp:8-gpus:8] TIMEOUT(120)
- perf/test_perf.py::test_perf[deepseek_r1_0528_fp8-bench-pytorch-float8-input_output_len:1000,2000-reqs:3000-ep:8-tp:8-gpus:8] TIMEOUT(100)
#qwen3.5_122b_a10b (MoE BF16 234G, 4-GPU): OOM on 80G GPUs, needs >90G
- perf/test_perf.py::test_perf[qwen3.5_122b_a10b-bench-pytorch-bfloat16-input_output_len:128,128-ep:4-tp:4-gpus:4]
- perf/test_perf.py::test_perf[qwen3.5_122b_a10b-bench-pytorch-bfloat16-input_output_len:500,2000-ep:4-tp:4-gpus:4]
Expand All @@ -338,6 +338,25 @@ llm_perf_core:
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp4-bench-pytorch-float4-maxbs:1-input_output_len:8000,1000-reqs:10-con:1-ep:8-tp:8-gpus:8] #min_latency
- perf/test_perf.py::test_perf[qwen3.5_397b_a17b_fp4-bench-pytorch-float4-maxbs:512-input_output_len:8000,1000-con:512-ep:8-tp:8-gpus:8] #max_throughput


# 10b: H20, B200, B300 test cases - same as 10 but excludes 96G-per-GPU parts.
# These checkpoints are ~645G, i.e. ~81G/GPU of weights at tp:8, which leaves
# only ~14G per GPU for KV cache, activations and comm buffers on a 96G part.
- condition:
ranges:
system_gpu_count:
gte: 8
compute_capability:
gte: 9.0
lte: 12.0
gpu_memory:
gt: 120000
tests:
# deepseek_r1_0528 (FP8 645G, 8-GPU ep=8)
- perf/test_perf.py::test_perf[deepseek_r1_0528_fp8-bench-pytorch-float8-input_output_len:8000,1000-reqs:20000-ep:8-tp:8-gpus:8] TIMEOUT(120)
- perf/test_perf.py::test_perf[deepseek_r1_0528_fp8-bench-pytorch-float8-input_output_len:1000,2000-reqs:3000-ep:8-tp:8-gpus:8] TIMEOUT(100)


# 11: RTX-6000 Server test cases
- condition:
ranges:
Expand All @@ -347,7 +366,9 @@ llm_perf_core:
gte: 12.0
lte: 12.0
tests:
- perf/test_perf.py::test_perf[qwen3_235b_a22b_fp4-bench-pytorch-float4-input_output_len:1000,2000-con:8-ep:8-tp:8-gpus:8]
# con:8 keeps the GPUs far from saturation, so wall time is dominated by the
# 2000-token decode of each request; does not fit the 60min default here.
- perf/test_perf.py::test_perf[qwen3_235b_a22b_fp4-bench-pytorch-float4-input_output_len:1000,2000-con:8-ep:8-tp:8-gpus:8] TIMEOUT(120)
- perf/test_perf.py::test_perf[qwen3_235b_a22b_fp4-bench-pytorch-float4-input_output_len:1000,2000-con:512-ep:8-tp:8-gpus:8]
# deepseek_r1_0528
- perf/test_perf.py::test_perf[deepseek_r1_0528_fp4-bench-pytorch-float4-maxbs:1-input_output_len:1000,2000-reqs:10-ep:4-tp:8-gpus:8] TIMEOUT(120)
Expand Down
Loading