perf(agentx): add Qwen3.5 GB300 SGLang Pareto curve / 性能：新增 Qwen3.5 GB300 SGLang Pareto 曲线 - #2812
perf(agentx): add Qwen3.5 GB300 SGLang Pareto curve / 性能：新增 Qwen3.5 GB300 SGLang Pareto 曲线#2812Ankur-singh wants to merge 2 commits into
Conversation
Add two PP4 disaggregated and three aggregate AgentX deployments with matching srt-slurm recipes. 中文：新增两个 PP4 分离式和三个聚合式 AgentX 部署，并添加匹配的 srt-slurm 配置。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Backfill the public pull request URL in the appended performance changelog entry. 中文：在追加的性能变更日志条目中回填公开拉取请求链接。
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 60873a8. Configure here.
| dp-attn: false | ||
| additional-settings: | ||
| - "SYNTHETIC_ACCEPTANCE=true" | ||
| - "SYNTHETIC_ACCEPTANCE_LENGTH=4.8" |
There was a problem hiding this comment.
C565 acceptance length does not match MTP
Medium Severity
SYNTHETIC_ACCEPTANCE_LENGTH is 4.8 for the C565 point, but that recipe uses speculative-num-steps: 4 and speculative-num-draft-tokens: 5. The committed Qwen3.5 golden table maps those MTP levels to 3.97 and 4.41; 4.8 is the entry for 6 speculative tokens. Throughput jobs will simulate a higher acceptance rate than the matching golden measurement, so this Pareto point will not be comparable to the other Qwen3.5 points.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 60873a8. Configure here.
| dp-attn: false | ||
| additional-settings: | ||
| - "SYNTHETIC_ACCEPTANCE=true" | ||
| - "SYNTHETIC_ACCEPTANCE_LENGTH=4.8" |
There was a problem hiding this comment.
Acceptance lengths use draft-token index
Medium Severity
The new C704 and aggregate points set SYNTHETIC_ACCEPTANCE_LENGTH to 4.8 and 5.04, which are the golden entries for 6 and 7 speculative tokens. Those recipes use speculative-num-steps of 5 and 6. Existing Qwen3.5 recipes and the SGLang injector resolve AL from step count (3 → 3.39), so these values sit one MTP level above the matching golden measurement and will slightly inflate throughput.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 60873a8. Configure here.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| INFMAX_CONTAINER_WORKSPACE: /infmax-workspace | ||
| RESULT_DIR: /logs/agentic | ||
| PORT: '8000' | ||
| IS_MULTINODE: 'false' |
There was a problem hiding this comment.
🔴 This new TP8 aggregate recipe spans 2 nodes (agg_nodes: 2, gpus_per_node: 4) but sets IS_MULTINODE: 'false' — the sibling single-node TP2 recipes correctly use 'false', but this one appears copy-pasted without updating the flag for its 2-node topology. Existing analogous multi-node TP8 recipes (e.g. glm5.2 gb200 agg-gb200-tp8-c4-mtp.yaml) set IS_MULTINODE: 'true' for the same agg_nodes: 2 shape. Fix: set IS_MULTINODE: 'true' whenever agg_nodes/decode_nodes/prefill_nodes > 1, matching every multi-node recipe's convention.
Extended reasoning...
benchmarks/benchmark_lib.sh:3483-3491 branches on IS_MULTINODE to decide power-monitoring mode in run_agentic_replay_and_write_outputs: when IS_MULTINODE=true it uses the coordinated multi-node measurement-window path, else it enables the single-node in-process GPU monitor (agentx_power_enabled=1), which only observes the local node's GPUs. With IS_MULTINODE='false' but the deployment actually running TP8 across 2 physical nodes, this benchmark point silently records incomplete/incorrect power data for the Pareto point (missing the second node's GPUs) instead of using the correct multi-node aggregation, unlike the base branch's existing multi-node recipes which all set the flag correctly.
Verification: normal. The new recipe agg-gb300-tp8-c7-b1-mtp-hicache-nightly-20260831.yaml describes a genuinely 2-node deployment: resources set gpus_per_node: 4, agg_nodes: 2, gpus_per_agg: 8 (lines 13-16) and sglang_config sets tensor-parallel-size: 8 (line 64), so one aggregate worker's TP8 spans 8 GPUs across 2 physical nodes. Yet line 126 sets IS_MULTINODE: 'false'. This is inconsistent with the…
| conc-list: [565] | ||
| prefill: | ||
| num-worker: 3 | ||
| tp: 1 | ||
| pp: 4 | ||
| ep: 1 | ||
| dp-attn: false | ||
| additional-settings: | ||
| - "SYNTHETIC_ACCEPTANCE=true" | ||
| - "SYNTHETIC_ACCEPTANCE_LENGTH=4.8" |
There was a problem hiding this comment.
🔴 The c565 pp-pareto search-space entry sets SYNTHETIC_ACCEPTANCE_LENGTH=4.8, identical to the c704 entry above it, but its recipe (disagg-gb300-3p4d-pp4-dep4-c565) uses speculative-num-steps=4/num-draft-tokens=5 vs c704's steps=6/draft-tokens=7 — the injected synthetic acceptance length no longer matches the golden AL for c565's actual MTP depth (golden_al_distribution/qwen3.5_mtp.yaml maps depth 4 to 3.97, not 4.8), so throughput numbers for that Pareto point are measured against the wrong acceptance rate. Fix: set SYNTHETIC_ACCEPTANCE_LENGTH for c565 to the golden AL value matching its own speculative-num-steps/num-draft-tokens (3.97 for depth 4), not a copy of the adjacent c704 block's value.
Extended reasoning...
configs/nvidia-master.yaml lines 7208-7229 (c704) and 7226-7245 (c565) are near-identical search-space blocks under qwen3.5-fp4-gb300-dynamo-sglang-agentic-pp-pareto; every field that should differ (conc-list, CONFIG_FILE) was updated for c565 except SYNTHETIC_ACCEPTANCE_LENGTH, which stayed at 4.8. The c565 recipe (disagg-gb300-3p4d-pp4-dep4-c565-mtp-hicache-nightly-20260831.yaml, lines 145-147) configures speculative-num-steps: 4, speculative-num-draft-tokens: 5 — a shallower MTP depth than c704's steps=6/draft-tokens=7 — so the correct golden AL (per golden_al_distribution/qwen3.5_mtp.yaml, thinking_on key 4 = 3.97) differs from c704's key 6 = 4.80. Because the throughput job injects SYNTHETIC_ACCEPTANCE_LENGTH=4.8 for c565 anyway, the synthetic acceptance-rate simulation used to measure this Pareto point does not correspond to the depth actually configured, producing a throughput number that misrepresents the c565 point relative to its real MTP settings — a silent correctness defect in a benchmark result that will be published as a Pareto curve point.
Verification: normal. The c565 pp-pareto entry at configs/nvidia-master.yaml:7239 sets SYNTHETIC_ACCEPTANCE_LENGTH=4.8, identical to the c704 entry at line 7219, but the two recipes have different MTP depths, so the injected synthetic acceptance length is wrong for c565. Verified the intended mapping is SYNTHETIC_ACCEPTANCE_LENGTH = golden_al_distribution/qwen3.5_mtp.yaml keyed by speculative-num-draft-tokens…
| - "Use PP4 prefill with DEP4 decode at concurrency 704 and 565, plus two distinct TP2 aggregate concurrency-44 recipes and one TP8 aggregate concurrency-7 recipe." | ||
| - "Keep full MTP and HiCache in every recipe; throughput uses the committed synthetic acceptance lengths while EVAL_ONLY keeps real MTP behavior." | ||
| - "Run the upstream SGLang image as shipped and pin the PP recipes plus router identity to ai-dynamo/dynamo PR 14064 head ad6f7d54ce7b99c53576b10482755c93c921fbac." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX |
There was a problem hiding this comment.
🟡 (optional) The new changelog entry's pr-link is left as the literal placeholder "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX" instead of the actual PR URL, unlike every other entry in this file. Fix: replace it with the real PR number before merge (e.g. via a pre-merge check that rejects any pr-link matching /pull/XXX or non-numeric PR id) so the changelog keeps its audit trail linking each performance change back to its originating PR.
Extended reasoning...
perf-changelog.yaml is append-only and used to trace every performance-affecting change back to its PR for review/audit; this entry documents five new Pareto points but its pr-link field cannot be dereferenced (pull/XXX does not exist), so anyone auditing this changelog entry after merge has no way to find the actual PR, commits, or discussion behind these recipe/config additions, unlike all preceding entries which have real numeric pull links (e.g. pull/2774, pull/2808 immediately above it).
Verification: nit. The new changelog entry ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX (perf-changelog.yaml:6845), a literal placeholder rather than the real PR number, so the entry cannot be dereferenced to its originating PR — confirmed in the diff (only line added at the file tail). This is real but purely an audit-trail/documentation gap: the entry is otherwise valid YAML…
1 participant
Summary / 摘要
Add five Qwen3.5 NVFP4 AgentX Pareto points on GB300 using the upstream
lmsysorg/sglang:nightly-dev-cu13-20260831-bb5e6198image.Add two PP4-prefill/DEP4-decode deployments at concurrency 704 and 565, two distinct TP2 aggregate fingerprints at concurrency 44, and one TP8 aggregate deployment at concurrency 7.
Add the five matching checked-in srt-slurm recipes and append one performance changelog entry.
使用上游
lmsysorg/sglang:nightly-dev-cu13-20260831-bb5e6198镜像，为 GB300 新增五个 Qwen3.5 NVFP4 AgentX Pareto 点。新增两个并发度为 704 和 565 的 PP4 prefill/DEP4 decode 部署、两个并发度为 44 的独立 TP2 聚合配置指纹，以及一个并发度为 7 的 TP8 聚合部署。
添加五个匹配的内置 srt-slurm 配方，并追加一条性能变更日志。
Runtime details / 运行时详情
All five points use MTP and HiCache. Throughput jobs inject the matching committed Qwen3.5 golden acceptance lengths;
EVAL_ONLYremoves simulated acceptance and retains real MTP verification.The PP recipes pin Dynamo router/source identity to
ad6f7d54ce7b99c53576b10482755c93c921fbac, the current head of ai-dynamo/dynamo#14064. That PR is still open and draft, so it remains an explicit dependency for the PP points.The SGLang image includes the support merged in sgl-project/sglang#36248. The recipes run the SGLang image as shipped; no engine source is mounted or patched.
五个点均使用 MTP 和 HiCache。吞吐量任务注入匹配的已提交 Qwen3.5 Golden AL；
EVAL_ONLY会移除模拟接受率并保留真实 MTP 验证。PP 配方将 Dynamo 路由器和源码标识固定到
ad6f7d54ce7b99c53576b10482755c93c921fbac，即 ai-dynamo/dynamo#14064 的当前提交。该 PR 仍为草稿且尚未合并，因此仍是 PP 点的明确依赖。SGLang 镜像包含已在 sgl-project/sglang#36248 合并的支持。配方直接运行未修改的 SGLang 镜像，不挂载或修补引擎源码。
Recipe applicability / 配方适用性
No separate SGLang cookbook PR is required: every added deployment uses the repository's multi-node srt-slurm path, including the one-node aggregate shapes, and the matching recipes are checked into this PR.
无需单独创建 SGLang cookbook PR：新增部署全部使用仓库的多节点 srt-slurm 路径（包括单节点聚合形态），匹配的配方已包含在本 PR 中。
Validation / 验证
Parsed the master config, runner config, changelog, and all five recipe YAML files.
Generated both exact config keys and the affected Qwen3.5 FP4 GB300 Dynamo-SGLang AgentX family; the selected changelog expands to five throughput jobs and four eval-only jobs with positive node counts
5/7/1/1/2.Verified master-to-recipe parity for image, worker counts, TP/PP/EP, data-parallel attention, and node allocation.
Ran
srtctlv1.0.38 preflight and dry-run successfully for all five recipes.Ran the synthetic-acceptance injection/removal check for all five recipes and
runners/test_slurm_utils.py(15 passed).Validated the committed changelog against
origin/main, preserved all historical changelog bytes, and passedgit diff --check.已解析主配置、runner 配置、变更日志及全部五个配方 YAML 文件。
已生成两个精确配置键及受影响的 Qwen3.5 FP4 GB300 Dynamo-SGLang AgentX 配置族；变更日志选择结果包含五个吞吐量任务和四个仅评测任务，节点数
5/7/1/1/2均为正数。已验证主配置与配方在镜像、worker 数、TP/PP/EP、数据并行注意力和节点分配方面保持一致。
全部五个配方均已通过
srtctlv1.0.38 preflight 和 dry-run。已对全部五个配方验证模拟接受率的注入与移除，并运行
runners/test_slurm_utils.py（15 项通过）。已基于
origin/main验证已提交的变更日志，保留全部历史字节，并通过git diff --check。Follow-up / 后续
The public full sweep and evals have not been started. This PR creation does not apply a sweep label or dispatch GPU work.
尚未启动公开 full sweep 和评测。本次创建 PR 不添加 sweep 标签，也不触发 GPU 任务。
Note
Low Risk
Config-only changes (recipes, master search space, changelog) with no runtime or application code paths modified.
Overview
Adds five GB300 Qwen3.5 NVFP4 AgentX Pareto points on
lmsysorg/sglang:nightly-dev-cu13-20260831-bb5e6198, wired through two newnvidia-master.yamlkeys (qwen3.5-fp4-gb300-dynamo-sglang-agentic-pp-paretoand...-agg-pareto) plus matching checked-in srt-slurm recipes undergb300-fp4/agentic/.Disaggregated (throughput) points use Dynamo KV routing (pinned to
ad6f7d54ce7b99c53576b10482755c93c921fbac), PP4 prefill, and DEP4 decode with MTP + HiCache at concurrency 704 (3p2d) and 565 (3p4d). Aggregate points cover two distinct TP2 concurrency-44 fingerprints (batch 1 vs 2, HiCache 32 vs 128) and one TP8 concurrency-7 shape on two nodes. Master entries inject synthetic acceptance lengths for throughput sweeps and point each scenario at itsCONFIG_FILErecipe; benchmarks run viaagentic_srt.sh.perf-changelog.yamldocuments the refresh and notes PP recipes intentionally omit PR #36248 breakable prefill CUDA Graph settings.Reviewed by Cursor Bugbot for commit 60873a8. Bugbot is set up for automated code reviews on this repo. Configure here.