[AMD][MI35X] Enable fused allreduce+RMSNorm for Qwen3.5 MXFP4 MI355X agentic-mtp - #2801
[AMD][MI35X] Enable fused allreduce+RMSNorm for Qwen3.5 MXFP4 MI355X agentic-mtp#2801ChangLiu0709 wants to merge 2 commits into
Conversation
…-mtp Replace INT8 QuickReduce with AITER fused allreduce+RMSNorm (--enable-aiter-allreduce-fusion) on the Qwen3.5 MXFP4 MI355X agentic-coding benchmark arm. The fused kernel combines TP allreduce, residual addition, and RMSNorm into a single HIP kernel launch, cutting 256 separate kernel launches down to 134 per decode cycle. Results (TP2, conc=4, 1-hour AgentX agentic replay): - Output throughput/GPU: 114.1 -> 120.8 tok/s (+5.9%) - Decode cycle time: 14,316 -> 11,440 us (-20.1%) - MTP acceptance length: 3.39 (unchanged) - Error rate: 0/892 = 0.0% Config key: qwen3.5-fp4-mi355x-sglang-agentic-mtp Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Refresh to collect TensorRT-LLM server metrics." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2774 | ||
|
|
||
| - config-keys: | ||
| - qwen3.5-fp4-mi355x-sglang-agentic-mtp | ||
| scenario-type: | ||
| - agentic-coding | ||
| description: | ||
| - "Enable AITER fused allreduce+RMSNorm (--enable-aiter-allreduce-fusion), replacing INT8 QuickReduce. +5.9% output throughput per GPU at TP2/conc4 agentic workload (114.1 -> 120.8 tok/s). -20.1% decode cycle time from kernel-level profiling." |
There was a problem hiding this comment.
🔴 New changelog entry omits the required pr-link field, which every prior entry in the file includes and which ChangelogEntry.pr_link (utils/matrix_logic/validation.py:1031) requires with no default. Fix: add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/<number> (or pr-link: TBD before the PR exists, per docs/configuration-procedures.md) to this entry so schema validation succeeds.
Extended reasoning...
ChangelogEntry is a pydantic model with extra='forbid' and pr_link: str = Field(alias="pr-link") — no default value, so a missing key raises a pydantic ValidationError when the changelog is parsed. Every other entry in perf-changelog.yaml (e.g. the immediately preceding one at line ~6819 for pull/2774) includes pr-link. Any CI step or matrix-logic tooling that loads perf-changelog.yaml via ChangelogMetadata/ChangelogEntry will fail on this new tail entry, blocking the PR's changelog validation step until pr-link is added.
Verification: normal. The new tail entry at perf-changelog.yaml:6825-6830 has config-keys, scenario-type, and description but no pr-link key (confirmed by reading the file tail; the immediately preceding entry at 6823 and every other entry include pr-link). ChangelogEntry (utils/matrix_logic/validation.py:1025-1031) declares pr_link: str = Field(alias="pr-link") with NO default, under `model_config =…
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33588417338 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33588417338 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33588417338 |
2 participants
Summary
Enable AITER fused allreduce+RMSNorm (
--enable-aiter-allreduce-fusion) on the Qwen3.5-397B-A17B-MXFP4 MI355X agentic-coding benchmark arm, replacing the INT8 QuickReduce path.Config keys touched:
qwen3.5-fp4-mi355x-sglang-agentic-mtpScope
Benchmark recipe change only (launch flags + env vars). No image bump — same
lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829.What changed in the recipe
export ROCM_QUICK_REDUCE_QUANTIZATION=INT8--enable-aiter-allreduce-fusionadded to SGLang server argsThe two paths are mutually exclusive — AITER custom allreduce fusion requires QuickReduce to be disabled.
Motivation
The AITER
allreduce_fusion_kernel_1stageHIP kernel fuses TP allreduce + residual addition + RMSNorm into a single kernel launch, eliminating ~122 redundant kernel launches per decode cycle (from 256 → 134 total AR+RMS launches).This targets the decode-phase bottleneck where Qwen3.5's 60-layer architecture (45 KDA + 15 MHA) generates ~256 standalone allreduce and RMSNorm kernel calls per cycle, each dominated by launch overhead (99.5% of per-call time) rather than compute or memory bandwidth.
Results
Kernel-level profiling (TP2, conc=4, EAGLE 3-step MTP)
Production smoke test (TP2, conc=4, 1-hour AgentX agentic replay)
Validation
Relying on the labeled
full-sweep-fail-fastsweep.Made with Cursor
Co-authors
@ChangLiu0709
@chunfangamd