Skip to content

[AMD][MI35X] Enable fused allreduce+RMSNorm for Qwen3.5 MXFP4 MI355X agentic-mtp #2801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ChangLiu0709 wants to merge 3 commits into
base: main
Choose a base branch
from
+22 −4
Open

[AMD][MI35X] Enable fused allreduce+RMSNorm for Qwen3.5 MXFP4 MI355X agentic-mtp #2801

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f895cef
[AMD][MI35X] Enable fused allreduce+RMSNorm for Qwen3.5 MXFP4 agentic…
ChangLiu0709 Sep 1, 2026
6ecec64
Update perf-changelog
chunfangamd Sep 2, 2026
4db558d
qwen3.5-fp4-mi355x: trim concurrency sweep, drop hicache configs
ChangLiu0709 Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,10 @@ export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export ROCM_QUICK_REDUCE_QUANTIZATION=INT8
# Fused AR+RMSNorm via AITER custom allreduce (replaces INT8 QuickReduce;
# the two are mutually exclusive). Measured +5.9% output throughput at
# TP2/conc4 agentic workload vs the QuickReduce baseline.
# export ROCM_QUICK_REDUCE_QUANTIZATION=INT8
export SGLANG_TIMEOUT_KEEP_ALIVE=1800

if [ "${EVAL_ONLY:-false}" != "true" ]; then
Expand Down Expand Up @@ -127,6 +130,7 @@ SGLANG_CMD=(
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4
--enable-aiter-allreduce-fusion
--enable-metrics
--enable-cache-report
"${CACHE_ARGS[@]}"
Expand Down
4 changes: 1 addition & 3 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,10 +372,8 @@ qwen3.5-fp4-mi355x-sglang-agentic-mtp:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32, 40] }
- { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20, 24, 28, 32] }
- { tp: 2, ep: 1, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 4, 8, 12, 16, 20] }
- { tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [40, 48, 56, 64] }
- { tp: 2, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [20, 24, 28, 32] }

qwen3.5-fp4-mi355x-sglang-disagg:
image: lmsysorg/sglang-rocm:v0.5.12.post1-rocm720-mi35x-20260523
Expand Down
16 changes: 16 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6821,3 +6821,19 @@
description:
- "Refresh to collect TensorRT-LLM server metrics."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2774

- config-keys:
- qwen3.5-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Enable AITER fused allreduce+RMSNorm (--enable-aiter-allreduce-fusion), replacing INT8 QuickReduce. +5.9% output throughput per GPU at TP2/conc4 agentic workload (114.1 -> 120.8 tok/s). -20.1% decode cycle time from kernel-level profiling."
Comment on lines 6822 to +6830

@claude claude Bot Sep 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 New changelog entry omits the required pr-link field, which every prior entry in the file includes and which ChangelogEntry.pr_link (utils/matrix_logic/validation.py:1031) requires with no default. Fix: add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/<number> (or pr-link: TBD before the PR exists, per docs/configuration-procedures.md) to this entry so schema validation succeeds.

Extended reasoning...

ChangelogEntry is a pydantic model with extra='forbid' and pr_link: str = Field(alias="pr-link") — no default value, so a missing key raises a pydantic ValidationError when the changelog is parsed. Every other entry in perf-changelog.yaml (e.g. the immediately preceding one at line ~6819 for pull/2774) includes pr-link. Any CI step or matrix-logic tooling that loads perf-changelog.yaml via ChangelogMetadata/ChangelogEntry will fail on this new tail entry, blocking the PR's changelog validation step until pr-link is added.

Verification: normal. The new tail entry at perf-changelog.yaml:6825-6830 has config-keys, scenario-type, and description but no pr-link key (confirmed by reading the file tail; the immediately preceding entry at 6823 and every other entry include pr-link). ChangelogEntry (utils/matrix_logic/validation.py:1025-1031) declares pr_link: str = Field(alias="pr-link") with NO default, under `model_config =…

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2801

- config-keys:
- qwen3.5-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Trim concurrency sweep: remove hicache KV-offloading configs (TP4/TP2 dram), cap TP4 no-offload conc-list at 32 (was 40)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2801