[AMD][MI35X] Serve Qwen3.5 MXFP4-AttnFP8-V2 on the MI355X SGLang

[yichiche]

Motivation

amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 is a newer AMD checkpoint of Qwen3.5-397B-A17B. It keeps the routed experts in MXFP4, additionally quantizes the shared expert to MXFP4 and fuses it into the routed MoE kernel, and runs self_attn / linear_attn in FP8 (per-channel static weights, per-token dynamic activations). Its published GSM8K is 97.24 vs. 97.71 for the bf16 Qwen/Qwen3.5-397B-A17B baseline (99.55% recovery). It supersedes amd/Qwen3.5-397B-A17B-MXFP4 as the checkpoint the MI355X SGLang single-node fixed-seq-len submission should represent.

This is a redo of #2695 on top of current main: that PR predates #2640 (image bump to v0.5.18-rocm720-mi35x-20260829, --kv-cache-dtype fp8_e4m3, --disable-shared-experts-fusion), which has since landed and made it conflict.

Modifications

In configs/amd-master.yaml, on qwen3.5-fp4-mi355x-sglang and qwen3.5-fp4-mi355x-sglang-mtp:

1. model: amd/Qwen3.5-397B-A17B-MXFP4 → amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2.
2. Drop the { tp: 4, conc-start: 4, conc-end: 16 } band from both arms, leaving TP2.

Why TP4 has to go. The first sweep on this PR (run 33484262630) crashed every TP4 job in prefill:

qwen3_5.py:686   projected_states_ba, _ = self.in_proj_ba(hidden_states)
 → quark/schemes/quark_w8a8_fp8.py:178   apply_fp8_linear(...)
 → fp8_utils.py:1987                     gemm_a8w8_bpreshuffle(...)
RuntimeError: gemm_a8w8_bpreshuffle failed for shape M=7237, N=32, K=4096: This GEMM is not supported!

The checkpoint's quantization_config is quant_method: quark with layer_quant_config entries for *linear_attn* and *self_attn* (fp8_e4m3, qscheme: per_channel, is_dynamic: true). So QuarkW8A8Fp8 sets per_token=True and, under the SGLANG_USE_AITER=1 the scripts already export, pre-shuffles the weights and dispatches to aiter gemm_a8w8_bpreshuffle.

in_proj_ba is MergedColumnParallelLinear(hidden_size=4096, output_sizes=[num_v_heads, num_v_heads]), i.e. N=128 total. TP2 shards it to N=64; TP4 shards it to N=32. aiter's CK dispatch for K≥1536 / M≥256 / N<1536 picks an NPerBlock=64 instance, and the bpreshuffle instances are generated with GemmSpecialization::Default — no MN padding, because the weight is pre-shuffled — so IsSupportedArgument rejects N=32. This is structural in the current CK instance set, not a tuning gap, so a newer image alone would not fix it. The band can come back once apply_fp8_linear falls back to torch._scaled_mm rowwise for N % 64 != 0; a comment in the config records that.

The image stays at lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260829, and isl/osl, runner, precision, framework, and all launch flags are untouched — no launch-script change in this PR at all. runners/launch_mi355x-amds.sh resolves the launch script from model-prefix, precision, and framework, so both arms keep using the same two scripts with the checkpoint arriving through $MODEL.

The qwen3.5-fp4-mi355x-atom arm still serves amd/Qwen3.5-397B-A17B-MXFP4 and keeps both TP bands; the AgentX and disagg fp4 arms are untouched.

Append the corresponding perf-changelog.yaml trigger covering both config keys.

An earlier revision of this PR also added SGLANG_USE_AITER_FP8_PER_TOKEN=1 to the two launch scripts, mirroring the checkpoint's published repro recipe. That has been removed: the variable is read only by Fp8LinearMethod (fp8.py:481), which handles quant_method: fp8 checkpoints. This checkpoint is quant_method: quark, so its attention linears go through QuarkW8A8Fp8, which never consults the variable and already enables per-channel weights, per-token activations, and the aiter shuffle from the checkpoint's own config. The export was a no-op here, and removing it does not affect the TP4 failure.

Accuracy Tests

GSM8K from the sweep on this PR, all four eval rows green on the new checkpoint (lm-eval, 5-shot, chat template):

arm	TP	conc	em_strict	em_flexible
qwen3.5-fp4-mi355x-sglang	2	64	0.9697	0.9568
qwen3.5-fp4-mi355x-sglang	2	256	0.9651	0.9530
qwen3.5-fp4-mi355x-sglang-mtp	2	64	0.9712	0.9575
qwen3.5-fp4-mi355x-sglang-mtp	2	128	pass	—

No accuracy-affecting logic changes in this repo; correctness of the checkpoint itself is upstream.

Benchmarking

Repo validation was run locally:

• python -m pytest utils/matrix_logic/ -q → 267 passed.
• python utils/matrix_logic/generate_sweep_configs.py full-sweep --config-files configs/amd-master.yaml --model-prefix qwen3.5 --precision fp4 --runner-type mi355x --scenario-type fixed-seq-len → 23 configs: 13 on amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (7 TP2 non-MTP, 6 TP2 MTP) at the unchanged image, plus the 10 unchanged ATOM configs still on amd/Qwen3.5-397B-A17B-MXFP4. Down from 29 because the two TP4 bands (6 configs) are gone.

End-to-end MI355X throughput numbers come from the sweep triggered on this PR (full-sweep-fail-fast). Because the checkpoint changes and the TP4 band is removed, the resulting numbers are not a like-for-like continuation of the previous MXFP4 series on these two config keys.

Conflicts

Supersedes #2695, which is the same checkpoint change against a pre-#2640 main. #2695 should be closed if this lands.

---

Note

Low Risk
Benchmark configuration and changelog only; no launch scripts or runtime logic in this repo.

Overview
Updates the MI355X SGLang fixed-seq-len benchmark matrix for Qwen3.5 FP4 by pointing qwen3.5-fp4-mi355x-sglang and qwen3.5-fp4-mi355x-sglang-mtp at amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 instead of the older MXFP4-only checkpoint (FP8 attention + fused shared-expert MoE). TP4 concurrency sweeps are removed from both keys, leaving TP2 only, because TP4 triggers unsupported aiter FP8 GEMM shapes during prefill.

A matching perf-changelog.yaml entry documents the checkpoint swap and TP4 removal for those two config keys. ATOM, agentic, and disagg arms are unchanged.

^{Reviewed by Cursor Bugbot for commit febaac4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 New changelog entry's pr-link is the literal placeholder string "PR_LINK_PLACEHOLDER" instead of a real PR URL, unlike every other entry in this append-only file. Fix: replace with the actual GitHub PR URL before merge; any tooling that parses pr-link (e.g. to build changelog links) will get a broken/non-URL value otherwise.

Extended reasoning...

perf-changelog.yaml is append-only and each prior entry (e.g. line 6730's pr-link: https://github.com/.../pull/2634) holds a real URL. This new entry at the tail sets pr-link: PR_LINK_PLACEHOLDER, so any consumer of this file (changelog renderer, doc generator, or a human clicking through) that expects a valid URL for every entry will fail or produce a dead link for this record specifically.

Verification: normal. The new entry at perf-changelog.yaml:6742 sets pr-link: PR_LINK_PLACEHOLDER, an invalid value. This is stronger than the candidate frames it (not just a dead link): the repo's own changelog gate rejects it. In utils/validate_perf_changelog.py, accepted placeholders are only PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"} (lines 24-27), and…

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33484262630
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33484262630

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33598359757
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33598359757