perf(agentx): refresh K3 MI355X vLLM recipe with DCP8 MTP arm by seungrokj · Pull Request #2810 · SemiAnalysisAI/InferenceX

seungrokj · 2026-09-03T00:11:00Z

Summary

Bump kimik3-fp4-mi355x-vllm-agentic-mtp to vLLM ROCm nightly 7c5dc571 and LMCache 0.5.5rc3.
Add a TP8/DCP8 spec-mtp DRAM-offload arm (conc 44/48/52).
Retune the LMCache tier (chunk-size 12288, max-gpu-workers 8) and the no-spec decode profile (GMU 0.9, 8192 batched tokens); switch the DCP attention backend to ROCM_AITER_MLA.

Test plan

Full-sweep validation of kimik3-fp4-mi355x-vllm-agentic-mtp

🤖 Generated with Claude Code

Note

Low Risk
Changes are confined to benchmark scripts, sweep config, and changelog; they do not alter application runtime code paths outside perf measurement.

Overview
Refreshes the Kimi-K3 FP4 MI355X vLLM agentic MTP benchmark recipe and matrix for higher concurrency and decode-context parallel (DCP).

The launch script now branches LMCache setup on DCP_SIZE: DCP runs pin 0.5.5rc3, use a 12288-token chunk and 8 GPU transfer workers; non-DCP stays on 0.5.5.dev60+rocm7.2 with chunk 3072 and 1 GPU worker. Spec/util profiles give conc 2/4/8 their own DSpark settings (matching 12/14), and the fallback no-spec path raises gpu-memory-utilization to 0.9. When DCP is enabled, decode attention switches from TRITON_MLA to ROCM_AITER_MLA.

kimik3-fp4-mi355x-vllm-agentic-mtp moves to a newer vLLM ROCm nightly image; the published search space is narrowed to a single LMCache DRAM arm at conc 14 while fuller arms (including TP8/DCP8 + LMCache rc3 at high conc) are left commented for staged sweeps. perf-changelog.yaml records the intended stack bump, DCP8 arm, and tuning details.

^{Reviewed by Cursor Bugbot for commit f48e07a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Bump the kimik3-fp4-mi355x-vllm-agentic-mtp image to vLLM ROCm nightly 7c5dc571 and LMCache to 0.5.5rc3, and add a TP8/DCP8 spec-mtp DRAM-offload arm (conc 44/48/52). Retune the LMCache tier (chunk-size 12288, max-gpu-workers 8) and the no-spec decode profile (GMU 0.9, 8192 batched tokens), and switch the DCP attention backend to ROCM_AITER_MLA. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-09-03T00:11:10Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

github-actions · 2026-09-03T00:11:10Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

github-actions · 2026-09-03T00:11:10Z

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

claude

Additional findings (outside the current diff — GitHub can't attach inline comments there):

🔴 perf-changelog.yaml — This PR bumps the vLLM image, retunes LMCache chunk-size/max-gpu-workers/GPU_MEM_UTIL/MAX_NUM_BATCHED_TOKENS, switches the DCP attention backend, and adds a brand-new dcp-size:8 search-space arm, yet perf-changelog.yaml is not updated, violating AGENTS.md's requirement that every performance-affecting change and recipe addition gets a new append-only entry. Fix: append a perf-changelog.yaml entry for config-key kimik3-fp4-mi355x-vllm-agentic-mtp describing the image bump, LMCache retune, and the new DCP8 arm, preserving all existing bytes and appending only at the tail. [also at: configs/amd-master.yaml:639 - This PR bumps the vLLM image, LMCache version/tuning, GPU_MEM_UTIL/MAX_NUM_BATCHED_TOKENS, attention backend, and adds…]

Extended reasoning...

AGENTS.md line 31 states every change that can affect benchmark performance and every recipe addition/modification requires a new perf-changelog.yaml entry, and the file is append-only. git diff confirms no changes to perf-changelog.yaml in this PR despite: image SHA bump (nightly-1dc464d... -> nightly-7c5dc571...), LMCache chunk-size 3072->12288, max-gpu-workers 1->8, GPU_MEM_UTIL 0.85->0.9, MAX_NUM_BATCHED_TOKENS 4096->8192, attention-backend TRITON_MLA->ROCM_AITER_MLA for DCP, and a wholly new dcp-size:8 search-space entry in configs/amd-master.yaml. Downstream consumers of perf-changelog.yaml (release notes, performance tracking tooling) will show no record of this recipe's tuning history, and future readers cannot audit why these values changed.

Verification: normal. AGENTS.md line 31 (a "Non-negotiable benchmark invariant"): "Every change that can affect benchmark performance and every recipe addition or modification requires a new perf-changelog.yaml entry. The file is append-only and byte-sensitive." This PR clearly qualifies on both counts. The diff (git diff f2be63a..HEAD --stat) touches only two files —… | normal. AGENTS.md line 31 (a…

github-actions · 2026-09-03T00:44:11Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33698645774
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33698645774

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions · 2026-09-03T01:05:12Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33698645774
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33698645774

github-actions · 2026-09-03T01:40:42Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33702093274
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33702093274

Set --max-gpu-workers to 8 only when DCP_SIZE>1 (else 1), and reduce the kimik3-fp4-mi355x-vllm-agentic-mtp search-space to a c14 (non-DCP) and c48 (DCP8) smoke test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1b24693. Configure here.}

…dent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Widen prefill activation headroom for the c48 DCP8 arm, which repeatedly OOM'd in the attention-residual forward at 8192 batched tokens under GMU 0.9. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… c14 Set --chunk-size 12288 only when DCP_SIZE>1 (else 3072), install the 0.5.5.dev60+rocm7.2 wheel from the 0.5.5rc3 release index, and activate the c14 LMCache-DRAM arm. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Use 0.5.5rc3 when DCP_SIZE>1 and 0.5.5.dev60+rocm7.2 otherwise, and pin the active c14 arm to dev60. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

seungrokj requested a review from a team September 3, 2026 00:11

seungrokj requested review from 1am9trash, billishyahao, chunfangamd and yctseng0211 as code owners September 3, 2026 00:11

github-project-automation Bot added this to InferenceMAX Board Sep 3, 2026

seungrokj and others added 2 commits September 3, 2026 09:11

docs(changelog): link K3 MI355X vLLM DCP8 MTP refresh PR

c281dbe

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

style(agentx): add trailing newline to K3 MI355X benchmark script

7b93468

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

seungrokj added AMD full-sweep-enabled agentx AgentX benchmarks, recipes, and infrastructure labels Sep 3, 2026

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh Outdated

claude Bot reviewed Sep 3, 2026

View reviewed changes

perf(agentx): drop c52 from K3 MI355X DCP8 arm

db7732d

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

seungrokj removed the full-sweep-enabled label Sep 3, 2026

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread configs/amd-master.yaml Outdated

perf(agentx): lower K3 c12/c14 GMU to 0.88; run c14-only smoke test

1701d42

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh

seungrokj and others added 3 commits September 3, 2026 11:30

perf(agentx): run K3 c12/c14 GMU 0.88 smoke test

aefefb3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): lower K3 c12/c14 GMU to 0.85

5cebe65

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): test K3 c14 GPU-resident (no KV offload), GMU 0.9

1b24693

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

cursor Bot reviewed Sep 3, 2026

View reviewed changes

Comment thread benchmarks/single_node/agentic/kimik3_fp4_mi355x_mtp.sh Outdated

seungrokj and others added 7 commits September 3, 2026 13:25

perf(agentx): bump K3 image to nightly-73029d42; c14 no-spec GPU-resi…

ffc4d74

…dent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): run K3 c48 DCP8 LMCache smoke test on nightly-73029d42

99d0671

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): run K3 c14 GPU-resident with MTP spec on nightly-73029d42

34d93c9

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): drop K3 high-conc max-num-batched-tokens to 4096 (OOM fix)

883d6e3

Widen prefill activation headroom for the c48 DCP8 arm, which repeatedly OOM'd in the attention-residual forward at 8192 batched tokens under GMU 0.9. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): activate K3 c48 DCP8 arm for OOM-fix validation

60acfb3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

perf(agentx): gate K3 LMCache wheel on DCP; run c14 on dev60

f48e07a

Use 0.5.5rc3 when DCP_SIZE>1 and 0.5.5.dev60+rocm7.2 otherwise, and pin the active c14 arm to dev60. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Navigation Menu

Uh oh!

perf(agentx): refresh K3 MI355X vLLM recipe with DCP8 MTP arm - #2810

perf(agentx): refresh K3 MI355X vLLM recipe with DCP8 MTP arm#2810
seungrokj wants to merge 16 commits into
mainfrom
amd/agentx_k3_vllm_0903_2

seungrokj commented Sep 3, 2026 •

edited by cursor Bot

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Uh oh!

claude Bot left a comment

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

seungrokj commented Sep 3, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

seungrokj commented Sep 3, 2026 •

edited by cursor Bot

Loading