[Noridom1]

Stacked on #43 (base feat/greenspec-refactor); once #43 merges the base moves to refactor/vng-benchmark and the diff stays the same.

What

New serving scenario family speedbench: SPEED-Bench prompts replayed through AgentX/aiperf as raw_payload request bodies, one request per dataset row (no duration, no ignore_eos). Spec-decode acceptance length comes from the same spec_decode block GreenSpec uses; arms: (base vs MTP) and the summary speedup column work unchanged.

• infx/serving/speedbench.py: selection (config, category, requests), sampling (max_tokens 4096, temperature 1.0, top_p 0.95 as in the golden-AL collectors), chat_template_kwargs for the thinking switch, request payloads. Only the first turn of a row is sent, as the aiperf/vLLM loaders do.
• client.py: prepare/run take either family through selection_family(). The leak guard and filtered console stay GreenSpec-only; SpeedBench data is public, nothing is filtered.
• Dataset lock: 6 entries (speedbench-qualitative, speedbench-throughput-{1k,2k,8k,16k,32k}), pinned revision + sha256. They point at Green-Lab/speed-bench-prepared (private copy with every row resolved; nvidia/SPEED-Bench masks ~60% of the qualitative rows). Needs GREENLAB_HF_TOKEN, already passed by feat(serving): greenspec scenario, spec-decode acceptance, arms, run summary #43.
• configs/serving/gemma4-speedbench-qualitative.yaml example, ARCHITECTURE.md section, tests.

Verified

• Smoke on rtx6000pro-hcm (run 40992115048, commit 657bb36): Gemma-4-31B FP8, base + mtp6, 20 coding requests, conc 1. Both arms 20/20; MTP6 acceptance length 4.94, accept rate 65.7%; summary table shows the speedup column. The dataset was read with the runner token and matched its sha256.
• ruff check / ruff format --check on inferencex-e2e/infx clean; 265 serving tests pass.

Not verified

• Sampling (temperature, top_p, enable_thinking=false) in the live run: aiperf artifacts do not keep the request body. Covered by unit tests only.
• Throughput configs (1k to 32k) and the full 880-row qualitative run were not run live.
• The 3.76x speedup in the smoke is 20 requests on an RTX 6000 Pro; it says nothing about real speedup.
• 4 test files are skipped locally (codeowners, git, srtctl missing from my venv); CI runs them.

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[github-actions]

inferencex-e2e/infx diff: +429 -9

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

[Noridom1]

Ready for review. SpeedBench adds a speedbench scenario family: SPEED-Bench prompts replayed via aiperf, with spec-decode acceptance length and base-vs-MTP speedup in the summary. Stacked on #43, so please merge #43 first; the base will move to refactor/vng-benchmark after that.

Verified: smoke on rtx6000pro-hcm (run 40992115048), base + mtp6, 20/20 requests each, MTP6 acceptance length 4.94. Lint and Tests are green.

Hi @thangquang09, please review. Make no mistake :D

[thangquang09]

Approve with changes — code, tests (266 pass at head), ruff and smoke runs all check out. Two doc fixes before merge:

1. serving-bench/SKILL.md — add speedbench to the scenario: enum (line 36) and a row to the variants table. Agents can't discover the new family as-is.
2. serving-runner-setup/SKILL.md (line 76) — "every repo in lock.yaml is public except greenspec-v1.1.0" is now wrong: this PR adds 6 more private entries. scenario: speedbench also needs GREENLAB_HF_TOKEN.

Suggestion: both misses are hand-copied facts drifting. Add private: true to lock entries and reword runner-setup to "every lock entry marked private needs GREENLAB_HF_TOKEN" — done forever.

Minor: consider dropping test_lock_pins_greenspec_and_speedbench_without_entries (pins lock.yaml contents; edited in both #43 and #45).

[Noridom1]

Thanks @thangquang09. Addressed in c8135ba, merged with the latest feat/greenspec-refactor (conflicts in lock.yaml and test_datasets.py resolved).

1. serving-bench/SKILL.md: speedbench is in the scenario: enum, with a row in the scenarios table and a variants row (config, category, requests).
2. serving-runner-setup/SKILL.md: reworded as you suggested. Every lock entry marked private: true needs GREENLAB_HF_TOKEN, and the text names both greenspec and speedbench.
3. Lock entries now carry private: true (the loader accepts it as an optional boolean, default false; a non-boolean is a one-line error). The seven private entries (greenspec-v1.1.1 and the six speedbench ones) are marked.
4. Dropped test_lock_pins_greenspec_and_speedbench_without_entries. A small test covers the private flag parsing instead.

Local: 272 tests pass, ruff check and format clean. The runner-setup rule is not enforced by any check; it is only reworded.

🤖 Generated with Claude Code

[thangquang09]

Confirmed mergeable after #43 merges (your branch is one commit behind feat/greenspec-refactor — e5ba0c2b8, the lock private: true commit — which is why this shows conflicting; I'll merge it in after #43). Both SGLang smokes verified: speedbench 41035309192 and greenspec v1.1.1 41036862025 (20/20 both arms, acceptance 4.125 / 3.7).