Repository navigation
feat(serving): greenspec scenario, spec-decode acceptance, arms, run summary - #43
Merged
Merged
Conversation
…y table Port of the GreenSpec bench (PR #42 on vng-benchmark) to the serving refactor. - scenario `greenspec`: raw_payload file built from one selection of the private Green-Lab/GreenSpec release (overrides: subset, lite, bucket, mode, max_tokens, requests); the release is pinned in datasets/lock.yaml. - infx/serving/greenspec.py: selection, request bodies, console filter and the artifact guard. The benchmark repo is public, so only an allowlist of files is uploaded, error text is withheld and server logs are stubbed. - infx/serving/specdec.py: acceptance length and rate from the engine's spec_decode counters before/after the replay, merged into the aggregate as `spec_decode`. - to_matrix: spec-decoding follows --speculative-config / --speculative-algorithm (was always none). - infx/serving/summary.py and a `summary` job in gn-benchmark.yml: one table of the run's points with the speedup of a speculative point over its base point. - config example, docs and skills updated; GREENLAB_HF_TOKEN is now needed on a runner that runs this scenario.
… throughput.output.tokens_per_second)
`arms:` gives each variant its own job and engine (serve: and env: shared, an arm adds `append` flags and overlays env), so a speculative arm and its base arm land in one run and the summary table can give the speedup. Point names are <name>-<arm>_c<conc>. Arms with identical engine command and env are rejected; arms: does not combine with endpoint:. - specdec: draft_depth (drafted tokens / draft events) in the spec_decode block; the summary table shows it next to the spec method (mtpx6). - configs/serving: the greenspec example now carries both arms.
… the private dataset
Noridom1
marked this pull request as ready for review
October 30, 2026 02:25
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
Collaborator
Author
|
Hi @thangquang09, please review, make no mistake :D |
v1.1.1 fixes tool parameters whose required keyword was {} instead of a list
(548 of 3072 rows); SGLang rejects those with HTTP 400, vLLM accepts them.
payload() also rewrites the keyword, so an older release still runs on both engines.
greenspec-v1.1.1 is marked private in lock.yaml; datasets.py takes an optional boolean (default false, one-line error otherwise) and the runner-setup skill keys off the flag instead of hand-copying the private entry names. Drop the test that pins lock.yaml contents; a small test covers the private flag parsing instead.
Collaborator
|
Confirmed mergeable. Verified locally at the latest head: 272/272 serving tests pass, ruff check + format clean; CI Lint/Tests green (Zizmor red is pre-existing on the base branch — same 4 findings with zizmor run on base, and this PR actually fixed one). All review points addressed, and the two follow-up fixes are verified by real runs:
Merging into |
thangquang09
added a commit
that referenced
this pull request
Oct 31, 2026
Pick up the greenspec v1.1.1 pin, the SGLang 0.0-gauge fix and the lock private flag (e5ba0c2) merged via #43. Conflicts resolved: lock.yaml keeps the six speedbench entries beside greenspec-v1.1.1, runner-setup SKILL.md keeps the wording that names both scenarios. 274 serving tests pass, ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the GreenSpec speculative-decoding bench (previously PR #42 on
vng-benchmark) to the serving-YAML architecture.greenspecscenario family (scenarios/greenspec.yaml,infx/serving/greenspec.py). The datasetGreen-Lab/GreenSpecis pinned inlock.yaml(greenspec-v1.1.1, revisionf7ea173f). Rows become request bodies in a node-local selection file and are replayed through aiperf (raw_payload). Overrides:subset,lite,bucket,mode,max_tokens,requests.required: {}in the tool schemas.greenspec.payload()also rewrites any remainingrequired: {}to[], so an older pin does not hit the HTTP 400 that SGLang returns for it (vLLM does not validate).infx/serving/specdec.py). Prometheus counters are snapshotted before and after the replay. The aggregate JSON gets aspec_decodeblock with acceptance length, acceptance rate and draft depth. Separate module so a future speedbench family can reuse it.arms:in serving YAML. Engine variants (for example base and MTP) shareserve:andenv:; each arm becomes its own job and point in one run. Duplicate arms are rejected. Not allowed withendpoint:.spec-decodingis derived per arm.infx/serving/summary.pyplus asummaryjob ingn-benchmark.yml). Writes a table to$GITHUB_STEP_SUMMARYwith throughput, TTFT, TPOT, acceptance and speedup over the base arm.question_idremains in an upload.GREENLAB_HF_TOKENis passed throughgn-benchmark.yml->benchmark-tmpl.ymllikeSERVING_API_KEY. Without it every arm fails with "GREENLAB_HF_TOKEN is unset" (9f6d288).configs/serving/gemma4-greenspec-diverse.yaml, docs (ARCHITECTURE.md,serving-benchskill), tests.Needs
The
GREENLAB_HF_TOKENrepository secret (documented inserving-runner-setup).Verified
refactor/vng-benchmark@ 708166e (the 18srt_slurm/launch failures in the wider suite needsrtctland are not from this change).rtx6000pro-hcm(run 40859952101, before the aiperf-native merge, agentx-harness 1.0.12; vLLM v0.30.0, MTP depth 6, c16, 50 requests): both arms 50/50 ok, the token reads the dataset, one RTX 6000 Pro serves--max-model-len 262144with FLASHINFER. The aggregate carriesspec_decode: acceptance length 5.03, accept rate 67.2%. The summary table renders withmtpx6in the spec column. The 0.97x speedup there is not meaningful: the KV cache holds about one 256k request, so c16 queues (TTFT p50 264-380 s).rtx6000pro-hcm, aiperf 0.13.0.dev20261027, 50 requests, c16): both arms 50/50 ok, 0 dropped.spec_decodeon the MTP arm: acceptance length 4.88, accept rate 64.7%. Thesummaryjob succeeded; the table renders locally from the artifacts withmtpx6and a 1.05x speedup (not meaningful on RTX, same KV limit).rtx6000pro-hcm(v1.1.1, 20diverserows, base arm): at c16 on one GPU a run failed with 2/16 requests over the aiperf 10% failure threshold. The SGLang log shows a KV pool of 130823 tokens against--context-length 262144, so the longest prompts probably do not fit (hypothesis, server error text is withheld by the leak guard). TP2 and--kv-cache-dtype fp8_e4m3variants were run at c2 to check this; fp8 KV changes numerics, so those runs are for error checking only. Results: see the comment below.cluster:hcm-hgx02(run 40679051440) also passed.Not yet verified
summaryjob writing the table to the GitHub run page was checked by rendering the artifacts locally, not on the page.cluster:hcm-hgx02(H200). No full 3072-row run.