feat(serving): speedbench scenario family (SPEED-Bench acceptance length)

.agents/skills/serving-bench/SKILL.md

@@ -33,7 +33,7 @@ storage:                            # optional: a non-GPU sidecar started first
 arms:                               # optional: engine variants in one run, e.g. base vs speculative (see ARCHITECTURE.md, Arms)
   base: {}
   mtp6: {append: "--speculative-config '...'"}
-scenario: agentx                    # agentx | tau2 | livekit | shared-prefix-e2 | hidden-prefix-e3 | synthetic | greenspec
+scenario: agentx                    # agentx | tau2 | livekit | shared-prefix-e2 | hidden-prefix-e3 | synthetic | greenspec | speedbench
 overrides: {duration: 180}          # optional variant, see the table below
 conc: [4, 8]                        # one benchmark job per value
 ```
@@ -50,6 +50,7 @@ Working examples, one per pattern: `inferencex-e2e/configs/serving/*.yaml`. They
 | `shared-prefix-e2` / `hidden-prefix-e3` | prefix-cache A/B traces (HiCache L2 vs Mooncake L3) | 1800 s |
 | `synthetic` | synthetic ISL/OSL with a shared-prefix cache hit, needs `workload:` | 300 s per point |
 | `greenspec` | speculative-decoding bench from production chat traffic (private; result has `spec_decode.acceptance_length`) | the whole selection, no duration |
+| `speedbench` | SPEED-Bench acceptance-length prompts (private copy of public data; result has `spec_decode.acceptance_length`) | the whole selection, no duration |
 
 Variants (`overrides:`, only these keys):
 
@@ -62,6 +63,7 @@ Variants (`overrides:`, only these keys):
 | LiveKit V1 / V2 trace | `overrides: {dataset: livekit-111868-v1}` (or `-v2`) |
 | a different trace idle-gap cap | `overrides: {trace_idle_gap_cap: 1}` |
 | GreenSpec subset, lite, bucket, cap | `scenario: greenspec` + `overrides: {subset: agentic_tools, lite: true, max_tokens: 4096, requests: 200}` (only these keys; no `duration`) |
+| SPEED-Bench config, category, cap | `scenario: speedbench` + `overrides: {config: qualitative, category: coding, requests: 40}` (config: `qualitative` or `throughput_{1k,2k,8k,16k,32k}`; no `duration`) |
 
 Synthetic grid, one job per point, all in one run:
 

.agents/skills/serving-runner-setup/SKILL.md

@@ -73,7 +73,7 @@ for d in $BASE/gharunner0*/actions-runner; do (cd "$d" && sudo ./svc.sh install
 
 The registration token comes from the settings page or `gh api -X POST repos/vngcloud/InferenceX/actions/runners/registration-token --jq .token` (valid exactly 1 hour, not configurable; mint it right before running `setup.sh`). `setup.sh` prints `All runners configured!` even when registration failed: check that every slot printed its own `[<name>] Done.` line.
 
-Every `datasets/lock.yaml` entry marked `private: true` (GreenSpec) needs `GREENLAB_HF_TOKEN`; the other repos are public. `infx/serving/datasets.py` downloads anonymously when `GREENLAB_HF_TOKEN` is unset, so a runner that is to run `scenario: greenspec` needs `GREENLAB_HF_TOKEN=…` in `actions-runner/.env`. Other scenarios need no `.env`.
+Every `datasets/lock.yaml` entry marked `private: true` (GreenSpec, SPEED-Bench) needs `GREENLAB_HF_TOKEN`; the other repos are public. `infx/serving/datasets.py` downloads anonymously when `GREENLAB_HF_TOKEN` is unset, so a runner that is to run `scenario: greenspec` or `scenario: speedbench` needs `GREENLAB_HF_TOKEN=…` in `actions-runner/.env`. Other scenarios need no `.env`.
 
 ## 3. Add the cluster record
 

inferencex-e2e/benchmarks/single_node/serving/datasets/lock.yaml

@@ -52,3 +52,42 @@ greenspec-v1.1.1:
   file: greenspec.jsonl
   sha256: 6a26785d07c235446e57ba34463a1ae9e2dcff938fec6246881492e511fb0f43
   private: true
+# SPEED-Bench, fully resolved: every row the public release masks is refetched from its source
+# (infx.serving.speedbench selects rows from the config's file). The prompts are public data, but the
+# resolved copy lives in a private repo: GREENLAB_HF_TOKEN must be able to read it.
+speedbench-qualitative:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: qualitative.jsonl
+  sha256: e9bbbe998472b79501ed79ba2bf126e58b63e153a6deef70ce2b63ea7a4de845
+  private: true
+speedbench-throughput-1k:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: throughput_1k.jsonl
+  sha256: 0b5665f61dcbf52dd357d515fd45f9e4c17c68eb902c788c81a4c9271b217377
+  private: true
+speedbench-throughput-2k:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: throughput_2k.jsonl
+  sha256: be247367a3ddd3c74b72d03122818d89c6e2e2fe956e6d66099d7ae0c45402a6
+  private: true
+speedbench-throughput-8k:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: throughput_8k.jsonl
+  sha256: 94f3ab2d3cd456654ee234faf888c459c5451069d6cfab6e051591b288e71739
+  private: true
+speedbench-throughput-16k:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: throughput_16k.jsonl
+  sha256: 77cbee1b4abc6b205ee9ad918bd8e1290a6514f4cf0ff9dc7db497afd1fffca2
+  private: true
+speedbench-throughput-32k:
+  repo: Green-Lab/speed-bench-prepared
+  revision: 184d7ed9fae19efa9d4d4e9959ac83cca83b3415
+  file: throughput_32k.jsonl
+  sha256: 04148f06e19c490856dcb9efc1b1fd4593354d4abd4bf07c6f5dc4ce91348c0b
+  private: true

inferencex-e2e/benchmarks/single_node/serving/scenarios/speedbench.yaml

@@ -0,0 +1,31 @@
+# SpeedBench family: SPEED-Bench acceptance-length prompts (nvidia/SPEED-Bench, fully resolved copy
+# Green-Lab/speed-bench-prepared; datasets/lock.yaml pins one jsonl per config).
+# infx.serving.speedbench selects the rows (overrides: config, category) and builds one request body per row
+# (first turn only, temperature/top_p/max_tokens/chat_template_kwargs from overrides) and writes them as the
+# raw_payload file IX_CUSTOM_DATASET_FILE points to; AgentX sends each body as is. Every request streams and
+# completes on its own (no ignore_eos): acceptance only means something on natural output. `requests` is the
+# selection size unless overrides.requests lowers it; the client sets it once it has counted the selection,
+# so the 1 below is only a placeholder. The dataset name below is a placeholder too: scenarios.py sets it
+# from overrides.config.
+schemaVersion: "2.0"
+benchmark:
+  unsafeOverride: true
+  endpoint:
+    type: chat
+    path: /v1/chat/completions
+    streaming: true
+    useServerTokenCount: true
+  datasets:
+  - name: speedbench-qualitative
+    type: file
+    format: raw_payload
+    path: ${IX_CUSTOM_DATASET_FILE}
+    sampling: sequential
+    randomSeed: 42
+  phases:
+  - name: profiling
+    kind: profiling
+    type: concurrency
+    requests: 1
+    failedRequestThreshold: 0.1
+randomSeed: 42

inferencex-e2e/configs/serving/gemma4-speedbench-qualitative.yaml

@@ -0,0 +1,40 @@
+# Pattern: SPEED-Bench acceptance length, thinking off, one category per job (speedbench scenario).
+# Gemma-4-31B FP8-block, vLLM, 1 GPU, one request in flight (the SPEED-Bench collectors' CONCURRENCY=1), MTP draft
+# depth 6 against the base engine in one run (arms:). Sampling is the golden AL matrices' (temperature 1.0,
+# top_p 0.95, 4096 output tokens); the thinking switch is chat_template_kwargs. Without `category:` the run
+# is all 880 prompts of the qualitative config (11 categories x 80); `config: throughput_8k` selects the
+# entropy tiers at 8k input. The run is one request per row of the selection (no duration): the results carry
+# spec_decode.acceptance_length. The prompts are public; the resolved copy is read with GREENLAB_HF_TOKEN.
+name: gemma4-speedbench-qualitative
+runner: cluster:hcm-hgx02
+image: vllm/vllm-openai:v0.30.0
+model: RedHatAI/gemma-4-31B-it-FP8-block
+gpus: 1
+precision: fp8block
+env:
+  NCCL_P2P_LEVEL: NVL
+  VLLM_DISABLE_COMPILE_CACHE: "1"
+  VLLM_ATTENTION_BACKEND: FLASHINFER
+serve: |
+  vllm serve RedHatAI/gemma-4-31B-it-FP8-block
+  --served-model-name RedHatAI/gemma-4-31B-it-FP8-block
+  --trust-remote-code
+  --tensor-parallel-size 1
+  --gpu-memory-utilization 0.92
+  --max-model-len 16384
+  --max-num-seqs 64
+  --no-enable-prefix-caching
+  --enable-auto-tool-choice
+  --tool-call-parser gemma4
+  --reasoning-parser gemma4
+arms:
+  base: {}
+  mtp6:
+    append: |
+      --speculative-config '{"method": "mtp", "model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 6}'
+scenario: speedbench
+overrides:
+  category: coding
+  chat_template_kwargs:
+    enable_thinking: false
+conc: [1]

inferencex-e2e/docs/serving/ARCHITECTURE.md

@@ -214,7 +214,7 @@ that several nodes need belong in a registry, where every node caches its own sq
 
 ## 6. Scenarios
 
-Seven base files in `benchmarks/single_node/serving/scenarios/`:
+Eight base files in `benchmarks/single_node/serving/scenarios/`:
 
 | Family | Workload | Dataset |
 |---|---|---|
@@ -225,6 +225,7 @@ Seven base files in `benchmarks/single_node/serving/scenarios/`:
 | `hidden-prefix-e3` | prefixes reused without being visible to the router | pinned file, Mooncake trace format |
 | `synthetic` | generated ISL/OSL with a shared-prefix share | synthetic |
 | `greenspec` | speculative-decoding bench cut from production chat traffic, one request per row | pinned private file (`Green-Lab/GreenSpec`), `raw_payload` format |
+| `speedbench` | SPEED-Bench acceptance-length prompts, one request per row | pinned files (`Green-Lab/speed-bench-prepared`, a private copy of public data), `raw_payload` format |
 
 Pinned files are listed in `benchmarks/single_node/serving/datasets/lock.yaml` with their URL and
 sha256; the client downloads and checks them before the run.
@@ -249,6 +250,16 @@ Three things differ from the other families:
   the aggregate result (`infx/serving/specdec.py`). A point without speculative decoding has no such block.
 - The runner needs `GREENLAB_HF_TOKEN` in `actions-runner/.env`: the release is not public.
 
+`speedbench` follows the same pattern (no duration, the `spec_decode` block, `GREENLAB_HF_TOKEN`) with its own
+keys: `config` (`qualitative`, default, or `throughput_1k` ... `throughput_32k`), `category` (one of
+that config's categories; default all), `max_tokens` (default 4096), `temperature` (1.0), `top_p` (0.95),
+`chat_template_kwargs` (the thinking switch, e.g. `enable_thinking: false`) and `requests`. The defaults are the
+golden AL matrices'. A request is the first turn of a row alone, as the aiperf and vLLM SPEED-Bench loaders send
+it. The result label carries every non-default key (`speedbench-qualitative-coding-ctk-enable_thinking-false`),
+so runs that differ in thinking mode never share a summary row. The files are the fully resolved copy of
+`nvidia/SPEED-Bench`: the public release masks about 60% of the qualitative rows. The prompts are public, so
+none of GreenSpec's guard applies: console, server log and results upload as for any other family.
+
 A base arm and a speculative arm are the `arms:` of one serving YAML (below). The `summary` job of
 `gn-benchmark.yml` prints one table of all points of a run on the run page, with the speedup of each
 speculative point over the base point of the same scenario and concurrency.

inferencex-e2e/infx/serving/client.py

@@ -20,10 +20,11 @@
 import urllib.request
 from collections.abc import Mapping
 from pathlib import Path
+from typing import Any
 
 from infx.bench import proc
 from infx.bench.agentic.venv import Runtime, bootstrap
-from infx.serving import greenspec, specdec
+from infx.serving import greenspec, specdec, speedbench
 
 OFFLINE = ("HF_HUB_OFFLINE", "TRANSFORMERS_OFFLINE", "HF_DATASETS_OFFLINE")
 DATASET_PATH = ".dataset_path"
@@ -193,6 +194,18 @@ def with_token_source(scenario_yaml: str, source: str) -> str:
     return yaml.safe_dump(doc, sort_keys=False)
 
 
+def selection_family(env: Mapping[str, str]) -> tuple[Any, Any] | None:
+    """The (module, Settings) of the selection family the job's serving YAML runs, else None.
+
+    greenspec and speedbench replay a selection of a pinned file, one request per row, and report
+    acceptance length. Only greenspec's file is internal: it alone goes through ``guard``.
+    """
+    for module in (greenspec, speedbench):
+        if (chosen := module.settings(env)) is not None:
+            return module, chosen
+    return None
+
+
 def with_requests(scenario_yaml: str, requests: int) -> str:
     """The scenario with its profiling phase set to ``requests`` requests (the size of the selection)."""
     import yaml  # in the AgentX venv
@@ -222,15 +235,16 @@ def prepare(env: Mapping[str, str], runtime: Runtime) -> int:
 
         try:
             path = fetch(env["SERVING_DATASET"])
-            if chosen := greenspec.settings(env):
+            if found := selection_family(env):
+                module, chosen = found
                 selection = Path(SELECTION_DIR) / "selection.jsonl"
-                count = greenspec.write_selection(path, chosen, served_model(env), selection)
+                count = module.write_selection(path, chosen, served_model(env), selection)
                 print(
-                    f"GreenSpec selection {greenspec.label(chosen)}: {count} requests", flush=True
+                    f"{module.TAG} selection {module.label(chosen)}: {count} requests", flush=True
                 )
                 path = selection
             (result_dir / DATASET_PATH).write_text(str(path))
-        except (DatasetError, greenspec.GreenSpecError) as error:
+        except (DatasetError, greenspec.GreenSpecError, speedbench.SpeedBenchError) as error:
             print(f"ERROR {error}", file=sys.stderr)
             return 1
     if url := env.get("SERVING_REMOTE_URL"):
@@ -265,7 +279,9 @@ def run(env: Mapping[str, str], runtime: Runtime) -> int:
     scenario = result_dir / "scenario.yaml"
     token_marker = result_dir / COUNT_SOURCE_MARKER
     source = token_marker.read_text().strip() if token_marker.exists() else None
-    chosen = greenspec.settings(env)
+    found = selection_family(env)
+    chosen = found[1] if found else None
+    internal = found is not None and found[0] is greenspec
     scenario_yaml = with_token_source(env["SERVING_SCENARIO_YAML"], source or "server")
     child = online(env) | {
         "AIPERF_DATASET_CONFIGURATION_TIMEOUT": "1800",
@@ -290,7 +306,7 @@ def run(env: Mapping[str, str], runtime: Runtime) -> int:
     metrics_urls = [url for url in env.get("AIPERF_SERVER_METRICS_URLS", "").split(",") if url]
     if chosen:
         specdec.save(result_dir / "spec_metrics_before.json", specdec.snapshot(metrics_urls))
-    replay = greenspec.tee_filtered if chosen else proc.tee
+    replay = greenspec.tee_filtered if internal else proc.tee
     replay_rc = replay(argv, result_dir / "benchmark.log", child)
     if chosen:
         specdec.save(result_dir / "spec_metrics_after.json", specdec.snapshot(metrics_urls))
@@ -334,7 +350,8 @@ def run(env: Mapping[str, str], runtime: Runtime) -> int:
             )
         else:
             specdec.merge(aggregate, spec)
-        guard_rc = guard(env, result_dir, aggregate, output_dir)
+        if internal:
+            guard_rc = guard(env, result_dir, aggregate, output_dir)
     return replay_rc or aggregate_rc or validate_rc or guard_rc
 
 

inferencex-e2e/infx/serving/greenspec.py

@@ -31,6 +31,7 @@
 from typing import Any
 
 FAMILY = "greenspec"
+TAG = "GreenSpec"  # how the family names itself in the job log
 META_KEYS = ("question_id", "category", "bucket", "lite", "isl", "osl", "thinking_src", "recorded")
 MAX_TOKEN_KEYS = ("max_tokens", "max_completion_tokens")
 SUBSETS = ("diverse", "agentic_coding", "agentic_tools", "llm_task")

inferencex-e2e/infx/serving/scenarios.py

@@ -13,7 +13,7 @@
 
 import yaml
 
-from infx.serving import greenspec
+from infx.serving import greenspec, speedbench
 from infx.serving.datasets import LockEntry, load_lock
 from infx.serving.slo import SLOError, parse_slo
 
@@ -26,6 +26,7 @@
     "hidden-prefix-e3",
     "synthetic",
     "greenspec",
+    "speedbench",
 )
 OVERRIDE_KEYS = (
     "duration",
@@ -239,6 +240,31 @@ def _greenspec(
     return Rendered(doc, greenspec.label(chosen), "file", name, False, 0.0)
 
 
+def _speedbench(
+    overrides: dict[str, Any],
+    point: Point | None,
+    search: bool,
+    slo: str | None,
+    lock: dict[str, LockEntry],
+) -> Rendered:
+    """The speedbench scenario with its selection; the run length is the selection, so there is no duration."""
+    if point is not None or search or slo:
+        raise ScenarioError("workload/slo: only for scenario synthetic, not speedbench")
+    try:
+        chosen = speedbench.normalize(overrides)
+    except speedbench.SpeedBenchError as error:
+        raise ScenarioError(str(error)) from None
+    if chosen.dataset not in lock:
+        raise ScenarioError(
+            f"scenario: dataset {chosen.dataset} of speedbench is not in the dataset lock"
+        )
+    doc = load_base("speedbench")
+    doc["benchmark"]["datasets"][0]["name"] = chosen.dataset
+    if chosen.requests:
+        _profiling(doc)["requests"] = chosen.requests
+    return Rendered(doc, speedbench.label(chosen), "file", chosen.dataset, False, 0.0)
+
+
 def render(
     family: str,
     overrides: dict[str, Any] | None = None,
@@ -252,6 +278,8 @@ def render(
     lock = load_lock() if lock is None else lock
     if family == "greenspec":
         return _greenspec(overrides, point, search, slo, lock)
+    if family == "speedbench":
+        return _speedbench(overrides, point, search, slo, lock)
     if family == "synthetic":
         if point is None:
             raise ScenarioError("workload: synthetic needs isl, osl and cache")

inferencex-e2e/infx/serving/schema.py

@@ -229,6 +229,7 @@ class ServingSpec(BaseModel):
         "hidden-prefix-e3",
         "synthetic",
         "greenspec",
+        "speedbench",
     ]
     overrides: dict[str, Any] = Field(default_factory=dict)
     workload: Workload | None = None