feat(serving): greenspec scenario, spec-decode acceptance, arms, run summary

.agents/skills/serving-bench/SKILL.md

@@ -30,7 +30,10 @@ router:                             # optional: any router in front of the engin
 storage:                            # optional: a non-GPU sidecar started first (e.g. Mooncake master)
   image: <image>
   serve: mooncake_master --port={{storage_master_port}} --http_metadata_server_port={{storage_meta_port}}
-scenario: agentx                    # agentx | tau2 | livekit | shared-prefix-e2 | hidden-prefix-e3 | synthetic
+arms:                               # optional: engine variants in one run, e.g. base vs speculative (see ARCHITECTURE.md, Arms)
+  base: {}
+  mtp6: {append: "--speculative-config '...'"}
+scenario: agentx                    # agentx | tau2 | livekit | shared-prefix-e2 | hidden-prefix-e3 | synthetic | greenspec
 overrides: {duration: 180}          # optional variant, see the table below
 conc: [4, 8]                        # one benchmark job per value
 ```
@@ -46,6 +49,7 @@ Working examples, one per pattern: `inferencex-e2e/configs/serving/*.yaml`. They
 | `livekit` | LiveKit voice-agent traces, V1.1 warm (dial warm-ups count in TTFT) | 1200 s |
 | `shared-prefix-e2` / `hidden-prefix-e3` | prefix-cache A/B traces (HiCache L2 vs Mooncake L3) | 1800 s |
 | `synthetic` | synthetic ISL/OSL with a shared-prefix cache hit, needs `workload:` | 300 s per point |
+| `greenspec` | speculative-decoding bench from production chat traffic (private; result has `spec_decode.acceptance_length`) | the whole selection, no duration |
 
 Variants (`overrides:`, only these keys):
 
@@ -57,6 +61,7 @@ Variants (`overrides:`, only these keys):
 | sampling instead of greedy | `overrides: {generation: {top_p: 0.95, temperature: 1.0}}` |
 | LiveKit V1 / V2 trace | `overrides: {dataset: livekit-111868-v1}` (or `-v2`) |
 | a different trace idle-gap cap | `overrides: {trace_idle_gap_cap: 1}` |
+| GreenSpec subset, lite, bucket, cap | `scenario: greenspec` + `overrides: {subset: agentic_tools, lite: true, max_tokens: 4096, requests: 200}` (only these keys; no `duration`) |
 
 Synthetic grid, one job per point, all in one run:
 

.agents/skills/serving-runner-setup/SKILL.md

@@ -73,7 +73,7 @@ for d in $BASE/gharunner0*/actions-runner; do (cd "$d" && sudo ./svc.sh install
 
 The registration token comes from the settings page or `gh api -X POST repos/vngcloud/InferenceX/actions/runners/registration-token --jq .token` (valid exactly 1 hour, not configurable; mint it right before running `setup.sh`). `setup.sh` prints `All runners configured!` even when registration failed: check that every slot printed its own `[<name>] Done.` line.
 
-No runner `.env` is needed for the datasets: every repo in `datasets/lock.yaml` is public on HF (checked 2026-10-25), and `infx/serving/datasets.py` downloads anonymously when `GREENLAB_HF_TOKEN` is unset. Add `GREENLAB_HF_TOKEN=…` to `actions-runner/.env` only if a private dataset is added to the lock.
+Every `datasets/lock.yaml` entry marked `private: true` (GreenSpec) needs `GREENLAB_HF_TOKEN`; the other repos are public. `infx/serving/datasets.py` downloads anonymously when `GREENLAB_HF_TOKEN` is unset, so a runner that is to run `scenario: greenspec` needs `GREENLAB_HF_TOKEN=…` in `actions-runner/.env`. Other scenarios need no `.env`.
 
 ## 3. Add the cluster record
 

.github/workflows/benchmark-tmpl.yml

@@ -6,6 +6,8 @@ on:
         required: true
       SERVING_API_KEY:
         required: false
+      GREENLAB_HF_TOKEN:
+        required: false
     inputs:
       serving:
         description: "VNG serving point (SERVING_JSON from infx/serving/to_matrix.py); empty for every other caller"
@@ -100,6 +102,7 @@ env:
   PYTHONPATH: ${{ github.workspace }}/inferencex-e2e
   SERVING_JSON: ${{ inputs.serving }}
   SERVING_API_KEY: ${{ secrets.SERVING_API_KEY }}
+  GREENLAB_HF_TOKEN: ${{ secrets.GREENLAB_HF_TOKEN }}
   INFERENCEX_E2E_ROOT: ${{ github.workspace }}
   PORT: '8888'
   INFMAX_CONTAINER_WORKSPACE: '/workspace'

.github/workflows/gn-benchmark.yml

@@ -2,7 +2,7 @@
 run-name: ${{ inputs.test-name }}
 # VNG serving-YAML benchmarks (inferencex-e2e/docs/serving/ARCHITECTURE.md).
 # Dispatch with inferencex-e2e/utils/serving/dispatch.sh, which validates first.
 on:
     workflow_dispatch:
         inputs:
             serving:
@@ -18,7 +18,7 @@
     contents: read
 
 jobs:
     plan:
         runs-on: ubuntu-latest
         outputs:
             points: ${{ steps.plan.outputs.points }}
@@ -48,7 +48,8 @@
                 include: ${{ fromJSON(needs.plan.outputs.points) }}
         secrets:
             INFERENCEX_OFFICIAL_RO_HF_TOKEN: ${{ secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN }}
             SERVING_API_KEY: ${{ matrix.api_key_secret && secrets[matrix.api_key_secret] || '' }}
+            GREENLAB_HF_TOKEN: ${{ secrets.GREENLAB_HF_TOKEN }}
         with:
             config: ${{ toJSON(matrix.config) }}
             serving: ${{ matrix.serving }}
@@ -60,3 +61,24 @@
             scenario-type: agentic-coding
             duration: ${{ matrix.config.duration }}
             test-name: ${{ inputs.test-name }}
+
+    summary:
+        name: summary
+        # One table of every point on the run page: throughput, latency, and for speculative decoding
+        # the acceptance length and the speedup over a base arm of the same scenario and concurrency.
+        needs: bench
+        if: ${{ !cancelled() }}
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+              with:
+                  persist-credentials: false
+            - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
+            - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+              with:
+                  path: results
+                  pattern: bmk_agentic_*
+            - working-directory: inferencex-e2e
+              env:
+                  PYTHONPATH: ${{ github.workspace }}/inferencex-e2e
+              run: uv run --no-project --python 3.12 python -P -m infx.serving.summary "$GITHUB_WORKSPACE/results" >> "$GITHUB_STEP_SUMMARY"

inferencex-e2e/benchmarks/single_node/serving/datasets/lock.yaml

@@ -2,6 +2,7 @@
 # Each entry pins one file at one HF commit; infx/serving/datasets.py downloads it
 # (token: GREENLAB_HF_TOKEN if set) and verifies sha256. `entries` is the dataset's
 # session count, copied into the scenario when a serving YAML picks this dataset.
+# `private: true` marks a repo that needs GREENLAB_HF_TOKEN to read (the runner-setup skill keys off it).
 tau2-cs:
   repo: Green-Lab/customer-service-agent-traces
   revision: 2c793380d8ba45bfde79f9a5c7b5991ed40ab2ea
@@ -42,3 +43,12 @@ livekit-111868-v1.1-warm:
   file: trace.v1.1-append-ideal-warm.jsonl
   sha256: e26542392a202764849c05018f2f8ea3f9574277ffa3400615f0b4df23b0eb2f
   entries: 671
+# GreenSpec v1.1.1: one flat file with every subset (infx.serving.greenspec selects rows from it).
+# Private repo: GREENLAB_HF_TOKEN must be able to read it. The benchmark repo is public, so the
+# greenspec scenario never uploads anything derived from the rows (see infx/serving/greenspec.py).
+greenspec-v1.1.1:
+  repo: Green-Lab/GreenSpec
+  revision: f7ea173f12917bcf1d3b9926a281a167d2b8243d
+  file: greenspec.jsonl
+  sha256: 6a26785d07c235446e57ba34463a1ae9e2dcff938fec6246881492e511fb0f43
+  private: true

inferencex-e2e/benchmarks/single_node/serving/scenarios/greenspec.yaml

@@ -0,0 +1,29 @@
+# GreenSpec family: the internal speculative-decoding bench cut from production chat traffic
+# (Green-Lab/GreenSpec, one flat greenspec.jsonl per release; datasets/lock.yaml pins the release).
+# infx.serving.greenspec selects the rows (overrides: subset, lite, bucket), builds one request body per
+# row and writes them as the raw_payload file IX_CUSTOM_DATASET_FILE points to; AgentX sends each body
+# as is. Every request streams and completes on its own (no ignore_eos): acceptance only means
+# something on natural output. `requests` is the selection size unless overrides.requests lowers it;
+# the client sets it once it has counted the selection, so the 1 below is only a placeholder.
+schemaVersion: "2.0"
+benchmark:
+  unsafeOverride: true
+  endpoint:
+    type: chat
+    path: /v1/chat/completions
+    streaming: true
+    useServerTokenCount: true
+  datasets:
+  - name: greenspec-v1.1.1
+    type: file
+    format: raw_payload
+    path: ${IX_CUSTOM_DATASET_FILE}
+    sampling: sequential
+    randomSeed: 42
+  phases:
+  - name: profiling
+    kind: profiling
+    type: concurrency
+    requests: 1
+    failedRequestThreshold: 0.1
+randomSeed: 42

inferencex-e2e/configs/serving/gemma4-greenspec-diverse.yaml

@@ -0,0 +1,40 @@
+# Pattern: speculative decoding against its base, in one run (arms:). GreenSpec, whole diverse subset,
+# Gemma-4-31B FP8-block, vLLM, 1 GPU, CCU 16, MTP draft depth 6. serve: is shared by every arm; an arm adds
+# flags (append) and env. Each arm is its own job and its own engine; the run's summary table lines them up
+# and gives the speedup of each speculative arm over the arm without one.
+# Same engine flags as the earlier SPEED-Bench cells (prefix caching off, 256k context). The run is one
+# request per row of the selection (no duration): the results carry spec_decode.acceptance_length.
+name: gemma4-greenspec-diverse
+runner: cluster:hcm-hgx02
+image: vllm/vllm-openai:v0.30.0
+model: RedHatAI/gemma-4-31B-it-FP8-block
+gpus: 1
+precision: fp8block
+env:
+  NCCL_P2P_LEVEL: NVL
+  VLLM_DISABLE_COMPILE_CACHE: "1"
+  VLLM_ATTENTION_BACKEND: FLASHINFER
+serve: |
+  vllm serve RedHatAI/gemma-4-31B-it-FP8-block
+  --served-model-name RedHatAI/gemma-4-31B-it-FP8-block
+  --trust-remote-code
+  --tensor-parallel-size 1
+  --gpu-memory-utilization 0.92
+  --max-model-len 262144
+  --max-num-seqs 64
+  --max-num-batched-tokens 16384
+  --enable-chunked-prefill
+  --long-prefill-token-threshold 8192
+  --no-enable-prefix-caching
+  --enable-auto-tool-choice
+  --tool-call-parser gemma4
+  --reasoning-parser gemma4
+arms:
+  base: {}
+  mtp6:
+    append: |
+      --speculative-config '{"method": "mtp", "model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 6}'
+scenario: greenspec
+overrides:
+  subset: diverse
+conc: [16]

inferencex-e2e/docs/serving/ARCHITECTURE.md

@@ -214,7 +214,7 @@ that several nodes need belong in a registry, where every node caches its own sq
 
 ## 6. Scenarios
 
-Six base files in `benchmarks/single_node/serving/scenarios/`:
+Seven base files in `benchmarks/single_node/serving/scenarios/`:
 
 | Family | Workload | Dataset |
 |---|---|---|
@@ -224,6 +224,7 @@ Six base files in `benchmarks/single_node/serving/scenarios/`:
 | `shared-prefix-e2` | requests sharing long prefixes | pinned file, Mooncake trace format |
 | `hidden-prefix-e3` | prefixes reused without being visible to the router | pinned file, Mooncake trace format |
 | `synthetic` | generated ISL/OSL with a shared-prefix share | synthetic |
+| `greenspec` | speculative-decoding bench cut from production chat traffic, one request per row | pinned private file (`Green-Lab/GreenSpec`), `raw_payload` format |
 
 Pinned files are listed in `benchmarks/single_node/serving/datasets/lock.yaml` with their URL and
 sha256; the client downloads and checks them before the run.
@@ -233,6 +234,43 @@ A serving YAML picks a family and may set `overrides`. The keys allowed are `dur
 `generation` and `trace_idle_gap_cap`. Any other key fails validation, so cache-bust, warmup and seed
 cannot change without a code review.
 
+`greenspec` takes its own keys instead: `subset` (`diverse`, `agentic_coding`, `agentic_tools`, `llm_task`),
+`lite`, `bucket`, `mode` (`target-default` or `recorded` sampling), `max_tokens` (output cap, default 8192)
+and `requests` (default: the whole selection). It has no duration; the run ends when the selection is sent.
+Three things differ from the other families:
+
+- The dataset is private and the benchmark repo is public, so the client (`infx/serving/greenspec.py`)
+  prints AgentX's console without error text, and before the upload keeps only an allowlist of files in
+  `results/`, strips error messages from the aiperf export, replaces server logs by a stub (the originals
+  stay on the node in `.greenspec-private-logs/`) and fails the job if a `question_id` is still in an
+  upload. The request bodies are written outside `results/` and deleted at the end.
+- The client snapshots the engine's `vllm:spec_decode_*` counters (`sglang:spec_accept_length` on SGLang)
+  before and after the replay and files `spec_decode` (`acceptance_length`, `acceptance_rate`, counts) in
+  the aggregate result (`infx/serving/specdec.py`). A point without speculative decoding has no such block.
+- The runner needs `GREENLAB_HF_TOKEN` in `actions-runner/.env`: the release is not public.
+
+A base arm and a speculative arm are the `arms:` of one serving YAML (below). The `summary` job of
+`gn-benchmark.yml` prints one table of all points of a run on the run page, with the speedup of each
+speculative point over the base point of the same scenario and concurrency.
+
+### Arms
+
+`arms:` runs engine variants of one serving YAML in one run: `serve:` and `env:` are shared, an arm adds
+flags with `append` and overlays `env`. Each arm is its own job on its own engine (the name of the point
+is `<name>-<arm>_c<conc>`), so arms multiply with `conc` and with a workload grid.
+
+```yaml
+serve: vllm serve <model> --tensor-parallel-size 1 ...
+arms:
+  base: {}
+  mtp6:
+    append: |
+      --speculative-config '{"method": "mtp", "model": "<draft>", "num_speculative_tokens": 6}'
+```
+
+Two arms that end up with the same command and env are rejected, and `arms:` does not combine with
+`endpoint:`. The `spec-decoding` field of each point follows its own command (`mtp`, `draft_model` or `none`).
+
 The synthetic family takes `workload:` instead:
 
 ```yaml

inferencex-e2e/infx/bench/agentic/replay.py

@@ -169,8 +169,10 @@ def replay_argv(cfg: ReplayConfig, cli: str | Path) -> list[str]:
     argv += ["--slice-duration", SLICE_DURATION_S]
     argv += ["--no-gpu-telemetry"]
     argv += [
-        "--trajectory-start-min-ratio", TRAJECTORY_START_MIN_RATIO,
-        "--trajectory-start-max-ratio", TRAJECTORY_START_MAX_RATIO,
+        "--trajectory-start-min-ratio",
+        TRAJECTORY_START_MIN_RATIO,
+        "--trajectory-start-max-ratio",
+        TRAJECTORY_START_MAX_RATIO,
     ]
     argv += [
         "--failed-request-threshold",