[DinhKhangHoang]

What

This adds the Gemma 4 LiveKit prod setup (customer 111868) as a serving YAML. One 8x H200 node runs two TP4 dynamo.vllm workers behind the Dynamo frontend KV router.

The refactored flow could not run that setup yet. schema.py rejected replicas > 1 with dynamo.frontend, and job.sh / bench.sh started one Dynamo engine with one system port. This is the same gap the gemma4-livekit-tp4-dyn-sgl-…-car128 config works around by running one engine at half the concurrency. #38 had this change for the old tree; this ports it to the new one.

Launcher (first commit)

• render.py: python3 -m dynamo.vllm counts as a Dynamo engine, with framework vllm.
• schema.py: replicas > 1 behind the Dynamo frontend is allowed for dynamo.vllm, as long as serve has no --kv-events-config of its own. dynamo.sglang stays at one replica.
• job.sh: picks a system port and a KV event port for each replica.
• bench.sh: each replica gets its GPU slice, its own DYN_SYSTEM_PORT and a ZMQ --kv-events-config. Without that, the second vLLM publisher can't bind. aiperf scrapes every system port.
  • With several replicas, bench.sh waits for each one's /health. The frontend lists the model as soon as the first engine registers, so the old check alone would start the client with one worker up.
  • A single dynamo.sglang engine starts exactly as before.

Config (second commit): configs/serving/gemma4-livekit-2tp4-dyn-vllm-prod.yaml

• Image vcr.vngcloud.vn/108942-public69/dynamo-stack/vllm-runtime:0.31.0-dyn1.6.0-g4p2-prod, public pull, used for both engine and router. It is vLLM 0.31.0 + ai-dynamo 1.6.0 plus:
  • [Bugfix] Retain one alignment block below sliding-window replay boundaries so a same-turn extension still hits the prefix cache vllm-project/vllm#57613: keeps one more alignment block below each sliding-window replay boundary, so Gemma 4's SWA groups lose fewer prefix hits.
  • fix(llm): keep Gemma 4 reasoning parsing on with the Gemma 4 tool parser ai-dynamo/dynamo#17486 backported: reasoning no longer leaks into content when thinking is off.
  • DYN_PARSER_VERSION=2 (unified parser route), also set in router.env so the route doesn't depend on the image. The legacy route leaks reasoning into content after a tool response and returns raw tool-call markup with tool_choice: none.
• Router: --router-mode=kv --router-prefill-load-scale=16, hard session affinity (30 min), --router-min-initial-workers=2. aiperf sends its per-conversation id as x-dynamo-session-id.
• Engine: --max-num-batched-tokens 8192 --long-prefill-token-threshold 2048 --prefix-cache-retention-interval 4096.
• scenario: livekit (v1.1-warm), conc: [96, 150]. c150 is the per-node prod load (300 CCU on 2 nodes).

Validation

• to_matrix --check: gemma4-livekit-2tp4-dyn-vllm-prod custom-agentic-livekit111868-v1.1-warm (1200s) c96,150 cluster:hcm-hgx02.

• gn-benchmark run 41402074324 of this YAML on this branch (hcm-hgx02, 1200 s per point, aiperf warm-up phase excluded):

  	c96	c150
  requests / errors	17 078 / 0	25 282 / 0
  req/s	14.2	21.0
  TTFT p50 / p90 / p95 / p99 (s)	0.14 / 0.32 / 0.51 / 0.83	0.19 / 0.48 / 0.73 / 1.33
  prompt tokens read from cache	92.3%	90.8%
  requests per worker	8 600 / 8 466	12 712 / 12 626

  Both replicas came up with their own KV event port, the frontend picked the v2 parser route (parser_version=V2), and the server and router logs have no tracebacks.

• Tool-call and API edge smoke on the -g4p2 image (our Slurm job 8218, 340 requests, Dynamo vs vllm serve): row for row the same result as the -g4p1 smoke the prod route was validated on (job 7594). vllm#57613 changes nothing on the API side.

• c96 is in line with what we had before on the V1 warm trace (p95 0.53 on the -g4strip image in CI run 39393810302, 0.55 on -g4p1 in our Slurm job 7604). It is not a strict before/after, since v1.1-warm drops the short rooms.

• c150, the per-node prod load, is at p95 0.73 s, above the 0.5 s target. This PR is the config as it runs in prod; tuning for c150 is a separate step.

• New tests:

  • test_bench.py runs bench.sh with two fake dynamo.vllm replicas: GPU slices, KV event endpoints and metrics URLs.
  • test_driver.py checks that job.sh gives each replica its own ports and that the frontend inherits none of them.
  • Both fail on the old scripts.

中文

内容

新增 Gemma 4 LiveKit 生产配置（客户 111868）：单台 8x H200 上两个 TP4 dynamo.vllm worker，前面是 Dynamo frontend KV router。

重构后的流程此前无法运行该配置：schema.py 拒绝 dynamo.frontend 搭配 replicas > 1，job.sh / bench.sh 只启动一个 Dynamo 引擎、一个 system 端口。gemma4-livekit-tp4-dyn-sgl-…-car128 配置正是用"单引擎、半并发"绕过这个限制。#38 在旧目录结构上做过同样的修改，本 PR 将其移植到新结构。

启动器（第一个 commit）

• render.py：python3 -m dynamo.vllm 视为 Dynamo 引擎，framework 为 vllm。
• schema.py：dynamo.vllm 可以在 Dynamo frontend 后设置 replicas > 1，前提是 serve 中没有自带 --kv-events-config；dynamo.sglang 仍限一个副本。
• job.sh：为每个副本选取 system 端口和 KV event 端口。
• bench.sh：每个副本有自己的 GPU 分片、DYN_SYSTEM_PORT 和 ZMQ --kv-events-config，否则第二个 vLLM publisher 无法绑定端口。aiperf 抓取所有 system 端口的指标。多副本时会等待每个副本的 /health，因为 frontend 在第一个引擎注册后就会列出模型。单个 dynamo.sglang 引擎的启动方式不变。

配置（第二个 commit）：configs/serving/gemma4-livekit-2tp4-dyn-vllm-prod.yaml

• 镜像 vcr.vngcloud.vn/108942-public69/dynamo-stack/vllm-runtime:0.31.0-dyn1.6.0-g4p2-prod（可匿名拉取），引擎与 router 共用。在 vLLM 0.31.0 + ai-dynamo 1.6.0 基础上加入：
  • [Bugfix] Retain one alignment block below sliding-window replay boundaries so a same-turn extension still hits the prefix cache vllm-project/vllm#57613：在每个滑动窗口重放边界下多保留一个对齐块，减少 Gemma 4 SWA 组的前缀命中损失。
  • fix(llm): keep Gemma 4 reasoning parsing on with the Gemma 4 tool parser ai-dynamo/dynamo#17486 的回移：关闭 thinking 时，推理内容不再泄漏到 content。
  • DYN_PARSER_VERSION=2（统一解析路径，同时在 router.env 中显式设置，不依赖镜像）：旧路径会在工具返回后把推理内容放进 content，并在 tool_choice: none 时输出原始工具调用标记。
• Router：--router-mode=kv --router-prefill-load-scale=16，硬会话亲和（30 分钟），--router-min-initial-workers=2。aiperf 以会话 id 作为 x-dynamo-session-id 发送。
• 引擎：--max-num-batched-tokens 8192 --long-prefill-token-threshold 2048 --prefix-cache-retention-interval 4096。
• scenario: livekit（v1.1-warm），conc: [96, 150]。c150 为单节点生产负载（2 个节点承载 300 CCU）。

验证

• to_matrix --check：gemma4-livekit-2tp4-dyn-vllm-prod custom-agentic-livekit111868-v1.1-warm (1200s) c96,150 cluster:hcm-hgx02。

• 本分支上该 YAML 的 gn-benchmark 运行 41402074324（hcm-hgx02，每个点 1200 s，不含 aiperf warm-up 阶段）：

  	c96	c150
  请求数 / 错误	17 078 / 0	25 282 / 0
  req/s	14.2	21.0
  TTFT p50 / p90 / p95 / p99 (s)	0.14 / 0.32 / 0.51 / 0.83	0.19 / 0.48 / 0.73 / 1.33
  从缓存读取的 prompt token	92.3%	90.8%
  每个 worker 的请求数	8 600 / 8 466	12 712 / 12 626

  两个副本各自使用独立的 KV event 端口，frontend 选择了 v2 解析路径（parser_version=V2），server 和 router 日志中没有 traceback。

• -g4p2 镜像的工具调用与 API 边界 smoke（我们的 Slurm job 8218，340 个请求，Dynamo 对比 vllm serve）：与验证生产路径时 -g4p1 的 smoke（job 7594）逐行结果一致，vllm#57613 对 API 行为没有影响。

• c96 与此前 V1 warm trace 的结果一致（CI run 39393810302 上 -g4strip 镜像 p95 0.53，我们 Slurm job 7604 上 -g4p1 为 0.55）。由于 v1.1-warm 去掉了较短的 room，这不是严格的前后对比。

• c150（单节点生产负载）p95 为 0.73 s，高于 0.5 s 目标。本 PR 是生产环境实际运行的配置，针对 c150 的调优另行处理。

• 新增测试：

  • test_bench.py 用两个模拟的 dynamo.vllm 副本运行 bench.sh，检查 GPU 分片、KV event 端点和指标 URL。
  • test_driver.py 检查 job.sh 为每个副本分配独立端口，且 frontend 不继承这些端口。
  • 两个测试在旧脚本上都会失败。

AI model disclosure

Prepared with Claude Opus 5.5 (claude-opus-5-5) in Claude Code: code, tests, config and this description. Reviewed by Khang Hoang.

🤖 Generated with Claude Code

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[github-actions]

inferencex-e2e/infx diff: +237 -12

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。