Repository navigation
Migrate vng-benchmark to the refactored tree (tree-swap merge) - #46
Merged
Merged
Conversation
… bounded replay and c160, TP4 c1 single admission / [AgentX] GB300 DSV4.1 Flash SGLang：升级 20261014 nightly、启用 fastokens、DEP2 开 bounded replay 并新增 c160、TP4 c1 单请求准入 (SemiAnalysisAI#4033) * Bump GB300 DSV4.1 Flash SGLang to the 20261013 nightly with fastokens The 20261013 nightly ships fastokens 0.3.2 (sgl-project/sglang#43104). The chat endpoint re-encodes the whole 300K-600K-token conversation each turn; HF tokenizers spends about 0.6 s of TTFT on it on Grace, fastokens about 0.03 s. Local one-hour runs, 20261012 to 20261013 + fastokens: TP4 c1 TTFT p90 1.38 to 0.28 s, 7,314 to 7,850 tok/s/GPU; TP2 c32 183.1 to 190.9 P90 tok/s/user; DEP2 c128 81.6 to 86.6 P90 at the same throughput. * Add a DEP2 c160 point to GB300 DSV4.1 Flash SGLang AgentX Same settings as DEP2 c128. A local one-hour run on the 20261012 image without fastokens gave 301,770 tok/s/GPU at 82.4 tok/s/user P90, against 278,104 at 84.2 for c128 in the same setup. * TP4 c1: admit one request and run decoder SWA bounded replay A subagent request that arrives during a decode now waits instead of stalling that decode with its prefill. Local one-hour runs on the 20261013 image with fastokens: 625.1 to 654.4 tok/s/user P90 at 7,848 tok/s/GPU, TTFT p50/p90 0.24/0.28 to 0.23/0.27 s; p99 0.97 to 1.13 s. * Move to the 20261014 nightly and run bounded replay at DEP2 The 20261014 image allows decoder SWA bounded replay with DP attention (sgl-project/sglang#43812). Local one-hour DEP2 c128 runs with fastokens: 271,676 tok/s/GPU at 86.6 P90 (20261013) to 280,752 at 88.5 (20261014) to 283,868 at 91.3 with bounded replay; TTFT p90 3.43 to 2.62 s. * Add DEP2 c160 with the HuggingFace tokenizer At c160, fastokens admits short subagent prefills faster and shortest-prefill-first then holds back long main-agent turns. Local one-hour c160 runs: 258,332 tok/s/GPU at 85.8 P90 with fastokens (20261013) against 296,603 at 79.2 (20261012) and 287,471 at 78.2 (20261013) with the HuggingFace tokenizer. * Run DEP2 c160 with a prefill-decode interval of 8 * Correct the fastokens version in the changelog entry [skip-sweep] The 20261014 image ships fastokens 0.3.4; sgl-project/sglang#43104 only sets the fastokens>=0.3.2 floor. Wording only: the recipe, master config and image are unchanged, so the pinned sweep still applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the recipe comment and simplify the changelog entry [skip-sweep] --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Po-Han Huang <pohanh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cam Quilici <cameron@semianalysis.com>
… bounded replay and c160, TP4 c1 single admission / [AgentX] GB300 DSV4.1 Flash SGLang：升级 20261014 nightly、启用 fastokens、DEP2 开 bounded replay 并新增 c160、TP4 c1 单请求准入 (SemiAnalysisAI#4033) * Bump GB300 DSV4.1 Flash SGLang to the 20261013 nightly with fastokens The 20261013 nightly ships fastokens 0.3.2 (sgl-project/sglang#43104). The chat endpoint re-encodes the whole 300K-600K-token conversation each turn; HF tokenizers spends about 0.6 s of TTFT on it on Grace, fastokens about 0.03 s. Local one-hour runs, 20261012 to 20261013 + fastokens: TP4 c1 TTFT p90 1.38 to 0.28 s, 7,314 to 7,850 tok/s/GPU; TP2 c32 183.1 to 190.9 P90 tok/s/user; DEP2 c128 81.6 to 86.6 P90 at the same throughput. * Add a DEP2 c160 point to GB300 DSV4.1 Flash SGLang AgentX Same settings as DEP2 c128. A local one-hour run on the 20261012 image without fastokens gave 301,770 tok/s/GPU at 82.4 tok/s/user P90, against 278,104 at 84.2 for c128 in the same setup. * TP4 c1: admit one request and run decoder SWA bounded replay A subagent request that arrives during a decode now waits instead of stalling that decode with its prefill. Local one-hour runs on the 20261013 image with fastokens: 625.1 to 654.4 tok/s/user P90 at 7,848 tok/s/GPU, TTFT p50/p90 0.24/0.28 to 0.23/0.27 s; p99 0.97 to 1.13 s. * Move to the 20261014 nightly and run bounded replay at DEP2 The 20261014 image allows decoder SWA bounded replay with DP attention (sgl-project/sglang#43812). Local one-hour DEP2 c128 runs with fastokens: 271,676 tok/s/GPU at 86.6 P90 (20261013) to 280,752 at 88.5 (20261014) to 283,868 at 91.3 with bounded replay; TTFT p90 3.43 to 2.62 s. * Add DEP2 c160 with the HuggingFace tokenizer At c160, fastokens admits short subagent prefills faster and shortest-prefill-first then holds back long main-agent turns. Local one-hour c160 runs: 258,332 tok/s/GPU at 85.8 P90 with fastokens (20261013) against 296,603 at 79.2 (20261012) and 287,471 at 78.2 (20261013) with the HuggingFace tokenizer. * Run DEP2 c160 with a prefill-decode interval of 8 * Correct the fastokens version in the changelog entry [skip-sweep] The 20261014 image ships fastokens 0.3.4; sgl-project/sglang#43104 only sets the fastokens>=0.3.2 floor. Wording only: the recipe, master config and image are unchanged, so the pinned sweep still applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the recipe comment and simplify the changelog entry [skip-sweep] --------- Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com> Co-authored-by: Po-Han Huang <pohanh@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Cam Quilici <cameron@semianalysis.com>
…enchmark 结果记录 run attempt (SemiAnalysisAI#4207) * [Results] record the run attempt on retried benchmark results / 重跑的 benchmark 结果记录 run attempt * [Results] drop aggregate dedupe; ingest already prefers per-job copies / 移除聚合去重 * [Results] read only the latest attempt of rerun AgentX points / 只读取重跑 AgentX 测试点的最新尝试
删除 matrix 包中的注释和文档字符串，并用显式检查替换带 lint 抑制注释的断言。
…8 / [Klaud Cold] Muse Spark Open MI355X：并发大于 8 时启用 HiCache DRAM 卸载 (SemiAnalysisAI#4201) * musespark mi355x: HiCache DRAM offload above conc 8 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf-changelog: link SemiAnalysisAI#4201 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Restore srt-slurm submodule pin to main (ae3d687) 8250395 accidentally staged a stale local submodule checkout (545233a). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…k Open H200 全部并发启用 HiCache 卸载 (SemiAnalysisAI#4199) * feat: offload every H200 Muse Spark point to HiCache * chore: link H200 HiCache entry to PR 4199 * fix: shrink H200 HiCache to 120 GB per rank to fit host RAM --------- Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
…GPU 为解码服务器分配 DRAM 预算 (SemiAnalysisAI#4211) * [AgentX] budget decode servers' DRAM from decode GPUs / [AgentX] 按解码 GPU 为解码服务器分配 DRAM 预算 * Export DRAM budgets for every scenario / 所有场景统一导出内存预算 * Address review: omit zero decode budget, reject per-worker hardware, name role in tiling errors / 处理审查意见 * Format generate.py / 格式化 generate.py
… / fix(klaud): 持有者收尾作业结束后释放恢复租约 (SemiAnalysisAI#4189) A recovery lease counted as held until the holder's whole planner run finished, which includes up to 6h of candidate sessions after the reconcile job. Waves start every 2h, so an earlier wave was almost always still running and later waves skipped those sessions, keeping 28 stale family claims alive. A lease is now free once the holder's reconcile job has completed. 此前恢复租约要等持有者的整个规划运行结束才释放，而规划运行在收尾作业之后还包含 最长 6 小时的候选会话。规划每 2 小时启动一次，前一轮几乎总是仍在运行，后续轮次 便跳过这些会话，导致 28 个陈旧的配置族声明一直存在。现在持有者的收尾作业完成后， 租约即可被接管。
…laud): 权重缺失后暂停该硬件上的模型 (SemiAnalysisAI#4193) DSR1 FP4 and FP8 weights are missing on b200-nscale and gb200-nv, and about 25 Klaud PRs in the last week each spent a GPU smoke run discovering it again for a different family or image. A readiness-blocked outcome can now carry reason-code weights-missing. For the cooldown window the planner skips every candidate with the same model, precision and hardware, using the source rows in that wave's klaud-plan artifact. DSR1 FP4 与 FP8 权重在 b200-nscale 和 gb200-nv 上缺失，过去一周约有 25 个 Klaud PR 各自为不同的配置族或镜像花一次 GPU 冒烟运行重新发现这一点。 readiness-blocked 结果现在可以带 reason-code weights-missing；在冷却窗口内， 规划器会根据该轮 klaud-plan 产物中的来源行，跳过模型、精度和硬件相同的所有候选。
…(klaud): 在途工作保持排队并重跑被抢占的作业 (SemiAnalysisAI#4161) * fix(klaud): regenerate pre-flag producers, record deferral reasons, survive install flakes - Producers older than the generator's --no-evals flag rejected it, so their baselines could never be verified and nine multi-node families were deferred every wave; retry without the flag and keep only throughput rows. - select now records why each candidate's baseline was deferred in selection.json and the step summary instead of discarding the error. - Both Klaud workflows retry the Claude CLI install once when the native binary is missing. - An attempt whose agent step never ran is retried next wave instead of cooling down for 24 hours. - 早于生成器 --no-evals 参数的 producer 会拒绝该参数，导致其基线无法验证， 九个多节点配置族每一轮都被推迟；现在会不带该参数重试，并只保留吞吐量行。 - select 现在会在 selection.json 和步骤摘要中记录每个候选基线被推迟的原因， 而不是丢弃错误。 - 两个 Klaud 工作流在缺少原生二进制时会重试一次 Claude CLI 安装。 - Agent 步骤从未运行的尝试会在下一轮重试，而不是冷却 24 小时。 * fix(klaud): match aggregate multinode baselines as the app publishes them The app mirrors an aggregate engine into both role columns, so a one-engine aggregate row publishes decode_num_workers equal to prefill's, while the matrix gives decode zero workers. Older result writers also published a one-worker aggregate as a single-node row. Either way no public row matched its producer entry, deferring nine agentic agg families with "No verified public baseline points". 应用会把聚合引擎镜像到 prefill 和 decode 两列，因此聚合行发布的 decode_num_workers 与 prefill 相同，而矩阵中 decode 为零个 worker；较旧的结果 写入器还会把单 worker 聚合发布为单节点行。两种情况下公开行都无法匹配其 producer 条目，导致九个 agentic agg 配置族因"No verified public baseline points"被推迟。 * feat(klaud): free abandoned candidate PRs, branches and claims each wave Sessions that hit the 6h job limit left draft PRs, klaud/auto branches and family claims behind, and recovery could not reconcile them, so their families stayed blocked indefinitely. Before selection, recovery now treats a candidate as abandoned once no planner session or final sweep is active: it cancels the candidate's leftover smoke runs, closes a draft PR that only Klaud committed to, deletes leftover branches of PRs Klaud itself closed and releases claims held by finished parents. Handoff, maintainer-touched and ready PRs are left alone. Recovery also bounds its targeted-run listing to the parent's lifetime, which exceeded GitHub's 1000-result cap for older parents. 达到 6 小时作业上限的会话会遗留草稿 PR、klaud/auto 分支和配置族声明，且恢复 流程无法收尾，导致这些配置族被无限期阻塞。现在恢复流程会在选择前，将没有活动 规划会话或最终 sweep 的候选视为已放弃：取消其残留的冒烟运行，关闭只有 Klaud 提交过的草稿 PR，删除 Klaud 自己关闭的 PR 遗留的分支，并释放已结束父运行持有 的声明。交接、维护者改动过或已就绪的 PR 保持不变。恢复流程还会将定向运行列表 限定在父运行的存续期内，此前较旧父运行的列表超过了 GitHub 1000 条结果上限。 * fix(klaud): keep in-flight work queued and rerun preempted jobs Capacity is now checked only when the planner selects a candidate. The session no longer checks or defers for capacity, so targeted attempts and the final sweep wait at the back of the queue instead of closing as capacity-deferred. Work cancelled by anyone but Klaud (scheduler, maintainer cancel-and-rerun, GitHub's 24-hour queue limit) is preemption: Klaud waits for an existing rerun or reruns the cancelled jobs itself, without using repairs or infrastructure retries. The unused check-capacity command and the session's dashboard key are removed. 容量现在只在规划器选择候选时检查。会话不再检查容量或因容量推迟，定向尝试和最终 sweep 会在队列末尾等待，而不是以 capacity-deferred 关闭。被 Klaud 以外的任何人 取消的工作（调度器、维护者的取消并重跑、GitHub 的 24 小时排队上限）视为抢占： Klaud 会等待已有的重跑，或自行重跑被取消的作业，不消耗修复次数或基础设施重试。 同时移除不再使用的 check-capacity 命令和会话中的看板密钥。
…emiAnalysisAI#4165) * fix(klaud): regenerate pre-flag producers, record deferral reasons, survive install flakes - Producers older than the generator's --no-evals flag rejected it, so their baselines could never be verified and nine multi-node families were deferred every wave; retry without the flag and keep only throughput rows. - select now records why each candidate's baseline was deferred in selection.json and the step summary instead of discarding the error. - Both Klaud workflows retry the Claude CLI install once when the native binary is missing. - An attempt whose agent step never ran is retried next wave instead of cooling down for 24 hours. - 早于生成器 --no-evals 参数的 producer 会拒绝该参数，导致其基线无法验证， 九个多节点配置族每一轮都被推迟；现在会不带该参数重试，并只保留吞吐量行。 - select 现在会在 selection.json 和步骤摘要中记录每个候选基线被推迟的原因， 而不是丢弃错误。 - 两个 Klaud 工作流在缺少原生二进制时会重试一次 Claude CLI 安装。 - Agent 步骤从未运行的尝试会在下一轮重试，而不是冷却 24 小时。 * fix(klaud): match aggregate multinode baselines as the app publishes them The app mirrors an aggregate engine into both role columns, so a one-engine aggregate row publishes decode_num_workers equal to prefill's, while the matrix gives decode zero workers. Older result writers also published a one-worker aggregate as a single-node row. Either way no public row matched its producer entry, deferring nine agentic agg families with "No verified public baseline points". 应用会把聚合引擎镜像到 prefill 和 decode 两列，因此聚合行发布的 decode_num_workers 与 prefill 相同，而矩阵中 decode 为零个 worker；较旧的结果 写入器还会把单 worker 聚合发布为单节点行。两种情况下公开行都无法匹配其 producer 条目，导致九个 agentic agg 配置族因"No verified public baseline points"被推迟。 * feat(klaud): free abandoned candidate PRs, branches and claims each wave Sessions that hit the 6h job limit left draft PRs, klaud/auto branches and family claims behind, and recovery could not reconcile them, so their families stayed blocked indefinitely. Before selection, recovery now treats a candidate as abandoned once no planner session or final sweep is active: it cancels the candidate's leftover smoke runs, closes a draft PR that only Klaud committed to, deletes leftover branches of PRs Klaud itself closed and releases claims held by finished parents. Handoff, maintainer-touched and ready PRs are left alone. Recovery also bounds its targeted-run listing to the parent's lifetime, which exceeded GitHub's 1000-result cap for older parents. 达到 6 小时作业上限的会话会遗留草稿 PR、klaud/auto 分支和配置族声明，且恢复 流程无法收尾，导致这些配置族被无限期阻塞。现在恢复流程会在选择前，将没有活动 规划会话或最终 sweep 的候选视为已放弃：取消其残留的冒烟运行，关闭只有 Klaud 提交过的草稿 PR，删除 Klaud 自己关闭的 PR 遗留的分支，并释放已结束父运行持有 的声明。交接、维护者改动过或已就绪的 PR 保持不变。恢复流程还会将定向运行列表 限定在父运行的存续期内，此前较旧父运行的列表超过了 GitHub 1000 条结果上限。 * feat(klaud): say why in non-validated closing comments Closing comments for uncertain, failed and other non-validated outcomes only listed the outcome, repairs and runs. finish now adds the Next prose of the session's latest attempt record as the reason: English in the visible part, Chinese in the details block. Only attempt records that Klaud-Cold posted under this session's parent run and candidate count; chunk parts, other authors, other sessions and prose failing the public-prose checks are ignored. Recovery-finished sessions show no reason, since their last attempt may predate the interruption. The prompt now asks the final attempt's Next to state why the session stopped. uncertain、failed 等未验证结果的关闭评论此前只列出结果、修复次数和运行。现在 finish 会把本会话最新尝试记录中的 Next 说明作为原因加入评论：英文放在可见部分， 中文放在 details 区块中。只采用 Klaud-Cold 在本会话父运行和候选下发布的尝试 记录；分块评论、其他作者、其他会话以及无法通过公开文本校验的说明都会被忽略。 由恢复流程收尾的会话不显示原因，因为其最后一次尝试可能早于中断。提示词现在要求 最终尝试的 Next 说明会话停止的原因。
…代码中解析目标集群 (SemiAnalysisAI#4166) * fix(klaud): regenerate pre-flag producers, record deferral reasons, survive install flakes - Producers older than the generator's --no-evals flag rejected it, so their baselines could never be verified and nine multi-node families were deferred every wave; retry without the flag and keep only throughput rows. - select now records why each candidate's baseline was deferred in selection.json and the step summary instead of discarding the error. - Both Klaud workflows retry the Claude CLI install once when the native binary is missing. - An attempt whose agent step never ran is retried next wave instead of cooling down for 24 hours. - 早于生成器 --no-evals 参数的 producer 会拒绝该参数，导致其基线无法验证， 九个多节点配置族每一轮都被推迟；现在会不带该参数重试，并只保留吞吐量行。 - select 现在会在 selection.json 和步骤摘要中记录每个候选基线被推迟的原因， 而不是丢弃错误。 - 两个 Klaud 工作流在缺少原生二进制时会重试一次 Claude CLI 安装。 - Agent 步骤从未运行的尝试会在下一轮重试，而不是冷却 24 小时。 * fix(klaud): match aggregate multinode baselines as the app publishes them The app mirrors an aggregate engine into both role columns, so a one-engine aggregate row publishes decode_num_workers equal to prefill's, while the matrix gives decode zero workers. Older result writers also published a one-worker aggregate as a single-node row. Either way no public row matched its producer entry, deferring nine agentic agg families with "No verified public baseline points". 应用会把聚合引擎镜像到 prefill 和 decode 两列，因此聚合行发布的 decode_num_workers 与 prefill 相同，而矩阵中 decode 为零个 worker；较旧的结果 写入器还会把单 worker 聚合发布为单节点行。两种情况下公开行都无法匹配其 producer 条目，导致九个 agentic agg 配置族因"No verified public baseline points"被推迟。 * feat(klaud): free abandoned candidate PRs, branches and claims each wave Sessions that hit the 6h job limit left draft PRs, klaud/auto branches and family claims behind, and recovery could not reconcile them, so their families stayed blocked indefinitely. Before selection, recovery now treats a candidate as abandoned once no planner session or final sweep is active: it cancels the candidate's leftover smoke runs, closes a draft PR that only Klaud committed to, deletes leftover branches of PRs Klaud itself closed and releases claims held by finished parents. Handoff, maintainer-touched and ready PRs are left alone. Recovery also bounds its targeted-run listing to the parent's lifetime, which exceeded GitHub's 1000-result cap for older parents. 达到 6 小时作业上限的会话会遗留草稿 PR、klaud/auto 分支和配置族声明，且恢复 流程无法收尾，导致这些配置族被无限期阻塞。现在恢复流程会在选择前，将没有活动 规划会话或最终 sweep 的候选视为已放弃：取消其残留的冒烟运行，关闭只有 Klaud 提交过的草稿 PR，删除 Klaud 自己关闭的 PR 遗留的分支，并释放已结束父运行持有 的声明。交接、维护者改动过或已就绪的 PR 保持不变。恢复流程还会将定向运行列表 限定在父运行的存续期内，此前较旧父运行的列表超过了 GitHub 1000 条结果上限。 * refactor(klaud): resolve candidate target clusters in code, not in review The overlap review agent had to infer each candidate's target clusters, and the planner pre-filtered candidates with a hardware-prefix heuristic. The planner now derives every family's target clusters from its generated rows and runners.yaml while rendering the live families, and records them on each candidate. choose and select both require all of them to be eligible. A family whose runner maps to no cluster is skipped with a warning. The review schema and prompt drop clusters, and capacity.json is no longer written for the agent. 重叠审查 Agent 此前需要自行推断每个候选的目标集群，规划器还按硬件前缀启发式 预筛候选。现在规划器在生成现有配置族时，根据生成的矩阵行和 runners.yaml 计算 每个配置族的全部目标集群，并记录在每个候选上。choose 和 select 都要求这些集群 全部满足容量条件。runner 无法映射到任何集群的配置族会被跳过并输出警告。审查 schema 和提示词不再包含 clusters，也不再为 Agent 写入 capacity.json。
…] Muse Spark Open H100 全部并发点启用 HiCache 卸载 (SemiAnalysisAI#4182) * feat: offload every H100 Muse Spark point to HiCache, sweep c1-c8 * chore: link H100 HiCache entry to PR 4182 * fix: lower H100 Muse Spark HiCache mem fraction to 0.95 after c4 OOM * fix: set H100 Muse Spark HiCache mem fraction to 0.965 0.95 shrank the GPU KV pool to 98k tokens and rejected ~196k-token trace inputs.
…Muse Spark Open B300 TP8 与 HiCache 配置 (SemiAnalysisAI#4156) * perf: tune Muse Spark Open B300 for TP8 and HiCache 为 B300 启用 TP8/EP8 与分级 HiCache，保留并发 1 至 128 的全部十个点。 * chore: link B300 TP8 performance entry to PR 4156 为 B300 TP8 性能记录补充 PR 链接。 * fix: keep B300 Muse Spark c128 inside DFlash decode graphs * chore: refresh PR SemiAnalysisAI#4156 for sweep reuse [skip-sweep] --------- Co-authored-by: Oseltamivir <58582368+Oseltamivir@users.noreply.github.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
…isAI#3807) * refactor(launch): share bench's masking echo and send warnings to stderr The launcher and the container-side bench commands each had an echo() helper, and only the launcher's masked *TOKEN* and *SECRET* values. The masking echo now lives in infx.bench.proc (stdlib-only, Python 3.10), which the launcher already imports; bench call() and tee() also mask the child environment's secrets. Two launcher warnings that went to stdout now go to stderr with the other diagnostics. * feat(launch): record one job_event.json per benchmark job Debugging a failed benchmark job meant reading a multi-hour Actions log plus the server logs. Each `python -m infx.launch run` now writes one record, job_event.json, into the workspace when it exits on any path: the run, job and point identity, image, recipe, cluster, Slurm job and nodes; the launch decisions (path, DCGM power, golden acceptance length, eval flags, KV offloading); seconds per stage (prepare, submit, queue_wait, run, collect); the outcome; and for a failure the first error's stage, type, message, exit code, retriable hint and the log to read first. On GitHub Actions a failure also emits one ::error annotation titled with its stage. The benchmark templates upload the record as job_event_<RESULT_FILENAME>, and collect-results joins the records it can download into events.jsonl in the job_events artifact (python -m infx.results.collect_events). A batch-wrapped launch continues the record of the launch it re-runs. The golden acceptance planner now reports the length it simulates, and the Slurm accounting read includes the job's node list. * fix(ci): collect job events after every benchmark and eval job Collect-results waits only for throughput jobs, so eval-only jobs were missing from events.jsonl and evals-only sweeps produced no job_events artifact. A dedicated collect-job-events template now runs after every sweep job in run-sweep.yml and e2e-tests.yml. Add the Chinese troubleshooting section. * refactor(launch): trim job event duplication and echo-only test assertions finish() reuses exited() for its exit-status fallback, the single-use PREPARE constant is inlined, and a cleanup's name is looked up only when its nonzero result is reported. The srt driver tests stop asserting environment values copied verbatim into the record, a non-negative stage duration check, and an annotation-absence check that could not fail without GITHUB_ACTIONS. * fix(launch): record the power rule's require flag in job events The stack's PowerDecision gained require (the client fails a point whose AgentX power is invalid), and the job event records the decision with asdict, so its Power record must accept the field. * docs: read job event records first when root-causing sweep failures Main removed docs/troubleshooting.md, where this branch described job_event.json. Keep the essentials in the debug-runs skill's root-cause step: where the records are uploaded and collected, what a failure's error fields mean, and one jq query. * feat(launch): record native single-node power failures in the job event Main's native single-node power telemetry (SemiAnalysisAI#3781) added two ways a point fails on power: a required power package that cannot be staged, and a failed AgentX power adapter run under REQUIRE_POWER. Both now record a typed job-event error (ArtifactError, PowerValidationFailed) in the collect stage instead of ending as a bare ExitStatus or CleanupFailed. Console output is unchanged. * feat(launch): make the job event answer what ran, where, how long and why it failed Schema v2 of job_event.json: - Slurm accounting times (slurm_submitted_at/started_at/ended_at, queue_s, batch_queue_s) from the one sacct reading; the misleading log-wait queue_wait stage folds into run. - error.detail: the evidence log's likeliest cause line plus its last 20 lines, also named in the ::error annotation. - versions, topology, per-point metrics and GitHub links (job.check_run_id, PR number, test name, ref under test, passed from run-sweep.yml and e2e-tests.yml). - isl/osl are null for AgentX points. - The generator CLI fingerprints every matrix row as changelog plans do. - collect-job-events adds each job's conclusion and failed step and synthesizes a NoLauncherRecord record for benchmark jobs that wrote none. * fix(launch): record a rejected recipe binding with the binder's error The srt-slurm recipe binders (single-node bind_point and multi-node bind_recipe) now stream their stderr to the job log and keep it in srt-bind.log. A failure is recorded as RecipeBindFailed whose message is the binder's last line, with srt-bind.log as evidence and its cause line and tail as detail, instead of a bare ExitStatus. The templates upload srt-bind.log with the server logs.
…SemiAnalysisAI#4244) Waves every 2h overlapped up to three 6h candidate sessions at once. 每 2 小时一轮会使最多三个 6 小时的候选会话同时重叠。
SemiAnalysisAI#4247) Once a Klaud session ended, recovery closed the candidate as unexpected-error unless its latest final sweep had passed. A final sweep that the scheduler cancelled to make room for other work was therefore closed with its PR, branch and family claim, even though the same run would be rerun. Even when recovery waited, the abandoned-work sweep could close the still-draft PR in the same pass. - reconcile: a latest final sweep that was cancelled with no job in its current attempt that failed, timed out, or hit its timeout-minutes (GitHub reports that as cancelled) is preempted. Recovery waits 30 minutes for someone else's rerun, then reruns the failed jobs itself, up to 10 attempts. GitHub errors during the check or the rerun keep the candidate for the next pass. After 24 hours without a rerun, and for real failures and closed PRs, recovery behaves as before. - recover: a preempted candidate stays live, so its PR, branch, claim and recovery lease survive the abandoned-work sweep. - reconcile: a final sweep that passed but failed validation now logs the reason instead of closing silently. - run-sweep.yml: upload-changelog-metadata also needs collect-evals, so an eval-only rerun uploads a manifest for the current attempt. - run-sweep.yml: sweeps started after a PR is labeled klaud-handoff no longer run as Klaud background work. 中文：会话结束后保留并重跑被抢占的最终扫描。此前 Klaud 会话结束后，只要最新的 最终扫描未通过，恢复流程就会以 unexpected-error 关闭候选；调度器为其他工作 让出资源而取消的最终扫描也会连同 PR、分支和配置族认领一起被关闭，尽管该运行 随后会被重跑；即便恢复流程选择等待，遗弃工作清理也可能在同一轮关闭仍为草稿的 PR。现在：当前尝试中没有作业失败、超时或触发 timeout-minutes（GitHub 将其 报告为 cancelled）、因取消而结束的最新最终扫描视为被抢占。恢复流程先等待 30 分钟让他人重跑，之后自行重跑失败作业，最多 10 次尝试；检查或重跑时的 GitHub 错误会保留候选到下一轮；超过 24 小时仍未重跑、真实失败以及已关闭的 PR 行为 不变。被抢占的候选保持存活，其 PR、分支、认领和恢复租约不会被遗弃工作清理 删除。通过但验证失败的最终扫描会记录原因。run-sweep.yml 中 upload-changelog-metadata 同时依赖 collect-evals，仅重跑评估时也会上传当前 尝试的清单；PR 加上 klaud-handoff 标签后启动的扫描不再以 Klaud 后台优先级运行。
Klaud final sweeps used full-sweep-fail-fast in every PR over the last seven days, but the candidate prompt still allowed full-sweep-enabled as an exception and validation accepted it. Targeted runs relied on the agent passing fail-fast=true, and the multi-node agentic eval matrix in e2e-tests.yml ignored the input entirely. - e2e-tests.yml: every matrix fails fast when klaud-run is set. - run-sweep.yml: the Klaud manifest marks a sweep as a valid full sweep only when full-sweep-fail-fast is its sole sweep label. - lifecycle: a validated PR must carry exactly full-sweep-fail-fast. - prompt: drop the full-sweep-enabled exception. 中文：Klaud 基准测试始终启用 fail-fast。过去 7 天所有 Klaud PR 的最终扫描 都使用了 full-sweep-fail-fast，但候选提示词仍允许以 full-sweep-enabled 作为 例外，验证逻辑也接受它；定向运行依赖代理传入 fail-fast=true，而 e2e-tests.yml 中多节点智能体评估矩阵完全忽略该输入。现在 klaud-run 时 e2e-tests.yml 的所有矩阵都启用 fail-fast；run-sweep.yml 的 Klaud 清单仅在 full-sweep-fail-fast 为唯一扫描标签时才视为有效的完整扫描；已验证的 PR 必须 只带 full-sweep-fail-fast；提示词删除 full-sweep-enabled 例外。
) The planner admitted a cluster while its running plus queued nodes stayed within 200% of its schedulable nodes. That counted human and Klaud work together, admitted clusters that were full with human jobs waiting, and let every candidate in a wave count the same idle nodes. Capacity is now per-cluster headroom from the scheduler status: idle nodes minus nodes that waiting Klaud jobs already requested, and only on clusters where no human job is waiting. A family needs headroom for its largest job on every target cluster, and candidates selected in one wave consume that headroom. Wave cadence and size are unchanged. 中文：Klaud 只在空闲集群上启动候选。此前规划器只要集群的运行加排队节点不超过 可调度节点的 200% 就放行，把人工作业和 Klaud 作业混在一起计算，导致满载且有 人工作业排队的集群也会被选中，同一波次的候选还会重复占用同一批空闲节点。现在 容量按集群从调度器状态计算余量：空闲节点减去排队中 Klaud 作业已申请的节点，且 仅限没有人工作业排队的集群。配置族的每个目标集群都必须容纳其最大的单个作业， 同一波次已选中的候选会扣减余量。波次频率和规模保持不变。
…wen3.5): GB300 TRT AgentX 切换到新的 Dynamo 配方 (SemiAnalysisAI#4195) * perf(qwen3.5): switch GB300 TRT AgentX to new Dynamo recipes Switch qwen3.5-fp4-gb300-dynamo-trt-agentic-disagg to TensorRT-LLM release:1.3.0a0.dev202610151219 and Dynamo from the 1.6.0.dev20261010 wheel with publish_metrics, and replace the search space with nine points from 1P1D c44 to 6P1D c1046. 将 qwen3.5-fp4-gb300-dynamo-trt-agentic-disagg 切换到 TensorRT-LLM release:1.3.0a0.dev202610151219，Dynamo 从 1.6.0.dev20261010 wheel 安装并开启 publish_metrics，并将搜索空间替换为从 1P1D c44 到 6P1D c1046 的九个点。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(qwen3.5): keep the GB300 TRT AgentX frontend on the head node srtctl runs post-eval on the head node against localhost, so a frontend on the first decode node refuses every eval request in multi-node layouts where decode has its own nodes. srtctl 在头节点上针对 localhost 运行 post-eval；当解码使用独立节点时，前端放在 第一个解码节点会导致所有评测请求被拒绝连接。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): update GB300 TRT AgentX engine configs and points Enable return_perf_metrics on prefill and decode, set a 2048 MB KV-cache bounce buffer on the colocated 1P1D roles, and replace the search space with 1P1D c38/c45/c48, 1P2D TEP4/TEP8 c48, 1P4D c38 (b1 and b4), 2P3D c136, 3P1D c565 and 3P2D c704. Dynamo settings are unchanged. 在 prefill 和 decode 上启用 return_perf_metrics，为同节点部署的 1P1D 角色设置 2048 MB 的 KV-cache 中转缓冲区，并将搜索空间替换为 1P1D c38/c45/c48、 1P2D TEP4/TEP8 c48、1P4D c38（b1 和 b4）、2P3D c136、3P1D c565 和 3P2D c704。 Dynamo 设置保持不变。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(qwen3.5): keep return_perf_metrics off on GB300 TRT AgentX Match the current recipes, which disable return_perf_metrics on prefill and decode; Dynamo publishes worker metrics. 与当前配方保持一致：prefill 和 decode 均关闭 return_perf_metrics，由 Dynamo 发布工作进程指标。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(qwen3.5): use the shared DRAM utilization for GB300 TRT AgentX 1P1D The DRAM budget is computed per role from that role's own GPU count, so 0.609524 already yields 128 GiB per rank for the colocated 1P1D roles; 0.457143 under-reported their allocation. Drop the header comments in the recipe. DRAM 预算按每个角色自身的 GPU 数计算，因此 0.609524 已为同节点部署的 1P1D 角色给出每个 rank 128 GiB；0.457143 会低报其分配量。同时删除配方中的头部注释。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kedar Potdar <115327600+kedarpotdar-nv@users.noreply.github.com>
…K3 GB300 完整曲线 (SemiAnalysisAI#4208) * perf: refresh the complete Kimi-K3 GB300 AgentX curve 升级现有 GB300 聚合及分离式配置的 vLLM 镜像，并加入使用概率草稿采样的四个低延迟点。 Signed-off-by: Xin Li <xinli@nvidia.com> * docs: link the full-curve changelog to PR 4208 将完整曲线更新记录关联到 PR SemiAnalysisAI#4208。 Signed-off-by: Xin Li <xinli@nvidia.com> * fix: configure Kimi K3 parsers for Dynamo serving Register the native K3 tool-call and reasoning parsers on aggregate and decode workers so required tool calls use the XTML path instead of generic JSON constraints. 为聚合和解码工作节点配置 Kimi K3 工具调用与推理解析器，使必选工具调用使用 XTML 路径，避免错误应用通用 JSON 约束。 * fix: update Dynamo for Kimi K3 streaming parser fixes Pin aggregate and disaggregated recipes to ffbbaaa6, which uses dynamo-parsers 8.1.0 and includes the K3 chunk-boundary fixes from frontend-crates PR 152. Keep the native-vLLM megakernel recipes unchanged. 将聚合和分离式配方的 Dynamo 固定到 ffbbaaa6，使用包含 K3 输出分块解析修复的 dynamo-parsers 8.1.0，保持原生 vLLM megakernel 配方不变。 * fix(kimik3): restore higher DSpark acceptance curve Use probabilistic draft sampling consistently so the existing golden-AL selector chooses 3.36 for four drafts and 3.84 for seven drafts. Keep evaluation on real block verification. 统一使用概率草稿采样，使现有 golden AL 选择器对 4 个和 7 个草稿分别使用 3.36 和 3.84；评测继续使用真实 block 验证。 Co-authored-by: Codex <noreply@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> --------- Signed-off-by: Xin Li <xinli@nvidia.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
…aud): 恢复未完成的会话并新增 klaud-abort (SemiAnalysisAI#4265) * feat(klaud): resume sessions that run out before finishing A Klaud session is a GitHub-hosted job capped at 6 hours. Preempted work now often outlives it, and the next wave's recovery then closed the candidate, cancelling its leftover runs. Recovery now resumes such a candidate in its original planner run, so the branch, PR, baseline, attempt comments and runs keep their identity: - An unfinished session (open bot draft, no outcome or cleanup record, fewer than 4 sessions, latest job failed, run ended under 24 hours ago and was not cancelled by a person) is approved to resume. - Recovery grants approved sessions whole planner runs at a time, oldest first, within the wave's candidate cap. Each granted PR gets a resume comment for the next attempt, then the run's failed jobs are rerun. - A rerun candidate job starts its agent only with a grant for that exact attempt; other rerun jobs end without a session. - Resumed sessions take slots from the wave's cap (select --resumed). - The agent step stops after 330 minutes, so recover-current still runs, and it keeps an unfinished resumable session for a later resume. - Only a planner run's first attempt can own a candidate PR, so a newer run's PR on the same branch is never adopted. - Sessions that recovery cannot reconcile stay open for up to 24 hours. - The candidate cooldown and weights pause age each session by its own diagnostics artifact. - prepare-baseline returns the frozen baseline from the PR body. 中文：恢复在完成前超时的会话。Klaud 会话是上限 6 小时的 GitHub 托管作业，被抢占的 工作现在常常比会话更久，此前下一波次的恢复流程会直接关闭候选并取消残留运行。现在 恢复流程会在原规划运行中继续该候选，分支、PR、基线、尝试评论和运行保持原有身份： 未完成的会话（机器人草稿 PR 仍打开、无结果或清理记录、会话少于 4 次、最近作业失败、 运行结束不足 24 小时且未被人工取消）可被批准恢复；恢复流程按最早的规划运行优先、 以整个运行为单位并在本波次候选上限内授予恢复，为每个 PR 发布对应下一次尝试的恢复 评论，再重跑该运行的失败作业；重跑的候选作业只有在该尝试有授权时才启动代理；恢复的 会话占用本波次的名额；代理步骤 330 分钟后停止，以便 recover-current 仍能运行并保留 可恢复的未完成会话；只有规划运行的第一次尝试能拥有候选 PR，不会接管更新运行的 PR； 无法收尾的会话最多保留 24 小时；候选冷却和权重暂停按各自的诊断产物计算时间； prepare-baseline 返回 PR 正文中冻结的基线。 * fix(klaud): keep ungranted resumable sessions re-runnable - resume-check fails a rerun job whose session is still resumable but has no grant for this attempt, so a later rerun can still resume it; such gate-only jobs no longer count as sessions. - Recovery grants planner runs strictly oldest first. - A failed grant deletes the grant comments it posted. - Approval is decided from the candidate's own latest job: failed, timed out, or cancelled at its time limit. A person's cancel is not resumed. - The prompt counts only commits after the last reported head, other than the perf-changelog entry, as used repairs. 中文：保持未获授权但可恢复的会话仍可重跑。对仍可恢复但本次尝试没有授权的重跑作业， resume-check 会让作业失败，以便之后的重跑仍能恢复它，这类仅运行检查的作业不再计入 会话次数；恢复流程严格按最早的规划运行优先授权；授权失败时删除已发布的授权评论； 是否批准恢复改为依据候选自身最近一次作业判断（失败、超时或因达到时间上限被取消）， 人工取消不会恢复；提示词只把最后一次报告的 head 之后、除 perf-changelog 条目以外的 提交计为已用修复次数。 * feat(klaud): resume cancelled sessions and add a klaud-abort label A cancelled candidate session is now treated like preempted Klaud work: recovery may resume it like a session that ran out. To stop a candidate for good, a maintainer applies klaud-abort instead. Then: - the session may stop without its own outcome; the job's cleanup step finishes it, - reconcile finishes it as the new `aborted` outcome, which cancels its owned runs, closes the PR and deletes the branch, and the closing comment says the label stopped it, - recovery and the resume gate never resume it, even with a grant. The resume comment now says the session ended rather than ran out. 中文：恢复被取消的会话并新增 klaud-abort 标签。被取消的候选会话现在与被抢占的 Klaud 工作同样处理，恢复流程可以像对待超时会话一样恢复它。如需彻底停止某个候选， 维护者应添加 klaud-abort 标签：会话可以在没有自身结果的情况下停止，由作业的收尾 步骤完成收尾；reconcile 以新的 `aborted` 结果收尾，取消其自有运行、关闭 PR 并删除 分支，关闭评论会说明由该标签停止；恢复流程和恢复检查即使存在授权也不会恢复它。 恢复评论现在表述为会话已结束，而不是已超时。 * feat(klaud): tell every candidate PR how to stop it Every Klaud candidate PR body now ends with a line saying that adding the klaud-abort label at any time stops the candidate for good, cancels its runs and closes the PR. The label name lives in models.py so the renderer and the lifecycle share it. 中文：在每个候选 PR 中说明如何停止它。每个 Klaud 候选 PR 正文末尾现在都会说明： 随时添加 klaud-abort 标签即可彻底停止该候选，取消其运行并关闭 PR。标签名定义在 models.py 中，供渲染器和生命周期共用。 * feat(klaud): keep aborted candidates out of the pool until unlabeled klaud-abort now means "stop now and don't retry". The planner skips a candidate while any of its Klaud PRs carries the label. Once the label is removed, the candidate is eligible again at the planner's next scan: an aborted outcome adds no cooldown, and a candidate whose latest PR ended in an abort skips the cooldown left by its earlier sessions. The planner's review job can now read issues to see labels and outcome comments, and the PR body explains both steps. 中文：在移除标签前将被中止的候选排除在候选池之外。klaud-abort 现在表示“立即停止 且不再重试”：只要该候选的任一 Klaud PR 带有此标签，规划器就会跳过它。移除标签后， 该候选会在规划器下次扫描时重新符合条件：aborted 结果不增加冷却时间，最近一个 PR 以中止结束的候选也会跳过其早先会话留下的冷却。规划器的审查作业现在可以读取 issues， 以查看标签和结果评论；PR 正文也说明了这两个步骤。 * feat(klaud): make klaud-abort the only Klaud control label Remove the klaud-handoff label and its handoff outcome. The prompt, Stop hook, diagnostics, recovery, resume approval and abandoned-work sweep no longer look for it, and run-sweep gives Klaud PRs background priority whatever their labels. Klaud still leaves merged PRs alone, recovery still skips PRs where a maintainer started runs after the session, and the abandoned-work sweep still skips PRs that are ready for review or carry maintainer commits. 中文：让 klaud-abort 成为唯一的 Klaud 控制标签。移除 klaud-handoff 标签及其 handoff 结果：提示词、Stop 钩子、诊断、恢复、恢复批准和遗弃工作清理都不再检查它， run-sweep 中 Klaud PR 无论带什么标签都以后台优先级运行。Klaud 仍不会处理已合并的 PR；对于会话结束后由维护者启动过运行的 PR，恢复流程仍会跳过；遗弃工作清理也仍会跳过 已标记为可审查或包含维护者提交的 PR。
… / 默认使用完整轨迹并显式列出 256k 例外 (SemiAnalysisAI#4287) * fix(agentx): default to the uncapped trace corpus 默认使用完整轨迹数据集，仅为明确列出的模型系列保留 256k 限制，并保持显式覆盖行为。 * fix(agentx): use the full trace corpus for Muse Spark 移除 Muse Spark 的 256k 例外，更新测试和文档，并记录七个受影响配置的性能变更。
… [Klaud Cold] 在完整轨迹数据集上重新 sweep Muse Spark Open B200 (SemiAnalysisAI#4291)
…gregated and DEP2/DEP16 disaggregated points / [NVIDIA][AgentX] GB300 DeepSeek-V4.1-Flash Dynamo+SGLang：TP4/TEP2 聚合与 DEP2/DEP16 分离式配置点 (SemiAnalysisAI#4180)
From bench/gemma4-livekit-sgl-pfgraph's 2x TP4 car128 arm. The refactor flow does not support Dynamo multi-replica yet (schema.py:398-406), so this is one TP4 engine at half the conc (c75/c100/c125 = same per-engine CCU as the original c150/c200/c250), min-initial-workers 1. Scenario custom-agentic- livekit111868-v1.1-warm -> livekit, runner hardware-hcm-8x_02 -> cluster:hcm-hgx02, host_cpus/host_mem_gb dropped (legacy-only). Handoff for the multi-replica gap: InferenceOptimization/docs/handoffs/2026-10-29-inferencex-dynamo-multi-replica-support.md
…gth) Port of the SPEED-Bench collectors to the serving architecture: one request per row of the fully resolved Green-Lab/speed-bench-prepared files (pinned per config in the dataset lock), first turn only as the aiperf/vLLM loaders send it, golden-matrix sampling defaults, thinking switch via chat_template_kwargs. Shares specdec (spec_decode block), arms: and the summary speedup column with greenspec; the data is public, so greenspec's upload guard is not applied.
… the private dataset
…261027 agentx-harness was a thin wrapper: an agentx CLI injecting --scenario agentx (cli_defaults + environment_defaults) over a fork of the aiperf package. Upstream deleted that alias spec, so swap to the upstream nightly and pass what the wrapper injected explicitly: - replay.py: --scenario inferencex-agentx-mvp is mandatory (aiperf silently no-ops the scenario without it) plus the retired agentx scenario defaults: streaming, server token count, stats 30 s, seed 42, trace idle cap 300 s, 393 entries, slice 1.0 s, no gpu telemetry, trajectory ratios 0.25/0.75, failed-threshold 0.10, warmup 10/lane, warmup grace 1800 s. - run.py/client.py: the scenario's environment defaults (dataset/service configure timeouts 1800 s, TCP user timeout 900 s) as only-if-unset env. - venv.py/event.py: bin/aiperf; the version regex accepts both pins. - The nightly (not released 0.13.0) is required: the release rejects the serving scenarios' warmupRequestsPerLane and traceIdleGapCapSeconds keys. - test_configs.py: count 10 -> 12 (two gemma4 livekit configs landed on the base branch without updating the test; fails on the base tip too). Artifact names, record models, and the results convention are unchanged. Evidence: stub A/B shows identical token streams and inter-turn gaps; GPU A/B (remote-maas GLM-5.3-Flash c30 3600 s, seed 42) matches the agentx twin within 0.3% on token streams and cache hit; livekit and tau2 scenario runs green.
perf(bench): replace agentx-harness with upstream aiperf
…at/speedbench-refactor
…style (ruff excludes tests)
…mote config overrides.warmup_requests_per_lane: 0 removes warmupRequestsPerLane from the rendered scenario, so aiperf skips the zero-idle cache-pressure warmup (only the mandatory snapshot primers run) -- needed for remote endpoints whose token/min quota the warmup burst exceeds (LiveKit agent gateway 429 MaxConcurrentGatewayLLMTpm). Adds configs/serving/remote-gemma4-livekitgw-c20.yaml (LiveKit Cloud gateway, gemma-4-31b-it, livekit v1.1-warm, CCU 20) and drops the config-count pin test (forbidden config pin). <details><summary>中文</summary> overrides.warmup_requests_per_lane: 0 会从渲染的 scenario 中移除 warmupRequestsPerLane,aiperf 只保留必发的 snapshot primer,跳过零间隔的 cache-pressure 预热——用于 token/min 配额会被预热突发打爆的远端 endpoint (LiveKit agent gateway 429 MaxConcurrentGatewayLLMTpm)。新增 configs/serving/remote-gemma4-livekitgw-c20.yaml(LiveKit Cloud 网关, gemma-4-31b-it,livekit v1.1-warm,CCU 20),并删除禁用的配置数量固定测试。 </details>
…ota) The gateway's MaxConcurrentGatewayLLMTpm quota (~2.4M tokens/min) rejects CCU 20: 13/125 profiling requests 429'd (10.4% > 10% threshold, run 40995671920). CCU 12 keeps the replay's token rate under the quota. <details><summary>中文</summary> 网关的 MaxConcurrentGatewayLLMTpm 配额(约 2.4M tokens/min)在 CCU 20 下扛不住: 125 个 profiling 请求中 13 个 429(10.4%,超过 10% 阈值,run 40995671920)。 降到 CCU 12,回放的 token 速率可控制在配额之内。 </details>
The replay's token demand (~2.9M tokens/min, arrival-driven by trace timing, unchanged from CCU 12 to 20) exceeds the plan's ~2.4M MaxConcurrentGatewayLLMTpm quota; CCU 6 caps in-flight lanes low enough to throttle demand under the quota. <details><summary>中文</summary> 回放的 token 需求(约 2.9M tokens/min,由 trace 时机驱动,CCU 12 到 20 几乎不变) 超过了套餐约 2.4M 的 MaxConcurrentGatewayLLMTpm 配额;CCU 6 通过压低在途 lane 数把需求压到配额之内。 </details>
v1.1.1 fixes tool parameters whose required keyword was {} instead of a list
(548 of 3072 rows); SGLang rejects those with HTTP 400, vLLM accepts them.
payload() also rewrites the keyword, so an older release still runs on both engines.
Resolve lock.yaml (greenspec-v1.1.1 + speedbench entries) and test_datasets.py. Review fixes: lock entries carry private: true and runner-setup keys off it, serving-bench lists the speedbench scenario and variants, drop the test that pins lock.yaml contents.
greenspec-v1.1.1 is marked private in lock.yaml; datasets.py takes an optional boolean (default false, one-line error otherwise) and the runner-setup skill keys off the flag instead of hand-copying the private entry names. Drop the test that pins lock.yaml contents; a small test covers the private flag parsing instead.
feat(serving): greenspec scenario, spec-decode acceptance, arms, run summary
Pick up the greenspec v1.1.1 pin, the SGLang 0.0-gauge fix and the lock private flag (e5ba0c2) merged via #43. Conflicts resolved: lock.yaml keeps the six speedbench entries beside greenspec-v1.1.1, runner-setup SKILL.md keeps the wording that names both scenarios. 274 serving tests pass, ruff clean.
feat(serving): speedbench scenario family (SPEED-Bench acceptance length)
…LO search on the winning c64 config (YaRN ctx 1M, hicache 64 GB/rank); dispatch run 41170766014
…m vng-benchmark Public ai-dynamo/sglang-runtime:1.6.1 for engine and router (frontend image lacks the sglang module), mem-fraction 0.75 -> 0.8, full-length agentx. c40 failed the SLO on vng-benchmark (run 41124818237: ITL p90 69.2ms); c28/c32 bracket the estimated 50ms crossing near CCU 29. Node: hcm-hgx02 only (node 01 held by an out-of-slurm vLLM gemma-4 server).
thangquang09
added a commit
that referenced
this pull request
Oct 31, 2026
…he tree-swap migration (PR #46)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Migrate
vng-benchmarkto the refactored tree. After this merge,vng-benchmarkis therefactor/vng-benchmarkcode (same tree, byte for byte), and the pre-migrationvng-benchmarklives on asarchive/vng-benchmark-20261031.What this PR is
A record of the migration decision, not a line-by-line review: 667 commits restructured onto the upstream layout (
inferencex-e2e/,collectivex/,operatorx/, ...). GitHub will show conflicts — that is expected. The merge is performed as a tree-swap: one merge commit whose tree is taken verbatim fromrefactor/vng-benchmark(parents: both branch tips). No hybrid, no conflict resolution.Audit summary (pre-migration, 2026-10-31)
result-report, runner/Slurm infra (accountgreennode, salloc right-sizing, gres,/mnt/hf_hub_cache), the 60-yaml sweep grid (now a parameterizedsyntheticscenario), synthetic acceptance injector, and the serving skills are all ported — several in improved form. The new flow is battle-tested on the cluster (successful runs 2026-10-28/30/31).qwen38-fp8-eagle-v0523-sweep-1gpu(run 41170766014) and the GLM-5.3 dynamo-1.6.1 mem-0.8 ladder (c40 + c28/c32).storage_*external-hit counting ininfx/results/agentic/backends/sglang.py(plan deviation feat: AIPerf agentic-replay for 3 trace datasets (+ optional --tokenizer) #9). Aggregate JSONs under-reportserver_external_cache_hit_ratefor L3 runs;result-reportis unaffected. Port when running a serious L3 A/B or opening an upstream PR.archive/vng-benchmark-20261031by design;serving-benchSKILL.md translation rules cover recreating any of them.After the merge
refactor/vng-benchmarkstays until the raw URL in.agents/skills/serving-runner-setup/SKILL.mdis repointed tovng-benchmark, then it can be deleted.