[Power] Power measurement roadmap

[edwingao28]

Links

• Dashboard: https://inferencex.semianalysis.com/inference?i_seq=8k%2F1k&i_metric=y_measuredJPerOutputToken
• H200 exact-pin validation: run 32324249431
• Producer: https://github.com/SemiAnalysisAI/InferenceX
• Consumer: https://github.com/SemiAnalysisAI/InferenceX-app
• Producer fork: edwingao28/srt-slurm#2 at e5c837f

Roadmap

Producer

• Single-node fixed-sequence power, NVIDIA + AMD feat(power): aggregate measured GPU power into agg result JSON #1558 [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• Multinode disaggregated power [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• Prefill/decode role watts Emit per-role average power on multinode disaggregated runs / 多节点分离式运行输出每角色平均功率 #2553
• Schema v2 whole-deployment energy semantics [Power] feat: stamp whole-deployment semantics with a schema version / 为功耗指标加盖整机语义版本号 #2599
• Single-node AgentX power [Power] feat: measure GPU power for single-node AgentX runs / 为单机 AgentX 运行接入 GPU 功耗测量 #2601
• GB200/GB300 FP4 DCGM lanes [Power] feat: extend dcgm energy lanes to gb dsv4/qwen3.5 fp4 / 扩展 dcgm 能耗采集到 gb dsv4 与 qwen3.5 fp4 #2507
• AMD multinode power
• Kimi-K3 power evaluation; retain its current vLLM/external-fork clone route until separately qualified

Multinode rollout [1/5]–[5/5]

• [1/5] Long-lived srt-slurm fork prerequisite (merged 2026-08-24) edwingao28/srt-slurm#2
• [2/5] H200 multinode AgentX infrastructure [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683 (merged 2026-08-24)
• [3/5] H200 GLM-5.2 + DSV4 recipe rollout [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684 (merged 2026-08-24; CI sweep 8/8 = 100%, GLM + DSV4 both publication_valid: true)
• [4/5] GB multinode recipe rollout [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687 — merged 2026-08-25; validation run 32780220262: GB200 qwen c768 + GB200 dsv4 c16384 + GB300 qwen c5120 all required=True status=complete publication_valid=True; dsv4-GB nightly-pin lanes still blocked by rotted images (pre-existing on main)
• [5/5] B200/B300 multinode rollout [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688 — merged 2026-08-26; validation run 32924589757: b200-dgxc + b200-nscale lanes zero failures, b200-nscale-slurm_02 vllm audit required=True complete publication_valid=True (c128/c256 windows valid), B300 both frameworks published valid power; remaining reds attributed to b300-014 enroot perms (cluster ops) and intermittent NIXL; kimik2.6 lanes blocked on runner capacity (also on main)

Multinode rollout validation

• H200 GLM-5.2 strict validation on exact producer e5c837f: run 32324249431 — required: true, publication_valid: true, 32/32 GPUs, schema v2, power_valid: 1
• H200 DSV4 optional recipe validation on producer a1b8c7af: run 32312491367
• H200 CI full validation on e5c837f via [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684: run 32767381601 — 8/8 lanes, GLM disagg + DSV4 agg both status=complete publication_valid=True
• Local and CI contract validation for [Power] Wire H200 multinode AgentX power / 接入 H200 多节点 AgentX 功耗链路 #2683, [Power] Require GLM-5.2 and add DSV4 H200 telemetry / 强制 GLM-5.2 并新增 DSV4 H200 功耗采集 #2684, [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687, and [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688
• H200 DSV4 exact-pin refresh on e5c837f (optional; remains required: false)
• H100 SA-Bench dedicated-infrastructure smoke on e5c837f if required for the fork scope
• Native GB hardware validation for [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 #2687
• Native B200/B300 hardware validation for [Power] Require B200 and B300 multinode telemetry / 强制启用 B200 和 B300 多节点功耗采集 #2688
• After each rollout merge, verify production ingestion and dashboard visibility: schema v2, power_valid scrubbing, whole-deployment J/token semantics, J/query, and role watts where applicable

Power validation series [1/4]–[4/4]

• [1/4] Single-node energy validation [1/4] [Power] feat: validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323
• [2/4] Multinode energy validation [2/4] [Power] feat: validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 #2437
• [3/4] GB200/GB300 official DCGM lane [3/4] [Power] feat: enable official dcgm energy lane for gb200/gb300 1p1d / 打开 gb200/gb300 官方 1P1D 能耗采集 #2456
• [4/4] AMD MI355X strict monitor lifecycle [4/4] [Power] feat: strict AMD gpu monitor lifecycle for MI355X / MI355X 单节点功率监控严格生命周期 #2494

Upstream srt-slurm

• [1/4] DCGM power artifact data layer #288
• [2/4] Multinode DCGM collection #289 (closed; carried in the fork pin)
• [3/4] Measurement window contract #290
• [4/4] Offline artifact validator + telemetry docs #291 (in review; not a blocker for the long-lived fork)

Dashboard

• Measured energy axes app#720
• Validity guard app#734
• J/query, Wh/query, and %TDP axes app#735
• AI-chart lower-is-better fixes fix(ai-chart): respect metric direction and missing values / 修复 AI 图表指标方向与缺失值 InferenceX-app#798