feat(agentx): fail incomplete profiling metric coverage / 检测 AgentX 分析指标覆盖不完整

[cquil11]

Summary

• require TTFT and inter-token-latency observations through at least 98% of each valid duration-based AgentX profiling phase
• exit non-zero with an actionable server-health error when coverage ends early
• retain JSON, CSV, JSONL, and log artifacts and mark the scenario result invalid with insufficient_profile_metric_coverage
• exclude warmup observations and intentionally short --unsafe-override smoke runs
• leave request scheduling, ordering, warmup handoff, and spawn/join logic unchanged

Validation

• full local pre-commit suite passed
• targeted changed-path suites passed (up to 170 tests per independent audit)
• four independent subagent mock-server audits on exact head f77c31a715850ce8f0eb9dd37ddc90ce0bffcdbe: 4 PASS, 0 FAIL
• healthy real mock profiles exited 0 with 99.8–100% TTFT coverage and 100% ITL coverage
• real mock servers deliberately stopped or suspended during profiling caused exit 1 with ProfileMetricCoverageError, no hang, retained aggregate artifacts, submission_valid: false, and insufficient_profile_metric_coverage
• exact 98% boundary, missing ITL, warmup exclusion, and short-smoke exclusion verified independently
• warmup-per-lane 1/10/100, fanout ordering, and SPAWN_JOIN gating tests passed
• rule replayed against InferenceX run 30730541420: all 28 healthy throughput jobs pass and all 6 known stalled jobs fail

中文说明

• 要求每个满足有效时长要求的 AgentX 分析阶段中，TTFT 和 token 间延迟观测至少覆盖 98% 的阶段时长
• 当指标覆盖提前结束时，以非零状态退出并给出可执行的服务端健康检查提示
• 保留 JSON、CSV、JSONL 和日志产物，并使用 insufficient_profile_metric_coverage 将场景结果标记为无效
• 预热观测和通过 --unsafe-override 运行的短时冒烟测试不计入该检查
• 不修改请求调度、顺序、预热交接及 spawn/join 逻辑

验证

• 完整本地 pre-commit 检查通过
• 针对改动路径的测试套件通过；每个独立审计最多执行 170 个测试
• 4 个独立子代理在精确提交 f77c31a715850ce8f0eb9dd37ddc90ce0bffcdbe 上完成 mock server 审计：4 个通过，0 个失败
• 健康的真实 mock 分析任务以 0 退出，TTFT 覆盖率为 99.8%–100%，ITL 覆盖率为 100%
• 在分析阶段主动停止或挂起真实 mock server 后，AIPerf 无卡死并以 1 退出，报告 ProfileMetricCoverageError，保留聚合产物，同时写入 submission_valid: false 和 insufficient_profile_metric_coverage
• 已独立验证精确 98% 边界、缺失 ITL、排除预热以及排除短时冒烟测试
• 每 lane 预热请求数为 1、10、100 的测试，以及 fanout 顺序和 SPAWN_JOIN 门控测试均通过
• 使用 InferenceX 工作流 30730541420 的结果回放规则：28 个健康吞吐任务全部通过，6 个已知卡死任务全部失败

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@f77c31a715850ce8f0eb9dd37ddc90ce0bffcdbe

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@f77c31a715850ce8f0eb9dd37ddc90ce0bffcdbe

---

Last updated for commit: f77c31a • Browse code

[github-actions]

Fern Docs Preview: generation failed — see the Actions log for details. This does not block merge; ask a maintainer to retry if needed.