Skip to content

feat(serving): mix scenario (open-loop trace replay), repo-local datasets, CI HTML report - #41

Open
nvcnvn wants to merge 14 commits into
refactor/vng-benchmarkfrom
feat/serving-mix-trace
Open

nvcnvn wants to merge 14 commits into
refactor/vng-benchmarkfrom
feat/serving-mix-trace

Conversation

@nvcnvn

@nvcnvn nvcnvn commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What

  • mix scenario: replays a mixed-traffic trace (bench repo studies/mixed-workload/mix.py) open loop at its own timestamps (fixed_schedule, blockSize: 64); conc only caps requests in flight. The rate is the load axis: one lock entry per rate, picked with overrides.dataset.
  • Repo-local datasets: a lock entry may pin path + sha256 (relative to inferencex-e2e/) instead of an HF repo/revision/file. First entry: mix-enterprise-12000-1h (6.8 MB, 12679 requests).
  • CI report: gn-benchmark.yml gets a report job that runs .agents/skills/result-report over the whole run, writes report.md to the run summary and uploads report.html (unzipped) plus report.md/data.json.
  • result-report "Over time" section: ~60 windows per run (2 s for a 2-minute smoke, 60 s for 1 h) rebuilt from aiperf's profile_export.jsonl with aiperf's own time-slice rules; at 1 s it matches aiperf's profile_export_aiperf_timeslices.json exactly (test fixture from a real run). Artifacts keep aiperf's 1 s slices. Markdown: collapsed table + Mermaid lines (render on the Summary page); HTML: SVG panels.
  • Report-only mode: gh workflow run gn-benchmark.yml --ref <branch> -f report-run=<run id> -f test-name=... re-renders a finished run's report with the branch's code, no GPU job.
  • Example config configs/serving/qwen3-8b-mix-enterprise-h200.yaml (Qwen3-8B, 1x H200 on hgx-h200-01, HiCache L1+L2), docs and skill tables updated.

Notes

  • inferencex-agentx-mvp locks agentic_replay; unsafeOverride: true keeps fixed_schedule and marks the run not submission-valid (same as E2/E3/LiveKit).
  • duration equals the trace length (3600 s): the scenario lock needs TTFT/ITL over 95% of it. A smaller overrides.duration cuts the replay.

Testing

  • pytest infx/tests/serving: 183 passed; result-report tests: 105 passed + 107 subtests (rebuilt 1 s windows == aiperf's 1 s slices).
  • aiperf validate mooncake-trace on the trace: 12679 rows OK.
  • 30 s dry run with agentx-harness 1.0.12 against a local stub: 96/96 requests (exactly the trace's first 30 s), process_agentic_result and validate_agentic_result pass.
  • 1x H200, duration: 3900 (https://github.com/vngcloud/InferenceX/actions/runs/40465657027): all 12679 requests served, 0 errors, 3.52 req/s (= offered), TTFT p50/p90/p99 46/405/2957 ms, cache hit 34%; failed only on aiperf's coverage check (92.3% of 3900 s < 95%). Fixed by duration: 3600.
  • Rerun with the fix on 1x H200 (hcm-hgx01, hgx02 was held by an 8-GPU job), passed: https://github.com/vngcloud/InferenceX/actions/runs/40478619956: 3.52 req/s (= offered), 12,047 tok/s, TTFT p50/p90/p99 49/442/3237 ms, ITL p90 17 ms, cache hit 34% (all GPU), errors 0.01%. Report on the run's Summary page.
  • 180 s smoke with 60 s slices on hgx-h200-01, passed, first CI report with the "Over time" section: https://github.com/vngcloud/InferenceX/actions/runs/40510497105
  • Re-rendered reports with the time view: https://github.com/vngcloud/InferenceX/actions/runs/40620395261 (1 h, 60 windows of 60 s), https://github.com/vngcloud/InferenceX/actions/runs/40620399491 (smoke, 36 windows of 5 s).
  • RTX PRO 6000 nodes: not used. Checkout works now that git is installed, but lmsysorg/sglang:v0.5.22 cannot be pulled from Docker Hub there (401s mid-download over a slow link, run 40482719267) until the image is staged on the nodes.

🤖 Generated with Claude Code

nvcnvn and others added 3 commits October 27, 2026 13:49
…ise mix 12000 tok/s 1h trace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n times (fixed_schedule)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ummary and report.html

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thanks for the contribution!

  • Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
  • PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
  • After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.
中文

感谢你的贡献!

  • **审阅:**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER,请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,再通过 Slack 联系核心维护者。必须严格遵循模板,并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have,才能触发签核验证。
  • **PR 验证:**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目,并且恰好带有一个主标签时,才会运行扫描:full-sweep-fail-fast(强烈推荐;canary 加逐矩阵 fail-fast)、full-sweep-enabled(有 canary;矩阵任务在失败后继续运行)或 non-canary-full-sweep-enabled(无 canary,也无 fail-fast)。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
  • **合并后:**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复;参见重新运行失败任务的说明。

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

inferencex-e2e/infx diff: +71 -15

Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package.

中文

大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。

nvcnvn and others added 11 commits October 27, 2026 14:11
…it for checkout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…verage at 92.3% < the lock's 95%

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 s slices made ~80 MB per hour and an unreadable time axis; 60 s keeps aiperf's own per-slice
stats at ~1.3 MB per hour. Whole-run metrics are unchanged; nothing in the repo read the 1 s slices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…exported

Markdown: a collapsed per-run table plus Mermaid lines for output tok/s and TTFT p90; HTML: SVG
panels over minutes. No re-aggregation: values are aiperf's per-slice stats. Runs with slices
under 30 s get a note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…drop the RTX example

The RTX nodes cannot pull lmsysorg/sglang:v0.5.22 from Docker Hub (401s mid-download over a slow
link) until the image is staged on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mermaid's auto y range drew a 9% change as a full-height climb; points sit at each slice's
start, so a 3-minute run ends at minute 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…benchmark

Re-running the old report job reuses that run's commit; this runs the current result-report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ult)

Other tools read the 1 s timeslice exports; coarser views belong in the report, not the artifact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est records

The artifacts keep aiperf's 1 s slices (other tools read them). The report rebuilds the same
per-slice numbers at ~60 windows per run from profile_export.jsonl with aiperf's rules; at 1 s it
matches profile_export_aiperf_timeslices.json exactly (TimelineTest, fixture from run 40478619956).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant