[AgentX] Validate vLLM cached-token tier metrics

[cquil11]

Purpose

Consume the physical cached-token source metric added by cquil11/vllm#2 and validate it through InferenceX AgentX collection.

The vLLM adapter surfaces vllm:prompt_tokens_cached_by_source as server_metrics.cache.cached_tokens_by_source, aggregated across endpoints and engines. The companion validation tooling exercises device, CPU, disk, generic external transfer, and NIXL paths, and enforces:

sum(physical cached-token sources) == cached-token total
local_cache_hit + external_kv_transfer == cached-token total

A bounded H100 smoke lane runs Qwen3-0.6B at TP1 with both GPU-resident KV and 13 GB of vLLM native CPU offload. It uses a checked-in growing-prefix Weka fixture that remains within the model's 40,960-token context window. The lane is observability plumbing, not a publishable performance submission.

It rejects:

• Missing built-in source series before warmup.
• Nonzero startup counters.
• Missing expected positive device or cpu attribution.
• Unexpected positive source labels.
• Physical or logical totals that do not conserve cached tokens.
• Duplicate or aliased Prometheus endpoints.

Image

ttl.sh/cquil11-vllm-tier-dfb25dbef4-20260828:24h

Digest: sha256:3592139a63122a7e2168805c71f705fd59156c85b5fc12e82fde0f91bc0d5f9c

The final vLLM Python files were applied to the tested CUDA runtime image without recompiling kernels. The tag is temporary; this PR remains a draft and needs a durable image before merge.

Validation

Final exact-commit workflow: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33133119809

Commit: 474973acbea52dbcd044cbd6c4624a9d18c0b22d

Both H100 jobs first observed exactly one endpoint and these zero-valued startup sources:

device, cpu, disk, mixed, external

Final exports:

Arm	Cached total	device	cpu	Logical cached total
GPU-only	764,688	764,688	0	764,688
DRAM offload	613,472	342,704	270,768	613,472

Direct successful jobs:

• DRAM: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33133119809/job/98726994884
• GPU-only: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33133119809/job/98726994885

The run-level conclusion is displayed as failure despite the Jobs API reporting every instantiated job as successful or intentionally skipped. Both AgentX jobs, collection, aggregation, and success-rate jobs are green.

Additional hardware coverage:

• H100 targeted live tests: device, CPU, and native NVMe/XFS disk passed.
• H200 targeted live tests: device, CPU, disk, startup-zero series, Qwen3 NIXL, and Qwen3.5 NIXL passed.
• Mixed one-request accounting observed 1,088 device + 2,160 tier = 3,248 cached tokens for both CPU and disk.
• Real Qwen3.5 NIXL decode observed 3,263 external + 1 local compute.

Local validation:

• Aggregation, parser/adversarial-label, and changelog tests: 65 passed.
• Matrix/schema tests: 255 passed.
• Bash syntax, shellcheck (with the repository's sourced-file exclusion), Python compilation, and git diff --check passed.
• CodeQL passed, including the linear-time Prometheus label parser added after adversarial input review.

An earlier run exposed two edge cases now covered by gates: a first-scrape baseline gap for lazily created labels, and duplicate collection through localhost plus 127.0.0.1. The vLLM startup initialization fixes the first; the canonical endpoint and single-endpoint assertion fix the second.

Related vLLM draft: cquil11/vllm#2

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档