Skip to content

cookbook: support NVMe canonical CPU cache - #89

Draft
nanjiangwill wants to merge 4 commits into
mainfrom
codex/cpu-cache-nvme-canonical
Draft

cookbook: support NVMe canonical CPU cache#89
nanjiangwill wants to merge 4 commits into
mainfrom
codex/cpu-cache-nvme-canonical

Conversation

@nanjiangwill

Copy link
Copy Markdown
Contributor

What changed

  • Document the CPU destination's two canonical-checkpoint placements: the existing in-memory default and host-local storage selected by --cpu-weight-cache-canonical-checkpoint-dir.
  • Pin the corresponding general and public K3 SGLang feature branches while their dependent PRs are under review.
  • Configure Kimi K3 MXFP4 to keep its 1.561 TB canonical checkpoint on ephemeral NVMe and retain only its TP8 rank images in RAM, lowering the validated memory limit from 4 TiB to 3 TiB.
  • Exercise the same path in the Kimi K2.6 NVFP4 profiler.
  • Ensure disk-destination profiles remove the CPU-only canonical-directory argument.

Why

CPU-prepared updates need rank-ready images resident for the short CPU-to-GPU engine pause, but the canonical Hugging Face checkpoint is only used during background reconstruction and compilation. Large models can place that canonical image on host-local NVMe to trade preparation latency and local I/O for substantially lower persistent host RAM without adding another update mode or RPC.

Dependencies

This draft temporarily pins:

After those PRs are rebased into their stable branches, this PR must replace the feature pins and commit table with the resulting stable commit IDs before it is ready to merge.

Validation

  • uv run ruff check .
  • uv run pytest -q (80 passed)
  • Ruff formatting on every changed Python file
  • Kimi K2.6 NVFP4 TP4: 633.3 GB after staging, 544.4 s preparation, 2.95 s engine pause, exact repeated token IDs/logprobs after update
  • Kimi K3 MXFP4 TP8 under a 3 TiB limit: 1.718 TB after staging, 3,018.9 s full-coverage preparation, 3.90 s engine pause, exact repeated token IDs/logprobs after update, zero cgroup max/OOM events

@nanjiangwill
nanjiangwill force-pushed the codex/cpu-cache-nvme-canonical branch from 102e86c to 3b3be5a Compare July 29, 2026 05:20
@nanjiangwill
nanjiangwill force-pushed the codex/cpu-cache-nvme-canonical branch from 3b3be5a to 99ae141 Compare July 29, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant