Persistent local-disk tier and DCP support for the in-process v1 connector

[chriswritescode-dev]

Two fixes needed to run GLM-5.2 MLA prefix offload through the in-process LMCacheConnectorV1 with DCP, both running in production on 4x RTX PRO 6000 (GLM-5.2-EXL3-TR3-3.0bpw, TP4/DCP4/MTP3, fp8 KV, L1 16 GiB DRAM + L2 200 GiB NVMe) on top of 0.5.2+glm52dcp.3.

1. DCP for the in-process v1 connector

#1 fixed MLA+DCP for the MP-server path (vllm_multi_process_adapter.py, kv_cache_groups.py). vllm_v1_adapter.py has no DCP handling, so with decode_context_parallel_size > 1 the default save_only_first_rank path stores rank 0's shard alone and scatters it back to every rank: the in-process connector saves and restores nothing usable.

dcp_gather.py gathers each rank's latent KV shard before a store and scatters it back on load, and ReqMeta.from_request_tracker now checks block capacity against the rank-local token count instead of the global one. Every hook returns immediately when DCP is inactive, so the non-DCP path is untouched.

Limitation: only the strided layout is handled, i.e. --dcp-kv-cache-interleave-size 1. Larger interleave sizes fall through to the previous behaviour rather than storing something wrong.

2. The local-disk tier survives a restart

LocalDiskBackend keeps its index only in memory and never enumerates its directory, so every restart both loses the whole L2 tier and leaks it: max_local_disk_size becomes a per-process allowance. Measured here on a 200 GiB cache: 522 GB on disk, 345.7 GB of it invisible to the running process.

disk_index_rebuild.py adopts the existing chunks at startup, keyed off the deterministic _key_to_path naming plus a byte-layout sidecar. Adoption requires the filename to parse back to the identical canonical path, the model name to match this run, chunk_size to match the config, and the file size to match the recorded layout, so a directory holding another model's chunks contributes nothing instead of producing a false hit. Over-cap chunks are deleted newest-kept-first, which enforces the cap against real on-disk bytes for the first time, and adoption order follows mtime so LRU recency roughly survives the restart.

Same file, three fixes the persistent tier made unavoidable:

• Re-put accounting. Overwriting an indexed key consumes no new bytes, but current_cache_size and usage grew on every put. A prefix-heavy workload re-saves the same chunks constantly, so the counter climbed until it crossed max_cache_size and evicted the entire real cache (observed as an eviction storm with 3.3 GB on disk against a 200 GiB cap).
• Vanished chunk served as a hit. read_file caught FileNotFoundError, popped the index entry and returned, after which load_bytes_from_disk did self.dict[key] and raised KeyError; without that lookup it would have handed back the never-written staging buffer as cached KV. read_file now reports success, both single-chunk callers treat a missing file as a miss, and the async prefetch path raises with the path instead of a bare KeyError. batched_get_blocking already guarded this correctly.
• remove() on an already-unlinked file. Eviction can unlink a file whose put task is still in flight; the next eviction raised FileNotFoundError out of wait_for_save and killed the engine. The index is the accounting source of truth, so the entry is dropped either way.

Tests

tests/v1/test_dcp_gather.py (10 tests, CPU-only, no torch.distributed) covers the interleave/deinterleave round trip against a brute-forced global order, heterogeneous layer widths and dtypes, the DCP4 packed round trip, rank-local token counts, and rank-symmetric collective counts in the store path.

tests/v1/storage_backend/test_disk_index_rebuild.py (14 tests) exercises the real insert_key, the real CacheEngineKey parser and the real LRU policy: adoption and byte accounting, mtime-ordered LRU, / mangling in model names, rejection of another model's chunks and of wrong-sized files, TP-agnostic key adoption, missing/mismatched/corrupt sidecars, cap enforcement, purge, and a guard that the module only touches attributes LocalDiskBackend.__init__ really assigns.

tests/v1/storage_backend/test_local_disk_backend.py gains TestVanishedChunkFile, which uses the existing real-backend fixtures. The two regression cases fail on this branch's parent with exactly the KeyError described above and pass with the fix; the third is a positive control that a present chunk still loads with its bytes intact. _write_key now delegates to a module-level register_chunk helper the new class shares.

Full file: 36 passed.

Notes

• The layout sidecar is currently written by the storing path in dcp_gather.py, so the first commit alone adopts nothing. Happy to move the write into LocalDiskBackend so the disk tier owns its own layout record and the persistence half becomes independent of the connector, if you prefer that shape.
• read_file's signature is left unannotated (as upstream) so mypy keeps treating it as untyped; the new bool return is documented in its callers instead.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• dev/*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f5a43d83-3add-4e75-96b1-8d1f89ef6c56

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}