kv-router: conditional disagg + overlap-driven KV migration on v1.4.1 by Pernekhan · Pull Request #35 · deepinfra/dynamo

Pernekhan · 2026-08-31T20:25:12Z

Draft — deployed and exercised, but gated off in production. Opening for review, not merge.

Ports the conditional-disagg + KV-migration work onto v1.4.1 (11 commits). The fork it came from was v1.1-era; main has since fast-forwarded to v1.4.1 (branch sync/v1.4.1), so this rebases that work onto what we actually run.

What's in it

Conditional disaggregation — DYN_MIN_REMOTE_PREFILL_TOKENS gates remote prefill by prompt size, so small requests prefill locally instead of paying a NIXL round-trip.
Overlap-driven KV migration — when the router sends a request to a worker other than the one holding its KV, optionally migrate the blocks instead of recomputing. Driven by effective_overlap_blocks rather than session identity, so it isn't limited to same-session reuse.
kv_migration_source on PreprocessedRequest — lets a decode worker act as a NIXL producer for the migration handshake.
Candidate telemetry — DYN_KV_MIGRATION_SAMPLE logs spill candidates without acting, for sizing the opportunity.
benchmarks/di-dsv4/ — the mirror/compare harness plus a README of measurement rules learned the hard way (warm before first measurement; >=6 windows; normalize per-GPU load).

Knobs: DYN_KV_MIGRATION (0 / dry / 1), DYN_KV_MIGRATION_MIN_TOKENS, DYN_KV_MIGRATION_SAMPLE.

Measured opportunity

Dry-mode telemetry on a 6-GPU aggregate fleet, 90s window:

spill rate        27.3%   (6.97/s of 25.5 req/s; log-based 28.0% agrees)
migratable tokens p50 3,072 · p90 14,336 · max 295,168 · mean 11,266
migratable volume 78,490 tok/s   vs 114,081 tok/s fleet uncached prefill

Treat 78k/114k as an upper bound, not a 69% saving — migratable_tokens is the overlap on the best worker, and the chosen worker may already hold part of it. The spill log doesn't carry the chosen worker's overlap, so the true delta is unmeasured.

We previously measured spill at 0.33–1.2% and shelved this as a tiny lever. That predated DYN_ROUTER_TRACK_OUTPUT_BLOCKS=true — the router couldn't see decode-generated blocks, so it couldn't detect the overlaps. The spills were happening; we were blind to them.

Why it's gated off

Migration needs a NIXL producer. In pure-aggregate mode (--disaggregation-mode=agg, no --kv-transfer-config) the worker never returns disaggregated_params, so kv_migration_handshake returns None silently — that path has no log. Symptom: KVMIGRATE_SPILL fires while producer handshake obtained stays at zero.

Do not run it in that state: each spill dispatches a real extra request to the source worker that can never succeed (~7/s of pure waste). Currently set to dry.

Enabling it for real means putting NixlConnector back on the workers, which reserves buffers and costs KV capacity — likely lowering hit rate and partly offsetting the gain. That trade is unmeasured.

Upstream status

NVIDIA has not shipped vLLM KV-block migration (checked against origin/main past v1.4.2). What exists upstream is request migration (lib/llm/src/migration.rs, --migration-limit) — re-dispatch on worker failure. SharedCacheType offers only None and Hicache, and HiCache queries sglang workers. So there's nothing to adopt in place of this.

🤖 Generated with Claude Code

https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw

Prompts below the threshold take the existing prefill-router passthrough and are served entirely on the decode worker (aggregate semantics, no KV-transfer handoff); larger prompts dispatch to dedicated prefill workers as before. Remote prefill only amortizes its handoff (route + prefill queue + NIXL + decode admission, ~450ms measured) above roughly 16k uncached tokens. Validated on DSv4-Flash roce-disagg 1P/1D: mixed real-shape trace TTFT p50 349ms (prod parity; 684-6576ms for all prior unconditional-disagg configs), gate confirmed by per-tier query attribution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit d892e43)

…ilure Discovery re-lists briefly drop instances from the transport map; a whale routed to a flapped prefill instance failed hard with 'Instance not found' (and the resulting 500 storm turned the endpoint-health-backed /live probe red, killing the frontend). Prefill is an optimization under conditional disaggregation: on any non-ResourceExhausted dispatch error, serve the request entirely on the decode worker instead of failing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 72c9995)

DYN_SESSION_FROM_PREFIX_TOKENS=N hashes the first N token ids into a SessionAffinityId when the caller sends none. Conversation renderings grow by appending, so every turn maps to the same key and pins to the same decode worker, making its prefix cache reusable. Overlap-credit scoring cannot achieve this at high load because the decode-load term dwarfs the per-request overlap credit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 4b4a22c)

…sion set DYN_LOCAL_CONTINUATIONS=1 keeps conversation continuations local regardless of raw prompt length: their history is already cached on the pinned decode worker, so only the new tail needs prefill. Only cold large prefills go remote. Approximated with a bounded recently-seen set keyed by the session prefix hash (1h TTL, 4M-entry cap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit eb02438)

Samples continuation requests and logs would-migrate decisions to size the D->D migration opportunity on production traffic. The dispatch leg (prefill on the pinned worker via bidirectional NIXL, decode elsewhere) follows in the next increment; design in kv-migration-design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 6ccc08c)

Wire-format half of decode-to-decode KV migration: the flag the router sets so a decode worker serves one producer step and publishes its NIXL handshake (see the DecodeWorkerHandler change). Additive and skipped when absent, so no behaviour changes until a router sets it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit af34dc3)

Producer behaviour was class-gated: prefill_request_kv_transfer_params() (do_remote_decode=True, max_tokens=1) is only ever called from PrefillWorkerHandler, so a decode worker could never offer its blocks to a peer. That made decode-to-decode KV migration impossible from the frontend at any price -- no header or routing trick can reach it. When the router marks a request with `kv_migration_source`, the decode handler now behaves like a prefill worker for exactly one step: it emits producer kv_transfer_params, generates a single token, and publishes the resulting handshake as `disaggregated_params` on the final chunk so the TARGET decode worker can pull the conversation's blocks over NIXL instead of recomputing them. Requires kv_role=kv_both, which our decode workers already run. Inert until a frontend sets the flag: with no `kv_migration_source` in the request, every path here is byte-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit db7220f)

Hand-ported (the originals conflict with 458 commits of upstream churn): - SchedulingResponse gains best_overlap_worker/best_overlap_blocks, fed by a max_overlap_worker() argmax over the per-worker overlap map - carried through FindBestMatchOutcome::Routed into WorkerSelection - migration dispatch in push_router::generate, gated on DYN_KV_MIGRATION and DYN_KV_MIGRATION_MIN_TOKENS, with kv_home as an index-lag fallback

…erry-pick The 1.3.0 fork's generate_tokens caller had this local; v1.4.1's does not, so the cherry-picked call site referenced an undefined name and every request failed with NameError -> HTTP 500. The parameter keeps its None default.

Missed in the port: the cherry-pick loop broke on the first push_router conflict, so 75154b1 never applied, and the hand-port only covered push_router.rs. The id was back to hashing min(len, N) tokens, which changes every turn for conversations shorter than N -- turn 2 looks like a new session and lands on a worker holding none of its blocks.

+
+def load(path):
+    rows = []
+    for line in open(path):


+                d[k] = v
+        try:
+            rows.append({k: float(v) for k, v in d.items() if k != 't'})
+        except ValueError:


Pernekhan and others added 11 commits August 25, 2026 17:22

benchmarks: add the DSv4 harness used for the dynamo/engine evaluation

3f6f08d

Pernekhan deployed to external_collaborator August 31, 2026 20:25 — with GitHub Actions Active

github-actions Bot added documentation Improvements or additions to documentation frontend backend::vllm router labels Aug 31, 2026

github-advanced-security Bot found potential problems Aug 31, 2026

View reviewed changes

Comment thread benchmarks/di-dsv4/summarize.py

def load(path):

rows = []

for line in open(path):

Comment thread benchmarks/di-dsv4/summarize.py

d[k] = v

try:

rows.append({k: float(v) for k, v in d.items() if k != 't'})

except ValueError:

Pernekhan mentioned this pull request Aug 31, 2026

kv-router: add block-weighted overlap counters #34

Draft

Navigation Menu

Uh oh!

kv-router: conditional disagg + overlap-driven KV migration on v1.4.1 - #35

kv-router: conditional disagg + overlap-driven KV migration on v1.4.1#35
Pernekhan wants to merge 11 commits into
mainfrom
claude/port-141-kv-migration

Pernekhan commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants