kv-router: add block-weighted overlap counters by Pernekhan · Pull Request #34 · deepinfra/dynamo

Pernekhan · 2026-08-31T20:24:44Z

The trap

dynamo_component_router_kv_hit_rate is a histogram of per-request overlap_blocks / isl_blocks, so sum/count is a mean of ratios (request-weighted). The engine's vllm:prefix_cache_hits_total / queries_total is a ratio of sums (block-weighted).

Those are not comparable, and on a skewed ISL distribution they diverge badly. On DeepSeek-V4-Flash-0731 — 44% of requests under 1800 tokens (<=7 blocks), mean ISL 14,211, tail past 130k tokens (~508 blocks) — the router read 0.320 against the engine's 0.767 and looked half blind.

That mis-comparison cost hours of investigation and nearly triggered an unnecessary indexer rewrite. The router was fine the whole time.

The fix

Add router_kv_overlap_blocks_total / router_kv_isl_blocks_total, incremented at routing time from selection.effective_overlap_blocks and isl_blocks. Dividing them gives a block-weighted router hit rate directly comparable to the engine's.

Verified live, immediately after deploy

router BLOCK-weighted   = 0.790   <- new counters
engine  BLOCK-weighted  = 0.767   <- vllm:prefix_cache_hits/queries
router REQUEST-weighted = 0.320   <- the old, misleading number

The router's view matches the engine within 3% and sits slightly ahead, which is expected — some blocks are evicted between the routing decision and execution.

Independently corroborated by a controlled test: 10/10 paired requests sharing an 8k-token prefix, block-weighted reuse 97.7% (range 96.0–98.5%), i.e. the router picked the prefix-holding worker every time.

The code comment states the aggregation difference explicitly so the next person doesn't walk into it.

🤖 Generated with Claude Code

https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw

Prompts below the threshold take the existing prefill-router passthrough and are served entirely on the decode worker (aggregate semantics, no KV-transfer handoff); larger prompts dispatch to dedicated prefill workers as before. Remote prefill only amortizes its handoff (route + prefill queue + NIXL + decode admission, ~450ms measured) above roughly 16k uncached tokens. Validated on DSv4-Flash roce-disagg 1P/1D: mixed real-shape trace TTFT p50 349ms (prod parity; 684-6576ms for all prior unconditional-disagg configs), gate confirmed by per-tier query attribution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit d892e43)

…ilure Discovery re-lists briefly drop instances from the transport map; a whale routed to a flapped prefill instance failed hard with 'Instance not found' (and the resulting 500 storm turned the endpoint-health-backed /live probe red, killing the frontend). Prefill is an optimization under conditional disaggregation: on any non-ResourceExhausted dispatch error, serve the request entirely on the decode worker instead of failing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 72c9995)

DYN_SESSION_FROM_PREFIX_TOKENS=N hashes the first N token ids into a SessionAffinityId when the caller sends none. Conversation renderings grow by appending, so every turn maps to the same key and pins to the same decode worker, making its prefix cache reusable. Overlap-credit scoring cannot achieve this at high load because the decode-load term dwarfs the per-request overlap credit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 4b4a22c)

…sion set DYN_LOCAL_CONTINUATIONS=1 keeps conversation continuations local regardless of raw prompt length: their history is already cached on the pinned decode worker, so only the new tail needs prefill. Only cold large prefills go remote. Approximated with a bounded recently-seen set keyed by the session prefix hash (1h TTL, 4M-entry cap). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit eb02438)

Samples continuation requests and logs would-migrate decisions to size the D->D migration opportunity on production traffic. The dispatch leg (prefill on the pinned worker via bidirectional NIXL, decode elsewhere) follows in the next increment; design in kv-migration-design.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit 6ccc08c)

Wire-format half of decode-to-decode KV migration: the flag the router sets so a decode worker serves one producer step and publishes its NIXL handshake (see the DecodeWorkerHandler change). Additive and skipped when absent, so no behaviour changes until a router sets it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit af34dc3)

Producer behaviour was class-gated: prefill_request_kv_transfer_params() (do_remote_decode=True, max_tokens=1) is only ever called from PrefillWorkerHandler, so a decode worker could never offer its blocks to a peer. That made decode-to-decode KV migration impossible from the frontend at any price -- no header or routing trick can reach it. When the router marks a request with `kv_migration_source`, the decode handler now behaves like a prefill worker for exactly one step: it emits producer kv_transfer_params, generates a single token, and publishes the resulting handshake as `disaggregated_params` on the final chunk so the TARGET decode worker can pull the conversation's blocks over NIXL instead of recomputing them. Requires kv_role=kv_both, which our decode workers already run. Inert until a frontend sets the flag: with no `kv_migration_source` in the request, every path here is byte-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw (cherry picked from commit db7220f)

Hand-ported (the originals conflict with 458 commits of upstream churn): - SchedulingResponse gains best_overlap_worker/best_overlap_blocks, fed by a max_overlap_worker() argmax over the per-worker overlap map - carried through FindBestMatchOutcome::Routed into WorkerSelection - migration dispatch in push_router::generate, gated on DYN_KV_MIGRATION and DYN_KV_MIGRATION_MIN_TOKENS, with kv_home as an index-lag fallback

…erry-pick The 1.3.0 fork's generate_tokens caller had this local; v1.4.1's does not, so the cherry-picked call site referenced an undefined name and every request failed with NameError -> HTTP 500. The parameter keeps its None default.

Missed in the port: the cherry-pick loop broke on the first push_router conflict, so 75154b1 never applied, and the hand-port only covered push_router.rs. The id was back to hashing min(len, N) tokens, which changes every turn for conversations shorter than N -- turn 2 looks like a new session and lands on a worker holding none of its blocks.

router_kv_hit_rate is a histogram of per-request overlap/isl, so sum/count is a mean of ratios. The engine's vllm:prefix_cache_hits/queries is a ratio of sums. On a skewed ISL distribution these diverge badly -- on DeepSeek-V4-Flash-0731 (44% of requests under 1800 tokens, mean 14211, tail past 130k) the router read 0.320 against the engine's 0.767 and looked half blind. Add router_kv_overlap_blocks_total / router_kv_isl_blocks_total so the router's predicted overlap can be compared with the engine directly. Measured on the same fleet right after deploy: block-weighted router 0.790 vs engine 0.767, i.e. the router's view was accurate all along. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

+
+def load(path):
+    rows = []
+    for line in open(path):


+                d[k] = v
+        try:
+            rows.append({k: float(v) for k, v in d.items() if k != 't'})
+        except ValueError:


Pernekhan · 2026-08-31T21:23:58Z

Superseded for review purposes — do not merge this as-is.

This branch is the exact build of the deployed frontend (dynamo-frontend:dyn141-blockwt, head fb8d2456ca), so it is useful as a record of what is running. But it is not a reviewable unit: it carries 12 commits, and PR #35 is a verified strict subset of it (git merge-base --is-ancestor returns true).

Split for review:

kv-router: add block-weighted overlap counters (standalone) #36 — the block-weighted counters alone, one commit on main, +36 lines, no behavior change. This is the mergeable piece.
kv-router: conditional disagg + overlap-driven KV migration on v1.4.1 #35 — conditional disagg + KV migration (draft, gated off by default).

Also worth recording: of the 12 commits here, the conditional-disagg and session-id work lives in prefill_router/mod.rs and is inert in aggregate mode — our frontend runs DYN_MIN_REMOTE_PREFILL_TOKENS=100000000, so that path never executes. The only commit that runs on every request in the deployed config is the metrics one now split into #36.

Pernekhan and others added 12 commits August 25, 2026 17:22

benchmarks: add the DSv4 harness used for the dynamo/engine evaluation

3f6f08d

Pernekhan deployed to external_collaborator August 31, 2026 20:24 — with GitHub Actions Active

github-actions Bot added documentation Improvements or additions to documentation frontend backend::vllm router labels Aug 31, 2026

github-advanced-security Bot found potential problems Aug 31, 2026

View reviewed changes

Comment thread benchmarks/di-dsv4/summarize.py

def load(path):

rows = []

for line in open(path):

Comment thread benchmarks/di-dsv4/summarize.py

d[k] = v

try:

rows.append({k: float(v) for k, v in d.items() if k != 't'})

except ValueError:

Pernekhan mentioned this pull request Aug 31, 2026

kv-router: add block-weighted overlap counters (standalone) #36

Open

Pernekhan marked this pull request as draft August 31, 2026 21:23

Navigation Menu

Uh oh!

kv-router: add block-weighted overlap counters - #34

kv-router: add block-weighted overlap counters#34
Pernekhan wants to merge 12 commits into
mainfrom
claude/kv-router-block-weighted-metrics

Pernekhan commented Aug 31, 2026

Uh oh!

Pernekhan commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants