kv-router: add block-weighted overlap counters (standalone) - #36
Open
Pernekhan wants to merge 1 commit into
Open
kv-router: add block-weighted overlap counters (standalone)#36Pernekhan wants to merge 1 commit into
Pernekhan wants to merge 1 commit into
Conversation
router_kv_hit_rate is a histogram of per-request overlap/isl, so sum/count is a mean of ratios. The engine's vllm:prefix_cache_hits/queries is a ratio of sums. On a skewed ISL distribution these diverge badly -- on DeepSeek-V4-Flash-0731 (44% of requests under 1800 tokens, mean 14211, tail past 130k) the router read 0.320 against the engine's 0.767 and looked half blind. Add router_kv_overlap_blocks_total / router_kv_isl_blocks_total so the router's predicted overlap can be compared with the engine directly. Measured on the same fleet right after deploy: block-weighted router 0.790 vs engine 0.767, i.e. the router's view was accurate all along. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single commit on top of
main, +36 lines across 3 files, no behavior change. Supersedes #34, which carried this same commit on top of 11 unrelated ones.The trap this closes
dynamo_component_router_kv_hit_rateis a histogram of per-requestoverlap_blocks / isl_blocks, sosum/countis a mean of ratios (request-weighted). The engine'svllm:prefix_cache_hits_total / queries_totalis a ratio of sums (block-weighted). They are not comparable, and on a skewed ISL distribution they diverge badly.On DeepSeek-V4-Flash-0731 — 44% of requests under 1800 tokens (≤7 blocks), mean ISL 14,211, tail past 130k tokens (~508 blocks) — the router read 0.320 against the engine's 0.767 and looked half blind.
It isn't. That mis-comparison consumed most of a debugging session and nearly triggered an unnecessary indexer rewrite on a subsystem that was working correctly.
The fix
Add
router_kv_overlap_blocks_total/router_kv_isl_blocks_total, incremented at routing time fromselection.effective_overlap_blocksandisl_blocks. Dividing them yields a block-weighted router hit rate directly comparable to the engine's.Verified live, immediately after deploy
Within 3% of the engine and slightly ahead, which is expected — some blocks are evicted between the routing decision and execution.
Independently corroborated: 10/10 paired requests sharing an 8k-token prefix showed 97.7% block-weighted reuse (range 96.0–98.5%), i.e. the router picked the prefix-holding worker every time.
Risk
None to serving. Two
IntCounters and their increments; no routing logic touched. The code comment states the aggregation difference explicitly so the next person doesn't walk into it.🤖 Generated with Claude Code
https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw