Skip to content

kv-cache: resolve TURBO_LAYER_ADAPTIVE mode per cache construction - #354

Merged
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
giveen:fix/turbo-layer-adaptive-per-cache
Sep 6, 2026
Merged

kv-cache: resolve TURBO_LAYER_ADAPTIVE mode per cache construction#354
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
giveen:fix/turbo-layer-adaptive-per-cache

Conversation

@giveen

@giveen giveen commented Sep 5, 2026

Copy link
Copy Markdown

Summary

Split out of #326 per review comment: #326 (comment)

adaptive_mode in llama_kv_cache::llama_kv_cache was a function-local static, so it was resolved once from whichever cache was constructed first in the process and then reused verbatim for every later construction, regardless of that cache's own type_v. This is a real bug for iswa/MTP setups that build more than one llama_kv_cache with different type_v in the same process, and for anything that sweeps --cache-type-v (e.g. llama-bench).

  • Hoisted the mode resolution out of the per-layer loop into a per-construction local (kv_adaptive_mode), computed once per llama_kv_cache::llama_kv_cache call before the layer loop starts.
  • Every layer within a given cache now sees the same mode (as before), and every new cache construction re-resolves the mode against its own type_v (the fix).
  • No functional change to the mode-dispatch logic itself — this only fixes when the mode is resolved.

Test plan

  • Builds clean (llama target)
  • Single-cache-per-process behavior unchanged (mode still resolved once per construction)

🤖 Generated with Claude Code

adaptive_mode was a function-local static, so it was computed once
from whichever cache was constructed first in the process and then
reused verbatim for every later construction regardless of that
cache's own type_v - a real problem for iswa/MTP setups that build
more than one kv_cache with different type_v in the same process, and
for anything that sweeps --cache-type-v (e.g. llama-bench).

Hoisted the mode resolution out of the per-layer loop into a
per-construction local computed once per llama_kv_cache::llama_kv_cache
call, before the loop starts, so every layer in a given cache sees the
same mode and every new cache re-resolves it against its own type_v.

Split out of PR TheTom#326 per review at
TheTom#326 (comment)

Assisted-by: Claude

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@TheTom
TheTom merged commit d7afd02 into TheTom:feature/turboquant-kv-cache Sep 6, 2026
9 of 24 checks passed
@TheTom

TheTom commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Merged. Per-construction resolution is the right shape; the second cache in an iswa or MTP process now gets its own mode instead of inheriting whichever was built first. The only visible change is that the auto-enable log line can print once per cache, which is fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants