Skip to content

server: do not propagate --kv-mean-center bias to draft model contexts#81

Open
cnndabbler wants to merge 1 commit into
PrismML-Eng:prismfrom
cnndabbler:fix/kvmc-draft-ctx
Open

server: do not propagate --kv-mean-center bias to draft model contexts#81
cnndabbler wants to merge 1 commit into
PrismML-Eng:prismfrom
cnndabbler:fix/kvmc-draft-ctx

Conversation

@cnndabbler

Copy link
Copy Markdown

Problem

--kv-mean-center sets common_params.kv_mean_center_path globally. The server builds draft-model params by copying the whole struct (auto params_dft = params_base;), so when a draft model is loaded via -md, both the draft VRAM-measurement probe and the real draft context inherit the bias path. The bias GGUF is calibrated for the target model's K geometry, so draft context creation fails:

  • with default f16 draft KV: llama_init_from_model: path_kv_mean_center requires the K cache type to be Q4_0 (got f16)
  • with -ctkd q4_0: load_kv_mean_center: bias tensor kv_bar.blk.3.k has 1024 elements, expected 512 (n_embd_head_k * n_head_kv)

Either way the probe reports [spec] failed to measure draft model memory and speculative decoding is silently disabled (common_speculative_init: no implementations specified) — making --kv-mean-center and -md mutually exclusive in practice.

Fix

Clear kv_mean_center_path on the copied draft params at both construction sites in tools/server/server-context.cpp (the VRAM-measurement probe and the real draft context), following the existing pattern of draft-specific overrides (cache_type_k/cache_type_v). The main context keeps the bias.

Coverage note: the only consumption site is common_context_params_to_llama (common/common.cpp), and these are the only two non-main-context derivations of common_params in the server for the -md path (the MTP branch shares the target model and is untouched).

Verified

RTX A5000, Ternary-Bonsai-27B Q2_0 + dspark drafter + --kv-mean-center + q4_0 KV (-ctkd/-ctvd q4_0), based on 62061f9:

  • bias loads for the main model only, no load_kv_mean_center errors
  • common_speculative_impl_draft_dspark: adding speculative implementation 'draft-dspark' — spec active alongside the bias
  • greedy decode ~59 tok/s with drafter vs ~48 AR; BFCL simple_python 95.0% with drafter vs 95.25% without (quality unaffected)

🤖 Generated with Claude Code

The bias GGUF is calibrated for the target model's K geometry. When a
draft model is loaded (-md), copying common_params wholesale made both
the VRAM-measurement probe and the real draft context inherit
kv_mean_center_path, failing with 'bias tensor ... has 1024 elements,
expected 512' and disabling speculative decoding. Clear the path on the
draft params in both sites so bias + drafter can coexist.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prevents target-model KV mean-centering bias from being applied to incompatible draft-model contexts.

Changes:

  • Clears the inherited bias path during draft VRAM measurement.
  • Clears it when creating the actual draft context.
  • Preserves bias behavior for the target and MTP contexts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@khosravipasha
khosravipasha requested a review from bri-prism July 18, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants