Docs: --kv-mean-center workflow for q4_0 K-cache (needed for 262k ctx on 16 GB cards)

[Astezelex]

Context: we ran what we believe is the first independent benchmark of Ternary Bonsai 27B on consumer hardware (RTX 5060 Ti 16 GB, fork commit b9591, CUDA build for sm_120, Linux). Overall results were good for the fork: ternary +24-31% decode vs a best-engine IQ2_XXS baseline, 262k context fits a 16 GB card with q4_0 KV, and the DSpark lossless-precision claim verified byte-identical Q4_1 vs bf16 (5/5 prompts at temp 0). Raw data and scripts: https://github.com/Astezelex/bonsai-27b-16gb-bench (also PrismML-Eng/Bonsai-demo#98). This report is friction we hit on the way.

The fork docs say a q4_0 K-cache "wants" a --kv-mean-center bias file for quality, but there is no guidance on generating or obtaining one per model, and no shipped file for Bonsai 27B. At 262k context (the fit headline for 16 GB cards) q4 KV is mandatory, so the quality delta matters.

Could you document the workflow (how to produce the bias file for a given model), or state that it is safe to omit for Bonsai 27B?

[khosravipasha]

Nice, this is impressive.

We did not have time to provide one, can you check if this guide is sufficient for now to unblock you? #54

Will take a look next week on our side to upload a sample centering gguf.

You can try other kv-cache quantization methods as well, this one is an experimental one that gave us early good results but its not fully integrated in hte llama.cpp fork yet, we will polish it more as we go forward.

[Astezelex]

Thanks, yes, #54 unblocks us. Our tested commit (b9591) already carries tools/kv-mean-center in the tree; we had only built the bench/cli/server targets, so it's just an extra build target plus the calibration run. One clarification for anyone cross-reading with PR PrismML-Eng/Bonsai-demo#98: our published 262k-fit number was measured with plain q4_0 KV, no centering bias, so it stands independent of this feature. A vendor sample centering GGUF for Bonsai 27B would be the ideal ending for the 16 GB story: at 262k the q4 K-cache is mandatory, so the bias file is effectively part of the deployment recipe. Happy to test the workflow on sm_120 and report back here.

[Astezelex]

Small addition useful for anyone landing here: the latest prebuilt release (prism-b9594-38c66ad) already ships llama-kv-mean-center in the linux-cuda tarball, so the #54 calibration workflow needs no source build. We'll report our Bonsai-27B calibration results on sm_120 here.