server: on-disk KV cache (--kv-cache-dir) — restore a conversation instead of re-prefilling it

[dzannotti]

Prefill on a Strix Halo costs minutes where the same KV state reads back from NVMe in seconds, so it is worth spending disk to never prefill a conversation twice.

Cherry-picked onto this fork's master rather than PR'd from an upstream-based branch, so the diff is the feature only. Compile-checked against this base.

Keying

An exact sha1 of the arriving conversation can never hit — every request carries one more message than anything already stored. So the lookup hashes the conversation truncated at its last assistant message, which is byte-identical to what the previous turn saved:

turn 1 [sys, u1]            -> no assistant message, no lookup
                               generate A1, store under key([sys, u1, A1])
turn 2 [sys, u1, A1, u2]    -> truncate -> key([sys, u1, A1])  HIT
                               generate A2, store under key([sys, u1, A1, u2, A2])

One exact key, an O(1) map lookup, no resident token index and no longest-common-prefix scan. Keys are built at the chat route where the messages are still visible and travel to the slot on the task. The reply is hashed as the client will echo it back (parsed content, reasoning stripped). Keys are bound to the weights path + arch description rather than --alias, so renaming an alias does not invalidate a cache and two models sharing one cannot read each other's state.

An entry

<key>.kv     target context state (llama_state_seq_save_file, carries the token list)
<key>.dft    draft context state, when speculative decoding is on
<key>.ckpt   context checkpoints
<key>.idx    index record, written last and deleted first, so it is the commit marker

.kv/.dft are written under .tmp and renamed; .idx goes last. A crash mid-write leaves a .tmp that startup sweeps; eviction deletes .idx first.

Checkpoints are not optional on SWA models

Without .ckpt this feature is worse than no cache. llama_state_seq_save_file only persists the SWA window, so after a restore llama_memory_seq_pos_min sits above pos_min_thold and update_slots will only reuse the prefix if a context checkpoint reaches further back. Measured on gemma-4-26B-A4B, turn 2:

	cache_n	prompt_n
cache off	1224	13
cache on, no checkpoints stored	0	1237
cache on, checkpoints stored	1219	18

Same reason the in-memory tier works — alloc() copies prompt.checkpoints alongside the blob.

Eviction

Runs before every lookup, so an expired conversation is never resurrected. Age first (--kv-cache-ttl, default 3 days), then least-recently-used against --kv-cache-max.

Flags

• --kv-cache-dir PATH — enable (default off)
• --kv-cache-max N — MiB, LRU budget, 0 = no limit
• --kv-cache-ttl N — seconds, default 259200
• --kv-cache-min-tokens N — default 256

--cache-ram 0 previously left no cache object at all; it now disables only the in-memory tier, and alloc() returns early rather than reading limit_size 0 as "no limit".

Measured

4 conversations x 2 turns interleaved 1 2 3 4 / 1 2 4 3 through 2 slots of 262144, kv_unified=false, --cache-ram 0, gfx1151/ROCm 10:

	control	with cache
turn-2 tokens presented	7,480	7,480
served from cache	0 (0%)	7,338 (98.1%)
prefilled	7,480	142

Restore measured at 47 ms for a 22.9k-token conversation vs 18.4 s to re-prefill; ~1.9 s vs ~3.5 min for a full 262k slot.

Cross-contamination: 8/8, covering four conversations whose system prompts differ by a single token, and four that are byte-identical after turn 1 (so they collapse onto one shared entry) then diverge. No conversation ever produced another's secret.

[dzannotti]

putting it as draft as i want to run a couple more tests on 27b and flash also

[dzannotti]

putting this back as open, i've verified against gemma4, ornith 1.5, qwen 3.8 flash and 3.8 27B and all displayed a reduced number of tokens being prefilled

[dzannotti]

Pushed a fourth commit: do not restore a conversation the slot already holds.

disk_load restored unconditionally whenever the key was found on disk, without checking what the slot already contained. On a returning turn that lands back on the slot holding its own state — which --slot-prompt-similarity makes likely — that reads hundreds of MiB to install a byte-identical copy, and discards the checkpoints built since. On a model with ~2 GiB entries that is 2 GiB read to change nothing.

The fix tracks the conversation key whose KV a slot currently holds, sets it on both store and restore, clears it in prompt_clear(), and skips the restore when it matches. The key stored after generation is exactly the key the next turn looks itself up by, so the comparison is direct rather than heuristic.

Both directions of a stale key are safe: a missed skip costs one needless read; a missed restore falls back to the ordinary common-prefix path and prefills more. Neither can produce wrong output.

Why it was not caught earlier: the verification workload is 4 conversations x 2 turns interleaved 1 2 3 4 / 1 2 4 3 through 2 slots, which evicts between every turn by design — so the "already resident" path was never exercised. Added a 1,2,1,2 case where both conversations stay resident.

Verified on gemma-4-26B-A4B, -np 2:

kv cache: restored  ->  0        (2 before the fix)
turn-2 cached       ->  98.7%    (3,667 of 3,717; 50 prefilled)

Zero disk reads, and reuse is unaffected because the ordinary prefix logic handles the resident case.

Compile-checked against this fork's base, as with the other three commits.