Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena

[giveen]

Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena

Summary

Adds experimental block KV cache streaming, ported and generalized from
Raymond Huang's feature/kv-stream-phase-arena
(https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming). His
implementation is Qwen3.5-only; this generalizes the same shared CUDA
phase-arena design to any architecture with a standard, uniform-per-layer KV
cache (plain dense/MoE, llama_memory_hybrid, iSWA), wires it into
TurboQuant's native direct_attention kernels (turbo2/3/4), and validates it
against real local models rather than by inspection.

With streaming on, only a resident subset of KV pages plus the active
compute workspace live in one bounded CUDA allocation; the rest of the KV
cache lives in host RAM and streams on demand. Enabled with
--kv-stream-arena-mib N; 0 (default) disables it and behavior is
unchanged.

All raw evidence (results.csv/.jsonl for every sweep and llama-perplexity
logs for every correctness check - every number in the tables below comes
straight from these) is committed at
benchmarks/results/kv-stream-pr-evidence/
(excludes the multi-GB saved-logits .kld files - regenerable with
--save-all-logits if needed).

What's new since the port

• Wired turbo2/3/4 into the native direct_attention path (measured ~2x
  prefill speedup on Qwen3.8-27B: ~1390 -> ~2930 t/s). This wiring benefits
  ordinary non-streamed attention unconditionally, but streamed
  attention only takes the direct_attention path when ggml-cuda is built
  with GGML_CUDA_FA_ALL_QUANTS (off by default) - without it, every
  streamed KV pair falls back to a slower F16-dequant path (measured
  ~2840 -> ~1200 t/s streamed prefill at 8K context, same model/types).
  See "Attention dispatch: direct vs F16 fallback" in docs/kv-stream.md.
• Fixed a dormant bug where process_ubatch() never switched the phase
  arena into decode layout for any plain (non-hybrid) llama_kv_cache -
  every model tested had been running decode with the prefill layout for
  its entire lifetime. Fixing it dropped streaming-vs-non-streaming mean
  KL-divergence from 0.003 to ~0.000000.
• Generalized streaming-target discovery off a fixed set of dynamic_casts
  to llama_memory_i::get_kv_stream_targets(), so an unsupported memory
  type fails closed with a clear error instead of a hard crash.
• Extended support to iSWA (dual full + sliding-window cache -
  Gemma-family, this fork's own laguna arch): the full-attention
  (kv_base) sub-cache streams, the sliding-window sub-cache stays
  resident (small by construction).
• Attempted DSV4 (DeepSeek-V4) support; found a real, un-root-caused
  correctness regression (mean KLD 0.07 vs ~0.000000 for every other
  architecture) and reverted it rather than ship it. DSA/MSA/MLA remain
  excluded or unverified for the same reason - see
  docs/kv-stream.md.
• Fixed two bugs found via code review, both with regression test coverage:
  • --swa-full silently conflicted with streaming (both iSWA sub-caches
    tried to bind the same single-lease arena); now rejected explicitly
    with a named reason instead of a generic construction failure.
  • A no_alloc dry-run context (used by -fit's auto-probe and by MTP's
    parent-memory-sizing pass) built a real CUDA arena anyway, then failed
    with a confusing "requires a memory type with a streamable KV cache"
    error - benign (the real context loads fine moments later) but
    alarming. Now skipped cleanly for no_alloc contexts.
• Added docs/kv-stream.md: requirements, architecture support matrix, why
  single-sequence-only is a design limitation and not a validation gate,
  arena sizing guidance, and a known-inefficiency note (see below).
• Added benchmarks/benchmark_kv_memory.py: sweeps context length and
  records GPU VRAM + host process RSS, with a same-context no-streaming
  baseline at every point (including past the point where the baseline
  OOMs), and plots both. Unlike the throughput sweep it needs no
  prefill/decode pass, so it's cheap enough to cover much wider ranges.

Architecture support

Shape	Support
Plain dense/MoE (one llama_kv_cache)	Yes
llama_memory_hybrid (recurrent + attention, e.g. Qwen3.5)	Yes, on the attention sub-cache
iSWA (dual full + sliding-window, Gemma-family / laguna)	Yes, on the full-attention sub-cache
MLA (DeepSeek-V2/V3-style)	Unverified - not excluded by architecture, but never tested against a real model
DSA, DSV4, MSA	Excluded - DSV4 has a proven correctness regression; DSA/MSA never attempted given that finding

Benchmarks

All runs: single RTX 5090-class GPU (32 GiB), -ctk q8_0 -ctv turbo4 -fa on --parallel 1, fixed 8192 MiB shared arena, ggml-cuda built with
GGML_CUDA_FA_ALL_QUANTS=ON, via benchmarks/benchmark_kv_stream.py
(prompt = context - 256 tokens, decode 256 tokens, prompt not cached). A
default build (flag off) gets materially lower streamed prefill throughput
at the same context/arena size - see "Attention dispatch: direct vs F16
fallback" in docs/kv-stream.md.

Qwen3.8-27B (dense/hybrid, no speculative decoding)

17 points, 8K -> 262K context. Smooth, continuous decline in both prefill
and decode throughput as context grows - no cliff, no crash, right through
the model's native 262K training context.

Context	Prefill t/s	Decode t/s	VRAM used
8K	2759	59.5	28.1 GiB
25K	2605	52.6	28.2 GiB
41K	2396	47.3	28.3 GiB
57K	2156	42.8	28.3 GiB
74K	1943	39.6	28.4 GiB
90K	1773	36.4	28.5 GiB
106K	1623	33.6	28.5 GiB
123K	1505	31.5	28.6 GiB
139K	1407	29.7	28.6 GiB
156K	1314	27.7	28.7 GiB
172K	1242	26.3	28.8 GiB
188K	1171	24.8	28.8 GiB
205K	1115	23.7	28.9 GiB
221K	1063	22.6	29.0 GiB
238K	1014	21.6	29.0 GiB
254K	967	20.8	29.1 GiB
262K	946	20.5	29.1 GiB

VRAM grows only ~1 GiB (28.1 -> 29.1 GiB, ~3.5%) across a 32x increase in
context (8K -> 262K) - the fixed 8192 MiB arena plus a small amount of
bookkeeping overhead that scales with layer/page counts, not with total
context. Full per-point data:
qwen3.8-27b-throughput/results.csv.

Correctness (streaming vs non-streaming), this checkpoint:

Context	PPL (baseline)	PPL (streaming)	Mean KLD	Max KLD	Same top-token
4K	5.7131	5.7275	-0.000000	0.000057	100.000%
32K	6.0108	6.0124	0.000000	0.000070	99.988%

Mean KLD is at the numerical noise floor at both scales. 32K is the more
meaningful check: host RSS measurements (see "VRAM and host RAM" below)
confirm real page streaming is active by that point, not just a
fully-resident no-op. Full methodology and a larger-scale attempt (and the
tooling limitation it hit) are in "Correctness: PPL and KL-divergence, at
increasing scale" below. Raw logs:
kld-logs/.

Qwen3.8-27B + MTP speculative decoding (--spec-type draft-mtp --spec-chain 8)

5 points, 8K -> 106K context. Streaming coexists cleanly with MTP on the
target context (MTP's own draft context never streams - see docs). Draft
acceptance on this benchmark's synthetic repeated-filler prompt was
measured at ~100%, so these decode numbers are a best-case ceiling, not a
typical-prompt expectation - treat the shape of the curve as
representative, not the absolute numbers.

Context	Prefill t/s	Decode t/s (MTP)	VRAM used
8K	2595	294.5	28.6 GiB
33K	2422	260.7	28.8 GiB
57K	2094	227.5	29.0 GiB
82K	1786	213.9	29.2 GiB
106K	1568	191.6	29.5 GiB

Full per-point data:
qwen3.8-27b-mtp-throughput/results.csv.

Required a small fix to benchmarks/benchmark_kv_stream.py: its exact
prompt/decode token-count check had never been run against speculative
decoding before, and MTP's chunked verification (up to spec_chain+1
tokens per step) can clip the last chunk a few tokens short of the target
right at the context boundary. Added a bounded tolerance derived from
--spec-chain; the reported decode_tps is unaffected either way (it's
the server's own measured rate, not derived from the target count).

Gemma-4-26B-A4B-It (MoE, iSWA)

6 points, 8K -> 131K context, using the proper instruction-tuned checkpoint
(gemma-4-26B-A4B-it-UD-Q5_K_S, Unsloth dynamic quant - an earlier pass
used a community "Uncensored" merge; see the correctness section below for
why it was swapped). Prefill is much faster than Qwen (MoE, ~4B active
params/token vs Qwen's dense 27B). Decode falls off far more steeply than
Qwen's dense/hybrid curve.

Context	Prefill t/s	Decode t/s	VRAM used
8K	9674	71.0	27.8 GiB
32K	6007	23.0	27.9 GiB
57K	4026	13.5	27.9 GiB
82K	2923	9.5	28.1 GiB
106K	2502	7.5	28.1 GiB
131K	2073	6.0	28.2 GiB

Full per-point data:
gemma4-it-throughput/results.csv.

Numbers are within noise of the earlier (worse-quant) checkpoint's run,
which rules out quantization/merge quality as the cause of the steep decode
falloff - it's reproduced almost exactly across two unrelated checkpoints of
the same architecture. VRAM is flat (27.8 -> 28.2 GiB) across the whole
16x context range - same arena-boundedness result as Qwen, on a
structurally different (MoE + iSWA) architecture. Streaming itself is
stable (no crash, no errors) through the whole sweep; the falloff looks
architectural (MoE expert routing and/or the mixed full+SWA cache
structure) rather than a streaming-specific regression, but hasn't been
root-caused against a non-streaming Gemma4 baseline - flagging as an open
question rather than asserting a cause.

VRAM and host RAM: where the data actually goes, and what happens without streaming

benchmarks/benchmark_kv_memory.py (new in this PR - see "What's new")
sweeps context length and records GPU VRAM and host process RSS, with a
same-context no-streaming run alongside the streaming one at every point.
It's cheap (no prefill/decode needed - the buffers are sized at context
construction) so it can cover a much wider range than the throughput sweep.

Qwen3.8-27B-AD, 131K -> 786K context (YaRN-scaled past the 262K
training context, 4x, --rope-scale 4 --yarn-orig-ctx 262144):

Context	Streaming VRAM	No-streaming VRAM	Streaming RSS	No-streaming RSS
131K	28.3 GiB	24.1 GiB	4.4 GiB	1.2 GiB
262K	28.3 GiB	27.7 GiB	7.7 GiB	1.3 GiB
393K	28.2 GiB	CUDA OOM (crash)	10.9 GiB	-
524K	28.2 GiB	not attempted (already OOM'd at 393K)	14.1 GiB	-
655K	28.3 GiB	not attempted (already OOM'd at 393K)	17.3 GiB	-
786K	28.2 GiB	not attempted (already OOM'd at 393K)	20.6 GiB	-

No-streaming hits a real CUDA error: out of memory abort at 393K (full
gdb-style backtrace:
qwen3.8-27b-memory/logs/baseline-393216.log -
inside the server's own warmup decode, not a clean pre-flight allocation
failure, an in-flight kernel abort, matching the "hard process abort near
the VRAM ceiling" failure mode in docs/kv-stream.md). Streaming keeps
going another 6x further (to 786K, the max tested) with VRAM pinned to
~28.3 GiB throughout, while host RSS absorbs the growth (4.4 -> 20.6 GiB).
Full per-point data:
qwen3.8-27b-memory/results.csv.

Gemma-4-26B-A4B-It, 65K -> 393K context:

Context	Streaming VRAM	No-streaming VRAM	Streaming RSS	No-streaming RSS
66K	28.1 GiB	20.9 GiB	1.8 GiB	1.3 GiB
131K	28.1 GiB	21.7 GiB	2.3 GiB	1.3 GiB
197K	28.1 GiB	22.4 GiB	2.9 GiB	1.4 GiB
262K	28.1 GiB	23.2 GiB	3.4 GiB	1.4 GiB
328K	28.1 GiB	24.0 GiB	3.9 GiB	1.4 GiB
393K	28.1 GiB	24.8 GiB	4.5 GiB	1.5 GiB

No-streaming never hit its ceiling in this range (21.4 -> 25.4 GiB, still
under budget at 393K) - most of Gemma4's layers are small fixed-window SWA
attention that doesn't grow with context, so its non-streaming footprint
grows much more slowly than Qwen's dense/hybrid cache. Streaming stays flat
at ~28 GiB across the same range. Not every architecture needs streaming at
the same context threshold, but streaming costs nothing extra either way.
Full per-point data:
gemma4-it-memory/results.csv.

Correctness: PPL and KL-divergence, at increasing scale

A short check is a weak check here. The mechanism this PR is actually
about - page residency, eviction, and ring-buffer streaming - only
activates once the resident window is a real fraction of the total
context. An early pass validated at -c 4096 against an 8192 MiB arena,
which likely fits entirely resident with little real eviction exercised.
Repeated at increasing scale, on the exact checkpoints benchmarked above
(llama-perplexity, wikitext-2-raw, non-streaming logits saved via
--kl-divergence-base, compared against a streaming
--kv-stream-arena-mib 8192 run of the same chunks):

Model	Context	PPL (baseline)	PPL (streaming)	Mean KLD	Max KLD	Same top-token
Qwen3.8-27B-AD	4K	5.7131	5.7275	-0.000000	0.000057	100.000%
Qwen3.8-27B-AD	32K	6.0108	6.0124	0.000000	0.000070	99.988%
Gemma-4-26B-A4B-It	4K	8067.36	16607.82	-0.000000	0.000056	100.000%

Mean KLD stays at the numerical noise floor at every scale tested, and
same-top-token stays at ~100% - streaming reproduces each model's own
output distribution, at a context large enough (32K) that host RSS
measurements above confirm real page streaming is active, not just a
fully-resident no-op. A 131K attempt on Qwen hit a tooling limitation in
llama-perplexity's own KL-divergence machinery (failed reading log-probs for chunk 0 against a 27 GB saved-logits file, failing immediately rather
than partway through - looks like a pre-existing limit unrelated to
streaming, not something this PR introduced) rather than a real finding
either way; flagging as an open item rather than a validated result at
that scale. Raw log:
kld-logs/qwen-streaming-131k-FAILED-tooling-limitation.log.

Gemma4-It's raw PPL is unusually high on wikitext for both runs equally
(confirmed coherent in normal chat use - see
kld-logs/gemma4-it-coherence-check.log)

• almost certainly a heavily instruction/reasoning-tuned model scoring
  poorly on raw non-chat continuation, a known effect, not a fork bug. Ruled
  out turbo K/V quantization and Flash Attention as causes by reproducing the
  same result with plain f16 KV cache and with -fa off (see
  kld-logs/gemma4-it-f16-check.log and kld-logs/gemma4-it-nofa-check.log).
  KLD - the metric that actually measures streaming fidelity, independent of
  whether the base distribution is "good" at modeling wikitext - shows no
  divergence regardless.

Gemma4 + MTP: not applicable

Confirmed on two independent checkpoints (the community merge and the
official instruct release) - neither has nextn tensors (checked via
direct GGUF metadata read), so --spec-type draft-mtp fails outright on
both: context type MTP requested but model doesn't contain MTP layers.
This is an architecture-wide limitation (the Gemma4 family wasn't trained
with an MTP/NextN head in these public releases), not a streaming
limitation or a quant/merge quality issue. No streaming-specific finding
here either way.

Known limitations (see docs/kv-stream.md for full detail)

• Single-sequence only (-np 1). Not a conservative gate - the
  resident-page design indexes pages by absolute buffer offset, the decode
  fast path assumes exactly one query token, and the adaptive
  repartitioning loop tracks one global counter. Real multi-sequence
  support needs a sequence dimension added to all three; not attempted.
• MTP's own draft-context cache never streams (explicitly zeroed in
  common/speculative.cpp - the draft/target contexts don't yet share one
  pool). Streaming on the target context is unaffected and verified
  working with MTP active (see benchmarks above).
• MTP verification batches use the prompt-phase arena layout, not the
  generation layout (server never calls llama_set_decode_phase()).
  Measured: 89 of 90 decode steps in a real MTP+streaming session ran in
  the prompt-phase layout (1246 resident pages/layer, 10 ring slots)
  instead of the generation layout (1273 resident pages/layer, 12 ring
  slots) - a real but modest gap at this context size. Fixing it requires
  resizing the generation-phase compute reservation for multi-token
  speculative batches, not just relaxing the existing TG1-only guard.
  Documented, not fixed, to avoid touching stable repartitioning logic for
  an unmeasured-at-scale win.
• No multi-GPU tensor-split streaming. Every eligible layer must be on
  one CUDA device today.

Testing

• test-kv-stream-config: 11 tests / 47 assertions, 0 failures.
• test-arg-parser: full suite passes, including new coverage for
  --kv-stream-arena-mib/--kv-stream-stage-mib (legacy alias), negative
  rejection, and 0-disables-cleanly.
• KL-divergence validation (streaming vs non-streaming,
  llama-perplexity --kl-divergence-base / --kl-divergence) on every
  supported architecture: mean KLD ~0.000000, >99% same-top-token for
  turbo/iSWA. DSV4 was the one architecture that failed this bar (0.07
  mean KLD) and is excluded as a result. Re-verified against the exact
  Qwen3.8-27B-AD and Gemma-4-26B-A4B-It checkpoints used in the benchmarks
  above at increasing context (4K and 32K for Qwen, 4K for Gemma4) - see
  "Correctness: PPL and KL-divergence, at increasing scale" - mean KLD at
  the noise floor at every scale tested, confirmed at a context (32K)
  large enough that host RSS growth shows real page streaming is active.
• benchmark_kv_memory.py (new): VRAM stays flat while no-streaming grows
  until a real CUDA OOM abort at 393K context (Qwen3.8-27B) - streaming
  ran 6x further with no VRAM growth. Full backtrace in
  qwen3.8-27b-memory/logs/baseline-393216.log.
• Live production smoke test: real llama-server invocation (mmproj +
  MTP + -c 226000 + streaming) served real chat completions with
  measured MTP draft acceptance for 2+ hours without incident.

Credits

Original shared-phase-arena design and its Python benchmark tooling by
Raymond Huang (https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming).
This PR generalizes it beyond Qwen3.5 to TurboQuant's supported
architectures and turbo K/V quantization types, and adds the fixes and
validation described above.

🤖 Generated with Claude Code

[giveen]

@TheTom that took a while but I think I am done.

[TheTom]

Read the whole thing at b08b515, src side and CUDA side separately, with flag-off as the deciding question. The good news first: with --kv-stream-arena-mib unset the FA kernels really do fold back to their old bodies (the output_partial template param and if constexpr leave the existing instantiations byte-identical, stream_k stays true), the refusals (single sequence, --swa-full, -fa off, no kv offload, draft contexts, DSA/DSV4/MSA/recurrent) are centralised in llama_kv_stream_config_validate and throw from the constructor, the copy-stream / event ordering between H2D pages and the partial FA kernels is right, and CUDA graph capture is disabled for any node touching a runtime. Reverting the DSV4 geometry change instead of shipping a 0.07 KLD was the right call. This is a much better base than #326.

Needs changes before it can land:

1. Rebase. It re-includes perplexity: respect explicit -np instead of silently overriding it #353 (-np in perplexity) and kv-cache: resolve TURBO_LAYER_ADAPTIVE mode per cache construction #354 (per-construction TURBO_LAYER_ADAPTIVE) verbatim; both merged yesterday. It is also one commit behind 4a54c52 (the TQ MMQ arch-config fallback), so the diff currently shows that fix as deleted. Rebase and I will merge with a merge commit, not a squash.

2. Per-layer mixed KV types are neither modelled nor refused. Page bytes are computed once from the cache-level type_k / type_v (llama-kv-cache.cpp around line 436) and kv_stream_buft is assigned to every layer, but the loop from line 502 can give individual layers q8_0, turbo4 or turbo2 under TURBO_LAYER_ADAPTIVE, and the turbo 128-padding then changes bytes per row inside a pool that assumes uniform pages. Mode 7 auto-enables with no env var whenever type_v == turbo2 and n_layer >= 8, so -ctv turbo2 --kv-stream-arena-mib N hits this silently. Every number in the body is -ctv turbo4. Either refuse streaming when kv_adaptive_mode != 0, or compute geometry per layer. A KLD run at -ctv turbo2 would show which.

3. test-kv-stream-cuda-attn fails on a default CUDA build. Line 550 asserts ATTENTION_DIRECT for every turbo pair, but that mode only exists under #ifdef GGML_CUDA_FA_ALL_QUANTS (fattn.cu around 1170), which is OFF by default (ggml/CMakeLists.txt:208). The test is registered under a bare if (GGML_CUDA). Guard the assertion or the registration. Separately, test-kv-stream-cuda-buffer allocates on device 0 with no device probe, so a GPU-less CUDA build goes red; skip cleanly when no device is present.

4. The "~2x prefill via direct_attention" headline is GGML_CUDA_FA_ALL_QUANTS=ON only. Default builds take the F16 dequant-to-workspace path. The body, docs/kv-stream.md and the benchmark tables should name the flag, and a default-build number belongs next to the FA_ALL_QUANTS one.

5. Two flag-off costs worth removing. ggml_cuda_graph_update_required now scans every node's kv_stream_generation before the cgraph->uid == graph->uid early-out, on every decode step, for everyone; hoist it behind an "any live runtime" check. set-rows.cu gained a runtime dst_row_base kernel argument on every SET_ROWS launch; a template parameter keeps the off path bit-identical. Smaller: process_ubatch builds a target vector per ubatch just to test emptiness.

Concerns, none individually blocking:

• State save/load, seq_rm, seq_cp and the K-shift path are untouched and untested with a streamed cache, and nothing refuses llama_state_* with the arena on. Either a stated refusal or a test.
• src/llama-kv-stream-softmax.{cpp,h} has no production caller; it is a CPU model of the CUDA fixup math compiled into libllama. Move it under tests/.
• The DSV4 plumbing you reverted the geometry for is still in the tree as unreachable code. Delete it and keep the exclusion.
• GGML_CUDA_PREFER_KV_HOST and GGML_CUDA_PREFER_MODEL_WEIGHTS are new env knobs that only act when streaming is off, unrelated to this feature and undocumented. Separate PR or drop.
• Head-dim-128 models still get the write-combined pinned buffer and then fall back to plain FA reading KV over PCIe every step. Correct but slow and not in the requirements list.
• Streamed output is close, not bit-identical (sequential online-softmax rescale across chunks vs flash_attn_combine_results), and the span tuner picks bounded vs unbounded from wall-clock timing, so a given step's kernel choice is timing-dependent. Both fine, both should be stated in the doc. kv_stream_normalize_chunk_results has no zero guard on the denominator.
• docs/kv-stream.md says a too-tight arena can SIGABRT a serving server mid-request. For an in-tree flag that needs to become an error return.
• kv_stream_arena_mib is inserted mid-struct in llama_context_params; append it so prebuilt consumers keep their offsets.
• The NDEBUG asserts in evaluate_and_capture went from buft == cuda_buffer_type(device) to buft_is_cuda, which drops the device identity check.
• GGML_ASSERT wrapping the dirty-row marking call (llama-kv-cache.cpp ~2150) puts a side effect inside an assert.
• 795 KB of PNGs under benchmarks/results. Charts belong in the PR body as uploaded images; the CSV/JSONL is defensible.

I'll run the GB10 gates (default build flag-off sweeps, FA_ALL_QUANTS build with -Werror, one streaming-on KLD at -ctv turbo4 and one at -ctv turbo2) as soon as that box has disk again; it is at 100% right now.

[TheTom]

Changes requested per the comment above: rebase off #353/#354, handle or refuse TURBO_LAYER_ADAPTIVE with streaming, fix the two CUDA tests that fail on default or GPU-less builds, and name the FA_ALL_QUANTS dependency on the headline number.

[TheTom]

One more from CI, and it is not the runner flake this time. The Windows arm64 cmake-pkg job fails at link:

FAILED: bin/Release/test-kv-stream-plan.exe
lld-link: error: undefined symbol: llama_kv_stream_plan_make(llama_kv_stream_plan_params const &)
lld-link: error: undefined symbol: llama_kv_stream_prefetch_dispatch(...)
lld-link: error: undefined symbol: llama_kv_stream_partition_adapt(...)
lld-link: error: undefined symbol: llama_kv_stream_feedback_delta_make(...)
lld-link: error: undefined symbol: llama_kv_stream_extent_make(...)

Those are internal libllama functions with no LLAMA_API export, and that job links the tests against the shared llama.dll, so the test cannot see them. The other three Windows legs were cancelled by fail-fast, so they would likely hit the same thing. Either compile the plan/softmax/config sources into the test target directly (the way test-kv-stream-softmax presumably needs to as well), or mark the test as static-only.

[tmark00]

Tried this on a 24 GB laptop card, which is roughly the situation the feature targets. The VRAM
saving is real, but prefill becomes quadratic in prompt length with the arena on, and — the part I
cannot explain — a larger arena makes it dramatically worse.

Setup: RTX 5090 Laptop (sm_120), driver 616.56, Windows 11, MSVC 14.51, branch at c8c35c39a,
unsloth/Qwen3.8-27B-GGUF Q5_K_M (qwen35, hybrid GDN, 24 of 65 layers hold KV), -c 131072,
-fa on, -np 1, -b 2048 -ub 512, q8_0/turbo3 KV, MTP adaptive drafting, -fit off. Single
38503-token prompt, 300 generated tokens, GPU idle (VRAM 0) before every run.

Totals

config	prefill	decode	VRAM
arena off	1163 t/s	49.0 t/s	23480 MB
--kv-stream-arena-mib 1024	156.6 t/s	15.5 t/s	21116 MB
--kv-stream-arena-mib 2048	155.6 t/s	18.7 t/s	22291 MB
--kv-stream-arena-mib 4096	did not finish	—	—

Off-arena figures are the mean of three clean runs (1167.95/1160.78/1160.50 prefill,
49.21/48.91/48.83 decode, VRAM identical at 23480 MB every time); arena-1024 likewise
(158.43/155.80/155.53, 15.50/15.67/15.42).

Prefill is quadratic with the arena on

Cumulative rate from the server's own prompt processing lines:

n_tokens	off	arena 1024	arena 2048	arena 4096
4138	1319	700	664	—
8234	1344	513	486	—
12330	1340	401	380	51.6
18474	1335	301	286	36.1
26666	—	—	213	25.7
34858	—	—	—	20.0

Off-arena stays flat at ~1340 t/s, i.e. linear prefill. With the arena the cumulative rate falls
monotonically. Per 2048-token chunk, arena 2048 goes 663 → 430 → 344 → 286 → 245 → 219 → 189 → 170
→ 154 → 141 → 130 → 121 t/s: chunk time grows linearly with position, so total prefill is O(n²).
Off-arena chunk time is constant (1.48–1.58 s per chunk throughout).

A bigger arena is much slower

This is the part that looks wrong rather than merely expensive. At the same 12330-token mark:

• --kv-stream-arena-mib 2048: 32.45 s
• --kv-stream-arena-mib 4096: 239.20 s

7.4x slower with twice the resident capacity. The 4096 run was still at 91% of the prompt
(34858/38503) after 29 minutes and I aborted it at the 1800 s client timeout. If more resident
pages mean more per-chunk work, that inverts the intended trade-off, and it is what makes the
feature hard to tune from outside: 1024 and 2048 behave nearly identically, then 4096 falls off a
cliff.

For completeness, the arena has a lower bound too — -c 262144 with --kv-stream-arena-mib 1024
refuses at startup with prefill phase does not fit shared CUDA arena: arena has no KV capacity after reserving compute space, and needs 2048 to start. So the usable window here is narrow.

Note on the baseline

On this model -c 262144 already loads without the arena (24049 MB of 24463), so I could not use
"fits at all" as the test. The KV is small because only 24 of 65 layers have a cache and turbo3 V
is compact — the ceiling here is model weights, not KV. That may be why my numbers look different
from a dense-model baseline.

What I did not check

Whether output stays correct under streaming — I compared throughput only. Also single arena
sizes, one prompt length, and one model; no sweep over -ub, which plausibly interacts with the
compute reservation inside the arena.

🤖 Generated with Claude Code