Context
The KVarN implementation loop (#180, branch feat/180-kvarn, commits 82efc4f..99c3773) concluded with the method proven working as advertised on this hardware: accuracy at FP16 parity (PPL +1.0-1.9%, math eval identical to fp32), 7.7x granted context on Qwen3-8B in 8 GB, decode >= fp32 on the 8B class, prefill 7.7-13x via chunked batching. Remaining headroom items, in priority order, with acceptance baselines calibrated on the RTX 4070 Laptop (8 GB, sm_89):
1. int8 tensor-core K-score GEMM for chunked prefill (top priority)
At 15.5k context the headline run (15.5k prompt + 300 decode = 98 s) is still prefill-dominated (~76 s). llm_kvarn_prefill_attention is bounded by the scalar nibble-decode K-score/V-agg loops: 203 t/s at 15.5k vs the fp32 flash class (~700 t/s). A tensor-core int8 GEMM against the 4-bit channel-major tile format would cut the headline to ~37 s.
Baselines to beat: 8B chunked prefill 460 t/s @4k, 203 t/s @15.5k (window 256). Tile byte-identity (re-chunking oracle in CudaKvarnPrefillTests) and the PPL gates (0.6B c=3072 15.60, 8B c=4096 8.99) must hold.
2. Split-tile KVarN decode attention (deferred twice, conditionally)
llm_kvarn_attention launches grid=numHeads blocks (16-32) on 36 SMs and walks tiles serially: 8B decode erodes ~0.9 t/s per 1k depth (30.2 @4k -> 20.0 @15.5k; still interactive, and fp32 has no presence at these depths in 8 GB). A split-tile two-kernel scheme (fixed worst-case grid + early-exit + LSE combine, mirroring AttentionSplitKv; graph-capturable with the existing 5b tracked-param machinery; apply the deferred inverse WHT after combining rotated tile partials) reclaims an estimated ~1/3 at 15.5k and 2-3x on small models at depth.
Decision gate per the 5a/5b reviews: measure 8B at 16-32k depth on the desktop; fund if the depth erosion inverts the fp32 advantage at depths KVarN exists to enable.
3. Minors
- Per-chunk logits skip in the batched prefill trunk (trivial; ~121 needless logits+download+sync points at 15.5k).
- head_dim 256 chunked prefill (48 KB static shared cap; needs the >48KB dynamic-shared opt-in or tiling) — currently falls back to per-token.
- f16 scales in the packed tile layout (~0.7 bits/elt saving; flagged in the Task-1 layout comment).
SharpInferenceServerOptions.TqMode (server currently exposes only the bool -> Lloyd-Max).- SnapKV x KVarN composition (Compact currently throws in KVarN mode by design).
- Vulkan port (research doc P3, optional).
- Multi-instance CUDA-graph id collision (two live CudaForwardPass on one backend would fight over ids -101/-102 — same pre-existing hazard as the dense single-graph path; no current call site composes this).
Pointers
🤖 Generated with Claude Code
https://claude.ai/code/session_018EWo1niax1g8A5E79RSALJ
Context
The KVarN implementation loop (#180, branch
feat/180-kvarn, commits 82efc4f..99c3773) concluded with the method proven working as advertised on this hardware: accuracy at FP16 parity (PPL +1.0-1.9%, math eval identical to fp32), 7.7x granted context on Qwen3-8B in 8 GB, decode >= fp32 on the 8B class, prefill 7.7-13x via chunked batching. Remaining headroom items, in priority order, with acceptance baselines calibrated on the RTX 4070 Laptop (8 GB, sm_89):1. int8 tensor-core K-score GEMM for chunked prefill (top priority)
At 15.5k context the headline run (15.5k prompt + 300 decode = 98 s) is still prefill-dominated (~76 s).
llm_kvarn_prefill_attentionis bounded by the scalar nibble-decode K-score/V-agg loops: 203 t/s at 15.5k vs the fp32 flash class (~700 t/s). A tensor-core int8 GEMM against the 4-bit channel-major tile format would cut the headline to ~37 s.Baselines to beat: 8B chunked prefill 460 t/s @4k, 203 t/s @15.5k (window 256). Tile byte-identity (re-chunking oracle in
CudaKvarnPrefillTests) and the PPL gates (0.6B c=3072 15.60, 8B c=4096 8.99) must hold.2. Split-tile KVarN decode attention (deferred twice, conditionally)
llm_kvarn_attentionlaunches grid=numHeads blocks (16-32) on 36 SMs and walks tiles serially: 8B decode erodes ~0.9 t/s per 1k depth (30.2 @4k -> 20.0 @15.5k; still interactive, and fp32 has no presence at these depths in 8 GB). A split-tile two-kernel scheme (fixed worst-case grid + early-exit + LSE combine, mirroring AttentionSplitKv; graph-capturable with the existing 5b tracked-param machinery; apply the deferred inverse WHT after combining rotated tile partials) reclaims an estimated ~1/3 at 15.5k and 2-3x on small models at depth.Decision gate per the 5a/5b reviews: measure 8B at 16-32k depth on the desktop; fund if the depth erosion inverts the fp32 advantage at depths KVarN exists to enable.
3. Minors
SharpInferenceServerOptions.TqMode(server currently exposes only the bool -> Lloyd-Max).Pointers
perplexityCLI command + scripts/kvarn-gate/ (wikitext fetcher, math eval)🤖 Generated with Claude Code
https://claude.ai/code/session_018EWo1niax1g8A5E79RSALJ