Skip to content

Centralize the KVarN support matrix + extend it to the server (TurboQuant head-dim gate) #437

Description

@pekkah

Context

Follow-up from #432 / #436 code review. The KVarN-vs-Lloyd-Max support matrix — which combinations of (SnapKV, head dim, backend, full-vs-partial offload, MoE, FP32 window) support KVarN — is currently hand-copied across three frontends:

  • RunCommand.Execute (inline kvarnBlocked ternary chain + a second explicit-kvarn validation block it must "mirror")
  • PerplexityCommand.ResolveAutoQuantizer (the only copy with unit tests)
  • InferenceEngineLoader.ResolveTq (local function + per-call-site reason strings)

The ground truth actually lives in Engine guards (CudaForwardPass KVarN head-dim ≤ 256, ForwardPass.EnableTurboQuant SnapKV throw, KVarNCompressor pow-2 [8,1024], TurboQuantCodebooks Lloyd-Max 128/256). The #432 warning text is copied 4×.

Why it matters

The partial-offload crash fixed in #436 (80c1445) was a direct symptom: the RunCommand copy downgraded auto-KVarN → Lloyd-Max on a head dim Lloyd-Max has no codebook for, because that copy didn't know the Lloyd-Max head-dim constraint. When the announced #180 follow-ups land (KVarN + SnapKV composition, KVarN for CUDA MoE / head dim > 256), each of the 3+ copies must change in lockstep; a missed site silently resolves auto → Lloyd-Max and reintroduces the exact #432 quality collapse on one surface only, with a now-wrong warning.

Proposed

  1. A single Engine-level helper next to TqQuantizer (e.g. static class TqSupport { ResolveAuto(headDim, isMoE, snapKv, backend, fullOffload, window, out reason); TryParseMode(...); }) that all three frontends call. Unit-test the matrix once.
  2. Extend it to the server: InferenceEngineLoader.cs:85 currently hard-rejects every head dim ∉ {128,256} whenever TurboQuant is on, before quantizer resolution — so KVarN's broader head-dim support (32/64/512/1024, which the CLI runs fine on the CPU path) is unreachable on the server, and explicit TqMode=kvarn on such a model gets a misleading "128 or 256" error. Relax the gate to the codec-aware envelope.
  3. Fold in the minor items from the fix(tq): default --tq-mode to auto — KVarN where supported, Lloyd-Max fallback with quality warning (#432) #436 review: the SnapKV-fallback warning that's only valid on CPU (misleading before the CUDA throw), and the server silently ignoring TqMode=kvarn when TurboQuant=false (the CLI rejects it).

Not blocking #436 — that PR is correct with the targeted guard; this is the deeper altitude fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EuBdnPNJ2XdaFp9PjYoJyz

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions