$\textcolor{red}{\textbf{(USER WAS BANNED FOR THIS POST)}}$
Summary
On a DeepSeek-V4-Flash model (DSV4, LLM_ARCH_DEEPSEEK4), using a quantized KV cache (--cache-type-k q8_0 --cache-type-v q8_0) causes a ~2.8x decode slowdown versus f16 KV, because the sparse-fa fused kernel (added in #27970 / 8e93a97) is only reachable through the mma_f16 path, which requires non-quantized K/V.
Environment
- llama.cpp:
67a17c1 (build 203)
- GPU: 4x RTX A6000 (Ampere SM86), tensor parallel (
-sm tensor)
- Model: DeepSeek-V4-Flash-0731, GGUF
Q8_K_XL
- Launch:
-c 1048576 -np 2 -sm tensor -fa on -ctk q8_0 -ctv q8_0
Measurements
Single-stream decode rate (marginal, measured over 100 vs 400 generated tokens via /completion):
| KV cache type |
decode |
| f16 (default) |
~33 tok/s (same setup, prior measurement) |
| q8_0 / q8_0 |
~11.7 tok/s |
During q8_0 decode, GPU utilization sits at only ~24-37% (memory/latency-bound), confirming a kernel-path regression rather than a compute limit.
Root cause
The sparse-fa dispatch lives only in the mma_f16 path. In ggml/src/ggml-cuda/fattn.cu, ggml_cuda_flash_attn_ext routes quantized K/V away from it:
if (turing_mma_available(cc) && Q->ne[0] != 40 && Q->ne[0] != 72) {
if (can_use_vector_kernel) {
if (!ggml_is_quantized(K->type) && !ggml_is_quantized(V->type)) {
// f16 -> may reach MMA_F16 (and its sparse-fa)
} else {
// quantized K/V -> forced onto VEC for single-query decode
if (Q->ne[1] == 1) { return BEST_FATTN_KERNEL_VEC; }
}
...
}
return BEST_FATTN_KERNEL_MMA_F16;
}So with q8_0 KV, decode never reaches ggml_cuda_flash_attn_ext_mma_f16_shall_use_sparse(); it lands on the vec kernel, which (a) has no sparse-attention path, (b) has no tensor-core MMA, and (c) must dequantize q8_0 on the fly.
Request
Add a quantized-KV (q8_0) path to the sparse-fa MMA kernel (dequantize q8_0 -> f16 inside the kernel, then run the sparse attention). This is the practical way to fit 1M context on 4x A6000, but the ~2.8x decode penalty currently makes q8_0 KV unusable for interactive workloads.
Happy to provide more measurements or test any patch.
Summary
On a DeepSeek-V4-Flash model (DSV4,
LLM_ARCH_DEEPSEEK4), using a quantized KV cache (--cache-type-k q8_0 --cache-type-v q8_0) causes a ~2.8x decode slowdown versus f16 KV, because the sparse-fa fused kernel (added in #27970 / 8e93a97) is only reachable through themma_f16path, which requires non-quantized K/V.Environment
67a17c1(build 203)-sm tensor)Q8_K_XL-c 1048576 -np 2 -sm tensor -fa on -ctk q8_0 -ctv q8_0Measurements
Single-stream decode rate (marginal, measured over 100 vs 400 generated tokens via
/completion):During q8_0 decode, GPU utilization sits at only ~24-37% (memory/latency-bound), confirming a kernel-path regression rather than a compute limit.
Root cause
The sparse-fa dispatch lives only in the
mma_f16path. Inggml/src/ggml-cuda/fattn.cu,ggml_cuda_flash_attn_extroutes quantized K/V away from it:So with
q8_0KV, decode never reachesggml_cuda_flash_attn_ext_mma_f16_shall_use_sparse(); it lands on theveckernel, which (a) has no sparse-attention path, (b) has no tensor-core MMA, and (c) must dequantize q8_0 on the fly.Request
Add a quantized-KV (q8_0) path to the sparse-fa MMA kernel (dequantize q8_0 -> f16 inside the kernel, then run the sparse attention). This is the practical way to fit 1M context on 4x A6000, but the ~2.8x decode penalty currently makes q8_0 KV unusable for interactive workloads.
Happy to provide more measurements or test any patch.