vulkan : only chunk batched mat-vec for the quant types that were measured

[LaurentZuijdwijk]

One of the two causes of #25. Written by an agent (Claude Fable 5.1); all numbers below were measured on a Strix Halo box in this session, against a baseline built from the merge-base (c7af5c6).

Problem

The batched mat-vec column chunking (#1) was measured on Qwen3.8-Flash-Next, whose hot shapes are q8_0 and q6_K. The K-quants and the q4_0 family were put on a 2-column chunk without a measurement. On a dense Qwen3.8-27B UD-Q4_K_XL that re-reads 16 GiB of weights once per chunk for nothing, because the single dispatch scales fine at 3, 5, 6 and 8 columns on this GPU and driver. Speculative verification lands exactly at those widths.

llama-bench -p 1,2,3,4,5,6,8 -n 0 -b 8 -r 3, prompt tok/s:

cols	master c7af5c6	master + GGML_VK_MMV_NO_SPLIT=1	ggml-org d08c787
1	12.0	11.8	11.9
2	23.5	23.0	23.0
3	23.0	32.9	33.1
4	30.7	39.7	39.2
5	28.5	43.1	46.7
6	30.8	42.7	51.3
8	28.8	47.4	59.9

(The gap between NO_SPLIT and mainline at 5 to 8 columns is upstream's own Strix Halo mat-vec row tuning, 2cdae80, which arrives with #23.)

Change

Restrict the chunking, and the 9..16 column chunk path, to q8_0 and q6_K, the two types where the slow variants were measured. Every other type takes upstream's dispatch. GGML_VK_MMV_NO_SPLIT=1 still disables the rest. The comment in ggml-vulkan.cpp records the measurement.

Measurements

Environment: Ryzen AI Max+ 395, Radeon 8060S (RADV, Mesa 26.1.7), Linux 7.0.0-30, Ubuntu 24.04. Model unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL. Server flags as in the issue: -ngl 999 -c 8192 --flash-attn on --parallel 1 --spec-type draft-mtp --spec-draft-n-max 4. One chat completion, 95-token prompt, ~185 generated tokens, temperature 0.1, thinking off. Decode tok/s, 3 runs each.

build	decode tok/s
master c7af5c6	9.6 / 9.8 / 9.9
this PR	14.8 / 15.1 / 15.2
master + GGML_VK_MMV_NO_SPLIT=1	15.6 / 15.9 / 15.7
ggml-org d08c787 (mainline)	19.7 / 19.7 / 20.2

The small difference to NO_SPLIT is the q6_K tensors of this model, which are still chunked. The rest of the gap to mainline is the n_rs_seq regression, addressed separately; with both fixes decode matches mainline (19.9 / 20.0 / 20.0).

Correctness

test-backend-ops test -b Vulkan0 -o MUL_MAT, same box, both builds from this session:

build	result	failing cases (all test_mul_mat_exact_batch, m=128, k=5120)
master c7af5c6	1078/1084	bf16 n=9, bf16 n=16, q5_1 n=4, q8_0 n=4, q4_K n=4, q5_K n=4
this PR	1081/1084	bf16 n=9, bf16 n=16, q8_0 n=4

No new failures. The q4_K, q5_K and q5_1 exact-batch cases pass now that those types take upstream's single dispatch; the bf16 and q8_0 cases are untouched by this change (bf16 was never chunked, q8_0 still is) and fail on master the same way. The change only alters which dispatch is chosen; the K-quant path is upstream's code.

Not verified

• Qwen3.8-Flash-Next, the model the chunking was tuned on, was not re-measured; q8_0 and q6_K keep their chunking so it should be unaffected.
• Whether q6_K chunking is right on dense models either; left as is for lack of a measurement.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LTnaB4ucGc5vkzctYns4Nq

[voidsurfer]

Independent check of this PR on the post-sync master (88c87dd), Vulkan first, then HIP for the shared cases. Model is the Huihui abliterated fine-tune of Qwen3.8-27B UD-Q4_K_XL: Q4_K and Q5_K dominant, 56 q6_K and 110 q8_0 tensors, so both kept and released chunk types are exercised.

Full Vulkan backend suite, both arms, same case inventory (19440 Vulkan rows):

• master: 6 failing cases, all MUL_MAT m=128 k=5120 (bf16 n=9, bf16 n=16, q5_1 n=4, q8_0 n=4, q4_K n=4, q5_K n=4)
• with this PR: 3 failing cases (bf16 n=9, bf16 n=16, q8_0 n=4)

So this PR clears q4_K, q5_1 and q5_K at that shape, as your own table says. No new failing case appears.

Two extra data points from our runs:

1. The q8_0 n=4 case that stays chunked still fails on Vulkan on both arms. The same case passes on HIP on both arms, where the Vulkan chunk path does not run. That case is now the only quantized one left, and it sits inside the two types this PR keeps.
2. Our decode gain with speculative : give MTP targets their recurrent rollback slots back #26 + this PR (+94.9% Vulkan, +42.3% HIP, MTP workload, n=5) is measured with q6_K still chunked, since this model carries 56 q6_K tensors. The Mesa 25.3.6 pp table in the issue (dabblingwithcode) points the same way: the residual pp gap at 5 to 8 columns tracks the q6_K tensors. We did not run a q6_K-restricted build, so we have no number of our own for that gate; the existing data suggests it is worth a measurement.

GGML_VK_MMV_NO_SPLIT=1 on master reproduces the fixed arm's plain-decode token stream exactly (228 of 228 tokens), which supports the dispatch as the cause of the plain-decode text difference on Vulkan.

Assisted-by: Codex (runs and checks done locally on real hardware by the poster).

-Astra