vulkan: fix ROCmFPx mat-vec cost at batch 3-8 on rocmfpx/wholesale-reference

[LaurentZuijdwijk]

Overview

While looking into "what needs merging re rocmfpx quants from laurentzuijdwijk/llama.cpp," I found this repo actually carries two independent ports of the same third-party format, not a merge relationship:

• laurentzuijdwijk/llama.cpp (personal fork, unrelated repo) hand-ported ciru-ai/ROCmFPX on 2026-08-20/21: CPU + Vulkan, 6 types. Its third commit, f0a2bd6b3, fixed a real perf bug: the Vulkan mat-vec shaders were mistuned for batch 3-8, exactly where speculative decoding verifies, so an FP4 27B model lost to a same-size K-quant under DFlash2 despite being 12% smaller.
• This repo's rocmfpx/wholesale-reference branch (b6a3f392c / 9abe628fb, 2026-08-27) is a separate, later hand-port of the same ciru-ai/ROCmFPX source, done with no apparent awareness of lz's fork. It's more complete in other ways (9 types instead of 6, plus HIP/CUDA kernels lz's port never touched), but it carries the identical mat-vec bug, unfixed - confirmed by direct diff, not by assumption.

So there is nothing to "merge" in the git sense (the two ports diverged from different upstream points and have different type layouts); what's actually transferable is the fix itself. This PR ports it onto rocmfpx/wholesale-reference's own code.

What changed, and what didn't

mul_mat_vecq.comp - ROCMFP4/ROCMFP4_FAST move out of the K_PER_ITER=8 bucket (shared there with ROCMFPX_FP6/FP8) into their own K_PER_ITER=32 bucket, same as lz's fix. FP6/FP8 are left alone - they were never part of the original fix and I have no evidence they share the bug.

mul_mat_vecq_funcs.glsl - lz's fork has one mmvq_dot_product shared between ROCMFP4/ROCMFP4_FAST; this branch has two separate functions. Applied the same "loop over the whole block instead of one 8-wide call" change to both, keeping this branch's existing split.

dequant_funcs.glsl - checked byte-for-byte before touching anything:

• fp3's dequantize4 had the exact same byte-window-plus-branch code lz's fix removed. Applied the fix as-is.
• fp6 was not touched. Lz's fix assumes fp6 is packed 6-bit-per-code (3 bytes = 4 codes); this branch instead stores fp6 as one full byte per code (data_a[...].qs[idx] direct indexing - no bit window, no branch). It's a structurally different layout, presumably from ciru-ai/ROCmFPX's original unpacked scheme that lz's own port explicitly corrected away from. Applying lz's fp6 diff here would silently misread every fp6 weight. There may still be a smaller, legitimate optimization available (hoisting the repeated per-weight scale lookup, since all 4 lanes of an aligned dequantize4 call share one scale) but that's a different, unverified change I did not make here - flagging it as a possible follow-up rather than guessing at it.

Measurements

None - this PR should be treated as unverified and is opened as a draft for that reason, per this repo's Benchmarking requirements. I have no Strix Halo hardware in this environment.

What I verified instead, since I can't benchmark:

• Diffed the pre-edit code against lz's pre-fix and post-fix versions to confirm the bug and the fix logic transplant correctly (repack()'s byte-window semantics, the QUANT_R==2 block layout, and the fp3 bit-packing are identical between the two ports; fp6 is not, which is why it's excluded).
• Compiled every touched shader variant with glslc --target-env=vulkan1.2: mul_mat_vecq.comp for ROCMFP4, ROCMFP4_FAST, ROCMFPX_FP3, ROCMFPX_FP6, ROCMFPX_FP8 (the last two to confirm the untouched branches still compile), and mul_mat_vec.comp for ROCMFPX_FP3 and ROCMFPX_FP6. All exit 0.
• Did not run test-backend-ops or llama-bench - no GPU available here. This needs those two things before it can leave draft: a test-backend-ops -o MUL_MAT / -o MUL_MAT_VEC pass for the touched types (to catch anything the compiler can't, like the k+4 accumulator split for ROCMFP4), and a llama-bench/GGML_VK_PERF_LOGGER run at batch 3-8 against the pre-fix baseline to confirm this branch actually sees the same magnitude of improvement lz measured (312->173 us etc.) - it's a different port, so the numbers won't necessarily match even if the bug is the same shape.

Additional information

Base branch is rocmfpx/wholesale-reference, not master - that branch is itself unmerged, which is a separate, prior question the repo owner should decide (whether/when to merge the wholesale port at all). This PR only fixes a bug within it.

Requirements

• I have read and agree with the contributing guidelines
• This change is Strix Halo specific: it's a shader perf fix for RDNA 3.5's mat-vec cost at the batch sizes speculative decoding verifies at
• AI usage disclosure: AGENT-AUTHORED. Written by Claude Opus 5 in Claude Code, at the repo owner's request, by diffing this repo's rocmfpx/wholesale-reference branch against the equivalent, verified fix in laurentzuijdwijk/llama.cpp
• What was NOT verified: no llama-bench, GGML_VK_PERF_LOGGER, or test-backend-ops run - no Strix Halo (or any Vulkan) hardware available in this environment. Shader compilation was checked with glslc; runtime correctness and the actual speedup on this branch's code were not

Assisted-by: Claude Opus 5