Vulkan: wrong matmul results on Adreno (subgroupSize=128) — warptile WM exceeds BM (implicit `WM <= BM` invariant violated for subgroupSize > 64)

[lkazarin]

Environment

• llama.cpp commit bf2c86d (also present in whisper.cpp's bundled ggml — its own tree must be patched too)
• Vulkan backend, Adreno 650 via Mesa Turnip, subgroupSize = 128 (double threadsize)
• Applies to any device that reports subgroupSize > 64, not just Adreno

Symptom

./test-backend-ops -b Vulkan0 -o MUL_MAT — 2 of 229 f16 cases FAIL with large error,
all of them batched + broadcast + partial-tile shapes:

MUL_MAT(type_a=f16,type_b=f32,m=64, n=45,k=128,bs=[8,1],nr=[4,1])  ERR = 0.414  FAIL
MUL_MAT(type_a=f16,type_b=f32,m=128,n=45,k=64, bs=[8,1],nr=[4,1])  ERR = 1.079  FAIL

Real workloads (whisper.cpp) produce garbage / looping output on the GPU while the
CPU backend on the same model is correct.

Root cause

In ggml_vk_load_shaders() (ggml/src/ggml-vulkan/ggml-vulkan.cpp), warptiles are built
with WM (index 4) and WARP (index 10) = subgroup_size_8 = max(device->subgroup_size, 8):

l_warptile = { 128, 128, 128, 16, subgroup_size_8 * 2, 64, 2, ..., subgroup_size_8 }; // BM=128, WM=subgroup*2
m_warptile = { 128,  64,  64, 16, subgroup_size_8,     32, 2, ..., subgroup_size_8 }; // BM=64,  WM=subgroup

These encode an implicit invariant WM <= BM. It holds for
subgroupSize ∈ {8,16,32,64} but breaks at 128: m_warptile gets WM = 128 > BM = 64,
l_warptile gets WM = 256 > BM = 128.

In mul_mm.comp this detonates in three ways for the degenerate tile:

1. BM / WM folds to 0 (unsigned) → warp_r = warp_i % (BM/WM) / warp_c = warp_i / (BM/WM) are udiv/umod by zero.
2. buf_a shared-memory indexing addresses rows 0..WM-1, but buf_a is only
   BM * SHMEM_STRIDE entries; for WM > BM it runs past its end into buf_b, so the
   A operands become B data.
3. Only the first WN of the BN output columns are ever written; columns WN..BN-1 are
   never computed/stored.

The small tile (s_warptile) uses BM = WM = subgroup_size_32 so it stays
self-consistent — which is exactly why only medium/large tiles fail (need n > 32
and m > 32); every passing case in the suite is a small-tile case.

Fix (verified working)

Clamp the tiling WARP for the l_/m_ warptiles to 64, leaving s_warptile untouched:

const uint32_t subgroup_size_8 = std::max(device->subgroup_size, 8u);
const uint32_t mm_warp_8       = std::min(subgroup_size_8, 64u);           // NEW: l_/m_ warptiles only
const uint32_t mul_mat_mm_warp_8 = std::min(mul_mat_subgroup_size_8, 64u); // NEW: mmqid l_/m_ warptiles

then use mm_warp_8 (resp. mul_mat_mm_warp_8) in every l_warptile* / m_warptile*
definition in place of subgroup_size_8 (resp. mul_mat_subgroup_size_8) — 18 lines.

⚠️ s_warptile must keep the unclamped value: its BLOCK_SIZE is subgroup_size_32
and it needs NUM_WARPS == 1 to keep its area at BM*BN. Clamping it too regresses
m=1 (matrix-vector) — verified.

Result after the fix: f16 229/229 and f32 173/173 pass, whisper.cpp transcribes correctly.

Note for maintainers

A cleaner central fix may be preferred (e.g. assert(WM <= BM) in the warptile builder and
select a valid tiling, or clamp WM to BM generically), but the scoped
mm_warp_8 = min(subgroup_size_8, 64) is minimal and verified. Since NUM_WARPS*WM*WN == BM*BN and WM <= BM both continue to hold for l_/m_ after clamping, occupancy is
unaffected on subgroupSize ≤ 64 devices (the min is a no-op there).

[aidaiprivate-source]

Kimi_Agent_AI-агенты.zip

[lkazarin]

Disclosure, in line with CONTRIBUTING.md: this report was written by an AI assistant (Claude, via Claude Code) working under my direction, and CONTRIBUTING.md point 4 explicitly prohibits using AI to write posts. I did not know about the policy when I opened this — flagging it now rather than leaving it undisclosed.

To be clear about what is and isn't mine: I directed the investigation, supplied and operated the hardware (OnePlus 8 Pro / Adreno 650, mainline Linux), and the finding is genuine — reproduced and verified by live debugging on that device, not speculation. The analysis and the wording, however, are the AI's.

The proposed fix is in #25735, which carries the same disclosure. If you would prefer this report reworded by me, or closed and reopened in my own words, just say so.

[0cc4m]

Sorry about the delay, I checked and in general the fix is fine, but it added a lot of AI comments and missed some other problematic configurations. I opened a PR for an alternative workaround, and once #25773 lands you could look into Adreno-specific warptiles.