vulkan: clamp l_/m_ warptile WM to <= BM (fix wrong matmul on subgroupSize > 64)

[lkazarin]

Overview

The l_/m_ warptiles set WM (and WARP) to subgroup_size_8 = max(device->subgroup_size, 8), which encodes an implicit WM <= BM invariant. That invariant holds for subgroupSize ∈ {8,16,32,64} but breaks at 128 — Adreno (Mesa Turnip) reports subgroupSize = 128 under double threadsize, so m_warptile gets WM=128 > BM=64 and l_warptile gets WM=256 > BM=128.

In mul_mm.comp the degenerate tile causes:

1. BM / WM to fold to 0 → warp_r = warp_i % (BM/WM) / warp_c = warp_i / (BM/WM) are udiv/umod by zero;
2. buf_a shared-memory indexing (rows 0..WM-1) to overrun its BM * SHMEM_STRIDE allocation into buf_b, so the A operands become B data;
3. columns WN..BN-1 of every output tile to be never computed or stored.

Result: silently wrong matmul results. s_warptile is unaffected (BM = WM = 32), which is why only medium/large tiles fail.

This PR clamps the tiling WARP for the l_/m_ warptiles to 64. s_warptile deliberately keeps the unclamped value (its BLOCK_SIZE is subgroup_size_32 and it needs NUM_WARPS == 1; clamping it regresses m=1). The min is a no-op on subgroupSize <= 64, so NVIDIA/AMD/Intel are unaffected.

Additional information

Reported in #25734.

Tested on Adreno 650 / Mesa Turnip (SM8250, mainline Linux / postmarketOS), test-backend-ops -b Vulkan0 -o MUL_MAT:

• before: 2/229 f16 cases FAIL (batched + broadcast + partial tile, e.g. m=64,n=45,k=128,bs=[8,1],nr=[4,1] ERR≈0.41, m=128,n=45,k=64 ERR≈1.08)
• after: f16 229/229 and f32 173/173 pass; whisper.cpp transcribes correctly on the GPU (previously produced garbage).

Note: this device also needs a separate Mesa/Turnip fix to avoid a GPU hang, filed at https://gitlab.freedesktop.org/mesa/mesa/-/work_items/15881 — that one is a driver bug, unrelated to this change.

Requirements

• I have read and agree with the contributing guidelines

• AI usage disclosure: YES — and it exceeds what CONTRIBUTING.md permits. Disclosing in full so maintainers can decide.

  This work was done with an AI assistant (Claude, via Claude Code) operating under my direction on my hardware. Specifically:

  • the investigation and root-cause analysis were performed by the AI;
  • the code change itself was written by the AI, not by me — so per CONTRIBUTING.md this is AI-generated code, not merely AI-assisted, and it does not meet the "majority of the code authored by a human contributor" bar;
  • this PR description, the original issue Vulkan: wrong matmul results on Adreno (subgroupSize=128) — warptile WM exceeds BM (implicit WM <= BM invariant violated for subgroupSize > 64) #25734 text, and the commit message were also written by the AI, which CONTRIBUTING.md point 4 explicitly prohibits.

  What is mine: I directed the whole investigation, supplied and operated the device, and made the decisions; the bug was found and the fix verified through live debugging on real hardware, not generated speculatively. The underlying finding is genuine and reproducible.

  I did not know about the AI policy when this was opened — that is on me, and I am not going to dress the text up to look otherwise. If this is not acceptable in its current form, please say so and I will either close it or resubmit the change authored by me in my own words. Happy to answer questions about any line of it.

[ggml-gh-bot]

Hi @lkazarin, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[0cc4m]

Superseded by #27726