Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The functional change is narrowly scoped to RDNA 3.5 Q6_K routing and appears consistent with existing architecture-gated heuristics; only a minor inline-comment maintainability tweak was noted.
Pull request overview
This PR adjusts the HIP/CUDA MMQ routing heuristic for AMD RDNA 3.5 (gfx1151) so Q6_K matmuls use the MMQ kernel for larger prompt-side column counts (up to ne11 <= 1024), matching the Strix Halo performance measurements described in the PR.
Changes:
- Extend the
GGML_TYPE_Q6_KMMQ eligibility threshold tone11 <= 1024specifically onGGML_CUDA_CC_IS_RDNA3_5(cc). - Keep existing thresholds unchanged for RDNA 3.0 and all other architectures.
File summaries
| File | Description |
|---|---|
| ggml/src/ggml-cuda/mmq.cu | Updates ggml_cuda_should_use_mmq() to route RDNA 3.5 Q6_K matmuls through MMQ up to ne11 = 1024. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // RDNA 3.5 (gfx1151): dequantize + hipBLASLt loses to MMQ up to ne11 = 1024; at 2048 the two | ||
| // paths trade places per shape (K = 4096 favours hipBLASLt, K >= 12288 favours MMQ), see PR. |
On gfx1151 the Q6_K MMQ kernel runs at a flat 16-17 TFLOPS while the dequantize + hipBLASLt path drops to 11-12 TFLOPS once K >= 8192 (the ffn_down shapes of 8B models) and pays a full f16 dequantize round trip per prefill pass. At ne11 = 2048 the two paths trade places per shape (K = 4096 favours hipBLASLt, K >= 12288 favours MMQ), so the gate stops at 1024. Keep RDNA 3.0 and other targets on the existing threshold. Assisted-by: Claude
5532a64 to
b8e4f25
Compare
There was a problem hiding this comment.
Validation: the Release HIP gfx1151 build passed. Q6_K MUL_MAT passed 1238/1238 ROCm correctness cases on Radeon 8060S/gfx1151.
Benchmark reproduced for a threshold-range kernel: against current master, test-backend-ops perf at Q6_K m=4096, n=512, k=14336 measured 5,342.90 us/run before this change and 3,507.31 us/run with it: 1.52x faster.
Scope: appropriate for the Strix repository. However it overlaps the broader #56 dispatch retune; choose one coherent final policy rather than merge competing threshold changes independently.
comment generated by my clanker Codex
Overview
On gfx1151
ggml_cuda_should_use_mmqsends Q6_K matmuls with ne11 > 256 to dequantize + hipBLASLt. That path isslower than MMQ up to ne11 = 1024 on this GPU: the Q6_K MMQ kernel runs at a flat 16-17 TFLOPS on the fork's RDNA 3.5
tile rows, while the Tensile GEMM drops from 20-22 TFLOPS at K = 5376 to 11-12 TFLOPS for K >= 8192 at ne11 = 512
(the ffn_down shapes of 8B models; PMC: L2 hit rate 68% -> 38%, DRAM fetch 1.2x -> 2.8x the unique bytes once the
per-workgroup f16 weight panel outgrows the 2 MB L2) and the dequantize pass adds a full f16 write + read of every
weight per prefill pass. At ne11 = 2048 the two paths trade places per shape (K = 4096: hipBLASLt 0.85-1.0x of MMQ
time, K >= 12288: MMQ 1.17-1.35x faster), so the gate stops at 1024 rather than going to
return true. This PRraises the Q6_K threshold to
ne11 <= 1024underGGML_CUDA_CC_IS_RDNA3_5; RDNA 3.0 and everything else keep theexisting threshold.
Effect: prefill +8-10% on Q4_K_M / Q5_K_M files (their ffn_down and attn_v tensors are Q6_K), +30% on pure Q6_K files,
up to +50% at -ub 384 where the Tensile path has a hole, no change at -ub 2048 (gate off), no change on files without
Q6_K tensors in the prompt path (Q2_K, Q3_K_M, IQ4_XS, Q4_0, Q8_0, tested), no change to decode (MMVQ path). Measured across 12 GGUFs and three more models (Gemma-4-E4B, Qwen3.6-35B-A3B MoE, Gemma-4-31B:
no change / no change / inconclusive) before this PR; the tables below are the merge-base re-run required here.
Interaction with upstream ggml-org#25940: its Q6_K vec_dot change makes the MMQ side another ~1.25x faster
on gfx1151 (19-20 TFLOPS), which only widens the margin this rule relies on.
Measurements
Baseline:
pass 1: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5pass 2: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q4_K_M ubatch axis -p 2048 -n 0 -ub 128..2048 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K ubatch axis
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 512 -n 128 -r 5Llama-3.1-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Llama-3.1-8B-Instruct-GGUF/Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q8_0 (no Q6_K tensors: control) -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q8_0.gguf -p 512 -n 128 -r 5After:
pass 1: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5pass 2: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q4_K_M ubatch axis -p 2048 -n 0 -ub 128..2048 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K ubatch axis
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 512 -n 128 -r 5Llama-3.1-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Llama-3.1-8B-Instruct-GGUF/Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q8_0 (no Q6_K tensors: control) -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q8_0.gguf -p 512 -n 128 -r 5Correctness:
test-backend-ops test -b ROCm0 -o MUL_MAT,MUL_MAT_IDon the change build:llama-perplexity -f wiki.test.raw -c 512 -b 512 -ub 512 --chunks 32 -ngl 99(this change only moves prompt-sized Q6_K matmuls from hipBLASLt f16 to MMQ int8, so PPL is the oracle):Additional information
an RDNA 3.5 Q6_K MMQ regression. A K-aware gate (MMQ for K >= 12288 at any ne11) would add another +17-35% on the
ffn_down shapes at -ub 2048, but
ggml_cuda_should_use_mmqdoes not receive K today; left as a follow-up.return true; its -ub 2048 row for the pure Q6_K file was 2.7% belowbaseline (1063.6 vs 1093.6 t/s), which is why the gate was lowered to 1024 and everything re-run. That log is
kept as
logs/strixpr-20260910-run1-returntrue.login the research repo. The decision data (kernel traces, PMC, K-sweeps) is in the research notes linkedfrom the branch description; happy to paste more here.
Requirements
belong in halo-box/llama.cpp instead
5-line change, ran the interleaved A/B harness and this PR's benchmark protocol, and wrote this description. The
owner reviewed the change and the numbers.
HIP_LAUNCH_BLOCKING=1; CDNA and RDNA 4 are notaffected by the gate but were not built or run; the Vulkan backend is untouched.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CzgPbiT4PgFRYgLXRob86e