Skip to content

HIP: tune MMVQ batch thresholds on RDNA3.5 - #28613

Open
SimonTeixidor wants to merge 1 commit into
ggml-org:masterfrom
SimonTeixidor:rdna35-mmvq-thresholds
Open

HIP: tune MMVQ batch thresholds on RDNA3.5#28613
SimonTeixidor wants to merge 1 commit into
ggml-org:masterfrom
SimonTeixidor:rdna35-mmvq-thresholds

Conversation

@SimonTeixidor

Copy link
Copy Markdown
Contributor

Overview

ggml_cuda_should_use_mmvq had no per-type tuning for RDNA3.5 and fell through
to a flat ne11 <= MMVQ_MAX_BATCH_SIZE (8). This PR adds a table for it which
gives a performance boost for batched generation.

Additional information

I introduced a temporary env var override which allowed me to sweep ne11 1-16 with
test-backend-ops perf -o MUL_MAT.

Results

threshold types
ne11 <= 2 Q4_K, Q5_K
ne11 <= 3 Q3_K, Q6_K
ne11 <= 4 Q8_0, Q2_K
ne11 <= 5 Q4_1, Q5_1, MXFP4, IQ4_NL, IQ3_XXS
ne11 <= 6 Q4_0, Q5_0, IQ3_S, IQ4_XS, IQ1_S
default (8) IQ2_XXS, IQ2_XS, IQ2_S, Q1_0, Q2_0, NVFP4

End-to-end (Strix Halo / gfx1151, Qwen3.8-27B-UD-IQ4_XS, llama-batched-bench, S_TG)

npl gain
1 -0.8% (identical to master, null control)
4 +6.4% +/- 2.0
6 +10.2% +/- 1.9
8 +20.5% +/- 1.9

test-backend-ops -o MUL_MAT: 1288/1288.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Claude Opus helped me write the sweep script and implement the PR.

Add per-type MMVQ batch-size tuning for RDNA3.5 in ggml_cuda_should_use_mmvq,
which speeds up batched generation.
@SimonTeixidor
SimonTeixidor requested a review from a team as a code owner September 8, 2026 19:15
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant