CUDA: let any expert count use the fast mm_ids_helper path by ServeurpersoCom · Pull Request #27978 · ggml-org/llama.cpp

ServeurpersoCom · 2026-08-29T17:46:31Z

Overview

CUDA: unlock the fast mm_ids_helper path for any expert count, enable it for 10 (Qwen3.8-Flash-Next)

Additional information

mm_ids_helper groups tokens by expert before the MoE matmuls. It has a fast path that spreads warp lanes across tokens, and a slow generic one that walks a single token per iteration with a warp reduction each time.

The fast path only accepted expert counts dividing 32, plus a hardcoded case for 6, so anything else quietly fell back to the slow path. Qwen3.8-Flash-Next uses 10, and nsys put this kernel at 13.3% of prefill time, ahead of flash attention.

The real constraint isn't the expert count itself but the number of lanes per token, and the loop already guarded the extra lanes. So the padding just rounds up to the next power of two. Every count already dispatched gets the same padding as before and is unchanged. I added 10; any other count is one line away.

RTX PRO 6000 at 55k context, first run discarded: prefill ~~2334 to 2600 t/s~~ 2139 -> 2351 t/s, +9.9% (current master 62acc89). Generation doesn't move, with a single token there's nothing to walk. Only tested on CUDA with 32 wide warps.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex < n_expert_used, so the padding generalizes to the next power of two. The 6 -> 8 case and every count already dispatched keep the exact same padding as before. n_expert_used = 10 now reaches the fast path. Measured on Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an RTX PRO 6000, warm runs with the first one discarded: prompt processing 2334 -> 2600 t/s Token generation is unaffected, since a single token leaves nothing to walk. Other expert counts reach the fast path by adding their case to the dispatch.

ServeurpersoCom · 2026-08-30T15:14:20Z

I checked exact head af8ae552a8914a4c84f6931c9eb8debcbf207615 on an RTX 3080 Ti (sm_86, CUDA 13.3). I compiled the proposed power-of-two helper and an sm_86 CUDA probe for the optimized-path n_expert_used = 10 lane guard, then ran the probe: neu_padded=16, two groups per warp, 20 valid-lane loads, 12 padded-lane skips, bad=0 (padded_ok=1). This is bounded compile/pad-guard evidence, not an end-to-end llama.cpp runtime or output-correctness test. I did not reproduce the RTX PRO 6000 2334 -> 2600 t/s result, so I cannot validate that performance claim. The patch adds optimized dispatch only for 10; other undispatched expert counts still use the existing generic path.

The perf claim needs a model whose expert count was not dispatched, so Qwen3.8-Flash-Next with 512 experts and 10 used. On anything using 2, 4, 6, 8, 16 or 32 there is nothing to measure, those already took the fast path.

I re-ran it on current master, reverting only f1793c1 for the control, two separate builds, alternated in both orders with the first pair discarded:

without 2161.5 2138.3 2137.8 2130.8
with 2418.6 2374.1 2349.1 2340.8

2139 -> 2351 t/s, +9.9% prefill at 55k context. A bit below the 2334 -> 2600 in the description, which came from a branch carrying other changes. (#28023 not merged for now)

* phase two * First PoC on MoE cache, with promising results on decode Assisted-by: GPT-5.6 Sol * CUDA: use the fast mm_ids_helper path for any n_expert_used (ggml-org#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex < n_expert_used, so the padding generalizes to the next power of two. The 6 -> 8 case and every count already dispatched keep the exact same padding as before. n_expert_used = 10 now reaches the fast path. Measured on Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an RTX PRO 6000, warm runs with the first one discarded: prompt processing 2334 -> 2600 t/s Token generation is unaffected, since a single token leaves nothing to walk. Other expert counts reach the fast path by adding their case to the dispatch. Assisted-by: GPT-5.6 Sol * ggml: keep MoE IDs alive during cache remap * llama: pipeline batched MoE cache transfers * examples: add persistent MoE corpus runner * docs: condense MoE cache experiment notes * ggml: remove unused inactive expert IDs * examples: keep MoE debug artifacts local * examples: remove MoE benchmark results * docs: remove local benchmark summary * examples: remove remaining MoE debug tooling * common: keep debug instrumentation local * ggml: route MoE cache ops without env override * llama: remove experimental MoE prefill overlap * llama: warn about mixed MoE cache layouts * ggml: simplify persistent MoE cache path * ggml: address MoE cache review feedback * guard test-moe-cache * llama: log MoE cache stats at trace level * llama: reject MoE cache on OpenCL * ggml: stabilize MoE cache graph topology --------- Co-authored-by: Pascal <admin@serveurperso.com>

KV restore batching (ggml-org#27991), kv-cells seq-scan early stop (ggml-org#28011), MOE fusion to specdec + multi-token (ggml-org#27621), mm_ids_helper templated fast path (ggml-org#27978), qwen4exp recurrent state rollback (ggml-org#28123), n_layer_nextn load order (ggml-org#28159), FA K/V XOR-swizzle smem tiles (ggml-org#25635), --lazy-mode -lzm (ggml-org#27837/ggml-org#27969). TQ3/TurboQuant stack and vitriol-* integration auto-merged clean; no conflicts. Experiment E1 of mining-experiment-master-plan-2026-09-01.

* phase two * First PoC on MoE cache, with promising results on decode Assisted-by: GPT-5.6 Sol * CUDA: use the fast mm_ids_helper path for any n_expert_used (ggml-org#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex < n_expert_used, so the padding generalizes to the next power of two. The 6 -> 8 case and every count already dispatched keep the exact same padding as before. n_expert_used = 10 now reaches the fast path. Measured on Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an RTX PRO 6000, warm runs with the first one discarded: prompt processing 2334 -> 2600 t/s Token generation is unaffected, since a single token leaves nothing to walk. Other expert counts reach the fast path by adding their case to the dispatch. Assisted-by: GPT-5.6 Sol * ggml: keep MoE IDs alive during cache remap * llama: pipeline batched MoE cache transfers * examples: add persistent MoE corpus runner * docs: condense MoE cache experiment notes * ggml: remove unused inactive expert IDs * examples: keep MoE debug artifacts local * examples: remove MoE benchmark results * docs: remove local benchmark summary * examples: remove remaining MoE debug tooling * common: keep debug instrumentation local * ggml: route MoE cache ops without env override * llama: remove experimental MoE prefill overlap * llama: warn about mixed MoE cache layouts * ggml: simplify persistent MoE cache path * ggml: address MoE cache review feedback * guard test-moe-cache * llama: log MoE cache stats at trace level * llama: reject MoE cache on OpenCL * ggml: stabilize MoE cache graph topology --------- Co-authored-by: Pascal <admin@serveurperso.com>

…#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex < n_expert_used, so the padding generalizes to the next power of two. The 6 -> 8 case and every count already dispatched keep the exact same padding as before. n_expert_used = 10 now reaches the fast path. Measured on Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an RTX PRO 6000, warm runs with the first one discarded: prompt processing 2334 -> 2600 t/s Token generation is unaffected, since a single token leaves nothing to walk. Other expert counts reach the fast path by adding their case to the dispatch.

ServeurpersoCom requested a review from a team as a code owner August 29, 2026 17:46

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Aug 29, 2026

am17an approved these changes Aug 30, 2026

View reviewed changes

ServeurpersoCom requested a review from ggerganov August 30, 2026 13:38

ggerganov approved these changes Aug 30, 2026

View reviewed changes

ServeurpersoCom merged commit f1793c1 into ggml-org:master Aug 30, 2026
20 of 23 checks passed

This comment was marked as low quality.

Sign in to view

dur-randir mentioned this pull request Aug 30, 2026

ggml-cpu : add mirror NUMA strategy (replicate weights on each node) #27986

Draft

Navigation Menu

Uh oh!

CUDA: let any expert count use the fast mm_ids_helper path - #27978

CUDA: let any expert count use the fast mm_ids_helper path#27978
ServeurpersoCom merged 1 commit into
ggml-org:masterfrom
ServeurpersoCom:mmid-optimize

ServeurpersoCom commented Aug 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

This comment was marked as low quality.

ServeurpersoCom commented Aug 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ServeurpersoCom commented Aug 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Uh oh!

This comment was marked as low quality.

ServeurpersoCom commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ServeurpersoCom commented Aug 29, 2026 •

edited

Loading

ServeurpersoCom commented Aug 30, 2026 •

edited

Loading