cuda: fusion hit counters, test guards and the #343 review follow-ups

[jasstrong]

Follow-up to #343, the non-blocking items from the review there.

What changed

• Fusion hit counters. The CUDA context now counts elementwise chains launched, tuned multi-ADD/MUL runs, and shared-quantize cache hits. They are read through ggml_backend_cuda_fusion_count(backend, name), also reachable via ggml_backend_reg_get_proc_address(reg, "ggml_backend_cuda_fusion_count") so test-backend-ops needs no CUDA header.
• Tests guard the feature, not just the numbers. test_case gets a required_fusion() hook; when the tested backend exposes the counter and it does not advance while the graph runs, the case fails with fusion did not fire. ELEM_CHAIN_FUSION requires the chain (the multi-ADD case requires the tuned kernel instead, see below), MUL_MAT_SHARED_SRC1 requires a cache hit for the n <= 8 Q8_0/Q4_0 cases (the mmvq path; native TQ weights quantize their activation on their own path, so those stay numeric-only). Backends without the counter are unaffected. Sanity check below: with GGML_CUDA_FUSE_CHAIN=0 the chain cases fail, with GGML_CUDA_Q8CACHE=0 the eight Q8_0/Q4_0 cache cases fail.
• New chain cases: SCALE with a nonzero bias (ggml_scale_bias), and a pure run of four same-shape f32 ADDs, which is the pattern the tuned multi-ADD kernel also takes.
• Precedence. You were right to ask: the tuned multi-ADD kernel is a little faster than the chain kernel for a pure run of ADDs (numbers below), so the chain detector now hands a pure same-layout ADD or MUL run, chained through src0, back to ggml_cuda_try_fuse()'s tuned path and keeps everything else. Both call sites carry a comment saying so.
• MOE_REDUCE_FUSION renamed to MUL_MAT_ID_REDUCE, since it no longer tests a fusion.
• Env vars: GGML_TQ_Q8CACHE becomes GGML_CUDA_Q8CACHE (the cache is not TQ-specific), and both it and GGML_CUDA_FUSE_CHAIN are documented in the AGENTS.md table. This is a breaking rename with no alias for the old name. It is safe here only because the variable landed in cuda: shared-quantize cache, residual and elementwise-chain fusions (generic half of #338) #343 and has not been in a release; anyone who picked it up from that branch has to change the name.

Second review round

• MUL_MAT_SHARED_SRC1's guard is narrowed from n <= 8 to n <= 4. ggml_cuda_should_use_mmvq picks the mmvq path from a per-architecture table whose lowest bound among the types registered here is ne11 <= 6 (Q8_0 on CDNA1), so n = 8 routes to MMQ there and the guard would have failed on a correct build.
• The guard is skipped when GGML_CUDA_DISABLE_FUSION is set and non-zero, matching the backend's own predicate, so the global kill switch stays usable while bisecting a numeric problem.
• An unknown counter name is now a hard failure rather than a silent pass, so a typo cannot disable the check, and print_test_console shows the failure reason: it was only reaching the CSV output, which left every failing case as a bare FAIL.
• fused_binary is split into fused_add and fused_mul, counted in their own branches; the multi-ADD case requires fused_add.
• required_fusion() is evaluated once, and every failure reason is reported so a numeric failure stays visible when the guard trips too.

Test

MI210 (gfx90a), GGML_TQ_MMQ=1 GGML_TQ_NATIVE=1:

suite	result
-o ELEM_CHAIN_FUSION (5 cases: base, gelu+softplus, self_mul, scale bias, add run)	5/5
-o MUL_MAT_SHARED_SRC1	12/12
-o MUL_MAT_RESIDUAL_FUSION	24/24
-o MUL_MAT_ID_REDUCE	6/6
-o MUL_MAT -p type_a=tq4_1s / tq3_1s	149/149, 158/158
guard check, GGML_CUDA_FUSE_CHAIN=0 -o ELEM_CHAIN_FUSION	1/5 (only the add run, which the tuned kernel takes)
guard check, GGML_CUDA_Q8CACHE=0 -o MUL_MAT_SHARED_SRC1	4/12 (only the TQ cases, which are numeric-only)

The branch sits on top of #348, since the tq4 suites hang on the base without it.

Timing, chain kernel vs tuned multi-ADD

test-backend-ops perf -o ELEM_CHAIN_FUSION, the add_run=1 case ({4096, 64} f32, four ADDs, 3 MB moved per run), three runs each, before the yield was added:

	us/run	GB/s
chain kernel (GGML_CUDA_FUSE_CHAIN=1, before the yield)	3.99 / 4.01 / 4.00	733
tuned ggml_cuda_op_fused_add (GGML_CUDA_FUSE_CHAIN=0)	3.75 / 3.96 / 3.96	740-780
chain enabled, with the yield (this branch)	3.75 / 3.77 / 3.98	736-780

For a mixed chain (SILU, MUL, ADD, ADD, SCALE with bias, CLAMP over the same shape) the chain kernel is the faster one: 2.50 us/run vs 2.52-2.54 us/run with it disabled.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NxP6x5bmUDYFvmouceN2mR

[TheTom]

The counters and the proc-address plumbing are right: each increments once per launch inside the branch that actually launches, never on a declined fusion; test-backend-ops stays free of CUDA headers and gets a null proc address on other backends; both guarded cases run whole-graph so ggml_cuda_try_fuse is what gets measured, bracketing only the backend-under-test run. Good.

Three things before I merge, all in the test file:

1. MUL_MAT_SHARED_SRC1's required_fusion() asserts a q8 cache hit for every n <= 8, but ggml_cuda_should_use_mmvq has a per-arch table: on CDNA1 it is ne11 <= 6 for Q8_0 and <= 7 for Q4_0, so the n = 8 cases route to MMQ, take no hit, and the guard reports "fusion did not fire" on a correct build. MI210 is CDNA2 and falls through to the default, which is why 12/12 is green there. Narrow the guard to n <= 4, or derive it from the same predicate.
2. GGML_CUDA_DISABLE_FUSION=1 (upstream's global kill switch) makes every ELEM_CHAIN_FUSION case fail, and setting it while bisecting a numeric bug is a normal thing to do. Skip the guard when that variable is set.
3. An unknown counter name returns -1 and the guard silently passes, so a typo disables the check with no signal. That should be a hard failure; failing loudly is the whole point.

Follow-up rather than blocker: MUL_MAT_RESIDUAL_FUSION (24 cases, whole-graph) still has no required_fusion(), so the mul_mat residual and GLU epilogue fusions stay unobservable, and that was the case the #343 review cared about most.

Nits: required_fusion() is evaluated twice at the call site; when both numerics and the guard fail the message hides the numeric failure; GGML_TQ_Q8CACHE to GGML_CUDA_Q8CACHE is a breaking rename with no alias, fine since #343 just landed but say so in the body; fused_binary cannot tell ADD from MUL, so the add_run guard would be satisfied by a stray fused MUL.

I'll run the four suites on a GB10 on the current head in the meantime; happy to push 1 to 3 myself if you'd rather rest the hands.

[jasstrong]

All three are fixed in af1b860, thank you. The CDNA1 one was a real hole rather than a style point.

1. MUL_MAT_SHARED_SRC1 narrowed to n <= 4. You are right that n = 8 is not universally an mmvq shape: the lowest bound in ggml_cuda_should_use_mmvq among the types this test registers is ne11 <= 6 for Q8_0 on CDNA1, so the guard would have reported a missing fusion on a correct build there. I kept it as a literal rather than deriving it from the predicate, because reaching that function from the test means pulling a CUDA header into test-backend-ops, which is what the proc-address indirection exists to avoid. The comment now names the bound and where it comes from, so whoever widens the registered n values knows what to re-check.

2. GGML_CUDA_DISABLE_FUSION skips the guard, matched to the backend's own predicate (set and non-zero), so =0 still runs it.

3. An unknown counter name is now a failure, not a silent pass.

Chasing that third one turned up something you did not ask for, and I would like your read on it: the console printer was discarding error_message entirely, so every failing case printed a bare FAIL and the reason only reached the CSV output. A guard that cannot say why it tripped is not much of a guard, so print_test_console now appends the message. That changes the failure line for every test, not just these, from FAIL to FAIL [compare failed] and so on. Say the word if you would rather have that as its own PR.

Also from your nits:

• required_fusion() is evaluated once into a local.
• Every failure reason is reported, so a numeric failure stays visible when the guard trips as well. The old expression hid test failed behind fusion did not fire.
• fused_binary is split into fused_add and fused_mul, counted inside their own branches, so a stray fused MUL can no longer satisfy the multi-ADD case.
• The GGML_TQ_Q8CACHE to GGML_CUDA_Q8CACHE rename is called out in the body as breaking with no alias.

On MUL_MAT_RESIDUAL_FUSION: agreed it is the case the #343 review cared about most, and I would rather send it separately than widen this one. The counter belongs at the mul_mat fusion dispatch sites, and I want to place it deliberately across the bias, GLU and residual paths rather than pick one and call it covered.

MI210 (gfx90a), GGML_TQ_MMQ=1 GGML_TQ_NATIVE=1, head af1b860:

suite	result
-o ELEM_CHAIN_FUSION	5/5
-o MUL_MAT_SHARED_SRC1	12/12
-o MUL_MAT_RESIDUAL_FUSION	24/24
-o MUL_MAT_ID_REDUCE	6/6
-o MUL_MAT -p type_a=tq4_1s / tq3_1s	149/149, 158/158
-o MUL_MAT_ID -p type_a=tq4_1s	44/44

The guard itself, checked in both directions:

case	result
GGML_CUDA_FUSE_CHAIN=0 -o ELEM_CHAIN_FUSION	1/5, FAIL [fusion 'elem_chain' did not fire]; the add-run case still passes because the chain hands pure ADD runs to the tuned kernel
GGML_CUDA_Q8CACHE=0 -o MUL_MAT_SHARED_SRC1	6/12, FAIL [fusion 'q8_cache_hits' did not fire]; the six failures are exactly the guarded Q8_0 and Q4_0 cases at n <= 4
counter name typo'd in a throwaway build	1/5, FAIL [unknown fusion counter 'elem_chian']
GGML_CUDA_DISABLE_FUSION=1	5/5 and 12/12, guard stands down
GGML_CUDA_DISABLE_FUSION=0	5/5, guard active

Thank you for the offer to push 1 to 3 yourself. The hands are slow but they work, and this was small enough to be worth doing here. I am glad you caught the CDNA1 table; that is not a case this machine can reach.

[TheTom]

All three land the way I hoped, and the CDNA1 bound being a literal with the source named in the comment is the right call given the header constraint. The console printer change is welcome here rather than as its own PR: a guard that cannot say why it tripped is exactly the thing this PR exists to fix, and FAIL [compare failed] is strictly more useful for every other test too. Splitting fused_binary into add and mul closes the stray-MUL hole. MUL_MAT_RESIDUAL_FUSION as a separate PR placed across the bias, GLU and residual dispatch sites is the better plan; no rush on it. Merging.