fix(matmul-nbits): free autotune dequant scratch when a fused config wins

[BoarQing]

Summary

Two commits. The first frees a large autotune scratch buffer that is never read; the second removes the triplication that forced the fix to be written three times.

1. Release the dq scratch when the autotune winner is fused

tuneWmmaConfig benchmarks both fused and dequant-then-gemm candidates, so it has to materialise the N*K fp16 scratch even when a fused candidate ends up winning. g_dq_buf is process-global and only ever grows, so one losing candidate on the widest matmul in the model pins that size for the life of the process.

On gemma-4-12B the vocabulary projection is 262144x3840, which is 1.875 GiB of scratch that nothing goes on to read. That is a large fraction of the 12.571 GiB carveout on gfx1151.

releaseDqBufferIfWinnerFused() frees it once the winner is known to need no scratch. The dispatch paths call ensureDqBuffer() again if a non-fused config is selected, so a fused winner simply stops holding memory it will not touch.

2. Unify the three WMMA autotune sweeps

Writing that fix exposed the actual problem: it had to be pasted into tuneWmmaConfig, tuneWmmaConfigU3 and tuneWmmaConfigU2 at the same point, because the three tuners are copies of a single sweep. Every autotune change has to be made three times, and when it isn't, the copies drift.

They have already drifted. The u3 and u2 tuners gained a clock-settle phase — without it the GPU is still on boost clock for the early configs and throttled by the late ones, which biased the sweep by table position (the u3 comment records dq+gemm configs, which sit last, looking 30-40% slower than they are). The 4-bit tuner never got it.

The sweep now lives once in tuneWmmaSweep(), templated on the config type because the three tables are distinct structs that share the fields the sweep reads. The caller passes a launch lambda and a skip predicate; remaining differences go in a WmmaTuneParams struct.

Each variant keeps exactly the values it had:

variant	warmup	iters	default_id	clock settle	skip predicate
4-bit	1	5	9	no	bn < 64 && N > 2*bn
u3	2	7	12	yes (id 12)	bn > N*2
u2	2	7	12	yes (id 12)	bn > N*2

Reconciling those would be a behaviour change rather than a refactor, so the differences are preserved and recorded in the struct instead of smoothed over. The missing 4-bit clock settle is deliberately left alone and flagged in a comment — see the note below.

Net for the refactor commit: 139 insertions, 195 deletions, and releaseDqBufferIfWinnerFused() is called from one place.

Test plan

• Builds clean on gfx1151, no new warnings from this file (only the pre-existing occupancy -Wpass-failed notes).
• gemma-4-12B int4-kquant, 270-token multimodal prompt + 8 decode steps: passes, and the 4-bit sweep runs all 8 shapes.
• Both branches of the fix are exercised in that run: one shape selects a dq+gemm winner (scratch correctly retained), the rest select fused (scratch released).
• The u3 and u2 instantiations are compile-verified but not exercised at runtime — this model/prompt only drives the 4-bit tuner. Worth covering on a runner with a u2/u3 model.
• The 1.875 GiB reclaim is not directly measured here; see the honesty note below.

On verifying that the refactor did not change tuning

Comparing selected config ids before and after the refactor shows differences, so I measured whether that means anything. Same prompt, autotune cache cleared before each run, four runs — two with the pre-refactor DLL and two with the post-refactor DLL:

shape (M=270)	old #1	old #2	new #1	new #2
N=15360 K=3840	42	13	13	13
N=2048 K=3840	4	4	4	0
N=3840 K=15360	11	10	12	13
N=3840 K=4096	38	39	13	40
N=3840 K=8192	12	13	12	12
N=4096 K=3840	50	50	50	50
N=512 K=3840	37	35	0	35
N=8192 K=3840	42	44	13	42

The same binary run twice disagrees on 6 of 8 shapes; old-vs-new disagrees on 5 of 8. N=3840 K=4096 returns four different winners across four runs, two of which are the same binary. Only one shape is stable throughout.

So the refactor is indistinguishable from run-to-run noise, which is the honest claim — I cannot assert bit-identical tuning, because the 4-bit tuner does not produce identical tuning against itself.

This is a pre-existing condition, not something introduced here, and it is the same phenomenon the disk cache exists to paper over ("several configs are within measurement noise of each other -- can settle on a DIFFERENT winner run-to-run"). It is also fairly direct evidence that the 4-bit tuner wants the clock-settle phase u3/u2 already have. I have not added it, since that is a behaviour change and belongs in its own PR, but I am happy to follow up.

Open question for reviewers

g_dq_buf and g_dq_buf_elems are mutated here outside any mutex. The autotune entry points are serialised by their per-variant tune mutex, but ensureDqBuffer() is also reachable from the dispatch paths. If two streams can dispatch concurrently this was already racy before this PR, and the fix does not make it worse — but it would be good to have someone who knows the threading model confirm whether that is actually reachable.

[github-actions]

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

1. The change is focused. Substantial work links the relevant issue
   or design discussion.
2. The PR documents relevant test results and updates affected
   documentation.
3. If AI tools provided substantial assistance, the description
   explains what was assisted and how it was validated, and commit
   trailers identify the tool. The contributor has reviewed and
   understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

[github-actions]

L2 Accuracy Results (EP vs CPU)

Model	Combined L2	Total Elems	Skipped NaN/Inf
conv_test_hybrid	4.8668E-07	64	0
GroupQueryAttention_seq256	25.2366	2621440	0
MatMulNBits_o_seq128	259.906	368640	0
QMoE_seq128	34.9552	368640	0

Threshold: 0.01 | Run: 3614 - Commit: 13e6f5f

[github-actions]

MorphiZen EP Performance Results

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.51	6.31	363	3	1244
GroupQueryAttention_seq128	4493.43	1.7029	11	6	311
matmul_down_seq128	525.36	2.36	74	3	352

EPContext Export Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.54	44.54	357	3	15591

EPContext Import Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.53	9.68	364	3	15761

OGA Benchmark Results

Model	Warmup	Reps	Prompt Len	Gen Tokens	TTFT (ms)	TPS	Peak Mem (GB)	GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32	1	5	128	128	169.2	77.5	1.33	13.53
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	1	5	128	128	272.1	40.7	1.22	6.43

OGA Wheel Smoke (Python benchmark_e2e.py)

Model	TTFT (ms)	TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	194	39.6

Run: 3614 - Commit: 13e6f5f

[BoarQing]

acc:http://xsjvitisaijenkins:8080/job/daily_hip_ep_oga/1680/
perf: http://xsjnts.xilinx.com/ROCm/compare?base_url=http%3A%2F%2Fxsjvitisaijenkins%3A8080%2Fjob%2Fdaily_hip_ep_oga%2F1633&base_job=&base=&curr_url=http%3A%2F%2Fxsjvitisaijenkins%3A8080%2Fjob%2Fdaily_hip_ep_oga%2F1679%2F&curr_job=&curr=

not ready

[BoarQing]

Verify:
http://xsjvitisaijenkins:8080/job/daily_hip_ep_oga/1681/
http://xsjvitisaijenkins:8080/job/daily_hip_ep_oga/1682/