metal : per-device tuned (Q, NE) for flash-attn vec (replay) by forforever73 · Pull Request #26570 · ggml-org/llama.cpp

forforever73 · 2026-08-04T11:23:30Z

Overview

Stacked on top of #26561.

Layer	Branch	Content
L1	`stack/metal-split`	Per-op source split (#24021), plus all kernels merged into `master` since then
L2 (this PR)	`stack/metal-fa-vec`	Per-device FlashAttention vector tuning (Q, NE) (#25750)
L3	`stack/metal-tuner`	Standalone FA-vector tuning tool (#26498)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, opus 5

forforever73 · 2026-08-04T11:27:30Z

Looks like the stack is wired up correctly :)

forforever73 · 2026-08-04T11:44:09Z

@ggerganov Yes, the button was there on both PRs and I used it from #26561. It worked as expected

ggerganov · 2026-08-11T06:27:00Z

        output_printer->print_failed_tests(failed_tests);

-        return n_ok == tests_run;
+        const bool slice_ok = run_fa_vec_slice(backend, backend_cpu.get());


This currently runs unconditionally for every test-backend-ops. For example:

test-backend-ops -o ADD

would still run it.

Thanks. Fixed — the slice now only runs when -o selects FLASH_ATTN_EXT, or when no -o is given (unchanged default).

ggerganov · 2026-08-11T07:50:11Z

Are you planning additional changes to the stack?

forforever73 · 2026-08-11T08:06:15Z

@ggerganov Yes, I have one more follow-up planned from #25750 the “Aggregate target-selection gate.” Do you think it would make more sense to stack it on top of this, or submit it as a separate optimization PR?

ggerganov · 2026-08-11T08:09:02Z

Aggregate target-selection gate: the θ=1.05 gate compares baseline vs target by summed time over the bucket's sampled points (ratio-of-sums), which weights the heaviest points most — a win concentrated there can pass the gate while lighter points barely benefit. Making it per-point / worst-case (as the τ compression pass already is) would be more robust.

Can you explain the goal in some simpler way - I don't understand the paragraph.

forforever73 · 2026-08-11T09:27:35Z

@ggerganov Sure. The table only overrides the baseline kernel when the win is large enough to justify a path that differs from today's backend. I use ~5% as the noise-floor threshold: below that, the difference is within the harness's run-to-run variation and doesn't warrant a table entry.

The threshold is applied to the sum of kernel time across the entry's sampled (KV-depth, batch-width) points. As a result, an entry can be net-positive even when individual points regress. Nearly all of the regressions occur at odd batch widths, where Q>1 configurations waste padded rows. For example, q5_0 with dk=320, dv=256 at 32k KV gains 8-13% at ne01=6/8/16, but regresses 10% at ne01=5 and 5% at ne01=7. The entry is still ~5% faster in aggregate, so it passes the gate.

Across the full table, this accounts for 26 of 1414 tuned sample points (1.8%), against a 1.33x median speedup. All five regressions above 5% are in the ne01>=5 bucket, which deliberately collapses everything from 5 onward into a single min-max-regret backstop, since real draft-n-max is typically <=3. Within ne01<=4, every regression is below 5%, i.e. within the noise floor the threshold is designed to filter out.

The aggregate-sum gate is a deliberate trade-off for that reason, and it is net-positive in the current sweep, but it isn't the final gating strategy.

ggerganov · 2026-08-12T10:35:40Z

Ok, up to you if you want to add it to the stack.

LMK when ready - I want to do 2 things before considering to merge:

Run tunings on my Macs + benches on some common models vs master
Check builds with whisper.cpp

forforever73 · 2026-08-12T10:46:35Z

OK, I’ve been a bit tied up lately, but I should be able to fill in the stack content sometime next week. I’ll let you know once it’s ready.

forforever73 · 2026-08-22T06:11:48Z

@ggerganov Ready now — the aggregate gate is replaced by a pointwise one: a candidate can only claim a bucket if it is no slower than the baseline at every sampled point it covers. And rebase on the latest master.

ggerganov · 2026-08-22T06:20:30Z

Great - let's see if we can get this in over the weekend. Will work on the plan above:

Run tunings on my Macs + benches on some common models vs master

Check builds with whisper.cpp

ggerganov · 2026-08-24T11:40:23Z

2ec0cd8 adds fa-vec tunings on M1 Pro, M2 Ultra and M5 Max

forforever73 · 2026-08-24T11:52:46Z

@ggerganov Nice, do you have any end-to-end performance results?

ggerganov · 2026-08-24T12:12:11Z

@ggerganov Nice, do you have any end-to-end performance results?

A quick test on M2 Ultra with the bench from #27390 (comment):

CPU	Model	Test	t/s master	t/s stack/metal-fa-vec	Speedup
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32	240.02	240.28	1.00
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32@d2048	179.87	235.69	1.31
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32@d4096	160.89	214.76	1.33
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32@d8192	143.58	175.74	1.22
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32@d16384	102.05	136.16	1.33
Apple M2 Ultra	qwen3 0.6B Q8_0	tg32@d32768	64.97	98.94	1.52

The base branch was changed.

ggerganov · 2026-08-24T16:30:52Z

+# ggml-metal-tuning
+
+Offline kernel tuner for the Metal backend.
+It sweeps a kernel's config grid on the machine it runs on and prints pasteable table rows for `ggml/src/ggml-metal/ggml-metal-tuning.cpp`.
+
+This is not a test: it never reports pass/fail on performance.
+A non-zero exit code means bad arguments or a wrong environment (no Metal device, missing proc bridges), never a perf result.
+
+| tuner | tunes | table |
+|---|---|---|
+| `fa-vec` | flash-attn vec `(Q, NE)` per `(dtype, head size, KV depth, batch width)` | `fa_vec_tuned_table` |
+
+## Adding a device to the FA-vec table
+
+Build on the target machine:
+
+```bash
+cmake -B build -DGGML_METAL=ON
+cmake --build build --target ggml-metal-tuning -j
+cmake --build build --target test-backend-ops -j
+```
+
+Sweep the grid (6 dtypes x 10 head sizes x 4 KV depths x 9 batch widths; a few hours):
+
+```bash
+./build/bin/ggml-metal-tuning fa-vec > fa_vec_rows.txt 2> fa_vec_sweep.log
+```


@forforever73 It would be useful to create a dedicated discussion with a summary of which families have been tuned and which haven't. This way we can get help from the community to run the tuning and collect data for more devices.

Sounds good. I can draft a checklist in a new Discussion thread to track the progress. But my laptop is off for the night, so I’ll have to wait for my morning. Or, if you prefer, you’re more than welcome to create it first

Added: #27668

* metal : per-device tuned (Q, NE) for flash-attn vec (ggml-org#25750) * rebase Q-generic FA vec body from 01dc936 (ggml-org#23114) * add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133) * add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback * add FA vec (Q,NE) perf sweep * fill tuning result * fold family table into a per-family representative SKU * refactor tuning result format * extend FA vec tuning to quantized KV caches * sync fa vec tuner bucketing with runtime, use pointwise tuning regret * update tuned table * format and cleanup * prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active * add device id -> token lookup for the offline tuning tool * add ggml-metal-tuning skeleton * add op-agnostic perf cell + median timing for the tuner * add FA-vec graph build + tensor init to the tuner * tools : add FA-vec (Q,NE) sweep, compression and table emit * cool down and re-measure the dirty window on thermal drift * test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice * tools : document the Metal tuner, point the table comment at it * abort on unknown KV type, single-source fa_vec_legal_ne * cleanup * honor -o in the FA vec (Q,NE) slice * retune FA-vec (Q, NE) under a pointwise no-harm gate * cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

nikwen · 2026-08-28T11:02:37Z

This is such a cool optimization, @forforever73. Thank you for it! 🙏

forforever73 · 2026-08-28T11:06:22Z

@nikwen Thank you! Let’s keep making the Metal backend better together !

* metal : per-device tuned (Q, NE) for flash-attn vec (ggml-org#25750) * rebase Q-generic FA vec body from 01dc936 (ggml-org#23114) * add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133) * add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback * add FA vec (Q,NE) perf sweep * fill tuning result * fold family table into a per-family representative SKU * refactor tuning result format * extend FA vec tuning to quantized KV caches * sync fa vec tuner bucketing with runtime, use pointwise tuning regret * update tuned table * format and cleanup * prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active * add device id -> token lookup for the offline tuning tool * add ggml-metal-tuning skeleton * add op-agnostic perf cell + median timing for the tuner * add FA-vec graph build + tensor init to the tuner * tools : add FA-vec (Q,NE) sweep, compression and table emit * cool down and re-measure the dirty window on thermal drift * test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice * tools : document the Metal tuner, point the table comment at it * abort on unknown KV type, single-source fa_vec_legal_ne * cleanup * honor -o in the FA vec (Q,NE) slice * retune FA-vec (Q, NE) under a pointwise no-harm gate * cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

forforever73 requested review from a team and ggerganov as code owners August 4, 2026 11:23

forforever73 changed the base branch from master to stack/metal-split August 4, 2026 11:25

This comment was marked as resolved.

Sign in to view

forforever73 force-pushed the stack/metal-split branch from 864c08e to 2934b5b Compare August 4, 2026 11:36

forforever73 force-pushed the stack/metal-fa-vec branch from 5d79b77 to aa4b599 Compare August 4, 2026 11:36

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Aug 4, 2026

forforever73 mentioned this pull request Aug 5, 2026

tools : split Metal FA-vec tuning into a standalone tuner #26620

Merged

forforever73 force-pushed the stack/metal-split branch from 2934b5b to 5522498 Compare August 5, 2026 08:22

forforever73 force-pushed the stack/metal-fa-vec branch from aa4b599 to 0f27935 Compare August 5, 2026 08:22

github-actions Bot added documentation Improvements or additions to documentation examples labels Aug 11, 2026

ggerganov reviewed Aug 11, 2026

View reviewed changes

forforever73 force-pushed the stack/metal-split branch from 5522498 to 89eea16 Compare August 22, 2026 05:54

forforever73 force-pushed the stack/metal-fa-vec branch from 040a9f2 to 1693e90 Compare August 22, 2026 05:55

forforever73 and others added 5 commits August 24, 2026 19:19

abort on unknown KV type, single-source fa_vec_legal_ne

f801d5d

cleanup

26bf53d

honor -o in the FA vec (Q,NE) slice

545c66e

retune FA-vec (Q, NE) under a pointwise no-harm gate

7658a24

cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max

efeda76

ggerganov force-pushed the stack/metal-fa-vec branch from 2ec0cd8 to efeda76 Compare August 24, 2026 16:21

ggerganov requested review from a team, CISC, JohannesGaessler, ngxson and pwilkin as code owners August 24, 2026 16:21

ggerganov changed the base branch from stack/metal-split to master August 24, 2026 16:21

ggerganov merged commit f280b26 into master Aug 24, 2026
2 checks passed

ggerganov deleted the stack/metal-fa-vec branch August 24, 2026 16:22

ggerganov reviewed Aug 24, 2026

View reviewed changes

BrewTestBot mentioned this pull request Aug 25, 2026

ggml 0.22.0 Homebrew/homebrew-core#300553

Merged

1 task

thypon mentioned this pull request Aug 31, 2026

Compile bug: tools/tuning hardcodes ${CMAKE_SOURCE_DIR}, breaks FetchContent consumers (ggml-metal-tuning.h not found) #28114

Open

8 tasks

Uh oh!

Conversation

forforever73 commented Aug 4, 2026

Overview

Requirements

Uh oh!

forforever73 commented Aug 4, 2026

Uh oh!

This comment was marked as resolved.

forforever73 commented Aug 4, 2026

Uh oh!

ggerganov Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

forforever73 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Aug 11, 2026

Uh oh!

forforever73 commented Aug 11, 2026

Uh oh!

ggerganov commented Aug 11, 2026

Uh oh!

forforever73 commented Aug 11, 2026

Uh oh!

ggerganov commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

forforever73 commented Aug 12, 2026

Uh oh!

forforever73 commented Aug 22, 2026

Uh oh!

ggerganov commented Aug 22, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

forforever73 commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 24, 2026

Uh oh!

Uh oh!

ggerganov Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

forforever73 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

nikwen commented Aug 28, 2026

Uh oh!

forforever73 commented Aug 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ggerganov commented Aug 12, 2026 •

edited

Loading