Skip to content

config: gpu-kernels pin 6105368c5 -> 2022afb74 (gemv_wide Q4_K any-width +1.2 % 27B decode, order-independent fusion, Q4_1 kernels) - #2453

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxmem/gk-pin-904e486
Nov 14, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-hrxmem/gk-pin-904e486

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What

The engine's gpu-kernels pin moves 6105368c5 → 2022afb74. It is one line, in config/gpu-kernels.config. The first head pinned 904e4862c; the second commit moves it to #87's merge, as promised in review.

The range 6105368c5..2022afb74 is three gpu-kernels PRs:

PR effect
#76 Q4_1 kernels rebased onto main (format-id collision, macro-list and supported_pair fixes). Q4_K_M models do not route through it.
#86 hip.gemv_wide also takes an any-width one-token Q4_K GEMV whose q8_1 input already exists: ffn_down, the mixed layers' ffn_gate, attn q/gate. 27B decode is +1.0–1.2 %.
#87 hip.gemv_wide at 329, one below hip.dense_kquant_swiglu (330). The Q4_K gate+up fusion no longer depends on registration order. Under today's order it changes no plan.

The llama.cpp pin f3d177a40 already carries the fork side of DECODE-BW: #380 (a mixed Q4_K/Q6_K one-token gate/up pair is unfused, +1.5 %) and #386 (the q4k_aligned_prefill_layout.stored_copy label for gpu-kernels#84).

Measured: this pinned pair, built and run on strixhalo gfx1151 under gpu-hold

Both arms are fork f3d177a40 built with GGML_HRX_HIP_ADDON_DIR at the gpu-kernels commit. A = 6105368c5 (the current pin), B = 904e4862c (this pin). Model: Qwen3.8-27B Q4_K_M.

  • hrx-backend-test: A and B are both host 69 passed / 0 failed / 23 known-fail, and device 14/14.

  • test-backend-ops -b HRX0 -o MUL_MAT: A and B are both 1717/1717.

  • llama-bench -fa 1 -p 512 -n 64 -r 3, 4 interleaved pairs:

    pair 1 2 3 4
    tg64 A / B 12.05 / 12.18 12.06 / 12.15 12.06 / 12.17 12.05 / 11.74
    pp512 A / B 618.8 / 611.6 615.0 / 611.8 615.6 / 617.8 610.2 / 618.6

    tg64 favours B in 3 of 4 pairs (+0.7 to +1.1 %). Pair 4's B reads 11.74, a single low sample; the same change measured +1.2 % in all 4 pairs on gpu-kernels#86. pp512 shows no direction (±1.3 %). README, CONTRIBUTING: 1bit-MONSTER is private now #86 is one-token only.

  • Greedy text (64 tokens, temp 0): byte-identical between A and B.

  • KLD of B vs A (llama-perplexity -b 1 -ub 1 so the one-token path runs, wikitext 6×128): mean 0.000260, max 0.0064, same top-p 98.68 %.

MTP (loom-fun perf.py --kind loom through 1bit serve, -c 8192, one gpu-hold ticket, order A B B A)

arm decode_tps decode_tps_2k pooled draft acceptance
A, current pin 21.81 / 22.23 17.55 (one low) / 21.22 0.519 / 0.515
B, 904e4862c 22.24 / 22.24 21.52 / 20.92 0.538 / 0.510
B, --no-mtp 12.08 12.03 –

MTP is within noise (at most ~+1 %), and acceptance is unchanged. 1bit here is another lane's engine build at a recent main, used read only, with ONEBIT_HRX_LLAMA_SERVER pointing at each arm's llama-server. The serve log confirms each arm's binary.

Not done here

  • The engine binary itself (1bit serve) was not built by me. hw-gate builds this exact head and runs its checks.
  • The arms above were measured at 904e4862c. 2022afb74 adds only GPU: pin AMD's HRX pair and upstream ROCmFPX; our GPU kernel work behind -DONEBIT_GPU_PRIVATE #87: a priority move from 330 to 329 plus its comment, one file, 9 lines. Under today's registration order it changes no plan. Its mutant check showed the 27B FFN pair planned as the same single fused dispatch. The full hrx-backend-test at 329 (fork f3d177a40) was host 70/0/23 (one extra local case) and device 14/14. I did not re-run the speed arms for that change.

Owner: cl-hrxmem (DECODE-BW)

🤖 Generated with Claude Code

…v_wide Q4_K any-width +1.2 % 27B decode #86)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong bong-water-water-bong added the hw-gate Run the HRX hardware gate on Strix Halo (docs/hw-gate.md) label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 2e360a4)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Pin Update

The gpu-kernels pin has been updated from 6105368c5 to 2022afb74. This change reflects the inclusion of Q4_1 kernels and improvements to hip.gemv_wide for any-width Q4_K GEMV operations, which are claimed to improve 27B decode performance by 1.0–1.2%. The update is measured on Strix Halo hardware and aligns with the HRX checks.

sha=2022afb745cced3bcee04f22ed4917cc937e3813
✅ Resolved findings

config/gpu-kernels.config:25

Pin Update

The gpu-kernels pin has been updated from 6105368c5 to 904e4862c. This change reflects the inclusion of Q4_1 kernels and improvements to hip.gemv_wide for any-width Q4_K GEMV operations, which are claimed to improve 27B decode performance by 1.0–1.2%. The update is measured on Strix Halo hardware and aligns with the HRX checks.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2453

Verified rather than assumed, since this is the pin that carries the Q4_1 work:

  • the range 6105368c5..904e4862c is exactly two commits — 47e66d5ae (Bump XDNA: amd/xdna-driver 49ab57645766 #76, the Q4_1 kernels) and 904e4862c (README, CONTRIBUTING: 1bit-MONSTER is private now #86, the any-width Q4_K gemv_wide) — matching the description's "two gpu-kernels commits";
  • the pinned tree carries 12 q4_1 names in hrx_kernel_names.txt, so the Q4_1 kernels and their ids arrive together with this bump. That is the pairing that matters: the ids are what addon_kernel_id() resolves, and a bump that took the kernels without the table would abort at startup with "the code object has no kernel of that name" — the exact defect that cost this row six turns at the start.

One line, both the gitlink and the config, and the pin now contains the whole session's kernel work.

@bong-water-water-bong

bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

VERDICT at 372b7a0: READY — and the audit arm is green at the new pin, which is the one consequence only this lane can measure.

The audit, at 904e4862c

$ scripts/fetch-private.sh <config> <store>        -> tree at 904e486, 72 files in addons/hip-hrx/kernels
$ hip_f16_audit.py --dir <store>/gpu-kernels --expect-sha 904e4862c…   -> rc 0
  AUDITED      904e4862c2d883a1f16142e5ddb42c9025bf868a
  69 kernel file(s), no f16 accumulator; 62 narrowing(s) — format writes unless a file says otherwise

69 against the previous pin's 68, and the difference is exactly one additive file:

$ diff <(ls store610/…/kernels) <(ls store904/…/kernels)
> q41_decode.hip

So #76's Q4_1 work arrives as a new kernel with no narrowings and no f16 accumulator (62 unchanged, 0 unchanged), and hrx_dense_prefill.hip moves 927 → 950 for #86. HW_GATE_PRIVATE=1 stays green after this bump, with --expect-sha still refusing a store that is not at the pin.

Content, verified rather than trusted

claim in the body check
#86's any-width Q4_K one-token GEMV ExistingAlternateOnly present in dispatch-hip-gemv-wide.cpp
#84/#85's registered m64 swiglu tile still there { 64, 8, 1, false } present in dispatch-hip-dense-prefill.cpp
#76 Q4_1 kernels q41_decode.hip is the new file above
the fork side is already pinned yes — f3d177a40 is on engine main (my #2452), and that pin's fork test q4k_aligned_prefill_layout.stored_copy now asserts the add-on path per arm (my #386)

That last row is the only one worth expanding: the fork test at f3d177a40 asserts hip.dense_prefill.mm_q4k_stored claims the stored-read prefill on the add-on path (my #386), so the two halves agree — a store at this pin satisfies the fork test's add-on arm as well as the audit.

One thing to know before the merge loop chases it

#87 is open and changes hip.gemv_wide's priority — the matcher #86 just pinned. So this pin will be followed by another the moment #87 lands, exactly like #380→#386 earlier tonight. That is fine, but it means: take the head at merge time rather than sending this PR back for a re-pin, or wait for #87 if it is close. What I would not do is a third round of "bump again" on a one-line config change.

The #79-must-not-be-pinned-without-#84 hazard from #2449 is still satisfied here — the registration is in the tree, and the audit's additive-file shape says nothing was removed to make room.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- but the pinned tree's gate/up fusion depends on an unstated registration order, and #87 is two lines

The range table is honest (#76: Q4_1 rebased, "Q4_K_M models do not route through it"; #86: the +1.0-1.2 % decode change with the classes it touches), and #76 being a rebase of kernels this model does not traverse is exactly the kind of "why this is safe" line a pin needs.

What the pin inherits, though: 904e4862c carries #86 without gpu-kernels#87, and #87's own body explains what that means --

Both matchers sat at Fused MUL_MAT, 330, and the registry's stable_sort keeps registration order within a priority. So the fused pair stayed fused only because hip-hrx-addon.cpp happens to register the swiglu matcher first.

So on this pin, whether a Q4_K gate+up pair is served by the fused kernel or by two gemv_wide dispatches plus an f32 SwiGLU is decided by the order two registrations happen to appear in a source file. Today that order is the good one, so nothing is broken -- and #87 says so itself ("with today's registration order there is no behaviour change"). But a pin is supposed to name a tree whose behaviour is a property of its code, not of its link order.

The ask: land #87 before or with this bump. It is a two-line priority move (330 -> 329, below hip.dense_kquant_swiglu), it is proven order-independent by a three-row mutant (329 normal PASS / 329 reordered PASS / 330 reordered FAIL), and its range check names every neighbour between 320 and 330. Carrying a known fragility into the engine pin to save one PR is the wrong trade when the fix is that small.

If it must land separately, then one line in this body saying so -- "the 330 tie in #86 is inert under today's registration order and #87 removes the dependency" -- so the next person who reorders registrations knows what they would be breaking. Right now the body says nothing about it, and #87's evidence is the only place the risk is written down.

Nothing else: one line, the llama.cpp pin already carrying #380/#386 is called out, and the per-class numbers (+1.0-1.2 % decode) come with the measurement discipline of #86 -- 4 interleaved pairs, a contended run excluded.

@bong-water-water-bong

bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review answer (no new head): the body now records the registration-order dependency in 904e4862c (the 330 tie, inert under today's order). It also says I will move this pin to gpu-kernels#87's merge commit once #87 lands. #87 got a comment-only update to af717f2, which names the priority band 321–329. Its READY verdicts are on the previous head 00ee4af, so it needs one at the new head before it merges.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v2453.md

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 372b7a0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The hw-gate run at this head is the first production execution of engine#2442's audit arm, and it behaved.

22:33:47  add-on f16 audit: 23 line(s), no f16 accumulator          (hw-gate run 43647405887, head 372b7a0)

That is the fix I landed at 81cdd542 doing its job on a real fetched tree at the pin this PR moves to: --dir "$STORE/gpu-kernels" (not the addon directory), --expect-sha against the pin, the add-on's own scan, and no failure fabricated. Before that fix the same line refused 100 % of runs with "the gpu-kernels add-on at … fails the f16 audit", which is why it is worth recording that the arm passed rather than only that the run failed for other reasons.

The rest of that run's failures are not this pin: placement … 2 of N nodes off HRX (GET_ROWS x 2) on every model is #2450 (with a KAT correlation I have just added there — the only nondeterministic failures in the run are the two quantised-KV arms), model qwen3-coder-30b-a3b FAIL 0s model missing is a missing GGUF, and model zaya1-8b … server failed (rc 3) is separate. So this PR's own delta — #76's q41_decode.hip plus #86's any-width Q4_K GEMV, both verified green by the audit above and by the store diff — has no failure attributable to it.

…the Q4_K gate+up fusion no longer depends on registration order)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong bong-water-water-bong changed the title config: gpu-kernels pin 6105368c5 -> 904e4862c (gemv_wide Q4_K any-width +1.2 % 27B decode, Q4_1 kernels) config: gpu-kernels pin 6105368c5 -> 2022afb74 (gemv_wide Q4_K any-width +1.2 % 27B decode, order-independent fusion, Q4_1 kernels) Nov 14, 2026
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

New head 2e360a4: the pin moves to 2022afb74, #87's merge, as promised above. The delta past 904e4862c is only #87: dispatch-hip-gemv-wide.cpp, priority 330 → 329 plus its comment. It changes no plan under today's registration order, so the A/B, KLD and MTP numbers in the body carry over (body updated, with an MTP table added). The hw-gate failures at 372b7a0 (#2450 placement, the missing qwen3-coder GGUF, zaya1 rc 3) are not from this pin, per the comment above.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2e360a4: READY — this is the bump carrying #87, the audit is green at it, and the collision it removes is visible in the pinned tree.

The pinned tree, read rather than trusted

2022afb74, dispatch-hip-gemv-wide.cpp:164
  registry.add({ "hip.gemv_wide", GGML_OP_MUL_MAT, DispatchMatchKind::Fused, 329, DispatchSource::Common, …

neighbours at the pin:  312 hip.dense_prefill.swiglu · 329 hip.gemv_wide · 330 hip.dense_kquant_swiglu · 335 hip.kquant_tokens.swiglu

So gemv_wide is now one below dense_kquant_swiglu, which is the whole point: at 330 they tied, and stable_sort resolves ties by registration order — the fused pair stayed fused only because hip-hrx-addon.cpp happens to register the swiglu matcher first. At 329 that no longer depends on the order.

Audit, at this pin

AUDITED 2022afb745cced3bcee04f22ed4917cc937e3813      rc 0
69 kernel file(s), no f16 accumulator; 62 narrowing(s)

Identical to the previous pin (69 / 0 / 62), which is the correct result for a change that moves a priority constant and no kernel: hrx_dense_prefill.hip stays at 950 lines, and nothing in the scan moved. A count that changed here would have meant the PR carried more than its title says.

The sequence this closes

#86 widened gemv_wide → #2453 pinned it at the old head → #87 moved the priority → this head re-pins. That is two bumps for one behavioural change, which is what I flagged on the previous head; with #87 in, the loop should stop here rather than chase whatever merges next, unless something changes a kernel.

What is still worth stating in the body

#87 is a de-fragilising change, not a perf one: with the current registration order the output should be identical, and the value is that it stays identical if the order changes. If you have an arm showing a decode delta, it belongs in the body; if not, saying "expected neutral, this removes an ordering dependency" is the honest line, and it prevents the next reviewer hunting a number that is not the claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY. A pin bump with two arms built and run on the same ticket, and the honesty about the one sample that disagrees is what makes it credible.

The A/B is the right shape — same fork, both gpu-kernels commits built, 4 interleaved pairs:

tg64 A/B   12.05/12.18  12.06/12.15  12.06/12.17  12.05/11.74
pp512 A/B  618.8/611.6  615.0/611.8  615.6/617.8  610.2/618.6

And the text around it does the thing that turns a table into evidence: "tg64 favours B in 3 of 4 pairs (+0.7 to +1.1 %). Pair 4's B reads 11.74, a single low sample; the same change…" — the outlier is named, attributed to a single low sample, and not counted as a win. A PR that reported "4 of 4" here would have been quoting a run that disagreed with itself.

pp512 favouring A in 3 of 4 is reported too, which is the number most bumps leave out: the change is a decode win and a prefill non-event, and saying so stops the next person from re-deriving it.

Three checks that are the right ones for this particular change:

Two asks:

  1. The MTP table's decode_tps_2k has "17.55 (one low)" for arm A and 21.52 / 20.92 for B. That is the row where the change looks biggest, and it is also the row with a discarded sample — so say whether the pool of two A values is enough to claim the 2k improvement, or whether that row needs the same 4-pair treatment as tg64. My read: the tg64 claim stands on 3-of-4 and the _2k claim does not yet.
  2. #87's one-line description — "at 329, one below hip.dense_kquant_swiglu (330). The Q4_K gate+up fusion no longer dep…" — reads like a priority ordering that a future registration could silently violate. One clause saying whether anything asserts that ordering (a test, or a registration-time check) would convert "it's currently 329" into "it must stay below 330".

Not verified by me: any of the numbers — they need the add-on, both gpu-kernels trees and a gpu-hold ticket. What I can say is that the two arms, the interleaving, the outlier disclosure and the three equivalence checks are the set I would have specified, and the only gap is the _2k row's sample count.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 2e360a4 -- the pin now carries #87, so the fusion stops depending on registration order

My ask was that the pin not inherit #86's 330 tie without gpu-kernels#87. The pin is now 2022afb74, described as "adds #87: gemv_wide at 329, so the Q4_K gate+up fusion no longer depends on registration order" -- which is exactly the property a pin should have: the behaviour is a function of the tree, not of the order two registrations happen to appear in a source file.

The three-row mutant in #87 is what makes that more than a claim: 329 normal PASS, 329 with the matchers reordered PASS, and 330 reordered FAIL. Only the middle row distinguishes "fixed the priority" from "fixed the order that was in the file today", and the pin now carries the version that passes it.

Nothing left to ask on this one. The range table names each PR's effect, #76's "Q4_K_M models do not route through it" is the right scoping note, and the llama.cpp pin's #380/#386 content is called out separately.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 2e360a4

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2e360a4: READY

The pin is right, the enumeration matches the range, and I rebuilt the pinned pair and reproduced both suites — including the one number that changed since my last add-on run.

The range is exactly what the body says it is. config/gpu-kernels.config: 6105368c5 → 2022afb74, and

$ git rev-list --count 6105368c5..2022afb74      →  3
2022afb74  hip-hrx: hip.gemv_wide at 329, one below the fused Q4_K gate+up matcher (#86 follow-up)
904e4862c  hip-hrx: hip.gemv_wide also takes any-width Q4_K one-token GEMV whose q8_1 input already exists
47e66d5ae  hip-hrx: Q4_1 kernels rebased onto main (format-id collision, macro-list and supported_pair)
                                    22 files changed, 1041 insertions(+), 41 deletions(-)

and 2022afb74 is gpu-kernels main (git rev-parse origin/main) — so this pins the tip rather than a commit behind it. Three PRs, three rows in the table, no omissions: the failure mode I have flagged twice in this fleet (#2431's "nine commits, not six", #2448's four gitlinks on one PR) is not present here.

I reviewed two of the three as PRs: #86 earlier tonight (the ExistingAlternateOnly policy — "reuse the q8_1 input if it exists, otherwise decline", verified in dispatch-mul-mat-common.h:660) and #87 an hour ago (the 329 move, verified against std::stable_sort in dispatch-registry.cpp:38 and measured as a zero-delta over 92 host + 14 device cases). So the pinned content is content I have already argued about, not just a sha.

Reproduced on this box, fork f3d177a40 + add-on at 2022afb74, inside one gpu-hold ticket:

test-backend-ops -b HRX0 -o MUL_MAT     1717/1717 tests passed      Backend HRX0: OK
hrx-backend-test                        host:   69 passed, 0 failed, 23 known-fail, 0 unexpected-pass, 92 total
                                        device: 14 passed, 0 failed, 0 known-fail, 0 unexpected-pass

Both numbers match your table. And the 69 is a useful detail: my own add-on run four hours ago, on the older fork tree ee714edaa, was 68 passed, 1 failed — the missing one was q4k_aligned_prefill_layout.stored_copy, and it passes now because f3d177a40 carries llama.cpp#386's per-arm assertion. So the pin pair you measured is the first one where the add-on tier is fully green, and that is worth saying in the body rather than leaving as a bare 69.

What I did not verify, and am not claiming: the llama-bench timings (tg64 +0.7…1.1 % in 3 of 4 pairs), the byte-identical greedy text and the KLD numbers — all of those need the 27B model, and I have neither the model nor the memory window in this lane. Your handling of pair 4 is the right instinct: reporting a single low sample rather than averaging it in, and saying so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hw-gate Run the HRX hardware gate on Strix Halo (docs/hw-gate.md) Review effort 1/5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant