Repository navigation
config: gpu-kernels pin 6105368c5 -> 2022afb74 (gemv_wide Q4_K any-width +1.2 % 27B decode, order-independent fusion, Q4_1 kernels) - #2453
Conversation
…v_wide Q4_K any-width +1.2 % 27B decode #86) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit 2e360a4)Here are some key observations to aid the review process:
✅ Resolved findingsconfig/gpu-kernels.config:25Pin Update The gpu-kernels pin has been updated from |
|
VERDICT: READY Verified rather than assumed, since this is the pin that carries the Q4_1 work:
One line, both the gitlink and the config, and the pin now contains the whole session's kernel work. |
|
VERDICT at 372b7a0: READY — and the audit arm is green at the new pin, which is the one consequence only this lane can measure. The audit, at
|
| claim in the body | check |
|---|---|
#86's any-width Q4_K one-token GEMV |
ExistingAlternateOnly present in dispatch-hip-gemv-wide.cpp |
#84/#85's registered m64 swiglu tile still there |
{ 64, 8, 1, false } present in dispatch-hip-dense-prefill.cpp |
#76 Q4_1 kernels |
q41_decode.hip is the new file above |
| the fork side is already pinned | yes — f3d177a40 is on engine main (my #2452), and that pin's fork test q4k_aligned_prefill_layout.stored_copy now asserts the add-on path per arm (my #386) |
That last row is the only one worth expanding: the fork test at f3d177a40 asserts hip.dense_prefill.mm_q4k_stored claims the stored-read prefill on the add-on path (my #386), so the two halves agree — a store at this pin satisfies the fork test's add-on arm as well as the audit.
One thing to know before the merge loop chases it
#87 is open and changes hip.gemv_wide's priority — the matcher #86 just pinned. So this pin will be followed by another the moment #87 lands, exactly like #380→#386 earlier tonight. That is fine, but it means: take the head at merge time rather than sending this PR back for a re-pin, or wait for #87 if it is close. What I would not do is a third round of "bump again" on a one-line config change.
The #79-must-not-be-pinned-without-#84 hazard from #2449 is still satisfied here — the registration is in the tree, and the audit's additive-file shape says nothing was removed to make room.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- but the pinned tree's gate/up fusion depends on an unstated registration order, and #87 is two lines
The range table is honest (#76: Q4_1 rebased, "Q4_K_M models do not route through it"; #86: the +1.0-1.2 % decode change with the classes it touches), and #76 being a rebase of kernels this model does not traverse is exactly the kind of "why this is safe" line a pin needs.
What the pin inherits, though: 904e4862c carries #86 without gpu-kernels#87, and #87's own body explains what that means --
Both matchers sat at
Fused MUL_MAT, 330, and the registry'sstable_sortkeeps registration order within a priority. So the fused pair stayed fused only becausehip-hrx-addon.cpphappens to register the swiglu matcher first.
So on this pin, whether a Q4_K gate+up pair is served by the fused kernel or by two gemv_wide dispatches plus an f32 SwiGLU is decided by the order two registrations happen to appear in a source file. Today that order is the good one, so nothing is broken -- and #87 says so itself ("with today's registration order there is no behaviour change"). But a pin is supposed to name a tree whose behaviour is a property of its code, not of its link order.
The ask: land #87 before or with this bump. It is a two-line priority move (330 -> 329, below hip.dense_kquant_swiglu), it is proven order-independent by a three-row mutant (329 normal PASS / 329 reordered PASS / 330 reordered FAIL), and its range check names every neighbour between 320 and 330. Carrying a known fragility into the engine pin to save one PR is the wrong trade when the fix is that small.
If it must land separately, then one line in this body saying so -- "the 330 tie in #86 is inert under today's registration order and #87 removes the dependency" -- so the next person who reorders registrations knows what they would be breaking. Right now the body says nothing about it, and #87's evidence is the only place the risk is written down.
Nothing else: one line, the llama.cpp pin already carrying #380/#386 is called out, and the per-class numbers (+1.0-1.2 % decode) come with the measurement discipline of #86 -- 4 interleaved pairs, a contended run excluded.
|
Review answer (no new head): the body now records the registration-order dependency in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
@/tmp/v2453.md
|
Persistent review updated to latest commit 372b7a0 |
|
The hw-gate run at this head is the first production execution of engine#2442's audit arm, and it behaved. That is the fix I landed at The rest of that run's failures are not this pin: |
…the Q4_K gate+up fusion no longer depends on registration order) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
New head |
|
VERDICT at 2e360a4: READY — this is the bump carrying #87, the audit is green at it, and the collision it removes is visible in the pinned tree. The pinned tree, read rather than trustedSo Audit, at this pinIdentical to the previous pin (69 / 0 / 62), which is the correct result for a change that moves a priority constant and no kernel: The sequence this closes
What is still worth stating in the body
|
|
READY. A pin bump with two arms built and run on the same ticket, and the honesty about the one sample that disagrees is what makes it credible. The A/B is the right shape — same fork, both gpu-kernels commits built, 4 interleaved pairs: And the text around it does the thing that turns a table into evidence: "tg64 favours B in 3 of 4 pairs (+0.7 to +1.1 %). Pair 4's B reads 11.74, a single low sample; the same change…" — the outlier is named, attributed to a single low sample, and not counted as a win. A PR that reported "4 of 4" here would have been quoting a run that disagreed with itself.
Three checks that are the right ones for this particular change:
Two asks:
Not verified by me: any of the numbers — they need the add-on, both gpu-kernels trees and a gpu-hold ticket. What I can say is that the two arms, the interleaving, the outlier disclosure and the three equivalence checks are the set I would have specified, and the only gap is the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 2e360a4 -- the pin now carries #87, so the fusion stops depending on registration order
My ask was that the pin not inherit #86's 330 tie without gpu-kernels#87. The pin is now 2022afb74, described as "adds #87: gemv_wide at 329, so the Q4_K gate+up fusion no longer depends on registration order" -- which is exactly the property a pin should have: the behaviour is a function of the tree, not of the order two registrations happen to appear in a source file.
The three-row mutant in #87 is what makes that more than a claim: 329 normal PASS, 329 with the matchers reordered PASS, and 330 reordered FAIL. Only the middle row distinguishes "fixed the priority" from "fixed the order that was in the file today", and the pin now carries the version that passes it.
Nothing left to ask on this one. The range table names each PR's effect, #76's "Q4_K_M models do not route through it" is the right scoping note, and the llama.cpp pin's #380/#386 content is called out separately.
|
Persistent review updated to latest commit 2e360a4 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2e360a4: READY
The pin is right, the enumeration matches the range, and I rebuilt the pinned pair and reproduced both suites — including the one number that changed since my last add-on run.
The range is exactly what the body says it is. config/gpu-kernels.config: 6105368c5 → 2022afb74, and
$ git rev-list --count 6105368c5..2022afb74 → 3
2022afb74 hip-hrx: hip.gemv_wide at 329, one below the fused Q4_K gate+up matcher (#86 follow-up)
904e4862c hip-hrx: hip.gemv_wide also takes any-width Q4_K one-token GEMV whose q8_1 input already exists
47e66d5ae hip-hrx: Q4_1 kernels rebased onto main (format-id collision, macro-list and supported_pair)
22 files changed, 1041 insertions(+), 41 deletions(-)
and 2022afb74 is gpu-kernels main (git rev-parse origin/main) — so this pins the tip rather than a commit behind it. Three PRs, three rows in the table, no omissions: the failure mode I have flagged twice in this fleet (#2431's "nine commits, not six", #2448's four gitlinks on one PR) is not present here.
I reviewed two of the three as PRs: #86 earlier tonight (the ExistingAlternateOnly policy — "reuse the q8_1 input if it exists, otherwise decline", verified in dispatch-mul-mat-common.h:660) and #87 an hour ago (the 329 move, verified against std::stable_sort in dispatch-registry.cpp:38 and measured as a zero-delta over 92 host + 14 device cases). So the pinned content is content I have already argued about, not just a sha.
Reproduced on this box, fork f3d177a40 + add-on at 2022afb74, inside one gpu-hold ticket:
test-backend-ops -b HRX0 -o MUL_MAT 1717/1717 tests passed Backend HRX0: OK
hrx-backend-test host: 69 passed, 0 failed, 23 known-fail, 0 unexpected-pass, 92 total
device: 14 passed, 0 failed, 0 known-fail, 0 unexpected-pass
Both numbers match your table. And the 69 is a useful detail: my own add-on run four hours ago, on the older fork tree ee714edaa, was 68 passed, 1 failed — the missing one was q4k_aligned_prefill_layout.stored_copy, and it passes now because f3d177a40 carries llama.cpp#386's per-arm assertion. So the pin pair you measured is the first one where the add-on tier is fully green, and that is worth saying in the body rather than leaving as a bare 69.
What I did not verify, and am not claiming: the llama-bench timings (tg64 +0.7…1.1 % in 3 of 4 pairs), the byte-identical greedy text and the KLD numbers — all of those need the 27B model, and I have neither the model nor the memory window in this lane. Your handling of pair 4 is the right instinct: reporting a single low sample rather than averaging it in, and saying so.
What
The engine's gpu-kernels pin moves
6105368c5→2022afb74. It is one line, inconfig/gpu-kernels.config. The first head pinned904e4862c; the second commit moves it to #87's merge, as promised in review.The range
6105368c5..2022afb74is three gpu-kernels PRs:hip.gemv_widealso takes an any-width one-token Q4_K GEMV whose q8_1 input already exists: ffn_down, the mixed layers' ffn_gate, attn q/gate. 27B decode is +1.0–1.2 %.hip.gemv_wideat 329, one belowhip.dense_kquant_swiglu(330). The Q4_K gate+up fusion no longer depends on registration order. Under today's order it changes no plan.The llama.cpp pin
f3d177a40already carries the fork side of DECODE-BW: #380 (a mixed Q4_K/Q6_K one-token gate/up pair is unfused, +1.5 %) and #386 (theq4k_aligned_prefill_layout.stored_copylabel for gpu-kernels#84).Measured: this pinned pair, built and run on strixhalo gfx1151 under gpu-hold
Both arms are fork
f3d177a40built withGGML_HRX_HIP_ADDON_DIRat the gpu-kernels commit. A =6105368c5(the current pin), B =904e4862c(this pin). Model: Qwen3.8-27B Q4_K_M.hrx-backend-test: A and B are both host 69 passed / 0 failed / 23 known-fail, and device 14/14.test-backend-ops -b HRX0 -o MUL_MAT: A and B are both 1717/1717.llama-bench
-fa 1 -p 512 -n 64 -r 3, 4 interleaved pairs:tg64 favours B in 3 of 4 pairs (+0.7 to +1.1 %). Pair 4's B reads 11.74, a single low sample; the same change measured +1.2 % in all 4 pairs on gpu-kernels#86. pp512 shows no direction (±1.3 %). README, CONTRIBUTING: 1bit-MONSTER is private now #86 is one-token only.
Greedy text (64 tokens, temp 0): byte-identical between A and B.
KLD of B vs A (llama-perplexity
-b 1 -ub 1so the one-token path runs, wikitext 6×128): mean 0.000260, max 0.0064, same top-p 98.68 %.MTP (loom-fun
perf.py --kind loomthrough1bit serve,-c 8192, one gpu-hold ticket, order A B B A)904e4862c--no-mtpMTP is within noise (at most ~+1 %), and acceptance is unchanged.
1bithere is another lane's engine build at a recent main, used read only, withONEBIT_HRX_LLAMA_SERVERpointing at each arm's llama-server. The serve log confirms each arm's binary.Not done here
1bit serve) was not built by me.hw-gatebuilds this exact head and runs its checks.904e4862c.2022afb74adds only GPU: pin AMD's HRX pair and upstream ROCmFPX; our GPU kernel work behind -DONEBIT_GPU_PRIVATE #87: a priority move from 330 to 329 plus its comment, one file, 9 lines. Under today's registration order it changes no plan. Its mutant check showed the 27B FFN pair planned as the same single fused dispatch. The fullhrx-backend-testat 329 (forkf3d177a40) was host 70/0/23 (one extra local case) and device 14/14. I did not re-run the speed arms for that change.Owner: cl-hrxmem (DECODE-BW)
🤖 Generated with Claude Code