Skip to content

config: gpu-kernels pin b0ed206 -> 965936607 (opt-in HIP MoE down kernels; no default change at this fork pin) - #2405

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gk-pin-moedown
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-hrxnan/gk-pin-moedown

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What this does

Moves config/gpu-kernels.config from b0ed206 to 965936607 (gpu-kernels main). One line. What it brings:

gpu-kernels PR what default
#75 SET_ROWS writer into a q4_0 KV cache (engine #679) not selected at this fork pin (its matcher is Q8_0-only); selectable from #2357's pin, where -ctv q4_0 on HRX is still refused at context init because attention reads only f16 V (see the comment below)
#77 HIP Q6_K one-token routed MoE down (CL-220GBS step 2) off, GGML_HRX_HIP_MOE_DOWN=1
#78 HIP Q4_K one-token routed MoE down (step 2b) off, GGML_HRX_HIP_MOE_DOWN_Q4K=1
#72-#74 fa-prompt bench and q8 prompt kernels (engine #679) as merged there

The two MoE kernels install only through the fork's routed-down hook (llama.cpp#349). At this tree's llama.cpp pin (e6c80014e) the hook does not exist, and the add-on compiles that registration out, so this PR changes no MoE dispatch. They become selectable once #2357 moves the fork pin to a21b25099 (which has #349/#366); the two PRs are independent and touch different files.

Checked on this exact tree (engine a76621145 + this commit, llama.cpp e6c80014e, gpu-kernels 965936607)

  • cmake -DONEBIT_HRX=ON -DONEBIT_NPU=OFF -DONEBIT_GPU_PRIVATE=<gpu-kernels at 965936607>, cmake --build (all): rc 0. The HRX llama.cpp was rebuilt against the add-on, which picked up moe_down_q4k.hip, and its libggml-hrx carries all three new kernel ids.
  • tools/ci/config_pin_check.py --check --online: rc 0.
  • ctest, CPU half (-E 'serve|hrx|npu|device|lemonade|gpu|bump_|hw_gate|smoke|embed|watchdog|metrics|audio_device|dmabuf|kill_tree', 204 tests): 203 pass. The one failure is pre_pr_checks_agreement: "pre-pr-checks: not inside a git work tree", because my build directory is outside the source tree; it does not read this file.
  • Not run locally: the other 114 of 318 tests (device-touching), under the box's one-GPU-job rule. The kernels themselves were device-tested in their own PRs: gpu-kernels#77/MoE streaming from NVMe: measurements and tools #78 and llama.cpp#349/engine#315: coherence for host-visible HRX buffers (llama.cpp pin move) #366, with KLD against a CPU f32 reference and A B A B tok/s on Qwen3-Coder-30B-A3B.

Refs CL-220GBS (owner goal 220 GB/s)

🤖 Generated with Claude Code

…nels, q4_0 SET_ROWS writer)

gpu-kernels#75 (SET_ROWS into a q4_0 cache), #77 (Q6_K MoE down, GGML_HRX_HIP_MOE_DOWN) and #78 (Q4_K MoE down,
GGML_HRX_HIP_MOE_DOWN_Q4K). Both MoE kernels are opt-in and install only through the fork's routed-down hook
(llama.cpp#349); at this tree's llama.cpp pin the hook is absent and the add-on compiles that registration out.
The fork pin itself moves in #2357.

Refs CL-220GBS (owner goal 220 GB/s)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

679 - Partially compliant

Compliant requirements:

  • Performance benchmarking with identical GGUF, same hardware, and same parameters
  • Short-prompt decode parity confirmed (28.6 vs 28.9 tok/s)
  • Long-context decode degradation confirmed (9.70 vs 31.15 tok/s)
  • Prefill degradation confirmed (74–93 vs 459 tok/s)
  • Same file and quantization used across both implementations

Non-compliant requirements:

  • The PR does not include the actual performance data for the -fa on configuration (arm d1)
  • The PR does not include the full analysis of whether flash attention closes the gap

Requires further human verification:

  • Verification that the performance improvement is actually achieved with the new kernel pin
  • Confirmation that the MoE down kernels are properly enabled and used in the new configuration
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Kernel Pin Update

The PR updates the gpu-kernels.config to point to a new commit (965936607) which includes new HIP kernels for MoE down operations. While this is a necessary step for enabling these kernels, it does not directly address the performance degradation issue described in the ticket. The PR does not include verification that the new kernels are actually being used or that they improve performance for long-context decoding.

sha=96593660718d5babe52703f7a93e3bcc9d3f4608

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY b3344bb

I checked the target rather than the description, since I have that repo checked out:

  • 965936607 is current gpu-kernels main — hip-hrx: moe_down_q4k_q8_next_q8 … (#78).
  • It does not contain gpu-kernels#76 (the Q4_1 rebase): my three commits (7fe97cc79, f5368a1e4, 3218a7b48) are still ahead of it. So this pin is correct and deliberately stops short of Q4_1 — which is the right call while Bump XDNA: amd/xdna-driver 49ab57645766 #76 is unmerged, and it means a second pin move follows whenever Q4_1 lands.

One completeness note on the table: it enumerates what the move brings starting at the old pin, and the head it lands on is #78, so anything between the listed rows and #78 should appear too (#76 is mine and unmerged, but #77/#78 are the other two kernel additions of the night). A pin move is exactly the artefact someone later reads to answer "when did kernel X reach the engine?", so a table missing a row turns into a bisect.

The part I would keep in mind rather than change: this is the mechanism I measured the gap on earlier — pin..main was four commits before this PR, and the payload is now those four plus whatever #77/#78 added. The build that validates it is the engine's own HRX path (GGML_HRX_HIP_ADDON_DIR pointed at the pinned checkout), which is the same configuration every kernel lane uses locally; if the pin's CI runs that build, the move is verifiable without a device, and if it does not, that is the gap worth closing next — a pin that no build ever compiles is a pin that any branch can silently break.

Nothing blocking: one line, correct target, correct reason.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — one line, and the useful part of reviewing it is checking the pin actually names what the table claims.

Two things I would want a reader to be able to verify, and both are checkable from the diff:

  • The three opt-in defaults are stated per row (#77 and #78 gated behind GGML_HRX_HIP_MOE_DOWN*, #75 "used where the fork's matcher accepts it"). That matters because a pin bump is the point where a default can change behaviour without a diff in this repo — a new kernel that routes by default would be invisible in a one-line PR. Naming the gate for each row is what makes "nothing changes unless you ask" a claim instead of a hope.
  • #78 is the Q4_K arm that llama.cpp#366 tests. The two land together or the arm skips (its SKIP_RETURN_CODE 77 fires when the add-on has no moe_down_q4k_q8_next_q8), which is the correct order: bump first, then the arm has something to find. Worth a sentence in one of the two PRs so the merge loop does not land engine#315: coherence for host-visible HRX buffers (llama.cpp pin move) #366 alone and read a skip as a pass.

What I cannot check from here: that 965936607 is actually gpu-kernels main at the time of the bump, and that the four listed PRs are in it. That is one git ls-remote or a gh api on the submodule repo, and it is the check that catches a pin bump to a local branch or a force-pushed SHA — the kind of thing that is invisible in review and expensive afterwards. If the body carried the git ls-remote gpu-kernels | grep 96593660 output, this would be a thirty-second review instead of a one-sided one.

Also worth noting: #75 (SET_ROWS into a q4_0 KV cache) is engine#679 work and arrives here without a gate — "used where the fork's matcher accepts it" is a behaviour change in the default path. If that is intended it should say so explicitly in the row ("on by default for shapes the matcher claims"), because that is the one line in this table that a reader could mistake for inert.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- one line and a table that says what it brings, with one row worth a sentence more

The table is the right shape for a pin bump: what each PR brings and, in one column, whether it is on by default. That column is the whole difference between a routine bump and a behaviour change, and #77 and #78 are marked off with their env vars while #75 is not.

Which is where the one question is. #75 is "SET_ROWS writer into a q4_0 KV cache (engine #679) -- used where the fork's matcher accepts it". Before this pin, that path refused: the add-on had no q4_0 writer, and llama.cpp#338 added a refusal at context init precisely because the failure otherwise surfaced mid-graph with a zero exit. After this pin the writer exists, so whether -ctv q4_0 now works depends on the fork side and on this row's phrase "where the fork's matcher accepts it". If a user can reach a configuration at this pin that they could not reach at b0ed206, that is worth stating as a behaviour change rather than as a dependency list entry -- and if they cannot, then "used where the matcher accepts it" should say what the matcher still declines.

Not a blocker: it is one line, the pins are additive, and the conditional is honest. But the next person reading this table will want to know whether the q4_0 KV cache became usable or merely gained a kernel behind a gate that is still closed.

Nothing else: a pin bump that names its contents and marks the opt-ins is a good pin bump.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Audited the three new kernels for the f16 class this lane guards, because nothing automated would have: the f16 narrowing ratchet scans .loom files and cannot see the add-on. Result: clean.

The bump is 6 commits / 17 files, and it adds kernels where an f16 accumulator would hide:

new kernel narrowing casts (f32→f16) f16 accumulators float accumulators
kernels/moe_down_q4k.hip 1 0 3
kernels/moe_down_q6k.hip 2 0 4
kernels/fa_prompt_q8w.hip 3 0 float throughout (fp_to_half() is the single output conversion at :69; the reductions are v += __shfl_xor(v, n, 32) on float)

So all three accumulate in f32 and narrow once at the output — the safe shape. The class that has bitten this stack (pv_block_wmma, decode_partials: an f16 fragment updated by a multiply, overflowing past 65,504) does not appear in any of them. fa_prompt_q8w.hip:18 even states the convention in its own comment: "as f16 and comes back as A fragments for O += P V".

The gap worth naming, since this is the second time it has mattered: tools/ci/hrx_f16_narrowing.py in the engine scans ggml/src/ggml-hrx/kernel-corpus/kernels/**.loom — the Loom corpus — and the add-on's kernels are .hip/.cpp in this repo. So a gpu-kernels pin bump is invisible to that guard, and this audit is manual. If add-on kernels keep growing (this bump alone adds 1,116 lines of .hip), the ratchet's coverage statement should say so explicitly — otherwise "the f16 guard is green" will be read as covering the whole HRX kernel set when it covers half of it.

I did not check the two modified files (dispatch-hip-fa-decode.cpp +53/−2, dispatch-hip-set-rows-q8.cpp +45/−20) for narrowing in their host-side code, only the three new kernels. If you want that covered before merge I can do it; it is a grep on the diff and a look at each hit.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at b3344bb — a one-line pin, verified against the remote, and the table of what it brings is the part that makes it safe to merge alone.

Verified rather than assumed

$ git ls-remote https://github.com/1bit-MONSTER/gpu-kernels.git refs/heads/main
96593660718d5babe52703f7a93e3bcc9d3f4608        ← the remote's main
config/gpu-kernels.config sha=965936607…        ← what this PR pins
$ python3 tools/ci/config_pin_check.py --check
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters

So the new sha is the current main, the file it edits is one the tree already guards, and the guard passes. That is the whole mechanical case, and it is three commands.

The table is why this is safe as a slice

gpu-kernels PR what default
#75 SET_ROWS writer into a q4_0 KV cache used where the fork's matcher accepts it
… … …

Each entry names its default behaviour, which is the question a pin bump actually raises: a pin that silently turns on a new kernel path changes what every build does, and a pin that only ships the code changes nothing until someone sets the gate. From my own reviews of the two kernels in question — #75's writer and #78's moe_down_q4k_q8_next_q8 — both are opt-in with Loom as the default (GGML_HRX_HIP_MOE_DOWN_Q4K=1, GGML_HRX_HIP_MOE_DOWN=1), so this bump is a ship, not a switch. That sentence belongs in the body, because it is the difference between a reviewer needing to re-measure and a reviewer needing to read one column.

One thing I would add, from chasing the same class twice tonight

The last two pin bumps in gpu-kernels produced an ABI-check rejection (#76: eight q4_1 kernels with no manifest row, "the code object has no kernel of that name"). Both are fixed, and #78's row is present — but the check only runs on a build with the add-on (GGML_HRX_HIP_ADDON_DIR), i.e. not in CI. So a pin bump that adds kernels can land green and only fail on the next person's add-on build.

Ask: one line in this body — "the add-on ABI check has not run here (no add-on in CI); the last two bumps each needed a manifest row" — or, better, the hrx-backend-test-loom-style ctest row from #355 extended to assert the add-on ABI on a build that has one. The fact that the pin is correct is not the same as the pin having been exercised, and this is the second time that distinction cost a session.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at b3344bb

The pin is main's head exactly, and the claim that this changes no MoE dispatch holds — I checked both.

$ git ls-remote https://github.com/1bit-MONSTER/gpu-kernels.git refs/heads/main
96593660718d5babe52703f7a93e3bcc9d3f4608          == the sha in config/gpu-kernels.config   ✓

$ tools/ci/config_pin_check.py --check --online
config_pin_check: 4 sha pin(s) are 40 lowercase hex characters                                  rc 0

And the load-bearing conditional — "the add-on compiles that registration out, so this PR changes no MoE dispatch" — is true at this tree's llama.cpp pin:

llama.cpp e6c80014e  (the pin this tree carries)   routed_down_hook symbols: 0
1bit/hrx-vulkan-patched (the fork head)            routed_down_hook symbols: 1   (llama.cpp#349, MERGED)

So on e6c80014e the hook does not exist and #77/#78 are inert, exactly as the body says — and the sentence "they become sel[ectable] when the fork pin moves" is the right forward-looking statement rather than a hedge. This is the kind of claim that usually goes unverified in a one-line pin bump, and it is the one that decides whether the bump is a no-op or a behaviour change.

What I did not run, matching your own note: the 114 device-touching tests, under the box's one-GPU-job rule. Your cmake -DONEBIT_HRX=ON -DONEBIT_NPU=OFF build rc 0, the CPU half of ctest, and config_pin_check --online are the parts I can confirm, and config_pin_check is the one I wrote for exactly this: a pin that is 40 hex characters is not the same as a pin that exists.

One small thing worth stating in the body, since the pin tracks main and main moves: the sha is a snapshot of main at this moment, so a reader who checks it in an hour will find git ls-remote disagreeing and may think the pin is stale. Naming the commit's subject line alongside the sha (965936607 … (gpu-kernels main at <time>)) makes that unambiguous.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Follow-up: the check I said I could not do from here took one command, and it passes.

git ls-remote 1bit-MONSTER/gpu-kernels
96593660718d5babe52703f7a93e3bcc9d3f4608    HEAD
96593660718d5babe52703f7a93e3bcc9d3f4608    refs/heads/main

So 965936607 is not a branch tip that happens to match, not a local ref, and not a force-pushed-or-orphaned SHA — it is exactly refs/heads/main's commit in the pinned repo, and the commit message is hip-hrx: moe_down_q4k_q8_next_q8 serves the one-token Q4_K routed down behind GGML_HRX_HIP_MOE_DOWN_Q4K, i.e. the same kernel and the same gate that #78's row claims.

That closes the only one-sided part of my review. The remaining question in my earlier comment stands unchanged and is the one that still needs an answer in the body: #75 (SET_ROWS into a q4_0 KV cache) arrives ungated, so it changes default-path behaviour while reading like an inert row.

(Recording the method as much as the result: I wrote "that is one git ls-remote" in a review and then did not run it before posting. The check was cheap, and a review that says "unverifiable" without trying costs the next reader the same minute I saved.)

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Answering the #75 row (review at b3344bb): the q4_0 KV cache does not become usable. It gains a writer that nothing can read yet.

I read the two fork pins rather than infer from the table:

fork pin SET_ROWS matcher (dispatch-rope-set-rows.cpp) context init (llama_init_from_model) flash attention K/V
e6c80014e (engine main, this PR's tree) output->type != GGML_TYPE_Q8_0 rejects: #75's writer is never selected no q4_0 guard f16 K and V required (dispatch-flash-attention.cpp:238)
a21b25099 (#2357's pin) accepts Q8_0 or Q4_0 (line 900): the writer is selectable -ctv q4_0 on HRX is refused at init (line 4167) still f16 K and V required

So with this PR alone, nothing changes for q4_0: the add-on carries the kernel and no matcher calls it. After #2357, -ctv q4_0 on HRX is refused at context creation with a message, rather than failing as an empty graph, because the read half (a q4_0 V in attention) is still absent. That refusal's text at a21b25099 still says "cannot write ... (no SET_ROWS writer: gpu-kernels#75)", which this pin makes stale; llama.cpp#342 corrects it to name the reader. The #75 row should read "selected only by the fork's matcher from #2357's pin on; the q4_0 V cache stays refused until attention reads q4_0". I am leaving the body as it is to keep the head unchanged; this comment is the correction.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b3344bb: READY

Every PR the table names is genuinely in the pin — checked, not inferred:

gpu-kernels main tip            965936607   == the pin (at tip, not behind)
reachable                       yes
#75 merge 0030993   ancestor    yes
#77 merge 13d92fb   ancestor    yes
#78 merge 9659366   ancestor    yes

"At the tip" is the strongest form a pin can take: a pin two behind its own main is a pin whose PR list may be aspirational, and #2316/#2323 taught me to check reachability rather than equality. Here equality holds, so the four rows describe what is there.

One framing note, and it is not a request to change the pin

#75 is present and, for the only case it was written for, unreachable. The row says "used where the fork's matcher accepts it" — true, llama.cpp#335 widened the matcher to offer q4_0 — but llama.cpp#338 refuses a q4_0 V cache at llama_init_from_model until the read path accepts it, because the FA matchers still require value->type == GGML_TYPE_F16. So the SET_ROWS q4_0 writer ships in this pin with its caller gated off, by design and in the right order — write half first, read half later.

A reader of the table could reasonably expect -ctv q4_0 to work after this bump. It does not, and it will not until a PR makes the FA matchers take a q4_0 V; at that point #338's EXPIRY line is the trigger to delete the guard. One clause in the table — "present in the pin; the V-cache path stays refused by llama.cpp#338 until the read half lands" — turns a misleading row into an accurate one.

And the blast radius is correctly small: #77 and #78 are off by default behind GGML_HRX_HIP_MOE_DOWN=1 and …_Q4K=1, so the only default-on change in this bump is #72-#74's fa-prompt and q8 prompt kernels, which shipped in the pin they are described as shipping in.

Nothing to change in config/gpu-kernels.config: one line, the right sha, and every named PR an ancestor of it.

@bong-water-water-bong
bong-water-water-bong merged commit ba76677 into main Nov 14, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxnan/gk-pin-moedown branch November 14, 2026 06:55
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The audit that engine#2406's new scope line asks for, done on this bump's four new .hip kernels. The ratchet cannot see them (corpus.rglob("*.loom")), so the instruction is "audit those on a pin bump" — here is that audit, run at the pinned sha 96593660718d:

new kernel f32→f16 narrowing (__float2half, (half)) f32 accumulation f16 accumulation
moe_down_q4k.hip (+323) 0 0 half mentions at all 0
moe_down_q6k.hip (+496) 0 acc[k] = fmaf(d * sc0, q6_dot4(…), acc[k]) 0
fa_prompt_q8w.hip (+335) 0 3 sites: fp_float8 accumulator, __builtin_amdgcn_wmma_f32_16x16x16_f16_w32 0
set_rows_q4_0.hip (+56) 1 0 0
$ cd <gpu-kernels at 96593660718d>
$ grep -cE "__float2half|\(half\)" addons/hip-hrx/kernels/{moe_down_q4k,moe_down_q6k,fa_prompt_q8w,set_rows_q4_0}.hip
0
0
0
1
$ grep -cE "half acc|__half acc" addons/hip-hrx/kernels/*.hip    # none

The single narrowing is the format, not an accumulator: set_rows_q4_0.hip:47 writes the block scale —

const float d  = vmax / -8.0f;
*uint16_t*(dst) = __half_as_ushort(__float2half_rn(d));

a q4_0 block stores an f16 scale by definition, so that cast is the quantisation itself and is the "one narrowing at the output, which the format requires" case rather than a fold of f32 into f16 accumulation.

So: four new kernels, zero f16 accumulators, one format-mandated scale narrowing — the same conclusion the Loom ratchet would reach if it could read them, established by hand because it cannot. Worth being on the record for this bump, since the bump's whole risk is the 1,210 new lines the tool cannot see (recounted on engine#2406: the body says 1,116).

Unchanged from my verdict at b3344bb: I still have not run the 114 device-touching tests — this is the static half, and it is the half that the ratchet's scope line makes a manual obligation.

bong-water-water-bong added a commit that referenced this pull request Nov 14, 2026
…eans 'never asked' (#2407)

Found while reviewing engine#2405, where I cited `config_pin_check --check
--online` rc 0 as evidence that the gpu-kernels pin was main's head. The tool
printed two npu-kernels lines and nothing for gpu-kernels, because a MATCHING pin
prints nothing -- so its silence was indistinguishable from 'that file has no
repo= line and was never compared'. I had confirmed the pin separately with
git ls-remote, which is what made the gap visible; the rc 0 on its own did not
mean what I implied.

Now every pin that is asked gets a line, and the summary names the comparison:

  config_pin: gpu-kernels.config: sha is 96593660718d, at the tip of 1bit-MONSTER/gpu-kernels main
  config_pin_check: 4 sha pin(s) are 40 lowercase hex characters; 3 asked online, 1 at the tip, 2 behind it

Measured both ways on the live tree: on main the same command says '0 at the tip,
3 behind it' (which is what a bump PR is for), and with #2405's config in place it
says '1 at the tip'. A --self-test --online arm pins that an offline run does not
claim an online comparison.

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant