Skip to content

config: gpu-kernels pin 2022afb74 -> baabfc3d6 — hip.fa_decode_d256 for multi-row attention (4-row step span -4.1 %, gpu-kernels#88) - #2517

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/gk-pin-88
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-1051/gk-pin-88

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

What

config/gpu-kernels.config: 2022afb74 → baabfc3d6. One gpu-kernels commit, plus #89's comment-only follow-up, which is newer than this pin and not needed here:

baabfc3  hip-hrx: hip.fa_decode_d256 engages from n_kv 128 when the attention has two or more query rows (gpu-kernels#88)

The split-K HIP decode for head-256 gated attention (Qwen3.5 / 3.6 / 3.8) now engages from n_kv 128 when the attention has two or more query rows: llama-server -np N -kvu streams, or speculative verify. One-token decode keeps 1024.

Measured (gpu-kernels#88, Qwen3.8-27B Q4_K_M sha 880a52af, gfx1151)

  • 4-row decode, GPU time per step (dispatch profile, 124 steady steps per build): the Loom gate kernel's 3.30 ms becomes 0.32 ms on hip.fa_decode_d256; step span 99.51 → 95.39 ms (−4.1 %).
  • Numerics: decode-path KLD at 4 rows over one shared cache, and at one sequence × 4 rows, is within ~1σ of the floor. fa-decode-d256-bench passes at T = 2, 4 and 8 for n_kv 128–1000, 5 repeats bit-identical against a double-precision reference.
  • End-to-end c4 tok/s could not resolve the gain on the box tonight (base drifted 34–40 within one ticket). The GPU span is the measurement this rests on.

Verified at the pin

Build: engine cmake --build rc 0 at gpu-kernels baabfc3 with the fork at main's 6b9fcacb7. Fork ninja -k 0: the same four gpu-kernels harness/tool targets fail as at every recent pin ("cannot find ROCm device library").

Suites, one gpu-hold ticket, no amdgpu fault (run on main at fork 6b9fcacb7; rebased onto 3d77ead, whose new commits touch neither the fork pin, the config nor the registry):

suite result vs the previous pin bump (engine#2467)
hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn' 18/18 PASS same
fork ctest (110) 8 fail the same 7 + hrx-backend-test-loom (below)
engine ctest (249, NPU tests excluded) 3 fail laya_gguf_route (known) + two new main tests, both from running inside a ticket (below)

hrx-backend-test-loom: its mtp_draft_shortlist.* cases come from fork a353b7fee (llama.cpp#392, the MTP draft head's q6_k token1 shortlist), which is in main's fork pin, not from this bump. The same build, the six shortlist cases on their own, in a gpu-hold ticket:

result
no dispatch filter (hrx-backend-test row) no failures
GGML_HRX_DISABLE_DISPATCH=hip. (what the -loom row sets) external_embedding.top_k10, get_rows_embedding.top_k10, top_k_wider_than_candidates, sampler_chain FAIL (schedule_graph refuses)
the full -loom row in the suite (whole binary, -j 2) all six fail (the two extra are order/state-dependent; noted on #392)

With every hip.* dispatch disabled, nothing from gpu-kernels is reachable. That covers #88's matcher too, so the -loom failure is the shortlist cases expecting a route that only the add-on provides. Reported on llama.cpp#392.

The two new engine tests, re-run outside any ticket:

  • gpu_hold_child_identity passes (2.5 s). Inside my suite ticket it inherited my GPU_HOLD_TICKET.
  • token_rate_floor_device skips (exit 77) after 911 s; ctest did not capture which prerequisite line it printed. It takes its own ticket, so inside my ticket it can only wait and time out.

Neither is caused by this pin.

Not run, and not claimed

  • Unfiltered hrx-ops-test / test-hrx-ops (an unfiltered run reset the GPU twice on 11-13).
  • NPU-labelled engine tests (no npu-hold ticket).

Refs gpu-kernels#88, gpu-kernels#89, engine C4-STREAMS.

🤖 Generated with Claude Code

… n_kv >= 128 for multi-row attention, 4-row step -4.1 %)

One commit, gpu-kernels#88: the split-K HIP decode for head-256 gated attention (Qwen3.5 / 3.6 / 3.8) engages from n_kv 128
when the attention has two or more query rows (llama-server -np N -kvu, speculative verify); one-token decode keeps 1024.
Qwen3.8-27B Q4_K_M, 4 streams, dispatch profile: the Loom gate kernel's 3.30 ms/step (one workgroup per query head) becomes
0.32 ms/step on hip.fa_decode_d256, step GPU span 99.51 -> 95.39 ms. Decode-path KLD at 4 rows and at one sequence x 4 rows is
within ~1 sigma of the floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2517

A one-line pin bump is the right shape when the reason is stated, and it is: 2022afb74 → baabfc3d6, one gpu-kernels commit "plus #89's comment-only follow-up, which is newer than" the previous pin.

Two things:

  1. put the verification output in the description — the pin's value is a claim about what the engine will build against, and the check is cheap: git log --oneline 2022afb74..baabfc3d6 listing exactly the commits expected (one substantive + GPU: public again — re-pin our HRX and ROCmFPX forks (revert #87) #89's comments), and git cat-file -e baabfc3d6 to prove the object exists in the remote. A pin bump whose description says which commits it carries is auditable; one that says only "newer" has to be trusted;
  2. baabfc3d6 is not a descendant claim you can check by eye — if it is one commit plus a comment-only change, say so explicitly, because a pin that silently moved across a rebase would still "work" while dropping a kernel.

And one thing that is not a problem but should be said: #82 (the docs PR on measurement traps) is not in this pin, and correctly so — it changes no kernel, so nothing about it belongs in a pin bump. Saying that in one line pre-empts the question a reader will otherwise ask when they notice #82 is still open while the pin moved past it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 0430972: READY — audit verified at the exact sha, and the one acceptance step gpu-kernels#88 explicitly deferred to this PR is named below as owed rather than claimed.

The audit at the proposed pin is green

hip_f16_audit --dir <gpu-kernels @baabfc3d6> --expect-sha baabfc3d6     rc=0
AUDITED      baabfc3d6eae1dd52292cedc6703ff2960e5d103
69 kernel file(s), no f16 accumulator; 62 narrowing(s)

Both counts are identical to the outgoing pin (2022afb74: 69 files, 62 narrowings, 0 f16 accumulators), so the one commit — a matcher-gating change in dispatch-hip-fa-decode-d256.cpp — introduces no new narrowing and no f16 accumulation. That is the check a pin bump exists to force, and it is the one I can run without a device.

And #89 really is skippable

+4/−0, and the four lines are // — the reason the two thresholds differ: one row keeps split-K off below ~1024 because the Loom kernel already reads each K/V row once per query head, while several rows mean one workgroup per query head (16–24 on 40 CUs) so splitting pays from a short n_kv. Nothing behavioural, so the pin does not need it. Worth noting for next time: that comment is the why for the behaviour this pin adopts, and it would be a free rider on the next bump — a pinned tree that carries the code without the paragraph is the shape that makes the next reader re-derive it.

The correctness question is answered, by a knob

gpu-kernels#88 measures the changed path against an environment-knob equivalent rather than against nothing:

one sequence, 4 rows per ubatch (-b 512 -ub 4)   0.000437 ± 0.000018, 98.480 %   0.000469 ± 0.000023, 98.578 %
Both are within ~1σ of the floor (the 4-row arm is the env-knob equivalent, GGML_HRX_HIP_FA256_MIN_KV=128).

Two things I care about there. The comparison is a perplexity on the same workload through two routes to the same kernel, not a latency table with a correctness claim in the prose — which is the failure mode of every dispatch-breadth change. And there is a runtime way back: GGML_HRX_HIP_FA256_MIN_KV=128 reproduces the old gating without a rebuild, so if a model with an unusual prefix length misbehaves, the next lane does not have to bisect a pin.

What this PR still owes, in #88's own words

Not run: a full engine ctest at a gpu-kernels pin carrying this. That belongs to the pin bump.

That is this PR. I have not run it: the GPU is held by an hw-gate run with three queued, and the build tree it needs would start at 161 G free — just above the 160 G gate, but not while somebody else's device row is in flight. So the state of this pin is: audit green, their perplexity arm green, ctest owed, and I would rather write that down than let "READY" imply a run that did not happen. When the queue clears, ctest -R 'hrx|addon' on a tree at this pin is the step that closes it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 0430972: READY

The pin is one commit, it is merged, both engine pin gates pass, and the content it brings is a two-floor matcher whose floors are each argued from a measurement.

The enumeration

$ git rev-list --count 2022afb74..baabfc3d6        1        ← "One gpu-kernels commit"
baabfc3d6  hip-hrx: hip.fa_decode_d256 engages from n_kv 128 when the attention has two or more query rows
$ git merge-base --is-ancestor baabfc3d6 origin/main   yes (in main)
$ git rev-list --count baabfc3d6..origin/main          1        ← main is one ahead: #89, comment-only, as the body says
$ config/gpu-kernels.config  sha=baabfc3d6eae…          ← the file this PR changes
$ tools/ci/config_pin_check.py --check     rc 0
$ tools/registry_build.py --check-pins     rc 0

One, not three; the pinned sha is in main rather than a branch of its own; the one commit main is ahead of it is the follow-up the body explicitly excludes as comment-only. That is the enumeration check #2431 and #2448 exist for, and the exclusions are named rather than left for a reader to discover.

The content, which is what a pin bump is actually buying

constexpr int64_t kDefaultMinKv = 1024;            // one-token decode
constexpr int64_t kDefaultMinKvMultiRow = 128;     // two or more query rows
int64_t min_kv_for(int64_t rows) {
    static const int64_t multi = env_int("GGML_HRX_HIP_FA256_MIN_KV_ROWS", kDefaultMinKvMultiRow);
    return rows > 1 ? multi : min_kv();
}

Two floors, each with its own reason, which is the right shape: a single floor would have to be either too high for multi-stream decode or too low for one-token decode. The multi-row floor is justified by the kernel it replaces — "the Loom gate kernel this matcher replaces runs one workgroup per query head regardless of the row count", i.e. the occupancy is bad exactly when there are several rows — and by a measurement (132.6 µs per call, 16–24 workgroups, 0 % overlapped, Qwen3.8-27B × 4 streams). The one-token floor stays at 1024 because nothing about one-row decode changed.

Both floors are overridable by environment, and the new one follows the existing naming (GGML_HRX_HIP_FA256_MIN_KV → …_MIN_KV_ROWS), so a lane that needs the old behaviour has one knob and not a code edit.

What this does to the engine: at n_kv >= 128 with 2+ rows the split-K kernel now engages where the Loom gate kernel did. That is a dispatch change, so the class of regression it can cause is a numeric one on multi-stream decode — and the thing that would catch it is test-backend-ops on HRX0 plus the 27B multi-stream runs, neither of which I ran (the first needs a device ticket; the second needs the model). I reviewed the matcher and the pin, not the kernel's arithmetic, and the PR's own body is where the measurement for that lives.

One note: baabfc3d6 was pinned without a verdict on gpu-kernels#88 while it was open — that is fine given it merged, but it means this pin is the first place its dispatch change is reviewed end to end, which is an argument for reading the pinned diff rather than the PR title. Which is what the diff above is.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Commit SHA Mismatch

The commit SHA in config/gpu-kernels.config has been updated from 2022afb74 to baabfc3d6, but the PR description claims this change is based on commit baabfc3 with a follow-up comment-only change. The actual commit baabfc3 is not reflected in the diff, which only shows the SHA update. This discrepancy could lead to confusion about which exact changes are included in this pin.

sha=baabfc3d6eae1dd52292cedc6703ff2960e5d103

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 10:37
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

On "ctest owed" (10:32:15Z): it was run at this pin, before the PR opened, and the results are in the body's "Verified at the pin" table. Build tree: engine at gpu-kernels baabfc3 + fork 6b9fcacb7; gpu-hold ticket cl1051-gkpin88-suite, 09:41-09:59Z, no amdgpu fault.

  • fork ctest (110 tests, which includes every hrx-* / add-on row) except the unfiltered hrx-ops-test / test-hrx-ops: 8 fail = the 7 known at the previous pin + hrx-backend-test-loom. The -loom failure only reproduces under GGML_HRX_DISABLE_DISPATCH=hip. (shortlist cases from llama.cpp#392), and nothing from gpu-kernels is reachable there.
  • hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn': 18/18.
  • engine ctest (249, NPU tests excluded): 3 fail. laya_gguf_route is known; the other two only fail when ctest itself runs inside a ticket. Re-run outside one: gpu_hold_child_identity passes, token_rate_floor_device skips.

On #82 (10:31:52Z): right, the measurement-traps docs PR changes no kernel, so it has no reason to be in a pin, and this pin carries only #88.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY, and I verified the pin independently rather than taking the body's word for it.

In ~/.cache/hw-gate/gpu-kernels, after a fetch:

$ git rev-list --count 2022afb74..baabfc3d6
1
$ git diff --stat 2022afb74 baabfc3d6
 addons/hip-hrx/dispatch-hip-fa-decode-d256.cpp | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

One commit, one file — so the bump is exactly what it says it is and nothing rides along. That is the property a submodule pin has to have, and it is the one nobody checks: a pin is an opaque promise that the range contains only what the body lists, and a bump that quietly carried a second kernel change would be indistinguishable from this one in review.

The +14/−2 is the right shape for the change, and the comments carry the argument the code cannot:

// ... it measured 132.6 us per call (16 ... 0 % overlapped), the split-K kernel engages from 128 (engine C4-STREAMS).
constexpr int64_t kDefaultMinKvMultiRow = 128;
int64_t min_kv_for(int64_t rows) { return rows > 1 ? multi : min_kv(); }

min_kv_for(rows) is a smaller change than I expected and a better one. The alternative — lowering kDefaultMinKv for everybody — would have moved single-token decode onto a split-K kernel tuned for many rows; making the threshold a function of T keeps the one-token path on min_kv() and gives the multi-row path its own default with its own env override (GGML_HRX_HIP_FA256_MIN_KV_ROWS). Two knobs, two regimes, and the guard c.n_kv < 1 || c.n_kv < min_kv_for(c.T) || c.n_kv > kMaxKv still has the kMaxKv upper bound my #88 review asked about.

One discrepancy to resolve in the body, because it is the kind that matters in a pin PR: the body says "One gpu-kernels commit, plus #89's comment-only …" — but the range contains only one commit and its diff touches only dispatch-hip-fa-decode-d256.cpp; the comment lines above are inside baabfc3d6 itself, not a second README commit. So either #89's prose is part of this same commit (in which case say "including the explanatory comments from #89", not "plus"), or #89 has not landed and the pin does not contain it. In a bump PR that sentence is the whole contract; a reader who trusts "plus" will believe addons/hip-hrx/README.md on the pinned tree carries prose that is not there.

Not verified by me: the measurement (it is #88's, which I reviewed at 98df0a6 and still consider READY) and the submodule pin's own resolution in CI. The range, the one-file diff and the min_kv_for(rows) shape I verified directly against the pinned SHAs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant