Repository navigation
config: gpu-kernels pin 2022afb74 -> baabfc3d6 — hip.fa_decode_d256 for multi-row attention (4-row step span -4.1 %, gpu-kernels#88) - #2517
Conversation
… n_kv >= 128 for multi-row attention, 4-row step -4.1 %) One commit, gpu-kernels#88: the split-K HIP decode for head-256 gated attention (Qwen3.5 / 3.6 / 3.8) engages from n_kv 128 when the attention has two or more query rows (llama-server -np N -kvu, speculative verify); one-token decode keeps 1024. Qwen3.8-27B Q4_K_M, 4 streams, dispatch profile: the Loom gate kernel's 3.30 ms/step (one workgroup per query head) becomes 0.32 ms/step on hip.fa_decode_d256, step GPU span 99.51 -> 95.39 ms. Decode-path KLD at 4 rows and at one sequence x 4 rows is within ~1 sigma of the floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY A one-line pin bump is the right shape when the reason is stated, and it is: Two things:
And one thing that is not a problem but should be said: #82 (the docs PR on measurement traps) is not in this pin, and correctly so — it changes no kernel, so nothing about it belongs in a pin bump. Saying that in one line pre-empts the question a reader will otherwise ask when they notice #82 is still open while the pin moved past it. |
|
VERDICT at 0430972: READY — audit verified at the exact sha, and the one acceptance step gpu-kernels#88 explicitly deferred to this PR is named below as owed rather than claimed. The audit at the proposed pin is greenBoth counts are identical to the outgoing pin ( And #89 really is skippable+4/−0, and the four lines are The correctness question is answered, by a knobgpu-kernels#88 measures the changed path against an environment-knob equivalent rather than against nothing: Two things I care about there. The comparison is a perplexity on the same workload through two routes to the same kernel, not a latency table with a correctness claim in the prose — which is the failure mode of every dispatch-breadth change. And there is a runtime way back: What this PR still owes, in #88's own words
That is this PR. I have not run it: the GPU is held by an |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 0430972: READY
The pin is one commit, it is merged, both engine pin gates pass, and the content it brings is a two-floor matcher whose floors are each argued from a measurement.
The enumeration
$ git rev-list --count 2022afb74..baabfc3d6 1 ← "One gpu-kernels commit"
baabfc3d6 hip-hrx: hip.fa_decode_d256 engages from n_kv 128 when the attention has two or more query rows
$ git merge-base --is-ancestor baabfc3d6 origin/main yes (in main)
$ git rev-list --count baabfc3d6..origin/main 1 ← main is one ahead: #89, comment-only, as the body says
$ config/gpu-kernels.config sha=baabfc3d6eae… ← the file this PR changes
$ tools/ci/config_pin_check.py --check rc 0
$ tools/registry_build.py --check-pins rc 0
One, not three; the pinned sha is in main rather than a branch of its own; the one commit main is ahead of it is the follow-up the body explicitly excludes as comment-only. That is the enumeration check #2431 and #2448 exist for, and the exclusions are named rather than left for a reader to discover.
The content, which is what a pin bump is actually buying
constexpr int64_t kDefaultMinKv = 1024; // one-token decode
constexpr int64_t kDefaultMinKvMultiRow = 128; // two or more query rows
int64_t min_kv_for(int64_t rows) {
static const int64_t multi = env_int("GGML_HRX_HIP_FA256_MIN_KV_ROWS", kDefaultMinKvMultiRow);
return rows > 1 ? multi : min_kv();
}Two floors, each with its own reason, which is the right shape: a single floor would have to be either too high for multi-stream decode or too low for one-token decode. The multi-row floor is justified by the kernel it replaces — "the Loom gate kernel this matcher replaces runs one workgroup per query head regardless of the row count", i.e. the occupancy is bad exactly when there are several rows — and by a measurement (132.6 µs per call, 16–24 workgroups, 0 % overlapped, Qwen3.8-27B × 4 streams). The one-token floor stays at 1024 because nothing about one-row decode changed.
Both floors are overridable by environment, and the new one follows the existing naming (GGML_HRX_HIP_FA256_MIN_KV → …_MIN_KV_ROWS), so a lane that needs the old behaviour has one knob and not a code edit.
What this does to the engine: at n_kv >= 128 with 2+ rows the split-K kernel now engages where the Loom gate kernel did. That is a dispatch change, so the class of regression it can cause is a numeric one on multi-stream decode — and the thing that would catch it is test-backend-ops on HRX0 plus the 27B multi-stream runs, neither of which I ran (the first needs a device ticket; the second needs the model). I reviewed the matcher and the pin, not the kernel's arithmetic, and the PR's own body is where the measurement for that lives.
One note: baabfc3d6 was pinned without a verdict on gpu-kernels#88 while it was open — that is fine given it merged, but it means this pin is the first place its dispatch change is reviewed end to end, which is an argument for reading the pinned diff rather than the PR title. Which is what the diff above is.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
On "ctest owed" (10:32:15Z): it was run at this pin, before the PR opened, and the results are in the body's "Verified at the pin" table. Build tree: engine at gpu-kernels
On #82 (10:31:52Z): right, the measurement-traps docs PR changes no kernel, so it has no reason to be in a pin, and this pin carries only #88. |
|
READY, and I verified the pin independently rather than taking the body's word for it. In One commit, one file — so the bump is exactly what it says it is and nothing rides along. That is the property a submodule pin has to have, and it is the one nobody checks: a pin is an opaque promise that the range contains only what the body lists, and a bump that quietly carried a second kernel change would be indistinguishable from this one in review. The +14/−2 is the right shape for the change, and the comments carry the argument the code cannot:
One discrepancy to resolve in the body, because it is the kind that matters in a pin PR: the body says "One gpu-kernels commit, plus #89's comment-only …" — but the range contains only one commit and its diff touches only Not verified by me: the measurement (it is #88's, which I reviewed at |
What
config/gpu-kernels.config:2022afb74→baabfc3d6. One gpu-kernels commit, plus #89's comment-only follow-up, which is newer than this pin and not needed here:The split-K HIP decode for head-256 gated attention (Qwen3.5 / 3.6 / 3.8) now engages from n_kv 128 when the attention has two or more query rows:
llama-server -np N -kvustreams, or speculative verify. One-token decode keeps 1024.Measured (gpu-kernels#88, Qwen3.8-27B Q4_K_M sha 880a52af, gfx1151)
hip.fa_decode_d256; step span 99.51 → 95.39 ms (−4.1 %).fa-decode-d256-benchpasses at T = 2, 4 and 8 for n_kv 128–1000, 5 repeats bit-identical against a double-precision reference.Verified at the pin
Build: engine
cmake --buildrc 0 at gpu-kernelsbaabfc3with the fork at main's6b9fcacb7. Forkninja -k 0: the same four gpu-kernels harness/tool targets fail as at every recent pin ("cannot find ROCm device library").Suites, one gpu-hold ticket, no amdgpu fault (run on main at fork 6b9fcacb7; rebased onto 3d77ead, whose new commits touch neither the fork pin, the config nor the registry):
hrx-ops-test --hrx-test-filter 'ssm_conv|^gdn'hrx-backend-test-loom(below)laya_gguf_route(known) + two new main tests, both from running inside a ticket (below)hrx-backend-test-loom: itsmtp_draft_shortlist.*cases come from forka353b7fee(llama.cpp#392, the MTP draft head's q6_k token1 shortlist), which is in main's fork pin, not from this bump. The same build, the six shortlist cases on their own, in a gpu-hold ticket:hrx-backend-testrow)GGML_HRX_DISABLE_DISPATCH=hip.(what the-loomrow sets)external_embedding.top_k10,get_rows_embedding.top_k10,top_k_wider_than_candidates,sampler_chainFAIL (schedule_graphrefuses)-loomrow in the suite (whole binary,-j 2)With every
hip.*dispatch disabled, nothing from gpu-kernels is reachable. That covers #88's matcher too, so the-loomfailure is the shortlist cases expecting a route that only the add-on provides. Reported on llama.cpp#392.The two new engine tests, re-run outside any ticket:
gpu_hold_child_identitypasses (2.5 s). Inside my suite ticket it inherited myGPU_HOLD_TICKET.token_rate_floor_deviceskips (exit 77) after 911 s; ctest did not capture which prerequisite line it printed. It takes its own ticket, so inside my ticket it can only wait and time out.Neither is caused by this pin.
Not run, and not claimed
hrx-ops-test/test-hrx-ops(an unfiltered run reset the GPU twice on 11-13).Refs gpu-kernels#88, gpu-kernels#89, engine C4-STREAMS.
🤖 Generated with Claude Code