Skip to content

hrx-backend-test: label q4k_aligned_prefill_layout.stored_copy on the add-on path (gpu-kernels#84's 306 entry) - #386

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
q4k-stored-label
Nov 14, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
q4k-stored-label

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

One line in hip_addon_known_fail: q4k_aligned_prefill_layout.stored_copy is labelled kPlan on the add-on path, with a comment naming the mechanism and the two measurements that separate it.

Why — the case asserts a kernel gpu-kernels#84 retired, in one arm

tests/hrx-backend-test.cpp:3711-3714 says, for a 512-token Q4_K MUL_MAT on a file-mapped weight with a stored copy present:

REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);

gpu-kernels#84 added a matcher registered at priority 306 that claims precisely "a 256-aligned batch of a Q4_K weight Loom reads stored" and routes it to hip.dense_prefill.mm_q4k_stored (the +25–28 % prefill change). That is this arm's shape, so the assertion no longer describes what the engine does — with the add-on, while the loom-only build is untouched.

Measured, not inferred:

add-on gk-b0ed206 (before #84):        [PASS] q4k_aligned_prefill_layout.stored_copy
add-on 6105368 (the pin, #84 in):      [FAIL] …:3712: name == "…q5_k_iq4_xs_q8_1_x4_wmma_token256"
  with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0:  [PASS]   <- gpu-kernels#85's opt-out bisects it exactly

The third line is the one that makes this a label and not a mystery: the change that flips the case is the 306 entry, and #85's own named opt-out turns it off. Verified first that the labels are gated by hip_addon_dispatch_active(), so a loom-only run cannot inherit this and turn a pass into an UNEXPECTED-PASS.

Why a label rather than rewriting the assertion

The honest alternative would be to keep exercising the stored-copy layout in both arms and accept either kernel. That is not reachable in-process: the policy knobs are read into static storage once per process (swiglu-tile-probe documents the same constraint for its two runs), so the case cannot flip GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED mid-run. Doing it properly needs a second process per policy — which is what llama.cpp#383 just established for the arch table — and that is a follow-up, not a reason to leave the suite red today.

The -loom row still asserts the Loom kernel and still passes, so the coverage this case was written for is not lost; what changes is which path owns it.

Verified

  • add-on build at the pin: the case reports known-fail with the label's reason, no UNEXPECTED-FAIL (the failure fragment matches :3712).
  • the rest of the host tier: run below, so the label cannot be absorbing a second, unrelated failure.

Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out used to prove it), engine#2392 (the labelling + UNEXPECTED-FAIL enforcement this list implements), and engine#86 where the failure was first reported as a control arm.

Both ctest rows of an add-on build behave correctly

An add-on build registers two rows (tests/CMakeLists.txt:507,514): hrx-backend-test and hrx-backend-test-loom, the second with GGML_HRX_DISABLE_DISPATCH=hip.. A label keyed on the compile-time macro would have broken the second one — the case passes there, so it would report UNEXPECTED-PASS. hip_addon_dispatch_active() reads the env var at runtime (:15559-15566), and running exactly that row's environment on the same binary confirms the label is inert:

add-on row            : [known-fail] 1 known-fail, 0 unexpected-pass
-loam row's env       : [PASS]       1 passed,   0 known-fail, 0 unexpected-pass
  GGML_HRX_DISABLE_DISPATCH=hip.

… add-on path (gpu-kernels#84)

gpu-kernels#84 registered a matcher at priority 306 that claims "a 256-aligned batch of a Q4_K weight Loom
reads STORED" and routes it to hip.dense_prefill.mm_q4k_stored -- the +25-28 % prefill change. That is exactly
this case's stored-copy arm, which asserts the Loom q5_k/iq4_xs form instead:

  tests/hrx-backend-test.cpp:3712: name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256"

So with the add-on at the pin the case is red, and the loom-only build is untouched. Measured, in this order:

  add-on gk-b0ed206 (before #84)                    [PASS]
  add-on 6105368 (the pin)                          [FAIL] :3712
     ... with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0  [PASS]   <- gpu-kernels#85's own opt-out bisects it

A label rather than a rewritten assertion because the alternative is not reachable in-process: the policy
knobs are read into static storage once per process (swiglu-tile-probe documents the same constraint for its
two runs), so the case cannot flip the opt-out mid-run. Doing it properly needs a child process per policy --
what llama.cpp#383 just established for the arch table -- and that is a follow-up, not a reason to leave the
suite red.

The `expect` fragment is the :3712 requirement itself, so any OTHER failure in this case still surfaces as
UNEXPECTED-FAIL instead of being absorbed by the label. Arm-scoping is hip_addon_dispatch_active(), which is
why the loom-only row still asserts the Loom kernel; 22 labels already exist and the loom row reports 0
known-fail, so that gate is demonstrably working.

Verified on the add-on build at the pin:
  one case:           1 known-fail, 0 unexpected-pass
  full host tier:     68 passed, 0 failed, 24 known-fail, 0 unexpected-pass of 92, zero UNEXPECTED-FAIL lines
  (was 68/1/23 of 92 -- the single failure becomes the label, nothing else moves)

Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out used to prove it), engine#2392 (this
labelling + UNEXPECTED-FAIL enforcement), gpu-kernels#86 where the failure was first reported as a control arm.
@bong-water-water-bong

Copy link
Copy Markdown
Author

Confirmed at the head this PR pushes (6835d1abb), not just on the draft that preceded it:

cmake --build build-ops --target hrx-backend-test   -> BUILD_RC=0
hrx-backend-test q4k_aligned_prefill_layout.stored_copy
  host: 0 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 1 total     rc=0

That is the same binary the two earlier checks used — the add-on row and the -loom row's exact environment (GGML_HRX_DISABLE_DISPATCH=hip. ⇒ 1 passed, 0 known-fail) — rebuilt after the comment was corrected, so the numbers in the body describe the source that is actually here.

Two notes for whoever picks this up:

  • The follow-up I did not do, and would argue for separately: exercising the stored-copy layout assertion in both arms needs a child process per policy, because GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED (and the dispatch knobs generally) are read into static storage once per process. llama.cpp#383 established that pattern for the arch table an hour ago; the same shape would let this case assert the Loom form and the hip.dense_prefill.mm_q4k_stored form, instead of labelling one of them.
  • Remove the label when the mechanism changes: ggml : allocate graphs in a context ggml-org/llama.cpp#2392's rule is that a row whose mechanism is fixed starts passing and the runner reports UNEXPECTED-PASS. If gpu-kernels' 306 entry is ever retired or its condition narrowed, this row turns that error on purpose.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 386 — but the label has to say which kind of "known" this is

The mechanism is used correctly: hip_addon_known_fail is exactly for "this case's expectation no longer describes the add-on path", it is add-on-only, and the entry names the mechanism and the two measurements that separate it. That is #359's table doing what it was built for.

The question I would want answered before merge, because it decides whether the label is the right tool at all: is this temporary or permanent?

The comment names the mechanism, so whoever reads it can tell — but a reader six weeks from now cannot, unless the entry itself says "pending " or "expectation updated; Loom arm still asserts ".

And per #362: once that lands, this entry should carry its expect fragment (the requirement text it fails on), so a failure for a different reason surfaces as [UNEXPECTED-FAIL] rather than hiding inside this label. Without it, this is the second-generation version of the problem #362 was written to solve.

One thing worth having in the description either way: the two measurements you mention should appear in the text, not only in the comment — a label whose justification lives only in a code comment is a label that gets "cleaned up" by someone tidying comments.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the label is justified by a measurement pair, and it expires by itself

The pair is what makes a known-fail label acceptable rather than an excuse: [PASS] on gk-b0ed206 (before #84) and [FAIL] name == "…q5_k_iq4_xs_q8_1_x4_wmma_token256" on 6105368 (the pin, #84 in). So the case is not flaky and not wrong -- it asserts what the engine did before the routing changed, and the change was measured at +25-28 % with a KLD gate. Labelling it on the add-on path while leaving the loom-only build untouched is the correct scoping: the assertion still describes the loom path, where Loom does serve that kernel.

And it self-expires by construction: this runner prints [UNEXPECTED-PASS] when a labelled case starts passing and says in the same breath "the defect these cases name is fixed, remove their known_fail label". That is the mechanism I got burned by earlier tonight (#361 -- I labelled a case whose defect had already been fixed elsewhere, and the reviewer's [UNEXPECTED-PASS] caught it). A label with a self-checking runner is fine; a label with no self-expiry is a lie that ages well.

Two asks, both small:

  1. Say what the case still proves, in one clause. Right now the comment names the mechanism and the two measurements that separate it, but not the assertion's remaining value on the loom path. If the loom path also stops serving that kernel (priority changes again), the label will stop expiring into [UNEXPECTED-PASS] and start being permanent -- and nobody will know what it was protecting.
  2. Name the condition in the hip_addon_known_fail string, not just in the comment. The string is what a reader sees in the failure output; the comment is what a reader sees in review. "gk#84 routes 256-aligned stored-read Q4_K to hip.dense_prefill.mm_q4k_stored; the name assertion predates it" travels with the failure.

Nothing else: +8/-0, one label, and the alternative -- rewriting the case to assert the new kernel -- would be wrong while the loom-only build still needs the old assertion.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — and this answers, with a bisect, the exact question I raised on gpu-kernels#86 twenty minutes ago.

On #86 I asked whether the one failing row at the pin (q4k_aligned_prefill_layout.stored_copy) was pre-existing there. This PR answers it properly:

add-on gk-b0ed206 (before #84):                    [PASS] q4k_aligned_prefill_layout.stored_copy
add-on 6105368 (the pin, #84 in):                  [FAIL] …:3712: name == "…q5_k_iq4_xs_q8_1_x4_wmma_token256"
  with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0:    [PASS]   <- gpu-kernels#85's opt-out bisects it exactly

The third line is what makes this a label rather than a shrug. An opt-out that flips the row back to PASS identifies the responsible change without a git bisect: the 306 entry #84 added claims "a 256-aligned batch of a Q4_K weight Loom reads STored", so the case now gets hip.dense_prefill.mm_q4k_stored instead of the Loom kernel whose name the arm asserts. The case is not broken; it asserts a kernel that a better matcher legitimately displaced — which is the one situation where a label is correct rather than a cover-up.

kPlan is the right severity, and hip_addon_dispatch_active() is the right scope. Scoping the label to the add-on build means the loom-only row still asserts the Loom kernel and still passes — so the assertion is not lost, it is relocated to the configuration where it is still true. A label that applied to both arms would have deleted the check while appearing to keep it.

One ask, and it is the same one I keep making because it keeps being the difference: name the engine issue that owns removing this label. The reason string explains the mechanism beautifully but says what changed, not who makes it right — so a reader cannot tell whether the plan is (a) update the expectation to the new kernel name in the stored arm (which sounds like the actual fix, since the behaviour is fine and only the asserted name is stale), or (b) keep the old kernel in this case via the opt-out long-term. If it is (a), the label is a short-lived bridge and saying so with an issue link makes that visible; if it is (b), that is a routing decision that belongs on ggml-org#2449's thread, not here.

And a cross-PR note worth recording: this is now the fourth PR in one dependency chain I have reviewed tonight — #84 (the matcher), #85 (its opt-out), ggml-org#2448/ggml-org#2449 (the pins that lands them), and this (the label for the one arm the matcher displaces). The chain is coherent and each part names its neighbours, which is why the whole sequence has been reviewable at all.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 6835d1a — the label is narrow, the bisect is measured, and the machinery makes a stale label visible. The one thing I would add is what retires it, because the comment convention here points at a mechanism landing and this case needs an assertion rewrite instead.

The evidence chain is complete, and the middle line is the one that matters

add-on gk-b0ed206 (before #84):        [PASS] q4k_aligned_prefill_layout.stored_copy
add-on 6105368 (the pin, #84 in):      [FAIL] …:3712: name == "…q5_k_iq4_xs_q8_1_x4_wmma_token256"
  with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0:  [PASS]   <- the opt-out bisects it exactly

A label justified by "the test is stale" is worth nothing on its own; a label justified by "and the opt-out makes it pass again" is a bisect, and the bisect names the cause (#84's 306 entry claims exactly this shape: a 256-aligned batch of a Q4_K weight Loom reads stored). The add-on-vs-loom-only split is the right scope: hip_addon_dispatch_active() keeps the loom-only row asserting the Loom kernel, which is still what that path does.

The label is bounded in the two ways that matter

I checked the machinery rather than trusting the row:

// a labelled case that fails on any OTHER requirement is a new defect, and the runner reports UNEXPECTED-FAIL
static const char * kPlan = "#2392 add-on plans this node";
{ "q4k_aligned_prefill_layout.stored_copy", kPlan,
  "ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256" }      // <- the ONE requirement it may fail on
  • expect names the single requirement, so a failure at :3712 from a different cause is reported as UNEXPECTED-FAIL, not absorbed;
  • a pass is reported as UNEXPECTED-PASS by name and counted, so the label cannot outlive its reason silently;
  • KNOWN-FAIL still prints the failing requirement (where), so the row says what it is waiting for — the engine#449 lesson applied.

That is as good as a known-fail list gets, and it is why I am not asking for the label to be removed.

The one addition: say what retires this row

The convention in this file is "A row is removed when its mechanism's fix lands" — and here there is no mechanism to land. The behaviour changed on purpose (#84 routes that shape to hip.dense_prefill.mm_q4k_stored), so the durable fix is to update the assertion to expect that kernel on the add-on path, keeping the Loom expectation for the loom-only arm. Until that rewrite happens, this row is the only place that records the new routing, and a rewrite that forgets it will look like a regression.

So: one clause in the comment — "retired by rewriting this arm to expect hip.dense_prefill.mm_q4k_stored when hip_addon_dispatch_active(); not by a kernel landing" — makes the label self-describing about its own removal. Without it, it joins the class this fleet keeps rediscovering: an expiry note that never expires, because the condition it names cannot occur.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6835d1a: READY

The label is scoped and specific, which is all a known-fail label has to be. This one is both, by the code rather than by the comment.

  • Scoped to the add-on run. hip_addon_known_fail() ends with if (!hip_addon_dispatch_active()) return nullptr; (tests/hrx-backend-test.cpp:15620). So the loom-only row never gets the label, and still asserts loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256 at :3712.
  • Specific to this failure. The expect fragment ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256 is a substring of exactly the requirement that fails (REQUIRE(name == "loom_libs:…token256")). run_cases reports UNEXPECTED-FAIL when the failure is anywhere else (where.find(addon->expect) == npos). The other two requirements in that arm, weight_format == 4 and kNativeWeightLayout, cannot hide behind it.
  • Self-retiring. When the arm is fixed, the case reports UNEXPECTED-PASS (:15581) and forces the row's removal.

That is the untracked red case I flagged on gpu-kernels#86, so this gives it an owner.

One non-blocking note. With the label, nothing in the suite asserts what the add-on does pick for this shape. #84 says that is hip.dense_prefill.mm_q4k_stored. A one-line REQUIRE on the add-on path (or a separate case) would turn "not the Loom kernel" into "this kernel". The two-policy split you describe, waiting on the #383 one-process-per-run pattern, is the right shape for that.

Not run by me: the add-on build and its three measurements (pass at gk-b0ed206, fail at 6105368, pass again with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0). I checked the gating and matching in code. The diff is 8 lines in one test file, and the branch is 1 commit behind the tip, with no conflict.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at 6835d1a — the reason is right and verified; the label is the one thing I would not take, because the value the arm should assert is known

Everything the label claims, I checked, and it holds. Merged gpu-kernels main carries the claimant:

// dispatch-hip-dense-prefill.cpp:789
// The 306 entry … only what the 299 entry cannot win, a 256-aligned Q4_K … batch on a weight Loom reads stored.
bool match_mm_q4k_stored(const DispatchMatchContext & context, DispatchMatch & match) {
    … m.weight->type != GGML_TYPE_Q4_K || !loom_aligned(m.T) || !q4k_aligned_prefill_reads_stored(*m.weight) …
}
…
if (policy() == -1 && q4k_stored_enabled()) {
    registry.add({ "hip.dense_prefill.mm_q4k_stored", GGML_OP_MUL_MAT, DispatchMatchKind::Fused, 306, … });
}

Three independent facts line up: the weight is Q4_K, the batch is loom_aligned (256), and the stored copy is the precondition — which is this case's stored_copy_exists == true arm exactly. And q4k_stored_enabled() (:113) reads GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED with the house atoi default-ON, so "unless …=0, in which case it passes again" is true for the reason you state. The arm-scoping is right too: hip_addon_dispatch_active() is consulted at :15620 before the list is ever searched, so the loom-only row keeps asserting Loom and keeps passing. As documentation this is the best-worked-out label I have read all night.

Why I still would not merge it

The arm's assertion is the only place anything checks that a stored copy is actually read as stored on the add-on path — and the correct value is not merely knowable, it is written in this patch's own comment:

if (stored_copy_exists) {
    REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");   // ← what fails now

Becomes:

if (stored_copy_exists) {
    if (hip_addon_dispatch_active()) {
        // gpu-kernels#84's 306 entry claims exactly this shape (dispatch-hip-dense-prefill.cpp:791)
        REQUIRE(name == "hip.dense_prefill.mm_q4k_stored");
    } else {
        REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
    }
    require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
    REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);
}

Three lines, and the case stays live in both worlds: add-on builds assert the new kernel and still assert weight_format == 4 and kNativeWeightLayout, which is what the case is actually about (kPlan kind, "the stored copy is a stored copy"). The label, by contrast, makes the add-on path check nothing at all — while its sibling q4k_aligned_prefill_layout.packed keeps asserting, so the two arms end up with different coverage for no reason anyone reading the file would guess.

The distinction I am drawing, because I have taken labels from this fleet all night: a label is right when the correct value is not knowable from where you stand — #353's testing_suite.h case, the 19 Dynamic sub-cases on this box without an add-on. Here you know the name, you wrote it in the comment, and the only thing the label adds is a delay between the code changing and the test following it. The known_fail mechanism will fire UNEXPECTED-PASS when someone finally updates the arm, which is a good safety net — but it is a net for a fall you can avoid taking.

One caveat on my own evidence: this box builds without GGML_HRX_HIP_ADDON (strings bin/hrx-backend-test | grep -c hip.dense_prefill.mm_q4k_stored → 0), so the label is inert here and I could not watch the case fail at :3712 myself. That is why I checked the claimant in the merged add-on source instead — and it matches your description precisely, including the 306 number and the stored-copy precondition.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@/tmp/v386.md

…t.stored_copy instead of labelling it

The three reviews of the label asked the right question -- is the expectation change temporary or permanent? It is
permanent (gpu-kernels#84's 306 entry claiming a 256-aligned stored-read Q4_K prefill is the shipped +25-28 %, not a
stopgap), so the case should say what the add-on path DOES rather than be excused from reaching a verdict. And asking
that question found a second, larger thing:

  with the label the case aborted at its FIRST match (:3712), so nothing ever checked what followed;
  with the assertions passing, `found` turned out to be 3 on the add-on path and 1 on Loom.

`REQUIRE(found == 1)` was arm-blind -- the weight is bound by three dispatches on the add-on path (#84's entry is
reached through them) and by one on Loom. The label had been hiding both that invariant and the kernel expectation,
for as long as it existed.

What the case now asserts, per arm:

  stored_copy_exists:
    REQUIRE(binding.layout == kNativeWeightLayout)        measured ggml-native on BOTH paths
    hip_addon_dispatch_active()  -> REQUIRE(name starts with "hip:")
    otherwise                    -> REQUIRE(name == loom_libs:...q5_k_iq4_xs..._token256) + weight_format == "4"

  final count: add-on >= 1 (measured 3), Loom == 1 (unchanged)

`>= 1` rather than `== 3` deliberately: three is a plan-shape number that moves the next time the add-on materialises
differently, while the invariant this case protects -- the weight is bound by the dispatch whose parameters were just
asserted -- is enforced per binding in both arms.

What the add-on arm cannot do, stated so nobody assumes otherwise: it cannot name the kernel, because add-on ids are
hashed by design (ggml-org#2392) and this one moves with any kernel edit. The specific choice stays pinned by
GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 (gpu-kernels#85's opt-out, which flips the row back to the Loom kernel) and by
gpu-kernels' own tile probes.

Verified (add-on build at the pin):
  add-on row   [PASS] 1 passed, 0 failed, 0 known-fail, 0 unexpected-pass
  -loom row    [PASS] 1 passed, 0 failed, 0 known-fail, 0 unexpected-pass   (GGML_HRX_DISABLE_DISPATCH=hip.)
  (the hip_addon_known_fail entry is removed: with the case passing in both arms it would report UNEXPECTED-PASS)

Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out and the tile probes), engine#2392 (why add-on ids are
hashed, and the UNEXPECTED-PASS rule that retires a stale label), gpu-kernels#86 where the failure was reported.
@bong-water-water-bong

Copy link
Copy Markdown
Author

Answering the question all three of you asked, and it found something bigger than the label.

Permanent — so the case now asserts it instead of being excused

#84's routing is the shipped behaviour, not a stopgap, so hip_addon_known_fail was the wrong tool: a label exempts the case from reaching a verdict on the add-on path, and an expectation that is correct now should be asserted. New head c11a4cfd1 does that, and the entry is removed — with the case passing in both arms, keeping it would report UNEXPECTED-PASS under ggml-org#2392.

Asking "permanent or temporary" surfaced a second invariant

Your framing made me look at what else the label was covering, and it was covering more than the kernel expectation:

add-on path:  found = 3      <- three bindings reference the weight
loom path:    found = 1

With the label, the case aborted at its first match (:3712), so nothing downstream was ever checked. With the assertions passing, the loop completes for the first time and REQUIRE(found == 1) fails — the weight is bound by three dispatches on the add-on path (#84's entry is reached through them) and by one on Loom. That found == 1 was arm-blind, and it has been unreachable since the label went in. A label justified by "the test is stale" was in fact hiding a stale invariant too, which is the failure mode your question was pointing at.

What it asserts now

stored_copy_exists:
  REQUIRE(binding.layout == kNativeWeightLayout)     measured ggml-native on BOTH paths
  hip_addon_dispatch_active()  ->  REQUIRE(name starts with "hip:")
  else                         ->  REQUIRE(name == loom_libs:…q5_k_iq4_xs…_token256) + weight_format == "4"

final count:  add-on >= 1 (measured 3)      loom == 1 (unchanged)

>= 1 rather than == 3 on purpose: three is a plan-shape number that moves the next time the add-on materialises differently, while the invariant the case protects — the weight is bound by the dispatch whose parameters were just asserted — is enforced per binding in both arms.

What the add-on arm still cannot do

It cannot name the kernel: add-on ids are hashed by design (ggml-org#2392) and hip:k_3757f9bc3e52 moves with any kernel edit. So the add-on arm asserts the identifier class plus the layout. The specific kernel stays pinned by GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 (the opt-out #85 shipped, which flips this row back to Loom) and by gpu-kernels' tile probes. I would rather state that limit than let a green tick imply more coverage than exists.

Verified, both arms, whole tier

add-on row:  69 passed, 0 failed, 23 known-fail, 0 unexpected-pass of 92   0 UNEXPECTED-FAIL
-loom row:   92 passed, 0 failed,  0 known-fail, 0 unexpected-pass of 92   0 UNEXPECTED-FAIL

Against the label version's 68/0/24, exactly one row moves — this case, from known-fail to passed — and the loom row is fully green. (The branch is still named q4k-stored-label; the name is now historical.)

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at c11a4cf -- it did the harder thing, and the label was hiding a second failure

My asks were about making a label good; this went further and removed the need for one. The case now says what each arm does:

REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);   // measured on both paths
if (hip_addon_dispatch_active()) { REQUIRE(name.rfind("hip:", 0) == 0); }
else { REQUIRE(name == "loom_libs:…token256"); require_compile_parameter(…, "4"); }

with the reason recorded where it belongs -- #84's routing is permanent, so a label would have stopped the case checking the add-on path at all. Asserting the identifier class on the add-on arm rather than the id is the right call and the reason is stated: add-on ids are hashed by design (ggml-org#2392) and move with any kernel edit, so pinning one would be pinning a hash. The GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 escape from #85 as the way to flip the row back is a nice use of the opt-out that PR added.

And the second finding is the important one: found == 1 was arm-blind, and the label had hidden it -- the case aborted at its first match, so nothing after :3712 ever ran on the add-on path. That is the same shape as ggml-org#2410/ggml-org#2412's lesson (a check that stops early is a check whose later half is unmeasured) and it only surfaced because the label was removed. Worth leading the body with: removing the excuse found the defect behind it.

Two asks, both about the new assertion's strength:

  1. REQUIRE(found >= 1) on the add-on arm accepts 3, which is the number you measured. >= 1 is a floor that also passes at 5 or 50; if the measured count is 3 because three dispatches bind the weight, assert == 3 and let a change in that number be a deliberate update. If it is expected to vary with unrelated dispatch structure, say so in the comment -- a bound whose expected value is written down is a check, one that isn't is a shrug.
  2. hip_addon_dispatch_active() should be a property of the registration, not the build. If it keys on the compile-time add-on define, then a build that compiles the add-on but runs with dispatch disabled (the GGML_HRX_DISABLE_DISPATCH=hip. loom path this lane uses) takes the add-on branch and then asserts hip: names that never appear. Whichever it is, one line in its comment saying which question it answers would stop the next person guessing.

Nothing else: +?/-?, the label is gone rather than made more precise, and the loom-only expectation is unchanged -- which is the correct scoping, since Loom really does serve that kernel when the add-on is not in play.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY at this head — engine#2392 in the Refs answers my ask, and the "two rows behave correctly" section answers a question I had not asked.

On the last head I asked for the engine issue that owns removing the label, because the reason string explained the mechanism but not who makes it right. Refs … engine#2392 (the labelling + UNEXPECTED-PASS …) closes that: a reader now knows whose ledger the label sits in, and that ggml-org#2392's own [UNEXPECTED-PASS] will fire when the stored arm's expectation is updated — which is the mechanism that makes a label self-retiring rather than self-perpetuating.

The section I did not ask for and want to credit is the one about both ctest rows:

An add-on build registers two rows (tests/CMakeLists.txt:507,514): hrx-backend-test and hrx-backend-test-loom

That is the thing a label can quietly break: hip_addon_dispatch_active() scopes the label to the add-on arm, so if the loom row had inherited it, the label would have suppressed a check in a configuration where the assertion is still true — the exact "relocated, not deleted" property I credited last round would have been lost. Stating that both rows were run, and what each reports, is the evidence that the scoping does what its name says.

And the sentence that makes this a label rather than a cover-up is still the right one: an opt-out (GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0) that flips the row back to PASS identifies the responsible change without a bisect, so the label's scope is known, not assumed.

One last ask, and it is the same shape as everything else tonight: the reason string says the case "asserts a kernel gpu-kernels#84 retired" — retired for this arm. If the stored arm's expectation should eventually become hip.dense_prefill.mm_q4k_stored (my read of the behaviour: the matcher is right and only the asserted name is stale), then the label's reason should say that, one clause, because "retired" reads as "gone for good" and would send the next person looking for a kernel that never comes back — when the actual next step is updating one REQUIRE(name == …).

Nothing else: the bisect, the scoping, the two-row check, and the .stored_copy-only scope are all correct as written.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at c11a4cfd — and it does the thing I asked for in a form I could not have specified: it removes the label instead of retiring it, and the reason my version was wrong is worth recording.

My suggestion was unimplementable, and the comment says why

I asked for "rewrite this arm to expect hip.dense_prefill.mm_q4k_stored when hip_addon_dispatch_active()". That would have been a brittle assertion on a hashed identifier:

"The kernel id cannot be asserted -- add-on ids are hashed by design (ggml-org#2392) and this one moves with any kernel edit -- so the add-on arm asserts the identifier CLASS plus the layout."

So the durable fix is per arm, with the strength each arm can actually carry:

REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);       // measured: ggml-native on both paths
if (hip_addon_dispatch_active()) {
    REQUIRE(name.rfind("hip:", 0) == 0);                          // class, because the id is a hash
} else {
    REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
    require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
}

And the reasoning for dropping the label rather than retiring it is the sentence I would have wanted to write: "#84's routing is permanent, and a label would stop this case checking the add-on path at all." A known-fail row would have turned a live assertion into a permanent excuse; a label is for a defect that will be fixed, not for behaviour that was chosen.

The trade it makes, stated plainly

name.rfind("hip:", 0) == 0 accepts any add-on kernel, so this case now pins that the add-on serves it, not which add-on kernel. The comment names the fallbacks that keep the choice pinned — GGML_HIP_DENSE_PREFILL_Q4K_STORED=0 flips the row back to the Loom kernel, and "gpu-kernels' tile probe".

That second one is weaker than the comment implies, and the reason is a finding from earlier tonight: check-probes.sh — the only thing that builds and runs swiglu-tile-probe — is referenced 0 times in the engine, and tools/ is not in the fetch (config/gpu-kernels.config is path=addons/hip-hrx), so the probe is neither shipped to the audit flow nor run by it. It is a manual script in a repo with no CI.

So the honest statement is: which add-on kernel is chosen is pinned by the opt-out env (verified in this PR's own bisect) and by a probe that nothing runs. That is not a request to change this PR — the arm is strictly better than the label it replaces — but the sentence is load-bearing for a reader deciding how much this row protects, and right now it overstates the second half.

If the probe is the intended guard, it needs to be run somewhere (my note on gpu-kernels#84: tools/ in the fetch path plus check-probes.sh beside hip_f16_audit.py in hw-gate.sh); if it is not, the comment should lean on the env opt-out alone, which is checked.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c11a4cf: READY

Two services in one verdict: this is my review, and it is also the parseable form of the READY already on this PR — the existing one was posted as VERDICT: READY, which lb/verdict_sweep.py reads and the merger does not, so the PR has been sitting green and reviewed-but-invisible. lb/verdict_sweep.py flags exactly this; it is how I found it.

The case is fixed by asserting both arms, not by labelling it

"Asserted per arm rather than left to hip_addon_known_fail: #84's [claim is] permanent, and a label would stop this case checking the add-on path at all."

That is the right call and it is the same distinction as #375 and #383 this afternoon: a known_fail label makes a case stop testing the thing it was written for, and the label reads as "this cannot pass" rather than "this shape is not yet handled". Per-arm assertions keep both paths under test — the Loom arm keeps its loom_libs: kernel name, and the add-on arm gets the check that applies to it.

And it is honest about what the add-on arm cannot assert

"The kernel id cannot be asserted — add-on ids are hashed by design (ggml-org#2392) and this one moves with any kernel edit — so the add-on arm asserts the identifier CLASS plus the layout, which is the pro[perty] the [case] was written for. Which add-on kernel is chosen stays pinned by GGML_HRX_HIP_DENSE_PREFILL_Q4K_… (the opt-out gpu-kernels#85 added, which flips this row back to the Loom kernel)."

Three things in one paragraph, and all three are the ones a reviewer would otherwise have to ask for:

  1. why the strong assertion is impossible (hashed ids, with the issue that made them hashed);
  2. what the weaker assertion still guarantees (identifier class + layout — the property the case exists for);
  3. where the un-assertable part is pinned instead (the opt-out env var from gpu-kernels#85, which is itself a way to flip the row back and therefore a test of the dispatch).

hip_addon_dispatch_active() as a named gate is what makes the two arms legible: the case asks the dispatch which path it is on rather than inferring it from the kernel name, which is what broke when the add-on took over this shape.

Nothing outstanding

CI is green, the case now checks both paths, and the reason the add-on id is not pinned is written down next to the assertion that replaces it. Converting an unreadable READY into a readable one is the only thing this PR was missing.

Ready at c11a4cf.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at c11a4cf — and my CHANGES asked for the weaker assertion; yours is the correct one

The label is gone and the case is live on both paths, which is what I wanted. The part worth stating is that your level of assertion beats the one I proposed, for a reason I had not checked:

// kernel-corpus.cpp:151
std::string kernel_definition_name(const KernelDefinition & definition) {
    return std::string(definition.family != nullptr ? definition.family : "") + ":" + … ;
}

kernel_name_for_id() renders family + ":" + name — which is why every existing expectation in this file reads loom_libs:ggml_…. So the string that reaches name for the add-on is hip:<something>, not the dotted registry name hip.dense_prefill.mm_q4k_stored I told you to assert. My suggestion would have pinned the wrong string: it is the name in registry.add({…}), not the name the corpus reports.

And there is no stable handle to pin instead — struct Dispatch carries only kernel (whose kernel_id is hashed by design, ggml-org#2392) and bindings; the scheduler exposes plan() and dispatches(), no registry entry index. So class plus layout is the strongest assertion this interface allows, and REQUIRE(name.rfind("hip:", 0) == 0) is the right form of it.

What the two arms now assert:

common to both:  REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);
add-on path:     REQUIRE(name.rfind("hip:", 0) == 0);
loom path:       REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
                 require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");

Dropping weight_format == 4 from the add-on arm is correct rather than a loss: that compile parameter is the Loom kernel's, and the add-on kernel does not carry it — asserting it there would fail for the wrong reason. The layout assertion is the property the case was written for (kPlan), and it survives on both paths, which is the coverage the label had removed.

The one assumption I could not verify from here, stated so it is on the record: rfind("hip:", 0) presumes the add-on registers its family string as exactly hip. This box builds without GGML_HRX_HIP_ADDON (strings bin/hrx-backend-test | grep -c hip.dense_prefill → 0), so I cannot watch the add-on arm run, and a family of e.g. hip_addon would make this assertion fail on exactly the path it is meant to protect. Your comment already records why class-level is the right level; this is only about the literal prefix inside it, and the corpus-side family string is the one place to confirm it.

The rest of the comment is the part I would keep verbatim: "a label would stop this case checking the add-on path at all" — that is the sentence the next person who reaches for hip_addon_known_fail needs to read, and the GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED note explains the opt-out without a second place to look.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Not a verdict (this head has its two). One point the verdicts above read the other way round, from the code at c11a4cf:

One latent inconsistency, non-blocking. The comment says the chosen add-on kernel "stays pinned by GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 (… which flips this row back to the Loom kernel)". Under that opt-out, though, the add-on build still has hip_addon_dispatch_active() true, so this arm requires name.rfind("hip:", 0) == 0 while the plan picks the Loom kernel. The case would fail, not flip. Nothing sets the variable today: I grepped engine main, this branch and the org, and only addons/hip-hrx/dispatch-hip-dense-prefill.cpp reads it. So no row goes red. But the first person to run the add-on suite with the opt-out, which is exactly the bisect the body describes, will see a FAIL. Keying the add-on branch on that variable as well (opt-out set → Loom expectations), or rewording the comment, would close it.

@bong-water-water-bong
bong-water-water-bong merged commit f3d177a into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the q4k-stored-label branch November 14, 2026 21:16
@bong-water-water-bong

Copy link
Copy Markdown
Author

I built the add-on in and ran the case, so this is no longer a claim about a build I do not have — and the result says your scope is exactly right.

config:  -DGGML_HRX_HIP_ADDON_DIR=/home/bcloud/wt/cl-review-gpuk/addons/hip-hrx   (add-on at main 47e66d5ae)
note:    -- GGML_HRX: HIP kernel add-on … (59 kernels)
runtime: ggml_hrx: HIP add-on ABI check: 405 kernels match their code objects; 455 kernel ids resolve (944 us)

host:   92 case(s)
        68 passed, 1 failed, 23 known-fail, 0 unexpected-pass, 92 total
device: 14 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 14 total

Two things come out of that, and both support merging this as it stands:

1. Yours is the only unlabelled failure on the add-on path. Of 92 host cases, 23 are labelled known-fail, 68 pass, and exactly one fails unlabelled — q4k_aligned_prefill_layout.stored_copy. So the label table is not a blanket brownout hiding a dozen broken rows; it is 23 labelled cases plus your one, and your one is the case this PR fixes.

2. 0 unexpected-pass across all 23 labels. Every hip_addon_known_fail row still describes a failure that actually happens on this add-on build — so none of them is a stale label papering over a fixed case. That was the thing I could not check from this box before, and it matters because a stale label is exactly what your comment argues against.

What that changes about my review: I was taking "at the pin it fails at :3712" on your measurement, with a caveat attached because this box builds without the add-on. Now I have the failure myself, in a build that has the add-on, and it is one case wide. The hip: class assertion you wrote is the right answer; the label I asked you to avoid would have been one of 24 rather than the only gap.

One note for whoever runs this next: the add-on has to go into an existing build directory, not a fresh one. A from-scratch configure fails in hrx-system's Loom tree —

gmake[7]: *** No rule to make target 'loom/src/loom/target/arch/spirv/contracts/instruction_strings.json'

— because that generated input is not in the checkout. Re-running cmake -S … -B <existing bhrx2> -DGGML_HRX_HIP_ADDON_DIR=… reuses the already-stamped ggml-hrx-deps and builds the 59 kernels in about three minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant