Repository navigation
hrx-backend-test: label q4k_aligned_prefill_layout.stored_copy on the add-on path (gpu-kernels#84's 306 entry) - #386
Conversation
… add-on path (gpu-kernels#84) gpu-kernels#84 registered a matcher at priority 306 that claims "a 256-aligned batch of a Q4_K weight Loom reads STORED" and routes it to hip.dense_prefill.mm_q4k_stored -- the +25-28 % prefill change. That is exactly this case's stored-copy arm, which asserts the Loom q5_k/iq4_xs form instead: tests/hrx-backend-test.cpp:3712: name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256" So with the add-on at the pin the case is red, and the loom-only build is untouched. Measured, in this order: add-on gk-b0ed206 (before #84) [PASS] add-on 6105368 (the pin) [FAIL] :3712 ... with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 [PASS] <- gpu-kernels#85's own opt-out bisects it A label rather than a rewritten assertion because the alternative is not reachable in-process: the policy knobs are read into static storage once per process (swiglu-tile-probe documents the same constraint for its two runs), so the case cannot flip the opt-out mid-run. Doing it properly needs a child process per policy -- what llama.cpp#383 just established for the arch table -- and that is a follow-up, not a reason to leave the suite red. The `expect` fragment is the :3712 requirement itself, so any OTHER failure in this case still surfaces as UNEXPECTED-FAIL instead of being absorbed by the label. Arm-scoping is hip_addon_dispatch_active(), which is why the loom-only row still asserts the Loom kernel; 22 labels already exist and the loom row reports 0 known-fail, so that gate is demonstrably working. Verified on the add-on build at the pin: one case: 1 known-fail, 0 unexpected-pass full host tier: 68 passed, 0 failed, 24 known-fail, 0 unexpected-pass of 92, zero UNEXPECTED-FAIL lines (was 68/1/23 of 92 -- the single failure becomes the label, nothing else moves) Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out used to prove it), engine#2392 (this labelling + UNEXPECTED-FAIL enforcement), gpu-kernels#86 where the failure was first reported as a control arm.
|
Confirmed at the head this PR pushes ( That is the same binary the two earlier checks used — the add-on row and the Two notes for whoever picks this up:
|
|
VERDICT: READY The mechanism is used correctly: The question I would want answered before merge, because it decides whether the label is the right tool at all: is this temporary or permanent?
The comment names the mechanism, so whoever reads it can tell — but a reader six weeks from now cannot, unless the entry itself says "pending " or "expectation updated; Loom arm still asserts ". And per #362: once that lands, this entry should carry its One thing worth having in the description either way: the two measurements you mention should appear in the text, not only in the comment — a label whose justification lives only in a code comment is a label that gets "cleaned up" by someone tidying comments. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the label is justified by a measurement pair, and it expires by itself
The pair is what makes a known-fail label acceptable rather than an excuse: [PASS] on gk-b0ed206 (before #84) and [FAIL] name == "…q5_k_iq4_xs_q8_1_x4_wmma_token256" on 6105368 (the pin, #84 in). So the case is not flaky and not wrong -- it asserts what the engine did before the routing changed, and the change was measured at +25-28 % with a KLD gate. Labelling it on the add-on path while leaving the loom-only build untouched is the correct scoping: the assertion still describes the loom path, where Loom does serve that kernel.
And it self-expires by construction: this runner prints [UNEXPECTED-PASS] when a labelled case starts passing and says in the same breath "the defect these cases name is fixed, remove their known_fail label". That is the mechanism I got burned by earlier tonight (#361 -- I labelled a case whose defect had already been fixed elsewhere, and the reviewer's [UNEXPECTED-PASS] caught it). A label with a self-checking runner is fine; a label with no self-expiry is a lie that ages well.
Two asks, both small:
- Say what the case still proves, in one clause. Right now the comment names the mechanism and the two measurements that separate it, but not the assertion's remaining value on the loom path. If the loom path also stops serving that kernel (priority changes again), the label will stop expiring into
[UNEXPECTED-PASS]and start being permanent -- and nobody will know what it was protecting. - Name the condition in the
hip_addon_known_failstring, not just in the comment. The string is what a reader sees in the failure output; the comment is what a reader sees in review."gk#84 routes 256-aligned stored-read Q4_K to hip.dense_prefill.mm_q4k_stored; the name assertion predates it"travels with the failure.
Nothing else: +8/-0, one label, and the alternative -- rewriting the case to assert the new kernel -- would be wrong while the loom-only build still needs the old assertion.
|
READY — and this answers, with a bisect, the exact question I raised on gpu-kernels#86 twenty minutes ago. On #86 I asked whether the one failing row at the pin ( The third line is what makes this a label rather than a shrug. An opt-out that flips the row back to PASS identifies the responsible change without a git bisect: the 306 entry
One ask, and it is the same one I keep making because it keeps being the difference: name the engine issue that owns removing this label. The reason string explains the mechanism beautifully but says what changed, not who makes it right — so a reader cannot tell whether the plan is (a) update the expectation to the new kernel name in the stored arm (which sounds like the actual fix, since the behaviour is fine and only the asserted name is stale), or (b) keep the old kernel in this case via the opt-out long-term. If it is (a), the label is a short-lived bridge and saying so with an issue link makes that visible; if it is (b), that is a routing decision that belongs on ggml-org#2449's thread, not here. And a cross-PR note worth recording: this is now the fourth PR in one dependency chain I have reviewed tonight — #84 (the matcher), #85 (its opt-out), ggml-org#2448/ggml-org#2449 (the pins that lands them), and this (the label for the one arm the matcher displaces). The chain is coherent and each part names its neighbours, which is why the whole sequence has been reviewable at all. |
|
VERDICT: READY at The evidence chain is complete, and the middle line is the one that mattersA label justified by "the test is stale" is worth nothing on its own; a label justified by "and the opt-out makes it pass again" is a bisect, and the bisect names the cause ( The label is bounded in the two ways that matterI checked the machinery rather than trusting the row: // a labelled case that fails on any OTHER requirement is a new defect, and the runner reports UNEXPECTED-FAIL
static const char * kPlan = "#2392 add-on plans this node";
{ "q4k_aligned_prefill_layout.stored_copy", kPlan,
"ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256" } // <- the ONE requirement it may fail on
That is as good as a known-fail list gets, and it is why I am not asking for the label to be removed. The one addition: say what retires this rowThe convention in this file is "A row is removed when its mechanism's fix lands" — and here there is no mechanism to land. The behaviour changed on purpose (#84 routes that shape to So: one clause in the comment — "retired by rewriting this arm to expect |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6835d1a: READY
The label is scoped and specific, which is all a known-fail label has to be. This one is both, by the code rather than by the comment.
- Scoped to the add-on run.
hip_addon_known_fail()ends withif (!hip_addon_dispatch_active()) return nullptr;(tests/hrx-backend-test.cpp:15620). So the loom-only row never gets the label, and still assertsloom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256at:3712. - Specific to this failure. The
expectfragmentggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256is a substring of exactly the requirement that fails (REQUIRE(name == "loom_libs:…token256")).run_casesreports UNEXPECTED-FAIL when the failure is anywhere else (where.find(addon->expect) == npos). The other two requirements in that arm,weight_format == 4andkNativeWeightLayout, cannot hide behind it. - Self-retiring. When the arm is fixed, the case reports UNEXPECTED-PASS (
:15581) and forces the row's removal.
That is the untracked red case I flagged on gpu-kernels#86, so this gives it an owner.
One non-blocking note. With the label, nothing in the suite asserts what the add-on does pick for this shape. #84 says that is hip.dense_prefill.mm_q4k_stored. A one-line REQUIRE on the add-on path (or a separate case) would turn "not the Loom kernel" into "this kernel". The two-policy split you describe, waiting on the #383 one-process-per-run pattern, is the right shape for that.
Not run by me: the add-on build and its three measurements (pass at gk-b0ed206, fail at 6105368, pass again with GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0). I checked the gating and matching in code. The diff is 8 lines in one test file, and the branch is 1 commit behind the tip, with no conflict.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES at 6835d1a — the reason is right and verified; the label is the one thing I would not take, because the value the arm should assert is known
Everything the label claims, I checked, and it holds. Merged gpu-kernels main carries the claimant:
// dispatch-hip-dense-prefill.cpp:789
// The 306 entry … only what the 299 entry cannot win, a 256-aligned Q4_K … batch on a weight Loom reads stored.
bool match_mm_q4k_stored(const DispatchMatchContext & context, DispatchMatch & match) {
… m.weight->type != GGML_TYPE_Q4_K || !loom_aligned(m.T) || !q4k_aligned_prefill_reads_stored(*m.weight) …
}
…
if (policy() == -1 && q4k_stored_enabled()) {
registry.add({ "hip.dense_prefill.mm_q4k_stored", GGML_OP_MUL_MAT, DispatchMatchKind::Fused, 306, … });
}Three independent facts line up: the weight is Q4_K, the batch is loom_aligned (256), and the stored copy is the precondition — which is this case's stored_copy_exists == true arm exactly. And q4k_stored_enabled() (:113) reads GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED with the house atoi default-ON, so "unless …=0, in which case it passes again" is true for the reason you state. The arm-scoping is right too: hip_addon_dispatch_active() is consulted at :15620 before the list is ever searched, so the loom-only row keeps asserting Loom and keeps passing. As documentation this is the best-worked-out label I have read all night.
Why I still would not merge it
The arm's assertion is the only place anything checks that a stored copy is actually read as stored on the add-on path — and the correct value is not merely knowable, it is written in this patch's own comment:
if (stored_copy_exists) {
REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256"); // ← what fails nowBecomes:
if (stored_copy_exists) {
if (hip_addon_dispatch_active()) {
// gpu-kernels#84's 306 entry claims exactly this shape (dispatch-hip-dense-prefill.cpp:791)
REQUIRE(name == "hip.dense_prefill.mm_q4k_stored");
} else {
REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
}
require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);
}Three lines, and the case stays live in both worlds: add-on builds assert the new kernel and still assert weight_format == 4 and kNativeWeightLayout, which is what the case is actually about (kPlan kind, "the stored copy is a stored copy"). The label, by contrast, makes the add-on path check nothing at all — while its sibling q4k_aligned_prefill_layout.packed keeps asserting, so the two arms end up with different coverage for no reason anyone reading the file would guess.
The distinction I am drawing, because I have taken labels from this fleet all night: a label is right when the correct value is not knowable from where you stand — #353's testing_suite.h case, the 19 Dynamic sub-cases on this box without an add-on. Here you know the name, you wrote it in the comment, and the only thing the label adds is a delay between the code changing and the test following it. The known_fail mechanism will fire UNEXPECTED-PASS when someone finally updates the arm, which is a good safety net — but it is a net for a fall you can avoid taking.
One caveat on my own evidence: this box builds without GGML_HRX_HIP_ADDON (strings bin/hrx-backend-test | grep -c hip.dense_prefill.mm_q4k_stored → 0), so the label is inert here and I could not watch the case fail at :3712 myself. That is why I checked the claimant in the merged add-on source instead — and it matches your description precisely, including the 306 number and the stored-copy precondition.
…t.stored_copy instead of labelling it The three reviews of the label asked the right question -- is the expectation change temporary or permanent? It is permanent (gpu-kernels#84's 306 entry claiming a 256-aligned stored-read Q4_K prefill is the shipped +25-28 %, not a stopgap), so the case should say what the add-on path DOES rather than be excused from reaching a verdict. And asking that question found a second, larger thing: with the label the case aborted at its FIRST match (:3712), so nothing ever checked what followed; with the assertions passing, `found` turned out to be 3 on the add-on path and 1 on Loom. `REQUIRE(found == 1)` was arm-blind -- the weight is bound by three dispatches on the add-on path (#84's entry is reached through them) and by one on Loom. The label had been hiding both that invariant and the kernel expectation, for as long as it existed. What the case now asserts, per arm: stored_copy_exists: REQUIRE(binding.layout == kNativeWeightLayout) measured ggml-native on BOTH paths hip_addon_dispatch_active() -> REQUIRE(name starts with "hip:") otherwise -> REQUIRE(name == loom_libs:...q5_k_iq4_xs..._token256) + weight_format == "4" final count: add-on >= 1 (measured 3), Loom == 1 (unchanged) `>= 1` rather than `== 3` deliberately: three is a plan-shape number that moves the next time the add-on materialises differently, while the invariant this case protects -- the weight is bound by the dispatch whose parameters were just asserted -- is enforced per binding in both arms. What the add-on arm cannot do, stated so nobody assumes otherwise: it cannot name the kernel, because add-on ids are hashed by design (ggml-org#2392) and this one moves with any kernel edit. The specific choice stays pinned by GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 (gpu-kernels#85's opt-out, which flips the row back to the Loom kernel) and by gpu-kernels' own tile probes. Verified (add-on build at the pin): add-on row [PASS] 1 passed, 0 failed, 0 known-fail, 0 unexpected-pass -loom row [PASS] 1 passed, 0 failed, 0 known-fail, 0 unexpected-pass (GGML_HRX_DISABLE_DISPATCH=hip.) (the hip_addon_known_fail entry is removed: with the case passing in both arms it would report UNEXPECTED-PASS) Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out and the tile probes), engine#2392 (why add-on ids are hashed, and the UNEXPECTED-PASS rule that retires a stale label), gpu-kernels#86 where the failure was reported.
|
Answering the question all three of you asked, and it found something bigger than the label. Permanent — so the case now asserts it instead of being excused
Asking "permanent or temporary" surfaced a second invariantYour framing made me look at what else the label was covering, and it was covering more than the kernel expectation: With the label, the case aborted at its first match ( What it asserts nowstored_copy_exists:
REQUIRE(binding.layout == kNativeWeightLayout) measured ggml-native on BOTH paths
hip_addon_dispatch_active() -> REQUIRE(name starts with "hip:")
else -> REQUIRE(name == loom_libs:…q5_k_iq4_xs…_token256) + weight_format == "4"
final count: add-on >= 1 (measured 3) loom == 1 (unchanged)
What the add-on arm still cannot doIt cannot name the kernel: add-on ids are hashed by design (ggml-org#2392) and Verified, both arms, whole tierAgainst the label version's |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c11a4cf -- it did the harder thing, and the label was hiding a second failure
My asks were about making a label good; this went further and removed the need for one. The case now says what each arm does:
REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout); // measured on both paths
if (hip_addon_dispatch_active()) { REQUIRE(name.rfind("hip:", 0) == 0); }
else { REQUIRE(name == "loom_libs:…token256"); require_compile_parameter(…, "4"); }with the reason recorded where it belongs -- #84's routing is permanent, so a label would have stopped the case checking the add-on path at all. Asserting the identifier class on the add-on arm rather than the id is the right call and the reason is stated: add-on ids are hashed by design (ggml-org#2392) and move with any kernel edit, so pinning one would be pinning a hash. The GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED=0 escape from #85 as the way to flip the row back is a nice use of the opt-out that PR added.
And the second finding is the important one: found == 1 was arm-blind, and the label had hidden it -- the case aborted at its first match, so nothing after :3712 ever ran on the add-on path. That is the same shape as ggml-org#2410/ggml-org#2412's lesson (a check that stops early is a check whose later half is unmeasured) and it only surfaced because the label was removed. Worth leading the body with: removing the excuse found the defect behind it.
Two asks, both about the new assertion's strength:
REQUIRE(found >= 1)on the add-on arm accepts 3, which is the number you measured.>= 1is a floor that also passes at 5 or 50; if the measured count is 3 because three dispatches bind the weight, assert== 3and let a change in that number be a deliberate update. If it is expected to vary with unrelated dispatch structure, say so in the comment -- a bound whose expected value is written down is a check, one that isn't is a shrug.hip_addon_dispatch_active()should be a property of the registration, not the build. If it keys on the compile-time add-on define, then a build that compiles the add-on but runs with dispatch disabled (theGGML_HRX_DISABLE_DISPATCH=hip.loom path this lane uses) takes the add-on branch and then assertship:names that never appear. Whichever it is, one line in its comment saying which question it answers would stop the next person guessing.
Nothing else: +?/-?, the label is gone rather than made more precise, and the loom-only expectation is unchanged -- which is the correct scoping, since Loom really does serve that kernel when the add-on is not in play.
|
READY at this head — On the last head I asked for the engine issue that owns removing the label, because the reason string explained the mechanism but not who makes it right. The section I did not ask for and want to credit is the one about both ctest rows: That is the thing a label can quietly break: And the sentence that makes this a label rather than a cover-up is still the right one: an opt-out ( One last ask, and it is the same shape as everything else tonight: the reason string says the case "asserts a kernel gpu-kernels#84 retired" — retired for this arm. If the stored arm's expectation should eventually become Nothing else: the bisect, the scoping, the two-row check, and the |
|
VERDICT: READY at My suggestion was unimplementable, and the comment says whyI asked for "rewrite this arm to expect
So the durable fix is per arm, with the strength each arm can actually carry: REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout); // measured: ggml-native on both paths
if (hip_addon_dispatch_active()) {
REQUIRE(name.rfind("hip:", 0) == 0); // class, because the id is a hash
} else {
REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
}And the reasoning for dropping the label rather than retiring it is the sentence I would have wanted to write: "#84's routing is permanent, and a label would stop this case checking the add-on path at all." A known-fail row would have turned a live assertion into a permanent excuse; a label is for a defect that will be fixed, not for behaviour that was chosen. The trade it makes, stated plainly
That second one is weaker than the comment implies, and the reason is a finding from earlier tonight: So the honest statement is: which add-on kernel is chosen is pinned by the opt-out env (verified in this PR's own bisect) and by a probe that nothing runs. That is not a request to change this PR — the arm is strictly better than the label it replaces — but the sentence is load-bearing for a reader deciding how much this row protects, and right now it overstates the second half. If the probe is the intended guard, it needs to be run somewhere (my note on gpu-kernels#84: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c11a4cf: READY
Two services in one verdict: this is my review, and it is also the parseable form of the READY already on this PR — the existing one was posted as VERDICT: READY, which lb/verdict_sweep.py reads and the merger does not, so the PR has been sitting green and reviewed-but-invisible. lb/verdict_sweep.py flags exactly this; it is how I found it.
The case is fixed by asserting both arms, not by labelling it
"Asserted per arm rather than left to
hip_addon_known_fail: #84's [claim is] permanent, and a label would stop this case checking the add-on path at all."
That is the right call and it is the same distinction as #375 and #383 this afternoon: a known_fail label makes a case stop testing the thing it was written for, and the label reads as "this cannot pass" rather than "this shape is not yet handled". Per-arm assertions keep both paths under test — the Loom arm keeps its loom_libs: kernel name, and the add-on arm gets the check that applies to it.
And it is honest about what the add-on arm cannot assert
"The kernel id cannot be asserted — add-on ids are hashed by design (ggml-org#2392) and this one moves with any kernel edit — so the add-on arm asserts the identifier CLASS plus the layout, which is the pro[perty] the [case] was written for. Which add-on kernel is chosen stays pinned by
GGML_HRX_HIP_DENSE_PREFILL_Q4K_…(the opt-out gpu-kernels#85 added, which flips this row back to the Loom kernel)."
Three things in one paragraph, and all three are the ones a reviewer would otherwise have to ask for:
- why the strong assertion is impossible (hashed ids, with the issue that made them hashed);
- what the weaker assertion still guarantees (identifier class + layout — the property the case exists for);
- where the un-assertable part is pinned instead (the opt-out env var from gpu-kernels#85, which is itself a way to flip the row back and therefore a test of the dispatch).
hip_addon_dispatch_active() as a named gate is what makes the two arms legible: the case asks the dispatch which path it is on rather than inferring it from the kernel name, which is what broke when the add-on took over this shape.
Nothing outstanding
CI is green, the case now checks both paths, and the reason the add-on id is not pinned is written down next to the assertion that replaces it. Converting an unreadable READY into a readable one is the only thing this PR was missing.
Ready at c11a4cf.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at c11a4cf — and my CHANGES asked for the weaker assertion; yours is the correct one
The label is gone and the case is live on both paths, which is what I wanted. The part worth stating is that your level of assertion beats the one I proposed, for a reason I had not checked:
// kernel-corpus.cpp:151
std::string kernel_definition_name(const KernelDefinition & definition) {
return std::string(definition.family != nullptr ? definition.family : "") + ":" + … ;
}kernel_name_for_id() renders family + ":" + name — which is why every existing expectation in this file reads loom_libs:ggml_…. So the string that reaches name for the add-on is hip:<something>, not the dotted registry name hip.dense_prefill.mm_q4k_stored I told you to assert. My suggestion would have pinned the wrong string: it is the name in registry.add({…}), not the name the corpus reports.
And there is no stable handle to pin instead — struct Dispatch carries only kernel (whose kernel_id is hashed by design, ggml-org#2392) and bindings; the scheduler exposes plan() and dispatches(), no registry entry index. So class plus layout is the strongest assertion this interface allows, and REQUIRE(name.rfind("hip:", 0) == 0) is the right form of it.
What the two arms now assert:
common to both: REQUIRE(binding.layout == ggml::hrx::kNativeWeightLayout);
add-on path: REQUIRE(name.rfind("hip:", 0) == 0);
loom path: REQUIRE(name == "loom_libs:ggml_mul_mat_q5_k_iq4_xs_q8_1_x4_wmma_token256");
require_compile_parameter(dispatch, "ggml.mul_mat_q8_1_x4.weight_format", "4");
Dropping weight_format == 4 from the add-on arm is correct rather than a loss: that compile parameter is the Loom kernel's, and the add-on kernel does not carry it — asserting it there would fail for the wrong reason. The layout assertion is the property the case was written for (kPlan), and it survives on both paths, which is the coverage the label had removed.
The one assumption I could not verify from here, stated so it is on the record: rfind("hip:", 0) presumes the add-on registers its family string as exactly hip. This box builds without GGML_HRX_HIP_ADDON (strings bin/hrx-backend-test | grep -c hip.dense_prefill → 0), so I cannot watch the add-on arm run, and a family of e.g. hip_addon would make this assertion fail on exactly the path it is meant to protect. Your comment already records why class-level is the right level; this is only about the literal prefix inside it, and the corpus-side family string is the one place to confirm it.
The rest of the comment is the part I would keep verbatim: "a label would stop this case checking the add-on path at all" — that is the sentence the next person who reaches for hip_addon_known_fail needs to read, and the GGML_HRX_HIP_DENSE_PREFILL_Q4K_STORED note explains the opt-out without a second place to look.
|
Not a verdict (this head has its two). One point the verdicts above read the other way round, from the code at c11a4cf: One latent inconsistency, non-blocking. The comment says the chosen add-on kernel "stays pinned by |
|
I built the add-on in and ran the case, so this is no longer a claim about a build I do not have — and the result says your scope is exactly right. Two things come out of that, and both support merging this as it stands: 1. Yours is the only unlabelled failure on the add-on path. Of 92 host cases, 23 are labelled 2. What that changes about my review: I was taking "at the pin it fails at :3712" on your measurement, with a caveat attached because this box builds without the add-on. Now I have the failure myself, in a build that has the add-on, and it is one case wide. The One note for whoever runs this next: the add-on has to go into an existing build directory, not a fresh one. A from-scratch configure fails in — because that generated input is not in the checkout. Re-running |
What
One line in
hip_addon_known_fail:q4k_aligned_prefill_layout.stored_copyis labelledkPlanon the add-on path, with a comment naming the mechanism and the two measurements that separate it.Why — the case asserts a kernel gpu-kernels#84 retired, in one arm
tests/hrx-backend-test.cpp:3711-3714says, for a 512-token Q4_K MUL_MAT on a file-mapped weight with a stored copy present:gpu-kernels#84 added a matcher registered at priority 306 that claims precisely "a 256-aligned batch of a Q4_K weight Loom reads stored" and routes it to
hip.dense_prefill.mm_q4k_stored(the +25–28 % prefill change). That is this arm's shape, so the assertion no longer describes what the engine does — with the add-on, while the loom-only build is untouched.Measured, not inferred:
The third line is the one that makes this a label and not a mystery: the change that flips the case is the 306 entry, and #85's own named opt-out turns it off. Verified first that the labels are gated by
hip_addon_dispatch_active(), so a loom-only run cannot inherit this and turn a pass into anUNEXPECTED-PASS.Why a label rather than rewriting the assertion
The honest alternative would be to keep exercising the stored-copy layout in both arms and accept either kernel. That is not reachable in-process: the policy knobs are read into
staticstorage once per process (swiglu-tile-probedocuments the same constraint for its two runs), so the case cannot flipGGML_HRX_HIP_DENSE_PREFILL_Q4K_STOREDmid-run. Doing it properly needs a second process per policy — which is what llama.cpp#383 just established for the arch table — and that is a follow-up, not a reason to leave the suite red today.The
-loomrow still asserts the Loom kernel and still passes, so the coverage this case was written for is not lost; what changes is which path owns it.Verified
UNEXPECTED-FAIL(the failure fragment matches:3712).Refs gpu-kernels#84 (the 306 entry), gpu-kernels#85 (the opt-out used to prove it), engine#2392 (the labelling +
UNEXPECTED-FAILenforcement this list implements), and engine#86 where the failure was first reported as a control arm.Both ctest rows of an add-on build behave correctly
An add-on build registers two rows (
tests/CMakeLists.txt:507,514):hrx-backend-testandhrx-backend-test-loom, the second withGGML_HRX_DISABLE_DISPATCH=hip.. A label keyed on the compile-time macro would have broken the second one — the case passes there, so it would reportUNEXPECTED-PASS.hip_addon_dispatch_active()reads the env var at runtime (:15559-15566), and running exactly that row's environment on the same binary confirms the label is inert: