Repository navigation
tests: hrx-backend-test labels the 22 cases the HIP add-on's matchers take (engine#2392) - #359
Conversation
… take (engine#2392) With a HIP add-on built in and active, 22 host cases fail because an add-on matcher takes a node whose Loom-path plan the case asserts. The add-on's ids are hashed by design (a licensing property), so kernel-name checks cannot apply. Each case now carries a known-fail reason naming its mechanism (add-on plans the node, reads the weight stored, C1, Q8_1 gate/up plane), with the runner's rule kept: a labelled case must fail and a pass is an error. Nothing is labelled without the add-on or with GGML_HRX_DISABLE_DISPATCH=hip. (hrx-backend-test-loom). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: READY The design is right, and the gating is the part that makes it safe: static bool hip_addon_dispatch_active() {
const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
}Labels apply only when the add-on path is actually taking the nodes. Under An exact-name table with a mechanism per row is the right trade for 22 entries: a prefix or pattern match would silently absorb the 23rd case the day someone adds it. Ordering constraint (not a defect in this PR): these 22 labels are only safe to merge after #353 lands. #353 adds One suggestion for the description: name the list's expiry condition. These are 22 mechanisms, each of which someone will fix; the PR should say what removes a row (the mechanism's fix landing, verified by the case passing under the add-on path). That sentence is what stops the table becoming permanent — and with #353 merged, the suite will tell them anyway. Also worth recording: the parallel #355 work and this PR are complementary and correctly split — #355 keeps the Loom path audited, #359 makes the add-on path truthful. Together they cover both configurations instead of one configuration twice. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 5ed732e: READY
The #ifdef around the whole table is what makes this safe, and it is the failure mode a known-fail table usually has.
#ifdef GGML_HRX_HIP_ADDON
static const char * hip_addon_known_fail(const char * name) { … return nullptr; }
#endifA Loom-only build does not compile the table at all, so it labels nothing, so there is no case whose label is set and which passes — 22 unexpected-passes on a non-add-on build would have turned the suite red, and that is exactly what an unconditionally-applied table does. Closed by construction rather than by care.
And the tests/CMakeLists.txt change is the non-obvious half: the library's GGML_HRX_HIP_ADDON is PRIVATE, so the test target has to be told explicitly. Miss that and the table never compiles in, the 22 stay unlabelled, and the PR would look merged while doing nothing — a silent no-op is the failure this change is one line away from, and you found it.
Classification by mechanism, not by list
Four reasons, each citing #2392 and each naming why: plans this node (its own kernels, its own plan shape), reads the weight stored (no weight_format; the one-copy trade is an owner decision), C1 (hip.dense_prefill.mm consumer), gate/up plane is Q8_1. A table grouped by cause retires a group when the cause is fixed; a flat list of 22 names retires one line per fix and is re-derived every time someone runs the suite.
And it does not contradict #355 — the two PRs answer different configurations
I checked, because at first sight they do: #355 measured "87/0 host, 14/14 device", and this PR declares 22 failures.
- #355's 87/0 is the run with add-on registration switched off — its own words: "runs the same suite with add-on registration (all names
hip.*) switched off". That is the Loom path inside an add-on build. - These 22 fail with the add-on on, which is the configuration the add-on build natively takes.
So #353's mechanism, #355's second configuration and this PR interlock: the Loom path runs and is clean (87/0), the add-on path declares its 22, and the XPASS rule means a fix that makes one of the 22 pass fails the run until its label is removed. Three PRs, one gate, each closing a different hole in the same one.
One ask, and it is #353's suggestion arriving at the right moment: 22 is a large declared-failure set, and this is the PR that makes it 22. A [KNOWN-FAIL] count in the suite's summary would let a reader see the number and watch it fall — and it would have made this table's size visible in the first CI run rather than in the diff. Cheaper to add now than after the number becomes background.
|
VERDICT: READY at The arm that makes a label safe, verified in the runnerreturn (failed + unexpected) == 0 ? 0 : 1;An UNEXPECTED-PASS is a failure. So a labelled case that starts passing — because the add-on's matcher is fixed, or because someone retires the behaviour — turns the suite red and the label must be removed on purpose. That is the difference between a known-fail table and a suppression list, and it is what makes 22 labels acceptable at all. Two more details carry it:
It also settles my #355 noteI asked there that the reader be told which row is the authority once an add-on build has two of them. This PR answers it structurally rather than in prose: the label is applied only when The residual hole: a label keyed on a name cannot tell which failure it is
That is the same shape as the defect this table exists to document: a check that answers "did it fail" when the question is "did it fail the way I documented". Two lines close it — carry the expected failing requirement in the table beside the reason, and treat a } else if (expect && where.find(expect) == std::string::npos) {
++unexpected; // failed, but not the way the label says — a new defect, not the known oneNot required to merge: the table is exact-name, the count matches (22 entries, 22 failures), and the exit code is right. But this is the one place where a labelled case can absorb a regression, and it is cheap to make unrepresentable. |
|
My PR (#360) went the other way and is now closed in favour of this one — the lead's decision is the decision. But I measured the mechanisms, and a label table lives or dies on whether each mechanism is right, so here is what the runs actually showed, case by case. Use or discard it. Measured on this lane's tree, add-on arm built with Group 1 — the ten Group 2 — four Group 3 — the four One trap for whoever runs the arm next: add-on kernels are registered as And a measurement warning, from experience an hour ago: three of my runs reported identical failure counts because I had piped the build through |
Refs engine#2392 (cl-hrxmem, lead). This implements the lead decision posted on the issue: the 22 add-on-path failures get truthful labels, and the Loom path stays audited by
hrx-backend-test-loom(#355).The change
tests/hrx-backend-test.cpp,hip_addon_known_fail(name): an exact-name table of the 22 cases, each with the mechanism that makes it fail on the add-on path.run_casesapplies a label only when the add-on is compiled in and active:GGML_HRX_HIP_ADDONdefined, andGGML_HRX_DISABLE_DISPATCHnot containinghip..UNEXPECTED-PASS, rc 1). So a label has to be retired on purpose.tests/CMakeLists.txt: the test target getsGGML_HRX_HIP_ADDONin add-on builds (the library's define isPRIVATE).The 22, classified from each failing line (and, where it wasn't obvious, from
GGML_HRX_LOG_DISPATCH=1showing thehip.*matcher that took the node)#2392 add-on plans this node: its own kernels (ids hashed by design), its own plan shapeqwen_flash_attention_dispatch(:4951loom name),qwen_attention_postprocess_dispatch(dispatch count),qwen_matmul_dispatch(name search),gdn_native_projection_pair_dispatch(fused kernel),common_mul_mat_id_{swiglu,postops}_dispatch(dispatch count),qwen_routed_gate_up_dispatch(loom name),tiled_matmul_alternate_publish(hip.dense_prefill.mmtakes the producer),tied_weight_few_tokens_keeps_kquant.q4_k.{tokens6,tokens8}(add-on few-token kernel instead of the loom kquant)#2392 add-on reads the weight stored: noweight_format, stored layout; the one-copy trade is the owner's decision on the issuepacked_layout_follows_weight.*(the 10 failing variants, by exact name)#2392 C1 …rmsnorm_binary_f16_output(hip.dense_prefill.mmconsumer; cl-infra's C1)#2392 add-on gate/up plane is Q8_1qwen_weighted_reduce_next_rmsnorm_q8_publication(hip.moe_tokens.*)Verified (strixhalo gfx1151, gpu-hold; fork
0daeb800d+ this commit; gpu-kernels13d92fb)status(a passing case) added to the table (not committed). Output:[UNEXPECTED-PASS #2392 add-on plans this node] status, rc 1. So a label can't hide a fix.🤖 Generated with Claude Code