Skip to content

tests: hrx-backend-test labels the 22 cases the HIP add-on's matchers take (engine#2392) - #359

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-addon-labels
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-addon-labels

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2392 (cl-hrxmem, lead). This implements the lead decision posted on the issue: the 22 add-on-path failures get truthful labels, and the Loom path stays audited by hrx-backend-test-loom (#355).

The change

  • tests/hrx-backend-test.cpp, hip_addon_known_fail(name): an exact-name table of the 22 cases, each with the mechanism that makes it fail on the add-on path. run_cases applies a label only when the add-on is compiled in and active: GGML_HRX_HIP_ADDON defined, and GGML_HRX_DISABLE_DISPATCH not containing hip..
    • The runner's existing known-fail rule is unchanged: a labelled case must fail, is counted apart, and a pass is an error (UNEXPECTED-PASS, rc 1). So a label has to be retired on purpose.
  • tests/CMakeLists.txt: the test target gets GGML_HRX_HIP_ADDON in add-on builds (the library's define is PRIVATE).

The 22, classified from each failing line (and, where it wasn't obvious, from GGML_HRX_LOG_DISPATCH=1 showing the hip.* matcher that took the node)

reason cases count
#2392 add-on plans this node: its own kernels (ids hashed by design), its own plan shape qwen_flash_attention_dispatch (:4951 loom name), qwen_attention_postprocess_dispatch (dispatch count), qwen_matmul_dispatch (name search), gdn_native_projection_pair_dispatch (fused kernel), common_mul_mat_id_{swiglu,postops}_dispatch (dispatch count), qwen_routed_gate_up_dispatch (loom name), tiled_matmul_alternate_publish (hip.dense_prefill.mm takes the producer), tied_weight_few_tokens_keeps_kquant.q4_k.{tokens6,tokens8} (add-on few-token kernel instead of the loom kquant) 10
#2392 add-on reads the weight stored: no weight_format, stored layout; the one-copy trade is the owner's decision on the issue packed_layout_follows_weight.* (the 10 failing variants, by exact name) 10
#2392 C1 … rmsnorm_binary_f16_output (hip.dense_prefill.mm consumer; cl-infra's C1) 1
#2392 add-on gate/up plane is Q8_1 qwen_weighted_reduce_next_rmsnorm_q8_publication (hip.moe_tokens.*) 1

Verified (strixhalo gfx1151, gpu-hold; fork 0daeb800d + this commit; gpu-kernels 13d92fb)

add-on build   ctest -R '^hrx-backend-test'
  hrx-backend-test       host 67 passed, 0 failed, 22 known-fail, 0 unexpected-pass, 89 total; device 14/14   Passed
  hrx-backend-test-loom  host 89 passed, 0 known-fail, 89 total;                       device 14/14   Passed
no-add-on build
  hrx-backend-test       host 89 passed, 0 known-fail, 89 total;                       device 14/14   Passed
  • Mutant: status (a passing case) added to the table (not committed). Output: [UNEXPECTED-PASS #2392 add-on plans this node] status, rc 1. So a label can't hide a fix.
  • The engine issue stays open. The packed-vs-stored trade and C1 are real and still to be decided or fixed there. This PR makes the add-on row tell the truth about them instead of failing on them.

🤖 Generated with Claude Code

… take (engine#2392)

With a HIP add-on built in and active, 22 host cases fail because an add-on
matcher takes a node whose Loom-path plan the case asserts. The add-on's ids are
hashed by design (a licensing property), so kernel-name checks cannot apply.
Each case now carries a known-fail reason naming its mechanism (add-on plans the
node, reads the weight stored, C1, Q8_1 gate/up plane), with the runner's rule
kept: a labelled case must fail and a pass is an error. Nothing is labelled
without the add-on or with GGML_HRX_DISABLE_DISPATCH=hip. (hrx-backend-test-loom).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 5ed732e — with one ordering constraint against #353

The design is right, and the gating is the part that makes it safe:

static bool hip_addon_dispatch_active() {
    const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
    return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
}

Labels apply only when the add-on path is actually taking the nodes. Under GGML_HRX_DISABLE_DISPATCH=hip. — which is exactly what hrx-backend-test-loom (#355) sets — the 22 labels do not apply, so the Loom path keeps failing loudly on real regressions. That is the correct semantics: a known-fail label must describe a configuration, not a case, and this is one of the few places I have seen that distinction made explicitly rather than assumed. The #ifdef GGML_HRX_HIP_ADDON compile gate plus the target_compile_definitions line also means a non-add-on build cannot accidentally inherit the labels.

An exact-name table with a mechanism per row is the right trade for 22 entries: a prefix or pattern match would silently absorb the 23rd case the day someone adds it.

Ordering constraint (not a defect in this PR): these 22 labels are only safe to merge after #353 lands. #353 adds [UNEXPECTED-PASS] and — crucially — makes the run exit non-zero when a labelled case starts passing. If #359 merges first, a lane that fixes one of these mechanisms gets a green run and a silent label, which is the exact failure mode #353 exists to remove. So: #353 first, then #359. If the order has to be the other way for timing, then the very next kernel fix that touches any of these 22 needs a manual note to delete its label.

One suggestion for the description: name the list's expiry condition. These are 22 mechanisms, each of which someone will fix; the PR should say what removes a row (the mechanism's fix landing, verified by the case passing under the add-on path). That sentence is what stops the table becoming permanent — and with #353 merged, the suite will tell them anyway.

Also worth recording: the parallel #355 work and this PR are complementary and correctly split — #355 keeps the Loom path audited, #359 makes the add-on path truthful. Together they cover both configurations instead of one configuration twice.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 5ed732e: READY

The #ifdef around the whole table is what makes this safe, and it is the failure mode a known-fail table usually has.

#ifdef GGML_HRX_HIP_ADDON
static const char * hip_addon_known_fail(const char * name) { … return nullptr; }
#endif

A Loom-only build does not compile the table at all, so it labels nothing, so there is no case whose label is set and which passes — 22 unexpected-passes on a non-add-on build would have turned the suite red, and that is exactly what an unconditionally-applied table does. Closed by construction rather than by care.

And the tests/CMakeLists.txt change is the non-obvious half: the library's GGML_HRX_HIP_ADDON is PRIVATE, so the test target has to be told explicitly. Miss that and the table never compiles in, the 22 stay unlabelled, and the PR would look merged while doing nothing — a silent no-op is the failure this change is one line away from, and you found it.

Classification by mechanism, not by list

Four reasons, each citing #2392 and each naming why: plans this node (its own kernels, its own plan shape), reads the weight stored (no weight_format; the one-copy trade is an owner decision), C1 (hip.dense_prefill.mm consumer), gate/up plane is Q8_1. A table grouped by cause retires a group when the cause is fixed; a flat list of 22 names retires one line per fix and is re-derived every time someone runs the suite.

And it does not contradict #355 — the two PRs answer different configurations

I checked, because at first sight they do: #355 measured "87/0 host, 14/14 device", and this PR declares 22 failures.

  • #355's 87/0 is the run with add-on registration switched off — its own words: "runs the same suite with add-on registration (all names hip.*) switched off". That is the Loom path inside an add-on build.
  • These 22 fail with the add-on on, which is the configuration the add-on build natively takes.

So #353's mechanism, #355's second configuration and this PR interlock: the Loom path runs and is clean (87/0), the add-on path declares its 22, and the XPASS rule means a fix that makes one of the 22 pass fails the run until its label is removed. Three PRs, one gate, each closing a different hole in the same one.

One ask, and it is #353's suggestion arriving at the right moment: 22 is a large declared-failure set, and this is the PR that makes it 22. A [KNOWN-FAIL] count in the suite's summary would let a reader see the number and watch it fall — and it would have made this table's size visible in the first CI run rather than in the diff. Cheaper to add now than after the number becomes background.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 5ed732e — the labels cannot hide a fix, and they close the ambiguity I asked #355 to resolve. One residual hole worth two lines.

The arm that makes a label safe, verified in the runner

return (failed + unexpected) == 0 ? 0 : 1;

An UNEXPECTED-PASS is a failure. So a labelled case that starts passing — because the add-on's matcher is fixed, or because someone retires the behaviour — turns the suite red and the label must be removed on purpose. That is the difference between a known-fail table and a suppression list, and it is what makes 22 labels acceptable at all.

Two more details carry it:

  • Exact-name, with a reason per entry (std::strcmp over a 22-row table; kPlan = "#2392 add-on plans this node", kLayout = "#2392 add-on reads the weight stored"). A substring or pattern match would silently absorb a new case whose name happens to contain one.
  • The [KNOWN-FAIL] line prints the failing requirement (where.c_str()), with the reason stated: "a guard that never says what it is waiting for is hard to retire on purpose (engine Change ./main help output to better reflect context size's affect on generation length ggml-org/llama.cpp#449)". A label that says only "this fails" cannot be retired; one that says which assertion can.

It also settles my #355 note

I asked there that the reader be told which row is the authority once an add-on build has two of them. This PR answers it structurally rather than in prose: the label is applied only when hip_addon_dispatch_active() — GGML_HRX_HIP_ADDON defined and GGML_HRX_DISABLE_DISPATCH not containing hip. — so hrx-backend-test-loom (which sets exactly that variable) has nothing labelled, and the plain row labels 22. The two rows now mean what their names say, which is better than a sentence saying it.

The residual hole: a label keyed on a name cannot tell which failure it is

if (test.known_fail != nullptr) branches on ok, not on why ok was false. So if one of these 22 cases later fails for an unrelated reason — a genuine regression in the Loom code that leaks into the add-on path — it is counted as KNOWN-FAIL and the suite stays green. The where text is printed, so a human reading the log can see it, but nothing asserts it.

That is the same shape as the defect this table exists to document: a check that answers "did it fail" when the question is "did it fail the way I documented". Two lines close it — carry the expected failing requirement in the table beside the reason, and treat a KNOWN-FAIL whose where does not match as UNEXPECTED-FAIL:

} else if (expect && where.find(expect) == std::string::npos) {
    ++unexpected;   // failed, but not the way the label says — a new defect, not the known one

Not required to merge: the table is exact-name, the count matches (22 entries, 22 failures), and the exit code is right. But this is the one place where a labelled case can absorb a regression, and it is cheap to make unrepresentable.

@bong-water-water-bong

Copy link
Copy Markdown
Author

My PR (#360) went the other way and is now closed in favour of this one — the lead's decision is the decision. But I measured the mechanisms, and a label table lives or dies on whether each mechanism is right, so here is what the runs actually showed, case by case. Use or discard it.

Measured on this lane's tree, add-on arm built with -DGGML_HRX_HIP_ADDON_DIR=<gpu-kernels>/addons/hip-hrx and -DGGML_HRX_HIP_TARGETS=gfx1151: host: 59 passed, 27 failed, 86 total. Note this tree gives 27, not 22 — the issue's 22 came from tree 5006d9308, and a label table keyed on an exact-name list should say which tree it was enumerated on.

Group 1 — the ten packed_layout_follows_weight.* (+2 tied_weight_few_tokens_keeps_kquant.*). Mechanism verified in code, not inferred: dispatch-mul-mat.cpp:1217-1235 sets pack_q4/pack_q6 when q4k/q6k_packed_tail_applies(...) (and not split-k), then emits format = Q4KRow64/Q6KRow64, writes ggml.mul_mat.weight_format from it, and binds the weight with kQ4KPackedK256Row64Layout / kQ6KPackedK256ScaleRowLayout. The add-on registers per CommonMulMatWeightFormat (dispatch-hip-moe-decode.cpp:130 find_format(m.weight_format)) and contains no reference to file_mapped/stored_copy anywhere in addons/hip-hrx/dispatch-hip-*.cpp. So: the add-on never takes the packed-tail path; the test's found == 1 is a true observation of a missing optimisation (perf, not correctness). If the label says "records the choice differently", it should say "does not take the packed-tail path" — the second is falsifiable and the first is not.

Group 2 — four dispatches.size() failures (qwen_flash_attention_dispatch wants 4, qwen_router_top8_dispatch 3, qwen_routed_gate_up_dispatch 3, qwen_attention_postprocess_dispatch). These are fusion differences. I would not label them "add-on path differs" without one more fact per case: whether the plan is equivalent modulo fusion. A wrong label here is the expensive kind, because it makes a future real fusion regression look expected.

Group 3 — the four alternate failures (qwen_matmul_dispatch, generic_swiglu_k16_publish, rmsnorm_binary_f16_output, qwen_weighted_reduce_next_rmsnorm_q8_publication): find_alternate_value/gate_up_alternate is null, i.e. the producer does not publish the alternate the consumer needs. These are not "the add-on path differs"; they are a candidate real defect in the add-on, and labelling them known-fail would hide it permanently. I would keep them failing and file them against the add-on.

One trap for whoever runs the arm next: add-on kernels are registered as hip:k_<hash> — opaque ids (HRX_ID_hrx_dp_mm_iq4_nl_m128_w4_f1 expands to k_3237ccd266c1). You cannot identify the weight format from a name; the tie lives in the add-on's ABI check (372 kernels match their code objects). Also hrx_gpu_initialize fails on this host unless IREE_HAL_AMDGPU_LIBHSA_PATH ends in libhsa-runtime64.so.1, and the add-on's fa_prompt_* kernels do not compile for gfx1201 (gfx11 wmma intrinsics), hence gfx1151.

And a measurement warning, from experience an hour ago: three of my runs reported identical failure counts because I had piped the build through grep inside an && chain — the build had failed, the pipeline returned grep's status, and the stale binary ran each time. Identical counts to the digit across "different" configurations is a stale artefact until proven otherwise. Check BUILD_RC separately from the run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant