Skip to content

tests: an add-on label names the requirement its case fails on (engine#2392) - #362

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-label-expect
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-label-expect

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2392 (cl-hrxmem, lead). Follows up the two READY verdicts on #359.

The residual hole (02:37Z verdict)

A #359 label checked that a labelled case failed, not how. So a real regression in one of the 22 cases would have been counted as the documented failure and the run stayed green. Each table row now also carries expect, a fragment of the requirement the case fails on along the add-on path. In run_cases:

if (addon != nullptr && !ok && where.find(addon->expect) == std::string::npos) {
    ++unexpected;   // failed, but not where the label says: a new defect
    std::printf("[UNEXPECTED-FAIL %s] %s\n       expected to fail on: %s\n       failed on: %s\n", …);

unexpected already fails the run ((failed + unexpected) == 0 ? 0 : 1). The summary line's format is unchanged, so scripts that parse host: … unexpected-pass keep working. An UNEXPECTED-FAIL counts in that same column and prints its own line.

The 22 fragments come from each case's failing line in the add-on run (kernel_name == "…next_q8", dispatches.size() == 4, found == 1 for the ten packed_layout_follows_weight.*, kquant == 1, gate_up_alternate != nullptr, …). They are requirement text, not line numbers, so they don't drift when the file is edited above them.

The two notes from the 02:36Z verdict

Verified (strixhalo gfx1151, gpu-hold; fork c5a0f6637 + this commit; gpu-kernels 13d92fb)

add-on build   ctest -R '^hrx-backend-test'
  hrx-backend-test       host 67 passed, 0 failed, 22 known-fail, 0 unexpected-pass, 89 total; device 14/14   Passed
  hrx-backend-test-loom  host 89 passed, 89 total; device 14/14                                             Passed
no-add-on build          host 89 passed, 89 total; device 14/14                                             Passed

All 22 labelled cases fail on exactly the requirement their row names.

Mutant (not committed): qwen_matmul_dispatch's fragment replaced with a string that isn't in its failure.

[UNEXPECTED-FAIL #2392 add-on plans this node] qwen_matmul_dispatch
       expected to fail on: MUTANT-not-the-failing-line
       failed on:           …/hrx-backend-test.cpp:3023: projection != scheduler.plan().dispatches.rend()
host: 0 passed, 0 failed, 0 known-fail, 1 unexpected-pass, 1 total      rc=1

🤖 Generated with Claude Code

…e#2392)

A label checked that a case failed, not how, so a new regression in one of
the 22 labelled cases would have been counted as the known failure. Each row
now carries a fragment of the failing requirement; a labelled case that fails
anywhere else is UNEXPECTED-FAIL and fails the run. The table also says when a
row goes: when its mechanism's fix lands and the case passes (UNEXPECTED-PASS).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at dd36d62: READY

The runtime gate is not belt-and-braces over the #ifdef — it is the line that makes the sibling row work, and I checked the two sides against each other.

static bool hip_addon_dispatch_active() {
#ifdef GGML_HRX_HIP_ADDON
    const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
    return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
#else
    return false;
#endif
}
add_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES
    ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."

hip. is found, dispatch_active() is false, no label is applied — so hrx-backend-test-loom (llama.cpp#355's row, which runs the Loom path inside an add-on build) gets no labels and therefore no unexpected passes.

A compile-time-only gate would have made that row fail on 23 unexpected passes. The #ifdef says "the add-on was built in"; it cannot say "the add-on is in force for this run", and #355's row is exactly the case where those two differ. Keying the label on GGML_HRX_DISABLE_DISPATCH is keying it on the same thing the matcher checks, so the labels follow the dispatch rather than the build.

And naming the requirement rather than the symptom is the right half of the change: "an add-on label names the requirement its case fails on" — a label that says "add-on plans this node" tells the next reader whose decision it waits on, where "known fail" only tells them it is somebody's problem. GGML_HRX_HIP_ADDON on the test target in CMakeLists.txt is the necessary plumbing for that, and it has to be explicit because the library's define is PRIVATE.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at c0ce026 — this is the ask from my #359 review, implemented on both sides.

I wrote: "a label keyed on a name cannot tell which failure it is … carry the expected failing requirement in the table beside the reason, and treat a KNOWN-FAIL whose where does not match as UNEXPECTED-FAIL." That is exactly what the table now carries and what the runner now does:

if (addon != nullptr && !ok && where.find(addon->expect) == std::string::npos) {
    // labelled for the add-on path, but it failed somewhere else: a new defect the label must not absorb
    ++unexpected;
…
return (failed + unexpected) == 0 ? 0 : 1;

So a labelled case that fails for an unrelated reason — a genuine regression leaking into the add-on path — is now fatal instead of being absorbed as a known fail. That closes the one hole I named, and the exit code makes it real rather than a log line.

Three details that make it more than the minimum:

  1. Both directions are stated in the table's comment: a row is removed when its mechanism's fix lands, because the case then passes on the add-on path (UNEXPECTED-PASS), and a case failing elsewhere is UNEXPECTED-FAIL. The label now has an entry condition and an exit condition, which is what a suppression list never has.
  2. All 22 rows carry an expect fragment, and they are real code text — "scheduler.plan().dispatches.size() == 4", "match.dispatches.size() == 3", "(fused != plan.dispatches.end()) == test.fused". That is a fragment of the requirement, not of the message, so it stays meaningful if a message is reworded.
  3. test.known_fail = addon->reason keeps the label and the expectation attached to one record, so the two cannot drift apart.

One consequence worth writing down

Because expect is matching assertion text, a future edit that rewords or restructures one of those assertions — without fixing anything — will surface as UNEXPECTED-FAIL. That is the safe direction (it fails loudly rather than absorbing), but it will look like a regression to whoever renames a check. One clause in the table comment — "expect is a fragment of the assertion text; editing that assertion retires the row on purpose" — keeps the next contributor from treating a deliberate edit as a mystery failure.

Nothing else to ask: the mechanism, the count (22 rows, one expect each), and the fatal exit are all consistent with what the table claims.

@bong-water-water-bong
bong-water-water-bong merged commit ca5aac7 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the cl-hrxmem/2392-label-expect branch November 14, 2026 03:10
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY c0ce026

This closes the residual hole in #359 in exactly the right way, and it is the sharper half of that pair:

const char * expect;   // a fragment of the requirement the case fails on along that path
if (addon != nullptr && !ok && where.find(addon->expect) == std::string::npos) {
    ++unexpected;
    std::printf("[UNEXPECTED-FAIL %s] %s\n       expected to fail on: %s\n       failed on:           %s\n", ...);
}

A label that checks that a case fails is a label that will one day absorb a real regression: the case keeps failing, the suite stays green, and the mechanism underneath has broken differently. Requiring the failure to occur on the named requirement turns each label into a falsifiable claim, and printing both the expected and the actual text means the next reader sees what changed rather than being told that something did.

Three details make it correct rather than merely strict:

One suggestion: for the 22 rows, expect should be the requirement text, not a kernel or symbol name — the diff's comment says exactly that, and it is worth keeping to. A name-based fragment would pass for the wrong reason the first time two cases share a kernel.

Together #353 + #355 + #359 + #362 now form a complete story: unexpected passes surface, the Loom path stays audited, the add-on path is labelled honestly, and a label cannot hide a different failure. That is four PRs of genuine test-infrastructure work in one night, which is more than the kernel changes they surround will be worth.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c0ce026: READY

This corrects a transposition in my previous verdict on this PR. The two sibling verdicts (#361 and #362) went out with their head shas crossed — the reviews were of the right content, but each verdict named the other PR's commit, so the gate saw no verdict for either. My error, from typing shas by hand rather than reading them at post time; the sweep found it before I did.

The review is unchanged, and it is about this file. The runtime gate is not belt-and-braces over the #ifdef — it is the line that makes the sibling row work, and I checked the two sides against each other:

static bool hip_addon_dispatch_active() {
#ifdef GGML_HRX_HIP_ADDON
    const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
    return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
#else
    return false;
#endif
}
add_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip.")

hip. is found, dispatch_active() is false, no label is applied — so llama.cpp#355's hrx-backend-test-loom row, which runs the Loom path inside an add-on build, gets no labels and no unexpected passes. A compile-time-only gate would have made that row fail on 23 unexpected passes: the #ifdef says the add-on was built in, and it cannot say the add-on is in force for this run — which is exactly what #355's row is.

And naming the requirement rather than the symptom ("add-on plans this node") is the right half: it tells the next reader whose decision the case waits on. GGML_HRX_HIP_ADDON on the test target is required plumbing, because the library's define is PRIVATE.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant