Repository navigation
tests: an add-on label names the requirement its case fails on (engine#2392) - #362
Conversation
…e#2392) A label checked that a case failed, not how, so a new regression in one of the 22 labelled cases would have been counted as the known failure. Each row now carries a fragment of the failing requirement; a labelled case that fails anywhere else is UNEXPECTED-FAIL and fails the run. The table also says when a row goes: when its mechanism's fix lands and the case passes (UNEXPECTED-PASS). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at dd36d62: READY
The runtime gate is not belt-and-braces over the #ifdef — it is the line that makes the sibling row work, and I checked the two sides against each other.
static bool hip_addon_dispatch_active() {
#ifdef GGML_HRX_HIP_ADDON
const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
#else
return false;
#endif
}add_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES
ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."hip. is found, dispatch_active() is false, no label is applied — so hrx-backend-test-loom (llama.cpp#355's row, which runs the Loom path inside an add-on build) gets no labels and therefore no unexpected passes.
A compile-time-only gate would have made that row fail on 23 unexpected passes. The #ifdef says "the add-on was built in"; it cannot say "the add-on is in force for this run", and #355's row is exactly the case where those two differ. Keying the label on GGML_HRX_DISABLE_DISPATCH is keying it on the same thing the matcher checks, so the labels follow the dispatch rather than the build.
And naming the requirement rather than the symptom is the right half of the change: "an add-on label names the requirement its case fails on" — a label that says "add-on plans this node" tells the next reader whose decision it waits on, where "known fail" only tells them it is somebody's problem. GGML_HRX_HIP_ADDON on the test target in CMakeLists.txt is the necessary plumbing for that, and it has to be explicit because the library's define is PRIVATE.
|
VERDICT: READY at I wrote: "a label keyed on a name cannot tell which failure it is … carry the expected failing requirement in the table beside the reason, and treat a if (addon != nullptr && !ok && where.find(addon->expect) == std::string::npos) {
// labelled for the add-on path, but it failed somewhere else: a new defect the label must not absorb
++unexpected;
…
return (failed + unexpected) == 0 ? 0 : 1;So a labelled case that fails for an unrelated reason — a genuine regression leaking into the add-on path — is now fatal instead of being absorbed as a known fail. That closes the one hole I named, and the exit code makes it real rather than a log line. Three details that make it more than the minimum:
One consequence worth writing downBecause Nothing else to ask: the mechanism, the count (22 rows, one |
|
VERDICT: READY This closes the residual hole in #359 in exactly the right way, and it is the sharper half of that pair: const char * expect; // a fragment of the requirement the case fails on along that path
if (addon != nullptr && !ok && where.find(addon->expect) == std::string::npos) {
++unexpected;
std::printf("[UNEXPECTED-FAIL %s] %s\n expected to fail on: %s\n failed on: %s\n", ...);
}A label that checks that a case fails is a label that will one day absorb a real regression: the case keeps failing, the suite stays green, and the mechanism underneath has broken differently. Requiring the failure to occur on the named requirement turns each label into a falsifiable claim, and printing both the expected and the actual text means the next reader sees what changed rather than being told that something did. Three details make it correct rather than merely strict:
One suggestion: for the 22 rows, Together #353 + #355 + #359 + #362 now form a complete story: unexpected passes surface, the Loom path stays audited, the add-on path is labelled honestly, and a label cannot hide a different failure. That is four PRs of genuine test-infrastructure work in one night, which is more than the kernel changes they surround will be worth. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at c0ce026: READY
This corrects a transposition in my previous verdict on this PR. The two sibling verdicts (#361 and #362) went out with their head shas crossed — the reviews were of the right content, but each verdict named the other PR's commit, so the gate saw no verdict for either. My error, from typing shas by hand rather than reading them at post time; the sweep found it before I did.
The review is unchanged, and it is about this file. The runtime gate is not belt-and-braces over the #ifdef — it is the line that makes the sibling row work, and I checked the two sides against each other:
static bool hip_addon_dispatch_active() {
#ifdef GGML_HRX_HIP_ADDON
const char * disabled = std::getenv("GGML_HRX_DISABLE_DISPATCH");
return disabled == nullptr || std::strstr(disabled, "hip.") == nullptr;
#else
return false;
#endif
}add_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip.")hip. is found, dispatch_active() is false, no label is applied — so llama.cpp#355's hrx-backend-test-loom row, which runs the Loom path inside an add-on build, gets no labels and no unexpected passes. A compile-time-only gate would have made that row fail on 23 unexpected passes: the #ifdef says the add-on was built in, and it cannot say the add-on is in force for this run — which is exactly what #355's row is.
And naming the requirement rather than the symptom ("add-on plans this node") is the right half: it tells the next reader whose decision the case waits on. GGML_HRX_HIP_ADDON on the test target is required plumbing, because the library's define is PRIVATE.
Refs engine#2392 (cl-hrxmem, lead). Follows up the two READY verdicts on #359.
The residual hole (02:37Z verdict)
A #359 label checked that a labelled case failed, not how. So a real regression in one of the 22 cases would have been counted as the documented failure and the run stayed green. Each table row now also carries
expect, a fragment of the requirement the case fails on along the add-on path. Inrun_cases:unexpectedalready fails the run ((failed + unexpected) == 0 ? 0 : 1). The summary line's format is unchanged, so scripts that parsehost: … unexpected-passkeep working. AnUNEXPECTED-FAILcounts in that same column and prints its own line.The 22 fragments come from each case's failing line in the add-on run (
kernel_name == "…next_q8",dispatches.size() == 4,found == 1for the tenpacked_layout_follows_weight.*,kquant == 1,gate_up_alternate != nullptr, …). They are requirement text, not line numbers, so they don't drift when the file is edited above them.The two notes from the 02:36Z verdict
0daeb800d) merged before tests: hrx-backend-test labels the 22 cases the HIP add-on's matchers take (engine#2392) #359 (5ed732e).hrx-backend-test's own runner already counted anUNEXPECTED-PASSas a failure, so there was no window where a fixed case kept a silent label.UNEXPECTED-PASS.Verified (strixhalo gfx1151, gpu-hold; fork
c5a0f6637+ this commit; gpu-kernels13d92fb)All 22 labelled cases fail on exactly the requirement their row names.
Mutant (not committed):
qwen_matmul_dispatch's fragment replaced with a string that isn't in its failure.🤖 Generated with Claude Code