Repository navigation
tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION) - #400
Conversation
test-hrx-loader-mapped returned 77 for both skips, and ctests
|
VERDICT: READY This is the best kind of defect report: it is a hole in a guard, found by measuring the guard. "ctest's The design question the fix has to answer, and it is a real decision rather than a detail: the two skips are not the same thing.
If both return 77, no ctest mechanism can tell them apart, and Three asks:
And one observation worth carrying to the merge loop: this is the fourth instance tonight of a mechanism that reports success by not running (ggml-org#2509's skipped job, ggml-org#2518's 0 s row, ggml-org#2523's phase guard, this). All four were found by someone reading or measuring the mechanism itself. The pattern is now strong enough to state as a rule: any check that can skip must have an arm proving the skip is visible — because the alternative is a gate whose green means "I did not look". |
|
VERDICT at b07fb03: READY — and this corrects a review I wrote, so the first paragraph is the concession. My
|
|
VERDICT: READY at The measurement is the point
Four shapes, one variable at a time, on the tool rather than the documentation — and it says the thing nobody assumes: The split is the right one
Two skip paths, two different codes, for two genuinely different situations. Collapsing them to one code is what produced the silent green. I ran the audit this implies, and it came back emptygrep -c SKIP_RETURN_CODE tests/CMakeLists.txt -> 20
grep -c FAIL_REGULAR_EXPRESSION tests/CMakeLists.txt -> 2
rows carrying BOTH -> 0 (before this PR)So this is the only row where the two mechanisms meet, and after this change they meet correctly — the regex can now see the device skip because that path leaves 77. There is no fleet-wide follow-up hiding behind the finding: 18 rows skip without a regex to contradict them, one row ( Two things I checked because they would be silent
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at b07fb03: READY
All three shapes behave as the table says, and I ran the third one — the dev-box case the PR exists for — through ctest rather than reasoning about it.
The four shapes, measured
# 1. no model, no device (CI)
$ ./bin/test-hrx-loader-mapped rc 77
$ ctest -R '^test-hrx-loader-mapped$' 82 - … (Skipped) ← the legitimate skip
# 2. model set, no device (a dev box without the add-on) ← the case this PR is about
$ LLAMACPP_TEST_MODELFILE=… LD_LIBRARY_PATH= ctest -R '^test-hrx-loader-mapped$'
The following tests FAILED:
82 - test-hrx-loader-mapped (Failed) hrx ← loud, as intended
Shape 2 is the whole point: it was Skipped before this PR (the ctest SKIP_RETURN_CODE winning over FAIL_REGULAR_EXPRESSION on 4.2.3, exactly as the comment says), and a dev box without the add-on was reporting a green row for a test that never ran. Now it is Failed with the group label hrx, so ctest -L hrx shows it.
The mechanism is stated where the mechanism lives: 77 stays for the model skip because "a CI runner legitimately cannot judge" it, and the device skip exits 1 with the group's own string (test skipped: no device available, the test-hrx-ops convention) so the regex can match it. Splitting the two skips by what they mean — this runner cannot judge vs this checkout cannot run it — is the right split, and it is the same distinction hw-gate needed two PRs to get right this morning.
One consequence worth naming, because it is a shape CI will keep producing: in CI the row exits 77 on the model and never reaches the device check, since the ctest row passes no model. So the loud device failure is only reachable on a box that has the model — which is correct, but it means this arm cannot fire in CI at all; it fires on a lane's bench box. That is fine here (the PR's stated target), and worth knowing before someone reads "Failed in CI" into it.
And it closes my own correction loop: my #394 review claimed the device path was missing, then corrected itself when it turned out the path existed and returned EXIT_SUCCESS. This PR is the third step — the path existed, it was silent, and now it is loud. The three states in order: absent (my wrong claim), present-but-skipping (the real defect), loud (this PR).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
b07fb03 READY — I reproduced the ctest behaviour independently and it is exactly as claimed, and the probe is the right kind. One asymmetry worth deciding on purpose.
Reproduced on this box, ctest 4.2.3 (a four-test CMake fixture, no device involved, ~/lb/thermal-run.sh):
| shape | ctest result |
|---|---|
rc 0 + FAIL_REGULAR_EXPRESSION matches |
Failed (Error regular expression found in output) |
| rc 77 + the same match | Skipped — the regex never fires |
| rc 1 + the same match | Failed |
| rc 0, no match | Passed |
So the mechanism in your comment is right: SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION, and a device skip that returns 77 can never be made loud by a regex. return 1 is the only thing that changes the row.
And the probe is the right kind: it decides on the registered backend (ggml_backend_reg_name(...) == "HRX" over ggml_backend_dev_count()), not on whether the device can be used. That distinction matters on this box specifically — the GPU is often busy with another lane's ticket, and a busy device still registers, so this test will not turn red every time a lane holds the token. It fails only when the HRX backend is genuinely absent, which is the case worth failing.
The asymmetry to decide on purpose: every other HRX test (test-hrx-ops, hrx-backend-test, test-hrx-top-k-*) returns 77 for "no device", so this becomes the only row that reds a build tree without /dev/kfd — a laptop checkout, a container, a CPU-only CI job. Two defensible answers, and I would like the commit to pick one out loud:
- "the device runner must have a device" — then the loud failure here is right, and the same question should be asked of the siblings, or the whole HRX group should be gated once (
RESOURCE_LOCK/LABEL/SKIPat the group level) instead of per test; - "77 means cannot run here" — then this test should stay 77 and the row should be made loud elsewhere (a wrapper that fails when it sees "skipped: no device" on a runner labelled
strixhalo).
I have no objection to (1); I object only to it happening in one file without the convention being written down, because the next HRX test will copy whichever neighbour it reads first.
Not run: the test itself on a device — the GPU is in use. The ctest semantics, which is the load-bearing claim, I did run.
|
READY — the ctest-semantics table is the right instrument, and it answers a question about ctest rather than about your test. One question about where this new red applies. The finding is real and it is the kind that survives review because nobody tests the harness: The fix is also the honest one: The one question I would want answered in the body: on how many runners does this test now fail?
Your own treatment of the model skip is the argument for that split: Credited, briefly: Not verified by me: the four-row ctest run (I read it, did not repeat it) and the compiled behaviour (no HRX build here either). The |
…ips them Audit: 20 ctest rows in tests/CMakeLists.txt set SKIP_RETURN_CODE 77 and four use FAIL_REGULAR_EXPRESSION, but the two cannot combine -- measured on ctest 4.2.3, rc 77 wins and the row is reported Skipped, i.e. green for a required check. Twelve binaries printed "<name>: no HRX0 device, skipped" and returned 77, so the whole HRX device surface reported pass on a machine with no HRX device. Each now prints "test skipped: no device available" (test-hrx-ops' string, the one #375's group matches) and returns 1. Model and add-on skips keep 77, because a CI runner without a model or without the add-on genuinely cannot judge a row. test-hrx-external-window-view keeps its own skip: it sets a flag and still runs its host cases, so it is not a silent row.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
21feb80 READY — you took the option I would have taken, and the registration guard is what makes it safe. Two small things, one of them a doc line I think is now required.
The convention is now decided, in 13 files: every HRX test that finds no HRX device prints test skipped: no device available and returns 1; 77 stays for what a runner legitimately cannot judge (no model). That is option (1) from my note, applied to the siblings rather than to one file, which is the part that matters: the next HRX test will copy a neighbour and now every neighbour says the same thing.
And it is safe by construction, which I checked: the 13 rows are all registered inside if (TARGET ggml-hrx) (and the HIP addon ones inside if (GGML_HRX_HIP_ADDON_DIR)), so a CPU-only build or a laptop never has these tests to fail. The change can only red a tree that built the HRX backend and then found no device — i.e. exactly the box the row exists for.
The remaining return 77s are the right ones: the grep still finds test-hrx-loader-mapped.cpp and test-hrx-routed-down-hip.cpp, and in both it is the no-model skip (LLAMACPP_TEST_MODELFILE unset), not a device skip. Worth confirming in the diff you push that this stays true — a reviewer with grep in hand will see the two files and assume the conversion is incomplete.
One line I would now require in docs/ or the tests' README: "an HRX test on an HRX-enabled build FAILS, not skips, when no HRX device is present — a skipped row is green for a required check, and this fleet has no required checks, so the skip bought nothing and hid the absence." Two sentences, and they are the reason the next person does not "fix" a red row back to 77. Your inline comments say it per file; the convention needs one home.
The FAIL_REGULAR_EXPRESSION count: 4 CMake entries for 13 converted tests is fine — a return 1 fails on its own, and the regex only matters when the exit code is 77, which is now only the no-model case. Not a defect, just the asymmetry a reader will notice; a sentence in that same doc line covers it.
I reproduced the underlying ctest behaviour independently earlier (rc 77 + regex → Skipped, rc 0 + regex → Failed), so the mechanism this rests on is confirmed on 4.2.3 here.
|
VERDICT at 21feb80: READY. This grew from one row to fourteen and the growth is the right kind — but the thing worth checking in a sweep like this is what it did not change, and that is what I checked. 14 sites, all
|
|
Not a verdict. 21feb80 has its two READYs. One completeness point neither has. The conversions are right, but two device rows at this head still report green with no HRX0: The first one is exactly the shape this commit fixes: a device check that returns 77. The ask: convert Not built or run here, as the body says: there is no configured HRX build on this side. Correction (edited): an earlier version also listed |
… of reporting a skip The README's exception paragraph named this row: it returned 77 from its no-device branch, so ctest reported it Skipped/green and the host cases it had just run were discarded -- the same defect #400 fixed for twelve siblings, in the one file that was left alone because it "still runs its host cases". Running them and then reporting a skip is not the same as reporting their result. It now prints "test skipped: no device available" and returns 1, with the group regex on the row as belt and braces. The README paragraph says fixed rather than recorded.
…t be judged (#402) * docs: an HRX test with no device fails, and 77 is only for what cannot be judged The convention #400 applied to thirteen files had no home in prose, and the README's "Backend test suites" section only covers the hrx-*-test family, so test-hrx-* was not documented at all. Two sentences a reader needs: an HRX-enabled build that finds no HRX device fails those rows rather than skipping them, and do not "fix" a red row back to return 77 -- on ctest 4.2.3 the skip return code wins over FAIL_REGULAR_EXPRESSION and the row reports ***Skipped with 100% tests passed. * docs: the device table leads with the outcome, and the model skip gets its own row The CHANGES at 205ad0d: the heading said "fails, it does not skip" while both cells opened with "prints test skipped", so a reader who scanned the table took away the opposite of the rule. Each cell now opens with the ctest outcome -- Failed / Skipped -- and puts the message second, because the message is what the binary prints while the outcome is what the row reports. The two mechanisms stay distinct: one family returns 0 and is made red by FAIL_REGULAR_EXPRESSION, the other returns 1 and is red on the exit code alone. Adds the model-skip row (77) so the heading's second half -- "77 is only for what cannot be judged" -- has a row that shows it, and one sentence on which side to re-check if ctest's precedence ever changes: the hrx-*-test family depends on it, test-hrx-* does not. * docs: name the two mechanisms in the device table, so a row cannot be simplified back The 13:02 review asked for the exit code and the ctest outcome rather than the printed line; the table now leads with the outcome and carries the model-missing row as Skipped (77). This adds where each outcome comes from: the `hrx-*-test` family returns 0 from tests/testing_suite.h and is caught by the FAIL_REGULAR_EXPRESSION registered in tests/CMakeLists.txt, while the `test-hrx-*` family returns 1 at its own call site. Verified against both sources at this head. * hrx README: name the one test-hrx-* row that still skips on a missing device Eight of the nine test-hrx-* rows report a missing device as Failed (return 1 plus FAIL_REGULAR_EXPRESSION). The ninth does not: test-hrx-external-window-view returns skipped ? 77 : 0, prints 'no HRX0 device, the value check is skipped', and its ctest row carries no FAIL_REGULAR_EXPRESSION -- so on a device-less runner it reports Skipped/green under a table that says Failed. Named rather than fixed here: the behaviour is a two-line change in another file and this commit is prose. * hrx README: the device table's first row described a mechanism the code does not have The row said `hrx-*-test` "has no device guard at all" and "prints no `test skipped: no device available for <suite>`". Both halves are false, and a table row is the wrong place for either: the string is in the SHARED harness, so a grep of `tests/*.cpp` cannot see it. tests/testing_suite.h:255-257 if (test_case->requires_device && !has_device) { fprintf(stderr, "test skipped: no device available for %s\n", config.suite_name.c_str()); return 0; } tests/CMakeLists.txt:695 set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test FAIL_REGULAR_EXPRESSION "test skipped: no device available") So the two families differ in HOW the report becomes a failure, not in whether it does: hrx-*-test the suite returns 0 and the registered regex is what fails the row (mechanism: regex) test-hrx-* the file returns 1 itself; three rows also carry the regex (mechanism: exit code) That is what the `convention` column now says, one word per row, and the intro states it once: both report the same way by design, the column says how each makes that report fail rather than skip. Also removed a broken continuation row (`| (removed row) | | | FAIL_REGULAR_EXPRESSION` …) that a half-applied edit left in the table, and kept the paragraph naming `test-hrx-external-window-view` - the one row that still returns 77 on a missing device (`skipped ? 77 : 0`, no regex registered) and therefore still skips, which is true and is the kind of exception a table should name. Verified, not read: testing_suite.h:255-257 prints and returns 0; CMakeLists.txt:695 registers the regex for those four rows; 14 of 14 test-hrx-* files print the message and return 1; three rows carry the regex; external-window-view has no regex. * tests: the external-window-view row fails on a missing device instead of reporting a skip The README's exception paragraph named this row: it returned 77 from its no-device branch, so ctest reported it Skipped/green and the host cases it had just run were discarded -- the same defect #400 fixed for twelve siblings, in the one file that was left alone because it "still runs its host cases". Running them and then reporting a skip is not the same as reporting their result. It now prints "test skipped: no device available" and returns 1, with the group regex on the row as belt and braces. The README paragraph says fixed rather than recorded. * tests: hrx-backend-test prints the family's no-device line, so its row fails with no device as the README says hrx-backend-test has its own device half rather than testing_suite.h's, and its no-device branch printed "device: skipped, no HRX devices available". The row's FAIL_REGULAR_EXPRESSION is "test skipped: no device available" (tests/CMakeLists.txt:695), which that line does not contain, and the branch returns 0 when no case names were given. So the row passed with no HRX device, and the README table's first row ("Failed") was false for one of the three rows it names (review at 15:14Z). It now prints "test skipped: no device available for hrx-backend-test (device half; the host cases above ran)". The registered regex matches it, so the row fails like hrx-buffer-test and hrx-loom-jit-test. The README row now says how each of the three makes the report, and the exception paragraph names this second one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: pi-lane <pi@local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
test-hrx-loader-mapped(from engine#2500's guard, merged as llama.cpp#394) has two skips and returned 77 for both. ctest'sFAIL_REGULAR_EXPRESSION— the mechanism #375 uses to stop a silently-skipping HRX row from reporting green — cannot see the output when the exit code isSKIP_RETURN_CODE.Measured on this box, ctest 4.2.3, four shapes:
test-hrx-ops, why #375 works)So the row was green on any runner without an HRX device, and "added to the no-device group" would not have changed that.
Change
test skipped: no device available. A device runner that cannot open an HRX device now fails the row.argv[1]orLLAMACPP_TEST_MODELFILE): a CI runner with no model legitimately cannot judge this test, and a skip that says why is the honest outcome there.Two lines of behaviour plus the reason in a comment.
testing_suite.h:255prints the same string and returns 0 for its device skip, which is the case #375's regex was written for; the exit code is what this row needs because it does not use that suite.Verified
g++ -fsyntax-only -std=c++17ontests/test-hrx-loader-mapped.cppwith the tree's include paths: clean.add_testper shape,SKIP_RETURN_CODE 77andFAIL_REGULAR_EXPRESSIONset exactly as intests/CMakeLists.txt).hrx_DIR/loomc_DIR), so the device path is not exercised on hardware. The mechanism it targets is the table above; the row's own build is the samellama_build_and_testas before.Still open, and outside this PR
Nothing in either repo passes this row a model (
search/codeforLLAMACPP_TEST_MODELFILEis 0 in1bit-MONSTER/engineand 0 in1bit-MONSTER/llama.cpp;scripts/hw-gate.shnever invokes ctest), so its measured result today is still "not measured". That needs a runner that has the gate's models on disk — engine-side, and a different file.