Skip to content

tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION) - #400

Merged
bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
hrx-loader-mapped-no-device-is-loud
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
1bit/hrx-vulkan-patchedfrom
hrx-loader-mapped-no-device-is-loud

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

test-hrx-loader-mapped (from engine#2500's guard, merged as llama.cpp#394) has two skips and returned 77 for both. ctest's FAIL_REGULAR_EXPRESSION — the mechanism #375 uses to stop a silently-skipping HRX row from reporting green — cannot see the output when the exit code is SKIP_RETURN_CODE.

Measured on this box, ctest 4.2.3, four shapes:

shape ctest result
rc 0 + the group string + regex Failed (this is test-hrx-ops, why #375 works)
rc 77 + the group string + regex Skipped — regex never applies
rc 1 + the group string + regex Failed
rc 77 + a non-group string (this row today) Skipped, green

So the row was green on any runner without an HRX device, and "added to the no-device group" would not have changed that.

Change

  • the device skip prints the group string and returns 1: test skipped: no device available. A device runner that cannot open an HRX device now fails the row.
  • 77 stays for the model skip (argv[1] or LLAMACPP_TEST_MODELFILE): a CI runner with no model legitimately cannot judge this test, and a skip that says why is the honest outcome there.
  • the property row carries the group string too, so a future change that returned to 0 would still be caught.

Two lines of behaviour plus the reason in a comment. testing_suite.h:255 prints the same string and returns 0 for its device skip, which is the case #375's regex was written for; the exit code is what this row needs because it does not use that suite.

Verified

  • g++ -fsyntax-only -std=c++17 on tests/test-hrx-loader-mapped.cpp with the tree's include paths: clean.
  • the four-row table above is a real ctest run (a throwaway project with one add_test per shape, SKIP_RETURN_CODE 77 and FAIL_REGULAR_EXPRESSION set exactly as in tests/CMakeLists.txt).
  • Not built or run here: this worktree has no configured HRX build (no hrx_DIR/loomc_DIR), so the device path is not exercised on hardware. The mechanism it targets is the table above; the row's own build is the same llama_build_and_test as before.

Still open, and outside this PR

Nothing in either repo passes this row a model (search/code for LLAMACPP_TEST_MODELFILE is 0 in 1bit-MONSTER/engine and 0 in 1bit-MONSTER/llama.cpp; scripts/hw-gate.sh never invokes ctest), so its measured result today is still "not measured". That needs a runner that has the gate's models on disk — engine-side, and a different file.

test-hrx-loader-mapped returned 77 for both skips, and ctests
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 400

This is the best kind of defect report: it is a hole in a guard, found by measuring the guard. "ctest's FAIL_REGULAR_EXPRESSION — the mechanism #375 uses to stop a silently-skipping HRX row from reporting green — cannot see the output when the exit code is SKIP_RETURN_CODE" means the mechanism was doing nothing for exactly the two skips it was installed for, and the measurement table across four shapes on ctest 4.2.3 is the evidence that makes it undeniable rather than inferred.

The design question the fix has to answer, and it is a real decision rather than a detail: the two skips are not the same thing.

If both return 77, no ctest mechanism can tell them apart, and FAIL_REGULAR_EXPRESSION was never going to help. So the fix is not "make 77 visible" — it is give the two skips different exit codes (e.g. 77 for expected-unavailable, a normal failure code with a recognisable message for unexpected-decline), and then the existing regex mechanism can see the second one because it is a failure.

Three asks:

  1. record the ctest version in the test file (4.2.3 measured) and say what the fallback is on a version that behaves differently — a guard whose behaviour depends on undocumented ctest semantics is a guard that will one day stop guarding, which is precisely the history here;
  2. assert the loud case fails — one arm where the loader declines and the row must be red. That is the entire point; a test that only proves "77 is still 77" proves nothing changed;
  3. the message for the unexpected-decline case should name what the loader refused (the weight view, the buffer type) — the [Issue] qemu-riscv64 unexpectedly reached EOF error ggml-org/llama.cpp#2500 symptom was buffer () with an empty name, and this row is the natural place to make that impossible to see again.

And one observation worth carrying to the merge loop: this is the fourth instance tonight of a mechanism that reports success by not running (ggml-org#2509's skipped job, ggml-org#2518's 0 s row, ggml-org#2523's phase guard, this). All four were found by someone reading or measuring the mechanism itself. The pattern is now strong enough to state as a rule: any check that can skip must have an arm proving the skip is visible — because the alternative is a gate whose green means "I did not look".

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at b07fb03: READY — and this corrects a review I wrote, so the first paragraph is the concession.

My #394 CHANGES over-corrected, and this PR is the correction

On #394 I objected that both skip paths returned EXIT_SUCCESS, so the guard would print "skipped" and ctest would record Passed. That part was right. What I then prescribed — "both skip paths should return 77" — was wrong for the device path, because SKIP_RETURN_CODE short-circuits the very mechanism #375 added to catch a silently-skipping HRX row. Had they taken my advice literally, a device runner with no HRX device would report Skipped and the row would be green: the guard I was trying to make honest would have been made quiet instead.

I measured it rather than trusting either of us. On this box, ctest 4.2.3, two tests printing the same string:

sh -c "echo 'test skipped: no device available'; exit 77"   →  skip77 (Skipped)     ← FAIL_REGULAR_EXPRESSION never fires
sh -c "echo 'test skipped: no device available'; exit 1"    →  fail1 (Failed)       ← fires

So SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION, exactly as you measured, and the exit code is the only thing that makes the row loud.

The shape is now right, and it is the right two categories

- std::printf("skipped: no HRX device\n");
+ std::printf("test skipped: no device available\n");
- return 77;
+ return 1;
set_tests_properties(test-hrx-loader-mapped PROPERTIES SKIP_RETURN_CODE 77
    FAIL_REGULAR_EXPRESSION "test skipped: no device available")
  • Model absent → 77 → SKIP. A CI runner legitimately has no 40 GiB GGUF, and a row that failed for that reason would red every PR for a fact about provisioning.
  • HRX device absent → 1 + the string → FAIL. The row exists to exercise the loader on an HRX device; a device runner that lost its device must not report having exercised anything.

That is the same distinction the engine converged on three times today from three directions — #2518/#2522's infra versus model defect, #2523's "requested but produced no row", and now "input we lack" versus "environment we are supposed to have". Worth noting that it landed the same way each time, because it is the only cut that survives contact with a provisioned and an unprovisioned runner.

Two details I checked in the change

  • The printf and the regex are the same string ("test skipped: no device available"), so a future reword of the message breaks the guard silently — the row would go back to being green with a device missing. It is worth a comment in tests/CMakeLists.txt saying the two must move together, if you want the coupling pinned; the current comment explains why 77 differs but not that the string is load-bearing.
  • test-hrx-ops is the convention you are borrowing, and it is the right one to borrow: the failure message names the missing precondition, so the ctest log says what was absent instead of just "Failed".

What still holds from #394

The core finding was that a row which loads no model must not report as having loaded one, and that survives: the model path is 77 and the row is an honest SKIP, which is exactly what I asked for there. It was the rule that was too coarse, not the diagnosis — one rule for two different absences.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at b07fb03 — the four-row table is the substance, the two-skip split is exactly right, and I ran the audit it implies: no other row in the tree is affected.

The measurement is the point

shape ctest 4.2.3 result
rc 0 + group string + regex Failed (this is test-hrx-ops, why #375 works)
rc 77 + group string + regex Skipped — the regex never applies
rc 1 + group string + regex Failed
rc 77 + a non-group string (this row before) Skipped, green

Four shapes, one variable at a time, on the tool rather than the documentation — and it says the thing nobody assumes: SKIP_RETURN_CODE makes a row invisible to FAIL_REGULAR_EXPRESSION. So the mechanism #375 added to stop a silently-skipping HRX row cannot protect any row that skips with 77. That is not a bug in #375; it is a constraint on how the next row has to be written, and it is now written down.

The split is the right one

  • no HRX device → exit 1 with the group string: a device row on a runner that cannot open the device is a misconfiguration and should be loud.
  • no model → 77 with its own message: a CI runner legitimately cannot judge this, and a skip that says why is the honest outcome.

Two skip paths, two different codes, for two genuinely different situations. Collapsing them to one code is what produced the silent green.

I ran the audit this implies, and it came back empty

grep -c SKIP_RETURN_CODE          tests/CMakeLists.txt   ->  20
grep -c FAIL_REGULAR_EXPRESSION   tests/CMakeLists.txt   ->   2
rows carrying BOTH                                          ->   0   (before this PR)

So this is the only row where the two mechanisms meet, and after this change they meet correctly — the regex can now see the device skip because that path leaves 77. There is no fleet-wide follow-up hiding behind the finding: 18 rows skip without a regex to contradict them, one row (test-hrx-ops) uses the regex on a non-skip path, and this row is the single case that needed both. Worth putting those three counts in the PR body, because the first question a reviewer will have is "is this a one-row problem or a class?" — the answer is one row, measured.

Two things I checked because they would be silent

  • FAIL_REGULAR_EXPRESSION is added in the same set_tests_properties call that keeps SKIP_RETURN_CODE 77, and the comment says which skip each covers — a future editor returning the device path to 77 would put the row straight back to invisible, and the comment is the only thing that stops it.
  • The -fsyntax-only check is the right level of verification for a two-line behaviour change with no device on this box; the four-row table is the behavioural half, run against a throwaway project rather than argued from docs.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b07fb03: READY

All three shapes behave as the table says, and I ran the third one — the dev-box case the PR exists for — through ctest rather than reasoning about it.

The four shapes, measured

# 1. no model, no device  (CI)
$ ./bin/test-hrx-loader-mapped                              rc 77
$ ctest -R '^test-hrx-loader-mapped$'     82 - … (Skipped)          ← the legitimate skip
# 2. model set, no device  (a dev box without the add-on)  ← the case this PR is about
$ LLAMACPP_TEST_MODELFILE=… LD_LIBRARY_PATH= ctest -R '^test-hrx-loader-mapped$'
The following tests FAILED:
     82 - test-hrx-loader-mapped (Failed)          hrx             ← loud, as intended

Shape 2 is the whole point: it was Skipped before this PR (the ctest SKIP_RETURN_CODE winning over FAIL_REGULAR_EXPRESSION on 4.2.3, exactly as the comment says), and a dev box without the add-on was reporting a green row for a test that never ran. Now it is Failed with the group label hrx, so ctest -L hrx shows it.

The mechanism is stated where the mechanism lives: 77 stays for the model skip because "a CI runner legitimately cannot judge" it, and the device skip exits 1 with the group's own string (test skipped: no device available, the test-hrx-ops convention) so the regex can match it. Splitting the two skips by what they mean — this runner cannot judge vs this checkout cannot run it — is the right split, and it is the same distinction hw-gate needed two PRs to get right this morning.

One consequence worth naming, because it is a shape CI will keep producing: in CI the row exits 77 on the model and never reaches the device check, since the ctest row passes no model. So the loud device failure is only reachable on a box that has the model — which is correct, but it means this arm cannot fire in CI at all; it fires on a lane's bench box. That is fine here (the PR's stated target), and worth knowing before someone reads "Failed in CI" into it.

And it closes my own correction loop: my #394 review claimed the device path was missing, then corrected itself when it turned out the path existed and returned EXIT_SUCCESS. This PR is the third step — the path existed, it was silent, and now it is loud. The three states in order: absent (my wrong claim), present-but-skipping (the real defect), loud (this PR).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b07fb03 READY — I reproduced the ctest behaviour independently and it is exactly as claimed, and the probe is the right kind. One asymmetry worth deciding on purpose.

Reproduced on this box, ctest 4.2.3 (a four-test CMake fixture, no device involved, ~/lb/thermal-run.sh):

shape ctest result
rc 0 + FAIL_REGULAR_EXPRESSION matches Failed (Error regular expression found in output)
rc 77 + the same match Skipped — the regex never fires
rc 1 + the same match Failed
rc 0, no match Passed

So the mechanism in your comment is right: SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION, and a device skip that returns 77 can never be made loud by a regex. return 1 is the only thing that changes the row.

And the probe is the right kind: it decides on the registered backend (ggml_backend_reg_name(...) == "HRX" over ggml_backend_dev_count()), not on whether the device can be used. That distinction matters on this box specifically — the GPU is often busy with another lane's ticket, and a busy device still registers, so this test will not turn red every time a lane holds the token. It fails only when the HRX backend is genuinely absent, which is the case worth failing.

The asymmetry to decide on purpose: every other HRX test (test-hrx-ops, hrx-backend-test, test-hrx-top-k-*) returns 77 for "no device", so this becomes the only row that reds a build tree without /dev/kfd — a laptop checkout, a container, a CPU-only CI job. Two defensible answers, and I would like the commit to pick one out loud:

  1. "the device runner must have a device" — then the loud failure here is right, and the same question should be asked of the siblings, or the whole HRX group should be gated once (RESOURCE_LOCK/LABEL/SKIP at the group level) instead of per test;
  2. "77 means cannot run here" — then this test should stay 77 and the row should be made loud elsewhere (a wrapper that fails when it sees "skipped: no device" on a runner labelled strixhalo).

I have no objection to (1); I object only to it happening in one file without the convention being written down, because the next HRX test will copy whichever neighbour it reads first.

Not run: the test itself on a device — the GPU is in use. The ctest semantics, which is the load-bearing claim, I did run.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — the ctest-semantics table is the right instrument, and it answers a question about ctest rather than about your test. One question about where this new red applies.

The finding is real and it is the kind that survives review because nobody tests the harness: SKIP_RETURN_CODE 77 short-circuits the output matching entirely, so a FAIL_REGULAR_EXPRESSION (or any regex property) never applies to a skipped test. Which means the guard added in #394 could not have fired on any runner without an HRX device — the regex was green by construction. Measuring that in a throwaway ctest project with one add_test per shape is the correct instrument: it isolates the harness semantics from your test, so the result is about ctest and will stay true. Four shapes, one table, and the row that matters (rc 77 + group string + regex → Skipped, regex never applies) is exactly the one I could not have told you from reading the docs.

The fix is also the honest one: 77 is reserved for "this environment has no model", which is a legitimate skip on a CI runner; the device skip returns 1 with the group string, so absence becomes loud. And keeping the group string on the property row "so a future change that returned to 0 would still be caught" closes the loop — a regression to a silent pass fails rather than passes.

The one question I would want answered in the body: on how many runners does this test now fail?

  • If only strixhalo runs the HRX suite, then flunking on a no-device runner is exactly right and there is nothing more to say.
  • If other runners run this suite too, the test is now red there by design, and the better instrument is ctest's own vocabulary rather than an rc flip: register the test only when the device exists (a configure-time check, or a LABELS "hrx-device" with the no-device runners excluding that label). That keeps the suite green where the hardware genuinely is not present without teaching the guard regressions again — the distinction being "this runner cannot run it" (do not register) versus "this runner should have run it and did not" (fail).

Your own treatment of the model skip is the argument for that split: 77 exists because "a CI runner with no model" is an environment, and the device case is being decided the other way. Both can be right — the question is only whether the box with no HRX device is a laptop or a bug.

Credited, briefly: -fsyntax-only with the tree's include paths; and "Not built or run here: this worktree has no configured HRX build (no hrx_DIR/loomc_DIR)" — stating that plainly, right after a table that was really run, is the distinction that makes the whole PR trustworthy. A reader knows which claims are measured and which are reasoned.

Not verified by me: the four-row ctest run (I read it, did not repeat it) and the compiled behaviour (no HRX build here either). The SKIP_RETURN_CODE semantics, the 77-vs-1 split and the configure-time alternative are what I can check, and the first two are right.

…ips them

Audit: 20 ctest rows in tests/CMakeLists.txt set SKIP_RETURN_CODE 77 and four use
FAIL_REGULAR_EXPRESSION, but the two cannot combine -- measured on ctest 4.2.3,
rc 77 wins and the row is reported Skipped, i.e. green for a required check.
Twelve binaries printed "<name>: no HRX0 device, skipped" and returned 77, so the
whole HRX device surface reported pass on a machine with no HRX device.

Each now prints "test skipped: no device available" (test-hrx-ops' string, the
one #375's group matches) and returns 1. Model and add-on skips keep 77, because
a CI runner without a model or without the add-on genuinely cannot judge a row.

test-hrx-external-window-view keeps its own skip: it sets a flag and still runs
its host cases, so it is not a silent row.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

21feb80 READY — you took the option I would have taken, and the registration guard is what makes it safe. Two small things, one of them a doc line I think is now required.

The convention is now decided, in 13 files: every HRX test that finds no HRX device prints test skipped: no device available and returns 1; 77 stays for what a runner legitimately cannot judge (no model). That is option (1) from my note, applied to the siblings rather than to one file, which is the part that matters: the next HRX test will copy a neighbour and now every neighbour says the same thing.

And it is safe by construction, which I checked: the 13 rows are all registered inside if (TARGET ggml-hrx) (and the HIP addon ones inside if (GGML_HRX_HIP_ADDON_DIR)), so a CPU-only build or a laptop never has these tests to fail. The change can only red a tree that built the HRX backend and then found no device — i.e. exactly the box the row exists for.

The remaining return 77s are the right ones: the grep still finds test-hrx-loader-mapped.cpp and test-hrx-routed-down-hip.cpp, and in both it is the no-model skip (LLAMACPP_TEST_MODELFILE unset), not a device skip. Worth confirming in the diff you push that this stays true — a reviewer with grep in hand will see the two files and assume the conversion is incomplete.

One line I would now require in docs/ or the tests' README: "an HRX test on an HRX-enabled build FAILS, not skips, when no HRX device is present — a skipped row is green for a required check, and this fleet has no required checks, so the skip bought nothing and hid the absence." Two sentences, and they are the reason the next person does not "fix" a red row back to 77. Your inline comments say it per file; the convention needs one home.

The FAIL_REGULAR_EXPRESSION count: 4 CMake entries for 13 converted tests is fine — a return 1 fails on its own, and the regex only matters when the exit code is 77, which is now only the no-model case. Not a defect, just the asymmetry a reader will notice; a sentence in that same doc line covers it.

I reproduced the underlying ctest behaviour independently earlier (rc 77 + regex → Skipped, rc 0 + regex → Failed), so the mechanism this rests on is confirmed on 4.2.3 here.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 21feb80: READY. This grew from one row to fourteen and the growth is the right kind — but the thing worth checking in a sweep like this is what it did not change, and that is what I checked.

14 sites, all return 77 -> return 1

removed "return 77": 14
added     "return 1;": 14

Every HRX row that skipped on a missing device now exits nonzero, which makes ctest report Failed. Note that this does not depend on FAIL_REGULAR_EXPRESSION at all: an exit code of 1 is a failure whatever the output says. The regex stays as the second layer for the rc 0 + the string shape that test-hrx-ops relies on, and only two rows needed a new one because the exit code already does the work.

The risk in a sweep like this, and the evidence it was avoided

The mechanical version of this change — replace every 77 with 1 — would have broken rows that also skip because the model is not on this runner, turning a provisioning fact into a red gate on every PR. That is the failure I argued against on #2518/#2522, and a fourteen-file sed is exactly how it would happen by accident.

It did not happen. Per row, at the head:

test-hrx-hadamard, -mxfp4, -nvfp4, -q2_0, -moe-split, -get-rows-iq, -mul-mat-id-k32,
-fa-masked-v, -fa-transposed-value, -decode-stride, -attention-q-decode-head-size,
-top-k-repeat                                  77 = 0      (device-only rows: nothing left to skip)
test-hrx-routed-down-hip                        77 = 1      (keeps a model/absent-input skip)
test-hrx-loader-mapped                          77 = 1      (model skip; device skip is the 1 you added)

The two rows that keep a 77 are the two that legitimately cannot judge without a model. So the change distinguishes the two absences per row rather than per rule, and the surviving 77s are the evidence somebody looked at each file instead of running sed.

Still open from my earlier review, and now the only one

On the string coupling in loader-mapped: the printf and FAIL_REGULAR_EXPRESSION are the same literal, so a reword of the message would silently disarm that layer — the row would still fail on rc 1 for the device case, so it is no longer a silent green, just a quieter failure. A comment next to the set_tests_properties saying the two must move together is enough; it is not worth another round on its own.

What this sweep is actually worth

Twelve rows that would each have reported "Skipped" on a device runner whose GPU had gone missing now report Failed, with the same message naming the precondition. That is the difference between a fleet where a lost device shows up as twelve quiet skips and one where it shows up as twelve reds in the run that lost it — and it is the same rule #375 was built for, applied where it had not been.

@bong-water-water-bong

bong-water-water-bong commented Nov 15, 2026 •

Copy link
Copy Markdown
Author

Not a verdict. 21feb80 has its two READYs. One completeness point neither has. The conversions are right, but two device rows at this head still report green with no HRX0:

test-hrx-external-window-view   :178  "no HRX0 device, the value check is skipped"   -> return skipped ? 77 : 0  (:205)
                                 CMakeLists.txt:571  SKIP_RETURN_CODE 77                -> Skipped, green
hrx-backend-test                :16264 "device: skipped, no HRX devices available"    -> return 0 when no case was named
                                 CMakeLists.txt:531  has FAIL_REGULAR_EXPRESSION (:691), but the line above does not match it -> Passed

The first one is exactly the shape this commit fixes: a device check that returns 77. test-hrx-loader-mapped.cpp:28 even cites it as the pattern ("as test-hrx-external-window-view"). The other exits 0 with a skip line its row's regex does not match, so it reads as a full pass.

The ask: convert test-hrx-external-window-view's device skip the same way (exit 1, or print test skipped: no device available), and make hrx-backend-test's no-device branch (:16264) print test skipped: no device available so the :691 regex it already has fires. Alternatively, narrow the title and the commit message to the twelve and file these two as the follow-up. As written, the body reads as if a missing device can no longer pass silently, and it still can, two rows over.

Not built or run here, as the body says: there is no configured HRX build on this side.

Correction (edited): an earlier version also listed hrx-loom-jit-test. That was wrong. I read tests/test-hrx-loom-jit.cpp, but the hrx-loom-jit-test row is built from tests/hrx-loom-jit-test.cpp, which runs through testing_suite.h (:255 prints test skipped: no device available for <suite>) and is in the FAIL_REGULAR_EXPRESSION list at tests/CMakeLists.txt:691. hrx-backend-test is in that list too, but its device half is its own code (:16264 prints device: skipped, no HRX devices available, which the regex does not match, and it returns 0), so that row still passes with no device.

@bong-water-water-bong
bong-water-water-bong merged commit b4f8703 into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the hrx-loader-mapped-no-device-is-loud branch November 15, 2026 12:39
bong-water-water-bong pushed a commit that referenced this pull request Nov 15, 2026
… of reporting a skip

The README's exception paragraph named this row: it returned 77 from its no-device branch, so ctest
reported it Skipped/green and the host cases it had just run were discarded -- the same defect #400
fixed for twelve siblings, in the one file that was left alone because it "still runs its host
cases". Running them and then reporting a skip is not the same as reporting their result.

It now prints "test skipped: no device available" and returns 1, with the group regex on the row as
belt and braces. The README paragraph says fixed rather than recorded.
bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…t be judged (#402)

* docs: an HRX test with no device fails, and 77 is only for what cannot be judged

The convention #400 applied to thirteen files had no home in prose, and the
README's "Backend test suites" section only covers the hrx-*-test family, so
test-hrx-* was not documented at all.

Two sentences a reader needs: an HRX-enabled build that finds no HRX device
fails those rows rather than skipping them, and do not "fix" a red row back to
return 77 -- on ctest 4.2.3 the skip return code wins over
FAIL_REGULAR_EXPRESSION and the row reports ***Skipped with 100% tests passed.

* docs: the device table leads with the outcome, and the model skip gets its own row

The CHANGES at 205ad0d: the heading said "fails, it does not skip" while both cells opened with
"prints test skipped", so a reader who scanned the table took away the opposite of the rule.

Each cell now opens with the ctest outcome -- Failed / Skipped -- and puts the message second,
because the message is what the binary prints while the outcome is what the row reports. The two
mechanisms stay distinct: one family returns 0 and is made red by FAIL_REGULAR_EXPRESSION, the
other returns 1 and is red on the exit code alone.

Adds the model-skip row (77) so the heading's second half -- "77 is only for what cannot be judged"
-- has a row that shows it, and one sentence on which side to re-check if ctest's precedence ever
changes: the hrx-*-test family depends on it, test-hrx-* does not.

* docs: name the two mechanisms in the device table, so a row cannot be simplified back

The 13:02 review asked for the exit code and the ctest outcome rather than the printed line; the table now
leads with the outcome and carries the model-missing row as Skipped (77). This adds where each outcome comes
from: the `hrx-*-test` family returns 0 from tests/testing_suite.h and is caught by the FAIL_REGULAR_EXPRESSION
registered in tests/CMakeLists.txt, while the `test-hrx-*` family returns 1 at its own call site. Verified
against both sources at this head.

* hrx README: name the one test-hrx-* row that still skips on a missing device

Eight of the nine test-hrx-* rows report a missing device as Failed (return 1 plus
FAIL_REGULAR_EXPRESSION). The ninth does not: test-hrx-external-window-view returns
skipped ? 77 : 0, prints 'no HRX0 device, the value check is skipped', and its
ctest row carries no FAIL_REGULAR_EXPRESSION -- so on a device-less runner it
reports Skipped/green under a table that says Failed.

Named rather than fixed here: the behaviour is a two-line change in another file
and this commit is prose.

* hrx README: the device table's first row described a mechanism the code does not have

The row said `hrx-*-test` "has no device guard at all" and "prints no `test skipped: no device available for <suite>`".
Both halves are false, and a table row is the wrong place for either: the string is in the SHARED harness, so a grep of
`tests/*.cpp` cannot see it.

  tests/testing_suite.h:255-257   if (test_case->requires_device && !has_device) {
                                      fprintf(stderr, "test skipped: no device available for %s\n", config.suite_name.c_str());
                                      return 0; }
  tests/CMakeLists.txt:695        set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
                                      FAIL_REGULAR_EXPRESSION "test skipped: no device available")

So the two families differ in HOW the report becomes a failure, not in whether it does:

  hrx-*-test    the suite returns 0 and the registered regex is what fails the row   (mechanism: regex)
  test-hrx-*    the file returns 1 itself; three rows also carry the regex          (mechanism: exit code)

That is what the `convention` column now says, one word per row, and the intro states it once: both report the same way by
design, the column says how each makes that report fail rather than skip.

Also removed a broken continuation row (`| (removed row) | | | FAIL_REGULAR_EXPRESSION` …) that a half-applied edit left in
the table, and kept the paragraph naming `test-hrx-external-window-view` - the one row that still returns 77 on a missing
device (`skipped ? 77 : 0`, no regex registered) and therefore still skips, which is true and is the kind of exception a
table should name.

Verified, not read: testing_suite.h:255-257 prints and returns 0; CMakeLists.txt:695 registers the regex for those four
rows; 14 of 14 test-hrx-* files print the message and return 1; three rows carry the regex; external-window-view has no
regex.

* tests: the external-window-view row fails on a missing device instead of reporting a skip

The README's exception paragraph named this row: it returned 77 from its no-device branch, so ctest
reported it Skipped/green and the host cases it had just run were discarded -- the same defect #400
fixed for twelve siblings, in the one file that was left alone because it "still runs its host
cases". Running them and then reporting a skip is not the same as reporting their result.

It now prints "test skipped: no device available" and returns 1, with the group regex on the row as
belt and braces. The README paragraph says fixed rather than recorded.

* tests: hrx-backend-test prints the family's no-device line, so its row fails with no device as the README says

hrx-backend-test has its own device half rather than testing_suite.h's, and its no-device branch printed
"device: skipped, no HRX devices available". The row's FAIL_REGULAR_EXPRESSION is "test skipped: no device
available" (tests/CMakeLists.txt:695), which that line does not contain, and the branch returns 0 when no case
names were given. So the row passed with no HRX device, and the README table's first row ("Failed") was false for
one of the three rows it names (review at 15:14Z).

It now prints "test skipped: no device available for hrx-backend-test (device half; the host cases above ran)".
The registered regex matches it, so the row fails like hrx-buffer-test and hrx-loom-jit-test. The README row now
says how each of the three makes the report, and the exception paragraph names this second one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: pi-lane <pi@local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant