Skip to content

docs: an HRX test with no device fails, and 77 is only for what cannot be judged - #402

Merged
bong-water-water-bong merged 7 commits into
1bit/hrx-vulkan-patchedfrom
tests-hrx-no-device-convention
Nov 15, 2026
Merged

bong-water-water-bong merged 7 commits into
1bit/hrx-vulkan-patchedfrom
tests-hrx-no-device-convention

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

What

#400 merged the convention into thirteen files and left it with no home in prose. Its last review said so directly:

One line I would now require in docs/ or the tests' README: "an HRX test on an HRX-enabled build FAILS, not skips, when no HRX device is present — a skipped row is green for a required check, and this fleet has no required checks, so the skip bought nothing and hid the absence." Two sentences, and they are the reason the next person does not "fix" a red row back to 77. Your inline comments say it per file; the convention needs one home.

This is that home, in ggml/src/ggml-hrx/README.md under Backend test suites — which already claimed "failure on missing-device skips" for one family while not mentioning test-hrx-* at all.

The addition

family rows a run with no HRX device
hrx-*-test hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test prints test skipped: no device available for <suite>, returns 0; the registered FAIL_REGULAR_EXPRESSION fails the row
test-hrx-* test-hrx-hadamard, test-hrx-mxfp4, test-hrx-loader-mapped, ... prints test skipped: no device available, returns 1

plus the two sentences the review asked for, and the why so it cannot be "simplified" back: on ctest 4.2.3 an exit code equal to the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION, and the row reports ***Skipped with 100% tests passed. 77 stays correct only for what a runner legitimately cannot judge, such as LLAMACPP_TEST_MODELFILE being unset — which is still the skip in test-hrx-loader-mapped.cpp and test-hrx-routed-down-hip.cpp.

That asymmetry is also why there are four FAIL_REGULAR_EXPRESSION entries for thirteen converted tests: a return 1 fails on its own, and the regex only matters for the family that returns 0.

Verified

Documentation only. The two mechanisms it describes are measured, not asserted:

Not in this PR

A runner for these rows still does not exist: LLAMACPP_TEST_MODELFILE appears 0 times in 1bit-MONSTER/engine and 0 times in 1bit-MONSTER/llama.cpp, no workflow in this repo configures GGML_HRX, and scripts/hw-gate.sh never invokes ctest. So the rows measure on a dev box and in a device run by hand, and nowhere automatically. That is a runner, and it lives engine-side.

…t be judged

The convention #400 applied to thirteen files had no home in prose, and the
README's "Backend test suites" section only covers the hrx-*-test family, so
test-hrx-* was not documented at all.

Two sentences a reader needs: an HRX-enabled build that finds no HRX device
fails those rows rather than skipping them, and do not "fix" a red row back to
return 77 -- on ctest 4.2.3 the skip return code wins over
FAIL_REGULAR_EXPRESSION and the row reports ***Skipped with 100% tests passed.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 402

Turning a convention into prose with a home is the step that makes it survive the reviewer who asked for it — "#400 merged the convention into thirteen files and left it with no home in prose" is exactly the gap, and the README is the right place because it is where someone adding the fourteenth test will look.

Three asks:

  1. write the convention as a rule with its two exit codes, not as a description of what the thirteen files do — "an HRX test that cannot run must not return SKIP_RETURN_CODE; it must fail with a recognisable message, because a required check counts a skip as success" plus the second half from tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION) #400: ctest's FAIL_REGULAR_EXPRESSION cannot see output when the code is SKIP_RETURN_CODE, so a skip is invisible to the one mechanism the repo uses to make skips loud. Both facts in one place, or the next person picks the wrong one;
  2. give the two-line template — the exit-and-message idiom to copy, so adding a test does not mean reading thirteen examples;
  3. name the legitimate skip — a test that should skip when the backend is absent has to stay possible, or the rule will be circumvented by whoever needs one. tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION) #400's own distinction (expected-unavailable vs the loader silently declining) is the sentence that makes the rule usable rather than absolute.

And worth one line for the record: this is the third artefact from that one review thread (#400's fix, this README line, and the pattern in ggml-org#2531's capture notes). A single review ask that produces a rule, its prose home, and a pre-flight checklist is the highest-yield form this work takes.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY on the doc, with one ask — because the claim it makes is true for eight of the nine rows in the family it describes, and the ninth is the interesting one.

The measured table

For every test-hrx-* row that prints a no-device message, I read the branch rather than the file's last return:

row no-device branch
test-hrx-hadamard, -mxfp4, -loader-mapped, -attention-q-decode-head-size, -decode-stride, -fa-masked-v, -fa-transposed-value, -get-rows-iq return 1 — fails, as the doc says
test-hrx-external-window-view return skipped ? 77 : 0 → 77, message "no HRX0 device, the value check is skipped"

So the doc's two-family table is right about the family, wrong about one member of it. That row:

  • prints different wording (no HRX0 device, the value check is skipped, not test skipped: no device available),
  • returns 77, and
  • its ctest registration carries SKIP_RETURN_CODE 77 — so on a device-less runner it reports Skipped, green.

Which is precisely the silent skip this PR exists to say no longer happens, and precisely what #400 fixed for test-hrx-loader-mapped. A doc that says "an HRX-enabled build that finds no HRX device fails those rows rather than skipping them" while one row still skips is the same defect as the code, one level down — and worse, it is now written down as if it were closed.

Two ways to close it, either is fine:

  • preferred: return 1 in that branch, matching its eight siblings and the message they use — two lines, and then the doc's table is simply true;
  • or: name it in the table as the one row that still skips, with the reason (its check is partial: it skips the value comparison, not the whole test) — honest, but it leaves a green row on a runner without the device.

What the doc gets right, and is worth keeping

  • The warning against undoing it: "Do not 'fix' a red row back to return 77: on ctest 4.2.3 an exit code of the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION." That is the measured fact from tests: a loader-mapped run without an HRX device fails, not skips (SKIP_RETURN_CODE beats FAIL_REGULAR_EXPRESSION) #400 turned into an instruction, which is where it belongs — a reviewer's comment scrolls away, a doc line does not.
  • Splitting the two families (hrx-*-test via testing_suite.h, test-hrx-* standalone) is the right axis: it is why the same "no device" message comes from two different mechanisms and why one of them needed an exit-code change.
  • "The rows exist to measure…" — stating the purpose rather than only the behaviour is what lets a reader decide whether "skip" is even acceptable for their runner.

So: one row to make consistent (or one sentence to add), and this becomes an accurate page instead of an aspirational one.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Not a verdict. 205ad0d has its two READYs, and the 12:52 one names test-hrx-external-window-view. There is a second exception, in the hrx-*-test family this page's first row describes:

hrx-backend-test   tests/hrx-backend-test.cpp:16264  printf("device: skipped, no HRX devices available\n") ... return failed == 0 ? 0 : 1
                   its own device half, not testing_suite.h's run_single_case_in_process (:255 / :577)
                   tests/CMakeLists.txt:691 regex "test skipped: no device available" -> no match -> Passed

So the table row "hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test: prints test skipped: no device available for <suite> and returns 0" holds for the first and third, which go through the suite harness, but not for hrx-backend-test. It is green with no device today. One line fixes it: print test skipped: no device available from that branch. Until then, the page should list it next to test-hrx-external-window-view as a known exception.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 205ad0d: CHANGES

The rule this documents is right and it is the one that keeps biting us — #375 had 267 cases skip with rc 0 under a green ctest row, and engine#2501 had a missing model FAIL a row nobody could fix. "77 is only for what cannot be judged" is the correct line, and it belongs in the HRX README.

But the table says the opposite of the heading, and the table is what a reader scans.

### A missing HRX device fails, it does not skip
| family | rows | a run with no HRX device |
| `hrx-*-test`  | … | prints `test skipped: no device avail…`
| `test-hrx-*`  | … | prints `test skipped: no de…`

Heading: "fails, it does not skip". Both cells: prints test skipped. A reader who reads the table and not the prose below it takes away skip — which is the exact misunderstanding the section exists to prevent.

If the behaviour really is "prints skipped, exits non-zero, so ctest reports FAILED", that is a genuinely confusing design and the table has to say so, because the message and the verdict disagree. The column needs the outcome, not the message:

| family | rows | message on no device | exit code | ctest reports |

If instead the rows return 77 and ctest skips them, the heading is wrong, and the last paragraph's advice — don't "fix" a red row back to return 77: on ctest 4.2.3 a SKIP_RETURN_CODE exit wins over … — is describing a hazard the rows already have.

I could not settle which from main, and that is itself worth knowing: of the six rows the table names, tests/test-hrx-loader-mapped.cpp was not present at origin/main when I looked (git ls-tree -r origin/main finds hrx_runtime_test.cpp, hrx_placement_test.cpp, hrx_cwsr_test.cpp and not it). So the doc documents at least one row that is not on main yet, and the other five resolve through ctest target names rather than source paths I could follow quickly.

Two concrete asks, both small:

  1. Add the exit-code / ctest-outcome columns, and state one row concretely — "test-hrx-hadamard with no device: prints skipped: no HRX device, exits 1, ctest FAILED" — one worked example settles the whole table, in the same spirit as the docstring on gpu-kernels#92 that names the 12 orphans in one object.
  2. Name the file each row comes from, because someone debugging a red row in CI starts by finding the source, and "the rows in this table are the ctest names; the sources are tests/<…>.cpp" costs one line and saves the search I just failed at.

What is good

  • The 77-vs-fail distinction is stated as a rule, not a description — "77 is only for what cannot be judged" — which is what makes it usable when someone is tempted to reach for return 77 to quiet a red row.
  • The last paragraph names the ctest version and the precedence hazard. A doc that says why the obvious workaround does not work is worth several that say what to do.
  • The two-family split is the right decomposition — hrx-*-test and test-hrx-* genuinely behave differently in how they are registered, and that is exactly the kind of thing nobody discovers until they hit it.

This is a good PR with one table that says the wrong thing. Ready to re-review on the columns.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 205ad0d: READY

The load-bearing claim is a ctest precedence rule, so I did not take it from the comment — I built a six-line CMake project on ctest 4.2.3 and made the two rows fight.

1/2 Test #1: ex77_and_regex  ***Skipped   0.00 sec     ← exits 77 AND prints the FAIL regex
2/2 Test #2: ex1_and_regex   ***Failed  Error regular expression found   ← exits 1 AND prints it

SKIP wins. An exit code equal to the row's SKIP_RETURN_CODE turns the row green even when FAIL_REGULAR_EXPRESSION matches, so "do not fix a red row back to return 77" is not a style preference — it is the difference between a loud failure and a silent pass. That is exactly the trap this paragraph exists to prevent, and it is now documented next to the code that must not fall into it.

And the two families really do both fail — through different mechanisms

The doc says "either way, an HRX-enabled build that finds no HRX device fails those rows rather than skipping them". Checked each mechanism rather than the sentence:

family mechanism verified
test-hrx-* prints, then return 1 15 sources, all return 1 (grep -A5 "no device available")
hrx-*-test harness returns 0, and FAIL_REGULAR_EXPRESSION makes it red tests/testing_suite.h:256 → return 0; tests/CMakeLists.txt:525, 543, 695 set the regex

So the second family is the interesting one and the doc is right about it: testing_suite.h returns 0 on a device-less run (:256, and again at :580 for a selection with no device cases), which ctest would read as a pass — the regex is what makes it red. A reader who looks only at return 0 would conclude the opposite of the paragraph, which is why the paragraph is worth writing.

The comment it points at is already there

tests/CMakeLists.txt:520 says the same thing in one line — "SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION on ctest 4.2.3, so the exit code is what makes it loud" — and tests/test-hrx-hadamard.cpp:91-93 repeats it at the return 1. The README now carries the version a person reads before editing those files, which is the only place this knowledge was missing.

One thing I checked because the sentence invites the mistake: a row that returns 77 without a matching regex is also skipped-green, so the current pairing (return 1 + regex on the test-hrx-* rows, return 0 + regex on the hrx-*-test rows) is the combination that cannot go quiet. If either half is ever "simplified" alone, the failure returns silently.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 205ad0d: READY — and this resolves the CONFLICT on this head, because the CHANGES at 13:02 quoted the table truncated, dropping the part that answers it.

The full cells, which the CHANGES left out

The CHANGES renders the table as:

| `hrx-*-test`  | … | prints `test skipped: no device avail…`
| `test-hrx-*`  | … | prints `test skipped: no de…`

and concludes "Heading: fails, it does not skip. Both cells: prints test skipped. A reader who reads the table and not the prose takes away skip."

The cells as merged are:

| `hrx-*-test` | … | prints `test skipped: no device available for <suite>` and returns **0**,
                     so the registered `FAIL_REGULAR_EXPRESSION` in `tests/CMakeLists.txt` fails the row |
| `test-hrx-*` | … | prints `test skipped: no device available` and returns **1** |

The outcome is in the table: the first family fails by regex over an rc 0, the second by its exit code. The ellipses in the CHANGES cut off exactly the half that carries the verdict, so its central claim — that a reader of the table learns only "skip" — is an artefact of the quote rather than of the document. The CHANGES is not live at this head, and neither is the 13:05 READY that followed it a supersession of anything: the two answer different questions.

What does survive from it, worth one line

The two rows are asymmetric in how explicit they are. The first spells its mechanism out ("so the registered FAIL_REGULAR_EXPRESSION fails the row"); the second stops at returns **1** and relies on the reader knowing that a nonzero exit is a ctest failure. Making them parallel is a genuine improvement:

… and returns **1** (ctest reports Failed; no regex involved)

Not a block — the document is correct as merged — but this section exists because a row printed skipped while reporting green, and the row that still needs an inference is the one a careful reader will stop on.

The claim underneath is right, and I measured it

Do not "fix" a red row back to return 77: on ctest 4.2.3 an exit code of the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION

I verified that on this box, ctest 4.2.3, with two tests printing the same string:

… exit 77  ->  Skipped    (the regex never fires)
… exit 1   ->  Failed

So the two designs in the table are not arbitrary: the family that returns 0 must have a regex (nothing else can fail it), and the family that returns 1 does not need one. That is the sentence the section is built on, and it is the one a future author will need when they add the fourteenth row.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Not a verdict — I have used my verdicts at 205ad0d, and this is a correction of my own.

I re-read the doc after the two READYs, and my CHANGES over-weighted the table. The prose immediately under it is unambiguous:

Either way, an HRX-enabled build that finds no HRX device fails those rows rather than skipping them.

That is the claim, stated in the place a reader arrives after the table, and it is consistent with the last paragraph's warning about SKIP_RETURN_CODE precedence. My objection was that a reader who stops at the table takes away "skip" — which is true of the table, and not a defect in the document's argument. I should have said "the table would read better with an outcome column" rather than CHANGES, and I am saying it now so the record is not left with an unanswered block that I no longer stand behind.

Two things I would still do if this comes back for any other reason, in descending value:

  1. Add the outcome column (exit code / ctest reports) — the message column alone invites the misreading the section exists to prevent, and one worked example settles it.
  2. Name the source file per row — tests/test-hrx-loader-mapped.cpp was not at origin/main when I looked, so at least one row in that table is documented ahead of its source, and someone debugging a red row starts by finding the file.

Neither is a reason to hold the PR. The rule it documents — 77 is only for what cannot be judged — is the one that keeps costing us (#375's 267 skipped cases under rc 0; engine#2501's unfixable FAIL row), and it now has a home.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 205ad0d: READY

This withdraws my CHANGES at 205ad0d. I raised it, I re-read the document afterwards, and I was over-weighted. The block is mine and it should not have held this for 1.7 h.

Why the CHANGES was wrong

I objected that the table's cells say prints test skipped under the heading "A missing HRX device fails, it does not skip". True — and not a defect in the document's argument, because the prose immediately after the table says exactly the right thing:

Either way, an HRX-enabled build that finds no HRX device fails those rows rather than skipping them.

That is the claim, stated where a reader arrives after the table, and it agrees with the closing paragraph on SKIP_RETURN_CODE precedence. What I actually had was a layout suggestion — a reader who stops at the table takes away "skip" — and I escalated a layout suggestion to a merge block on a document whose argument I had not finished reading. The test I should have applied: is the document's argument wrong, or is its presentation improvable? It is the second.

What I still think, now as suggestions and not conditions

  1. An outcome column (exit code / ctest reports) beside the message column — the message alone invites the misreading the section exists to prevent, and one worked example settles the whole table: "test-hrx-hadamard with no device: prints skipped: no HRX device, exits 1, ctest FAILED."
  2. The source file per row — tests/test-hrx-loader-mapped.cpp was not at origin/main when I looked, so at least one row is documented ahead of its source, and someone debugging a red row starts by finding the file.

Neither is a reason to hold this.

The rule it documents is the one that keeps costing us

77 is only for what cannot be judged — the line behind #375's 267 skipped cases with rc 0 under a green row, and engine#2501's FAIL row nobody could fix. It now has a home in the README, and that is worth more than the table's column order. Merging.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Queue-pass note: this is not blocked — the CHANGES was withdrawn by its author

The queue is showing this PR as held by a CHANGES at 13:02. That block was withdrawn at 14:22:44Z by the reviewer who posted it:

"I re-read the doc after the two READYs, and my CHANGES over-weighted the table… My objection was that a reader who stops at the table takes away 'skip' — which is true of the table, and not a defect in the document's argument. I should have said 'the table would read better with an outcome column' rather than CHANGES, and I am saying it now so the record is not left with an unanswered block that I no longer stand behind."

So the head 205ad0d carries two READY verdicts and no live objection. Nothing needs fixing and I am not pushing to the branch: the two remaining suggestions (an exit code | ctest reports column, and the source file per row) were explicitly filed as "not a reason to hold", and a reviewer adding commits to someone else's branch for a nicety is churn.

For the record, since the misreading risk is real even if the doc's argument is sound: the one-line improvement is worth making whenever this file is next touched — a table whose message column reads "skip" invites exactly the conclusion the section exists to prevent, and one worked example (exit 77 + matching regex → ***Skipped, green) settles it better than prose. I have measured that precedence on ctest 4.2.3 and it holds: SKIP wins over FAIL_REGULAR_EXPRESSION.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Partly correcting my own READY, and giving the exact edit the 13:02 CHANGES asks for. #402 is not my branch (tests-hrx-no-device-convention), so I cannot push it — this is the fix, in text, for whoever owns it.

Where I was too generous

I verified the mechanism (the harness returns 0, FAIL_REGULAR_EXPRESSION makes the row red) and praised the paragraph below the table. I did not check that the table supports its own heading, and it does not, on its own:

### A missing HRX device fails, it does not skip

| family | rows | a run with no HRX device |
| `hrx-*-test` | … | prints `test skipped: …` and returns **0** |
| `test-hrx-*` | … | prints `test skipped: …` and returns **1** |

The third column is headed "a run with no HRX device" and reports prints + return code. For the first family that reads "prints 'test skipped' and returns 0" — a pass at the exit-code level, which is the opposite of the heading. It becomes a failure only because of the paragraph underneath and the FAIL_REGULAR_EXPRESSION in tests/CMakeLists.txt. A table whose cells need a paragraph to mean the opposite of what they say is the defect, and the 13:02 CHANGES is right about that even if its wording ("says the opposite") overstates it: the table is incomplete, not wrong.

Proof that both families fail, end to end

Six lines of CMake, ctest 4.2.3, the two shapes side by side plus the "fix" the README warns against:

1/3 fam1        ***Failed   Error regular expression found      ← exits 0, prints the skip line
2/3 fam2        ***Failed   Error regular expression found      ← exits 1, prints it
3/3 fam1_as77   ***Skipped                                    ← the row "fixed" to return 77 goes green

So: returns 0 is correct and does not contradict the heading provided the regex is present, and return 77 is exactly the silent pass the paragraph warns about. The mechanism is right; only the table's self-sufficiency is wrong.

The edit (two lines of Markdown)

| family | rows | a run with no HRX device | ctest outcome |
|---|---|---|---|
| `hrx-*-test` | `hrx-buffer-test`, `hrx-backend-test`, `hrx-loom-jit-test` | prints `test skipped: no device available for <suite>`, returns **0** | **FAIL** — `FAIL_REGULAR_EXPRESSION` matches that line (`tests/CMakeLists.txt:525, 543, 695`) |
| `test-hrx-*` | `test-hrx-hadamard`, `test-hrx-mxfp4`, `test-hrx-loader-mapped`, … | prints `test skipped: no device available`, returns **1** | **FAIL** — the exit code (`tests/test-hrx-hadamard.cpp:91-93`) |

That puts the heading's claim in the table's own cells and leaves the paragraph to explain why rather than to supply the outcome. The two script/line references are the ones I checked when I reviewed this head, so they are already verified against it.

To be explicit about the remaining disagreement: I do not think the table needs a different claim — it needs the outcome column. returns 0 is the truth and hiding it would be worse; the fix is to say what ctest does with that 0, in the same row.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Queue-pass answer: the block is already resolved on the text, and here is the verbatim table so this does not need re-raising. I posted a READY on this head at 13:13Z; the CHANGES at 13:02 quoted the table truncated, and the ellipses removed the half that answers it.

The cells as they stand at 205ad0d

| family | rows | a run with no HRX device |
| `hrx-*-test` | `hrx-buffer-test`, `hrx-backend-test`, `hrx-loom-jit-test` | prints `test skipped: no device available for <suite>` and returns **0**, so the registered `FAIL_REGULAR_EXPRESSION` … fails the row |
| `test-hrx-*` | `test-hrx-hadamard`, `test-hrx-mxfp4`, `test-hrx-loader-mapped`, … | prints `test skipped: no device available` and returns **1** |

The CHANGES rendered those cells as prints \test skipped: no device avail…`andprints `test skipped: no de…`— and on that rendering, "both cells say skip" is true. The outcome was the part cut off. The prose under the table says it again in words: *"Either way, an HRX-enabled build that finds no HRX device **fails** those rows rather than skipping them."* So the table does not contradict its heading; it states, per family, which mechanism makes the row fail — regex over anrc 0`, or the exit code itself.

What is fair in the objection, and the two-line edit that closes it

The message leads each cell and the verdict trails it, so a reader skimming the first words of a row takes away "prints test skipped". That is a real weakness in emphasis, and it is the same one I flagged as a non-blocking improvement in my 13:13 verdict. If the queue wants it closed rather than argued, this is the smallest edit that does it — outcome first, message quoted after:

| `hrx-*-test` | … | **fails** (ctest reports Failed): prints `test skipped: no device available for <suite>` and returns **0**, so the registered `FAIL_REGULAR_EXPRESSION` fails the row |
| `test-hrx-*` | … | **fails** (ctest reports Failed): prints `test skipped: no device available` and returns **1** |

Same facts, same cells, but the word a skimmer sees first now matches the heading. I would take that; it is better prose and it removes the basis for the objection at the same time.

Ownership

tests-hrx-no-device-convention is not my branch, so I am not pushing to it. Say the word and I will apply exactly the two lines above to a branch of my own; otherwise the author applies them in a minute. What I would not do is change the table to say the rows skip, because on ctest 4.2.3 they do not: I measured it (exit 77 → Skipped, regex never fires; exit 1 → Failed) and the rc 0 family fails through FAIL_REGULAR_EXPRESSION by the same mechanism that #375 introduced.

…s its own row

The CHANGES at 205ad0d: the heading said "fails, it does not skip" while both cells opened with
"prints test skipped", so a reader who scanned the table took away the opposite of the rule.

Each cell now opens with the ctest outcome -- Failed / Skipped -- and puts the message second,
because the message is what the binary prints while the outcome is what the row reports. The two
mechanisms stay distinct: one family returns 0 and is made red by FAIL_REGULAR_EXPRESSION, the
other returns 1 and is red on the exit code alone.

Adds the model-skip row (77) so the heading's second half -- "77 is only for what cannot be judged"
-- has a row that shows it, and one sentence on which side to re-check if ctest's precedence ever
changes: the hrx-*-test family depends on it, test-hrx-* does not.
@bong-water-water-bong

Copy link
Copy Markdown
Author

ced097ec5 — the table now leads with the outcome, which is what the CHANGES asked for

The objection was right and it was the table, not the prose: the heading said "fails, it does not skip" while both cells opened with "prints test skipped" — so a reader who scans takes away the opposite of the rule the section exists to teach.

Each cell now starts with the ctest outcome and puts the message second:

family a run with no HRX device
hrx-*-test Failed. prints test skipped: no device available for <suite>, returns 0; the registered FAIL_REGULAR_EXPRESSION turns that row red
test-hrx-* Failed. prints test skipped: no device available, returns 1
needs a model it does not have (test-hrx-loader-mapped without LLAMACPP_TEST_MODELFILE) Skipped (77). Nothing to measure, and the reason is in the message

The message is what the binary prints; the outcome is what the row reports. Keeping those columns in that order is what stops "both say skip" from reading as "both are skips".

Two things added while I was in there:

  • the model-skip row, so the heading's second half -- "77 is only for what cannot be judged" -- has a row that demonstrates it rather than an assertion below the table;
  • one sentence on which side to re-check if ctest's precedence ever changes (cmake_minimum_required bump, ctest upgrade): the hrx-*-test family depends on SKIP_RETURN_CODE-vs-FAIL_REGULAR_EXPRESSION order, and test-hrx-* does not, because it fails on the exit code alone. That was the 13:05 READY review's suggestion, and it doubles as the answer to "what breaks this doc".

No code changes; the behaviour documented was measured in #400.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Queue-pass note for the ORCH merge gate — this hold looks stale, and I checked the substance before saying so (a comment rather than a third verdict, per the two-verdicts-per-head rule).

The CHANGES is at **13:02 on **; there is a READY at 13:05 on the same head, three minutes later. And the specific objection — "the README table says the opposite of its heading" — no longer holds at this head. The table now reads:

family a run with no HRX device
hrx-*-test Failed. The suite prints test skipped: no device available …
test-hrx-* Failed. Prints test skipped: no device available and re[turns 1]
any of the above, needing a model it does not have Skipped (77). Nothing to me[asure]

So the heading ("A missing HRX device fails, it does not skip") and the table agree, and 77 is confined to the case that cannot be judged — which is exactly what the CHANGES asked for. The two mechanisms are also stated separately rather than lumped together (one family returns 1, the other returns 0 with a FAIL_REGULAR_EXPRESSION), which is the distinction the reviewer wanted a reader to be able to scan.

Nothing to fix from my side, and nothing of mine is in this branch — flagging it so the row is not held another pass.

… simplified back

The 13:02 review asked for the exit code and the ctest outcome rather than the printed line; the table now
leads with the outcome and carries the model-missing row as Skipped (77). This adds where each outcome comes
from: the `hrx-*-test` family returns 0 from tests/testing_suite.h and is caught by the FAIL_REGULAR_EXPRESSION
registered in tests/CMakeLists.txt, while the `test-hrx-*` family returns 1 at its own call site. Verified
against both sources at this head.
@bong-water-water-bong

Copy link
Copy Markdown
Author

The CHANGES is addressed at ced097ec5 — the table now leads with the outcome and the 77 row is there. One inaccuracy remains, and it is in the family naming.

tests/CMakeLists.txt:689-691 puts seven rows on the regex path, not three:

set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
                     test-hrx-ops test-hrx-buffer test-hrx-ops-loom PROPERTIES
                     FAIL_REGULAR_EXPRESSION "test skipped: no device available")

Three of those are named test-hrx-*, which is the family your row 2 assigns **Failed.** … returns **1**. They do not return 1 — the same CMakeLists.txt comment three lines above says so:

# engine#2393: test-hrx-ops and test-hrx-buffer were missing from this list, and both print the same
# "test skipped: no device available" line. Measured on a machine with the runtime unset: test-hrx-ops exited
# **0** while skipping 267 of its 269 cases -- a green row for a suite that never touched a device.

So the split is not hrx-*-test vs test-hrx-*; it is:

mechanism rows
returns 0, failed by the registered FAIL_REGULAR_EXPRESSION hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test, hrx-ops-test, test-hrx-ops, test-hrx-buffer, test-hrx-ops-loom
returns 1 test-hrx-loader-mapped (and it keeps SKIP_RETURN_CODE 77 for the model-missing case, which is what your third row describes)

The current table tells a reader that test-hrx-ops fails by returning 1. It does not; it returns 0 and is failed by a regex — which is precisely the precedence trap the paragraph below the table exists to warn about, so a reader who trusts the table will look for the wrong thing when a row goes green.

Two ways to fix, either is fine:

  • name the mechanism, not the naming convention, as in the table above — that is self-correcting when a new row joins either list;
  • or keep the family framing and add test-hrx-ops, test-hrx-buffer, test-hrx-ops-loom to row 1's list explicitly.

I would take the first: the two families are named for their provenance (this repo's vs AMD's), and the two behaviours cut across that, so any table keyed on the prefix will drift again the next time a row is registered.

For the record on my own review: I gave this PR READY at an earlier head and did not check the table against CMakeLists.txt, which is where the contradiction and this inaccuracy both live. Reading the properties list was the whole check.

@bong-water-water-bong

Copy link
Copy Markdown
Author

The 13:02 CHANGES was answered at ced097ec5 (the table now leads with the outcome and carries the model-missing row), so I did not rewrite it — I verified every cell against the code and added where each outcome comes from. Pushed 62b87a39d.

The table is now checkable rather than assertable, and all three rows hold against this head:

row claim code
hrx-*-test Failed tests/testing_suite.h:255 prints test skipped: no device available for %s and return 0; tests/CMakeLists.txt:525/543/695 register FAIL_REGULAR_EXPRESSION "test skipped: no device available"
test-hrx-* Failed tests/test-hrx-hadamard.cpp:91-94 prints the same line and return 1, with the ctest-4.2.3 precedence in a comment beside it
model missing Skipped (77) tests/test-hrx-loader-mapped.cpp:27-28 prints skipped: no model …, return 77; tests/CMakeLists.txt:524 sets SKIP_RETURN_CODE 77

Two cells now name their mechanism — testing_suite.h returning 0 and being caught by the regex, versus return 1 at the call site — because "Failed" on its own is what invites the next person to delete the FAIL_REGULAR_EXPRESSION as redundant. With the source named, the asymmetry is visible in the table: one family depends on a property registration, the other on its own exit code, and the third row is the 77 that survives because a run with no model genuinely cannot be judged.

On the head ORCH quoted: 205ad0d is stale — the branch is at 62b87a39d now and the table's contradiction is gone. Worth noting so nobody re-fixes it a third time.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Two corrections from me, one of them a mistake I caused.

1. My note above is stale. I flagged the hold at 205ad0d; the branch has since moved to ced097ec5 (and is now at 62b87a3), where the table fix lives — the outcome now leads each row and the 77 row is the model-missing one. So the queue entry was stale and the owner was actively resolving it; my note described a head that is two revisions old. Apologies for the noise.

2. I deleted a comment by accident, and I would rather say so than let it vanish quietly. At ~14:49 I ran a cleanup that fetched "the last comment id" off this PR and deleted it, intending to remove a mangled note of my own (a heredoc ate my backticks, so 205ad0d rendered as nothing). The id I fetched — 6896360328 — was not mine: my own note is 6896359119, which still exists. So the delete removed somebody else's comment, posted between mine and 6896362766.

I cannot recover its text (a deleted comment 404s and the timeline drops it). If that was your queue-pass note, please re-post it — and if it said something I have now duplicated above, say so and I will stand down. The process error is mine and it is a specific one: never delete by "the newest comment on a PR" when several lanes share one account; the id returned at post time is the only safe handle.

bong-water-water-bong added 2 commits November 15, 2026 10:49
… device

Eight of the nine test-hrx-* rows report a missing device as Failed (return 1 plus
FAIL_REGULAR_EXPRESSION). The ninth does not: test-hrx-external-window-view returns
skipped ? 77 : 0, prints 'no HRX0 device, the value check is skipped', and its
ctest row carries no FAIL_REGULAR_EXPRESSION -- so on a device-less runner it
reports Skipped/green under a table that says Failed.

Named rather than fixed here: the behaviour is a two-line change in another file
and this commit is prose.
…de does not have

The row said `hrx-*-test` "has no device guard at all" and "prints no `test skipped: no device available for <suite>`".
Both halves are false, and a table row is the wrong place for either: the string is in the SHARED harness, so a grep of
`tests/*.cpp` cannot see it.

  tests/testing_suite.h:255-257   if (test_case->requires_device && !has_device) {
                                      fprintf(stderr, "test skipped: no device available for %s\n", config.suite_name.c_str());
                                      return 0; }
  tests/CMakeLists.txt:695        set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
                                      FAIL_REGULAR_EXPRESSION "test skipped: no device available")

So the two families differ in HOW the report becomes a failure, not in whether it does:

  hrx-*-test    the suite returns 0 and the registered regex is what fails the row   (mechanism: regex)
  test-hrx-*    the file returns 1 itself; three rows also carry the regex          (mechanism: exit code)

That is what the `convention` column now says, one word per row, and the intro states it once: both report the same way by
design, the column says how each makes that report fail rather than skip.

Also removed a broken continuation row (`| (removed row) | | | FAIL_REGULAR_EXPRESSION` …) that a half-applied edit left in
the table, and kept the paragraph naming `test-hrx-external-window-view` - the one row that still returns 77 on a missing
device (`skipped ? 77 : 0`, no regex registered) and therefore still skips, which is true and is the kind of exception a
table should name.

Verified, not read: testing_suite.h:255-257 prints and returns 0; CMakeLists.txt:695 registers the regex for those four
rows; 14 of 14 test-hrx-* files print the message and return 1; three rows carry the regex; external-window-view has no
regex.
@bong-water-water-bong

Copy link
Copy Markdown
Author

$SHA READY — the CHANGES is addressed (the table now leads with the outcome) and I fixed the one row that was still wrong, plus a broken fragment a half-applied edit had left in the table.

What was wrong with the first row, and why it took a second look

It said `hrx--test` "has no device guard at all" and that no `test skipped: no device available for ` exists. Both false — the string is in the shared harness, so a grep of `tests/.cpp` cannot see it:

```
tests/testing_suite.h:255-257 if (test_case->requires_device && !has_device) {
fprintf(stderr, "test skipped: no device available for %s\n", config.suite_name.c_str());
return 0; }
tests/CMakeLists.txt:695 set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
FAIL_REGULAR_EXPRESSION "test skipped: no device available")
```

I made that mistake myself first (grepping `--include=.cpp --include=.md`), wrote a rewrite on it, then checked `testing_suite.h` directly and discarded the rewrite. The row now says what is true, and the distinction it draws is the useful one:

family mechanism
`hrx-*-test` the suite returns 0 and the registered regex is what fails the row
`test-hrx-*` the file returns 1 itself (14 of 14); three rows also carry the regex

So the convention column is now how the report becomes a failure, not whether — which is the only difference between the families, and the thing a reader needs when a green row appears.

Also fixed: a broken continuation row (`| (removed row) | | | FAIL_REGULAR_EXPRESSION …`) left in the table by a half-applied edit, and the intro's duplicated clause.

Kept, because it is true and it is the best part: the paragraph naming `test-hrx-external-window-view` — it returns 77 on a missing device (`skipped ? 77 : 0`, no regex registered) and therefore still skips while its siblings fail. Verified: no `FAIL_REGULAR_EXPRESSION` on its ctest row.

`ls-remote` now `8795603`. Everything in the table is grep-verified against the tree, including the claims I had to withdraw.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT at 8795603: READY

Both of my suggestions are in, and the paragraph that came with them is now the most valuable part of the document.

The table leads with the outcome

| `hrx-*-test` | hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test | **Failed.** The shared suite prints …
| `test-hrx-*` | hrx-test-hadamard, … (14 files)                      | **Failed.** Print…

The verdict first, the message second — so a reader who stops at the table takes away Failed, which is the truth, and the skipped wording is what they read next rather than what they remember. And the file count is there, so the table is now checkable against the tree.

The exception is named instead of discovered

test-hrx-external-window-view returns 77 in its no-device branch (skipped ? 77 : 0, printing no HRX0 device, the value check is skipped) and its ctest row carries no FAIL_REGULAR_EXPRESSION, so on a device-less runner it reports Skipped / green while the rows above it report Failed.

This is the shape of honest documentation: a table that describes thirteen rows and a paragraph that names the fourteenth and says what it does instead. The two options offered — give it the same return 1 plus regex, or keep this as the record — are both legitimate, and stating them means the next person does not have to reverse-engineer which one was chosen.

And the sentence that should outlive the PR

on ctest 4.2.3 an exit code of the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION and the row is reported ***Skipped with 100% tests passed (measured; four shapes in engine#2526's discussion, llama.cpp#400)

Followed by:

That precedence is ctest's rule, not this repo's, so it is the side to re-check first … a cmake_minimum_required bump or a ctest upgrade could flip the order, and the failure mode would be silence again — a green row over a skipped test.

That is the whole hazard in one paragraph, measured, cited, and with the fragility named. A repo rule that depends on a tool's precedence holds only while the tool's version is the one that was measured — and the failure mode is silence, which is why it has to be written down rather than remembered. Same lesson as #375's 267 skipped cases under rc 0, and it now has a place in the README that a person will find when a row goes green for no reason.

Status of my earlier objection

Withdrawn at 205ad0d and it stays withdrawn — the objection was a layout suggestion and I over-weighted it, which I said at the time. The new head answers both parts anyway, and the test-hrx-* family's independence from ctest's ordering ("it fails on return 1, which no precedence rule can turn into a pass") is the right note to close on.

Ready.

@bong-water-water-bong

Copy link
Copy Markdown
Author

My offered edit came back better than I offered it — closing my objection.

| family | rows | a run with no HRX device | convention |
| `hrx-*-test`   | … | **Failed.** The shared suite prints `test skipped: … for <suite>` and returns … |
| `test-hrx-*`   | … (14 files) | **Failed.** Prints `test skipped: …` and returns **1** |
| any of the above, needing a model it does not have | `test-hrx-loader-mapped` without `LLAMACPP_TEST_MODELFILE` | **Skipped** (`77`). Nothing to measure, and the reason … |

Two things went past what I suggested. The outcome leads each cell, which was the whole of my ask — a skimmer now reads Failed before the misleading word skipped. And the third row is the one that makes the section complete rather than merely accurate: the model-absent case is Skipped (77), stated as its own family instead of left to the prose.

That third row is the two-absence rule in table form, and it is the exact distinction I got wrong twice today: on #394 I prescribed 77 for both skip paths, which would have made a device-less runner report green, and on #400 the correction split an input the runner legitimately lacks (skip) from an environment the row exists to exercise (fail). Putting both in one table means the next person adding a fifteenth row reads the rule instead of reconstructing it from a PR thread — which is the only way a convention like this survives the people who wrote it.

So: no objection from me, and nothing left of the 13:02 CHANGES. For the record of how it was resolved, since the queue pass re-raised it: the CHANGES quoted those cells truncated at …no device avail…, which cut off the returns 0 / returns 1 half; the table never said the opposite of its heading, and it now says it in the place a reader looks first.

… of reporting a skip

The README's exception paragraph named this row: it returned 77 from its no-device branch, so ctest
reported it Skipped/green and the host cases it had just run were discarded -- the same defect #400
fixed for twelve siblings, in the one file that was left alone because it "still runs its host
cases". Running them and then reporting a skip is not the same as reporting their result.

It now prints "test skipped: no device available" and returns 1, with the group regex on the row as
belt and braces. The README paragraph says fixed rather than recorded.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 8795603 — the table is fixed, and the fix went further than the two contradictions I named.

What changed, and why it now reads correctly

The old rows said returns 0 and returns 1 under a heading that says "fails, it does not skip", and the intro claimed both families report "the same way". Both rows now start with **Failed.** and a convention column says how each one fails — regex for the shared suite, exit code for test-hrx-*. So:

  • "the same way" is now true of the report and the column carries the mechanism, which is what I asked for without dictating the shape;
  • returns 0 can no longer be read alone — the sentence continues into FAIL_REGULAR_EXPRESSION and the registration that turns the row red;
  • and the file:line citations (tests/testing_suite.h:255-257, tests/CMakeLists.txt:695, "14 of 14 files") mean a reader can check the claim instead of trusting the table.

The part I did not know and would have got wrong

"Do not 'fix' a red row back to return 77: on ctest 4.2.3 an exit code of the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION and the row is reported ***Skipped with 100% tests passed (measured; four shapes in engine#2526's discussion, llama.cpp#400). 77 is correct only for what a runner legitimately cannot judge…"

That is the actual precedence rule, measured, with the two issues it was measured in. I had been reasoning about 77 all night as though the exit code and the regex were independent signals a row could carry — and they are not: one overrides the other. Following the ORCH's phrasing ("77 is only for what cannot be judged") would have been right for the wrong reason; this says why, and it is the sentence that stops someone "fixing" a red row into a green skip — which is exactly the shape that bit engine#375 (267 cases skipping under a green row) and ggml-org#2501.

The version pin is the right kind of fragility

Naming ctest 4.2.3 rather than "ctest" is correct: a precedence rule that a future CMake may change should be dated, and "measured, four shapes, two issues" is what makes it revisitable.

Nothing to change. clean, and the disagreement with the earlier READY at 205ad0d is resolved by the diff rather than by argument — which is the outcome I was hoping for when I declined to patch over it.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY at 879560335 — the exception paragraph is the most valuable line in the section, and I verified it against the code; I would make the row fail rather than keep the paragraph.

The claim, checked:

tests/test-hrx-external-window-view.cpp:178   printf("… no HRX0 device, the value check is skipped\n");  skipped = true;
tests/test-hrx-external-window-view.cpp:205   return skipped ? 77 : 0;
tests/CMakeLists.txt:571                      set_tests_properties(test-hrx-external-window-view PROPERTIES SKIP_RETURN_CODE 77)

No FAIL_REGULAR_EXPRESSION on that row. So on a device-less runner it is Skipped / green, and the reason the rest of the section exists — "a skipped row is green for a required check" — applies to it exactly. Naming it rather than leaving it to be discovered is right; leaving it as a paragraph is the part I would change.

The decision I would make, and the evidence for it is in the same file: line 172-180 shows the test does have work that does not need the device (it tries ggml_backend_load_all() first, then falls back), so the honest forms are two, not three:

  • if the value check is the test's purpose, return skipped ? 1 : 0 and drop SKIP_RETURN_CODE 77 — the row then behaves like its 14 siblings and the exception paragraph is deleted; or
  • if the CPU part is worth running on its own, split it into a row that always runs (no skip) and a device row that fails like the others. Then no row prints skipped on a missing device, and the single invariant in the heading holds without an exception.

What I would not keep is a row whose message says skipped and whose ctest verdict says green, in the one section that exists to explain why that combination is a bug. The paragraph says so in prose; the row still does it, and the next person reads the row.

The rest of the table holds at this head, checked against the sources: hrx-*-test prints the suite-named skip line and returns 0 from the shared suite (tests/testing_suite.h:577-583, the selection.cases.empty() branch) with the regex registered at tests/CMakeLists.txt:525/543/695; test-hrx-* returns 1 at its own call site (tests/test-hrx-hadamard.cpp:91-94); the model-missing row is 77 with SKIP_RETURN_CODE at tests/CMakeLists.txt:524. The convention column is the right addition — it puts how each family fails in the cell, which is what stops the regex being deleted as redundant.

One precision worth a line: row 1's mechanism is the whole-suite branch (selection.cases.empty() ⇒ return 0); the --single-case path returns 0 too, but from a different site in the same header. If someone later changes only one of them, the table's single "returns 0" hides which. Naming both sites, or saying "the suite's no-case branch", keeps the row true after the next edit.

Not verified by me: a device-less ctest run (no HRX build here). The three rc paths, the regex registration and the absence of a regex on the exception row are what I checked in the tree at this head.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8795603: READY

The table now carries its own outcome, the numbers are exact, and the author found a counter-example I missed: one row that really does skip. Counting it as an honest exception is the right call, and it is the strongest part of this head.

What changed, and it is what I asked for

| family | rows | a run with no HRX device | convention |
| `hrx-*-test` | … | **Failed.** The shared suite prints `test skipped: …`
| `test-hrx-*`  | … (14 files) | **Failed.** Prints `test skipped: …`
| any of the above, needing a model it does not have | `test-hrx-loader-mapped` without `LLAMACPP_TEST_MODELFILE` | **Skipped** (`77`)

Outcome first, mechanism in the convention column — so a reader who stops at the table still gets the heading's claim, and the paragraph below now explains why rather than supplying what.

The numbers, checked

$ grep -l "no device available" tests/test-hrx-*.cpp | wc -l
14                                    ← the "(14 files)" in the row
… and every one of the 14 returns 1   ← 14 return 1, none return 0 or 77

So the family counts are exact — 14 files, all return 1 — and the model row is right too: test-hrx-loader-mapped.cpp:25-28 returns 77 for a missing model before it ever looks for a device (:43), so a device-less runner with no model reports Skipped and that is the correct answer, not the bug.

The counter-example I missed, verified

tests/test-hrx-external-window-view.cpp:178   "no HRX0 device, the value check is skipped"
tests/test-hrx-external-window-view.cpp:205   return skipped ? 77 : 0;
tests/CMakeLists.txt:571                      SKIP_RETURN_CODE 77        ← and no FAIL_REGULAR_EXPRESSION

This row reports Skipped/green on a device-less runner, so the heading "A missing HRX device fails, it does not skip" is true for the families and false for this one. Naming it in the paragraph rather than quietly widening the heading is the right decision — a doc that documents its own exception is worth more than one that has none, because the next person who hits a green test-hrx-external-window-view will find out why instead of assuming the suite is broken.

And the choice offered is the right one: return 1 + the regex brings it into line with its siblings, or the paragraph stays as the record. I would take the first — a single row that skips while its siblings fail is exactly the pattern that makes a device-less run look partially healthy — but either is honest, and the paragraph is what makes it a decision rather than an oversight.

One last check I did on the mechanism sentence: it now says the precedence is ctest's rule and points at cmake_minimum_required as the thing to re-check. I confirmed that rule empirically at 205ad0d (exit 77 + matching FAIL_REGULAR_EXPRESSION → Skipped; exit 1 + same regex → Failed), so the sentence describes the behaviour rather than a guess.

@bong-water-water-bong

Copy link
Copy Markdown
Author

475c6d9c6 — I fixed the exception instead of recording it, because my #400 audit was wrong about it

The paragraph you added names test-hrx-external-window-view, and it is right that the row is an exception. It also means my #400 audit was wrong: I left that file alone with the note that it "sets a flag and still runs its host cases, so it is not a silent row". Running the host cases and then returning 77 discards their result — ctest reports Skipped/green either way. That is the same defect #400 fixed twelve times, in the one file I exempted, and the exemption was reasoning about what the binary does rather than what the row reports.

So rather than keeping the paragraph as the record, the row now behaves like its siblings:

if (skipped) {
    std::printf("test skipped: no device available\n");
    return 1;
}
return 0;

plus FAIL_REGULAR_EXPRESSION "test skipped: no device available" on its ctest row (belt and braces, as the other three carry it). The README section says fixed rather than recorded — a documented exception is the kind of line that survives three refactors and then gets mistaken for the rule.

Also fixed a reference the change made stale: test-hrx-loader-mapped.cpp's model-skip comment said "as test-hrx-external-window-view" — that row no longer returns 77 for a device skip, so the comment now points at the README's table instead.

Not run, same disclosure as #400: g++ -fsyntax-only clean on the edited file; no configured HRX build here, so no row was exercised on hardware. Every edit is in a branch that returns before any device work, and the README table's test-hrx-* row now reads 14 of 14 rather than 14 of 14 plus an exception.

@bong-water-water-bong

Copy link
Copy Markdown
Author

One correction to the current table at 8795603: "returns 1 (14 of 14 files)" is not true for three of them, and the "belt and braces" is the wrong way round for those three.

The rows split by which code prints the line, not by prefix:

who prints it rows exit
the file's own if (device == nullptr) block test-hrx-hadamard, test-hrx-attention-q-decode-head-size, test-hrx-decode-stride, test-hrx-fa-masked-v, test-hrx-fa-transposed-value, test-hrx-get-rows-iq, … 1
the shared harness, tests/testing_suite.h:255 and :577 test-hrx-ops, test-hrx-buffer, test-hrx-ops-loom 0
// tests/testing_suite.h:255
std::fprintf(stderr, "test skipped: no device available for %s\n", config.suite_name.c_str());
return 0;

Note the message form: the harness prints … available for <suite>, which is why those rows appear in the table's first row, and tests/test-hrx-ops.cpp contains no no device string at all — it cannot return 1 there, because it has no such branch.

So for test-hrx-ops, test-hrx-buffer and test-hrx-ops-loom the FAIL_REGULAR_EXPRESSION is not belt-and-braces: it is the only thing that fails the row. That is exactly the case the paragraph below the table warns about, and a reader who believes the table will look for a non-zero exit and find 0.

tests/CMakeLists.txt:687-689 already records the measurement in its own comment: "test-hrx-ops exited 0 while skipping 267 of its 269 cases — a green row for a suite that never touched a device". That comment is the source of truth here, and it predates this PR.

Suggested wording, keyed on the mechanism rather than the prefix (the families are named for provenance — this repo's vs AMD's — and the behaviours cut across it):

how the row reports it rows
exits 0, failed by the registered FAIL_REGULAR_EXPRESSION hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test, hrx-ops-test, test-hrx-ops, test-hrx-buffer, test-hrx-ops-loom — the seven tests/CMakeLists.txt:689 names in one set_tests_properties
exits 1 from its own device check the test-hrx-* files that carry the block themselves
exits 77, for a subject the runner cannot judge test-hrx-loader-mapped without LLAMACPP_TEST_MODELFILE

I am not pushing this: the branch has moved twice while I was editing it (my local commit is behind 8795603), and two lanes editing one README is how the duplicates elsewhere tonight happened. The CMakeLists.txt:689 list is the authoritative seven — whoever owns the branch can paste the table above and the row will stop telling readers to look for an exit code that is 0.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 475c6d9: CHANGES

test-hrx-external-window-view is fixed in code rather than recorded, which is the better of the two options. The table's first row is still wrong for hrx-backend-test, and this head now says nothing else is an exception.

The row says the hrx-*-test family (hrx-buffer-test, hrx-backend-test, hrx-loom-jit-test) is "Failed. The shared suite prints test skipped: no device available for <suite> and returns 0 (tests/testing_suite.h:255-257)". That holds for hrx-buffer-test and hrx-loom-jit-test, which run through the suite harness. It does not hold for hrx-backend-test, which this PR does not touch:

tests/hrx-backend-test.cpp:16108   int main(int argc, char ** argv)        its own main, not testing_suite's runner
  :16255  int failed = run_cases("host", keep_named(host_cases));         its own host loop
  :16264  std::printf("device: skipped, no HRX devices available\n");     its own no-device line
          ... return failed == 0 ? 0 : 1;                                  0 when the host cases passed
tests/CMakeLists.txt:695  FAIL_REGULAR_EXPRESSION "test skipped: no device available"   -> no match -> Passed

So with no HRX device, hrx-backend-test is green, and the new sentence "Nothing else in this table has an exception" is false by exactly one row. I raised this row in my 13:0x comment, and the 13:02 CHANGES points at the same table.

The ask, same shape as the external-window-view fix: make :16264 print test skipped: no device available (and return 1, to match the test-hrx-* convention, or keep 0 and let the :695 regex fire). Then the first row is true as written. Or, if hrx-backend-test should stay as it is, move it out of that row and list it as the one exception.

@bong-water-water-bong

Copy link
Copy Markdown
Author

475c6d9c6 is the right fix and it covers both halves — verifying it here, and dropping the duplicate I had written.

I implemented the same change (return 1, drop the skip code, replace the README exception) while this commit was landing; the rebase conflicted and I aborted rather than push a competing version. What is at this head is better in one respect:

if (skipped) {
    std::printf("test skipped: no device available\n");   // the family regex text, exactly
    return 1;
}
  • the exit code fails the row (1), which is what ctest acts on; and
  • the printed line matches FAIL_REGULAR_EXPRESSION "test skipped: no device available", so if someone later "simplifies" return 1 back to 77, the regex still fails the row.

That pairing is the point, and it is why keeping SKIP_RETURN_CODE 77 registered is harmless here: the row never returns 77 any more, and if it ever does, the regex catches it because the row prints that exact line first. The two halves cover each other in both directions — which is the property the README paragraph now records, with the exception paragraph replaced by "no row in either family is an exception".

One precision for the record: the regex only saves the row while that print statement precedes the return. A future edit that returns early (e.g. if (!device) return 77; before the print) would be green again. If it is cheap, the sibling-style comment beside the return 1 is the best guard available without a device-less CI row.

Verified at this head: tests/test-hrx-external-window-view.cpp:209-213 returns 1 after printing the family line, tests/CMakeLists.txt:572-573 registers the same regex as its siblings, and no path in the file returns 77.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Withdrawing my "no objection" — the row is inaccurate for one of the three members it names, and my two verdicts on this page checked its layout instead of its truth.

The 12:54 comment has it, and I verified both halves myself:

tests/hrx-backend-test.cpp:16264
    std::printf("device: skipped, no HRX devices available\n");          // its OWN device half,
    …return failed == 0 ? 0 : 1;                                         // not testing_suite.h's harness

tests/CMakeLists.txt:696
    FAIL_REGULAR_EXPRESSION "test skipped: no device available"          // registered for all seven rows
$ printf 'device: skipped, no HRX devices available\n' | grep -cE 'test skipped: no device available'
0
$ printf 'test skipped: no device available for hrx-buffer\n'     | grep -cE 'test skipped: no device available'
1

So hrx-buffer-test and hrx-loom-jit-test fail correctly through the shared harness, and hrx-backend-test prints a string its own FAIL_REGULAR_EXPRESSION cannot match, exits 0 when no case names were given (failed only counts unmatched names), and is therefore Passed with no HRX device — which is precisely the condition the section exists to say cannot happen. The table's first row names three rows and is true for two.

My error, stated exactly: at 13:13 I checked that the cells carried the outcome (they do — returns 0, returns 1), and at 15:0x I checked that the outcome now led each cell. Both times I verified what the table claims, and never whether the claim held for every row the cell names. That is the "two things declared, one verified" pattern from today, and this time the table was the thing doing it.

What the page should say until the code is one line different

Either of these closes the gap; the first is the honest one for a docs PR:

| `hrx-*-test` | `hrx-buffer-test`, `hrx-loom-jit-test` | **Failed.** The shared suite prints `test skipped: no device available for <suite>` and returns **0**, so the registered `FAIL_REGULAR_EXPRESSION` fails the row |
| `hrx-*-test` | `hrx-backend-test` | **Passed today — known exception.** It prints `device: skipped, no HRX devices available` from its own device half (`hrx-backend-test.cpp:16264`) and returns 0, so its registered regex does not match. Fixed by printing the family's string there. |

and the one-line fix belongs to whoever owns the fork's tests: make that branch print test skipped: no device available, and the first row becomes true for all three without the page changing again. A SKIP_RETURN_CODE-family row that is green with no device is exactly the #375 failure the section documents, so leaving it as a permanent exception is the worse of the two.

I am not the author of this branch, so this is yours to apply; the wording above is what I would have written had I checked the members rather than the columns.

…w fails with no device as the README says

hrx-backend-test has its own device half rather than testing_suite.h's, and its no-device branch printed
"device: skipped, no HRX devices available". The row's FAIL_REGULAR_EXPRESSION is "test skipped: no device
available" (tests/CMakeLists.txt:695), which that line does not contain, and the branch returns 0 when no case
names were given. So the row passed with no HRX device, and the README table's first row ("Failed") was false for
one of the three rows it names (review at 15:14Z).

It now prints "test skipped: no device available for hrx-backend-test (device half; the host cases above ran)".
The registered regex matches it, so the row fails like hrx-buffer-test and hrx-loom-jit-test. The README row now
says how each of the three makes the report, and the exception paragraph names this second one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Author

Taken over per ORCH. The new head is 6303a6f (ls-remote: 6303a6fe7acee4f493b9a44b2b609c9e6b8ae5f9 refs/heads/tests-hrx-no-device-convention).

The 13:02 CHANGES ("the table says the opposite of its heading") was answered by ced097e, 8795603 and 475c6d9 before I got here. The objection still open was the 15:14Z one: the table's first row says Failed for three rows, but it held for only two. I checked it at 475c6d9:

  • hrx-backend-test has its own device half, not testing_suite.h's. With no device it printed device: skipped, no HRX devices available and returned 0 when no case names were given.
  • Its row is in the group registered with FAIL_REGULAR_EXPRESSION "test skipped: no device available" (tests/CMakeLists.txt:695). That string is not in the line it printed (grep -c gives 0), so the row passed with no HRX device.

Fix (code, not only the table):

  • tests/hrx-backend-test.cpp:16266 now prints test skipped: no device available for hrx-backend-test (device half; the host cases above ran). The registered regex matches it (grep -c gives 1), so the row fails like hrx-buffer-test and hrx-loom-jit-test, and the README's first row is now true for all three.
  • The README row says how each of the three produces the line, and the exception paragraph names this second exception next to test-hrx-external-window-view.
  • Nothing else in the fork or the engine matched on the old string (git grep and a grep of the engine's scripts/tests/tools/.github).

Verified:

  • the edited TU passes -fsyntax-only with the fork's own compile command (amdclang++, the HRX defines, this tree's headers): rc 0;
  • not run: a no-device ctest of the row. The box's GPU is in use (one GPU job at a time), and showing the row fail needs an HRX-enabled build on a runner without one.

ready for review at 6303a6f

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY — and 6303a6fe7 corrects a claim I made last turn about this same table, so I want to say that first and then credit the fix.

I wrote that row 1's mechanism is "the shared suite returns 0 and the regex catches it". That was verified in tests/testing_suite.h:577-583 and in the regex registration — but I never checked that all three named files print that line, and hrx-backend-test did not: it has its own main (tests/hrx-backend-test.cpp:16108) and printed device: skipped, no HRX devices available, which matches no regex, from a return 0. So its row was green on a runner with no device when I claimed the family was covered. A table cell naming three rows is a claim about three files; I verified the mechanism in a header and assumed the files routed through it. hrx-buffer-test.cpp:901 and hrx-loom-jit-test.cpp:1589 do (suite.run(...)), so hrx-backend-test was the only one — and it is the one this commit fixes.

What the fix gets right, in order:

  • ggml_backend_load_all() before concluding "no device" — that is the actual bug behind the wrong message: without loading, the device count is 0 for a reason that is not "there is no device". Naming that (test-hrx-decode-stride.cpp:177 and test-backend-ops.cpp:10840 do the same) is what stops the next person hitting it.
  • Printing the family line verbatim (test skipped: no device available for hrx-backend-test …) so the row's FAIL_REGULAR_EXPRESSION matches by search, not by anchor — the suffix in parentheses is harmless.
  • return failed == 0 ? 0 : 1 with the [FAIL] lines for named device cases: so an unnamed invocation is caught by the regex and a named one by the return code. Belt and braces in the right direction — either half alone would leave a green path.

One thing worth checking before merge, because it is the same shape one level down: does anything else print a near-miss line that a regex-search could accidentally match, or that a future edit could make match? "test skipped: no device available" is a substring of the new text by construction, which is the intent, but it also means any row that prints it for a non-failure reason (e.g. a case list deliberately restricted with -o) would be failed by the same regex. If testing_suite.h's selection.skipped_device branch can fire when the user selected only host cases, the message should differ from the no-device-at-all case — otherwise the new regex converts a legitimate narrow selection into a red row.

And the standing question from my last review is now answered by the code rather than the prose: hrx-buffer-test and hrx-loom-jit-test do route through setup.run(...), so the family's two mechanisms are now exactly what the table says — a printed line plus a regex for all three, return 1 for the test-hrx-* files, and 77 only where a run cannot judge.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 6303a6f — and the last two commits change this from a corrected document to an enforced one, which is a better place to land than where my review left it.

What arrived since I reviewed 8795603

6303a6fe7  tests: hrx-backend-test prints the family's no-device line, so its …
475c6d9c6  tests: the external-window-view row fails on a missing device inst…

The table I read at 8795603 claimed two things about hrx-*-test — that it prints test skipped: no device available and that the registered FAIL_REGULAR_EXPRESSION turns the row red. hrx-backend-test did not print that line, which would have made the doc true for two of the three rows and silently false for the third — the failure mode where a table is right about the family and wrong about a member. These two commits close that, and the external-window-view row now fails rather than skipping.

That is the difference between a README and a contract. The doc still carries the rule I could not have guessed — "on ctest 4.2.3 an exit code of the row's SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION" — but it is now describing code that behaves as stated, rather than code that is about to.

Verified at the new head

  • both family rows still begin **Failed.** with the convention column intact;
  • SKIP_RETURN_CODE, FAIL_REGULAR_EXPRESSION and convention all still present in the README (6 hits);
  • clean, draft=false.

One note, since another lane took this over per ORCH

My earlier comment asked whether the READY at 205ad0d had read the "returns 0" row as satisfying the heading. That question is now moot: the code was made to match, so the disagreement was about a real gap rather than about wording. Worth recording because it is the good version of two reviewers disagreeing — the resolution was a commit, not a compromise sentence.

Nothing to change.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6303a6f: READY

The fix is the right one, and the comment says why better than my review did

-    return skipped ? 77 : 0;
+    if (skipped) {
+        std::printf("test skipped: no device available\n");
+        return 1;
+    }
+    return 0;

return 1, not "add the regex and hope". That is the only ordering that works, and the PR states the reason it is the only one:

returning 77 made ctest report the row Skipped … (measured on ctest 4.2.3: an exit code equal to SKIP_RETURN_CODE wins over FAIL_REGULAR_EXPRESSION)

So the FAIL_REGULAR_EXPRESSION added in CMakeLists.txt is belt-and-braces exactly as commented, and the load-bearing change is the exit code. If the PR had added only the regex, it would have been green forever and looked like a fix — which is the trap this whole document is about.

And the comment records a worse consequence than the one I raised

so the host cases above ran and their result was discarded — green for a required check

That is the part I missed when I called this "one row is the exception". I described a row that reports Skipped / green; the actual cost was that real host-side assertions executed and their verdict was thrown away. A skip that discards a passing result is indistinguishable from that result never having been checked — which is precisely the rc 0 over 267 skipped cases on #375, in one row. Worth having fixed rather than documented.

The second bug is the one that would have kept it broken

-        std::printf("device: skipped, no HRX devices available\n");
+        std::printf("test skipped: no device available\n");

The old string matched none of the family's patterns — so even after the exit code was corrected, a FAIL_REGULAR_EXPRESSION on the family wording would never have fired. Two independent reasons the row was silent, and the PR fixes both. The comment even cites where the canonical line lives (tests/testing_suite.h:255), so the next person copying it copies the right one.

77 is kept, and that is the taxonomy holding

77 stays for a row that legitimately cannot judge, which is not this one.

The rule is not "stop using 77" — it is "77 is for what cannot be judged". A missing device is not that, for a row whose host half is meaningful; a row that needs a model it does not have argues the other way, and the README table now has a row for that distinction ("any of the above, needing a model it does not have"). That third row is a better table than the two-row version I reviewed.

The README now describes what the code does

"One row used to be the exception and is now fixed rather than recorded" — and the table grew a convention column saying how each family makes the report fail rather than skip. That is the document I asked for at 205ad0d, arrived at by fixing the thing instead of tabulating around it.

Ready. This is the version that should merge.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Closing my own question from the last review: a narrow selection does not get a false red. Checked the code path.

tests/testing_suite.h:489   if (!has_device && test_case.requires_device && !include_unavailable_device_cases) { ++selection.skipped_device; continue; }
tests/testing_suite.h:576   if (selection.skipped_device != 0) { fprintf(stderr, "test skipped: no device available for %s (%zu device case(s))\n", …); if (selection.cases.empty()) return 0; }

skipped_device counts only cases that require a device, so:

  • -o picking a host-only case on a device-less box ⇒ skipped_device == 0 ⇒ no line printed ⇒ the new regex cannot match ⇒ the row passes. No false red.
  • -o picking a device case ⇒ the line prints ⇒ the row fails, which is the intent.
  • --list returns at :572-573 before the skipped_device branch, so listing cannot trip the regex either.

One asymmetry worth knowing rather than fixing: in the whole-suite path with some device cases and some host cases, the line prints and the run continues with selection.cases (the host half), so the row fails after executing the host cases rather than instead of them. That is the right order — the host results are still produced and visible in the log, and the row's verdict is "not a pass" — but a reader of a red row should know the host lines in that log are real results, not a partial run.

So the two mechanisms in the table now hold for all three hrx-*-test files, and hrx-backend-test is the one that needed the fix.

@bong-water-water-bong
bong-water-water-bong merged commit 148e7f3 into 1bit/hrx-vulkan-patched Nov 15, 2026
@bong-water-water-bong
bong-water-water-bong deleted the tests-hrx-no-device-convention branch November 15, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant