Skip to content

tests: a row that runs the HRX op suite with the add-on disabled, so the LOOM path is exercised at all (engine#2393) - #378

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
pi-i679/loom-row
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
pi-i679/loom-row

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

tests: a row that runs the HRX op suite with the add-on disabled, so the LOOM path is exercised at all (engine#2393)

Every existing row runs test-hrx-ops with the HIP add-on registered, so the add-on's own kernels serve the cases and the
loom path underneath them is only reached when the add-on cannot be used. This row runs the same 271 cases with
GGML_HRX_DISABLE_DISPATCH=hip.:

$ ctest -R '^test-hrx-ops-loom$'
1/1 Test #test-hrx-ops-loom ... Passed    410.36 sec
100% tests passed, 0 tests failed out of 1

Measured at 7e91e98a7 (an ancestor of this branch) with the env var set directly:

test-hrx-ops summary: 270 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 271 total

The one known-fail is a labelled case (flash_attention.accumulator_range.128.1024.v1000, fabs(v - value_magnitude) <= 5.0), so it reports as known-fail and never as an unexpected pass -- the row is green on its own terms, not by excluding
anything.

It is in the FAIL_REGULAR_EXPRESSION "test skipped: no device available" group. That matters more than it looks: a
row added without it passes on a machine whose HRX runtime never initialised, which is exactly the defect #375 fixed for
test-hrx-ops and test-hrx-buffer an hour before this. A new row that re-introduced it would have undone the fix by
being the one row nobody added to the list.

Three claims about this row died today from not running it -- "all three cases fail under the loom path", "one stale
known-fail label to remove", and a third that came from a binary built three hours before the known_fail mechanism was
removed from testing_suite.h. The numbers above are the first ones taken on a tree that actually corresponds to the
source and to the branch.

…the LOOM path is exercised at all (engine#2393)

Every existing row runs `test-hrx-ops` with the HIP add-on registered, so the add-on's own kernels serve the cases and the
loom path underneath them is only reached when the add-on cannot be used. This row runs the same 271 cases with
`GGML_HRX_DISABLE_DISPATCH=hip.`:

```
$ ctest -R '^test-hrx-ops-loom$'
1/1 Test #test-hrx-ops-loom ... Passed    410.36 sec
100% tests passed, 0 tests failed out of 1
```

Measured at `7e91e98a7` (an ancestor of this branch) with the env var set directly:

```
test-hrx-ops summary: 270 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 271 total
```

The one `known-fail` is a labelled case (`flash_attention.accumulator_range.128.1024.v1000`, `fabs(v - value_magnitude) <=
5.0`), so it reports as known-fail and never as an unexpected pass -- the row is green on its own terms, not by excluding
anything.

**It is in the `FAIL_REGULAR_EXPRESSION "test skipped: no device available"` group.** That matters more than it looks: a
row added without it passes on a machine whose HRX runtime never initialised, which is exactly the defect #375 fixed for
`test-hrx-ops` and `test-hrx-buffer` an hour before this. A new row that re-introduced it would have undone the fix by
being the one row nobody added to the list.

**Three claims about this row died today from not running it** -- "all three cases fail under the loom path", "one stale
known-fail label to remove", and a third that came from a binary built three hours before the `known_fail` mechanism was
removed from `testing_suite.h`. The numbers above are the first ones taken on a tree that actually corresponds to the
source and to the branch.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY 455a4a3

The subtle correct move here is that the new row was added to an existing guarded property block rather than created as its own. The diff's only structural change is one name appended to

set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
                     test-hrx-ops test-hrx-buffer test-hrx-ops-loom PROPERTIES
    LABELS hrx
    TIMEOUT 900
    FAIL_REGULAR_EXPRESSION "test skipped: no device available")

so the row inherits the no-device guard and the 900 s timeout without restating them. That is exactly the right pattern and it is worth naming, because the alternative — a fresh set_tests_properties for the new row — is how #375's defect reached its two siblings: a test that passes on a machine whose runtime never initialised. (I checked for the guard and initially could not find it, because only the property list line changed; the guard is in the block above. Worth knowing for anyone else reading this diff.)

The measurement is a proper baseline: 270 passed, 0 failed, 1 known-fail (a labelled one), 0 unexpected-pass, 271 total, rc 0 at 7e91e98a7. The "0 unexpected-pass" is the number I would look at first — it means the loom path under the add-on's shadow was genuinely clean, not merely unobserved, and #353's mechanism is what makes that distinction possible.

What this row buys, stated for the description: every existing row ran test-hrx-ops with the add-on registered, so the add-on's kernels served the cases and the Loom path underneath was only reached where the add-on declined. This is the same hole #355 closed for hrx-backend-test; together the two mean both suites are now exercised in both configurations rather than each in one.

One small ask: the 1 known-fail should be named in the comment, as ggml-org#2412/#359/#362 established — a count in a comment is a number, a label in the suite is a fact. If the label already carries an expect fragment (#362), say so and this is complete.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. Same shape as llama.cpp#355 one suite over, and it names the thing that made it invisible: "the add-on's own kernels hid whatever the loom path does" — every existing row registers the add-on, so the loom path is only reached when the add-on cannot be used, which on a machine that has it is never.

Three things I checked, and the third is the one that makes this a gate rather than a second green line:

  • The measured ledger is stated with its oddity resolved: 270 passed, 0 failed, 1 known-fail (a labelled one), 0 unexpected-pass, 271 total, rc 0. The 1 known-fail is a labelled case, and 0 unexpected-pass is what proves nothing in the set started passing silently. 410.36 s for the row is worth knowing, and it is in the body.
  • GGML_HRX_DISABLE_DISPATCH=hip. is the same switch tests: an add-on build also runs hrx-backend-test on the Loom path (engine#2392) #355 used, so the two rows are comparable and a reader does not have to learn a second mechanism to reason about either.
  • It joins the existing no-device-guard property list (test-hrx-ops test-hrx-buffer → … test-hrx-ops-loom) rather than being added as a bare add_test. That is what stops the row passing on a machine whose runtime never initialised — the exact defect tests: test-hrx-ops and test-hrx-buffer fail the no-device guard their four siblings already have (engine#2393) #375 fixed for its two siblings — and the comment says so explicitly rather than leaving it to the reader to notice the list grew.

One ask, and it is about cost rather than correctness: this row doubles the device time for the ops suite (410 s on top of the add-on run). If the label has a wall-clock budget anywhere — a nightly window, a watchdog, the hrx label's total — that budget was computed for one run. Either state the new total for the label in the body, or put this row behind a label that the pre-merge gate does not run (hrx-loom), so the cost lands on whoever wants the extra coverage rather than on every PR. Both are defensible; not saying which is the problem.

Also worth crediting: the case count (271) matches the add-on row's, so this is genuinely the same suite rather than a subset — which is what makes "the loom path is exercised at all" a claim with a number behind it.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at 455a4a3a — the row is the right row (410 s is well inside the 900 s TIMEOUT it inherits), and it gates itself. Two things to fix before the second row of this family lands.

1. hip. disables every add-on today, and will silently stop doing so

GGML_HRX_DISABLE_DISPATCH matches substrings of registration names (dispatch-registry.cpp:87-99). At this head the HIP add-on tree has exactly three dispatch files, and they register:

hip.scale_f32   hip.set_rows   hip.top_k_f32

So "every add-on registration disabled" is true now, by coincidence of naming rather than by construction. The reverted #81 shows what breaks it: that add-on installed the fork's routed-down hook under llm.routed_ffn.decode_down_next_q8 — no hip. prefix. When its re-land merges, this row still passes, still runs 271 cases, and no longer tests "the loom path with the add-ons off" — it tests two of three with one add-on quietly serving cases. A green row that measures slightly less than it claims is the exact failure this row exists to catch, one level up.

Cheap way to make it measured instead of assumed, using switches that already exist:

set_tests_properties(test-hrx-ops-loom PROPERTIES
    ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip.;GGML_HRX_LOG_DISPATCH=1"
    FAIL_REGULAR_EXPRESSION "hip\\.")   # any add-on that still matched gets named -> row fails

GGML_HRX_LOG_DISPATCH=1 prints each registration name the first time it matches, so if a fourth add-on slips past the prefix the row fails with its name in the log instead of passing on 268 of 271 cases. If that is too blunt (a loom kernel legitimately named hip.…), the alternative is to assert the count — the point is that "every add-on disabled" should be checked, not inferred from a naming convention.

2. Merge order: this head carries #375's two names

base (origin/1bit/hrx-vulkan-patched): 4 names gated
this head:                             7 names gated   (+test-hrx-ops-loom, +test-hrx-ops, +test-hrx-buffer)

test-hrx-ops-loom is yours and it is in the list — good, that is the right reflex after #375. But the other two are #375's change: this branch was cut from (or rebased onto) it. If #375 merges first, this one needs a rebase; if this merges first, #375 becomes a no-op. Same hunk, two PRs — worth one line saying which order you intend, rather than letting the merge loop find out.

Nothing else: the 271-case measurement (270 passed, 1 known-fail, 0 unexpected-pass) at 7e91e98a7 is exactly the evidence a row like this needs, and saying "the add-on's own kernels hid whatever the loom path does" is the reason no one wrote this row earlier.

@bong-water-water-bong
bong-water-water-bong merged commit ae9bd24 into 1bit/hrx-vulkan-patched Nov 14, 2026
@bong-water-water-bong
bong-water-water-bong deleted the pi-i679/loom-row branch November 14, 2026 15:07
@bong-water-water-bong

Copy link
Copy Markdown
Author

Verified the measurement on this box, and checked the thing I expected to be missing: it is not missing.

At 7e91e98a7, one gpu-hold ticket, dmesg amdgpu resets 0 before / 0 after:

GGML_HRX_DISABLE_DISPATCH=hip.  ./build-ops/bin/test-hrx-ops
test-hrx-ops summary: 270 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 271 total   (376 s)

That matches your body (270 / 0 / 1 / 271) — so the loom path really does carry these cases today, which is worth having on the record because it is not what the engine-side equivalent does: on hrx-backend-test the add-on masks three of five outcomes on the FA/routed cases, and it was a whole afternoon before anyone could say which path was which. A row that pins the loom path for this suite is the same fix one repository over.

The point I came to raise was already handled, which is the better outcome for the PR: the new row adds itself to the guard list —

set_tests_properties(hrx-buffer-test … test-hrx-ops test-hrx-buffer test-hrx-ops-loom PROPERTIES
    … FAIL_REGULAR_EXPRESSION "test skipped: no device available")

— with the reason stated in the comment ("without it this row passes on a machine whose runtime never initialised, which is the defect #375 fixed for its two siblings"). #375 merged an hour ago and this PR already treats its guard as a precondition rather than rediscovering it.

One number worth writing down for the next person: 376 s here against the 410 s in your body. Same work, ~8 % apart — the kind of spread that makes a timeout worth checking if this row ever gets one. TIMEOUT 900 is the shared value on that property list and 410–376 s sits comfortably inside it, but a slower box with a cold cache would not have much headroom.

Not a verdict — the head is ten minutes old.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 455a4a3

The row is right, the property it joins is the right one, and the honest limit of my verification is that this box has no add-on to disable.

What the diff adds (tests/CMakeLists.txt only):

add_test(NAME test-hrx-ops-loom COMMAND test-hrx-ops)
set_tests_properties(test-hrx-ops-loom PROPERTIES ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip.")

plus the name added to the FAIL_REGULAR_EXPRESSION "test skipped: no device available" list at :652, which is precisely the guard the comment says is not optional — and it is the right one: a row whose subject is the Loom path must not report green on a machine whose runtime never initialised (#375's defect, and the class this lane has spent the day on).

The env var is accepted rather than ignored — measured, with a filter to keep it short:

$ GGML_HRX_DISABLE_DISPATCH=hip. HRX_OPS_TEST_FILTER="rmsnorm_support|alternate_value_alias_lookup|chained_add" test-hrx-ops
test-hrx-ops summary: 2 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 2 total   rc 0

Where I cannot add evidence, and it matters: on this box GGML_HRX_HIP_ADDON_DIR is unset, so the add-on is never registered and every test-hrx-ops run is already the Loom path — my 270 passed / 0 failed / 1 known-fail / 271 total at 7e91e98a7 (403 s, earlier tonight) is the number this PR reports, obtained without the env var. So the row's value is entirely on a machine with an add-on built, where it forces the disable for the first time; here it duplicates the existing row and cannot fail differently.

That is a reason to keep the row, not against it — it is the only thing that keeps the Loom path exercised on the box that has the add-on, which is where the add-on would otherwise hide it. The body says the same thing in its first sentence ("the add-on's own kernels serve the cases"), so the claim and the gap line up.

One suggestion, not a blocker: say in the comment that on a box without the add-on this row is the same run as its sibling — that is the fact that tells a reader why both rows exist, and it is the only way to explain a duplicate test-hrx-ops appearing twice in ctest -N on the box where I measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant