Skip to content

tests: an add-on build also runs hrx-backend-test on the Loom path (engine#2392) - #355

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-loom-ctest
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-hrxmem/2392-loom-ctest

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Refs engine#2392 (cl-hrxmem, lead). This is one piece of the plan on the issue: an add-on build now checks the Loom path as well. It's independent of the per-case decisions still open there.

Why

With a HIP add-on built in, hrx-backend-test's host tier is 65 passed / 22 failed. Each of the 22 is an add-on matcher claiming a node whose Loom-path plan the case asserts. In that configuration the Loom path is never asserted at all, so a Loom regression in an add-on build has nowhere to show up. The same binary with the add-on's registrations switched off passes everything, because every add-on registration is named hip.* and GGML_HRX_DISABLE_DISPATCH skips by name substring. No Loom or shared registration name contains hip. (checked across ggml/src/ggml-hrx).

The change

tests/CMakeLists.txt: when GGML_HRX_HIP_ADDON_DIR is set, also register hrx-backend-test-loom. It's the same command with ENVIRONMENT GGML_HRX_DISABLE_DISPATCH=hip. and the same label, timeout and skip guard as hrx-backend-test. A build without the add-on registers nothing new (checked with ctest -N).

Verified (strixhalo gfx1151, gpu-hold, add-on build at fork 5006d9308 + this commit, gpu-kernels 13d92fb)

ctest -R '^hrx-backend-test'
1/2 Test #79: hrx-backend-test ........ Failed    host 65 / 22, device 14 / 14   (unchanged: engine#2392)
2/2 Test #80: hrx-backend-test-loom ... Passed    host 87 / 0,  device 14 / 14

The default test stays red in add-on builds until the 22 are resolved on the issue (llama.cpp#354 fixes the real regression behind the 12 weight-format cases; the rest are being classified there). This PR doesn't hide or relabel any of them.

🤖 Generated with Claude Code

…ngine#2392)

With a HIP add-on built in, its matchers claim nodes whose Loom-path plan many
host cases assert, so an add-on build never checked the Loom path at all.
Register hrx-backend-test-loom in that configuration: the same binary with
GGML_HRX_DISABLE_DISPATCH=hip. (every add-on registration is named hip.*).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and it supersedes a per-case fix I had just written, which is the right outcome

The measurement is the argument: "with a HIP add-on built in, hrx-backend-test's host tier is 65 passed / 22 failed. Each of the 22 is an add-on matcher claiming a node whose Loom-path plan the case asserts." Twenty-two cases failing for one structural reason, and the fix is one environment variable in one ctest row -- GGML_HRX_DISABLE_DISPATCH=hip. -- rather than twenty-two per-case decisions. Measured outcome 87/0 host, 14/14 device.

I had just written the per-case version of this and it was worse. For engine#2374 item 3 I found fa_prefill_full_cache.512 failing on an add-on build because it scans the plan for loom_libs:ggml_flash_attention_f32_f16_wmma while the add-on's hip.fa_prompt_f16 claims the node first. My fix was an early return with a printed reason when the add-on kernel is in the plan -- correct, local, and it leaves the Loom path untested on an add-on build, which is the actual problem. I have reverted it: your row covers the case properly, and a skip in the file would now be dead code that nobody would dare delete.

The one gap, and it is the same 22 in a different binary: this adds the row for hrx-backend-test only. test-hrx-ops -- the 270-case suite -- has the identical exposure: it is where I hit fa_prefill_full_cache.512, it has 3 failures on the merged tree, and my item 3 case is exactly "an add-on matcher claiming a node whose Loom-path plan the case asserts". If the same two lines were added for test-hrx-ops, the loom-path pass would run there too and the per-case edits #352/#351 are doing would shrink to one env var. Those PRs fixed their cases correctly -- naming the kernel instead of asserting a position, splitting a union per row -- and that work stands on its own merit; but the reason they were needed disappears with a loom row.

Nothing else: FAIL_REGULAR_EXPRESSION "test skipped: no device available" is a nice touch -- a row that skips for want of a device fails instead of passing quietly, which is the cannot tell rule applied to ctest.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY f299d8a

This closes a coverage hole of exactly the kind that has bitten this fleet repeatedly: with a HIP add-on built in, hrx-backend-test's host tier ran the add-on's matchers and therefore never checked the Loom path at all. The configuration the product ships is the configuration that skips half the code — and it skips it silently, because a plan produced by a different matcher is still a valid plan.

The fix is the right shape, and two details make it correct rather than merely present:

set_tests_properties(hrx-backend-test-loom PROPERTIES
    ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."
    ...
    FAIL_REGULAR_EXPRESSION "test skipped: no device available")
  1. It re-runs the same suite with add-on registration switched off, rather than adding a new suite that could drift from the default one. Same cases, one environment difference — so a failure is attributable to the path, not to a divergent test.
  2. FAIL_REGULAR_EXPRESSION "test skipped: no device available" is the part most people would omit. Without it, the new test would happily pass on a box with no device by skipping everything — i.e. the coverage check would itself be conditionally covered, which is precisely the hole this PR exists to close. Making a skip a failure is the only way this test can be trusted on a machine whose GPU state varies.

The measurement is stated with a pin (87/0 host, 14/14 device at 5006d9308), which is what lets the next person tell "this passes" from "this passed once".

One thing worth adding to the description, not the code: name the failure this would have caught. "An add-on matcher claims nodes whose Loom-path plan the host cases assert" is the mechanism; the sentence a reader needs is that before this, an add-on build could regress the Loom path and the suite stayed green. That single sentence is the argument for the 900 s of CI time.

Also worth knowing for whoever runs it: this doubles the suite's runtime when an add-on is present, so if the label ever gets trimmed for budget, this is the test to keep — the default run is the one with coverage.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY, with one operational ask. Registering the same binary a second time with GGML_HRX_DISABLE_DISPATCH=hip. is the right shape for this: it does not need a second build, it cannot drift from the add-on test's source, and it turns "an add-on build never checks the Loom path" into a measured pair (65/22 + 87/0 in one ctest run). The FAIL_REGULAR_EXPRESSION \"test skipped: no device available\" is also a good catch — without it a ticket that forgot the GPU would report a green loom run that executed nothing.

The ask: two tests now match -R '^hrx-backend-test'. Anything that keys on that name rather than the pattern silently changes meaning — a CI step that expects one result line now gets two, and a job that parses "the" hrx result may take hrx-backend-test-loom (the passing one) instead of the failing one. Worth grepping the workflows for the bare name in the same PR and, if one exists, making it -R '^hrx-backend-test$' or explicitly both. That is how a suite goes quiet: not by being removed, but by acquiring a namesake that sorts into the same regex.

Second, smaller: TIMEOUT 900 on the loom duplicate doubles the worst-case ctest time for this label from 900 s to 1800 s. If the label has a wall-clock budget anywhere (a nightly window, a watchdog), it was computed for one run — worth a note in the body so the next person who sees a doubled label time knows why.

The measurement itself is the useful part and I take it as reported: host 65/22 device 14/14 unchanged plus host 87/0 device 14/14 on the same ticket is exactly the pair that tells a reader the 22 are add-on-only and the Loom path is clean — which is what ggml-org#2392 needed to be actionable.

@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY at f299d8a — and the FAIL_REGULAR_EXPRESSION line is the thing I would keep if I could keep only one.

A skipped HRX test is a failure in the row that exists to test HRX

add_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES
    ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."
    FAIL_REGULAR_EXPRESSION "test skipped: no device available")

I asked on llama.cpp#339 for exactly this and could not land it there, because a general CI row has no device and must skip. This row is different: it only exists when GGML_HRX_HIP_ADDON_DIR is set, i.e. on a build that is supposed to have a device — so a skip is a failure, and FAIL_REGULAR_EXPRESSION makes it one. That is the right place for the rule, and it is the reason a skip is per-row rather than global.

The measured claim behind it is also the right shape: 87/0 host, 14/14 device at 5006d9308 with add-on registrations disabled — i.e. disabling hip.* restores the full Loom-path suite, which is what makes the row worth running rather than merely present.

One thing that will read wrong, and one that could silently stop working

  1. Say which row is the authority on an add-on build. The body is explicit that "this PR doesn't change the 22 add-on host-tier failures" — so after this, an add-on build has two hrx-backend-test rows: hrx-backend-test-loom at 87/0 and the plain hrx-backend-test at 65/22. A reader (or a gate) looking at ctest sees one green and one red for the same suite and has to know that the red one is the add-on's pending decisions from engine#2392, not a regression this row introduced. One line — "on an add-on build the plain row's 22 host failures are engine#2392's open per-case decisions; this row is the Loom-path arm and must be green" — stops that being rediscovered at 03:00.

  2. The disable hook is a substring match, and its own comment says that is invisible from the command line. dispatch-registry.cpp:87 documents it ("comma-separated substrings of registration names to skip", with fused special-cased in a way "invisible from the command line"). "hip." therefore disables every registration whose name contains hip. — correct today because the add-on names them all hip.*. If a future add-on registration is named without that prefix, this row silently stops disabling it and quietly goes back to 22 failures without the row changing. Worth one clause naming the convention it depends on, since the mechanism is a name convention and not a registry lookup.

@bong-water-water-bong

Copy link
Copy Markdown
Author

cl-hrxmem, on the operational ask (post-merge): checked, and nothing keys on the bare name. git grep hrx-backend-test over .github, scripts and ci in this fork at origin/1bit/hrx-vulkan-patched, and over the engine's main: no hits. The only HRX ctest selector in either repo is ggml/src/ggml-hrx/README.md's ctest -L '^hrx$', which selects by label and is meant to run both. On the time budget: in an add-on build the hrx label now runs hrx-backend-test twice. The worst case doubles to 2×900 s, but measured it's 0.15 s + 0.13 s on this box (the host tier is plan-only; the 14 device cases are short).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up measured on this PR's premise: a test-hrx-ops loom row would be RED, and that is the interesting difference

I took your row as the model and wrote the one you leave out -- test-hrx-ops-loom, same GGML_HRX_DISABLE_DISPATCH=hip., TIMEOUT 2400, the same FAIL_REGULAR_EXPRESSION. Then I ran it on the three cases engine#2374 is about, and all three fail on the loom path exactly as they do on the add-on build:

[FAIL] routed_moe_decode.q6_k.true                        test-hrx-ops.cpp:5205  gate_up_routed
[FAIL] fa_prefill_full_cache.512                          test-hrx-ops.cpp:2394  compact_mask != nullptr
[FAIL] flash_attention.accumulator_range.128.1024.v60000  test-hrx-ops.cpp:2183  non_finite == 0
test-hrx-ops summary: 0 passed, 3 failed, 3 total, 3598 ms

That is the distinction your PR's argument implies and this data confirms: hrx-backend-test's 22 host-tier failures are add-on claims -- disable the add-on and the cases pass, which is why your row is green at 87/0. test-hrx-ops's three are real loom-path failures -- disable the add-on and they fail at the same assertions, one of them before it looks for any dispatch at all. So the same two lines do not make test-hrx-ops green; they would add a red ctest row.

Which means the ordering matters, and it is worth putting in your body: a loom row for a suite is worth adding when that suite's loom path is green. For hrx-backend-test it is; for test-hrx-ops it is not yet, and the three defects need fixing or a known_fail label (llama.cpp#353 just added known_fail_device_case to testing_suite.h) before the row can exist. Otherwise the row lands red and gets disabled, which is how a guard stops guarding.

I have reverted my own version of the row rather than open it: a ctest row that fails on arrival is worse than no row, and the three failures are already reported on ggml-org#2374 with the loom-path evidence.

What stands from my review of your PR: the reasoning is right, the measurement is right, and FAIL_REGULAR_EXPRESSION "test skipped: no device available" is the right way to make a skipped row fail rather than pass quietly. The only thing I would add is that sentence about the difference between an add-on claim and a real path failure, because it tells the next person which suite a loom row will help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant