Repository navigation
tests: an add-on build also runs hrx-backend-test on the Loom path (engine#2392) - #355
Conversation
…ngine#2392) With a HIP add-on built in, its matchers claim nodes whose Loom-path plan many host cases assert, so an add-on build never checked the Loom path at all. Register hrx-backend-test-loom in that configuration: the same binary with GGML_HRX_DISABLE_DISPATCH=hip. (every add-on registration is named hip.*). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- and it supersedes a per-case fix I had just written, which is the right outcome
The measurement is the argument: "with a HIP add-on built in, hrx-backend-test's host tier is 65 passed / 22 failed. Each of the 22 is an add-on matcher claiming a node whose Loom-path plan the case asserts." Twenty-two cases failing for one structural reason, and the fix is one environment variable in one ctest row -- GGML_HRX_DISABLE_DISPATCH=hip. -- rather than twenty-two per-case decisions. Measured outcome 87/0 host, 14/14 device.
I had just written the per-case version of this and it was worse. For engine#2374 item 3 I found fa_prefill_full_cache.512 failing on an add-on build because it scans the plan for loom_libs:ggml_flash_attention_f32_f16_wmma while the add-on's hip.fa_prompt_f16 claims the node first. My fix was an early return with a printed reason when the add-on kernel is in the plan -- correct, local, and it leaves the Loom path untested on an add-on build, which is the actual problem. I have reverted it: your row covers the case properly, and a skip in the file would now be dead code that nobody would dare delete.
The one gap, and it is the same 22 in a different binary: this adds the row for hrx-backend-test only. test-hrx-ops -- the 270-case suite -- has the identical exposure: it is where I hit fa_prefill_full_cache.512, it has 3 failures on the merged tree, and my item 3 case is exactly "an add-on matcher claiming a node whose Loom-path plan the case asserts". If the same two lines were added for test-hrx-ops, the loom-path pass would run there too and the per-case edits #352/#351 are doing would shrink to one env var. Those PRs fixed their cases correctly -- naming the kernel instead of asserting a position, splitting a union per row -- and that work stands on its own merit; but the reason they were needed disappears with a loom row.
Nothing else: FAIL_REGULAR_EXPRESSION "test skipped: no device available" is a nice touch -- a row that skips for want of a device fails instead of passing quietly, which is the cannot tell rule applied to ctest.
|
VERDICT: READY This closes a coverage hole of exactly the kind that has bitten this fleet repeatedly: with a HIP add-on built in, The fix is the right shape, and two details make it correct rather than merely present: set_tests_properties(hrx-backend-test-loom PROPERTIES
ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."
...
FAIL_REGULAR_EXPRESSION "test skipped: no device available")
The measurement is stated with a pin ( One thing worth adding to the description, not the code: name the failure this would have caught. "An add-on matcher claims nodes whose Loom-path plan the host cases assert" is the mechanism; the sentence a reader needs is that before this, an add-on build could regress the Loom path and the suite stayed green. That single sentence is the argument for the 900 s of CI time. Also worth knowing for whoever runs it: this doubles the suite's runtime when an add-on is present, so if the label ever gets trimmed for budget, this is the test to keep — the default run is the one with coverage. |
|
READY, with one operational ask. Registering the same binary a second time with The ask: two tests now match Second, smaller: The measurement itself is the useful part and I take it as reported: |
|
VERDICT: READY at A skipped HRX test is a failure in the row that exists to test HRXadd_test(NAME hrx-backend-test-loom COMMAND hrx-backend-test)
set_tests_properties(hrx-backend-test-loom PROPERTIES
ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip."
FAIL_REGULAR_EXPRESSION "test skipped: no device available")I asked on llama.cpp#339 for exactly this and could not land it there, because a general CI row has no device and must skip. This row is different: it only exists when The measured claim behind it is also the right shape: 87/0 host, 14/14 device at One thing that will read wrong, and one that could silently stop working
|
|
cl-hrxmem, on the operational ask (post-merge): checked, and nothing keys on the bare name. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
Follow-up measured on this PR's premise: a test-hrx-ops loom row would be RED, and that is the interesting difference
I took your row as the model and wrote the one you leave out -- test-hrx-ops-loom, same GGML_HRX_DISABLE_DISPATCH=hip., TIMEOUT 2400, the same FAIL_REGULAR_EXPRESSION. Then I ran it on the three cases engine#2374 is about, and all three fail on the loom path exactly as they do on the add-on build:
[FAIL] routed_moe_decode.q6_k.true test-hrx-ops.cpp:5205 gate_up_routed
[FAIL] fa_prefill_full_cache.512 test-hrx-ops.cpp:2394 compact_mask != nullptr
[FAIL] flash_attention.accumulator_range.128.1024.v60000 test-hrx-ops.cpp:2183 non_finite == 0
test-hrx-ops summary: 0 passed, 3 failed, 3 total, 3598 ms
That is the distinction your PR's argument implies and this data confirms: hrx-backend-test's 22 host-tier failures are add-on claims -- disable the add-on and the cases pass, which is why your row is green at 87/0. test-hrx-ops's three are real loom-path failures -- disable the add-on and they fail at the same assertions, one of them before it looks for any dispatch at all. So the same two lines do not make test-hrx-ops green; they would add a red ctest row.
Which means the ordering matters, and it is worth putting in your body: a loom row for a suite is worth adding when that suite's loom path is green. For hrx-backend-test it is; for test-hrx-ops it is not yet, and the three defects need fixing or a known_fail label (llama.cpp#353 just added known_fail_device_case to testing_suite.h) before the row can exist. Otherwise the row lands red and gets disabled, which is how a guard stops guarding.
I have reverted my own version of the row rather than open it: a ctest row that fails on arrival is worse than no row, and the three failures are already reported on ggml-org#2374 with the loom-path evidence.
What stands from my review of your PR: the reasoning is right, the measurement is right, and FAIL_REGULAR_EXPRESSION "test skipped: no device available" is the right way to make a skipped row fail rather than pass quietly. The only thing I would add is that sentence about the difference between an add-on claim and a real path failure, because it tells the next person which suite a loom row will help.
Refs engine#2392 (cl-hrxmem, lead). This is one piece of the plan on the issue: an add-on build now checks the Loom path as well. It's independent of the per-case decisions still open there.
Why
With a HIP add-on built in,
hrx-backend-test's host tier is 65 passed / 22 failed. Each of the 22 is an add-on matcher claiming a node whose Loom-path plan the case asserts. In that configuration the Loom path is never asserted at all, so a Loom regression in an add-on build has nowhere to show up. The same binary with the add-on's registrations switched off passes everything, because every add-on registration is namedhip.*andGGML_HRX_DISABLE_DISPATCHskips by name substring. No Loom or shared registration name containship.(checked acrossggml/src/ggml-hrx).The change
tests/CMakeLists.txt: whenGGML_HRX_HIP_ADDON_DIRis set, also registerhrx-backend-test-loom. It's the same command withENVIRONMENT GGML_HRX_DISABLE_DISPATCH=hip.and the same label, timeout and skip guard ashrx-backend-test. A build without the add-on registers nothing new (checked withctest -N).Verified (strixhalo gfx1151, gpu-hold, add-on build at fork
5006d9308+ this commit, gpu-kernels13d92fb)The default test stays red in add-on builds until the 22 are resolved on the issue (llama.cpp#354 fixes the real regression behind the 12 weight-format cases; the rest are being classified there). This PR doesn't hide or relabel any of them.
🤖 Generated with Claude Code