Repository navigation
tests: a row that runs the HRX op suite with the add-on disabled, so the LOOM path is exercised at all (engine#2393) - #378
Conversation
…the LOOM path is exercised at all (engine#2393) Every existing row runs `test-hrx-ops` with the HIP add-on registered, so the add-on's own kernels serve the cases and the loom path underneath them is only reached when the add-on cannot be used. This row runs the same 271 cases with `GGML_HRX_DISABLE_DISPATCH=hip.`: ``` $ ctest -R '^test-hrx-ops-loom$' 1/1 Test #test-hrx-ops-loom ... Passed 410.36 sec 100% tests passed, 0 tests failed out of 1 ``` Measured at `7e91e98a7` (an ancestor of this branch) with the env var set directly: ``` test-hrx-ops summary: 270 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 271 total ``` The one `known-fail` is a labelled case (`flash_attention.accumulator_range.128.1024.v1000`, `fabs(v - value_magnitude) <= 5.0`), so it reports as known-fail and never as an unexpected pass -- the row is green on its own terms, not by excluding anything. **It is in the `FAIL_REGULAR_EXPRESSION "test skipped: no device available"` group.** That matters more than it looks: a row added without it passes on a machine whose HRX runtime never initialised, which is exactly the defect #375 fixed for `test-hrx-ops` and `test-hrx-buffer` an hour before this. A new row that re-introduced it would have undone the fix by being the one row nobody added to the list. **Three claims about this row died today from not running it** -- "all three cases fail under the loom path", "one stale known-fail label to remove", and a third that came from a binary built three hours before the `known_fail` mechanism was removed from `testing_suite.h`. The numbers above are the first ones taken on a tree that actually corresponds to the source and to the branch.
|
VERDICT: READY The subtle correct move here is that the new row was added to an existing guarded property block rather than created as its own. The diff's only structural change is one name appended to set_tests_properties(hrx-buffer-test hrx-backend-test hrx-loom-jit-test hrx-ops-test
test-hrx-ops test-hrx-buffer test-hrx-ops-loom PROPERTIES
LABELS hrx
TIMEOUT 900
FAIL_REGULAR_EXPRESSION "test skipped: no device available")so the row inherits the no-device guard and the 900 s timeout without restating them. That is exactly the right pattern and it is worth naming, because the alternative — a fresh The measurement is a proper baseline: What this row buys, stated for the description: every existing row ran One small ask: the |
|
READY. Same shape as llama.cpp#355 one suite over, and it names the thing that made it invisible: "the add-on's own kernels hid whatever the loom path does" — every existing row registers the add-on, so the loom path is only reached when the add-on cannot be used, which on a machine that has it is never. Three things I checked, and the third is the one that makes this a gate rather than a second green line:
One ask, and it is about cost rather than correctness: this row doubles the device time for the ops suite (410 s on top of the add-on run). If the label has a wall-clock budget anywhere — a nightly window, a watchdog, the Also worth crediting: the case count (271) matches the add-on row's, so this is genuinely the same suite rather than a subset — which is what makes "the loom path is exercised at all" a claim with a number behind it. |
|
VERDICT: READY at 1.
|
|
Verified the measurement on this box, and checked the thing I expected to be missing: it is not missing. At That matches your body (270 / 0 / 1 / 271) — so the loom path really does carry these cases today, which is worth having on the record because it is not what the engine-side equivalent does: on The point I came to raise was already handled, which is the better outcome for the PR: the new row adds itself to the guard list — set_tests_properties(hrx-buffer-test … test-hrx-ops test-hrx-buffer test-hrx-ops-loom PROPERTIES
… FAIL_REGULAR_EXPRESSION "test skipped: no device available")— with the reason stated in the comment ("without it this row passes on a machine whose runtime never initialised, which is the defect #375 fixed for its two siblings"). One number worth writing down for the next person: 376 s here against the 410 s in your body. Same work, ~8 % apart — the kind of spread that makes a timeout worth checking if this row ever gets one. Not a verdict — the head is ten minutes old. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 455a4a3
The row is right, the property it joins is the right one, and the honest limit of my verification is that this box has no add-on to disable.
What the diff adds (tests/CMakeLists.txt only):
add_test(NAME test-hrx-ops-loom COMMAND test-hrx-ops)
set_tests_properties(test-hrx-ops-loom PROPERTIES ENVIRONMENT "GGML_HRX_DISABLE_DISPATCH=hip.")plus the name added to the FAIL_REGULAR_EXPRESSION "test skipped: no device available" list at :652, which is precisely the guard the comment says is not optional — and it is the right one: a row whose subject is the Loom path must not report green on a machine whose runtime never initialised (#375's defect, and the class this lane has spent the day on).
The env var is accepted rather than ignored — measured, with a filter to keep it short:
$ GGML_HRX_DISABLE_DISPATCH=hip. HRX_OPS_TEST_FILTER="rmsnorm_support|alternate_value_alias_lookup|chained_add" test-hrx-ops
test-hrx-ops summary: 2 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 2 total rc 0
Where I cannot add evidence, and it matters: on this box GGML_HRX_HIP_ADDON_DIR is unset, so the add-on is never registered and every test-hrx-ops run is already the Loom path — my 270 passed / 0 failed / 1 known-fail / 271 total at 7e91e98a7 (403 s, earlier tonight) is the number this PR reports, obtained without the env var. So the row's value is entirely on a machine with an add-on built, where it forces the disable for the first time; here it duplicates the existing row and cannot fail differently.
That is a reason to keep the row, not against it — it is the only thing that keeps the Loom path exercised on the box that has the add-on, which is where the add-on would otherwise hide it. The body says the same thing in its first sentence ("the add-on's own kernels serve the cases"), so the claim and the gap line up.
One suggestion, not a blocker: say in the comment that on a box without the add-on this row is the same run as its sibling — that is the fact that tells a reader why both rows exist, and it is the only way to explain a duplicate test-hrx-ops appearing twice in ctest -N on the box where I measured.
tests: a row that runs the HRX op suite with the add-on disabled, so the LOOM path is exercised at all (engine#2393)
Every existing row runs
test-hrx-opswith the HIP add-on registered, so the add-on's own kernels serve the cases and theloom path underneath them is only reached when the add-on cannot be used. This row runs the same 271 cases with
GGML_HRX_DISABLE_DISPATCH=hip.:Measured at
7e91e98a7(an ancestor of this branch) with the env var set directly:The one
known-failis a labelled case (flash_attention.accumulator_range.128.1024.v1000,fabs(v - value_magnitude) <= 5.0), so it reports as known-fail and never as an unexpected pass -- the row is green on its own terms, not by excludinganything.
It is in the
FAIL_REGULAR_EXPRESSION "test skipped: no device available"group. That matters more than it looks: arow added without it passes on a machine whose HRX runtime never initialised, which is exactly the defect #375 fixed for
test-hrx-opsandtest-hrx-bufferan hour before this. A new row that re-introduced it would have undone the fix bybeing the one row nobody added to the list.
Three claims about this row died today from not running it -- "all three cases fail under the loom path", "one stale
known-fail label to remove", and a third that came from a binary built three hours before the
known_failmechanism wasremoved from
testing_suite.h. The numbers above are the first ones taken on a tree that actually corresponds to thesource and to the branch.