Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions ggml/src/ggml-hrx/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,22 @@ ctest --test-dir build -L '^hrx$' --no-tests=error --parallel 2 --verbose
```

Use verbose mode to retain output from passing cases as well as failures. Consumers executing the discovered commands outside CTest must also honor the returned test properties, including timeout and failure regex.

### A missing HRX device fails, it does not skip

Two families of rows cover this backend. Both report a missing device the same way **by design**, and the
`convention` column says how each one makes that report *fail* rather than skip:

| family | rows | a run with no HRX device | convention |
|---|---|---|---|
| `hrx-*-test` | `hrx-buffer-test`, `hrx-backend-test`, `hrx-loom-jit-test` | **Failed.** Each prints `test skipped: no device available for <suite>` and returns **0**: `hrx-buffer-test` and `hrx-loom-jit-test` through the shared suite (`tests/testing_suite.h:255-257`), `hrx-backend-test` from its own device half (`tests/hrx-backend-test.cpp:16266`; it printed `device: skipped, no HRX devices available` until this PR, which the regex did not match, so that row **passed** with no device). The rows are registered with `FAIL_REGULAR_EXPRESSION "test skipped: no device available"` (`tests/CMakeLists.txt:695`), and that registration is what turns the row red | **regex** |
| `test-hrx-*` | `test-hrx-hadamard`, `test-hrx-mxfp4`, `test-hrx-loader-mapped`, … (14 files) | **Failed.** Prints `test skipped: no device available` and returns **1** (14 of 14 files); three of the rows also carry the same `FAIL_REGULAR_EXPRESSION` as belt and braces | **exit code** |
| any of the above, needing a model it does not have | `test-hrx-loader-mapped` without `LLAMACPP_TEST_MODELFILE` | **Skipped** (`77`). Nothing to measure, and the reason is in the message — the one case where a skip is the honest answer |

Either way, **an HRX-enabled build that finds no HRX device fails those rows rather than skipping them**. The rows exist to measure the device; a skipped row is green for a required check, so a skip bought nothing and hid the absence.

**One row used to be the exception and is now fixed rather than recorded**: `test-hrx-external-window-view` returned **77** from its no-device branch (`skipped ? 77 : 0`), so ctest reported it **Skipped / green** and the host cases it had just run were discarded. It now prints the same string and returns **1** like its siblings, and carries the same `FAIL_REGULAR_EXPRESSION`. `hrx-backend-test` was the second: it has its own device half rather than the shared suite, and its no-device line did not contain the registered string, so its row was green with no device; it now prints the family's string. Nothing else in this table has an exception.

Do not "fix" a red row back to `return 77`: on ctest 4.2.3 an exit code of the row's `SKIP_RETURN_CODE` wins over `FAIL_REGULAR_EXPRESSION` and the row is reported `***Skipped` with `100% tests passed` (measured; four shapes in engine#2526's discussion, llama.cpp#400). `77` is correct only for what a runner legitimately cannot judge, such as a row that needs `LLAMACPP_TEST_MODELFILE` and has none.

That precedence is ctest's rule, not this repo's, so it is the side to re-check first if the behaviour ever changes: a `cmake_minimum_required` bump or a ctest upgrade could flip the order, and the failure mode would be silence again -- a green row over a skipped test. The `test-hrx-*` family does not depend on the order (it fails on the exit code alone); the `hrx-*-test` family does.
4 changes: 3 additions & 1 deletion tests/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -568,7 +568,9 @@ if (TARGET ggml-hrx)
target_link_libraries(test-hrx-external-window-view PRIVATE ggml-hrx ggml hrx::hrx loomc::loomc)
target_include_directories(test-hrx-external-window-view PRIVATE ../ggml/include ../ggml/src ../ggml/src/ggml-hrx)
add_test(NAME test-hrx-external-window-view COMMAND test-hrx-external-window-view)
set_tests_properties(test-hrx-external-window-view PROPERTIES SKIP_RETURN_CODE 77)
# the device skip exits 1 (see the file); the regex is belt and braces, as on its siblings
set_tests_properties(test-hrx-external-window-view PROPERTIES SKIP_RETURN_CODE 77
FAIL_REGULAR_EXPRESSION "test skipped: no device available")

# the fused GATED_DELTA_NET pattern when the scheduler has already elided one of its layout aliases (the state
# RESHAPE visited before the L2_NORM root, engine #1925): host only
Expand Down
4 changes: 3 additions & 1 deletion tests/hrx-backend-test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16261,7 +16261,9 @@ int main(int argc, char ** argv) {
ggml_backend_load_all();
}
if (ggml_backend_hrx_get_device_count() == 0) {
std::printf("device: skipped, no HRX devices available\n");
// the family's no-device line (tests/testing_suite.h:255): the row's FAIL_REGULAR_EXPRESSION matches it and fails
// the row; "device: skipped, no HRX devices available" alone matched nothing, so the row passed with no device
std::printf("test skipped: no device available for hrx-backend-test (device half; the host cases above ran)\n");
for (const std::string & name : unmatched) {
std::printf("[FAIL] no case named '%s' (device cases cannot run without an HRX device)\n", name.c_str());
++failed;
Expand Down
10 changes: 9 additions & 1 deletion tests/test-hrx-external-window-view.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -202,5 +202,13 @@ int main() {

std::printf("test-hrx-external-window-view: %d failures\n", failures);
REQUIRE(failures == 0);
return skipped ? 77 : 0;
// A missing device fails this row, like its siblings (README, "A missing HRX device fails"): returning 77 made
// ctest report the row Skipped, so the host cases above ran and their result was discarded -- green for a
// required check (measured on ctest 4.2.3: an exit code equal to SKIP_RETURN_CODE wins over
// FAIL_REGULAR_EXPRESSION). 77 stays for a row that legitimately cannot judge, which is not this one.
if (skipped) {
std::printf("test skipped: no device available\n");
return 1;
}
return 0;
}
2 changes: 1 addition & 1 deletion tests/test-hrx-loader-mapped.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ int main(int argc, char ** argv) {
const char * path = argc > 1 ? argv[1] : std::getenv("LLAMACPP_TEST_MODELFILE");
if (path == nullptr || path[0] == '\0') {
std::printf("skipped: no model (argv[1] or LLAMACPP_TEST_MODELFILE)\n");
return 77; // ctest SKIP_RETURN_CODE, as test-hrx-external-window-view
return 77; // ctest SKIP_RETURN_CODE: no model is a row this runner cannot judge (README, device table)
}
llama_backend_init();
ggml_backend_dev_t hrx = nullptr;
Expand Down