diff --git a/ggml/src/ggml-hrx/README.md b/ggml/src/ggml-hrx/README.md index e3152cfaf11..799670a049a 100644 --- a/ggml/src/ggml-hrx/README.md +++ b/ggml/src/ggml-hrx/README.md @@ -81,3 +81,22 @@ ctest --test-dir build -L '^hrx$' --no-tests=error --parallel 2 --verbose ``` Use verbose mode to retain output from passing cases as well as failures. Consumers executing the discovered commands outside CTest must also honor the returned test properties, including timeout and failure regex. + +### A missing HRX device fails, it does not skip + +Two families of rows cover this backend. Both report a missing device the same way **by design**, and the +`convention` column says how each one makes that report *fail* rather than skip: + +| family | rows | a run with no HRX device | convention | +|---|---|---|---| +| `hrx-*-test` | `hrx-buffer-test`, `hrx-backend-test`, `hrx-loom-jit-test` | **Failed.** Each prints `test skipped: no device available for ` and returns **0**: `hrx-buffer-test` and `hrx-loom-jit-test` through the shared suite (`tests/testing_suite.h:255-257`), `hrx-backend-test` from its own device half (`tests/hrx-backend-test.cpp:16266`; it printed `device: skipped, no HRX devices available` until this PR, which the regex did not match, so that row **passed** with no device). The rows are registered with `FAIL_REGULAR_EXPRESSION "test skipped: no device available"` (`tests/CMakeLists.txt:695`), and that registration is what turns the row red | **regex** | +| `test-hrx-*` | `test-hrx-hadamard`, `test-hrx-mxfp4`, `test-hrx-loader-mapped`, … (14 files) | **Failed.** Prints `test skipped: no device available` and returns **1** (14 of 14 files); three of the rows also carry the same `FAIL_REGULAR_EXPRESSION` as belt and braces | **exit code** | +| any of the above, needing a model it does not have | `test-hrx-loader-mapped` without `LLAMACPP_TEST_MODELFILE` | **Skipped** (`77`). Nothing to measure, and the reason is in the message — the one case where a skip is the honest answer | + +Either way, **an HRX-enabled build that finds no HRX device fails those rows rather than skipping them**. The rows exist to measure the device; a skipped row is green for a required check, so a skip bought nothing and hid the absence. + +**One row used to be the exception and is now fixed rather than recorded**: `test-hrx-external-window-view` returned **77** from its no-device branch (`skipped ? 77 : 0`), so ctest reported it **Skipped / green** and the host cases it had just run were discarded. It now prints the same string and returns **1** like its siblings, and carries the same `FAIL_REGULAR_EXPRESSION`. `hrx-backend-test` was the second: it has its own device half rather than the shared suite, and its no-device line did not contain the registered string, so its row was green with no device; it now prints the family's string. Nothing else in this table has an exception. + +Do not "fix" a red row back to `return 77`: on ctest 4.2.3 an exit code of the row's `SKIP_RETURN_CODE` wins over `FAIL_REGULAR_EXPRESSION` and the row is reported `***Skipped` with `100% tests passed` (measured; four shapes in engine#2526's discussion, llama.cpp#400). `77` is correct only for what a runner legitimately cannot judge, such as a row that needs `LLAMACPP_TEST_MODELFILE` and has none. + +That precedence is ctest's rule, not this repo's, so it is the side to re-check first if the behaviour ever changes: a `cmake_minimum_required` bump or a ctest upgrade could flip the order, and the failure mode would be silence again -- a green row over a skipped test. The `test-hrx-*` family does not depend on the order (it fails on the exit code alone); the `hrx-*-test` family does. diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index c39bcd685d9..8b785ca3310 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -568,7 +568,9 @@ if (TARGET ggml-hrx) target_link_libraries(test-hrx-external-window-view PRIVATE ggml-hrx ggml hrx::hrx loomc::loomc) target_include_directories(test-hrx-external-window-view PRIVATE ../ggml/include ../ggml/src ../ggml/src/ggml-hrx) add_test(NAME test-hrx-external-window-view COMMAND test-hrx-external-window-view) - set_tests_properties(test-hrx-external-window-view PROPERTIES SKIP_RETURN_CODE 77) + # the device skip exits 1 (see the file); the regex is belt and braces, as on its siblings + set_tests_properties(test-hrx-external-window-view PROPERTIES SKIP_RETURN_CODE 77 + FAIL_REGULAR_EXPRESSION "test skipped: no device available") # the fused GATED_DELTA_NET pattern when the scheduler has already elided one of its layout aliases (the state # RESHAPE visited before the L2_NORM root, engine #1925): host only diff --git a/tests/hrx-backend-test.cpp b/tests/hrx-backend-test.cpp index 8e96bac3127..7a3abf81e77 100644 --- a/tests/hrx-backend-test.cpp +++ b/tests/hrx-backend-test.cpp @@ -16261,7 +16261,9 @@ int main(int argc, char ** argv) { ggml_backend_load_all(); } if (ggml_backend_hrx_get_device_count() == 0) { - std::printf("device: skipped, no HRX devices available\n"); + // the family's no-device line (tests/testing_suite.h:255): the row's FAIL_REGULAR_EXPRESSION matches it and fails + // the row; "device: skipped, no HRX devices available" alone matched nothing, so the row passed with no device + std::printf("test skipped: no device available for hrx-backend-test (device half; the host cases above ran)\n"); for (const std::string & name : unmatched) { std::printf("[FAIL] no case named '%s' (device cases cannot run without an HRX device)\n", name.c_str()); ++failed; diff --git a/tests/test-hrx-external-window-view.cpp b/tests/test-hrx-external-window-view.cpp index 4dc5fb8ffee..30d987e66b7 100644 --- a/tests/test-hrx-external-window-view.cpp +++ b/tests/test-hrx-external-window-view.cpp @@ -202,5 +202,13 @@ int main() { std::printf("test-hrx-external-window-view: %d failures\n", failures); REQUIRE(failures == 0); - return skipped ? 77 : 0; + // A missing device fails this row, like its siblings (README, "A missing HRX device fails"): returning 77 made + // ctest report the row Skipped, so the host cases above ran and their result was discarded -- green for a + // required check (measured on ctest 4.2.3: an exit code equal to SKIP_RETURN_CODE wins over + // FAIL_REGULAR_EXPRESSION). 77 stays for a row that legitimately cannot judge, which is not this one. + if (skipped) { + std::printf("test skipped: no device available\n"); + return 1; + } + return 0; } diff --git a/tests/test-hrx-loader-mapped.cpp b/tests/test-hrx-loader-mapped.cpp index d3a5ba605a3..a35a78a648f 100644 --- a/tests/test-hrx-loader-mapped.cpp +++ b/tests/test-hrx-loader-mapped.cpp @@ -25,7 +25,7 @@ int main(int argc, char ** argv) { const char * path = argc > 1 ? argv[1] : std::getenv("LLAMACPP_TEST_MODELFILE"); if (path == nullptr || path[0] == '\0') { std::printf("skipped: no model (argv[1] or LLAMACPP_TEST_MODELFILE)\n"); - return 77; // ctest SKIP_RETURN_CODE, as test-hrx-external-window-view + return 77; // ctest SKIP_RETURN_CODE: no model is a row this runner cannot judge (README, device table) } llama_backend_init(); ggml_backend_dev_t hrx = nullptr;