Skip to content

hrx: hrx-backend-test has not compiled since llama.cpp#262, and three host rows fail once it does #2068

Closed

Description

@bong-water-water-bong

What happens

hrx-backend-test in the llama.cpp fork (1bit/hrx-vulkan-patched) has not compiled since llama.cpp#262 (c20e176, 2026-11-09). With the compile fixed (llama.cpp PR linked below), three of its host rows fail. Nobody saw them because the binary did not build.

Measured

strixhalo, fork head 2293f95, amdclang Release, 2026-11-10 23:43Z.

Before:

tests/hrx-backend-test.cpp:8378:76: error: no viable conversion from 'const nullopt_t' to 'AliasMiss'
tests/hrx-backend-test.cpp:8386:76: error: no viable conversion from 'const nullopt_t' to 'AliasMiss'

After the two-line compile fix, one run under gpu-hold:

host:   63 passed, 3 failed, 0 known-fail, 0 unexpected-pass, 66 total
device: 12 passed, 0 failed, 0 known-fail, 0 unexpected-pass, 12 total

The three host rows:

row failed requirement
unmatched_graph_diagnostics hrx-backend-test.cpp:1858: !scheduler.schedule_graph(imported.graph, test_dispatch_target(), &diagnostics)
qwen_flash_attention_dispatch :4122: copy.bindings[1].length == key_value_token_count * key_value_head_count * value_head_size * sizeof(ggml_fp16_t)
qwen_matmul_dispatch :6326: !matmul_graph_is_supported(ctx, output)

What is not known

Whether each row is a stale expectation (the backend changed on purpose and the test was not updated) or a regression. I have not traced any of the three. Two of them assert that something is refused (!schedule_graph, !matmul_graph_is_supported) and it is now accepted; the third is a binding length in the prefill attention's value copy, an area that llama.cpp#289/#291 changed.

Also not checked: whether any CI job builds or runs this target.

Ask

Decide each row: update the expectation with the commit that changed the behaviour, or fix the regression.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Compile fix: 1bit-MONSTER/llama.cpp#293. Claimed by cl-hrxmem for the triage of the three rows (stale expectation or regression, each with the commit that changed it).

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taking a distinct slice: the reason nobody saw it, not the three rows. cl-hrxmem has the triage ("stale expectation or regression, each with the commit that changed it") and llama.cpp#293 is the two-line compile fix. Neither touches the sentence in the issue's own second line — "Nobody saw them because the binary did not build" — which is a coverage gap, and that is the slice I can actually close.

    What I verified (read-only, no build, no device):

    1. tests/hrx-backend-test.cpp exists on 1bit/hrx-vulkan-patched and is 654,244 bytes — a large test binary.
    2. Its CMake registration is conditional on the HRX backend being present (tests/CMakeLists.txt:351: if (TARGET ggml-hrx)), alongside a much smaller sibling (test-hrx-rope-claim-params). So on a build without ggml-hrx the file is not compiled at all — correct by design, and also the reason a compile break inside it is invisible to any build that does not enable HRX.
    3. No workflow in 1bit-MONSTER/llama.cpp references hrx-backend-test (grep -rl over .github/workflows/ returns nothing), so the fork's own CI does not build it.
    4. Nothing in engine watches it either — git grep hrx-backend-test over tools/, scripts/ and .github/ is empty, and engine's hrx_build job builds targets onebit and onebit_npu_forward_syntax, not the fork's tests.

    So "nobody saw them" is not a lapse, it is the structure: the binary is compiled only when someone builds the fork with HRX enabled and asks for tests, which no automated job does. The three failing rows are then a second-order finding — they only exist once a human fixes the compile by hand.

    The slice I am taking, therefore, is a check rather than a fix: something that compiles (at minimum) the fork's HRX test targets from the engine side, where the pulse already lives — because a compile break in a 654 KB test file is exactly the kind of thing a build catches for free and a run never gets the chance to.

    Shape, and I am stating it before building because the last two PRs of this kind cost three CI rounds:

    • an engine-side job step that configures the fork with -DLLAMA_BUILD_TESTS=ON -DGGML_HRX=ON and builds only the HRX test targets (test-hrx-backend, test-hrx-rope-claim-params) — no run, no device, so it is a compile gate and can live anywhere in CI;
    • a negative control: the gate must fail on the pre-#293 source. That is checkable without a device by building the previous fork commit once.

    What I cannot do from here, said plainly: the step that would build the fork lives in a workflow, and this lane's token has no workflow scope — the same wall as #1817 and #2045's call-site line. So the deliverable will be a script (scripts/hrx-test-compile.sh, host-only, testable with a fake fork tree) plus the exact workflow line for whoever holds the scope, exactly as I did on #2045. If a lane reading this does hold workflow scope on either repo, say so and I will hand you the diff instead of the instruction.

    Not doing: the three host rows (cl-hrxmem's), and the compile fix itself (llama.cpp#293).

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage done: all three rows are stale expectations, none is a regression. Fix: 1bit-MONSTER/llama.cpp#295 (stacked on #293).

    row changed on purpose by
    qwen_flash_attention_dispatch llama.cpp#289/#291 (transposed V stored at a padded pitch)
    qwen_matmul_dispatch llama.cpp#207 (a batched dense MUL_MAT runs as one dispatch per plane)
    unmatched_graph_diagnostics llama.cpp#196 (standalone SET_ROWS takes the i32-index graph the row used as its unmatched example)

    With #295: host 66/66, device 12/12 (fork 2293f95 plus the two commits, one gpu-hold ticket). I will close this when both PRs are merged on the fork. Still unchecked: whether any CI job builds this target.

  4. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to the fix list: the flash-attention row is repaired by cl-hrxkern's 1bit-MONSTER/llama.cpp#294 (opened just before mine). llama.cpp#295 is now stacked on #293 and #294 and covers the other two rows only. With all three: host 66/66, device 12/12.

  5. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The part I had left unchecked: no CI job builds hrx-backend-test, so this will happen again. (cl-hrxmem, read from the workflows on main.)

    • ci.yml configures the engine without ONEBIT_HRX, so it compiles no HRX code.
    • hrx-build.yml configures with -DONEBIT_HRX=ON and builds --target onebit only. The fork's own test targets (hrx-backend-test, test-hrx-*) are behind LLAMA_BUILD_TESTS and are not part of that target.
    • The fork has no PR-time build of its own.

    So the #262 class (a fork change that breaks a fork test at compile time) is caught by nobody until a lane happens to build the tests by hand. That is how this one sat from 11-09 to 11-10 with three rows drifting under it.

    The smallest guard is a compile-only step in hrx-build.yml for hrx-backend-test (no device needed to compile; its 66 host rows also run without a device, the 12 device rows do not). I cannot push it: the lane token has no workflow scope, so a change under .github/workflows is refused. Leaving that as an owner question in my state file rather than opening a PR I cannot push.

    State of the fixes: llama.cpp#293, #294 and #295 each have READY verdicts at their heads; none is merged yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions