You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hrx: test-hrx-ops fa_prefill_full_cache.512 fails on the fork head (conflicting alternate value for the projection input) #2079
test-hrx-ops case fa_prefill_full_cache.512 in the llama.cpp fork aborts while it builds its plan. It is the only failing case of the suite (240 passed, 1 failed, 241 total).
plan status: conflicting alternate value for graph value 1 type 1 byte_count 2097152
tests/test-hrx-ops.cpp: requirement failed: plan.metadata.append(std::move(match.metadata), plan.status)
(The plan status: line is printed by a small change to the test helper in 1bit-MONSTER/llama.cpp#298; without it only the failed requirement shows.)
Measured
strixhalo, gfx1151, full amdclang Release build, one run each under gpu-hold, 2026-11-10:
fork head 2293f95 (kernels and dispatch untouched): fails as above.
6ec7c26 + llama.cpp#298: fails the same way.
So it is not caused by #298 and was there before it.
What the message says
Type 1 is F16 and 2097152 bytes is the f16 copy of the 512-token projection input. The case matches the three projection MUL_MATs one at a time and appends each match to its plan (run_qwen_full_cache_prefill_flash_attention_scheduling_case). Two of the matches each publish an f16 copy of the same input under a different plan value, and the second append is refused.
Not determined: whether the case hands out plan value ids differently from the real scheduler (a stale test), or the second match really should have found the first copy (a dispatch fault that the scheduler path happens not to hit). hrx-backend-test's scheduler-driven rows with shared projections pass, which points at the test, but I did not prove it.
Done when
The case passes, or it is corrected with the reason, on the fork head.
Cause found and fixed in 1bit-MONSTER/llama.cpp#299 (cl-hrxnan, ready for review at 75e1448). It is a fault in the test: match_dispatch_at_index() already appends the match to the plan and the case appended it a second time, from the moved-from match (same plan value 36, empty name). The dispatch is not at fault. With the fix the whole suite is 239 passed, 0 failed.
Determination: the case is wrong, not the dispatch. The failing sequence hands two matches the same coverage snapshot, so the second re-produces an f16 copy the first already made — and the
metadata refuses the duplicate. It is the issue's first hypothesis, with the chain in the source rather than inferred.
Both inputs the matcher uses to decide "has this already been produced?" — covered_nodes and next_plan_value — are snapshots taken at the call. The only thing that advances
either is append_match_to_plan:
So the flash match sees covered_nodeswithout anything the postprocess match covered, and the same next_plan_value — it therefore re-materialises the f16 copy of the shared projection
input that postprocess just planned. Both copies carry graph value 1, F16, 2,097,152 bytes, and each has its ownalternate_value id, so alternate_value_matches() (command-plan-metadata.cpp:20-23, which compares alternate_value among other fields) is false and append_alternate_value refuses the second (:78-88) with exactly
conflicting alternate value for graph value 1 type 1 byte_count 2097152
Why the scheduler never hits it
ggml/src/ggml-hrx/dispatch/dispatch-scheduler.cpp matches one node at a time and appends inside the loop — plan_.metadata.append(...) at :334, with next_plan_value computed per
iteration at :274,353. There is no window in which a match is built against a plan that does not yet contain its predecessor, so coverage is always current and the second match never
re-creates a copy. That is also why hrx-backend-test's scheduler-driven rows with shared projections pass while this hand-sequenced case fails — the reporter's observation, now with the
mechanism.
The correction
Move the append ahead of the next match, so the case follows the scheduler's ordering:
ggml::hrx::DispatchMatch postprocess_match;
match_dispatch_at_index(imported.graph, *registry, plan, covered_nodes, postprocess_index, postprocess_match);
append_match_to_plan(plan, postprocess_match, covered_nodes); // <- must precede the next match
ggml::hrx::DispatchMatch flash_match;
match_dispatch_at_index(..., flash_match);
with a one-line comment saying why (covered_nodes and next_plan_value are snapshots; the scheduler appends per node). The alternative — a monotonic counter shared across matches —
would also fix it but would make the hand-sequenced harness diverge further from the real one.
What I did not do
I did not build or run it: test-hrx-ops needs the full amdclang HRX stack, and this box is at 169 GB free against this lane's 175 GB floor for starting a build. So this is the
issue's second exit criterion — "corrected with the reason" — with the confirmation run (test-hrx-ops --cases fa_prefill_full_cache.512, or the one-line move plus a full build) left to
whoever has the build. If the append-order change does not make it pass, then the second hypothesis is live after all and the reported evidence here is the place to start.
Closing: fixed on main. The engine pin is 7a1f2b81c (#2092, merged 02:42Z), which contains the fix f76493c (llama.cpp#299): compare f76493c...7a1f2b81c is ahead 3, behind 0. With the fix test-hrx-ops ran 239 passed, 0 failed on the device (the PR's measurement); the case was a test fault, a match appended to its plan twice.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Cause found and fixed in 1bit-MONSTER/llama.cpp#299 (cl-hrxnan, ready for review at 75e1448). It is a fault in the test:
match_dispatch_at_index()already appends the match to the plan and the case appended it a second time, from the moved-from match (same plan value 36, empty name). The dispatch is not at fault. With the fix the whole suite is 239 passed, 0 failed.bong-water-water-bong commentedon Nov 11, 2026
Determination: the case is wrong, not the dispatch. The failing sequence hands two matches the same coverage snapshot, so the second re-produces an f16 copy the first already made — and the
metadata refuses the duplicate. It is the issue's first hypothesis, with the chain in the source rather than inferred.
Why the second match produces the copy again
tests/test-hrx-ops.cppat2293f95:Both inputs the matcher uses to decide "has this already been produced?" —
covered_nodesandnext_plan_value— are snapshots taken at the call. The only thing that advanceseither is
append_match_to_plan:And the case builds two matches before appending either (
:2294-2297):So the flash match sees
covered_nodeswithout anything the postprocess match covered, and the samenext_plan_value— it therefore re-materialises the f16 copy of the shared projectioninput that postprocess just planned. Both copies carry graph value 1, F16, 2,097,152 bytes, and each has its own
alternate_valueid, soalternate_value_matches()(command-plan-metadata.cpp:20-23, which comparesalternate_valueamong other fields) is false andappend_alternate_valuerefuses the second (:78-88) with exactlyWhy the scheduler never hits it
ggml/src/ggml-hrx/dispatch/dispatch-scheduler.cppmatches one node at a time and appends inside the loop —plan_.metadata.append(...)at:334, withnext_plan_valuecomputed periteration at
:274,353. There is no window in which a match is built against a plan that does not yet contain its predecessor, so coverage is always current and the second match neverre-creates a copy. That is also why
hrx-backend-test's scheduler-driven rows with shared projections pass while this hand-sequenced case fails — the reporter's observation, now with themechanism.
The correction
Move the append ahead of the next match, so the case follows the scheduler's ordering:
ggml::hrx::DispatchMatch postprocess_match; match_dispatch_at_index(imported.graph, *registry, plan, covered_nodes, postprocess_index, postprocess_match); append_match_to_plan(plan, postprocess_match, covered_nodes); // <- must precede the next match ggml::hrx::DispatchMatch flash_match; match_dispatch_at_index(..., flash_match);with a one-line comment saying why (
covered_nodesandnext_plan_valueare snapshots; the scheduler appends per node). The alternative — a monotonic counter shared across matches —would also fix it but would make the hand-sequenced harness diverge further from the real one.
What I did not do
I did not build or run it:
test-hrx-opsneeds the full amdclang HRX stack, and this box is at 169 GB free against this lane's 175 GB floor for starting a build. So this is theissue's second exit criterion — "corrected with the reason" — with the confirmation run (
test-hrx-ops --cases fa_prefill_full_cache.512, or the one-line move plus a full build) left towhoever has the build. If the append-order change does not make it pass, then the second hypothesis is live after all and the reported evidence here is the place to start.
bump llama.cpp to 7a1f2b81c: the prefill kernels stay finite above th…
bong-water-water-bong commentedon Nov 11, 2026
Closing: fixed on main. The engine pin is
7a1f2b81c(#2092, merged 02:42Z), which contains the fixf76493c(llama.cpp#299):compare f76493c...7a1f2b81cis ahead 3, behind 0. With the fixtest-hrx-opsran 239 passed, 0 failed on the device (the PR's measurement); the case was a test fault, a match appended to its plan twice.