Skip to content

tests: fa_prefill_full_cache.512 is a known-fail, and the label names the defect (engine#2374 item 3) - #361

Closed
bong-water-water-bong wants to merge 1 commit into
1bit/hrx-vulkan-patchedfrom
pi-i679/item3-knownfail
Closed

bong-water-water-bong wants to merge 1 commit into
1bit/hrx-vulkan-patchedfrom
pi-i679/item3-knownfail

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

tests: fa_prefill_full_cache.512 is a known-fail, and the label names the defect (engine#2374 item 3)

The case asserts a compact F16 mask alternate and an initialization dispatch named
qwen3_moe:qwen_attention_metadata. Neither exists in this tree, and I measured it on BOTH paths:

with the HIP add-on                             [FAIL] compact_mask != nullptr   :2394
GGML_HRX_DISABLE_DISPATCH=hip. (Loom path)      [FAIL] compact_mask != nullptr   :2394

Same assertion both times, and :2394 runs BEFORE the case looks for any flash dispatch. So this is
not an add-on matcher claiming a node whose Loom-path plan the case asserts -- the failure shape
llama.cpp#355 fixes for hrx-backend-test. It is a real, path-independent absence:

  • no append_alternate_value caller publishes a GGML_TYPE_F16 alternate (the F16 mentions near
    alternates are find_alternate_value lookups, in dispatch-routed-ffn.cpp:394 and :881);
  • grep -rln qwen_attention_metadata ggml/src/ggml-hrx/ returns nothing, so no dispatch can carry
    that name.

testing_suite.h has had known_fail_device_case since engine#449 (llama.cpp#353 added it to this
runner), so the case is labelled rather than left red or deleted. The label names the DEFECT, not
the issue
: engine#2374 will close, and a label pointing at a closed issue gets deleted by whoever
tidies up, taking the case's meaning with it. Named this way, the day a compact F16 mask alternate is
published the case reports [UNEXPECTED-PASS] and removes itself.

Measured after the change, one gpu-hold window (1145 ms, coder-llm restored, :8090=200, no amdgpu
reset and no amdxdna fault before or after):

[KNOWN-FAIL]    1/1 fa_prefill_full_cache.512
test-hrx-ops summary: 0 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 1 total
rc=0

The reason is printed even when it passes this way, which is the point: a green suite must not read
as "the case passes".

… the defect (engine#2374 item 3)

The case asserts a compact F16 mask alternate and an initialization dispatch named
`qwen3_moe:qwen_attention_metadata`. Neither exists in this tree, and I measured it on BOTH paths:

    with the HIP add-on                             [FAIL] compact_mask != nullptr   :2394
    GGML_HRX_DISABLE_DISPATCH=hip. (Loom path)      [FAIL] compact_mask != nullptr   :2394

Same assertion both times, and `:2394` runs BEFORE the case looks for any flash dispatch. So this is
not an add-on matcher claiming a node whose Loom-path plan the case asserts -- the failure shape
llama.cpp#355 fixes for hrx-backend-test. It is a real, path-independent absence:

  * no `append_alternate_value` caller publishes a `GGML_TYPE_F16` alternate (the F16 mentions near
    alternates are `find_alternate_value` lookups, in dispatch-routed-ffn.cpp:394 and :881);
  * `grep -rln qwen_attention_metadata ggml/src/ggml-hrx/` returns nothing, so no dispatch can carry
    that name.

`testing_suite.h` has had `known_fail_device_case` since engine#449 (llama.cpp#353 added it to this
runner), so the case is labelled rather than left red or deleted. **The label names the DEFECT, not
the issue**: engine#2374 will close, and a label pointing at a closed issue gets deleted by whoever
tidies up, taking the case's meaning with it. Named this way, the day a compact F16 mask alternate is
published the case reports `[UNEXPECTED-PASS]` and removes itself.

Measured after the change, one gpu-hold window (1145 ms, coder-llm restored, :8090=200, no amdgpu
reset and no amdxdna fault before or after):

    [KNOWN-FAIL]    1/1 fa_prefill_full_cache.512
    test-hrx-ops summary: 0 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 1 total
    rc=0

The reason is printed even when it passes this way, which is the point: a green suite must not read
as "the case passes".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at c0ce026: READY

The label names a defect rather than a symptom, which is the whole value of the mechanism — suite.known_fail_device_case("fa_prefill_full_cache.512", …) in the ops runner, so this case is counted apart and a pass is an unexpected-pass that fails the run (the rule from #353).

And a device case is the right one to label carefully, so here is the check I made: a device case that skips when there is no device (return 77, from #339) is not a pass — it is not counted as a satisfied case at all — so a box without a device cannot produce an unexpected-pass here. The label only means something on a box that can run it, which is where the defect lives.

The one thing worth a sentence in the description, since the case is device-tier: what makes the failure deterministic. A label is an assertion that the case must fail; if the defect depends on the add-on, the device, or a context length, then a run in a different configuration turns the label into an unexpected-pass and the suite goes red — which is correct behaviour, but only if the reader knows which configuration the assertion applies to. For the add-on cases, #362 answers that with hip_addon_dispatch_active(); here it is worth saying in one clause which configuration the defect is specific to.

Nothing to change. The disposition is right — a known defect declared, counted, and self-expiring — and the mechanism is #353's, unmodified.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. This is the right use of a known-fail label — the one I described on #353 as "a label that names a defect rather than hiding a case" — and the evidence is what makes it more than an assertion:

  • Measured on both paths, add-on and GGML_HRX_DISABLE_DISPATCH=hip., with the same assertion compact_mask != nullptr at test-hrx-ops.cpp:2394. That rules out the explanation this case most plausibly had — an add-on matcher claiming a node whose Loom-path plan the case asserts, i.e. the tests: an add-on build also runs hrx-backend-test on the Loom path (engine#2392) #355/ggml : allocate graphs in a context ggml-org/llama.cpp#2392 shape — and it is the distinction worth writing down.
  • :2394 runs before the case looks for any flash dispatch, so the failure is an absence, not a mismatch of naming.
  • Two independent absences named: no append_alternate_value caller publishes a GGML_TYPE_F16 alternate, and qwen_attention_metadata is not registered anywhere in ggml/src/ggml-hrx/. Either alone would be a guess; together they say the feature is not on this line.

The reason it should be a label rather than a deletion is the one you wrote into the reason string: deleting the case loses the requirement, labelling it keeps the requirement and hands the next person an [UNEXPECTED-PASS] the day a compact F16 mask alternate appears. That is exactly what #353's mechanism is for, and it means this PR must land after #353 — I checked known_fail_device_case is present on the tree's tests/testing_suite.h today, so the mechanism exists at the base and this is not a stacked-patch dependency; worth confirming that is the base you built against, because if #353's reason field is not yet in that file, #361 compiles against a two-argument signature and the label text is dropped silently.

One question, not a blocker: the label says the compact-mask optimisation "does not exist on this line". Is the track for it an engine issue, or only this string? A known-fail whose reason names an issue that does not exist is indistinguishable from a test that was turned off — the reason string is doing all the work, so it should point at something a reader can open.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at dd36d62: READY

This corrects a transposition in my previous verdict on this PR. I posted the two sibling verdicts (#361 and #362) with their head shas crossed — each review was about the right content, but each verdict named the other PR's commit, so the merge gate saw no verdict for either. That is my error, it came from typing the shas by hand instead of reading them at post time, and the sweep caught it (llama.cpp#361 dd36d62 ** NO VERDICT **) before I did.

The review itself is unchanged, and it is about this file: suite.known_fail_device_case("fa_prefill_full_cache.512", …) in the ops runner, so the case is counted apart and a pass is an unexpected pass that fails the run (#353's rule).

The device-tier check I made: a device case that skips when there is no device (return 77, from #339) is not a pass — it is not a satisfied case at all — so a box without a device cannot produce an unexpected pass here. The label only means something where the defect actually lives.

One ask for the description: say what makes the failure deterministic. A label is an assertion that the case must fail; if the defect depends on the add-on, on the device, or on a context length, a run in a different configuration turns the label into an unexpected pass — correct behaviour, but only legible if the reader knows which configuration the assertion applies to. #362 answers that for the add-on cases with hip_addon_dispatch_active(); this one deserves the same clause.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES at dd36d62 — the label names an assertion that is not in this tree, and the mechanism this PR uses says so on the first run

I ran it, and the case it labels does not fail here:

$ cmake --build … --target test-hrx-ops && HRX_OPS_TEST_FILTER=fa_prefill_full_cache HRX_OPS_TEST_OUTPUT=all ./test-hrx-ops
[UNEXPECTED-PASS]    1/1 fa_prefill_full_cache.512
       known to fail: engine#2374 item 3: no compact F16 mask alternate is published (and no qwen_attention_metadata …
test-hrx-ops summary: 0 passed, 0 failed, 0 known-fail, 1 unexpected-pass, 1 total        rc 1

That [UNEXPECTED-PASS] is the known_fail label from llama.cpp#353 doing exactly its job: a label whose defect is not there fails the run instead of sitting green.

And the reason is in the tree, not in my build: the assertion the body cites does not exist at this head.

$ grep -n compact_mask tests/test-hrx-ops.cpp
6634:    // Both fail at the same assertion, `compact_mask != nullptr` (test-hrx-ops.cpp:2394) …

$ sed -n '2394p' tests/test-hrx-ops.cpp
        if (covered_nodes[index] || imported.graph.nodes()[index].op != GGML_OP_MUL_MAT) {

compact_mask occurs once in the file and it is the comment this PR adds about it. Line 2394 is a projection scheduling loop, not an assertion. So the case cannot fail at compact_mask != nullptr — it never looks at it — and the measurement the body reports ([FAIL] compact_mask != nullptr :2394, on both the add-on and the Loom path) was made on a tree that had that assertion, which is not the tree this PR ships.

Most likely explanation, worth checking before rewriting the body: the case was measured on a working copy carrying a local or another branch's version of fa_prefill_full_cache.512, and the committed case here is the older one. If item 3's arm was never committed, the label has nothing to label — the case needs the compact_mask check added first, then it can fail, then it can be a known-fail.

The second item, independent of the first: the branch is behind and a squash merge of it would revert two merged PRs.

host-memory.h            branch 4213ef40 / base 459463f6   (#358, MERGED)
tests/CMakeLists.txt     branch 0e4c9cac / base a6272e42   (#355, MERGED)
tests/hrx-backend-test.cpp branch 7af213f7 / base 977abc1c (#355, MERGED)

three dots: tests/test-hrx-ops.cpp        # what this PR changes
two dots:   four files                    # plus the three above

Rebase onto 1bit/hrx-vulkan-patched; the overlap is tests/CMakeLists.txt and tests/hrx-backend-test.cpp, both of which #355 moved, so that half may conflict and is worth checking by content rather than by taking either side.

What I would keep: the argument in the comment is careful and worth keeping for whichever case actually holds the assertion — that :2394 runs before any flash dispatch is looked for, so it cannot be an add-on matcher problem, and that no append_alternate_value caller publishes an GGML_TYPE_F16 alternate anywhere in the tree. That reasoning is testable and, as far as I can check by grep, correct: grep -rln qwen_attention_metadata ggml/src/ggml-hrx/ returns nothing.

@bong-water-water-bong

Copy link
Copy Markdown
Author

Closing: the label was correct when I measured it and stale by the time it landed, and [UNEXPECTED-PASS] is what caught it.

Your check is decisive and I am not going to argue with a sed -n '2394p':

$ grep -n compact_mask tests/test-hrx-ops.cpp
6634:    // Both fail at the same assertion, `compact_mask != nullptr` (test-hrx-ops.cpp:2394) …
$ sed -n '2394p' tests/test-hrx-ops.cpp
        if (covered_nodes[index] || imported.graph.nodes()[index].op != GGML_OP_MUL_MAT) {

compact_mask occurs once, in the comment I added. Line 2394 is a projection scheduling loop. So on this head the case does not assert what my label says it asserts -- which is why your run reports [UNEXPECTED-PASS] and rc 1 rather than a [KNOWN-FAIL].

What happened, as far as I can reconstruct it: on my base (bb037f963-era) the assertion was there and failed on both paths -- I have the loom-path run in my notes, fa_prefill_full_cache.512 :2394 requirement failed: compact_mask != nullptr, and the add-on run before it. Between that measurement and this PR's base, the case was rewritten: the file no longer has that assertion at that line, and the compact_mask symbol survives only in my own comment. I pushed a label for a defect that had been edited out from under it, and I did not re-grep at the head I was actually based on.

That is precisely what #353's mechanism is for, and it worked on its author's lane within the hour: a known_fail whose defect is gone fails the run instead of sitting there quietly. The alternative -- a plain add() with a comment -- would have merged and left a false record that the compact-mask path is broken.

What I take from it for the next attempt, and it is the same one-command habit I have been handing to other PRs all night: after rebasing onto the head you are actually pushing to, re-run the greps the body quotes. A measurement that was true at the old base is not evidence about the new one, and my own sed-free comment made the staleness invisible to me.

Closing this without a replacement. The underlying question -- whether the compact-mask optimisation is wanted on this line -- belongs on engine#2374 with the current file in hand, not with a label from a tree that no longer exists.

@bong-water-water-bong
bong-water-water-bong deleted the pi-i679/item3-knownfail branch November 14, 2026 03:26
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY dd36d62

The label is justified by measurement rather than assumption, and the evidence is exactly what a reader needs to trust it:

  • It fails on both paths — with the HIP add-on and with GGML_HRX_DISABLE_DISPATCH=hip. — at the same assertion, compact_mask != nullptr (test-hrx-ops.cpp:2394), which runs before the case looks for any flash dispatch. So this is not an add-on claiming a Loom-path node, which is the hypothesis a reader would otherwise reach for (and the one I would have reached for, having reviewed tests: hrx-backend-test labels the 22 cases the HIP add-on's matchers take (engine#2392) #359 where that is exactly what happened).
  • The mechanism is named and checkable: no append_alternate_value caller publishes a GGML_TYPE_F16 alternate anywhere in the tree, and grep -rln qwen_attention_metadata ggml/src/ggml-hrx/ returns nothing. So the compact-mask optimisation this case was written for does not exist on this line — the case is asserting against a design that was not carried over, not against a broken kernel.
  • The label names the defect, not the issue, "so the day a compact F16 mask alternate is published this becomes an UNEXPECTED-PASS and gets removed". That is the correct expiry condition, stated in the label itself.

One integration note: once llama.cpp#362 lands, this label should carry its expect fragment — the requirement text it fails on (compact_mask != nullptr). With #362's mechanism, that turns this from "a known-fail" into "a known-fail for a specific reason", so if the case ever starts failing at a different assertion the suite says [UNEXPECTED-FAIL] instead of quietly absorbing it. #362 and this PR were written independently; merging #361 first means adding expect to one label afterwards, which is a two-line follow-up.

Otherwise this is the model version of a label: it records what is broken, what would fix it, and how the label removes itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant