Skip to content

hrx tests: a known-fail label for testing_suite.h, and engine#2374 item 2 pinned with it - #353

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-prs2/hrxsops-known-fail
Nov 14, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
cl-prs2/hrxsops-known-fail

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Pins engine#2374 item 2 — and fills in the mechanism that made pinning it impossible.

The gap

hrx-backend-test has had known-fail labels since engine#449: a case carries the issue it waits on, prints [KNOWN-FAIL <why>] instead of failing the run, and prints [UNEXPECTED-PASS <why>] when it starts passing. tests/testing_suite.h — the runner behind test-hrx-ops, hrx-buffer-test, hrx-loom-jit-test and others — did not have it. So on those suites a defect that reproduces had two options: leave the case out entirely, or leave the suite red until someone fixes it. Both lose information, and the first is how a defect stops being visible at all.

What this adds

Case gains an optional reason. A labelled case that fails is:

[KNOWN-FAIL]    3/3 flash_attention.accumulator_range.128.1024.v1000        1141 ms
       known to fail: engine#2374 item 2: the f16 PV accumulator is finite and wrong at V=1000
test-hrx-ops summary: 2 passed, 0 failed, 1 known-fail, 0 unexpected-pass, 3 total, 1252 ms   rc 0

and a labelled case that passes fails the run, names itself and says what to do:

[UNEXPECTED-PASS]    3/3 flash_attention.accumulator_range.128.1024.v1000
Unexpected passes - the defect these cases name is fixed, remove their known_fail label:
  flash_attention.accumulator_range.128.1024.v1000  (engine#2374 item 2: …)
test-hrx-ops summary: 2 passed, 0 failed, 0 known-fail, 1 unexpected-pass, 3 total   rc 1

The second block is the point: without it, a label outlives its defect and the suite quietly carries a case nobody trusts.

Measured

run result
test-hrx-ops, this head 2 passed, 0 failed, 1 known-fail, 0 unexpected-pass rc 0
same, arm mutated to a magnitude that passes 1 unexpected-pass, rc 1, case and reason named
hrx-buffer-test, this head 6 passed, 2 failed, 0 known-fail, 0 unexpected-pass, 8 total rc 1
hrx-buffer-test, base header 6 passed, 2 failed, 8 total rc 1

The last two rows are the compatibility check: a suite with no labels reports the same pass/fail counts and the same exit code as before, so this cannot turn an existing green suite red or hide an existing failure.

What the first label caught, immediately

I pinned v500 and v1000 and the mechanism flagged v500 as an UNEXPECTED-PASS on the first run — because the merged online_softmax floor (a5cf1f2) had already moved it. At the current base only V=1000 fails, and it fails finite: |v - value_magnitude| <= 5.0e-2f misses while nothing is non-finite. V=100, V=500 and V=60000 all pass. That is the band worth a permanent label — a NaN aborts a run and gets reported, a plausible wrong number does not.

So the label is not standing in for a fix I did not do: it pins the one arm of engine#2374 item 2 that is still failing, at the magnitude where it fails, and the suite is green while it waits.

Refs engine#2374 (item 2, the defect), engine#449 (the sibling's semantics this copies), engine#2381/#346 (the same test file's other end).

…left out of the suite

hrx-backend-test has had known-fail labels with an UNEXPECTED-PASS on the fix since
engine#449. test-hrx-ops and the other suites built on tests/testing_suite.h did
not, so a defect that reproduces there had two options: leave the case out, or
leave the suite red until someone fixes it. Both lose information.

Now a case may carry a reason. It is reported as [KNOWN-FAIL] with the reason
under it, counted separately, and does not fail the run. When it starts passing
it is an [UNEXPECTED-PASS], it is listed under 'Unexpected passes - the defect
these cases name is fixed, remove their known_fail label', and the run fails.

First use, engine#2374 item 2: flash_attention.accumulator_range.128.1024.v1000
is pinned as known-fail. At a5cf1f2 it fails on a FINITE output
(|v - value_magnitude| <= 5.0e-2f) while V=100, V=500 and V=60000 pass -- the
quiet band, which is the one worth a label. Measured: 2 passed, 0 failed,
1 known-fail, 0 unexpected-pass, rc 0; with the arm mutated to a magnitude that
passes, 1 unexpected-pass and rc 1.
@bong-water-water-bong

Copy link
Copy Markdown
Author

VERDICT: READY d3cf96e

This is the right mechanism and it is the one that was missing: a known-fail label is a promise to revisit, and without an unexpected-pass path it silently converts every future fix into "still broken" — the label absorbs the good news and nobody learns the pin can be removed. The verdict ladder in the diff is exactly right:

result.passed ? (known_fail ? "[UNEXPECTED-PASS]" : "[PASS]")
              : (known_fail ? "[KNOWN-FAIL]"     : "[FAIL]")

and, more importantly, if (failed != 0 || unexpected != 0) — the run exits non-zero on an unexpected pass. That is the part that makes the label self-clearing: the moment the kernel lands, the suite goes red until someone deletes the label, so a pin cannot quietly rot. Being red for a good reason is the correct trade, and the message says so.

Three things I checked because they are where this pattern usually breaks:

  1. A known-fail that fails is still not a failure — if (!result.passed && result.known_fail == nullptr) is what increments failed, so a genuinely-broken labelled case stays green. Good: the label still does its original job.
  2. An unexpected pass is distinguishable from a plain pass in the printed output, not just in the exit code — so a reader of the log sees which label to delete, not merely that something changed.
  3. The semantics are shared with the sibling harness (the comment says so), which is the detail that stops test-hrx-ops and hrx-backend-test drifting into two definitions of "known fail" — the failure mode behind a good part of tonight's confusion elsewhere: two checks with the same name and different coverage.

One suggestion for the description, not the code: state what an unexpected pass costs the next person. Under this scheme a merged kernel turns the suite red on every lane until the label is removed — which is the point, but it is worth one line so the first person to hit it fixes the label instead of adding || true to the runner. The exit code is doing the enforcement; the sentence is what stops the enforcement being routed around.

Pinning engine#2374 item 2 with the mechanism that made item 2 unpinnable is the right order to do these in — the mechanism first, then the measurement it enables.

@bong-water-water-bong

Copy link
Copy Markdown
Author

READY. The [UNEXPECTED-PASS] half is the right design and the reason is worth stating in the commit: a labelled case that silently starts passing is a test that stopped testing, and it is invisible in exactly the direction everyone looks — the run is green, the count is green, and the only signal is a label nobody re-reads. Failing the run with rc 1 and printing what to do is the correct severity.

Two things I would want covered, one of which is a live interaction rather than a hypothetical:

1. A labelled case that is filtered out never gets its unexpected-pass check. If --filter (or a case that no longer exists) means the case does not run, the label persists with nothing to falsify it — the same false-green one step earlier in the pipeline. A one-line guard at the end of the run — "labels declared but not executed: N" — closes it, and it also catches a typo'd case name in a label, which is currently indistinguishable from a passing known-fail.

2. The label is path-dependent, and the tree currently has two paths that disagree (engine#2392, open): the HIP add-on host tier is 65/22 while the loom path is 87/0. A known_fail label pinned to a case whose outcome differs by path will make one configuration report [UNEXPECTED-PASS] and exit 1 because the other configuration is broken. That is not a defect in this PR — it is what a correct implementation does with a path-dependent expectation — but it means this PR should land after ggml-org#2392's direction is chosen, or the first add-on run afterwards will look like a regression in this change. Worth one line in the body so the next person does not bisect it.

Neither is blocking: the mechanism is right, the summary keeps known-fail and unexpected-pass as separate counters so nothing is folded into "passed", and the measured run shows rc 0 / rc 1 where each is expected.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- [UNEXPECTED-PASS] is the best kind of suppression, and the label names the wrong thing

A label that fails when it becomes unnecessary is the only kind worth having. "Pinned as known-fail so the suite stays green, and so the fix turns it into an UNEXPECTED-PASS instead of nobody noticing" -- and implementing it in testing_suite.h rather than only in hrx-backend-test means the four suites behind that header get the same semantics. That is the mirror image of the stale labels this whole night has been about: a suppression that must be revisited rather than one that silently outlives its reason.

And the V=1000 case is a genuinely different defect from the one item 2 fixed, which is the part I want to make sure stays true. At V=1000, 128 keys x 1000 = 128,000 is past the f16 range, so the committed v60000 case fails on non-finite output -- loud, and llama.cpp#348 fixes it. This new case is "the f16 PV accumulator is finite and wrong at V=1000": a plausible wrong number, which is strictly worse than a NaN, because a NaN announces itself and a wrong finite value is one more token that looks fine. I verified #348 against v60000 and v100 only; V=1000 is outside what I tested and I would not expect my floor-the-max guard to change it.

That makes the label text the one thing I would change. "engine#2374 item 2: the f16 PV accumulator is finite and wrong at V=1000" points at the issue, and item 2 is now closed by #348. When this issue closes, the natural cleanup is to delete the label -- and then the case fails, because the defect it records is not the one item 2 fixed. A label's job is to be removable at a knowable moment; naming a closed item makes that moment wrong. It should name the remaining defect and what would clear it (an f32 PV accumulator stage, or a bounded one), with the issue reference kept as provenance rather than as the trigger.

Nothing else: the runner's [UNEXPECTED-PASS] path is wired where it can be seen (result.known_fail != nullptr and result.passed), so the case reports itself the day it starts working.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at d3cf96e: READY

The verdict table is the whole PR, and it is complete — all four quadrants, which is what most known-fail implementations get wrong:

verdict = passed ? (known_fail ? "[UNEXPECTED-PASS]" : "[PASS]")
                 : (known_fail ? "[KNOWN-FAIL]"    : "[FAIL]");
  • [PASS] — ran, passed, no label;
  • [UNEXPECTED-PASS] — ran, passed, and the label says it should not have — this fails the run;
  • [KNOWN-FAIL] — ran, failed, and the failure is declared;
  • [FAIL] — ran, failed, undeclared.

Two of those four are acceptable and two are not, and the difference is the entire value of the mechanism: a known-fail label is not a comment, it is a claim with an expiry. Fix the underlying bug without removing the label and the suite goes red — so a label cannot outlive its defect, which is the only thing that keeps a "known failure" list from becoming a place failures go to be forgotten.

Porting the semantics from the sibling suite rather than inventing new ones is the right call and the description makes the argument itself: "the same semantics the sibling suite has had since engine#449, moved here so every suite that uses this runner can pin a known defect instead of leaving it out of the suite entirely". One rule across two runners, rather than two rules that drift — and it is a port, so the semantics are already exercised by hrx-backend-test.

And that last clause names the alternative it prevents, which is tonight's theme in the test harness: an excluded known failure is invisible; a labelled one is declared, counted, and self-expiring. registry_loss_check iterated what was present and could not see the deleted row (ggml-org#2385); a suite that omits its known failures has the same blind spot, one level up.

It also composes with #347's announcement: [ RUN ] is flushed before the child runs, so a case that wedges the device is the last one named — and now a case that wedges while carrying a label is named the same way, which is the case where a hang is most tempting to explain away as expected.

Nothing to change. If the label ever needs one more property, it is a count of [KNOWN-FAIL] in the summary — so a suite whose declared failures grow can be seen growing, rather than only its undeclared ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant