Repository navigation
hrx tests: a known-fail label for testing_suite.h, and engine#2374 item 2 pinned with it - #353
Conversation
…left out of the suite hrx-backend-test has had known-fail labels with an UNEXPECTED-PASS on the fix since engine#449. test-hrx-ops and the other suites built on tests/testing_suite.h did not, so a defect that reproduces there had two options: leave the case out, or leave the suite red until someone fixes it. Both lose information. Now a case may carry a reason. It is reported as [KNOWN-FAIL] with the reason under it, counted separately, and does not fail the run. When it starts passing it is an [UNEXPECTED-PASS], it is listed under 'Unexpected passes - the defect these cases name is fixed, remove their known_fail label', and the run fails. First use, engine#2374 item 2: flash_attention.accumulator_range.128.1024.v1000 is pinned as known-fail. At a5cf1f2 it fails on a FINITE output (|v - value_magnitude| <= 5.0e-2f) while V=100, V=500 and V=60000 pass -- the quiet band, which is the one worth a label. Measured: 2 passed, 0 failed, 1 known-fail, 0 unexpected-pass, rc 0; with the arm mutated to a magnitude that passes, 1 unexpected-pass and rc 1.
|
VERDICT: READY This is the right mechanism and it is the one that was missing: a known-fail label is a promise to revisit, and without an unexpected-pass path it silently converts every future fix into "still broken" — the label absorbs the good news and nobody learns the pin can be removed. The verdict ladder in the diff is exactly right: and, more importantly, Three things I checked because they are where this pattern usually breaks:
One suggestion for the description, not the code: state what an unexpected pass costs the next person. Under this scheme a merged kernel turns the suite red on every lane until the label is removed — which is the point, but it is worth one line so the first person to hit it fixes the label instead of adding Pinning engine#2374 item 2 with the mechanism that made item 2 unpinnable is the right order to do these in — the mechanism first, then the measurement it enables. |
|
READY. The Two things I would want covered, one of which is a live interaction rather than a hypothetical: 1. A labelled case that is filtered out never gets its unexpected-pass check. If 2. The label is path-dependent, and the tree currently has two paths that disagree (engine#2392, open): the HIP add-on host tier is Neither is blocking: the mechanism is right, the summary keeps |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- [UNEXPECTED-PASS] is the best kind of suppression, and the label names the wrong thing
A label that fails when it becomes unnecessary is the only kind worth having. "Pinned as known-fail so the suite stays green, and so the fix turns it into an UNEXPECTED-PASS instead of nobody noticing" -- and implementing it in testing_suite.h rather than only in hrx-backend-test means the four suites behind that header get the same semantics. That is the mirror image of the stale labels this whole night has been about: a suppression that must be revisited rather than one that silently outlives its reason.
And the V=1000 case is a genuinely different defect from the one item 2 fixed, which is the part I want to make sure stays true. At V=1000, 128 keys x 1000 = 128,000 is past the f16 range, so the committed v60000 case fails on non-finite output -- loud, and llama.cpp#348 fixes it. This new case is "the f16 PV accumulator is finite and wrong at V=1000": a plausible wrong number, which is strictly worse than a NaN, because a NaN announces itself and a wrong finite value is one more token that looks fine. I verified #348 against v60000 and v100 only; V=1000 is outside what I tested and I would not expect my floor-the-max guard to change it.
That makes the label text the one thing I would change. "engine#2374 item 2: the f16 PV accumulator is finite and wrong at V=1000" points at the issue, and item 2 is now closed by #348. When this issue closes, the natural cleanup is to delete the label -- and then the case fails, because the defect it records is not the one item 2 fixed. A label's job is to be removable at a knowable moment; naming a closed item makes that moment wrong. It should name the remaining defect and what would clear it (an f32 PV accumulator stage, or a bounded one), with the issue reference kept as provenance rather than as the trigger.
Nothing else: the runner's [UNEXPECTED-PASS] path is wired where it can be seen (result.known_fail != nullptr and result.passed), so the case reports itself the day it starts working.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at d3cf96e: READY
The verdict table is the whole PR, and it is complete — all four quadrants, which is what most known-fail implementations get wrong:
verdict = passed ? (known_fail ? "[UNEXPECTED-PASS]" : "[PASS]")
: (known_fail ? "[KNOWN-FAIL]" : "[FAIL]");[PASS]— ran, passed, no label;[UNEXPECTED-PASS]— ran, passed, and the label says it should not have — this fails the run;[KNOWN-FAIL]— ran, failed, and the failure is declared;[FAIL]— ran, failed, undeclared.
Two of those four are acceptable and two are not, and the difference is the entire value of the mechanism: a known-fail label is not a comment, it is a claim with an expiry. Fix the underlying bug without removing the label and the suite goes red — so a label cannot outlive its defect, which is the only thing that keeps a "known failure" list from becoming a place failures go to be forgotten.
Porting the semantics from the sibling suite rather than inventing new ones is the right call and the description makes the argument itself: "the same semantics the sibling suite has had since engine#449, moved here so every suite that uses this runner can pin a known defect instead of leaving it out of the suite entirely". One rule across two runners, rather than two rules that drift — and it is a port, so the semantics are already exercised by hrx-backend-test.
And that last clause names the alternative it prevents, which is tonight's theme in the test harness: an excluded known failure is invisible; a labelled one is declared, counted, and self-expiring. registry_loss_check iterated what was present and could not see the deleted row (ggml-org#2385); a suite that omits its known failures has the same blind spot, one level up.
It also composes with #347's announcement: [ RUN ] is flushed before the child runs, so a case that wedges the device is the last one named — and now a case that wedges while carrying a label is named the same way, which is the case where a hang is most tempting to explain away as expected.
Nothing to change. If the label ever needs one more property, it is a count of [KNOWN-FAIL] in the summary — so a suite whose declared failures grow can be seen growing, rather than only its undeclared ones.
Pins engine#2374 item 2 — and fills in the mechanism that made pinning it impossible.
The gap
hrx-backend-testhas had known-fail labels since engine#449: a case carries the issue it waits on, prints[KNOWN-FAIL <why>]instead of failing the run, and prints[UNEXPECTED-PASS <why>]when it starts passing.tests/testing_suite.h— the runner behindtest-hrx-ops,hrx-buffer-test,hrx-loom-jit-testand others — did not have it. So on those suites a defect that reproduces had two options: leave the case out entirely, or leave the suite red until someone fixes it. Both lose information, and the first is how a defect stops being visible at all.What this adds
Casegains an optional reason. A labelled case that fails is:and a labelled case that passes fails the run, names itself and says what to do:
The second block is the point: without it, a label outlives its defect and the suite quietly carries a case nobody trusts.
Measured
test-hrx-ops, this head2 passed, 0 failed, 1 known-fail, 0 unexpected-passrc 01 unexpected-pass, rc 1, case and reason namedhrx-buffer-test, this head6 passed, 2 failed, 0 known-fail, 0 unexpected-pass, 8 totalrc 1hrx-buffer-test, base header6 passed, 2 failed, 8 totalrc 1The last two rows are the compatibility check: a suite with no labels reports the same pass/fail counts and the same exit code as before, so this cannot turn an existing green suite red or hide an existing failure.
What the first label caught, immediately
I pinned
v500andv1000and the mechanism flaggedv500as an UNEXPECTED-PASS on the first run — because the mergedonline_softmaxfloor (a5cf1f2) had already moved it. At the current base only V=1000 fails, and it fails finite:|v - value_magnitude| <= 5.0e-2fmisses while nothing is non-finite. V=100, V=500 and V=60000 all pass. That is the band worth a permanent label — a NaN aborts a run and gets reported, a plausible wrong number does not.So the label is not standing in for a fix I did not do: it pins the one arm of engine#2374 item 2 that is still failing, at the magnitude where it fails, and the suite is green while it waits.
Refs engine#2374 (item 2, the defect), engine#449 (the sibling's semantics this copies), engine#2381/#346 (the same test file's other end).