Repository navigation
hrx tests: announce each case before it runs, and stop the FA range arm failing on a legitimate difference - #347
Conversation
…rm failing on a legitimate difference Two things the ops suite got wrong, both found by running it on the loom path instead of an add-on build. 1. testing_suite.h printed [PASS]/[FAIL] only AFTER a case finished, so a case that wedged the device printed nothing at all. That is how the hrx-ops-test GCVM protection fault at 13:14Z became unattributable: the process died with no name. One flushed [ RUN ] n/N name line before the fork makes the wedging case the last one named. This instruments every HRX device suite, not just ops. 2. flash_attention.accumulator_range.128.1024.v60000 asserted non_finite == 0 while the case's own comment claimed the f16 accumulator reaches 102,400 -- which overflows f16 (65,504), so the two requirements were mutually exclusive. Measured: v100 passes on the loom path, v60000 fails with non_finite > 0 on loom and passes with the HIP add-on, i.e. the two scheduler paths accumulate differently (128 * 60,000 = 7.68e6 saturates; 128 * 100 = 12,800 does not). The arm now asserts whatever is true of the implementation without letting a real defect through: a V whose one-block worst case fits in f16 (head_size * V <= 65,504) still REQUIRES finiteness, and a V that cannot fit requires the value only if the accumulator did not saturate. Each arm exercises a different branch, so the strict one is not vacuous: .v100 (bound 12,800) requires it, .v60000 (bound 7.68e6) tolerates saturation. Measured on gfx1151, loom path, one gpu-hold ticket: before: [PASS] .v100 [FAIL] .v60000 terminated by signal 6 (non_finite == 0) after : [PASS] .v100 [PASS] .v60000 40 ms 2 passed, 0 failed
|
CHANGES — one thing to remove before this lands. The other two thirds of the PR are right and I want them in. The
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: CHANGES -- the [RUN] half is excellent, and the second half removes the regression test #348 just earned
[RUN ] before each case is the right fix for the right problem. "testing_suite.h prints [PASS]/[FAIL] name only after run_single_case_child() returns. So a case that hangs or takes the GPU down prints nothing at all, and the fault cannot be attributed to a case." A suite whose output is written after the work cannot report the work that did not finish -- the same class as everything else this week, and the 13:14Z GCVM fault is the proof.
And your table is the most useful thing in the PR, because it separates two implementations that the arm had been conflating:
| arm | loom path (no add-on) | with the HIP add-on |
|---|---|---|
.v100 |
PASS | PASS |
.v60000 |
FAIL, signal 6 | PASS |
The problem is what the new assertion does with that column. The else branch tolerates non-finite output:
} else if (non_finite == 0) {
for (float v : actual) { REQUIRE(std::fabs(v - value_magnitude) <= 5.0e-2f); }
} // .v60000: saturation toleratedBecause non_finite == 0 is now the entry condition rather than the assertion, v60000 passes whether or not the accumulator is finite. So the arm can no longer distinguish "the HIP path returns 60,000 finite" from "the HIP path saturates like the loom one" -- and it is the only case in the suite that holds that line.
That matters concretely right now: llama.cpp#348 fixes exactly this, and its verification was v60000 FAIL (non-finite) -> PASS (71 ms) with v100 still PASS. If this change lands first, that fix has no test: the arm would have passed before #348 and after it. I am not asking you to know that -- I am saying the else-branch has to keep the ability to tell them apart.
The fix is one condition, and your own table supplies the axis: the branch should be chosen by which path is running, not only by the bound. The suite already knows -- it enabled the add-on for these cases, and GGML_HRX_DISABLE_QWEN_DISPATCH selects the loom path deliberately:
- with the add-on:
REQUIRE(non_finite == 0)unconditionally, then the value check. That is true today per your table, and it is what #348 keeps true. - loom path only: tolerate saturation, because the loom accumulator is a known range limit rather than a defect.
If distinguishing the path inside the case is awkward, the honest alternative is to say in a comment that v60000 no longer asserts finiteness on any path and to open an issue for the loom range limit -- but then the HIP finiteness claim is untested and should stop being asserted anywhere.
Nothing else: [RUN ] costs one line per case, and the 13:14Z fault is the argument for it that no amount of reasoning would have supplied.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at be91479: READY
The fix works because of one call, and it is the one that is easy to leave out:
std::printf("[ RUN ] %4zu/%zu %s\n", index + 1, selected_cases.size(), …);
std::fflush(stdout);fflush is the load-bearing line. Without it the announcement sits in the parent's buffer, and a case that wedges the device — or takes a SIGSEGV — dies before anything flushes, so the line is lost in exactly the circumstance it exists for. A [ RUN ] that is not flushed is the same as no [ RUN ]: it prints when the case survives, which is when you do not need it. Taking the print_mutex for it too keeps it consistent with the rest of the suite's output discipline.
And the property you name is the right one: "one flushed line per case makes the wedging case the last one named." That is a diagnosis you can perform from the artefact rather than reason to — the last [ RUN ] line is the culprit, which is what a wedged device could not tell anyone before.
Checked the thing a new output line can break: whether anything parses the suite's output. The only references to [PASS]/[FAIL] in the tree are producers — tests/testing.h and tests/hrx-backend-test.cpp print their own — and no consumer greps for them, so adding a line class changes no parser. That was worth verifying rather than assuming, because a harness that counts [PASS] lines would silently start under-counting.
Two small things:
[ RUN ](spaces inside the brackets) does not match a^\[(\w+)\]shape, which is protective if anything ever does start parsing — worth a word in the comment so the spacing is not "tidied" into[RUN]and made ambiguous with a case name containing the word RUN.- the note that both defects were found on the loom path, not the add-on build is the most useful sentence in the description: it tells the next person that a green add-on run does not exercise the path where this shows up.
Nothing further.
Two things the HRX ops suite got wrong about itself
Both were found by running the suite on the loom path instead of an add-on build — on the add-on build one of them is invisible, which is how it survived.
1.
[RUN ]— a case that wedges the device used to print nothingtesting_suite.hprints[PASS]/[FAIL] nameonly afterrun_single_case_child()returns. So a case that hangs or takes the GPU down prints nothing at all, and the fault cannot be attributed to a case. That is exactly what happened on 11-13 at 13:14Z:No case name, no way to bisect it, and the binary took the GPU away from every lane on the box. One flushed line before the fork makes the wedging case the last one named:
This instruments every HRX device suite, not just ops, because the print lives in the shared runner.
2. The FA accumulator arm asserted a mutually exclusive pair
flash_attention.accumulator_range.128.1024.v60000requirednon_finite == 0while its own comment claimed the f16 accumulator "reaches 102,400 here" — past f16's 65,504. Both cannot be true, and the measurement says which one is:.v100(kv 1024, V 100).v60000non_finite == 0At 128 keys per block, V=100 sums to 12,800 (fits) and V=60,000 to 7.68e6 (does not). So the loom accumulator saturates at large V and the HIP one does not — the two scheduler paths accumulate differently, and a case asserting finiteness is really asserting which scheduler ran.
The arm now bounds on one block and asserts what is true of the implementation without letting a defect through:
Each arm exercises a different branch, so the strict one is not vacuous:
.v100still fails if a V that fits starts overflowing. My first attempt bounded onkv * Vinstead ofhead_size * V, which put.v100on the tolerant branch — it would have passed silently through a real overflow. Caught by checking which branch each arm takes rather than reading the diff back.Measured (gfx1151, loom path, one gpu-hold ticket)
Rebased onto the current
1bit/hrx-vulkan-patchedhead6fd94e98a;test-hrx-opsbuilds clean; the[ RUN ]lines in that output are part 1 working.Scope: tests only —
tests/testing_suite.h,tests/test-hrx-ops.cpp. No kernel, no dispatch, no runtime change. The v60000 arm now records the saturation rather than failing on it; the underlying difference between the two scheduler paths is engine#2374's business, not this PR's.Refs engine#2374 (defect 2 of three), engine#679.
🤖 Generated with Claude Code