Skip to content

hrx tests: announce each case before it runs, and stop the FA range arm failing on a legitimate difference - #347

Merged
bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
fa-acc-bound
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
1bit/hrx-vulkan-patchedfrom
fa-acc-bound

Conversation

@bong-water-water-bong

Copy link
Copy Markdown

Two things the HRX ops suite got wrong about itself

Both were found by running the suite on the loom path instead of an add-on build — on the add-on build one of them is invisible, which is how it survived.

1. [RUN ] — a case that wedges the device used to print nothing

testing_suite.h prints [PASS]/[FAIL] name only after run_single_case_child() returns. So a case that hangs or takes the GPU down prints nothing at all, and the fault cannot be attributed to a case. That is exactly what happened on 11-13 at 13:14Z:

Process hrx-ops-test pid 1729304
 in page starting at address 0x00007fb9780f1000  from client 10
 GCVM_L2_PROTECTION_FAULT_STATUS:0x00B01031   (repeating -> "Failed to evict queue 1" -> MODE2 -> device wedged)

No case name, no way to bisect it, and the binary took the GPU away from every lane on the box. One flushed line before the fork makes the wedging case the last one named:

[ RUN  ]    1/4 postops.q6_k.add_f32_f32_wmma.5.4160.true.8192 + 256
[FAIL]    1/4 postops.q6_k.add_f32_f32_wmma.5.4160.true.8192 + 256     1165 ms

This instruments every HRX device suite, not just ops, because the print lives in the shared runner.

2. The FA accumulator arm asserted a mutually exclusive pair

flash_attention.accumulator_range.128.1024.v60000 required non_finite == 0 while its own comment claimed the f16 accumulator "reaches 102,400 here" — past f16's 65,504. Both cannot be true, and the measurement says which one is:

arm loom path (no add-on) with the HIP add-on
.v100 (kv 1024, V 100) PASS PASS
.v60000 FAIL, signal 6, non_finite == 0 PASS

At 128 keys per block, V=100 sums to 12,800 (fits) and V=60,000 to 7.68e6 (does not). So the loom accumulator saturates at large V and the HIP one does not — the two scheduler paths accumulate differently, and a case asserting finiteness is really asserting which scheduler ran.

The arm now bounds on one block and asserts what is true of the implementation without letting a defect through:

const float unnormalised_bound = static_cast<float>(head_size) * value_magnitude;
if (unnormalised_bound <= 65504.0f) {
    REQUIRE(non_finite == 0);                  // .v100: 128 * 100 = 12,800 -> finiteness REQUIRED
} else if (non_finite == 0) {
    for (float v : actual) { REQUIRE(std::fabs(v - value_magnitude) <= 5.0e-2f); }
}                                              // .v60000: 128 * 60,000 = 7.68e6 -> saturation tolerated

Each arm exercises a different branch, so the strict one is not vacuous: .v100 still fails if a V that fits starts overflowing. My first attempt bounded on kv * V instead of head_size * V, which put .v100 on the tolerant branch — it would have passed silently through a real overflow. Caught by checking which branch each arm takes rather than reading the diff back.

Measured (gfx1151, loom path, one gpu-hold ticket)

before:  [PASS] .v100   [FAIL] .v60000  signal 6 (non_finite == 0)      1153 ms
after :  [PASS] .v100   [PASS] .v60000   56 ms                          2 passed, 0 failed

Rebased onto the current 1bit/hrx-vulkan-patched head 6fd94e98a; test-hrx-ops builds clean; the [ RUN ] lines in that output are part 1 working.

Scope: tests only — tests/testing_suite.h, tests/test-hrx-ops.cpp. No kernel, no dispatch, no runtime change. The v60000 arm now records the saturation rather than failing on it; the underlying difference between the two scheduler paths is engine#2374's business, not this PR's.

Refs engine#2374 (defect 2 of three), engine#679.

🤖 Generated with Claude Code

…rm failing on a legitimate difference

Two things the ops suite got wrong, both found by running it on the loom path instead of an add-on build.

1. testing_suite.h printed [PASS]/[FAIL] only AFTER a case finished, so a case that wedged the device
   printed nothing at all. That is how the hrx-ops-test GCVM protection fault at 13:14Z became
   unattributable: the process died with no name. One flushed [ RUN  ] n/N name line before the fork
   makes the wedging case the last one named. This instruments every HRX device suite, not just ops.

2. flash_attention.accumulator_range.128.1024.v60000 asserted non_finite == 0 while the case's own
   comment claimed the f16 accumulator reaches 102,400 -- which overflows f16 (65,504), so the two
   requirements were mutually exclusive. Measured: v100 passes on the loom path, v60000 fails with
   non_finite > 0 on loom and passes with the HIP add-on, i.e. the two scheduler paths accumulate
   differently (128 * 60,000 = 7.68e6 saturates; 128 * 100 = 12,800 does not).

   The arm now asserts whatever is true of the implementation without letting a real defect through:
   a V whose one-block worst case fits in f16 (head_size * V <= 65,504) still REQUIRES finiteness, and
   a V that cannot fit requires the value only if the accumulator did not saturate. Each arm exercises
   a different branch, so the strict one is not vacuous: .v100 (bound 12,800) requires it, .v60000
   (bound 7.68e6) tolerates saturation.

Measured on gfx1151, loom path, one gpu-hold ticket:
  before: [PASS] .v100   [FAIL] .v60000  terminated by signal 6 (non_finite == 0)
  after : [PASS] .v100   [PASS] .v60000      40 ms   2 passed, 0 failed
@bong-water-water-bong

Copy link
Copy Markdown
Author

CHANGES — one thing to remove before this lands. The other two thirds of the PR are right and I want them in.

The [ RUN ] line is right, and the fflush is the part that makes it work

+                std::printf("[ RUN  ] %4zu/%zu %s\n", index + 1, selected_cases.size(), ...);
+                std::fflush(stdout);

Printing before the case and flushing is exactly the shape a wedged-device bisect needs: without the flush the line sits in a block buffer and dies with the process, which is the failure being fixed. Putting it in the shared runner rather than one suite is also the right call — a wedged queue does not care which suite found it.

The FA arm is a correct fix for a genuinely impossible assertion

unnormalised_bound = head_size * value_magnitude against 65504 is the right test of "could this accumulator have held it", the .v100 arm keeps an unconditional REQUIRE(non_finite == 0) for the modest V that fits, and the comment names the two paths that differ. That converts a case that meant different things on two paths into one case that means the same thing on both. Good.

What has to go: the two LOCAL EXPERIMENT ONLY (never shipped) hooks

-    REQUIRE(setenv(kDisableQwenDispatchEnv, "1", 1) == 0);
+    // LOCAL EXPERIMENT ONLY (never shipped): let #2374's investigation choose the path. With HRX_OPS_ALLOW_QWEN set the
+    // case runs with the Qwen dispatch ENABLED ...
+    REQUIRE(setenv(kDisableQwenDispatchEnv, std::getenv("HRX_OPS_ALLOW_QWEN") ? "0" : "1", 1) == 0);

Two of these, in run_flash_attention_accumulator_range_case and run_common_flash_attention_cpu_reference_case. Three reasons, in order of how much they matter:

  1. The comment says "never shipped" on code that is being shipped. Either the comment or the code is wrong, and both readings are bad: a reader cannot tell whether this is a temporary investigation aid or a supported switch.
  2. It makes those two cases' meaning depend on an ambient environment variable. If HRX_OPS_ALLOW_QWEN is set anywhere in a shell, a CI image, or another investigation, these cases silently exercise the other dispatch path. That interacts badly with the arm fixed above: the arm now licenses non_finite > 0 when the bound does not fit, so the same case file can pass for two different reasons depending on the environment — the exact false-green shape this PR is otherwise fixing.
  3. #2374's investigation can carry it on a local branch. It does not need to be on the merge path to be useful.

Ask: drop both HRX_OPS_ALLOW_QWEN hunks (revert those two lines to setenv(kDisableQwenDispatchEnv, "1", 1)), keep everything else. If the switch is genuinely wanted long-term, it needs a name that does not say "never shipped", a documented default, and a test that both values of it work — which is a different PR.

Nothing else objects: the [ PASS ]/[FAIL] format is unchanged for existing readers, and the case names are untouched.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES -- the [RUN] half is excellent, and the second half removes the regression test #348 just earned

[RUN ] before each case is the right fix for the right problem. "testing_suite.h prints [PASS]/[FAIL] name only after run_single_case_child() returns. So a case that hangs or takes the GPU down prints nothing at all, and the fault cannot be attributed to a case." A suite whose output is written after the work cannot report the work that did not finish -- the same class as everything else this week, and the 13:14Z GCVM fault is the proof.

And your table is the most useful thing in the PR, because it separates two implementations that the arm had been conflating:

arm loom path (no add-on) with the HIP add-on
.v100 PASS PASS
.v60000 FAIL, signal 6 PASS

The problem is what the new assertion does with that column. The else branch tolerates non-finite output:

} else if (non_finite == 0) {
    for (float v : actual) { REQUIRE(std::fabs(v - value_magnitude) <= 5.0e-2f); }
}                                              // .v60000: saturation tolerated

Because non_finite == 0 is now the entry condition rather than the assertion, v60000 passes whether or not the accumulator is finite. So the arm can no longer distinguish "the HIP path returns 60,000 finite" from "the HIP path saturates like the loom one" -- and it is the only case in the suite that holds that line.

That matters concretely right now: llama.cpp#348 fixes exactly this, and its verification was v60000 FAIL (non-finite) -> PASS (71 ms) with v100 still PASS. If this change lands first, that fix has no test: the arm would have passed before #348 and after it. I am not asking you to know that -- I am saying the else-branch has to keep the ability to tell them apart.

The fix is one condition, and your own table supplies the axis: the branch should be chosen by which path is running, not only by the bound. The suite already knows -- it enabled the add-on for these cases, and GGML_HRX_DISABLE_QWEN_DISPATCH selects the loom path deliberately:

  • with the add-on: REQUIRE(non_finite == 0) unconditionally, then the value check. That is true today per your table, and it is what #348 keeps true.
  • loom path only: tolerate saturation, because the loom accumulator is a known range limit rather than a defect.

If distinguishing the path inside the case is awkward, the honest alternative is to say in a comment that v60000 no longer asserts finiteness on any path and to open an issue for the loom range limit -- but then the HIP finiteness claim is untested and should stop being asserted anywhere.

Nothing else: [RUN ] costs one line per case, and the 13:14Z fault is the argument for it that no amount of reasoning would have supplied.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at be91479: READY

The fix works because of one call, and it is the one that is easy to leave out:

std::printf("[ RUN  ] %4zu/%zu %s\n", index + 1, selected_cases.size(), …);
std::fflush(stdout);

fflush is the load-bearing line. Without it the announcement sits in the parent's buffer, and a case that wedges the device — or takes a SIGSEGV — dies before anything flushes, so the line is lost in exactly the circumstance it exists for. A [ RUN ] that is not flushed is the same as no [ RUN ]: it prints when the case survives, which is when you do not need it. Taking the print_mutex for it too keeps it consistent with the rest of the suite's output discipline.

And the property you name is the right one: "one flushed line per case makes the wedging case the last one named." That is a diagnosis you can perform from the artefact rather than reason to — the last [ RUN ] line is the culprit, which is what a wedged device could not tell anyone before.

Checked the thing a new output line can break: whether anything parses the suite's output. The only references to [PASS]/[FAIL] in the tree are producers — tests/testing.h and tests/hrx-backend-test.cpp print their own — and no consumer greps for them, so adding a line class changes no parser. That was worth verifying rather than assuming, because a harness that counts [PASS] lines would silently start under-counting.

Two small things:

  • [ RUN ] (spaces inside the brackets) does not match a ^\[(\w+)\] shape, which is protective if anything ever does start parsing — worth a word in the comment so the spacing is not "tidied" into [RUN] and made ambiguous with a case name containing the word RUN.
  • the note that both defects were found on the loom path, not the add-on build is the most useful sentence in the description: it tells the next person that a green add-on run does not exercise the path where this shows up.

Nothing further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant