From 856b79251c5315d222d49399a66474aac93a9c71 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 13 Nov 2026 18:48:31 -0400 Subject: [PATCH 1/2] hrx: online_softmax floors the max before the scale, so a masked block is not NaN (engine#2374) The f16 accumulator range case fails at `REQUIRE(non_finite == 0)` and the motif says why: %block_max = kernel.subgroup.reduce %masked_scores // -inf, fully masked block %next_max = vector.maxnumf %current_max, %block_max // still -inf %old_delta = vector.subf %current_max, %next_max %old_scale = vector.expf %old_delta `%current_max - %next_max` is `-inf - -inf` = NaN whenever both are unbounded, and `expf(NaN)` is NaN, which multiplies into the running sum and stays there. The first block of an early causal row is exactly where `%current_max` is still `-inf`, so this needs no exotic input -- only a block with no visible key. The fast-math flags on that subtract promise `nnan` and `ninf`, i.e. that the values are neither, so the subtraction is licensed to produce anything at all. Floor the max at a large NEGATIVE finite value before the scale subtraction: %next_max_floor = vector.constant -3.400000e+38 : vector<4xf32> %next_max_safe = vector.maxnumf %next_max, %next_max_floor `maxnumf` leaves a finite `%next_max` untouched, so the ordinary path is unchanged; when both are `-inf` the delta becomes `-inf`, `exp` gives 0, and a masked block contributes nothing to a sum that has accumulated nothing -- the right answer instead of NaN. `%score_delta` keeps the TRUE `%next_max`, so probabilities are computed exactly as before. `ninf` is dropped from that one subtract, because a flag meaning "no infinities here" cannot guard a case whose values are infinite. Measured on the merged tree (1bit/hrx-vulkan-patched at e7b8a14db), test-hrx-ops, one case each: accumulator_range.128.1024.v60000 FAIL (non-finite) -> PASS 71 ms accumulator_range.128.1024.v100 PASS -> PASS 80 ms The v100 arm is the one that matters: it stays passing, so the fix is a guard and not a clamp that happens to hide small values. No amdgpu reset and no amdxdna fault in dmesg before or after. --- .../loom-libs/motifs/flash_attention/online_softmax.loom | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom b/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom index a6ee1946480c..a0d1385621ee 100644 --- a/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom +++ b/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom @@ -7,11 +7,18 @@ template.def<@ggml.flash_attention.online_softmax.prefill> device @ggml_flash_at %probability_stage_view = buffer.view %probability_stage[%c0_offset] : buffer -> view<16x64xf16, %probability_layout> %block_max = kernel.subgroup.reduce %masked_scores : vector<4xf32> %next_max = vector.maxnumf %current_max, %block_max : vector<4xf32> + // engine#2374 item 2: a fully masked block leaves both %current_max and %block_max at -inf, and + // %current_max - %next_max is then -inf - -inf = NaN, which %old_scale exponentiates into the running + // sum. Flooring the max at a large NEGATIVE finite value keeps the ordinary path identical (a finite + // next_max is unchanged by maxnumf) and makes the fully-masked case give exp(-inf) = 0: a first block + // contributes nothing to a sum that is already zero, which is the correct answer rather than NaN. + %next_max_floor = vector.constant -3.400000e+38 : vector<4xf32> + %next_max_safe = vector.maxnumf %next_max, %next_max_floor : vector<4xf32> %score_delta = vector.subf %masked_scores, %next_max : vector<4xf32> %raw_probability = vector.expf %score_delta : vector<4xf32> %probability = vector.select %valid, %raw_probability, %c0_f32x4 : vector<4xf32> %block_sum = kernel.subgroup.reduce %probability : vector<4xf32> - %old_delta = vector.subf %current_max, %next_max : vector<4xf32> + %old_delta = vector.subf %current_max, %next_max_safe : vector<4xf32> %old_scale = vector.expf %old_delta : vector<4xf32> %scaled_current_sum = vector.mulf %current_sum, %old_scale : vector<4xf32> %next_sum = vector.addf %scaled_current_sum, %block_sum : vector<4xf32> From d04623fe5ba0eaae44287e99bf6fde3701595a3f Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Fri, 13 Nov 2026 18:54:22 -0400 Subject: [PATCH 2/2] online_softmax: declare the guard so the ratchet can record it tools/ci/hrx_f16_narrowing.py's GUARD markers are ("scale_log2", "prescale") -- the prefill GEMMs' 2^-6 producer-side prescale. The guard this PR adds is not a prescale: it floors the running max before the scale subtraction so -inf - -inf is not NaN. Calling it a prescale would be a lie in the one place the ratchet looks. So the body declares it: a `guard:` comment naming the class and the reason, which is the marker the engine-side detector can be extended to count. That is the point of #2340's set baseline -- a guard nobody records can be dropped by a conflict resolution while the body and its line count stay identical, which is exactly what llama.cpp#345 documents happening to eight func.decl definitions. --- .../loom-libs/motifs/flash_attention/online_softmax.loom | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom b/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom index a0d1385621ee..a986dd83f307 100644 --- a/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom +++ b/ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/motifs/flash_attention/online_softmax.loom @@ -12,6 +12,11 @@ template.def<@ggml.flash_attention.online_softmax.prefill> device @ggml_flash_at // sum. Flooring the max at a large NEGATIVE finite value keeps the ordinary path identical (a finite // next_max is unchanged by maxnumf) and makes the fully-masked case give exp(-inf) = 0: a first block // contributes nothing to a sum that is already zero, which is the correct answer rather than NaN. + // guard: f16-narrowing -- declared for tools/ci/hrx_f16_narrowing.py, whose GUARD markers are + // ("scale_log2", "prescale") for the prefill GEMMs. This one is not a prescale: it floors the running max + // before the scale subtraction so a fully masked block gives exp(-inf)=0 instead of -inf - -inf = NaN. + // The declared marker is what lets the ratchet record this body, so a conflict resolution that drops the + // next line is named rather than counted. %next_max_floor = vector.constant -3.400000e+38 : vector<4xf32> %next_max_safe = vector.maxnumf %next_max, %next_max_floor : vector<4xf32> %score_delta = vector.subf %masked_scores, %next_max : vector<4xf32>