From c3a6e1cd7cb3c6825b91a98a149f7ccde7dabdb9 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Tue, 10 Nov 2026 21:39:18 -0400 Subject: [PATCH] bump llama.cpp to 7a1f2b81c: the prefill kernels stay finite above the f16 range, and hrx-backend-test builds again (fixes #2068) Pin 2293f95ea -> 7a1f2b81c (seven commits ahead, none behind): llama.cpp #292 and #298 (engine #1962), #293/#294/#295 (engine #2068), #300 (engine #2069), #299 (engine #2079). registry/architectures.json records the new pin; docs/hrx.md describes the prefill fixes and what is still open under #1962. Co-Authored-By: Claude Fable 5.2 --- docs/hrx.md | 28 +++++++++++++++++++++++++--- registry/architectures.json | 2 +- third_party/llama.cpp | 2 +- 3 files changed, 27 insertions(+), 5 deletions(-) diff --git a/docs/hrx.md b/docs/hrx.md index 184f21fd2..ce5e5fdb9 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -300,9 +300,31 @@ Measured on gfx1151 (2026-11-10): the dense inputs of Qwen3-0.6B, Qwen3-4B, ZAYA1-8B and flint. Cases at those two values are in the suite, so the shift has a test on each side. -Still open under #1962: two prefill kernels that re-quantise the activations into 8-bit blocks -(Q5_K and Q6_K weights at 512 tokens) keep a per-block term in half precision and still return NaN -above 65504. And the limit is moved, not removed: an activation above 4.19e6 would overflow again. +The prefill kernels followed at pin `7a1f2b81c`. The two small-batch Q6_K kernels were already done +(llama.cpp #288). The Q4_K and Q6_K prefill kernels read a half-precision copy of the activations; +that copy is now stored at 2^-6 and the sums are scaled back (llama.cpp #298). The Q5_K/IQ4_XS +prefill kernel reads the activations as 8-bit blocks, each with a scale and a scaled sum in half +precision. The scale fits up to 8.3e6; the scaled sum is about as large as the largest activation in +the block, so it was the term that overflowed. The kernel now rebuilds a sum that is not finite from +the 8-bit values it already holds, and uses a finite one as stored (llama.cpp #292). + +Measured on gfx1151 (2026-11-11, fork `7a1f2b81c`, full build): + +- `test-backend-ops -o MUL_MAT` with a resident weight (2048 x 4096), Q4_K, Q5_K, Q6_K, IQ4_XS, Q8_0 + and Q4_0 at 1 to 512 tokens, with activation spikes of 3.0, 9.3e4 and 2e6: 180 of 180 pass (run at + llama.cpp #298's head). Before #292 and #298, Q5_K at 512 tokens and Q4_K and Q6_K at 256 and 512 + tokens returned NaN at 9.3e4. The whole op is 617 of 617; `MUL_MAT_ID` is 845 of 845. +- Perplexity on wikitext-2 (`-c 512 -b 512`, 16 chunks): Qwen3-4B-Q4_K_M 17.1836 (17.1864 before + #298), Qwen3-4B-IQ4_XS 17.6304 (the same before and after #292). +- Prefill rate on Qwen3-4B-Q4_K_M across #298, eight alternating runs on a loaded box: 2683 against + 2735 tok/s (ratio 0.98, pairs between 0.88 and 1.14). No regression shown; a cost of 2 to 3 % is + not excluded. + +Still open under #1962: the final projection of the draft head (a shortlist kernel) narrows its +activations too and has no test that reaches it. The other kernels that read the 8-bit blocks (the +fused SwiGLU, the routed kernels and the low-token Q4 paths) use the stored sum as it is and are not +measured above 65504. And the limits are moved, not removed: 4.19e6 for the half-precision copies, +8.3e6 for the 8-bit blocks. ### Fixed: Qwen3-VL on `HRX0`: it decodes again, and its rotation runs on the device ([engine#1467](https://github.com/1bit-MONSTER/engine/issues/1467)) diff --git a/registry/architectures.json b/registry/architectures.json index ef1ebe7c4..cbe8229e0 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -1,7 +1,7 @@ { "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { - "llama.cpp (hrx)": "2293f95ea49dafec71a2460739d95ea31cd27bfc" + "llama.cpp (hrx)": "7a1f2b81ce17aaa84a40b766d1b06e5fd4601196" }, "counts": { "hrx": 298, diff --git a/third_party/llama.cpp b/third_party/llama.cpp index 2293f95ea..7a1f2b81c 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit 2293f95ea49dafec71a2460739d95ea31cd27bfc +Subproject commit 7a1f2b81ce17aaa84a40b766d1b06e5fd4601196