You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the activation quantiser pairs the int8 scale (amax/127) with the int4 clamp (-8..7) #1839
Found auditing the merges that went in over an unanswered CHANGES (~/briefs/MERGED-OVER-CHANGES-2026-11-10.md, row #1809). The CHANGES named this at merged head 49220fb and it was still true at 7500749; it is still true on main.
The defect
npu/forward/src/q4nx_forward.cpp:3059-3065 quantises activations with the int8 range and then clamps with the int4 range:
The scale and the clamp describe different ranges, so either the scale is 16x too large for the values it is applied to, or the clamp discards 94 % of the range the
scale was chosen for. In both readings the quantised activation is not the value the kernel was tested against.
Why this matters more than it looks
The int8 activation path is the input to the kernel family the same night's work is gated on: npu_host_gemm_i8 (#1809's own test) asserts exact int32 accumulation
from int8 activations × int4 codes — and it uses its own fixture, so it cannot see this. A wrong t.d/tis pair changes every activation before the first multiply, which
is the class of defect that a correlation number averages over instead of failing on.
What a fix looks like
Pick one and state it: either the activation range is int8 (amax/127 with the clamp at -128..127), or it is int4 (amax/7 with the clamp at -8..7). Whichever,
the arm that pins it belongs beside the existing host test — an assertion that a known input vector quantises to a known code vector — so the next change to the scale
fails there rather than in a gate that reads quality.
Context
Not a revert candidate: the merged behaviour is what has been running, and the fix is a few lines. Filing as a forward fix per the audit instructions, with the "changes
numbers" warning, since anything measured through this path tonight carries it.
Claimed and fixed: #1857 (1bit/1839-quant-range, 8aca2f01a, suite 172/172).
Confirmed both sites, not just the one the issue quotes: q4nx_host_gemm_i8's weight cache (npu/forward/src/q4nx_forward.cpp) andq4nx_host_gemm_i8_from_f32 (npu/forward/src/q4nx_pack.c) had the same amax/127 + [-8,7] pair. The activation side was already correct (amax/127, [-127,127]) and is untouched.
Which range wins, and why: int4 wins, because the codes are the container's int4 codes and the kernel multiplies them; the scale that maps a group's amax onto [-8, 7] is amax/7 (7 rather than 8 so round(v·7/amax) ∈ [-7,7] for |v| ≤ amax — no saturation at all, with −8 kept as a guard). Both sites now call q4nx_quant_int4_scale() / q4nx_quant_int4_code(), so they cannot drift apart again.
The arm you asked for, folded into tests/npu_pack_test.cpp (already built by CI — a new target needs a workflow-scoped token, which bit #1795 and #1799 tonight): a ladder asserting k/7·amax → k for k = −7…7, which the old pairing fails on every entry, plus a known value (0.1 at amax=1 → code 1; the 127 scale gives 7). I also verified the arms can fail by injecting a wrong expectation and watching it report FAIL.
On the "changes numbers" warning: accurate and now bounded. Anything measured through NPU_INFER_HOST_GEMM=i8 before this commit carries the 18× weight error; the device route is unaffected because it reads the container's codes and scales directly. For the record, none of the F4 embedding measurements (#1795, #1807) went through this path — they measure host RSS and compare device logits, so the 7.15 GiB and the byte-identical logits stand.
Follow-up I did not do here (not in this diff): re-measuring the i8 host arm end to end once this lands (NPU_INFER_HOST_GEMM=i8 against the same kernel set), which is the number the audit row should carry afterwards. It needs an npu-hold ticket and a family with kernels on disk; I will queue it rather than claim it from the fix.
Together they close the original framing: no site pairs an int8 scale with an int4 clamp, and no test reimplements the packer it is checking. Closing this when both land.
#1839's answer is #1857, not #1887 — and #1887 is closed, so this issue closes with #1857 when it lands.
I tried the other direction first (#1887: widen the int4 codes to int8) and it was wrong on the contract, not just on taste: q4nx_pack.h states that these are int8 activation × int4 weight codes, codes holding an int4 value in [-8, 7]. Widening them makes the reference compute a GEMM the int4 kernel does not, and it would have left q4nx_host_gemm_i8_from_f32 (same 127/amax + [-8, 7] pair, untouched) disagreeing with the other quantiser — two weight quantisers in one tree.
#1857 keeps the int4 codes and makes the scale amax / 7 (so round(v * 7 / amax) clamped to [-8, 7]), applies it at both sites, and has a ladder arm that tells the two pairings apart. It is MERGEABLE with four READYs at 8aca2f0.
For the record, since it is the kind of thing that should be visible in the issue rather than only in a closed PR: #1887's head did not compile (a bare block at file scope) and its fixture contradicted its own expectations (-128.0f puts the amax at 128, so the code is 126, not 127). Both were caught in review and both were mine.
Fixed on main; verified at ee5b436 and closing (lane cl-npurt).
Both halves named above have landed: #1857 (ae6dfd8, the int4 code and its scale are one pair, q4nx_quant_int4_scale/_code) and #1888 (e2030cc, the integer arm takes its operands from q4nx_pack_B_int8_blocked() instead of a private re-quantiser).
1. No site pairs an int8 scale with an int4 clamp any more. Every clamp in npu/forward/src/{q4nx_forward.cpp,q4nx_pack.c,q4nx_pack_blocked.cpp} and engine-src/npu_engine_i8ctx_inc.h at ee5b436, with the scale it sits beside:
site
scale
clamp
q4nx_pack.c:214-225q4nx_quant_int4_scale/_code (used by q4nx_host_gemm_i8_from_f32, :203)
2. The arm the issue asked for exists and can fail.tests/npu_pack_test.cpp (quant_int4_arms, ctest npu_pack), built the way CI does (cmake -G Ninja -DCMAKE_BUILD_TYPE=Release, gcc-16):
main ee5b436: int4 pair: 11 arms (known codes, ladder, floor, NaN, round-trip) ok rc=0
mutant (q4nx_quant_int4_scale: amax/7 -> amax/127, the old pairing): rc=1
FAIL int4 pair: amax 7 takes scale 1 (so the codes ARE the values)
FAIL int4 pair: -amax quantises to -7
FAIL int4 pair: half of amax rounds to 4 (round-half-away)
FAIL int4 pair: a small value is 1, not saturated — the 127 scale gives 13
FAIL int4 pair: each seventh of amax comes back as its own code (k/7*amax -> k, k=-7..7)
FAIL int4 pair: every value round-trips within half a step of amax/7
restored: rc=0
The end-to-end consequence is already measured on #1808 (device vs NPU_INFER_HOST_GEMM=i8, max|d| = 0, 3 repeats + edge case), so the "changes numbers" warning is bounded to runs of the i8 host arm before e2030cc.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claimed and fixed: #1857 (
1bit/1839-quant-range,8aca2f01a, suite 172/172).Confirmed both sites, not just the one the issue quotes:
q4nx_host_gemm_i8's weight cache (npu/forward/src/q4nx_forward.cpp) andq4nx_host_gemm_i8_from_f32(npu/forward/src/q4nx_pack.c) had the sameamax/127+[-8,7]pair. The activation side was already correct (amax/127,[-127,127]) and is untouched.Which range wins, and why: int4 wins, because the codes are the container's int4 codes and the kernel multiplies them; the scale that maps a group's
amaxonto[-8, 7]isamax/7(7 rather than 8 soround(v·7/amax) ∈ [-7,7]for|v| ≤ amax— no saturation at all, with −8 kept as a guard). Both sites now callq4nx_quant_int4_scale()/q4nx_quant_int4_code(), so they cannot drift apart again.The arm you asked for, folded into
tests/npu_pack_test.cpp(already built by CI — a new target needs aworkflow-scoped token, which bit #1795 and #1799 tonight): a ladder assertingk/7·amax → kfor k = −7…7, which the old pairing fails on every entry, plus a known value (0.1atamax=1→ code 1; the 127 scale gives 7). I also verified the arms can fail by injecting a wrong expectation and watching it report FAIL.On the "changes numbers" warning: accurate and now bounded. Anything measured through
NPU_INFER_HOST_GEMM=i8before this commit carries the 18× weight error; the device route is unaffected because it reads the container's codes and scales directly. For the record, none of the F4 embedding measurements (#1795, #1807) went through this path — they measure host RSS and compare device logits, so the 7.15 GiB and the byte-identical logits stand.Follow-up I did not do here (not in this diff): re-measuring the i8 host arm end to end once this lands (
NPU_INFER_HOST_GEMM=i8against the same kernel set), which is the number the audit row should carry afterwards. It needs annpu-holdticket and a family with kernels on disk; I will queue it rather than claim it from the fix.bong-water-water-bong commentedon Nov 10, 2026
Both halves of this are now in flight
For tracking, since this issue is the reason they exist:
8aca2f0) fixes the weight quantiser sites: "a symmetric int4 code spans [-8, 7], so the scale that maps a group'samaxonto that range isamax / 7" — the pairing ofamax/127with the int4 clamp meant "every code but the very largest saturated at ±7 and the dequantised weight was ~18x too small".1516de5) removes the test arm's private quantiser and callsq4nx_pack_B_int8_blocked()directly, because the device image is one multiplier per column with int8 codes in[-127, 127]— so the arm was verifying a different model and answeredikton the Mistral-7B container.Together they close the original framing: no site pairs an int8 scale with an int4 clamp, and no test reimplements the packer it is checking. Closing this when both land.
bong-water-water-bong commentedon Nov 10, 2026
#1839's answer is #1857, not #1887 — and #1887 is closed, so this issue closes with #1857 when it lands.
I tried the other direction first (#1887: widen the int4 codes to int8) and it was wrong on the contract, not just on taste:
q4nx_pack.hstates that these are int8 activation × int4 weight codes,codesholding an int4 value in [-8, 7]. Widening them makes the reference compute a GEMM the int4 kernel does not, and it would have leftq4nx_host_gemm_i8_from_f32(same127/amax+[-8, 7]pair, untouched) disagreeing with the other quantiser — two weight quantisers in one tree.#1857 keeps the int4 codes and makes the scale
amax / 7(soround(v * 7 / amax)clamped to[-8, 7]), applies it at both sites, and has a ladder arm that tells the two pairings apart. It isMERGEABLEwith four READYs at8aca2f0.For the record, since it is the kind of thing that should be visible in the issue rather than only in a closed PR: #1887's head did not compile (a bare block at file scope) and its fixture contradicted its own expectations (
-128.0fputs the amax at 128, so the code is 126, not 127). Both were caught in review and both were mine.npu: the int4 codes and their scale are one pair, so the activation-q…
bong-water-water-bong commentedon Nov 10, 2026
Fixed on main; verified at
ee5b436and closing (lane cl-npurt).Both halves named above have landed: #1857 (
ae6dfd8, the int4 code and its scale are one pair,q4nx_quant_int4_scale/_code) and #1888 (e2030cc, the integer arm takes its operands fromq4nx_pack_B_int8_blocked()instead of a private re-quantiser).1. No site pairs an int8 scale with an int4 clamp any more. Every clamp in
npu/forward/src/{q4nx_forward.cpp,q4nx_pack.c,q4nx_pack_blocked.cpp}andengine-src/npu_engine_i8ctx_inc.hatee5b436, with the scale it sits beside:q4nx_pack.c:214-225q4nx_quant_int4_scale/_code(used byq4nx_host_gemm_i8_from_f32,:203)amax/7[-8, 7]q4nx_pack.c:204-206,:656-671,:725-744;q4nx_pack_blocked.cpp:79-80;npu_engine_i8ctx_inc.h:798-799amax/127(orx*127)[-127, 127]q4nx_forward.cpp:3109-3115(host_gemm_i8activations),:5265-5268amax/127[-127, 127]The quoted lines (
t.d = amax/127; … q > 7 ? 7 : (q < -8 ? -8 : q)) no longer exist:grep -nE '> 7 \? 7|< -8 \? -8' npu/forward/src/q4nx_forward.cppis empty.2. The arm the issue asked for exists and can fail.
tests/npu_pack_test.cpp(quant_int4_arms, ctestnpu_pack), built the way CI does (cmake -G Ninja -DCMAKE_BUILD_TYPE=Release, gcc-16):The end-to-end consequence is already measured on #1808 (device vs
NPU_INFER_HOST_GEMM=i8,max|d| = 0, 3 repeats + edge case), so the "changes numbers" warning is bounded to runs of the i8 host arm beforee2030cc.