Skip to content

npu: the activation quantiser pairs the int8 scale (amax/127) with the int4 clamp (-8..7) #1839

Closed

Description

@bong-water-water-bong

Found auditing the merges that went in over an unanswered CHANGES (~/briefs/MERGED-OVER-CHANGES-2026-11-10.md, row #1809). The CHANGES named this at merged head
49220fb and it was still true at 7500749; it is still true on main.

The defect

npu/forward/src/q4nx_forward.cpp:3059-3065 quantises activations with the int8 range and then clamps with the int4 range:

t.d  = amax / 127.0f;      // int8 scale: -128 .. 127
tis  = 127.0f / amax;
q    = q > 7 ? 7 : (q < -8 ? -8 : q);   // int4 clamp: -8 .. 7

The scale and the clamp describe different ranges, so either the scale is 16x too large for the values it is applied to, or the clamp discards 94 % of the range the
scale was chosen for. In both readings the quantised activation is not the value the kernel was tested against.

Why this matters more than it looks

The int8 activation path is the input to the kernel family the same night's work is gated on: npu_host_gemm_i8 (#1809's own test) asserts exact int32 accumulation
from int8 activations × int4 codes — and it uses its own fixture, so it cannot see this. A wrong t.d/tis pair changes every activation before the first multiply, which
is the class of defect that a correlation number averages over instead of failing on.

What a fix looks like

Pick one and state it: either the activation range is int8 (amax/127 with the clamp at -128..127), or it is int4 (amax/7 with the clamp at -8..7). Whichever,
the arm that pins it belongs beside the existing host test — an assertion that a known input vector quantises to a known code vector — so the next change to the scale
fails there rather than in a gate that reads quality.

Context

Not a revert candidate: the merged behaviour is what has been running, and the fix is a few lines. Filing as a forward fix per the audit instructions, with the "changes
numbers" warning, since anything measured through this path tonight carries it.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed and fixed: #1857 (1bit/1839-quant-range, 8aca2f01a, suite 172/172).

    Confirmed both sites, not just the one the issue quotes: q4nx_host_gemm_i8's weight cache (npu/forward/src/q4nx_forward.cpp) and q4nx_host_gemm_i8_from_f32 (npu/forward/src/q4nx_pack.c) had the same amax/127 + [-8,7] pair. The activation side was already correct (amax/127, [-127,127]) and is untouched.

    Which range wins, and why: int4 wins, because the codes are the container's int4 codes and the kernel multiplies them; the scale that maps a group's amax onto [-8, 7] is amax/7 (7 rather than 8 so round(v·7/amax) ∈ [-7,7] for |v| ≤ amax — no saturation at all, with −8 kept as a guard). Both sites now call q4nx_quant_int4_scale() / q4nx_quant_int4_code(), so they cannot drift apart again.

    The arm you asked for, folded into tests/npu_pack_test.cpp (already built by CI — a new target needs a workflow-scoped token, which bit #1795 and #1799 tonight): a ladder asserting k/7·amax → k for k = −7…7, which the old pairing fails on every entry, plus a known value (0.1 at amax=1 → code 1; the 127 scale gives 7). I also verified the arms can fail by injecting a wrong expectation and watching it report FAIL.

    On the "changes numbers" warning: accurate and now bounded. Anything measured through NPU_INFER_HOST_GEMM=i8 before this commit carries the 18× weight error; the device route is unaffected because it reads the container's codes and scales directly. For the record, none of the F4 embedding measurements (#1795, #1807) went through this path — they measure host RSS and compare device logits, so the 7.15 GiB and the byte-identical logits stand.

    Follow-up I did not do here (not in this diff): re-measuring the i8 host arm end to end once this lands (NPU_INFER_HOST_GEMM=i8 against the same kernel set), which is the number the audit row should carry afterwards. It needs an npu-hold ticket and a family with kernels on disk; I will queue it rather than claim it from the fix.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Both halves of this are now in flight

    For tracking, since this issue is the reason they exist:

    Together they close the original framing: no site pairs an int8 scale with an int4 clamp, and no test reimplements the packer it is checking. Closing this when both land.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    #1839's answer is #1857, not #1887 — and #1887 is closed, so this issue closes with #1857 when it lands.

    I tried the other direction first (#1887: widen the int4 codes to int8) and it was wrong on the contract, not just on taste: q4nx_pack.h states that these are int8 activation × int4 weight codes, codes holding an int4 value in [-8, 7]. Widening them makes the reference compute a GEMM the int4 kernel does not, and it would have left q4nx_host_gemm_i8_from_f32 (same 127/amax + [-8, 7] pair, untouched) disagreeing with the other quantiser — two weight quantisers in one tree.

    #1857 keeps the int4 codes and makes the scale amax / 7 (so round(v * 7 / amax) clamped to [-8, 7]), applies it at both sites, and has a ladder arm that tells the two pairings apart. It is MERGEABLE with four READYs at 8aca2f0.

    For the record, since it is the kind of thing that should be visible in the issue rather than only in a closed PR: #1887's head did not compile (a bare block at file scope) and its fixture contradicted its own expectations (-128.0f puts the amax at 128, so the code is 126, not 127). Both were caught in review and both were mine.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed on main; verified at ee5b436 and closing (lane cl-npurt).

    Both halves named above have landed: #1857 (ae6dfd8, the int4 code and its scale are one pair, q4nx_quant_int4_scale/_code) and #1888 (e2030cc, the integer arm takes its operands from q4nx_pack_B_int8_blocked() instead of a private re-quantiser).

    1. No site pairs an int8 scale with an int4 clamp any more. Every clamp in npu/forward/src/{q4nx_forward.cpp,q4nx_pack.c,q4nx_pack_blocked.cpp} and engine-src/npu_engine_i8ctx_inc.h at ee5b436, with the scale it sits beside:

    site scale clamp
    q4nx_pack.c:214-225 q4nx_quant_int4_scale/_code (used by q4nx_host_gemm_i8_from_f32, :203) amax/7 [-8, 7]
    q4nx_pack.c:204-206, :656-671, :725-744; q4nx_pack_blocked.cpp:79-80; npu_engine_i8ctx_inc.h:798-799 amax/127 (or x*127) [-127, 127]
    q4nx_forward.cpp:3109-3115 (host_gemm_i8 activations), :5265-5268 amax/127 [-127, 127]

    The quoted lines (t.d = amax/127; … q > 7 ? 7 : (q < -8 ? -8 : q)) no longer exist: grep -nE '> 7 \? 7|< -8 \? -8' npu/forward/src/q4nx_forward.cpp is empty.

    2. The arm the issue asked for exists and can fail. tests/npu_pack_test.cpp (quant_int4_arms, ctest npu_pack), built the way CI does (cmake -G Ninja -DCMAKE_BUILD_TYPE=Release, gcc-16):

    main ee5b436:            int4 pair: 11 arms (known codes, ladder, floor, NaN, round-trip)   ok   rc=0
    mutant (q4nx_quant_int4_scale: amax/7 -> amax/127, the old pairing):                         rc=1
      FAIL int4 pair: amax 7 takes scale 1 (so the codes ARE the values)
      FAIL int4 pair: -amax quantises to -7
      FAIL int4 pair: half of amax rounds to 4 (round-half-away)
      FAIL int4 pair: a small value is 1, not saturated — the 127 scale gives 13
      FAIL int4 pair: each seventh of amax comes back as its own code (k/7*amax -> k, k=-7..7)
      FAIL int4 pair: every value round-trips within half a step of amax/7
    restored:                                                                                    rc=0
    

    The end-to-end consequence is already measured on #1808 (device vs NPU_INFER_HOST_GEMM=i8, max|d| = 0, 3 repeats + edge case), so the "changes numbers" warning is bounded to runs of the i8 host arm before e2030cc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions