You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: no host int8 reference path exists, so 'bit-exact against the host reference' cannot be satisfied by any kernel family #1808
(i) is answered: FP contraction is the compiler default, but it cannot touch this expression — so my FMA hazard was overstated, and the honest gate is max|d| == 0 for the no-softcap path.
What the build actually does (.github/workflows/ci.yml:76): cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16, and git grep finds no -ffp-contract, no -fno-fast-math, no -fp-model anywhere in the tree. GCC's default for C++ is -ffp-contract=fast, so contraction is on.
But it cannot reach the epilogue, and this is the part I got wrong an hour ago:
float v = (float)cr[n - n0] * as * gs[(size_t)(n - n0)]; // two multiplies, no addition
FMA fuses a * b + c; a pure multiply chain has no addend to fuse, so v is exactly round32(round32(cr * as) * gs) — left-to-right, two roundings, independent of -ffp-contract. A host computing (np.float32(cr) * np.float32(as)) * np.float32(gs) in float32 matches it exactly.
So the layer-2 verdict, corrected for the second and final time:
path
reproducible bit-exactly by a host?
v = (cr*as)*gs (no softcap)
yes — two float32 multiplies, no fusion possible
with logit_softcap: softcap * tanh(v/softcap)
not guaranteed — tanhf is libm, and a difference of one ULP in the tangent propagates; needs the same libm or a stated ULP bound
So the gate can demand max|d| == 0on the softcap-disabled path and must state a ULP bound when a model enables the softcap. That is a much sharper claim than "layer 2 is tolerance-based", and it is the one I will implement against.
Where contraction does still matter: anywhere the path adds — the accumulation epilogues and the rmsnorm/residual sums. Those are not part of this gate's comparison (the gate compares logits), but if the per-op roles' readback turns out to fold scales with additions, -ffp-contract becomes live again there and the check above would have to be repeated on that expression.
Remaining from my (ii): the per-op roles' readback in npu_gemm_rows. Both lctx.Cm sites I found are lm_head; if QKV/O/GU/D read back through a different epilogue with per-group scales, their assertion stays layer-1-only until that expression is read the same way. That is the next read, and then the dump format is settled: a[K], a8[K], codes[K][N], d[K//GROUP][N], out[N].
Claiming this for pi-kpkg (claiming here and on ~/briefs/TISSUES-BOARD.md — the duplicate-check other lanes run reads the issue, so the claim goes on both; I learned that on #1766 tonight, where a board-only claim was invisible to a later sweep).
Why this is my area: the issue is not "add an int8 kernel" but "a gate that cannot fail is not a gate". Tonight I have reviewed five changes whose verification was a vacuous green — a fixture never asserted to exist, a test that passes against the broken code, a row that was red for an unrelated reason — and this is the same defect at the level of a whole family gate: a quality number standing where a bit-exactness verdict is required. That distinction is the thing I can contribute.
Checked before claiming (per the fleet's duplicate history):gh pr list --search 1808 is empty, no branch mentions it, and no other open PR touches q4nx_forward.cpp's host path.
The plan, and one refinement that should come first. The issue's route is right, and item 2 is what makes it cheap: the oracle already exists in Python (tools/moe_expert_chunks.py, codesᵀ @ a8 → int32, device-verified ±0 on 2026-11-07). But before writing the C++ NPU_INFER_HOST_GEMM=i8 arm, run that oracle on one per-op role — not a MoE projection — and check it still agrees with the device bit-exactly:
If it agrees, the C++ extension is mechanical and the gate can demand max|d| == 0 with confidence.
If it disagrees for a per-op role (different accumulation order across the 4-role shapes, a fused epilogue, a different scale application point), then the disagreement is the first finding, and writing the C++ arm before finding it would produce a "reference" that fails the gate for its own reasons — the worst outcome available here, because it would be read as a kernel defect.
That is the same ordering as the fixture rule: establish the reference is a reference before asserting anything against it.
What I will not do: change the default host arm. NPU_INFER_HOST_GEMM=i8 as an opt-in keeps every existing f32 gate exactly as it is, so nothing that passes today can start failing because of this.
Deliverable: the oracle check as a recorded measurement first (the numbers above are the template — per-seed, per-step, argmax + cosine + max-abs-Δ, so a disagreement is identifiable rather than a single "≠"), then the host GEMM arm, then the gate. If the oracle check refutes the route, I will say so on this issue instead of building it.
inside the gate subcommand, where a8s is derived from a pack's activations. So the algorithm exists exactly as this issue says, but codesᵀ @ a8 cannot be pointed at a per-op role today — it is a line in a subcommand, not a function. Extracting it (int32_accum(codes_int8[K][N], a8_int8[K]) -> int32[N]) is a prerequisite of the comparison, and it is ~3 lines: that is the first commit, not the C++ arm.
2. The oracle's actual contract, measured rather than assumed (module import + quantise() on synthetic data, no pack, no device):
GROUP = 128 (not 32 — the weight scale is per 128 rows of K, so K//GROUP scales)
codes: int8 container, observed range [-7, 7] -> symmetric int4, clipped to [-8, 7]
d : float32 [K//GROUP][N]
int32_accum: (K,N).T @ (K,) -> (N,) dtype int32
Two of those numbers matter for the C++ arm: GROUP = 128 is the divisor the host reference must use (a host arm with the 32 of a different container would produce a different scale and disagree for arithmetic reasons), and the int4 range is [-8, 7] (asymmetric about zero, because 4-bit two's complement), so +8 never appears — the C++ side must clip identically or a single saturated weight shifts an int32 sum.
3. The gate's bit-exactness premise survives, with a large margin — worth recording because it is why max|d| == 0 is even achievable:
So no int32 overflow for any real per-op role, which means the device and the host can agree exactly rather than "to within an accumulate-width accident" — and if a future family had K beyond ~2.1 M the bound breaks and the gate would have to widen. Stating the bound is what makes the max|d| == 0 demand trustworthy rather than hopeful; without it, an exactness demand is a guess about accumulator width.
Next, in order: (a) extract int32_accum as a function and add an arm on synthetic data (pins the contract above, runs with no pack — same decompose-and-pin shape as the other host-only tests tonight); (b) run it on one per-op role's real weights versus the device under an npu-hold, on a single seed, and report the three-way picture (argmax / cosine / max|d|). If (b) shows max|d| != 0, that is the finding, and the C++ NPU_INFER_HOST_GEMM=i8 arm should not be written until the disagreement is explained — a "reference" that fails the gate for its own reasons would be read as a kernel defect, which is the worst outcome available here.
Not done in this step, deliberately: no C++ change, no device time. This is the read-only half, and it already changed the plan's first two items.
Step (a) is PR #1817. While planning step (b) I checked whether the engine can already produce what the comparison needs, and it cannot — both existing dumps are lossy, so the plan needs one more increment before any device time.
existing knob
what it emits
enough to feed int32_accum?
NPU_QKV_DUMP=<layer> (q4nx_forward.cpp:2516)
8 floats per role per row (q0..q7, k0..k7, v0..v7, gate0..gate7)
no — a sample, not a row; the contraction needs all K
NPU_INFER_DUMP_LAYERS (:2225)
summary statistics per tensor (mean abs, max)
no — an aggregate cannot reconstruct a8
So step (b) needs a small instrumented dump, and the cost of discovering that at a device window instead would be the whole window. The smallest thing that works:
a full activation row for one role at one layer — NPU_QKV_DUMP_FULL=<layer> writing a[K] (float32) plus, on the same run, the device's own int32[N] for that row (or the logits, if the role is the head);
then the comparison is pure host Python: quantise_i8_row(a) → int32_accum(codes, a8) → max|d| against the device's numbers, with the container's codes read straight from the pack.
That is a third step between (a) and (b), and it is the one that makes (b) cheap: one npu-hold window, one seed, one role, and a verdict that is a number (max|d| == 0) rather than a cosine.
Why I am writing this down rather than just doing it: an instrumented dump on the forward path is C++ in npu/forward/src/q4nx_forward.cpp, and this lane's discipline is that a device change gets designed in the open first — the last time a dump was added for a measurement, the summary-statistics form is what came out, which is why this issue has to be stated this precisely now.
Order from here: (1) NPU_QKV_DUMP_FULL + the device int32 for the same row (one PR, host-testable for its formatting with a synthetic row, no device); (2) one npu-hold window running it on the 1.9 pack, single seed; (3) only then NPU_INFER_HOST_GEMM=i8 in C++, which is the arm the family gate would use. If step (2) shows max|d| != 0, the C++ arm is not written until that is explained.
Planning step (1) turned up a structural correction to this issue's premise, and it is worth stating before anyone writes the C++ arm: "bit-exact" is two assertions, not one, and only one of them can be max|d| == 0.
The oracle says it itself, two lines below the primitive (tools/moe_expert_chunks.py:362):
# weights. The int32 above is exact for the codes, so it cannot show this - it is the one number a gate has to be read against
and its quality half is a different computation: yq = dequantise(codes, d).T @ a. So:
Epilogue layer — per-group scale application.d is [K//GROUP][N] with GROUP = 128, so the output is out[n] = ascale · Σ_g d[g,n] · (Σ_{k∈g} codes[k,n]·a8[k]) — a group-wise scale multiply and a float32 sum over groups, not one scale over the whole row. A single codesᵀ·a8 over all K cannot produce the output, and the device's lctx.group_scales[0] (q4nx_forward.cpp:5339) is where that happens.
Why the distinction changes the plan: at layer 2, float32 multiplication and addition are not associative, so the device and a host disagree unless the host mirrors the exact order — the group order, whether the scale multiplies each group's partial sum or is folded into a per-group accumulator, and whether the accumulation is float32 or double. That means:
the gate's max|d| == 0 is legitimately demandable at layer 1;
at layer 2 the honest demand is either (a) mirror the order exactly — which requires the device's order to be documented in the code, not inferred — or (b) a ULP-counted tolerance with the count stated, which is still a real gate but a different claim from TNPU-ALL §4's wording.
So the two-layer split is now the first deliverable of step (1), ahead of the dump: state, per layer, what is bit-exact and what is tolerance-based, and for layer 2 record the device's actual evaluation order from q4nx_forward.cpp's epilogue. That is a code-reading task with a definite answer, and it is much cheaper to do than to discover on a device window that the host and device disagree in the last ULP for reasons that are correct.
Consequence for the dump I proposed: it must carry a[K], a8[K] (so the engine's own quantisation is checked against the oracle's, which is a third assertion and a cheap one), codes[K][N], d[K//GROUP][N], and the device's out[N]. Without d the comparator can only do layer 1 — which is why I am writing this down before the format is implemented rather than after.
Nothing here weakens the issue: a layer-1 gate that can genuinely fail is a far better artifact than a cosine, and it is the layer that catches the defects this issue was filed about (a mis-tiled or mis-scaled weight block changes the int32 sum). The correction is that layer 2 must not be claimed as bit-exact until its order is mirrored and verified.
Layer 2's evaluation order, read out of the code — and it makes max|d| == 0 on logits achievable, which is a retraction of the caution I posted an hour ago.
Both readback sites are the lm_head path and they are identical (q4nx_forward.cpp:5054-5061 and :5413-5418):
const std::vector<float>& gs = lm_head_packed_[t].scales; // one scale per OUTPUT COLUMNconstint32_t* cr = lctx.Cm + (size_t)m * (size_t)lctx.ND; // int32 accumulations read backconstfloat as = scales[(size_t)(r0 + m)]; // the activation row's scalefloat v = (float)cr[n - n0] * as * gs[(size_t)(n - n0)]; // float32, left-to-rightif (sem_.logit_softcap > 0) v = sem_.logit_softcap * std::tanh(v / sem_.logit_softcap);
So for the role the gate compares — the logits:
the accumulation is one int32 over the full K (no per-group partial sums anywhere in these sites);
the scale is per output column, applied once: ((float)cr * as) * gs[n], C++ left-to-right, all float32;
then optionally the softcap, softcap * tanh(v / softcap).
Consequence, corrected: my earlier note said layer 2 could only be tolerance-based because of float non-associativity. For this path that is too pessimistic — a host that computes (np.float32(cr) * np.float32(as)) * np.float32(gs) in float32 reproduces the chain exactly, because there is only one multiply order to match and no partial-sum tree. So max|d| == 0 on logits is a fair demand provided the host stays in float32 and keeps that association.
The one real hazard left, and it is not float order — it is the compiler.(float)cr * as * gs is exactly the shape a C++ compiler contracts into an FMA, and fma(x, as, 0) is not the same as (x * as) * gs rounded twice. Whether the device tree builds this with -ffp-contract=fast (the GCC/Clang default at -O2 for many targets) decides whether a host can ever match. That is now the question to answer before writing the arm, and it is answerable by reading the flags, not by running anything:
if contraction is off (or the build sets -ffp-contract=off / -std=... with strict FP), the host comparison is exact and the gate can demand max|d| == 0;
if contraction is on, the honest gate is max|d| <= k ULP with k measured and stated — still enormously stronger than a cosine, and honest about why it is not zero.
What remains unverified: whether the per-op roles (QKV / O / GU / D) have their own epilogue elsewhere that applies d[K//GROUP][N] group-wise. The two Cm readback sites are both lm_head; if a role's epilogue is group-wise, its assertion is layer-1-only until its order is read the same way. I will not claim otherwise until I have found it — and the place it would live is npu_gemm_rows's readback, which is the next thing to read.
So the deliverable order is now: (i) read the FP flags for contraction (cheap, decides the claim), (ii) read npu_gemm_rows's readback for the per-op roles, (iii) then the dump, with the format I specified (a, a8, codes, d, out).
(i) is answered: FP contraction is the compiler default, but it cannot touch this expression — so my FMA hazard was overstated, and the honest gate is max|d| == 0 for the no-softcap path.
What the build actually does (.github/workflows/ci.yml:76): cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16, and git grep finds no -ffp-contract, no -fno-fast-math, no -fp-model anywhere in the tree. GCC's default for C++ is -ffp-contract=fast, so contraction is on.
But it cannot reach the epilogue, and this is the part I got wrong an hour ago:
float v = (float)cr[n - n0] * as * gs[(size_t)(n - n0)]; // two multiplies, no addition
FMA fuses a * b + c; a pure multiply chain has no addend to fuse, so v is exactly round32(round32(cr * as) * gs) — left-to-right, two roundings, independent of -ffp-contract. A host computing (np.float32(cr) * np.float32(as)) * np.float32(gs) in float32 matches it exactly.
So the layer-2 verdict, corrected for the second and final time:
path
reproducible bit-exactly by a host?
v = (cr*as)*gs (no softcap)
yes — two float32 multiplies, no fusion possible
with logit_softcap: softcap * tanh(v/softcap)
not guaranteed — tanhf is libm, and a difference of one ULP in the tangent propagates; needs the same libm or a stated ULP bound
So the gate can demand max|d| == 0on the softcap-disabled path and must state a ULP bound when a model enables the softcap. That is a much sharper claim than "layer 2 is tolerance-based", and it is the one I will implement against.
Where contraction does still matter: anywhere the path adds — the accumulation epilogues and the rmsnorm/residual sums. Those are not part of this gate's comparison (the gate compares logits), but if the per-op roles' readback turns out to fold scales with additions, -ffp-contract becomes live again there and the check above would have to be repeated on that expression.
Remaining from my (ii): the per-op roles' readback in npu_gemm_rows. Both lctx.Cm sites I found are lm_head; if QKV/O/GU/D read back through a different epilogue with per-group scales, their assertion stays layer-1-only until that expression is read the same way. That is the next read, and then the dump format is settled: a[K], a8[K], codes[K][N], d[K//GROUP][N], out[N].
Correction, and it invalidates the plan I posted above: the host int8 reference already exists — it was added for this issue, by someone else, and I should have checked before planning around its absence.
At current main in npu/forward/src/q4nx_forward.cpp:
if (constchar* hg = getenv("NPU_INFER_HOST_GEMM")) {
host_gemm_ = true;
// engine#1808: "i8" selects the integer arm, whose int32 accumulation is the only thing a// kernel family can be bit-exact against (an int8 device cannot match an f32 host by// construction). Any other value keeps the f32 path …
host_gemm_i8_ = (hg[0] == 'i' && hg[1] == '8');
}
and host_gemm_i8() implements it: "the inner sum is exact int32 with no float in it (q4nx_host_gemm_i8_grouped), so the comparison against the device is about accumulation and epilogue, not about a float both sides happen to share" — with G = 128 groups, per-tensor cached quantisation (measured: per-call quantisation made a 7B token ~7.5 G scalar ops and never finished inside a 9-minute ticket).
Carried by #1809 — commit b6db79e93, "npu: the integer host arm a kernel gate needs, because f32 cannot be bit-exact with int8 (Refs #1808)" — plus d3e8bd5f4. So:
The issue's premise is resolved in code.NPU_INFER_HOST_GEMM=i8 is the opt-in arm; =1 (or anything else) keeps f32, so no existing caller moved. An f32-only observation is what the issue recorded, and it is a knob away from the thing it says does not exist.
My dump plan is unnecessary. I proposed NPU_QKV_DUMP_FULL to compare a Python oracle against the device. host_gemm_i8 computes the reference in process, on the same weights, in the same run — which is strictly better than a dump round-trip: no format to define, no risk of comparing against a different pack, and it already handles the per-group epilogue (t.groups, t.d) that my plan was going to have to add.
What is actually still owed is the gate, not the arm — the max|d| == 0 assertion (softcap-off path, where my layer-2 read shows a pure float32 multiply chain that a host can match exactly; a stated ULP bound when logit_softcap is on), on 3 seeds plus an edge case, wired into the family gate.
Consequently I am standing down from this issue rather than working it alongside the lane that implemented i8: two lanes on one issue is how this fleet produced seven duplicate/divergence pairs tonight, and #1809 already has the better artifact. I will say plainly what I got wrong: I claimed #1808, checked gh pr list --search 1808 and branches (which found nothing), read the f32 arm, and planned an implementation without ever reading the NPU_INFER_HOST_GEMM handler — four lines that would have ended the investigation. The duplicate check that would have caught this is not "is there a PR for this issue" but "does the code the issue describes already exist", and I did not run it until I went looking for the per-op roles' epilogue.
What I am leaving behind, since it is still useful to whoever writes the gate: the layer-2 evaluation order (v = (float)cr * as * gs[col], per-column scale, float32, no FMA possible in a pure multiply chain; tanhf when softcap is on) and the int32 exactness bound (8·127·K, headroom ×1032 at K=2048). Those are gate-design facts, and they are independent of which arm provides the numbers.
Progress on this issue, and a sharper statement of what is left — the remaining 0.3 is not where I expected, and the way to settle it is an int32 comparison, not a float one.
The arm this issue asked for now exists and works (merged in #1809, corrected in #1830): weight image from the engine's own q4nx_pack_B_int8_blocked(), exact int32 accumulation, the same epilogue factors. Three-arm gate on the 4096 family, one ticket:
comparison
argmax steps 0-3
cosine
max abs logit Δ
device vs f32
same ×3, DIFF at 3
0.9960-0.9974
1.24-1.53 → 10.26
device vs int8 arm
same ×4
0.99927-0.99986
0.30-0.36
So the arm already answers the question this issue was opened with — is the kernel wrong, or is it quantisation? — and the answer is quantisation. But max|d| = 0.30 is not bit-exactness, and I have now ruled out by code inspection the two explanations I expected:
the activation scale: the device computes ascales[m] = amax/127 from A itself (xdna_target.cpp:982), which is exactly what the arm does;
the weight scale and codes: both sides call the same packer over the same f32 matrix with the same roundf/clamp, and for these shapes the padded and unpadded dims coincide (KD,ND = 4096/6144, 4096/4096, 4096/28672, 14336/4096), so the blocked permutation is the same.
Therefore the next step is not another whole-model run but an INT32-LEVEL comparison. The float logits can differ from epilogue ordering ((float)cm * asc * gs[n] on the device versus d[n] * (ascale * acc) here) while the integer accumulators agree exactly — and if the int32s agree, then "bit-exact at the accumulator" is the honest claim the plan's §4 can be read as asking for, with the float residue attributed to multiplication order rather than to operands.
Concretely: dump kctx.Cm (the device's post-readback int32 row) for one GEMM and the arm's acc for the same tensor and activation, and diff them as integers. That is one small debug hook plus one ≤9 min ticket, and it either closes this issue or names the operand that still differs. Worth stating plainly: it needs a device dump hook, so it is not a host-only change like the last three.
Excluded by measurement, not argument: float ordering cannot explain the arm's 0.30 residue — the gap is exactly 0.
The cheapest remaining explanation for device-vs-int8-arm max|d| = 0.30 was epilogue associativity: the device computes ((cm * asc) * gs) (xdna_target.cpp:1157) and the arm computes (gs * (asc * cm)), and float multiplication is not associative.
So the A/B now computes both orders for every element of every call and reports the largest gap. Over a whole request, on all four roles (QKV 4096×6144, O 4096×4096, GU 4096×28672, D 14336×4096):
epilogue_order_gap=0 (every call)
Excluded. So the 0.30 is in the operands: a code or a scale that differs between the device's packed image and q4nx_pack_B_int8_blocked()'s, even though both start from the same f32 matrix and the same roundf/clamp/amax floor.
That leaves exactly one measurement for this issue: dump the device's post-readback int32 (kctx.Cm) for one GEMM and diff it against the arm's accumulator, as integers.
If the int32s differ → the operand difference is real and the diff indexes it (which columns, by how much).
If they match → the accumulators are bit-exact and the float difference is entirely the epilogue's two multiplications, which the probe above bounds at 0 elsewhere; that would make "bit-exact at the accumulator" the honest §4 claim for int8 kernels.
Commit 10fc175ef on #1830 adds the probe behind the existing NPU_DBG_I8_AB var, so the exclusion is repeatable rather than a one-off. The int32 dump itself needs a device hook and one ≤9 min ticket — the first step in this thread that is not host-only.
The integer comparison is done, and it is a much better result than a cosine: the device's accumulators and the arm's are BIT-IDENTICAL for a whole token's pass.
Added env-gated int32 dumps on both sides (NPU_I8_INT32_DUMP: the device's post-readback kctx.Cm row, the arm's accumulator) and diffed them as integers, one npu-hold -t 9 ticket (169 s), same container, one token:
matched calls 760 (shape-aware pairing; the two sides dump in different orders)
exactly equal int32 accumulators: 400 / 760
first non-zero diff: call 400, N=6144, differing 6144/6144 columns, max|d| = 251163
calls before it: all zero
Reading it: 400 calls is a whole pass over the model — 12 calls per layer × 32 layers, plus the head. So the
device's integer GEMM arithmetic and this arm's agree bit for bit while both are on the same state. The
divergence then begins exactly at the second token's first QKV, and every call after it differs.
What that excludes, by measurement: the weight codes, the packing, the blocked permutation, the activation scale
and the epilogue ordering — all five are now ruled out for the first pass, since an operand difference anywhere would
have shown up there as a non-zero int32.
What it points at: the second token's activation, i.e. state — the KV cache (or whatever the device keeps
between steps) differs from what the host arm recomputes. That is consistent with the earlier float observation
(cos 0.9993 at step 0, drifting after): the first step agrees, and every later step is a consequence of the first
state difference.
For the plan's §4 wording: "bit-exact at the accumulator" is now a measured claim for a full token's pass rather
than an aspiration — the strongest statement this lane has been able to make about a kernel family, and it is stronger
than the cosine it replaces. Bit-exact logits over multiple steps still needs the state question answered, which is
the next step and is not a kernel-code problem.
Narrowed again, and one of my own hypotheses is now excluded: the two arms keep K/V in the SAME host buffers, so the post-token-1 divergence is not a KV-format difference.
Checked instead of assumed:
kcache_/vcache_ are std::vector<std::vector<float>>owned by the model and shared by both arms (model_.kcache, q4nx_forward.h:519) — the device arm does not keep its own copy in another dtype;
the forward's own header states the non-GEMM work, including GQA attention and the KV cache, runs on the host in float32;
the XDNA target has no KV buffers at all (grep -i kv over xdna_target.cpp finds no allocation).
So for the second token both arms read the same f32 K/V built by the same host code. Given that the int32 accumulators are bit-identical for the whole first pass, the only way token 2 can diverge is that token 1's float output differed — i.e. the difference is in the epilogue, not in the state and not in the GEMM.
And that is the one comparison my order probe did not make: it multiplied my factors in two orders, not my factors against the device'sgs. The device's per-column scale comes from the engine-src packB_blocked(); the arm now takes its scales from q4nx_pack_B_int8_blocked() — same rule, two different implementations, and nothing has yet compared them value by value.
So the next measurement is small and precise: dump the device's gs array (one line in xdna_target.cpp next to the Cm * asc * gs[n] loop) and diff it against the arm's d for the same tensor. Equal ⇒ the epilogue's remaining difference is elsewhere and the divergence needs re-deriving; not equal ⇒ the two packers disagree on a scale and that is the whole story.
For the record, the family's practical status does not change: device vs int8 arm is argmax-identical 4/4 steps at cos ≥ 0.9993, which is the gate evidence a row can cite today.
The scale comparison needs a keyed dump, not index alignment — and index alignment has now failed twice, so that is the next thing to change rather than retry.
Dumped the device's per-column gs (beside the Cm * asc * gs[n] epilogue) and the arm's d, one token, same container, then paired them by index:
Those numbers are not a result: the two sides call the GEMM a different number of times (the arm is reached from paths the device's launcher is not, and vice versa), so index pairing compares call 150 of one against call 150 of the other. The previous int32 comparison had the same problem and only worked because I paired on shape, which is a partial key — it matched 760 of 1408 and could not tell two same-shape calls apart.
So the next step is a format change, not another run: write the dump as gs_L<layer>_<role>_N<n>.bin on both sides, so the pairing is exact and the comparison is meaningful. Everything else is already in place (the hooks are five lines each, env-gated), and the same key fixes the int32 comparison — which is also why the "bit-identical for the first 400 calls" figure should be re-read as "bit-identical over the 400 calls that the shape pairing could match", not as a stronger claim than it is.
I am recording that qualification rather than letting a stronger sentence stand on a partial key.
The 4096 family's GEMMs are BIT-EXACT against an int8 host reference. Measured, with the pairing finally done by key.
Dumps paired by index first — meaningless, because the two sides reach the GEMM a different number of times (1024 arm dumps vs 384 device dumps on the same request). Keyed by (layer, N, K, ordinal):
The full chain that closes the GEMM question, each link measured:
property
result
activation scale
same formula both sides (amax/127, xdna_target.cpp:982)
epilogue multiplication order
gap 0 on every call (rate-order probe)
int32 accumulators, first token's whole pass
identical (shape-paired; see the caveat below)
per-column weight scales, keyed pairing
384/384 exactly equal, worst relative diff 0
So the device's GEMMs and the arm's compute the same integers from the same operands. What remains outside the GEMM: the two still differ by max|d| = 0.30 in logits, and the non-GEMM work (RoPE, attention, norms) is shared host code — so the remaining candidates are the fused/selected kernel paths (fused_gu=1 splits GU differently from the arm's single GEMM) and the head path.
Caveat I am keeping in the record, not dropping: the int32 comparison paired on shape, a partial key (760 of 1408 dumps matched), so it establishes bit-identity for the calls that pairing could match, not for every call. The weight-scale result above uses the full key and is not subject to that.
For a family row, the honest and now well-supported claim is:device vs int8 reference — argmax-identical 4/4 steps, cos ≥ 0.9993, and the GEMMs are bit-exact (weight scales 384/384 identical, accumulators identical where paired). The 0.30 logit residue is attributable to the path around the GEMM, not to the kernel.
The 0.30 is the LM HEAD, and it is by design: in host mode run_lm_head() is a plain float matvec, while the device's head reads a packed, quantised table.
Found by reading the code rather than running another ticket (q4nx_forward.cpp, run_lm_head):
if (host_gemm_) {
// Offline mode: plain float matvec against the (possibly dequantized) tied table,// matching the Python reference. No I8Ctx is touched.for (int n = 0; n < NV; n++) { float s = 0; for (int k = 0; k < H; k++) s += h[k] * inv_es0 * lm_at(n, k); ... }
}
and on the device path the head goes through run_lm_head_gemv / lm_head_packed_ with dequant_pb16i(...), i.e. a quantised table.
So the arm's integer path covers every body GEMM (npu_gemm / npu_gemm_rows) and not the head, which it computes in f32. That closes the last question on this issue without a measurement:
body GEMMs: bit-exact — weight scales 384/384 identical (worst relative difference 0), int32 accumulators identical for the first token's pass, epilogue order gap 0, same activation-scale formula;
logits: differ by max|d| = 0.30, and the only place the two arms compute the same tensor differently is the head's precision.
So "bit-exact logits" needs the integer head in the arm (or a device head in f32, which is not what ships). That is a bounded next step — one more call site rather than a new investigation — and until it exists the honest claim is "body GEMMs bit-exact, head differs by quantisation", which is what #1740 now carries.
Worth saying plainly: this is the fifth time tonight that reading the code answered what a run would have only hinted at, and it is free.
Design for the last piece (15 lines, per the plan's rule for anything over about a day): the integer head, so "bit-exact logits" becomes a claim this lane can make.
Why it is the last piece: the body GEMMs are bit-exact against the arm (weight scales 384/384 identical, worst relative difference 0; accumulators identical for the first token's pass; epilogue order gap 0; same activation-scale formula). The only tensor the two arms compute differently is the head: in host mode run_lm_head() is a plain f32 matvec — "No I8Ctx is touched" in its own comment — while the device reads a packed quantised table (lm_head_packed_, run_lm_head_gemv, dequant_pb16i).
What already exists to build on:q4nx_pack_B_int8_blocked() (the engine's own packer, one multiplier per column, int8 codes) and q4nx_host_gemm_i8_grouped() (exact int32 accumulation). The body arm already uses both; the head needs the same two calls with K=H, N=V.
Shape: the head is the one tensor big enough that a full int8 image costs real memory — 4096 x 151936 x 1 B = 622 MB at this vocabulary, against the body's per-layer images. So it is built once, in chunks by vocabulary block, and cached with the same (layer=-1, wkey, K, N)-style key the body uses.
Correction to the body's shortcut: the body arm caches a whole tensor's image; the head cannot. Chunking means the MAC loop must walk blocks (for block in V/CHUNK) and apply each block's scales, i.e. a new call rather than a flag on the existing one.
Cost per token: 4096 x 151936 int32 MACs ~ 6.2e8, which is more than the whole body's GEMM work at M=16. Expect the arm to slow noticeably; that is fine for a reference (it is off by default) but it should be measured, not discovered.
Correctness bar it unlocks: with the integer head in place, all arithmetic the device performs has an integer twin, so max|d| == 0 on logits becomes achievable and §4's wording can be satisfied literally rather than read down to "at the accumulator".
Risk to state up front: if the device's head packing differs from packB_blocked's (it may use the pb16i two-plane form, given head_pb16i_), the arm must reproduce that form or the claim stays "bit-exact except the head", honestly labelled.
Order of work: (a) dump the device head's gs and a few code bytes and compare against q4nx_pack_B_int8_blocked output for the same rows — same keyed-dump method that settled the body, one ticket; (b) only then write the arm's head path against whichever image the device actually reads; (c) re-run the three-arm gate and quote max|d|.
What I am not doing tonight: starting (b) on the assumption of which packing the device uses. The body cost four wrong turns to that exact mistake, and the keyed dump in (a) is what made the body answerable.
This is the honest next thread, written down before the next session rather than improvised at the end of this one.
The design note's biggest risk is closed by reading the code: the device's head tiles are packed with the SAME packer the body arm already uses.
The note said: "if the device's head packing differs from packB_blocked's (it may use the pb16i two-plane form, given head_pb16i_), the arm must reproduce that form or the claim stays 'bit-exact except the head'." Checked:
// q4nx_forward.cpp, the non-ternary head build
pw.bo = lctx.make_weight_bo(*dev_);
lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales); // -> packB_blocked
So for a b8 container like the Mistral 1.9 pack the head is packB_blocked: one multiplier per column, int8 codes — the same rule q4nx_pack_B_int8_blocked() implements, and the same one the body arm already reproduces bit-exactly (weight scales 384/384 identical). The pb16i two-plane path is a separate branch (head_pb16i_), taken only when the container carries a ternary image, and it is not the case here.
That makes the integer head a repetition rather than a new mechanism:
per vocabulary tile t of width T (the widest design's N), build codes/d with q4nx_pack_B_int8_blocked over the head rows [n0, n1), K = H — chunked because a full head image is 622 MB at this vocabulary;
quantise the hidden row to int8 with amax/127 (the device does the same, scales[m] = amax/127 at :5068);
exact int32 MAC per column, then d[n] * (ascale * acc);
cache per tile with a key that cannot collide (the body's lesson: (layer, wkey, K, N)).
Two things to keep from the body's four wrong turns: the arms must read the same image (so use the packer, not a re-quantisation) and the cache key must identify the tile, not just a role.
Worth stating: the head path in the forward already computes per-row scales identically to the arm (amax/127, inv_es factored the same way), so steps 2 and 3 are the only new lines.
BIT-EXACT. Device logits and the integer reference agree to the last bit, on every step of a full request.
step 0: argmax 3916 == 3916 cos 1.000000 max|d| 0
step 1: argmax 278 == 278 cos 1.000000 max|d| 0
step 2: argmax 28725 == 28725 cos 1.000000 max|d| 0
step 3: argmax 22478 == 22478 cos 1.000000 max|d| 0
BIT-EXACT over 4 step(s) at max|d| tolerance 0.0
One npu-hold -t 9 ticket, Mistral-7B-Instruct-v0.2 1.9 container on kernels-h4096-b8, greedy, device arm against NPU_INFER_HOST_GEMM=i8.
What closed it. The head was the last place the two arms computed the same tensor differently — host mode's run_lm_head() is a plain f32 matvec ("No I8Ctx is touched"), the device reads a packed quantised table. Its tiles come from lctx.packB_into(...) → packB_blocked, so the arm now builds them with the same rule via q4nx_pack_B_int8_blocked() (one multiplier per column, int8 codes), chunked by vocabulary block (8192 columns; a full image is 622 MB here), cached per block, MAC in exact int32 through q4nx_layout::blocked_index().
The chain, each link measured rather than argued:
activation scale — same formula (amax/127, xdna_target.cpp:982);
epilogue multiplication order — gap 0 on every call;
int32 accumulators — identical for a whole token's pass (shape-paired, caveat kept);
head — now integer on both sides ⇒ logits bit-exact.
This is TNPU-ALL §4's bar met literally — "bit-exact against the host reference, 3 repeats, edge case" — instead of being read down to "at the accumulator". The family row can now say bit-exact, not cos ≥ 0.9955.
Honest remaining items: 3 repeats and an explicit edge case on this exact configuration are the next formal step (this run was one request, four steps), and the same gate has to be run for the other three 4096 sets (im12288, mha, chatglm), which have no pack on this box that imports yet.
TNPU-ALL §4 is met in full for this family: three repeats, an edge case, and device determinism — all BIT-EXACT.
One npu-hold -t 9 ticket (279 s), kernels-h4096-b8, Mistral-7B-Instruct-v0.2 1.9, greedy, four generated steps per request:
arm
result
device vs integer reference — repeat 1
BIT-EXACT (max|d| = 0, cos 1.000000)
repeat 2
BIT-EXACT
repeat 3
BIT-EXACT
device determinism, run 1 vs run 2
BIT-EXACT
device determinism, run 1 vs run 3
BIT-EXACT
edge case: prompt Hi, device vs reference
BIT-EXACT (all four steps, argmax identical)
So this is the plan's bar taken literally — "bit-exact against the host reference on the device, on at least 3 seeds plus an edge case, and the same request repeated 3 times returns identical bytes" — rather than read down to a cosine or to "at the accumulator".
What it took, in order, each link measured: same activation-scale formula → epilogue multiplication-order gap 0 → int32 accumulators identical for a token's pass → per-column weight scales 384/384 exactly equal → an integer head built with the engine's own packer (the last non-integer op) → logits identical to the last bit.
Still not claimed here: the same gate for the other three 4096 sets (h4096-im12288, h4096-mha, h4096-chatglm), which have no pack on this box that imports yet. That is the next task, and it is now a repeatable one-command sequence.
Fixed on main; re-measured independently at ee5b436 and closing (lane cl-npurt).
The title's premise no longer holds: the host int8 reference path is on main — the integer arm (#1809, b6db79e), its operands taken from the engine's own packer (#1888, e2030cc), and the integer head in run_lm_head (same merge). NPU_INFER_HOST_GEMM=i8 selects it.
The bit-exact results above were taken on the branch before it merged, so I repeated the gate on a build of main with my own script and comparer (a byte comparison of the --dump-logits float32 vectors, nothing shared with the earlier tooling):
1bit built from ee5b436 (-DONEBIT_NPU=ON), Mistral-7B-Instruct-v0.2 1.9 container, greedy, 4 generated steps per request.
Reference arm: NPU_INFER_HOST_GEMM=i8 1bit forward-serve … --dump-logits (no device opened), prompts The capital of France is and Hi.
Device arm: three fresh serves under one npu-hold -t 9 ticket (84 s); the first also answers Hi.
reference vs device serve 1: step 0..3 argmax 3916/278/28725/22478 both sides max|d| 0 bytes IDENTICAL BIT-EXACT over 4 step(s)
reference vs device serve 2: BIT-EXACT over 4 step(s)
reference vs device serve 3: BIT-EXACT over 4 step(s)
device serve 1 vs 2, 1 vs 3: BIT-EXACT over 4 step(s) each
edge prompt "Hi": step 0..3 argmax 28725/22478/28747/315 both sides max|d| 0 bytes IDENTICAL BIT-EXACT over 4 step(s)
Text on both arms: Paris, Question and , Question: I.
Not covered, and not a reason to keep this issue open: only this one container and kernel set were gated; the other three 4096 sets have no importing pack on the box (recorded above), and a model with logit_softcap on goes through tanh, where bit-exactness is not guaranteed by construction. #1830 is superseded by #1888 and can be closed by its author.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claiming this for pi-kpkg (claiming here and on
~/briefs/TISSUES-BOARD.md— the duplicate-check other lanes run reads the issue, so the claim goes on both; I learned that on #1766 tonight, where a board-only claim was invisible to a later sweep).Why this is my area: the issue is not "add an int8 kernel" but "a gate that cannot fail is not a gate". Tonight I have reviewed five changes whose verification was a vacuous green — a fixture never asserted to exist, a test that passes against the broken code, a row that was red for an unrelated reason — and this is the same defect at the level of a whole family gate: a quality number standing where a bit-exactness verdict is required. That distinction is the thing I can contribute.
Checked before claiming (per the fleet's duplicate history):
gh pr list --search 1808is empty, no branch mentions it, and no other open PR touchesq4nx_forward.cpp's host path.The plan, and one refinement that should come first. The issue's route is right, and item 2 is what makes it cheap: the oracle already exists in Python (
tools/moe_expert_chunks.py,codesᵀ @ a8 → int32, device-verified ±0 on 2026-11-07). But before writing the C++NPU_INFER_HOST_GEMM=i8arm, run that oracle on one per-op role — not a MoE projection — and check it still agrees with the device bit-exactly:max|d| == 0with confidence.That is the same ordering as the fixture rule: establish the reference is a reference before asserting anything against it.
What I will not do: change the default host arm.
NPU_INFER_HOST_GEMM=i8as an opt-in keeps every existing f32 gate exactly as it is, so nothing that passes today can start failing because of this.Deliverable: the oracle check as a recorded measurement first (the numbers above are the template — per-seed, per-step, argmax + cosine + max-abs-Δ, so a disagreement is identifiable rather than a single "≠"), then the host GEMM arm, then the gate. If the oracle check refutes the route, I will say so on this issue instead of building it.
npu: NPU_INFER_HOST_GEMM=i8 selects the integer arm, so a family can …
bong-water-water-bong commentedon Nov 10, 2026
First step done, and it is the "is this a reference at all?" half — plus one blocker found in the plan's item 2.
1. The primitive is real but not callable without a pack. It is
tools/moe_expert_chunks.py:337:inside the
gatesubcommand, wherea8sis derived from a pack's activations. So the algorithm exists exactly as this issue says, butcodesᵀ @ a8cannot be pointed at a per-op role today — it is a line in a subcommand, not a function. Extracting it (int32_accum(codes_int8[K][N], a8_int8[K]) -> int32[N]) is a prerequisite of the comparison, and it is ~3 lines: that is the first commit, not the C++ arm.2. The oracle's actual contract, measured rather than assumed (module import +
quantise()on synthetic data, no pack, no device):Two of those numbers matter for the C++ arm:
GROUP = 128is the divisor the host reference must use (a host arm with the 32 of a different container would produce a different scale and disagree for arithmetic reasons), and the int4 range is[-8, 7](asymmetric about zero, because 4-bit two's complement), so+8never appears — the C++ side must clip identically or a single saturated weight shifts an int32 sum.3. The gate's bit-exactness premise survives, with a large margin — worth recording because it is why
max|d| == 0is even achievable:So no int32 overflow for any real per-op role, which means the device and the host can agree exactly rather than "to within an accumulate-width accident" — and if a future family had K beyond ~2.1 M the bound breaks and the gate would have to widen. Stating the bound is what makes the
max|d| == 0demand trustworthy rather than hopeful; without it, an exactness demand is a guess about accumulator width.Next, in order: (a) extract
int32_accumas a function and add an arm on synthetic data (pins the contract above, runs with no pack — same decompose-and-pin shape as the other host-only tests tonight); (b) run it on one per-op role's real weights versus the device under annpu-hold, on a single seed, and report the three-way picture (argmax / cosine /max|d|). If (b) showsmax|d| != 0, that is the finding, and the C++NPU_INFER_HOST_GEMM=i8arm should not be written until the disagreement is explained — a "reference" that fails the gate for its own reasons would be read as a kernel defect, which is the worst outcome available here.Not done in this step, deliberately: no C++ change, no device time. This is the read-only half, and it already changed the plan's first two items.
bong-water-water-bong commentedon Nov 10, 2026
Step (a) is PR #1817. While planning step (b) I checked whether the engine can already produce what the comparison needs, and it cannot — both existing dumps are lossy, so the plan needs one more increment before any device time.
int32_accum?NPU_QKV_DUMP=<layer>(q4nx_forward.cpp:2516)q0..q7,k0..k7,v0..v7,gate0..gate7)KNPU_INFER_DUMP_LAYERS(:2225)mean abs,max)a8So step (b) needs a small instrumented dump, and the cost of discovering that at a device window instead would be the whole window. The smallest thing that works:
NPU_QKV_DUMP_FULL=<layer>writinga[K](float32) plus, on the same run, the device's ownint32[N]for that row (or the logits, if the role is the head);quantise_i8_row(a)→int32_accum(codes, a8)→max|d|against the device's numbers, with the container'scodesread straight from the pack.That is a third step between (a) and (b), and it is the one that makes (b) cheap: one
npu-holdwindow, one seed, one role, and a verdict that is a number (max|d| == 0) rather than a cosine.Why I am writing this down rather than just doing it: an instrumented dump on the forward path is C++ in
npu/forward/src/q4nx_forward.cpp, and this lane's discipline is that a device change gets designed in the open first — the last time a dump was added for a measurement, the summary-statistics form is what came out, which is why this issue has to be stated this precisely now.Order from here: (1)
NPU_QKV_DUMP_FULL+ the device int32 for the same row (one PR, host-testable for its formatting with a synthetic row, no device); (2) onenpu-holdwindow running it on the 1.9 pack, single seed; (3) only thenNPU_INFER_HOST_GEMM=i8in C++, which is the arm the family gate would use. If step (2) showsmax|d| != 0, the C++ arm is not written until that is explained.tools: expose the int8 reference contract so a family gate can be bit…
bong-water-water-bong commentedon Nov 10, 2026
Planning step (1) turned up a structural correction to this issue's premise, and it is worth stating before anyone writes the C++ arm: "bit-exact" is two assertions, not one, and only one of them can be
max|d| == 0.The oracle says it itself, two lines below the primitive (
tools/moe_expert_chunks.py:362):and its quality half is a different computation:
yq = dequantise(codes, d).T @ a. So:codesᵀ · a8in int32. Order-independent, exact, and achievable asmax|d| == 0. The bound I pinned in tools: expose the int8 reference contract so a family gate can be bit-exact (engine#1808) #1817 (8·127·Kinside int32 with headroom ×1032 at K=2048) is what makes this layer trustworthy rather than hopeful. A host arm can be bit-identical here.dis[K//GROUP][N]withGROUP = 128, so the output isout[n] = ascale · Σ_g d[g,n] · (Σ_{k∈g} codes[k,n]·a8[k])— a group-wise scale multiply and a float32 sum over groups, not one scale over the whole row. A singlecodesᵀ·a8over allKcannot produce the output, and the device'slctx.group_scales[0](q4nx_forward.cpp:5339) is where that happens.Why the distinction changes the plan: at layer 2, float32 multiplication and addition are not associative, so the device and a host disagree unless the host mirrors the exact order — the group order, whether the scale multiplies each group's partial sum or is folded into a per-group accumulator, and whether the accumulation is float32 or double. That means:
max|d| == 0is legitimately demandable at layer 1;So the two-layer split is now the first deliverable of step (1), ahead of the dump: state, per layer, what is bit-exact and what is tolerance-based, and for layer 2 record the device's actual evaluation order from
q4nx_forward.cpp's epilogue. That is a code-reading task with a definite answer, and it is much cheaper to do than to discover on a device window that the host and device disagree in the last ULP for reasons that are correct.Consequence for the dump I proposed: it must carry
a[K],a8[K](so the engine's own quantisation is checked against the oracle's, which is a third assertion and a cheap one),codes[K][N],d[K//GROUP][N], and the device'sout[N]. Withoutdthe comparator can only do layer 1 — which is why I am writing this down before the format is implemented rather than after.Nothing here weakens the issue: a layer-1 gate that can genuinely fail is a far better artifact than a cosine, and it is the layer that catches the defects this issue was filed about (a mis-tiled or mis-scaled weight block changes the int32 sum). The correction is that layer 2 must not be claimed as bit-exact until its order is mirrored and verified.
npu: the integer host arm a kernel gate needs, because f32 cannot be …
bong-water-water-bong commentedon Nov 10, 2026
Layer 2's evaluation order, read out of the code — and it makes
max|d| == 0on logits achievable, which is a retraction of the caution I posted an hour ago.Both readback sites are the lm_head path and they are identical (
q4nx_forward.cpp:5054-5061and:5413-5418):So for the role the gate compares — the logits:
((float)cr * as) * gs[n], C++ left-to-right, all float32;softcap * tanh(v / softcap).Consequence, corrected: my earlier note said layer 2 could only be tolerance-based because of float non-associativity. For this path that is too pessimistic — a host that computes
(np.float32(cr) * np.float32(as)) * np.float32(gs)in float32 reproduces the chain exactly, because there is only one multiply order to match and no partial-sum tree. Somax|d| == 0on logits is a fair demand provided the host stays in float32 and keeps that association.The one real hazard left, and it is not float order — it is the compiler.
(float)cr * as * gsis exactly the shape a C++ compiler contracts into an FMA, andfma(x, as, 0)is not the same as(x * as) * gsrounded twice. Whether the device tree builds this with-ffp-contract=fast(the GCC/Clang default at-O2for many targets) decides whether a host can ever match. That is now the question to answer before writing the arm, and it is answerable by reading the flags, not by running anything:-ffp-contract=off/-std=...with strict FP), the host comparison is exact and the gate can demandmax|d| == 0;max|d| <= k ULPwithkmeasured and stated — still enormously stronger than a cosine, and honest about why it is not zero.What remains unverified: whether the per-op roles (QKV / O / GU / D) have their own epilogue elsewhere that applies
d[K//GROUP][N]group-wise. The twoCmreadback sites are both lm_head; if a role's epilogue is group-wise, its assertion is layer-1-only until its order is read the same way. I will not claim otherwise until I have found it — and the place it would live isnpu_gemm_rows's readback, which is the next thing to read.So the deliverable order is now: (i) read the FP flags for contraction (cheap, decides the claim), (ii) read
npu_gemm_rows's readback for the per-op roles, (iii) then the dump, with the format I specified (a, a8, codes, d, out).bong-water-water-bong commentedon Nov 10, 2026
(i) is answered: FP contraction is the compiler default, but it cannot touch this expression — so my FMA hazard was overstated, and the honest gate is
max|d| == 0for the no-softcap path.What the build actually does (
.github/workflows/ci.yml:76):cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16, andgit grepfinds no-ffp-contract, no-fno-fast-math, no-fp-modelanywhere in the tree. GCC's default for C++ is-ffp-contract=fast, so contraction is on.But it cannot reach the epilogue, and this is the part I got wrong an hour ago:
FMA fuses
a * b + c; a pure multiply chain has no addend to fuse, sovis exactlyround32(round32(cr * as) * gs)— left-to-right, two roundings, independent of-ffp-contract. A host computing(np.float32(cr) * np.float32(as)) * np.float32(gs)in float32 matches it exactly.So the layer-2 verdict, corrected for the second and final time:
v = (cr*as)*gs(no softcap)logit_softcap:softcap * tanh(v/softcap)tanhfis libm, and a difference of one ULP in the tangent propagates; needs the same libm or a stated ULP boundSo the gate can demand
max|d| == 0on the softcap-disabled path and must state a ULP bound when a model enables the softcap. That is a much sharper claim than "layer 2 is tolerance-based", and it is the one I will implement against.Where contraction does still matter: anywhere the path adds — the accumulation epilogues and the rmsnorm/residual sums. Those are not part of this gate's comparison (the gate compares logits), but if the per-op roles' readback turns out to fold scales with additions,
-ffp-contractbecomes live again there and the check above would have to be repeated on that expression.Remaining from my (ii): the per-op roles' readback in
npu_gemm_rows. Bothlctx.Cmsites I found are lm_head; if QKV/O/GU/D read back through a different epilogue with per-group scales, their assertion stays layer-1-only until that expression is read the same way. That is the next read, and then the dump format is settled:a[K],a8[K],codes[K][N],d[K//GROUP][N],out[N].bong-water-water-bong commentedon Nov 10, 2026
Correction, and it invalidates the plan I posted above: the host int8 reference already exists — it was added for this issue, by someone else, and I should have checked before planning around its absence.
At current
maininnpu/forward/src/q4nx_forward.cpp:and
host_gemm_i8()implements it: "the inner sum is exact int32 with no float in it (q4nx_host_gemm_i8_grouped), so the comparison against the device is about accumulation and epilogue, not about a float both sides happen to share" — withG = 128groups, per-tensor cached quantisation (measured: per-call quantisation made a 7B token ~7.5 G scalar ops and never finished inside a 9-minute ticket).Carried by #1809 — commit
b6db79e93, "npu: the integer host arm a kernel gate needs, because f32 cannot be bit-exact with int8 (Refs #1808)" — plusd3e8bd5f4. So:NPU_INFER_HOST_GEMM=i8is the opt-in arm;=1(or anything else) keeps f32, so no existing caller moved. An f32-only observation is what the issue recorded, and it is a knob away from the thing it says does not exist.NPU_QKV_DUMP_FULLto compare a Python oracle against the device.host_gemm_i8computes the reference in process, on the same weights, in the same run — which is strictly better than a dump round-trip: no format to define, no risk of comparing against a different pack, and it already handles the per-group epilogue (t.groups,t.d) that my plan was going to have to add.max|d| == 0assertion (softcap-off path, where my layer-2 read shows a pure float32 multiply chain that a host can match exactly; a stated ULP bound whenlogit_softcapis on), on 3 seeds plus an edge case, wired into the family gate.Consequently I am standing down from this issue rather than working it alongside the lane that implemented
i8: two lanes on one issue is how this fleet produced seven duplicate/divergence pairs tonight, and #1809 already has the better artifact. I will say plainly what I got wrong: I claimed #1808, checkedgh pr list --search 1808and branches (which found nothing), read the f32 arm, and planned an implementation without ever reading theNPU_INFER_HOST_GEMMhandler — four lines that would have ended the investigation. The duplicate check that would have caught this is not "is there a PR for this issue" but "does the code the issue describes already exist", and I did not run it until I went looking for the per-op roles' epilogue.What I am leaving behind, since it is still useful to whoever writes the gate: the layer-2 evaluation order (
v = (float)cr * as * gs[col], per-column scale, float32, no FMA possible in a pure multiply chain;tanhfwhen softcap is on) and the int32 exactness bound (8·127·K, headroom ×1032 at K=2048). Those are gate-design facts, and they are independent of which arm provides the numbers.npu: the integer arm's code cache collided across layers, which is wh…
bong-water-water-bong commentedon Nov 10, 2026
Progress on this issue, and a sharper statement of what is left — the remaining 0.3 is not where I expected, and the way to settle it is an int32 comparison, not a float one.
The arm this issue asked for now exists and works (merged in #1809, corrected in #1830): weight image from the engine's own
q4nx_pack_B_int8_blocked(), exact int32 accumulation, the same epilogue factors. Three-arm gate on the 4096 family, one ticket:So the arm already answers the question this issue was opened with — is the kernel wrong, or is it quantisation? — and the answer is quantisation. But
max|d| = 0.30is not bit-exactness, and I have now ruled out by code inspection the two explanations I expected:ascales[m] = amax/127from A itself (xdna_target.cpp:982), which is exactly what the arm does;roundf/clamp, and for these shapes the padded and unpadded dims coincide (KD,ND = 4096/6144, 4096/4096, 4096/28672, 14336/4096), so the blocked permutation is the same.Therefore the next step is not another whole-model run but an INT32-LEVEL comparison. The float logits can differ from epilogue ordering (
(float)cm * asc * gs[n]on the device versusd[n] * (ascale * acc)here) while the integer accumulators agree exactly — and if the int32s agree, then "bit-exact at the accumulator" is the honest claim the plan's §4 can be read as asking for, with the float residue attributed to multiplication order rather than to operands.Concretely: dump
kctx.Cm(the device's post-readback int32 row) for one GEMM and the arm'saccfor the same tensor and activation, and diff them as integers. That is one small debug hook plus one ≤9 min ticket, and it either closes this issue or names the operand that still differs. Worth stating plainly: it needs a device dump hook, so it is not a host-only change like the last three.bong-water-water-bong commentedon Nov 10, 2026
Excluded by measurement, not argument: float ordering cannot explain the arm's 0.30 residue — the gap is exactly 0.
The cheapest remaining explanation for device-vs-int8-arm
max|d| = 0.30was epilogue associativity: the device computes((cm * asc) * gs)(xdna_target.cpp:1157) and the arm computes(gs * (asc * cm)), and float multiplication is not associative.So the A/B now computes both orders for every element of every call and reports the largest gap. Over a whole request, on all four roles (QKV 4096×6144, O 4096×4096, GU 4096×28672, D 14336×4096):
Excluded. So the 0.30 is in the operands: a code or a scale that differs between the device's packed image and
q4nx_pack_B_int8_blocked()'s, even though both start from the same f32 matrix and the sameroundf/clamp/amax floor.That leaves exactly one measurement for this issue: dump the device's post-readback int32 (
kctx.Cm) for one GEMM and diff it against the arm's accumulator, as integers.Commit
10fc175efon #1830 adds the probe behind the existingNPU_DBG_I8_ABvar, so the exclusion is repeatable rather than a one-off. The int32 dump itself needs a device hook and one ≤9 min ticket — the first step in this thread that is not host-only.bong-water-water-bong commentedon Nov 10, 2026
The integer comparison is done, and it is a much better result than a cosine: the device's accumulators and the arm's are BIT-IDENTICAL for a whole token's pass.
Added env-gated int32 dumps on both sides (
NPU_I8_INT32_DUMP: the device's post-readbackkctx.Cmrow, the arm's accumulator) and diffed them as integers, onenpu-hold -t 9ticket (169 s), same container, one token:Reading it: 400 calls is a whole pass over the model — 12 calls per layer × 32 layers, plus the head. So the
device's integer GEMM arithmetic and this arm's agree bit for bit while both are on the same state. The
divergence then begins exactly at the second token's first QKV, and every call after it differs.
What that excludes, by measurement: the weight codes, the packing, the blocked permutation, the activation scale
and the epilogue ordering — all five are now ruled out for the first pass, since an operand difference anywhere would
have shown up there as a non-zero int32.
What it points at: the second token's activation, i.e. state — the KV cache (or whatever the device keeps
between steps) differs from what the host arm recomputes. That is consistent with the earlier float observation
(cos 0.9993 at step 0, drifting after): the first step agrees, and every later step is a consequence of the first
state difference.
For the plan's §4 wording: "bit-exact at the accumulator" is now a measured claim for a full token's pass rather
than an aspiration — the strongest statement this lane has been able to make about a kernel family, and it is stronger
than the cosine it replaces. Bit-exact logits over multiple steps still needs the state question answered, which is
the next step and is not a kernel-code problem.
bong-water-water-bong commentedon Nov 10, 2026
Narrowed again, and one of my own hypotheses is now excluded: the two arms keep K/V in the SAME host buffers, so the post-token-1 divergence is not a KV-format difference.
Checked instead of assumed:
kcache_/vcache_arestd::vector<std::vector<float>>owned by the model and shared by both arms (model_.kcache,q4nx_forward.h:519) — the device arm does not keep its own copy in another dtype;grep -i kvoverxdna_target.cppfinds no allocation).So for the second token both arms read the same f32 K/V built by the same host code. Given that the int32 accumulators are bit-identical for the whole first pass, the only way token 2 can diverge is that token 1's float output differed — i.e. the difference is in the epilogue, not in the state and not in the GEMM.
And that is the one comparison my order probe did not make: it multiplied my factors in two orders, not my factors against the device's
gs. The device's per-column scale comes from the engine-srcpackB_blocked(); the arm now takes its scales fromq4nx_pack_B_int8_blocked()— same rule, two different implementations, and nothing has yet compared them value by value.So the next measurement is small and precise: dump the device's
gsarray (one line inxdna_target.cppnext to theCm * asc * gs[n]loop) and diff it against the arm'sdfor the same tensor. Equal ⇒ the epilogue's remaining difference is elsewhere and the divergence needs re-deriving; not equal ⇒ the two packers disagree on a scale and that is the whole story.For the record, the family's practical status does not change: device vs int8 arm is argmax-identical 4/4 steps at cos ≥ 0.9993, which is the gate evidence a row can cite today.
bong-water-water-bong commentedon Nov 10, 2026
The scale comparison needs a keyed dump, not index alignment — and index alignment has now failed twice, so that is the next thing to change rather than retry.
Dumped the device's per-column
gs(beside theCm * asc * gs[n]epilogue) and the arm'sd, one token, same container, then paired them by index:Those numbers are not a result: the two sides call the GEMM a different number of times (the arm is reached from paths the device's launcher is not, and vice versa), so index pairing compares call 150 of one against call 150 of the other. The previous int32 comparison had the same problem and only worked because I paired on shape, which is a partial key — it matched 760 of 1408 and could not tell two same-shape calls apart.
So the next step is a format change, not another run: write the dump as
gs_L<layer>_<role>_N<n>.binon both sides, so the pairing is exact and the comparison is meaningful. Everything else is already in place (the hooks are five lines each, env-gated), and the same key fixes the int32 comparison — which is also why the "bit-identical for the first 400 calls" figure should be re-read as "bit-identical over the 400 calls that the shape pairing could match", not as a stronger claim than it is.I am recording that qualification rather than letting a stronger sentence stand on a partial key.
bong-water-water-bong commentedon Nov 10, 2026
The 4096 family's GEMMs are BIT-EXACT against an int8 host reference. Measured, with the pairing finally done by key.
Dumps paired by index first — meaningless, because the two sides reach the GEMM a different number of times (1024 arm dumps vs 384 device dumps on the same request). Keyed by
(layer, N, K, ordinal):The full chain that closes the GEMM question, each link measured:
amax/127,xdna_target.cpp:982)So the device's GEMMs and the arm's compute the same integers from the same operands. What remains outside the GEMM: the two still differ by
max|d| = 0.30in logits, and the non-GEMM work (RoPE, attention, norms) is shared host code — so the remaining candidates are the fused/selected kernel paths (fused_gu=1splits GU differently from the arm's single GEMM) and the head path.Caveat I am keeping in the record, not dropping: the int32 comparison paired on shape, a partial key (760 of 1408 dumps matched), so it establishes bit-identity for the calls that pairing could match, not for every call. The weight-scale result above uses the full key and is not subject to that.
For a family row, the honest and now well-supported claim is: device vs int8 reference — argmax-identical 4/4 steps, cos ≥ 0.9993, and the GEMMs are bit-exact (weight scales 384/384 identical, accumulators identical where paired). The 0.30 logit residue is attributable to the path around the GEMM, not to the kernel.
bong-water-water-bong commentedon Nov 10, 2026
The 0.30 is the LM HEAD, and it is by design: in host mode
run_lm_head()is a plain float matvec, while the device's head reads a packed, quantised table.Found by reading the code rather than running another ticket (
q4nx_forward.cpp,run_lm_head):and on the device path the head goes through
run_lm_head_gemv/lm_head_packed_withdequant_pb16i(...), i.e. a quantised table.So the arm's integer path covers every body GEMM (
npu_gemm/npu_gemm_rows) and not the head, which it computes in f32. That closes the last question on this issue without a measurement:max|d| = 0.30, and the only place the two arms compute the same tensor differently is the head's precision.So "bit-exact logits" needs the integer head in the arm (or a device head in f32, which is not what ships). That is a bounded next step — one more call site rather than a new investigation — and until it exists the honest claim is "body GEMMs bit-exact, head differs by quantisation", which is what #1740 now carries.
Worth saying plainly: this is the fifth time tonight that reading the code answered what a run would have only hinted at, and it is free.
bong-water-water-bong commentedon Nov 10, 2026
Design for the last piece (15 lines, per the plan's rule for anything over about a day): the integer head, so "bit-exact logits" becomes a claim this lane can make.
run_lm_head()is a plain f32 matvec — "No I8Ctx is touched" in its own comment — while the device reads a packed quantised table (lm_head_packed_,run_lm_head_gemv,dequant_pb16i).q4nx_pack_B_int8_blocked()(the engine's own packer, one multiplier per column, int8 codes) andq4nx_host_gemm_i8_grouped()(exact int32 accumulation). The body arm already uses both; the head needs the same two calls withK=H,N=V.(layer=-1, wkey, K, N)-style key the body uses.for block in V/CHUNK) and apply each block's scales, i.e. a new call rather than a flag on the existing one.max|d| == 0on logits becomes achievable and §4's wording can be satisfied literally rather than read down to "at the accumulator".packB_blocked's (it may use the pb16i two-plane form, givenhead_pb16i_), the arm must reproduce that form or the claim stays "bit-exact except the head", honestly labelled.gsand a few code bytes and compare againstq4nx_pack_B_int8_blockedoutput for the same rows — same keyed-dump method that settled the body, one ticket; (b) only then write the arm's head path against whichever image the device actually reads; (c) re-run the three-arm gate and quotemax|d|.This is the honest next thread, written down before the next session rather than improvised at the end of this one.
bong-water-water-bong commentedon Nov 10, 2026
The design note's biggest risk is closed by reading the code: the device's head tiles are packed with the SAME packer the body arm already uses.
The note said: "if the device's head packing differs from
packB_blocked's (it may use the pb16i two-plane form, givenhead_pb16i_), the arm must reproduce that form or the claim stays 'bit-exact except the head'." Checked:So for a b8 container like the Mistral 1.9 pack the head is
packB_blocked: one multiplier per column, int8 codes — the same ruleq4nx_pack_B_int8_blocked()implements, and the same one the body arm already reproduces bit-exactly (weight scales 384/384 identical). The pb16i two-plane path is a separate branch (head_pb16i_), taken only when the container carries a ternary image, and it is not the case here.That makes the integer head a repetition rather than a new mechanism:
tof widthT(the widest design's N), buildcodes/dwithq4nx_pack_B_int8_blockedover the head rows[n0, n1),K = H— chunked because a full head image is 622 MB at this vocabulary;amax/127(the device does the same,scales[m] = amax/127at:5068);d[n] * (ascale * acc);(layer, wkey, K, N)).Two things to keep from the body's four wrong turns: the arms must read the same image (so use the packer, not a re-quantisation) and the cache key must identify the tile, not just a role.
Worth stating: the head path in the forward already computes per-row scales identically to the arm (
amax/127,inv_esfactored the same way), so steps 2 and 3 are the only new lines.bong-water-water-bong commentedon Nov 10, 2026
BIT-EXACT. Device logits and the integer reference agree to the last bit, on every step of a full request.
One
npu-hold -t 9ticket, Mistral-7B-Instruct-v0.2 1.9 container onkernels-h4096-b8, greedy, device arm againstNPU_INFER_HOST_GEMM=i8.What closed it. The head was the last place the two arms computed the same tensor differently — host mode's
run_lm_head()is a plain f32 matvec ("No I8Ctx is touched"), the device reads a packed quantised table. Its tiles come fromlctx.packB_into(...)→packB_blocked, so the arm now builds them with the same rule viaq4nx_pack_B_int8_blocked()(one multiplier per column, int8 codes), chunked by vocabulary block (8192 columns; a full image is 622 MB here), cached per block, MAC in exact int32 throughq4nx_layout::blocked_index().The chain, each link measured rather than argued:
amax/127,xdna_target.cpp:982);This is TNPU-ALL §4's bar met literally — "bit-exact against the host reference, 3 repeats, edge case" — instead of being read down to "at the accumulator". The family row can now say bit-exact, not cos ≥ 0.9955.
Honest remaining items: 3 repeats and an explicit edge case on this exact configuration are the next formal step (this run was one request, four steps), and the same gate has to be run for the other three 4096 sets (
im12288,mha,chatglm), which have no pack on this box that imports yet.bong-water-water-bong commentedon Nov 10, 2026
TNPU-ALL §4 is met in full for this family: three repeats, an edge case, and device determinism — all BIT-EXACT.
One
npu-hold -t 9ticket (279 s),kernels-h4096-b8, Mistral-7B-Instruct-v0.2 1.9, greedy, four generated steps per request:Hi, device vs referenceSo this is the plan's bar taken literally — "bit-exact against the host reference on the device, on at least 3 seeds plus an edge case, and the same request repeated 3 times returns identical bytes" — rather than read down to a cosine or to "at the accumulator".
What it took, in order, each link measured: same activation-scale formula → epilogue multiplication-order gap 0 → int32 accumulators identical for a token's pass → per-column weight scales 384/384 exactly equal → an integer head built with the engine's own packer (the last non-integer op) → logits identical to the last bit.
Still not claimed here: the same gate for the other three 4096 sets (
h4096-im12288,h4096-mha,h4096-chatglm), which have no pack on this box that imports yet. That is the next task, and it is now a repeatable one-command sequence.npu: the integer arm's operands come from the engine's packer, not a …
bong-water-water-bong commentedon Nov 10, 2026
Fixed on main; re-measured independently at
ee5b436and closing (lane cl-npurt).The title's premise no longer holds: the host int8 reference path is on main — the integer arm (#1809,
b6db79e), its operands taken from the engine's own packer (#1888,e2030cc), and the integer head inrun_lm_head(same merge).NPU_INFER_HOST_GEMM=i8selects it.The bit-exact results above were taken on the branch before it merged, so I repeated the gate on a build of main with my own script and comparer (a byte comparison of the
--dump-logitsfloat32 vectors, nothing shared with the earlier tooling):1bitbuilt fromee5b436(-DONEBIT_NPU=ON), Mistral-7B-Instruct-v0.2 1.9 container, greedy, 4 generated steps per request.NPU_INFER_HOST_GEMM=i8 1bit forward-serve … --dump-logits(no device opened), promptsThe capital of France isandHi.npu-hold -t 9ticket (84 s); the first also answersHi.Text on both arms:
Paris, Questionand, Question: I.Not covered, and not a reason to keep this issue open: only this one container and kernel set were gated; the other three 4096 sets have no importing pack on the box (recorded above), and a model with
logit_softcapon goes throughtanh, where bit-exactness is not guaranteed by construction. #1830 is superseded by #1888 and can be closed by its author.