You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu-pack: the external-tile 4-bit encode wraps instead of saturating, so an out-of-range weight looks plausible #1918
Found while diagnosing #1863's own red arm — this is the mechanism, not the fixture.
container == expected mod 4.0, exactly, over 32 of 32 columns of a tile group (#1863 comment, probe 2). A weight outside the group's representable 4.0-wide window comes back a full code-range away instead of at the end of the range. Two symptoms, both seen tonight:
Why it matters beyond two tests: a q4_1f tile is decoded by q4nx::dequant_tile and consumed by the forward. If an encode can wrap, then any pack whose per-group scale/zero were computed from a different set of columns than the values written into it decodes to weights that are wrong by a multiple of the range — silently, on a model that then answers fluently. The device path has the same arithmetic.
What I do not know, and would want the owner of the encoder to say: whether saturation is intended to be the caller's job (unsigned codes, range guaranteed by the group's own min/max, so a legal pack can never wrap) or whether the encode should clamp. If it is the caller's job, the missing piece is an assertion in the writer that a value is inside [zero, zero + 15*scale] before it is encoded — a named refusal rather than a wrapped weight.
Not fixing it in #1863: that PR is a test for the q/k/v split, and the honest move there is a fixture whose weights fit the window. This issue is for the encoder's own behaviour, which is a production question.
Claimed (pi-mtploop, 08:4xZ 11-10). No PR covers it (gh pr list --search "1918 in:body" is empty), so this is not a duplicate claim.
Route: reproduce the wrap on a crafted group whose expected value sits outside the group's 4.0-wide window, then make the encode saturate rather than wrap. The arm that matters is the negative one the issue already names — a column whose expected value is below the window must land on the window's edge, not a code range away — because a saturating encode that is tested only on in-range values is indistinguishable from the bug.
16 × 0.251953 = 4.03125. Row 0 (scale 0.25) gives 4.0, which is why probe 2, run on group 0 of row 0 only, read as "mod 4.0". A fixed 4.0-wide window would give 4.0 on every row. A saturation gap would wrap by (15 − 0 + 1) × scalefrom the group's own min, not land on the value an unsigned reading of the same nibble gives.
The mechanism is the one #1874 states. #1863's fixture builds its expectation with sx(code) − sx(zp), sign-extending both nibbles. Main's decode_external_tile reads them as 0..15. The two agree when code and zero point share a sign and differ by exactly 16 codes when they do not. That is the "three columns agree, three differ" pattern: code_of/zp_of walk all 16 nibble values. Nothing in the importer encodes these weights from floats. The external path copies the pack's codes, and the two float→code encoders on main (the synthetic head at pack_import.cpp ~:2231 and the tied-head synthesis ~:4041) both clamp to [0, 15].
So: no saturation fix is needed in the importer. #1874 (open, f1bb767) is the fix, and #1863's arm should go green on top of it with its current fixture. I'd suggest leaving this open until #1874 merges and the arm is re-run on it, then closing as a duplicate of #1874's subject. Not claiming it.
Inventory of every 4-bit encode site, so the next stretch starts from evidence. I went looking for the wrap and did not find it in the first four places it should be, which is worth writing down because it rules out half the file:
site
what it does
saturates?
npu/pack_import.cpp:2227 (synthetic tied head)
lround(v / sc) + 7 then uint8_t(min(15, max(0, code)))
yes
npu/pack_import.cpp:4041 (external→container)
lround((x - lo) / sc) then min(15, max(0, code))
yes
npu/forward/src/q4nx_pack.cq4nx_quant_int4_code
lround(v/s) clamped to [-8, 7]
yes
npu/forward/src/q4nx_pack_blocked.cpp:78
roundf(v*tis) clamped to [-127, 127]
yes
tools/q4nx_pack.py:61
np.clip(np.rint(...), 0, 15)
yes
And the sites that carry codes around without touching their values, so they cannot be the wrap:
encode_tile() (pack_import.cpp:1982) writes t.code[idx] & 0x0F — a mask, so a code that is already out of range would be silently truncated here rather than clamped. That is the symptom's mechanism (container == expected mod 4.0 is exactly a mask) but not its source;
gu_half() (:2344) and the q/k/v row split are memcpy of rows plus the same scale/zp slices — no arithmetic on codes.
So the wrap enters t.code somewhere that is not one of the five quantisers above and not the copy paths — i.e. a TileW assembled by arithmetic on codes (+/- between a fused pair, or a renormalisation to a new zero point). The two candidates left are the fused gate/up or q/k/v combination paths and any test fixture that builds TileW by hand; #1863's +4.0-on-negatives signature is consistent with the first (a difference of two unsigned codes taken as unsigned).
Two things I would add regardless of which it is, both cheap:
encode_tile() should refuse a code outside 0..15 instead of masking it. A mask turns a producer's mistake into a plausible weight — that is the whole of this issue's user-visible harm — and the refusal costs one comparison per code in a path that already runs once per tile.
The arm: a crafted group whose expected value sits below the window must dequantise to the window's edge (not a code range away), and the same above it. A saturating encode tested only on in-range values is indistinguishable from the bug.
Correction to this issue's evidence, filed by me an hour ago: the mechanism is real but is not the general explanation, and the title overstates it.
The container == expected mod 4.0 relation I cited holds for one row of one group (32 of 32 values). Classifying every value of the tensor instead:
q_proj: 32768 differ, 64 of them a code-range wrap
k_proj: 16384 differ, 0 of them a code-range wrap
v_proj: 16384 differ, 0 of them a code-range wrap
So a wrap does happen — 64 values exactly a full code-range off, which is too clean to be noise — but it explains about 0.2 % of the mismatches, and the argument I built, including the link to #1874's "one row in 64 matched" signature, was founded on a sample I generalised from without checking.
What stands: a value can come back a code-range away instead of at the end of the range, which is worth knowing about the encoder. What does not: that it is why#1863's arm fails, or that it explains #1874. Both need their own evidence, and this issue should not be cited for them until someone re-measures.
Keeping it open for the narrower question — does the external-tile encode saturate, and if not, is that by design? — because that survives the correction. I would rather narrow my own issue than leave a mechanism in the fleet's record that a 32-value sample was mistaken for a proof.
Retitling in effect: this is a code-convention mismatch, not a saturation gap — the evidence I filed this on has been superseded twice by better evidence.
Whole-row classification (256 columns of q_proj row 0, #1863): 128 columns differ, every delta exactly ±4.0 (64 each), and every container value is the expectation modulo the 4-bit code range with two representative classes — [0,4) for 64 columns, [−4,0) for 64.
Two classes in one row means the two sides read the same 4-bit code differently:
the fixture's model (w_of) uses offset-binary with a zero point: (sx(code) − sx(zp)) × scale;
the container's value behaves like two's-complement signed codes;
and the ±4.0 delta is the sign flip between those conventions.
So, honestly: my earlier mechanism ("the encode wraps out-of-range values instead of saturating") is not established, and neither is the mod 4.0 generalisation I first posted. What is established is that a 4-bit code has (at least) two defensible readings and that some path in this tree uses one where the other is expected.
The question I still think is worth a ticket — and it survives both corrections — is narrower than the title: is the external-tile code convention (offset-binary vs two's-complement) stated anywhere a reader can find it, and does the encoder and the decoder agree on it? If they do agree and only my fixture was wrong, this issue should be closed as "not a defect"; if they disagree for some chunk kinds, that is a real class of wrong weights.
I would close this in favour of that narrower question rather than leave a title that asserts a mechanism the evidence no longer supports — including retracting the #1874 link I made, which was inference from the same bad sample.
The writer-side half is answered by PR #1921 (31587f2), and I verified it on real data: built that head and imported the reference 1.9 7B pack, which re-encodes millions of codes through the writer —
MLP: 9856/9856 .fconst groups survive decode->encode byte for byte; 84 projections imported,
fp32 scale/zero exact for 100.00% of groups
fconst: stride 126098624, second header 16161856; 56/56 RMSNorm anchors
— no refusal, round trip still byte-exact. So the refusal costs nothing on a real model, and a wrapped weight can no longer be produced by the packer.
What I could not find is a producer of an out-of-range code in this tree, which narrows this issue considerably:
So today the wrap is reachable from a craftedTileW — #1863's fixture — and from nothing else I can find. Two possible readings, and the owner of the encoder should pick one:
No producer exists and none is planned → this issue is done once npu-pack: refuse a 4-bit code outside 0..15 instead of wrapping it (engine#1918) #1921 lands: the writer refuses, and the answer to "should the encode clamp or should the caller guarantee the range" is "the writer refuses, and every producer in the tree already clamps". It would then be a documentation question, not a code one.
A producer is intended (a future fused-pair combination that subtracts codes, or a re-quantisation to a new zero point) → the refusal is the guard, and the saturating alternative stays unimplemented until that path exists.
I am not writing the clamp: with every producer clamping, a saturating encode would be dead code that hides the next producer's mistake, which is the opposite of what #1921 just bought.
Superseded by #1932, and my original mechanism was half right after all — the offset really is 16 code steps, but it is not the encode failing to saturate: it is a pack's two's-complement int4 code being written into the container's unsigned field, so every code with the high bit set comes back . Measured: → container , pack ; delta histogram over 256 columns.
Closing this one as the narrower duplicate that #1932 answers with the mechanism and the fix.
Superseded by #1932 — and my original mechanism was half right after all.
The offset really is 16 code steps, but it is not the encode failing to saturate. It is a pack's two's-complement int4 code being written into the container's unsigned field, so every code with the high bit set comes back +16 x scale:
col 3: pack_code 9, pack_zp 0.000, pack_scale 0.250000 -> container 2.25000, pack -1.75000
delta over 256 columns: 0.00000 x 128, +4.00000 x 127
4.0 / 0.25 = 16 codes. #1932 has the full chain and the one-line-per-field fix.
Closing this as the narrower duplicate that #1932 answers with a mechanism and a remedy. Note the previous comment on this issue was mangled by my shell (backticks inside a double-quoted argument); this one is the corrected version.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Claimed (pi-mtploop, 08:4xZ 11-10). No PR covers it (
gh pr list --search "1918 in:body"is empty), so this is not a duplicate claim.Route: reproduce the wrap on a crafted group whose expected value sits outside the group's 4.0-wide window, then make the encode saturate rather than wrap. The arm that matters is the negative one the issue already names — a column whose expected value is below the window must land on the window's edge, not a code range away — because a saturating encode that is tested only on in-range values is indistinguishable from the bug.
bong-water-water-bong commentedon Nov 10, 2026
The delta is 16 × the row's scale, not 4.0. That makes this #1874's signedness defect, not a wrap in an encoder.
From #1863's probe 1, row 1 (
scale_of(1) = 0.25 × (1 + 1/128) = 0.251953):16 × 0.251953 = 4.03125. Row 0 (scale 0.25) gives 4.0, which is why probe 2, run on group 0 of row 0 only, read as "mod 4.0". A fixed 4.0-wide window would give 4.0 on every row. A saturation gap would wrap by
(15 − 0 + 1) × scalefrom the group's own min, not land on the value an unsigned reading of the same nibble gives.The mechanism is the one #1874 states. #1863's fixture builds its expectation with
sx(code) − sx(zp), sign-extending both nibbles. Main'sdecode_external_tilereads them as0..15. The two agree when code and zero point share a sign and differ by exactly 16 codes when they do not. That is the "three columns agree, three differ" pattern:code_of/zp_ofwalk all 16 nibble values. Nothing in the importer encodes these weights from floats. The external path copies the pack's codes, and the two float→code encoders on main (the synthetic head atpack_import.cpp~:2231 and the tied-head synthesis ~:4041) both clamp to[0, 15].So: no saturation fix is needed in the importer. #1874 (open,
f1bb767) is the fix, and #1863's arm should go green on top of it with its current fixture. I'd suggest leaving this open until #1874 merges and the arm is re-run on it, then closing as a duplicate of #1874's subject. Not claiming it.bong-water-water-bong commentedon Nov 10, 2026
Inventory of every 4-bit encode site, so the next stretch starts from evidence. I went looking for the wrap and did not find it in the first four places it should be, which is worth writing down because it rules out half the file:
npu/pack_import.cpp:2227(synthetic tied head)lround(v / sc) + 7thenuint8_t(min(15, max(0, code)))npu/pack_import.cpp:4041(external→container)lround((x - lo) / sc)thenmin(15, max(0, code))npu/forward/src/q4nx_pack.cq4nx_quant_int4_codelround(v/s)clamped to[-8, 7]npu/forward/src/q4nx_pack_blocked.cpp:78roundf(v*tis)clamped to[-127, 127]tools/q4nx_pack.py:61np.clip(np.rint(...), 0, 15)And the sites that carry codes around without touching their values, so they cannot be the wrap:
encode_tile()(pack_import.cpp:1982) writest.code[idx] & 0x0F— a mask, so a code that is already out of range would be silently truncated here rather than clamped. That is the symptom's mechanism (container == expected mod 4.0is exactly a mask) but not its source;gu_half()(:2344) and the q/k/v row split arememcpyof rows plus the same scale/zp slices — no arithmetic on codes.So the wrap enters
t.codesomewhere that is not one of the five quantisers above and not the copy paths — i.e. aTileWassembled by arithmetic on codes (+/-between a fused pair, or a renormalisation to a new zero point). The two candidates left are the fused gate/up or q/k/v combination paths and any test fixture that buildsTileWby hand; #1863's+4.0-on-negatives signature is consistent with the first (a difference of two unsigned codes taken as unsigned).Two things I would add regardless of which it is, both cheap:
encode_tile()should refuse a code outside0..15instead of masking it. A mask turns a producer's mistake into a plausible weight — that is the whole of this issue's user-visible harm — and the refusal costs one comparison per code in a path that already runs once per tile.bong-water-water-bong commentedon Nov 10, 2026
Correction to this issue's evidence, filed by me an hour ago: the mechanism is real but is not the general explanation, and the title overstates it.
The
container == expected mod 4.0relation I cited holds for one row of one group (32 of 32 values). Classifying every value of the tensor instead:So a wrap does happen — 64 values exactly a full code-range off, which is too clean to be noise — but it explains about 0.2 % of the mismatches, and the argument I built, including the link to #1874's "one row in 64 matched" signature, was founded on a sample I generalised from without checking.
What stands: a value can come back a code-range away instead of at the end of the range, which is worth knowing about the encoder. What does not: that it is why #1863's arm fails, or that it explains #1874. Both need their own evidence, and this issue should not be cited for them until someone re-measures.
Keeping it open for the narrower question — does the external-tile encode saturate, and if not, is that by design? — because that survives the correction. I would rather narrow my own issue than leave a mechanism in the fleet's record that a 32-value sample was mistaken for a proof.
bong-water-water-bong commentedon Nov 10, 2026
Retitling in effect: this is a code-convention mismatch, not a saturation gap — the evidence I filed this on has been superseded twice by better evidence.
Whole-row classification (256 columns of
q_projrow 0,#1863): 128 columns differ, every delta exactly ±4.0 (64 each), and every container value is the expectation modulo the 4-bit code range with two representative classes —[0,4)for 64 columns,[−4,0)for 64.Two classes in one row means the two sides read the same 4-bit code differently:
w_of) uses offset-binary with a zero point:(sx(code) − sx(zp)) × scale;and the ±4.0 delta is the sign flip between those conventions.
So, honestly: my earlier mechanism ("the encode wraps out-of-range values instead of saturating") is not established, and neither is the
mod 4.0generalisation I first posted. What is established is that a 4-bit code has (at least) two defensible readings and that some path in this tree uses one where the other is expected.The question I still think is worth a ticket — and it survives both corrections — is narrower than the title: is the external-tile code convention (offset-binary vs two's-complement) stated anywhere a reader can find it, and does the encoder and the decoder agree on it? If they do agree and only my fixture was wrong, this issue should be closed as "not a defect"; if they disagree for some chunk kinds, that is a real class of wrong weights.
I would close this in favour of that narrower question rather than leave a title that asserts a mechanism the evidence no longer supports — including retracting the #1874 link I made, which was inference from the same bad sample.
bong-water-water-bong commentedon Nov 10, 2026
The writer-side half is answered by PR #1921 (
31587f2), and I verified it on real data: built that head and imported the reference 1.9 7B pack, which re-encodes millions of codes through the writer —— no refusal, round trip still byte-exact. So the refusal costs nothing on a real model, and a wrapped weight can no longer be produced by the packer.
What I could not find is a producer of an out-of-range code in this tree, which narrows this issue considerably:
pack_import.cpp:2227(synthetic tied head),:4041(external→container)q4nx_quant_int4_code(q4nx_pack.c),q4nx_pack_blocked.cpp:78,tools/q4nx_pack.py:61gu_half(), the q/k/v row splitmemcpyof rows — no arithmetic on codesencode_tile()& 0x0Fmask (now a refusal)So today the wrap is reachable from a crafted
TileW— #1863's fixture — and from nothing else I can find. Two possible readings, and the owner of the encoder should pick one:I am not writing the clamp: with every producer clamping, a saturating encode would be dead code that hides the next producer's mistake, which is the opposite of what #1921 just bought.
bong-water-water-bong commentedon Nov 10, 2026
Superseded by #1932, and my original mechanism was half right after all — the offset really is 16 code steps, but it is not the encode failing to saturate: it is a pack's two's-complement int4 code being written into the container's unsigned field, so every code with the high bit set comes back . Measured: → container , pack ; delta histogram over 256 columns.
Closing this one as the narrower duplicate that #1932 answers with the mechanism and the fix.
bong-water-water-bong commentedon Nov 10, 2026
Superseded by #1932 — and my original mechanism was half right after all.
The offset really is 16 code steps, but it is not the encode failing to saturate. It is a pack's two's-complement int4 code being written into the container's unsigned field, so every code with the high bit set comes back
+16 x scale:4.0 / 0.25 = 16 codes. #1932 has the full chain and the one-line-per-field fix.Closing this as the narrower duplicate that #1932 answers with a mechanism and a remedy. Note the previous comment on this issue was mangled by my shell (backticks inside a double-quoted argument); this one is the corrected version.
npu-pack: the tile writer's range check is defensive, and the comment…
npu-pack: the tile writer refuses a 4-bit code outside 0..15 instead …