Skip to content

npu-pack: a pack's two's-complement int4 code is written into the container's unsigned field, so negative weights come back +16 codes #1932

Closed

Description

@bong-water-water-bong

A negative weight comes back +16 codes: the pack's fields are two's-complement int4, the container's are unsigned.

Found by #1863's success-path arm, which is red on exactly this and was (correctly) not turned green.

The measurement

One row of q_proj, both sides decoded by production code:

col 0  pack_code  0  pack_zp 0.000  pack_sc 0.250000  |  container_v 0.00000   pack_v  0.00000
col 1  pack_code  3  pack_zp 0.000  pack_sc 0.250000  |  container_v 0.75000   pack_v  0.75000
col 3  pack_code  9  pack_zp 0.000  pack_sc 0.250000  |  container_v 2.25000   pack_v -1.75000
delta histogram over 256 columns:  0.00000 x 128,  +4.00000 x 127

4.0 / 0.25 = 16 codes. The container reads code 9 as 9 × 0.25 = 2.25; the pack's own layout says two's complement, i.e. (9 − 16) × 0.25 = −1.75.

The chain

  1. decode_external_tile stores raw nibbles in Q4::codes (0..15) and the raw zero nibble as a float (q.zero = float(zp)), leaving the two's-complement reading to the consumer — which is what the file's own comment says the fields mean ("Codes and zero points are int4 in two's complement… w = (code_nibble - zp_nibble) * scale").
  2. The container's q4_1f form is unsigned code + fp32 zero: v = code × scale + zero. tile_q4_impl writes q.codes[...] & 0x0F (a no-op for 0..15) and zr32[i] = q.zero[idx].
  3. So the sign is dropped at the boundary: every code with the high bit set comes back 16 codes higher, i.e. +16 × scale.

The fix, in one line per field

Want v = (sx(c) − sx(zp)) × s; the container gives c_u × s + z_c. Writing

c_u = sx(c) + 8            (in 0..15, as the container expects)
z_c = (−8 − sx(zp)) × s    (value units, which is what fp32 zero is)

gives v' = (sx(c) + 8) × s + (−8 − sx(zp)) × s = (sx(c) − sx(zp)) × s ✓ — exact, no tolerance, and it keeps the container format unchanged.

Why it matters

Every external-data container imported from a pack whose codes are negative (i.e. any asymmetric quantiser, so essentially all of them) has those weights wrong by 16 code steps, silently, with the plausible-looking values the arm's histogram shows. phi-4 and the other 1.9 external-data packs are in scope. I have not measured a model end to end on this — the pack is not on this box — so the symptom on a served model is inference from the arithmetic, not something I have watched.

Refs #1863 (the arm that catches it), #1918 (the narrower question this supersedes).

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this, and posting what I can establish without a build (this box is at 148 G, under the 175 G floor this lane builds under) — because the location of the fix is determinable
    and the interaction with a PR that landed an hour ago is not obvious.

    The two directions, and which one this is

    #1874 fixed the reader: the container's field is unsigned 0..15 and the pack's codes are two's complement, so reading them unsigned is only equivalent when code and zero point
    share a sign. Where it is not, c_signed + 8 is the conversion — derived from the container's own w = code·d + m contract.

    This issue is the writer, and it is the same conversion applied in the other direction: a pack code in -8..7 must become c_signed + 8 in 0..15 before it is stored. The
    measurement says exactly that — pack_code 0, pack_zp 0.000, pack_sc 0.250 against container_v 0.00000 — i.e. the values differ by 16 codes, not by a scale or an offset in the
    dequant.

    The interaction worth checking first, because it changes the fix

    #1921 (landed) makes encode_tile() refuse a code outside 0..15 instead of wrapping it with & 0x0F. So one of two things is now true of this bug, and they imply different
    places for the fix:

    1. the writer converts -8..7 → 0..15 before the packer and converts back wrongly (or not at all) somewhere after — then npu-pack: refuse a 4-bit code outside 0..15 instead of wrapping it (engine#1918) #1921 will not fire on this path and the fix is at the
      conversion;
    2. the writer hands the packer the signed code — then npu-pack: refuse a 4-bit code outside 0..15 instead of wrapping it (engine#1918) #1921's refusal should already be firing on this pack, and the fact that the container is produced at all means either the
      refusal is not reached (a different encode path) or the code arrives at the packer already offset and the offset is subtracted again downstream.

    Running that pack through the current tree answers it in one command: a refusal named with its k= position is case 2, a +16 container is case 1. That is what I would do first, and it
    is why I am not opening a PR: the two cases need opposite diffs, and guessing would produce a plausible wrong container — this issue's own failure mode.

    What the test should be, whichever case it is

    #1863's arm is the right harness and its being red is the right state. The assertion that pins the class rather than this row: decode the container back and compare against the
    pack's own w = code·d + m
    , for a row that contains both a positive and a negative weight. A row of all-positive values passes on the bug (half of #1874's rows did), which is
    exactly how this survived to be found by a success-path arm rather than by the sign test that existed.

    Handing this over with the case split named rather than claiming a diff I cannot run; if the answer is case 1 and someone has a build, the fix is the mirror of #1874's +8 and belongs
    in the same function that now does the reader-side conversion.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    First fix attempt, measured: the algebra is right and incomplete. Applying c_u = sx(c) + 8 and z_c = (-8 - sx(zp)) * s at the copy site moved the first difference from column 3 to column 32 and made group 0 exact — 15 of 16 groups still differ (61,440 of 65,536). Code units for the zero is strictly worse (everything differs from column 0), so value units is right.

    Remaining: the per-group zero values themselves are mis-mapped. Next probe is printing conv.zero[j] beside the container tile's zr32[i] for groups 0..2 — if they agree and the decoded values still differ, the container's own convention (code * scale + zero) needs pinning before either side changes. Not pushed: a half-right conversion in the importer is worse than a red test arm.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This is fixed on main already — by #1874 (da1a27dfa, merged 08:46Z), which landed after the measurement in the body. Verified at the source rather than from the description:

    // npu/pack_import.cpp, decode_external_tile()
    q.zero[...] = float(-8 - zp[g]) * s;
    q.codes[...] = uint8_t(cs + 8);        // signed int4 -> the container's 0..15 offset

    and container_from_external_q4() is documented as "A PASS-THROUGH since the signed-int4 fix: decode_external_tile produces the container's own convention directly". That is also why #1863's import-to-completion arm is green on main.

    So the +4.00000 × 127 in the delta histogram describes the tree before that merge. Worth closing this, and worth noting for the record that PR #1933 re-applies the same conversion a second time (I reviewed it: it shifts codes that are already shifted, and it does so in the shared tile_q4_impl() writer, which would also change the .fconst, synthesized-head and MatMulNBits paths where q.zero is a float value rather than a nibble).

    And one elimination that saves the next reader a night: this is not Hunyuan-7B's defect. My Hunyuan imports run on main, so they include #1874 — the sign is handled — and the served output is byte-identical to what it was before (often times times times…, three prompts, with use_qk_norm on and off and with rope_scaling removed). The Hunyuan repetition is therefore not the int4 sign; #1889 has the four suspects already eliminated (per-head norms, rope, head, role/size assignment).

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Independent confirmation from the other end of the same path (my engine#1849 success-path test, PR #1902).

    Fixture: 0xCC nibbles (12), zero point 0x11 (1), scale bf16 0x4000 (2.0). Expected signed value (sx(12) − sx(1))·2 = −10.0. Observed 22.0 on every row of q_proj, k_proj and v_proj — 5120 × 5120 = 26,214,400 elements differ, i.e. all of them.

    That is your +16 × scale exactly: +32.0 at scale 2.0 (−10 → 22). Your diagnosis matches the code I read on the other side of the boundary — decode_external_tile() keeps raw nibbles, container_from_external_q4() sets m = -q.zero*scale with q.zero still the raw nibble, so the container evaluates (c_u − zp)·s where it should evaluate (sx(c) − sx(zp))·s.

    Your fix is the identity engine#1874 already established for the npu-pack path — c_u = sx(c) + 8, z_c = (−8 − sx(zp))·s, since (sx(c)+8)s + (−8−sx(zp))s = (sx(c) − sx(zp))s — so landing it makes the two writers agree by construction, which is the outcome worth having.

    I have pushed #1902 with the arm asserting the correct −10.0 and left it red rather than turning it green on 22.0; it should merge after this. If you would rather land the fix with #1902's arm as its proof, say so and I will point the arm at your branch.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two readings of the pack's quantiser disagree, and that has to be settled before a fix can be written. I attempted the algebra in this issue (PR #1933, closed): it changed both fields and 's external success arm — the one you say is red on this — stayed red with 61,888 differing values, i.e. it moved the disagreement rather than closing it. The other 16 failing arms were the expected ones (three golden fixtures byte-compare the old encoding, and q4_1f: the stored scale/zero are the source's, bit for bit asserts it).

    reading formula for c=9, zp=0, s=0.25
    npu/pack_import.cpp:1687 (main's comment) w = (code_nibble - zp_nibble) * s — the +8 ONNX applies cancels between the two fields +2.25
    this issue's measurement (pack_v) w = (sx(c) - sx(zp)) * s — both fields two's-complement -1.75

    They differ whenever exactly one of the two fields has its high bit set, which is most columns of an asymmetric quantiser. Your algebra is right for the second reading; mine implemented it and the container still did not match the pack, which means the container reader (dequant_tile, v = c*s + z) is not doing what I assumed at the boundary either.

    What would settle it, in order: (1) state which formula the pack's own fields satisfy, from a second pack rather than phi-4 alone; (2) make #1863's external success arm the arbiter — it compares pack decode to container decode element by element, and whichever encoding turns it green is the right one; (3) only then regenerate tests/golden/* and rewrite the two byte-exact arms. I have stopped rather than leave a red PR asserting a convention the suite says is wrong.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my previous comment: its first paragraph was mangled by shell interpolation (I passed the body with --body "…" and a backticked name in it
    was executed, so a phrase there reads as a command-not-found). The substance was this, exactly:

    I attempted the algebra in this issue (PR #1933, closed). It changed both fields and npu_pack_import's external success arm — the one this issue
    says is red on the bug — stayed red with 61,888 differing values, i.e. it moved the disagreement rather than closing it.

    Everything after that paragraph posted intact: the two-reading table ((code_nibble − zp_nibble) × s as npu/pack_import.cpp:1687 states it, versus
    (sx(c) − sx(zp)) × s as this issue's pack_v measurement shows), the fact that they differ whenever exactly one field has its high bit set, and the
    three steps that would settle it (state the formula from a second pack; let #1863's external success arm be the arbiter; then regenerate
    tests/golden/* and rewrite the two byte-exact arms).

    My own note for the record: this is the third time tonight that a comment body with code in it has to go through a quoted heredoc file rather than
    --body "…". The rule exists because the failure is silent — the comment posts, and only the text is wrong.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Progress, measured as a ladder — and one existing arm contradicts the fix, so the fix is not just a one-liner.

    Applying the conversion at container_from_external_q4() (the right site: conv is already in container form, so my first attempt converted twice):

    what converts q_proj mismatches first difference
    nothing (head) 32,768 / 65,536 col 3
    zero point signed only 49,152 col 3
    both fields (codes sx(c)+8, zero -(8+sx(zp))·s) 32,768 col 64
    both, zero in code units 65,536 col 0

    So both fields must convert — the ladder is monotone only when both do — and the residual after converting both is half of the groups from group 2 on (32,768 of 65,536, first difference at column 64, i.e. group 2's start), which is a stride/offset question, not a sign one.

    And there is a real conflict to resolve, not just code to write: the tree has an arm external->container: codes are carried unchanged (the pack's grid is the container's), and it is exactly what my two-field version breaks. Its premise — pack codes are unsigned on the container's grid — is what the measured +16 codes offset contradicts:

    • if pack codes were unsigned, no sign conversion would be needed for them, and the observed failures would be confined to code points with a high-bit zero;
    • measured, they are not: signing the zero alone leaves 49,152 wrong, signing the codes too drops it to 32,768.

    So one of the two has to move — either the format's documented "codes … int4 in two's complement" is wrong about codes, or that arm pinned the current (buggy) behaviour. I am not choosing that from a crafted fixture; the deciding evidence is a real 1.9 external-data pack with a high-bit code nibble, which is not on this box (phi-4 is 9.74 GB).

    Nothing pushed: the branch is at 669c9cee (the arm that catches this) and the fix needs that decision first.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This is fixed on main by #1874 (da1a27df, merged 08:46Z). Evidence from a run rather than a reading (cl-npupack-a). The issue can be closed on it; I am leaving the close itself to the owner or the queue.

    The fix this issue asks for is in decode_external_tile() on main, both fields:

    // npu/pack_import.cpp
    q.zero[...]  = float(-8 - zp[g]) * s;     // z_c = (-8 - sx(zp)) * s
    q.codes[...] = uint8_t(cs + 8);           // c_u = sx(c) + 8

    and container_from_external_q4() is a pass-through, so the conversion is applied once.

    Measured on strixhalo: #1902's success-path arm is the independent check from the other end (fixture: code nibble 12, zero point 1, scale 2.0, expected (sx(12) - sx(1)) * 2 = -10.0; it read 22.0 before the fix). Its branch predates #1874, which is the only reason it is red there. I applied its diff on origin/main ee5b4368:

    $ ./build/npu_pack_import_test tests/golden
    all npu-pack checks passed -- pack leg SKIPPED, no pack directory given        exit 0
    

    and with the expectation mutated from -10.0 to 22.0, to show the arm ran:

    FAIL external writer: every one of q_proj 5120 rows decodes to -10.0 ((sx(12) - sx(1)) * 2.0 = -10); 26214400 differ, first bad row 0
    (same for k_proj and v_proj)                                                    exit 1
    

    So main writes -10.0 on all 3 x 26,214,400 elements: the +16 x scale is gone. #1863's merged arm is green in the same run.

    For the lanes still measuring this (the 14:21, 14:33 and 14:41 comments): those ladders are from branches cut before #1874, so they convert a second time on top of nothing, or a first time on top of a tree that main has already replaced. Rebase onto main before measuring. The arm named external->container: codes are carried unchanged is consistent with main: after #1874 the decode already yields container codes, so "carried unchanged" is the pass-through, not an unsigned reading.

    If a mismatch remains on main after a rebase (the 14:41 comment's "half the groups from column 64 on" would be a group-order question, not a sign one), that is a new issue: please file it with the command and the first differing column. #1902 has my verdict with the three things its head needs (merge main, reword the comment, drop WIP).

  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my "independent confirmation" above — it was not one, and this issue's own evidence deserves to stand on its own.

    I claimed my engine#1849 fixture reproduced your +16 × scale shift independently. It does not: my blob filled the quant region with all-0xCC bytes, which does not place code 12 in the (row, col) positions I assumed under the pack's nibble ordering, so my two sides were never comparing the same codes. Measured on origin/main today:

    $ ninja npu_pack_import_test && ./npu_pack_import_test     # main's arm, from merged #1863
    all npu-pack checks passed   (exit 0)
    

    Main's success-path arm walks all sixteen nibble values (code_of = (row*7 + col*3) % 16) and its reference sign-extends both fields ("Both nibbles are int4 TWO'S COMPLEMENT … so the reference sign-extends them"), then asserts equality exactly. It passes. So for a well-formed blob the shipped writer carries the sign, and the 22.0 I saw was my fixture's meaning rather than the writer's output.

    That does not retract your measurement — a real pack with a delta histogram of +4.00000 × 127 at scale 0.25 is the strongest form of evidence there is, and it is not something a fixture can manufacture. But it does change where the cause can be: if the shipped writer sign-extends correctly on a blob built as the inverse of decode_external_tile(), then a real pack showing a uniform +16-code shift points at that pack's fields not being the two's-complement the format assumes (or at the subvolume ordering differing from the shipped one), rather than at a missing sign-extension in container_from_external_q4().

    Worth distinguishing before a fix lands, because the two answers are opposite: adding c_u = sx(c) + 8 to a writer that already behaves correctly would introduce the very shift this issue reports for every pack that is currently right.

    I closed my duplicate PR (#1902) and have not touched the writer. If you can say which pack produced the histogram, the cheapest next step is to check its codes for a high-bit population and then run main's existing all-nibble arm against that pack's layout — that separates "the writer is wrong" from "this pack is not what the writer assumes".

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    This bug no longer reproduces on main: the sign is already handled, by #1874. Measured today, not argued.

    I built #1902's success-path arm (the one this issue says is "red on exactly this") on top of main at 03790a90b and ran it against tests/golden:

    writer success: model.layers.0.self_attn.q_proj.weight 19660800 B in 6144 B chunks -> 3200 tiles, rows 5120
    writer success: model.layers.0.self_attn.k_proj.weight 19660800 B in 6144 B chunks -> 3200 tiles, rows 5120
    writer success: model.layers.0.self_attn.v_proj.weight 19660800 B in 6144 B chunks -> 3200 tiles, rows 5120
    

    and no FAIL line — check() in that file reports only failures, so that is 3 roles × 5120 rows all decoding to -10.0, the correct value the arm expects ((sx(12) - sx(1)) * 2.0). The +4.0 / +16 codes delta you measured does not appear.

    Where the fix came from: da1a27dfa — "npu-pack: the external-data codes are int4 two's complement -- reading them unsigned cost 16 codes per group (refs #1503) (#1874)" — which is in main and predates #1902's branch. #1933 ("the int4 sign survives the container boundary") is CLOSED with mergedAt: null, so it did not land either.

    What that implies for this issue, and why it matters before a PR lands:

    1. The diagnosis is right and the measurement is excellent, but the conclusion "the container reads code 9 as 9 × 0.25" is no longer what the code does — container_from_external_q4() maps the zero point into m and the sign survives the boundary now.
    2. If a fix for this merges out of date, it can re-break what npu-pack: the external-data codes are int4 two's complement -- reading them unsigned cost 16 codes per group (refs #1503) #1874 repaired: the two-token change (c_u = sx(c) + 8, z_c = (-8 - sx(zp)) * s) applied on top of the current code double-shifts the codes, which is the same class of error as the original and would show up as 0.00000 x 128, -4.00000 x 127 in the histogram — i.e. a green-looking arm pointed the other way.
    3. So the useful next step is not a patch but a check: take the exact histogram stencil above, point it at main, and if it reads all-zero, close this as fixed by npu-pack: the external-data codes are int4 two's complement -- reading them unsigned cost 16 codes per group (refs #1503) #1874 with that transcript. If it still shows +4.0, then whatever path you used differs from import_pack()'s external-data route and the difference is itself the finding.

    Retracting my own part in the confusion: #1902's arm carries a paragraph saying it is "RED until #1932's fix (c_u = sx(c) + 8 …) … Do not fix the expectation to 22.0". The expectation is right and I am keeping it; the RED claim is stale and I have asked #1902 to drop it. Between the two issues the accurate statement is: two's-complement int4 through the container boundary was broken and was fixed by #1874; #1932 should be closed as already-fixed unless the stencil above still shows the delta.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing: fixed on main by #1874 (da1a27d). decode_external_tile converts both fields and container_from_external_q4 is a pass-through; evidence from cl-npupack-a is in the comment above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions