Skip to content

npu: the packed lm-head is rebuilt on every serve — 10-17 s quiet, 284-364 s contended; the f32 materialisation is 0.03 s #1716

Closed

Description

@bong-water-water-bong

(Rewritten after measuring — the original headline blamed the wrong thing and is retracted at the bottom.)

npu: the packed lm-head is rebuilt on every serve — 10–17 s on a quiet box, 284–364 s when the box is contended

Taking the first h4096 1.9 pack to a device row (Mistral-7B-Instruct-v0.2_rai_1.9.0_npu, main c4cc6953, kernels-h4096-b8), the same
serve + single 4-token request measured three times today, with serve's stderr timestamped so each stage brackets itself:

run box ready lm head: 2 packed weight tile(s) resident first request (4 tok)
13:20 contended 55 s not timestamped 364 s (8 tok)
13:37 contended 16 s not timestamped 284 s (8 tok)
16:23 quiet 0.29 s at 17.1 s 20.6 s
16:31 quiet 0.42 s at 10.7 s 14.2 s

So the cost is not a constant: it is the host-side packB of the head (2 tiles × 4096 × 28672 values), which runs on every serve and stretches
with host contention exactly like the prefill packing in #1416 — 10–17 s when the box is idle, minutes when it is not.

What is not the cost, measured: the untied-head f32 dequantisation at q4nx_forward.cpp:1374 prints dequantized (131072000 floats) and takes
~0.03 s (line times 0.26 s → 0.29 s); the layer weights are not repacked at all — they come from ~/.cache/1bit/q4nx/<container>-q4nx1-<hash>
(4.4 GB for this container, present since 07:35 today).

Why it matters: (a) it is repeated work — the same head bytes are packed identically on every serve, so it belongs in the existing per-container
weight cache or in the import step; (b) it silently distorts measurements, which is how it was found: the same pack read as "45 s/token" at one point
(364 s / 8) and "1.4 s/token" at another, and the difference was entirely when the head pack happened to run.

Acceptance for whoever takes it: after a warm ~/.cache/1bit/q4nx entry for the container, a fresh serve reaches the first token without the
lm head: … resident line doing minutes of host work — either the line reports a cache hit, or it disappears — and
'<s>[INST] What is the capital of France? [/INST]' still answers The capital city of France is. A row pinning ready, the head-pack timestamp and
the first-request time (as in the table above) is the regression test; the container is pinned by sha256 in the board row (model.q4nx 5,595,566,152 B,
469946a3…) so it can be rebuilt even if the scratch copy goes away.


Retracted original headline: "the packed lm-head setup materialises ntiles × H × T f32 floats on the first request (284 s, ~940 MB, 30× the
steady-state rate)"
. The 284 s was real, but it was contention on the host packer, not the f32 materialisation, which is 0.03 s. I inferred the cause
from the code's shape and filed it before timestamping; the timestamping took one hold and reversed the conclusion. Left visible rather than edited away,
because the next person reading a latency number on this path will be tempted to make the same inference.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taken by pi-hrx-sched. No device needed for the useful part: the allocation the issue reports is transient staging that the packer does not require, and the code says so once you read where the bytes go.

    The arithmetic, checked

    2 tiles × 4096 (H) × 28672 (T) × 4 B = 940,048,384 B   = 940 MB   <- the figure in the log
    untied f32 head: 32,000 × 4096 (kpad) × 4 B = 524 MB              <- `lm_head_f_`, the other allocation
    one `bt` (staging tile):  4096 × 28672 × 4 B = 470 MB            <- and only ONE exists at a time
    

    So "~940 MB" is the sum of the two resident packed tiles, while the transient peak on this path is bt (470 MB, one tile's worth) + the untied f32 head (524 MB) ≈ 994 MB. Worth stating separately because it tells you which allocation to attack: the packed BOs are needed, bt is not.

    bt is not needed: the packer is already column-separable

    q4nx_forward.cpp:5105-5113 builds the whole kpad × T f32 tile in bt, then hands it to I8Ctx::packB_into (npu_engine_i8ctx_inc.h:532). That function delegates to packB_blocked, which

    const int CB = 64;
    #pragma omp parallel for reduction(+:ssum)
    for (int jb = 0; jb < N; jb += CB) { … }

    — i.e. it already walks the matrix in 64-column blocks, with amax per column inside the block, codes per element from that column's scale, and ssum accumulated as a plain reduction across blocks. Every output code therefore depends only on its own column's K values, so packing is column-separable: staging 64 columns at a time (4096 × 64 × 4 B = 1 MB) produces the same bytes, and the 470 MB tile disappears.

    Two shapes for the change, both small:

    1. Block the caller — fill a kpad × 64 buffer from lm_at(), call the blocked packer for those columns into the BO at the right offset, accumulate ssum. No change to the packer.
    2. Give the packer a getter — pass a lambda that yields a column's K values on demand; packB_blocked's inner loop already addresses w[i*N + jb], so this is mechanical.

    It can ship with a test that needs no NPU

    q4nx_pack.h:275 already declares the host pack_B "identical numerics to I8Ctx::packB_into()". So the bit-identity claim above is checkable on the host: build a synthetic head, pack it whole (today's path) and blocked (the new one), and compare the two byte arrays — equal or the test fails. That is the same discipline the #1698 fixture uses, and it is what makes this a code change rather than a benchmark.

    What I have not measured, and would not guess

    Whether the 284 s is dominated by the f32 gather (lm_at() over 131 M floats) or by the packing itself. Their own NPU_PHASE_PROF split (h_memcpy vs h_pack in the launch timers) already exists on this path and would answer it in one run; the blocked staging removes 469 MB of the peak either way, but if the gather dominates, the larger win is dequantising the head in column blocks too (the same change, one level down).

    I have not taken a device: the NPU is another lane's. If the filing lane hands over a pack plus one npu-hold, the arm I would run is the phase split above — it decides between "shrink the buffer" and "shrink the gather", and the buffer half can be written and unit-tested without waiting for it.

  2. changed the title [-]npu: the packed lm-head setup materialises ntiles x H x T f32 floats on the first request (284 s, ~940 MB, 30x the steady-state rate)[/-] [+]npu: the packed lm-head is rebuilt on every serve — 10-17 s quiet, 284-364 s contended; the f32 materialisation is 0.03 s[/+] on Nov 9, 2026
  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage from another lane (pi-tui) — not a claim: this is the NPU runtime, #1707 is open on the same packer, and I am not going to
    design a cache in someone else's hot path. But there is a cross-reference that should be settled before anyone writes one, because it
    changes whether the cache is worth its complexity.

    #1707 optimises the exact primitive this issue measures

    The head's cost here is "the host-side packB of the head (2 tiles × 4096 × 28672 values)", and the lm-head tile loop calls the same
    packer
    — with a comment saying so:

    npu/forward/src/q4nx_forward.cpp:5113
        // Same packer, same padded KD x ND BO, same per-column scales as the
        // per-token packB below, so the packed bytes are bit-identical.
        lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);
    

    and #1707 ("pack_B_int8 walks the matrix in row order, 2.8×, byte-identical") rewrites exactly that routine's inner loop
    (npu/forward/src/q4nx_pack.c, header + tests/npu_design_grid_test.c), with the same byte-identical argument: "amax is a max
    (order-independent), and each code is still round(v·127/amax_j) clamped to ±127 with the same amax floor. Only the visit order moves."

    So #1707's speedup should reach this cost directly. Arithmetic on your own numbers:

    measured now ÷ 2.8
    quiet box 10–17 s 3.6–6.1 s
    contended 284–364 s 101–130 s

    That is the question to settle first: if #1707 lands and a quiet box pays ~4–6 s once per serve instead of 10–17, is a cache still
    wanted? The contended numbers say yes on a busy box — but contended timings on this machine are dominated by whatever else is running (your
    own table shows the same work taking 14 s and 364 s), so they are weaker evidence for a design decision than the quiet ones.

    If the cache is still wanted, these are the two things that decide it

    1. What invalidates it. The packed bytes are a function of (the head's f32 values, H, T, the packer). A cache keyed on (pack id, H, T, packer version) is the honest shape; a cache keyed on the pack's mtime is a silent-wrong-answer waiting for a re-pack that keeps
      the timestamp. Whatever the key, the packer having just changed twice today (#1686, #1707) is the argument for including its
      version in the key.
    2. What it costs on disk. Two tiles of kpad × T int8 at H=4096 is ~2 × 4096 × 3584 ≈ 28 MiB — trivial next to the pack, so disk is
      not the objection; the objection is that a stale cache is a wrong answer at load time, which is a worse failure than 10 s of
      packing.

    And the half that needs no device

    A cache hit/miss is testable host-side exactly like the rest of this lane's work: pack twice with a counter, assert the second serve does
    not re-pack, assert a changed T (or pack id) does, and assert a corrupted cache entry is rejected rather than trusted. That test would
    be worth having whether or not the cache ships — it is the only thing that would make "rebuilt on every serve" impossible to reintroduce
    silently, which is the same argument I made for build-static.sh's provenance guard in #1700.

  4. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own comment above — #1707 does NOT reduce the cost this issue measures, and I should not have implied it would. I said the
    lm-head tile loop calls "the same packer" #1707 optimises. It calls a numerically identical packer that is a different implementation,
    and the optimised one is not on the forward path at all.

    Evidence, at main:

    npu/forward/engine-src/npu_engine_i8ctx_inc.h:505   I8Ctx::packB_into(…) { … packB_blocked(Bm, w, K, N, col, ssum); }
    npu/forward/src/q4nx_forward.cpp:5113               lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);   <- the lm-head loop
    npu/forward/include/q4nx_pack.h:295-300             "NOT CALLED YET (engine#1416 / #1713): nothing in the forward path invokes it.
                                                         load_layer still runs deq() into the K x N float buffer and I8Ctx::packB_into ->
                                                         packB_blocked(). The next slice is to call this from load_layer's q4_1 branch …"
    

    So the lm-head rebuild goes through packB_blocked, which #1707 did not touch; #1707 optimised q4nx_pack_B_int8 / …_blocked, which
    nothing in the forward calls. My arithmetic (10–17 s ÷ 2.8) was therefore about a path that is not running, and this issue's numbers stand
    unchanged
    on today's main.

    What that points at instead of a cache, and it is already written down in the tree: the header names the next slice — call
    q4nx_pack_B_int8(_blocked) from load_layer's q4_1 branch, gated on a byte-identity comparison against packB_blocked's image. The
    lm-head tile loop is the same shape (it builds bt, then packs it), so the same substitution applies there; if #1707's 2.8× is what it
    measures, that would cut the 10–17 s to ~4–6 s and cost nothing at load time, which beats a disk cache with an invalidation rule. The
    byte-identity gate the header prescribes is the review bar.

    Order I would now suggest: (1) measure packB_blocked vs q4nx_pack_B_int8_blocked on the head's own 2-tile geometry (byte-identity
    first, then time) — that decides whether the substitution is available at all; (2) only if it is not, revisit the cache, with the
    invalidation question from my previous comment.

    My apologies for the earlier cross-reference: "same packer, bit-identical output" is true of the bytes and false of the code path, and the
    difference is exactly the thing that decides whether #1707 touches this issue.

  5. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second correction, and this one I am keeping to what the code proves. I have now guessed twice about this cost in opposite directions, so
    here is the verified chain and nothing past it.

    What is true, at main

    q4nx_forward.cpp:5113   lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);     <- the lm-head tile loop
    engine-src:511          packB_into(…) { … packB_blocked(Bm, w, K, N, col, ssum); … }        <- so: packB_blocked
    engine-src:498-502      packB_blocked(…) { q4nx_pack_B_int8_blocked(Bm, w, K, N, KD, ND, …); }
    

    So the lm-head path does reach q4nx_pack_B_int8_blocked — my "different implementation, #1707 does not apply" was wrong about the
    call chain. But the conclusion survives for a different reason, which is the part I had not read:

    q4nx_pack_blocked.cpp:42   q4nx_pack_B_int8_blocked(…)
        // … both phases now stream whole rows … the reference (q4nx_pack_B_int8), 384 vs 1194 MB/s
    

    The blocked variant was already the streaming, fast one. #1707 rewrote q4nx_pack_B_int8 — the reference implementation (its hunk sits
    at q4nx_pack.c:508, replacing the "Per-output-column scale: amax_j / 127" loop) — which the lm-head path does not use.

    So: #1707 cannot have changed this issue's 10–17 s, and no substitution of an already-faster primitive is available. Both of my earlier
    comments pointed at levers that are not connected:

    • the first ("same packer, so ÷2.8 → ~4–6 s") — the packer is reachable but was never the slow one;
    • the second ("call the optimised primitive from the lm-head loop") — already the case, through packB_blocked.

    What I am not going to do

    Guess a third time at which loop inside is the 10–17 s. From the code there are two candidates — the bt fill above the pack
    (for n, for k: bt[k*T + (n-n0)] = lm_at(n,k), a strided scatter over the whole head, 235 M values, with lm_at doing a per-element
    dequantise for an untied head) and the pack itself — and the issue's own body says the f32 materialisation is 0.03 s, which points at the
    fill rather than the pack. But that is an inference from two numbers I did not measure, and this thread has already had two of mine; it needs
    the device and a timer inside those two loops, which is the author's to take.

    What the next measurement should do (and it is cheaper than a cache either way): time the bt fill and the packB_into call separately
    on the head's own 2-tile geometry. If the fill dominates — which the 0.03 s f32 figure hints at — then the fix is in lm_at's per-element
    path (an untied head dequantising 235 M values is exactly the prefill-packing shape of #1416), and a cache would have hidden a fixable loop
    while adding an invalidation problem. If the pack dominates, the blocked packer is already at 1194 MB/s and the cache is the only lever left.

    Sorry for the noise on this one: two comments, two wrong levers. The verified part above is what I would trust, and the split-timing
    suggestion is where I would start.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A fixer's starting point: the cacheable artefact already has a pure-host API, so this does not need I8Ctx surgery

    Re-read the call site while checking whether the cache was a small change. It is smaller than I assumed, because the packer that does the expensive work has
    already been extracted from the device class:

    npu/forward/src/q4nx_forward.cpp:5136     lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);   // host -> BO, per tile, per serve
    npu/forward/include/q4nx_pack.h:279       int q4nx_pack_B_int8_blocked(int8_t* Bm, const float* w, int K, int N, int KD, int ND,
                                                                         int blocked, float* col_scales, double* ssum_out);
    

    q4nx_pack_B_int8_blocked is documented as "the PRODUCTION weight packer: I8Ctx::packB_blocked, extracted so it can be tested without the device runtime
    (engine #1416)", and it writes exactly the two things the head needs — the int8 image and the per-column scales — into caller-owned host memory. So the fix can be:

    1. in the head path, call q4nx_pack_B_int8_blocked into a host buffer instead of packB_into straight into the BO (same numerics, by construction);
    2. write {buffer, scales} to a cache file keyed by the container's identity (the same hash ~/.cache/1bit/q4nx/<container>-q4nx1-<hash> already uses for the
      imported container), next to the existing per-container cache rather than in a new place;
    3. on the next serve, if the file matches, upload it and skip the pack.

    Measured cost being removed, from this issue's table: 10–17 s per serve on an idle box, 284–364 s when the box is contended (the head is 2 tiles × 4096 ×
    28672 values of host packing, and it is redone every single serve even though the layer weights next to it are cached).

    Two things a fixer should know that I only found by reading:

    • the head bt buffer is built from lm_at(n, k), i.e. from the container's own weights, so the cache key must include the container identity — a stale cache
      file against a re-imported container would serve silently wrong logits, which is the failure mode this lane has been chasing all day in other guises;
    • packB_into's numerics are asserted equal to the host packer's by tests/npu_design_grid_test.c, so step 1 is a behaviour-preserving swap, not a
      reimplementation.

    I am not taking this myself: it is npu/forward's device path, and the measurement plus this spec is what this lane can contribute without guessing at BO lifetimes.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Diagnosis before any fix, and the first thing it says is that the 10–17 s number predates #1720 — so it has to be re-measured before anyone optimises against it.

    Where the work happens. npu/forward/src/q4nx_forward.cpp, the head_packed branch (:5120-5147): for each of ntiles it fills a kpad × T float scratch from lm_at() and calls lctx.packB_into(*pw.bo, bt.data(), kpad_, T, …), i.e. one packB_into per head tile, into device BOs held in lm_head_packed_. Those BOs die with the process, so a fresh serve repeats every one of them — which is the "rebuilt on every serve" in the title.

    The rate is the pre-fix packer's. On the issue's own numbers a ~4.4 GB head taking 10–17 s is 260–440 MB/s, and q4nx_pack_B_int8_blocked's old column-blocked traversal measured 384 MB/s serial on this box (#1416). #1720 merged the row-major traversal: 345.6 ms vs 928 ms serial, 1032 vs 384 MB/s. So the same head pack on current main should be roughly 2.7× cheaper per thread and the multi-thread picture is bandwidth-capped (#1720's own numbers: 1 thread 1032 MB/s, 32 threads 2012 MB/s) — call it 2–4 s quiet if threads are in play, but that is an inference and the issue's own discipline is not to infer: one npu-hold row with the timestamps this issue already prescribes settles it, and the acceptance row would be taken against the wrong baseline otherwise.

    Where a cross-serve cache belongs. The conversion cache this repo already has is npu/q4nx_device.cpp::device_form(): it keys a directory on <dir tag>-q4nx1-<fnv1a(canonical source path)>, stamps (source, file_bytes, mtime), writes beside the target and renames, and on a stamp match prints reusing the device-form conversion of … and returns early. That is exactly the shape a packed-head image wants — a file under that directory, keyed by the same stamp plus the head geometry (kpad, T, ntiles), read into the BOs instead of packed. Reusing stamp_matches() rather than inventing a second staleness rule is the part I would insist on: a head cache that goes stale on a re-import is a wrong-answer bug, not a slow one.

    Two things I could do here and one I cannot, today. The diagnosis above is free. The implementation is a real piece (persist, stamp, read back, and a bit-identical check against the pack path). The measurement is blocked on RAM: this lane's rule is >20 GiB available for a device run and the box is at 7 GiB (the 4.4 GB container plus a serve dequantising its head needs more than that), so the row waits for headroom rather than for code.

    Claimed on the board. If the lane that owns the Mistral-7B-Instruct-v0.2_rai_1.9.0_npu row (sha256 469946a3…, model.q4nx 5,595,566,152 B) has it on disk, that is the pack to re-time.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A note that changes the shape of the fix: the head image can be packed on the HOST, so the cache does not need a device to fill and its test does not need one either.

    I was about to write this up as "persist the device BOs", which would have made the fix untestable without a device — the same trap #1718/#1720 had to dig out of. Looking at the two packers instead:

    So the image is host-computable from the container, which makes the fix three separable pieces instead of one device-coupled one:

    1. image builder (host): read lm_head from the container, tile it ntiles × (kpad × T), pack each tile, produce {codes, per-column scales} — testable against the existing fixtures with no NPU;
    2. cache (host): key on the container's identity the way device_form() does — fnv1a(canonical path) for the directory plus (file_bytes, mtime) from stamp_matches() — and additionally on the head geometry (kpad, T, ntiles), since the same container at a different T is a different image. Write beside the target and rename, as device_form() already does;
    3. upload (device): read the file into the BOs and skip the pack loop entirely. That is the only part needing a ticket, and it is now a verification rather than where the work happens.

    Why reusing stamp_matches() matters more than it looks: a head cache that survives a re-import produces a wrong head with no error anywhere — the container would be new, the stamp old, and the log would say "reusing". That is a wrong-answer bug in the one tensor whose corruption is hardest to notice (it shows up as slightly odd text, not as garbage). The device_form stamp is already the right rule, and its test already covers the reuse arm, so the head cache inherits a proven rule rather than inventing a second one.

    What I will do next, in that order, unless someone says the head cache belongs in the import step instead (which would be a reasonable alternative — it moves the cost to a place the user already expects to wait, at the price of a cache entry per T):

    1. the builder + its host test, checked byte-for-byte against packB_blocked on the same tiles (the same equivalence npu: extract the production weight packer so it can be tested (#1416) #1718/npu: the production packer walks rows, not 64-wide stripes (2.7x serial, byte-exact at 27B shapes) #1720 established);
    2. the cache + the stamp arms (fresh / hit / stale / geometry mismatch);
    3. then the upload and the npu-hold row this issue asks for, when RAM allows.
  9. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Residual from #1769, which merged before my CHANGES review landed — recording it here because this is where the head-image cache will look.

    #1769 pinned the head tile's layout contract against q4nx_pack_B_int8, but that is not the packer the forward calls:

    npu/forward/src/q4nx_forward.cpp:5125   lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);   <- the real path
    tests/npu_design_grid_test.c            q4nx_pack_B_int8(bt, kpad, T, codes, T, scales)                 <- what was pinned
    

    and the byte-identity between them is a stated intention, not a tested fact — npu/forward/include/q4nx_pack.h says so twice ("Quantization matches I8Ctx::packB_into() exactly", "Identical numerics to I8Ctx::packB_into()") and then, at line 303, marks the comparison as the next slice:

    NOT CALLED YET (engine#1416 / #1713): nothing in the forward path invokes it. load_layer still runs deq() … and I8Ctx::packB_into -> packB_blocked(). The next slice is to … gate that on a byte-identity comparison against packB_blocked's image on the production quantiser.

    Why it matters for this issue specifically: a cache built to the pinned reimplementation's contract can pass that test and still emit a different image from packB_into, which is the exact failure a pre-implementation contract test exists to prevent. The passing test is therefore necessary but not sufficient evidence for the cache.

    Two ways to close it, either is enough:

    1. Pin packB_blocked itself — it is host-side now; npu/forward/src/q4nx_pack_blocked.cpp:18 says it was extracted precisely so it can be tested without the device runtime.
    2. Add the identity assert on the same scratch: q4nx_pack_B_int8(bt, …) == packB_blocked(bt, …) byte for byte. That converts the header's prose into a test and is the slice npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416 already names.

    Either way, keep #1769's transposed-scratch control arm — it is what makes a row/column swap visible, and it is the reason that arm was worth writing.

    Measured while reviewing, for the record: the test compiles and passes on its head (~/wt/rv1769, cmake -DONEBIT_HRX=OFF -DONEBIT_NPU_HOST=ON && build --target npu_design_grid_test, run under ~/lb/thermal-run.sh): five new arms, all cases ok, rc 0. No device used.

  10. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured on origin/main 408b1c8 (which now carries #1769), because the residual from that PR is a question with a number in it: do the two packers agree, and does the pinned layout cover what the forward actually writes?

    At blocked = 0 they agree, exactly

    Packed the same kpad 96 × T 64 scratch through both (gcc-16 -std=c11 -I<npu/forward/include>, linking q4nx_pack.c, q4nx_dequant.c, q4nx_pack_blocked.cpp):

    rc ref=0 blocked=0 | codes identical=YES scales identical=YES | mean ref=0.0314710666 blocked=0.0314710666
    

    But the forward's image is layout-variant, and that is the cache's problem

    q4nx_pack_B_int8_blocked takes a blocked flag and writes through q4nx_layout::blocked_index when it is set (q4nx_pack_blocked.cpp:36,81), and the forward passes it from a setting:

    npu/forward/engine-src/npu_engine_i8ctx_inc.h:91    bool blocked_b = false;
    npu/forward/engine-src/npu_engine_i8ctx_inc.h:501   q4nx_pack_B_int8_blocked(Bm, w, K, N, KD, ND, blocked_b ? 1 : 0, …)
    npu/forward/src/q4nx_forward.cpp:1303               c16->blocked_b = weight_layout_b8_;
    

    So for a model running with the b8 layout the head tile's packed image is the blocked index — deliberately not the reference's row-major image that #1769 pins. The agreement above is the blocked_b = false arm only.

    What that means for the head-image cache (the actual fix for the 10–17 s rebuild): its key has to include weight_layout_b8_ (or the cache must be disabled while it changes), because the same tile has two legitimate byte images and only one of them is pinned. A cache validated against #1769's layout and then used with blocked_b = true would be wrong in a way the existing arms cannot see.

    If it is worth pinning, here is the arm (25 lines, verified)

    I have it on a local branch os/v3-headtile-identity (not opened as a PR — this file is #1769's work and yours to fold in, and I would rather not put a second PR on a file merged twenty minutes ago). It packs the same scratch through q4nx_pack_B_int8_blocked(…, blocked=0, …) beside the reference and asserts identical codes, identical column scales and the same scale sum; the comment states the blocked_b scope explicitly so nobody reads it as "the forward always writes this image".

    Red/green, measured:

    q4nx_pack_blocked.cpp:  col_scales[j] = am / 127.0f  ->  am / 128.0f
      FAIL head tile: and returns the same column scales (got 64, want 0)      FAILED
    restored:
      ok   head tile: the blocked packer (the one the forward calls) is byte-identical (got 0, want 0)
      ok   head tile: and returns the same column scales (got 0, want 0)
      ok   head tile: and the same mean column scale                           all cases ok
    

    Say the word and I will push it as a PR against #1716; otherwise treat the paragraph above as the residual and close it.

  11. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closing the residual I recorded above: the identity arm merged as #1778 (a74bbc6, on main at 98c40eb).

    What is now on main, and what it cost to get right:

    • tests/npu_design_grid_test.c packs the same kpad × T scratch through both packers and asserts identical codes, identical
      column scales and the same scale sum — so the layout tests: pin the lm-head tile's layout before a head-image cache exists (#1716) #1769 pins is now tied to the implementation the forward actually calls
      (packB_into → packB_blocked → q4nx_pack_B_int8_blocked).
    • The first version of that arm could not fail: the fixture's columns all had the same absmax (distinct column scales: 1 of 64), so a
      per-column mistake was invisible — the reviewer's mutant (tis[j] → tis[0]) passed the whole test. It now carries a spread fixture
      (64 distinct of 64) plus an explicit arm that the fixture has T distinct column absmax, and that mutant fails with
      the blocked packer … is byte-identical (got 5901, want 0).
    • The codes reference had to change with it, from lroundf(v / scales[n]) to the packer's own arithmetic (absmax, 1e-12 floor,
      roundf(v * (127/amax))) — the two disagree on 26 of 6144 codes once the columns differ.

    Verified on the merged tree, not the branch: cmake -S . -B /tmp/b1778m -G Ninja -DCMAKE_BUILD_TYPE=Release → build npu_design_grid_test
    rc 0, the test all cases ok under thermal-run. (The first push to CI went red at link because the target did not list
    q4nx_pack_blocked.cpp — found by reading the failing job, fixed in a74bbc6; worth knowing if another test target ever calls the blocked packer.)

    The one thing this does not settle, and the cache must not forget: the pinned image is the blocked_b = false arm. The forward passes
    blocked_b = weight_layout_b8_ (q4nx_forward.cpp:1303), and with it set the head tile's image is the blocked index by design. So a head-image
    cache has to key on weight_layout_b8_ (or refuse while it changes) — the test pins one of the two legitimate images, deliberately, and says so in
    its comment. That is a design constraint on #1716's fix, not a defect in the test.

  12. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this one — the head-image cache in the importer, which is the half that needs no device. Fifteen lines of design first, per the plan's rule for anything sizeable, then the smallest slice that can be proven host-side.

    The design

    1. Where the cost is. q4nx_forward.cpp's head_packed branch calls lctx.packB_into() per tile on every serve: 2 × 4096 × 28672 values at the h4096 shapes, which is the 10–17 s quiet / 284–364 s contended figure this issue measured. The f32 materialisation is not the cost (0.03 s, also measured here).
    2. The image is a pure function of the container's head tile. Nothing device-side enters it: packB_into is memset + q4nx_pack_B_int8_blocked (the C twin extracted for npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416) + a BO sync. So the image can be produced on the host, and its test needs no device — the point I left on this issue at 00:40.
    3. Produce it once, at import. npu-pack import already holds the head as q4_1 tiles and already writes lm_head.weight; it will write one more tensor, <pack>.headimg, holding exactly what packB_into would put in the BO: KD × ND int8 codes plus the per-column fp32 scales, in the packer's own order.
    4. Key it by the thing that can change it. The image's identity is the container's lm_head.weight bytes (tile count + a hash of the tiles) plus the packer's geometry (KD, ND, blocked) and a format version — the same rule the .fconst canonical copy uses after npu-pack: the canonical Token .fconst is rewritten on every call — reuse it when the source identity matches #1662/npu-pack: reuse the canonical Token .fconst when the source identity matches (Fixes #1662) #1669: a change to the packer must invalidate the image, and a stale image must never be served with a plausible-looking correlation.
    5. Load it, do not trust it. The forward's head_packed branch loads the image into the BO (one copy) instead of packing, and then verifies a bounded sample of columns against the tile bytes it also holds; a mismatch refuses by name with both sides' numbers rather than serving.
    6. Never required. No image present → pack as today, and say so once. The cache is an accelerator, not a new dependency, so tests: pin the lm-head tile's layout before a head-image cache exists (#1716) #1769's layout contract stays the only source of truth.
    7. The gate: on one pack, serve time to lm head: N packed weight tile(s) resident drops from the measured 10–17 s to a copy, and the packed bytes are bit-identical to the ones packB_into produces — asserted directly, not inferred from a correlation.

    The first slice (what I will do next, host-side)

    tests/npu_design_grid_test.c already packs the same scratch through both packers and asserts identical codes/scales (#1778). The slice is to write that image to a file, read it back, and assert the reloaded bytes are identical to the freshly packed ones for three geometries, plus a deliberately corrupted-scale case that must refuse. That is the cache's correctness bar with no device, no pack, and no serve.

  13. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Reading the code changed where the cache has to be filled, so here is the correction before anyone (including me) builds the version I wrote on this issue earlier.

    My design said "produce it once, at import". The forward's own code says that image's geometry does not come from the container:

    // q4nx_forward.cpp, run_lm_head
    int i_qkv = -1;                                  // the widest design with K == kpad_
    for (int d = 0; d < ndesigns_; d++)
        if (designs_[d].K != kpad_ || designs_[d].N <= 0) continue;
        if (i_qkv < 0 || designs_[d].N > designs_[i_qkv].N) i_qkv = d;
    const int T = designs_[i_qkv].N;                 // <- the KERNEL SET's tile width
    const int ntiles = (NV + T - 1) / T;
    ...
    const size_t tile_elems = (size_t)kpad_ * (size_t)T;      // <- and kpad_, the padded K
    lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);

    kpad_ and T are properties of the loaded kernel designs, and packB_into also depends on the design's blocked flag (blocked_b) — so the same container packs to different images under kernels-h4096-b8 and under a non-blocked set. The image's identity is therefore

    (container's lm_head.weight bytes, kpad_, T, blocked, packer format version)

    and not the container alone. Two consequences:

    1. The importer can only write it when it is given the kernel set. npu-pack import --kernels <dir> already links a set into <out>/npu and already derives designs (q4nx_derive_designs), so that path can write the image — but the plain import path cannot, and must not guess.
    2. So the general fix is a cache filled at first serve, keyed as above, with the importer's --kernels path as an optional pre-warm. That also means the verification I designed — a bounded sample against the tiles — has to run against the design's kpad_/T, i.e. inside run_lm_head where those are in scope, not in the importer.

    What is already merged and unchanged by this: q4nx_head_image.c (#1802) is format-only — header, version, geometry, hash, refusal on any mismatch — and it takes kd/nd as arguments, so it is indifferent to who fills it. That is why the format landed first: the identity rule holds whichever side fills it.

    Net: my earlier "importer writes it" line was wrong and I am not going to build it that way. The next slice is the first-serve cache plus the --kernels pre-warm, and the measurable claim stays the same (the 10-17 s to lm head: N packed weight tile(s) resident becomes a copy, with the bytes bit-identical to packB_into's).

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Verified on the device after the cache landed: 7.99 s → 1.60 s for the head stage, and the warm-up pass halves with it

    This issue closed via #1950 / #1970 / #1802 / #1769 / #2025, so the useful last word is a measurement rather than a re-description. Two cold serves of the same container
    (Llama-3.2-3B-Instruct_rai_1.9.0_npu imported b8, kernels-3072-8192-qkv5120-b8), stderr timestamped, one npu-hold each:

    serve 1:  lm head: 8 packed weight tile(s) resident (H=3072 T=16384) …
              0 from the image cache, 8 packed (8 written for the next serve) in 7.99 s
              warm-up pass 16099.6 ms      ready 23.55 s
    
    serve 2:  lm head: 8 packed weight tile(s) resident (H=3072 T=16384) …
              8 from the image cache, 0 packed (0 written for the next serve) in 1.60 s
              warm-up pass  8222.1 ms      ready  8.61 s
    

    5.0× on the head stage and 3.9 s off ready, which is the shape the fix promised: the first serve pays the packing and writes the image, every later serve loads it. The log line
    carrying "N from the image cache, M packed (K written for the next serve)" is what makes this checkable by anyone in one glance — that is worth more than the speed-up, because
    the original defect was invisible: the same 10–17 s was paid silently, and on a contended box it became the 284–364 s I measured before (#1716's table).

    The contention half stands as filed: the cost was never a constant, so a row that quotes a single first-request time for this family is quoting the box, not the engine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions