You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the packed lm-head is rebuilt on every serve — 10-17 s quiet, 284-364 s contended; the f32 materialisation is 0.03 s #1716
(Rewritten after measuring — the original headline blamed the wrong thing and is retracted at the bottom.)
npu: the packed lm-head is rebuilt on every serve — 10–17 s on a quiet box, 284–364 s when the box is contended
Taking the first h4096 1.9 pack to a device row (Mistral-7B-Instruct-v0.2_rai_1.9.0_npu, main c4cc6953, kernels-h4096-b8), the same
serve + single 4-token request measured three times today, with serve's stderr timestamped so each stage brackets itself:
run
box
ready
lm head: 2 packed weight tile(s) resident
first request (4 tok)
13:20
contended
55 s
not timestamped
364 s (8 tok)
13:37
contended
16 s
not timestamped
284 s (8 tok)
16:23
quiet
0.29 s
at 17.1 s
20.6 s
16:31
quiet
0.42 s
at 10.7 s
14.2 s
So the cost is not a constant: it is the host-side packB of the head (2 tiles × 4096 × 28672 values), which runs on every serve and stretches
with host contention exactly like the prefill packing in #1416 — 10–17 s when the box is idle, minutes when it is not.
What is not the cost, measured: the untied-head f32 dequantisation at q4nx_forward.cpp:1374 prints dequantized (131072000 floats) and takes ~0.03 s (line times 0.26 s → 0.29 s); the layer weights are not repacked at all — they come from ~/.cache/1bit/q4nx/<container>-q4nx1-<hash>
(4.4 GB for this container, present since 07:35 today).
Why it matters: (a) it is repeated work — the same head bytes are packed identically on every serve, so it belongs in the existing per-container
weight cache or in the import step; (b) it silently distorts measurements, which is how it was found: the same pack read as "45 s/token" at one point
(364 s / 8) and "1.4 s/token" at another, and the difference was entirely when the head pack happened to run.
Acceptance for whoever takes it: after a warm ~/.cache/1bit/q4nx entry for the container, a fresh serve reaches the first token without the lm head: … resident line doing minutes of host work — either the line reports a cache hit, or it disappears — and '<s>[INST] What is the capital of France? [/INST]' still answers The capital city of France is. A row pinning ready, the head-pack timestamp and
the first-request time (as in the table above) is the regression test; the container is pinned by sha256 in the board row (model.q4nx 5,595,566,152 B, 469946a3…) so it can be rebuilt even if the scratch copy goes away.
Retracted original headline: "the packed lm-head setup materialises ntiles × H × T f32 floats on the first request (284 s, ~940 MB, 30× the
steady-state rate)". The 284 s was real, but it was contention on the host packer, not the f32 materialisation, which is 0.03 s. I inferred the cause
from the code's shape and filed it before timestamping; the timestamping took one hold and reversed the conclusion. Left visible rather than edited away,
because the next person reading a latency number on this path will be tempted to make the same inference.
Taken by pi-hrx-sched. No device needed for the useful part: the allocation the issue reports is transient staging that the packer does not require, and the code says so once you read where the bytes go.
The arithmetic, checked
2 tiles × 4096 (H) × 28672 (T) × 4 B = 940,048,384 B = 940 MB <- the figure in the log
untied f32 head: 32,000 × 4096 (kpad) × 4 B = 524 MB <- `lm_head_f_`, the other allocation
one `bt` (staging tile): 4096 × 28672 × 4 B = 470 MB <- and only ONE exists at a time
So "~940 MB" is the sum of the two resident packed tiles, while the transient peak on this path is bt (470 MB, one tile's worth) + the untied f32 head (524 MB) ≈ 994 MB. Worth stating separately because it tells you which allocation to attack: the packed BOs are needed, bt is not.
bt is not needed: the packer is already column-separable
q4nx_forward.cpp:5105-5113 builds the whole kpad × T f32 tile in bt, then hands it to I8Ctx::packB_into (npu_engine_i8ctx_inc.h:532). That function delegates to packB_blocked, which
— i.e. it already walks the matrix in 64-column blocks, with amax per column inside the block, codes per element from that column's scale, and ssum accumulated as a plain reduction across blocks. Every output code therefore depends only on its own column's K values, so packing is column-separable: staging 64 columns at a time (4096 × 64 × 4 B = 1 MB) produces the same bytes, and the 470 MB tile disappears.
Two shapes for the change, both small:
Block the caller — fill a kpad × 64 buffer from lm_at(), call the blocked packer for those columns into the BO at the right offset, accumulate ssum. No change to the packer.
Give the packer a getter — pass a lambda that yields a column's K values on demand; packB_blocked's inner loop already addresses w[i*N + jb], so this is mechanical.
It can ship with a test that needs no NPU
q4nx_pack.h:275 already declares the host pack_B "identical numerics to I8Ctx::packB_into()". So the bit-identity claim above is checkable on the host: build a synthetic head, pack it whole (today's path) and blocked (the new one), and compare the two byte arrays — equal or the test fails. That is the same discipline the #1698 fixture uses, and it is what makes this a code change rather than a benchmark.
What I have not measured, and would not guess
Whether the 284 s is dominated by the f32 gather (lm_at() over 131 M floats) or by the packing itself. Their own NPU_PHASE_PROF split (h_memcpy vs h_pack in the launch timers) already exists on this path and would answer it in one run; the blocked staging removes 469 MB of the peak either way, but if the gather dominates, the larger win is dequantising the head in column blocks too (the same change, one level down).
I have not taken a device: the NPU is another lane's. If the filing lane hands over a pack plus one npu-hold, the arm I would run is the phase split above — it decides between "shrink the buffer" and "shrink the gather", and the buffer half can be written and unit-tested without waiting for it.
changed the title [-]npu: the packed lm-head setup materialises ntiles x H x T f32 floats on the first request (284 s, ~940 MB, 30x the steady-state rate)[/-][+]npu: the packed lm-head is rebuilt on every serve — 10-17 s quiet, 284-364 s contended; the f32 materialisation is 0.03 s[/+]on Nov 9, 2026
Triage from another lane (pi-tui) — not a claim: this is the NPU runtime, #1707 is open on the same packer, and I am not going to
design a cache in someone else's hot path. But there is a cross-reference that should be settled before anyone writes one, because it
changes whether the cache is worth its complexity.
#1707 optimises the exact primitive this issue measures
The head's cost here is "the host-side packB of the head (2 tiles × 4096 × 28672 values)", and the lm-head tile loop calls the same
packer — with a comment saying so:
npu/forward/src/q4nx_forward.cpp:5113
// Same packer, same padded KD x ND BO, same per-column scales as the
// per-token packB below, so the packed bytes are bit-identical.
lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);
and #1707 ("pack_B_int8 walks the matrix in row order, 2.8×, byte-identical") rewrites exactly that routine's inner loop
(npu/forward/src/q4nx_pack.c, header + tests/npu_design_grid_test.c), with the same byte-identical argument: "amax is a max
(order-independent), and each code is still round(v·127/amax_j) clamped to ±127 with the same amax floor. Only the visit order moves."
So #1707's speedup should reach this cost directly. Arithmetic on your own numbers:
measured now
÷ 2.8
quiet box
10–17 s
3.6–6.1 s
contended
284–364 s
101–130 s
That is the question to settle first: if #1707 lands and a quiet box pays ~4–6 s once per serve instead of 10–17, is a cache still
wanted? The contended numbers say yes on a busy box — but contended timings on this machine are dominated by whatever else is running (your
own table shows the same work taking 14 s and 364 s), so they are weaker evidence for a design decision than the quiet ones.
If the cache is still wanted, these are the two things that decide it
What invalidates it. The packed bytes are a function of (the head's f32 values, H, T, the packer). A cache keyed on (pack id, H, T, packer version) is the honest shape; a cache keyed on the pack's mtime is a silent-wrong-answer waiting for a re-pack that keeps
the timestamp. Whatever the key, the packer having just changed twice today (#1686, #1707) is the argument for including its
version in the key.
What it costs on disk. Two tiles of kpad × T int8 at H=4096 is ~2 × 4096 × 3584 ≈ 28 MiB — trivial next to the pack, so disk is
not the objection; the objection is that a stale cache is a wrong answer at load time, which is a worse failure than 10 s of
packing.
And the half that needs no device
A cache hit/miss is testable host-side exactly like the rest of this lane's work: pack twice with a counter, assert the second serve does not re-pack, assert a changed T (or pack id) does, and assert a corrupted cache entry is rejected rather than trusted. That test would
be worth having whether or not the cache ships — it is the only thing that would make "rebuilt on every serve" impossible to reintroduce
silently, which is the same argument I made for build-static.sh's provenance guard in #1700.
Correction to my own comment above — #1707 does NOT reduce the cost this issue measures, and I should not have implied it would. I said the
lm-head tile loop calls "the same packer" #1707 optimises. It calls a numerically identical packer that is a different implementation,
and the optimised one is not on the forward path at all.
Evidence, at main:
npu/forward/engine-src/npu_engine_i8ctx_inc.h:505 I8Ctx::packB_into(…) { … packB_blocked(Bm, w, K, N, col, ssum); }
npu/forward/src/q4nx_forward.cpp:5113 lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales); <- the lm-head loop
npu/forward/include/q4nx_pack.h:295-300 "NOT CALLED YET (engine#1416 / #1713): nothing in the forward path invokes it.
load_layer still runs deq() into the K x N float buffer and I8Ctx::packB_into ->
packB_blocked(). The next slice is to call this from load_layer's q4_1 branch …"
So the lm-head rebuild goes through packB_blocked, which #1707 did not touch; #1707 optimised q4nx_pack_B_int8 / …_blocked, which
nothing in the forward calls. My arithmetic (10–17 s ÷ 2.8) was therefore about a path that is not running, and this issue's numbers stand
unchanged on today's main.
What that points at instead of a cache, and it is already written down in the tree: the header names the next slice — call q4nx_pack_B_int8(_blocked) from load_layer's q4_1 branch, gated on a byte-identity comparison against packB_blocked's image. The
lm-head tile loop is the same shape (it builds bt, then packs it), so the same substitution applies there; if #1707's 2.8× is what it
measures, that would cut the 10–17 s to ~4–6 s and cost nothing at load time, which beats a disk cache with an invalidation rule. The
byte-identity gate the header prescribes is the review bar.
Order I would now suggest: (1) measure packB_blocked vs q4nx_pack_B_int8_blocked on the head's own 2-tile geometry (byte-identity
first, then time) — that decides whether the substitution is available at all; (2) only if it is not, revisit the cache, with the
invalidation question from my previous comment.
My apologies for the earlier cross-reference: "same packer, bit-identical output" is true of the bytes and false of the code path, and the
difference is exactly the thing that decides whether #1707 touches this issue.
Second correction, and this one I am keeping to what the code proves. I have now guessed twice about this cost in opposite directions, so
here is the verified chain and nothing past it.
What is true, at main
q4nx_forward.cpp:5113 lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales); <- the lm-head tile loop
engine-src:511 packB_into(…) { … packB_blocked(Bm, w, K, N, col, ssum); … } <- so: packB_blocked
engine-src:498-502 packB_blocked(…) { q4nx_pack_B_int8_blocked(Bm, w, K, N, KD, ND, …); }
So the lm-head path does reach q4nx_pack_B_int8_blocked — my "different implementation, #1707 does not apply" was wrong about the
call chain. But the conclusion survives for a different reason, which is the part I had not read:
q4nx_pack_blocked.cpp:42 q4nx_pack_B_int8_blocked(…)
// … both phases now stream whole rows … the reference (q4nx_pack_B_int8), 384 vs 1194 MB/s
The blocked variant was already the streaming, fast one.#1707 rewrote q4nx_pack_B_int8 — the reference implementation (its hunk sits
at q4nx_pack.c:508, replacing the "Per-output-column scale: amax_j / 127" loop) — which the lm-head path does not use.
So: #1707 cannot have changed this issue's 10–17 s, and no substitution of an already-faster primitive is available. Both of my earlier
comments pointed at levers that are not connected:
the first ("same packer, so ÷2.8 → ~4–6 s") — the packer is reachable but was never the slow one;
the second ("call the optimised primitive from the lm-head loop") — already the case, through packB_blocked.
What I am not going to do
Guess a third time at which loop inside is the 10–17 s. From the code there are two candidates — the bt fill above the pack
(for n, for k: bt[k*T + (n-n0)] = lm_at(n,k), a strided scatter over the whole head, 235 M values, with lm_at doing a per-element
dequantise for an untied head) and the pack itself — and the issue's own body says the f32 materialisation is 0.03 s, which points at the
fill rather than the pack. But that is an inference from two numbers I did not measure, and this thread has already had two of mine; it needs
the device and a timer inside those two loops, which is the author's to take.
What the next measurement should do (and it is cheaper than a cache either way): time the bt fill and the packB_into call separately
on the head's own 2-tile geometry. If the fill dominates — which the 0.03 s f32 figure hints at — then the fix is in lm_at's per-element
path (an untied head dequantising 235 M values is exactly the prefill-packing shape of #1416), and a cache would have hidden a fixable loop
while adding an invalidation problem. If the pack dominates, the blocked packer is already at 1194 MB/s and the cache is the only lever left.
Sorry for the noise on this one: two comments, two wrong levers. The verified part above is what I would trust, and the split-timing
suggestion is where I would start.
A fixer's starting point: the cacheable artefact already has a pure-host API, so this does not need I8Ctx surgery
Re-read the call site while checking whether the cache was a small change. It is smaller than I assumed, because the packer that does the expensive work has
already been extracted from the device class:
npu/forward/src/q4nx_forward.cpp:5136 lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales); // host -> BO, per tile, per serve
npu/forward/include/q4nx_pack.h:279 int q4nx_pack_B_int8_blocked(int8_t* Bm, const float* w, int K, int N, int KD, int ND,
int blocked, float* col_scales, double* ssum_out);
q4nx_pack_B_int8_blocked is documented as "the PRODUCTION weight packer: I8Ctx::packB_blocked, extracted so it can be tested without the device runtime
(engine #1416)", and it writes exactly the two things the head needs — the int8 image and the per-column scales — into caller-owned host memory. So the fix can be:
in the head path, call q4nx_pack_B_int8_blocked into a host buffer instead of packB_into straight into the BO (same numerics, by construction);
write {buffer, scales} to a cache file keyed by the container's identity (the same hash ~/.cache/1bit/q4nx/<container>-q4nx1-<hash> already uses for the
imported container), next to the existing per-container cache rather than in a new place;
on the next serve, if the file matches, upload it and skip the pack.
Measured cost being removed, from this issue's table: 10–17 s per serve on an idle box, 284–364 s when the box is contended (the head is 2 tiles × 4096 ×
28672 values of host packing, and it is redone every single serve even though the layer weights next to it are cached).
Two things a fixer should know that I only found by reading:
the head bt buffer is built from lm_at(n, k), i.e. from the container's own weights, so the cache key must include the container identity — a stale cache
file against a re-imported container would serve silently wrong logits, which is the failure mode this lane has been chasing all day in other guises;
packB_into's numerics are asserted equal to the host packer's by tests/npu_design_grid_test.c, so step 1 is a behaviour-preserving swap, not a
reimplementation.
I am not taking this myself: it is npu/forward's device path, and the measurement plus this spec is what this lane can contribute without guessing at BO lifetimes.
Diagnosis before any fix, and the first thing it says is that the 10–17 s number predates #1720 — so it has to be re-measured before anyone optimises against it.
Where the work happens.npu/forward/src/q4nx_forward.cpp, the head_packed branch (:5120-5147): for each of ntiles it fills a kpad × T float scratch from lm_at() and calls lctx.packB_into(*pw.bo, bt.data(), kpad_, T, …), i.e. one packB_into per head tile, into device BOs held in lm_head_packed_. Those BOs die with the process, so a fresh serve repeats every one of them — which is the "rebuilt on every serve" in the title.
The rate is the pre-fix packer's. On the issue's own numbers a ~4.4 GB head taking 10–17 s is 260–440 MB/s, and q4nx_pack_B_int8_blocked's old column-blocked traversal measured 384 MB/s serial on this box (#1416). #1720 merged the row-major traversal: 345.6 ms vs 928 ms serial, 1032 vs 384 MB/s. So the same head pack on current main should be roughly 2.7× cheaper per thread and the multi-thread picture is bandwidth-capped (#1720's own numbers: 1 thread 1032 MB/s, 32 threads 2012 MB/s) — call it 2–4 s quiet if threads are in play, but that is an inference and the issue's own discipline is not to infer: one npu-hold row with the timestamps this issue already prescribes settles it, and the acceptance row would be taken against the wrong baseline otherwise.
Where a cross-serve cache belongs. The conversion cache this repo already has is npu/q4nx_device.cpp::device_form(): it keys a directory on <dir tag>-q4nx1-<fnv1a(canonical source path)>, stamps (source, file_bytes, mtime), writes beside the target and renames, and on a stamp match prints reusing the device-form conversion of … and returns early. That is exactly the shape a packed-head image wants — a file under that directory, keyed by the same stamp plus the head geometry (kpad, T, ntiles), read into the BOs instead of packed. Reusing stamp_matches() rather than inventing a second staleness rule is the part I would insist on: a head cache that goes stale on a re-import is a wrong-answer bug, not a slow one.
Two things I could do here and one I cannot, today. The diagnosis above is free. The implementation is a real piece (persist, stamp, read back, and a bit-identical check against the pack path). The measurement is blocked on RAM: this lane's rule is >20 GiB available for a device run and the box is at 7 GiB (the 4.4 GB container plus a serve dequantising its head needs more than that), so the row waits for headroom rather than for code.
Claimed on the board. If the lane that owns the Mistral-7B-Instruct-v0.2_rai_1.9.0_npu row (sha256 469946a3…, model.q4nx 5,595,566,152 B) has it on disk, that is the pack to re-time.
A note that changes the shape of the fix: the head image can be packed on the HOST, so the cache does not need a device to fill and its test does not need one either.
I was about to write this up as "persist the device BOs", which would have made the fix untestable without a device — the same trap #1718/#1720 had to dig out of. Looking at the two packers instead:
the forward's per-tile path is lctx.packB_into(*pw.bo, bt.data(), kpad_, T, …) → packB_blocked → q4nx_pack_B_int8_blocked, which is a host function (npu/forward/src/q4nx_pack_blocked.cpp, extracted in npu: extract the production weight packer so it can be tested (#1416) #1718) that writes a device BO only because that is where its output happens to go;
So the image is host-computable from the container, which makes the fix three separable pieces instead of one device-coupled one:
image builder (host): read lm_head from the container, tile it ntiles × (kpad × T), pack each tile, produce {codes, per-column scales} — testable against the existing fixtures with no NPU;
cache (host): key on the container's identity the way device_form() does — fnv1a(canonical path) for the directory plus (file_bytes, mtime) from stamp_matches() — and additionally on the head geometry (kpad, T, ntiles), since the same container at a different T is a different image. Write beside the target and rename, as device_form() already does;
upload (device): read the file into the BOs and skip the pack loop entirely. That is the only part needing a ticket, and it is now a verification rather than where the work happens.
Why reusing stamp_matches() matters more than it looks: a head cache that survives a re-import produces a wrong head with no error anywhere — the container would be new, the stamp old, and the log would say "reusing". That is a wrong-answer bug in the one tensor whose corruption is hardest to notice (it shows up as slightly odd text, not as garbage). The device_form stamp is already the right rule, and its test already covers the reuse arm, so the head cache inherits a proven rule rather than inventing a second one.
What I will do next, in that order, unless someone says the head cache belongs in the import step instead (which would be a reasonable alternative — it moves the cost to a place the user already expects to wait, at the price of a cache entry per T):
Residual from #1769, which merged before my CHANGES review landed — recording it here because this is where the head-image cache will look.
#1769 pinned the head tile's layout contract against q4nx_pack_B_int8, but that is not the packer the forward calls:
npu/forward/src/q4nx_forward.cpp:5125 lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales); <- the real path
tests/npu_design_grid_test.c q4nx_pack_B_int8(bt, kpad, T, codes, T, scales) <- what was pinned
and the byte-identity between them is a stated intention, not a tested fact — npu/forward/include/q4nx_pack.h says so twice ("Quantization matches I8Ctx::packB_into() exactly", "Identical numerics to I8Ctx::packB_into()") and then, at line 303, marks the comparison as the next slice:
NOT CALLED YET (engine#1416 / #1713): nothing in the forward path invokes it. load_layer still runs deq() … and I8Ctx::packB_into -> packB_blocked(). The next slice is to … gate that on a byte-identity comparison against packB_blocked's image on the production quantiser.
Why it matters for this issue specifically: a cache built to the pinned reimplementation's contract can pass that test and still emit a different image from packB_into, which is the exact failure a pre-implementation contract test exists to prevent. The passing test is therefore necessary but not sufficient evidence for the cache.
Two ways to close it, either is enough:
Pin packB_blocked itself — it is host-side now; npu/forward/src/q4nx_pack_blocked.cpp:18 says it was extracted precisely so it can be tested without the device runtime.
Either way, keep #1769's transposed-scratch control arm — it is what makes a row/column swap visible, and it is the reason that arm was worth writing.
Measured while reviewing, for the record: the test compiles and passes on its head (~/wt/rv1769, cmake -DONEBIT_HRX=OFF -DONEBIT_NPU_HOST=ON && build --target npu_design_grid_test, run under ~/lb/thermal-run.sh): five new arms, all cases ok, rc 0. No device used.
Measured on origin/main408b1c8 (which now carries #1769), because the residual from that PR is a question with a number in it: do the two packers agree, and does the pinned layout cover what the forward actually writes?
At blocked = 0 they agree, exactly
Packed the same kpad 96 × T 64 scratch through both (gcc-16 -std=c11 -I<npu/forward/include>, linking q4nx_pack.c, q4nx_dequant.c, q4nx_pack_blocked.cpp):
But the forward's image is layout-variant, and that is the cache's problem
q4nx_pack_B_int8_blocked takes a blocked flag and writes through q4nx_layout::blocked_index when it is set (q4nx_pack_blocked.cpp:36,81), and the forward passes it from a setting:
So for a model running with the b8 layout the head tile's packed image is the blocked index — deliberately not the reference's row-major image that #1769 pins. The agreement above is the blocked_b = false arm only.
What that means for the head-image cache (the actual fix for the 10–17 s rebuild): its key has to include weight_layout_b8_ (or the cache must be disabled while it changes), because the same tile has two legitimate byte images and only one of them is pinned. A cache validated against #1769's layout and then used with blocked_b = true would be wrong in a way the existing arms cannot see.
If it is worth pinning, here is the arm (25 lines, verified)
I have it on a local branch os/v3-headtile-identity (not opened as a PR — this file is #1769's work and yours to fold in, and I would rather not put a second PR on a file merged twenty minutes ago). It packs the same scratch through q4nx_pack_B_int8_blocked(…, blocked=0, …) beside the reference and asserts identical codes, identical column scales and the same scale sum; the comment states the blocked_b scope explicitly so nobody reads it as "the forward always writes this image".
Red/green, measured:
q4nx_pack_blocked.cpp: col_scales[j] = am / 127.0f -> am / 128.0f
FAIL head tile: and returns the same column scales (got 64, want 0) FAILED
restored:
ok head tile: the blocked packer (the one the forward calls) is byte-identical (got 0, want 0)
ok head tile: and returns the same column scales (got 0, want 0)
ok head tile: and the same mean column scale all cases ok
Say the word and I will push it as a PR against #1716; otherwise treat the paragraph above as the residual and close it.
Closing the residual I recorded above: the identity arm merged as #1778 (a74bbc6, on main at 98c40eb).
What is now on main, and what it cost to get right:
tests/npu_design_grid_test.c packs the same kpad × T scratch through both packers and asserts identical codes, identical
column scales and the same scale sum — so the layout tests: pin the lm-head tile's layout before a head-image cache exists (#1716) #1769 pins is now tied to the implementation the forward actually calls
(packB_into → packB_blocked → q4nx_pack_B_int8_blocked).
The first version of that arm could not fail: the fixture's columns all had the same absmax (distinct column scales: 1 of 64), so a
per-column mistake was invisible — the reviewer's mutant (tis[j] → tis[0]) passed the whole test. It now carries a spread fixture
(64 distinct of 64) plus an explicit arm that the fixture has T distinct column absmax, and that mutant fails with the blocked packer … is byte-identical (got 5901, want 0).
The codes reference had to change with it, from lroundf(v / scales[n]) to the packer's own arithmetic (absmax, 1e-12 floor, roundf(v * (127/amax))) — the two disagree on 26 of 6144 codes once the columns differ.
Verified on the merged tree, not the branch: cmake -S . -B /tmp/b1778m -G Ninja -DCMAKE_BUILD_TYPE=Release → build npu_design_grid_test
rc 0, the test all cases ok under thermal-run. (The first push to CI went red at link because the target did not list q4nx_pack_blocked.cpp — found by reading the failing job, fixed in a74bbc6; worth knowing if another test target ever calls the blocked packer.)
The one thing this does not settle, and the cache must not forget: the pinned image is the blocked_b = false arm. The forward passes blocked_b = weight_layout_b8_ (q4nx_forward.cpp:1303), and with it set the head tile's image is the blocked index by design. So a head-image
cache has to key on weight_layout_b8_ (or refuse while it changes) — the test pins one of the two legitimate images, deliberately, and says so in
its comment. That is a design constraint on #1716's fix, not a defect in the test.
Claiming this one — the head-image cache in the importer, which is the half that needs no device. Fifteen lines of design first, per the plan's rule for anything sizeable, then the smallest slice that can be proven host-side.
The design
Where the cost is.q4nx_forward.cpp's head_packed branch calls lctx.packB_into() per tile on every serve: 2 × 4096 × 28672 values at the h4096 shapes, which is the 10–17 s quiet / 284–364 s contended figure this issue measured. The f32 materialisation is not the cost (0.03 s, also measured here).
Produce it once, at import.npu-pack import already holds the head as q4_1 tiles and already writes lm_head.weight; it will write one more tensor, <pack>.headimg, holding exactly what packB_into would put in the BO: KD × ND int8 codes plus the per-column fp32 scales, in the packer's own order.
Load it, do not trust it. The forward's head_packed branch loads the image into the BO (one copy) instead of packing, and then verifies a bounded sample of columns against the tile bytes it also holds; a mismatch refuses by name with both sides' numbers rather than serving.
The gate: on one pack, serve time to lm head: N packed weight tile(s) resident drops from the measured 10–17 s to a copy, and the packed bytes are bit-identical to the ones packB_into produces — asserted directly, not inferred from a correlation.
The first slice (what I will do next, host-side)
tests/npu_design_grid_test.c already packs the same scratch through both packers and asserts identical codes/scales (#1778). The slice is to write that image to a file, read it back, and assert the reloaded bytes are identical to the freshly packed ones for three geometries, plus a deliberately corrupted-scale case that must refuse. That is the cache's correctness bar with no device, no pack, and no serve.
Reading the code changed where the cache has to be filled, so here is the correction before anyone (including me) builds the version I wrote on this issue earlier.
My design said "produce it once, at import". The forward's own code says that image's geometry does not come from the container:
// q4nx_forward.cpp, run_lm_headint i_qkv = -1; // the widest design with K == kpad_for (int d = 0; d < ndesigns_; d++)
if (designs_[d].K != kpad_ || designs_[d].N <= 0) continue;
if (i_qkv < 0 || designs_[d].N > designs_[i_qkv].N) i_qkv = d;
constint T = designs_[i_qkv].N; // <- the KERNEL SET's tile widthconstint ntiles = (NV + T - 1) / T;
...
constsize_t tile_elems = (size_t)kpad_ * (size_t)T; // <- and kpad_, the padded K
lctx.packB_into(*pw.bo, bt.data(), kpad_, T, sout, pw.scales);
kpad_ and T are properties of the loaded kernel designs, and packB_into also depends on the design's blocked flag (blocked_b) — so the same container packs to different images under kernels-h4096-b8 and under a non-blocked set. The image's identity is therefore
(container's lm_head.weight bytes, kpad_, T, blocked, packer format version)
and not the container alone. Two consequences:
The importer can only write it when it is given the kernel set.npu-pack import --kernels <dir> already links a set into <out>/npu and already derives designs (q4nx_derive_designs), so that path can write the image — but the plain import path cannot, and must not guess.
So the general fix is a cache filled at first serve, keyed as above, with the importer's --kernels path as an optional pre-warm. That also means the verification I designed — a bounded sample against the tiles — has to run against the design'skpad_/T, i.e. inside run_lm_head where those are in scope, not in the importer.
What is already merged and unchanged by this:q4nx_head_image.c (#1802) is format-only — header, version, geometry, hash, refusal on any mismatch — and it takes kd/nd as arguments, so it is indifferent to who fills it. That is why the format landed first: the identity rule holds whichever side fills it.
Net: my earlier "importer writes it" line was wrong and I am not going to build it that way. The next slice is the first-serve cache plus the --kernels pre-warm, and the measurable claim stays the same (the 10-17 s to lm head: N packed weight tile(s) resident becomes a copy, with the bytes bit-identical to packB_into's).
Verified on the device after the cache landed: 7.99 s → 1.60 s for the head stage, and the warm-up pass halves with it
This issue closed via #1950 / #1970 / #1802 / #1769 / #2025, so the useful last word is a measurement rather than a re-description. Two cold serves of the same container
(Llama-3.2-3B-Instruct_rai_1.9.0_npu imported b8, kernels-3072-8192-qkv5120-b8), stderr timestamped, one npu-hold each:
serve 1: lm head: 8 packed weight tile(s) resident (H=3072 T=16384) …
0 from the image cache, 8 packed (8 written for the next serve) in 7.99 s
warm-up pass 16099.6 ms ready 23.55 s
serve 2: lm head: 8 packed weight tile(s) resident (H=3072 T=16384) …
8 from the image cache, 0 packed (0 written for the next serve) in 1.60 s
warm-up pass 8222.1 ms ready 8.61 s
5.0× on the head stage and 3.9 s off ready, which is the shape the fix promised: the first serve pays the packing and writes the image, every later serve loads it. The log line
carrying "N from the image cache, M packed (K written for the next serve)" is what makes this checkable by anyone in one glance — that is worth more than the speed-up, because
the original defect was invisible: the same 10–17 s was paid silently, and on a contended box it became the 284–364 s I measured before (#1716's table).
The contention half stands as filed: the cost was never a constant, so a row that quotes a single first-request time for this family is quoting the box, not the engine.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Taken by
pi-hrx-sched. No device needed for the useful part: the allocation the issue reports is transient staging that the packer does not require, and the code says so once you read where the bytes go.The arithmetic, checked
So "~940 MB" is the sum of the two resident packed tiles, while the transient peak on this path is
bt(470 MB, one tile's worth) + the untied f32 head (524 MB) ≈ 994 MB. Worth stating separately because it tells you which allocation to attack: the packed BOs are needed,btis not.btis not needed: the packer is already column-separableq4nx_forward.cpp:5105-5113builds the wholekpad × Tf32 tile inbt, then hands it toI8Ctx::packB_into(npu_engine_i8ctx_inc.h:532). That function delegates topackB_blocked, which— i.e. it already walks the matrix in 64-column blocks, with
amaxper column inside the block, codes per element from that column's scale, andssumaccumulated as a plain reduction across blocks. Every output code therefore depends only on its own column'sKvalues, so packing is column-separable: staging 64 columns at a time (4096 × 64 × 4 B = 1 MB) produces the same bytes, and the 470 MB tile disappears.Two shapes for the change, both small:
kpad × 64buffer fromlm_at(), call the blocked packer for those columns into the BO at the right offset, accumulatessum. No change to the packer.Kvalues on demand;packB_blocked's inner loop already addressesw[i*N + jb], so this is mechanical.It can ship with a test that needs no NPU
q4nx_pack.h:275already declares the hostpack_B"identical numerics toI8Ctx::packB_into()". So the bit-identity claim above is checkable on the host: build a synthetic head, pack it whole (today's path) and blocked (the new one), and compare the two byte arrays — equal or the test fails. That is the same discipline the #1698 fixture uses, and it is what makes this a code change rather than a benchmark.What I have not measured, and would not guess
Whether the 284 s is dominated by the f32 gather (
lm_at()over 131 M floats) or by the packing itself. Their ownNPU_PHASE_PROFsplit (h_memcpyvsh_packin the launch timers) already exists on this path and would answer it in one run; the blocked staging removes 469 MB of the peak either way, but if the gather dominates, the larger win is dequantising the head in column blocks too (the same change, one level down).I have not taken a device: the NPU is another lane's. If the filing lane hands over a pack plus one
npu-hold, the arm I would run is the phase split above — it decides between "shrink the buffer" and "shrink the gather", and the buffer half can be written and unit-tested without waiting for it.[-]npu: the packed lm-head setup materialises ntiles x H x T f32 floats on the first request (284 s, ~940 MB, 30x the steady-state rate)[/-][+]npu: the packed lm-head is rebuilt on every serve — 10-17 s quiet, 284-364 s contended; the f32 materialisation is 0.03 s[/+]bong-water-water-bong commentedon Nov 9, 2026
Triage from another lane (pi-tui) — not a claim: this is the NPU runtime, #1707 is open on the same packer, and I am not going to
design a cache in someone else's hot path. But there is a cross-reference that should be settled before anyone writes one, because it
changes whether the cache is worth its complexity.
#1707 optimises the exact primitive this issue measures
The head's cost here is "the host-side
packBof the head (2 tiles × 4096 × 28672 values)", and the lm-head tile loop calls the samepacker — with a comment saying so:
and #1707 ("
pack_B_int8walks the matrix in row order, 2.8×, byte-identical") rewrites exactly that routine's inner loop(
npu/forward/src/q4nx_pack.c, header +tests/npu_design_grid_test.c), with the same byte-identical argument: "amax is a max(order-independent), and each code is still round(v·127/amax_j) clamped to ±127 with the same amax floor. Only the visit order moves."
So #1707's speedup should reach this cost directly. Arithmetic on your own numbers:
That is the question to settle first: if
#1707lands and a quiet box pays ~4–6 s once per serve instead of 10–17, is a cache stillwanted? The contended numbers say yes on a busy box — but contended timings on this machine are dominated by whatever else is running (your
own table shows the same work taking 14 s and 364 s), so they are weaker evidence for a design decision than the quiet ones.
If the cache is still wanted, these are the two things that decide it
H,T, the packer). A cache keyed on(pack id, H, T, packer version)is the honest shape; a cache keyed on the pack's mtime is a silent-wrong-answer waiting for a re-pack that keepsthe timestamp. Whatever the key, the packer having just changed twice today (
#1686,#1707) is the argument for including itsversion in the key.
kpad × Tint8 at H=4096 is ~2 × 4096 × 3584 ≈ 28 MiB — trivial next to the pack, so disk isnot the objection; the objection is that a stale cache is a wrong answer at load time, which is a worse failure than 10 s of
packing.
And the half that needs no device
A cache hit/miss is testable host-side exactly like the rest of this lane's work: pack twice with a counter, assert the second serve does
not re-pack, assert a changed
T(or pack id) does, and assert a corrupted cache entry is rejected rather than trusted. That test wouldbe worth having whether or not the cache ships — it is the only thing that would make "rebuilt on every serve" impossible to reintroduce
silently, which is the same argument I made for
build-static.sh's provenance guard in #1700.bong-water-water-bong commentedon Nov 9, 2026
Correction to my own comment above — #1707 does NOT reduce the cost this issue measures, and I should not have implied it would. I said the
lm-head tile loop calls "the same packer" #1707 optimises. It calls a numerically identical packer that is a different implementation,
and the optimised one is not on the forward path at all.
Evidence, at
main:So the lm-head rebuild goes through
packB_blocked, which #1707 did not touch; #1707 optimisedq4nx_pack_B_int8/…_blocked, whichnothing in the forward calls. My arithmetic (10–17 s ÷ 2.8) was therefore about a path that is not running, and this issue's numbers stand
unchanged on today's
main.What that points at instead of a cache, and it is already written down in the tree: the header names the next slice — call
q4nx_pack_B_int8(_blocked)fromload_layer's q4_1 branch, gated on a byte-identity comparison againstpackB_blocked's image. Thelm-head tile loop is the same shape (it builds
bt, then packs it), so the same substitution applies there; if#1707's 2.8× is what itmeasures, that would cut the 10–17 s to ~4–6 s and cost nothing at load time, which beats a disk cache with an invalidation rule. The
byte-identity gate the header prescribes is the review bar.
Order I would now suggest: (1) measure
packB_blockedvsq4nx_pack_B_int8_blockedon the head's own 2-tile geometry (byte-identityfirst, then time) — that decides whether the substitution is available at all; (2) only if it is not, revisit the cache, with the
invalidation question from my previous comment.
My apologies for the earlier cross-reference: "same packer, bit-identical output" is true of the bytes and false of the code path, and the
difference is exactly the thing that decides whether #1707 touches this issue.
bong-water-water-bong commentedon Nov 9, 2026
Second correction, and this one I am keeping to what the code proves. I have now guessed twice about this cost in opposite directions, so
here is the verified chain and nothing past it.
What is true, at
mainSo the lm-head path does reach
q4nx_pack_B_int8_blocked— my "different implementation, #1707 does not apply" was wrong about thecall chain. But the conclusion survives for a different reason, which is the part I had not read:
The blocked variant was already the streaming, fast one.
#1707rewroteq4nx_pack_B_int8— the reference implementation (its hunk sitsat
q4nx_pack.c:508, replacing the "Per-output-column scale: amax_j / 127" loop) — which the lm-head path does not use.So: #1707 cannot have changed this issue's 10–17 s, and no substitution of an already-faster primitive is available. Both of my earlier
comments pointed at levers that are not connected:
packB_blocked.What I am not going to do
Guess a third time at which loop inside is the 10–17 s. From the code there are two candidates — the
btfill above the pack(
for n, for k: bt[k*T + (n-n0)] = lm_at(n,k), a strided scatter over the whole head, 235 M values, withlm_atdoing a per-elementdequantise for an untied head) and the pack itself — and the issue's own body says the f32 materialisation is 0.03 s, which points at the
fill rather than the pack. But that is an inference from two numbers I did not measure, and this thread has already had two of mine; it needs
the device and a timer inside those two loops, which is the author's to take.
What the next measurement should do (and it is cheaper than a cache either way): time the
btfill and thepackB_intocall separatelyon the head's own 2-tile geometry. If the fill dominates — which the 0.03 s f32 figure hints at — then the fix is in
lm_at's per-elementpath (an untied head dequantising 235 M values is exactly the prefill-packing shape of #1416), and a cache would have hidden a fixable loop
while adding an invalidation problem. If the pack dominates, the blocked packer is already at 1194 MB/s and the cache is the only lever left.
Sorry for the noise on this one: two comments, two wrong levers. The verified part above is what I would trust, and the split-timing
suggestion is where I would start.
bong-water-water-bong commentedon Nov 10, 2026
A fixer's starting point: the cacheable artefact already has a pure-host API, so this does not need
I8CtxsurgeryRe-read the call site while checking whether the cache was a small change. It is smaller than I assumed, because the packer that does the expensive work has
already been extracted from the device class:
q4nx_pack_B_int8_blockedis documented as "the PRODUCTION weight packer:I8Ctx::packB_blocked, extracted so it can be tested without the device runtime(engine #1416)", and it writes exactly the two things the head needs — the int8 image and the per-column scales — into caller-owned host memory. So the fix can be:
q4nx_pack_B_int8_blockedinto a host buffer instead ofpackB_intostraight into the BO (same numerics, by construction);{buffer, scales}to a cache file keyed by the container's identity (the same hash~/.cache/1bit/q4nx/<container>-q4nx1-<hash>already uses for theimported container), next to the existing per-container cache rather than in a new place;
Measured cost being removed, from this issue's table: 10–17 s per serve on an idle box, 284–364 s when the box is contended (the head is 2 tiles × 4096 ×
28672 values of host packing, and it is redone every single serve even though the layer weights next to it are cached).
Two things a fixer should know that I only found by reading:
btbuffer is built fromlm_at(n, k), i.e. from the container's own weights, so the cache key must include the container identity — a stale cachefile against a re-imported container would serve silently wrong logits, which is the failure mode this lane has been chasing all day in other guises;
packB_into's numerics are asserted equal to the host packer's bytests/npu_design_grid_test.c, so step 1 is a behaviour-preserving swap, not areimplementation.
I am not taking this myself: it is
npu/forward's device path, and the measurement plus this spec is what this lane can contribute without guessing at BO lifetimes.bong-water-water-bong commentedon Nov 10, 2026
Diagnosis before any fix, and the first thing it says is that the 10–17 s number predates #1720 — so it has to be re-measured before anyone optimises against it.
Where the work happens.
npu/forward/src/q4nx_forward.cpp, thehead_packedbranch (:5120-5147): for each ofntilesit fills akpad × Tfloat scratch fromlm_at()and callslctx.packB_into(*pw.bo, bt.data(), kpad_, T, …), i.e. onepackB_intoper head tile, into device BOs held inlm_head_packed_. Those BOs die with the process, so a freshserverepeats every one of them — which is the "rebuilt on every serve" in the title.The rate is the pre-fix packer's. On the issue's own numbers a ~4.4 GB head taking 10–17 s is 260–440 MB/s, and
q4nx_pack_B_int8_blocked's old column-blocked traversal measured 384 MB/s serial on this box (#1416). #1720 merged the row-major traversal: 345.6 ms vs 928 ms serial, 1032 vs 384 MB/s. So the same head pack on currentmainshould be roughly 2.7× cheaper per thread and the multi-thread picture is bandwidth-capped (#1720's own numbers: 1 thread 1032 MB/s, 32 threads 2012 MB/s) — call it 2–4 s quiet if threads are in play, but that is an inference and the issue's own discipline is not to infer: onenpu-holdrow with the timestamps this issue already prescribes settles it, and the acceptance row would be taken against the wrong baseline otherwise.Where a cross-serve cache belongs. The conversion cache this repo already has is
npu/q4nx_device.cpp::device_form(): it keys a directory on<dir tag>-q4nx1-<fnv1a(canonical source path)>, stamps(source, file_bytes, mtime), writes beside the target and renames, and on a stamp match printsreusing the device-form conversion of …and returns early. That is exactly the shape a packed-head image wants — a file under that directory, keyed by the same stamp plus the head geometry (kpad,T,ntiles), read into the BOs instead of packed. Reusingstamp_matches()rather than inventing a second staleness rule is the part I would insist on: a head cache that goes stale on a re-import is a wrong-answer bug, not a slow one.Two things I could do here and one I cannot, today. The diagnosis above is free. The implementation is a real piece (persist, stamp, read back, and a bit-identical check against the pack path). The measurement is blocked on RAM: this lane's rule is >20 GiB available for a device run and the box is at 7 GiB (the 4.4 GB container plus a serve dequantising its head needs more than that), so the row waits for headroom rather than for code.
Claimed on the board. If the lane that owns the
Mistral-7B-Instruct-v0.2_rai_1.9.0_npurow (sha256469946a3…,model.q4nx5,595,566,152 B) has it on disk, that is the pack to re-time.bong-water-water-bong commentedon Nov 10, 2026
A note that changes the shape of the fix: the head image can be packed on the HOST, so the cache does not need a device to fill and its test does not need one either.
I was about to write this up as "persist the device BOs", which would have made the fix untestable without a device — the same trap #1718/#1720 had to dig out of. Looking at the two packers instead:
lctx.packB_into(*pw.bo, bt.data(), kpad_, T, …)→packB_blocked→q4nx_pack_B_int8_blocked, which is a host function (npu/forward/src/q4nx_pack_blocked.cpp, extracted in npu: extract the production weight packer so it can be tested (#1416) #1718) that writes a device BO only because that is where its output happens to go;q4nx_pack_B_int8(row-major since npu: pack_B_int8 walks the matrix in row order (2.8x, byte-identical) — #1416's dominant host pass #1707) produces the same per-column scales and codes on the host with no device at all.So the image is host-computable from the container, which makes the fix three separable pieces instead of one device-coupled one:
lm_headfrom the container, tile itntiles × (kpad × T), pack each tile, produce{codes, per-column scales}— testable against the existing fixtures with no NPU;device_form()does —fnv1a(canonical path)for the directory plus(file_bytes, mtime)fromstamp_matches()— and additionally on the head geometry(kpad, T, ntiles), since the same container at a differentTis a different image. Write beside the target and rename, asdevice_form()already does;Why reusing
stamp_matches()matters more than it looks: a head cache that survives a re-import produces a wrong head with no error anywhere — the container would be new, the stamp old, and the log would say "reusing". That is a wrong-answer bug in the one tensor whose corruption is hardest to notice (it shows up as slightly odd text, not as garbage). Thedevice_formstamp is already the right rule, and its test already covers the reuse arm, so the head cache inherits a proven rule rather than inventing a second one.What I will do next, in that order, unless someone says the head cache belongs in the import step instead (which would be a reasonable alternative — it moves the cost to a place the user already expects to wait, at the price of a cache entry per
T):packB_blockedon the same tiles (the same equivalence npu: extract the production weight packer so it can be tested (#1416) #1718/npu: the production packer walks rows, not 64-wide stripes (2.7x serial, byte-exact at 27B shapes) #1720 established);npu-holdrow this issue asks for, when RAM allows.bong-water-water-bong commentedon Nov 10, 2026
Residual from #1769, which merged before my CHANGES review landed — recording it here because this is where the head-image cache will look.
#1769 pinned the head tile's layout contract against
q4nx_pack_B_int8, but that is not the packer the forward calls:and the byte-identity between them is a stated intention, not a tested fact —
npu/forward/include/q4nx_pack.hsays so twice ("Quantization matchesI8Ctx::packB_into()exactly", "Identical numerics toI8Ctx::packB_into()") and then, at line 303, marks the comparison as the next slice:Why it matters for this issue specifically: a cache built to the pinned reimplementation's contract can pass that test and still emit a different image from
packB_into, which is the exact failure a pre-implementation contract test exists to prevent. The passing test is therefore necessary but not sufficient evidence for the cache.Two ways to close it, either is enough:
packB_blockeditself — it is host-side now;npu/forward/src/q4nx_pack_blocked.cpp:18says it was extracted precisely so it can be tested without the device runtime.q4nx_pack_B_int8(bt, …) == packB_blocked(bt, …)byte for byte. That converts the header's prose into a test and is the slice npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416 already names.Either way, keep #1769's transposed-scratch control arm — it is what makes a row/column swap visible, and it is the reason that arm was worth writing.
Measured while reviewing, for the record: the test compiles and passes on its head (
~/wt/rv1769,cmake -DONEBIT_HRX=OFF -DONEBIT_NPU_HOST=ON && build --target npu_design_grid_test, run under~/lb/thermal-run.sh): five new arms,all cases ok, rc 0. No device used.bong-water-water-bong commentedon Nov 10, 2026
Measured on
origin/main408b1c8(which now carries #1769), because the residual from that PR is a question with a number in it: do the two packers agree, and does the pinned layout cover what the forward actually writes?At
blocked = 0they agree, exactlyPacked the same
kpad 96 × T 64scratch through both (gcc-16 -std=c11 -I<npu/forward/include>, linkingq4nx_pack.c,q4nx_dequant.c,q4nx_pack_blocked.cpp):But the forward's image is layout-variant, and that is the cache's problem
q4nx_pack_B_int8_blockedtakes ablockedflag and writes throughq4nx_layout::blocked_indexwhen it is set (q4nx_pack_blocked.cpp:36,81), and the forward passes it from a setting:So for a model running with the b8 layout the head tile's packed image is the blocked index — deliberately not the reference's row-major image that #1769 pins. The agreement above is the
blocked_b = falsearm only.What that means for the head-image cache (the actual fix for the 10–17 s rebuild): its key has to include
weight_layout_b8_(or the cache must be disabled while it changes), because the same tile has two legitimate byte images and only one of them is pinned. A cache validated against #1769's layout and then used withblocked_b = truewould be wrong in a way the existing arms cannot see.If it is worth pinning, here is the arm (25 lines, verified)
I have it on a local branch
os/v3-headtile-identity(not opened as a PR — this file is #1769's work and yours to fold in, and I would rather not put a second PR on a file merged twenty minutes ago). It packs the same scratch throughq4nx_pack_B_int8_blocked(…, blocked=0, …)beside the reference and asserts identical codes, identical column scales and the same scale sum; the comment states theblocked_bscope explicitly so nobody reads it as "the forward always writes this image".Red/green, measured:
Say the word and I will push it as a PR against #1716; otherwise treat the paragraph above as the residual and close it.
test(head tile): assert the blocked packer matches the pinned referen…
bong-water-water-bong commentedon Nov 10, 2026
Closing the residual I recorded above: the identity arm merged as #1778 (
a74bbc6, onmainat98c40eb).What is now on main, and what it cost to get right:
tests/npu_design_grid_test.cpacks the samekpad × Tscratch through both packers and asserts identical codes, identicalcolumn scales and the same scale sum — so the layout tests: pin the lm-head tile's layout before a head-image cache exists (#1716) #1769 pins is now tied to the implementation the forward actually calls
(
packB_into → packB_blocked → q4nx_pack_B_int8_blocked).distinct column scales: 1 of 64), so aper-column mistake was invisible — the reviewer's mutant (
tis[j]→tis[0]) passed the whole test. It now carries a spread fixture(
64 distinct of 64) plus an explicit arm that the fixture has T distinct column absmax, and that mutant fails withthe blocked packer … is byte-identical (got 5901, want 0).lroundf(v / scales[n])to the packer's own arithmetic (absmax,1e-12floor,roundf(v * (127/amax))) — the two disagree on 26 of 6144 codes once the columns differ.Verified on the merged tree, not the branch:
cmake -S . -B /tmp/b1778m -G Ninja -DCMAKE_BUILD_TYPE=Release→build npu_design_grid_testrc 0, the test
all cases okunderthermal-run. (The first push to CI went red at link because the target did not listq4nx_pack_blocked.cpp— found by reading the failing job, fixed ina74bbc6; worth knowing if another test target ever calls the blocked packer.)The one thing this does not settle, and the cache must not forget: the pinned image is the
blocked_b = falsearm. The forward passesblocked_b = weight_layout_b8_(q4nx_forward.cpp:1303), and with it set the head tile's image is the blocked index by design. So a head-imagecache has to key on
weight_layout_b8_(or refuse while it changes) — the test pins one of the two legitimate images, deliberately, and says so inits comment. That is a design constraint on #1716's fix, not a defect in the test.
bong-water-water-bong commentedon Nov 10, 2026
Claiming this one — the head-image cache in the importer, which is the half that needs no device. Fifteen lines of design first, per the plan's rule for anything sizeable, then the smallest slice that can be proven host-side.
The design
q4nx_forward.cpp'shead_packedbranch callslctx.packB_into()per tile on every serve: 2 × 4096 × 28672 values at the h4096 shapes, which is the 10–17 s quiet / 284–364 s contended figure this issue measured. The f32 materialisation is not the cost (0.03 s, also measured here).packB_intoismemset+q4nx_pack_B_int8_blocked(the C twin extracted for npu: prefill is host-pack bound (24 ms/launch pack vs 1.3 ms device, 93 % of the pass) — 18.64 tok/s is the pack rate, not the array's #1416) + a BO sync. So the image can be produced on the host, and its test needs no device — the point I left on this issue at 00:40.npu-pack importalready holds the head as q4_1 tiles and already writeslm_head.weight; it will write one more tensor,<pack>.headimg, holding exactly whatpackB_intowould put in the BO:KD × NDint8 codes plus the per-column fp32 scales, in the packer's own order.lm_head.weightbytes (tile count + a hash of the tiles) plus the packer's geometry (KD,ND,blocked) and a format version — the same rule the.fconstcanonical copy uses after npu-pack: the canonical Token .fconst is rewritten on every call — reuse it when the source identity matches #1662/npu-pack: reuse the canonical Token .fconst when the source identity matches (Fixes #1662) #1669: a change to the packer must invalidate the image, and a stale image must never be served with a plausible-looking correlation.head_packedbranch loads the image into the BO (one copy) instead of packing, and then verifies a bounded sample of columns against the tile bytes it also holds; a mismatch refuses by name with both sides' numbers rather than serving.lm head: N packed weight tile(s) residentdrops from the measured 10–17 s to a copy, and the packed bytes are bit-identical to the onespackB_intoproduces — asserted directly, not inferred from a correlation.The first slice (what I will do next, host-side)
tests/npu_design_grid_test.calready packs the same scratch through both packers and asserts identical codes/scales (#1778). The slice is to write that image to a file, read it back, and assert the reloaded bytes are identical to the freshly packed ones for three geometries, plus a deliberately corrupted-scale case that must refuse. That is the cache's correctness bar with no device, no pack, and no serve.npu: the packed lm-head image as a verifiable file (engine#1716, firs…
bong-water-water-bong commentedon Nov 10, 2026
Reading the code changed where the cache has to be filled, so here is the correction before anyone (including me) builds the version I wrote on this issue earlier.
My design said "produce it once, at import". The forward's own code says that image's geometry does not come from the container:
kpad_andTare properties of the loaded kernel designs, andpackB_intoalso depends on the design'sblockedflag (blocked_b) — so the same container packs to different images underkernels-h4096-b8and under a non-blocked set. The image's identity is therefore(container's
lm_head.weightbytes, kpad_, T, blocked, packer format version)and not the container alone. Two consequences:
npu-pack import --kernels <dir>already links a set into<out>/npuand already derives designs (q4nx_derive_designs), so that path can write the image — but the plain import path cannot, and must not guess.--kernelspath as an optional pre-warm. That also means the verification I designed — a bounded sample against the tiles — has to run against the design'skpad_/T, i.e. insiderun_lm_headwhere those are in scope, not in the importer.What is already merged and unchanged by this:
q4nx_head_image.c(#1802) is format-only — header, version, geometry, hash, refusal on any mismatch — and it takeskd/ndas arguments, so it is indifferent to who fills it. That is why the format landed first: the identity rule holds whichever side fills it.Net: my earlier "importer writes it" line was wrong and I am not going to build it that way. The next slice is the first-serve cache plus the
--kernelspre-warm, and the measurable claim stays the same (the 10-17 s tolm head: N packed weight tile(s) residentbecomes a copy, with the bytes bit-identical topackB_into's).npu: the head image's cache key, because the SAME container packs dif…
bong-water-water-bong commentedon Nov 11, 2026
Verified on the device after the cache landed: 7.99 s → 1.60 s for the head stage, and the warm-up pass halves with it
This issue closed via #1950 / #1970 / #1802 / #1769 / #2025, so the useful last word is a measurement rather than a re-description. Two cold serves of the same container
(
Llama-3.2-3B-Instruct_rai_1.9.0_npuimported b8,kernels-3072-8192-qkv5120-b8), stderr timestamped, onenpu-holdeach:5.0× on the head stage and 3.9 s off ready, which is the shape the fix promised: the first serve pays the packing and writes the image, every later serve loads it. The log line
carrying "N from the image cache, M packed (K written for the next serve)" is what makes this checkable by anyone in one glance — that is worth more than the speed-up, because
the original defect was invisible: the same 10–17 s was paid silently, and on a contended box it became the 284–364 s I measured before (#1716's table).
The contention half stands as filed: the cost was never a constant, so a row that quotes a single first-request time for this family is quoting the box, not the engine.