Repository navigation
Description
Our NPU loader materialises the token embedding table in RAM: a memcpy of the whole bf16 table, or a float dequant of the whole quantised table. For the 27B that is 0.74 GiB on disk becoming 2.96 GiB resident, for a table where one row per token is ever read.
Today, with the lines
npu/forward/src/q4nx_forward.cpp, forward init:
- bf16 tables —
embed_.resize((size_t)NV * H)followed bymemcpy(embed_.data(), e, NV*H*sizeof(uint16_t)). The whole table is copied out of the container. - quantised tables —
embed_f_.resize((size_t)NV * (size_t)kpad_)and thenq4nx_dequant_plain_i8(...)/dequant_any(...)over the whole table at 4 bytes per element. The 27B's I8 table is 0.74 GiB on disk and becomes 2.96 GiB resident, three quarters of which is the 4× expansion of a table that is read one row at a time. - For a tied model the same buffer is also the LM head (
lm_w_), so the head's packing reads it too — which is the second half of this issue, not a reason to keep the copy.
Measured, from the containers on this box (embedding tensor size × dtype, model.q4nx header)
| model | embedding on disk | resident today | what we would hold mapped |
|---|---|---|---|
bonsai2-27b-NPU (I8) |
0.74 GiB | 2.96 GiB (f32 dequant) | 0.74 GiB |
bonsai2-27b-NPU-b8 (I8) |
0.74 GiB | 2.96 GiB | 0.74 GiB |
qwen3-coder-30b-a3b-moe (BF16) |
0.58 GiB | 0.58 GiB | 0.58 GiB |
ZR1-1.5B-npu-bf16 (BF16) |
0.43 GiB | 0.43 GiB | 0.43 GiB |
d3-lite-bf16 (BF16) |
0.47 GiB | 0.47 GiB | 0.47 GiB |
Qwen3-0.6B (BF16) |
0.29 GiB | 0.29 GiB | 0.29 GiB |
The bf16 rows cost page-cache-backed bytes we then duplicate privately; the I8 rows cost 4× the file in private anonymous memory. The saving is not a tweak: it is the difference between "resident = container size" and "resident = dequantised size", and it grows with the vocabulary, which is where the largest tables are.
What it should do
- Map
model.q4nxread-only once (npu/file_map.halready exists and is the right tool) and take every tensor's bytes from the mapping instead of copying the ones we can read in place. - One row per token, gathered at use: for a bf16 table, the row is
base + token*H*2and is converted to floats into a small scratch (Hfloats, 10 KB at H=2560). For a quantised table, dequantise that one row with the per-row (or per-32-column) scale into the same scratch. No whole-table buffer exists at any point. - The tied head streams rows through whatever packs it today; it must not require a materialised table either. The head-image cache work (engine#1716) is the natural place for this, and a streaming source is a prerequisite for it rather than a competitor.
- Per-layer embedding tables take the same treatment: they are small individually but there is one per layer, and today each is materialised.
Gate
- Identical output is the bar: 3 prompts × 3 repeats, byte-identical text before and after, plus the logit dump comparison (
--dump-logits) bit-identical on the same prompts. - Host RSS before/after on three models, measured the same way each time —
VmRSSof the serving process after load and after one decode, one bf16 model, one quantised table, one 27B-class pack:- expectation from the table above: the 27B row drops from ~2.96 GiB to ~0.74 GiB of private memory (the mapping is page-cache, reclaimable, and not counted as anonymous).
- No decode-speed regression: tok/s within the control-pair noise floor on the same model, measured in the same ticket.
- The row then records the container/mode, the model, and the RSS numbers before and after.
Blocked right now, and what I will do about it
The measurement needs the NPU lane and the box has 1.1 GiB available of 128 GiB at the time of filing (MemFree 1.1 GB, mem_info_gtt_used 101 GiB), with two llama-server processes holding GTT outside any gpu-hold ticket (3 h 21 m and 6 h 56 m old) — reported to the owner rather than killed from here. The code change and its host-side tests do not need the device, so they come first; the RSS rows wait for a window.
Refs: engine#881 (the kpad-before-first-dequant ordering this must preserve), #1725 (a gemma-4 embedding component that is f32 and unquantised), #1716 (head packing), #671 (pack loader).
Activity
bong-water-water-bong commentedon Nov 10, 2026
Design decisions after reading the storage layout, recorded here so the next step does not need to re-derive them.
1. The container is already mapped read-only; nothing new needs mapping.
Model::Model(npu/model.cpp)map_file()smodel.q4nxand everyTensor::datapoints into that mapping. So this issue is not "add mmap" — it is "stop copying and stop expanding what is already mapped". That makes the diff much smaller than the framing suggests, and it means the per-layer embedding tables only need the same treatment as the token table.2. Both readers are already element-wise accessors, so no call site changes.
Q4nxNpuForward::embed_at(token, k)andlm_at(n, k)are the only readers ofembed_/embed_f_/lm_head_f_, and both are called withkinnermost (for (n …) { for (k …) }at the host head and at both tile-packing loops). So a row cache — gather one row, serve H elements from it, move to the next row — is exactly the right granularity and needs no restructuring of the head or the packing loops. That is also what "one row per token" means concretely here.3. bf16 and plain I8 need no row buffer at all. bf16: the element is
bf16_to_f32(((const uint16_t*)src)[token*H + k])straight from the mapping. Plain I8 (esz == en, the per-32-column-scale case, which is what the 27B and gemma-4 containers use): one int8 byte × one float scale index. Both are O(1) per element with no buffer, and they cover every embedding dtype currently on this box (the header dump above is BF16 or I8 throughout).4. The tiled/chunked quantised case needs one new primitive, and it belongs in the public tree.
dequant_anyrefuses anything that is not a multiple of the tile grid, and the tile layout is public (npu/forward/include/q4nx_dequant.h:Q4NX_TILE_ROWS 32,Q4NX_TILE_COLS 256,Q4NX_TILE_BYTES 5120), so a row is not byte-contiguous: it lives as rowr % 32inside each ofcols/256tiles. The primitive to add iswhich reads only the
cols/256tiles covering rowr(20 tile reads at kpad 5120, ~100 KB touched) instead of the 655 KB a 32-row band costs. It is testable host-side against the existingq4nx_dequant_tensor_chunkedoutput with no device and no model, in the same shape astests/npu_padded_grid_test.cpp.5. The tie/lm_head question resolves itself.
lm_atfalls back toembed_atwhen the model is tied, so both go through the same row cache; an untied model's separatelm_head_f_gets the identical treatment. One code path, not two.Order of work, smallest first: bf16 + plain I8 (no new primitive, removes the copy for every bf16 model and the 4× expansion for the 27B/gemma-class ones) → the tile-row primitive + chunked embeddings → per-layer tables → RSS rows on three models under one
npu-holdticket each.Status right now: blocked on box memory, not on the design. At 23:4x the box had 1.1 GiB available of 128 GiB with
mem_info_gtt_usedat 101 GiB (twollama-serverprocesses 3 h 21 m and 6 h 56 m old outside anygpu-holdticket, reported to the owner), and load average 44. Acc1pluscannot run there, so I have not written code I cannot compile and test — the first commit will be the bf16/I8 row cache with its host-side test, as soon as the box has room.bong-water-water-bong commentedon Nov 10, 2026
Two things to add from the FastFlowLM v1.0.12 digest I did tonight — one is a detail of our code that makes the resident figure worse than the issue's own arithmetic, the other is external corroboration that the fix is safe.
1. For a quantised container we hold both buffers, not one.
embed_.resize((size_t)NV * H)is at :669, before the dtype branch at :684, and the quantised path then addsembed_f_.resize((size_t)NV * kpad_)at :688. So a quantised table costs the bf16 buffer plus the float one:The quantised path never reads
embed_(it readsembed_f_viaembed_f32_,:1481), so the first resize is pure waste on every container whose embedding is I8 — which is exactly the 27B's. Movingembed_.resize()into the bf16 branch is a one-line change that removes it; the read-one-row fix then removes the other.2. There is already a one-row reader in our own tree.
npu/lane.cpp:157:That is the pattern the forward needs — one row per token, straight out of the mapping, no table-sized buffer. The forward's problem is not that row reads are missing from the codebase; it is that the forward chose the whole-table form while the NPU lane path next to it did not.
3. Vendor corroboration that the fix keeps the numbers. FastFlowLM v1.0.12 (2026-11-05) did exactly this and reports the effect: token and per-layer embedding tables are now read from
model.q4nxthrough a read-only memory map, one row per token, and they state "Output, decode speed and time to first token are unchanged", with private memory going 3.9 GiB → 0.3 GiB ongemma4-it:e4b/nemotron3.5-bolt:e4b(4.2 → 0.5 GiB on the DFlash tag; 0.46/0.92/1.16 GiB saved on qwen3.5 0.8b/2b/4b). Same container format, same access pattern we would use — so the risk of the one-row mmap changing output is a claim a shipping vendor already made against the same shape, which is worth something when we weigh it.npu: read the token embedding table in place, one row per token, inst…
npu: read the token embedding table in place, one row per token, inst…
bong-water-water-bong commentedon Nov 10, 2026
Measured the question the merged PR left open ("no container takes the plain-I8 branch"), with a temporary print in the forward init rather than by reading the heuristic. It is not a bug in the branch condition — the pack is not the dtype its header says.
en = NV*H = 248320 * 5120 = 1,271,398,400, but the tensor holds 794,624,000 bytes — and 794,624,000 / 5120 = 155,200, which is exactly the row count in the headershape. So the bytes fit two readings at once:[155200, 5120]— whatdtypeandshapesay[248320, 5120]on the container's K grid — what the runtime does(248320/32) * (5120/256) = 155,200tiles of 5,120 B = 0.625 B per elementdequant_anypicks (b): 155,200 tiles × 5,120 B = 794,624,000 bytes exactly, i.e. a 4-bit packed table, which decodes to 1.27 G elements and materialises as 4.74 GiB of f32. The same table is 0.74 GiB in the file — a 6.4× expansion while the model loads.Consequences, and they change what the rest of this issue has to do:
r, read thecols/256 = 20tiles that cover it (20 × 5,120 B = ~100 KB touched per token) instead of decoding 155,200 tiles.Next commit on this issue is
q4nx_dequant_tile_row()in the public tree (npu/forward/src/q4nx_dequant.c, declared besideq4nx_dequant_tensor_chunked), tested host-side againstq4nx_dequant_tensor_chunkedoutput the same waytests/npu_dequant_bf16_test.ctests the plain-I8 row read — bit for bit, plus the tile-straddling and last-partial-row arms. Thenembed_at()takes the tile-packed branch and the 27B's 4.74 GiB goes away.bong-water-water-bong commentedon Nov 10, 2026
Status of the two halves the assignment names, with what is measured and what is not.
Token embedding table: done, measured, merged/queued.
#1795(merged): bf16 and plain-int8 tables read in place, one row per token. Three models,RssAnonafter load: 436 → 139 MiB (Qwen3-0.6B), 1669 → 1224 MiB (ZR1-1.5B), 714 → 229 MiB (d3-lite); each delta equals that model's own table (297/297, 445/445, 485/481), output byte-identical on 3 prompts each.#1807(open, one verdict): the tile-packed path. On the device route the 27B goes 9773 → 2658 MiBRssAnonafter load (7.15 GiB returned) with the step-1 vocab-wide float32 logits byte-identical (c93ff216…both arms).Per-layer embedding tables: located, costed, and not implementable-as-measured on this box.
The per-layer path is real and it does materialise:
ple_model_proj_isNL * ple_dim * Hfloats built at load by transposing eachmodel.per_layer_model_proj.weight_layer{l}from its row-grouped[ple_dim/32, H, 32]bf16 layout into a dense f32 matrix (npu/forward/src/q4nx_forward.cpp, theple_dim_ > 0block). Its cost isNL * ple_dim * H * 4bytes — and it is consumed per layer as a context projection, so "one row per token" here means doing that transpose arithmetic in the accessor instead of in a load-time copy, exactly the shape of the work that landed for the token table.What stops the measurement, and it is not the code: no container on this box has these tensors at all. Scanning every
model.q4nxhere forper_layer_model_proj: 0 hits. The only gemma-family thing on disk is~/scratch-packs/gemma3-270m-npu2, which is not a converted container and is not a per-layer-embedding model. So there is nothing to measure on, and the plan's rule is measured numbers only — I am not going to land a change here and quote a formula as if it were a result.Order I would do it in when a gemma-4-class pack exists (W4 owns those packs,
#1725is the neighbouring embedding defect): import one, recordRssAnonbefore/after the change on the same three prompts, and confirm the per-layer context projection is byte-identical by the same logits comparison used for the 27B. The code change itself is ~40 lines in the accessor plus one test arm on the transpose arithmetic ([a*32+g][c]from[a][c][g]), which is the part that can be wrong while looking right.If the owner wants this half done before a pack exists, say so and I will land the accessor change behind the unit test and mark the row unmeasured — but that is a weaker deliverable than the token half and I would rather it not be counted as F4 done.
npu: read the tile-packed embedding table one row per token too, so t…
bong-water-water-bong commentedon Nov 10, 2026
Status at main
ee5b436, with one new measurement that changes the "per-layer tables" half (lane cl-npurt).What is on main: #1795 (
bf16and plain-int8 tables read in place), #1807 (tile-packed tables, one row per token; 27BRssAnonafter load 9773 → 2658 MiB, step-1 logits byte-identical), #1871 (the per-row path is chosen by the row reader's own rule). The tied head needs no table either:lm_at()falls through toembed_at().Item 4, per-layer embedding tables — already not materialised. The status comment above looked for
per_layer_model_projand found no pack. A gemma-4 pack is on the box now (gemma4-e2b-npu2,model.q4nx4,671,875,302 B), and its header shows which tensor is the table:model.per_layer_token_embd.weight[262144, 8960]model.embed_tokens.weight[262144, 1536]model.per_layer_model_proj.weight_layer{0..34}[8, 1536, 32]eachple_model_proj_)The per-layer token table is read by
load_ple_block(): one 32-row band dequantised from the mapping per lookup (32 × 8960 floats), never the table.ple_model_proj_is a 256 × 1536 projection per layer, 55 MB in all, not a vocabulary-sized table.Measured,
1bit forward-servebuilt fromee5b436,NPU_INFER_HOST_GEMM=1(no device opened),/proc/<pid>/status:So a container whose two tables would be 10.9 GiB as floats serves in under 0.7 GiB of private memory. (This row is about memory only: it ran in host mode, and I did not judge the text it produced.)
What is still open against this issue's own gate: the no decode-speed regression row — tok/s before and after on the same model in one ticket — is not in #1795 or #1807. It needs a pre-#1795 build next to a current one and one 27B-class ticket. I am leaving the issue open for that single row rather than closing on memory alone.
bong-water-water-bong commentedon Nov 10, 2026
The missing gate row: decode speed before and after, 27B, one ticket (lane cl-npurt).
Two builds side by side, same compiler line and NPU runtime: before =
d805b90^(the parent of #1795), after = main8f10bfbplus #1970 (which does not touch decode). Modelbonsai2-27b-NPU,1bit forward-serveon the device, onenpu-holdticket (967 s), serves interleaved before / after / before / after, fresh process each time, two requests per serve ("The capital of France is", 6 tokens, temperature 0). Numbers are the serve's owntimings.predicted_msovercompletion_tokens.Paris.\nThe capital ofParis.\nThe capital ofParis.\nThe capital ofParis.\nThe capital ofEvery "after" run sits inside the spread of the "before" runs, so there is no decode-speed regression from reading the embedding row at use; the text is the same in all eight runs. The after build logs
embedding dtype I8 -> read in place from the container, one row per token (engine#1784): 4.74 GiB not materialised.Limits, stated plainly: 6-token runs (this pack decodes at about 7–8 s per token on this route, so longer runs do not fit a ticket), a busy box (the ticket queued 416 s behind another lane), and the noise floor above is wide. The row can rule out a regression larger than that floor, not a 1–2 % one.
RssAnonreadings from this ticket are not usable (the box was reclaiming memory between serves: the same "before" binary read 10.2 GB and then 4.1 GB after load); the memory rows are the ones already recorded in #1807 and in my comment above.With this, every line of the issue's gate has a recorded row: identical output and RSS in #1795/#1807, per-layer tables in the comment above, decode speed here. Closing.