Skip to content

npu: the loader materialises the token embedding table in RAM (0.74 GiB I8 -> 2.96 GiB resident); map it read-only and read one row per token #1784

Closed

Description

@bong-water-water-bong

Our NPU loader materialises the token embedding table in RAM: a memcpy of the whole bf16 table, or a float dequant of the whole quantised table. For the 27B that is 0.74 GiB on disk becoming 2.96 GiB resident, for a table where one row per token is ever read.

Today, with the lines

npu/forward/src/q4nx_forward.cpp, forward init:

  • bf16 tables — embed_.resize((size_t)NV * H) followed by memcpy(embed_.data(), e, NV*H*sizeof(uint16_t)). The whole table is copied out of the container.
  • quantised tables — embed_f_.resize((size_t)NV * (size_t)kpad_) and then q4nx_dequant_plain_i8(...) / dequant_any(...) over the whole table at 4 bytes per element. The 27B's I8 table is 0.74 GiB on disk and becomes 2.96 GiB resident, three quarters of which is the 4× expansion of a table that is read one row at a time.
  • For a tied model the same buffer is also the LM head (lm_w_), so the head's packing reads it too — which is the second half of this issue, not a reason to keep the copy.

Measured, from the containers on this box (embedding tensor size × dtype, model.q4nx header)

model embedding on disk resident today what we would hold mapped
bonsai2-27b-NPU (I8) 0.74 GiB 2.96 GiB (f32 dequant) 0.74 GiB
bonsai2-27b-NPU-b8 (I8) 0.74 GiB 2.96 GiB 0.74 GiB
qwen3-coder-30b-a3b-moe (BF16) 0.58 GiB 0.58 GiB 0.58 GiB
ZR1-1.5B-npu-bf16 (BF16) 0.43 GiB 0.43 GiB 0.43 GiB
d3-lite-bf16 (BF16) 0.47 GiB 0.47 GiB 0.47 GiB
Qwen3-0.6B (BF16) 0.29 GiB 0.29 GiB 0.29 GiB

The bf16 rows cost page-cache-backed bytes we then duplicate privately; the I8 rows cost 4× the file in private anonymous memory. The saving is not a tweak: it is the difference between "resident = container size" and "resident = dequantised size", and it grows with the vocabulary, which is where the largest tables are.

What it should do

  1. Map model.q4nx read-only once (npu/file_map.h already exists and is the right tool) and take every tensor's bytes from the mapping instead of copying the ones we can read in place.
  2. One row per token, gathered at use: for a bf16 table, the row is base + token*H*2 and is converted to floats into a small scratch (H floats, 10 KB at H=2560). For a quantised table, dequantise that one row with the per-row (or per-32-column) scale into the same scratch. No whole-table buffer exists at any point.
  3. The tied head streams rows through whatever packs it today; it must not require a materialised table either. The head-image cache work (engine#1716) is the natural place for this, and a streaming source is a prerequisite for it rather than a competitor.
  4. Per-layer embedding tables take the same treatment: they are small individually but there is one per layer, and today each is materialised.

Gate

  • Identical output is the bar: 3 prompts × 3 repeats, byte-identical text before and after, plus the logit dump comparison (--dump-logits) bit-identical on the same prompts.
  • Host RSS before/after on three models, measured the same way each time — VmRSS of the serving process after load and after one decode, one bf16 model, one quantised table, one 27B-class pack:
    • expectation from the table above: the 27B row drops from ~2.96 GiB to ~0.74 GiB of private memory (the mapping is page-cache, reclaimable, and not counted as anonymous).
  • No decode-speed regression: tok/s within the control-pair noise floor on the same model, measured in the same ticket.
  • The row then records the container/mode, the model, and the RSS numbers before and after.

Blocked right now, and what I will do about it

The measurement needs the NPU lane and the box has 1.1 GiB available of 128 GiB at the time of filing (MemFree 1.1 GB, mem_info_gtt_used 101 GiB), with two llama-server processes holding GTT outside any gpu-hold ticket (3 h 21 m and 6 h 56 m old) — reported to the owner rather than killed from here. The code change and its host-side tests do not need the device, so they come first; the RSS rows wait for a window.

Refs: engine#881 (the kpad-before-first-dequant ordering this must preserve), #1725 (a gemma-4 embedding component that is f32 and unquantised), #1716 (head packing), #671 (pack loader).

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design decisions after reading the storage layout, recorded here so the next step does not need to re-derive them.

    1. The container is already mapped read-only; nothing new needs mapping. Model::Model (npu/model.cpp) map_file()s model.q4nx and every Tensor::data points into that mapping. So this issue is not "add mmap" — it is "stop copying and stop expanding what is already mapped". That makes the diff much smaller than the framing suggests, and it means the per-layer embedding tables only need the same treatment as the token table.

    2. Both readers are already element-wise accessors, so no call site changes. Q4nxNpuForward::embed_at(token, k) and lm_at(n, k) are the only readers of embed_ / embed_f_ / lm_head_f_, and both are called with k innermost (for (n …) { for (k …) } at the host head and at both tile-packing loops). So a row cache — gather one row, serve H elements from it, move to the next row — is exactly the right granularity and needs no restructuring of the head or the packing loops. That is also what "one row per token" means concretely here.

    3. bf16 and plain I8 need no row buffer at all. bf16: the element is bf16_to_f32(((const uint16_t*)src)[token*H + k]) straight from the mapping. Plain I8 (esz == en, the per-32-column-scale case, which is what the 27B and gemma-4 containers use): one int8 byte × one float scale index. Both are O(1) per element with no buffer, and they cover every embedding dtype currently on this box (the header dump above is BF16 or I8 throughout).

    4. The tiled/chunked quantised case needs one new primitive, and it belongs in the public tree. dequant_any refuses anything that is not a multiple of the tile grid, and the tile layout is public (npu/forward/include/q4nx_dequant.h: Q4NX_TILE_ROWS 32, Q4NX_TILE_COLS 256, Q4NX_TILE_BYTES 5120), so a row is not byte-contiguous: it lives as row r % 32 inside each of cols/256 tiles. The primitive to add is

    int q4nx_dequant_tile_row(const uint8_t* data, size_t nbytes, int rows, int cols,
                              int chunk_bytes, int r, float* out /* cols */);

    which reads only the cols/256 tiles covering row r (20 tile reads at kpad 5120, ~100 KB touched) instead of the 655 KB a 32-row band costs. It is testable host-side against the existing q4nx_dequant_tensor_chunked output with no device and no model, in the same shape as tests/npu_padded_grid_test.cpp.

    5. The tie/lm_head question resolves itself. lm_at falls back to embed_at when the model is tied, so both go through the same row cache; an untied model's separate lm_head_f_ gets the identical treatment. One code path, not two.

    Order of work, smallest first: bf16 + plain I8 (no new primitive, removes the copy for every bf16 model and the 4× expansion for the 27B/gemma-class ones) → the tile-row primitive + chunked embeddings → per-layer tables → RSS rows on three models under one npu-hold ticket each.

    Status right now: blocked on box memory, not on the design. At 23:4x the box had 1.1 GiB available of 128 GiB with mem_info_gtt_used at 101 GiB (two llama-server processes 3 h 21 m and 6 h 56 m old outside any gpu-hold ticket, reported to the owner), and load average 44. A cc1plus cannot run there, so I have not written code I cannot compile and test — the first commit will be the bf16/I8 row cache with its host-side test, as soon as the box has room.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two things to add from the FastFlowLM v1.0.12 digest I did tonight — one is a detail of our code that makes the resident figure worse than the issue's own arithmetic, the other is external corroboration that the fix is safe.

    1. For a quantised container we hold both buffers, not one. embed_.resize((size_t)NV * H) is at :669, before the dtype branch at :684, and the quantised path then adds embed_f_.resize((size_t)NV * kpad_) at :688. So a quantised table costs the bf16 buffer plus the float one:

    shape bf16 buffer float dequant both resident
    0.5B (V 151,936 H 896) 0.25 GiB 0.51 GiB 0.76 GiB
    27B-class (V 151,936 H 5,120) 1.45 GiB 2.90 GiB 4.35 GiB

    The quantised path never reads embed_ (it reads embed_f_ via embed_f32_, :1481), so the first resize is pure waste on every container whose embedding is I8 — which is exactly the 27B's. Moving embed_.resize() into the bf16 branch is a one-line change that removes it; the read-one-row fix then removes the other.

    2. There is already a one-row reader in our own tree. npu/lane.cpp:157:

    const size_t row = size_t(d.hidden) * 2;
    std::memcpy(act->map(), model.tensor("model.embed_tokens.weight").data + size_t(token) * row, row);

    That is the pattern the forward needs — one row per token, straight out of the mapping, no table-sized buffer. The forward's problem is not that row reads are missing from the codebase; it is that the forward chose the whole-table form while the NPU lane path next to it did not.

    3. Vendor corroboration that the fix keeps the numbers. FastFlowLM v1.0.12 (2026-11-05) did exactly this and reports the effect: token and per-layer embedding tables are now read from model.q4nx through a read-only memory map, one row per token, and they state "Output, decode speed and time to first token are unchanged", with private memory going 3.9 GiB → 0.3 GiB on gemma4-it:e4b / nemotron3.5-bolt:e4b (4.2 → 0.5 GiB on the DFlash tag; 0.46/0.92/1.16 GiB saved on qwen3.5 0.8b/2b/4b). Same container format, same access pattern we would use — so the risk of the one-row mmap changing output is a claim a shipping vendor already made against the same shape, which is worth something when we weigh it.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Measured the question the merged PR left open ("no container takes the plain-I8 branch"), with a temporary print in the forward init rather than by reading the heuristic. It is not a bug in the branch condition — the pack is not the dtype its header says.

    [F4DEBUG] etd=model.embed_tokens.weight dtype=I8 data_size=794624000 en=1271398400 NV=248320 H=5120 kpad_=5120 equal=0
    [forward] embedding dtype I8 -> dequantized (1271398400 floats, 4.74 GiB)
    

    en = NV*H = 248320 * 5120 = 1,271,398,400, but the tensor holds 794,624,000 bytes — and 794,624,000 / 5120 = 155,200, which is exactly the row count in the header shape. So the bytes fit two readings at once:

    reading what it means for the same 794,624,000 bytes
    (a) plain int8, [155200, 5120] — what dtype and shape say 794.6 MB / 794.6 M elements = 1 byte per element
    (b) packed tiles, [248320, 5120] on the container's K grid — what the runtime does (248320/32) * (5120/256) = 155,200 tiles of 5,120 B = 0.625 B per element

    dequant_any picks (b): 155,200 tiles × 5,120 B = 794,624,000 bytes exactly, i.e. a 4-bit packed table, which decodes to 1.27 G elements and materialises as 4.74 GiB of f32. The same table is 0.74 GiB in the file — a 6.4× expansion while the model loads.

    Consequences, and they change what the rest of this issue has to do:

    1. The plain-I8 branch is not "unreachable by accident" — it is unreachable for these packs by construction, because a table stored as packed tiles cannot also be 1 byte per element. The branch is for the genuinely-plain Gemma4-class case; no pack on this box is one. Worth a one-line comment on the branch rather than a code change.
    2. The bf16 work that merged (npu: read the token embedding table in place, one row per token, instead of materialising it (engine#1784) #1795) is the small half. The 0.74 GiB → 4.74 GiB case is the tile-packed path, and that is the number the assignment's "3.9 GiB → 0.3 GiB" is about. Row-gather for it means: for row r, read the cols/256 = 20 tiles that cover it (20 × 5,120 B = ~100 KB touched per token) instead of decoding 155,200 tiles.
    3. It also explains why the 27B's before/after RSS was noise in the npu: read the token embedding table in place, one row per token, instead of materialising it (engine#1784) #1795 measurement: both binaries took path (b), unchanged.

    Next commit on this issue is q4nx_dequant_tile_row() in the public tree (npu/forward/src/q4nx_dequant.c, declared beside q4nx_dequant_tensor_chunked), tested host-side against q4nx_dequant_tensor_chunked output the same way tests/npu_dequant_bf16_test.c tests the plain-I8 row read — bit for bit, plus the tile-straddling and last-partial-row arms. Then embed_at() takes the tile-packed branch and the 27B's 4.74 GiB goes away.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status of the two halves the assignment names, with what is measured and what is not.

    Token embedding table: done, measured, merged/queued.

    • #1795 (merged): bf16 and plain-int8 tables read in place, one row per token. Three models, RssAnon after load: 436 → 139 MiB (Qwen3-0.6B), 1669 → 1224 MiB (ZR1-1.5B), 714 → 229 MiB (d3-lite); each delta equals that model's own table (297/297, 445/445, 485/481), output byte-identical on 3 prompts each.
    • #1807 (open, one verdict): the tile-packed path. On the device route the 27B goes 9773 → 2658 MiB RssAnon after load (7.15 GiB returned) with the step-1 vocab-wide float32 logits byte-identical (c93ff216… both arms).

    Per-layer embedding tables: located, costed, and not implementable-as-measured on this box.

    The per-layer path is real and it does materialise: ple_model_proj_ is NL * ple_dim * H floats built at load by transposing each model.per_layer_model_proj.weight_layer{l} from its row-grouped [ple_dim/32, H, 32] bf16 layout into a dense f32 matrix (npu/forward/src/q4nx_forward.cpp, the ple_dim_ > 0 block). Its cost is NL * ple_dim * H * 4 bytes — and it is consumed per layer as a context projection, so "one row per token" here means doing that transpose arithmetic in the accessor instead of in a load-time copy, exactly the shape of the work that landed for the token table.

    What stops the measurement, and it is not the code: no container on this box has these tensors at all. Scanning every model.q4nx here for per_layer_model_proj: 0 hits. The only gemma-family thing on disk is ~/scratch-packs/gemma3-270m-npu2, which is not a converted container and is not a per-layer-embedding model. So there is nothing to measure on, and the plan's rule is measured numbers only — I am not going to land a change here and quote a formula as if it were a result.

    Order I would do it in when a gemma-4-class pack exists (W4 owns those packs, #1725 is the neighbouring embedding defect): import one, record RssAnon before/after the change on the same three prompts, and confirm the per-layer context projection is byte-identical by the same logits comparison used for the 27B. The code change itself is ~40 lines in the accessor plus one test arm on the transpose arithmetic ([a*32+g][c] from [a][c][g]), which is the part that can be wrong while looking right.

    If the owner wants this half done before a pack exists, say so and I will land the accessor change behind the unit test and mark the row unmeasured — but that is a weaker deliverable than the token half and I would rather it not be counted as F4 done.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Status at main ee5b436, with one new measurement that changes the "per-layer tables" half (lane cl-npurt).

    What is on main: #1795 (bf16 and plain-int8 tables read in place), #1807 (tile-packed tables, one row per token; 27B RssAnon after load 9773 → 2658 MiB, step-1 logits byte-identical), #1871 (the per-row path is chosen by the row reader's own rule). The tied head needs no table either: lm_at() falls through to embed_at().

    Item 4, per-layer embedding tables — already not materialised. The status comment above looked for per_layer_model_proj and found no pack. A gemma-4 pack is on the box now (gemma4-e2b-npu2, model.q4nx 4,671,875,302 B), and its header shows which tensor is the table:

    tensor dtype / shape on disk as f32
    model.per_layer_token_embd.weight I8 [262144, 8960] 2.35 GB 9.4 GB
    model.embed_tokens.weight I8 [262144, 1536] 0.40 GB 1.5 GiB
    model.per_layer_model_proj.weight_layer{0..34} BF16 [8, 1536, 32] each 27.5 MB total 55 MB (ple_model_proj_)

    The per-layer token table is read by load_ple_block(): one 32-row band dequantised from the mapping per lookup (32 × 8960 floats), never the table. ple_model_proj_ is a 256 × 1536 projection per layer, 55 MB in all, not a vocabulary-sized table.

    Measured, 1bit forward-serve built from ee5b436, NPU_INFER_HOST_GEMM=1 (no device opened), /proc/<pid>/status:

    [forward] embedding dtype I8 -> read in place from the container, one row per token (engine#1784): 1.50 GiB not materialised
    after load:        RssAnon 549,560 kB  (537 MiB)
    after one request: RssAnon 704,104 kB  (688 MiB)
    

    So a container whose two tables would be 10.9 GiB as floats serves in under 0.7 GiB of private memory. (This row is about memory only: it ran in host mode, and I did not judge the text it produced.)

    What is still open against this issue's own gate: the no decode-speed regression row — tok/s before and after on the same model in one ticket — is not in #1795 or #1807. It needs a pre-#1795 build next to a current one and one 27B-class ticket. I am leaving the issue open for that single row rather than closing on memory alone.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The missing gate row: decode speed before and after, 27B, one ticket (lane cl-npurt).

    Two builds side by side, same compiler line and NPU runtime: before = d805b90^ (the parent of #1795), after = main 8f10bfb plus #1970 (which does not touch decode). Model bonsai2-27b-NPU, 1bit forward-serve on the device, one npu-hold ticket (967 s), serves interleaved before / after / before / after, fresh process each time, two requests per serve ("The capital of France is", 6 tokens, temperature 0). Numbers are the serve's own timings.predicted_ms over completion_tokens.

    serve run 1 ms/token run 2 ms/token text
    before 1 7150.8 7907.1 Paris.\nThe capital of
    after 1 7669.7 7595.1 Paris.\nThe capital of
    before 2 8169.4 8043.9 Paris.\nThe capital of
    after 2 7482.7 7155.5 Paris.\nThe capital of
    • before: mean 7818 ms/token, range 7151–8169 (the control pair alone spans 14 %)
    • after: mean 7476 ms/token, range 7156–7670

    Every "after" run sits inside the spread of the "before" runs, so there is no decode-speed regression from reading the embedding row at use; the text is the same in all eight runs. The after build logs embedding dtype I8 -> read in place from the container, one row per token (engine#1784): 4.74 GiB not materialised.

    Limits, stated plainly: 6-token runs (this pack decodes at about 7–8 s per token on this route, so longer runs do not fit a ticket), a busy box (the ticket queued 416 s behind another lane), and the noise floor above is wide. The row can rule out a regression larger than that floor, not a 1–2 % one. RssAnon readings from this ticket are not usable (the box was reclaiming memory between serves: the same "before" binary read 10.2 GB and then 4.1 GB after load); the memory rows are the ones already recorded in #1807 and in my comment above.

    With this, every line of the issue's gate has a recorded row: identical output and RSS in #1795/#1807, per-layer tables in the comment above, decode speed here. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions