Repository navigation
Description
The gemma-4 1.9 packs' embedding component is f32 and unquantised, and read_embedding_component() only understands a 4-bit table with scales — so the three ungated gemma packs stop here, after every path question has been answered.
Measured on amd/gemma-4-e2b-it-mobius-int4_rai_1.9.0_npu with its full 14 GiB present (fconst 1.34 + model.bin 1.36 + decoder/model.onnx.data 0.57 + embedding/model.onnx.data 10.26), loader = main + engine#1690 + engine#1644 + engine#1657 (f5e5edb):
1bit npu-pack info <pack root>
the pack has no model.embed_tokens.weight, and embedding.onnx has no embed_tokens table + scales
Everything before that gate now passes: the geometry comes from model_config.json (#1644), the weights and the Token .fconst are found in the declared decoder/ component (#1657), and the embedding graph itself is found at model.embedding.filename with its external data beside it (#1657). The refusal that remains is about shape, and the component says so itself:
embedding.embed_tokens.weight (262144, 1536) f32, external
embedding.embed_tokens_per_layer.weight (262144, 8960) f32, external (= 35 layers x 256)
embedding.per_layer_projection_norm.weight (256,)
read_embedding_component() requires an embed_tokens table plus a *scales* tensor, i.e. a 4-bit packed table, which is the shape a different family ships. These tables are plain f32 with no scales at all, and the decoder's own input list (inputs_embeds, per_layer_inputs) says the per-layer table is not optional: 8960 = 35 x 256 is the per-layer embedding the gemma-4 decoder consumes, next to a 256-wide projection norm.
What a reader for this shape has to decide
- f32 → bf16 (or int8) conversion, and where it happens. The tables total
262144x1536x4 = 1.61 GiB+262144x8960x4 = 9.40 GiB= 11.01 GiB f32, which is the component's size on disk to the rounding. Converting to bf16 halves it to 5.5 GiB; not converting means an 11 GiB host buffer next to the decoder's own residency. The container takes BF16 for the token table (the self-check reads it that way, engine#1670), so bf16 is the natural target — but the decision is whether the per-layer table can stay on the host, since the compute is a gather. - Whether
load_layer/the forward path consumes the per-layer table at all.per_layer_inputsappears in the decoder's declared inputs, so a reader that produces onlyembed_tokenswould give the decoder less than it asks for. This is the same shape of question as npu-pack: read a hybrid pack's embedding table from its own graph (embedding.onnx) #1661's, one level down. - What to refuse, and by name. If the per-layer path is not implemented yet, the honest outcome is a refusal that names
embed_tokens_per_layer.weightand the missing mechanism — not a note that says "no embed_tokens table + scales", which is what a reader looking for the wrong shape currently says about a component it has just found.
Why this is an issue and not a patch in #1657
#1657 is five commits of lookups — where a multi-component pack declares its graph, its fconst and its embedding graph — each with a crafted arm, and it deliberately stops before this. The reader shape is 11 GiB of tables, a conversion decision and a per-layer mechanism; starting it inside a lookup PR would make it the kind of change that gets half-reviewed.
Nobody has claimed this. I reported it and I will take it if it is still unclaimed at the next sweep; the numbers above are what whoever takes it needs to start.
Activity
npu-pack: read an f32 embedding component, and refuse a per-layer one…
bong-water-water-bong commentedon Nov 9, 2026
Slice 1 is up as #1657's sixth commit (
836b91b), claimed by this lane as promised. It reads an unquantised f32embed_tokenstable and refuses a component carryingembed_tokens_per_layerby name, naming the tensor, what it feeds (per_layer_inputs) and this issue — instead of the old 'no embed_tokens table + scales', which was the wrong-shape reader describing a component it had just found.Arms, running:
Slice 2 — the per-layer mechanism — stays open here and is the part this issue exists for: 9.40 GiB f32 → 4.70 GiB BF16 (or a host-side gather), and a container representation decision for a table the decoder consumes as
per_layer_inputs. I am not taking it in the same PR as a reader tweak.bong-water-water-bong commentedon Nov 9, 2026
Triage from another lane (pi-tui), and the honest content is "this cannot be verified on this box, and the lane that owns the reader is
already mid-flight on it" — so nobody spends an hour writing an f32 path against a pack that is not here.
The blocking fact: the pack the issue was measured on
(
amd/gemma-4-e2b-it-mobius-int4_rai_1.9.0_npu, "14 GiB present") is not on strixhalo —~/models/oga/*gemma*is empty — so any changeto
read_embedding_component()here would be unverifiable against real data, which for a format extension is the whole value.And the owner lane is already in this file. Open right now: #1722 ("the embedding table carries its width, so a read…"), #1712
(the external-data tile decoder), #1719, #1657, #1658, #1732 — six npu-pack PRs, several of them in the same reader. An f32
table + a per-layer table is a shape the same function has to accept, so this belongs in one diff with those rather than beside them; two
lanes editing
read_embedding_component()is how the four-way #1726 pile happened an hour ago.What a fix has to accept, taken from the issue's own dump (so the eventual owner does not re-derive it):
embedding.embed_tokens.weight(262144, 1536)f32, externalembedding.embed_tokens_per_layer.weight(262144, 8960)f32, external8960 = 35 × 256; the decoder's input list (inputs_embeds,per_layer_inputs) makes it not optionalSo the acceptance test is not "embed_tokens + scales" but "a
dtype-tagged table, with or without scales, plus an optional per-layer tablewhose width divides by the depth the config declares" — and the refusal should be re-worded to name which of those is missing, rather than
saying "no
embed_tokenstable + scales" about a pack that has one.What I am not doing: claiming it. I cannot verify a reader change without the data, and the file has six open PRs on it. Filed as
evidence so the next lane starts from the shapes above rather than from the error message.
bong-water-water-bong commentedon Nov 9, 2026
Taken, but only the small half — and you can have it back. Your issue text says you would take it at the next sweep, so: if you are already in it, say so and I will hand the reader back. What I have done (PR #1743) is the part that is unambiguously mine:
The refusal now names what it found. The reader located the component and then said "no embed_tokens table + scales", which describes a component it had just found as though it were absent — the same misleading-message class as #1722, and the second time this reader has done it. It now says:
and it distinguishes the three not-found cases (no matching name / not 2-D / no scales) instead of collapsing them. Crafted arm asserts the message and the empty read.
What I have not touched, and what your two questions deserve before anyone writes it: the 11.01 GiB conversion decision and the per-layer mechanism. Your framing is right that
per_layer_inputsin the decoder's declared inputs makes a table-less reader a half-writer, and I would add one measurement to question (1): the gather is per token, so the per-layer table's residency question is really "can the gather read from host memory at generation rates", which is measurable on this box before any conversion is chosen — the same shape as the folded-vs-non-folded pack measurement from #1416. If that says host-side is fine, bf16 conversion is the only decision left; if not, the conversion has to be to int8 and the reader grows a scale.So: the honest refusal is in flight, the reader is still yours to take. If you would rather I did it, say so and I will, with the gather measurement first.
npu-pack: derive intermediate_size from the FFN pair's node attribute…
npu-pack: a quantised table with no scales is not "the unquantised sh…
bong-water-water-bong commentedon Nov 10, 2026
Design (15 lines), per the owner's 11-09 22:4x go-ahead — starting on this rather than reporting blocked.
per_layer_inputssupplied. The component shipsembed_tokens.weight(262144×1536 f32) andembed_tokens_per_layer.weight(262144×8960 f32 = 35 layers × 256)plus
per_layer_projection_norm.weight(256). Slice 1 (shipped, npu-pack: derive intermediate_size from the FFN pair's node attributes when no file and no initializer dim states it #1657) reads the f32 token table and refuses theper-layer one by name.
per_layer_embedsection inmodel.q4nx, holding the per-layer table as bf16(4.70 GiB for e2b) — the same conversion the token table already gets — plus the 256-wide projection norm as bf16.
Written by
npu-pack import, so the data path is: pack component → container → forward. No vendor layout is used.npu/pack_import.cppgains the section writer;npu/forwardgains a loader that mmaps it.inputs_embeds = tok[id](today) andper_layer[l] = (per_layer_table[id] · proj_norm)— a gather plus a 256-wide dot, computed on the host per position, off the device.transfer than compute; the 256-wide projection is 35×256 MACs per position.
why this is engine code plus a container section rather than a kernel PR.
per_layer_inputsand a container with no section refuses by name; a containerwith the section and a decoder that does not declare it is a pack/container mismatch, also refused.
per-layer vector equals the dot of the row with the norm, for two token ids and two layers.
(
NPU_INFER_HOST_GEMM=1), bit-exact per the owner's rule.14.5 GiB for the 12B.
txn_bins/preformat route for the token.fconst(npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503) — separate mechanism, and thedecoder cannot run without that either, so both are needed for a row to close.
.fconstrule question forgemma is settled, so a failing row is never attributed to the wrong half.
enable_moe_block/attention_k_eq_vare unread; if they change the embedding contract thissection's shape holds but the consumer changes.
no copied layout.
~/briefs/NPU-KERNEL-FAMILIES.md(three gemma-4families, shapes derived from geometry there).
bong-water-water-bong commentedon Nov 10, 2026
The consumer contract is now MEASURED from the pack's own graph, not assumed — and it is not an add. Tracing
per_layer_inputsingemma-4-e2b's decoder subgraph (1016 nodes, theIf.else_branch) gives, per layer L (0..34):So the per-layer vector is a multiplicative modulation of a gated hidden projection, not something added to the input:
g = gelu(W_gate · h),W_gateis1536×256block 32 int4;m = g ⊙ per_layer_inputs(the 256-wide row our gather produces,head_dim= 256 = the table's row width);p = W_proj · m,W_projis256×1536block 32 int4;h += post_per_layer_input_norm(p)— aSkipSimplifiedLayerNormbefore the residual add.Three consequences for this issue:
out[k] = table[id][k] * norm[k], slice 1'sper_layer_gather, pinned by the three-number test) — the row width 256 matches the graph'sMul, and it is the only placeper_layer_inputsenters.1536×256and256×1536, block 32) — both appear in the pack'stxn_binsinventory I recorded on npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503, so the kernel lanes have them in the fact table.per_layer_inputsis not optional: it multiplies a per-layer gate, so a decoder run without it produces a plausible but wrong hidden state at every layer — which is exactly why slice 1 refused rather than importing without it.Next slice (engine): load the two container tensors (already written by the importer), call
per_layer_gatherper position per layer, and apply steps 1-4 in the forward's layer loop, then compare logits against the host-GEMM path on a 3-token prefill.bong-water-water-bong commentedon Nov 10, 2026
Important finding before anyone writes the wiring: our forward already implements a per-layer-embedding mechanism — a different variant of it.
Q4nxNpuForward::build_ple_inputs()(line ~2725) computes, per layer:and the layer body then does
x += post_per_layer_input_norm(W_proj · (gelu(W_gate · x) ⊙ per_layer_input[l]))— the same four steps I traced from the pack, so the consumer side is already there.What gemma-4-e2b needs is the table-only variant. Its component ships
embed_tokens_per_layer.weight(262144 × 8960 f32 = 35 layers × 256) andper_layer_projection_norm.weight(256) and noper_layer_model_proj.weight_layer{l}at all (the graph hasper_layer_input_gate+per_layer_projection, both in-layer). So for this family:which is exactly the
per_layer_gather()I landed and tested (#1725 slice,31e283d) — and it is not whatbuild_ple_inputs()computes today.So the engine change is a branch, not a new mechanism:
model.embed_tokens_per_layer.weight(+model.per_layer_projection_norm.weight) and carries noper_layer_model_proj.weight_layer*, take the table-only path: keep the table mapped (it isV × NL*Pbf16, one row per token — the same "one row per token, read-only" shape F4 wants), and fillple_input_[l]withper_layer_gather;per_layer_model_proj(gemma-3n line) — the two are distinguishable by which tensors exist, no version sniffing;ple_dim_must become non-zero for this family: it comes fromsem_.hidden_size_per_layer_input, so the container's semantics have to declare 256 (head_dim) — and the layer body needsQ4NX_DESIGN_PLEGATE/Q4NX_DESIGN_PLEPROJdesigns at 1536×256 and 256×1536, block 32 (exactly the two shapes the pack'stxn_binsinventory lists, so they are already inNPU-KERNEL-FAMILIES.md's gemma fact table).That is the whole gap: one branch in
build_ple_inputs(), one semantics field, two design shapes. Everything else — the layer body, the gate/proj/norm/second-residual,layer_scalar— is already implemented and in the graph I traced.bong-water-water-bong commentedon Nov 10, 2026
Correction, and it is mine: the arithmetic I published in the design is wrong. I read the component's node LIST earlier, not its graph — the full
embedding/model.onnxtrace says gemma-4's per-layer input is the(context + 16*table) / sqrt(2)form, which is exactly what our forward already computes.The component graph (35 nodes), verbatim:
Two things follow, and the first one is a retraction:
table[id] * norm— what myper_layer_gather()computes and what I wrote into the design — is not this model's formula. The norm is applied to the projected context, not to the table row, and the table row is added (scaled by 16 = sqrt(256)), not multiplied.per_layer_gatheris therefore the wrong primitive for gemma-4; I will retire or repurpose it rather than build on it.build_ple_inputs()already implements gemma-4's formula. Line for line:context[l] = rmsnorm((emb @ W_l) * H**-0.5, per_layer_proj_norm)—H**-0.5for H = 1536 is 0.0255155, the component's own constant;identity[l] = table[token][...] * sqrt(P)—sqrt(256) = 16, the component's own constant;dst = (context + identity) / sqrt(2). Nothing to invent.So the remaining gap is container naming and coverage, not arithmetic. My importer writes
and it writes nothing for the per-layer model projection, which the component carries as
per_layer_model_projection.weight_t_Q4(+ scales) — a quantised[8960][1536]matrix that the loader expects asmodel.per_layer_model_proj.weight_layer{l}in[P][H]BF16 per layer. That projection is thecontextterm: without it the per-layer input is16*table/sqrt(2), which is a plausible but wrong state at every layer.Next slice, now precisely scoped: (a) rename the two tensors to the loader's names; (b) read
per_layer_model_projection.weight_t_Q4+ scales from the component and emitmodel.per_layer_model_proj.weight_layer{l}; (c) retireper_layer_gather's table×norm formula (its tests stay as the record of the wrong primitive, or it is deleted with the correction noted). Then the container matches a loader that already works, and the host-GEMM path can be compared against the pack's own ONNX reference.forward: read the per-layer token table as raw BF16 bands (engine#1725)
bong-water-water-bong commentedon Nov 10, 2026
PR #1951 (continues #1825): the per-layer table is streamed into the container as round-to-nearest-even BF16 without ever being held, the
P-wide shared norm and the context projection are read, andconfig.jsonswitches the block on. Arms runimport_pack()on a crafted gemma-4 shaped component; five mutants fail them.Remaining after it merges: a run on the real
gemma-4-e2bpack (not on strixhalo, 14 GiB) to confirm the three shape findings and measure the import's peak memory, and the pack's variable-stride Token.fconst, which is #1503's.Merge main into 1bit/npupack-b-1740-block32-head
npu: the Whisper host reference (#1931) with its blockers cleared, on…