Skip to content

npu-pack: the gemma-4 embedding component is f32 and unquantised, so the 1.9 packs stop at the reader's shape (4-bit table + scales) #1725

Closed

Description

@bong-water-water-bong

The gemma-4 1.9 packs' embedding component is f32 and unquantised, and read_embedding_component() only understands a 4-bit table with scales — so the three ungated gemma packs stop here, after every path question has been answered.

Measured on amd/gemma-4-e2b-it-mobius-int4_rai_1.9.0_npu with its full 14 GiB present (fconst 1.34 + model.bin 1.36 + decoder/model.onnx.data 0.57 + embedding/model.onnx.data 10.26), loader = main + engine#1690 + engine#1644 + engine#1657 (f5e5edb):

1bit npu-pack info <pack root>
the pack has no model.embed_tokens.weight, and embedding.onnx has no embed_tokens table + scales

Everything before that gate now passes: the geometry comes from model_config.json (#1644), the weights and the Token .fconst are found in the declared decoder/ component (#1657), and the embedding graph itself is found at model.embedding.filename with its external data beside it (#1657). The refusal that remains is about shape, and the component says so itself:

embedding.embed_tokens.weight              (262144, 1536)  f32, external
embedding.embed_tokens_per_layer.weight    (262144, 8960)  f32, external   (= 35 layers x 256)
embedding.per_layer_projection_norm.weight (256,)

read_embedding_component() requires an embed_tokens table plus a *scales* tensor, i.e. a 4-bit packed table, which is the shape a different family ships. These tables are plain f32 with no scales at all, and the decoder's own input list (inputs_embeds, per_layer_inputs) says the per-layer table is not optional: 8960 = 35 x 256 is the per-layer embedding the gemma-4 decoder consumes, next to a 256-wide projection norm.

What a reader for this shape has to decide

  1. f32 → bf16 (or int8) conversion, and where it happens. The tables total 262144x1536x4 = 1.61 GiB + 262144x8960x4 = 9.40 GiB = 11.01 GiB f32, which is the component's size on disk to the rounding. Converting to bf16 halves it to 5.5 GiB; not converting means an 11 GiB host buffer next to the decoder's own residency. The container takes BF16 for the token table (the self-check reads it that way, engine#1670), so bf16 is the natural target — but the decision is whether the per-layer table can stay on the host, since the compute is a gather.
  2. Whether load_layer/the forward path consumes the per-layer table at all. per_layer_inputs appears in the decoder's declared inputs, so a reader that produces only embed_tokens would give the decoder less than it asks for. This is the same shape of question as npu-pack: read a hybrid pack's embedding table from its own graph (embedding.onnx) #1661's, one level down.
  3. What to refuse, and by name. If the per-layer path is not implemented yet, the honest outcome is a refusal that names embed_tokens_per_layer.weight and the missing mechanism — not a note that says "no embed_tokens table + scales", which is what a reader looking for the wrong shape currently says about a component it has just found.

Why this is an issue and not a patch in #1657

#1657 is five commits of lookups — where a multi-component pack declares its graph, its fconst and its embedding graph — each with a crafted arm, and it deliberately stops before this. The reader shape is 11 GiB of tables, a conversion decision and a per-layer mechanism; starting it inside a lookup PR would make it the kind of change that gets half-reviewed.

Nobody has claimed this. I reported it and I will take it if it is still unclaimed at the next sweep; the numbers above are what whoever takes it needs to start.

Activity

  1. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Slice 1 is up as #1657's sixth commit (836b91b), claimed by this lane as promised. It reads an unquantised f32 embed_tokens table and refuses a component carrying embed_tokens_per_layer by name, naming the tensor, what it feeds (per_layer_inputs) and this issue — instead of the old 'no embed_tokens table + scales', which was the wrong-shape reader describing a component it had just found.

    Arms, running:

    f32 embedding: 2048 values (why: )
    f32 embedding (per-layer): 0 values, why: the component carries embedding.embed_tokens_per_layer.weight …
    

    Slice 2 — the per-layer mechanism — stays open here and is the part this issue exists for: 9.40 GiB f32 → 4.70 GiB BF16 (or a host-side gather), and a container representation decision for a table the decoder consumes as per_layer_inputs. I am not taking it in the same PR as a reader tweak.

  2. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Triage from another lane (pi-tui), and the honest content is "this cannot be verified on this box, and the lane that owns the reader is
    already mid-flight on it"
    — so nobody spends an hour writing an f32 path against a pack that is not here.

    The blocking fact: the pack the issue was measured on
    (amd/gemma-4-e2b-it-mobius-int4_rai_1.9.0_npu, "14 GiB present") is not on strixhalo — ~/models/oga/*gemma* is empty — so any change
    to read_embedding_component() here would be unverifiable against real data, which for a format extension is the whole value.

    And the owner lane is already in this file. Open right now: #1722 ("the embedding table carries its width, so a read…"), #1712
    (the external-data tile decoder), #1719, #1657, #1658, #1732 — six npu-pack PRs, several of them in the same reader. An f32
    table + a per-layer table is a shape the same function has to accept, so this belongs in one diff with those rather than beside them; two
    lanes editing read_embedding_component() is how the four-way #1726 pile happened an hour ago.

    What a fix has to accept, taken from the issue's own dump (so the eventual owner does not re-derive it):

    tensor shape note
    embedding.embed_tokens.weight (262144, 1536) f32, external plain table, no scales — the refusal's premise is a 4-bit shape this family does not ship
    embedding.embed_tokens_per_layer.weight (262144, 8960) f32, external 8960 = 35 × 256; the decoder's input list (inputs_embeds, per_layer_inputs) makes it not optional

    So the acceptance test is not "embed_tokens + scales" but "a dtype-tagged table, with or without scales, plus an optional per-layer table
    whose width divides by the depth the config declares" — and the refusal should be re-worded to name which of those is missing, rather than
    saying "no embed_tokens table + scales" about a pack that has one.

    What I am not doing: claiming it. I cannot verify a reader change without the data, and the file has six open PRs on it. Filed as
    evidence so the next lane starts from the shapes above rather than from the error message.

  3. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Taken, but only the small half — and you can have it back. Your issue text says you would take it at the next sweep, so: if you are already in it, say so and I will hand the reader back. What I have done (PR #1743) is the part that is unambiguously mine:

    The refusal now names what it found. The reader located the component and then said "no embed_tokens table + scales", which describes a component it had just found as though it were absent — the same misleading-message class as #1722, and the second time this reader has done it. It now says:

    embedding.onnx's embedding.embed_tokens.weight is 8 x 16 f32 with no *scales* tensor: that is the UNQUANTISED shape
    (gemma-4's), which this reader does not implement yet (engine#1725), not a component that is missing
    

    and it distinguishes the three not-found cases (no matching name / not 2-D / no scales) instead of collapsing them. Crafted arm asserts the message and the empty read.

    What I have not touched, and what your two questions deserve before anyone writes it: the 11.01 GiB conversion decision and the per-layer mechanism. Your framing is right that per_layer_inputs in the decoder's declared inputs makes a table-less reader a half-writer, and I would add one measurement to question (1): the gather is per token, so the per-layer table's residency question is really "can the gather read from host memory at generation rates", which is measurable on this box before any conversion is chosen — the same shape as the folded-vs-non-folded pack measurement from #1416. If that says host-side is fine, bf16 conversion is the only decision left; if not, the conversion has to be to int8 and the reader grows a scale.

    So: the honest refusal is in flight, the reader is still yours to take. If you would rather I did it, say so and I will, with the gather measurement first.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Design (15 lines), per the owner's 11-09 22:4x go-ahead — starting on this rather than reporting blocked.

    1. Goal: make the gemma-4 1.9 decoders runnable, i.e. their per_layer_inputs supplied. The component ships
      embed_tokens.weight (262144×1536 f32) and embed_tokens_per_layer.weight (262144×8960 f32 = 35 layers × 256)
      plus per_layer_projection_norm.weight (256). Slice 1 (shipped, npu-pack: derive intermediate_size from the FFN pair's node attributes when no file and no initializer dim states it #1657) reads the f32 token table and refuses the
      per-layer one by name.
    2. Container section: a new per_layer_embed section in model.q4nx, holding the per-layer table as bf16
      (4.70 GiB for e2b) — the same conversion the token table already gets — plus the 256-wide projection norm as bf16.
      Written by npu-pack import, so the data path is: pack component → container → forward. No vendor layout is used.
    3. Engine reader: npu/pack_import.cpp gains the section writer; npu/forward gains a loader that mmaps it.
    4. Forward path: the embedding step becomes inputs_embeds = tok[id] (today) and per_layer[l] = (per_layer_table[id] · proj_norm) — a gather plus a 256-wide dot, computed on the host per position, off the device.
    5. Why host-side: the gather is O(1) per token and the table is 4.7 GiB, so putting it on the NPU would cost more
      transfer than compute; the 256-wide projection is 35×256 MACs per position.
    6. Operator surface: nothing new on the device — no new kernel family is required for the embedding path, which is
      why this is engine code plus a container section rather than a kernel PR.
    7. Refusal: a decoder that declares per_layer_inputs and a container with no section refuses by name; a container
      with the section and a decoder that does not declare it is a pack/container mismatch, also refused.
    8. Test (host, no device): crafted component with a known table → import → re-read the section → assert the gathered
      per-layer vector equals the dot of the row with the norm, for two token ids and two layers.
    9. Test (device, npu-hold): a 3-token prefill whose logits are compared against the same pack on the host-GEMM path
      (NPU_INFER_HOST_GEMM=1), bit-exact per the owner's rule.
    10. Size gate: import refuses unless free disk ≥ section size × 1.2, because the section is 4.7 GiB for e2b and
      14.5 GiB for the 12B.
    11. Not in this slice: the txn_bins/preformat route for the token .fconst (npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503) — separate mechanism, and the
      decoder cannot run without that either, so both are needed for a row to close.
    12. Order: section + reader + host test first (this PR), then the device arm once the .fconst rule question for
      gemma is settled, so a failing row is never attributed to the wrong half.
    13. Risk: the 12B's enable_moe_block / attention_k_eq_v are unread; if they change the embedding contract this
      section's shape holds but the consumer changes.
    14. Own code only: the only reference used is the pack's own ONNX graph (names, dims, dtypes) — no vendor binary,
      no copied layout.
    15. Tracking: this issue; the kernel-family side is claimed in ~/briefs/NPU-KERNEL-FAMILIES.md (three gemma-4
      families, shapes derived from geometry there).
  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The consumer contract is now MEASURED from the pack's own graph, not assumed — and it is not an add. Tracing per_layer_inputs in gemma-4-e2b's decoder subgraph (1016 nodes, the If.else_branch) gives, per layer L (0..34):

    per_layer_inputs ──Reshape──┐
    hidden h ──MatMulNBitsBf K=1536 N=256 block=32 bits=4──► Activation(Gelu) ──► Mul ──► MatMulNBitsBf K=256 N=1536 block=32 ──►
                                                                           (with the per-layer row)                    SLRN post_per_layer_input_norm
                                                                                                                                │
                                                                                        layer stream h ────────────────► Add ◄───┘
    

    So the per-layer vector is a multiplicative modulation of a gated hidden projection, not something added to the input:

    1. g = gelu(W_gate · h), W_gate is 1536×256 block 32 int4;
    2. m = g ⊙ per_layer_inputs (the 256-wide row our gather produces, head_dim = 256 = the table's row width);
    3. p = W_proj · m, W_proj is 256×1536 block 32 int4;
    4. h += post_per_layer_input_norm(p) — a SkipSimplifiedLayerNorm before the residual add.

    Three consequences for this issue:

    • The gather's arithmetic is right (out[k] = table[id][k] * norm[k], slice 1's per_layer_gather, pinned by the three-number test) — the row width 256 matches the graph's Mul, and it is the only place per_layer_inputs enters.
    • The wiring needs two weight matmuls per layer that are already in the decoder's own graph (1536×256 and 256×1536, block 32) — both appear in the pack's txn_bins inventory I recorded on npu-pack: the 1.9 packs split on the MatMulNBits K block size (32, and mixed 32/128), before the hidden-896 limit is reached #1503, so the kernel lanes have them in the fact table.
    • per_layer_inputs is not optional: it multiplies a per-layer gate, so a decoder run without it produces a plausible but wrong hidden state at every layer — which is exactly why slice 1 refused rather than importing without it.

    Next slice (engine): load the two container tensors (already written by the importer), call per_layer_gather per position per layer, and apply steps 1-4 in the forward's layer loop, then compare logits against the host-GEMM path on a 3-token prefill.

  6. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Important finding before anyone writes the wiring: our forward already implements a per-layer-embedding mechanism — a different variant of it. Q4nxNpuForward::build_ple_inputs() (line ~2725) computes, per layer:

    context[l]        = rmsnorm( (emb @ W_l) * H**-0.5, per_layer_proj_norm )      W_l = model.per_layer_model_proj.weight_layer{l}   [P][H]
    token_identity[l] = table[token][l*P:(l+1)*P] * sqrt(P)                        table = ple_block_ (loaded per token)
    per_layer_input[l]= (context[l] + token_identity[l]) / sqrt(2)
    

    and the layer body then does x += post_per_layer_input_norm(W_proj · (gelu(W_gate · x) ⊙ per_layer_input[l])) — the same four steps I traced from the pack, so the consumer side is already there.

    What gemma-4-e2b needs is the table-only variant. Its component ships embed_tokens_per_layer.weight (262144 × 8960 f32 = 35 layers × 256) and per_layer_projection_norm.weight (256) and no per_layer_model_proj.weight_layer{l} at all (the graph has per_layer_input_gate + per_layer_projection, both in-layer). So for this family:

    per_layer_input[l][i] = table[token][l*P + i] * proj_norm[l*P + i]       (P = 256 = head_dim; no context term, no 1/sqrt(2))
    

    which is exactly the per_layer_gather() I landed and tested (#1725 slice, 31e283d) — and it is not what build_ple_inputs() computes today.

    So the engine change is a branch, not a new mechanism:

    1. at init, if the container carries model.embed_tokens_per_layer.weight (+ model.per_layer_projection_norm.weight) and carries no per_layer_model_proj.weight_layer*, take the table-only path: keep the table mapped (it is V × NL*P bf16, one row per token — the same "one row per token, read-only" shape F4 wants), and fill ple_input_[l] with per_layer_gather;
    2. keep the existing projection variant untouched for the packs that ship per_layer_model_proj (gemma-3n line) — the two are distinguishable by which tensors exist, no version sniffing;
    3. ple_dim_ must become non-zero for this family: it comes from sem_.hidden_size_per_layer_input, so the container's semantics have to declare 256 (head_dim) — and the layer body needs Q4NX_DESIGN_PLEGATE / Q4NX_DESIGN_PLEPROJ designs at 1536×256 and 256×1536, block 32 (exactly the two shapes the pack's txn_bins inventory lists, so they are already in NPU-KERNEL-FAMILIES.md's gemma fact table).

    That is the whole gap: one branch in build_ple_inputs(), one semantics field, two design shapes. Everything else — the layer body, the gate/proj/norm/second-residual, layer_scalar — is already implemented and in the graph I traced.

  7. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction, and it is mine: the arithmetic I published in the design is wrong. I read the component's node LIST earlier, not its graph — the full embedding/model.onnx trace says gemma-4's per-layer input is the (context + 16*table) / sqrt(2) form, which is exactly what our forward already computes.

    The component graph (35 nodes), verbatim:

    context   = RMSNormalization( (inputs_embeds @ per_layer_model_projection.weight_t_Q4) * 0.02551551815399144,
                                  per_layer_projection_norm.weight )                      # 8960 wide, normalised AFTER the projection
    identity  = Gather(embed_tokens_per_layer.weight, ids) * 16.0
    per_layer_inputs = (context + identity) * 0.7071067811865476                              # = (context + identity)/sqrt(2)
    

    Two things follow, and the first one is a retraction:

    1. table[id] * norm — what my per_layer_gather() computes and what I wrote into the design — is not this model's formula. The norm is applied to the projected context, not to the table row, and the table row is added (scaled by 16 = sqrt(256)), not multiplied. per_layer_gather is therefore the wrong primitive for gemma-4; I will retire or repurpose it rather than build on it.
    2. The forward's existing build_ple_inputs() already implements gemma-4's formula. Line for line: context[l] = rmsnorm((emb @ W_l) * H**-0.5, per_layer_proj_norm) — H**-0.5 for H = 1536 is 0.0255155, the component's own constant; identity[l] = table[token][...] * sqrt(P) — sqrt(256) = 16, the component's own constant; dst = (context + identity) / sqrt(2). Nothing to invent.

    So the remaining gap is container naming and coverage, not arithmetic. My importer writes

    model.embed_tokens_per_layer.weight   BF16 {V, 8960}       ← the loader reads model.per_layer_token_embd.weight (+ .scale)
    model.per_layer_projection_norm.weight BF16 {8960}         ← the loader reads model.per_layer_proj_norm.weight
    

    and it writes nothing for the per-layer model projection, which the component carries as per_layer_model_projection.weight_t_Q4 (+ scales) — a quantised [8960][1536] matrix that the loader expects as model.per_layer_model_proj.weight_layer{l} in [P][H] BF16 per layer. That projection is the context term: without it the per-layer input is 16*table/sqrt(2), which is a plausible but wrong state at every layer.

    Next slice, now precisely scoped: (a) rename the two tensors to the loader's names; (b) read per_layer_model_projection.weight_t_Q4 + scales from the component and emit model.per_layer_model_proj.weight_layer{l}; (c) retire per_layer_gather's table×norm formula (its tests stay as the record of the wrong primitive, or it is deleted with the correction noted). Then the container matches a loader that already works, and the host-GEMM path can be compared against the pack's own ONNX reference.

  8. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    PR #1951 (continues #1825): the per-layer table is streamed into the container as round-to-nearest-even BF16 without ever being held, the P-wide shared norm and the context projection are read, and config.json switches the block on. Arms run import_pack() on a crafted gemma-4 shaped component; five mutants fail them.

    Remaining after it merges: a run on the real gemma-4-e2b pack (not on strixhalo, 14 GiB) to confirm the three shape findings and measure the import's peak memory, and the pack's variable-stride Token .fconst, which is #1503's.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions