Skip to content

npu-pack: read the 1.7 Gemma-3-4b-it-mm embedding (f32 lm_head.weight table, 0-D embed_scale) (engine#2663) - #2664

Draft
bong-water-water-bong wants to merge 2 commits into
mainfrom
cl-npuperf/gemma3mm-embedding
Draft

bong-water-water-bong wants to merge 2 commits into
mainfrom
cl-npuperf/gemma3mm-embedding

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2663

Draft until the real-pack row lands. It is queued as an ordinary row on build/1bit-2663, a separate static link of this head, after the running 1.7 sweep. Disk is at the row floor, so only one pack can be on disk at a time.

What

The 1.7 Gemma-3-4b-it-mm-onnx-ryzenai-npu embedding graph (gemma-3-embedding.onnx, named by genai_config.json) is a Gather over lm_head.weight [262208, 2560] f32, followed by a multiply by the 0-D model.language_model.embed_tokens.embed_scale. The reader refused it ("no embed_tokens table + scales") for two reasons:

  1. The table carries no embed_tokens name. Now, when no table is named, the component's only 2-D f32 initializer is taken; with two candidates neither is.
  2. The 0-D embed_scale was taken as the 4-bit table's scales. A 0-D tensor is never taken as scales now, since those are 2-D [V, groups].

The container stores a gemma embedding pre-scaled: q4nx_forward.cpp's lookup applies no gemma scale, and its tied head divides sqrt(H) back out. So embed_scale is applied to the f32 table, and only when it equals sqrt(hidden). Any other value is refused by name, because nothing else is divided back out.

Also fixed (found by the first arm, which segfaulted): OnnxTensor::raw points into the graph file's mapping, and parse_onnx() unmaps that mapping before returning. Inline payloads of 64 B or less are now copied at parse time (OnnxTensor::small), and the scalar is read from that copy.

Proof so far

  • npu_pack_import_test: new test_embedding_gemma3mm_component, 4 arms:
  • Mutants:
    • hardcoded want = 16: both H-2560 arms FAIL, while the H-256 arms pass under it, which is the gap the review named;
    • no-scale fails arm 1;
    • no-fallback fails arms 1-3;
    • no-sqrt-check fails arm 3;
    • guessing between two fails arm 4.
    • Restored, everything passes.

Pending:

  • the real pack: npu-pack info gaps, import, and the NPU rows (kernels families/kernels-2560-10240-hd256-b8, text only);
  • full ctest.

Steps past the embedding (intermediate_size from the graph, the sliding/global RoPE split) are first reached on that run.

🤖 Generated with Claude Code

… table, 0-D embed_scale) (engine#2663)

The component's table is lm_head.weight [262208, 2560] f32 and its 0-D
embed_tokens.embed_scale was taken as the 4-bit scales. Take the only 2-D f32
initializer as the table when none is named, never a 0-D tensor as scales,
and apply embed_scale into the container's pre-scaled form only when it is
sqrt(hidden). parse_onnx now copies inline payloads <= 64 B: `raw` pointed
into a mapping unmapped before the reader looked (the first arm segfaulted).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit deb803c)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2663 - Partially compliant

Compliant requirements:

  • Fix the issue where npu-pack info stops with "no model.embed_tokens.weight" and "embedding.onnx has no embed_tokens table + scales"
  • Handle the case where the embedding table is named lm_head.weight instead of containing embed_tokens
  • Correctly handle the 0-D embed_scale tensor that should not be treated as 4-bit scales
  • Ensure the embedding table is read as a 2-D f32 tensor and scaled by the embed_scale when it matches sqrt(hidden)
  • Add tests to verify the fix works correctly

Non-compliant requirements:

  • None

Requires further human verification:

  • None

2668 - Partially compliant

Compliant requirements:

  • Copy inline payloads of 64 B or less into OnnxTensor::small during parsing to ensure data remains accessible

Non-compliant requirements:

Requires further human verification:

  • Verification that all existing readers of raw after parse are properly handled to avoid accessing freed memory
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Tensor Handling for Embedding Scales

The code previously treated a 0-D tensor named embed_scale as the 4-bit table's scales, which is incorrect since scales should be 2-D tensors. This has been fixed by adding a check to ensure that only 2-D tensors are considered as scales, and by correctly handling the embed_scale as a separate scalar multiplier.

if (t.dims.empty() && t.name.find("embed_scale") != std::string::npos) { embed_scale = &t; continue; }
Unmapped Memory Access

The code previously accessed raw data from tensors that pointed into the graph file's mapping, which was unmapped before returning from parse_onnx(). This has been fixed by copying small inline payloads (64 B or less) into a new small vector during parsing, ensuring that the data remains accessible for the tensor's lifetime.

const OnnxTensor* embed_scale = nullptr;   // engine#2663: a 0-D multiplier the component applies after its Gather

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Review asks, no verdict (the PR declares itself a draft until the real-pack row lands)

Two heuristics are being loosened in the same change, and each has a different failure direction — that is the thing to be careful about, not the specific pack.

  1. "when no table is named, the component's only 2-D f32 initializer is taken; with two candidates neither is" — a fallback by uniqueness. Safe against ambiguity, but it decides on a property of the whole component, so a future pack with one 2-D f32 initializer that is not the embedding table will be taken silently. The honest guard is the shape: [262208, 2560] is a vocabulary × hidden table, and a fallback that also checks rows == vocab_size (or rows > hidden) is nearly free and rules most of that out;
  2. "the 0-D embed_scale was taken as the 4-bit table's scales. A 0-D tensor is never taken as scales now" — a negative rule by rank, which is the right direction (scales are 2-D [V, groups]), but it is stated as a rank exclusion rather than as a shape contract. If the rule is "scales are 2-D", then say what the positive predicate is, or the next odd rank re-opens the question.

Three asks:

  1. state which of the two heuristics this pack actually needs — if CPU reference: GGUF reader, dequant, Qwen3 fp32 forward, golden test #1 alone fixes the import, Tokenizer: byte-level BPE from GGUF, exact against HF tokenizers #2 is a second rule shipped on one pack's evidence, and that is worth knowing because each loosening is a surface for a future wrong accept;
  2. the queued row needs a negative case as well — the real-pack row proves the good pack imports; it does not prove a pack with two 2-D f32 initializers still refuses. The refusal path is the one a future pack will hit, and it is cheap to keep: a fixture with two candidates must still say "neither is" rather than picking one;
  3. say how this relates to the GET_ROWS / embedding by-design class — lm_head.weight as the embedding table is exactly the node class llama.cpp's host-weight-placement.cpp treats as legitimately host-side ("the token-embedding GET_ROWS over a weight in the loader's file mapping", hw-gate: a placement node the engine calls BY DESIGN is not a misplacement (engine#2450 gate half) #2511/serve: GET /metrics with the engine-only series Lemonade cannot see (P5) #395). If the NPU importer now decides the same question with different rules, the two should at least cite each other, or one of them will drift.

And one thing worth keeping: "Disk is at the row floor, so only one pack can be on disk at a time" — that is a real constraint stated plainly, and it explains why the row is queued rather than run. Same as #2651's build/1bit relink note: the deferral is deliberate, not pending.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 8434609: READY. Three arms — positive, control, refusal — and the refusal is the one that matters: the reader does not guess between candidates, and it does not silently apply a scale the forward will not divide back out.

The scale check is the right shape

const double want = std::sqrt(double(Hf));
if (!(std::fabs(double(v) - want) <= 1e-3 * want)) {
    if (why) *why = "embed_scale is " + std::to_string(v) + ", not sqrt(hidden) = " + std::to_string(want) + …
  • relative, not absolute — a scale is multiplicative, so the tolerance has to be too, and 1e-3 * want is the correct form;
  • want is computed from the table's own dims (Hf), not from a per-model literal, so this generalises to the next mm component without a new constant;
  • the refusal names both numbers, so a pack that fails says "embed_scale is 50.4, not sqrt(hidden) = 50.5964" rather than "invalid";
  • the success path prints the scale it applied ("scaled by its embed_scale 50.5964 (= sqrt(2560)) into the container's pre-scaled form"), which is the provenance that makes a later numerical difference diagnosable.

And "two unnamed 2-D candidates are not guessed between" is the rule I would have asked for: the failure mode with two plausible tables is a container built from the wrong one, which no downstream check can see.

The arms are the set I would write

check(scaled,  "… lm_head.weight is the table and embed_scale (16 = sqrt(256)) is applied")
check(got == tok, "… control -- without embed_scale the raw table is read")
check(… why.find("embed_scale is") != npos, "… a scale other than sqrt(H) is refused")

Positive, negative control, and refusal. The middle one is the one usually missing: without it, an implementation that always applied some scale would still pass the positive arm, because the fixture is built to scale. With it, "the scale is optional and its absence reads the raw table" is a tested property rather than an intention.

One question about the tolerance's justification

1e-3 relative is 0.05 absolute on 50.5964. That comfortably absorbs a bf16- or f32-rounded sqrt(H) (a few parts in 10⁴), so the check will not refuse a correct pack — but it also accepts a pack whose scale is genuinely 0.4 % wrong, and that error would land on every embedding row, silently, inside the tolerance.

So: what is 1e-3 sized against? If the answer is "an f32 sqrt of an onnx-rounded H", then a number two orders of magnitude tighter (1e-5) would still pass every correct pack and refuse a wrong one; if the answer is "some exporter rounds the scale to 3 significant figures", say so beside the constant, because then 1e-3 is exactly right and a future reader will otherwise tighten it and break a real pack. Either way the constant deserves the sentence the rest of this function already gets.

Draft state, unchanged from what the description says

"Draft until the real-pack row lands … a separate static link of this head, after the running 1.7 sweep. Disk is at the row floor, so only one pack can be on disk at a time." That is the right sequencing and my READY covers the reader and its three arms only — the real-pack row is what should take it out of draft.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 8434609: READY on the design and both fixes. One ask about the arm, and it is the same trap I flagged on #2597: the fixture's hidden size is the number whose square root the proof quotes, so a hardcoded 16 would pass every host arm.

Both fixes are right, and the third thing it found is the best of them

  1. No embed_tokens name ⇒ take the component's only 2-D f32 initializer, "with two candidates it guesses neither" — ambiguity is declined rather than resolved by ordering, which is the property that makes a heuristic safe.
  2. A 0-D tensor is never taken as the 4-bit table's scales (if (t.name.find("scale") != npos) { if (t.dims.size() == 2) scales = &t; }) — the dims check gates a name match, which is why a embed_scale named scale cannot masquerade as [V, groups].
  3. And the arm found a real bug on its first run: OnnxTensor::raw points into the graph file's mapping, which parse_onnx() unmaps before returning. Inline payloads ≤ 64 B are now copied; larger inline tables are stated as still unread, with the reason ("the packs measured so far keep their tables in external data"). A test that segfaults and produces a fix, plus a named residual, is the best value a first arm can have.

The guard computes the scale, which is exactly why the arm needs a second H

const double want = std::sqrt(double(Hf));
if (!(std::fabs(double(v) - want) <= 1e-3 * want)) {
    … "embed_scale is " + v + ", not sqrt(hidden) = " + want + …

want is derived from Hf, the tolerance is relative (1e-3), and the refusal names both numbers — all three correct. But the fixture's hidden is 256, whose square root is 16, which is the number the proof quotes (embed_scale = 16 = sqrt(256)). So the arm as written cannot distinguish "computes √H" from "compares against 16": a hardcoded 16 passes it exactly.

The real pack is [262208, 2560] ⇒ √2560 ≈ 50.6, so a hardcoded 16 would fail on the device row and be caught there — but only after a queued real-pack run, and only for this pack. One fixture with a different H closes it at host speed:

H = 2560, embed_scale = 50.59644256269407  -> accepted, and the table scaled by that value
H = 2560, embed_scale = 16                 -> refused, "not sqrt(hidden) = 50.59…"

The second assertion is the one that matters: it is the #2597 shape (a fixture whose numbers coincide with the claim) and the gpu-kernels#107 shape (a claim about which artefact was measured) in one arm.

The residual deserves a ticket, or it will be a paragraph

"Not fixed here, stated so it isn't missed: the existing inline-f32 table branch and the per-layer inline table read raw after [the mapping is gone]"

Naming it is right and is why the PR is trustworthy. But two other branches with the same dangling-pointer hazard, written in a merged PR's body, will be found by whoever next segfaults — not by whoever reads. A one-line issue referencing #2663 (or a TODO(engine#2663) at each site) makes it greppable from the code, which is where the bug will be met.

The draft discipline is repeated correctly

build/1bit-2663, a separate static link of this head, so the queued row does not relink anything in use — the same shape #2651 arrived at after relinking build/1bit mid-sweep. Twice in one morning is a convention now, and worth one line in docs/npu.md so the third person does not learn it the hard way.

…ass (engine#2663)

At H 256 the scale the arm quoted (16) is sqrt(256), so `want = 16` passed every
host arm. At the real pack's width, sqrt(2560) = 50.596 is accepted and applied
and 16 is refused naming 50.59; the hardcoded-16 mutant fails both. The inline
residual is tracked as engine#2668.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Re the 10:54Z ask: done at deb803c.

The PR is still a draft until the real-pack row lands. It is queued on build/1bit-2663, linked from 8434609; deb803c changes only the test and the docs, so the binary's code is the same.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit deb803c

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: the 1.7 Gemma-3-4b-it-mm embedding is an f32 table named lm_head.weight, and its 0-D embed_scale is taken as the 4-bit scales

1 participant