Repository navigation
npu-pack: read the 1.7 Gemma-3-4b-it-mm embedding (f32 lm_head.weight table, 0-D embed_scale) (engine#2663) - #2664
bong-water-water-bong wants to merge 2 commits into
Conversation
… table, 0-D embed_scale) (engine#2663) The component's table is lm_head.weight [262208, 2560] f32 and its 0-D embed_tokens.embed_scale was taken as the 4-bit scales. Take the only 2-D f32 initializer as the table when none is named, never a 0-D tensor as scales, and apply embed_scale into the container's pre-scaled form only when it is sqrt(hidden). parse_onnx now copies inline payloads <= 64 B: `raw` pointed into a mapping unmapped before the reader looked (the first arm segfaulted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit deb803c)Here are some key observations to aid the review process:
|
Review asks, no verdict (the PR declares itself a draft until the real-pack row lands)Two heuristics are being loosened in the same change, and each has a different failure direction — that is the thing to be careful about, not the specific pack.
Three asks:
And one thing worth keeping: "Disk is at the row floor, so only one pack can be on disk at a time" — that is a real constraint stated plainly, and it explains why the row is queued rather than run. Same as #2651's |
|
VERDICT at 8434609: READY. Three arms — positive, control, refusal — and the refusal is the one that matters: the reader does not guess between candidates, and it does not silently apply a scale the forward will not divide back out. The scale check is the right shapeconst double want = std::sqrt(double(Hf));
if (!(std::fabs(double(v) - want) <= 1e-3 * want)) {
if (why) *why = "embed_scale is " + std::to_string(v) + ", not sqrt(hidden) = " + std::to_string(want) + …
And "two unnamed 2-D candidates are not guessed between" is the rule I would have asked for: the failure mode with two plausible tables is a container built from the wrong one, which no downstream check can see. The arms are the set I would writePositive, negative control, and refusal. The middle one is the one usually missing: without it, an implementation that always applied some scale would still pass the positive arm, because the fixture is built to scale. With it, "the scale is optional and its absence reads the raw table" is a tested property rather than an intention. One question about the tolerance's justification
So: what is Draft state, unchanged from what the description says"Draft until the real-pack row lands … a separate static link of this head, after the running 1.7 sweep. Disk is at the row floor, so only one pack can be on disk at a time." That is the right sequencing and my READY covers the reader and its three arms only — the real-pack row is what should take it out of draft. |
|
VERDICT at Both fixes are right, and the third thing it found is the best of them
The guard computes the scale, which is exactly why the arm needs a second Hconst double want = std::sqrt(double(Hf));
if (!(std::fabs(double(v) - want) <= 1e-3 * want)) {
… "embed_scale is " + v + ", not sqrt(hidden) = " + want + …
The real pack is The second assertion is the one that matters: it is the The residual deserves a ticket, or it will be a paragraph
Naming it is right and is why the PR is trustworthy. But two other branches with the same dangling-pointer hazard, written in a merged PR's body, will be found by whoever next segfaults — not by whoever reads. A one-line issue referencing The draft discipline is repeated correctly
|
…ass (engine#2663) At H 256 the scale the arm quoted (16) is sqrt(256), so `want = 16` passed every host arm. At the real pack's width, sqrt(2560) = 50.596 is accepted and applied and 16 is refused naming 50.59; the hardcoded-16 mutant fails both. The inline residual is tracked as engine#2668. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re the 10:54Z ask: done at deb803c.
The PR is still a draft until the real-pack row lands. It is queued on |
|
Persistent review updated to latest commit deb803c |
Fixes #2663
Draft until the real-pack row lands. It is queued as an ordinary row on
build/1bit-2663, a separate static link of this head, after the running 1.7 sweep. Disk is at the row floor, so only one pack can be on disk at a time.What
The 1.7
Gemma-3-4b-it-mm-onnx-ryzenai-npuembedding graph (gemma-3-embedding.onnx, named bygenai_config.json) is aGatheroverlm_head.weight[262208, 2560] f32, followed by a multiply by the 0-Dmodel.language_model.embed_tokens.embed_scale. The reader refused it ("no embed_tokens table + scales") for two reasons:embed_tokensname. Now, when no table is named, the component's only 2-D f32 initializer is taken; with two candidates neither is.embed_scalewas taken as the 4-bit table's scales. A 0-D tensor is never taken as scales now, since those are 2-D[V, groups].The container stores a gemma embedding pre-scaled:
q4nx_forward.cpp's lookup applies no gemma scale, and its tied head dividessqrt(H)back out. Soembed_scaleis applied to the f32 table, and only when it equals sqrt(hidden). Any other value is refused by name, because nothing else is divided back out.Also fixed (found by the first arm, which segfaulted):
OnnxTensor::rawpoints into the graph file's mapping, andparse_onnx()unmaps that mapping before returning. Inline payloads of 64 B or less are now copied at parse time (OnnxTensor::small), and the scalar is read from that copy.rawafter the parse the same way. Every pack measured so far keeps its tables in external data, so neither path has been hit.Proof so far
npu_pack_import_test: newtest_embedding_gemma3mm_component, 4 arms:lm_head.weightis read and scaled byembed_scale= 16 = sqrt(256), exact;embed_scalethe raw table is read;embed_scale10 is refused by name;want = 16: both H-2560 arms FAIL, while the H-256 arms pass under it, which is the gap the review named;Pending:
npu-pack infogaps, import, and the NPU rows (kernelsfamilies/kernels-2560-10240-hd256-b8, text only);Steps past the embedding (intermediate_size from the graph, the sliding/global RoPE split) are first reached on that run.
🤖 Generated with Claude Code