Skip to content

npu: config.json's vocab_size is never checked against the embedding table it overrides #1843

Closed

Description

@bong-water-water-bong

vocab_size from config.json wins even when the embedding table disagrees, and nothing compares them

Found auditing #1803 (merged ef00e33) against the CHANGES at its merged head. Taking the vocabulary from config.json first is right — it fixes
both defects the PR names — but the cross-check the review asked for is not on main.

Where: npu/model.cpp:82-93

if (c.contains("vocab_size")) {
    dims_.vocab = c.at("vocab_size");           // <- never compared with the table
} else {
    const Tensor* emb = find("model.embed_tokens.weight");
    ...
    dims_.vocab = emb->shape.size() >= 3 ? int(emb->shape[0] * kTileRows) : int(emb->shape.at(0));
}

Two sources now exist for the same number and the config always wins. If a pack's config.json states a vocab_size larger than the table's row count, the
embedding read runs past the end of the tensor; if it is smaller, the tail rows are unreachable. Neither is detected here, and both are the "answers
quietly wrong" class rather than a crash.

What it needs (small):

if (c.contains("vocab_size")) {
    dims_.vocab = c.at("vocab_size");
    const Tensor* emb = find("model.embed_tokens.weight");        // when present, it is evidence
    if (emb) {
        const int rows = emb->shape.size() >= 3 ? int(emb->shape[0] * kTileRows) : int(emb->shape.at(0));
        if (rows != dims_.vocab)
            throw std::runtime_error(dir + "/config.json states vocab_size " + std::to_string(dims_.vocab) +
                                     " but the embedding table has " + std::to_string(rows) + " rows");
    }
}

with an arm for the mismatch case next to the two the PR already added (config-silent fallback, and the plain-table shape). The npu_model_bounds_test
arms are the right home and need no device.

Why a refusal rather than picking one: the 1.9 Decision packs have no table at all (the case the PR fixes), so an absent table must stay silent; a
present but disagreeing table is a damaged or mislabelled pack, and neither number can be trusted to be the right one.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Tried to run it. Cannot, from this box's current state — and the reason is the issue itself.

    I own this one (it is my lane's run), so after clearing my review queue I went to execute the walk rather than wait for a table. Here is what exists:

    needed state now
    Qwen3-1.7B checkpoint ~/.cache/huggingface/hub/models--Qwen--Qwen3-1.7B is 4.4 MB — metadata and config only, no weights (the model is ~3.4 GB)
    gguf_pack.py not in the repo (git ls-tree -r origin/main | grep -c → 0)
    fconst_build.py not in the repo (0)
    fconst_qwen3.c not in the repo (0)
    built llama.cpp with the fconst layer none present
    /tmp artifacts from the earlier run purged — /tmp/fc19_s/ and gguf_f16.gguf are gone, only a 1-byte empty.gguf remains
    disk 143 G free, under the 175 G the owner's build rule requires

    Why that is the finding, not an excuse

    Every tool that produced the numbers in the shipped W2 row is uncommitted. The row cites 8a95e04f — a commit — but no one reading that table can rebuild the fixture it was measured on, because the two builders and the three .c layers live only in a scratch directory that no longer exists. ~/t_sweep was removed during the 15.8 GB disk break and took them with it.

    That is precisely the gap npu #51 just closed for the kernel vendor two hours ago, in almost the same words: "Not reproducible from a commit yet … Whoever ran these rows: commit it to a branch and put its sha here, or the engine PR it became." The review lane applied that standard to them; it applies to us.

    What would let this run

    1. commit gguf_pack.py, fconst_build.py and the three fconst_qwen3.c layers — the fixture builders are as much of the reference as the number is;
    2. download the 3.4 GB checkpoint (it will not fit the 143 G disk without the cache being pruned — see the q4nx cache issue, which was 160 GiB before it was pruned by hand);
    3. build (a few GB), then run one 128-token CPU-only walk and post the row.

    I am not going to fake a row, and I am not going to paste numbers from an earlier invocation whose artifacts no longer exist — that is the exact thing §2 forbids and the thing this issue was opened to prevent.

    Decision I am making instead of waiting: the W2 row is marked unreproducible until those three tools land in the tree. If anyone has them in a worktree, that is a two-command fix and I will run the walk the same hour.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correction to my own comment above — I was wrong, and the way I was wrong matters

    I wrote that "every tool that produced the shipped W2 row is uncommitted". That is false, and I should have checked for the capability instead of grepping for the four filenames I remembered from the spec.

    tools/fconst19_measure.py is in the tree, with tests/fconst19_measure_test.py beside it. It is exactly the tool that produces the rows tests/npu_pack_import_test.cpp pins: it measures a 1.9 pack against its 1.8 twin, derives the per-block prefix and the rotated bundle's tile count / tile size / rotation from content displacement rather than byte equality (because "a constant run — SmolLM2 has a 2,560-byte all-zero run — matches at any position and moves the boundary"), and then requires the result to equal the 1.8 file byte for byte:

    Anything less prints the residual and refuses: a near-miss rule is worse than none.

    That is the same discipline as #1861's literals and npu #51's provenance demand, and it has been in the tree the whole time. My comment above sent a reader looking for a missing tool that is not missing.

    What is actually missing, corrected

    needed for the W2 walk state
    rule derivation for the fconst19 rows committed — tools/fconst19_measure.py (+ test)
    the two vendor packs (1.8 and 1.9 of the same model) not on this box; these are vendor artifacts, not something we build
    the walk harness (run a pack under llama.cpp, compare 128 tokens) not committed — this is the real gap
    the 3.4 GB f16 checkpoint for an independent reference not downloaded; disk is 142 G, below the owner's 150 G floor
    fconst_qwen3.c a synthetic fixture kernel for the arena/flag experiment (#1839), not part of W2's row — I conflated the two

    So the W2 row's measurement side is reproducible and well built. What it lacks is the walk harness — the thing that takes a pack, runs N tokens under a known build, and prints the comparison — and the input packs. That is a much narrower gap than "everything was scratch", and it is the part worth committing.

    Miss #7, named

    "I asserted 'not in the repo' from a filename grep, not from a capability search." Four exact names, four zeros, and I reported it as though the function did not exist. A grep -i fconst over the tree would have found the tool. The check I should have run is the one I keep telling other lanes to run: search for what the artifact does, not for what I remember it being called.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Misfiled: my two comments above belong to a different subject, and I am flagging it rather than leaving them here

    The two comments I posted here (2026-11-10 08:52 and 09:12) are about the W2 device walk — fixture builders missing from the tree, /tmp scratch, a corrected claim about tools/fconst19_measure.py. None of that is about vocab_size.

    This issue is: "config.json's vocab_size is never checked against the embedding table it overrides" — a real, separate defect, and it is still open and unaddressed by my comments. A reader arriving here for the vocab check should not have to read two essays about pack scratch to find that out.

    Where the misfiled content actually belongs:

    Apologies for the noise on your issue. I will keep W2 material off it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions