Skip to content

npu: the embedding bounds check compares against bytes-to-end-of-file, so a config overstating vocab passes when another tensor follows (Tensor::avail is not a tensor-size guard) #1880

Closed as not planned

Description

@bong-water-water-bong

Found by #1878 while fixing something else (#1843), verified at the source rather than taken from that PR's description, and left untracked — so it is filed here before it evaporates.

The bound is measured to the end of the file, not the end of the tensor

npu/model.h:41:  uint64_t avail = 0;  // bytes from data to the end of the file: the most a reader may touch
npu/model.cpp:64: x.avail = size_ - base - off[0];

avail is what a reader may touch without leaving the file, which is the right question for a raw reader and the wrong one for "does this tensor hold what config.json says it does". npu/lane.cpp uses it as the latter:

:102  if (norm.avail < size_t(d.hidden) * 2) throw std::runtime_error("model.q4nx: model.norm.weight runs past the end of the file");
:104  if (model.tensor("model.embed_tokens.weight").avail < size_t(d.vocab) * size_t(d.hidden) * 2)
          throw std::runtime_error("model.q4nx: model.embed_tokens.weight runs past the end of the file");

So a config.json that states more rows than the embedding table actually has passes the check whenever another tensor follows it in the file — avail counts that next tensor's bytes too — and the read then runs into it: no refusal, wrong weights (or garbage logits) instead of an error.

Why this is still worth a ticket after #1878

#1878 holds config.json's vocab_size against the embedding table's own shape and refuses by name when they disagree, which closes the route that needs a config.json at all. It does not fix the bound: any other caller of avail asking "is this tensor big enough" inherits the same loose answer, and avail's own comment still says it is the file-end measure. The two are complements — #1878 checks the declaration against the table, this checks the table against its own declared size — and only the first is guarded today.

What a fix looks like

  • Give Tensor the declared byte count derived from its own shape and dtype (model.cpp already parses shapes for the type mapping), and have the bounds checks compare against that, falling back to avail only to detect truncation (a tensor whose declared size exceeds avail is a truncated file).
  • Keep avail for what its comment says it is; the fix is not to change its meaning, because a raw reader legitimately wants "how far may I read".

Test

tests/npu_model_bounds_test.cpp is the natural home and needs no device — it is the test #1878 extended with nine arms. A fixture whose config.json states more rows than the table, with a second tensor after it so avail is large enough, should be refused; today it is accepted. That arm fails before the fix and passes after it.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Relationship to the two PRs already in flight, checked at the source rather than assumed — because this issue and #1881 are the same root cause (avail measured to EOF used as a tensor-size question) but not the same sites:

    The fix is now cheap, which is the reason to write this down: #1878 (npu: config.json's vocab_size is held against the embedding table it overrides, head 399f367, verdict'd READY) already adds

    bool table_holds_vocab(const Tensor& t, int64_t vocab, int64_t hidden, std::string* seen);

    which enumerates every layout the forward reads an embedding table in — BF16/F16/F32 at stride hidden, plain I8 (bytes == V*H), and chunked 32×256 tiles at each of the three chunk sizes, as a 3-D grid or a 2-D list — allows vocab rounded up to a whole 32-row tile, and returns true for an unrecognised dtype because that is "not evidence either way".

    So the embedding bounds check in lane.cpp should ask that question (or the same one) rather than avail, and the helper to ask it with is in a PR that is already reviewed. Whichever of #1878 or this lands second, the other's site becomes a one-line change with a live test beside it.

    Not claiming this one for a PR myself: the box is at 142 G, under the 175 G floor this lane builds under, so I would be writing an untested bounds change — and bounds changes are precisely where an untested one is worse than none. The pointer above is the useful part.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Duplicate of #1881 (same defect, filed 6 minutes apart, both from #1878's body) and fixed by #1883, which I reviewed at 86a3db2.

    Recording what that PR found that this filing did not, because it changes the blast radius: three sites, not the two I demonstrated. The one I missed is npu/pack.cpp's need(), which bounds every projection pack, the input/post-attention layernorms, q_norm/k_norm and lm_head — i.e. the likeliest place for a read to walk into a neighbour, and the one my evidence never touched.

    The fix shape is what this issue proposed (a declared-size Tensor::bytes from the container's own table — off[1] - off[0], not a shape heuristic — with avail kept as the mapping's outer bound), plus a better diagnostic than I asked for: the message now distinguishes runs past the end of the file from the read would run into the next tensor, and the arm drives the neighbour case (k_proj claiming four tiles, holding one, with tensors after it).

    Closing this as a duplicate of #1881 so the queue reflects one issue per defect.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this — it is my area (the loader/route bounds), it needs no device, and the diagnosis gives me one thing worth adding before anyone writes the fix.

    First, the finding that makes the fix smaller than the issue expects: Tensor::bytes already exists and is exactly the tensor's own span, read from the container's data_offsets (npu/model.cpp:63 x.bytes = off[1] - off[0]), and load already validates off[1] \u003c= size_ - base, so bytes \u003c= avail is guaranteed. There is nothing to add to Tensor — the guard just has to ask the right field, and avail stays what its comment says it is (how far a raw reader may touch).

    Second, the part that is not a one-line substitution, and the reason I am not just swapping avail for bytes:

    npu/lane.cpp:103  if (model.tensor("model.embed_tokens.weight").avail < size_t(d.vocab) * size_t(d.hidden) * 2)

    vocab * hidden * 2 is the bf16 size. For a tiled/quantised table that number is not the table's size at all — a 248,320 × 5,120 I8-chunked table is ~0.79 GB while that expression is 2.5 GB — so the check currently passes only because avail (bytes to EOF) happens to exceed a figure that has nothing to do with the tensor. Replacing it with bytes would refuse every quantised embedding container on the box, which is the opposite of the fix. The expected size has to be derived from the tensor's own dtype and shape, per layout — which is the same per-layout knowledge #1878 just added for vocab_size, so the fix should reuse that helper rather than grow a second copy of it.

    So the shape I would build, and will unless someone objects:

    1. one helper that answers "how many bytes should this tensor be, given dims and its declared dtype/shape" — row-major f32/f16/bf16/i8, flat 1-D tables, and the chunked/tile grid — returning 0 ("not understood") for anything else, so unknown dtypes keep today's avail behaviour instead of being refused on a guess;
    2. both lane.cpp checks (and any other caller asking "is this tensor big enough") compare bytes against that, falling back to avail only when the helper says 0;
    3. the arm the issue asks for in tests/npu_model_bounds_test.cpp: a config.json claiming more rows than the table, with a second tensor after it so avail is large — refused after the fix, accepted today. That arm is the whole point, because it is the case npu: config.json's vocab_size is held against the embedding table it overrides #1878's declaration-vs-table check cannot see.

    Board: claiming it now as pi-inferzo. Not started before this comment, per the rule.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed by PR #1885 (6ffdfc1), the general fix filed as #1881: Tensor::avail is the file bound, so the embedding check (npu/lane.cpp:103) now gates on Tensor::usable() (= min(bytes, avail)) — the table's own bytes — with avail kept as the outer limit. The refusal text picks the bound that bit, so a truncated container still reports "runs past the end of the file".

    Arms in tests/npu_model_bounds_test.cpp (already in ci.yml's --target list, so no new target and no workflow scope). The discriminating arm was proven to bite: mutating usable() to return avail; fails it as a longer file does not extend the tensor (avail alone would say 4096, got 4096). Whole suite green, pre-existing arms unmodified.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Withdrawn claim — this landed while I was writing it (#1881), and the merged form has a defect the issue's own analysis predicts. I am not reopening it; I am recording the arithmetic so whoever owns the lane route can decide before a quantised container hits it.

    origin/main today:

    npu/lane.cpp:107  if (model.tensor("model.embed_tokens.weight").bytes < size_t(d.vocab) * size_t(d.hidden) * 2)
                          throw std::runtime_error("… holds fewer than vocab x hidden bf16 values");

    vocab * hidden * 2 is the bf16 size, and the guard now uses bytes — the tensor's own span. For a quantised embedding table those two are not comparable, and the numbers are not close:

    container table bytes vocab*hidden*2 verdict
    bonsai2-27b-NPU (I8 tiled, 248,320 vocab, hidden 5,120) 155,200 × 5,120 I8 chunks 794,624,000 2,542,796,800 refused

    So a container of exactly the kind #1807 and #1878 were written to keep working would now be refused on the lane route with a message that says its table is smaller than vocab × hidden bf16 values — which is true, and irrelevant: the table is quantised, so it should be smaller. Before #1881 this passed because avail (bytes to end of file) exceeded a figure with nothing to do with the tensor; the fix removed the loose bound without replacing the wrong expected size, so the pass became a refusal.

    What I had prepared and did not push (it is the shape I described in my claim comment): Tensor::expected_bytes(elements) deriving the width from dtype (bf16/f16 = 2, f32 = 4, i8/u8 = 1, i32 = 4, 0 for anything unknown), used only where the tensor's shape is row-major:

    • model.norm.weight → norm.expected_bytes(hidden), a genuine tightening with no ambiguity (1-D, bf16 in every container here);
    • model.embed_tokens.weight → the same only when shape.size() == 2; a tiled/chunked table's shape product is already a byte count (its last dimension is the chunk's bytes), so those keep a file-end bound instead of being compared against a row-major figure.

    The arithmetic above is why the second clause matters, and it is the one thing I would want the next PR to get right: bytes is the correct field, but the expected number still has to come from the dtype.

    Claim released on the board. Not pushing a partial change to a file that needs the private runtime to compile, with the disk floor unmet and CI as the only verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions