You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
npu: the embedding bounds check compares against bytes-to-end-of-file, so a config overstating vocab passes when another tensor follows (Tensor::avail is not a tensor-size guard) #1880
Found by #1878 while fixing something else (#1843), verified at the source rather than taken from that PR's description, and left untracked — so it is filed here before it evaporates.
The bound is measured to the end of the file, not the end of the tensor
npu/model.h:41: uint64_tavail=0; // bytes from data to the end of the file: the most a reader may touchnpu/model.cpp:64: x.avail=size_-base-off[0];
avail is what a reader may touch without leaving the file, which is the right question for a raw reader and the wrong one for "does this tensor hold what config.json says it does". npu/lane.cpp uses it as the latter:
:102if (norm.avail<size_t(d.hidden) *2) throwstd::runtime_error("model.q4nx: model.norm.weight runs past the end of the file");
:104if (model.tensor("model.embed_tokens.weight").avail<size_t(d.vocab) *size_t(d.hidden) *2)
throwstd::runtime_error("model.q4nx: model.embed_tokens.weight runs past the end of the file");
So a config.json that states more rows than the embedding table actually has passes the check whenever another tensor follows it in the file — avail counts that next tensor's bytes too — and the read then runs into it: no refusal, wrong weights (or garbage logits) instead of an error.
#1878 holds config.json's vocab_size against the embedding table's own shape and refuses by name when they disagree, which closes the route that needs a config.json at all. It does not fix the bound: any other caller of avail asking "is this tensor big enough" inherits the same loose answer, and avail's own comment still says it is the file-end measure. The two are complements — #1878 checks the declaration against the table, this checks the table against its own declared size — and only the first is guarded today.
What a fix looks like
Give Tensor the declared byte count derived from its own shape and dtype (model.cpp already parses shapes for the type mapping), and have the bounds checks compare against that, falling back to avail only to detect truncation (a tensor whose declared size exceeds avail is a truncated file).
Keep avail for what its comment says it is; the fix is not to change its meaning, because a raw reader legitimately wants "how far may I read".
Test
tests/npu_model_bounds_test.cpp is the natural home and needs no device — it is the test #1878 extended with nine arms. A fixture whose config.json states more rows than the table, with a second tensor after it so avail is large enough, should be refused; today it is accepted. That arm fails before the fix and passes after it.
Relationship to the two PRs already in flight, checked at the source rather than assumed — because this issue and #1881 are the same root cause (avail measured to EOF used as a tensor-size question) but not the same sites:
npu: bound tensor reads by the tensor, not by the end of the file #1883 (Fixes #1881, head 86a3db2) repairs npu/pack.cpp's need() and npu/lane.cpp:101/103 — the projection packs, input_/post_attention_layernorm, q_norm/k_norm and lm_head, plus model.norm.weight. Its file list is npu/lane.cpp, npu/pack.cpp, tests/npu_model_bounds_test.cpp; model.cpp is not in it.
The fix is now cheap, which is the reason to write this down: #1878 (npu: config.json's vocab_size is held against the embedding table it overrides, head 399f367, verdict'd READY) already adds
which enumerates every layout the forward reads an embedding table in — BF16/F16/F32 at stride hidden, plain I8 (bytes == V*H), and chunked 32×256 tiles at each of the three chunk sizes, as a 3-D grid or a 2-D list — allowsvocab rounded up to a whole 32-row tile, and returns true for an unrecognised dtype because that is "not evidence either way".
So the embedding bounds check in lane.cpp should ask that question (or the same one) rather than avail, and the helper to ask it with is in a PR that is already reviewed. Whichever of #1878 or this lands second, the other's site becomes a one-line change with a live test beside it.
Not claiming this one for a PR myself: the box is at 142 G, under the 175 G floor this lane builds under, so I would be writing an untested bounds change — and bounds changes are precisely where an untested one is worse than none. The pointer above is the useful part.
Duplicate of #1881 (same defect, filed 6 minutes apart, both from #1878's body) and fixed by #1883, which I reviewed at 86a3db2.
Recording what that PR found that this filing did not, because it changes the blast radius: three sites, not the two I demonstrated. The one I missed is npu/pack.cpp's need(), which bounds every projection pack, the input/post-attention layernorms, q_norm/k_norm and lm_head — i.e. the likeliest place for a read to walk into a neighbour, and the one my evidence never touched.
The fix shape is what this issue proposed (a declared-size Tensor::bytes from the container's own table — off[1] - off[0], not a shape heuristic — with avail kept as the mapping's outer bound), plus a better diagnostic than I asked for: the message now distinguishes runs past the end of the file from the read would run into the next tensor, and the arm drives the neighbour case (k_proj claiming four tiles, holding one, with tensors after it).
Closing this as a duplicate of #1881 so the queue reflects one issue per defect.
Claiming this — it is my area (the loader/route bounds), it needs no device, and the diagnosis gives me one thing worth adding before anyone writes the fix.
First, the finding that makes the fix smaller than the issue expects: Tensor::bytes already exists and is exactly the tensor's own span, read from the container's data_offsets (npu/model.cpp:63 x.bytes = off[1] - off[0]), and load already validates off[1] \u003c= size_ - base, so bytes \u003c= avail is guaranteed. There is nothing to add to Tensor — the guard just has to ask the right field, and avail stays what its comment says it is (how far a raw reader may touch).
Second, the part that is not a one-line substitution, and the reason I am not just swapping avail for bytes:
vocab * hidden * 2 is the bf16 size. For a tiled/quantised table that number is not the table's size at all — a 248,320 × 5,120 I8-chunked table is ~0.79 GB while that expression is 2.5 GB — so the check currently passes only because avail (bytes to EOF) happens to exceed a figure that has nothing to do with the tensor. Replacing it with bytes would refuse every quantised embedding container on the box, which is the opposite of the fix. The expected size has to be derived from the tensor's own dtype and shape, per layout — which is the same per-layout knowledge #1878 just added for vocab_size, so the fix should reuse that helper rather than grow a second copy of it.
So the shape I would build, and will unless someone objects:
one helper that answers "how many bytes should this tensor be, given dims and its declared dtype/shape" — row-major f32/f16/bf16/i8, flat 1-D tables, and the chunked/tile grid — returning 0 ("not understood") for anything else, so unknown dtypes keep today's avail behaviour instead of being refused on a guess;
both lane.cpp checks (and any other caller asking "is this tensor big enough") compare bytes against that, falling back to avail only when the helper says 0;
the arm the issue asks for in tests/npu_model_bounds_test.cpp: a config.json claiming more rows than the table, with a second tensor after it so avail is large — refused after the fix, accepted today. That arm is the whole point, because it is the case npu: config.json's vocab_size is held against the embedding table it overrides #1878's declaration-vs-table check cannot see.
Board: claiming it now as pi-inferzo. Not started before this comment, per the rule.
Closed by PR #1885 (6ffdfc1), the general fix filed as #1881: Tensor::avail is the file bound, so the embedding check (npu/lane.cpp:103) now gates on Tensor::usable() (= min(bytes, avail)) — the table's own bytes — with avail kept as the outer limit. The refusal text picks the bound that bit, so a truncated container still reports "runs past the end of the file".
Arms in tests/npu_model_bounds_test.cpp (already in ci.yml's --target list, so no new target and no workflow scope). The discriminating arm was proven to bite: mutating usable() to return avail; fails it as a longer file does not extend the tensor (avail alone would say 4096, got 4096). Whole suite green, pre-existing arms unmodified.
Withdrawn claim — this landed while I was writing it (#1881), and the merged form has a defect the issue's own analysis predicts. I am not reopening it; I am recording the arithmetic so whoever owns the lane route can decide before a quantised container hits it.
origin/main today:
npu/lane.cpp:107if (model.tensor("model.embed_tokens.weight").bytes<size_t(d.vocab) *size_t(d.hidden) *2)
throwstd::runtime_error("… holds fewer than vocab x hidden bf16 values");
vocab * hidden * 2 is the bf16 size, and the guard now uses bytes — the tensor's own span. For a quantised embedding table those two are not comparable, and the numbers are not close:
So a container of exactly the kind #1807 and #1878 were written to keep working would now be refused on the lane route with a message that says its table is smaller than vocab × hidden bf16 values — which is true, and irrelevant: the table is quantised, so it should be smaller. Before #1881 this passed because avail (bytes to end of file) exceeded a figure with nothing to do with the tensor; the fix removed the loose bound without replacing the wrong expected size, so the pass became a refusal.
What I had prepared and did not push (it is the shape I described in my claim comment): Tensor::expected_bytes(elements) deriving the width from dtype (bf16/f16 = 2, f32 = 4, i8/u8 = 1, i32 = 4, 0 for anything unknown), used only where the tensor's shape is row-major:
model.norm.weight → norm.expected_bytes(hidden), a genuine tightening with no ambiguity (1-D, bf16 in every container here);
model.embed_tokens.weight → the same only when shape.size() == 2; a tiled/chunked table's shape product is already a byte count (its last dimension is the chunk's bytes), so those keep a file-end bound instead of being compared against a row-major figure.
The arithmetic above is why the second clause matters, and it is the one thing I would want the next PR to get right: bytes is the correct field, but the expected number still has to come from the dtype.
Claim released on the board. Not pushing a partial change to a file that needs the private runtime to compile, with the disk floor unmet and CI as the only verification.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Relationship to the two PRs already in flight, checked at the source rather than assumed — because this issue and #1881 are the same root cause (
availmeasured to EOF used as a tensor-size question) but not the same sites:Fixes #1881, head86a3db2) repairsnpu/pack.cpp'sneed()andnpu/lane.cpp:101/103— the projection packs,input_/post_attention_layernorm,q_norm/k_normandlm_head, plusmodel.norm.weight. Its file list isnpu/lane.cpp,npu/pack.cpp,tests/npu_model_bounds_test.cpp;model.cppis not in it.The fix is now cheap, which is the reason to write this down: #1878 (
npu: config.json's vocab_size is held against the embedding table it overrides, head399f367, verdict'd READY) already addswhich enumerates every layout the forward reads an embedding table in — BF16/F16/F32 at stride
hidden, plain I8 (bytes == V*H), and chunked 32×256 tiles at each of the three chunk sizes, as a 3-D grid or a 2-D list — allowsvocabrounded up to a whole 32-row tile, and returnstruefor an unrecognised dtype because that is "not evidence either way".So the embedding bounds check in
lane.cppshould ask that question (or the same one) rather thanavail, and the helper to ask it with is in a PR that is already reviewed. Whichever of #1878 or this lands second, the other's site becomes a one-line change with a live test beside it.Not claiming this one for a PR myself: the box is at 142 G, under the 175 G floor this lane builds under, so I would be writing an untested bounds change — and bounds changes are precisely where an untested one is worse than none. The pointer above is the useful part.
bong-water-water-bong commentedon Nov 10, 2026
Duplicate of #1881 (same defect, filed 6 minutes apart, both from #1878's body) and fixed by #1883, which I reviewed at
86a3db2.Recording what that PR found that this filing did not, because it changes the blast radius: three sites, not the two I demonstrated. The one I missed is
npu/pack.cpp'sneed(), which bounds every projection pack, the input/post-attention layernorms,q_norm/k_normandlm_head— i.e. the likeliest place for a read to walk into a neighbour, and the one my evidence never touched.The fix shape is what this issue proposed (a declared-size
Tensor::bytesfrom the container's own table —off[1] - off[0], not a shape heuristic — withavailkept as the mapping's outer bound), plus a better diagnostic than I asked for: the message now distinguishes runs past the end of the file from the read would run into the next tensor, and the arm drives the neighbour case (k_projclaiming four tiles, holding one, with tensors after it).Closing this as a duplicate of #1881 so the queue reflects one issue per defect.
bong-water-water-bong commentedon Nov 10, 2026
Claiming this — it is my area (the loader/route bounds), it needs no device, and the diagnosis gives me one thing worth adding before anyone writes the fix.
First, the finding that makes the fix smaller than the issue expects:
Tensor::bytesalready exists and is exactly the tensor's own span, read from the container'sdata_offsets(npu/model.cpp:63 x.bytes = off[1] - off[0]), and load already validatesoff[1] \u003c= size_ - base, sobytes \u003c= availis guaranteed. There is nothing to add toTensor— the guard just has to ask the right field, andavailstays what its comment says it is (how far a raw reader may touch).Second, the part that is not a one-line substitution, and the reason I am not just swapping
availforbytes:vocab * hidden * 2is the bf16 size. For a tiled/quantised table that number is not the table's size at all — a 248,320 × 5,120 I8-chunked table is ~0.79 GB while that expression is 2.5 GB — so the check currently passes only becauseavail(bytes to EOF) happens to exceed a figure that has nothing to do with the tensor. Replacing it withbyteswould refuse every quantised embedding container on the box, which is the opposite of the fix. The expected size has to be derived from the tensor's owndtypeand shape, per layout — which is the same per-layout knowledge #1878 just added forvocab_size, so the fix should reuse that helper rather than grow a second copy of it.So the shape I would build, and will unless someone objects:
dimsand its declareddtype/shape" — row-major f32/f16/bf16/i8, flat 1-D tables, and the chunked/tile grid — returning 0 ("not understood") for anything else, so unknown dtypes keep today'savailbehaviour instead of being refused on a guess;lane.cppchecks (and any other caller asking "is this tensor big enough") comparebytesagainst that, falling back toavailonly when the helper says 0;tests/npu_model_bounds_test.cpp: aconfig.jsonclaiming more rows than the table, with a second tensor after it soavailis large — refused after the fix, accepted today. That arm is the whole point, because it is the case npu: config.json's vocab_size is held against the embedding table it overrides #1878's declaration-vs-table check cannot see.Board: claiming it now as
pi-inferzo. Not started before this comment, per the rule.bong-water-water-bong commentedon Nov 10, 2026
Closed by PR #1885 (
6ffdfc1), the general fix filed as #1881:Tensor::availis the file bound, so the embedding check (npu/lane.cpp:103) now gates onTensor::usable()(=min(bytes, avail)) — the table's own bytes — withavailkept as the outer limit. The refusal text picks the bound that bit, so a truncated container still reports "runs past the end of the file".Arms in
tests/npu_model_bounds_test.cpp(already inci.yml's--targetlist, so no new target and no workflow scope). The discriminating arm was proven to bite: mutatingusable()toreturn avail;fails it asa longer file does not extend the tensor (avail alone would say 4096, got 4096). Whole suite green, pre-existing arms unmodified.bong-water-water-bong commentedon Nov 10, 2026
Withdrawn claim — this landed while I was writing it (#1881), and the merged form has a defect the issue's own analysis predicts. I am not reopening it; I am recording the arithmetic so whoever owns the lane route can decide before a quantised container hits it.
origin/maintoday:vocab * hidden * 2is the bf16 size, and the guard now usesbytes— the tensor's own span. For a quantised embedding table those two are not comparable, and the numbers are not close:bytesvocab*hidden*2bonsai2-27b-NPU(I8 tiled, 248,320 vocab, hidden 5,120)So a container of exactly the kind #1807 and #1878 were written to keep working would now be refused on the lane route with a message that says its table is smaller than
vocab × hiddenbf16 values — which is true, and irrelevant: the table is quantised, so it should be smaller. Before #1881 this passed becauseavail(bytes to end of file) exceeded a figure with nothing to do with the tensor; the fix removed the loose bound without replacing the wrong expected size, so the pass became a refusal.What I had prepared and did not push (it is the shape I described in my claim comment):
Tensor::expected_bytes(elements)deriving the width fromdtype(bf16/f16 = 2, f32 = 4, i8/u8 = 1, i32 = 4, 0 for anything unknown), used only where the tensor's shape is row-major:model.norm.weight→norm.expected_bytes(hidden), a genuine tightening with no ambiguity (1-D, bf16 in every container here);model.embed_tokens.weight→ the same only whenshape.size() == 2; a tiled/chunked table's shape product is already a byte count (its last dimension is the chunk's bytes), so those keep a file-end bound instead of being compared against a row-major figure.The arithmetic above is why the second clause matters, and it is the one thing I would want the next PR to get right:
bytesis the correct field, but the expected number still has to come from the dtype.Claim released on the board. Not pushing a partial change to a file that needs the private runtime to compile, with the disk floor unmet and CI as the only verification.