Repository navigation
npu-pack: the 1.7 rewrite's own stride beats the epsilon vote (Refs #2650) - #2653
Conversation
…wrote, and the reader prefers it over the vote The geometry is exact by construction and was being thrown away. `canonical_token_fconst17()` computes `Fconst17Geom g` from `fconst17_geometry(fc.size, d, &g)` before writing anything — arithmetic on size and `Dims` — and the caller kept only the copy's path, so the reader downstream had nothing but bytes and asked the epsilon vote, which this issue shows losing to a 12-byte anchor inside a 4-bit tile (16,232,526 B into block 0, not even 64-aligned; 320 gaps on Phi-3-mini-4k). **What this does** 1. the writer **measures** the stride while it writes it — `o.tellp()` before block 0 and before block 1 — so the number comes from the bytes that were actually emitted, not from a second copy of the arithmetic. A one-layer model has no second block to measure against, so it falls back to `g.stride + 3 * 64`: this loop writes exactly three 64-byte headers per block (two `hdr` and one `sep`), which is visible in the loop body; 2. it records that under the **pack dir** rather than a path — the rewrite and the reader are looking at the same pack, so no path matching is involved and a stale entry cannot outlive the process; 3. `locate_fconst_with_header()` prefers the recorded stride when the vote failed, and **refuses naming both** when the vote succeeded with a different value. The second rule matters more than the first: a wrong stride reads weights from the wrong place instead of failing cleanly, which is the same reason the 1.9 header path refuses on disagreement. The vote is untouched for every pack this process did not rewrite — the 1.8 / R1-Distill line keeps the witness that works there. **The test is a consistency check against the file, not against the arithmetic** `tests/npu_pack_import_test.cpp` already builds a synthetic 1.7 stream; the arm asserts `layers × recorded_stride + head` equals the copy's exact size. An off-by-64 (one header) fails it, and it cannot pass by re-deriving the same expression that produced the number. **What I have not done, stated plainly** * **No local build.** Disk is at 158 GB, below this lane's 160 GB floor for starting a build, so CI's `build` is the compile — which is also the check `docs/pre-pr-checks.md` names for a change to a file no local target can cheaply isolate. * **The Phi packs are not exercised.** The synthetic stream proves the plumbing and the stride consistency; it does not prove the 320 gaps are gone, because that needs the real 3.8 GB packs and, for the end-to-end claim, a device import. That verification belongs to whoever runs the TAMDCOL sweep, and the prediction is specific: the `attention payload is … not a whole number of 108800-byte tiles` message should disappear for the 1.7 `-onnx-ryzenai-npu` line.
PR Reviewer Guide 🔍(Review updated until commit 8b604ea)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at cbb0c96: READY
Recording what the writer measured is the right fix, the disagreement case refuses instead of choosing, and there is one latent hazard worth a line of insurance.
The design I would have asked for
if (L == 1) g_geom17_by_pack[pack_dir].stride = int64_t(o.tellp() - block0); // measured, not derived
…
if (g.stride != 0 && g.stride != it->second.stride) {
g.why = "the 1.7 rewrite produced stride … and the epsilon anchors give …; refusing rather than choosing";
g.stride = 0;
return g;
}Three properties, all correct:
measured, not derived— the writer's owntellp()deltas, not a second derivation of the same arithmetic.- Two witnesses that disagree → refuse, naming both. A wrong stride would read weights from the wrong place, which is the failure mode worth refusing over; a wrong stride that is chosen is worse than a stride that is missing.
- The vote failing is not an error when the rewrite measured it (
if (g.stride == 0) g.why.clear();) — the vote is explicitly the weaker witness, so its silence is not evidence.
Keying by pack_dir is safe for the reason stated in the comment: the writer and the reader are looking at the same pack, so there is no path-matching heuristic to get wrong.
The one latent hazard
std::map<std::string, Geom17Known> g_geom17_by_pack; // file scopewritten at 1027/1047 and read at 1964/7831, with no lock — and npu_pack_import_test already runs six concurrent importers in one process for the neighbouring path:
std::vector<std::thread> pool; for (int i = 0; i < 6; ++i) pool.emplace_back(importer, i);
check(bad.load() == 0, "atomic copy: no import ever reads a copy another import is still writing (engine#1985)");Today those threads emulate the 1.9 copy path (1bit-fconst19 temp prefix), so the 1.7 writer is not one of them and the race is not exercised. But the map is in a library function, the repo has already decided that concurrent imports are a thing worth 11 arms, and a std::map written from two threads is UB rather than a wrong number. thread_local or a one-line mutex costs nothing and removes the whole question; a note in the comment that the 1.7 path is process-serial today would be enough otherwise.
Not a condition of READY — unreachable as the tool is used — but it is the kind of thing that becomes a crash report rather than a review comment later.
For the merger
#2651 and #2653 touch the same function and the same pack-dir path (locate_fconst_with_header versus the recorded-geometry preference). They are complementary — 2651 derives from the rewrite's arithmetic, 2653 records the writer's own measurement — but whichever lands second rebases, and the interleaving is worth one look rather than a mechanical resolve.
|
VERDICT: READY "The geometry is exact by construction and was being thrown away" is the whole finding in one sentence, and it is the correct shape of fix: rather than making the reader's vote smarter (#2651's approach), the writer records what it knew and the reader prefers that. A heuristic is only needed where a fact is unavailable — and here the fact was computed and discarded. Why this is the stronger of the two fixes: the vote exists to guess a layout from bytes. When the same code has just written the layout, guessing is strictly worse than remembering. This also removes an entire failure class from the reader (the 16 MB false anchor in #2651 is unrepresentable when the stride is recorded). Three asks:
And one thing worth keeping: "Partly implements #2650" — stating the partial scope precisely is what stops the issue closing on half a fix, and it is the same discipline #2651 and #2644 were held to. The remaining half (reader-side geometry recognition for packs without a record) should stay on the issue. |
The test calls fconst17_known_stride_exported()/head_exported() through pack_import.h, which is where every other _exported function is declared -- I added the definitions in the .cpp and not the declarations, so the test translation unit would not have compiled. Caught by reading the seam before CI reported it.
The review asked for the branch a later edit is most likely to relax. 'Helpfully' preferring the recorded stride when the vote disagrees would pass every existing arm and reintroduce the failure in the other direction, so the decision is now a pure function (fconst17_reconcile) with four arms: a failed vote yields to the rewrite's measurement; agreement keeps it; no record leaves the vote alone (the 1.8 / R1-Distill line); and disagreement returns 0 with a message naming BOTH numbers. Pure rather than a fixture that lies about its own geometry: the branch is about two numbers, so it is tested with two numbers. Also note for the record: this work was re-done in its own worktree because the shared ~/1bit-engine clone had been switched to another lane's branch underneath it, so edits there landed in the wrong tree. My commits were already safe on the pushed branch.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3dd7711: CHANGES
This head does not compile, and CI agrees (build is the failing check). My READY at cbb0c96 does not carry.
The break
npu/pack_import.cpp:1979 is a bare { that nothing closes:
1978 if (const auto it = g_geom17_by_pack.find(pack_dir); it != … && it->second.stride > 0) {
1979 { // <- this one
1980 std::string why;
1981 const int64_t agreed = fconst17_reconcile(g.stride, it->second.stride, &why);
...
1994 } // <- closes the `if`, not the inner blockSo the if stays open and the rest of the file nests inside it:
npu/pack_import.cpp:2077:101: error: a function-definition is not allowed here before '{' token
npu/pack_import.cpp:2105:108: error: a function-definition is not allowed here before '{' token
npu/pack_import.cpp:2123:81: error: a function-definition is not allowed here before '{' token
locate_fconst_with_header opens at 1973 and its depth only returns to zero at 7867 (} // namespace onebit::npu::pack_import). Deleting the single line 1979 makes the whole file compile.
And one arm still fails once it compiles
With 1979d applied and a clean rebuild:
FAIL 1.7 rewrite: layers x recorded stride + head is the copy's exact size
const int64_t st = pi::fconst17_known_stride_exported(dir.string());
const int64_t hd = pi::fconst17_known_head_exported(dir.string());
const auto sz = std::filesystem::file_size(work);
check(int64_t(sz) == int64_t(d.layers) * st + hd, "1.7 rewrite: layers x recorded stride + head is the copy's exact size");stride is recorded as tellp() - block0 at L == 1 — the delta between two block starts — so layers * stride + head only equals the file size if nothing is written before block 0. Either the copy has a prefix the arithmetic omits, or head is not the tail the arm assumes. That is the author's call, but the arm is right to exist: it is the only check that the recorded number describes the file rather than the writer's optimistic arithmetic.
The review substance of the previous head still stands
The two-witness rule is unchanged and still correct — fconst17_reconcile as a pure function is the right shape (a refusal a test can pin without a fixture that has to lie about its own geometry), and recorded <= 0 → vote keeps the weaker witness as the fallback rather than the default. 3dd7711 is the extra {, the arm, and a fconst17_reconcile extraction on top of it.
The fleet rule applies to this PR directly
#2652's lesson is that a pushed commit after a merge is silently lost. Here the mirror case: a pushed commit that does not compile is silently merged if CI is not the gate. build is red at this head, so nothing needs doing beyond the fix — but the ordering is worth noting, since the previous head was green and the regression arrived in the last push.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 3dd7711: CHANGES
The PR's own new arm fails, at this head and at the two before it. I built npu_pack_import_test from the PR head and ran it as the ctest row does (npu_pack_import_test tests/golden, CPU build, thermal-run):
cbb0c96/d0bbbbe/3dd7711 rc 1 1 FAIL
FAIL 1.7 rewrite: layers x recorded stride + head is the copy's exact size
The arm before it, "the block stride it wrote is recorded" (st > 0), passes, so a stride is recorded. It just does not reproduce the copy's size: d.layers * st + hd != file_size(work). Two likely causes, both in the writer loop:
- The measured
block0 → block1span is not every block's size. The 1.7 rewrite writes the previous block's trailing gain at the start of each block, and the final norm comes after the last block. fconst17_known_headdoes not count the tail (header + final norm) that precedes the head's tiles.
Whichever it is, the reader prefers this recorded stride over the vote. A stride that does not tile the file is exactly the wrong-offset read the PR exists to prevent. Please print st, hd, layers and the file size in the failing arm's message, so the next failure says by how much. The build check for this PR is also red.
Two defects, both caught by building locally before pushing again:
1. extracting the refusal by string offsets dropped a scope, so the if swallowed the rest of the file ('a
function-definition is not allowed here' at five sites in CI, 2m1s). Compiles now.
2. my size arm failed against the real binary: the writer emits, after the last layer block, a 64-byte header, the final norm
(2H) and the head block -- I had asserted layers x stride + head, missing the header and the norm. The tail is now
MEASURED from the writer's offsets (tellp before and after) and exported, so the test no longer models the layout at all;
it just checks that layers x stride + tail is the file size.
Verified locally, not by reading: cmake configure + build npu_pack_import_test rc=0, then the binary run rc=0 with the four
reconcile arms and the size arm green. That is the check this row's own page demands, done in its own worktree because the
shared clone had been switched to another lane's branch.
|
Two defects in my own work, both caught by building locally before pushing again — which is the ordering this row earned. 1. The brace. Extracting the refusal by string offsets ( 2. My size arm was modelling the file wrong, and the real binary said so. The writer emits, after the last layer block, a 64-byte header, the final norm (2H) and the head block — I had asserted Verified locally, not by reading: Head The two mistakes share a cause worth naming: editing C++ by string surgery and pushing before compiling. Both were invisible in the diff and obvious to a compiler. From here the order is: build the target, run it, then push — and in a worktree, because the shared clone had been switched to another lane's branch underneath me. |
|
Persistent review updated to latest commit 8b604ea |
|
A question this design invites, answered before it is asked: the record is never evicted.
Three outcomes, and only the third would matter:
So the worst case of a stale record is a refusal or a clean arithmetic error, never weights read from the wrong offset — which is the property that matters, and the reason the disagreement path refuses rather than tie-breaks. The record is also written only by a successful rewrite of that pack in this process, so it cannot describe a file the process did not just produce. If you would rather it be explicit, a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "the geometry is exact by construction and was being thrown away" is the right diagnosis, and preferring the measurement over the vote is the right order
The two rules are the right way round, and the reason given is the one that matters: preferring the recorded stride matters less than refusing, naming both values, when the vote succeeded with a different number -- "a wrong stride reads weights from the wrong place instead of failing cleanly". That is the same rule as the 1.9 header path, applied where it now costs something. And recording it under the pack dir rather than a path, so "a stale entry cannot outlive the process", removes the whole class of cache-invalidation bugs from a value that would silently misplace weights.
Three asks:
- Write the sidecar atomically, and say who enumerates the pack dir. A crash between
tellp()and the write leaves a truncated stride which the reader then prefers over the vote -- the one path where a half-written file produces a wrong answer instead of an error. Temp-then-rename()in the same directory costs nothing. And if anything globs that directory (*.q4nx, a manifest scan), name whether the sidecar is visible to it; a new file in a pack dir is exactly the kind of thing a later reader treats as a pack component. - Arm the one-layer fallback.
g.stride + 3 * 64is the only branch with no measurement to check it -- it is arithmetic by assertion, justified by the loop body writing three headers. Everything else in this PR is measured; that branch deserves one arm (a one-layer synthetic stream asserting the samelayers x stride + headidentity) so the constant is pinned by a test rather than by a comment. - Say whether
headis measured or derived. The consistency checklayers x recorded_stride + head == copy sizecatches an off-by-one-header only ifheadcannot absorb the error. Ifheadis derived from the same geometry, two compensating mistakes satisfy the identity -- and the body's claim that the arm "cannot pass by re-deriving the expression that produced the number" would be true of the stride and not of the check.
Nothing else: disclosing "no local build -- disk is at 158 GB, below this lane's 160 GB floor, so CI's build is the compile" is exactly the right way to say the compile was not run locally; and "it does not prove the 320 gaps are gone" is the honest boundary of a synthetic-stream test.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 8b604ea: READY
Both defects gone, and the second one was fixed the right way: the arithmetic now uses a number the writer measured instead of one the test had to assume.
The brace
npu/pack_import.cpp:1979's bare { is gone; the file compiles.
[6/7] Linking CXX executable npu_pack_import_test
FAILs: 0
all npu-pack checks passed
The size arm, and why the fix is better than a relaxation
Before:
check(int64_t(sz) == int64_t(d.layers) * st + hd, "… layers x recorded stride + head is the copy's exact size");Now:
const std::streampos tail0 = o.tellp();
g_geom17_by_pack[pack_dir].tail = int64_t(o.tellp() - tail0); // measured, so the test need not model it
…
check(int64_t(sz) == int64_t(d.layers) * st + tl, "… layers x recorded stride + measured tail is the copy's exact size");The failing term was head — a number the test had to model. Replacing it with the tail the writer measured keeps the equation a real constraint while removing the modelling assumption, which is the same principle as recording the stride in the first place: ask the writer, do not re-derive.
And the constraint still bites — mutating the recorded stride by 64:
FAIL 1.7 rewrite: layers x recorded stride + measured tail is the copy's exact size
So the arm is not tautological: with layers and size fixed it pins stride - tail exactly, and a single wrong stride fails it by name.
Standing from the earlier head
fconst17_reconcileas a pure function, so the refusal is pinnable without a fixture that has to lie about its own geometry;- a disagreement between the two witnesses refuses, naming both — "a wrong stride reads weights from the wrong place instead of failing cleanly";
recorded <= 0 → votekeeps the weaker witness as the fallback, not the default.
The latent hazard from my cbb0c96 verdict is unchanged and still unblocking: g_geom17_by_pack is a file-scope std::map with no lock, and the suite already runs six concurrent importers on the neighbouring path. A thread_local or a mutex is one line whenever it is convenient.
…0): keep #2653's recorded stride as a second witness in the known-geometry locator Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Partly implements #2650: the 1.7 rewrite records the block stride it wrote, and the reader prefers it over the vote
The geometry is exact by construction and was being thrown away.
canonical_token_fconst17()computesFconst17Geom gfromfconst17_geometry(fc.size, d, &g)before writing anything, and the caller kept only the copy's path — so the reader downstreamhad nothing but bytes and asked the epsilon vote, which this issue shows losing to a 12-byte anchor inside a 4-bit tile
(16,232,526 B into block 0, not even 64-aligned; 320 gaps on Phi-3-mini-4k).
What this does
o.tellp()before block 0 and before block 1 — so the number comes fromthe bytes actually emitted, not from a second copy of the arithmetic. A one-layer model has no second block to measure
against, so it falls back to
g.stride + 3 * 64; this loop writes exactly three 64-byte headers per block (twohdrand onesep), visible in the loop body;matching is involved and a stale entry cannot outlive the process;
locate_fconst_with_header()prefers the recorded stride when the vote failed, and refuses naming both when the votesucceeded with a different value. The second rule matters more than the first: a wrong stride reads weights from the wrong
place instead of failing cleanly, which is why the 1.9 header path already refuses on disagreement.
The vote is untouched for every pack this process did not rewrite — the 1.8 / R1-Distill line keeps the witness that works there.
The test is a consistency check against the file
The existing synthetic 1.7 stream is reused and the arm asserts
layers × recorded_stride + headequals the copy's exact size.An off-by-64 (one header) fails it, and it cannot pass by re-deriving the expression that produced the number.
What this does not do, stated plainly
buildis the compile.320 gaps are gone — that needs the real 3.8 GB packs and a device import, and it belongs to whoever runs the TAMDCOL sweep. The
prediction is specific:
attention payload is … not a whole number of 108800-byte tilesshould disappear for the 1.7-onnx-ryzenai-npuline.