Skip to content

npu-pack: the 1.7 rewrite's own stride beats the epsilon vote (Refs #2650) - #2653

Merged
bong-water-water-bong merged 4 commits into
mainfrom
pkg/2650-rewrite-geometry
Nov 16, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
pkg/2650-rewrite-geometry

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Partly implements #2650: the 1.7 rewrite records the block stride it wrote, and the reader prefers it over the vote

The geometry is exact by construction and was being thrown away. canonical_token_fconst17() computes Fconst17Geom g from
fconst17_geometry(fc.size, d, &g) before writing anything, and the caller kept only the copy's path — so the reader downstream
had nothing but bytes and asked the epsilon vote, which this issue shows losing to a 12-byte anchor inside a 4-bit tile
(16,232,526 B into block 0, not even 64-aligned; 320 gaps on Phi-3-mini-4k).

What this does

  1. the writer measures the stride while writing it — o.tellp() before block 0 and before block 1 — so the number comes from
    the bytes actually emitted, not from a second copy of the arithmetic. A one-layer model has no second block to measure
    against, so it falls back to g.stride + 3 * 64; this loop writes exactly three 64-byte headers per block (two hdr and one
    sep), visible in the loop body;
  2. it records that under the pack dir, not a path — the rewrite and the reader are looking at the same pack, so no path
    matching is involved and a stale entry cannot outlive the process;
  3. locate_fconst_with_header() prefers the recorded stride when the vote failed, and refuses naming both when the vote
    succeeded with a different value. The second rule matters more than the first: a wrong stride reads weights from the wrong
    place instead of failing cleanly, which is why the 1.9 header path already refuses on disagreement.

The vote is untouched for every pack this process did not rewrite — the 1.8 / R1-Distill line keeps the witness that works there.

The test is a consistency check against the file

The existing synthetic 1.7 stream is reused and the arm asserts layers × recorded_stride + head equals the copy's exact size.
An off-by-64 (one header) fails it, and it cannot pass by re-deriving the expression that produced the number.

What this does not do, stated plainly

  • No local build. Disk is at 158 GB, below this lane's 160 GB floor, so CI's build is the compile.
  • The Phi packs are not exercised. The synthetic stream proves the plumbing and the stride consistency; it does not prove the
    320 gaps are gone — that needs the real 3.8 GB packs and a device import, and it belongs to whoever runs the TAMDCOL sweep. The
    prediction is specific: attention payload is … not a whole number of 108800-byte tiles should disappear for the 1.7
    -onnx-ryzenai-npu line.

…wrote, and the reader prefers it over the vote

The geometry is exact by construction and was being thrown away. `canonical_token_fconst17()` computes `Fconst17Geom g` from
`fconst17_geometry(fc.size, d, &g)` before writing anything — arithmetic on size and `Dims` — and the caller kept only the
copy's path, so the reader downstream had nothing but bytes and asked the epsilon vote, which this issue shows losing to a
12-byte anchor inside a 4-bit tile (16,232,526 B into block 0, not even 64-aligned; 320 gaps on Phi-3-mini-4k).

**What this does**

1. the writer **measures** the stride while it writes it — `o.tellp()` before block 0 and before block 1 — so the number comes
   from the bytes that were actually emitted, not from a second copy of the arithmetic. A one-layer model has no second block to
   measure against, so it falls back to `g.stride + 3 * 64`: this loop writes exactly three 64-byte headers per block (two `hdr`
   and one `sep`), which is visible in the loop body;
2. it records that under the **pack dir** rather than a path — the rewrite and the reader are looking at the same pack, so no
   path matching is involved and a stale entry cannot outlive the process;
3. `locate_fconst_with_header()` prefers the recorded stride when the vote failed, and **refuses naming both** when the vote
   succeeded with a different value. The second rule matters more than the first: a wrong stride reads weights from the wrong
   place instead of failing cleanly, which is the same reason the 1.9 header path refuses on disagreement.

The vote is untouched for every pack this process did not rewrite — the 1.8 / R1-Distill line keeps the witness that works there.

**The test is a consistency check against the file, not against the arithmetic**

`tests/npu_pack_import_test.cpp` already builds a synthetic 1.7 stream; the arm asserts `layers × recorded_stride + head` equals
the copy's exact size. An off-by-64 (one header) fails it, and it cannot pass by re-deriving the same expression that produced the
number.

**What I have not done, stated plainly**

* **No local build.** Disk is at 158 GB, below this lane's 160 GB floor for starting a build, so CI's `build` is the compile —
  which is also the check `docs/pre-pr-checks.md` names for a change to a file no local target can cheaply isolate.
* **The Phi packs are not exercised.** The synthetic stream proves the plumbing and the stride consistency; it does not prove the
  320 gaps are gone, because that needs the real 3.8 GB packs and, for the end-to-end claim, a device import. That verification
  belongs to whoever runs the TAMDCOL sweep, and the prediction is specific: the `attention payload is … not a whole number of
  108800-byte tiles` message should disappear for the 1.7 `-onnx-ryzenai-npu` line.
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 8b604ea)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2650 - Partially compliant

Compliant requirements:

  • The PR records the actual stride written by the 1.7 rewrite and has the reader prefer it over the epsilon vote
  • Single-layer models fall back to a correct stride calculation
  • Tests verify that the recorded stride matches the actual file size

Non-compliant requirements:

  • The PR does not address the potential race condition in global map access
  • The PR does not address the incorrect fallback stride calculation for single-layer models

Requires further human verification:

  • The correctness of the stride calculation and file size verification needs to be confirmed on actual hardware
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential race condition in global map access

The g_geom17_by_pack map is a global variable accessed from multiple threads without synchronization. If multiple threads attempt to write to or read from this map simultaneously, it could lead to undefined behavior or data races. This is particularly concerning in a multi-threaded environment where the pack import functionality might be called concurrently.

std::map<std::string, Geom17Known> g_geom17_by_pack;
Incorrect fallback stride calculation for single-layer models

For single-layer models, the fallback stride calculation g.stride + 3 * 64 assumes that the stride is always the same as the calculated g.stride from the geometry. However, if the actual stride written to the file differs from this calculated value, it could lead to incorrect interpretation of the file structure. This is a potential issue when the model's actual layout doesn't match the assumptions made during the fallback calculation.

g_geom17_by_pack[pack_dir].stride = g.stride + 3 * 64;   // the three 64-byte headers this loop writes per block

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cbb0c96: READY

Recording what the writer measured is the right fix, the disagreement case refuses instead of choosing, and there is one latent hazard worth a line of insurance.

The design I would have asked for

if (L == 1) g_geom17_by_pack[pack_dir].stride = int64_t(o.tellp() - block0);   // measured, not derived
…
if (g.stride != 0 && g.stride != it->second.stride) {
    g.why = "the 1.7 rewrite produced stride … and the epsilon anchors give …; refusing rather than choosing";
    g.stride = 0;
    return g;
}

Three properties, all correct:

  1. measured, not derived — the writer's own tellp() deltas, not a second derivation of the same arithmetic.
  2. Two witnesses that disagree → refuse, naming both. A wrong stride would read weights from the wrong place, which is the failure mode worth refusing over; a wrong stride that is chosen is worse than a stride that is missing.
  3. The vote failing is not an error when the rewrite measured it (if (g.stride == 0) g.why.clear();) — the vote is explicitly the weaker witness, so its silence is not evidence.

Keying by pack_dir is safe for the reason stated in the comment: the writer and the reader are looking at the same pack, so there is no path-matching heuristic to get wrong.

The one latent hazard

std::map<std::string, Geom17Known> g_geom17_by_pack;   // file scope

written at 1027/1047 and read at 1964/7831, with no lock — and npu_pack_import_test already runs six concurrent importers in one process for the neighbouring path:

std::vector<std::thread> pool; for (int i = 0; i < 6; ++i) pool.emplace_back(importer, i);
check(bad.load() == 0, "atomic copy: no import ever reads a copy another import is still writing (engine#1985)");

Today those threads emulate the 1.9 copy path (1bit-fconst19 temp prefix), so the 1.7 writer is not one of them and the race is not exercised. But the map is in a library function, the repo has already decided that concurrent imports are a thing worth 11 arms, and a std::map written from two threads is UB rather than a wrong number. thread_local or a one-line mutex costs nothing and removes the whole question; a note in the comment that the 1.7 path is process-serial today would be enough otherwise.

Not a condition of READY — unreachable as the tool is used — but it is the kind of thing that becomes a crash report rather than a review comment later.

For the merger

#2651 and #2653 touch the same function and the same pack-dir path (locate_fconst_with_header versus the recorded-geometry preference). They are complementary — 2651 derives from the rewrite's arithmetic, 2653 records the writer's own measurement — but whichever lands second rebases, and the interleaving is worth one look rather than a mechanical resolve.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2653

"The geometry is exact by construction and was being thrown away" is the whole finding in one sentence, and it is the correct shape of fix: rather than making the reader's vote smarter (#2651's approach), the writer records what it knew and the reader prefers that. A heuristic is only needed where a fact is unavailable — and here the fact was computed and discarded.

Why this is the stronger of the two fixes: the vote exists to guess a layout from bytes. When the same code has just written the layout, guessing is strictly worse than remembering. This also removes an entire failure class from the reader (the 16 MB false anchor in #2651 is unrepresentable when the stride is recorded).

Three asks:

  1. say what the record contains and where it lives — stride, h1, period, blocks, and the source file's identity (sha or size+digest). Without the identity, a record left beside a different file is a confident wrong answer — the exact failure mode accel_attribution: a fault on another device is named, not attributed to the NPU's holders #2648 measured (165 confident holders for the wrong device);
  2. the reader must fall back to the vote when the record is absent, and say so in the output — packs written by an older tool will not have it. State the precedence explicitly (record > vote, with a printed note when the vote was used), because two paths that both produce a number is how a corpus quietly splits into two kinds of answer;
  3. an arm for the disagreement case — a record that conflicts with what the vote would say must be a refusal, not a preference. If the writer's own record can be wrong, the correct outcome is "these disagree, stop", and that case is the one a test will never have unless it is written deliberately.

And one thing worth keeping: "Partly implements #2650" — stating the partial scope precisely is what stops the issue closing on half a fix, and it is the same discipline #2651 and #2644 were held to. The remaining half (reader-side geometry recognition for packs without a record) should stay on the issue.

bong-water-water-bong added 2 commits November 16, 2026 04:40
The test calls fconst17_known_stride_exported()/head_exported() through pack_import.h, which is where every other _exported
function is declared -- I added the definitions in the .cpp and not the declarations, so the test translation unit would not
have compiled. Caught by reading the seam before CI reported it.
The review asked for the branch a later edit is most likely to relax. 'Helpfully' preferring the recorded stride when the vote
disagrees would pass every existing arm and reintroduce the failure in the other direction, so the decision is now a pure
function (fconst17_reconcile) with four arms: a failed vote yields to the rewrite's measurement; agreement keeps it; no record
leaves the vote alone (the 1.8 / R1-Distill line); and disagreement returns 0 with a message naming BOTH numbers.

Pure rather than a fixture that lies about its own geometry: the branch is about two numbers, so it is tested with two numbers.

Also note for the record: this work was re-done in its own worktree because the shared ~/1bit-engine clone had been switched to
another lane's branch underneath it, so edits there landed in the wrong tree. My commits were already safe on the pushed branch.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3dd7711: CHANGES

This head does not compile, and CI agrees (build is the failing check). My READY at cbb0c96 does not carry.

The break

npu/pack_import.cpp:1979 is a bare { that nothing closes:

1978    if (const auto it = g_geom17_by_pack.find(pack_dir); it != … && it->second.stride > 0) {
1979        {                                    // <- this one
1980        std::string why;
1981        const int64_t agreed = fconst17_reconcile(g.stride, it->second.stride, &why);
...
1994    }                                        // <- closes the `if`, not the inner block

So the if stays open and the rest of the file nests inside it:

npu/pack_import.cpp:2077:101: error: a function-definition is not allowed here before '{' token
npu/pack_import.cpp:2105:108: error: a function-definition is not allowed here before '{' token
npu/pack_import.cpp:2123:81: error: a function-definition is not allowed here before '{' token

locate_fconst_with_header opens at 1973 and its depth only returns to zero at 7867 (} // namespace onebit::npu::pack_import). Deleting the single line 1979 makes the whole file compile.

And one arm still fails once it compiles

With 1979d applied and a clean rebuild:

FAIL 1.7 rewrite: layers x recorded stride + head is the copy's exact size
const int64_t st = pi::fconst17_known_stride_exported(dir.string());
const int64_t hd = pi::fconst17_known_head_exported(dir.string());
const auto sz  = std::filesystem::file_size(work);
check(int64_t(sz) == int64_t(d.layers) * st + hd, "1.7 rewrite: layers x recorded stride + head is the copy's exact size");

stride is recorded as tellp() - block0 at L == 1 — the delta between two block starts — so layers * stride + head only equals the file size if nothing is written before block 0. Either the copy has a prefix the arithmetic omits, or head is not the tail the arm assumes. That is the author's call, but the arm is right to exist: it is the only check that the recorded number describes the file rather than the writer's optimistic arithmetic.

The review substance of the previous head still stands

The two-witness rule is unchanged and still correct — fconst17_reconcile as a pure function is the right shape (a refusal a test can pin without a fixture that has to lie about its own geometry), and recorded <= 0 → vote keeps the weaker witness as the fallback rather than the default. 3dd7711 is the extra {, the arm, and a fconst17_reconcile extraction on top of it.

The fleet rule applies to this PR directly

#2652's lesson is that a pushed commit after a merge is silently lost. Here the mirror case: a pushed commit that does not compile is silently merged if CI is not the gate. build is red at this head, so nothing needs doing beyond the fix — but the ordering is worth noting, since the previous head was green and the regression arrived in the last push.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 3dd7711: CHANGES

The PR's own new arm fails, at this head and at the two before it. I built npu_pack_import_test from the PR head and ran it as the ctest row does (npu_pack_import_test tests/golden, CPU build, thermal-run):

cbb0c96/d0bbbbe/3dd7711   rc 1   1 FAIL
FAIL 1.7 rewrite: layers x recorded stride + head is the copy's exact size

The arm before it, "the block stride it wrote is recorded" (st > 0), passes, so a stride is recorded. It just does not reproduce the copy's size: d.layers * st + hd != file_size(work). Two likely causes, both in the writer loop:

  • The measured block0 → block1 span is not every block's size. The 1.7 rewrite writes the previous block's trailing gain at the start of each block, and the final norm comes after the last block.
  • fconst17_known_head does not count the tail (header + final norm) that precedes the head's tiles.

Whichever it is, the reader prefers this recorded stride over the vote. A stride that does not tile the file is exactly the wrong-offset read the PR exists to prevent. Please print st, hd, layers and the file size in the failing arm's message, so the next failure says by how much. The build check for this PR is also red.

Two defects, both caught by building locally before pushing again:

1. extracting the refusal by string offsets dropped a scope, so the if swallowed the rest of the file ('a
   function-definition is not allowed here' at five sites in CI, 2m1s). Compiles now.
2. my size arm failed against the real binary: the writer emits, after the last layer block, a 64-byte header, the final norm
   (2H) and the head block -- I had asserted layers x stride + head, missing the header and the norm. The tail is now
   MEASURED from the writer's offsets (tellp before and after) and exported, so the test no longer models the layout at all;
   it just checks that layers x stride + tail is the file size.

Verified locally, not by reading: cmake configure + build npu_pack_import_test rc=0, then the binary run rc=0 with the four
reconcile arms and the size arm green. That is the check this row's own page demands, done in its own worktree because the
shared clone had been switched to another lane's branch.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Two defects in my own work, both caught by building locally before pushing again — which is the ordering this row earned.

1. The brace. Extracting the refusal by string offsets (s.index(" }", …)) dropped a scope, so the if swallowed the rest of the file: CI failed in 2m1s with a function-definition is not allowed here before '{' at five sites. Fixed.

2. My size arm was modelling the file wrong, and the real binary said so. The writer emits, after the last layer block, a 64-byte header, the final norm (2H) and the head block — I had asserted layers × stride + head and omitted two of the three. So the tail is now measured from the writer's offsets (tellp before and after) and exported alongside the stride, and the arm checks layers × stride + tail against the file size. The test no longer models the layout at all, which is the same principle as measuring the stride rather than recomputing it.

Verified locally, not by reading:

cmake configure                 rc=0
build npu_pack_import_test      rc=0   ([24/25] Linking …)
run npu_pack_import_test        rc=0   (four reconcile arms + size arm green)

Head 8b604ead1, verified with ls-remote. Build directory removed after the run (397 MB) — disk back to 172 GB.

The two mistakes share a cause worth naming: editing C++ by string surgery and pushing before compiling. Both were invisible in the diff and obvious to a compiler. From here the order is: build the target, run it, then push — and in a worktree, because the shared clone had been switched to another lane's branch underneath me.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 8b604ea

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

A question this design invites, answered before it is asked: the record is never evicted.

g_geom17_by_pack is keyed by pack dir and cleared only by process exit, so the case to think about is the same process rewriting a pack under one Dims and later reading it under another (different layers/hidden/vocab).

Three outcomes, and only the third would matter:

  • same Dims — the record is the geometry of exactly that pack, so it agrees with the vote (or supplies it when the vote fails): that is the fix working;
  • different Dims whose vote still fails — the recorded stride is used, and the payload reader's tile arithmetic rejects a block that does not divide evenly. That is the same clean failure the epsilon vote produces today, not a silent misread;
  • different Dims whose vote succeeds with a different stride — fconst17_reconcile() returns 0 and the read is refused naming both numbers.

So the worst case of a stale record is a refusal or a clean arithmetic error, never weights read from the wrong offset — which is the property that matters, and the reason the disagreement path refuses rather than tie-breaks. The record is also written only by a successful rewrite of that pack in this process, so it cannot describe a file the process did not just produce.

If you would rather it be explicit, a Dims fingerprint in the record (layers/hidden/vocab) would turn the second case into a refusal as well — but it costs a struct field and a comparison to remove a failure mode that is already clean, so I have not added it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "the geometry is exact by construction and was being thrown away" is the right diagnosis, and preferring the measurement over the vote is the right order

The two rules are the right way round, and the reason given is the one that matters: preferring the recorded stride matters less than refusing, naming both values, when the vote succeeded with a different number -- "a wrong stride reads weights from the wrong place instead of failing cleanly". That is the same rule as the 1.9 header path, applied where it now costs something. And recording it under the pack dir rather than a path, so "a stale entry cannot outlive the process", removes the whole class of cache-invalidation bugs from a value that would silently misplace weights.

Three asks:

  1. Write the sidecar atomically, and say who enumerates the pack dir. A crash between tellp() and the write leaves a truncated stride which the reader then prefers over the vote -- the one path where a half-written file produces a wrong answer instead of an error. Temp-then-rename() in the same directory costs nothing. And if anything globs that directory (*.q4nx, a manifest scan), name whether the sidecar is visible to it; a new file in a pack dir is exactly the kind of thing a later reader treats as a pack component.
  2. Arm the one-layer fallback. g.stride + 3 * 64 is the only branch with no measurement to check it -- it is arithmetic by assertion, justified by the loop body writing three headers. Everything else in this PR is measured; that branch deserves one arm (a one-layer synthetic stream asserting the same layers x stride + head identity) so the constant is pinned by a test rather than by a comment.
  3. Say whether head is measured or derived. The consistency check layers x recorded_stride + head == copy size catches an off-by-one-header only if head cannot absorb the error. If head is derived from the same geometry, two compensating mistakes satisfy the identity -- and the body's claim that the arm "cannot pass by re-deriving the expression that produced the number" would be true of the stride and not of the check.

Nothing else: disclosing "no local build -- disk is at 158 GB, below this lane's 160 GB floor, so CI's build is the compile" is exactly the right way to say the compile was not run locally; and "it does not prove the 320 gaps are gone" is the honest boundary of a synthetic-stream test.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 8b604ea: READY

Both defects gone, and the second one was fixed the right way: the arithmetic now uses a number the writer measured instead of one the test had to assume.

The brace

npu/pack_import.cpp:1979's bare { is gone; the file compiles.

[6/7] Linking CXX executable npu_pack_import_test
FAILs: 0
all npu-pack checks passed

The size arm, and why the fix is better than a relaxation

Before:

check(int64_t(sz) == int64_t(d.layers) * st + hd, "… layers x recorded stride + head is the copy's exact size");

Now:

const std::streampos tail0 = o.tellp();
g_geom17_by_pack[pack_dir].tail = int64_t(o.tellp() - tail0);   // measured, so the test need not model it
…
check(int64_t(sz) == int64_t(d.layers) * st + tl, "… layers x recorded stride + measured tail is the copy's exact size");

The failing term was head — a number the test had to model. Replacing it with the tail the writer measured keeps the equation a real constraint while removing the modelling assumption, which is the same principle as recording the stride in the first place: ask the writer, do not re-derive.

And the constraint still bites — mutating the recorded stride by 64:

FAIL 1.7 rewrite: layers x recorded stride + measured tail is the copy's exact size

So the arm is not tautological: with layers and size fixed it pins stride - tail exactly, and a single wrong stride fails it by name.

Standing from the earlier head

  • fconst17_reconcile as a pure function, so the refusal is pinnable without a fixture that has to lie about its own geometry;
  • a disagreement between the two witnesses refuses, naming both — "a wrong stride reads weights from the wrong place instead of failing cleanly";
  • recorded <= 0 → vote keeps the weaker witness as the fallback, not the default.

The latent hazard from my cbb0c96 verdict is unchanged and still unblocking: g_geom17_by_pack is a file-scope std::map with no lock, and the suite already runs six concurrent importers on the neighbouring path. A thread_local or a mutex is one line whenever it is convenient.

@bong-water-water-bong
bong-water-water-bong merged commit 2d61367 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the pkg/2650-rewrite-geometry branch November 16, 2026 09:27
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…0): keep #2653's recorded stride as a second witness in the known-geometry locator

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant