Skip to content

npu-pack: locate a 1.7 headerless canonical copy from the rewrite's arithmetic, not the epsilon vote (engine#2650) - #2651

Merged
bong-water-water-bong merged 3 commits into
mainfrom
cl-npuperf/fconst17-known-geometry
Nov 16, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
cl-npuperf/fconst17-known-geometry

Conversation

@bong-water-water-bong

@bong-water-water-bong bong-water-water-bong commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2650

Ready at 9acc544. The device rows and the full ctest are below.

What

#2638 rewrites a 1.7 -onnx-ryzenai-npu Token file into its 1.8 form. The canonical copy was then located by locate_fconst()'s epsilon vote. The vote's anchor is 12 bytes (epsilon + 10 zeros) and occurs inside 4-bit tiles. On Phi-3-mini-4k one sits 16,232,526 B into block 0, the vote took it as the second header, and 320 tensors were refused. Phi-3.5-mini and Phi-3-mini-128k have the identical geometry and read with 0 gaps.

locate_fconst_with_header() now recognises a canonical copy of a 1.7 source by fconst17_geometry(). It sets stride, h1, period and blocks from the same arithmetic the rewrite used, then checks the copy's size and every block's two headers. It refuses by block when one is not where the rewrite put it. All other packs take the vote exactly as before.

Proof so far

  • npu_pack_import_test: new test_fconst17_known_geometry, a synthetic 3-layer 1.7 stream with an epsilon + 10-zero anchor planted 1,000 B into block 0's q tile.
    • control: the plain vote is fooled (h1 differs);
    • the importer's locator gives the exact stride and h1;
    • a copy with block 1's second header zeroed is refused, naming block 1.
  • Mutant: with the new branch disabled (if (false && ...)), both new arms FAIL ("stride 0, h1 0, the epsilon headers do not repeat with a single period"). Restored, it passes.
  • On a real pack: build/1bit was relinked with this change at 08:18:01Z, mid-sweep. The first pack read after that (08:18:14Z) is Llama-2-7b-chat-hf-onnx-ryzenai-npu. It went through the npu-pack: read the 1.7 -onnx-ryzenai-npu Token .fconst; eps and llama3 RoPE from the pack itself (Fixes #2634) #2638 rewrite (note printed), then through this locator: 0 gaps, import rc 0. On the NPU it answered "Paris" / "Great! 12 times 7 is equal to 84." / "Neil Armstrong ... Apollo 11 mission in 1969", all stop, 0.97-1.0 tok/s, dmesg clean. In that sweep, the three refused packs were read before the relink (Qwen2-7B at 08:16:51Z); each was then re-read and served on this head in the device proof below.
  • Second commit (9acc544): the zero-padded head. CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu was read after the relink and still refused. It never reached the rewrite because its file is 3,656,038,400 B, exactly 515 x 143,616-B tiles more than the arithmetic for vocab 32016 (501 head tiles). Llama-2-7b-chat fits that arithmetic to the byte. HTTP range reads of the published file show:
    • every layer's post and end gains sit where the arithmetic puts them (bf16-gain-like 0.95-1.00), through L31;
    • head tiles 0..499 are data and tile 500 is 76 % zero (16 of 64 rows);
    • tiles 501..1015 are all zero.
      The size gate now takes whole head tiles of padding (padded head <= 65,536 rows), and the rewrite drops them only after checking every byte is zero.
    • Arms: zero padding fits and rewrites to the 1.8 stream byte for byte; one non-zero byte means it is not rewritten; padding past 65,536 rows does not fit.
    • Mutants: no-pad, no-zero-check and no-bound each FAIL exactly their own arm(s). Restored, everything passes.

Device proof (head 9acc544)

The binary is build/1bit-2650: a separate static link of this head, so the running sweep's binary was not relinked. Each row is npu-hold, with dmesg clean before and after:

pack before this head
Phi-3-mini-4k-instruct-onnx-ryzenai-npu 320 gaps 0 gaps; "Paris" / "12 times 7 equals 84." / "Neil Armstrong ... July 20, 1969, during the Apollo 11 mission", all stop, 1.3-1.7 tok/s
Qwen2-7B-onnx-ryzenai-npu 280 gaps 0 gaps; Paris / 84 / Neil Armstrong, Apollo 11, 1969, all stop, 0.7-1.0 tok/s
CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu 320 gaps 0 gaps; Paris / "Sure! 12 times 7 is 84." / Neil Armstrong, Apollo 11, 1969, all stop, 0.7-0.85 tok/s

Phi-4-mini-instruct-onnx-ryzenai-npu is still refused, for a different reason. Its Token file is exactly 32 layer strides (1,758,724,096 B) with no head: the model is tied, so the export leaves the head out. Building the head from the embedding is new work and is tracked in #2658, so this PR doesn't wait on it.

Full ctest (9acc544, llama.cpp at its pin 148e7f3)

ctest -j 6 -E 'npu_dmabuf_import|_device|_npu_run' under thermal-run: 352/353 pass. The one row is npu_rt_shim (Not Run): its test binary does not link in this tree (undefined npu_rt_api::ext::bo::bo(device const&, int, unsigned long, void*)), and this PR does not touch it. An earlier run, made before I re-pinned this worktree's llama.cpp, also failed submodule_pin_check, a local checkout fault that the re-pin cleared. tool_help_check and npu_pack_import each failed once under -j 6 load and pass alone and on the re-run.

🤖 Generated with Claude Code

…rithmetic, not the epsilon vote (engine#2650)

The vote's 12-byte anchor (epsilon + 10 zeros) occurs inside 4-bit tiles. On
Phi-3-mini-4k-instruct-onnx-ryzenai-npu one sits 16,232,526 B into block 0 and
was taken as the second header: 320 gaps although the #2638 rewrite was right
(Phi-3.5 / Phi-3-128k, same layout, 0 gaps). Phi-4-mini and Qwen2-7B likewise.
The rewrite wrote every header, so the locator now takes the geometry from
fconst17_geometry() and only verifies each header is where it was put.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit ab01b06)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2650 - Partially compliant

Compliant requirements:

  • The canonical copy of a 1.7 headerless pack is now located from the rewrite's arithmetic, not by the epsilon vote
  • The fix handles zero-padded head tiles after the head
  • Documentation in docs/npu.md has been updated to reflect the new behavior

Non-compliant requirements:

  • The PR does not address the specific case of Phi-4-mini and Qwen2-7B models that were also refused the same way

Requires further human verification:

  • Verification that the fix correctly handles all models mentioned in the ticket description (Phi-3-mini-4k, Phi-3.5-mini, Phi-3-mini-128k, Phi-4-mini, Qwen2-7B)
  • Verification that the zero-padding logic works correctly for all edge cases
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Padding Size Check

The padding size check in fconst17_geometry() uses uint64_t(g->head) + (size - want) <= uint64_t(65536 / 64) * uint64_t(g->tile_h) which may not correctly validate the maximum padding allowed. This could lead to accepting files with padding that exceeds the intended limit.

if (size > want && (size - want) % uint64_t(g->tile_h) == 0 &&
    uint64_t(g->head) + (size - want) <= uint64_t(65536 / 64) * uint64_t(g->tile_h)) {
    g->pad = int64_t(size - want);
    return "";
}
Potential Logic Error in Geometry Validation

In locate_fconst_with_header(), the validation loop checks L <= dims.layers which could lead to checking an extra block beyond the actual number of layers, potentially causing false positives in the header validation. This is a potential logic error that could affect correctness.

for (int L = 0; size_ok && L <= dims.layers && bad < 0; ++L) {
    const int64_t b = int64_t(L) * g.stride;
    if (!eps_at(fc, b, dims.eps_bits) || (L < dims.layers && !eps_at(fc, b + g.h1, dims.eps_bits))) bad = L;
}
Test Coverage for Edge Cases

The test test_fconst17_known_geometry() creates a synthetic 1.7 stream but doesn't cover all edge cases. Specifically, it doesn't test what happens when the size of the canonical copy doesn't match the expected size based on the rewrite's arithmetic, or when the headers are completely missing from the copy.

        std::string nz = f17 + std::string(size_t(3 * T), '\0');
        nz[nz.size() - 7] = 1;
        const std::string tokn = (dir / "padded-nz.fconst").string();
        put_file(tokn, nz);
        check(pi::canonical_token_fconst17_exported(tokn, dir.string(), d, work + ".n") == tokn,
              "1.7 fconst: padding with a non-zero byte is not rewritten");
        check(!pi::fconst17_geometry_exported(f17.size() + 1024 * T, d).empty(),
              "1.7 fconst: padding past 65,536 head rows does not fit");
    }
    // the graph's RMSNorm epsilon (a node with a float attribute), read back
    std::string node, attr;
    pb::str(node, 4, "SimplifiedLayerNormalization");
    pb::str(attr, 1, "epsilon");
    pb::tag(attr, 2, 5);
    const float e = 1e-5f;
    char eb[4];
    std::memcpy(eb, &e, 4);
    attr.append(eb, 4);
    pb::ints(attr, 20, 1);   // AttributeProto.type = FLOAT
    pb::str(node, 5, attr);
    put_file((dir / "eps.onnx").string(), pb::model({}, {node, node}));
    int n_eps = 0;
    bool conflict = true;
    const float got_eps = pi::graph_norm_eps_exported((dir / "eps.onnx").string(), &n_eps, &conflict);
    check(got_eps == 1e-5f && n_eps == 2 && !conflict, "1.7 fconst: the graph's RMSNorm epsilon is read (" +
                                                           std::to_string(got_eps) + ", " + std::to_string(n_eps) + " nodes)");
    // #2638 review: two nodes that DISAGREE are a conflict, and the decision says so instead of falling back in silence
    const auto eps_node = [&](float v) {
        std::string nd, at;
        pb::str(nd, 4, "SimplifiedLayerNormalization");
        pb::str(at, 1, "epsilon");
        pb::tag(at, 2, 5);
        char vb[4];
        std::memcpy(vb, &v, 4);
        at.append(vb, 4);
        pb::ints(at, 20, 1);
        pb::str(nd, 5, at);
        return nd;
    };
    put_file((dir / "eps2.onnx").string(), pb::model({}, {eps_node(1e-5f), eps_node(1e-6f)}));
    int n2 = 0;
    bool conflict2 = false;
    (void)pi::graph_norm_eps_exported((dir / "eps2.onnx").string(), &n2, &conflict2);
    check(conflict2 && n2 == 2, "1.7 fconst: two RMSNorm nodes stating different epsilons are a conflict");
    std::string msg;
    const float kept = pi::graph_eps_decision(false, 1e-6f, 2, true, 1e-5f, &msg);
    check(kept == 1e-6f && msg.find("WARNING") != std::string::npos && msg.find("DIFFERENT") != std::string::npos,
          "1.7 fconst: on a conflict the configured eps is kept and a WARNING says why (" + msg + ")");
    const float taken = pi::graph_eps_decision(false, 1e-6f, 18, false, 1e-5f, &msg);
    check(taken == 1e-5f && msg.find("taken from the graph") != std::string::npos,
          "1.7 fconst: agreeing nodes give their eps, with a note");
    const float stated = pi::graph_eps_decision(true, 1e-6f, 18, false, 1e-5f, &msg);
    check(stated == 1e-6f && msg.empty(), "1.7 fconst: a stated config.json eps wins, silently");
    fs::remove_all(dir);
}

// engine#2650: the canonical copy of a 1.7 headerless pack is located from the rewrite's arithmetic, not by the epsilon
// vote. A 12-byte anchor (epsilon + 10 zeros) planted inside block 0's q tiles fools the vote -- the control -- and the
// importer's locator still gives the exact stride and second header.
void test_fconst17_known_geometry() {
✅ Resolved findings

npu/pack_import.cpp:1991-1994

Potential Logic Error in Geometry Validation

In locate_fconst_with_header(), the validation logic for checking if the canonical copy's headers are in the correct positions might be flawed. The loop checks L <= dims.layers which could lead to checking an extra block beyond the actual number of layers, potentially causing false positives in the header validation.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Review asks, no verdict (the PR declares itself a draft until the device rows land)

The mechanism is the strongest part and it should survive into the final description unchanged: "The vote's anchor is 12 bytes (epsilon + 10 zeros) and occurs inside 4-bit tiles. On Phi-3-mini-4k one sits 16,232,526 B into block 0, the vote took it as the second header, and 320 tensors were refused." That is a false positive of a heuristic, with its address — and the contrast ("Phi-3.5-mini and Phi-3-mini-128k have the identical geometry and read with 0 gaps") is what proves it is the anchor and not the geometry.

Four asks, in the order they decide the shape of the fix:

  1. a vote should never be decided by a single anchor — the epsilon vote found one 12-byte match at 16 MB into a tile and treated it as a header. Whatever the final form, the vote needs either a second corroborating field (the bytes after the anchor must parse as a plausible header) or a position bound (a real header lies within the first N bytes of the block). One anchor anywhere in the file is a coin flip over 16 MB of tile data;
  2. state how many packs the new geometry check fixes and what it would reject — fconst17_geometry() recognising a canonical 1.7 source by arithmetic is a good discriminator, but it is now the thing that decides which rewrite path runs. Two lists: the packs that read with 0 gaps because of it, and any pack whose geometry matches but is not 1.7 (a false accept would rewrite a file that needed no rewrite);
  3. the queued device rows must include a negative case — the three proof rows are all known-good packs. Add one that the old vote refused and the new geometry check does not claim (i.e. it must still refuse, cleanly), or the change is only proven for the packs that motivated it;
  4. the ctest/build/1bit contention is worth an explicit note — "rebuilding everything relinks the binary that the batch is serving from" is a real constraint that will bite the next lane too. One line in the description saying the ctest is deferred for that reason (rather than "pending") tells a reader the deferral is deliberate.

What I am not doing: recording a verdict. The proof rows are queued, the ctest is deferred behind a serving binary, and the author said so — that is exactly the case where a READY on code reading would be worthless.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 969af37: READY on the mechanism and the arm; the device rows are the author's own open item and I am not clearing a draft without them.

The arm is the best-formed test I have reviewed tonight

for (auto& ch : b) { seed = seed * 1103515245u + 12345u; ch = char((seed >> 16) | 1); }   // never 0: no anchors
…
f17[1000] = char(d.eps_bits & 0xFF), f17[1001] = char(d.eps_bits >> 8);   // the coincidence, planted on purpose

Three properties, each of which most arms lack:

  1. The background is provably anchor-free — | 1 forces every random byte odd, so no accidental epsilon + 10 zeros can exist and the only anchor in the file is the one the test planted. Without that, a pass could mean "the coincidence did not occur in this seed".
  2. The coincidence is planted at a computed place — 1000 B into q's first tile of block 0, built from the same attn_tile_bytes/group_bytes arithmetic the locator uses, so the fixture and the code agree about geometry by construction rather than by coincidence.
  3. The failing method is run as a control — "a 12-byte anchor … fools the vote -- the control -- and the importer's locator still gives the exact stride and second header". That is the difference between an arm and a demonstration: it proves the fixture reproduces the bug, so the new method passing means something.

The pathology itself is worth the fix on its own: a 12-byte signature inside a file made of packed 4-bit tiles is not a discriminator, and the measurement (16,232,526 B into block 0, 320 tensors refused) is the kind of number that turns "possible" into "happened".

One ask: prove the locator does not consult the epsilon at all

The claim is "not by the epsilon vote", but the fixture plants the anchor using this model's epsilon (rms_norm_eps = 1e-5f → eps_bits). A model with a different epsilon plants a different 12-byte anchor, and the arm as written would still pass — so it demonstrates "the planted anchor does not fool the locator", not "the locator never looks at eps_bits".

Two cheap ways to close that, either is enough:

  • a second arm with a different epsilon (1e-6, which Phi-3.5 uses) and the same planted geometry — if both pass, the arithmetic is what locates;
  • better: an arm where eps_bits is deliberately wrong — set to something no tile contains — and assert the locator still returns the same stride and second header. That one cannot pass if any code path consults the epsilon, which is exactly the claim in the title.

And the draft state, since it matters for what this verdict is worth

"WIP / draft until the device rows land … The full ctest runs once build/1bit is free, since rebuilding everything relinks the binary that the batch is serving from." That is the right sequencing — the row that proves it on a real Phi-3-mini-4k is the one that counts, and rebuilding under a running batch would be the wrong trade. My READY covers the locator arithmetic and the arm; it does not stand in for those rows, and I would want to see them before this leaves draft.

The one-line lesson worth putting in docs/npu.md

A fixed byte pattern is not a locator in a file whose payload is dense in that pattern's alphabet. The 4-bit tiles contain arbitrary byte pairs, so any short magic will appear; a locator has to come from arithmetic the format already guarantees.

The page currently describes the fix; that sentence is why the next person does not add a third vote with a longer anchor.

…-Instruct, engine#2650)

CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu is 515 x 143,616-B tiles larger than
the arithmetic (vocab 32016 = 501 head tiles); range reads show those 515 tiles
are all zero, and every layer's gains sit where the arithmetic puts them. The
size gate now takes whole head tiles of padding (padded head <= 65,536 rows);
the rewrite drops them only after checking every byte is zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 969af37: READY on the design and the arms. It is a draft for a good reason, it carries the control that most detector changes skip, and it bypasses the detector it knows to be fooled rather than teaching it a new trick.

The control is the part worth reading

"test_fconst17_known_geometry, a synthetic 3-layer 1.7 stream with an epsilon + 10-zero anchor planted [inside block 0]: control: the plain vote is fooled (h1 differs); the importer's locator gives the exact stride and h1; a copy with block 1's second header zeroed is refused, naming block 1."

Three arms and the first one is the one that is usually missing: the old method is demonstrated to fail on the input the new method handles. Without it, a reviewer cannot tell whether fconst17_geometry() fixes a real failure or simply adds a second path that happens to work on a fixture. With it, the change is justified by the other method's observable failure on the same bytes.

The third arm is the second-best: a malformed copy is refused, naming block 1 — a locator that cannot say which block it distrusted makes a 16 MB offset hunt.

The known-fooled detector is not asked, which is the right resolution

// a 1.7 headerless pack's canonical copy was WRITTEN by canonical_token_fconst17(), so its geometry is
// known exactly; the epsilon vote is not asked.
if (!src.empty() && !ec && tok != src && fconst17_geometry(src_size, dims, &g17).empty()) { … }

For the class where the vote is known to be wrong the code does not consult it at all — it computes the geometry from what the writer did and validates the result. That is the difference between adding a detector and removing a bad one's authority, and it is the choice I would have argued for: a heuristic that is fooled by a 12-byte anchor planted in the q tiles should not be the last word on a file whose layout we wrote ourselves.

The real-world instance is stated with its offset — "Phi-3-mini-4k-instruct-onnx-ryzenai-npu's anchor occurs 16,232,526 B into block 0, and the vote took it as the second header" — which is what makes the synthetic control a model of a real file rather than a convenient one.

The export exists for the test, and that is fine

locate_fconst_with_header_exported() is a thin shim that lets the test drive the locator without a device. Naming it _exported rather than making the production symbol visible keeps the seam obvious, and it is why three arms can run in a CPU-only test.

The draft status is honest and correctly scoped

"WIP / draft until the device rows land. The proof rows for Phi-3-mini-4k, Phi-4-mini and Qwen2-7B are queued behind the running [NPU job]."

Three named models, a stated reason for waiting, and no claim about rows that have not run. The host-side arms stand on their own; the draft label is about the device proof, which is the right thing to be missing.

One note, not an ask

When the device rows land, the number worth recording beside them is the one that would have been wrong under the vote: for Phi-3-mini-4k, h1 taken from the anchor at 16,232,526 B versus the true header. That single pair of numbers is what a future reader needs to recognise the failure if it ever returns, and it costs one line in the PR body.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9acc544: READY

The fix is the right shape — stop voting on where a header is, when the rewrite that wrote it knows exactly — and the numbers in the header are what make it believable.

The bug, as measured: the epsilon vote takes block 0's first two anchors as the header pair, and "that pattern can occur inside 4-bit tiles" — on Phi-3-mini-4k-instruct-onnx-ryzenai-npu one occurs 16,232,526 B into block 0, the vote took it as the second header, and the pack was refused on 320 tensors although its rewrite was right. Phi-3.5-mini and Phi-3-mini-128k have the identical layout and read with 0 gaps; Phi-4-mini (320) and Qwen2-7B (280) were refused the same way.

That contrast is the strongest evidence in the PR: identical layout, different payload, opposite verdicts — which only a content-dependent heuristic can produce, and which is exactly what the vote is. A layout bug would have failed all four.

What replaces it: canonical_token_fconst17() wrote the copy, so its geometry is known and the locator checks each header is where it was put, refusing by block when it is not:

g.why = "the 1.7 headerless pack's canonical copy is not where its rewrite put it (block N's headers | the size)"

That is a check with a defined right answer instead of a vote between candidates — and the refusal names the block, so a real corruption is diagnosable rather than reported as a tensor count.

The arms cover both directions, which is what I look for on a heuristic removal:

  • a zero-padded head rewrites to the 1.8 stream byte for byte (the false-refusal case — the pack that used to be rejected);
  • a copy with a non-zero byte in the pad is handled (the padded-nz case);
  • f17.size() + 3*T fits (whole zero tiles after the head) while +1024*T does not — the geometry still bounds the size, so "known geometry" did not become "any size accepted".

Not verified: I did not build or run npu_pack_import_test — disk is at 158 G, under the 160 G floor I build under — so the 320-tensor and 0-gap figures are the author's, and the arms above are what I read rather than what I ran.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 9acc544

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 969af37: READY

The root cause is stated with the coincidence that produced it, and the test proves the coincidence is real before proving the fix works.

The cause, as stated

"The vote takes block 0's first two anchors as the header pair, and its 12-byte anchor (epsilon + 10 zeros) occurs inside 4-bit tiles: on Phi-3-mini-4k-instruct-onnx-ryzenai-npu one did, at an odd offset 16 232 526 B into block 0, and the pack was refused on 320 tensors ("attention payload is 16 226 318 B, not a whole number of 108 800-byte tiles") although the rewrite was right — the identical layout of Phi-3.5 and Phi-3-128k read with 0 gaps."

That is a complete diagnosis: a 12-byte pattern (epsilon followed by ten zeros) is not a rare enough fingerprint to identify a header inside 4 KiB of 4-bit weights, one false match is enough to move h1, and the failure surfaces 320 tensors later as an arithmetic complaint that names the payload rather than the locator.

And the fix is the right shape: the canonical copy was written by canonical_token_fconst17(), so its geometry is known by construction — asking a heuristic to rediscover what the writer knew is how the whole class of failure entered.

The control, which is the part I did not expect

// the coincidence: epsilon then 10 zeros, 1000 B into q's first tile of block 0
check(said.empty() && vote.h1 != h1, "known geometry: the planted anchor fools the epsilon vote (h1 …");
check(g.stride == stride && g.h1 == h1 && g.hdr.size() == 1 && g.why.empty(), …);

The first check asserts the bug still reproduces — the planted anchor really does move the vote's h1 — and the second asserts the new locator is unaffected. A test that only checked the second would pass on a day when the vote happened to be right, and would keep passing after the vote changed. This one cannot.

Two more properties are armed: a copy whose header is not where the rewrite put it is refused (so the arithmetic path is not simply permissive), and the synthetic stream is generated with ch = char((seed >> 16) | 1) — "never 0: no anchors" — so the only anchor in the fixture is the planted one.

The mutant

locate_fconst_with_header → return FconstGeom{}:

FAIL known geometry: the canonical copy is located from the rewrite's arithmetic (stride 0, h1 0, )
FAIL known geometry: a header missing from the copy is refused by block ()
FAIL hybrid import: a pack whose geometry is only in text_config imports, tail envelope included
FAIL post-attn header: the import read the crafted stride

Four by name, including the refusal arm — the mutation does not merely lose the new path, it also removes the check that the other paths still enforce.

all npu-pack checks passed.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 9acc544

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 9acc544

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9acc54: READY

The padded-head tolerance is bounded twice — by tile count and by the all-zero check — and the zero check bites.

The shape

CodeLlama-7b-Instruct-hf-onnx-ryzenai-npu: vocab 32 016 -> 501 head tiles, the last holding 16 rows,
then 515 more tiles of zeros. 1 016 tiles, 65 024 rows, while its 32 layer blocks sit exactly where
the arithmetic puts them.

So the size gate now accepts whole head tiles after the head, and it does two things that keep that from becoming a hole:

  1. A ceiling: "up to 65 536 padded head rows … so an unrelated larger file is not [accepted]". Without it, "size ≥ expected" would be the gate, which accepts anything.
  2. Every padded byte must be zero — "with one non-zero byte the file is not rewritten."

Mutant — accept the padding without the zero check:

FAIL 1.7 fconst: padding with a non-zero byte is not rewritten

fails by name. That is the arm that matters: it is the difference between "this export pads its head" and "this file happens to be longer", and it is checked on the same fixture that proves the padding is real.

How this sits with the rest

g->pad is reported out of the geometry function and the zero check lives in the caller (pad is "whole head tiles after the head … must be all zero"), so the arithmetic stays a pure predicate and the refusal stays where the bytes are. The arm count grew by two and nothing else moved; all npu-pack checks passed.

My earlier verdict at 969af37 stands unchanged for the locator itself — including the control that asserts the planted anchor still fools the epsilon vote before asserting the new path is unaffected.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9acc544: READY

I built this head's npu_pack_import_test myself, ran it, and ran the PR's own mutant. I did no device run and no full ctest; the device table and the 352/353 are the author's.

Ran

I made a fresh worktree at 9acc544 and configured it on its own (Release, Ninja, out of tree), building only npu_pack_import_test under thermal-run.

  • npu_pack_import_test tests/golden gives rc 0, "all npu-pack checks passed". The new known-geometry arms and the three padding arms are in that run:
    • zero padding is rewritten;
    • a non-zero byte means no rewrite (padded-nz.fconst ... those bytes are not zero, so it is not rewritten);
    • padding past 65,536 rows does not fit.
  • Mutant: I disabled the new locator branch with if (false && !src.empty() && ...) at pack_import.cpp's new block in locate_fconst_with_header, rebuilt, and confirmed the binary rebuilt. It gives rc 1, 2 checks failed, exactly the two the body names:
    • known geometry: the canonical copy is located from the rewrite's arithmetic (stride 0, h1 0, the epsilon headers do not repeat with a single period)
    • known geometry: a header missing from the copy is refused by block
    • Source restored; the diff is empty.
  • The test left nothing under /tmp/fconst17*.

Read

  • fconst17_geometry padding:
    • Only whole tile_h multiples after the head are accepted, and only while head + pad stays within 1,024 tiles (65,536 rows). A vocabulary above 65,536 rows therefore gets no padding allowance, and its exact-size path is unchanged.
    • canonical_token_fconst17 drops the padding only after checking every byte is zero. A non-zero byte leaves the file to the 1.8 reader. That is the right failure direction: it refuses rather than silently truncating.
  • locate_fconst_with_header:
    • The new branch fires only when the SOURCE Token file fits the 1.7 arithmetic and tok != src, i.e. it is reading the rewrite's canonical copy.
    • When a rewrite was refused (non-zero padding), tok == src and the vote runs as before, so other packs keep today's behaviour.
    • The expected size is layers x stride + env + head, with no padding. That matches what the rewrite writes once the padding is dropped, so CodeLlama's padded source and its unpadded copy agree.
    • Every block's two headers are checked, plus the final block's leading one (L <= layers). A miss is refused by block number.

Non-blocking

The arithmetic now exists twice: once in canonical_token_fconst17 (the writer) and once in the locator's g.h1 / g.stride (the reader). One helper returning FconstGeom from Fconst17Geom would stop the two drifting apart. The known-geometry arm would catch a drift today, so this is optional.

…0): keep #2653's recorded stride as a second witness in the known-geometry locator

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at ab01b06

Change since 9acc544: a merge of origin/main (not a rebase: the reviewed commits are kept, and no force push was needed). Main had picked up #2653, which reaches #2650 by a second route: the writer records the stride from its own offsets, and fconst17_reconcile checks the vote against it. Only pack_import.h conflicted, and both sets of declarations are kept.

How the two routes now fit together in locate_fconst_with_header:

Re-run on the merged tree:

Not re-run: the device rows. The binary they ran on contained this PR's locator and the padded-head change. The merge adds only the agreement check and #2653's code, and neither touches the bytes read.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the control that shows the old vote is fooled is what makes this reviewable, and one case is still unproven

The evidence has the right shape: a synthetic 3-layer stream with the epsilon-10-zero anchor planted 1,000 B into block 0's q tile, then the control -- "the plain vote is fooled (h1 differs)" -- before the new locator gets it right. That is the difference between a fix and a claim: the arm demonstrates the old path failing on the same bytes the new path handles. Plus a named refusal (a copy with block 1's second header zeroed is refused, naming block 1) and a mutant that makes both arms fail when the branch is disabled. Deriving stride/h1/period/blocks from fconst17_geometry() -- the same arithmetic the rewrite used -- rather than re-fitting them is also the right choice: the copy is a product of that arithmetic, so recognising it should test that identity rather than search for a plausible one.

The ask, and it is the only thing standing between this and "proven":

  1. The failing pack itself has not been re-read. "The three refused packs were all read before th[e relink]" -- so the device run that proves the fix is Llama-2-7b-chat-hf, a different pack, and Phi-3.5-mini / Phi-3-mini-128k are argued by geometry. Phi-3-mini-4k is the case with the 320 gaps inside block 0's q tile at 16,232,526 B, and it is the one whose behaviour changed. If that pack is on the box, re-reading it and reporting 320 -> 0 is one command and closes the issue; I checked ~/models and it is not there, so the ask becomes: say that plainly in the body (the failing pack is absent, so the closure rests on geometry + a different pack), or fetch it. Right now a reader will assume the fix was demonstrated on the failing case, and the sentence "the three refused packs were all read before the relink" is easy to miss.
  2. Say whether the geometry check can pass on a non-1.7 source. fconst17_geometry() is applied when the copy is recognised as a 1.7 canonical form; if a 1.9 file could coincidentally satisfy size == layers*stride + head with its two headers where expected, the vote would be skipped for a pack that needs it. One sentence on what distinguishes the two, or on why the size-plus-header test cannot coincide, is the cheap version of the arm I would otherwise ask for.

Nothing else: All other packs take the vote exactly as before keeps the blast radius honest, and 0.97-1.0 tok/s on the NPU with a clean dmesg is the right device row for a correctness fix -- the number is not the claim.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at ab01b06: READY (second verdict; the first was READY-on-design at 969af37, when the device rows were still queued). The draft is lifted and the evidence now covers all three packs — including the one that still fails.

The device proof is the right shape: gaps to zero, with the answers quoted

pack before this head
Phi-3-mini-4k 320 gaps 0 gaps; "Paris" / "12 times 7 equals 84." / "Neil Armstrong … July 20, 1969"
Qwen2-7B 280 gaps 0 gaps; Paris / 84 / Neil Armstrong, Apollo 11, 1969, all stop, 0.7–1.0 tok/s
CodeLlama-7b 320 gaps 0 gaps; Paris / "Sure! 12 times 7 is 84." / Neil Armstrong, Apollo 11, 1969

Three packs, three counts to zero, and the answers quoted rather than a pass/fail — a gap count of zero with wrong answers would be worse than the gaps. Qwen2-7B's decode rate is stated (0.7–1.0 tok/s) instead of omitted, which is the detail someone might otherwise ask about later.

And Phi-4-mini is still refused, for a different reason — its Token file is exactly 32 layer strides. A PR that says which input it does not handle, and why the reason changed, is more useful than one that quietly narrows its own claim.

The measured binary is identified, and the sweep was not disturbed

"The binary is build/1bit-2650: a separate static link of this head, so the running sweep's binary was not relinked."

That is gpu-kernels#107's lesson applied to a parallel process: the failure mode there was measuring an object that the build never rebuilt; the failure mode here would be relinking the binary a 24-hour sweep is using. The body first admits build/1bit was relinked mid-sweep at 08:18:01Z and then fixes the method — an admission and a correction in the same PR.

Three mutants, each failing its own arm

no-pad, no-zero-check, no-bound each FAIL exactly their own arm(s) — and the first branch has its own mutant (if (false && …) ⇒ both new arms fail). That is one more mutant than the arms, which is the right ratio: an arm whose mutant does not bite is an arm that cannot.

The second commit's arithmetic is checked semantically, not just structurally

  • every layer's post and end gains sit where the arithmetic puts them (bf16-gain-like 0.95–1.00) through L31 — a check that the parse means something, not only that it parses;
  • head tiles 0..499 are data, tile 500 is 76 % zero, tiles 501..1015 all zero ⇒ the padding is real and bounded;
  • the size gate takes whole head tiles (padded head ≤ 65,536 rows) and drops them only after the zero check;
  • arms: zero padding rewrites byte for byte, one non-zero byte means no rewrite, padding past the bound is refused.

Full suite

ctest -j 6 -E 'npu_dmabuf_import|_device|_npu_run' under thermal-run: 352/353, the single non-pass being npu_rt_shim (Not Run — the known link failure that has been red on main all night, unrelated to this change).

Nothing to change

My design point from the first verdict stands and is now exercised on real packs: for a 1.7 source the locator computes the geometry from what canonical_token_fconst17() wrote and validates it, rather than asking the epsilon vote that a 12-byte anchor can fool. Three packs that used to fail now read, and the one that does not says why.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at ab01b06 — and the arm is the reason: it reproduces the bug, proves the fix, and pins the refusal, in one function.

The arm does three things, and the first is the one most tests skip

check(said.empty() && vote.h1 != h1, "… the planted anchor fools the epsilon vote (h1 N vs M) -- the control");
check(g.stride == stride && g.h1 == h1 && g.hdr.size() == 1 && g.why.empty(),
      "… the canonical copy is located from the rewrite's arithmetic …");
check(gb.stride == 0 && gb.why.find("block 1") != std::string::npos,
      "… a header missing from the copy is refused by block");

The control asserts the vote is WRONG — vote.h1 != h1 — rather than only asserting the new path is right. That is what makes the test evidence about the bug: the synthetic stream plants the epsilon + 10 zeros anchor 1,000 B into block 0's q tile, and the test fails if a future change to locate_fconst() happens to make the vote land correctly, because then it is no longer demonstrating the defect it exists for. A test that only checked the fix would keep passing if the vote regressed into the same answer by luck.

The negative arm is the other half: a copy whose block-1 header was moved is refused, and refused with a reason that names block 1 rather than "located some other way". gb.stride == 0 plus a substring on the reason is the pair I would have asked for — a refusal that says nothing would pass a stride == 0 check on its own.

Why the mechanism is believable

The anchor is a 12-byte coincidence, and the fix replaces a vote with arithmetic derived from the same rewrite that wrote the file:

  • fconst17_geometry() sets stride, h1, period and blocks from the rewrite's own numbers, then checks the copy's size and every block's two headers.
  • Refusal is by block, so a caller learns which header disagreed.
  • Other packs are untouched — the vote is still the path for every pack that is not a 1.7 headerless canonical copy, and the padding arm (padding past 65,536 head rows does not fit, and a non-zero padding byte not rewritten) pins that the new size rule has bounds rather than accepting whatever follows the head.

The measured symptom matches the stated cause: one anchor at 16,232,526 B in Phi-3-mini-4k block 0 was taken as the second header and 320 tensors were refused, while Phi-3.5-mini and Phi-3-mini-128k — identical geometry — read with 0 gaps. A 12-byte coincidence that appears in one model's q tiles and not its siblings is exactly what an epsilon vote cannot survive and arithmetic can.

What I did not ask for

Nothing. The negative arm plus the control covers the failure directions I would have written into a review request, and the why field carrying the block number means the next person debugging a refusal gets the block rather than a boolean.

blocked is build in progress only.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ab01b06

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ab01b06: READY

The claim I could not check last time now cross-checks against a recorded failure, the arithmetic closes exactly, and the new test carries a control that makes the old mechanism fail — which is the form this fix needed.

The central number is consistent, and I checked it two ways

The doc says the 12-byte anchor (epsilon + 10 zeros) occurs 16,232,526 B into block 0 of Phi-3-mini-4k-instruct. The recorded gap message from the failing pack is:

attention payload is 16226318 B, not a whole number of 108800-byte tiles
16,232,526 - 16,226,318 = 6,208 B   -- exactly the fconst envelope the 128k import log names ("6208 B envelope")
16,226,318 / 108,800    = 149.139  -- not a whole tile, which is the gap message verbatim

So the vote's "second header" lands 6,208 B past the true payload end — inside block 0's q tiles — and the two independent numbers differ by exactly the envelope size. That is the kind of consistency a copied number does not have, and it is why I believe the diagnosis rather than just the sentence.

The padding form closes too

vocab 32,016 / 64 = 500.25      -> 501 head tiles
501 + 515 padded  = 1,016 tiles = 65,024 rows  <= the 65,536-row bound

And the gate refuses on the things that would make it wrong: padding past 65,536 rows, or any non-zero byte in the padding (zero = true scan, then the file is left to the 1.8 reader). A padded file is not silently taken for a 1.7 pack.

The test is the right shape: it fails the old mechanism

// A 12-byte anchor (epsilon + 10 zeros) planted inside block 0's q tiles fools the vote
//   -- the control -- and the importer's locator still gives the exact stride and second header.

A test that only shows the new locator working would not prove the vote was wrong. This one plants the collision, shows the vote taken in by it (the control), and shows the arithmetic unaffected. The fixture also fills q tiles with | 1 so no anchor can appear by accident — the trap that makes such a fixture vacuous is closed deliberately.

Plus the negative arm: "a copy whose headers are not where the rewrite put them is refused, not located some other way" — that is the property that keeps the locator from becoming a best-effort search.

Not verified

The arms themselves (I did not build; a review does not take a build tree at 162 G free and build/hw_gate are the CI's job), and the 1.8 side of "left to the 1.8 reader" — I am trusting that reader's refusal rather than re-running it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Re the review at 10:01Z ("the failing pack itself has not been re-read"): it was re-read and served, after the sentence you quoted was written.

Phi-3-mini-4k-instruct-onnx-ryzenai-npu was downloaded again as published and run through build/1bit-2650, a separate static link of 9acc544 (batch2650, npu-hold, dmesg clean before and after):

  • npu-pack info: 320 -> 0 gaps;
  • import rc 0;
  • on the NPU: "The capital of France is Paris." / "12 times 7 equals 84." / "Neil Armstrong. He achieved this historic milestone on July 20, 1969, during the Apollo 11 mission", all stop, 1.3-1.7 tok/s.

Qwen2-7B (280 -> 0) and CodeLlama-7b (320 -> 0) were run the same way. The body's "Device proof (head 9acc544)" table has all three. The pack isn't in ~/models because the row script deletes each pack after its row is written (brief rule).

The sentence "the three refused packs were all read before the relink" describes the 08:18Z sweep binary only. I've reworded it in the body so it can't be read as covering the proof runs.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

One shape your new arithmetic will meet, and currently refuses with the wrong reason — from #2658 (a tied Phi-4-mini pack), with the numbers verified against your own constants:

fconst17_geometry() computes want = layers x stride + head and refuses when size != want. A model that ties lm_head to the embedding exports no head tiles, so its Token file is exactly layers × stride:

stride      = 54,960,128        (your arithmetic, hidden 3072 / IM 8192 / 24-8 heads / hd 128)
32 x stride = 1,758,724,096     = the file's exact size
head        = 3,126 x 108,800 = 340,108,800 B   <- absent

So the geometry path declines a stream whose stride it has just computed correctly, and the caller falls through to the epsilon vote, which reports "fewer than 6 RMSNorm epsilon headers in the first GiB" — a message that names a cause the pack does not have.

Suggestion, if you agree: accept size == layers × stride as a named variant (g->head = 0 plus a flag), so the tied case is placed and the refusal can say "the pack carries no head tiles" rather than blaming the epsilons. That is a small addition to what you already have, and it keeps the shape of the function — it knows the stride, so being short by exactly a head is distinguishable from a wrong geometry.

Not asking you to take the head-synthesis half; that is #2658's work and needs the pack reproduced first. I am only flagging that your want as written cannot place a tied 1.7 pack, and that the fallback message sends the reader to the wrong machinery.

@bong-water-water-bong
bong-water-water-bong merged commit 3a0b155 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-npuperf/fconst17-known-geometry branch November 16, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

npu-pack: a 1.7 -onnx-ryzenai-npu canonical copy is located by the epsilon vote, and a 4-bit tile coincidence breaks Phi-3-mini-4k (320 gaps)

1 participant