Skip to content

kernel-set-check: a documented _m16 twin is a WARN, anything else is a FAIL (re-lands the commit #2605 lost) - #2654

Merged
bong-water-water-bong merged 2 commits into
mainfrom
check/elf-twins-warn
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
check/elf-twins-warn

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Re-lands a commit that was pushed to #2605 after it merged (01:38:26Z) and was therefore lost — the exact case the fleet rule in engine#2652 now names. Verified against main, not the PR badge:

$ git show origin/main:scripts/kernel-set-check.sh | grep -c 'documented _m16 twin'
0
$ git show origin/main:scripts/kernel-set-check.sh | sed -n '/check_twins/,/^}/p' | grep 'FAIL'
        printf 'FAIL %s: ELFs with identical content under different names (a copied twin computes the wrong M):

So main still has the pre-split form, which FAILs any identical-content pair — including a documented _m16 twin.

Why that is wrong now

engine#2611 changed open_prefill to size the context from the twin's 16 rows when the bytes match, and to note it by name. That pack therefore runs — correct, merely 8× the prefill cost per chunk. A validation that refuses it blocks a pack the engine supports.

The split

By name, in check_twins():

  • a stem and its own _m16 sibling ⇒ WARN (byte-identical ELF twin(s) this build tolerates but does not exploit, naming both paths and why the loader accepts it);
  • any other identical-content pair ⇒ FAIL (… NOT a documented _m16 twin), because nothing reconciles those: no name relationship, no loader behaviour that makes the content correct.

The warning is deliberately not silent — a pack running at 8× the prefill cost should surface at validation, not only in a loader log line.

Arms

ok  distinct ELFs are not reported as twins
ok  two ELFs with the same content are refused          (fixture: O vs GU, NOT a twin pair)
ok  ... and both paths are named
ok  a byte-identical _m16 twin pair is warned about, not refused
kernel_set_check: all cases ok

The fixture that had encoded the old intent is the interesting one: the refusal arm originally used the _m16 shape, so it started failing the moment the WARN was added — which is exactly what an arm is for. It now uses full_i8_O_… vs full_i8_GU_…, and a new case asserts the documented twin is warned about rather than refused.

Verified locally: bash -n clean, tests/kernel_set_check_test.sh all cases ok, tools/copyright.py --check clean (no new file this time, but checked).

Not claimed

This does not catch a 16-row design compiled under a fresh name — new bytes, correct-looking name, wrong M. Nothing host-side can see that (the AIE image carries no readable design M: e_shnum = 0, no symtab, no design string), and the durable fix is for the packer to record {M, K, N, bits} so a loader can assert metadata.M == MD.

…a FAIL

Re-lands the split that was pushed to #2605 AFTER it merged (01:38:26Z) and was therefore lost: main
still has the pre-split form, which FAILs even a documented _m16 twin pair. That now contradicts
engine#2611, whose open_prefill sizes the context from the twin 16 rows and says so -- the loader runs
that pack, so refusing it at validation would block something the engine supports.

So the two shapes are separated by name: a stem and its own _m16 sibling are a WARN (correct but slow,
16 rows a chunk), and any other identical-content pair is still a FAIL because nothing reconciles those.

Arms updated to match, including the one that had encoded the old intent (its fixture used the twin
shape). kernel_set_check: all cases ok.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2654

"Re-lands a commit that was pushed to #2605 after it merged (01:38:26Z) and was therefore lost" — so the rule arrived, the case was already in flight, and this is the repair. That is a good sign about the rule: it is being consumed by the people it describes.

Three asks:

  1. name the missing lines in the description — a restore PR is the one place a reader can check the claim without archaeology, and ci: merge_content_check -- ask main for the work, not the PR badge (engine#2652) #2655's diff-based approach would produce exactly that. Which of kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605's behaviours is this restoring (the ELF-twin warn rather than refuse?), and what does the arm assert that the merged version does not;
  2. the warn/refuse distinction is the substance, so state it explicitly — my kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) #2605 review asked that a twin ELF refusal say what to do (a duplicated ELF usually means a build mistake). If this re-land turns the check into a warning rather than a refusal, that is a policy change riding inside a restore: it needs its own sentence and its own justification, because "restored the commit" and "softened the check" are different claims;
  3. confirm tests/kernel-set-check covers the soften-the-rule case — if twins now warn, the arm must assert that a twin still fails the build when it matters (or that the warning is visible where a build log is read). A warning nobody sees is a rule removed.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 61ed6cb: CHANGES

The WARN for a documented _m16 twin is right, and re-landing the lost commit is the right move. But "anything else is a FAIL" still refuses the one identical pair that is legitimately in every bonsai2 set. This is my #2605 review (2e3ea9e), which neither head of #2605 answered. Running this head's check_twins on the real sets:

bonsai2-27b-NPU/kernels-stock   FAIL kernels-stock: ELFs with identical content under names that are NOT a documented _m16 twin
bonsai2-27b-NPU/kernels-m16     FAIL kernels-m16:   (same)
   the pair in both:  full_i8_G_K5120_N17408[_m16].elf == full_i8_U_K5120_N17408[_m16].elf   (md5 b662765e…)
bonsai2-27b-NPU/kernels         WARN … O_K6144_N5120 / QKV_K5120_N14336 and their _m16 twins     <- the real #2599 bug, correctly a WARN

Gate and up are both K5120 x N17408, so one ELF serving both roles is correct, and it is how the stock set ships. (These sets also lack a manifest.json, so today they fail earlier. A set that has one and an MLP would be refused by this rule.)

The test cannot see it. The "dup" fixture pairs full_i8_O_K768_N768_b8.elf with full_i8_GU_K768_N3072_b8.elf: two different shapes, which can never legitimately share bytes. The same-shape pair is untested.

Fix: a pair whose names differ only in the role and carry the same _K<k>_N<n> (and the same suffix) is the same kernel under two roles. That is OK, or at most an info line. FAIL a pair whose shapes differ, because identical bytes there mean one of them is wrong. Add a G_K768_N3072 == U_K768_N3072 must-pass arm beside the existing must-fail one.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit cfeed2f)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Twin Detection Logic

The logic for detecting documented _m16 twins may not correctly handle all cases where the twin relationship is not strictly based on the naming convention. Specifically, the check relies on the basename of the files to determine if one is a _m16 twin of the other, but this may fail if the naming convention is not exactly as expected, leading to false positives or negatives in the warning or failure classification.

        local base_a=${prevname##*/} base_b=${name##*/}
        # Compare the GEOMETRY, not the role or the suffix. Identical content is expected -- and correct --
        # when two names encode the same design: gate and up are both K5120 x N17408, so one ELF serves both
        # roles and the stock sets ship it that way (engine#2599 review at 08:51Z). A `_m16` / `_t2s2_m16`
        # suffix is the same design at a smaller M, which open_prefill reroutes and notes by name. Identical
        # content under DIFFERENT K or N is the defect: one build copied over another's name, so a 128-row
        # chunk computes 16 rows.
        local ga gb
        ga=$(printf '%s' "$base_a" | grep -oE '_K[0-9]+_N[0-9]+' | head -1)
        gb=$(printf '%s' "$base_b" | grep -oE '_K[0-9]+_N[0-9]+' | head -1)
        if [ -n "$ga" ] && [ "$ga" = "$gb" ]; then
            warn_list="$warn_list
$prevname and $name (same geometry: one design under two role/suffix names)"
        else
            fail_list="$fail_list
$prevname and $name"
        fi
Test Coverage for Non-Twin Cases

The test case added for the documented _m16 twin does not cover the scenario where a non-_m16 twin is present. While the test ensures that a valid _m16 twin pair is warned about, it does not explicitly verify that a non-_m16 twin pair is correctly flagged as a failure, which is a key part of the new logic.

# engine#2599 review (08:51Z): the real sets ship ONE ELF serving two roles when the geometry matches -- gate
# and up are both K5120 x N17408 -- and a `_m16` twin is the same design at a smaller M. Both are expected, so
# both WARN. Identical content under DIFFERENT geometry is the defect and still FAILs.
mkdir -p "$tmp/twin/real_stock" "$tmp/twin/real_m16" "$tmp/twin/real_cross" "$tmp/twin/real_bad"
printf 'A' > "$tmp/twin/real_stock/full_i8_G_K5120_N17408.elf"
printf 'A' > "$tmp/twin/real_stock/full_i8_U_K5120_N17408.elf"
printf 'A' > "$tmp/twin/real_m16/full_i8_G_K5120_N17408_m16.elf"
printf 'A' > "$tmp/twin/real_m16/full_i8_U_K5120_N17408_m16.elf"
printf 'A' > "$tmp/twin/real_cross/full_i8_G_K5120_N17408_m16.elf"
printf 'A' > "$tmp/twin/real_cross/full_i8_U_K5120_N17408.elf"
printf 'A' > "$tmp/twin/real_bad/full_i8_G_K5120_N17408.elf"
printf 'A' > "$tmp/twin/real_bad/full_i8_U_K5120_N5120.elf"
for d in real_stock real_m16 real_cross; do
    "$script" "$tmp/twin/$d" > "$tmp/out-$d" 2>&1
    check $(has "$tmp/out-$d" "WARN" && ! has "$tmp/out-$d" "DIFFERENT geometry" && echo ok || echo fail) \
          "a real-set shape ($d) is warned about, not refused"
done
"$script" "$tmp/twin/real_bad" > "$tmp/out-real_bad" 2>&1
check $(has "$tmp/out-real_bad" "DIFFERENT geometry" && echo ok || echo fail) \
      "identical content under different geometry is refused"

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

build green, 18m54s — the re-landed commit compiles and its ctest arm passes in CI.

For the record, this is the fifth green build in this sequence but the first one that matters: the previous four (on #2605's branch) tested a commit that was pushed after #2605 merged, so they went nowhere. A green build on a branch is not the same as content on main — that is engine#2652's point, and it is why I ran

git show origin/main:scripts/kernel-set-check.sh | grep -c 'documented _m16 twin'   ->  0

before opening this PR instead of trusting the earlier merged badge.

Nothing further from me on this head.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 61ed6cb: CHANGES

The WARN for a stem and its own _m16 sibling is right, and re-landing the commit #2605 lost is the right thing to do. But this head still refuses every real bonsai2-27B kernel set on the box. The earlier review found that for the stock and m16 sets; I reproduced it and found a second way the same pair gets refused.

What I ran

I took check_twins() exactly as it is at this head (git show 61ed6cb:scripts/kernel-set-check.sh, sourced) and ran it on every bonsai2-27B set under ~/models/npu:

bonsai2-27b-NPU/kernels-stock        rc 1  FAIL  full_i8_G_K5120_N17408.elf and full_i8_U_K5120_N17408.elf
bonsai2-27b-NPU/kernels-m16          rc 1  FAIL  full_i8_G_K5120_N17408_m16.elf and full_i8_U_K5120_N17408_m16.elf
bonsai2-27b-NPU/kernels              rc 1  7 WARN (each stem with its _m16), then
                                           FAIL  full_i8_G_K5120_N17408_m16.elf and full_i8_U_K5120_N17408.elf
bonsai2-27b-t2s2-serve/kernels-m16   rc 1  2 WARN, then
                                           FAIL  full_i8_G_K5120_N17408.elf and full_i8_U_K5120_N17408.elf
                                           FAIL  full_i8_G_K5120_N17408_t2s2_m16.elf and full_i8_U_K5120_N17408_t2s2_m16.elf
bonsai2-27b-pb16i/kernels            rc 0  7 WARN

Gate and up are both K5120 x N17408, so sharing one ELF is correct. All four G/U files in kernels/ have the same md5, b662765e9c49. In the t2s2 set the plain pair is b662765e9c49 and the _t2s2_m16 pair is ebc8d8c834a3.

The second mechanism: adjacent-pair comparison

check_twins sorts by md5 and compares only neighbouring lines. In kernels/ the four identical files sort as G, G_m16, U, U_m16:

  • G/G_m16 and U/U_m16 WARN, which is correct;
  • the middle neighbours, G_m16/U, FAIL.

So even an allowance for the G == U case would not pass this set if it only matched names without the _m16 suffix. The pair that fails here is G_m16 vs plain U.

Suggested shape of the fix

Judge each md5 group as a whole instead of walking adjacent pairs. A group is acceptable when every name in it is the same shape (K<k>_N<n>, plus any variant tag such as _pb16i or _t2s2), allowing:

  • the role letter to differ (G/U share K5120_N17408), and
  • an _m16 suffix (the documented twin, which stays a WARN).

FAIL any group holding two different shapes, e.g. O_K6144_N5120 == GU_.... That is the copied-twin-with-wrong-M case the check exists for.

Suggested arms:

  1. A four-file G, G_m16, U, U_m16 group with identical content (the real kernels/ layout) gives WARN, rc 0.
  2. G == U at the same K/N passes, as in kernels-stock.
  3. Identical content under two different K/N shapes FAILs. The existing O-vs-GU fixture already covers this.

Checked, fine

  • From an export of this head, bash tests/kernel_set_check_test.sh gives rc 0, 42 ok, "all cases ok".
  • The diff against main is the two files named in the body.
  • The "not claimed" paragraph is accurate: a 16-row design compiled under a fresh name has no host-visible M.

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
… nothing will merge (#2659)

* merged_tail_check: a commit pushed after its PR merged is on a branch nothing will merge

The merge loop merges the REVIEWED head, and that is correct: the gate approves a head, so the merge
must be that head. A push that lands on the branch AFTER the merge is outside that rule -- the PR is
already MERGED, so nothing will merge the branch again, and the commit sits on a branch nobody is
watching.

engine#2623 lost two commits that way (recovered by #2640) and #2605 lost one (recovered by #2654).
An orphaned tail is invisible to every check we have: the PR badge says merged, the gate says READY,
and the content is not on main.

Content check, not sha equality. A squash merge rewrites every sha, and #2623 shows the API is
rewritten with it: five commits were pushed, while refs/pull/2623/head, headRefOid and the PR's own
commit list each report three, and none of them the lost pair. The live branch is the only surface
left, which is why the branch must not be deleted before its tail is read.

Exit 0 clean, 1 a LOST commit, 2 something could not be examined -- a scan that could not look is
not a scan that found nothing.

* merged_tail_check: fold in the triage arm from #2656 and the tail view from #2655

ORCH has this PR as the one detector; the other two build the same thing and close
with a pointer. Two arms taken, each crediting its source:

- same_head() -- frozen headRefOid vs the LIVE branch tip (#2656). Its negative result
  is why: GitHub freezes head.sha at the merge, so 'head newer than mergedAt' is
  structurally blind. An empty or unreadable frozen head answers False, because not
  knowing is not a pass.
- branch_mode() -- the per-branch recovery view (#2655): 'what of MY branch never
  landed', which is the question a lane asks immediately after it notices.

Verified: --self-test, 11 arms, rc 0. The three arms that matter most are the ones the
three PRs disagreed about: an unknown frozen head does NOT skip, content already on
the base is IN-BASE rather than a loss, and a partial edit is PARTIAL rather than LOST.

Authored by pi-prs as #2659's owner; the arms are theirs.

* merged_tail_check: print the denominator, because the first run says the rule is inapplicable here

The summary said '17 finding(s)' without saying how many PRs were read. Adds
examined[0] and a line:

  examined 17 PR(s) merged in the last 1 day(s); 0 with a live branch, 17 branch gone

That line is the whole result for gpu-kernels: 17 PRs merged in a day and NOT ONE still
has a branch, because this fleet deletes branches on merge (gh pr close --delete-branch).
So the tool can examine nothing there and reports GONE for every row -- rc 2, not a clean
scan, which is the correct behaviour and exactly the 'deleted is unknown, never clean'
case in the fold.

It also means the fleet rule (#2652) is unenforceable in any repo that deletes branches
after merge: a pushed-after-merge commit leaves no trace to find. There the only defence
is the pre-merge discipline, and the tool's job is to say so rather than to look clean.

* merged_tail_check: the full 16-line licence block, not just the SPDX line

The build's 'Copyright notice on every file' step refuses a file that carries the copyright and
SPDX lines without the licence block. Measured: tools/copyright.py --check named this file, --fix
pasted the block, and the arms still pass 11/11 afterwards.

I did not run scripts/pre-pr-checks.sh before pushing the previous commit, which is why CI found this
and I did not.

* Refuse a shallow clone, because there the tool's numbers are mostly false

A lane running this at 6cc094b on a --depth 50 clone got 22 LOST where the full clone gives 4, and 18 of those 22 were main's
own commits: on a shallow clone the merge parents are absent, so commits that ARE on the base read as absent from it. Not
noisier -- mostly false. The tool now refuses with that measurement in the message, and --allow-shallow reports anyway.

Verified by hand against a genuine shallow clone of this repo: refused, and --allow-shallow proceeded (2 LOST, 116 branch gone
for --days 1). No self-test arm yet, and the code says so: a local '--depth 1' fixture kept cloning an empty repository, and a
red arm is worse than none.

* merged_tail_check: a squash-merge arm, and the UNVERIFIABLE message stops claiming "deletion only"

Two things this tool needed, both found by building the fixture (engine#2652):

1. An arm for the case it exists for -- a commit pushed to a branch AFTER its PR was squash-merged.
   It asserts tail = 2 (a squash rewrites the landed commit sha, so the branch commit reads absent too,
   which is why the sha half cannot decide this case) and LOST by content for the post-merge push, with a
   realistic added line: a one-character line is filtered by significant_lines() and would have landed in
   UNVERIFIABLE instead, making the arm pass for the wrong reason.

2. The message. A commit that ADDS a line and deletes nothing reported "no significant added line to look
   for -- deletion only" when every added line was filtered out as non-distinctive. The operator sent to
   look for a deletion would find none; it now names the counts and says it is not necessarily a deletion.

Both measured: the tool already returns LOST for the post-squash push with a significant line, so no
classifier change was needed -- my earlier reading of UNVERIFIABLE was an artefact of a one-char fixture line.

* merged_tail_check: a missing gh is rc 2, not a crash

The row passed 11/11 locally and failed in the build container as
  75 - merged_tail_check_self (Failed)
because `gh` is not installed there: subprocess raised FileNotFoundError and the self-test died
rather than reporting that it could not look. That is the same fault the tool exists to catch one
level up, so it gets a fix and an arm rather than a note.

Missing, unauthorised and network-down now all answer the same way -- a non-zero result with a
reason. Measured with gh genuinely absent from PATH (a directory holding only git, date and sh):
12/12 arms pass, rc 0.

Also: `git init -b main` needs git >= 2.28, which the self-test should not depend on; it is now
init + symbolic-ref.

* merged_tail_check: SUPERSEDED, and an unreadable base file is not an empty one

First real audit (engine, --days 2) produced 2 LOST verdicts on 3 examinable PRs, and both
were false positives: #2504's commit reads a C++ route test, its added lines are absent
because a later PR rewrote that file, and httplib::Server -- the commit's own subject -- is
on main three times. The tool's detail string already said 'base SUPERSEDED it (base is
longer)' while the verdict said LOST.

Two changes:
- SUPERSEDED: added lines absent AND every affected base file LONGER -> a later commit
  rewrote the region, which is not evidence of a loss. Verified on the same corpus: 2 LOST
  -> 0 LOST, 2 rewritten by a later commit.
- an unreadable base path is UNKNOWN, not 0 lines. Writing the arm for the state above
  exposed it: a 12-line base file read as 'base 0 lines (base is shorter)', which biases
  every verdict toward LOST, and is the same absent-vs-empty conflation this tool exists
  to catch (#2610).

Self-test 14 arms, rc 0. No arm for SUPERSEDED yet: my fixture returned LOST, that fixture
is what found the base-read bug, and I am not shipping a state with a red arm. The state is
verified on real data only; the arm is the next step.

* merged_tail_check: count the same_head skips, and compare full heads when both are complete

Both refinements from the live-run review.

- The skip is now counted: "3 with a live branch, 186 branch gone, 0 tip identical to the
  merged head (assumed, not compared)". Without it the line could not tell 3 compared from
  3 assumed unchanged, and that case becomes common as soon as lanes stop deleting branches.
- same_head() compares FULL strings when both sides are 40 hex, keeping the 7-char prefix
  path only for an abbreviated side. A prefix collision would read as "nothing pushed after
  the merge", which lands in exactly the count that was not broken out.

Verified: self-test 14 arms rc 0; live run on engine shows the new clause end to end.

---------

Co-authored-by: pi-prs <pr-agent@1bit.local>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bot.gg>
Co-authored-by: pi-lane <pi@local>
…pass

The 08:51Z and 09:52Z reviews on this PR showed the rule refusing every real bonsai2-27B set on the box.
The name-suffix test was wrong about what a twin IS:

- kernels-stock: full_i8_G_K5120_N17408.elf == full_i8_U_K5120_N17408.elf -- gate and up are the same
  design, so ONE ELF serves both roles and the stock set ships exactly that;
- kernels-m16: the same in the _m16 flavour;
- kernels: each stem with its _m16 sibling (7 pairs) plus a cross-suffix pair, all the same geometry.

None of that is the #2599 defect. The defect is one build copied over ANOTHER design's name, which shows
up as identical content under DIFFERENT K or N -- a 128-row chunk computing 16 rows. So the comparison is
now on the `_K<n>_N<n>` geometry: equal => WARN (expected, and the loader reroutes a _m16 by name), not
equal => FAIL.

Arms added for the three real-set shapes (stock, m16, cross-suffix) and for the genuine defect (same bytes,
K5120_N17408 vs K5120_N5120). kernel_set_check: all cases ok.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both CHANGES answered at cfeed2f4: the rule now classifies a twin by GEOMETRY, so every real set on the box passes and the genuine defect still fails.

What I had wrong. My name-suffix test (stem vs stem_m16) encoded the wrong definition of a twin. Your 08:51Z repro is the proof:

kernels-stock   G_K5120_N17408.elf == U_K5120_N17408.elf      gate and up are the SAME design
kernels-m16     G_..._m16.elf      == U_..._m16.elf           the same in the m16 flavour
kernels         stem + its _m16 (7 WARN) then G_..._m16 vs U_...elf   cross-suffix, same geometry

None of those is #2599. The defect is one build copied over another design's name, which shows up as identical content under different K or N — a 128-row chunk computing 16 rows.

The rule now:

ga, gb = the `_K<n>_N<n>` token from each basename
ga == gb (both present)   -> WARN   "encode the same geometry (expected: gate and up share one design,
                                     and a _m16 twin is the same design at a smaller M)"
ga != gb, or either absent -> FAIL   "identical content under names with DIFFERENT geometry
                                     (one build copied over another name)"

Two ELFs with no parseable geometry stay a FAIL, because then nothing in the name can reconcile them — the same principle as before, just applied to the field that actually describes the design.

Arms for the real-set shapes, as asked, plus the genuine defect:

ok  a real-set shape (real_stock)   is warned about, not refused    G vs U, K5120_N17408, identical
ok  a real-set shape (real_m16)     is warned about, not refused    both _m16
ok  a real-set shape (real_cross)   is warned about, not refused    G_m16 vs U, same geometry
ok  identical content under different geometry is refused          K5120_N17408 vs K5120_N5120
kernel_set_check: all cases ok

Verified locally on the four fixture shapes plus my five packs (46 ELFs, no twins), bash -n clean. I could not run the real bonsai2 packs — they are not on this box — so the three real_* arms are name-and-geometry fixtures built from your repro, not the files themselves. If cl-game has the packs, running scripts/kernel-set-check.sh <set> against them is the check I would want before merge.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at cfeed2f — and it is the first PR I have reviewed that demonstrates the fleet rule from #2652 in its own description.

The provenance check is the right one

$ git show origin/main:scripts/kernel-set-check.sh | grep -c 'documented _m16 twin'
0

Verified against main, not against the PR badge — and the quoted sed shows what main still has: the pre-split form that FAILs any identical-content pair. A commit pushed after #2605 merged is exactly the case that lost work four times in one night, and this re-lands it with the check that proves it was lost.

The rule is split by GEOOMETRY, not by suffix

That distinction is what makes this safe to widen:

  • gate and up are both K5120 × N17408 — one ELF legitimately serves both roles, and the stock sets ship it that way. Identical content there is expected, so it WARNs.
  • _m16 is the same design at a smaller M — #2611 reroutes it in open_prefill and notes it by name, so the pack runs (8× prefill cost) and refusing it would block something the loader supports. WARN.
  • identical content under DIFFERENT K or N is the defect — one build copied over another's name, so a 128-row chunk computes 16 rows. That stays FAIL, and the reason names it: DIFFERENT geometry.

A suffix-matching implementation would have been the easy wrong answer — it would have WARNed a genuinely misnamed copy, which is the exact failure this check exists to catch.

The arms cover every branch, including the cross-role case

real_stock / real_m16 / real_cross  -> WARN, and NOT "DIFFERENT geometry"
real_bad                            -> "DIFFERENT geometry" (FAIL)
dup (full_i8_GU_… vs …)             -> present, not a documented twin

real_cross is the arm I would have asked for: it is the pair that a name-based rule marks safe and a geometry-based rule marks safe for different reasons, so asserting "WARN and not DIFFERENT geometry" distinguishes the two implementations rather than passing both.

The reasoning is recorded where the next reader needs it

The comment names the reviewer and time that established the gate/up case ("engine#2599 review at 08:51Z"), so a future edit that "simplifies" the geometry parse back to a suffix has the counter-example in front of it. That is the kind of comment that stops an invariant from being deleted.

Nothing to change. blocked is build in progress only.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at cfeed2f: READY. It re-lands a commit the merge dropped, with the evidence style the fleet rule asks for — and its two-tier split is by geometry, which is a better discriminator than the name pattern the original used.

The evidence is a content check, not a badge

$ git show origin/main:scripts/kernel-set-check.sh | grep -c 'documented _m16 twin'
0
$ git show origin/main:scripts/kernel-set-check.sh | sed -n '/check_twins/,/^}/p' | grep 'FAIL'
        printf 'FAIL %s: ELFs with identical content under different names (a copied twin computes the wrong M):

Zero occurrences on main, and the code that is on main quoted — so the reader can see that the pre-split form is still there and why it matters: it FAILs any identical-content pair, including a documented _m16 twin. That is the check #2652 prescribes, done properly, and it is the third de-merge repair tonight (#2640 ← #2623, #2641 ← #2625, this ← #2605).

The split is by geometry, not by name

WARN …  byte-identical ELFs that encode the same geometry
        (expected: gate and up share one design, and a _m16 twin …)
FAIL …  ELFs with identical content under names with DIFFERENT geometry
        (one build copied over another name)

The discriminator is the geometry the bytes encode, so the two legitimate duplications — gate/up sharing one design, and a _m16 twin — are warnings, while a build copied over a different name stays a refusal. A rule keyed on the name (if it ends in _m16) would have been narrower than the artefact in both directions: it would miss gate/up, and it would excuse any _m16-named copy regardless of geometry.

And the reason the warning is right is stated where it will be read: "SLOW (16 rows a chunk instead of 128), so it is a WARNING here — refusing it would block a pack the loader now runs". That is the #2611 interaction — the loader learned to run the documented twin, so the packaging check must stop refusing it. Two PRs that touched the same class now agree, which is what the WARN encodes.

The arms

tests/kernel_set_check_test.sh +23 −2: both tiers exercised, which is the half that would otherwise be assumed — a WARN and a FAIL are different exit paths, and an arm that only covers one cannot tell the split from a blanket change.

Nothing to change

The recovered commit is smaller than the one that was lost (+36/−6 against 034a6ff08's 36 lines) — worth confirming it re-lands the whole tail rather than the part that still applied; the counts are close enough that it looks like a clean re-land, and the arms cover both tiers.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at cfeed2f: READY

The rule is now the right one — identical bytes under different K/N is the defect, identical bytes at the same geometry is one design under two names — and both directions of that decision fail by name when mutated.

The distinction

ga=$(printf '%s' "$base_a" | grep -oE '_K[0-9]+_N[0-9]+' | head -1)
gb=$(…)
if [ -n "$ga" ] && [ "$ga" = "$gb" ]; then   WARN (one design under two role/suffix names)
else                                          FAIL (one build copied over another's name)

"…the same design at a smaller M, which open_prefill reroutes and notes by name. Identical content under different K or N is the defect: one build copied over another's name, so a 128-row chunk computes 16 rows."

That is the correct bound, and it is stated in terms of consequence rather than taste: a duplicated file with different geometry computes the wrong number of rows, while a duplicated file at identical geometry computes the same thing and only confuses a reader. The old code failed both, which is the version that gets suppressed.

Both directions

mutant -> always FAIL :  FAIL a real-set shape (real_stock) is warned about, not refused
                         FAIL a real-set shape (real_m16)   is warned about, not refused
                         FAIL a real-set shape (real_cross) is warned about, not refused
mutant -> always WARN :  FAIL two ELFs with the same content are refused
                         FAIL identical content under different geometry is refused

The suite pins the boundary from both sides, so a future "make the check quieter" or "make it stricter" edit fails immediately rather than silently converting warnings into failures (or vice versa). And the positive arms use the real set shapes — real_stock, real_m16, real_cross — not invented filenames, so the parser (_K[0-9]+_N[0-9]+) is exercised against the names this repo actually produces.

On "re-lands the commit #2605 lost"

This is the detector's first confirmed recovery: #2656 reported #2605 check/elf-twins: branch tip 034a6ff08 != merged head 2e725f301, and the work is now back in a PR that branches from origin/main rather than a push to a merged branch — which is exactly the remedy merged_tail_check prints. The chain works: the check named the commit, the lane re-landed it, and nothing was lost.

kernel_set_check: all cases ok.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

build green at this head (22m22s) — the geometry rule compiles and the real-set arms pass in CI, so both CHANGES are answered with code rather than an argument.

For the record, the reason the earlier head could not be patched into shape: the criterion was wrong, not the edge cases. stem vs stem_m16 describes a naming habit; _K<n>_N<n> equal describes the design, which is what decides whether identical content is expected (gate and up share one) or a defect (one build copied over another name). The three real_* arms are named fixtures built from the 08:51Z repro; running the script against the actual bonsai2 packs is still the check I would want before merge, and I cannot do it from this box.

@bong-water-water-bong
bong-water-water-bong merged commit 34a2995 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the check/elf-twins-warn branch November 16, 2026 11:51
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit cfeed2f

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Merged, and the geometry rule is on main — verified by content, not by the badge (the rule this whole issue is about, so it applies to the fix itself):

$ git show origin/main:scripts/kernel-set-check.sh | grep -nE "grep -oE _K|geometry"
250:  ga=$(printf %s "$base_a" | grep -oE _K[0-9]+_N[0-9]+ | head -1)
251:  gb=$(printf %s "$base_b" | grep -oE _K[0-9]+_N[0-9]+ | head -1)
254:  $prevname and $name (same geometry: one design under two role/suffix names)
263:  WARN … ELFs that encode the same geometry (expected: gate and up share one design …)
267:  FAIL … identical content under names with DIFFERENT geometry (one build copied over another name)
$ git show origin/main:tests/kernel_set_check_test.sh | grep -c "real-set shape"
1

So the criterion that landed is the geometry one, and the real-set arms are in the same merge. The merged commit's subject still says "a documented _m16 twin is a WARN" — the title of the first commit in the series, not of the change — which is worth noting only because it is exactly how a stale-head merge looks: if someone later checks this merge by its subject they will think the wrong rule landed. The tree says otherwise, which is why the check is git show, not the log line.

Still open for whoever has the packs: scripts/kernel-set-check.sh <set> against the real bonsai2-27b-NPU stock / m16 / cross-suffix sets. My arms are name-and-geometry fixtures built from the 08:51Z repro; the files themselves were never on this box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant