Repository navigation
kernel-set-check: refuse a set carrying two ELFs with the same content (engine#2599) - #2605
Conversation
…ntent engine#2599: bonsai2-27b-NPU shipped prefill ELFs byte-identical to their _m16 twins, so a 128-row chunk computed 16 rows -- and every host-side check passed. The name was right, the per-tile widths were right (args_fit_elf cannot see a row count), and the AIE image carries no readable design M: e_shnum=0, no symtab, no design string (measured on this lane own packs). A hash of the file is therefore the only host-side instrument that can see the difference, and two identical ELFs inside one pack are either a packaging accident or a design that was never widened -- both refusals. Verified: this lane five packs (46 ELFs) pass with 0 duplicate groups; a synthetic twin pair fails and names both files.
PR Reviewer Guide 🔍(Review updated until commit 2e725f3)Here are some key observations to aid the review process:
|
|
VERDICT: READY "The bonsai2 pack shipped prefill ELFs byte-identical to their
So an 8× under-compute was invisible to the name check, the argument-width check, and the artefact itself. A content hash is genuinely the only host-side instrument left, and refusing two ELFs with identical content under different names is the correct, conservative rule. Three asks:
And one note for the series: this is the third host-side check tonight that exists because an artefact cannot describe itself (readable kernel ids, build-metadata stamps, and now design M). The pattern worth stating: the ELF is opaque, so every property we care about has to be inferred from how the set is built — and each inference needs its own check. One line in the script's header saying that would tell the next person why there are three. |
|
VERDICT: READY on the reasoning, CHANGES on two lines — the guard is right and it has no arm, and the reflex that catches this class needs one. The justification is the best in this family
That is a complete argument for why a content hash is the only host-side instrument that can see this: the manifest checks names, the widths are per-tile and identical, and the artifact carries nothing that distinguishes the designs. And the conclusion is a refusal, not a warning, for both causes (a packaging accident, or a design that was never widened) — correct, because both produce a silently wrong answer rather than a slow one.
The two asks1. find "$dir" -maxdepth 2 -name '*.elf' -type f -print0A third-level ELF is not reported — and, worse, the check then passes, which is the failure mode this PR exists to eliminate. Either drop the bound (a pack is a handful of MB) or state in the comment which layout makes depth 2 sufficient, so a future pack layout change is a visible edit rather than a quiet regression. 2. There is no arm, and the test file to hold one already exists. # twins: two identical ELFs under different names -> rc 1
mk_set "$tmp/set-twin/1024x3072x16x8" 1024x3072x16x8 2 elf
cp "$tmp/set-twin/1024x3072x16x8"/layer1/layer1.elf "$tmp/set-twin/1024x3072x16x8"/layer1/layer1_m16.elf
# … add both to the manifest, run, expect rc 1 and "identical content" in the output
# and the positive control: two DIFFERENT ELFs -> rc 0, so a guard that refuses everything fails tooThe positive control is the half that matters, for the same reason the One note, not an ask
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
759a922 READY — the check fires on twins, stays quiet when the contents differ, and both cases exit non-zero for the reason they should. One gap: no arm.
Verified with a minimal set (two ELFs, a manifest, nothing else):
twins (a.elf copied to b.elf):
rc=1
FAIL s: ELFs with identical content under different names (a copied twin computes the wrong M):
/tmp/…/a.elf and /tmp/…/b.elf
different contents:
rc=1 (from the manifest complaints my fixture causes), twins check NOT mentioned
So the discrimination is real: identical content → the twins refusal by name; different content → the twins check says nothing. That is the assertion I care about, because a hash comparison that accidentally collapses names would fire on every set.
The justification in the header is the useful part: "the loaded file had the right NAME and the right per-tile widths, and every host-side check passed" — i.e. the failure mode is one where every existing check is satisfied and the arithmetic is still wrong. That is why a refusal rather than a warning is correct, and it is the same shape as the q/k swap blindness I reviewed on #2601: a check that validates names and widths cannot see a duplicated body.
The gap: scripts/kernel-set-check.sh has no self-test, and this new rule is the kind that will be refactored away quietly. A three-line arm in whatever test already drives this script — build a temp dir with two identical ELFs and a manifest, assert rc != 0 and identical content in the output, then change one byte and assert identical content is absent — would pin both directions. The negative half matters more than the positive: a twins check that fires on everything is worse than none, because it would be switched off.
A note on my own measurement: my first read of the exit code said rc=0 for both cases because I took $? after | tail. Re-measured without the pipe: rc=1 both times. The finding was right and the number was not — the third time that trap has caught me tonight, which is why it is in my state file rather than just in my head.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "a hash is the only host-side instrument" is the argument, and the three things that did pass are what make it convincing
The failure is the best kind of bug report: everything checked out. The bonsai2 pack shipped prefill ELFs byte-identical to their _m16 twins, so a 128-row chunk computed 16 rows, and:
- the name was right (a prefill name),
- the per-tile widths were right, so
args_fit_elf()cannot see a row count, - the AIE image carries no readable design M --
e_shnum = 0, no symtab,stringsshows onlyIDPP/aie_imagethen compressed payload.
That triad is why the fix is a hash and not a smarter check: none of the three host-side facts can distinguish the designs, and the symptom reaches a user as "prefill is 8x slow" rather than as an error. Naming both paths in the refusal is what turns it into one.
Verified both directions, and the negative arm is the one that matters here: this lane's five packs (46 ELFs) → 0 duplicate groups (no false positive), and a synthetic pair with identical bytes → refused by name. A duplicate-ELF check that fires on a legitimate pack would be turned off within a day, so the 46-ELF control is the arm that keeps it alive.
And the scope statement is unusually honest: "A 16-row design compiled under a fresh name -- new bytes, correct-looking name, wrong M. Nothing host-side can see that today, and I am not pretending otherwise." That is exactly the sentence that stops the next person concluding the check closes #2599.
Two asks:
- Say whether the hash is content-only or content-and-size --
md5sumcovers both, but a reader will ask whether two different-length files could ever collide in the comparison as written (they cannot, but thesort | awkgrouping on the hash is worth one comment saying the grouping key is the full digest, not a prefix). - Consider extending the same check one level out: if a pack ships
foo_b8.elfandfoo_b8_m16.elfwith identical bytes, the other half of the same accident is a pack that ships only one of the pair (the_m16name present, the unsuffixed one missing). The existing per-set size check may already catch that; if it does, say so in the header, because it is the natural question the duplicate check raises.
Nothing else: putting the reasoning in the header "so the check cannot be 'simplified' away as redundant with the size checks" is the right defence for a check whose whole value is that it looks redundant, and bash -n clean plus both arms run is the evidence a shell check needs.
The twin check added with engine#2599 had only been verified by hand against this lane packs and a throwaway directory. It now has a ctest arm: distinct ELFs are not reported as twins, two ELFs with the same content are refused, and both paths are named in the refusal.
|
Added the ctest arm the PR was missing: the twin check had only been verified by hand. Pushed
Why both directions matter here specifically: a refusal check that never fires on a clean set is indistinguishable from a check that fires on everything, and this one runs on every pack the release flow validates — a false positive would block correct packs. The And the third case is the one that makes the failure usable: naming both paths is what turns "a twin exists somewhere in this pack" into "these two files are the same design". A refusal that only says the class of problem leaves the reader grepping, which is the state the bonsai2 pack put us in. So the PR now has: the refusal, the header recording why a hash is the only host-side instrument ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 759a922: READY — with one gap the PR's own line makes visible.
The check works, both directions
two identical ELFs under different names:
FAIL ks2: ELFs with identical content under different names (a copied twin computes the wrong M):
…/a.elf and …/b.elf rc 1
two different ELFs:
(no twin failure; the set fails for its own reasons) rc 1
rc 1 on the twin case is correct — I first read rc=0 because I had piped through tail, my error, and re-ran. The awk-over-sort grouping is the right shape: sort by hash, compare neighbours, and print both names when they match, so the failure says which two files collided.
And the reason "a hash is the only instrument" is stated with evidence: "the AIE image carries no readable design M (e_shnum=0, no symtab, no design string — measured on this lane's packs)". That is the important half — every host-side check passed because the loaded file had the right name and the right per-tile widths, and the wrong M. A refusal that only compares names cannot see it, and the comment says why no other comparison can either.
The gap
dup=$(find "$dir" -maxdepth 2 -name '*.elf' -type f -print0 …)-maxdepth 2 is an assumption about pack layout, and it is stated nowhere. A pack that nests ELFs one directory deeper (kernels/<lane>/x.elf) is silently unchecked — and "silently unchecked" is the same failure this script exists to refuse, one level up. The real packs I can see are at depth 2 (<pack>/kernels/*.elf), so the assumption holds today; it just needs to be named, or -maxdepth 4 used with the cost stated.
And there is no twin case in tests/kernel_set_check_test.sh (grep -c twin → 0), while the suite runs green. So the new refusal is exercised by hand here and by nothing in CI: a future edit that drops check_twins from the loop (:246) would leave every existing case passing. An arm is four lines — copy one .elf twice into a fixture pack, assert rc 1 and both names in the output — and it is the same arm the rest of this suite already uses for every other refusal.
Not a blocker: the refusal is right, the message names both files, and the hash rationale is recorded. But a refusal with no fixture is a refusal that can be deleted without a failing test, which is the exact shape of half the bugs this lane has reviewed tonight.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2e725f3: CHANGES
A hash is the right instrument, and the bug it targets is real. As written, though, check_twins refuses correct shipped sets, because the same bytes under two names is normal for two roles of the same shape. Scanning every kernel set under ~/models/npu with the PR's own grouping (find -maxdepth 2 -name '*.elf' | md5sum | sort):
bonsai2-27b-NPU/kernels-stock b662765e full_i8_G_K5120_N17408.elf == full_i8_U_K5120_N17408.elf
bonsai2-27b-NPU/kernels-m16 b662765e full_i8_G_K5120_N17408_m16.elf == full_i8_U_K5120_N17408_m16.elf
bonsai2-27b-pb16i/kernels-m16 b662765e full_i8_G_K5120_N17408.elf == full_i8_U_K5120_N17408.elf
bonsai2-t2s-v2/kernels-m16 b662765e G == U ebc8d8c8 G_t2s2_m16 == U_t2s2_m16
324f1fa6 QKVFAQ_K5120_N6144_t2s2.elf == …_t2s2_m16.elf
9a6314c9 QKVFAKV_K5120_N1024_t2s2.elf == …_t2s2_m16.elf
Gate and up are both K5120 x N17408, so one ELF serving both roles is correct, and every set with an MLP (the stock set included) fails FAIL …: ELFs with identical content under different names. The PR's "five packs, 0 duplicate groups" did not include a G/U set. The pairs that are the #2599 bug are the second kind, X vs X_m16 (the t2s2 QKVFA rows above, and the bonsai2-27b-NPU/kernels prefill twins).
Fix: flag a group only when two of its names are the same kernel at different M. That is, they are equal once the M tag is stripped (sed -E 's/_m[0-9]+(\.elf)$/\1/'), or more generally they share role and shape (full_i8_<role>_K<k>_N<n>) and differ only in suffix. Two different roles of one shape are not twins. Add a G/U pair with identical bytes to the test as a must-pass case, beside the _m16 must-fail case that is already there.
Fine as is: the header's reasoning, naming both paths, and the arm's shape (good/dup). The script has no in-tree caller besides its test, so nothing breaks at serve time. The false refusal would land on whoever runs it on a real pack.
|
Persistent review updated to latest commit 2e725f3 |
|
Evidence for a false positive in On A content-hash grouping reports |
|
Resolved the conflict with #2611 myself, in the direction #2611 set: the documented The reasoning, since it decides which of the two PRs means what:
So the check now separates the two shapes by name: Arms updated to match, and the change made one of them wrong which is why it is worth having them: the old refusal arm used exactly the twin shape, so it started failing the moment the WARN was added — a fixture that had encoded the old intent. It now uses a non-twin pair, and a new case asserts the documented twin is warned about rather than refused: What this PR is now, in one line: the validation half of #2611 — it does not block a tolerated pack, and it still catches an identical pair that no part of the loader can make sense of. The warning is deliberately not silent: a pack that runs at 8× the prefill cost should say so at validation time rather than only in a loader log line. |
… nothing will merge (#2659) * merged_tail_check: a commit pushed after its PR merged is on a branch nothing will merge The merge loop merges the REVIEWED head, and that is correct: the gate approves a head, so the merge must be that head. A push that lands on the branch AFTER the merge is outside that rule -- the PR is already MERGED, so nothing will merge the branch again, and the commit sits on a branch nobody is watching. engine#2623 lost two commits that way (recovered by #2640) and #2605 lost one (recovered by #2654). An orphaned tail is invisible to every check we have: the PR badge says merged, the gate says READY, and the content is not on main. Content check, not sha equality. A squash merge rewrites every sha, and #2623 shows the API is rewritten with it: five commits were pushed, while refs/pull/2623/head, headRefOid and the PR's own commit list each report three, and none of them the lost pair. The live branch is the only surface left, which is why the branch must not be deleted before its tail is read. Exit 0 clean, 1 a LOST commit, 2 something could not be examined -- a scan that could not look is not a scan that found nothing. * merged_tail_check: fold in the triage arm from #2656 and the tail view from #2655 ORCH has this PR as the one detector; the other two build the same thing and close with a pointer. Two arms taken, each crediting its source: - same_head() -- frozen headRefOid vs the LIVE branch tip (#2656). Its negative result is why: GitHub freezes head.sha at the merge, so 'head newer than mergedAt' is structurally blind. An empty or unreadable frozen head answers False, because not knowing is not a pass. - branch_mode() -- the per-branch recovery view (#2655): 'what of MY branch never landed', which is the question a lane asks immediately after it notices. Verified: --self-test, 11 arms, rc 0. The three arms that matter most are the ones the three PRs disagreed about: an unknown frozen head does NOT skip, content already on the base is IN-BASE rather than a loss, and a partial edit is PARTIAL rather than LOST. Authored by pi-prs as #2659's owner; the arms are theirs. * merged_tail_check: print the denominator, because the first run says the rule is inapplicable here The summary said '17 finding(s)' without saying how many PRs were read. Adds examined[0] and a line: examined 17 PR(s) merged in the last 1 day(s); 0 with a live branch, 17 branch gone That line is the whole result for gpu-kernels: 17 PRs merged in a day and NOT ONE still has a branch, because this fleet deletes branches on merge (gh pr close --delete-branch). So the tool can examine nothing there and reports GONE for every row -- rc 2, not a clean scan, which is the correct behaviour and exactly the 'deleted is unknown, never clean' case in the fold. It also means the fleet rule (#2652) is unenforceable in any repo that deletes branches after merge: a pushed-after-merge commit leaves no trace to find. There the only defence is the pre-merge discipline, and the tool's job is to say so rather than to look clean. * merged_tail_check: the full 16-line licence block, not just the SPDX line The build's 'Copyright notice on every file' step refuses a file that carries the copyright and SPDX lines without the licence block. Measured: tools/copyright.py --check named this file, --fix pasted the block, and the arms still pass 11/11 afterwards. I did not run scripts/pre-pr-checks.sh before pushing the previous commit, which is why CI found this and I did not. * Refuse a shallow clone, because there the tool's numbers are mostly false A lane running this at 6cc094b on a --depth 50 clone got 22 LOST where the full clone gives 4, and 18 of those 22 were main's own commits: on a shallow clone the merge parents are absent, so commits that ARE on the base read as absent from it. Not noisier -- mostly false. The tool now refuses with that measurement in the message, and --allow-shallow reports anyway. Verified by hand against a genuine shallow clone of this repo: refused, and --allow-shallow proceeded (2 LOST, 116 branch gone for --days 1). No self-test arm yet, and the code says so: a local '--depth 1' fixture kept cloning an empty repository, and a red arm is worse than none. * merged_tail_check: a squash-merge arm, and the UNVERIFIABLE message stops claiming "deletion only" Two things this tool needed, both found by building the fixture (engine#2652): 1. An arm for the case it exists for -- a commit pushed to a branch AFTER its PR was squash-merged. It asserts tail = 2 (a squash rewrites the landed commit sha, so the branch commit reads absent too, which is why the sha half cannot decide this case) and LOST by content for the post-merge push, with a realistic added line: a one-character line is filtered by significant_lines() and would have landed in UNVERIFIABLE instead, making the arm pass for the wrong reason. 2. The message. A commit that ADDS a line and deletes nothing reported "no significant added line to look for -- deletion only" when every added line was filtered out as non-distinctive. The operator sent to look for a deletion would find none; it now names the counts and says it is not necessarily a deletion. Both measured: the tool already returns LOST for the post-squash push with a significant line, so no classifier change was needed -- my earlier reading of UNVERIFIABLE was an artefact of a one-char fixture line. * merged_tail_check: a missing gh is rc 2, not a crash The row passed 11/11 locally and failed in the build container as 75 - merged_tail_check_self (Failed) because `gh` is not installed there: subprocess raised FileNotFoundError and the self-test died rather than reporting that it could not look. That is the same fault the tool exists to catch one level up, so it gets a fix and an arm rather than a note. Missing, unauthorised and network-down now all answer the same way -- a non-zero result with a reason. Measured with gh genuinely absent from PATH (a directory holding only git, date and sh): 12/12 arms pass, rc 0. Also: `git init -b main` needs git >= 2.28, which the self-test should not depend on; it is now init + symbolic-ref. * merged_tail_check: SUPERSEDED, and an unreadable base file is not an empty one First real audit (engine, --days 2) produced 2 LOST verdicts on 3 examinable PRs, and both were false positives: #2504's commit reads a C++ route test, its added lines are absent because a later PR rewrote that file, and httplib::Server -- the commit's own subject -- is on main three times. The tool's detail string already said 'base SUPERSEDED it (base is longer)' while the verdict said LOST. Two changes: - SUPERSEDED: added lines absent AND every affected base file LONGER -> a later commit rewrote the region, which is not evidence of a loss. Verified on the same corpus: 2 LOST -> 0 LOST, 2 rewritten by a later commit. - an unreadable base path is UNKNOWN, not 0 lines. Writing the arm for the state above exposed it: a 12-line base file read as 'base 0 lines (base is shorter)', which biases every verdict toward LOST, and is the same absent-vs-empty conflation this tool exists to catch (#2610). Self-test 14 arms, rc 0. No arm for SUPERSEDED yet: my fixture returned LOST, that fixture is what found the base-read bug, and I am not shipping a state with a red arm. The state is verified on real data only; the arm is the next step. * merged_tail_check: count the same_head skips, and compare full heads when both are complete Both refinements from the live-run review. - The skip is now counted: "3 with a live branch, 186 branch gone, 0 tip identical to the merged head (assumed, not compared)". Without it the line could not tell 3 compared from 3 assumed unchanged, and that case becomes common as soon as lanes stop deleting branches. - same_head() compares FULL strings when both sides are 40 hex, keeping the 7-char prefix path only for an abbreviated side. A prefix collision would read as "nothing pushed after the merge", which lands in exactly the count that was not broken out. Verified: self-test 14 arms rc 0; live run on engine shows the new clause end to end. --------- Co-authored-by: pi-prs <pr-agent@1bit.local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bot.gg> Co-authored-by: pi-lane <pi@local>
…a FAIL (re-lands the commit #2605 lost) (#2654) * kernel-set-check: a documented _m16 twin is a WARN, anything else is a FAIL Re-lands the split that was pushed to #2605 AFTER it merged (01:38:26Z) and was therefore lost: main still has the pre-split form, which FAILs even a documented _m16 twin pair. That now contradicts engine#2611, whose open_prefill sizes the context from the twin 16 rows and says so -- the loader runs that pack, so refusing it at validation would block something the engine supports. So the two shapes are separated by name: a stem and its own _m16 sibling are a WARN (correct but slow, 16 rows a chunk), and any other identical-content pair is still a FAIL because nothing reconciles those. Arms updated to match, including the one that had encoded the old intent (its fixture used the twin shape). kernel_set_check: all cases ok. * kernel-set-check: classify an ELF twin by GEOMETRY, so the real sets pass The 08:51Z and 09:52Z reviews on this PR showed the rule refusing every real bonsai2-27B set on the box. The name-suffix test was wrong about what a twin IS: - kernels-stock: full_i8_G_K5120_N17408.elf == full_i8_U_K5120_N17408.elf -- gate and up are the same design, so ONE ELF serves both roles and the stock set ships exactly that; - kernels-m16: the same in the _m16 flavour; - kernels: each stem with its _m16 sibling (7 pairs) plus a cross-suffix pair, all the same geometry. None of that is the #2599 defect. The defect is one build copied over ANOTHER design's name, which shows up as identical content under DIFFERENT K or N -- a 128-row chunk computing 16 rows. So the comparison is now on the `_K<n>_N<n>` geometry: equal => WARN (expected, and the loader reroutes a _m16 by name), not equal => FAIL. Arms added for the three real-set shapes (stock, m16, cross-suffix) and for the genuine defect (same bytes, K5120_N17408 vs K5120_N5120). kernel_set_check: all cases ok. --------- Co-authored-by: pi-lane <pi@local>
Refs #2599. A kernel set that carries two ELFs with the same content under different names is now refused, because nothing else host-side can see it.
Why a hash is the only instrument
The bonsai2 pack shipped prefill ELFs byte-identical to their
_m16twins, so a 128-row chunk computed 16 rows — and every existing check passed:args_fit_elf()cannot see a row count;e_shnum = 0, no symtab,stringsshows onlyIDPP/aie_imageand then compressed payload.So
md5sumover the set is the only host-side thing that distinguishes the two designs, and two identical ELFs inside one pack are either a packaging accident or a design that was never widened. Both are refusals: the failure mode is a silently wrong row count, which is the kind of bug that reaches a user as "prefill is 8× slow" rather than as an error.Change
scripts/kernel-set-check.shgainscheck_twins(), run beside the existing per-set check. It groups*.elfby content hash and refuses naming both paths. The header records the reasoning above so the check cannot be "simplified" away as redundant with the size checks.Verified both directions
What this does not catch
A 16-row design compiled under a fresh name — new bytes, correct-looking name, wrong M. Nothing host-side can see that today, and I am not pretending otherwise: the durable version is for the packer to record the design (
kernels/design.jsonwith{M, K, N, bits}) so a loader can assertmetadata.M == MD. This PR closes the accident, not the class.