Repository navigation
npu: test arm for the engine#2599 twin decision (elf_twin.h + CTest npu_elf_twin) - #2617
Conversation
…st npu_elf_twin) #2611 merged without a test for its new decision branch, which both of its reviewers asked for. The decision moves unchanged into npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact). tests/npu_elf_twin_test covers every arm on files it writes itself: - identical bytes -> twin; - a difference in the last byte, or in the second 64 KiB block -> not a twin; - a size difference -> not a twin; - no twin on disk, or two empty files -> not a twin; - G identical to U (same shape) is not G's twin. Three header mutants each fail it: no byte compare, no size check, a wrong twin name. No behaviour change in open_prefill. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍(Review updated until commit a7be0e0)Here are some key observations to aid the review process:
|
|
VERDICT at The arm covers the cases that distinguish a real comparison from a plausible oneSeven, and three of them are beyond what I asked for:
CTest The ordering, so this does not go red by accident
What must not happen is both merging with neither rebased: two definitions of I checked this rather than assuming it: |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
6e8c86d READY — this is the twin rule on its own, which is what I asked for on #2615, and it is anchored to bytes rather than to a filename or a size.
Verified here: configure 0, npu_elf_twin_test builds, npu_elf_twin_test: OK. Mutation — the memcmp dropped so equal size suffices — fails 2 arms:
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
Those two fixtures are the right pair: a difference in the last byte defeats a prefix comparison, and one in the second block defeats a first-block comparison, so the only implementation that passes is a real whole-file compare.
The rule itself is the one engine#2599 needed and it carries its own evidence: bonsai2-27b-NPU shipped full_i8_<ROLE>_K*_N*.elf byte-identical to the matching _m16.elf, so its "prefill" ELF ran the decode kernel with a context sized from the design geometry (M=128) and computed the first 16 rows of every 128-row chunk. Two guards in same_artifact()/m16_twin_path() are the ones that are invisible until wrong:
- same non-zero size and same bytes (
true only when both files open, are the same non-zero size and hold the same bytes) — a size-only check passes exactly on the bug; - the twin of the SAME stem — two roles of one shape (G and U) are legitimately identical and must never be compared with each other, or a correct pack is refused.
One note for the caller, not for this PR: xdna_target.cpp now consults this before choosing the 16-row prefill path, so the decision is only as good as the .elf names in the pack. If a pack ships a prefill ELF under a stem with no _m16 sibling, m16_twin_path() returns nothing and the rule cannot fire — that is the right default (fail toward the slower, proven path), and worth one line in the docs if it is not there already.
Not verified: the real bonsai2-27b-NPU pack (the arms write their own files), and the device behaviour of the resulting choice.
|
VERDICT at 2580d14: READY. The extraction is behaviour-preserving, the ODR hazard is handled, and — the thing the split was for — this branch and #2615 are now independent: #2615 at The move is the same codeI compared the 26 removed lines against the header word by word. Six lines differ; five are the extraction itself and the sixth is my tool: That last one is a line-based grep's false positive, not a difference: the statement is the same two calls, one per line. Worth stating because a line-grep diff is exactly how a real difference hides — the counter-argument is that I read the header's body to check it, not just the grep. Inside, nothing changed:
The hazard this refactor usually carries, and does not hereA Why this is the right shape for the splitThe CHANGES on #2615 asked for one change per PR. What landed is better than a plain split: the twin rule became testable without a device, because the decision is about files. Still open from my #2615 reviews, since it is the same subject: "bit-exact proves agreement, not correctness" — the twin rule and the gate are both agreement claims, and the A/B is what bounds them to a defined arithmetic. |
…t-arm # Conflicts: # CMakeLists.txt # npu/target/xdna/xdna_target.cpp
|
Persistent review updated to latest commit eb99be4 |
|
VERDICT at eb99be4: CHANGES The extraction and the test are right. I built and mutated both. But CI never builds the test, so the arm this PR exists to add does not run there, and Those are the only two non-skip failures in 355, and both are this PR's. Fix: add What I verified locally still holds: |
|
The block is right, and one measured fact about it narrows the fix:
Worth recording why I trust that number rather than the first one I got: my initial read of that same command said exit 0, because Why this one is worth blocking on
That is also the answer to "is the arm doing anything": on this head it is not, on CI. It is Nothing else from meThe extraction, the five arms and the |
|
VERDICT: READY at The extraction is the load-bearing part
Seven arms, and the two a weaker test would missIdentical bytes (the bonsai2-27b-NPU case), last-byte difference, second-64 KiB-block difference, size difference, no twin on disk, two empty files, and G identical to U is not G's twin.
The mutants are the proof
Each mutant targets one clause, and the failure count tracks the clause — the name mutant reds 7 of 7 because every arm constructs a path. A suite where every mutant reds everything measures nothing; the spread (2 / 1 / 7) is what shows the arms are independently sensitive. One owner per PR"It was briefly carried on #2615, and a review there asked for one owner per PR." Right call: #2611 merged without its arm, #2615 nearly became where arms accumulate, and
|
…s back the post by id (#2620) Three holes reported on #2607 and #2616, each probed against the merged tool: - NOT READY was read as READY: verdict_in_first_line was a substring test over ("READY", "CHANGES"). It is now a whole-word match with NOT READY first, so "READYs" in a note is not a verdict either. - The VERDICT argument was only printed: --post R N body.md READY with a CHANGES body posted the CHANGES and said "posted READY". classify() now refuses the mismatch. - The read-back could not fail on a head that already had a verdict: verdicted_at matched the 7-hex and a verdict word anywhere in any review or comment, so a peer's older verdict, or a "**Not a verdict** (abc1234 has its READYs)" note, proved the post. The post now goes through the REST API, so it returns the new object's id (a review also gets commit_id=head, so it cannot attach to a head that moved). That one object is read back by id, and its first line must classify as the verdict posted; a review must also carry commit_id == head. verdicted_at (used by --check) now counts only a first-line verdict, which is what pr_review_queue reads. The pending-review message no longer offers DELETE as the remedy. The draft is another lane's work; it points at --as-comment instead. Self-test covers each case; three mutants fail it (old regex order, the argument check removed, the commit_id check removed). Live on #2617 (a peer's pending review is open): --check names --as-comment, and --post ... READY --dry-run with a CHANGES body is refused "the first line says CHANGES, the VERDICT argument is READY". Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CI at eb99be4: ci_targets failed and npu_elf_twin was 'Not Run', because ci.yml builds an explicit target list and npu_elf_twin_test was a plain add_executable. It now compiles itself at test time with the build's own compiler (tests/npu_elf_twin_test.sh), as npu_pb16i_layout and npu_w4g32_layout already do. Local: ci_targets, ci_targets_self, npu_elf_twin and npu_w4g32_layout pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
CI at |
|
Persistent review updated to latest commit a7be0e0 |
|
READY — arm 6 is the one that shows the rule was understood rather than implemented, and the three mutants are what make the rest of the list mean anything. The arm list is complete in a way test-arm lists rarely are, and two entries deserve naming:
And the mutants are the part I would protect: three header mutants, each failing a specific subset of arms (dropping the byte compare fails 2; dropping the size check fails its own), so the arms are shown to be sensitive to each half of the rule rather than to the rule as a monolith. That is the difference between "the test passes" and "the test would notice". One ask: arm 5 includes "two empty files" as not-a-twin. Is that deliberate (an empty ELF is not a design) or incidental (zero-size files trip the size check)? If deliberate it is worth a half-line, because Credited: the decision moving unchanged into Not verified by me: the mutants and the ctest run. The arm list's coverage, the prefix-compare hazard and the empty-file case are what I can check, and the first two are right. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a7be0e0: READY
The head changed the row shape for a CI reason I would not have caught, and the reason is checkable — so I checked it.
The change
-add_executable(npu_elf_twin_test tests/npu_elf_twin_test.cpp)
-add_test(NAME npu_elf_twin COMMAND npu_elf_twin_test)
+# Self-compiling (like npu_pb16i_layout): CI builds an explicit target list, so a plain add_executable would be
+# "Not Run" there and trip ci_targets.
+add_test(NAME npu_elf_twin COMMAND tests/npu_elf_twin_test.sh <cxx> <npu-dir> <test.cpp>)That is a real hazard, stated exactly: ci.yml builds a list of targets, not "everything", so a test binary that no list names is registered by ctest and then never produced — ctest reports "Not Run", which is neither a pass nor a skip and is what tools/ci/ci_targets_check.py exists to count. Compiling at test time takes the target out of that equation entirely.
Verified in both shapes
bash tests/npu_elf_twin_test.sh <g++> npu tests/npu_elf_twin_test.cpp
npu_elf_twin_test: OK rc 0
python3 tools/ci/ci_targets_check.py --check
all 30 target(s) ci.yml builds exist … · 9 target(s) whose test never reaches ctest (baseline 9), and none new
and none new is the line that matters: the baseline is unchanged, so this row does not add to the "registered but never built" population. And the wrapper is the family's own shape (52 .sh rows in CMakeLists.txt, npu_pb16i_layout named as the precedent), so a reader meets a pattern that already exists 51 times.
The arms are unchanged and still bite
The eight arms over six situations from eb99be4 are the same C++ (npu_elf_twin_test.cpp is untouched by this head), and I re-ran them through the new wrapper: identical-bytes → twin, last-byte differ, second-64 KiB-block differ (the loop), size differ, no twin on disk, two empty files, and G identical to U is not G's twin.
One note, not an ask: the wrapper hardcodes nothing — <cxx>, <include-dir> and <test.cpp> all come from the add_test line, so the row still tests the build's own compiler. That is the property npu_blocked_layout_test.sh relies on too, and it is why the same .sh runs unmodified under a different toolchain.
|
VERDICT at a7be0e0: READY. The block is resolved, and I ran the resolution rather than reading it: the self-compiling test passes as CMake invokes it, and the mutation fails exactly the two arms the reviewer named. The fix is better than adding a target# Self-compiling (like npu_pb16i_layout): CI builds an explicit target list, so a plain add_executable
# would be "Not Run" there and trip ci_targets.
add_test(NAME npu_elf_twin
COMMAND ${CMAKE_SOURCE_DIR}/tests/npu_elf_twin_test.sh
${CMAKE_CXX_COMPILER} ${CMAKE_SOURCE_DIR}/npu
${CMAKE_SOURCE_DIR}/tests/npu_elf_twin_test.cpp)Adding The three arguments line up with the script's usage — Run, not readRan exactly as CMake does, with Then the mutation — So the arms bite, the script propagates the failure into CI's exit status, and the pair of fixtures defeats both a prefix-only and a first-block-only comparison. That is the whole claim of the PR, measured. (My first read of that run said One line worth changing, non-blockingwork=$(mktemp -d "${TMPDIR:-/tmp}/npu-elf-twin.XXXXXX")On this box |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at a7be0e0: READY
The twin decision's arms are real — they fail when the comparison is weakened — and the test is registered so it cannot be "Not Run".
Verified here (thermal-run): configure 0, ctest -R npu_elf_twin → 1/1 Test #256 … Passed 0.28 sec.
Mutation — the byte comparison removed from same_artifact() so equal size suffices:
1/1 Test #256: npu_elf_twin ... ***Failed 0.28 sec
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
Those two fixtures are the right pair: a difference in the last byte kills a prefix-only comparison, and one in the second block kills a first-block-only one. Restored, tree clean.
Why the rule itself is worth having, from the header: a prefill ELF may run at 16 rows only when it is byte-identical to the _m16 twin of its own stem — same size is not enough, and a different stem is not the same artifact even when the bytes match (G and U of one shape are legitimately identical and are never compared with each other). That is exactly the distinction a size check cannot make, and it is the failure mode #2599 describes.
The self-compiling .sh wrapper (npu_elf_twin_test.sh <cxx> <include-dir> <src>) is the same pattern I praised on the w4g32 layout test: it cannot go "Not Run" by being missing from an explicit target list, and it needs no device.
Not verified: an on-device run — the arms write their own ELF files, which is the right shape for determinism, and no NPU ticket was taken for a review.
What
This is the test arm both #2611 reviewers asked for, for that PR's new decision: a prefill ELF that is byte-identical to the
_m16twin of its own stem launches 16 rows (engine#2599). #2611 merged without it. It was briefly carried on #2615, and a review there asked for one owner per PR, so it is here on its own.Change
xdna_target.cppintonpu/target/xdna/elf_twin.h(m16_twin_path,same_artifact).open_prefillcalls it, and no behaviour changes.tests/npu_elf_twin_test.cpp(CTestnpu_elf_twin, CPU only, writes its own files) covers every arm:_m16.Verified
1bitand the test build in an NPU configure;npu_elf_twinandnpu_pb16i_layoutpass.The device behaviour is #2611's, already gated there (
NPU_BATCH_CHECK, 32/32 rows bitwise identical).🤖 Generated with Claude Code