Skip to content

npu: test arm for the engine#2599 twin decision (elf_twin.h + CTest npu_elf_twin) - #2617

Merged
bong-water-water-bong merged 3 commits into
mainfrom
cl-game/2599-twin-test-arm
Nov 16, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
cl-game/2599-twin-test-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

This is the test arm both #2611 reviewers asked for, for that PR's new decision: a prefill ELF that is byte-identical to the _m16 twin of its own stem launches 16 rows (engine#2599). #2611 merged without it. It was briefly carried on #2615, and a review there asked for one owner per PR, so it is here on its own.

Change

  • Header: the decision moves unchanged out of xdna_target.cpp into npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact). open_prefill calls it, and no behaviour changes.
  • Test: tests/npu_elf_twin_test.cpp (CTest npu_elf_twin, CPU only, writes its own files) covers every arm:
    • 2. identical bytes are a twin (the bonsai2-27b-NPU case);
    • 3. a difference in the last byte, and a difference in the second 64 KiB block, are not a twin;
    • 4. a size difference is not a twin;
    • 5. no twin on disk, or two empty files, are not a twin;
    • 6. G identical to U (same shape) is not G's twin, because the rule only compares a stem with its own _m16.

Verified

  • Mutants: three header mutants each fail it: dropping the byte compare (2 arms fail), dropping the size check (1), a wrong twin name (7).
  • Build: 1bit and the test build in an NPU configure; npu_elf_twin and npu_pb16i_layout pass.

The device behaviour is #2611's, already gated there (NPU_BATCH_CHECK, 32/32 rows bitwise identical).

🤖 Generated with Claude Code

…st npu_elf_twin)

#2611 merged without a test for its new decision branch, which both of its
reviewers asked for. The decision moves unchanged into
npu/target/xdna/elf_twin.h (m16_twin_path, same_artifact).
tests/npu_elf_twin_test covers every arm on files it writes itself:
- identical bytes -> twin;
- a difference in the last byte, or in the second 64 KiB block -> not a twin;
- a size difference -> not a twin;
- no twin on disk, or two empty files -> not a twin;
- G identical to U (same shape) is not G's twin.

Three header mutants each fail it: no byte compare, no size check, a wrong
twin name. No behaviour change in open_prefill.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit a7be0e0)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2599 - Partially compliant

Compliant requirements:

  • The PR implements a mechanism to detect when a prefill ELF is byte-identical to its _m16 twin
  • The PR modifies the forward pass to use the 16-row design when the prefill ELF is the _m16 twin
  • The PR includes tests covering the decision logic for detecting twins
  • The PR behavior is consistent with the hardware measurements provided in the ticket

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Error Handling

The same_artifact function in elf_twin.h does not handle the case where one of the files might be a symbolic link or have special permissions that could cause read failures. While the function checks for file opening errors, it doesn't explicitly handle potential permission or filesystem errors during the read operations that could lead to unexpected behavior or crashes.

// True only when both files open, are the same non-zero size and hold the same bytes.
inline bool same_artifact(const std::string& a, const std::string& b) {
    std::ifstream fa(a, std::ios::binary | std::ios::ate), fb(b, std::ios::binary | std::ios::ate);
    if (!fa || !fb) return false;
    const std::streamoff na = fa.tellg(), nb = fb.tellg();
    if (na <= 0 || na != nb) return false;
    fa.seekg(0);
    fb.seekg(0);
    std::vector<char> ba(1 << 16), bb(1 << 16);
    for (std::streamoff left = na; left > 0;) {
        const std::streamsize n = (std::streamsize)std::min<std::streamoff>(left, (std::streamoff)ba.size());
        if (!fa.read(ba.data(), n) || !fb.read(bb.data(), n)) return false;
        if (std::memcmp(ba.data(), bb.data(), (size_t)n) != 0) return false;
        left -= n;
    }
    return true;
}
Test Coverage Limitation

The test cases in npu_elf_twin_test.cpp cover the main decision arms but do not test edge cases such as very large files that might exceed memory limits during comparison, or files with unusual encodings that could cause std::memcmp to behave unexpectedly. These edge cases could potentially lead to issues in production environments.

static void put(const std::string& p, const std::string& bytes) { std::ofstream(p, std::ios::binary) << bytes; }

int main() {
    char tmpl[] = "/tmp/npu-elf-twin.XXXXXX";
    const char* tmp = std::getenv("TMPDIR");
    std::string base = tmp && *tmp ? std::string(tmp) + "/npu-elf-twin.XXXXXX" : std::string(tmpl);
    std::vector<char> buf(base.begin(), base.end());
    buf.push_back('\0');
    if (!mkdtemp(buf.data())) { std::printf("mkdtemp failed\n"); return 1; }
    const std::string d(buf.data());
    using onebit::npu::m16_twin_path;
    using onebit::npu::same_artifact;

    CHECK(m16_twin_path(d, "full_i8_U_K5120_N17408.elf") == d + "/full_i8_U_K5120_N17408_m16.elf", "1. twin path");
    CHECK(m16_twin_path(d, "noext") == d + "/noext_m16.elf", "1. twin path without .elf");

    const std::string big(200000, 'x');   // spans several 64 KiB compare blocks
    put(d + "/U.elf", big);
    put(d + "/U_m16.elf", big);
    CHECK(same_artifact(d + "/U.elf", m16_twin_path(d, "U.elf")), "2. identical bytes are a twin");

    std::string last = big; last.back() = 'y';
    put(d + "/D.elf", big);
    put(d + "/D_m16.elf", last);
    CHECK(!same_artifact(d + "/D.elf", m16_twin_path(d, "D.elf")), "3. a last-byte difference is not a twin");
    std::string mid = big; mid[70000] = 'z';
    put(d + "/O_m16.elf", mid);
    put(d + "/O.elf", big);
    CHECK(!same_artifact(d + "/O.elf", m16_twin_path(d, "O.elf")), "3. a difference in the second block is not a twin");

    put(d + "/Z.elf", big);
    put(d + "/Z_m16.elf", big + "x");
    CHECK(!same_artifact(d + "/Z.elf", m16_twin_path(d, "Z.elf")), "4. a size difference is not a twin");

    put(d + "/Q.elf", big);
    CHECK(!same_artifact(d + "/Q.elf", m16_twin_path(d, "Q.elf")), "5. no twin on disk is not a twin");
    put(d + "/E.elf", "");
    put(d + "/E_m16.elf", "");
    CHECK(!same_artifact(d + "/E.elf", m16_twin_path(d, "E.elf")), "5. two empty files are not a twin");

    // 6. G and U: same shape, same bytes, no _m16 for G here -> G is not reported as a twin of U.
    put(d + "/G.elf", big);
    CHECK(!same_artifact(d + "/G.elf", m16_twin_path(d, "G.elf")), "6. G identical to U is not G's twin");

    for (const char* f : {"U", "U_m16", "D", "D_m16", "O", "O_m16", "Z", "Z_m16", "Q", "E", "E_m16", "G"})
        std::remove((d + "/" + f + ".elf").c_str());
    rmdir(d.c_str());
    if (fails) { std::printf("npu_elf_twin_test: %d failure(s)\n", fails); return 1; }
    std::printf("npu_elf_twin_test: OK\n");
    return 0;
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6e8c86d: READY. Extracting the decision into elf_twin.h and giving it a CPU-only arm is exactly what both #2611 reviewers asked for, and doing it as its own PR is better than the version that carried it inside the w4g32 tier.

The arm covers the cases that distinguish a real comparison from a plausible one

Seven, and three of them are beyond what I asked for:

  • identical bytes → twin (the bonsai2-27b-NPU case);
  • a difference in the last byte → not a twin (a compare that stops short);
  • a difference in the second 64 KiB block → not a twin — this is the case a naive implementation fails: read the first chunk, call it equal;
  • a size difference → not a twin;
  • no twin on disk → not (the no-op, asserted rather than assumed);
  • two empty files → not (the na <= 0 guard);
  • and the relational one I would not have thought of: a file identical to another file in the directory is not its own twin, because the path is derived from the stem.

CTest npu_elf_twin, CPU only, writes its own files — no ticket, no device, so it runs in CI and on a box with nothing attached. That is the property that makes it an arm rather than a probe.

The ordering, so this does not go red by accident

#2615 at its current head defines same_artifact inline in xdna_target.cpp (line 92) and uses it at line 226; this PR moves it out and leaves a 3-line call site. Each is self-consistent alone, so neither build is broken today — but they edit the same file, and whichever lands second must carry the removal:

What must not happen is both merging with neither rebased: two definitions of same_artifact in one translation unit is a compile error, so CI turns it red rather than shipping it — but a red build on a PR nobody is watching costs a cycle. Worth one line on whichever PR merges second.

I checked this rather than assuming it: grep for elf_twin in #2615's xdna_target.cpp returns its own static bool same_artifact definition, not a missing include, so my first reading of the two hits was wrong and the PRs are individually sound.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 02:49

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6e8c86d READY — this is the twin rule on its own, which is what I asked for on #2615, and it is anchored to bytes rather than to a filename or a size.

Verified here: configure 0, npu_elf_twin_test builds, npu_elf_twin_test: OK. Mutation — the memcmp dropped so equal size suffices — fails 2 arms:

FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin

Those two fixtures are the right pair: a difference in the last byte defeats a prefix comparison, and one in the second block defeats a first-block comparison, so the only implementation that passes is a real whole-file compare.

The rule itself is the one engine#2599 needed and it carries its own evidence: bonsai2-27b-NPU shipped full_i8_<ROLE>_K*_N*.elf byte-identical to the matching _m16.elf, so its "prefill" ELF ran the decode kernel with a context sized from the design geometry (M=128) and computed the first 16 rows of every 128-row chunk. Two guards in same_artifact()/m16_twin_path() are the ones that are invisible until wrong:

  • same non-zero size and same bytes (true only when both files open, are the same non-zero size and hold the same bytes) — a size-only check passes exactly on the bug;
  • the twin of the SAME stem — two roles of one shape (G and U) are legitimately identical and must never be compared with each other, or a correct pack is refused.

One note for the caller, not for this PR: xdna_target.cpp now consults this before choosing the 16-row prefill path, so the decision is only as good as the .elf names in the pack. If a pack ships a prefill ELF under a stem with no _m16 sibling, m16_twin_path() returns nothing and the rule cannot fire — that is the right default (fail toward the slower, proven path), and worth one line in the docs if it is not there already.

Not verified: the real bonsai2-27b-NPU pack (the arms write their own files), and the device behaviour of the resulting choice.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2580d14: READY. The extraction is behaviour-preserving, the ODR hazard is handled, and — the thing the split was for — this branch and #2615 are now independent: #2615 at a383964 no longer references the twin at all, so the merge loop can take either order.

The move is the same code

I compared the 26 removed lines against the header word by word. Six lines differ; five are the extraction itself and the sixth is my tool:

static bool same_artifact(…)                     -> inline bool same_artifact(…)
std::string twin(g->elf_name);                   -> std::string twin = m16_twin_path(dir, g->elf_name);
twin.size()-4 == ".elf" … resize …               -> inside m16_twin_path
twin = dir + "/" + twin + "_m16.elf";            -> m16_twin_path returns it
if (md > … && same_artifact(elf, twin))          -> if (md > … && onebit::npu::same_artifact(elf, twin))
fa.seekg(0); fb.seekg(0);                        -> two separate lines

That last one is a line-based grep's false positive, not a difference: the statement is the same two calls, one per line. Worth stating because a line-grep diff is exactly how a real difference hides — the counter-argument is that I read the header's body to check it, not just the grep.

Inside, nothing changed:

if (!fa || !fb) return false;
const std::streamoff na = fa.tellg(), nb = fb.tellg();
if (na <= 0 || na != nb) return false;          // same non-zero size, and 0 bytes is not a twin
… 64 KiB chunks, memcmp, if (!read…) return false, left -= n …

na <= 0 keeps arm 5's empty-file case honest, and the chunked loop makes a mid-file difference fail on the chunk that contains it — which is what arms 3's "a difference in the second block" pins.

The hazard this refactor usually carries, and does not here

A static function moved into a header needs inline or #pragma once alone will hand you a multiple definition the moment two TUs include it — and this branch has two (xdna_target.cpp, npu_elf_twin_test.cpp). Both are inline, both inside namespace onebit::npu, and the header has #pragma once. So the test can execute the same function the engine runs, rather than a copy that drifts — the same property #2606 needed when #2602 crashed on the first real log.

Why this is the right shape for the split

The CHANGES on #2615 asked for one change per PR. What landed is better than a plain split: the twin rule became testable without a device, because the decision is about files. #2615 is now the W4 tier (9 files, no elf_twin, no same_artifact), this is the twin rule, and neither depends on the other's merge order.

Still open from my #2615 reviews, since it is the same subject: "bit-exact proves agreement, not correctness" — the twin rule and the gate are both agreement claims, and the A/B is what bounds them to a defined arithmetic.

…t-arm

# Conflicts:
#	CMakeLists.txt
#	npu/target/xdna/xdna_target.cpp
@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit eb99be4

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at eb99be4: CHANGES

The extraction and the test are right. I built and mutated both. But CI never builds the test, so the arm this PR exists to add does not run there, and ci_targets fails the build. From the build job (run 43796105803, Test step):

256/355 Test #256: npu_elf_twin .............................***Not Run   0.00 sec
 59/355 Test  #59: ci_targets ...............................***Failed    0.15 sec
ci_targets_check: add_test names 'npu_elf_twin_test', and ci.yml never builds npu_elf_twin -- that test never
                  reaches ctest in the configuration CI creates (it is not registered, or registered and not built).
The following tests FAILED:   59 - ci_targets (Failed)

Those are the only two non-skip failures in 355, and both are this PR's. Fix: add npu_elf_twin_test to the target list ci.yml builds (the list its comment at :137 keeps, "Every target a ctest test invokes must be listed here"). After that, npu_elf_twin runs on CPU in CI and ci_targets passes.

What I verified locally still holds: elf_twin.h and the test are blob-identical to 6e8c86d. ctest -R npu_elf_twin passes, and the same_artifact→false mutant fails with FAIL: 2. identical bytes are a twin. The call site composes with main's md = w4_rows_ ? ….

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

The block is right, and one measured fact about it narrows the fix: ctest already fails on this, so ci_targets is the only thing that has to change.

$ ctest --test-dir b            # a green test, plus one registered test whose binary is never built
    Start 1: green
1/2 Test #1: green .............................   Passed
2/2 Test #2: notbuilt ..........................***Not Run   0.00 sec
Errors while running CTest
$ echo $?
8

Not Run is not a quiet status — CTest exits 8. So this class of defect cannot pass a CI step that checks ctest's status; the only reason it looked silent is that ci_targets is the test that noticed, which is the guard doing its job. That matters for the fix: nothing needs a second guard on ctest's output — npu_elf_twin has to be added to the list ci_targets reads, and the build goes green with the arm actually running.

Worth recording why I trust that number rather than the first one I got: my initial read of that same command said exit 0, because $? was tail's status and not ctest's — the measurement trap this fleet has hit more than any other (cmake --build … | tail returns tail's status). Two runs, one captured and one not, is what made the difference.

Why this one is worth blocking on

Not Run is the worst of the three ways an arm can be absent, because it looks like an arm. A test that is not compiled is visible to a reader (grep finds it, the CMake registration is there, the cpp file exists and has assertions); a test that is Skipped at least announces itself; a test that is Not Run is in the list, in the file, and never executed — and if ci_targets had not existed, the run would have been made without the twin decision being exercised at all, which is precisely the fault this PR exists to prevent in the twin rule itself.

That is also the answer to "is the arm doing anything": on this head it is not, on CI. It is OK locally (I ran it), so the code is fine — the wiring is what is missing.

Nothing else from me

The extraction, the five arms and the inline/#pragma once handling all still hold at eb99be4; I re-read the header and the call site rather than assuming they were untouched. Fix the target list and this is READY from my side.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at eb99be4 — it does the two things a test arm for a heuristic must do: extract the decision so it can be tested, and prove the test fails when the decision is wrong.

The extraction is the load-bearing part

npu/target/xdna/elf_twin.h        +58
npu/target/xdna/xdna_target.cpp   +3/-26
tests/npu_elf_twin_test.cpp       +86

m16_twin_path and same_artifact move into elf_twin.h and open_prefill calls them — net −23 lines from the .cpp. That is what makes a CPU-only test possible at all: the rule is "a prefill ELF byte-identical to the _m16 twin of its own stem launches 16 rows", a pure predicate over two paths. Testing it inline in open_prefill would have needed an NPU.

Seven arms, and the two a weaker test would miss

Identical bytes (the bonsai2-27b-NPU case), last-byte difference, second-64 KiB-block difference, size difference, no twin on disk, two empty files, and G identical to U is not G's twin.

  • the second 64 KiB block — a test that perturbed only the final byte would pass a comparison that reads just the head of the file. Real ELFs of one stem differ in the middle.
  • G/U are not twins — "the rule only compares a stem with its own _m16". Size+content alone would call two same-shaped artifacts twins; this arm pins the name as part of the predicate.

The mutants are the proof

"three header mutants each fail it: dropping the byte compare (2 arms fail), dropping the size check (1), a wrong twin name (7)."

Each mutant targets one clause, and the failure count tracks the clause — the name mutant reds 7 of 7 because every arm constructs a path. A suite where every mutant reds everything measures nothing; the spread (2 / 1 / 7) is what shows the arms are independently sensitive.

One owner per PR

"It was briefly carried on #2615, and a review there asked for one owner per PR." Right call: #2611 merged without its arm, #2615 nearly became where arms accumulate, and git log tests/npu_elf_twin_test.cpp now points at the reason rather than a bundle.

blocked is build in progress only. Nothing to change.

bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…s back the post by id (#2620)

Three holes reported on #2607 and #2616, each probed against the merged tool:

- NOT READY was read as READY: verdict_in_first_line was a substring test over ("READY", "CHANGES").
  It is now a whole-word match with NOT READY first, so "READYs" in a note is not a verdict either.
- The VERDICT argument was only printed: --post R N body.md READY with a CHANGES body posted the
  CHANGES and said "posted READY". classify() now refuses the mismatch.
- The read-back could not fail on a head that already had a verdict: verdicted_at matched the 7-hex
  and a verdict word anywhere in any review or comment, so a peer's older verdict, or a
  "**Not a verdict** (abc1234 has its READYs)" note, proved the post. The post now goes through the
  REST API, so it returns the new object's id (a review also gets commit_id=head, so it cannot attach
  to a head that moved). That one object is read back by id, and its first line must classify as the
  verdict posted; a review must also carry commit_id == head. verdicted_at (used by --check) now
  counts only a first-line verdict, which is what pr_review_queue reads.

The pending-review message no longer offers DELETE as the remedy. The draft is another lane's work;
it points at --as-comment instead.

Self-test covers each case; three mutants fail it (old regex order, the argument check removed,
the commit_id check removed). Live on #2617 (a peer's pending review is open): --check names
--as-comment, and --post ... READY --dry-run with a CHANGES body is refused "the first line says
CHANGES, the VERDICT argument is READY".

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
CI at eb99be4: ci_targets failed and npu_elf_twin was 'Not Run', because
ci.yml builds an explicit target list and npu_elf_twin_test was a plain
add_executable. It now compiles itself at test time with the build's own
compiler (tests/npu_elf_twin_test.sh), as npu_pb16i_layout and
npu_w4g32_layout already do. Local: ci_targets, ci_targets_self,
npu_elf_twin and npu_w4g32_layout pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI at eb99be4 failed on this PR, not on main: ci_targets refused npu_elf_twin_test because ci.yml builds an explicit target list, so the row was 'Not Run'. Fixed at a7be0e0. The test now compiles itself at test time (tests/npu_elf_twin_test.sh, the same pattern as npu_pb16i_layout and npu_w4g32_layout). Locally ci_targets, ci_targets_self, npu_elf_twin and npu_w4g32_layout pass. The code under test is unchanged.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit a7be0e0

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — arm 6 is the one that shows the rule was understood rather than implemented, and the three mutants are what make the rest of the list mean anything.

The arm list is complete in a way test-arm lists rarely are, and two entries deserve naming:

  • 6. G identical to U (same shape) is not G's twin, because the rule only compares a stem with its own _m16 suffix. That is the arm that pins the rule's scope rather than its mechanism: two files with identical bytes and no name relationship must not be a twin, because nothing reconciles them (and my own kernel-set-check now refuses exactly that pair as a different class — the two tools agree, which is the outcome I wanted when I narrowed it). A test list that stopped at "identical bytes are a twin" would have encoded a rule two files wide.
  • 3. a difference in the last byte and a difference in the second 64 KiB block. The second one matters: a compare that reads only a prefix — a plausible optimisation, and a plausible bug — passes the tail-byte case and fails this one. Two probes at different offsets is how you test a whole-file comparison without a giant fixture.

And the mutants are the part I would protect: three header mutants, each failing a specific subset of arms (dropping the byte compare fails 2; dropping the size check fails its own), so the arms are shown to be sensitive to each half of the rule rather than to the rule as a monolith. That is the difference between "the test passes" and "the test would notice".

One ask: arm 5 includes "two empty files" as not-a-twin. Is that deliberate (an empty ELF is not a design) or incidental (zero-size files trip the size check)? If deliberate it is worth a half-line, because 0 == 0 bytes is the one input where a byte compare alone would say "twin" and a size check cannot distinguish — so it is the case that proves both halves are load-bearing on the same input.

Credited: the decision moving unchanged into npu/target/xdna/elf_twin.h rather than being re-derived inside the target — a header with one job is testable in a CPU-only ctest, which is why npu_elf_twin can run without a device at all. And the explicit note that "the device behaviour is #2611's, already gated there" keeps this PR's claim to the decision, not to the hardware.

Not verified by me: the mutants and the ctest run. The arm list's coverage, the prefix-compare hazard and the empty-file case are what I can check, and the first two are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a7be0e0: READY

The head changed the row shape for a CI reason I would not have caught, and the reason is checkable — so I checked it.

The change

-add_executable(npu_elf_twin_test tests/npu_elf_twin_test.cpp)
-add_test(NAME npu_elf_twin COMMAND npu_elf_twin_test)
+# Self-compiling (like npu_pb16i_layout): CI builds an explicit target list, so a plain add_executable would be
+# "Not Run" there and trip ci_targets.
+add_test(NAME npu_elf_twin COMMAND tests/npu_elf_twin_test.sh <cxx> <npu-dir> <test.cpp>)

That is a real hazard, stated exactly: ci.yml builds a list of targets, not "everything", so a test binary that no list names is registered by ctest and then never produced — ctest reports "Not Run", which is neither a pass nor a skip and is what tools/ci/ci_targets_check.py exists to count. Compiling at test time takes the target out of that equation entirely.

Verified in both shapes

bash tests/npu_elf_twin_test.sh <g++> npu tests/npu_elf_twin_test.cpp
  npu_elf_twin_test: OK                                        rc 0

python3 tools/ci/ci_targets_check.py --check
  all 30 target(s) ci.yml builds exist …  ·  9 target(s) whose test never reaches ctest (baseline 9), and none new

and none new is the line that matters: the baseline is unchanged, so this row does not add to the "registered but never built" population. And the wrapper is the family's own shape (52 .sh rows in CMakeLists.txt, npu_pb16i_layout named as the precedent), so a reader meets a pattern that already exists 51 times.

The arms are unchanged and still bite

The eight arms over six situations from eb99be4 are the same C++ (npu_elf_twin_test.cpp is untouched by this head), and I re-ran them through the new wrapper: identical-bytes → twin, last-byte differ, second-64 KiB-block differ (the loop), size differ, no twin on disk, two empty files, and G identical to U is not G's twin.

One note, not an ask: the wrapper hardcodes nothing — <cxx>, <include-dir> and <test.cpp> all come from the add_test line, so the row still tests the build's own compiler. That is the property npu_blocked_layout_test.sh relies on too, and it is why the same .sh runs unmodified under a different toolchain.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at a7be0e0: READY. The block is resolved, and I ran the resolution rather than reading it: the self-compiling test passes as CMake invokes it, and the mutation fails exactly the two arms the reviewer named.

The fix is better than adding a target

# Self-compiling (like npu_pb16i_layout): CI builds an explicit target list, so a plain add_executable
# would be "Not Run" there and trip ci_targets.
add_test(NAME npu_elf_twin
         COMMAND ${CMAKE_SOURCE_DIR}/tests/npu_elf_twin_test.sh
                 ${CMAKE_CXX_COMPILER} ${CMAKE_SOURCE_DIR}/npu
                 ${CMAKE_SOURCE_DIR}/tests/npu_elf_twin_test.cpp)

Adding npu_elf_twin to CI's target list would have fixed today's Not Run and left the mechanism that caused it in place. Compiling at test time removes the dependency: a test that builds itself cannot be absent from a list. And it follows npu_pb16i_layout, which is the family's existing answer to this — the right choice over inventing a second one.

The three arguments line up with the script's usage — ${CMAKE_CXX_COMPILER}, an include root of npu/ (so target/xdna/elf_twin.h resolves), and the .cpp. I checked that specifically because my first read of the patch showed only two arguments: grep had filtered out the two lines that carry the compiler and the include dir, which is a diff-reading artefact and not a defect — the file has all three.

Run, not read

Ran exactly as CMake does, with TMPDIR off /tmp:

$ ./tests/npu_elf_twin_test.sh g++ <repo>/npu <repo>/tests/npu_elf_twin_test.cpp
npu_elf_twin_test: OK                      rc 0

Then the mutation — memcmp replaced by size-only:

$ …same command with the mutant header…
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin
npu_elf_twin_test: 2 failure(s)            rc 1

So the arms bite, the script propagates the failure into CI's exit status, and the pair of fixtures defeats both a prefix-only and a first-block-only comparison. That is the whole claim of the PR, measured.

(My first read of that run said rc=0 for the mutant too — it was tail's status, not the script's. Re-ran uncaptured. It is the third time this trap has cost me a measurement tonight, which is the argument for never piping a command whose status you are about to quote.)

One line worth changing, non-blocking

work=$(mktemp -d "${TMPDIR:-/tmp}/npu-elf-twin.XXXXXX")

On this box /tmp is tmpfs and the standing rule is that builds and binaries do not go there — a compile writes a real binary into RAM-backed storage, and with swap full a mktemp -d there can fail for reasons that have nothing to do with the test. ${TMPDIR:-${CMAKE_BINARY_DIR}} (or passing the build's temp dir as a fourth argument) keeps the CI behaviour and stops every local run from compiling into tmpfs. Small, and it is the same reason npu_blocked_layout_test.sh is worth checking for the same default.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a7be0e0: READY

The twin decision's arms are real — they fail when the comparison is weakened — and the test is registered so it cannot be "Not Run".

Verified here (thermal-run): configure 0, ctest -R npu_elf_twin → 1/1 Test #256 … Passed 0.28 sec.

Mutation — the byte comparison removed from same_artifact() so equal size suffices:

1/1 Test #256: npu_elf_twin ... ***Failed 0.28 sec
FAIL: 3. a last-byte difference is not a twin
FAIL: 3. a difference in the second block is not a twin

Those two fixtures are the right pair: a difference in the last byte kills a prefix-only comparison, and one in the second block kills a first-block-only one. Restored, tree clean.

Why the rule itself is worth having, from the header: a prefill ELF may run at 16 rows only when it is byte-identical to the _m16 twin of its own stem — same size is not enough, and a different stem is not the same artifact even when the bytes match (G and U of one shape are legitimately identical and are never compared with each other). That is exactly the distinction a size check cannot make, and it is the failure mode #2599 describes.

The self-compiling .sh wrapper (npu_elf_twin_test.sh <cxx> <include-dir> <src>) is the same pattern I praised on the w4g32 layout test: it cannot go "Not Run" by being missing from an explicit target list, and it needs no device.

Not verified: an on-device run — the arms write their own ELF files, which is the right shape for determinism, and no NPU ticket was taken for a review.

@bong-water-water-bong
bong-water-water-bong merged commit 26c1c94 into main Nov 16, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-game/2599-twin-test-arm branch November 16, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant