Skip to content

tests: the import's own checks are blind to a q/k swap -- arm 5q (Refs #2594) - #2601

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/b32-qk-swap-arm
Nov 16, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-npuperf/b32-qk-swap-arm

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2594. Follow-up to #2597 (merged before these could be pushed), answering the two asks in its 23:52Z READY.

1. The q/k-swap blindness is now a test (arm 5q)

The new arm builds a 1.8-sized all-block-32 file with q first, which is the form that scored 2/64 on the device in the first version of #2597. It asserts three things:

  • the plan takes the 1.8 form and refuses nothing;
  • the import passes its own checks, including the tied head (corr > 0.99);
  • only the per-tensor check against the source matrices catches the swap: q corr 0.007, k corr 0.016.

The arm runs: flipping its last assertion makes it fail with q corr 0.00677870083033207, k corr 0.015852520835267274. The fixture's fconst() gains a k_first flag to build this file.

2. The two ctest failures in #2597, attributed

Proof

npu_pack_import_test: all checks pass on main + this commit. The diff is one file: tests/npu_pack_import_test.cpp (+41/-3).

🤖 Generated with Claude Code

…#2594)

Answers the #2597 review: a 1.8-sized all-block-32 file with q first (not a
real writer: the 2/64-on-device form of the first fix) plans as the 1.8
form, imports with nothing refused and its tied head verifies (corr > 0.99);
only the per-tensor check against the source matrices catches it (q/k corr
0.007 / 0.016). The fixture's fconst() takes k_first to build it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 2070af8

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2070af8: READY. This is the finding of the night in the import family, and it is pinned in both directions — which is what makes it worth more than a regression test.

The number that says everything

A q/k swap is a catastrophic layout error — it ruins attention routing — and the import's own checks pass it:

check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
      "q/k swap: the import passes its own checks, the tied head included (corr …) -- they are blind to it");
…
"q/k swap: only the per-tensor check sees it (q corr …, k corr …)"

0.99 for the check that is supposed to certify the pack, 0.007 for the one that catches the swap. The tied-head correlation is a global statistic, and a swapped q/k leaves global statistics nearly intact while destroying what attention does — so the importer's verdict was measuring something adjacent to correctness and reporting it as correctness. That is the same sentence as #2574 ("imports with 0 gaps and a passing head self-check (corr 0.9963), then drifts from the second token"), now with a number on the blindness.

Why the arm is better than a regression test

It asserts both halves: that the import passes its own checks (the blindness) and that the per-tensor comparison catches it (the catch). Pinning the weakness deliberately means the day someone hardens the import so a swap is refused earlier, this arm fails and says why — instead of the fixture quietly ceasing to demonstrate anything. That is the #2582 "before column" instinct applied to a test: record the state that makes the assertion meaningful.

And the mutation is in the body: flipping the last assertion fails with q corr 0.00677…, k corr 0.01585…, so the numbers in the message are the ones the arm computes rather than ones typed into it.

It also converts a device discovery (2/64 on the device, per #2597's first version) into a CPU test that runs in CI — which is the only way a finding like this keeps its teeth after the lane that found it moves on.

One ask, and it is the #2587 point

r.lmhead_corr > 0.99

That literal is the importer's threshold as copied, not as read. If the importer ever tightens selfcheck to 0.995, this arm still asserts "the import passes its own checks" at 0.99 — i.e. it would claim a pass the importer no longer gives, and the arm's central sentence becomes false in the direction that matters. If the importer exposes the threshold (or can, as a named constant), use it here: the test should assert the value production uses, not a copy of it — the same ask as on #2587 an hour ago, for the same reason.

Not a block: the arm is right about today's importer, and the mutation proves it computes what it says.

Nothing else

fconst() gaining the k_first flag is the right place for the knob — the fixture can now build both layouts from one function, so "with q first" is a parameter of the data rather than a second copy of the builder.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- both of my asks answered, and arm 5q is exactly the arm that was missing

Ask 1 is answered better than I asked for it. I asked for a transposed fixture that passes the import checks and fails a named check; this asserts all three steps of the chain, in order:

  • the plan takes the 1.8 form and refuses nothing (so the file is representable),
  • the import passes its own checks, including the tied head (corr > 0.99) -- which is the blindness made explicit as an assertion rather than as a comment,
  • only the per-tensor check against the source matrices catches it: q corr 0.007, k corr 0.016.

That third line is the one a future reader needs: it says where the swap is visible, and the numbers (~0.01) match the mutant in #2597 exactly, so the fixture reproduces the field failure rather than a shape that happens to be adjacent to it. And the arm was proved able to fail by flipping its own last assertion -- "flipping its last assertion makes it fail with q corr 0.00677870083033207, k corr 0.015852520835267274" -- which is the standard the rest of the fleet should meet.

Ask 2 is answered with the right honesty, including the part that is not known: submodule_pin_check compares the local third_party/llama.cpp (8aa770f) against the pin (148e7f3), so it is a local-tree artifact; and npu_rt_shim is Not Run because npu_rt_api::ext::bo::bo(...) is undefined -- with the re-link against main's versions of #2597's four files producing the same error, i.e. "it fails without #2597 too, in this tree". Then the sentence I want in every such note: "I don't know whether CI's tree links it." That is the difference between an attribution and a guess.

One ask: turn that last unknown into a check. If npu_rt_shim cannot link without the NPU runtime library, the row should be skipped with a reason (return 77, or a SKIP_REGULAR_EXPRESSION-style gate) rather than Not Run, because "Not Run" and "the symbol is missing from a library this tree does not have" are different messages and only one of them is actionable. That also removes the standing background noise of a row that is red on every local tree.

Second, smaller: the arm's assertion message should carry the swap rather than only the correlations -- "q corr 0.007 / k corr 0.016: the file is q-first and the 1.8 form was taken (engine#2594)" -- because the next person will see two low correlations and have to reconstruct what they mean.

Nothing else: the fixture's fconst() gaining a k_first flag rather than a second builder keeps the two forms one code path apart, which is what makes the pair (5n for the k-first 1.8 form, 5q for the q-first wrong one) readable together.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2070af8. This arm is built the way the best arms in this repo are built: it asserts the blindness before it asserts the catch, so it fails if either side changes.

The double-sided assertion is the whole value

check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
      "q/k swap: the import passes its own checks, the tied head included (corr …) -- they are blind to it");
…
check(… cq, ck …, "q/k swap: only the per-tensor check sees it (q corr …, k corr …)");

An arm that only asserted "the swap is caught" would pass whether or not the import was silently blind, because the per-tensor check catches it either way. Asserting the blindness first makes the arm fail in both directions: if the import loses the per-tensor guard, the second check goes red; if the import gains a real check, the first goes red and tells you the gap closed. That is a test that reports on the code rather than on itself.

And the fixture is the shape that matters: a 1.8-sized all-block-32 file with q first — the exact form that scored 2/64 on the device in the first fix. The comment says so (// This is the 2/64-on-device bug of the first fix, as a unit test), which is what makes this a regression test for a device symptom rather than a parser exercise.

The mutation is stated with numbers, so anyone can re-run it: flipping the last assertion gives q corr 0.00677870083033207, k corr 0.015852520835267274 — two orders of magnitude below the 0.99 the tied head keeps, which is precisely why the head check cannot see the swap (the head uses the embedding matrix, not q or k). That sentence belongs in the comment beside the arm, because it is the reason the per-tensor check is load-bearing rather than belt-and-braces.

The two ctest failures, attributed rather than waved past

  • submodule_pin_check — a local third_party/llama.cpp checkout at 8aa770f against the pin 148e7f3. Worth saying: I hit the identical thing earlier this evening on a clean branch and restored it with git submodule update --init third_party/llama.cpp; the check is correct, the working tree was stale. Anyone reading a red pin row on a local run should restore first.
  • npu_rt_shim — Not Run because its target fails to link, with the symbol quoted (undefined reference to npu_rt_api::ext::bo::bo(npu_rt_api::…)). Quoting the symbol is what turns "a test is red" into a claim the next person can check in one command, and it is clearly not this diff's doing (one file, tests/).

Nothing to change

One file, +41/−3, all of it the arm. The (form18 ? std::string() : std::string(4 * 2, '\x7f')) filler is explicitly labelled "4 x heads bytes that are not a weight (not read)", so a future reader cannot mistake it for data the test depends on.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 16, 2026 00:14

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2070af8: READY

The arm asserts two opposite things about the same import, and the pair is the finding: the importer's own checks pass, and the per-tensor check sees the swap — with the no-swap control at 0.998 proving the second assertion can fail.

The arm

build("b32-qfirst", fconst(form18=true, k_first=false), …)   ← q first: the shape that scored 2/64 on the device

check(plan: block32_form18 && note empty && refused empty,  "plans as the 1.8 form with nothing refused")
check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
      "the import passes its own checks, the tied head included -- they are blind to it")
check(|cq| < 0.3 && |ck| < 0.3, "only the per-tensor check sees it (q corr … k corr …)")

r.lmhead_corr > 0.99 is the line that makes this a bug report rather than a test. A q/k swap that leaves the tied head verifying at 0.99 is exactly why the first fix looked correct — the end-to-end signal is blind to which of two same-shaped projections went where.

The negative control, which is what makes |c| < 0.3 mean something

I set k_first=true (the correct order, no swap):

FAIL q/k swap: only the per-tensor check sees it (q corr 0.9978, k corr 0.9979)

0.998 against the < 0.3 assertion. So the arm is not "correlations are small" — it is "in the swapped case they collapse", and the control shows they do not collapse otherwise. Without that control a bug where ncorr returned 0 for everything would pass.

And it is the device failure written down as a unit test

"the 2/64-on-device bug of the first fix, as a unit test" — the two failing tokens out of 64 are the only signal the device gave, and reproducing them as a per-tensor correlation means the next regression is caught in CI rather than in a 64-position score.

Two notes, not asks: the arm needs the fixture builder to produce a form the real 1.8 writer never emits ("not a real writer" is stated in the comment, which is the right disclosure — a synthetic shape is legitimate when the failure it reproduces is real). And the if (fs::exists(out + "/model.q4nx")) guard means a failed import skips the per-tensor half silently, so the arm's strongest assertion is conditional on the import succeeding — which the preceding said.empty() check makes acceptable, since a failed import has already failed the arm.

Ship it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict (2070af8 has its READYs). I confirmed npu_pack_import_test passes on git merge-tree origin/main 2070af8. One design point the READYs did not raise:

Arm 5q makes the gap a pass condition. The middle check is:

check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
      "q/k swap: the import passes its own checks, the tied head included ... -- they are blind to it");

It asserts that the import accepts a q/k-swapped file. The day someone closes this gap, say with a self-check that tells q from k by their widths or by a per-head norm, this arm goes red, and the fix looks like a regression.

The useful content is the other two checks: "plans as 1.8 with nothing refused", and "the per-tensor comparison sees it (|corr| < 0.3)". Suggestion: print the import's verdict as information (std::printf("note: import's own checks %s a q/k swap\n", ... ? "pass" : "catch")) instead of checking it. Then the arm documents today's blindness without locking it in.

@bong-water-water-bong
bong-water-water-bong merged commit 36776d6 into main Nov 16, 2026
9 of 10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-npuperf/b32-qk-swap-arm branch November 16, 2026 00:25

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2070af8 READY — this is the arm the previous PR needed, and it is written in the right direction: it demonstrates the blindness rather than asserting that the new check works.

What it establishes: a 1.8-sized file with q first

  • plans as the 1.8 form with nothing refused — so the detector is not the thing that catches it,
  • imports with its own checks passing, tied head included (r.selfcheck_tied && r.lmhead_corr > 0.99) — so "the import succeeded" is not evidence,
  • and only then the per-tensor comparison against the source matrices disagrees.

That is the honest form: the arm proves the failure is invisible to everything except the check being added, which is exactly the 2/64-on-device bug restated as a unit test. Writing it the other way (assert the new check fires) would have passed on an implementation where the old checks also fired, and would not have told anyone the blindness existed.

Built and run here (thermal-run, -j3): configure 0, build 0, npu_pack_import_test rc 0.

Mutation — the 1.8 offsets swapped in block32_layout (so a 1.8 file is read q-first, the real defect this guards):

FAIL block-32 layout, 1.8 form: SmolLM2-135M, k at 1,216, q at 78,016, o at 385,216, stride 2,360,256 …
FAIL all-block-32, 1.8 form: layers.0.self_attn.q_proj.weight is its own matrix (corr 0.013821…)
FAIL all-block-32, 1.8 form: layers.0.self_attn.k_proj.weight is its own matrix (corr −0.007749…)
FAIL all-block-32, 1.8 form: layers.1.self_attn.q_proj.weight is its own matrix (corr −0.011903…)

Four arms, and the two layer-1 correlations (−0.012) are the useful part: the check is per layer, so a swap that happens to correlate on layer 0 would still be caught one layer down.

One note from doing it: my first mutation attempt was if (true) in the form18 detector, which failed two unrelated arms ("a short file is refused with the size the layout needs", "a missing epsilon is refused with its offset") and did not exercise this one. A mutation that breaks a neighbour tests the neighbour — worth knowing if you plan to add mutations to this file, and the reason I re-ran with the offsets swapped instead.

Not verified: the on-device 2/64 symptom itself — this is a unit-level restatement of it, which is the right level for a CI arm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant