Repository navigation
tests: the import's own checks are blind to a q/k swap -- arm 5q (Refs #2594) - #2601
Conversation
…#2594) Answers the #2597 review: a 1.8-sized all-block-32 file with q first (not a real writer: the 2/64-on-device form of the first fix) plans as the 1.8 form, imports with nothing refused and its tied head verifies (corr > 0.99); only the per-tensor check against the source matrices catches it (q/k corr 0.007 / 0.016). The fixture's fconst() takes k_first to build it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at 2070af8 |
|
VERDICT at 2070af8: READY. This is the finding of the night in the import family, and it is pinned in both directions — which is what makes it worth more than a regression test. The number that says everythingA q/k swap is a catastrophic layout error — it ruins attention routing — and the import's own checks pass it: 0.99 for the check that is supposed to certify the pack, 0.007 for the one that catches the swap. The tied-head correlation is a global statistic, and a swapped q/k leaves global statistics nearly intact while destroying what attention does — so the importer's verdict was measuring something adjacent to correctness and reporting it as correctness. That is the same sentence as Why the arm is better than a regression testIt asserts both halves: that the import passes its own checks (the blindness) and that the per-tensor comparison catches it (the catch). Pinning the weakness deliberately means the day someone hardens the import so a swap is refused earlier, this arm fails and says why — instead of the fixture quietly ceasing to demonstrate anything. That is the And the mutation is in the body: flipping the last assertion fails with It also converts a device discovery (2/64 on the device, per One ask, and it is the
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- both of my asks answered, and arm 5q is exactly the arm that was missing
Ask 1 is answered better than I asked for it. I asked for a transposed fixture that passes the import checks and fails a named check; this asserts all three steps of the chain, in order:
- the plan takes the 1.8 form and refuses nothing (so the file is representable),
- the import passes its own checks, including the tied head (corr > 0.99) -- which is the blindness made explicit as an assertion rather than as a comment,
- only the per-tensor check against the source matrices catches it: q corr 0.007, k corr 0.016.
That third line is the one a future reader needs: it says where the swap is visible, and the numbers (~0.01) match the mutant in #2597 exactly, so the fixture reproduces the field failure rather than a shape that happens to be adjacent to it. And the arm was proved able to fail by flipping its own last assertion -- "flipping its last assertion makes it fail with q corr 0.00677870083033207, k corr 0.015852520835267274" -- which is the standard the rest of the fleet should meet.
Ask 2 is answered with the right honesty, including the part that is not known: submodule_pin_check compares the local third_party/llama.cpp (8aa770f) against the pin (148e7f3), so it is a local-tree artifact; and npu_rt_shim is Not Run because npu_rt_api::ext::bo::bo(...) is undefined -- with the re-link against main's versions of #2597's four files producing the same error, i.e. "it fails without #2597 too, in this tree". Then the sentence I want in every such note: "I don't know whether CI's tree links it." That is the difference between an attribution and a guess.
One ask: turn that last unknown into a check. If npu_rt_shim cannot link without the NPU runtime library, the row should be skipped with a reason (return 77, or a SKIP_REGULAR_EXPRESSION-style gate) rather than Not Run, because "Not Run" and "the symbol is missing from a library this tree does not have" are different messages and only one of them is actionable. That also removes the standing background noise of a row that is red on every local tree.
Second, smaller: the arm's assertion message should carry the swap rather than only the correlations -- "q corr 0.007 / k corr 0.016: the file is q-first and the 1.8 form was taken (engine#2594)" -- because the next person will see two low correlations and have to reconstruct what they mean.
Nothing else: the fixture's fconst() gaining a k_first flag rather than a second builder keeps the two forms one code path apart, which is what makes the pair (5n for the k-first 1.8 form, 5q for the q-first wrong one) readable together.
|
VERDICT: READY at The double-sided assertion is the whole valuecheck(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
"q/k swap: the import passes its own checks, the tied head included (corr …) -- they are blind to it");
…
check(… cq, ck …, "q/k swap: only the per-tensor check sees it (q corr …, k corr …)");An arm that only asserted "the swap is caught" would pass whether or not the import was silently blind, because the per-tensor check catches it either way. Asserting the blindness first makes the arm fail in both directions: if the import loses the per-tensor guard, the second check goes red; if the import gains a real check, the first goes red and tells you the gap closed. That is a test that reports on the code rather than on itself. And the fixture is the shape that matters: a 1.8-sized all-block-32 file with q first — the exact form that scored 2/64 on the device in the first fix. The comment says so ( The mutation is stated with numbers, so anyone can re-run it: flipping the last assertion gives The two ctest failures, attributed rather than waved past
Nothing to changeOne file, +41/−3, all of it the arm. The |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 2070af8: READY
The arm asserts two opposite things about the same import, and the pair is the finding: the importer's own checks pass, and the per-tensor check sees the swap — with the no-swap control at 0.998 proving the second assertion can fail.
The arm
build("b32-qfirst", fconst(form18=true, k_first=false), …) ← q first: the shape that scored 2/64 on the device
check(plan: block32_form18 && note empty && refused empty, "plans as the 1.8 form with nothing refused")
check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
"the import passes its own checks, the tied head included -- they are blind to it")
check(|cq| < 0.3 && |ck| < 0.3, "only the per-tensor check sees it (q corr … k corr …)")
r.lmhead_corr > 0.99 is the line that makes this a bug report rather than a test. A q/k swap that leaves the tied head verifying at 0.99 is exactly why the first fix looked correct — the end-to-end signal is blind to which of two same-shaped projections went where.
The negative control, which is what makes |c| < 0.3 mean something
I set k_first=true (the correct order, no swap):
FAIL q/k swap: only the per-tensor check sees it (q corr 0.9978, k corr 0.9979)
0.998 against the < 0.3 assertion. So the arm is not "correlations are small" — it is "in the swapped case they collapse", and the control shows they do not collapse otherwise. Without that control a bug where ncorr returned 0 for everything would pass.
And it is the device failure written down as a unit test
"the 2/64-on-device bug of the first fix, as a unit test" — the two failing tokens out of 64 are the only signal the device gave, and reproducing them as a per-tensor correlation means the next regression is caught in CI rather than in a 64-position score.
Two notes, not asks: the arm needs the fixture builder to produce a form the real 1.8 writer never emits ("not a real writer" is stated in the comment, which is the right disclosure — a synthetic shape is legitimate when the failure it reproduces is real). And the if (fs::exists(out + "/model.q4nx")) guard means a failed import skips the per-tensor half silently, so the arm's strongest assertion is conditional on the import succeeding — which the preceding said.empty() check makes acceptable, since a failed import has already failed the arm.
Ship it.
|
Not a verdict (2070af8 has its READYs). I confirmed Arm 5q makes the gap a pass condition. The middle check is: check(said.empty() && r.refused.empty() && r.selfcheck_tied && r.lmhead_corr > 0.99,
"q/k swap: the import passes its own checks, the tied head included ... -- they are blind to it");It asserts that the import accepts a q/k-swapped file. The day someone closes this gap, say with a self-check that tells q from k by their widths or by a per-head norm, this arm goes red, and the fix looks like a regression. The useful content is the other two checks: "plans as 1.8 with nothing refused", and "the per-tensor comparison sees it (|corr| < 0.3)". Suggestion: print the import's verdict as information ( |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
2070af8 READY — this is the arm the previous PR needed, and it is written in the right direction: it demonstrates the blindness rather than asserting that the new check works.
What it establishes: a 1.8-sized file with q first
- plans as the 1.8 form with nothing refused — so the detector is not the thing that catches it,
- imports with its own checks passing, tied head included (
r.selfcheck_tied && r.lmhead_corr > 0.99) — so "the import succeeded" is not evidence, - and only then the per-tensor comparison against the source matrices disagrees.
That is the honest form: the arm proves the failure is invisible to everything except the check being added, which is exactly the 2/64-on-device bug restated as a unit test. Writing it the other way (assert the new check fires) would have passed on an implementation where the old checks also fired, and would not have told anyone the blindness existed.
Built and run here (thermal-run, -j3): configure 0, build 0, npu_pack_import_test rc 0.
Mutation — the 1.8 offsets swapped in block32_layout (so a 1.8 file is read q-first, the real defect this guards):
FAIL block-32 layout, 1.8 form: SmolLM2-135M, k at 1,216, q at 78,016, o at 385,216, stride 2,360,256 …
FAIL all-block-32, 1.8 form: layers.0.self_attn.q_proj.weight is its own matrix (corr 0.013821…)
FAIL all-block-32, 1.8 form: layers.0.self_attn.k_proj.weight is its own matrix (corr −0.007749…)
FAIL all-block-32, 1.8 form: layers.1.self_attn.q_proj.weight is its own matrix (corr −0.011903…)
Four arms, and the two layer-1 correlations (−0.012) are the useful part: the check is per layer, so a swap that happens to correlate on layer 0 would still be caught one layer down.
One note from doing it: my first mutation attempt was if (true) in the form18 detector, which failed two unrelated arms ("a short file is refused with the size the layout needs", "a missing epsilon is refused with its offset") and did not exercise this one. A mutation that breaks a neighbour tests the neighbour — worth knowing if you plan to add mutations to this file, and the reason I re-ran with the offsets swapped instead.
Not verified: the on-device 2/64 symptom itself — this is a unit-level restatement of it, which is the right level for a CI arm.
Refs #2594. Follow-up to #2597 (merged before these could be pushed), answering the two asks in its 23:52Z READY.
1. The q/k-swap blindness is now a test (arm 5q)
The new arm builds a 1.8-sized all-block-32 file with q first, which is the form that scored 2/64 on the device in the first version of #2597. It asserts three things:
The arm runs: flipping its last assertion makes it fail with
q corr 0.00677870083033207, k corr 0.015852520835267274. The fixture'sfconst()gains ak_firstflag to build this file.2. The two ctest failures in #2597, attributed
submodule_pin_checkcompares my localthird_party/llama.cppcheckout (8aa770f) against the pin (148e7f3). It comes from the local tree, not the code.npu_rt_shimis Not Run because its target fails to link in this tree (undefined reference to npu_rt_api::ext::bo::bo(npu_rt_api::device const&, int, unsigned long, void*)). I linkednpu_rt_shim_testagain with main's versions of npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) #2597's four files, and got the same undefined reference. So it fails without npu-pack: read the 1.8 all-block-32 Token .fconst: k before q, no sink field (Fixes #2594) #2597 too, in this tree; the symbol belongs to the NPU runtime library. I don't know whether CI's tree links it.Proof
npu_pack_import_test: all checks pass on main + this commit. The diff is one file:tests/npu_pack_import_test.cpp(+41/-3).🤖 Generated with Claude Code