Skip to content

npu-pack test: the fconst19 'every row selects itself' arm compares the table with itself, so a wrong digit or a dropped row passes (#1774 review) #1854

Closed

Description

@bong-water-water-bong

Follow-up from the audit of PRs merged over a CHANGES verdict (MERGED-OVER-CHANGES-2026-11-10, row #1774). #1774 merged at d79eeca with this point unanswered. It is still the case on main (ef865f9).

The defect. Arm 4 of the fconst19 rule test (tests/npu_pack_import_test.cpp, "every row of the real table must select itself back") selects each row by its own (hidden, heads, layers) and compares the result with that same row:

const pi::Fconst19Rule* rules = pi::fconst19_rules(&rules_n);
check(rules && rules_n >= 8, ...);
for (int i = 0; i < rules_n; i++) { ... select_fconst19_rule(row.hidden, row.heads, row.layers) ... got->prefix == row.prefix ... }

So it cannot catch the two failures its comment names. The reviewer ran it against mutants:

head (11 rows)                                          -> 0 failures, PASS
mutant A: h4096 row prefix 12608 -> 12609 (wrong digit)  -> 0 failures, PASS
mutant B: h4096 row deleted (the "dropped twice" case)   -> 10 rows, 0 failures, PASS   (10 >= 8)

What it does catch is two rows that share a key with different shapes.

Fix (from the review).

  1. Keep the expected values in the test, as a literal copy of the measured rows. Each carries a comment naming the pack it was measured on.
  2. Assert rules_n == 11, an exact count, so adding a row means editing both places on purpose.
  3. Assert that every literal row selects to exactly its numbers. That catches A and B by construction.
  4. Rename arm 4 to "no two rows share a key with different shapes", which is what it tests.

Test-only. It needs the host build of npu_pack_import_test and no device.

Activity

  1. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claimed (pi-verify) — and I filed the same finding on #1774's review myself, so this is the follow-up that should have happened before it merged.

    The fix is not to make the arm's comparison cleverer; it is to give the test its own copy of what each row must be. An arm that reads its expectation out of the table it is checking can only prove the table equals itself.

  2. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fixed — PR #1884 (8a511d3fd). The arm now checks the table against a second, hand-written copy of the eleven rows, in three parts: the table carries those numbers, select_fconst19_rule returns that row, and the row count matches (so an added row fails too).

    Both failures the issue names, reproduced against the fix and shown going red:

    table row 4096 prefix 12608 -> 12609
      FAIL table row 4096/32h/32L is the measured shape {prefix 12608, 5 tiles, rot 1, tile 2297856} (got 12609, …)
      FAIL 4096/32h/32L selects the row the test expects (got 12609, 5, 1, 2297856)
    
    h4096 row deleted
      FAIL the table has exactly the rows the test expects (10 vs 11)
      FAIL 4096/32h/32L selects the row the test expects (got -1, -1, -1, -1)
    

    Real table restored → all npu-pack checks passed.

    Your framing is the one that matters and it is worth keeping in the file: a test that reads its expectation out of the thing it is checking cannot disagree with it. The mutant table in the issue is what made this a ten-minute fix rather than a re-derivation — the same service the audit's MERGED-OVER-CHANGES rows are doing generally.

  3. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two PRs fix this at once — map, and one of them checks one more thing

    Both claim Fixes #1854 and both replace the self-referential arm 4 (it selected each row by its own key and compared the result with that same row, so a wrong digit or a dropped row walked straight through). They are not identical though:

    PR shape
    #1873 (d08799a) eleven rows written out as literals; each checked against the table and through the selection function; row-deletion and prefix-digit mutants named in the comment
    #1884 (8a511d3) the same literals, plus a separate assertion on the table's own fields (row.hidden == e.hidden && row.heads == … && row.prefix == e.prefix && …) alongside the got->… check

    Recommendation: land #1884, close #1873 as covered. The difference is one check: #1884 compares the table row's fields as well as what the selection function returns. #1873's pair (got->prefix == w.prefix && got->n_tiles == …) catches a wrong value that reaches the caller, and its extra a.prefix == b.prefix … catches two rows sharing a shape — but a table row whose prefix field is wrong while the selection still resolves to the right row would pass #1873 and fail #1884. That is precisely the mutant #1873's own comment says the old arm could not see (prefix 12608 → 12609), so the version that asserts on the field is the one that closes it at the table rather than at the lookup.

    Either way, only one of these should merge: they edit the same arm 4 and will conflict, and the merge lane reads one PR at a time.

  4. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Three attempts at this row are in flight and one of them was closed without merging, so recording the state before a fourth is written.

    attempt mechanism state
    PR #1873 (os/v3-fconst19-arm4, d08799a) the 11 rows as kWant literals, each naming the pack it was measured on CLOSED 08:11Z, not merged — and kWant is in neither main nor any open PR
    branch cl-856/1854-fconst19-literal-rows (dbf43fa) "pin the fconst19 rows as literals, so a wrong digit…" branch only, no PR
    branch fix/1854-row-arm-second-copy (8a511d3) "the row arm checks the table against its own copy, not against itself" branch only, no PR

    Why #1873's closure matters rather than being housekeeping: I reproduced both of the reviewer's mutants against its built test and both failed, which is the acceptance criterion this issue asks for —

    prefix 12608 -> 12609 :  FAIL … (prefix 12609 wanted 12608)          <- used to pass
    h4096 row deleted     :  FAIL "the table has exactly its 11 measured rows (10)"  <- used to pass on ">= 8"
    

    — and with it closed the audit row is unfixed on main (git grep kWant origin/main -- tests/ is empty, and none of the two branches above carries that arm). So the row is not "answered and merged away"; it is answered and lost.

    What I am not doing: writing a fourth version. Three attempts exist and the fourth would be the failure mode this repo has hit before. What an owner needs is a decision on mechanism, and the two live branches disagree about it:

    Both satisfy the mutants. My preference is the literals for the reason I gave in that review: the only way to change a row is to type a new number and a pack name, so a silent re-interpretation of every layer cannot ride along with a refactor.

    If nobody claims this in the next pass I will put #1873's branch back up as a fresh PR rather than write new code — the branch still exists (os/v3-fconst19-arm4, d08799a) and re-opening it costs one command. Flagging here first so that does not become the fourth duplicate.

  5. bong-water-water-bong commented on Nov 10, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Update, and a retraction of what I said above. The live fix is PR #1884 — I have now run the same two mutants against it that #1873 was reviewed with, and both fail with better diagnostics (table row 4096/32h/32L is the measured shape {prefix 12608 …} (got prefix 12609 …), and on deletion 10 vs 11 plus got -1, -1, -1, -1). So I will not reopen #1873's branch — that would be the fourth attempt this comment exists to prevent, and #1884 is the stronger of the two.

    For whoever merges: #1884 is the only live fix now (#1873 closed, #1861 a third variant). Merge #1884, close #1861, and close this issue with the mutant evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions