From 7023f956ce4d53533869b83e4bb95d12a866537f Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Thu, 5 Nov 2026 23:19:56 -0400 Subject: [PATCH] mtp: a new MTP leaderboard record cannot arrive outside the records guard #1206's guard compares the seven MTP records' published values against the fixtures, but it names those seven ids in a dict: a record added later -- a new target, a re-measured window, a k=6 row -- would land in bench/npu-leaderboard/ and be ignored by the check, looking exactly like a covered one. That is the "one table row per record" hole #1213 closed for the generated page, one field over, and the same defect this lane keeps paying for: a check that reports success on the records it was told about rather than the records that exist. The guard now also reads the directory: every file whose id starts 27b-mtp- must be one of the ids it derives, and the readable-directory count is asserted first so an empty listing cannot pass as coverage. Measured (pure python, no model, no device): before: the guard reported `all checks passed` with an extra record present on disk. after: an 8th record added as a copy of the hit-rate one -> `FAIL every MTP record on disk is checked by this guard: uncovered ['27b-mtp-extra-20261105']`, `1 CHECK(S) FAILED`; record removed -> `all checks passed`, `the guard's record directory is readable (7 MTP record(s))`. the existing arms are unchanged: 7 of 7 records equal what the fixtures derive (75.14 / 47.51 / 34.81 / 29.28 %, 46.69 %, 2.87 / 2.82), the +0.01 anti-vacuity arm still bites. tools/copyright.py --check rc 0. refs #805 (P6's leaderboard line); extends #1206. --- tests/mtp_depth_pool_test.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tests/mtp_depth_pool_test.py b/tests/mtp_depth_pool_test.py index 5e721a8f1..20eb958c4 100755 --- a/tests/mtp_depth_pool_test.py +++ b/tests/mtp_depth_pool_test.py @@ -25,7 +25,8 @@ 2. the echoed summary a harness prints after its exit line is ignored, so a cell is never counted twice; 3. a depth with no denominator on either side is reported as unmeasured, never as a +0.00 pp pass; 4. the numbers the MTP leaderboard records publish (all-chains rates, hit rate, both tokens-per-pass figures) are - re-derived from the same fixtures and compared to the record JSON, so a record cannot drift from its evidence. + re-derived from the same fixtures and compared to the record JSON, so a record cannot drift from its evidence -- + and every MTP record in the directory has to be one the guard names, so a new one cannot arrive uncovered. usage: tests/mtp_depth_pool_test.py (pure python, no model, no device) """ @@ -325,6 +326,15 @@ def record_mismatches(expect): wobbled[first] += 0.01 check(record_mismatches(wobbled) != [], f"the record comparison bites: {first} +0.01 is reported, not accepted") + # ... and the guard covers the DIRECTORY, not just the seven ids it happens to name: an MTP record added + # later that nobody adds here would look exactly like a covered one, which is how the "one table row per + # record" hole was closed for the page (#1213) and how a check ends up reporting success on the records it + # was told about rather than the records that exist. + on_disk = sorted(f[:-5] for f in os.listdir(RECORDS) + if f.startswith("27b-mtp-") and f.endswith(".json")) + check(len(on_disk) >= len(derived), f"the guard's record directory is readable ({len(on_disk)} MTP record(s))") + uncovered = [rid for rid in on_disk if rid not in derived] + check(not uncovered, f"every MTP record on disk is checked by this guard: uncovered {uncovered}") else: check(False, "tests/fixtures/mtp_clause55 is incomplete, so the records cannot be re-derived")