Skip to content

Harness: each DepGates self-test runs only the rule it plants for - #1386

Merged
GLinnik21 merged 8 commits into
mainfrom
lane/deps-only
Nov 14, 2026
Merged

GLinnik21 merged 8 commits into
mainfrom
lane/deps-only

Conversation

@GLinnik21

Copy link
Copy Markdown
Collaborator

What

host harness tests (make check-python-harness, python3 tests/test_harness.py) spent 2647 of its 2678 summed test-seconds in the DepGates class: ~70 self-tests, each running ALL of ci/check-deps.sh (~25 s of CPU) to prove one planted violation is caught by one rule. This PR makes each self-test run only that rule.

Measured back to back on the same Mac (PLX_TEST_JOBS=4, load 8-12, shared), before = origin/main (includes #1384), after = this branch:

before after
Ran N tests in (module wall) 736 tests in 550.6 s 742 tests in 165.9 s (154.8 s on the literal run)
children CPU, user + sys 1638.8 + 277.1 = 1915.9 s 370.6 + 59.3 = 429.9 s (-78%)
DepGates summed per-test seconds 2039.6 s (82 tests) 494.7 s (88 tests, -76%)

The "after" numbers include the new tests below (the every-rule-alone sweep is ~85 s of CPU of them). Single-rule runs cost about 1 s for the owner rules and 5-7 s for the rules that need the test-module classification (3.7 s of rust_test_modules.py runs, now paid only by the 11 rules that read it); a full run is unchanged at ~20-25 s.

How the selector works

ci/check-deps.sh --only RULE[,RULE...] runs just those rules; --list-rules prints the 43 rule names (the RULES array). Each rule is a block if want <name>; then ... fi # <name> (one-line gates: want <name> && gate ...), with the body left at the margin so the diff and blame of the rules stay what they were: the script gains lines and rewrites only six one-line gate calls. The selected run executes the same code as the full run, not a copy. The three shared preludes (the six owner rules' grep, the 17-run test-module classification, the ledger prefilter) are computed only when a rule that reads them is selected, at the position they always had; is_wholly_test exits 2 if asked before its prelude ran.

It is an argument, not an environment variable, on purpose: an exported variable would silently shrink make check and CI's gate; an argument cannot be inherited.

Why the default run is unchanged

No argument selects every rule. Verified by running origin/main's script and this one from the same checkout: stdout and stderr are byte-identical and the exit code the same (0 on the green tree), and on a tree with planted libm, threads, workers and owned-damage violations (4 gates fail in both). Every run, selected or not, ends by checking that RULES and the want guards name the same rules in the same order, which prints nothing when they agree.

Tests that pin it, in DepGates:

  • test_every_rule_is_selectable_and_nothing_else_reports: the rule names the full run reports (ok — <name> lines) equal --list-rules, in order, apart from allowlist-counts, the one rule that prints nothing when green (named in the test, so a second silent rule needs an edit there). A rule that reports but has no guard, or a guard whose name is missing, fails.
  • test_every_rule_prints_the_same_alone_as_in_the_full_run: each of the 43 rules alone prints exactly the lines the full run printed for it, exit 0, empty stderr. This is the guard against a rule block leaning on something another rule's block computed: under set -u an unset variable inside $( ... ) kills only that subshell and the rule reads green, so it shows only when the rule runs by itself.
  • test_a_selection_runs_its_rules_in_the_scripts_order, test_a_misspelt_or_empty_selector_is_an_error_not_a_green_run (a misspelt or empty selector, a trailing comma, an unknown flag all exit 2 and run nothing).

How a mis-selected test fails

Each self-test declares the rules it grades (@_grades("threads", "libm")) and every helper (_plant, _plant_all, _plant_many, _mutate, _prepend, _prepend_files, _append_to, _deps) ends in one _deps(); a test with no declaration fails rather than running everything or nothing. _check_deps_verdict refuses a run that exits other than 0 or 1, one with unbound variable on stderr, and one in which a selected rule printed no ok — <rule> / check-deps: <rule>: line (anchored: mutators is not mutators-visibility); _assert_flagged refuses a rule the test did not select. test_a_test_that_selects_the_wrong_rule_fails_instead_of_passing plants a libm violation under a ticks selection and requires the grading to fail; test_the_gate_runner_refuses_a_run_that_is_no_verdict_for_its_rules pins each refusal on fabricated runs.

The two whole-script tests (test_the_private_copy_grades_exactly_like_the_real_tree and the shared real-tree run) still run every rule.

Worth a reviewer's eye: the "accepts / ignores / exempts" tests (green expectations) now mean "this rule passes", not "no rule objects"; the runner's verdict-line check is what stops them passing vacuously, and each was given the rules of its red twin.

Not changed

Gates

On the committed tree (git status --short empty before and after), origin/main merged in (#1384 and #1385 are on it):

$ PLX_TEST_JOBS=4 python3 tests/test_harness.py
Ran 742 tests in 154.757s
OK

$ bash ci/check-deps.sh        # exit 0
check-deps: all gates green    # stdout and stderr identical to origin/main's script

$ python3 ci/test_check_ports.py
Ran 13 tests in 0.034s
OK

Not run: make check / make check-nodef (no Rust or Makefile change), pr-reviewer, doc-claim-auditor.

🤖 Generated with Claude Code

GLinnik21 and others added 8 commits November 14, 2026 06:37
… batch two loops and share the real-tree run

ci/check-ports.py lexed every file of its scopes and classified every scope's wholly-test
files on each run (4.1 s of CPU of the ~25 s a ci/check-deps.sh run costs), once per DepGates
test. It now reads a file's source first and tokenizes it only when it holds Instant,
SystemTime or elapsed (the wall-clock regex and the alias test both need one), and computes a
scope's wholly-test set only when a file of that scope has a wall-clock site. Verified
byte-identical on the checkout and on trees with a planted production site, a wholly-test
site, an Instant alias, an .elapsed ( site and an unlexable file.

tests/test_harness.py: the two ui-owner loops plant their files in one copy and grade one
gate run by file name (6 runs -> 2), and the two tests that grade the checkout share one run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s collation

case "$flavor" in *[!a-z]* took "Debug" for lower case under macOS bash 3.2 with
en_US.UTF-8 (the CI runner's locale), so --app=Debug exited 0 instead of 2 and the daily
build bench's check row failed in ReadBack.test_bad_arguments on 2026-11-13. The class is
spelled out; the test runs the argument under that locale too (red before the fix: 0 != 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper name

ci/check-python-steps.txt: ORDER is the START order of a 4-lane runner, and
test_check_machine_shape.py (225 s, one CPU-bound process) sat 110th, so it started ~190 s
into the job and the job lasted 413 s. The nine slowest steps now start first; a list
scheduler over the step durations of CI run 43537750750 gives 412 s for the old order (the
job's own 413 s) and 317 s for the new one, the sum of all step times over four lanes.

tests/test_harness.py: the helper that plants several files across crates in one gate run is
_plant_all (a _plant_many for one crate already exists).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll rule's lex errors

The wall rule's prefilter (a file without Instant/SystemTime/elapsed is not lexed)
and its lazily computed wholly-test sets changed what an unlexable file does:

- A wholly-test file holding a wall word that cannot be lexed used to be skipped
  (the wholly-test set was consulted before lexing) and failed after the prefilter.
  It is skipped again: only a file that fails to lex is classified, which costs
  nothing on a valid tree. Classifying every file with a wall word first was
  measured at about twice the gate's time.
- A file with no wall word and an unterminated raw string, or a scope whose
  `wholly_test_files` hits an unbalanced token group and holds no wall site, no
  longer fails this rule. ci/check-list-caps.py lexes every production file under
  every rust-modules/*/src and rust-modules/src and computes wholly_test_files over
  each, and lets the ValueError fail its step; the comment at the prefilter says so.

ci/test_check_ports.py pins each case, including that check-list-caps reports the
files the wall rule no longer lexes. The ORDER header of ci/check-python-steps.txt
now matches the file: the nine steps of the 2026-11-14 CI run first, then the
thirteen others of the sixteen measured 2026-10-07.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ci/check-deps.sh` gains `--only RULE[,RULE...]` and `--list-rules`. Each of
its 43 rules is now a block guarded by `want <name>`; with no argument every
guard passes and the script is the one `make check` and CI run (same lines on
stdout, same exit code, on a green and on a red tree). A name that is no rule,
an empty list or another argument exits 2 before any rule runs. Three shared
preludes (the owner-rule grep, the 17-run test-module classification and the
ledger file prefilter) are computed only when a rule that reads them is
selected; `is_wholly_test` exits 2 if asked before its prelude ran. RULES and
the guards are cross-checked at the end of every run.

The ~70 DepGates self-tests each ran the whole script (~25 s of CPU) to prove
one planted violation is caught. Each now declares the rules it grades
(`@_grades(...)`) and runs only those through one `_deps()`; the runner refuses
an exit other than 0 or 1, an unbound-variable message on stderr, and a
selected rule that printed no verdict line, and `_assert_flagged` refuses a
rule the test did not select. The whole-script tests keep running every rule.

New tests pin that the full run reports exactly the rules `--list-rules` names,
that every rule alone prints what the full run prints for it, that a
misspelt or empty selector is an error, and that a test selecting the wrong
rule fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second --only used to overwrite the first, so `--only wall --only libm`
ran just libm and reported green. It now exits 2 through deps_usage like the
other bad selectors, and the bad-selector test lists the three spellings.

The which-tier skill said the harness shells out to check-deps.sh (11 s a
run) per self-test; each DepGates self-test now runs only its own rule(s)
through --only, and only the whole-script tests and the shared real-tree run
run every rule. The timing figure is dropped. check-ports.py's note on who
runs the wall rule now names the every-rule sweep, which runs --only wall too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GLinnik21
GLinnik21 merged commit 2e1b78d into main Nov 14, 2026
15 checks passed
@GLinnik21
GLinnik21 deleted the lane/deps-only branch November 14, 2026 05:46
GLinnik21 added a commit that referenced this pull request Nov 14, 2026
cargo test builds every unit of the host unit suite (the 18 test binaries, the workspace libs under them and the third-party libs) with the test profile. That profile inherited the main checkout's full DWARF, which lanes and CI had already dropped. rust-modules/Cargo.toml now sets [profile.test] debug = "line-tables-only". Panic backtraces keep file:line; only debugger locals in host test builds are lost. CARGO_PROFILE_TEST_DEBUG=2 (or --config profile.test.debug=2) brings full DWARF back. The simulator's cargo build, the ARM library and clippy/check do not use this profile and are unchanged. In lanes and CI the unit graph is byte-identical, because their configs already set dev to line tables and test inherits it.

Measured on an M4 in scratch trees with no lane config (build-bench, base/head alternated, medians of 3 per run): the non-incremental leaf-edit rebuild went from 43.3 s to 34.7 s for plx_base (-20%), from 23.2 s to 19.1 s for plx_ui dwell.rs (-18%) and from 23.0 s to 19.2 s for the plx_ui hub (-17%). The incremental loop shows no clear change. target plus target-fast shrink by 19%.

The root manifest is pinned by two read-only certificates. With the owner's approval, both were re-certified:
- ci/proofs/serde-json-immutable.json: the manifest hash, and the ui/src/lib.rs hash, which had been stale since 6a74d2a
- ci/proofs/platform-generated-enums.json: the manifest hash

The comments above check-python-harness (Makefile) and the harness job (ci.yml) now describe #1386's per-rule `ci/check-deps.sh --only` runs. Prose about the test profile's debug level in build-gc.sh, agent-reference.md, module-layers.md, cargo-seed.py and the host-rust action is corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant