Repository navigation
Harness: each DepGates self-test runs only the rule it plants for - #1386
Merged
Merged
Conversation
… batch two loops and share the real-tree run ci/check-ports.py lexed every file of its scopes and classified every scope's wholly-test files on each run (4.1 s of CPU of the ~25 s a ci/check-deps.sh run costs), once per DepGates test. It now reads a file's source first and tokenizes it only when it holds Instant, SystemTime or elapsed (the wall-clock regex and the alias test both need one), and computes a scope's wholly-test set only when a file of that scope has a wall-clock site. Verified byte-identical on the checkout and on trees with a planted production site, a wholly-test site, an Instant alias, an .elapsed ( site and an unlexable file. tests/test_harness.py: the two ui-owner loops plant their files in one copy and grade one gate run by file name (6 runs -> 2), and the two tests that grade the checkout share one run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s collation case "$flavor" in *[!a-z]* took "Debug" for lower case under macOS bash 3.2 with en_US.UTF-8 (the CI runner's locale), so --app=Debug exited 0 instead of 2 and the daily build bench's check row failed in ReadBack.test_bad_arguments on 2026-11-13. The class is spelled out; the test runs the argument under that locale too (red before the fix: 0 != 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper name ci/check-python-steps.txt: ORDER is the START order of a 4-lane runner, and test_check_machine_shape.py (225 s, one CPU-bound process) sat 110th, so it started ~190 s into the job and the job lasted 413 s. The nine slowest steps now start first; a list scheduler over the step durations of CI run 43537750750 gives 412 s for the old order (the job's own 413 s) and 317 s for the new one, the sum of all step times over four lanes. tests/test_harness.py: the helper that plants several files across crates in one gate run is _plant_all (a _plant_many for one crate already exists). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll rule's lex errors The wall rule's prefilter (a file without Instant/SystemTime/elapsed is not lexed) and its lazily computed wholly-test sets changed what an unlexable file does: - A wholly-test file holding a wall word that cannot be lexed used to be skipped (the wholly-test set was consulted before lexing) and failed after the prefilter. It is skipped again: only a file that fails to lex is classified, which costs nothing on a valid tree. Classifying every file with a wall word first was measured at about twice the gate's time. - A file with no wall word and an unterminated raw string, or a scope whose `wholly_test_files` hits an unbalanced token group and holds no wall site, no longer fails this rule. ci/check-list-caps.py lexes every production file under every rust-modules/*/src and rust-modules/src and computes wholly_test_files over each, and lets the ValueError fail its step; the comment at the prefilter says so. ci/test_check_ports.py pins each case, including that check-list-caps reports the files the wall rule no longer lexes. The ORDER header of ci/check-python-steps.txt now matches the file: the nine steps of the 2026-11-14 CI run first, then the thirteen others of the sixteen measured 2026-10-07. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ci/check-deps.sh` gains `--only RULE[,RULE...]` and `--list-rules`. Each of its 43 rules is now a block guarded by `want <name>`; with no argument every guard passes and the script is the one `make check` and CI run (same lines on stdout, same exit code, on a green and on a red tree). A name that is no rule, an empty list or another argument exits 2 before any rule runs. Three shared preludes (the owner-rule grep, the 17-run test-module classification and the ledger file prefilter) are computed only when a rule that reads them is selected; `is_wholly_test` exits 2 if asked before its prelude ran. RULES and the guards are cross-checked at the end of every run. The ~70 DepGates self-tests each ran the whole script (~25 s of CPU) to prove one planted violation is caught. Each now declares the rules it grades (`@_grades(...)`) and runs only those through one `_deps()`; the runner refuses an exit other than 0 or 1, an unbound-variable message on stderr, and a selected rule that printed no verdict line, and `_assert_flagged` refuses a rule the test did not select. The whole-script tests keep running every rule. New tests pin that the full run reports exactly the rules `--list-rules` names, that every rule alone prints what the full run prints for it, that a misspelt or empty selector is an error, and that a test selecting the wrong rule fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A second --only used to overwrite the first, so `--only wall --only libm` ran just libm and reported green. It now exits 2 through deps_usage like the other bad selectors, and the bad-selector test lists the three spellings. The which-tier skill said the harness shells out to check-deps.sh (11 s a run) per self-test; each DepGates self-test now runs only its own rule(s) through --only, and only the whole-script tests and the shared real-tree run run every rule. The timing figure is dropped. check-ports.py's note on who runs the wall rule now names the every-rule sweep, which runs --only wall too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GLinnik21
added a commit
that referenced
this pull request
Nov 14, 2026
cargo test builds every unit of the host unit suite (the 18 test binaries, the workspace libs under them and the third-party libs) with the test profile. That profile inherited the main checkout's full DWARF, which lanes and CI had already dropped. rust-modules/Cargo.toml now sets [profile.test] debug = "line-tables-only". Panic backtraces keep file:line; only debugger locals in host test builds are lost. CARGO_PROFILE_TEST_DEBUG=2 (or --config profile.test.debug=2) brings full DWARF back. The simulator's cargo build, the ARM library and clippy/check do not use this profile and are unchanged. In lanes and CI the unit graph is byte-identical, because their configs already set dev to line tables and test inherits it. Measured on an M4 in scratch trees with no lane config (build-bench, base/head alternated, medians of 3 per run): the non-incremental leaf-edit rebuild went from 43.3 s to 34.7 s for plx_base (-20%), from 23.2 s to 19.1 s for plx_ui dwell.rs (-18%) and from 23.0 s to 19.2 s for the plx_ui hub (-17%). The incremental loop shows no clear change. target plus target-fast shrink by 19%. The root manifest is pinned by two read-only certificates. With the owner's approval, both were re-certified: - ci/proofs/serde-json-immutable.json: the manifest hash, and the ui/src/lib.rs hash, which had been stale since 6a74d2a - ci/proofs/platform-generated-enums.json: the manifest hash The comments above check-python-harness (Makefile) and the harness job (ci.yml) now describe #1386's per-rule `ci/check-deps.sh --only` runs. Prose about the test profile's debug level in build-gc.sh, agent-reference.md, module-layers.md, cargo-seed.py and the host-rust action is corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
host harness tests(make check-python-harness,python3 tests/test_harness.py) spent 2647 of its 2678 summed test-seconds in theDepGatesclass: ~70 self-tests, each running ALL ofci/check-deps.sh(~25 s of CPU) to prove one planted violation is caught by one rule. This PR makes each self-test run only that rule.Measured back to back on the same Mac (
PLX_TEST_JOBS=4, load 8-12, shared), before =origin/main(includes #1384), after = this branch:Ran N tests in(module wall)DepGatessummed per-test secondsThe "after" numbers include the new tests below (the every-rule-alone sweep is ~85 s of CPU of them). Single-rule runs cost about 1 s for the owner rules and 5-7 s for the rules that need the test-module classification (3.7 s of
rust_test_modules.pyruns, now paid only by the 11 rules that read it); a full run is unchanged at ~20-25 s.How the selector works
ci/check-deps.sh --only RULE[,RULE...]runs just those rules;--list-rulesprints the 43 rule names (theRULESarray). Each rule is a blockif want <name>; then ... fi # <name>(one-line gates:want <name> && gate ...), with the body left at the margin so the diff and blame of the rules stay what they were: the script gains lines and rewrites only six one-linegatecalls. The selected run executes the same code as the full run, not a copy. The three shared preludes (the six owner rules' grep, the 17-run test-module classification, the ledger prefilter) are computed only when a rule that reads them is selected, at the position they always had;is_wholly_testexits 2 if asked before its prelude ran.It is an argument, not an environment variable, on purpose: an exported variable would silently shrink
make checkand CI's gate; an argument cannot be inherited.Why the default run is unchanged
No argument selects every rule. Verified by running
origin/main's script and this one from the same checkout: stdout and stderr are byte-identical and the exit code the same (0 on the green tree), and on a tree with plantedlibm,threads,workersandowned-damageviolations (4 gates fail in both). Every run, selected or not, ends by checking thatRULESand thewantguards name the same rules in the same order, which prints nothing when they agree.Tests that pin it, in
DepGates:test_every_rule_is_selectable_and_nothing_else_reports: the rule names the full run reports (ok — <name>lines) equal--list-rules, in order, apart fromallowlist-counts, the one rule that prints nothing when green (named in the test, so a second silent rule needs an edit there). A rule that reports but has no guard, or a guard whose name is missing, fails.test_every_rule_prints_the_same_alone_as_in_the_full_run: each of the 43 rules alone prints exactly the lines the full run printed for it, exit 0, empty stderr. This is the guard against a rule block leaning on something another rule's block computed: underset -uan unset variable inside$( ... )kills only that subshell and the rule reads green, so it shows only when the rule runs by itself.test_a_selection_runs_its_rules_in_the_scripts_order,test_a_misspelt_or_empty_selector_is_an_error_not_a_green_run(a misspelt or empty selector, a trailing comma, an unknown flag all exit 2 and run nothing).How a mis-selected test fails
Each self-test declares the rules it grades (
@_grades("threads", "libm")) and every helper (_plant,_plant_all,_plant_many,_mutate,_prepend,_prepend_files,_append_to,_deps) ends in one_deps(); a test with no declaration fails rather than running everything or nothing._check_deps_verdictrefuses a run that exits other than 0 or 1, one withunbound variableon stderr, and one in which a selected rule printed nook — <rule>/check-deps: <rule>:line (anchored:mutatorsis notmutators-visibility);_assert_flaggedrefuses a rule the test did not select.test_a_test_that_selects_the_wrong_rule_fails_instead_of_passingplants alibmviolation under aticksselection and requires the grading to fail;test_the_gate_runner_refuses_a_run_that_is_no_verdict_for_its_rulespins each refusal on fabricated runs.The two whole-script tests (
test_the_private_copy_grades_exactly_like_the_real_treeand the shared real-tree run) still run every rule.Worth a reviewer's eye: the "accepts / ignores / exempts" tests (green expectations) now mean "this rule passes", not "no rule objects"; the runner's verdict-line check is what stops them passing vacuously, and each was given the rules of its red twin.
Not changed
Makefile: the comment abovecheck-python-harness("ci/check-deps.sh(11 s a run) once or more per test ... 175 s") is stale after this and Harness: cheaper wall gate, fewer DepGates runs, slowest python gates first, locale-proof sched-trace flavour check #1384; left alone because touching the Makefile would need amake build-benchtable.ci/check-ports.py: its comment said the wall gate runs "~100 times by tests/test_harness.py"; reworded.ci/allow/ledger line was added.Gates
On the committed tree (
git status --shortempty before and after),origin/mainmerged in (#1384 and #1385 are on it):Not run:
make check/make check-nodef(no Rust or Makefile change),pr-reviewer,doc-claim-auditor.🤖 Generated with Claude Code