Repository navigation
Harness: cheaper wall gate, fewer DepGates runs, slowest python gates first, locale-proof sched-trace flavour check - #1384
Merged
Merged
Conversation
… batch two loops and share the real-tree run ci/check-ports.py lexed every file of its scopes and classified every scope's wholly-test files on each run (4.1 s of CPU of the ~25 s a ci/check-deps.sh run costs), once per DepGates test. It now reads a file's source first and tokenizes it only when it holds Instant, SystemTime or elapsed (the wall-clock regex and the alias test both need one), and computes a scope's wholly-test set only when a file of that scope has a wall-clock site. Verified byte-identical on the checkout and on trees with a planted production site, a wholly-test site, an Instant alias, an .elapsed ( site and an unlexable file. tests/test_harness.py: the two ui-owner loops plant their files in one copy and grade one gate run by file name (6 runs -> 2), and the two tests that grade the checkout share one run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s collation case "$flavor" in *[!a-z]* took "Debug" for lower case under macOS bash 3.2 with en_US.UTF-8 (the CI runner's locale), so --app=Debug exited 0 instead of 2 and the daily build bench's check row failed in ReadBack.test_bad_arguments on 2026-11-13. The class is spelled out; the test runs the argument under that locale too (red before the fix: 0 != 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper name ci/check-python-steps.txt: ORDER is the START order of a 4-lane runner, and test_check_machine_shape.py (225 s, one CPU-bound process) sat 110th, so it started ~190 s into the job and the job lasted 413 s. The nine slowest steps now start first; a list scheduler over the step durations of CI run 43537750750 gives 412 s for the old order (the job's own 413 s) and 317 s for the new one, the sum of all step times over four lanes. tests/test_harness.py: the helper that plants several files across crates in one gate run is _plant_all (a _plant_many for one crate already exists). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll rule's lex errors The wall rule's prefilter (a file without Instant/SystemTime/elapsed is not lexed) and its lazily computed wholly-test sets changed what an unlexable file does: - A wholly-test file holding a wall word that cannot be lexed used to be skipped (the wholly-test set was consulted before lexing) and failed after the prefilter. It is skipped again: only a file that fails to lex is classified, which costs nothing on a valid tree. Classifying every file with a wall word first was measured at about twice the gate's time. - A file with no wall word and an unterminated raw string, or a scope whose `wholly_test_files` hits an unbalanced token group and holds no wall site, no longer fails this rule. ci/check-list-caps.py lexes every production file under every rust-modules/*/src and rust-modules/src and computes wholly_test_files over each, and lets the ValueError fail its step; the comment at the prefilter says so. ci/test_check_ports.py pins each case, including that check-list-caps reports the files the wall rule no longer lexes. The ORDER header of ci/check-python-steps.txt now matches the file: the nine steps of the 2026-11-14 CI run first, then the thirteen others of the sixteen measured 2026-10-07. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was slow, and why
Host harness tests (median 375 s, six runs cancelled at the 10 min timeout on 11-12/13). 2647 of the module's 2678 summed test-seconds are the
DepGatesclass: each of its ~80 tests runs the whole ofci/check-deps.sh(about 25 s of CPU a run here) over a private copy of the tree. Nothing else in the module costs anything. The job's average rose from 305 s (11-12) to 427 s (11-13) the day #1358 landedci/check-ports.py, thewallrule: it lexes every file of 12 scopes in pure Python and classifies every scope's wholly-test files, 4.1 s of CPU in every one of those runs.Host python/shell/C gates (median 231 s, 413 s in run 43537750750).
check-parallel.pystarts steps in manifest order on 4 lanes.ci/test_check_machine_shape.pyis one 225 s CPU-bound process and was the 110th entry, so it started ~190 s into the job and ended it at 413 s. A list scheduler over that run's 136 step durations reproduces the old order's makespan (412 s against the real 413 s) and gives 317 s for the new order, which is the sum of all step times over four lanes.Daily build bench
checkrow (2026-11-13).tools/tv-sched-trace.shchecked--app's flavour with*[!a-z]*. A range collates by locale: under macOS bash 3.2 withen_US.UTF-8(the runner's) it takes "Debug" for lower case, so--app=Debugexited 0 instead of 2. It is a locale dependency, not a timing flake:LC_ALL=en_US.UTF-8 /bin/bash -c 'case Debug in *[!a-z]*) ...'reproduces it here.Changes
ci/check-ports.py: a file is tokenized only when its source holdsInstant,SystemTimeorelapsed(the wall regex and the alias test both need one of them), and a scope's wholly-test set is computed only when a file of that scope has a wall site. Output and exit code are identical to the old script for every file that contains a wall word, on the checkout and on trees with a planted production site, a wholly-test site, anInstant asalias and an.elapsed (site; that includes an unlexable file with a wall word, which still fails unless it is a wholly-test file (the old script never lexed those; the new one lexes first and consults the wholly-test set only when the lex fails, so a valid tree pays nothing). A file WITHOUT a wall word is no longer lexed by this rule, so an unlexable one (an unterminated raw string) no longer failswall;ci/check-list-caps.py(a step ofci/check-python-steps.txt) lexes every production file under everyrust-modules/*/srcandrust-modules/srcand fails closed on it, andwholly_test_filesof each root likewise, while a scope's wholly-test set is computed here only when a file in it has a wall site.ci/test_check_ports.pypins all of it, including thatcheck-list-capsreports whatwallno longer lexes.ci/check-deps.shis otherwise untouched.tests/test_harness.py: the twoui-ownerloops (3 variants each, 6 gate runs) plant their files in one copy and grade one run, requiring each variant by its file name in the gate'spath:line:textreport (6 runs -> 2);test_check_deps_is_greenandtest_the_private_copy_grades_exactly_like_the_real_treeshare one run over the checkout behind a lock (nothing in the class edits the checkout). No test was skipped, deleted or weakened; the "allowed" controls keep their own runs.ci/check-python-steps.txt: the nine slowest steps start first (a move, no step added or dropped;ci/test_ci_split.pypasses).tools/tv-sched-trace.sh+tools/test_tv_sched_trace.py: the flavour class is spelled out;test_bad_argumentsalso runs--app=Debugunderen_US.UTF-8(red before the fix with0 != 2, green after).Measured
The Mac was shared and oversubscribed throughout (load 8-70), so wall times are not comparable; CPU and run counts are.
ci/check-deps.shCPU per run, 3 alternating old/new runs back to back (macOS, BSD grep, Python 3.9)check-ports.pyalone, user CPUtests/test_harness.pyhost python/shell/C gates, scheduler model on CI step timesHarness job estimate for the 4-vCPU runner, scaling the CI median by runs x CPU per run: 375 s x (70/75) x (28.1/31.9) is about 310 s. That is a real cut but not the ~240 s target; CI on this PR gives the actual number. What is still not done: the other ~25 s of each gate run is ~55 inline greps/awks of
ci/check-deps.sh, so the next lever is letting a self-test run only the rule it plants for (anONLYswitch around the ~45 blocks), or caching the lexer output per file content.Not touched
The comment above
check-python-harness:in the Makefile still says "11 s a run" and "175 s"; both are stale. Editing the Makefile requires amake build-benchtable, so it is left for whoever next touches that file.Gates
On the committed tree (
git status --shortempty before and after):make checkandmake check-python-restwere not run locally; CI gates the PR.🤖 Generated with Claude Code