Skip to content

Harness: cheaper wall gate, fewer DepGates runs, slowest python gates first, locale-proof sched-trace flavour check - #1384

Merged
GLinnik21 merged 4 commits into
mainfrom
lane/harness-speed
Nov 14, 2026
Merged

GLinnik21 merged 4 commits into
mainfrom
lane/harness-speed

Conversation

@GLinnik21

@GLinnik21 GLinnik21 commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

What was slow, and why

Host harness tests (median 375 s, six runs cancelled at the 10 min timeout on 11-12/13). 2647 of the module's 2678 summed test-seconds are the DepGates class: each of its ~80 tests runs the whole of ci/check-deps.sh (about 25 s of CPU a run here) over a private copy of the tree. Nothing else in the module costs anything. The job's average rose from 305 s (11-12) to 427 s (11-13) the day #1358 landed ci/check-ports.py, the wall rule: it lexes every file of 12 scopes in pure Python and classifies every scope's wholly-test files, 4.1 s of CPU in every one of those runs.

Host python/shell/C gates (median 231 s, 413 s in run 43537750750). check-parallel.py starts steps in manifest order on 4 lanes. ci/test_check_machine_shape.py is one 225 s CPU-bound process and was the 110th entry, so it started ~190 s into the job and ended it at 413 s. A list scheduler over that run's 136 step durations reproduces the old order's makespan (412 s against the real 413 s) and gives 317 s for the new order, which is the sum of all step times over four lanes.

Daily build bench check row (2026-11-13). tools/tv-sched-trace.sh checked --app's flavour with *[!a-z]*. A range collates by locale: under macOS bash 3.2 with en_US.UTF-8 (the runner's) it takes "Debug" for lower case, so --app=Debug exited 0 instead of 2. It is a locale dependency, not a timing flake: LC_ALL=en_US.UTF-8 /bin/bash -c 'case Debug in *[!a-z]*) ...' reproduces it here.

Changes

  • ci/check-ports.py: a file is tokenized only when its source holds Instant, SystemTime or elapsed (the wall regex and the alias test both need one of them), and a scope's wholly-test set is computed only when a file of that scope has a wall site. Output and exit code are identical to the old script for every file that contains a wall word, on the checkout and on trees with a planted production site, a wholly-test site, an Instant as alias and an .elapsed ( site; that includes an unlexable file with a wall word, which still fails unless it is a wholly-test file (the old script never lexed those; the new one lexes first and consults the wholly-test set only when the lex fails, so a valid tree pays nothing). A file WITHOUT a wall word is no longer lexed by this rule, so an unlexable one (an unterminated raw string) no longer fails wall; ci/check-list-caps.py (a step of ci/check-python-steps.txt) lexes every production file under every rust-modules/*/src and rust-modules/src and fails closed on it, and wholly_test_files of each root likewise, while a scope's wholly-test set is computed here only when a file in it has a wall site. ci/test_check_ports.py pins all of it, including that check-list-caps reports what wall no longer lexes. ci/check-deps.sh is otherwise untouched.
  • tests/test_harness.py: the two ui-owner loops (3 variants each, 6 gate runs) plant their files in one copy and grade one run, requiring each variant by its file name in the gate's path:line:text report (6 runs -> 2); test_check_deps_is_green and test_the_private_copy_grades_exactly_like_the_real_tree share one run over the checkout behind a lock (nothing in the class edits the checkout). No test was skipped, deleted or weakened; the "allowed" controls keep their own runs.
  • ci/check-python-steps.txt: the nine slowest steps start first (a move, no step added or dropped; ci/test_ci_split.py passes).
  • tools/tv-sched-trace.sh + tools/test_tv_sched_trace.py: the flavour class is spelled out; test_bad_arguments also runs --app=Debug under en_US.UTF-8 (red before the fix with 0 != 2, green after).

Measured

The Mac was shared and oversubscribed throughout (load 8-70), so wall times are not comparable; CPU and run counts are.

before after
ci/check-deps.sh CPU per run, 3 alternating old/new runs back to back (macOS, BSD grep, Python 3.9) 31.9 s 28.1 s
check-ports.py alone, user CPU 4.12 s 1.58 s
gate runs in tests/test_harness.py 75 70
host python/shell/C gates, scheduler model on CI step times 412 s 317 s

Harness job estimate for the 4-vCPU runner, scaling the CI median by runs x CPU per run: 375 s x (70/75) x (28.1/31.9) is about 310 s. That is a real cut but not the ~240 s target; CI on this PR gives the actual number. What is still not done: the other ~25 s of each gate run is ~55 inline greps/awks of ci/check-deps.sh, so the next lever is letting a self-test run only the rule it plants for (an ONLY switch around the ~45 blocks), or caching the lexer output per file content.

Not touched

The comment above check-python-harness: in the Makefile still says "11 s a run" and "175 s"; both are stale. Editing the Makefile requires a make build-bench table, so it is left for whoever next touches that file.

Gates

On the committed tree (git status --short empty before and after):

$ PLX_TEST_JOBS=4 python3 tests/test_harness.py
Ran 736 tests in 687.203s
OK
$ python3 -W ignore tools/test_tv_sched_trace.py
Ran 11 tests in 34.495s
OK
$ python3 ci/test_ci_split.py
Ran 16 tests in 0.028s
OK

make check and make check-python-rest were not run locally; CI gates the PR.

🤖 Generated with Claude Code

GLinnik21 and others added 4 commits November 14, 2026 06:37
… batch two loops and share the real-tree run

ci/check-ports.py lexed every file of its scopes and classified every scope's wholly-test
files on each run (4.1 s of CPU of the ~25 s a ci/check-deps.sh run costs), once per DepGates
test. It now reads a file's source first and tokenizes it only when it holds Instant,
SystemTime or elapsed (the wall-clock regex and the alias test both need one), and computes a
scope's wholly-test set only when a file of that scope has a wall-clock site. Verified
byte-identical on the checkout and on trees with a planted production site, a wholly-test
site, an Instant alias, an .elapsed ( site and an unlexable file.

tests/test_harness.py: the two ui-owner loops plant their files in one copy and grade one
gate run by file name (6 runs -> 2), and the two tests that grade the checkout share one run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s collation

case "$flavor" in *[!a-z]* took "Debug" for lower case under macOS bash 3.2 with
en_US.UTF-8 (the CI runner's locale), so --app=Debug exited 0 instead of 2 and the daily
build bench's check row failed in ReadBack.test_bad_arguments on 2026-11-13. The class is
spelled out; the test runs the argument under that locale too (red before the fix: 0 != 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… helper name

ci/check-python-steps.txt: ORDER is the START order of a 4-lane runner, and
test_check_machine_shape.py (225 s, one CPU-bound process) sat 110th, so it started ~190 s
into the job and the job lasted 413 s. The nine slowest steps now start first; a list
scheduler over the step durations of CI run 43537750750 gives 412 s for the old order (the
job's own 413 s) and 317 s for the new one, the sum of all step times over four lanes.

tests/test_harness.py: the helper that plants several files across crates in one gate run is
_plant_all (a _plant_many for one crate already exists).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll rule's lex errors

The wall rule's prefilter (a file without Instant/SystemTime/elapsed is not lexed)
and its lazily computed wholly-test sets changed what an unlexable file does:

- A wholly-test file holding a wall word that cannot be lexed used to be skipped
  (the wholly-test set was consulted before lexing) and failed after the prefilter.
  It is skipped again: only a file that fails to lex is classified, which costs
  nothing on a valid tree. Classifying every file with a wall word first was
  measured at about twice the gate's time.
- A file with no wall word and an unterminated raw string, or a scope whose
  `wholly_test_files` hits an unbalanced token group and holds no wall site, no
  longer fails this rule. ci/check-list-caps.py lexes every production file under
  every rust-modules/*/src and rust-modules/src and computes wholly_test_files over
  each, and lets the ValueError fail its step; the comment at the prefilter says so.

ci/test_check_ports.py pins each case, including that check-list-caps reports the
files the wall rule no longer lexes. The ORDER header of ci/check-python-steps.txt
now matches the file: the nine steps of the 2026-11-14 CI run first, then the
thirteen others of the sixteen measured 2026-10-07.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GLinnik21
GLinnik21 merged commit a680294 into main Nov 14, 2026
15 checks passed
@GLinnik21
GLinnik21 deleted the lane/harness-speed branch November 14, 2026 04:47
GLinnik21 added a commit that referenced this pull request Nov 14, 2026
Brings in #1382 and #1384; nothing in them touches the hostsim suite or
the files this branch edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant