Repository navigation
Check: run the hostsim unit pass only over the crates that declare hostsim - #1381
Merged
Merged
Conversation
…stsim `check-cargo-unit-hostsim` ran all 18 test binaries a second time, but seven crates declare no `hostsim` feature and have no `feature = "hostsim"` in their source: plx_net, plx_ui, plx_appkit, plx_screens, plx_media_codecs, plx_play_uri and plx_playback_contract. In CI run 43528579834 the two passes printed the same test names for every one of them (and for 7 more that stay), and the hostsim-only tests are in plxnative-modules (+208), plx_media (+280) and plx_play_feed (+17), all kept. What `hostsim` changes below them (base paths steerable, gfx glyph-cache capacity, the simulator's env-driven render scale and locale overrides, GL-context-only code) is either unset or unreachable in a unit run, or makes the hostsim pass the laxer one. The three crates that declare the feature only to forward it (plx_machine, plx_data, plx_session) stay. The recipe's `-p` list becomes HOSTSIM_SUITE, a subset of UNIT_SUITE. ci/test_test_crate.py pins the subset, the recipe, and that every crate left out has no `hostsim` feature and no hostsim cfg, so a crate that gains one fails the gate until it is added. ci/test_ci_split.py's literal pin and the agent-reference paragraph follow. In the hostsim job's log those four large binaries took 78 s (plx_screens), 58.7 (plx_appkit), 33.4 (plx_ui) and 20.4 (plx_net) of the 474 s binary total, and their test binaries are no longer compiled under hostsim either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Makefile comment said the second run of the seven left-out crates repeated the default pass, and the guard docstring called them crates the feature cannot reach. Both were inexact: the old pass linked those crates' test binaries against hostsim-built plx_base, plx_gfx, plx_platform, plx_media and plx_play_feed (ENV_STEERABLE paths, glyph-cache sizes, ffi_host present). The crates' own test set is identical in the two passes and none of their tests observes those dependency-side differences, which the guard does not check; it checks that the crate declares no hostsim feature and has no feature = "hostsim" cfg in its src. Reword both. docs/agent-reference.md's testing section still said make check runs the whole host suite a second time under hostsim; it now says the crates that declare hostsim (HOSTSIM_SUITE) do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1380 (net transport test change); nothing in it touches the hostsim suite or the files this branch edits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was slow and why
make check-cargo-unit-hostsim(thehost unit tests (hostsim)CI job, median 395 s) re-ran all 18 test binaries under--features hostsim. Seven crates do not declare the feature and have nofeature = "hostsim"in their source:plx_net,plx_ui,plx_appkit,plx_screens,plx_media_codecs,plx_play_uri,plx_playback_contract.Evidence (CI run 43528579834, both unit jobs' logs): for those seven, the default and hostsim passes print the identical set of test names and counts (also for
plx_base,plx_gfx,plx_platform,plx_plex,plx_session,plx_machine,plx_data, which stay). The names that differ areplxnative-modules(+208 hostsim-only, 6 default-onlycursor_door_tests),plx_media(+280) andplx_play_feed(+17), all kept. In the hostsim job those seven binaries took 78.0 (screens), 58.7 (appkit), 33.4 (ui), 20.4 (net), 13.6, 2.1 and 0.0 s of the 474 s binary total (123.9 s wall), and their test targets were also compiled under hostsim.What hostsim changes below those crates, each checked in source:
base::pathsbecomes steerable (ENV_STEERABLE; the default pass is the stricter one, bare/tmp);base::surface,gfxandplatform::i18nhostsim code keys onPLXNATIVE_RENDER_SCALE,PLXNATIVE_LOCALE,PLXNATIVE_FORMAT_LOCALE,PLXNATIVE_CLOCK(unset in the suite), or needs a GL context (bind_core_profile_vao,under_blend); gfx glyph-cache capacity is larger (320/480 slots against 80/160);plexandtelemetryhostsim only gatetarget_arch = "arm"code, a no-op on a host;platform's remaining sites areany(hostsim, test, test-support), true in both passes. So the old second run of the seven also linked their test binaries against hostsim-builtplx_base,plx_gfx,plx_platform,plx_mediaandplx_play_feed(the steerable paths and the larger glyph caches above, andffi_hostpresent inplx_play_feed). The seven crates' own test set is identical in both passes and none of their tests observes those dependency-side differences (per the review of this PR); the guard below does not check that.The change
HOSTSIM_SUITE(the 11 crates that declarehostsim, inUNIT_SUITE's order) becomes the hostsim recipe's-plist.ci/test_test_crate.pygainstest_the_hostsim_pass_is_exactly_the_crates_that_declare_hostsim: HOSTSIM_SUITE is an ordered subset of UNIT_SUITE, the recipe names exactly it, and every crate left out has nohostsimfeature and no hostsim cfg. A crate that gains one fails the gate until it is added.ci/test_ci_split.py's literal pin and thedocs/agent-reference.mdbuild-section paragraph follow. TheMakefilecomment, the guard's docstring and the Tier 1 testing paragraph ofdocs/agent-reference.mdsay what is dropped and why: the crate's own test set is identical, and the dependency-side hostsim differences are not observed by its tests and not checked by the guard (which checks that the crate declares nohostsimfeature and has nofeature = "hostsim"cfg in its src).origin/mainwas merged into the branch three times while the PR was open (Net: bound a never-pausing peer's cancel latency by what the transport controls (fixes the bench's failing unit suite) #1380; Media: the window-reread test times its wait from before the stop's sleep (flaky on loaded runners) #1382 and Harness: cheaper wall gate, fewer DepGates runs, slowest python gates first, locale-proof sched-trace flavour check #1384; CI: build the host crates with line tables only #1383 and Tests: the 8 slowest unit tests do the same checks for a fraction of the CPU #1385); none of those touch the hostsim suite or the files this PR edits.Still proves
The same as before for every crate whose own tests can differ under hostsim; the four big binaries (
plx_screens,plx_appkit,plx_ui,plx_net) now run once, in the default pass, with the same test names as before. Not covered any more: the seven crates' test binaries against the hostsim-built dependencies listed above. A future test in one of them that comes to depend on a dependency-side hostsim difference (without a hostsim cfg in the crate itself, which the guard would catch) is run by the default pass only, against the default-built dependencies, until the crate joinsHOSTSIM_SUITE.Gates
314f423db,git status --shortempty before and after:python3 ci/test_test_crate.py:Ran 13 tests in 1.948s/OK;python3 ci/test_ci_split.py:Ran 16 tests in 0.018s/OK.bd5e77ae2(run 43553308180) andc6403a750(run 43558881851): every job green,host unit tests (hostsim)included. The run for314f423dbis the next one.2561618bd),make test-hostsim C="plxnative-modules plx_base plx_machine plx_platform plx_gfx plx_plex plx_telemetry plx_data plx_session plx_media plx_play_feed"(all 11 selected packages accept--features hostsim; per-binary lines in the order printed):The hostsim-only counts match the CI log (
plxnative-modules1587,plx_media2414,plx_play_feed323). This ismake test-hostsim's incremental tree on a 10-core Mac under other lanes' load, not the CI recipe, so the wall is only indicative (the same 18 binaries summed 474 s in CI's hostsim job).CI: the
host unit tests (hostsim)jobJob seconds are
started_attocompleted_atof the job; the wall line is the job's owncargo-test-parallel:line. The baseline is the 2026-11-06..14 median 395 s, p90 472 s (18 binaries); run 43528579834, on the old recipe, summed 474 s over 18 binaries at 123.9 s wall.cargo-test-parallel2561618bdbd5e77ae2c6403a750314f423db(head; includes #1383 and #1385, so not comparable)Three comparable samples (a fourth, not comparable, below), all below the baseline median, but two of them (140.4 s and 135.1 s) with a longer test wall than the old 18-binary run, so this is a smaller and noisier saving than the binary sums suggest: the wall is set by
plxnative-modules(140.3 s and 135.0 s in the first and third samples) and the compile step, and the runners vary a lot on identical code (the unchangedhost unit tests (default features)job ran its 18 binaries in 184.5, 119.6 and 115.9 s wall on the same three heads). Not a percentage claim.#1383(line tables only for the host crates) and#1385(the slowest unit tests) reached the branch after the third sample, so the fourth row is not comparable with the first three.Build benchmark (
make build-bench, AGENTS.md: a PR that touchesMakefile)MakefilegainsHOSTSIM_SUITEand a shorter-plist in one recipe; nothing the benchmark builds changes, and its unit-suite row runs the DEFAULT suite (the default-featurecargo test --libover the 18 packages,tools/build-bench.py'stestsscenario, throughUNIT_RUNfromprint-bench-config; the row saysdefault features), so it cannot show the hostsim saving: that shows in the CI job above. The tables are a no-regression check.make build-bench ARGS='--json ...', defaults (3 runs per scenario, interleaved,rustc front end: serial), pinned rustc 1.100.0, Apple M4 10 cores, with$HOME/.cargo/binand Homebrew onPATH; the benchmark holds themake checklock exclusively, so no othermake checkshared the box; other work not under that lock may have. Same machine, same sitting (2026-11-14, 06:46 to 08:18 +03). Base is a separate detached worktree at5ca2777b5(origin/mainwhen measured); head runs are in the PR worktree. Base and head run 1 each started with no target dir (the benchmark's untimed warm-up build); head run 2 reused head run 1's. The incremental rows (notarget-fast) and the ARM staticlib row (no archive) are skipped on every side; nothing was narrowed with--only.git diff --stat 5ca2777b5 bd5e77ae2 -- rust-modules tools build_support Cargo.toml Cargo.lock .cargo .github build.rsis empty; the two commits differ only inMakefile,ci/test_ci_split.py,ci/test_test_crate.pyanddocs/agent-reference.md. Head run 2 (c6403a750, run after merging Media: the window-reread test times its wait from before the stop's sleep (flaky on loaded runners) #1382 and Harness: cheaper wall gate, fewer DepGates runs, slowest python gates first, locale-proof sched-trace flavour check #1384) adds main's one media test file,tools/tv-sched-trace.shand the python gates. The head314f423dbadds CI: build the host crates with line tables only #1383 and Tests: the 8 slowest unit tests do the same checks for a fraction of the CPU #1385 on top, which were not benchmarked.WARNING:lines (10 cores): base max 27.9 (22 warnings, median 17.2); head run 1 max 91.4 (45 warnings, median 38.6); head run 2 max 15.6 (23 warnings, median 11.5). The source of the load was not determined; run 1 ran under the heaviest of it, and almost every row of it is 1.3-2.7x slower than base on identical code. Run 2, on a calmer box, has a min-max range that overlaps base's in every row but one (app coldstart.rs, where run 2 is the faster), and its unit-suite row is 59.7 s against 57.8 s. Per AGENTS.md's reading rule a delta inside the spread is not a result: there is no measured regression, and none is expected from a change that touches no compiled input.5ca2777b5bd5e77ae2c6403a750Each cell is median (min - max) over 3 runs. The benchmark's own output, verbatim:
base `5ca2777b5`
build-bench
5ca2777b50af· rustc 1.100.0 (0ef1bc386 2026-11-09) · Apple M4, 10 cores, Darwin 27.0.0 · 3 run(s) per scenario, interleaved · rustc front end: serialmake test-fastonce or pass --cold to build it first)make test-fastonce or pass --cold to build it first)make(the benchmark never triggers the FFmpeg build)WARNING: noop run 1: load 10.62 > 10 cores
WARNING: leaf run 1: load 13.74 > 10 cores
WARNING: machine run 1: load 17.19 > 10 cores
WARNING: platform run 1: load 15.60 > 10 cores
WARNING: gfx run 1: load 12.13 > 10 cores
WARNING: net run 1: load 10.66 > 10 cores
WARNING: session run 2: load 10.35 > 10 cores
WARNING: machine run 3: load 12.38 > 10 cores
WARNING: platform run 3: load 17.40 > 10 cores
WARNING: gfx run 3: load 20.91 > 10 cores
WARNING: net run 3: load 18.50 > 10 cores
WARNING: ui run 3: load 18.19 > 10 cores
WARNING: plex run 3: load 15.00 > 10 cores
WARNING: telemetry run 3: load 15.33 > 10 cores
WARNING: data run 3: load 16.61 > 10 cores
WARNING: session run 3: load 22.30 > 10 cores
WARNING: media run 3: load 24.14 > 10 cores
WARNING: appkit run 3: load 27.89 > 10 cores
WARNING: screens run 3: load 24.47 > 10 cores
WARNING: app run 3: load 20.81 > 10 cores
WARNING: hub run 3: load 18.66 > 10 cores
WARNING: tests run 3: load 15.99 > 10 cores
WARNING: the host was loaded (load above core count or swap over 90% full): these numbers are noisy; compare medians of repeated runs only.
head run 1 `bd5e77ae2`
build-bench
bd5e77ae2ff6· rustc 1.100.0 (0ef1bc386 2026-11-09) · Apple M4, 10 cores, Darwin 27.0.0 · 3 run(s) per scenario, interleaved · rustc front end: serialmake test-fastonce or pass --cold to build it first)make test-fastonce or pass --cold to build it first)make(the benchmark never triggers the FFmpeg build)WARNING: leaf run 1: load 12.13 > 10 cores
WARNING: machine run 1: load 14.92 > 10 cores
WARNING: platform run 1: load 18.18 > 10 cores
WARNING: gfx run 1: load 31.78 > 10 cores
WARNING: net run 1: load 31.57 > 10 cores
WARNING: ui run 1: load 43.58 > 10 cores
WARNING: plex run 1: load 31.77 > 10 cores
WARNING: telemetry run 1: load 33.13 > 10 cores
WARNING: data run 1: load 41.39 > 10 cores
WARNING: session run 1: load 61.97 > 10 cores
WARNING: media run 1: load 76.86 > 10 cores
WARNING: appkit run 1: load 74.61 > 10 cores
WARNING: screens run 1: load 63.69 > 10 cores
WARNING: app run 1: load 52.23 > 10 cores
WARNING: hub run 1: load 48.35 > 10 cores
WARNING: tests run 1: load 36.59 > 10 cores
WARNING: noop run 2: load 36.23 > 10 cores
WARNING: leaf run 2: load 36.23 > 10 cores
WARNING: machine run 2: load 63.79 > 10 cores
WARNING: platform run 2: load 70.87 > 10 cores
WARNING: gfx run 2: load 91.42 > 10 cores
WARNING: net run 2: load 79.69 > 10 cores
WARNING: ui run 2: load 64.27 > 10 cores
WARNING: plex run 2: load 47.31 > 10 cores
WARNING: telemetry run 2: load 48.92 > 10 cores
WARNING: data run 2: load 38.58 > 10 cores
WARNING: session run 2: load 31.08 > 10 cores
WARNING: media run 2: load 37.22 > 10 cores
WARNING: appkit run 2: load 37.81 > 10 cores
WARNING: screens run 2: load 57.69 > 10 cores
WARNING: app run 2: load 51.67 > 10 cores
WARNING: hub run 2: load 47.42 > 10 cores
WARNING: tests run 2: load 48.93 > 10 cores
WARNING: noop run 3: load 60.17 > 10 cores
WARNING: leaf run 3: load 60.17 > 10 cores
WARNING: machine run 3: load 45.02 > 10 cores
WARNING: platform run 3: load 34.00 > 10 cores
WARNING: gfx run 3: load 27.96 > 10 cores
WARNING: net run 3: load 23.85 > 10 cores
WARNING: ui run 3: load 19.03 > 10 cores
WARNING: plex run 3: load 15.53 > 10 cores
WARNING: telemetry run 3: load 13.79 > 10 cores
WARNING: data run 3: load 13.81 > 10 cores
WARNING: session run 3: load 11.85 > 10 cores
WARNING: media run 3: load 10.48 > 10 cores
WARNING: the host was loaded (load above core count or swap over 90% full): these numbers are noisy; compare medians of repeated runs only.
head run 2 `c6403a750`
build-bench
c6403a7505c5· rustc 1.100.0 (0ef1bc386 2026-11-09) · Apple M4, 10 cores, Darwin 27.0.0 · 3 run(s) per scenario, interleaved · rustc front end: serialmake test-fastonce or pass --cold to build it first)make test-fastonce or pass --cold to build it first)make(the benchmark never triggers the FFmpeg build)WARNING: machine run 1: load 10.08 > 10 cores
WARNING: platform run 1: load 15.61 > 10 cores
WARNING: gfx run 1: load 13.92 > 10 cores
WARNING: net run 1: load 10.97 > 10 cores
WARNING: ui run 1: load 10.13 > 10 cores
WARNING: noop run 2: load 10.92 > 10 cores
WARNING: leaf run 2: load 10.92 > 10 cores
WARNING: machine run 2: load 12.88 > 10 cores
WARNING: platform run 2: load 12.80 > 10 cores
WARNING: gfx run 2: load 12.89 > 10 cores
WARNING: net run 2: load 11.74 > 10 cores
WARNING: ui run 2: load 12.09 > 10 cores
WARNING: plex run 2: load 10.99 > 10 cores
WARNING: telemetry run 2: load 10.51 > 10 cores
WARNING: data run 2: load 10.42 > 10 cores
WARNING: session run 2: load 10.26 > 10 cores
WARNING: machine run 3: load 12.38 > 10 cores
WARNING: platform run 3: load 14.39 > 10 cores
WARNING: gfx run 3: load 11.69 > 10 cores
WARNING: net run 3: load 10.74 > 10 cores
WARNING: ui run 3: load 11.24 > 10 cores
WARNING: plex run 3: load 12.66 > 10 cores
WARNING: telemetry run 3: load 11.51 > 10 cores
WARNING: the host was loaded (load above core count or swap over 90% full): these numbers are noisy; compare medians of repeated runs only.
🤖 Generated with Claude Code