diff --git a/.github/actions/host-rust/action.yml b/.github/actions/host-rust/action.yml index 02caeadb0f..abd07c37a3 100644 --- a/.github/actions/host-rust/action.yml +++ b/.github/actions/host-rust/action.yml @@ -44,17 +44,21 @@ runs: - name: Build the host crates with line tables only # Nothing in CI attaches a debugger to a host test binary, and a panic backtrace needs only the - # line table. The default `dev`/`test` profile writes full DWARF for every crate, dependencies - # included, which is most of what rustc's codegen, the linker and the cache archive carry for - # the 18 test binaries. This is the per-user cargo config, so it reaches the bare `cargo` - # lines and every `make` target of the host jobs alike. A dependency built with the old + # line table. A `dev` build writes full DWARF for every crate, dependencies included, which is + # most of what rustc's codegen, the linker and the cache archive carry (the 18 test binaries + # too, before rust-modules/Cargo.toml gave `[profile.test]` line tables of its own: `cargo test` + # builds with `test`, which inherits `dev`). This is the per-user cargo config, so it reaches + # the bare `cargo` lines and every `make` target of the host jobs alike, and it trims what the + # manifest does not: the `dev` builds (clippy, `cargo check`) and the third-party crates' + # debug info (`debug = false`). A dependency built with the old # profile is a different unit to cargo, and rust-cache's key does not see this file (it hashes # the manifests, the lockfile, the toolchain files and the workspace's `.cargo/config.toml`, # not the home one), so the cache step below carries its own `prefix-key`: without it the # full-DWARF cache restores as an exact hit, every dependency is rebuilt, and the post step - # finds the cache up to date and saves nothing. It is NOT the manifest: a developer's checkout - # keeps full DWARF (docs/agent-reference.md, the lane policy), and the profiles the ARM binary - # is built with do not inherit `dev`, so the shipped library is untouched. + # finds the cache up to date and saves nothing. The `dev` half is NOT in the manifest: a developer's checkout + # keeps full DWARF in `dev` builds (docs/agent-reference.md, the lane policy; its host tests + # get line tables from the manifest's `[profile.test]`), and the profiles the ARM binary is + # built with do not inherit `dev`, so the shipped library is untouched. shell: bash run: | mkdir -p "$HOME/.cargo" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4d3340e629..3389a59845 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -446,9 +446,11 @@ jobs: - name: make check-python-rest run: make check-python-rest - # tests/test_harness.py alone: about 190 s of what used to be a 360 s `check-python` job. Most of - # it is its `DepGates` class running ci/check-deps.sh over private copies of the tree on every - # core of the runner, so it gains nothing from sharing a runner with the other gates. + # tests/test_harness.py alone, split off a 360 s `check-python` job. Most of it is its `DepGates` + # class: about 70 self-tests that each run only the rules they plant a violation for + # (`ci/check-deps.sh --only`, #1386) over private copies of the tree, spread over every core of + # the runner, so it gains nothing from sharing a runner with the other gates. The step took + # 144-196 s on main's commits of 2026-11-14 after #1386 (380-500 s on the ones before it). # `make check-python` is still the two halves, run side by side locally. host-python-harness: name: host harness tests diff --git a/Makefile b/Makefile index bb7d53d69a..1a89831757 100644 --- a/Makefile +++ b/Makefile @@ -1957,15 +1957,15 @@ check-python: harness='$(MAKE) --no-print-directory check-python-harness' check-python-harness: - @# The harness's own host unit tests (tests/test_harness.py, stdlib unittest). THE MOST - @# EXPENSIVE STEP IN `check` BY FAR — 661 s of a 1100 s run, measured 2026-10-01 on a cold lane - @# at load ~5, and 630 s of that is the `DepGates` class running the whole of - @# `ci/check-deps.sh` (11 s a run) once or more per test to prove each structure gate still - @# catches a planted violation. Those self-tests now edit a private copy of the gate's inputs - @# and run on a thread pool (`PLX_TEST_JOBS`, default min(8, cpus)): the module takes 175 s - @# instead of 661 s. (It was 980 s of 1217 s before `check-deps.sh` stopped forking a process - @# per candidate line.) Measure before budgeting, and if this number needs to come down - @# further, `check-deps.sh` itself is the place. run.py + @# The harness's own host unit tests (tests/test_harness.py, stdlib unittest): the harness + @# branch of `check-python` and the whole of CI's `host-python-harness` job. Most of it is the + @# `DepGates` class: about 70 self-tests, each planting ONE violation in a private copy of the + @# gate's inputs and running `ci/check-deps.sh --only ` to prove that + @# rule still catches it (since #1386 a self-test runs its own rules, not all 43), on a thread + @# pool (`PLX_TEST_JOBS`, default min(8, cpus)). Measured 2026-11-14 on CI: this step took + @# 144-196 s on the commits since #1386 and 380-500 s on the ones before it. Measure before + @# budgeting; `check-deps.sh`'s per-rule cost, then the self-test count, is where more would + @# come from. run.py @# decides WHAT gets driven on the one television and had no test of any kind until 2026-08-22. @# What it pins is the code path a full manifest.local.json never enters: an `item` key this @# installation cannot resolve SKIPS the cases that need it instead of killing the run. A diff --git a/ci/proofs/platform-generated-enums.json b/ci/proofs/platform-generated-enums.json index d605a09d8a..fcfd161a24 100644 --- a/ci/proofs/platform-generated-enums.json +++ b/ci/proofs/platform-generated-enums.json @@ -26,7 +26,7 @@ }, "text_hashing": "sha256-utf8-bytes-crlf-to-lf-local-root-version-v1", "manifests": { - "Cargo.toml": "fbeb900767c6d3a7091b0366cdaf79753f45384be9b39a649dd8e7decbf40fb6", + "Cargo.toml": "c57b1c2d662d460a6d71f6de158397f1cb2f2fecb167bfa738631d77b661248b", "appkit/Cargo.toml": "1e3a33f392110f269537b12f12879c6bc15309aca304b189d6b0346edf8a1186", "base/Cargo.toml": "ebe0b95553bd44f33f422f7d8e22b75ddeb53c7ee81bbe738f675690694568f7", "data/Cargo.toml": "90a763a8a9e22bda7e95f8ede4dc10ec6e92a62b90bd0fd2e622c617d75ec6c7", diff --git a/ci/proofs/serde-json-immutable.json b/ci/proofs/serde-json-immutable.json index 6a423f3f0f..b84b7d7881 100644 --- a/ci/proofs/serde-json-immutable.json +++ b/ci/proofs/serde-json-immutable.json @@ -373,7 +373,7 @@ }, "lock_sha256": "c56af60124266be5ee0b7dfa7ae81b4f0ff38b047fe672f40a009f6e1c22d3d3", "manifests": { - "Cargo.toml": "fbeb900767c6d3a7091b0366cdaf79753f45384be9b39a649dd8e7decbf40fb6", + "Cargo.toml": "c57b1c2d662d460a6d71f6de158397f1cb2f2fecb167bfa738631d77b661248b", "appkit/Cargo.toml": "1e3a33f392110f269537b12f12879c6bc15309aca304b189d6b0346edf8a1186", "base/Cargo.toml": "ebe0b95553bd44f33f422f7d8e22b75ddeb53c7ee81bbe738f675690694568f7", "data/Cargo.toml": "90a763a8a9e22bda7e95f8ede4dc10ec6e92a62b90bd0fd2e622c617d75ec6c7", @@ -602,7 +602,7 @@ ], "edition": "2021", "lib": "ui/src/lib.rs", - "lib_sha256": "c77775a6faa9583579ee576ea9f9a51a7b9c92d7ef692161761c1e471faf3873" + "lib_sha256": "616bd784cbf479441275a483f83ed2d88df05db406731bb2ead7a8d70845e557" } }, "text_hashing": "sha256-utf8-bytes-crlf-to-lf-local-root-version-v1" diff --git a/ci/readonly_dependencies.py b/ci/readonly_dependencies.py index b0ec65d11a..70f0661c9e 100644 --- a/ci/readonly_dependencies.py +++ b/ci/readonly_dependencies.py @@ -18,7 +18,7 @@ import tarfile CERTIFICATE_PATH = Path(__file__).with_name('proofs') / 'serde-json-immutable.json' -CERTIFICATE_SHA256 = '7a299a63a440720a7f7313fe985519efb77b6f6a2963b5b0d92fc3bc3be987be' +CERTIFICATE_SHA256 = '29e62dc168d2691cebe44a1ce21b59d186002fed59df18bdb2c2b86d76f0497d' SCHEMA = 'plx.readonly.locked-json.v1' REGISTRY = 'registry+https://github.com/rust-lang/crates.io-index' SUPPORTED = ['alloc', 'arbitrary_precision', 'default', 'float_roundtrip', 'raw_value', 'std', 'unbounded_depth'] diff --git a/ci/readonly_generated.py b/ci/readonly_generated.py index 8e222d851a..a0b8bdd6e9 100644 --- a/ci/readonly_generated.py +++ b/ci/readonly_generated.py @@ -13,7 +13,7 @@ from readonly_dependencies import _local_sha, _sha, _json, _manifest_paths, _configs, _workspace_hashes, WORKSPACE_HASHING CERTIFICATE_PATH = Path(__file__).with_name('proofs') / 'platform-generated-enums.json' -CERTIFICATE_SHA256 = 'b61be9b470960a80bbb097dfa466fa5fd4229f89ef8f1234bb302edabf9f8396' +CERTIFICATE_SHA256 = 'c737368bf0642a51f34c3424f9545eaa8becb72dc2206ae5f1a40ddcc3c7f67e' SOURCES = ['platform/Cargo.toml', 'platform/src/lib.rs', 'platform/src/i18n/mod.rs', 'platform/build.rs', 'platform/build_support/catalog.rs', 'build_support/install_identities.rs', 'machine/src/lib.rs', 'machine/src/machine.rs'] diff --git a/docs/agent-reference.md b/docs/agent-reference.md index 6bd6f0005f..c9a2c32d1d 100644 --- a/docs/agent-reference.md +++ b/docs/agent-reference.md @@ -753,7 +753,10 @@ the pinned Sentry Native cross-build), and `sshpass` (Homebrew; deploy/run use y `tools/build-gc.sh` now installs `.claude/worktrees/.cargo/config.toml` with `incremental = false`, which every cargo reads and which stops above the main checkout. Since 2026-09-18 the same file also sets `[profile.dev] debug = "line-tables-only"` and `debug = false` - for third-party packages in lanes (main keeps full DWARF). A lane's target dirs are also no longer + for third-party packages in lanes (main keeps full DWARF for `dev`: the simulator, clippy and + `cargo check`). A host `cargo test` build is the `test` profile, whose line tables + `rust-modules/Cargo.toml` sets for every checkout, main included; a debugger on a host test needs + `CARGO_PROFILE_TEST_DEBUG=2`. A lane's target dirs are also no longer all built from scratch: in a linked worktree the Makefile seeds an ABSENT target dir with an APFS clone of the third-party output (registry crates, once also the build-std sysroot, about 40% of a lane's target bytes then) from `~/.cache/plxnative/cargo-seed/`, via `tools/cargo-seed.py`; the app crate is diff --git a/docs/module-layers.md b/docs/module-layers.md index 6f91e95f6b..7acababeac 100644 --- a/docs/module-layers.md +++ b/docs/module-layers.md @@ -1941,8 +1941,8 @@ The last no-LTO figure is over the 12,426,000 limit, which made Splits 9 and 10 because whole-program optimisation with a single codegen unit sees the same code the single crate did. They also restore the cross-crate inlining the single crate had, which the per-frame UI code leans on. Only release-profile builds pay for it (the ARM staticlib and the storage helper when built as `release`, the -Linux simulator of `make sim-linux` and `make macapp`); the dev profile that `make check`, the host -tests and the macOS simulator use is untouched. The price is build time: CI's ARM library build +Linux simulator of `make sim-linux` and `make macapp`); the `dev` and `test` profiles that the macOS +simulator, `make check` and the host tests use are untouched. The price is build time: CI's ARM library build went from 2m40 to 4m15, and a local `ARM_PROFILE=release` one from 44 s to 94 s. A plain local `make` / `make deploy` no longer pays it: the Makefile builds the ARM staticlib with `[profile.tvdev]` (no LTO, 16 codegen units), which is faster and larger, and keeps `release` for CI, `RELEASE=1`, `SYMBOLS=1` and diff --git a/rust-modules/Cargo.toml b/rust-modules/Cargo.toml index d76401c786..a9912a2090 100644 --- a/rust-modules/Cargo.toml +++ b/rust-modules/Cargo.toml @@ -299,9 +299,9 @@ serde = { version = "1", default-features = false, features = ["derive", "alloc" # back the cross-crate inlining that the UI's per-frame code relied on. # # What it costs is build time, paid only by release-profile builds: CI's ARM library build went from -# 2m40 to 4m15 and a local `ARM_PROFILE=release` one from 44 s to 94 s. `make check` and the host tests run the dev profile -# and do not pay it, and neither does a plain local `make` / `make deploy`: the Makefile builds those -# with `[profile.tvdev]` below. Release-profile builds are CI, `RELEASE=1`, `SYMBOLS=1`, +# 2m40 to 4m15 and a local `ARM_PROFILE=release` one from 44 s to 94 s. `make check` and the host tests (the `test` +# profile, which inherits `dev`) do not pay it, and neither does a plain local `make` / `make deploy`: +# the Makefile builds those with `[profile.tvdev]` below. Release-profile builds are CI, `RELEASE=1`, `SYMBOLS=1`, # `FLAVOR=stable|nightly` and `make ARM_PROFILE=release`. [profile.release] opt-level = 2 @@ -321,6 +321,25 @@ inherits = "release" lto = "off" codegen-units = 16 +# The host UNIT-TEST build (`cargo test`: `make check`'s `check-cargo-unit-*`, `make test-fast`, +# `make test-crate`, `make test-hostsim`, `make build-bench`): every unit of such a build, the +# workspace libraries and the third-party crates under the test binaries included, is built with +# THIS profile and not with `dev` (`cargo test --unit-graph` names the profile of each unit). Line +# tables keep a panic backtrace's file:line and drop the variable-level DWARF; nothing here attaches +# a debugger to a test binary. Measured 2026-11-14 on a checkout with full-DWARF `dev` (the main +# checkout's setting), `make build-bench`, medians: the non-incremental edit rebuild of the test +# build took 14-22% less (plx_base leaf 39.8-43.5 s -> 33.9-35.9 s; plx_ui leaf 21.7-23.4 s -> +# 18.5-20.1 s) and `target/` + `target-fast/` shrank 19% (9.6 GB -> 7.8 GB); the incremental +# `make test-fast` loop was not clearly moved (a plx_ui edit 6.0 -> 6.2 s, a plx_base edit 16.1-21.8 s +# -> 15.5-16.1 s with the baseline drifting between runs). `[profile.dev]` is untouched: the +# simulator, clippy and `cargo check` keep full DWARF in a developer's checkout. Lanes and CI already +# trim `dev` through a cargo config (tools/build-gc.sh, .github/actions/host-rust/action.yml), +# which `test` inherits, so for them this line changes nothing (the unit graph is identical); the +# main checkout is where it takes effect. A debugger on a host test needs +# `CARGO_PROFILE_TEST_DEBUG=2` (`CARGO_PROFILE_DEV_DEBUG` no longer reaches a `cargo test` build). +[profile.test] +debug = "line-tables-only" + [dev-dependencies] # The `cfg(test)` fixtures of the base layer (`testlock`, `testnet`, ...): a dependency is never # built `cfg(test)`, so the tests here ask for them by feature. diff --git a/tools/build-gc.sh b/tools/build-gc.sh index 7a97c3a38f..af50bea831 100755 --- a/tools/build-gc.sh +++ b/tools/build-gc.sh @@ -107,8 +107,8 @@ install_worktree_cargo_policy() { # # Cargo merges config upward from the working directory, so this file applies to every linked # worktree under `.claude/worktrees/` and to NOTHING else: the main checkout lives above this -# directory and keeps its incremental cache and full DWARF, which is the policy the Makefile -# states beside `RUST_FEATFLAGS`. The Makefile can only enforce it for its own cargo invocations; +# directory and keeps its incremental cache and full DWARF in `dev` builds, which is the policy the +# Makefile states beside `RUST_FEATFLAGS`. The Makefile can only enforce it for its own cargo invocations; # an agent running `cargo test` or `cargo check` by hand bypassed it, and 12.9 GB of lane # incremental caches is what that cost. # @@ -137,17 +137,25 @@ incremental = false # 11% cut to the whole tree and a 26% cut to the object files directly, from ONE crate's `--lib` # closure. The saving is modest, plainly: most of the object-file bulk is code, not DWARF. # -# `CARGO_PROFILE_DEV_DEBUG=2` in the environment still overrides this, the same escape hatch as -# `CARGO_INCREMENTAL=1` above: an env var outranks a config file, so a lane genuinely attaching -# `lldb`/`rust-lldb` to a host test can buy full DWARF back for itself without touching this file. +# `CARGO_PROFILE_DEV_DEBUG=2` in the environment still overrides this for a `dev` build (the +# simulator, clippy, `cargo check`), the same escape hatch as `CARGO_INCREMENTAL=1` above: an env +# var outranks a config file. A host TEST is another profile (since 2026-11-14): `cargo test` builds +# every unit, third-party crates included, with `[profile.test]`, and rust-modules/Cargo.toml sets +# that one to line tables itself, which a `[profile.dev]` setting (this file's or the +# environment's) no longer reaches. A lane genuinely attaching `lldb`/`rust-lldb` to a host test +# buys full DWARF back for itself with `CARGO_PROFILE_TEST_DEBUG=2`; third-party crates keep the +# `debug = false` below either way. # -# Checked before landing that nothing else in the nearer config chain already sets this and would -# make the addition a no-op or a conflict: neither `rust-modules/Cargo.toml` (only -# `[profile.release]`) nor `rust-modules/.cargo/config.toml` (only `[target...]` rustflags and -# formerly `[unstable] build-std`) mentions `[profile.dev]` debug. Cargo resolves a manifest `[profile]` -# against a config `[profile]` by letting config win, and the nearest config wins over a farther -# one — both rules favour this file over anything upstream, which is the reason it is safe to add -# here rather than in the crate manifest. +# Checked before landing, for `dev` builds, that nothing else in the nearer config chain already +# sets this and would make the addition a no-op or a conflict: neither `rust-modules/Cargo.toml` +# (its profiles are `release`, `tvdev` and, since 2026-11-14, `test`; none is `dev`) nor +# `rust-modules/.cargo/config.toml` (only `[target...]` rustflags and formerly `[unstable] +# build-std`) mentions `[profile.dev]` debug. Cargo resolves a manifest `[profile]` against a +# config `[profile]` by letting config win, and the nearest config wins over a farther one — both +# rules favour this file over anything upstream, which is the reason it is safe to add here rather +# than in the crate manifest. A `cargo test` build takes its debug level from `[profile.test]` +# instead (the paragraph above), so for it the `debug` line below is redundant; the +# `debug = false` for third-party crates is what this file still adds there. [profile.dev] debug = "line-tables-only" [profile.dev.package."*"] diff --git a/tools/cargo-seed.py b/tools/cargo-seed.py index cfd4abea9e..8ead422b34 100755 --- a/tools/cargo-seed.py +++ b/tools/cargo-seed.py @@ -26,7 +26,7 @@ * **It clones or it does nothing.** `clonefile(2)` is called directly (not `cp -c`, which quietly falls back to a full copy, spending the very disk this exists to save). Another volume, a filesystem without cloning, or any platform other than macOS: a no-op, never a copy. -* **Linked worktrees only.** The main checkout keeps its own incremental cache and full DWARF, +* **Linked worktrees only.** The main checkout keeps its own incremental cache and full DWARF in `dev` builds, which no lane could share, so the Makefile does not call this there and the script refuses too. `PLX_CARGO_SEED=off` disables it everywhere. * **A key guards the seed**: the sha256 of the toolchain (`rustc + -vV` and `rustc -vV`),