fix(cluster): pin and cache the bootstrap toolchain outside the dir the wipe deletes (#392)

[FumingPower3925]

Closes #392.

What was wrong

#388 cached the actions-runner tarball. The same playbook still fetched four more tools from four internet services on every run, into the directory it wipes, behind skip-if-present guards that therefore never skipped, with no retries:

task	source	version before
Install uv	astral.sh	unpinned
Create ansible venv	python-build-standalone via uv	minor only
Install ansible-core	PyPI	unpinned
Install ansible.posix	galaxy.ansible.com	unpinned

Nightly 34981852841 never ran a cell because of the last one. ansible-galaxy collection install ansible.posix gave msa2-client nothing for 2 m 29 s and the whole bootstrap failed, although msa2-server and msr1 had already registered. Measured from the hosts in the same minute: Galaxy's API had a 23.3 s time-to-first-byte and returned 302-with-no-body to msa2-server, while GitHub on the same hosts delivered 48.7 and 61.8 MB/s. The pipe was fine; Galaxy was not.

The change

Pin, then cache. Caching an unpinned install would silently freeze whichever version installed first, with nothing recording which. Pins are uv 0.12.15, python 3.13, ansible-core 2.21.4, ansible.posix 2.2.2. All four were field-proven by the 14:29Z bootstrap on all three hosts, and the versioned uv installer really pins (APP_VERSION="0.12.15"). Each lives in a directory under runner_cache_dir keyed by its pin, so a bump installs fresh instead of reusing a stale copy.

Six workflows hardcode the tool paths — so they stay, as symlinks. matrix-nightly-tier, matrix-weekend-tier, benchmark-tier, matrix-checkptr-tier, matrix-race-tier and matrix-pr-tier never read the exports in "Start runner". Each re-derives RUNNER_ROOT=/tmp/actions-runner-$(hostname -s) and puts ${RUNNER_ROOT}/ansible-venv/bin on GITHUB_PATH and ${RUNNER_ROOT}/ansible-collections in ANSIBLE_COLLECTIONS_PATH. Moving the venv would have broken every cluster tier; a grep of every consumer before touching the path is what caught it. The playbook now recreates those two paths each run as symlinks into the cache, and none of the six files changes.

That is safe only if teardown's wipe does not follow the links. It is ansible.builtin.file state=absent, and in ansible-core 2.21.4 ensure_absent for a directory is exactly shutil.rmtree(b_path, ignore_errors=False). Measured on macOS and on Linux aarch64 CPython 3.13.15 (shutil.rmtree.avoids_symlink_attacks = True): both rmtree and rm -rf of a directory containing a symlink leave the target intact.

uv venv --managed-python. The first end-to-end run found a real flaw: uv prefers a matching python already on PATH over downloading into UV_PYTHON_INSTALL_DIR, so the "cached" venv linked to the test controller's python and a fresh container could not execute it. The cluster only escaped because its system python is 3.14. Verified in isolation with a decoy python3.13 planted on PATH, uv 0.12.15:

A  no flag           venv python -> /opt/leak/cpython-3.13.15-linux-aarch64-gnu/bin/python3.13
B  --managed-python  venv python -> /opt/cache-py/cpython-3.13.15-linux-aarch64-gnu/bin/python3.13

UV_PYTHON_INSTALL_DIR and UV_CACHE_DIR also move uv's python and cache under /tmp, which this playbook's own "nothing lands outside /tmp" rule always required and uv's $HOME defaults quietly broke.

A cache that skips on presence is a liability, so ansible.posix is trusted only if its MANIFEST.json names the pinned version; otherwise it is fetched from Galaxy, and if Galaxy will not serve it, built from the collection's GitHub source tag. ansible.posix publishes no release assets, so its git source is the only non-Galaxy route; git 2.53.0 is on all three hosts. uv is likewise only trusted if uv --version reports the pin.

Guard and controls

TestBootstrapToolchainIsPinnedAndCachedOutsideTheWipedDir checks the pins, that every fetch uses its pin, that no install target or skip guard sits under runner_root, that every tool lives under runner_cache_dir, the manifest check, --managed-python, and the six-workflow symlink contract from both ends — it counts the workflows that consume each legacy path (logs 6 and 6) and fails if that count ever reaches zero.

Each injection was asserted applied before the test ran, and reverted with cp confirmed by SHA-256:

	injection	result
C1	unpin ansible-core	FAILS
C2	move the venv guard back under runner_root	FAILS
C3	drop the ansible-collections symlink	FAILS
C4	ansible_posix_version: "latest"	FAILS
C5	drop the MANIFEST.json integrity check	FAILS
C6	drop --managed-python	FAILS

ansible-playbook --syntax-check is rc=0 on both playbooks (the unedited baseline was also rc=0).

End-to-end: the shipped tasks, actually run

The harness is built by parsing this playbook and keeping its eight toolchain task definitions verbatim (it asserts all eight are found, so it cannot silently shrink). Ubuntu 26.04 container with no python3.13, so uv genuinely downloads python into the cache. Every result below is read from the filesystem, not the play recap.

	scenario	ground truth
S0	uv venv --clear on an existing venv, uv 0.12.15	rc=0 — the flag is safe
S1	cold cache, decoy python3.13 on PATH	rc=0; venv python resolves inside …/tools/uv-python/; uv 0.12.15, core 2.21.4, posix 2.2.2; ansible-doc ansible.posix.sysctl resolves through the legacy workflow paths
S2	teardown's exact ansible.builtin.file state=absent	runner_root gone; uv, venv, collection and uv-python all survive
S4	cached manifest corrupted to 0.0.0	version check fails, rescue reinstalls, manifest back to 2.2.2
S5	Galaxy unreachable, collections deleted	Galaxy retries and fails; GitHub-source install succeeds; manifest 2.2.2
S3	warm cache, --network none, run by the cached venv's own ansible-playbook	rc=0, rescued=0; uv task ok (no fetch); links recreated; offline contract works

What this does not claim

• The e2e ran linux/arm64 (msr1's architecture) only. amd64 is covered by the first live cluster bootstrap after merge, which is also where the cache surviving a real teardown gets checked on all three hosts, the way fix(cluster): cache the actions-runner tarball outside the dir the wipe deletes #388 was.
• On a cold cache every host will show rescued=1 for ansible.posix, because a missing manifest is what routes into the install. That is the design, not a failure.
• Locally I ran the root package suite and the guard tests, not the full go test ./...: another agent was building on the same host, which has run out of process slots three times today. CI runs the full suite here.

[FumingPower3925]

Review response — round 3

Thank you for the review. It was right on the point that matters most: the branch still trusted three cached states on presence alone, which is precisely what it exists to remove. Every minor finding is fixed below. Nits are fixed where they are cheap and correct, and the rest are deferred with the reason.

Ansible runtime

finding	change	guarded by
venv creates: bin/python accepts a dangling symlink (glob → lexists), so the promised self-heal never ran	venv and ansible-core are one task, trusted only when a completion stamp written last exists and ansible-playbook --version runs and reports [core 2.21.4]; a miss removes and rebuilds the venv	C8 (stamp), C9 (presence guard reintroduced)
a slow Galaxy on a cold cache burns ~11 min before the git fallback	every Galaxy attempt runs under coreutils timeout {{ galaxy_attempt_timeout }} (120 s) with retries: 1; the fallback runs under its own galaxy_git_fallback_timeout (300 s)	C11
changed_when raises on a result with no stdout and hides the real cause	(uv_install.stdout | default('')), and the same on the new venv task	—
force: true on the link task disables the target-exists check	force removed; a stat fails the play if either cache directory is missing	C12, C12b (force: yes spelling)

Cache poisoning and partial state

finding	change	guarded by
bin/ansible-playbook exists before uv finishes an install	covered by the stamp plus the live probe above	C8, C9
ansible.posix MANIFEST.json is written first, so a killed install passes the version check forever	ansible.posix is trusted only with a completion stamp and ansible-galaxy collection verify --offline, which checks every file against its recorded hash, and a MANIFEST.json version equal to the pin (verify does not hash the manifest itself — see below)	C10, C15
dangling venv python	covered by the probe	C9
version-keyed directories accumulate in RAM-backed /tmp	deferred — measured at tens of MB per version, and a reboot clears them; documented in RUNNER_BOOTSTRAP.md with the purge command	—
an orphaned install from a cancelled bootstrap could race the next run	deferred — both stamps are written last, so any interleaving turns into a rebuild rather than a poisoned cache	—

Consumers and docs

finding	change
RUNNER_BOOTSTRAP.md says nothing persists between runs	intro corrected; new Persistent tool cache section (layout, keyed by pin, how each tool is verified, symlinks, reboot clears it, purge command); operator step 5 now says /tmp/celeris-runner-tarballs/ is expected to remain
teardown's "Confirm pristine state" overstates what it checks	renamed Confirm the runner dir is gone, with a comment that the tool cache survives by design (the only reference to the old name was that line)
cluster-runner-down comment claims it mirrors runner-setup.yml	comment corrected to say it is a plain install on the GitHub-hosted image, not the self-hosted provisioning
the playbook's "nothing lands outside /tmp" claim is false for ansible-galaxy	ANSIBLE_HOME now points into the tool cache for every ansible.posix task, so its temp dirs and API cache stay inside it; comment updated (C14)
"runner-scoped" wording	reworded: both paths are symlinks into the host's version-keyed tool cache

Supply chain

finding	change
only ansible-core is pinned; its five dependencies are frozen by the cache at whatever resolved first	uv pip install --exclude-newer 2026-09-15T00:00:00Z (ansible-core 2.21.4 was uploaded 2026-09-08T16:51:47Z), and the cutoff is part of the venv directory name, so moving it builds a fresh venv (C7)
the manifest check does not catch a partial install	collection verify --offline, as above (C10)
the git fallback installs a movable tag	installs commit e98d9a0756458be1ac710988498000973889075c — verified against the GitHub API as the lightweight tag 2.2.2, whose galaxy.yml says version: 2.2.2 (C13)
/tmp pre-creation by another local user	owner: "{{ ansible_user_id }}" on both cache directories, so a directory someone else created first fails the play rather than being trusted
Galaxy install without signature verification; curl | sh without a checksum; uv receipt written to $HOME	deferred — Galaxy publishes no signatures for 2.2.2; the uv installer already verifies the sha256 of each archive against the immutable release digests, as your review confirmed; the receipt only enables uv self update, and the version check would catch any drift on the next run

Gate

Syntax-check rc=0 on both playbooks, go vet clean, all five guard tests pass on the good tree, and all 16 controls fail as they must. Each injection is asserted to have applied before the test runs, and each is reverted with cp and confirmed by SHA-256.

One thing the gate caught was mine. The first version of the new guard failed on the good tree: a substring check for force: true matched the comment explaining why there is no force. That made control C12 look like it passed when the good tree already printed its message. The guard now matches force as a YAML key in any truthy spelling, and C12 plus a new C12b (force: yes) are re-proven against a good tree that passes.

End-to-end: the shipped task definitions, actually run

Same harness as before: it parses this playbook and keeps its eight toolchain task definitions verbatim, and fails if any is missing. It runs in an Ubuntu 26.04 container (linux/arm64) with a decoy python3.13 planted on PATH. The test controller's own ANSIBLE_HOME, module temp dir and uv cache are kept out of /root, so anything found there afterwards can only have come from the playbook. Every result below is read from the filesystem, not from the play recap.

	scenario	ground truth
S1	cold cache, decoy python on PATH	rc=0; the venv's python resolves inside the cache; core 2.21.4; ansible.posix 2.2.2; both stamps present; dependencies frozen at cryptography 50.0.1, jinja2 3.1.6, packaging 26.3, PyYAML 6.0.3, resolvelib 1.2.1; ansible-doc ansible.posix.sysctl resolves through the legacy workflow paths
S2	teardown's exact ansible.builtin.file state=absent	runner_root gone; uv, the venv, both stamps, uv-python and ansible-home all survive
S4	cached MANIFEST.json version altered to 0.0.0	cached check fails, rescue reinstalls, manifest back to 2.2.2
S6	a plugin file tampered, manifest and stamp left intact	verify --offline fails, rescue reinstalls, the tampered line is gone
S7	cached python deleted, so the venv interpreter dangles	probe fails, venv rebuilt, its interpreter runs from inside the cache
S8a	venv completion stamp removed	venv rebuilt, stamp back
S8b	ansible/cli/playbook.py deleted, stamp intact	probe fails, venv rebuilt, file back
S5	Galaxy unreachable, collections deleted	both Galaxy attempts fail; the git install of commit e98d9a07 succeeds; 2.2.2; stamp present
S9	Galaxy accepts TCP and never answers, galaxy_attempt_timeout=20	exactly two attempts (one retry line, final rc 124), fallback succeeds, 59 s in total
S10	anything written under /root	/root/.ansible, /root/.cache/uv and /root/.local/share/uv all absent; ansible-galaxy's temp dirs and cache are under ansible-home in the tool cache
S3	warm cache, --network none, run by the cached venv's own ansible-playbook	rc=0, rescued=0; uv, core and ansible.posix all ok — verify --offline works with no network; links recreated; ansible-doc resolves

Two things the e2e caught that the guard alone did not:

• A regression in my own first round-3 cut. I had replaced the warm path's manifest version check with verify --offline. But verify does not hash MANIFEST.json itself, so a manifest altered to 0.0.0 passed the cached check and the play reported success. The warm path now compares the version too, guarded by TestCachedAnsiblePosixCheckComparesTheManifestVersion and control C15. S4 above is the re-run.
• /root/.cache/uv appeared in an earlier run. It came from the harness's own uv commands, not from the playbook; once those were given their own cache directory, S10 reads absent.

Coverage limit: the e2e is linux/arm64, which is msr1's architecture. amd64 — and the cache actually surviving a real teardown on all three hosts — gets verified live on the first cluster bootstrap after merge, as with #388.