The cluster bootstrap still re-fetches four tools from four internet services every run, two of them unpinned: a Galaxy timeout on one host failed nightly 34981852841

[FumingPower3925]

Nightly 34981852841 never ran a cell, for the second time today. The bootstrap failed on one of three hosts and that failed the whole job, although msa2-server and msr1 had already started and registered their runners.

TASK [Install ansible.posix collection]          14:30:05Z -> 14:32:38Z
fatal: [msa2-client]: FAILED! => {"cmd": [".../ansible-galaxy", "collection", "install",
  "--collections-path", "/tmp/actions-runner-msa2-client/ansible-collections", "ansible.posix"],
  "delta": "0:02:29.086517", "rc": 1,
  "stderr": "[ERROR]: Failed to download collection tar from 'default' due to the following
             unforeseen error: The read operation timed out"}
msa2-client : ok=11 changed=8 unreachable=0 failed=1
msa2-server : ok=16 changed=11 unreachable=0 failed=0
msr1        : ok=16 changed=11 unreachable=0 failed=0

It was Galaxy, not the network

The cluster uplink had been fixed about ten minutes earlier (#390). Measured from the hosts moments after the failure, with a GitHub download on the same host in the same minute as a control:

host	galaxy.ansible.com	GitHub control
msa2-client	ansible-posix-2.2.2.tar.gz, 166 KB in 1.03 s, ttfb 0.77 s	50 MiB at 48.7 MB/s, ttfb 0.20 s
msa2-server	HTTP 302, 0 bytes after 12.4 s, ttfb 2.59 s	50 MiB at 61.8 MB/s, ttfb 0.20 s
msa2-client	Galaxy API index: HTTP 200 but ttfb 23.3 s	—

The pipe was fast. Galaxy was answering slowly or not at all, and one host's two-and-a-half-minute stall was enough.

#387 fixed one of five instances of this pattern

#388 cached the actions-runner tarball outside the directory the bootstrap wipes. The same playbook fetches four more tools from the internet on every run, into that same wiped directory, each behind a skip-if-present guard that can therefore never skip:

task (ansible/runner-setup.yml)	fetches from	version	guard
Install uv	astral.sh/uv/install.sh (+ GitHub)	unpinned, latest installer	[ -x {{ runner_root }}/uv/uv ] — inside the wipe
Create ansible venv via uv	python-build-standalone via uv	--python 3.13 (minor only)	creates: {{ runner_root }}/ansible-venv/bin/python — inside the wipe
Install ansible-core via uv pip	PyPI	unpinned	creates: {{ runner_root }}/ansible-venv/bin/ansible-playbook — inside the wipe
Install ansible.posix collection	galaxy.ansible.com	unpinned	creates: {{ runner_root }}/ansible-collections/... — inside the wipe

None retries. Four independent external services, any one of which can fail the cluster run.

The comments say this is deliberate ("lives inside runner_root so runner-teardown.yml wipes it"), for the same pristine-by-design reason as the tarball. The argument from #387 carries over: the pristine rule exists because a stale .runner configuration pins a deleted registration. It is about runner configuration, not about re-downloading tooling.

The unpinned installs are their own problem

ansible-core and ansible.posix install whatever is latest at bootstrap time. So the version of the tool that drives every validation run can change between two runs with no commit in any repository, which makes the apparatus itself a moving part. That is at odds with the repository's frozen-dependency policy for everything else it deploys (committed lockfiles, --locked, --frozen-lockfile, --require-hashes).

It also means naive caching would be wrong: caching an unpinned install silently freezes whichever version happened to install first, with nothing recording which one it was. Pinning has to come first, or together.

Proposed fix

1. Pin uv, ansible-core and ansible.posix to explicit versions. Python is already pinned to a minor.
2. Cache them outside runner_root, under the existing runner_cache_dir from fix(cluster): cache the actions-runner tarball outside the dir the wipe deletes #388, keyed by version so a bump installs fresh instead of reusing a stale copy. Point ANSIBLE_COLLECTIONS_PATH and the venv PATH export in "Start runner" at the cache.
3. Retry each cold fetch, and treat a cached artifact that fails its own check the way fix(cluster): cache the actions-runner tarball outside the dir the wipe deletes #388 does: discard it and re-fetch once, never skip it forever.
4. Consider a non-Galaxy source for ansible.posix. With a persistent cache Galaxy is only hit on a cold cache, but a cold cache is the first run after every reboot.
5. Extend TestRunnerTarballIsCachedOutsideTheWipedDir so no network-fetching task may keep its artifact or guard under runner_root, with a control per task. fix(cluster): cache the actions-runner tarball outside the dir the wipe deletes #388's guard caught its author's own first draft; one covering only the tarball left these four uncovered.

[FumingPower3925]

Verified live on the cluster: both the cold and the warm path

This closed when #393 merged, before any real bootstrap had run it. Two cluster runs since then exercise both paths on all three hosts (msa2-server, msa2-client, msr1).

Cold: nightly 34994706167. Runs on an empty tool cache. Bootstrap took ~4 min.

task	all 3 hosts
Ensure tool cache dir	changed
Install uv	changed (a real install)
Install ansible-core into a uv venv	changed (a real install)
Check the cached ansible.posix	failed on the missing stamp, so the rescue ran, which is correct for a cold cache
Install ansible.posix from Galaxy	changed (no git fallback needed)
Manifest version assert / verify by hash / stamp	ok / ok / changed
Point the legacy runner_root paths at the cache	changed
Recap	ok=23 changed=11 failed=0 rescued=1

The same nightly's teardown skipped "Remove installed toolchain directories" on every host and wiped only the runner dir.

Warm: race tier 35024823796. The next bootstrap on the same hosts, taking ~2 min.

task	all 3 hosts
Download actions-runner tarball	ok (cached)
Install uv	ok
Install ansible-core into a uv venv	ok
Check the cached ansible.posix is complete and the pinned version	ok: stamp, verify --offline and manifest version all pass
Install ansible.posix from Galaxy	did not run
Point the legacy runner_root paths at the cache	changed (re-created after the runner-dir wipe, by design)
Recap	ok=18 changed=6 failed=0 rescued=0

So the toolchain survives teardown, is trusted only after it is verified, and a warm bootstrap makes no uv, Python or Galaxy downloads.

The same nightly also confirmed #394 live: 66 of 66 result documents carry tier_1.engine_counters with 23 keys, and 64 of 64 series files have the 10 new columns.