fix(cluster): cache the actions-runner tarball outside the dir the wipe deletes

[FumingPower3925]

Closes #387.

The defect

ansible/runner-setup.yml wiped runner_root, then downloaded the actions-runner tarball into runner_root with force: false. That flag is supposed to skip a download when the file is already present — but the file it looks for lives in the directory the previous task just deleted, so it could never be there. Dead code, and 215 MiB re-fetched per host on every cluster run.

What it cost

run	tier	bootstrap
34918161309	nightly	1 m 40 s
34940523791	checkptr	2 m 58 s
34736980002	weekend	2 m 05 s
…9 more before today		all under 3 m
34961642523	race, today	12 m 20 s
34971158913	nightly, today	15 m 17 s — job cancelled at its cap

The nightly's matrix validate was skipped and no cell ran. Nothing else was in the matrix-tier-cluster group, so this was a timeout, not an eviction. The download alone was 13 m 05 s of it; every other task in that playbook took 3–20 seconds.

225,628,509 bytes in 785 s is ≈287 KB/s.

Retrying was already tried — dc6e702 ci: retry actions-runner tarball download — and cannot help: get_url's timeout is the URL-open timeout, not a transfer cap, so retries: 5 never fires on a slow-but-progressing transfer. It just runs until the job is killed.

The change

Cache outside the wipe. runner_cache_dir, force: false against it, unarchive from there into the freshly wiped runner_root. After the first run per host per runner version the download disappears entirely.

The wipe stays exactly as it was. Its comment explains why a stale .runner is unsafe — it pins a registration GitHub has since deleted and run.sh exits with "the runner registration has been deleted from the server" — but that is an argument about the runner configuration, not about bytes GitHub hands back byte-identical for a filename that already contains the version.

The cache is /tmp/celeris-runner-tarballs, not /tmp/actions-runner-cache. The obvious name is runner_root on a host called cache, and the wipe would delete it. My first draft used the obvious name and the guard test rejected it — included below, because a guard catching its own author on the first run is the only evidence that it bites.

A bad cache self-heals. A cache turns one corrupt download into every future run's problem, and a skipped-because-present file looks exactly like success. The extract is therefore the integrity check: block/rescue discards the cached tarball and fetches it once more if it will not unpack.

timeout-minutes: 15 → 25 on the bootstrap job in all six cluster workflows. A cold cache still moves 215 MiB three times; 15 had no margin even on a good day.

Controls

Every injection was diffed against a backup before the test ran, and reverted with cp, confirmed by SHA-256 match (not just git status).

Control	Injection	Result
1	get_url dest back inside runner_root	FAIL — a get_url writes to "{{ runner_root }}/{{ runner_tarball }}" instead of the cache
2	cache dir renamed to /tmp/actions-runner-cache	FAIL — sits under runner_root … the wipe deletes it and force:false becomes dead code again
3	teardown given a task removing the cache	FAIL — runner-teardown.yml references runner_cache_dir
4	one workflow's bootstrap back to timeout-minutes: 15	FAIL — matrix-race-tier.yml: bootstrap timeout-minutes=15, want >= 20

Vacuity: TestClusterBootstrapHasHeadroomForAColdCache logs checked 6 cluster workflow(s) and fails outright if it finds zero, so it cannot pass over an empty set.

runner-teardown.yml is byte-identical to HEAD — it only ever removed runner_root, and control 3 guards that it stays that way.

What I ran

gofmt -l . clean, go vet ./... clean, and the root package green both untagged and under -tags mage. I did not run the full go test ./... locally: another agent is working in this repo and the heavy validation package has exhausted this host's process table three times today. CI runs the full suite on this PR, including the tagged job added by #385.

Not addressed here

Whether the cluster's link to github.com was degraded today specifically, or whether 287 KB/s is its normal ceiling for a large object, is unmeasured — my own SSH sessions to the cluster hung repeatedly while investigating, and one 3-packet ping run showed 100 % loss between two runs that showed 0 %. That is worth its own look, but it is not what this PR claims to fix: the point is that the bootstrap should not be moving 645 MiB on a good day either.

[FumingPower3925]

Following up on this PR's "Not addressed here" section, which said the link question was unmeasured. It is measured now, and one piece of what I wrote there was wrong.

Wrong: I cited my SSH sessions hanging as a signal. They were not one. Tailscale SSH was waiting on an interactive re-authentication check (login.tailscale.com/a/...); once it cleared, SSH to all three hosts was immediate. That symptom had nothing to do with the link.

Right, but for reasons I had not established: the link is degraded. Tracked as #390.

The decisive comparison is the same task on the same hosts twelve hours apart: TASK [Download actions-runner tarball] ran 01:40:34 → 01:40:48Z in nightly 34918161309 — 14 seconds for the same 645 MiB that took 785 seconds at 12:50Z. A factor of 56 on the same day.

Measured live on an idle cluster: 1.1 MB/s single-stream from GitHub on each of two hosts, but 867 KB/s aggregate when both run at once — two streams total less than one, so it is a single contended pipe, which predicts the 785 s almost exactly. speed.cloudflare.com came back at 193 KB/s, five times slower again, so it is not GitHub.

That does not change anything in this PR. Moving 645 MiB per bootstrap was wasteful at 46 MB/s too; it just was not visible until the pipe narrowed. But it does mean the cache is not a fix for #390, and I am holding the 24-hour soak and the benchmark until the link recovers.