Cluster bootstrap re-downloads 215 MiB per host every run and timed out the nightly; its force:false cache guard points inside the directory the previous task deletes

[FumingPower3925]

Nightly tier run 34971158913 never reached a single cell. The bootstrap cluster runners job hit its timeout-minutes: 15 and GitHub cancelled it 15 m 17 s in, mid-TASK [Install uv]; matrix validate was then skipped. Nothing else was in the matrix-tier-cluster group — this was not an eviction.

What ate the budget

12:50:31  TASK [Download actions-runner tarball]
13:03:36  changed: [msr1] / [msa2-client] / [msa2-server]      <- 13m 05s
13:03:56  TASK [Extract tarball]
13:04:32  TASK [Register runner with GitHub]
13:04:54  ##[error]The operation was canceled.                  <- 15m 17s, the cap

Every other task in that run took 3–20 seconds.

The tarball is 225,628,509 bytes (215 MiB), fetched per host, on every cluster run. 785 s for 215 MiB is ≈287 KB/s.

This is not a one-off, and it is not only today

Bootstrap durations, measured from the job records:

run	tier	bootstrap
34003151551	weekend	1m 40s
34368601903	weekend	2m 08s
34616620237	weekend	2m 32s
34724025365	checkptr	1m 47s
34727620439	race	2m 10s
34731237016	checkptr	1m 41s
34736980002	weekend	2m 05s
34876253223	nightly	3m 02s
34918161309	nightly	1m 40s
34940523791	checkptr	2m 58s
34961642523	race, today	12m 20s
34971158913	nightly, today	15m 17s — TIMED OUT

So the historical norm is under three minutes and today's two runs are 12 and 15. The download is the variable; everything else is constant. Today's race tier survived only because 12 m 20 s happens to be under 15 m — it had 2 m 40 s of margin, on a job whose cap has never been sized for a 215 MiB transfer.

There is already a commit dc6e702 ci: retry actions-runner tarball download (transient network timeout), so this download has been fragile before. The retry (retries: 5, delay: 10, timeout: 60) does not help here: get_url's timeout is the URL-open timeout, not a transfer cap, so a slow-but-progressing transfer is never retried — it just runs until the job dies.

Why the cache never helps today

ansible/runner-setup.yml does this, in order:

- name: Wipe stale runner dir
  file: { path: "{{ runner_root }}", state: absent }
- name: Ensure runner root dir
  ...
- name: Download actions-runner tarball
  get_url:
    dest: "{{ runner_root }}/{{ runner_tarball }}"
    force: false          # <- can never match: the dir was just deleted

force: false is intended to skip a download when the file is already there, but the destination lives inside the directory the previous task just removed, so it is dead code. Every bootstrap re-downloads 215 MiB on all three hosts unconditionally.

The wipe itself is correct and should stay — its comment explains why (a stale .runner pins a registration GitHub has deleted, and run.sh then exits with "the runner registration has been deleted from the server"). But that rationale is about the runner configuration, not about the tarball bytes. The tarball is immutable and already content-addressed by version in its filename.

Proposed fix

1. Download into a cache directory outside runner_root — /tmp/actions-runner-cache/ keeps the playbook's "nothing lands outside /tmp" rule — with force: false, which then actually means something. Unarchive from the cache into the freshly wiped runner_root.
2. Make a bad cache self-healing: wrap the extract in block/rescue, and on failure delete the cached tarball and re-download once. Otherwise one corrupt cached file poisons every future run silently, which is worse than a slow one.
3. Confirm runner-teardown.yml removes only runner_root and leaves the cache (it appears to — worth asserting in a guard test rather than reading it once).
4. Raise the bootstrap timeout-minutes from 15. A cold cache still has to move 215 MiB, and 15 minutes has no margin for that even on a good day.
5. A guard test in the workflow_*_test.go family: the download destination must not be under runner_root, and teardown must not remove the cache path.

With the cache warm, the download disappears from every run after the first per host per runner version, and a slow-network day stops being able to block the release sequence.

[FumingPower3925]

Verified on real hosts, not just in CI

#388's first live run was nightly 34981852841. That run failed, for an unrelated reason (a Galaxy timeout, now #392), but it exercised every part of this change on real hardware:

• TASK [Extract tarball] ran 14:29:30 → 14:29:44Z, 14 seconds, extracting from the cache on all three hosts.

• The play recap shows rescued=0 on every host, so the cached tarball passed its own integrity check and the re-fetch path never ran.

• The cache survived teardown. After teardown cluster runners finished at 14:37:08Z, and with zero runners still registered on the repository, /tmp/celeris-runner-tarballs still held the right artifact on each host:

  msa2-client  actions-runner-linux-x64-2.335.1.tar.gz
  msa2-server  actions-runner-linux-x64-2.335.1.tar.gz
  msr1         actions-runner-linux-arm64-2.335.1.tar.gz
  

  Each file matches its host's architecture, and teardown did not remove any of them. That is what the guard test asserts statically, now observed on disk.

The bootstrap as a whole is still not solid. The same playbook fetches four more tools on every run into the directory it wipes, two of them unpinned. That is #392.