16 unretried `curl | sh` rustup installs across 12 workflows — no retry, and no pipefail, so a truncated installer can pass the stage

[avrabe]

[fathom (gale)] — root-caused from a real red on #285, which only bumps a tool pin and had no business failing a Zephyr kernel test.

The failure

msgq_usage (qemu_cortex_m3) failed in step 4, "Install Rust + ARM target" — before any gale code ran:

curl: (35) Recv failure: Connection reset by peer
error: rustup could not choose a version of rustup to run, because one wasn't
       specified explicitly, and no default is configured.
##[error]Process completed with exit code 1

The curl https://sh.rustup.rs | sh was reset at the network level, so no default toolchain was installed, and the next line (rustup target add thumbv7m-none-eabi) failed. External cause, but a fragility we own.

Two distinct problems

1. No retry, on a step that runs dozens of times per PR.

grep -rn "sh.rustup.rs" .github/workflows/*.yml   ->  16 sites across 12 workflows

zephyr-tests.yml alone has four, and they feed the big matrices (~45 qemu_cortex_m3 jobs, mps2/an385, the SMP set). A single PR therefore performs dozens of unretried network installs; with several PRs open the expected number of resets stops being small. I noticed this precisely because I had four PRs in flight — my own queue pressure made a latent fragility visible.

2. curl … | sh with no pipefail can pass a TRUNCATED install.

The step has no shell: override, so it runs under Actions' default bash -e {0} — which sets -e but not -o pipefail. In curl -sSf … | sh -s -- -y, the pipeline's exit status is sh's. If curl dies mid-transfer, sh executes a partial installer and can exit 0.

This time the truncation was caught by the following line failing. That was luck, not design: a partial install that happens to leave a usable rustup shim would pass the stage and fail later somewhere far less legible — or not at all.

Proposed fix

Split download from execution, and let curl do the retrying:

- name: Install Rust + ARM target
  run: |
    set -euo pipefail
    apt-get update -qq && apt-get install -y -qq curl > /dev/null 2>&1
    curl --proto '=https' --tlsv1.2 -sSf \
         --retry 5 --retry-all-errors --retry-delay 2 --connect-timeout 15 \
         https://sh.rustup.rs -o /tmp/rustup-init.sh
    sh /tmp/rustup-init.sh -y --default-toolchain stable
    . /root/.cargo/env
    rustup target add thumbv7m-none-eabi

Three changes, each doing one thing:

• --retry 5 --retry-all-errors — --retry alone does not cover (35) Recv failure; --retry-all-errors is the flag that makes connection resets retryable. This is the actual fix for the observed failure.
• -o then sh — a truncated download can no longer be executed, because curl's non-zero exit stops the step before sh runs. This is the fix for the silent variant.
• set -euo pipefail — belt and braces; with -o file the pipe is gone anyway.

Kill-criterion

Point the URL at a host that resets the connection (or block it) and confirm the step fails at the download, not three lines later — and that no rustup-init.sh is executed. A retry that has never been shown to catch anything is not a retry.

Scope / honesty

• This is an infrastructure failure with an identified mechanism, not a "flaky test". I am not proposing to re-run and move on; the re-run is to unblock chore(synth): bump the pin 0.52.0 -> 0.57.0 — the isolation core goes 8/9 -> 9/9 #285, the fix is this issue.
• I have not proven the silent-truncation path has ever actually occurred here. The (35) reset is observed; the missing-pipefail hazard is read off the step definition and Actions' documented default shell. Recorded as a latent hazard, not an incident.
• Deliberately not landing this yet: four PRs are already saturating the runners, and a 17-site workflow change would add another full matrix. It should land when the queue drains.
• Related but separate: these steps set RUSTUP_HOME: /mnt/.rustup while sourcing /root/.cargo/env — the /root disk-exhaustion workaround. Not touched here.

[avrabe]

[fathom (gale) — this is broader than rustup: three distinct unretried network fetches failed in CI today]

I filed this about curl | sh rustup installs. That was the first instance, not the class. Over roughly six hours, three different unretried network fetches failed on this repo's CI, each taking down an unrelated check:

#	step	failure	check taken down
1	curl … sh.rustup.rs | sh	curl: (35) Recv failure: Connection reset by peer	msgq_usage (qemu_cortex_m3) on #285
2	bazel fetch of rules_lean…mathlib	_lean_toolchain/bin/lake: 9: Syntax error: Unterminated quoted string	Lean Proofs (7 files) on #288
3	west init -> git clone …/zephyr.git	fatal: unable to access …: Could not resolve host: github.com	sleep (qemu_cortex_m3) on #288

(1) and (3) are confirmed network failures — a connection reset and a DNS resolution failure, both quoted verbatim from the job logs.

(2) is a hypothesis, and I am flagging it as one. On my machine _lean_toolchain/bin/lake is a genuine 51 KB Mach-O binary, so the Linux artefact should likewise be an ELF. A valid ELF is exec'd by the kernel, not parsed by sh — so for sh to reach line 9 and report an unterminated quoted string, the file was almost certainly truncated or an error page. That is the same signature as a partial download being executed. A re-run is in flight and will separate transient from structural; if it goes green, (2) joins (1) and (3).

Why this changes the fix

The original proposal added --retry-all-errors to one curl in 16 places. That is still right, but it addresses one of three. The pattern is:

Any unretried network fetch in a CI step is a single point of failure for every check in that job, and none of gale's are retried or integrity-checked.

Three consequences worth separating:

• Loud failures (1, 3) cost a re-run. Annoying, and they make a green matrix a coin-flip when 60 jobs each do several fetches.
• Silent failures (2) are the dangerous shape: a truncated artefact that executes and produces a downstream error nowhere near the real cause. It took reading a lake shim's line number to get from "Lean proofs failed" to "the fetch failed". That is the class where the original curl -sSf | sh without pipefail also lives — a partial installer piped into sh.
• Queue pressure multiplies it. Four PRs x ~60 jobs x several fetches each is on the order of a thousand unretried network operations per round. That is how (1) surfaced; the odds of some fetch failing approach one.

Revised ask

Not just rustup. An audit of the fetch points, with retry and integrity checking where the artefact is later executed:

1. curl … sh.rustup.rs | sh — 16 sites, 12 workflows (original proposal; -o then sh, --retry-all-errors).
2. west init / west update — network clones with no retry. This is the one that hit today.
3. The bazel rules_lean / mathlib fetch — worth checking whether rules_lean verifies what it downloads. patches/rules_lean_mathlib_timeout.patch shows we have patched it before.
4. pip3 install west, cargo install wasm-tools/wac-cli — same class, not yet observed failing.

I am not proposing to fix all four now, and I am deliberately not opening a PR: gale's matrix is ~60 jobs per PR and the queue is the constraint. Recording the pattern while the evidence is fresh, since "CI was flaky today" is exactly the framing that loses it.

Kill-criterion, unchanged in spirit

For each fetch point: make the source fail (point at a host that resets, or block DNS) and confirm the step fails at the fetch, with the fetch named — not three lines later in an unrelated tool, and not silently by executing a truncated file.

[avrabe]

[fathom (gale) — a fourth mechanism, and it is NOT a fetch]

This issue now covers three unretried-network-fetch failures. A fourth appeared on #308 that does not belong to that family, and I would rather record the distinction than let it be absorbed:

Zephyr C Coverage — step 5 "Initialize west workspace"
##[error]The runner has received a shutdown signal. This can happen when the
         runner service is stopped, or a manually started runner is canceled.
##[error]Process completed with exit code 1

GitHub terminated the runner mid-job. No retry, integrity check or pipefail fix addresses that — the process was killed from outside. Re-running is the only response, and it is the correct one.

Why the distinction matters for this issue's fix

The four so far:

#	mechanism	fixable here?
1	curl … sh.rustup.rs — connection reset	yes — --retry-all-errors
2	bazel fetch of mathlib — wrong-platform lake	no — was rules_lean#30, fixed upstream
3	west init git clone — DNS failure	yes — retry
4	runner shutdown mid-job	no — nothing in the repo can prevent it

If the retry work lands and someone measures "CI still goes red sometimes", (2) and (4) are why — and neither is evidence the retries failed. Worth having written down before that conversation happens rather than after.

One thing (4) does argue for

It is an argument for the merge gate (#294), not for retries. A runner shutdown produces a red that is indistinguishable, at a glance, from a real failure — and with required_status_checks.contexts == [] nothing forces anyone to look. The safe response to (4) is "re-run and check", and today the repo does not require that anyone does.

Re-run triggered on #308; the other 57 checks were green.

[avrabe]

[fathom (gale) — a fifth mechanism on the same PR, and this one has a known history]

#308 went red twice more while draining, neither a fetch:

(4) runner shutdown — reported above.

(5) container setup abort. smp_semaphore (qemu_x86_64, SMP):

started 16:01:58 -> completed 16:47:58   (46 minutes)
step 2 "Initialize containers"  in_progress
steps 3-12                      pending
log blob: BlobNotFound

The job spent 46 minutes in container initialisation, never reached checkout, and produced no log at all. That is the empty-log / empty-steps signature, and it is worth naming because it looks nothing like a test failure — the check goes red with no output to read, and the instinct is to assume the test is broken.

gale has seen this shape before as a /root disk-exhaustion vector in the Zephyr jobs (image pull + workspace + SDK extract + rustup rust-std, mitigated by moving to /mnt). I am not claiming that is the cause here — I have no log to prove it, which is exactly the problem — only that the signature matches and the next occurrence is worth checking against it rather than re-run-and-forget.

Running tally, for whoever implements the retries

#	mechanism	fixable by retry/integrity work?
1	rustup curl — connection reset	yes
2	mathlib fetch — wrong-platform lake	no (was rules_lean#30, fixed upstream)
3	west init clone — DNS failure	yes
4	runner shutdown mid-job	no
5	container setup abort, 46 min, no log	unknown — needs a log first

Two of five are addressable by this issue's proposed fix. That is still worth doing — (1) and (3) both cost a full matrix re-run — but "CI is flaky" is at least five distinct things here, and conflating them will make the retry work look ineffective when it is not.

For (5) specifically the useful next step is observability, not a retry: a job that dies in setup with no log cannot be diagnosed at all. Capturing df -h and the container pull result before the workspace step would turn the next occurrence into evidence.

[avrabe]

[fathom (gale) — a live instance, discriminated, plus a second defect at the same line]

This issue predicted the failure; here is one, caught in the act, with the control
that distinguishes it from an outage.

The failure

PR #319, llvm-lto-test (pipe, tests/kernel/pipe/pipe_api):

  6. failure  Install matching Clang + LLD
  7. skipped  Initialize west workspace
  8. skipped  Setup SDK paths
  9. skipped  Build and run test (LLVM+LTO)

The line:

wget -qO- https://apt.llvm.org/llvm.sh | bash -s -- "${RUST_LLVM}" all

It was transient, and here is the control rather than the assertion

Two independent checks, because "flaky" is not a diagnosis:

1. apt.llvm.org was reachable — fetched directly during the failure window,
   HTTP 200 with the real script body (8476 bytes).
2. A sibling leg passed the identical step in the same run.
   llvm-lto-test (msgq, …) reached success while pipe had already failed at
   step 6 — same workflow, same runner pool, same install.

One leg failing while its sibling succeeds on the same fetch is a transient
network failure, and it is fatal only because nothing retries. That is this
issue, reproduced.

The second defect, at the same line, and it is worse

GitHub's default shell for run: is bash -e, not bash -eo pipefail. In
wget … | bash, the pipeline reports bash's status. A failed or truncated
download feeds bash empty stdin, bash exits 0, and the step passes with
clang never installed — the failure then surfaces later as an unrelated-looking
build error.

Demonstrated mechanically (404 URL, set -e, with and without pipefail):

WITHOUT pipefail: exit 0  <-- failed download reported SUCCESS
WITH pipefail:    non-zero

So the issue title understates it: these steps do not merely fail without retry,
they can succeed without installing anything.

Current population

pipe-to-shell installs: 15 occurrences across 8 workflows
apt.llvm.org specifically: 4 sites

PR #320 fixes the four apt.llvm.org sites

set -o pipefail, three attempts with 15s/30s backoff, explicit non-zero exit
with a reason. Scoped to four rather than fifteen for a reason:

one of them is in release.yml. A transient fetch there either aborts a
release or — via the pipefail defect — produces a release built without the
matching clang, silently. That is a different severity from a retryable CI leg,
and it is why these went first.

The remaining 11 stay here. #320 is the pattern to copy.

[avrabe]

[fathom (gale) — survey of the remaining 11 sites, and a trap in the obvious fix]

#320 fixes the 4 apt.llvm.org sites. Surveying the other 11 before anyone
batches them, because the obvious fix has a hazard I walked into myself.

The trap

The natural patch is set -o pipefail at the top of the step. That persists for
the whole step, so it also applies to any later pipeline in the same block —
and something --version | head -1 is a pipeline where head closing early can
hand the producer SIGPIPE, making a perfectly healthy command report 141.

I did exactly this in the first version of #320 and caught it on self-review, not
from CI — the collateral failure would have surfaced later looking like an
unrelated flake. The fix is to scope pipefail to the pipeline under test:

if ( set -o pipefail; curl -sSf … | sh -s -- … ); then

Which of the 11 are exposed

workflow	step	later pipes in the same step
nightly.yml	Install Rust + target	2
renode-tests.yml	Install Rust + target	2
coverage.yml	Install Rust + ARM target	0
llvm-lto.yml ×2	Install Rust stable + ARM target	0
release-wasm.yml	Install Rust + wasm32 target	0
release.yml	Install Rust + wasm32 target	0
zephyr-tests.yml ×4	Install Rust + {ARM, x86_64} target	0

Two sites would regress under the naive patch. The other nine would be fine
either way — but the scoped form is correct everywhere, so there is no reason to
apply two different shapes.

Note on severity ordering

These 11 are all rustup/toolchain installs. #320's four were apt.llvm.org, and
one of those sits in release.yml — where the pipefail defect means a release
built without the matching clang, silently. That is why they went first.
release.yml also appears in this list (the rustup site), so it is worth doing
next rather than last.