ci: an apt/PPA outage fails a leg the same way a defect does, and every PR re-derives that from a log

[Yaraslaut]

Filed after the third occurrence in one day of CI legs going red for reasons entirely outside this repository. I flagged the pattern on PR #661 after the second and said it would be worth a ticket if a third appeared. It has.

The three occurrences, all measured

1. 2026-09-21 ~07:30Z — PR #661, four legs. ubuntu-toolchain-r PPA returning 503:

E: Failed to fetch https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu/pool/main/g/gcc-16/libgcc-s1_16-…_amd64.deb   503  Service Unavailable
##[error]Process completed with exit code 100

Confirmed across all four failing jobs (503 counts 6, 4, 6, 4). Re-running the failed jobs later cleared them and the PR merged unchanged.

2. 2026-09-21 ~12:42Z — PR #665, seven legs. Launchpad again, different symptom:

lazr.restfulclient.errors.ServerError: HTTP Error 500: Internal Server Error
b'GPGKeyTemporarilyNotFoundError'
##[error]Process completed with exit code 1

Six of the seven show that; the seventh (Linux / Qt6 WebSockets) shows the 503 form instead. Same root, two symptoms.

3. Same window — PR #671, three legs, same shape.

The cost, which is not the outage

Each occurrence cost a diagnosis cycle on a PR that had nothing to do with it, and in each case the diagnosis was only possible because someone read a job log. Concretely, on #661 the sequence was: four legs red → hypothesis that the branch's timing-shaped test was at fault → log read → PPA outage → re-run → merged unchanged. The branch was never the problem, and nothing in the check list said so.

Worse, the failures are indistinguishable from real ones at the level anyone looks first. A red Valgrind memcheck and a red Linux / all optional features (gcc) look exactly like a change that broke Valgrind and GCC. On #665 the shape was actively misleading: the PR had just been rebased onto a changed include/morph/core/bridge.hpp specifically because the combination was untested, so seven red legs was the expected shape of a real regression. They were all apt.

Verification status: reproduced three times, cause read from logs; no fix designed

What I have is nine job logs across three PRs. What I have not done: checked whether the repository could detect this condition reliably, measured how often it happens over a longer window, or established whether GitHub's own retry behaviour already covers part of it. This is a report of a pattern, not a worked solution.

What would resolve it

The goal is to make "the environment failed" distinguishable from "the change failed" without a human reading a log. Options, none of them measured:

• Retry the dependency-installation step with backoff. Cheapest, and it would have absorbed all three of these outright — apt-level transient failures are the well-trodden case for it. Risk: it hides a genuine and persistent packaging breakage behind three slow attempts.
• Fail the step with a distinguishable marker — an annotation or a distinct exit path saying the failure was package installation, so a reader sees it in the check list rather than in the log.
• Cache more aggressively so a Launchpad outage does not reach the runner at all. The Cache apt packages step already exists; on bank/gui + ci: clear the eleven bank-GUI sources' clang-tidy debt, and gate the Q_OBJECT header split that no build catches early (fixes #656, fixes #659) #665 it reported Cache not found for input keys: apt-valgrind-…, so the miss is part of the story.

What would change the verdict

Close as wontfix if the measured frequency is low enough that three in one day was an unlucky window rather than a rate — that is a question about a longer sample than I have. Re-open if a fourth occurrence costs another PR a diagnosis cycle.

Related

• ladder: the ledger scenario corpus fails intermittently with "database is locked" — observed once, not reproduced #658 — the ledger scenario corpus's intermittent SQLite database is locked. Different cause, same category: a red leg that is not the branch's fault and that every reader must diagnose from scratch.
• clang-tidy-diff run locally is silently blind to Catch2 TEST_CASE cognitive-complexity findings the CI job reports #666 — a local gate that is silently blind where CI is not. The mirror image: there, silence is wrong; here, noise is wrong.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Fourth occurrence, and a different upstream — 2026-09-21 ~14:53Z, PR #671

Two legs, Linux / gcc-debug and Linux / gcc-release, both in the Install sccache step:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
##[error]Process completed with exit code 2

The installer was handed something that is not a tarball — almost certainly an error page from the download host — and piped it to tar unchecked. Confirmed on both legs; no compilation ran.

This is a different upstream from the three already recorded (ubuntu-toolchain-r 503, Launchpad GPG 500, and the unrelated database is locked flake in #658). That strengthens the ticket rather than diluting it: the common factor is not one unreliable host, it is that any setup-step failure is presented to a reader identically to a defect.

It also shows the cost sharply. PR #671 is a container change inside Strand, and "both GCC legs red while every clang leg passes" is precisely the shape a real portability bug would take. A reader who trusted the check list would have started looking for one. Nothing compiled.

One addition to this issue's options list, specific to this occurrence and cheaper than the others: the installer pipes a download straight into tar without checking it. curl --fail (or verifying the magic bytes) would turn "not in gzip format" into "download failed with HTTP N" at the point of failure, with no retry policy and no CI-wide design decision needed. That does not fix the class, but it fixes the one case where the log actively misleads.

Still not measured: the frequency over any window longer than today, and whether GitHub's own retry behaviour already covers part of it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VptDWG2fKr2vBnLSJcgzgW

[Yaraslaut]

Triage: valid — but the remaining work is blocked on a decision, not on effort

Verified on master @ a9cb5649.

The frequency question the issue left open is now answered

The issue offers one way out: "Close as wontfix if the measured frequency is
low enough that three in one day was an unlucky window rather than a rate — that
is a question about a longer sample than I have."

The longer sample exists now, and it does not support wontfix. Two further
occurrences of the same class followed the three in the body, with different
causes, which is the point — this is not one flaky host:

• sccache tarball, PR core/cmake: stop a serial dispatch rebuilding a std::deque it never fills, and say what -Wno-shadow-uncaptured-local really suppresses (fixes #660, refs #662) #671, both GCC legs. Recorded verbatim in the tree at
  .github/workflows/ci.yml:338-343: the download host served an error page,
  curl reported it as a successful transfer, and the only diagnostic anyone saw
  was gzip: stdin: not in gzip format / tar: Child returned status 1.
• jurplel/install-qt-action's moving major tag, fixed by pinning to
  @v4.3.1 — commit 948e26b0, "so a moving major tag cannot break Qt", now at
  11 sites across ci.yml, wasm-demo.yml and wasm-ladder.yml.

Five distinct environment failures, four distinct causes, inside roughly two days.
That is a rate, not a window. The wontfix branch is closed.

What has landed is the easy half, and the tree says so

#684 and #683 (refs #672) made failing downloads name themselves, at ten
sites plus the sccache step. The comment those landed with is unusually clear
about what it did not do, and it is the best statement of this ticket's
remaining scope — ci.yml:355-364:

This is not a retry policy and it does not make the outage less likely -- it
makes the log name the thing that happened, which is the half of morph#672 that
needs no CI-wide decision. The rest of that class -- a retry policy, or a marker
that distinguishes "the environment failed" from "the change failed" in the
check list, and the apt/PPA outages that are the other three quarters of the
ticket -- is still open.

I confirmed there is no retry anywhere in ci.yml: a grep for retry,
nick-fields/retry, --retry and a bounded for loop returns only those three
comment lines disclaiming one.

So the diagnosis cost the issue is actually about — "each occurrence cost a
diagnosis cycle on a PR that had nothing to do with it" — is undiminished. Naming
the failure helps whoever reads the log; it does not stop a PR author having to
read one.

Why this is valid and still must not be dispatched yet

The problem is real, measured, and rising. But the three options the issue lists
are not interchangeable and the choice between them is a CI-wide policy
decision — the tree's own comment calls it that:

1. Retry with backoff. Would have absorbed all five. The issue states the risk
   itself: "it hides a genuine and persistent packaging breakage behind three slow
   attempts." That is a real cost to accept deliberately, not a detail.
2. A distinguishable marker in the check list. Does not prevent anything; makes
   the environment failure visible without a log read. Lower risk, smaller payoff.
3. Cache more aggressively. Narrowest; the issue already notes the apt cache
   missed on bank/gui + ci: clear the eleven bank-GUI sources' clang-tidy debt, and gate the Q_OBJECT header split that no build catches early (fixes #656, fixes #659) #665 (Cache not found for input keys: apt-valgrind-...).

Per the batching rule in running-issue-lanes, a ticket needing a decision nobody
in the lane can make does not go in a batch — it blocks every other ticket sharing
its PR on a question that may take days. This ticket is therefore labelled
valid and deliberately left undispatched until an owner picks 1, 2 or 3 (they
are not exclusive; 1+2 is coherent). One sentence on this issue unblocks it.

Recommendation, for whoever makes that call

Take 1 and 2 together, bounded: two attempts with a short backoff on the
dependency-installation steps only, and on final failure emit a marker naming
package installation as the cause. Two attempts rather than three keeps the
"slow attempts hiding a real breakage" cost to one extra round trip, and the
marker means the case retry does not absorb still costs nobody a log read.
Explicitly not in scope: retrying build, test or upload steps, where a retry would
mask exactly the failures CI exists to catch.

What I did not verify

• I did not measure how often GitHub's own infrastructure already retries these
  steps, which the issue also lists as unchecked. It remains unchecked.
• I did not test that a retry would in fact have cleared any of the five; for
  occurrences 1 and 2 the issue records that a manual re-run did, which is
  suggestive, not equivalent.

What would change the verdict

• wontfix only if a month-long sample shows the two-day cluster was
  anomalous. The current sample says the opposite.
• Close as fixed when an environment failure on a PR is distinguishable from a
  change failure without opening a job log — and prove it per AGENTS.md by forcing
  the failure (point a step at a URL that 503s) and watching the marker appear.
  A retry that is never observed to fire is a control measuring nothing.

[Yaraslaut]

Closed — out of scope under the CI reduction

This issue proposes adding or tightening CI enforcement. morph#755 moved in the opposite direction deliberately: the check surface had reached 57 checks per PR across 51 jobs and 31 checker scripts, while more than half the open backlog was CI maintaining itself. The meta-gates — the checks policing other checks — were removed.

The finding itself is not disputed and the analysis stands; what has changed is the appetite for enforcing it in CI. Closed as not planned, not as invalid.

No compile, test, sanitizer, coverage or platform coverage was removed — every build, sanitizer, Valgrind, coverage and platform leg still runs. The trade is that pins, citations and conventions of this kind are now conventions rather than gates.

Re-open if the cost/benefit changes — the evidence recorded here would not need re-gathering.