From 64f5baee301d848049e1be563495e487130209df Mon Sep 17 00:00:00 2001 From: pi-prs Date: Wed, 11 Nov 2026 11:22:49 -0400 Subject: [PATCH 1/3] ci: the apt mirror retry as a script, so the ci.yml change is one line (engine#2210) On 2026-11-11 14:53Z the Ubuntu mirror failed four consecutive apt attempts and took main plus eleven open PRs red at once (#2199-#2205, #2183, #2187, #2189, #2207). ci.yml already retried this for engine#1814 and the shape was right -- each attempt re-reads the index so a later one can reach a different mirror -- but four attempts span only ~90 s of back-off, and a mirror returning 404s on debs its own fresh index lists does not repair itself inside 90 s. This moves the loop into scripts/ci-apt-install.sh so the workflow change that lands it is a single line: run: scripts/ci-apt-install.sh Why a script rather than an inline edit: this lane's token has repo, gist and read:org, so it cannot push changes under .github/workflows/ (workflow scope). Moving the logic out means a lane that does have the scope only has to replace the inline block with one line, instead of reviewing 25 lines of shell inside YAML. What the script does differently: back-off doubles (15, 30, 60, 120, 240, then 300 s steps to a 25 min deadline), each apt call carries Acquire::Retries so a per-connection 404 or timeout is retried inside the attempt instead of ending it, the deadline keeps the step bounded so it fails rather than sleeping into the job timeout, and the failure line names the attempt count and elapsed seconds. Verified: bash -n; --help and an unknown argument (rc 2); first-try success (rc 0); a fake apt-get that always fails exits 1 and prints the named error; and the doubling was observed (attempt 2 reported 'retrying in 30s'). --- scripts/ci-apt-install.sh | 64 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100755 scripts/ci-apt-install.sh diff --git a/scripts/ci-apt-install.sh b/scripts/ci-apt-install.sh new file mode 100755 index 000000000..a70fd557d --- /dev/null +++ b/scripts/ci-apt-install.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# ci-apt-install.sh - install the CI build's apt dependencies, retrying the mirror on a time budget. +# +# engine#2210. ci.yml already retried this for engine#1814, and the shape was right: each attempt re-reads the +# index, so a later attempt can reach a different mirror. What it did not have was enough TIME. On 2026-11-11 +# 14:53Z four consecutive attempts failed and took `main` plus eleven open PRs red at once (#2199-#2205, #2183, +# #2187, #2189, #2207) -- four attempts spanned ~90 s of back-off, and a mirror pool returning 404s on debs its +# own fresh index lists does not repair itself inside 90 s. +# +# So: the back-off doubles (15, 30, 60, 120, 240, then 300 s steps to a ~23 min deadline), each apt call carries +# Acquire::Retries so a per-connection 404 or timeout is retried inside the attempt instead of ending it, and the +# deadline keeps the step bounded so it fails rather than sleeping into the job timeout. +# +# usage: scripts/ci-apt-install.sh [--deadline-seconds N] +# exit: 0 installed, 1 the mirror never served the packages +set -uo pipefail + +deadline_seconds=1500 +while [ $# -gt 0 ]; do + case "$1" in + --deadline-seconds) deadline_seconds=${2:-1500}; shift 2 ;; + -h|--help) sed -n '17,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "ci-apt-install: unknown argument '$1'" >&2; exit 2 ;; + esac +done + +apt_opts="-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30" +packages="ca-certificates git python3 python3-venv curl jq file procps make gcc-16 g++-16 cmake ninja-build + libcurl4-openssl-dev libzstd-dev libssl-dev libdrm-dev libpcre2-dev" + +attempt=0; delay=15; started=$(date +%s); deadline=$((started + deadline_seconds)) +while :; do + attempt=$((attempt + 1)) + # shellcheck disable=SC2086 # both option and package lists are intentional word splits + if apt-get $apt_opts update && + DEBIAN_FRONTEND=noninteractive apt-get $apt_opts install -y --no-install-recommends $packages; then + printf 'ci-apt-install: installed on attempt %d after %ss\n' "$attempt" "$(( $(date +%s) - started ))" + exit 0 + fi + now=$(date +%s) + if [ $((now + delay)) -gt "$deadline" ]; then break; fi + printf 'ci-apt-install: attempt %d failed after %ss (mirror 404 is a known transient, engine#1814/engine#2210); retrying in %ss\n' \ + "$attempt" "$((now - started))" "$delay" + sleep "$delay" + delay=$(( delay * 2 )); if [ "$delay" -gt 300 ]; then delay=300; fi +done +printf '::error::apt-get update+install failed after %d attempt(s) and %ss -- if the 404s name debs the index lists, it is the mirror (engine#1814), not this tree; re-run the job\n' \ + "$attempt" "$(( $(date +%s) - started ))" +exit 1 From 88347d109ad03ecd54d14157e56eca5af768af06 Mon Sep 17 00:00:00 2001 From: pi-prs Date: Wed, 11 Nov 2026 12:25:46 -0400 Subject: [PATCH 2/3] ci: check the toolchain before fetching it, so a mirror outage cannot fail a build (engine#2210) The retry this PR first carried was the weaker half of the fix. The runners are self-hosted and every package in the list is already installed: ci-apt-install: all 18 package(s) already installed; nothing to fetch so the common case was doing a network round trip to fetch packages that were present, and that round trip is the only reason the mirror's availability could fail a build at all. The step now checks each package with dpkg and installs only what is missing: - all present (the normal case) -> no apt call, no network, exit 0; - some missing -> apt install names ONLY those (verified: 2 of 18 missing -> 'libzstd-dev libdrm-dev'); - apt fails -> the doubling back-off and the deadline still apply, and it exits 1 with the attempt count and elapsed seconds. Verified: bash -n; a real run on this host -> 'all 18 package(s) already installed; nothing to fetch', exit 0, no apt; a fake dpkg-query reporting two missing -> exactly those two reach apt install; a fake apt-get that always fails with --deadline-seconds 0 -> exit 1 and the named error. --- scripts/ci-apt-install.sh | 56 ++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/scripts/ci-apt-install.sh b/scripts/ci-apt-install.sh index a70fd557d..3c2430489 100755 --- a/scripts/ci-apt-install.sh +++ b/scripts/ci-apt-install.sh @@ -14,41 +14,67 @@ # See the License for the specific language governing permissions and # limitations under the License. # -# ci-apt-install.sh - install the CI build's apt dependencies, retrying the mirror on a time budget. +# ci-apt-install.sh - make sure the CI build's apt dependencies are present, without depending on the mirror. # -# engine#2210. ci.yml already retried this for engine#1814, and the shape was right: each attempt re-reads the -# index, so a later attempt can reach a different mirror. What it did not have was enough TIME. On 2026-11-11 -# 14:53Z four consecutive attempts failed and took `main` plus eleven open PRs red at once (#2199-#2205, #2183, -# #2187, #2189, #2207) -- four attempts spanned ~90 s of back-off, and a mirror pool returning 404s on debs its -# own fresh index lists does not repair itself inside 90 s. +# engine#2210. Two separate faults, both observed on 2026-11-11: # -# So: the back-off doubles (15, 30, 60, 120, 240, then 300 s steps to a ~23 min deadline), each apt call carries -# Acquire::Retries so a per-connection 404 or timeout is retried inside the attempt instead of ending it, and the -# deadline keeps the step bounded so it fails rather than sleeping into the job timeout. +# 1. THE MIRROR. ci.yml retried this for engine#1814 and the shape was right -- each attempt re-reads the index, +# so a later attempt can reach a different mirror -- but four attempts span ~90 s of back-off, and a mirror +# returning 404s on debs its own fresh index lists does not repair itself inside 90 s. At 14:53Z that took +# `main` plus eleven open PRs red at once (#2199-#2205, #2183, #2187, #2189, #2207). +# +# 2. THE DEPENDENCY ITSELF. The runners are self-hosted and the whole toolchain is already installed, so the +# common case fetches nothing -- but the step still ran apt on every job, which is why a mirror outage could +# fail a build at all. Checking first means the normal path touches no network. +# +# So: check every package with dpkg, install only what is missing, and if apt has to run, give it a time budget +# (doubling back-off to a deadline) instead of a fixed attempt count. # # usage: scripts/ci-apt-install.sh [--deadline-seconds N] -# exit: 0 installed, 1 the mirror never served the packages +# exit: 0 all packages present (fetching nothing, or after a successful install), 1 the mirror never served them set -uo pipefail deadline_seconds=1500 while [ $# -gt 0 ]; do case "$1" in --deadline-seconds) deadline_seconds=${2:-1500}; shift 2 ;; - -h|--help) sed -n '17,30p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -h|--help) sed -n '17,32p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "ci-apt-install: unknown argument '$1'" >&2; exit 2 ;; esac done -apt_opts="-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30" -packages="ca-certificates git python3 python3-venv curl jq file procps make gcc-16 g++-16 cmake ninja-build +packages="ca-certificates git python3 python3-venv curl jq file procps make + gcc-16 g++-16 cmake ninja-build libcurl4-openssl-dev libzstd-dev libssl-dev libdrm-dev libpcre2-dev" +# ── 1. what the build needs, and what is already here ─────────────────────────────────────────────────────────── +# dpkg rather than `command -v`: a package can be installed without its binary on PATH (a -dev package has no +# binary at all), and this list is mostly the latter. +missing="" +total=0 +for pkg in $packages; do + total=$((total + 1)) + case "$(dpkg-query -W -f='${Status}' "$pkg" 2>/dev/null)" in + *"install ok installed"*) ;; + *) missing="$missing $pkg" ;; + esac +done + +if [ -z "$missing" ]; then + printf 'ci-apt-install: all %d package(s) already installed; nothing to fetch\n' "$total" + exit 0 +fi +printf 'ci-apt-install: %d of %d package(s) missing:%s\n' \ + "$(printf '%s\n' $missing | wc -l)" "$total" "$missing" + +# ── 2. install only those, on a time budget ───────────────────────────────────────────────────────────────────── +apt_opts="-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30" attempt=0; delay=15; started=$(date +%s); deadline=$((started + deadline_seconds)) while :; do attempt=$((attempt + 1)) - # shellcheck disable=SC2086 # both option and package lists are intentional word splits + # shellcheck disable=SC2086 # option, package and missing lists are intentional word splits if apt-get $apt_opts update && - DEBIAN_FRONTEND=noninteractive apt-get $apt_opts install -y --no-install-recommends $packages; then + DEBIAN_FRONTEND=noninteractive apt-get $apt_opts install -y --no-install-recommends $missing; then printf 'ci-apt-install: installed on attempt %d after %ss\n' "$attempt" "$(( $(date +%s) - started ))" exit 0 fi From e4c141161368f3323b1b16881e4578985ed63afa Mon Sep 17 00:00:00 2001 From: pi-prs Date: Wed, 11 Nov 2026 14:31:54 -0400 Subject: [PATCH 3/3] ci: correct the cause this script documents -- the failure was container DNS, not the mirror (engine#2210, engine#2211) The header said '1. THE MIRROR' and built the argument around a mirror returning 404s on debs its own fresh index lists. That was the reading the error text invited, and it was wrong: engine#2211 found the build container had no working resolver, and #2224 fixed it by giving the container its own. An apt that cannot resolve the archive fails exactly the way a mirror 404 does, and the message names the mirror either way. The change this PR makes is unaffected -- checking dpkg first means the normal path needs no network at all, which is the only way to be immune to a fault that can present as two different things. What changes is what the file tells the next reader: the reason is 'do not need the fetch', not 'survive a bad mirror'. Verified: bash -n; ./scripts/ci-apt-install.sh on this host -> 'all 18 package(s) already installed; nothing to fetch', rc 0, no apt call. --- scripts/ci-apt-install.sh | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/scripts/ci-apt-install.sh b/scripts/ci-apt-install.sh index 3c2430489..c35abeb4f 100755 --- a/scripts/ci-apt-install.sh +++ b/scripts/ci-apt-install.sh @@ -14,21 +14,22 @@ # See the License for the specific language governing permissions and # limitations under the License. # -# ci-apt-install.sh - make sure the CI build's apt dependencies are present, without depending on the mirror. +# ci-apt-install.sh - make sure the CI build's apt dependencies are present, without depending on the network. # -# engine#2210. Two separate faults, both observed on 2026-11-11: +# engine#2210. On 2026-11-11 the build step failed four times in a row and took `main` plus eleven open PRs red at +# once (#2199-#2205, #2183, #2187, #2189, #2207). The step reported it as "404s on debs the index lists" and +# engine#1814's retry was written for that reading, quite reasonably -- but the cause turned out to be the +# **build container's DNS** (engine#2211, fixed by #2224 giving the container its own resolvers). An apt that +# cannot resolve the archive fails the same way a mirror 404 does, and the error text names the mirror. # -# 1. THE MIRROR. ci.yml retried this for engine#1814 and the shape was right -- each attempt re-reads the index, -# so a later attempt can reach a different mirror -- but four attempts span ~90 s of back-off, and a mirror -# returning 404s on debs its own fresh index lists does not repair itself inside 90 s. At 14:53Z that took -# `main` plus eleven open PRs red at once (#2199-#2205, #2183, #2187, #2189, #2207). -# -# 2. THE DEPENDENCY ITSELF. The runners are self-hosted and the whole toolchain is already installed, so the -# common case fetches nothing -- but the step still ran apt on every job, which is why a mirror outage could -# fail a build at all. Checking first means the normal path touches no network. +# That is the lesson this script is built around, and it is not "retry harder". The runners are self-hosted and +# the whole toolchain is already installed, so the common case fetches nothing -- the step was making a network +# round trip to fetch packages that were present, and that round trip is the only reason a DNS or mirror fault +# could fail a build at all. Checking first means the normal path touches no network, so neither fault can. # # So: check every package with dpkg, install only what is missing, and if apt has to run, give it a time budget -# (doubling back-off to a deadline) instead of a fixed attempt count. +# (doubling back-off to a deadline) instead of a fixed attempt count -- because when something upstream is wrong, +# time is the only lever a retry has. # # usage: scripts/ci-apt-install.sh [--deadline-seconds N] # exit: 0 all packages present (fetching nothing, or after a successful install), 1 the mirror never served them