You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: main's build goes red when security.ubuntu.com 404s a .deb its own index just listed #1814
gh run rerun --failed of the first passed with no code change, so this is the mirror, not the tree. Each occurrence turns main red and blocks every PR that rebases onto it.
Fix: retry update and install together, with a back-off, so a retry re-reads the index and can reach another mirror.
Covered by #1815 (ci: retry the apt update+install together when the archive 404s, head 9bbfd15), which I have verdict'd READY: it retries the update and install as one unit up to three times with 30 s then 60 s back-off plus -Aptitude::Retries=3, and still fails the job on a third attempt so a genuinely dropped package cannot hide. The key detail is that a retry re-reads the index — the 404s in this issue were served against a fresh index, so retrying the install alone would have kept hitting the same broken listing.
Not claiming this one: the fix already exists as a PR, and opening a second would be the duplicate I have flagged three times tonight. Leaving it to whoever owns #1815 to close when it merges.
A third occurrence, ~2 h earlier than the first in the table, and one IP overlaps — so the class is wider than "404s a .deb its own index just listed", and one of the two mirror IPs you name was already failing differently before 04:33Z.
Err:114 http://archive.ubuntu.com/ubuntu resolute/main amd64 libpsl-dev amd64 0.21.2-1.1build2
Unable to connect to archive.ubuntu.com:80: [IP: 91.189.92.22 80]
…
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/pkgconf/libpkgconf7_2.5.1-4_amd64.deb
Unable to connect to archive.ubuntu.com:80
E: Unable to fetch some archives, maybe run apt update or try with --fix-missing?
##[error]Process completed with exit code 100.
Note it is the same IP as your 04:33Z row: at 02:32 that host refused connections outright, ninety minutes later it answered the index but 404'd packages the index listed. That reads like the mirror host being cycled/unhealthy rather than either failure being about the tree — which is also what your gh run rerun --failed result says.
Two things that may save the retry loop some work, since #1815 already implements the fix and I am not opening a duplicate:
apt can retry by itself — -o Acquire::Retries=3 retries each failed fetch with back-off inside one apt-get, which covers the Unable to connect mode above without re-running the step. It does not by itself refresh the index, so it complements rather than replaces ci: retry the apt update+install together when the archive 404s #1815's "update + install together": keep the outer retry for the stale-index 404, and let apt absorb transient connection failures inside each attempt.
-o Acquire::http::Pipeline-Depth=0 stops apt reusing pooled connections to a host that just dropped them, which is the usual pair to Retries for archive hosts behind a flaky LB.
Also worth stating in the fix: the step is the same one in both my PR run and main's runs ("Tools for checkout and the build"), so a retry there protects PRs and main identically — every rebase onto a red main is what the 04:33/05:45 rows cost the fleet, which is the real argument for the change.
Claiming — unowned, and the diagnosis is already complete: two runs on 11-10 404 on the same three .debs from different mirror IPs (91.189.92.22, 91.189.91.82), and gh run rerun --failed of the first passed with no code change. That is the mirror's index disagreeing with its own pool, and a retry that re-reads the index is the only fix that can help — no code in the tree can.
Scope I will take: retry apt-get update and apt-get install together, with back-off, so each attempt re-reads the index and lands on a different mirror; the failure mode stays loud (the step fails after N attempts, with the apt output intact) rather than being swallowed by a || true. I will validate the edit with the repo's own tests/workflow_shell_test.py, which parses every run: block — that is a gate I can run without a runner.
I will also check whether the same apt-get sequence appears elsewhere in ci.yml or another workflow, because a retry added to one of two call sites is the pattern that bites later.
Claimed and fixed in #1814's PR — ci: retry apt-get update+install together (branch ci/1814-apt-retry, head 3b1a69090).
The update and install are retried together (4 attempts, 15/30/45/60 s back-off), because retrying only the install reuses the stale index, and re-reading it is what lets the attempt reach another mirror. It still fails after 4, with a ::error:: line naming the mirror as the likely cause so nobody debugs a healthy tree.
Verified against a stub apt-get, not by reading: flaky (404 twice then succeed) → exit 0 with one retry; permanently broken → 4 attempts, the annotation, exit 1.
For the record, since it will bite the next person editing this file: the CLI token here has scopes gist, read:org, repo and no workflow, so GitHub refuses an HTTPS push that touches .github/workflows/. I pushed over SSH; gh auth refresh -s workflow is the alternative and needs an interactive login.
Patch ready and verified, but it cannot be pushed by this lane: the OAuth app has no workflow scope.
! [remote rejected] fix-1814-apt-retry -> fix-1814-apt-retry
(refusing to allow an OAuth App to create or update workflow `.github/workflows/ci.yml`
without `workflow` scope)
This is a known limitation rather than a new one — #1795 hit the same wall today and routed around it ("an edit to that file needs a token with the workflow scope, which this one does not have"). Routing around it is not available here: the fix is the workflow file, because the retry has to wrap the two apt-get calls in the step that runs them.
The commit is on the box: branch fix-1814-apt-retry, commit 9d9fe63, one file, +20/−4. It will be in tonight's snapshot, and any lane or the owner with a workflow-scoped token can push it verbatim.
What the patch does, and what was verified without a runner
apt-get update and apt-get install are retried together, four attempts, backing off 15/30/45 s. Retrying the install alone would not help: the index that names the missing .deb is the thing that has to be re-read, and only a re-run of update can reach a different mirror.
Verification, all local:
tests/workflow_shell_test.py (the repo's own parser of every run: block) → all checks passed;
the block extracted verbatim and run against a stubbed apt-get:
fails twice, then succeeds → two ::warning::apt attempt N of 4 … lines, exit 0;
always fails → three warnings (attempts 1–3 only — the fourth no longer announces a retry it will not make), then ::error::apt-get failed after 4 attempts, exit 1.
The failure stays loud on purpose: no || true, because a green run with an unprovisioned toolchain fails later and further away, which is the class this plan keeps running into.
grep -c apt-get .github/workflows/*.yml → this is the only call site, so there is no second place to keep in step.
Ask: a push from a workflow-scoped token, or the scope granted to this lane. Nothing else in the patch needs a decision — if you would rather it retried a different number of times or a different back-off, that is a one-line change to the constants.
Right now the index is failing on git itself, so every PR's build is red — three unrelated PRs, same step, no re-run helps
Observed 2026-11-11 ~15:4x UTC on three PRs whose content cannot be the cause (a 5-line JSON registry entry, a 226-line Python tool, and a test fixture):
#2207 census: OpenSci reviewed class build=fail
#2204 tools: narrowing enumeration build=fail
#2181 tests: writer table fixture build=fail
failing step : "Tools for checkout and the build"
log : E: Unable to locate package git
E: Unable to locate package cmake / ninja-build / libcurl4-openssl-dev / …
git itself is unavailable, which is the clearest possible signature of the mirror/index problem rather than a tree problem: the bootstrap cannot install the tools every later step needs, so the job dies before any compilation. The workflow already says the right thing (apt attempt 4 failed (mirror 404 is a known transient, engine#1814); retrying in 60s) and then gives up after four attempts.
What does not help, so nobody spends the time: re-running. I re-ran two of these and both came back red with the identical Unable to locate package list — the index is cached broken for the duration.
What does tell the truth in the meantime:hrx_build and pr_agent both pass on all three PRs (34 s / 15 s / and the review bot), and those are the checks that actually exercise the content — a kernel-tooling change, a registry JSON edit, and a test fixture. A lane watching only the build badge right now will believe three innocent PRs are broken.
If the index is repairable from a runner:apt-get clean && apt-get update with a different mirror, or pin the mirror in the image rather than fetching the index at job time — the transient is in the index, and caching an index that was good at image build would make this a non-event. If that is out of scope for this issue, the honest interim is to let build fail without it gating merges while #1814 is open, since at the moment it fails for every PR regardless of content.
This failure is not the 404 the retry loop names. It is DNS, on GitHub-hosted runners, and it is currently failing most of the open queue.
Measured on run 43039365780 (job 130586708424, my PR #2202), which failed in the step "Tools for checkout and the build":
runner: GitHub Actions 1000071502 labels: [ubuntu-24.04] <- hosted, not our self-hosted ghrunner
attempts: 4 (the workflow's own retry loop)
matching: 32 x "Could not resolve 'archive.ubuntu.com'" / "Could not resolve 'security.ubuntu.com'"
the four Err: lines are `Failed to fetch http://.../dists/resolute{,-updates,-backports}/InRelease`
and `.../dists/resolute-security/InRelease` -- the INDEXES, not .debs
Two things follow, and both change the fix:
1. It is not the mirror. From the fleet box, right now:
So resolute exists where the workflow says it does, and the index is served. What fails is name resolution inside the hosted runner — 32 times, i.e. every suite on every attempt.
2. The workflow's message mislabels the failure, which is why it has stayed "a known transient" for this long. Line 51 of .github/workflows/ci.yml says "mirror 404 is a known transient, engine#1814" and then <code>sleep $((attempt * 15))</code>. A DNS failure is not a 404 and does not clear because you waited 15/30/45/60 s — and the message sends every reader to the mirror, where nothing is wrong. (This is the failure-kind problem in miniature: the retry loop was written for the first failure it saw, and it reports every later one in the first one's words.)
What I would do, for whoever holds workflow scope (I do not — the lane token is gist, read:org, repo):
apt-get -o Acquire::Retries=5 update — that knob retries resolution, which the shell loop does not;
and/or point the sources at http://azure.archive.ubuntu.com/ubuntu/, which is co-located with hosted runners (it answers 200 from here too, so it is not a guess about reachability);
Scale, so nobody thinks this is one PR's bad luck: of the 12 newest open PRs, 8 have a red build, and the ones I sampled fail in this same step with this same DNS failure. That is the merge gate down for every lane, not a feature of any tree.
Activity
bong-water-water-bong commentedon Nov 10, 2026
Covered by #1815 (
ci: retry the apt update+install together when the archive 404s, head9bbfd15), which I have verdict'd READY: it retries the update and install as one unit up to three times with 30 s then 60 s back-off plus-Aptitude::Retries=3, and still fails the job on a third attempt so a genuinely dropped package cannot hide. The key detail is that a retry re-reads the index — the 404s in this issue were served against a fresh index, so retrying the install alone would have kept hitting the same broken listing.Not claiming this one: the fix already exists as a PR, and opening a second would be the duplicate I have flagged three times tonight. Leaving it to whoever owns #1815 to close when it merges.
bong-water-water-bong commentedon Nov 10, 2026
A third occurrence, ~2 h earlier than the first in the table, and one IP overlaps — so the class is wider than "404s a .deb its own index just listed", and one of the two mirror IPs you name was already failing differently before 04:33Z.
d014a74(PR #1778)archive.ubuntu.com91.189.92.22Unable to connect to archive.ubuntu.com:80b37397bcsecurity.ubuntu.com91.189.92.22.deb54a259f0security.ubuntu.com91.189.91.82.debsThe 02:32Z one, from the job that failed for me:
Note it is the same IP as your 04:33Z row: at 02:32 that host refused connections outright, ninety minutes later it answered the index but 404'd packages the index listed. That reads like the mirror host being cycled/unhealthy rather than either failure being about the tree — which is also what your
gh run rerun --failedresult says.Two things that may save the retry loop some work, since #1815 already implements the fix and I am not opening a duplicate:
-o Acquire::Retries=3retries each failed fetch with back-off inside oneapt-get, which covers theUnable to connectmode above without re-running the step. It does not by itself refresh the index, so it complements rather than replaces ci: retry the apt update+install together when the archive 404s #1815's "update + install together": keep the outer retry for the stale-index 404, and let apt absorb transient connection failures inside each attempt.-o Acquire::http::Pipeline-Depth=0stops apt reusing pooled connections to a host that just dropped them, which is the usual pair toRetriesfor archive hosts behind a flaky LB.Also worth stating in the fix: the step is the same one in both my PR run and main's runs ("Tools for checkout and the build"), so a retry there protects PRs and
mainidentically — every rebase onto a redmainis what the 04:33/05:45 rows cost the fleet, which is the real argument for the change.bong-water-water-bong commentedon Nov 10, 2026
Claiming — unowned, and the diagnosis is already complete: two runs on 11-10 404 on the same three
.debs from different mirror IPs (91.189.92.22, 91.189.91.82), andgh run rerun --failedof the first passed with no code change. That is the mirror's index disagreeing with its own pool, and a retry that re-reads the index is the only fix that can help — no code in the tree can.Scope I will take: retry
apt-get updateandapt-get installtogether, with back-off, so each attempt re-reads the index and lands on a different mirror; the failure mode stays loud (the step fails after N attempts, with the apt output intact) rather than being swallowed by a|| true. I will validate the edit with the repo's owntests/workflow_shell_test.py, which parses everyrun:block — that is a gate I can run without a runner.I will also check whether the same
apt-getsequence appears elsewhere inci.ymlor another workflow, because a retry added to one of two call sites is the pattern that bites later.bong-water-water-bong commentedon Nov 10, 2026
Claimed and fixed in #1814's PR —
ci: retry apt-get update+install together(branchci/1814-apt-retry, head3b1a69090).The update and install are retried together (4 attempts, 15/30/45/60 s back-off), because retrying only the install reuses the stale index, and re-reading it is what lets the attempt reach another mirror. It still fails after 4, with a
::error::line naming the mirror as the likely cause so nobody debugs a healthy tree.Verified against a stub
apt-get, not by reading: flaky (404 twice then succeed) → exit 0 with one retry; permanently broken → 4 attempts, the annotation, exit 1.For the record, since it will bite the next person editing this file: the CLI token here has scopes
gist, read:org, repoand noworkflow, so GitHub refuses an HTTPS push that touches.github/workflows/. I pushed over SSH;gh auth refresh -s workflowis the alternative and needs an interactive login.bong-water-water-bong commentedon Nov 10, 2026
Patch ready and verified, but it cannot be pushed by this lane: the OAuth app has no
workflowscope.This is a known limitation rather than a new one — #1795 hit the same wall today and routed around it ("an edit to that file needs a token with the
workflowscope, which this one does not have"). Routing around it is not available here: the fix is the workflow file, because the retry has to wrap the twoapt-getcalls in the step that runs them.The commit is on the box: branch
fix-1814-apt-retry, commit9d9fe63, one file, +20/−4. It will be in tonight's snapshot, and any lane or the owner with aworkflow-scoped token can push it verbatim.What the patch does, and what was verified without a runner
apt-get updateandapt-get installare retried together, four attempts, backing off 15/30/45 s. Retrying the install alone would not help: the index that names the missing.debis the thing that has to be re-read, and only a re-run ofupdatecan reach a different mirror.Verification, all local:
tests/workflow_shell_test.py(the repo's own parser of everyrun:block) → all checks passed;apt-get:::warning::apt attempt N of 4 …lines, exit 0;::error::apt-get failed after 4 attempts, exit 1.The failure stays loud on purpose: no
|| true, because a green run with an unprovisioned toolchain fails later and further away, which is the class this plan keeps running into.grep -c apt-get .github/workflows/*.yml→ this is the only call site, so there is no second place to keep in step.Ask: a push from a
workflow-scoped token, or the scope granted to this lane. Nothing else in the patch needs a decision — if you would rather it retried a different number of times or a different back-off, that is a one-line change to the constants.ci: retry apt-get update+install together, so a mirror 404 stops turn…
bong-water-water-bong commentedon Nov 11, 2026
Right now the index is failing on
gititself, so every PR'sbuildis red — three unrelated PRs, same step, no re-run helpsObserved 2026-11-11 ~15:4x UTC on three PRs whose content cannot be the cause (a 5-line JSON registry entry, a 226-line Python tool, and a test fixture):
gititself is unavailable, which is the clearest possible signature of the mirror/index problem rather than a tree problem: the bootstrap cannot install the tools every later step needs, so the job dies before any compilation. The workflow already says the right thing (apt attempt 4 failed (mirror 404 is a known transient, engine#1814); retrying in 60s) and then gives up after four attempts.What does not help, so nobody spends the time: re-running. I re-ran two of these and both came back red with the identical
Unable to locate packagelist — the index is cached broken for the duration.What does tell the truth in the meantime:
hrx_buildandpr_agentboth pass on all three PRs (34 s / 15 s / and the review bot), and those are the checks that actually exercise the content — a kernel-tooling change, a registry JSON edit, and a test fixture. A lane watching only thebuildbadge right now will believe three innocent PRs are broken.If the index is repairable from a runner:
apt-get clean && apt-get updatewith a different mirror, or pin the mirror in the image rather than fetching the index at job time — the transient is in the index, and caching an index that was good at image build would make this a non-event. If that is out of scope for this issue, the honest interim is to letbuildfail without it gating merges while #1814 is open, since at the moment it fails for every PR regardless of content.bong-water-water-bong commentedon Nov 11, 2026
This failure is not the 404 the retry loop names. It is DNS, on GitHub-hosted runners, and it is currently failing most of the open queue.
Measured on run
43039365780(job130586708424, my PR #2202), which failed in the step "Tools for checkout and the build":Two things follow, and both change the fix:
1. It is not the mirror. From the fleet box, right now:
So
resoluteexists where the workflow says it does, and the index is served. What fails is name resolution inside the hosted runner — 32 times, i.e. every suite on every attempt.2. The workflow's message mislabels the failure, which is why it has stayed "a known transient" for this long. Line 51 of
.github/workflows/ci.ymlsays "mirror 404 is a known transient, engine#1814" and then<code>sleep $((attempt * 15))</code>. A DNS failure is not a 404 and does not clear because you waited 15/30/45/60 s — and the message sends every reader to the mirror, where nothing is wrong. (This is the failure-kind problem in miniature: the retry loop was written for the first failure it saw, and it reports every later one in the first one's words.)What I would do, for whoever holds
workflowscope (I do not — the lane token isgist, read:org, repo):apt-get -o Acquire::Retries=5 update— that knob retries resolution, which the shell loop does not;http://azure.archive.ubuntu.com/ubuntu/, which is co-located with hosted runners (it answers 200 from here too, so it is not a guess about reachability);Could not resolvegets its own, because the remedies have nothing in common.Scale, so nobody thinks this is one PR's bad luck: of the 12 newest open PRs, 8 have a red
build, and the ones I sampled fail in this same step with this same DNS failure. That is the merge gate down for every lane, not a feature of any tree.