You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: every build since 14:15Z fails because the job container cannot resolve archive.ubuntu.com (DNS, not the #1814 404) #2211
Since 14:15Z on 2026-11-11, every ci → build job fails in the first step, "Tools for checkout and the build". This covers main and every PR head, re-runs included. The step's message blames the mirror (engine#1814), but this time the cause is different: the job container cannot resolve the Ubuntu archive hosts at all. No mirror ever answers, so re-running does not help.
Err:1 http://archive.ubuntu.com/ubuntu resolute InRelease
Could not resolve 'archive.ubuntu.com'
Err:4 http://security.ubuntu.com/ubuntu resolute-security InRelease
Could not resolve 'security.ubuntu.com'
...
##[error]apt-get update+install failed 4 times -- if the 404s name .debs the index lists, it is the mirror (engine#1814), not this tree; re-run the job
The last green ci run was main at 13:53Z..github/workflows/ci.yml has not changed since 31f4fd4 (2026-11-10 07:03Z), so this is not a tree change.
The hosts are up. From strixhalo at the same time, getent hosts archive.ubuntu.com resolves, and InRelease returns HTTP/1.1 200 OK.
Where it runs: the runner is GitHub-hosted (Hosted Compute Agent, Azure centralus), runs-on: ubuntu-24.04 with container: ubuntu:26.04, options: --init. Name resolution fails inside the job container's network, not upstream.
Proposed fix (a workflow edit; this lane's token cannot push .github/workflows/*)
Give the container resolvers of its own, so it does not depend on the Docker network's embedded DNS:
Related: #2212 moves the apt retry into scripts/ci-apt-install.sh with a 25-minute deadline. The outage here has lasted more than 70 minutes and is DNS, so the --dns container option is still needed. Details: #2212 (comment)
Not the image. The last green build (13:5xZ, westus2) and a 15:4xZ red one pulled the same ubuntu:26.04 digest sha256:6a06111c3b0a…. Both used runner provisioner 20261028.624 and runner image ubuntu-24.04 20261101.360.1.
Not one region. The red jobs from 14:54Z to 15:45Z ran in centralus, eastus, eastus2, westus, westus2, westus3 and southcentralus. westus2 is where the last green run was.
Nothing in the tree, image or runner version differs between green and red, so the failure is in the hosted container network's name resolution. Two ways out: a ci.yml edit (--dns in container options:, which needs a workflow-scoped token), or waiting for GitHub. Re-running a PR does not help.
so the network has working DNS for these names — and note both answers are IPv6 only.
2. No commit changed .github/workflows/ci.yml near 14:15Z. Its last change is 31f4fd43, 2026-11-10 07:03Z (the apt retry). So the failure is not a configuration change that landed with it.
Together those say the break is on the runner side, inside the job container, and the two most likely mechanisms are distinguishable in one command on a runner:
DNS inside the container: docker run --rm ubuntu:26.04 getent hosts archive.ubuntu.com — if that fails while the host resolves, it is the container's resolver (a container: job uses the runner's Docker DNS, and options: --init does not touch it);
IPv6-only answers: if the resolver is fine but apt cannot reach the addresses, the failure would surface as connect, not resolve — the logs say Could not resolve, which points at the first.
What I cannot do from here: the fix is in .github/workflows/ci.yml (or the runner's Docker config), and neither is reachable from a lane token — X-Oauth-Scopes: gist, read:org, repo has no workflow scope (that is why #2076's script is still waiting for its workflow step). So this needs the workflow owner or cl-infra, and the useful thing I can add is the bound above plus the one-command discriminator.
And a mitigation that does not need workflow scope, if the resolver turns out to be the cause: apt-get -o Acquire::ForceIPv4=true would not help a resolve failure, so it is worth establishing which of the two it is before anyone patches the step again — #1814 and #1815 both changed this same step for a mirror problem, and this failure is a different one.
Activity
bong-water-water-bong commentedon Nov 11, 2026
Related: #2212 moves the apt retry into
scripts/ci-apt-install.shwith a 25-minute deadline. The outage here has lasted more than 70 minutes and is DNS, so the--dnscontainer option is still needed. Details: #2212 (comment)🤖 Generated with Claude Code
bong-water-water-bong commentedon Nov 11, 2026
Two causes ruled out (15:50Z, still red):
ubuntu:26.04digestsha256:6a06111c3b0a…. Both used runner provisioner 20261028.624 and runner image ubuntu-24.04 20261101.360.1.Nothing in the tree, image or runner version differs between green and red, so the failure is in the hosted container network's name resolution. Two ways out: a ci.yml edit (
--dnsin containeroptions:, which needs a workflow-scoped token), or waiting for GitHub. Re-running a PR does not help.🤖 Generated with Claude Code
bong-water-water-bong commentedon Nov 11, 2026
Two facts that bound this from outside the container
1. The host resolves both archive hosts. From strixhalo:
so the network has working DNS for these names — and note both answers are IPv6 only.
2. No commit changed
.github/workflows/ci.ymlnear 14:15Z. Its last change is31f4fd43, 2026-11-10 07:03Z (the apt retry). So the failure is not a configuration change that landed with it.Together those say the break is on the runner side, inside the job container, and the two most likely mechanisms are distinguishable in one command on a runner:
docker run --rm ubuntu:26.04 getent hosts archive.ubuntu.com— if that fails while the host resolves, it is the container's resolver (acontainer:job uses the runner's Docker DNS, andoptions: --initdoes not touch it);aptcannot reach the addresses, the failure would surface as connect, not resolve — the logs sayCould not resolve, which points at the first.What I cannot do from here: the fix is in
.github/workflows/ci.yml(or the runner's Docker config), and neither is reachable from a lane token —X-Oauth-Scopes: gist, read:org, repohas noworkflowscope (that is why #2076's script is still waiting for its workflow step). So this needs the workflow owner orcl-infra, and the useful thing I can add is the bound above plus the one-command discriminator.And a mitigation that does not need workflow scope, if the resolver turns out to be the cause:
apt-get -o Acquire::ForceIPv4=truewould not help a resolve failure, so it is worth establishing which of the two it is before anyone patches the step again — #1814 and #1815 both changed this same step for a mirror problem, and this failure is a different one.