Skip to content

ci: every build since 14:15Z fails because the job container cannot resolve archive.ubuntu.com (DNS, not the #1814 404) #2211

Closed

Description

@bong-water-water-bong

Since 14:15Z on 2026-11-11, every ci → build job fails in the first step, "Tools for checkout and the build". This covers main and every PR head, re-runs included. The step's message blames the mirror (engine#1814), but this time the cause is different: the job container cannot resolve the Ubuntu archive hosts at all. No mirror ever answers, so re-running does not help.

Evidence

  • 12 of 12 build jobs between 14:15Z and 15:05Z failed. Each log has 32 Could not resolve lines and no Failed to fetch … 404. Example (main, and census: an answered FastFlowLM gap stays answered, and the d3 pin watches its paths (Fixes #2171) #2189's re-run at 14:40Z):
    Err:1 http://archive.ubuntu.com/ubuntu resolute InRelease
      Could not resolve 'archive.ubuntu.com'
    Err:4 http://security.ubuntu.com/ubuntu resolute-security InRelease
      Could not resolve 'security.ubuntu.com'
    ...
    ##[error]apt-get update+install failed 4 times -- if the 404s name .debs the index lists, it is the mirror (engine#1814), not this tree; re-run the job
    
  • The last green ci run was main at 13:53Z. .github/workflows/ci.yml has not changed since 31f4fd4 (2026-11-10 07:03Z), so this is not a tree change.
  • The hosts are up. From strixhalo at the same time, getent hosts archive.ubuntu.com resolves, and InRelease returns HTTP/1.1 200 OK.
  • Where it runs: the runner is GitHub-hosted (Hosted Compute Agent, Azure centralus), runs-on: ubuntu-24.04 with container: ubuntu:26.04, options: --init. Name resolution fails inside the job container's network, not upstream.

Proposed fix (a workflow edit; this lane's token cannot push .github/workflows/*)

  1. Give the container resolvers of its own, so it does not depend on the Docker network's embedded DNS:
    options: --init --dns 1.1.1.1 --dns 8.8.8.8
  2. Fix the retry message: when the log has Could not resolve rather than a 404, it should say so, instead of pointing at ci: main's build goes red when security.ubuntu.com 404s a .deb its own index just listed #1814 and telling people to re-run.

Until then, re-running a red PR's build does not help: #2189's re-run at 14:40Z failed the same way.

🤖 Generated with Claude Code

Activity

  1. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Related: #2212 moves the apt retry into scripts/ci-apt-install.sh with a 25-minute deadline. The outage here has lasted more than 70 minutes and is DNS, so the --dns container option is still needed. Details: #2212 (comment)

    🤖 Generated with Claude Code

  2. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two causes ruled out (15:50Z, still red):

    • Not the image. The last green build (13:5xZ, westus2) and a 15:4xZ red one pulled the same ubuntu:26.04 digest sha256:6a06111c3b0a…. Both used runner provisioner 20261028.624 and runner image ubuntu-24.04 20261101.360.1.
    • Not one region. The red jobs from 14:54Z to 15:45Z ran in centralus, eastus, eastus2, westus, westus2, westus3 and southcentralus. westus2 is where the last green run was.

    Nothing in the tree, image or runner version differs between green and red, so the failure is in the hosted container network's name resolution. Two ways out: a ci.yml edit (--dns in container options:, which needs a workflow-scoped token), or waiting for GitHub. Re-running a PR does not help.

    🤖 Generated with Claude Code

  3. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Two facts that bound this from outside the container

    1. The host resolves both archive hosts. From strixhalo:

    $ getent hosts archive.ubuntu.com   → 2a06:bc80:10:1000::3fd, 2620:2d:4000:1::101
    $ getent hosts security.ubuntu.com  → 2a06:bc80:0:1000::16, 2a06:bc80:0:1000::216
    

    so the network has working DNS for these names — and note both answers are IPv6 only.

    2. No commit changed .github/workflows/ci.yml near 14:15Z. Its last change is 31f4fd43, 2026-11-10 07:03Z (the apt retry). So the failure is not a configuration change that landed with it.

    Together those say the break is on the runner side, inside the job container, and the two most likely mechanisms are distinguishable in one command on a runner:

    • DNS inside the container: docker run --rm ubuntu:26.04 getent hosts archive.ubuntu.com — if that fails while the host resolves, it is the container's resolver (a container: job uses the runner's Docker DNS, and options: --init does not touch it);
    • IPv6-only answers: if the resolver is fine but apt cannot reach the addresses, the failure would surface as connect, not resolve — the logs say Could not resolve, which points at the first.

    What I cannot do from here: the fix is in .github/workflows/ci.yml (or the runner's Docker config), and neither is reachable from a lane token — X-Oauth-Scopes: gist, read:org, repo has no workflow scope (that is why #2076's script is still waiting for its workflow step). So this needs the workflow owner or cl-infra, and the useful thing I can add is the bound above plus the one-command discriminator.

    And a mitigation that does not need workflow scope, if the resolver turns out to be the cause: apt-get -o Acquire::ForceIPv4=true would not help a resolve failure, so it is worth establishing which of the two it is before anyone patches the step again — #1814 and #1815 both changed this same step for a mirror problem, and this failure is a different one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions