Boot deadlocks forever in network stage: wait_for_network's 'systemctl start systemd-networkd-wait-online' never returns (26.1-0ubuntu3~26.04.1, OpenStack/netplan)

[AryanHamedani]

Bug report

After upgrading an Ubuntu 26.04 (resolute) OpenStack VM from cloud-init 26.1-0ubuntu2 to the SRU 26.1-0ubuntu3~26.04.1, the first reboot never completes: the network stage blocks forever inside wait_for_network() on systemctl start systemd-networkd-wait-online.service. cloud-init-network.service ships TimeoutSec=0, and ssh.socket/docker.service are ordered after cloud-init, so the host ends up answering ICMP with zero listeners — effectively bricked (only provider-console/disk surgery recovers it). The hang reproduced deterministically on every boot attempt (4/4), including after a full stop/start of the VM.

The same image with 26.1-0ubuntu2 (fresh VM, same cloud, same datasource) executes the identical call in 0.125 s — systemd starts networkd alongside wait-online and boot proceeds.

Environment

• Ubuntu 26.04 LTS (resolute) cloud image, OpenStack provider (DMI: OpenStack Foundation OpenStack Nova), ConfigDrive (config-2) present, netplan renderer, single NIC (ens3, DHCP4)
• Broken boot: cloud-init 26.1-0ubuntu3~26.04.1, systemd 259.5-0ubuntu3.4, kernel 7.0.0-30-generic (all three landed in one apt-get upgrade/dist-upgrade three days before the first reboot — so I cannot fully isolate cloud-init vs. systemd; the behavioral delta is in this code path)
• Working boot (for comparison): same image, cloud-init 26.1-0ubuntu2, systemd 259.5-0ubuntu3, kernel 7.0.0-15-generic

Evidence (from the dead host's disk, post-mortem)

/var/log/cloud-init.log — these are the final lines; nothing follows for 28+ minutes until the VM was power-cycled:

main.py[DEBUG]: Will wait for network connectivity before continuing
stages.py[DEBUG]: Using distro class <class 'cloudinit.distros.ubuntu.Distro'>
activators.py[DEBUG]: Using selected activator: <class 'cloudinit.net.activators.NetplanActivator'> from priority: ['netplan', 'eni', 'network-manager', 'networkd']
subp.py[DEBUG]: Running command ['systemctl', 'is-enabled', 'NetworkManager.service'] with allowed return codes [0] (shell=False, capture=True)
subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)

Journal of the same boot: after Starting cloud-init-network.service - Cloud-init: Network Stage... at uptime ~6.5 s, systemd logs zero further job transitions for the rest of the boot (26+ min observed). systemd-networkd.service is never started in the main system (it ran only in the initrd, which got a DHCP lease fine and handed off). No Ordering cycle/Deleting job messages. The console shows only:

[***   ] Job cloud-init-network.service/start running (9min 2s / no limit)

So the start job for systemd-networkd-wait-online.service (BindsTo=/After=systemd-networkd.service) is enqueued but never becomes runnable, networkd's pulled-in job never runs either, and since the caller is the unit the rest of boot is ordered behind (Before=sysinit.target), the wait can never be satisfied — a deadlock that systemd's transaction cycle-checker cannot see because the jobs arrive in a second transaction.

On the working 26.1-0ubuntu2 boot, the same sequence logs:

subp.py[DEBUG]: Running command ['systemctl', 'start', 'systemd-networkd-wait-online.service'] with allowed return codes [0] (shell=False, capture=True)
performance.py[DEBUG]: Running ['systemctl', 'start', 'systemd-networkd-wait-online.service'] took 0.125 seconds

with the journal showing Starting systemd-networkd.service... immediately after Starting cloud-init-network.service....

Impact

A routine SRU upgrade turned into an unbootable production host on its next (unrelated, provider-initiated) reboot. Because cloud-init-network.service has TimeoutSec=0, there is no bound at all: sshd never starts, and the machine cannot be recovered without out-of-band access.

Workarounds we validated

• Drop-in on cloud-init-network.service with TimeoutStartSec=300 — the stage's job fails at the bound and boot proceeds (sshd/docker come up; After= is satisfied by a finished job). This seems worth shipping as a belt-and-braces default given the blast radius of TimeoutSec=0.
• Booting with cloud-init=disabled also recovers the host.

Suggested angle

wait_for_network() issuing a blocking systemctl start from inside a unit that gates sysinit.target means any condition that keeps the started unit's job queued (rather than failing fast) converts to an infinite boot hang. Either the call needs its own timeout, or cloud-init-network.service should not ship TimeoutSec=0, or the wait should verify systemd-networkd.service is active/activatable before blocking on wait-online.

I no longer have the dead disk attached (provider reclaimed it), but I extracted full journals and logs for the broken and working boots before it went and can share more excerpts on request.

[AryanHamedani]

Update with a significant datapoint: this is not specific to the 26.1-0ubuntu3~26.04.1 SRU.

I ran a controlled reboot test on the replacement VM — same environment (OpenStack/ConfigDrive/netplan, single DHCP NIC), stock cloud-init 26.1-0ubuntu2 and systemd 259.5-0ubuntu3 from the released 26.04 cloud image, never upgraded — and the identical deadlock occurred on its second boot:

cloud-init-network.service: failed (Result: timeout)   # our local TimeoutStartSec=300 drop-in
performance.py[DEBUG]: Running ['systemctl', 'start', 'systemd-networkd-wait-online.service'] took 301.228 seconds

(The only reason this VM survived is the local TimeoutStartSec=300 drop-in described above — without it this would have been a second bricked host.)

So the distinguishing variable is not the cloud-init version but whether systemd-networkd.service happens to be active when the network stage runs. On this image networkd's only install-time pull-in is multi-user.target, which is ordered after cloud-init — so on a boot where nothing else has started networkd early, wait_for_network()'s blocking systemctl start systemd-networkd-wait-online.service enqueues jobs that never become runnable and boot hangs forever. First boots appeared to work only because something in the first-boot path (netplan generate/apply ordering) had already started networkd.

Adding Wants=systemd-networkd.service + After=systemd-networkd.service to cloud-init-network.service (we did it via a drop-in) makes the wait return in <1 s and boots complete cleanly (35 s vs 367 s degraded / infinite unguarded) — which suggests the packaged unit may simply be missing that pull-in for the networkd/netplan case, in addition to the TimeoutSec=0 blast-radius concern.

[AryanHamedani]

Proposed root-cause fix: #7053.

Since my last comment I completed the forensics on the affected host: the missing link in the chain is network-pre.target. A third-party VPN helper unit on the box declares Wants=network-pre.target + Before=network-pre.target while keeping default dependencies (so it is also After=sysinit.target). That transitively orders systemd-networkd.service (After=network-pre.target) behind sysinit.target — i.e. behind cloud-init-network.service itself — so the networkd start job pulled in by the runtime systemctl start systemd-networkd-wait-online.service can never run while the network stage is blocked on it. The PR includes a minimal QEMU reproducer (stock 26.1 image + a 10-line trap unit) that hangs unpatched and boots clean with the fix.