fix(net): avoid boot deadlock when starting systemd-networkd-wait-online - #7053
Open
AryanHamedani wants to merge 1 commit into
Open
fix(net): avoid boot deadlock when starting systemd-networkd-wait-online#7053AryanHamedani wants to merge 1 commit into
AryanHamedani wants to merge 1 commit into
Conversation
NetworkdActivator.wait_for_network() runs a plain blocking `systemctl start systemd-networkd-wait-online.service`. On Ubuntu this is issued at runtime from cloud-init's network stage (see 5772), which is ordered Before=sysinit.target. systemd-networkd-wait-online.service is BindsTo/After systemd-networkd.service, and systemd-networkd.service is After=network-pre.target. Any unit ordered Before=network-pre.target that keeps default dependencies (a common pattern for third-party firewall/VPN helpers) is also After=sysinit.target, so the networkd start job pulled in by the wait is transitively ordered after the very cloud-init unit issuing the start. The job never becomes runnable, and because it belongs to a separate transaction systemd cannot detect the cycle. With cloud-init-network.service shipping TimeoutSec=0, boot hangs forever with no sshd and no console login - an unbootable host. Start systemd-networkd explicitly and pass --job-mode=ignore-dependencies for both jobs. The skipped ordering constraints are unsatisfiable by construction while cloud-init's network stage is running, and this mirrors the fix applied to the analogous sshd restart deadlock in cc_set_passwords (5935). Fixes canonicalGH-7052
AryanHamedani
force-pushed
the
fix/networkd-wait-online-boot-deadlock
branch
from
August 29, 2026 10:05
03d53e5 to
5374d93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Commit Message
Additional Context
This is the root-cause fix for #7052 (production host bricked by this deadlock; full journal forensics in the issue). The precise deadlock chain, established from the journals of an affected host (Ubuntu 26.04, cloud-init 26.1, systemd 259, OpenStack/ConfigDrive, netplan renderer):
cloud-init-network.service(Before=sysinit.target,TimeoutSec=0) reachesDistro.wait_for_network()→ blockingsystemctl start systemd-networkd-wait-online.service.BindsTo=+After=systemd-networkd.service, so networkd's start job is pulled into the new transaction — but never becomes runnable.systemd-networkd.serviceisAfter=network-pre.target(shipped systemd unit).Wants=network-pre.target+Before=network-pre.targetwhile keeping default dependencies, i.e. it is alsoAfter=sysinit.target. That makesnetwork-pre.target(and hence networkd) transitively ordered aftersysinit.target→ aftercloud-init-network.serviceitself.systemctl startbelongs to a second transaction, so systemd's ordering-cycle detection cannot see the loop (the cloud-init job is running, not queued). Boot hangs forever: host answers pings, but sshd/getty never start.Supporting evidence from the affected host (a locally-added
TimeoutStartSec=300drop-in used to break the hang for observation): the momentcloud-init-network.servicewas killed at t=306s, the queued networkd job started within 200ms and wait-online completed in 55ms — the jobs were runnable the whole time except for the ordering edge behind cloud-init itself.Notes on the chosen fix:
--job-mode=ignore-dependenciesare unsatisfiable by construction in this situation: a unit that is bothBefore=network-pre.targetand (via default deps)After=sysinit.targetcan never be honored on a boot where cloud-init's pre-sysinit network stage must bring the network online. Experimentally, addingWants=/After=systemd-networkd.servicetocloud-init-network.serviceinstead makes systemd detect the (then in-transaction) cycle and delete the third-party unit's start job — i.e. the constraint gets broken either way; this fix just does it without hanging or deleting jobs.systemd-networkd.serviceis started explicitly (first) becauseignore-dependenciesalso suppresses theBindsTo=pull-in, and because on the affected boots networkd is not yet active when the wait fires. Its netlink sockets are already listening at that point (they haveDefaultDependencies=no), so a dependency-less start is safe; on healthy boots where networkd is already active both calls are no-ops.cc_set_passwords(fix: don't deadlock when starting network service with systemctl #5935).TimeoutSec=0oncloud-init-network.servicemeans any future variant of this class of hang is an unbootable host rather than a slow boot; bounding the stage (or thesubpcall) may be worth a separate discussion.Test Steps
Reproduced and verified in local QEMU VMs (Ubuntu 26.04
resolute-server-cloudimg-amd64.img, NoCloud seed). The seed's user-data containsbootcmd(so_should_wait_on_network→ wait) and installs a minimal "third-party firewall helper" trap unit:Unpatched VM (stock cloud-init 26.1): first boot provisions and reboots; the second boot hangs forever — serial console:
Patched VM (same seed and trap unit, plus a runcmd applying this diff to the installed cloud-init): second boot completes normally. networkd is inactive when the stage begins (only its sockets are listening after the initrd hand-off); the stage itself now starts it:
The trap unit then runs later in boot without any deleted jobs (
Finished fake-vpn-helper.servicebeforeReached target network-pre.target).Unit tests:
python -m pytest tests/unittests— 5,746 passed; the newTestActivatorsWaitForNetworktests were written first and fail on main. Linters: black/isort/ruff/mypy/pylint clean at the pinned versions.Merge type