Skip to content

fix(net): avoid boot deadlock when starting systemd-networkd-wait-online - #7053

Open
AryanHamedani wants to merge 1 commit into
canonical:mainfrom
AryanHamedani:fix/networkd-wait-online-boot-deadlock
Open

fix(net): avoid boot deadlock when starting systemd-networkd-wait-online#7053
AryanHamedani wants to merge 1 commit into
canonical:mainfrom
AryanHamedani:fix/networkd-wait-online-boot-deadlock

Conversation

@AryanHamedani

Copy link
Copy Markdown

Proposed Commit Message

fix(net): avoid boot deadlock when starting systemd-networkd-wait-online

NetworkdActivator.wait_for_network() runs a plain blocking
`systemctl start systemd-networkd-wait-online.service`. On Ubuntu this
is issued at runtime from cloud-init's network stage (see 5772), which
is ordered Before=sysinit.target. systemd-networkd-wait-online.service
is BindsTo/After systemd-networkd.service, and systemd-networkd.service
is After=network-pre.target. Any unit ordered Before=network-pre.target
that keeps default dependencies (a common pattern for third-party
firewall/VPN helpers) is also After=sysinit.target, so the networkd
start job pulled in by the wait is transitively ordered after the very
cloud-init unit issuing the start. The job never becomes runnable, and
because it belongs to a separate transaction systemd cannot detect the
cycle. With cloud-init-network.service shipping TimeoutSec=0, boot
hangs forever with no sshd and no console login - an unbootable host.

Start systemd-networkd explicitly and pass
--job-mode=ignore-dependencies for both jobs. The skipped ordering
constraints are unsatisfiable by construction while cloud-init's
network stage is running, and this mirrors the fix applied to the
analogous sshd restart deadlock in cc_set_passwords (5935).

Fixes GH-7052

Additional Context

This is the root-cause fix for #7052 (production host bricked by this deadlock; full journal forensics in the issue). The precise deadlock chain, established from the journals of an affected host (Ubuntu 26.04, cloud-init 26.1, systemd 259, OpenStack/ConfigDrive, netplan renderer):

  1. cloud-init-network.service (Before=sysinit.target, TimeoutSec=0) reaches Distro.wait_for_network() → blocking systemctl start systemd-networkd-wait-online.service.
  2. The started unit is BindsTo= + After=systemd-networkd.service, so networkd's start job is pulled into the new transaction — but never becomes runnable.
  3. systemd-networkd.service is After=network-pre.target (shipped systemd unit).
  4. A third-party VPN helper on the host declares Wants=network-pre.target + Before=network-pre.target while keeping default dependencies, i.e. it is also After=sysinit.target. That makes network-pre.target (and hence networkd) transitively ordered after sysinit.target → after cloud-init-network.service itself.
  5. The blocking systemctl start belongs to a second transaction, so systemd's ordering-cycle detection cannot see the loop (the cloud-init job is running, not queued). Boot hangs forever: host answers pings, but sshd/getty never start.

Supporting evidence from the affected host (a locally-added TimeoutStartSec=300 drop-in used to break the hang for observation): the moment cloud-init-network.service was killed at t=306s, the queued networkd job started within 200ms and wait-online completed in 55ms — the jobs were runnable the whole time except for the ordering edge behind cloud-init itself.

Notes on the chosen fix:

  • The ordering constraints skipped by --job-mode=ignore-dependencies are unsatisfiable by construction in this situation: a unit that is both Before=network-pre.target and (via default deps) After=sysinit.target can never be honored on a boot where cloud-init's pre-sysinit network stage must bring the network online. Experimentally, adding Wants=/After=systemd-networkd.service to cloud-init-network.service instead makes systemd detect the (then in-transaction) cycle and delete the third-party unit's start job — i.e. the constraint gets broken either way; this fix just does it without hanging or deleting jobs.
  • systemd-networkd.service is started explicitly (first) because ignore-dependencies also suppresses the BindsTo= pull-in, and because on the affected boots networkd is not yet active when the wait fires. Its netlink sockets are already listening at that point (they have DefaultDependencies=no), so a dependency-less start is safe; on healthy boots where networkd is already active both calls are no-ops.
  • Same approach as the analogous deadlock fix in cc_set_passwords (fix: don't deadlock when starting network service with systemctl #5935).
  • Possible follow-up, deliberately out of scope here: TimeoutSec=0 on cloud-init-network.service means any future variant of this class of hang is an unbootable host rather than a slow boot; bounding the stage (or the subp call) may be worth a separate discussion.

Test Steps

Reproduced and verified in local QEMU VMs (Ubuntu 26.04 resolute-server-cloudimg-amd64.img, NoCloud seed). The seed's user-data contains bootcmd (so _should_wait_on_network → wait) and installs a minimal "third-party firewall helper" trap unit:

[Unit]
Description=Fake third-party VPN/firewall helper (default dependencies)
Wants=network-pre.target
Before=network-pre.target

[Service]
Type=oneshot
ExecStart=/usr/bin/true
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
  • Unpatched VM (stock cloud-init 26.1): first boot provisions and reboots; the second boot hangs forever — serial console:

             Starting cloud-init-network.service - Cloud-init: Network Stage...
    [   ***] Job cloud-init-network.service/start running (4min 29s / no limit)   <- spins forever
    
  • Patched VM (same seed and trap unit, plus a runcmd applying this diff to the installed cloud-init): second boot completes normally. networkd is inactive when the stage begins (only its sockets are listening after the initrd hand-off); the stage itself now starts it:

             Starting cloud-init-network.service - Cloud-init: Network Stage...
             Starting systemd-networkd.service - Network Management...
    [  OK  ] Started systemd-networkd.service - Network Management.
             Starting systemd-networkd-wait-onl… - Wait for Network to be Online...
    [  OK  ] Finished systemd-networkd-wait-onl…ce - Wait for Network to be Online.
    [  OK  ] Finished cloud-init-network.service - Cloud-init: Network Stage.
    ...
    [  OK  ] Reached target multi-user.target - Multi-User System.
    repro-b login:
    

    The trap unit then runs later in boot without any deleted jobs (Finished fake-vpn-helper.service before Reached target network-pre.target).

Unit tests: python -m pytest tests/unittests — 5,746 passed; the new TestActivatorsWaitForNetwork tests were written first and fail on main. Linters: black/isort/ruff/mypy/pylint clean at the pinned versions.

Merge type

  • Squash merge using "Proposed Commit Message"
  • Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

NetworkdActivator.wait_for_network() runs a plain blocking
`systemctl start systemd-networkd-wait-online.service`. On Ubuntu this
is issued at runtime from cloud-init's network stage (see 5772), which
is ordered Before=sysinit.target. systemd-networkd-wait-online.service
is BindsTo/After systemd-networkd.service, and systemd-networkd.service
is After=network-pre.target. Any unit ordered Before=network-pre.target
that keeps default dependencies (a common pattern for third-party
firewall/VPN helpers) is also After=sysinit.target, so the networkd
start job pulled in by the wait is transitively ordered after the very
cloud-init unit issuing the start. The job never becomes runnable, and
because it belongs to a separate transaction systemd cannot detect the
cycle. With cloud-init-network.service shipping TimeoutSec=0, boot
hangs forever with no sshd and no console login - an unbootable host.

Start systemd-networkd explicitly and pass
--job-mode=ignore-dependencies for both jobs. The skipped ordering
constraints are unsatisfiable by construction while cloud-init's
network stage is running, and this mirrors the fix applied to the
analogous sshd restart deadlock in cc_set_passwords (5935).

Fixes canonicalGH-7052
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant