fix (EC2/GCE): wait for NIC on allowlisted platforms by goldberl · Pull Request #7065 · canonical/cloud-init

goldberl · 2026-09-01T21:05:36Z

Proposed Commit Message

fix (EC2/GCE): wait for NIC on allowlisted platforms

On some GCE and AWS EC2 instances, cloud-init-local runs before
network interfaces are fully initialized by the kernel. This causes
early datasource detection to fail, preventing metadata fetching and
SSH access.

To avoid introducing boot delays across all platforms, implement
allowlist-gated NIC polling and centralize the retry logic in a shared
network helper:
 * add wait_for_candidate_nics() in cloudinit/net/init.py
 * use helper from DataSourceEc2 and DataSourceGCE when allowlisted
 * EC2 gate: DMI system-product-name (e.g. hpc7a.96xlarge)
 * GCE gate: DMI baseboard-product-name (e.g. izumi)

Fixes first-boot race conditions on affected AWS and GCP instances
without impacting unaffected instance types or breaking unit tests.

Add unit coverage for:
 * helper retry/timeout behavior
 * EC2 allowlisted vs non-allowlisted polling paths
 * GCE allowlisted vs non-allowlisted polling paths

Fixes: GH-6697, GH-6737, LP-2144694

Signed-off-by: Leah Goldberg <leah.goldberg@canonical.com>

Additional info

This fix will emit logs on affected AWS and GCP instances when NIC polling is enabled, such as:

# AWS (Ubuntu Noble Minimal - hpc7a.96xlarge)
# Race condition reproduced on boot (~100% reproduction rate)
# The allowlist check was triggered, the initial check detected no
# primary NICs, and the interface was successfully acquired after
# 1 retry (1.001s total duration).


ubuntu@ip-172-31-24-66:~$ sudo cat /var/log/cloud-init.log | grep -iE "polling allowlist|Candidate NIC polling|No primary NICs"

2026-09-04 17:59:15,747 - DataSourceEc2.py[DEBUG]: Product hpc7a.96xlarge is in NIC polling allowlist, waiting for NICs

2026-09-04 17:59:15,747 - net[DEBUG]: No primary NICs found with access to IMDS, waiting 1 seconds to retry

2026-09-04 17:59:16,748 - net[DEBUG]: Candidate NIC polling completed in 1.001 seconds (timeout=60, sleep_interval=1): ['enp34s0']

---------------------------------------

# GCP (Ubuntu Noble - izumi / c3-metal)
# Race condition did not hit (~1% reproduction rate)
# The allowlist check was verified (timeout=60), and the candidate
# NIC was discovered on the initial polling attempt without needing
# a retry loop.

ubuntu@goldberl-c3-noble-test-0:~$ sudo cat /var/log/cloud-init.log | grep -iE "polling allowlist|Candidate NIC polling|No primary NICs"

2026-09-04 18:05:12,301 - DataSourceGCE.py[DEBUG]: Board izumi is in NIC polling allowlist, waiting for NICs

2026-09-04 18:05:14,765 - net[DEBUG]: Candidate NIC polling completed in 2.464 seconds (timeout=60, sleep_interval=1): ['enp5s0f0']

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

blackboxsw

Thank you for all this work @goldberl! I think this looks like a contained approach which limits exposure to boot-time costs to known instance types.

I have a number of requests and discussion points inline. Please do feel free to push back on things that you think are unreasonable.

Can you also please add a comment with the related log entries emitted by this PR on a working instance?

goldberl · 2026-09-04T13:39:05Z

Hi @blackboxsw, thank you for the detailed review. I've made the following changes to the PR:

Used performance.Timed to track and log execution duration.
Added [bug]: EC2 datasource failing on first boot due to missing NICs #6697 and [bug] DataSourceGCELocal fails on GCE c3-metal instances due to NIC not ready at first boot #6737 reference in comments for context.
Updated docstring for wait_for_candidate_nics to mention carrier checks and documented behavior when timeout=0.
Updated debug logs in wait_for_candidate_nics to mention IMDS primary NICs and added timeout exceeded logs when timeout > 0.
Cleaned up DMI lookup by removing unnecessary or "" fallbacks.
Simplified flow in EC2 and GCE datasources to always invoke net.wait_for_candidate_nics(), passing timeout=60 for allowlisted platforms and timeout=0 for non-allowlisted platforms (option 3 from your comment).
Simplified the polling loop conditional to while not candidate_nics:, running find_candidate_nics() inside and breaking on success or timeout.
Updated EC2, GCE, and net unit tests to cover the unified flow and new log conditions.

I'll run tests on AWS and GCP instances to capture updated logs and update the PR description with those

…latforms On some GCE and AWS EC2 instances, cloud-init-local runs before network interfaces are fully initialized by the kernel. This causes early datasource detection to fail, preventing metadata fetching and SSH access. To avoid introducing boot delays across all platforms, implement allowlist gated NIC polling and centralize the retry logic in a shared network helper: * add wait_for_candidate_nics() in cloudinit/net/__init__.py * use helper from DataSourceEc2 and DataSourceGCE only when allowlisted * EC2 gate: DMI system-product-name (e.g. hpc7a.96xlarge) * GCE gate: DMI baseboard-product-name (e.g. izumi) Fixes first-boot race conditions on affected AWS and GCP instances without impacting unaffected instance types or breaking unit tests. Add unit coverage for: * helper retry/timeout behavior * EC2 allowlisted vs non-allowlisted polling paths * GCE allowlisted vs non-allowlisted polling paths Fixes: canonicalGH-6697, canonicalGH-6737, LP-2144694 Signed-off-by: Leah Goldberg <leah.goldberg@canonical.com>

blackboxsw

@goldberl thank you for this iteration.

While looking at the end product here, the use of timeout as dual purpose where 0 means just run once and > 0 means max_wait it makes it a bit harder to discern the intent of the timeout param.

I think we may need to go with the alternative solution to pre-flight check:

if not wait_on_nics:
   find_candidate_nics()
else:
   wait_on_candidate_nics()

That should then ensure a clear path between conditions which require calling wait_for_candidate_nics versus those we expect to succeed on a single call.

blackboxsw · 2026-09-08T14:14:16Z

+                break
+
+            LOG.debug(
+                "No primary NICs found with access to IMDS, waiting %s "


We don't truly know yet if IMDS is accessible, just that the NIC has carrier. So let's be more 'correct' about this in the logged line.

Suggested change

"No primary NICs found with access to IMDS, waiting %s "

"No primary NICs found with carrier, waiting %s "

blackboxsw · 2026-09-08T14:14:46Z

+            "Timed out after %s seconds waiting for primary NICs with access "
+            "to IMDS",


Same with this message, IMDS may or may not be present on certain platforms. But, waiting on carrier is specifically what we are doing.

blackboxsw · 2026-09-08T14:16:24Z

                LOG.debug("FreeBSD doesn't support running dhclient with -sf")
                return False
-            candidate_nics = net.find_candidate_nics()
+            product_name = dmi.read_dmi_data("system-product-name")


We may want to .lower() to ensure we don't get hit by case-sensitivity.

blackboxsw · 2026-09-08T18:13:34Z

+
+    def test_timeout_logs_when_timeout_is_exceeded(self, mocker, caplog):
+        mocker.patch("cloudinit.net.find_candidate_nics", return_value=[])
+        mocker.patch("cloudinit.net.time.sleep")


We are mocking time.sleep in these unittests but not time.monotonic which means we leak calls to the underlying host during these test runs which adds runtime to our unittests. Please mock time.monotonic where appropriate and validate that we don't block for 1 second while running tox -e py3 -- tests/unittests/net/test_init.py

blackboxsw · 2026-09-08T18:25:45Z

+            wait_for_nics = board_name in NETWORK_POLLING_BOARD_ALLOW_LIST
+            if wait_for_nics:
+                LOG.debug(
+                    "Board %s is in NIC polling allowlist, waiting for NICs",


Let's generalize this log Waiting for NICs with carrier flag. and pull it into wait_for_candidate_nics. No need to report our instance type/allowlist/etc. Let's avoid duplicating in GCE and Ec2 where possible.

blackboxsw · 2026-09-08T18:30:20Z

+                )
+            candidate_nics = net.wait_for_candidate_nics(
+                timeout=60 if wait_for_nics else 0, sleep_interval=1
+            )


Thank you for this iteration @goldberl.

There are a couple of things that make we want to avoid this choice now:

the duplication of this logging in both GCE and EC2 which really should be inside wait_for_candidate_nics

the unclear dual-purpose of the timeout param inside wait_for_candidate_nics where a 0 timeout is special and means don't retry, whereas the timeout > 0 means max_wait time once find_candidate_nics or retries complete.

We are already defining a wait_for_nics local anyway which allows GCE or Ec2 to avoid calling wait_for_candidate_nics in conditions where is isn't planning on waiting. This avoids a perception in the future that Ec2 or GCE will be waiting in all cases, and then we need to parse the slightly obscure timeout = 60 if wait_4_nix or 0 and then interpret what that means by re-reading the wait_for_candidate_nics implementation.

Instead of this, I think we probably should just go with the prior alternative proposed:

set a local variable find_candidate_nics which will be either net.find_candidate_nics or net.wait_on_candidate_nics based on whether the platform is in ALLOW_LIST.

No need to addtiionally log "Instance Id in ALLOW_LIST" we can discern that based on a leading log "Waiting on candidate NICs with carrier" or something like that when wait_for_candidate_nics is called.

blackboxsw · 2026-09-08T18:33:09Z

+
+    def test_timeout_logs_when_timeout_is_exceeded(self, mocker, caplog):
+        mocker.patch("cloudinit.net.find_candidate_nics", return_value=[])
+        mocker.patch("cloudinit.net.time.sleep")


We are mocking time.sleep in these unittests but not time.monotonic which means we leak calls to the underlying host during these test runs which adds runtime to our unittests. Please mock time.monotonic where appropriate and validate that we don't block for 1 second while running tox -e py3 -- tests/unittests/net/test_init.py

blackboxsw self-assigned this Sep 2, 2026

blackboxsw requested changes Sep 2, 2026

View reviewed changes

goldberl force-pushed the nic-rety-logic-with-allowlist branch from 9ae9014 to 8517063 Compare September 4, 2026 13:34

goldberl force-pushed the nic-rety-logic-with-allowlist branch from 8517063 to 29aa35b Compare September 4, 2026 13:48

goldberl force-pushed the nic-rety-logic-with-allowlist branch from 29aa35b to f9d5234 Compare September 4, 2026 13:56

goldberl requested a review from blackboxsw September 4, 2026 18:10

blackboxsw requested changes Sep 8, 2026

View reviewed changes

	"No primary NICs found with access to IMDS, waiting %s "
	"No primary NICs found with carrier, waiting %s "

		"Timed out after %s seconds waiting for primary NICs with access "
		"to IMDS",

Uh oh!

Conversation

goldberl commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Commit Message

Additional info

Merge type

Uh oh!

blackboxsw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

goldberl commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blackboxsw left a comment

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

blackboxsw Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

goldberl commented Sep 1, 2026 •

edited

Loading

goldberl commented Sep 4, 2026 •

edited

Loading