fix: 3024, offline non-MLX interfaces so we skip onboard NICs by stoo-davies · Pull Request #4120 · NVIDIA/infra-controller

stoo-davies · 2026-07-24T17:46:16Z

Fixes issue 3024.
We will offline any non-MLX interfaces before starting forge-dpu-agent, so we don't try to register via the wrong IP address.
Seen on a GB300 setup where the onboard NIC was connected up, and showed up first in the routing table.

Related issues

#3024

Type of Change

Add - New feature or capability
Change - Changes in existing functionality
Fix - Bug fixes
Remove - Removed features or deprecated functionality
Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

This PR contains breaking changes

Testing

Unit tests added/updated
Integration tests added/updated
Manual testing performed
No testing required (docs, internal refactor, etc.)

Additional Notes

Tested on an x86 host, a GB200 and a GB300.
The GB300 test below was one of the machines we hit the original bug on.

Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP16p3s0f0np0 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP16p3s0f1np1 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP18p3s0f0np0 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP18p3s0f1np1 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP22s22np0 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP2p3s0f0np0 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enP2p3s0f1np1 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Disabling network interface: interface=enP5p9s0 vendor=0x8086
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enp3s0f0np0 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enp3s0f1np1 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enp3s0f2np2 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=enp3s0f3np3 vendor=0x15b3
Jul 24 17:34:16 scout forge-scout-pre.sh[3007]: Keeping network interface: interface=lo reason=loopback

Also adds the anynic kernel command line arg to skip the checks, useful for booting from test interfaces.

Signed-off-by: Stoo Davies <189670495+stoo-davies@users.noreply.github.com>

copy-pr-bot · 2026-07-24T17:46:19Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-07-24T17:46:25Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a31a4045-302e-4d98-9af2-35b56418cefc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Walkthrough

Adds a Scout NIC filtering helper that preserves loopback and Mellanox interfaces, packages it into both Scout images with iproute2, and invokes it during startup with failure-tolerant handling. The x86_64 startup flow also adds an additional udev synchronization step.

Changes

Scout NIC filtering

Layer / File(s)	Summary
NIC filtering script `pxe/common_files/forge-scout-network.sh`	Adds configurable cmdline and sysfs inputs, an `anynic` bypass, vendor-based interface preservation, and disabling of other interfaces through `ip`.
Helper packaging `pxe/Makefile.toml`, `pxe/mkosi.profiles/scout-oss-/mkosi.conf`, `pxe/mkosi.profiles/scout-oss-/mkosi.postinst.chroot`	Stages and installs `forge-scout-network.sh` for both architectures and adds `iproute2` to their package lists.
Boot-time invocation `pxe/mkosi.profiles/scout-oss-*/mkosi.extra/opt/forge/forge-scout-pre.sh`	Runs the network helper without aborting startup when it fails; the x86_64 script additionally waits for udev to settle.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScoutImage
  participant forge-scout-pre.sh
  participant forge-scout-network.sh
  participant SysfsNet
  participant ip
  ScoutImage->>forge-scout-pre.sh: Start boot preparation
  forge-scout-pre.sh->>forge-scout-network.sh: Invoke NIC filtering
  forge-scout-network.sh->>SysfsNet: Inspect network interfaces
  SysfsNet-->>forge-scout-network.sh: Return interface and vendor data
  forge-scout-network.sh->>ip: Disable non-preserved interfaces
  ip-->>forge-scout-network.sh: Return operation status
  forge-scout-network.sh-->>forge-scout-pre.sh: Return success or failure
  forge-scout-pre.sh-->>ScoutImage: Continue startup

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately captures the core change: offlining non-MLX interfaces to avoid onboard NIC selection.
Description check	✅ Passed	The description matches the implemented change and explains the issue, behavior, and test context.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

stoo-davies · 2026-07-24T17:53:42Z

@coderabbitai review

coderabbitai · 2026-07-24T17:53:48Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@pxe/mkosi.profiles/scout-oss-aarch64/mkosi.extra/opt/forge/forge-scout-pre.sh`:
- Line 41: Move the /opt/forge/forge-scout-network.sh invocation in both
forge-scout-pre.sh variants to after the final udevadm settle and module-loading
steps, so all interfaces are filtered; alternatively, make its non-zero exit
fail pre-start instead of suppressing it with a warning. Apply the same change
to
pxe/mkosi.profiles/scout-oss-aarch64/mkosi.extra/opt/forge/forge-scout-pre.sh:41-41
and
pxe/mkosi.profiles/scout-oss-x86_64/mkosi.extra/opt/forge/forge-scout-pre.sh:41-41.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 04d3dfda-a8b7-4a96-a3c2-4c4065428296

📥 Commits

Reviewing files that changed from the base of the PR and between 65d1653 and b9451a4.

📒 Files selected for processing (8)

pxe/Makefile.toml
pxe/common_files/forge-scout-network.sh
pxe/mkosi.profiles/scout-oss-aarch64/mkosi.conf
pxe/mkosi.profiles/scout-oss-aarch64/mkosi.extra/opt/forge/forge-scout-pre.sh
pxe/mkosi.profiles/scout-oss-aarch64/mkosi.postinst.chroot
pxe/mkosi.profiles/scout-oss-x86_64/mkosi.conf
pxe/mkosi.profiles/scout-oss-x86_64/mkosi.extra/opt/forge/forge-scout-pre.sh
pxe/mkosi.profiles/scout-oss-x86_64/mkosi.postinst.chroot

Signed-off-by: Stoo Davies <189670495+stoo-davies@users.noreply.github.com>

stoo-davies · 2026-07-24T18:14:31Z

/ok to test 9ebbba5

github-actions · 2026-07-24T18:16:07Z

🌿 Preview your docs: https://nvidia-preview-pull-request-4120.docs.buildwithfern.com/infra-controller

stoo-davies added 2 commits July 20, 2026 16:06

Fix: 3024, down any non-mlx interfaces

e460a8c

Signed-off-by: Stoo Davies <189670495+stoo-davies@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into scout-nic-ignore

b9451a4

Signed-off-by: Stoo Davies <189670495+stoo-davies@users.noreply.github.com>

coderabbitai Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread pxe/mkosi.profiles/scout-oss-aarch64/mkosi.extra/opt/forge/forge-scout-pre.sh Outdated

Addressing rabbit feedback.

9ebbba5

Signed-off-by: Stoo Davies <189670495+stoo-davies@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: 3024, offline non-MLX interfaces so we skip onboard NICs#4120

fix: 3024, offline non-MLX interfaces so we skip onboard NICs#4120
stoo-davies wants to merge 3 commits into
NVIDIA:mainfrom
stoo-davies:scout-nic-ignore

stoo-davies commented Jul 24, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

Review skipped

Uh oh!

stoo-davies commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

stoo-davies commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

stoo-davies commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related issues

Type of Change

Breaking Changes

Testing

Additional Notes

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

stoo-davies commented Jul 24, 2026

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stoo-davies commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

stoo-davies commented Jul 24, 2026 •

edited

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading