[Issue]: Race condition between Node Reboot during Driver Upgrade and DRA/Metrics Exporter DaemonSets

[kaiwalyajoshi]

Problem Description

We're currently running v1.5.0 and v1.5.1-beta.0 AMD GPU Operators.
We're also testing upgrades between v1.5.0 -> v1.5.1-beta.0

These operators are running with the following configs:
v1.5.0

deviceConfig:
  spec:
    driver:
      enable: true
      blacklist: true
      version: "30.20.1"
      upgradePolicy:
        enable: true
    devicePlugin:
      enableDevicePlugin: false
      enableNodeLabeller: false
    metricsExporter:
      enable: true
    draDriver:
      enable: true

v1.5.1-beta.0

deviceConfig:
  spec:
    driver:
      enable: true
      blacklist: true
      version: "31.30"
      upgradePolicy:
        enable: true
    devicePlugin:
      enableDevicePlugin: false
      enableNodeLabeller: false
    metricsExporter:
      enable: true
    draDriver:
      enable: true

Note that we're retaining deviceConfig.spec.driver.upgradePolicy.rebootRequired: true from the upstream Helm AMD GPU Operator charts.

With the above, we have noticed a race condition between the GPU Operator performing its reboot while the images are getting pulled for DRA & Metrics Exporter. This means that when the GPU Node restarts, the DRA & Metrics Exporter images may not have pulled all of its layers and its pods can crashloop with a exec format error message.

Operating System

24.04.4 LTS (Noble Numbat)

CPU

AMD EPYC 9555 64-Core Processor

GPU

MI350P

ROCm Version

7.2.x

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

rocminfo --support output

Paste output here

Additional Information

Additional RCA Information here:
NKP-v2.18.0-Exec-Format-Errors-Executive-Summary.md.pdf

[yansun1996]

Thanks for the detailed report and the RCA write-up. We've reproduced and root-caused this, and the fix is merged in #606 (shipping in v1.5.1). Sharing our analysis and recommendations below.

Root cause

The failure has two parts:

1. Exposure window (operator lifecycle): During a driver-version transition, KMM installs the target kernel module and the upgrade policy moves the node toward reboot, while kubelet/containerd are still pulling and unpacking images for GPU-dependent operands (DRA driver, metrics exporter). If the node reboots inside that window, an image can be recorded as present while its unpacked overlayfs snapshot is left incomplete.
2. Persistent corruption (containerd/overlayfs): After reboot, containerd reuses that node-local snapshot as-is, so the affected pods crash-loop with exec format error. This second part is a containerd-layer behavior and is not specific to our operands — any containerd-managed image can be left in this state by a reboot that interrupts an in-flight unpack (including sudden reboots from power loss, etc.).

Fix (v1.5.1)

The operator's pre-reboot drain now evicts all GPU operand pods before the node reboots, including the DRA driver pod, which was previously missing from the eviction list. Combined with the existing cordon + NoSchedule taint, no operand images are being pulled or unpacked between the drain and the reboot, which closes the exposure window for operator-managed workloads. The upgrade also syncs filesystems before the reboot.

The resulting sequence is:

taint target GPU node → drain operand pods (including DRA pod) → unload old driver + sync + reboot → load new driver → bring up operands

Recommendations

• Let the operator orchestrate DRA. If you're using the operator together with DRA, deploy the operator and let it manage the DRA driver as an operand. The operator handles the full operand lifecycle during driver upgrade internally, which is what allows it to drain the DRA pod correctly before reboot.
• Split operator upgrade and driver upgrade into separate steps rather than doing both at once.
• Consider hardening containerd globally. Because the underlying overlayfs exposure isn't specific to our stack, containerd's image-integrity option is worth evaluating as a broader mitigation:

  [plugins."io.containerd.cri.v1.images"]
    image_pull_with_sync_fs = true

  This syncs the filesystem after image unpack and helps protect any containerd-managed image against reboot-interrupted-unpack corruption, beyond just the operator's operands.

Closing this as resolved by #606. Please give v1.5.1 a try when it's available and reopen if you still see the race. Feedback and PRs are always welcome.

[yansun1996]

Resolved by #606 (cherry-pick of the upstream fix). Shipping in v1.5.1.