[RFC] Recovering from Xid 79 / GPU fallen off the bus without rebooting Linux

[edwardgushchin]

Hi NVIDIA maintainers,

Per CONTRIBUTING.md, I am opening this RFC before turning a local prototype into pull requests.

The problem

A discrete GPU can occasionally disappear from PCIe and report Xid 79: GPU has fallen off the bus. When that GPU drives the desktop, the result is much larger than one failed application: the monitors go black, remote desktop may remain connected without receiving new frames, and every GPU-facing process becomes unusable.

At the same time, Linux itself can still be alive. Networking, storage, SSH, and unrelated system services may continue to work normally. Rebooting the whole operating system is therefore a very expensive recovery boundary for a failure isolated to one PCIe device and its clients.

The goal of this work is simple: when the GPU fails, recover as much as the hardware state safely allows, keep the Linux boot and non-GPU services alive, and restore the graphical stack without requiring an OS reboot.

How the prototype solves it

The key was to stop treating every GPU failure as the same kind of reset. The prototype classifies the state after an Xid/RM recovery decision and chooses one of two recovery paths.

1. The GPU is still reachable and supports FLR

When PCI configuration is readable and RM selects the gpu-pf-flr recovery action, the current driver object tree can be preserved:

1. A delayed worker takes the existing global power-state and PM write locks.
2. NvKMS, UVM, and user channels are quiesced through the normal system-PM suspend path.
3. The PCI function receives a Function Level Reset.
4. The normal resume path restores RM/GSP, UVM, modeset, and clients.
5. PCI identity and driver state are verified before the transaction is declared complete.
6. A session helper reapplies the compositor's unchanged current monitor layout so DRM scanout and remote capture are recreated.

The Xid callback never resets hardware directly. It records the primary event and schedules classification after a short settle window, allowing a following Xid 154 and the final RM recovery action to participate in the decision.

2. The GPU has genuinely fallen off the bus

After Xid 79, RM has already marked the GPU disconnected/lost and invalidated its channels. Restoring PCIe connectivity alone cannot make the old RM/GSP object tree valid again, so the prototype deliberately does not attempt an in-place resume.

Instead, the kernel publishes a recovery request to a privileged host supervisor. The supervisor:

1. remembers which GPU-facing services were active and drains their device users;
2. unbinds every function belonging to the GPU slot;
3. verifies that the slot is directly behind an isolated Root Port with no unrelated devices in reset scope;
4. performs a Secondary Bus Reset;
5. rebinds the GPU functions, recreating fresh driver/RM state;
6. restarts only the graphical and GPU-facing services that were active before recovery.

This path cannot preserve in-flight GPU work or the old graphical session, but it preserves the Linux boot, SSH, storage, networking, and unrelated services. No OS reboot is requested or performed.

Result

On the test system (GA102 / RTX 3090 Ti, open kernel modules 610.57.04, GNOME Wayland), rebuilding the GPU stack and replaying the current compositor layout returned all three physical displays and remote desktop frames while the Linux boot and SSH connection stayed alive.

The current prototype combines the delayed kernel classifier, PM/FLR coordinator, topology-gated host supervisor, recovery telemetry, and compositor watcher into one automatic policy. It fails closed when the recovery action or PCIe topology is ambiguous, and none of the reset paths use hard-coded PCI addresses.

The implementation also keeps the mechanism/policy boundary explicit: the kernel classifies the RM/GPU state and performs the PM-preserving transaction, while destructive host orchestration and desktop-specific layout replay remain in userspace.

Proposed upstream shape

I would like to split the work into reviewable pieces rather than submit the complete prototype as one patch:

1. accurate PM resume error propagation and recovery telemetry;
2. the PM/FLR recovery coordinator;
3. delayed Xid/RM classification and a generic kernel-to-userspace lost-link notification;
4. keep compositor integration and host service policy outside the production kernel patch.

Questions for maintainers

1. Is preserving Linux uptime after Xid 79, with an explicit loss of GPU clients but no OS reboot, within scope for this repository?
2. What kernel-to-userspace mechanism would be appropriate for the lost-link handoff: uevent, generic netlink, devlink health reporting, or an existing NVIDIA interface?
3. Is the proposed split suitable for review, or should the first change be limited to telemetry/error propagation?
4. Is there an existing RM recovery/reinitialization path or locking rule that should replace any part of this design?
5. What hardware matrix and recovery evidence would you require for the first code PR?

I can provide the source audit, state-machine traces, recovery telemetry, and hardware logs. I would prefer to align the design with maintainer guidance before publishing the patch series.

Thanks.

[edwardgushchin]

A real lost-link event on the same GA102 / RTX 3090 Ti system exposed a missing requirement in this proposal: teardown must be able to finish when the GPU can no longer complete display work.

First, a clarification of the original Result section: the earlier display restoration demonstrated the reachable-GPU recovery/layout-replay path. It did not validate end-to-end recovery from an already inaccessible GPU. The supervised unbind/SBR/rebind path remains experimental; the field event below did not recover successfully.

On September 6, with open modules 610.57.04 plus the local prototype on CachyOS 7.2.0 / GNOME Wayland, the classifier recorded primary Xid 79 followed by Xid 154 and inaccessible PCI configuration. Linux and SSH remained usable. All readable PCI configuration bytes for the GPU subsequently returned 0xff, and NVML could no longer obtain a device handle. The original cause of the PCIe loss is still unknown; no pre-fault temperature measurement was captured.

The supervisor stopped the graphical stack and GPU clients, but GPU unbind did not return. The kernel recorded a blocked modeset worker and the unbinding task inside a display-notifier wait. Relevant functions from the captured stack, listed from the waiting function toward the userspace write:

nvEvoWaitForCoreNotifier
...
drm_atomic_helper_shutdown
nv_drm_dev_unload
nv_drm_dev_destroy
nv_pci_remove_helper
pci_device_remove
device_release_driver_internal
unbind_store
kernfs_fop_write_iter
__x64_sys_write

A later normal reboot request entered shutdown, but the unbinding task remained after repeated SIGKILL attempts. The machine ultimately required a manual hardware reset. The final shutdown stage was not captured, so I cannot attribute every part of the shutdown stall exclusively to this task.

The relevant source is EvoCheckNotifier() in src/nvidia-modeset/src/nvkms-dma.c. At base commit e4a5faa, the short timeout allows the lost-notification fallback only when the channel GET offset equals PUT. Consequently, expiration alone does not bound the wait: an inaccessible or non-progressing queue can keep the call inside the loop indefinitely. A userspace deadline started after the blocking sysfs write cannot address this.

I have prepared a narrow local candidate that returns FALSE after the existing short timeout if GET reads 0xffffffff, without synthesizing a completed notifier. It preserves the existing behavior for completed notifications, nonblocking polling, a drained channel with a lost notification, and a slow but reachable channel. A host-side harness compiling the actual C wait/timeout functions reproduces the nonterminating old behavior with simulated all-ones MMIO and passes with the candidate.

This is candidate-level evidence only. MMIO GET itself was not sampled during the field failure; the captured all-ones data was PCI configuration space. The candidate has not been loaded or tested on a real lost-link event. Module compilation completed with objtool/rethunk warnings, and this is not a claim of an upstream-ready or hardware-validated recovery fix. In particular, the outer wait API is void, so returning FALSE from the internal helper does not provide end-to-end fatal-error propagation.

The question for maintainers is now more specific: what is the supported NvKMS/RM/DRM teardown path once a GPU is known to be inaccessible? Should waits such as this one consult an explicit lost-device state, propagate a fatal error through the wait/HAL callers, or be bypassed by a dedicated removal path? Is an all-ones MMIO check acceptable as a narrow guard, or is there a more authoritative existing signal?

A lost-device teardown mechanism that does not require further GPU completion appears to be a prerequisite for the proposed supervised recovery path. I can provide the focused diff, deterministic regression harness, and sanitized stack excerpts for review.