Skip to content

vpci: skip the device while its state unit holds it closed - #5075

Open
Robert Nowotny (bitranox) wants to merge 4 commits into
microsoft:mainfrom
bitranox:vpci-closed-device-on-reset
Open

Robert Nowotny (bitranox) wants to merge 4 commits into
microsoft:mainfrom
bitranox:vpci-closed-device-on-reset

Conversation

@bitranox

@bitranox Robert Nowotny (bitranox) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

A VM reset runs the state units' resets concurrently (StateUnits::run_op joins them), and two of them can collide on a VPCI device:

  • The VMBus server's reset closes the VPCI channel. VpciChannel::close calls release_all, which powers the device off and clears its BARs through lock() on the device's CloseableMutex.
  • The device's own ArcMutexChipsetDeviceUnit holds that mutex closed for the whole of its reset().

No unit dependency orders the two. When they overlap, lock() panics with lock should not be closed, which aborts the VM worker. We hit this once, on an in-guest reset of a Linux guest with an emulated MANA NIC on VPCI, and could not reproduce it on demand.

While close runs, the channel's config space accesses (bars, set_bars, set_power) now go through one helper that uses lock_if_open() and skips the device while it is closed. PciBusCfgAccessCallbackView in bus.rs already handles a closed device this way. Skipping leaves turning the device off to the device's own reset, and three VPCI devices had none:

  • GdmaDevice: gdma: return the device to its initial state on reset #4932 makes its reset return the config space to its initial state. Until that lands, a skipped close leaves MMIO enabled on it.
  • RelayedVpciDevice: the second commit makes its reset clear the MMIO-enable bit, through the same path a guest write takes, so a TDISP device is deactivated too. The BARs are left alone, since the client only shadows BAR writes until MMIO is enabled again. The third commit has the reset first finish any write still in flight: a stopped device's lock stays open, so the channel's own MMIO-off can arrive between stop and reset.
  • AssignedPciDevice (WHP): its reset is still a TODO, so until it has one, a skipped close leaves its MMIO mapped. That reset is virt_whp: reset the assigned PCI device #5076.

Guest requests (a BAR assignment, a release, a power change) keep the plain lock(), as before: the guest gets a success reply for them, so they must not be skipped quietly. The fourth commit narrows the skip to close for that reason.

The other way to fix this would be to order the resets, so that a VPCI device's reset waits until the VMBus server's reset has closed its channel. I went with the smaller change; if you'd rather have the ordering, I can rework it that way.

#4978 covers a different window, the bus mutex held closed across a hot removal, and the two PRs touch different files. This race is between the device's own reset and the VMBus server's reset, a case its notes don't cover.

Testing

  • close_while_the_device_is_closed_does_not_touch_it holds the device's mutex closed and closes the channel. Without the fix it panics with lock should not be closed; with it, the channel leaves the device alone. close_with_the_device_open_powers_it_off checks the normal path still powers the device off.
  • reset_turns_mmio_off in vpci_relay turns MMIO on through the relay, with and without TDISP, resets the device, and checks that MMIO is off on the host. It fails against the old empty reset. reset_finishes_writes_issued_after_stop issues that MMIO-off write after stop with another write queued behind it, then resets, and checks both complete. It hit the assertion the second commit had added.
  • guest_power_request_needs_the_device_open sends a power change to a closed device outside close and expects the lock() panic, so a guest request is never skipped.
  • On this branch: cargo test -p vpci -p vpci_relay (34 passed), clippy with -D warnings, and the workspace clippy, build, doc and fmt checks. This branch was not booted, and the relay changes have no OpenHCL run behind them, only the unit tests.
  • The same change in our downstream build went through 16 reboot and reset cycles of the Linux guest that crashed, plus two runs of the in-guest reset where the crash happened, with no panic. The race showed up only once, so those runs mostly show that resets still work; the unit test is what covers the race itself.

A VM reset runs the state units' resets concurrently. The VMBus server's
reset closes each VPCI channel, and VpciChannel::close powers the device
off and clears its BARs through lock() on the device's CloseableMutex.
If the device's ArcMutexChipsetDeviceUnit is in its own reset at that
moment, it holds the mutex closed, lock() panics with "lock should not be
closed", and the panic aborts the VM worker.

Route the channel's config space accesses (bars, set_bars, set_power)
through lock_if_open() and skip the device while it is closed. Its reset
returns the config space to its initial state, so the skipped writes
have nothing left to do. PciBusCfgAccessCallbackView in bus.rs already
handles a closed device this way.
@bitranox
Robert Nowotny (bitranox) requested a review from a team as a code owner November 13, 2026 13:08
Copilot AI balanced review requested due to automatic review settings November 13, 2026 13:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Channel close can skip PCI cleanup when overlapping no-op device resets, leaving MMIO enabled or stale BAR mappings after the channel’s resource state is released.

1 open issue
What changed in this PR

This PR addresses a VPCI channel-close panic during concurrent VM resets by skipping PCI configuration access while the device’s state unit holds its mutex closed. It targets a different race from the hot-removal issue in #4978.

Changes:

  • Routes BAR reads, BAR writes, and power changes through a shared lock_if_open() helper.
  • Adds tests for channel closure with the device closed and normal power-off with it open.
File Description
vm/​devices/​pci/​vpci/​src/​device.rs Adds guarded configuration access and channel-close tests.

Review effort: Balanced

Comment thread vm/devices/pci/vpci/src/device.rs Resolved
RelayedVpciDevice::reset did nothing. When a VM reset closes the VPCI
channel while the device's state unit holds the device closed for its
reset, the channel now skips the device, so powering it off has to
happen in the device's own reset. Clear the MMIO-enable bit through the
same path a guest write takes, so a TDISP device is deactivated too.

The BARs are left alone: until MMIO is enabled again a BAR write only
changes the client's shadow copy, and with MMIO off they decode nothing.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 13:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Reset cleanup remains unsafe: a deferred TDISP deactivation can abort the relay worker, and Windows assigned-device resets can leave MMIO mappings active after resource bookkeeping is discarded.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Clean up assigned-device MMIO mappings during reset

vm/​devices/​pci/​vpci/​src/​device.rs:1252

AssignedPciDevice is also attached through VPCI on Windows, but its reset is still a TODO (vmm_core/virt_whp/src/device.rs:456-458). If channel close skips power-off during that reset, MMIO mappings remain active even though release_all() discards the channel's resource bookkeeping. Implement reset cleanup for assigned devices too, or ensure skipped channel cleanup completes before the device reopens.

1 resolved since last review

Review effort: Balanced

Comment thread vm/devices/pci/vpci_relay/src/lib.rs Resolved
A stopped device's lock stays open, so the VPCI channel can still turn
MMIO off between stop() and reset(). On a TDISP device that write starts
a deactivation, which nothing polls while the device is stopped, and the
reset's check that nothing was pending fired. Run the same drain stop()
uses at the start of reset().
Copilot AI balanced review requested due to automatic review settings November 13, 2026 14:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A critical pause/resume race can acknowledge guest BAR or power writes as successful while the device is closed, silently dropping them without replay on resume.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

High severity Paused device acknowledges BAR/power writes without applying them

vm/​devices/​pci/​vpci/​src/​device.rs:1252

During pause, ArcMutexChipsetDeviceUnit::stop closes the device independently of the VPCI channel, so a guest request can reach this helper while the device is closed. set_bars/set_power then skip the write but handle_packet still reports SUCCESS, and resume neither resets the device nor replays the request. Limit skipping to reset cleanup; order or defer normal protocol accesses so acknowledged writes are applied.

1 resolved since last review

Review effort: Balanced

The skip also covered guest requests: a BAR assignment, a release or a
power change that found the device closed was dropped, yet the guest
still got a success reply. A closed device is only expected while a VM
reset closes the channel, so take the skip there and keep the plain
lock everywhere else.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 15:10
@bitranox

Copy link
Copy Markdown
Contributor Author

The fourth commit takes the skip out of the guest request paths. Only close skips a closed device now; a BAR assignment, a release or a power change from the guest goes through lock() again, so it is applied or it panics as it does on main, and is never acknowledged without being applied. guest_power_request_needs_the_device_open pins that.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-sensitive TDISP teardown needs human validation of reset ordering, transport progress, and isolation cleanup because mock-host tests omit BAR mappings and no OpenHCL runtime validation was performed.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants