Skip to content

openvmm: VPCI hot removal closes the chipset bus while VPs can still access it #1156

Closed

Description

@tangruize

Dynamic VPCI removal closes the bus mutex while waiting for VMBus revocation, although running VPs can still access its MMIO intercepts. An access in that interval encounters a closed chipset lock and panics.

In vmm_core/src/device_builder.rs, DynamicVpciDevice::remove closes the bus before awaiting the channel revoke:

    pub async fn remove(self) {
        self.vpci_bus.close().revoke().await;
        self.vpci_unit.remove().await;
        self.pci_unit.remove().await;
    }

VpciBus::revoke borrows the bus until its awaited channel revocation finishes. That keeps the closeable mutex closed across the wait. The VPCI intercepts are removed only by the following vpci_unit.remove, while chipset dispatch uses the ordinary lock path and expects the mutex to remain open.

Trigger

Remove an offered VPCI device while its VMBus revocation is pending and a running VP accesses the VPCI MMIO bus. The removal sequence creates the closed-lock window independently of guest ejection acknowledgement. This concerns standard VMs with dynamic VPCI devices; microVMs have no VMBus or VPCI.

Fix

Make VpciBus::revoke synchronously move out the channel and mark the bus revoked, returning an owned waiter that does not borrow the bus. Obtain that waiter under an ordinary short lock, release the lock, and then await revocation.

This preserves bus-before-device removal ordering without closing chipset access over the asynchronous VMBus wait.

Activity

  1. tangruize commented on Nov 1, 2026

    @tangruize
    ContributorAuthor

    Proposed patch

    Patch against OpenVMM 998b9f2402050cd16d1faa897bc831ef1434be48. Save the contents below as fix.patch and run git apply fix.patch.

    fix.patch
    diff --git a/vm/devices/pci/vpci/src/bus.rs b/vm/devices/pci/vpci/src/bus.rs
    index 9e1d6a05b..4bcf6586d 100644
    --- a/vm/devices/pci/vpci/src/bus.rs
    +++ b/vm/devices/pci/vpci/src/bus.rs
    @@ -238,13 +238,17 @@ pub fn eject_control(&self) -> VpciBusEject {
     
         /// Revokes the VMBus channel and waits for revocation to complete.
         ///
    +    /// Marks the channel revoked synchronously and returns a wait that does not
    +    /// borrow the bus, so chipset access can continue during revocation.
    +    ///
         /// If an offer is in progress, the offering task revokes its channel when
         /// the offer completes.
    -    pub async fn revoke(&mut self) {
    -        if let VpciBusChannelState::Offered { channel } =
    -            std::mem::replace(&mut self.channel, VpciBusChannelState::Revoked)
    -        {
    -            channel.revoke().await;
    +    pub fn revoke(&mut self) -> impl Future<Output = ()> + use<> {
    +        let state = std::mem::replace(&mut self.channel, VpciBusChannelState::Revoked);
    +        async move {
    +            if let VpciBusChannelState::Offered { channel } = state {
    +                channel.revoke().await;
    +            }
             }
         }
     }
    @@ -548,6 +552,34 @@ mod tests {
     
         const BASE_ADDR: u64 = 0x1000_0000;
     
    +    #[async_test]
    +    async fn bus_access_is_available_between_revoke_and_completion(_driver: DefaultDriver) {
    +        let device = Arc::new(CloseableMutex::new(DeferWriteDevice::new()));
    +        let (bus, _offer) = VpciBus::new_unoffered(
    +            VpciBusConfig {
    +                instance_id: Guid::new_random(),
    +                vtom: None,
    +                vnode: None,
    +            },
    +            device,
    +            &mut ExternallyManagedMmioIntercepts,
    +            VpciInterruptMapper::new(TestVpciInterruptController::new()),
    +        )
    +        .unwrap();
    +        let bus = Arc::new(CloseableMutex::new(bus));
    +        let wait = bus.lock().revoke();
    +        {
    +            let mut bus = bus.lock();
    +            assert!(matches!(bus.channel, VpciBusChannelState::Revoked));
    +            let mut data = [0; 4];
    +            assert!(matches!(
    +                bus.bus_device.mmio_read(0, &mut data),
    +                IoResult::Ok | IoResult::Err(_)
    +            ));
    +        }
    +        wait.await;
    +    }
    +
         /// A minimal PCI device that returns `IoResult::Ok` for all operations
         /// until `start_deferring` is called, after which `pci_cfg_write` defers
         /// completion until driven by `poll_device`.
    diff --git a/vmm_core/src/device_builder.rs b/vmm_core/src/device_builder.rs
    index 440d98a9d..7c851a1b8 100644
    --- a/vmm_core/src/device_builder.rs
    +++ b/vmm_core/src/device_builder.rs
    @@ -56,7 +56,8 @@ pub async fn eject(&self) -> anyhow::Result<()> {
     
         /// Removes the VPCI bus before removing its underlying PCI device.
         pub async fn remove(self) {
    -        self.vpci_bus.close().revoke().await;
    +        let revoked = self.vpci_bus.lock().revoke();
    +        revoked.await;
             self.vpci_unit.remove().await;
             self.pci_unit.remove().await;
         }
  2. ppenna commented on Nov 11, 2026

    @ppenna
    Contributor

    Closing: this is an OpenVMM bug outside NVX, and its fix is filed upstream as microsoft/openvmm#4978, vmm_core: keep a removed VPCI bus open while its channel is revoked. Tracking continues there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions