Skip to content

openvmm: KVM snapshots silently lose pending SMI and SMM execution state #1158

Closed

Description

@tangruize

Snapshot capture can succeed while a KVM CPU has an SMI pending or is executing in System Management Mode (SMM). The saved state cannot represent either condition, so restore loses the event or resumes outside the captured SMM handler.

In vmm_core/virt/src/state.rs, the generated save_all checks only whether each element exists for the partition; it does not check whether those elements can represent the CPU's current execution state:

fn save_all(&mut self) -> Result<$save_state, $crate::state::StateError<Self::Error>> {
    let mut save_state = $save_state::default();
    $(
        if <$ty as $crate::state::StateElement<$caps, $vp>>::is_present(self.caps()) {
            save_state.$get = Some(self.$get().map_err(|err| $crate::state::StateError{phase: concat!("save ", stringify!($id)), err})?);
        }
    )*

    Ok(save_state)
}

No saved CPU element contains SMM state or a pending SMI, yet all available getters can succeed. Separately, KvmVpStateAccess::set_activity omits KVM_VCPUEVENT_VALID_SMM, so writing reset or restored activity onto an existing CPU does not clear its SMM-related state.

Trigger

Deliver an SMI through an IO-APIC/MSI route and capture before the CPU consumes it, or capture while the CPU is in SMM. Restore the resulting snapshot: the pending SMI or interrupted handler state is missing. Resetting or restoring onto a CPU already in SMM exposes the stale-destination-state problem.

Fix

fix.patch adds a saveability check before any element is read. The KVM backend inspects get_vcpu_events()?.smi and refuses capture when smm or pending is nonzero. Its activity setter supplies KVM_VCPUEVENT_VALID_SMM with zero SMM fields, explicitly clearing SMM, pending SMI and latched INIT on restore/reset.

This refuses unsupported capture rather than implementing SMM snapshots. SMI delivery remains immediate; a capture retry requires the guest to complete its SMM handler.

Activity

  1. tangruize commented on Nov 1, 2026

    @tangruize
    ContributorAuthor

    Proposed patch

    Patch against OpenVMM 998b9f2402050cd16d1faa897bc831ef1434be48. Save the contents below as fix.patch and run git apply fix.patch.

    fix.patch
    diff --git a/Cargo.lock b/Cargo.lock
    index 55754c811..e67618ebc 100644
    --- a/Cargo.lock
    +++ b/Cargo.lock
    @@ -10239,6 +10239,7 @@ dependencies = [
      "libc",
      "memory_range",
      "open_enum",
    + "pal_async",
      "pal_event",
      "parking_lot",
      "pci_core",
    diff --git a/vmm_core/virt/src/state.rs b/vmm_core/virt/src/state.rs
    index f30ba98e1..f14f56934 100644
    --- a/vmm_core/virt/src/state.rs
    +++ b/vmm_core/virt/src/state.rs
    @@ -73,6 +73,15 @@ pub trait $trait {
                     /// Commits any state changes made with the `set_*` methods.
                     fn commit(&mut self) -> Result<(), Self::Error>;
     
    +                /// Checks that the state can be saved: that the elements a save
    +                /// reads hold all of it. A backend refuses a state they cannot
    +                /// hold (KVM: a VP in SMM or with an SMI pending, whose SMM state
    +                /// no element holds). `save_all` calls it first, before it reads
    +                /// any element. By default, `Ok`.
    +                fn check_saveable(&mut self) -> Result<(), Self::Error> {
    +                    Ok(())
    +                }
    +
                     $(
                     /// Gets the specified state.
                     fn $get(&mut self) -> Result<$ty, Self::Error>;
    @@ -83,6 +92,7 @@ pub trait $trait {
                     /// Save all state that can be restored by a call to restore.
                     #[allow(unused_mut)]
                     fn save_all(&mut self) -> Result<$save_state, $crate::state::StateError<Self::Error>> {
    +                    self.check_saveable().map_err(|err| $crate::state::StateError{phase: "save check", err})?;
                         let mut save_state = $save_state::default();
                         $(
                             if <$ty as $crate::state::StateElement<$caps, $vp>>::is_present(self.caps()) {
    diff --git a/vmm_core/virt_kvm/Cargo.toml b/vmm_core/virt_kvm/Cargo.toml
    index 28d2745f7..c765b5296 100644
    --- a/vmm_core/virt_kvm/Cargo.toml
    +++ b/vmm_core/virt_kvm/Cargo.toml
    @@ -36,6 +36,10 @@ tracelimit.workspace = true
     zerocopy.workspace = true
     
     [dev-dependencies]
    +vm_topology.workspace = true
    +vmcore.workspace = true
    +pal_async.workspace = true
    +guestmem.workspace = true
     test_with_tracing.workspace = true
     
     [target.'cfg(all(target_os = "linux", target_arch = "aarch64"))'.dependencies]
    diff --git a/vmm_core/virt_kvm/src/arch/x86_64/mod.rs b/vmm_core/virt_kvm/src/arch/x86_64/mod.rs
    index d6b9daafa..91d74485e 100644
    --- a/vmm_core/virt_kvm/src/arch/x86_64/mod.rs
    +++ b/vmm_core/virt_kvm/src/arch/x86_64/mod.rs
    @@ -12,6 +12,8 @@
     pub(crate) mod snp;
     mod time_abi;
     mod vm_state;
    +#[cfg(test)]
    +mod smm_tests;
     mod vp_state;
     
     pub(crate) use time_abi::KvmTimeAbi;
    diff --git a/vmm_core/virt_kvm/src/arch/x86_64/smm_tests.rs b/vmm_core/virt_kvm/src/arch/x86_64/smm_tests.rs
    new file mode 100644
    index 000000000..935ff27d3
    --- /dev/null
    +++ b/vmm_core/virt_kvm/src/arch/x86_64/smm_tests.rs
    @@ -0,0 +1,151 @@
    +// Copyright (c) Microsoft Corporation.
    +// Licensed under the MIT License.
    +
    +use super::*;
    +use guestmem::GuestMemory;
    +use pal_async::DefaultPool;
    +use test_with_tracing::test;
    +use virt::BindProcessor;
    +use virt::Hypervisor;
    +use virt::PartitionConfig;
    +use virt::ProtoPartitionConfig;
    +use virt::x86::vp;
    +use vm_topology::memory::MemoryLayout;
    +use vm_topology::processor::TopologyBuilder;
    +use vmcore::vmtime::VmTime;
    +use vmcore::vmtime::VmTimeKeeper;
    +use x86defs::apic::DeliveryMode;
    +
    +/// Builds a KVM partition with `vp_count` VPs (no guest memory mapped: the
    +/// VPs never run) and calls `f` with it and its VP binders.
    +fn with_partition<R>(
    +    vp_count: u32,
    +    f: impl FnOnce(&KvmPartition, Vec<KvmProcessorBinder>) -> R,
    +) -> R {
    +    DefaultPool::run_with(async |driver| {
    +        let keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
    +        let vmtime = keeper.builder().build(&driver).await.unwrap();
    +        let topology = TopologyBuilder::new_x86().build(vp_count).unwrap();
    +        let mem_layout = MemoryLayout::new(0x400000, &[], &[], &[], None).unwrap();
    +        let guest_memory = GuestMemory::allocate(mem_layout.end_of_ram() as usize);
    +        let (partition, binders) = Kvm::new()
    +            .unwrap()
    +            .new_partition(ProtoPartitionConfig {
    +                processor_topology: &topology,
    +                hv_config: None,
    +                vmtime: &vmtime,
    +                isolation: virt::ProtoPartitionIsolation::None,
    +                nested_virt: false,
    +                user_mode_memory_faults: false,
    +                lazy_memory_registration: false,
    +                time_abi: None,
    +            })
    +            .unwrap()
    +            .build(PartitionConfig {
    +                mem_layout: &mem_layout,
    +                guest_memory: &guest_memory,
    +                cpuid: &[],
    +                vtl0_alias_map: None,
    +                fault_resolver: None,
    +            })
    +            .unwrap();
    +        assert!(
    +            !partition.caps().hv1,
    +            "a microVM partition has no Hyper-V enlightenments"
    +        );
    +        f(&partition, binders)
    +    })
    +}
    +
    +/// A device's MSI to `apic_id` (physical destination, edge).
    +fn msi(mode: DeliveryMode, apic_id: u32, vector: u8) -> MsiRequest {
    +    MsiRequest::new_x86(mode, apic_id, false, vector, false)
    +}
    +
    +/// The VP's SMM state in KVM (KVM_GET_VCPU_EVENTS's `smi`).
    +fn smm_state(vp: &KvmProcessor<'_>) -> kvm::kvm_vcpu_events__bindgen_ty_4 {
    +    vp.kvm.get_vcpu_events().unwrap().smi
    +}
    +
    +/// Puts the VP in SMM, as its run does at an SMI, with an INIT latched if
    +/// `latched_init` (an INIT that SMM blocks).
    +fn enter_smm(vp: &KvmProcessor<'_>, latched_init: bool) {
    +    let mut events = vp.kvm.get_vcpu_events().unwrap();
    +    events.flags = kvm::KVM_VCPUEVENT_VALID_SMM;
    +    events.smi.smm = 1;
    +    events.smi.latched_init = latched_init.into();
    +    vp.kvm.set_vcpu_events(&events).unwrap();
    +}
    +
    +/// Whether `err`, a save's error, is the refusal of VP `vp_index`'s SMM
    +/// state.
    +fn is_smm_refusal(err: &virt::state::StateError<KvmError>, vp_index: u32) -> bool {
    +    matches!(
    +        std::error::Error::source(err).and_then(|err| err.downcast_ref::<KvmError>()),
    +        Some(KvmError::SmmState { vp_index: index }) if *index == vp_index
    +    )
    +}
    +
    +/// BUG-15 (before the fix, the pending SMI was lost across a save and a
    +/// restore). A device's SMI reaches a running VP as KVM's pending SMI: the
    +/// IO-APIC copies the guest's SMI delivery mode into its MSI, KVM_SIGNAL_MSI
    +/// calls kvm_inject_smi, and KVM enters SMM at the VP's next run, executing
    +/// guest RAM at SMBASE. The VP's saved state has no SMM state, so the save of
    +/// a VP with an SMI pending is refused (`check_saveable`, before the save reads
    +/// any element), naming the VP; the VP keeps its SMI.
    +#[test]
    +#[ignore = "requires /dev/kvm"]
    +fn a_save_with_an_smi_pending_is_refused() {
    +    with_partition(1, |partition, mut binders| {
    +        let mut vp = binders[0].bind().unwrap();
    +        let apic_id = vp.inner.vp_info.apic_id;
    +        {
    +            partition.request_msi(Vtl::Vtl0, msi(DeliveryMode::SMI, apic_id, 0));
    +        }
    +        assert_eq!(smm_state(&vp).pending, 1, "KVM has the SMI pending");
    +
    +        let err = vp.access_state(Vtl::Vtl0).save_all().unwrap_err();
    +        assert!(is_smm_refusal(&err, 0), "{err:?}");
    +        assert_eq!(smm_state(&vp).pending, 1, "the refusal changed nothing");
    +    })
    +}
    +
    +/// BUG-15: the save of a VP in SMM, or in SMM with an INIT latched (SMM blocks
    +/// INIT), is refused (`check_saveable`), naming the VP.
    +#[test]
    +#[ignore = "requires /dev/kvm"]
    +fn a_save_of_a_vp_in_smm_is_refused() {
    +    with_partition(1, |_partition, mut binders| {
    +        let mut vp = binders[0].bind().unwrap();
    +        for latched_init in [false, true] {
    +            enter_smm(&vp, latched_init);
    +            let err = vp.access_state(Vtl::Vtl0).save_all().unwrap_err();
    +            assert!(
    +                is_smm_refusal(&err, 0),
    +                "latched INIT {latched_init}: {err:?}"
    +            );
    +        }
    +    })
    +}
    +
    +/// BUG-15: setting a VP's activity, which a restore and a VP's reset do,
    +/// leaves it outside SMM with no SMI pending and no INIT latched, as a saved
    +/// state (and a reset state) describes it; a VP's reset kept them before.
    +#[test]
    +#[ignore = "requires /dev/kvm"]
    +fn restore_and_reset_leave_smm() {
    +    with_partition(1, |_partition, mut binders| {
    +        let mut vp = binders[0].bind().unwrap();
    +        let vp_info = vp.inner.vp_info;
    +        let outside = kvm::kvm_vcpu_events__bindgen_ty_4::default();
    +
    +        enter_smm(&vp, true);
    +        let activity = vp::Activity::default();
    +        vp.access_state(Vtl::Vtl0).set_activity(&activity).unwrap();
    +        assert_eq!(smm_state(&vp), outside, "restore");
    +
    +        enter_smm(&vp, true);
    +        vp.access_state(Vtl::Vtl0).reset_all(&vp_info).unwrap();
    +        assert_eq!(smm_state(&vp), outside, "reset");
    +    })
    +}
    diff --git a/vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs b/vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs
    index 59dd45bac..1d271a857 100644
    --- a/vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs
    +++ b/vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs
    @@ -169,6 +169,22 @@ fn commit(&mut self) -> Result<(), Self::Error> {
             Ok(())
         }
     
    +    /// A VP in SMM, or with an SMI pending, holds state that a VP's saved
    +    /// state cannot (OpenVMM saves no SMM state), so a save refuses it before
    +    /// it reads anything. (KVM's `latched_init` is not checked: KVM reports any
    +    /// INIT it has not yet processed, which a stop's Record step applies before
    +    /// a save; an INIT stays latched past that only while SMM blocks it, and a
    +    /// VP in SMM is refused.)
    +    fn check_saveable(&mut self) -> Result<(), Self::Error> {
    +        let smi = self.kvm().get_vcpu_events()?.smi;
    +        if smi.smm != 0 || smi.pending != 0 {
    +            return Err(KvmError::SmmState {
    +                vp_index: self.vp.inner.vp_info.base.vp_index.index(),
    +            });
    +        }
    +        Ok(())
    +    }
    +
         fn registers(&mut self) -> Result<vp::Registers, Self::Error> {
             let regs = self.kvm().get_regs()?;
     
    @@ -360,6 +376,7 @@ fn set_activity(&mut self, value: &vp::Activity) -> Result<(), Self::Error> {
             if interrupt_shadow {
                 event_flags |= kvm::KVM_VCPUEVENT_VALID_SHADOW;
             }
    +        event_flags |= kvm::KVM_VCPUEVENT_VALID_SMM;
     
             let mut events = kvm::kvm_vcpu_events {
                 exception: kvm::kvm_vcpu_events__bindgen_ty_1 {
    @@ -383,6 +400,12 @@ fn set_activity(&mut self, value: &vp::Activity) -> Result<(), Self::Error> {
                 },
                 sipi_vector: 0,
                 flags: event_flags,
    +            smi: kvm::kvm_vcpu_events__bindgen_ty_4 {
    +                smm: 0,
    +                pending: 0,
    +                smm_inside_nmi: 0,
    +                latched_init: 0,
    +            },
                 exception_has_payload: 0,
                 exception_payload: 0,
                 ..Default::default()
    diff --git a/vmm_core/virt_kvm/src/lib.rs b/vmm_core/virt_kvm/src/lib.rs
    index 5ad901a6c..46d7804b8 100644
    --- a/vmm_core/virt_kvm/src/lib.rs
    +++ b/vmm_core/virt_kvm/src/lib.rs
    @@ -67,6 +67,11 @@ pub enum KvmError {
         State(#[from] Box<StateError<KvmError>>),
         #[error("invalid state while restoring: {0}")]
         InvalidState(&'static str),
    +    #[cfg(guest_arch = "x86_64")]
    +    #[error(
    +        "VP {vp_index} is in SMM or has an SMI pending, which a snapshot cannot hold; retry the snapshot"
    +    )]
    +    SmmState { vp_index: u32 },
         #[error("unsupported isolation configuration: {0}")]
         UnsupportedIsolationConfiguration(&'static str),
         #[error("misaligned gic base address")]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions