Skip to content

openvmm: in virt_whp, a VP saved in cli; hlt runs past the HLT after restore, but still reads back as Halted #1474

Closed

Description

@q5438722

Summary

This happens with the default --hypervisor whp (the hypervisor APIC), with or without --hv. OpenVMM saves a VP that is halted with interrupts off as MpState::Halted, which is correct. Restoring that state with restore_all succeeds, and activity() still returns Halted. But on its next run the VP executes the instructions after the HLT.

A VP idling through HV_X64_MSR_GUEST_IDLE (saved as Idle) behaves the same way. With sti; hlt, only a restore onto the same VP wakes it. KVM keeps the VP halted in all these cases.

Repro

This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs. The guest is real-mode code: out 0x80, 'H'; cli; hlt; out 0x80, 'W'. The test below is abridged from that file.

fn suspended_vp_stays_suspended(config: Config, code: &'static [u8], mark: u8, fresh: bool) {
    with_time(move |time| {
        let mut vm = run_to_suspend(config, code, mark, time); // run until 'H', stop
        let saved = vm.save(0);
        let expected = vm.activity(0).mp_state; // Halted
        if fresh {
            // As `--restore-snapshot`: a new partition with the saved RAM.
            let ram = vm.ram();
            drop(vm);
            let mut hv = whp(config);
            vm = Vm::new(&mut hv, config, 1, time);
            vm.mem.write_at(0, &ram).unwrap();
        }
        vm.restore(0, &saved).unwrap();
        assert_eq!(vm.activity(0).mp_state, expected, "read back after restore_all");
        let woke = vm.run_until_mark(b'W', Duration::from_secs(1));
        assert!(!woke, "the VP was saved as {expected:?} but ran past the suspending instruction");
    })
}

#[test]
fn halted_vp_stays_halted_after_restore_into_fresh_partition() {
    suspended_vp_stays_suspended(WHP, HALT, b'H', true);
}
// Also: ..._onto_same_vp, and idle_vp_stays_idle_after_restore_into_fresh_partition (WHP_HV1).
$ cargo test -p virt_whp --test vp_state -- halted_vp idle_vp
the VP was saved as Halted but ran past the suspending instruction after restore_all (marks "W")

All three tests fail on main 42d4fe619. The read-back right after restore_all passes in each of them.

Cause

The behavior comes from WHP itself. restore_all sets registers, then activity (halt_suspend = 1), then the APIC page (apic.rs#L292-L319). We tested this by restoring subsets of the elements in different orders:

  • Restoring any single element does not wake the VP.
  • Restoring registers and apic together does, with or without activity.
  • No order keeps the VP halted. If activity is written after apic, the halt is dropped and reads back as Running. That would trip restore_all's debug check (state.rs#L98-L113), so changing the order is not a fix.

Fix Suggestion

fix.patch works around this inside virt_whp, for the hypervisor APIC only. On restore of Halted or Idle, if the instruction before RIP is HLT (or the RDMSR of the idle MSR), it moves RIP back onto that instruction and restores the VP as Running. The VP then suspends itself again on its first instruction. Until the VP runs, the getters keep returning the saved mp_state and RIP, so restore_all's check and a re-save are unchanged. With the patch, all three restore paths keep the VP suspended, and a sweep of single-field state mutations gives the same results as before.

fix.patch (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..4df6f36d7 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -175,7 +175,100 @@ fn now(&mut self) -> VmTime {
     }
 }
 
+/// A halt or idle suspend state that `restore_activity` turned into a
+/// re-execution of the suspending instruction.
+///
+/// With the hypervisor's APIC, WHP does not keep a VP in a halt or idle
+/// suspend state that is restored along with the VP's register and APIC
+/// state: on its next run the VP continues after the HLT (or the RDMSR of
+/// HV_X64_MSR_GUEST_IDLE), even though the activity register still reads
+/// back as suspended, and a suspend state written after the APIC state is
+/// dropped. Instead of restoring the suspend state, RIP is pointed back at
+/// the suspending instruction and the VP is restored as running, so that it
+/// executes the instruction again and the hypervisor suspends it itself.
+///
+/// Until the VP runs, the state accessors still report the saved state:
+/// `mp_state` and `rip` (the architectural RIP, after the instruction).
+#[derive(Debug, Copy, Clone)]
+pub(crate) struct RestoredSuspend {
+    pub mp_state: vp::MpState,
+    pub rip: u64,
+}
+
 impl WhpProcessor<'_> {
+    /// Points RIP back at the instruction that put the VP into `mp_state`
+    /// (HLT for halt, RDMSR for the guest idle MSR) if that is the
+    /// instruction just before RIP. Returns the original RIP if it did.
+    fn rewind_to_suspend_instruction(
+        &mut self,
+        vtl: Vtl,
+        mp_state: vp::MpState,
+    ) -> Result<Option<u64>, Error> {
+        let expected: &[u8] = match mp_state {
+            vp::MpState::Halted => &[0xf4],     // hlt
+            vp::MpState::Idle => &[0x0f, 0x32], // rdmsr
+            vp::MpState::Running | vp::MpState::WaitForSipi => return Ok(None),
+        };
+        let wvp = self.vp.whp(vtl);
+        let (cs, rip, efer) = get_registers!(
+            wvp,
+            [
+                whp::RegisterSegment::Cs,
+                whp::Register64::Rip,
+                whp::Register64::Efer,
+            ]
+        )
+        .for_op("get registers")?;
+        let long_mode = efer & x86defs::X64_EFER_LMA != 0
+            && x86defs::SegmentAttributes::from(cs.Attributes).long();
+        let start = rip.wrapping_sub(expected.len() as u64);
+        for (i, &byte) in expected.iter().enumerate() {
+            let offset = start.wrapping_add(i as u64);
+            let gva = if long_mode {
+                offset
+            } else {
+                cs.Base.wrapping_add(offset) & 0xffff_ffff
+            };
+            let Ok(gpa) = wvp
+                .translate_gva(
+                    gva,
+                    whp::abi::WHvTranslateGvaFlagValidateExecute
+                        | whp::abi::WHvTranslateGvaFlagPrivilegeExempt,
+                )
+                .for_op("translate gva")?
+            else {
+                return Ok(None);
+            };
+            let mut b = [0];
+            if self.vp.partition.gm.read_at(gpa, &mut b).is_err() || b[0] != byte {
+                return Ok(None);
+            }
+        }
+        wvp.set_register(whp::Register64::Rip, start)
+            .for_op("set rip")?;
+        Ok(Some(rip))
+    }
+
+    /// Sets the register state. With the hypervisor's APIC, a new RIP
+    /// invalidates the check in [`Self::rewind_to_suspend_instruction`], so
+    /// redo it for a VP that is in a halt or idle suspend state.
+    pub(crate) fn restore_registers(
+        &mut self,
+        vtl: Vtl,
+        value: &vp::Registers,
+    ) -> Result<(), Error> {
+        let LocalApicKind::Offloaded = self.vp.partition.vtlp(vtl).lapic else {
+            return self.vp.set_register_state(vtl, value);
+        };
+        let activity = self.save_activity(vtl)?;
+        self.state.vtls[vtl].restored_suspend = None;
+        self.vp.set_register_state(vtl, value)?;
+        if matches!(activity.mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+            self.restore_activity(vtl, &activity)?;
+        }
+        Ok(())
+    }
+
     pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error> {
         let activity = match self.vp.partition.vtlp(vtl).lapic {
             LocalApicKind::Emulated(_) => {
@@ -203,6 +296,9 @@ pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error>
                     &self.vp.whp(vtl).get_apic().for_op("get apic state")?,
                 );
                 activity.nmi_pending = apic.hv_apic_nmi_pending();
+                if let Some(restored) = self.state.vtls[vtl].restored_suspend {
+                    activity.mp_state = restored.mp_state;
+                }
                 activity
             }
         };
@@ -247,7 +343,35 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
                 self.vp.set_register_state(vtl, &value)?;
             }
             LocalApicKind::Offloaded => {
-                self.vp.set_register_state(vtl, value)?;
+                // Undo an earlier rewind so that RIP is architectural again.
+                if let Some(restored) = self.state.vtls[vtl].restored_suspend.take() {
+                    self.vp
+                        .whp(vtl)
+                        .set_register(whp::Register64::Rip, restored.rip)
+                        .for_op("set rip")?;
+                }
+                let mut mp_state = value.mp_state;
+                if let Some(rip) = self.rewind_to_suspend_instruction(vtl, mp_state)? {
+                    self.state.vtls[vtl].restored_suspend = Some(RestoredSuspend { mp_state, rip });
+                    mp_state = vp::MpState::Running;
+                } else if matches!(mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+                    tracelimit::warn_ratelimited!(
+                        ?mp_state,
+                        "restoring a suspend state without its instruction before rip; \
+                         the hypervisor may resume the vp"
+                    );
+                }
+                self.vp.set_register_state(
+                    vtl,
+                    &vp::Activity {
+                        mp_state,
+                        nmi_pending: value.nmi_pending,
+                        nmi_masked: value.nmi_masked,
+                        interrupt_shadow: value.interrupt_shadow,
+                        pending_event: value.pending_event,
+                        pending_interruption: value.pending_interruption,
+                    },
+                )?;
                 // Set the NMI pending bit via the APIC.
                 let mut apic = vp::ApicRegisters::from_page(
                     &self.vp.whp(vtl).get_apic().for_op("get apic state")?,
diff --git a/vmm_core/virt_whp/src/lib.rs b/vmm_core/virt_whp/src/lib.rs
index f3088f4f4..871e2cb1f 100644
--- a/vmm_core/virt_whp/src/lib.rs
+++ b/vmm_core/virt_whp/src/lib.rs
@@ -249,6 +249,11 @@ struct PerVtlRunState {
     deliverability_notifications: HvDeliverabilityNotificationsRegister,
     // Only used when `hv` is `None`.
     vp_assist_page: u64,
+    /// A restored halt or idle state that is being re-entered by executing
+    /// the suspending instruction again (see `apic::RestoredSuspend`).
+    #[cfg(guest_arch = "x86_64")]
+    #[inspect(debug)]
+    restored_suspend: Option<apic::RestoredSuspend>,
 }
 
 impl PerVtlRunState {
@@ -269,6 +274,8 @@ pub fn new(lapic: &LocalApicKind, hv: &Hv1State, vp_info: &TargetVpInfo, vtl: Vt
             },
             deliverability_notifications: 0.into(),
             vp_assist_page: 0,
+            #[cfg(guest_arch = "x86_64")]
+            restored_suspend: None,
         }
     }
 
@@ -359,6 +366,8 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
             hv,
             deliverability_notifications,
             vp_assist_page,
+            #[cfg(guest_arch = "x86_64")]
+            restored_suspend,
         } = self;
 
         #[cfg(guest_arch = "x86_64")]
@@ -375,6 +384,10 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
 
         *deliverability_notifications = 0.into();
         *vp_assist_page = 0;
+        #[cfg(guest_arch = "x86_64")]
+        {
+            *restored_suspend = None;
+        }
     }
 }
 
diff --git a/vmm_core/virt_whp/src/vp.rs b/vmm_core/virt_whp/src/vp.rs
index 30549ffe6..6e57e33b5 100644
--- a/vmm_core/virt_whp/src/vp.rs
+++ b/vmm_core/virt_whp/src/vp.rs
@@ -371,6 +371,14 @@ pub(crate) async fn run_vp(
             // Set the lazy EOI bit just before running.
             let lazy_eoi = self.sync_lazy_eoi();
 
+            // From here on the VP executes the instruction that a restore
+            // rewound to, so its register state is architectural again.
+            #[cfg(guest_arch = "x86_64")]
+            {
+                let vtl = self.state.active_vtl;
+                self.state.vtls[vtl].restored_suspend = None;
+            }
+
             let mut runner = self.current_whp().runner();
             let exit = runner
                 .run()
diff --git a/vmm_core/virt_whp/src/vp_state.rs b/vmm_core/virt_whp/src/vp_state.rs
index a187c12d6..434ef1808 100644
--- a/vmm_core/virt_whp/src/vp_state.rs
+++ b/vmm_core/virt_whp/src/vp_state.rs
@@ -89,11 +89,15 @@ fn commit(&mut self) -> Result<(), Self::Error> {
         }
 
         fn registers(&mut self) -> Result<vp::Registers, Self::Error> {
-            self.run.vp.get_register_state(self.vtl)
+            let mut regs: vp::Registers = self.run.vp.get_register_state(self.vtl)?;
+            if let Some(restored) = self.run.state.vtls[self.vtl].restored_suspend {
+                regs.rip = restored.rip;
+            }
+            Ok(regs)
         }
 
         fn set_registers(&mut self, value: &vp::Registers) -> Result<(), Self::Error> {
-            self.run.vp.set_register_state(self.vtl, value)
+            self.run.restore_registers(self.vtl, value)
         }
 
         fn activity(&mut self) -> Result<vp::Activity, Self::Error> {

Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.

Metadata

Metadata

Labels

area: openvmmInitial title-keyword triage suggestion

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions