Skip to content

openvmm: in virt_whp, with the user-mode APIC, an INIT IPI aborts the VMM and snapshot restore always fails #1472

Open

Description

@q5438722

Summary

With --hypervisor whp:user_mode_apic (or --hv with whp:no_enlightenments, which forces the user-mode APIC), restore_activity writes all four activity registers to WHP, including InternalActivityState. When the partition's LocalApicEmulationMode is None, WHP rejects writes to that register with HV_STATUS_ACCESS_DENIED (0xC0350006), even when the value is unchanged.

This breaks two things:

  • INIT IPIs. When an AP receives INIT, inject_init → x86_init → set_activity hits the same error and unwraps it, so the VMM aborts. Every multiprocessor guest sends INIT while starting its APs.
  • Snapshot restore. restore_all fails at activity for every saved state, even an unchanged save of the same VM.

Repro

This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs.

The tests below are abridged from that file:

fn init_ipi_to_ap(config: Config) {
    with_time(move |time| {
        let mut hv = whp(config);
        let mut vm = Vm::new(&mut hv, config, 2, time);
        // BSP in 32-bit protected mode: ICR high = APIC ID 1, ICR low = 0x4500 (INIT).
        vm.mem.write_at(CODE, SEND_INIT).unwrap();
        vm.set_registers(0, protected_mode);
        assert!(vm.run_until_mark(b'S', Duration::from_secs(5)));
        // Give the AP time to process the INIT.
        vm.start();
        std::thread::sleep(Duration::from_millis(500));
        vm.stop();
        assert_eq!(vm.activity(1).mp_state, vp::MpState::WaitForSipi);
    })
}

#[test]
fn user_mode_apic_init_ipi_to_ap() {
    init_ipi_to_ap(WHP_USER_MODE_APIC);
}

// user_mode_apic_restore_unchanged_state: save_all on a fresh VP, then
// restore_all into a fresh partition of the same configuration.
// The controls hypervisor_apic_* (default WHP config) pass.
cargo test -p virt_whp --test vp_state -- user_mode_apic_init_ipi_to_ap user_mode_apic_restore_unchanged_state user_mode_apic_hv1_restore_unchanged_state hypervisor_apic

Output on main 42d4fe619:

panicked at vmm_core\virt_whp\src\apic.rs:487:45:
called `Result::unwrap()` on an `Err` value: Whp { operation: "set registers", source: Access to the specified object was denied. (os error -1070268410) }

restore_all ... failed: state access error, phase restore "activity": whp error, failed to set registers: Access to the specified object was denied.

Cause

  • The Emulated branch of restore_activity writes the whole Activity with set_register_state (apic.rs#L247). The first of its four registers is InternalActivityState.
  • Calling the WHP API directly, only InternalActivityState is denied with LocalApicEmulationMode = None. The other three can be written. With XApic/X2Apic emulation all four can be written.
  • x86_init calls set_activity for APs (vp.rs#L2066), and inject_init unwraps the result (apic.rs#L487).

virt_backend_tests only covers user_mode_apic: false, so CI doesn't exercise this path.

Suggested fix

In the Emulated branch, write only PendingInterruption, InterruptState and PendingEvent. In this mode OpenVMM tracks halt and wait-for-SIPI itself (self.state.halted, lapic.startup_suspend), and the value being written is always Running, the register's current value. A patch is attached (fix.patch). It also fixes the INIT path.

Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.

Suggested fix (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..0b87ca6e4 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -14,15 +14,18 @@
 use crate::WhpProcessor;
 use crate::WhpResultExt;
 use crate::WhpVpRef;
+use hvdef::HvRegisterValue;
 use hvdef::HvVtlEntryReason;
 use hvdef::HvX64PendingEventReg0;
 use hvdef::HvX64PendingInterruptionRegister;
+use hvdef::HvX64RegisterName;
 use hvdef::Vtl;
 use inspect::Inspect;
 use std::sync::atomic::Ordering;
 use virt::VpIndex;
 use virt::io::CpuIo;
 use virt::irqcon::MsiRequest;
+use virt::state::HvRegisterState;
 use virt::x86::MsrError;
 use virt::x86::vp;
 use virt_support_apic::ApicClient;
@@ -34,6 +37,7 @@
 use whp::get_registers;
 use whp::set_registers;
 use x86defs::RFlags;
+use zerocopy::FromZeros;
 
 impl WhpPartitionInner {
     pub fn lint(&self, vp_index: VpIndex, vtl: Vtl, index: usize) {
@@ -244,7 +248,36 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
                     pending_interruption: value.pending_interruption,
                 };
 
-                self.vp.set_register_state(vtl, &value)?;
+                // The run state lives in `self.state` and the emulated APIC
+                // above, not in the hypervisor. With the APIC emulated in
+                // user mode (`WHvX64LocalApicEmulationModeNone`), WHP denies
+                // writes to the internal activity state register, even of
+                // its current value, so write only the other activity
+                // registers.
+                let mut values = [HvRegisterValue::new_zeroed(); 4];
+                debug_assert_eq!(
+                    value.names(),
+                    &[
+                        HvX64RegisterName::InternalActivityState,
+                        HvX64RegisterName::PendingInterruption,
+                        HvX64RegisterName::InterruptState,
+                        HvX64RegisterName::PendingEvent0,
+                    ]
+                );
+                value.get_values(values.iter_mut());
+                let [_, pending_interruption, interrupt_state, pending_event] = values;
+                set_registers!(
+                    self.vp.whp(vtl),
+                    [
+                        (
+                            whp::Register64::PendingInterruption,
+                            pending_interruption.as_u64()
+                        ),
+                        (whp::Register64::InterruptState, interrupt_state.as_u64()),
+                        (whp::Register128::PendingEvent, pending_event.as_u128()),
+                    ]
+                )
+                .for_op("set activity registers")?;
             }
             LocalApicKind::Offloaded => {
                 self.vp.set_register_state(vtl, value)?;

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: openvmmInitial title-keyword triage suggestionarea: snapshotInitial title-keyword triage suggestion

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions