Repository navigation
Description
Summary
With --hypervisor whp:user_mode_apic (or --hv with whp:no_enlightenments, which forces the user-mode APIC), restore_activity writes all four activity registers to WHP, including InternalActivityState. When the partition's LocalApicEmulationMode is None, WHP rejects writes to that register with HV_STATUS_ACCESS_DENIED (0xC0350006), even when the value is unchanged.
This breaks two things:
- INIT IPIs. When an AP receives INIT,
inject_init→x86_init→set_activityhits the same error and unwraps it, so the VMM aborts. Every multiprocessor guest sends INIT while starting its APs. - Snapshot restore.
restore_allfails atactivityfor every saved state, even an unchanged save of the same VM.
Repro
This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs.
The tests below are abridged from that file:
fn init_ipi_to_ap(config: Config) {
with_time(move |time| {
let mut hv = whp(config);
let mut vm = Vm::new(&mut hv, config, 2, time);
// BSP in 32-bit protected mode: ICR high = APIC ID 1, ICR low = 0x4500 (INIT).
vm.mem.write_at(CODE, SEND_INIT).unwrap();
vm.set_registers(0, protected_mode);
assert!(vm.run_until_mark(b'S', Duration::from_secs(5)));
// Give the AP time to process the INIT.
vm.start();
std::thread::sleep(Duration::from_millis(500));
vm.stop();
assert_eq!(vm.activity(1).mp_state, vp::MpState::WaitForSipi);
})
}
#[test]
fn user_mode_apic_init_ipi_to_ap() {
init_ipi_to_ap(WHP_USER_MODE_APIC);
}
// user_mode_apic_restore_unchanged_state: save_all on a fresh VP, then
// restore_all into a fresh partition of the same configuration.
// The controls hypervisor_apic_* (default WHP config) pass.cargo test -p virt_whp --test vp_state -- user_mode_apic_init_ipi_to_ap user_mode_apic_restore_unchanged_state user_mode_apic_hv1_restore_unchanged_state hypervisor_apic
Output on main 42d4fe619:
panicked at vmm_core\virt_whp\src\apic.rs:487:45:
called `Result::unwrap()` on an `Err` value: Whp { operation: "set registers", source: Access to the specified object was denied. (os error -1070268410) }
restore_all ... failed: state access error, phase restore "activity": whp error, failed to set registers: Access to the specified object was denied.
Cause
- The Emulated branch of
restore_activitywrites the wholeActivitywithset_register_state(apic.rs#L247). The first of its four registers isInternalActivityState. - Calling the WHP API directly, only
InternalActivityStateis denied withLocalApicEmulationMode = None. The other three can be written. WithXApic/X2Apicemulation all four can be written. x86_initcallsset_activityfor APs (vp.rs#L2066), andinject_initunwraps the result (apic.rs#L487).
virt_backend_tests only covers user_mode_apic: false, so CI doesn't exercise this path.
Suggested fix
In the Emulated branch, write only PendingInterruption, InterruptState and PendingEvent. In this mode OpenVMM tracks halt and wait-for-SIPI itself (self.state.halted, lapic.startup_suspend), and the value being written is always Running, the register's current value. A patch is attached (fix.patch). It also fixes the INIT path.
Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.
Suggested fix (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..0b87ca6e4 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -14,15 +14,18 @@
use crate::WhpProcessor;
use crate::WhpResultExt;
use crate::WhpVpRef;
+use hvdef::HvRegisterValue;
use hvdef::HvVtlEntryReason;
use hvdef::HvX64PendingEventReg0;
use hvdef::HvX64PendingInterruptionRegister;
+use hvdef::HvX64RegisterName;
use hvdef::Vtl;
use inspect::Inspect;
use std::sync::atomic::Ordering;
use virt::VpIndex;
use virt::io::CpuIo;
use virt::irqcon::MsiRequest;
+use virt::state::HvRegisterState;
use virt::x86::MsrError;
use virt::x86::vp;
use virt_support_apic::ApicClient;
@@ -34,6 +37,7 @@
use whp::get_registers;
use whp::set_registers;
use x86defs::RFlags;
+use zerocopy::FromZeros;
impl WhpPartitionInner {
pub fn lint(&self, vp_index: VpIndex, vtl: Vtl, index: usize) {
@@ -244,7 +248,36 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
pending_interruption: value.pending_interruption,
};
- self.vp.set_register_state(vtl, &value)?;
+ // The run state lives in `self.state` and the emulated APIC
+ // above, not in the hypervisor. With the APIC emulated in
+ // user mode (`WHvX64LocalApicEmulationModeNone`), WHP denies
+ // writes to the internal activity state register, even of
+ // its current value, so write only the other activity
+ // registers.
+ let mut values = [HvRegisterValue::new_zeroed(); 4];
+ debug_assert_eq!(
+ value.names(),
+ &[
+ HvX64RegisterName::InternalActivityState,
+ HvX64RegisterName::PendingInterruption,
+ HvX64RegisterName::InterruptState,
+ HvX64RegisterName::PendingEvent0,
+ ]
+ );
+ value.get_values(values.iter_mut());
+ let [_, pending_interruption, interrupt_state, pending_event] = values;
+ set_registers!(
+ self.vp.whp(vtl),
+ [
+ (
+ whp::Register64::PendingInterruption,
+ pending_interruption.as_u64()
+ ),
+ (whp::Register64::InterruptState, interrupt_state.as_u64()),
+ (whp::Register128::PendingEvent, pending_event.as_u128()),
+ ]
+ )
+ .for_op("set activity registers")?;
}
LocalApicKind::Offloaded => {
self.vp.set_register_state(vtl, value)?;
Activity