Skip to content

openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160

Closed

Description

@tangruize

When restoring a snapshot into fresh KVM vCPUs, OpenVMM only saves and restores the VP state elements listed in vmm_core/virt/src/x86/vp.rs and the 8 syscall MSRs in VirtualMsrs::names:

    (8, "pat", pat, set_pat, Pat),
    (9, "msrs", virtual_msrs, set_virtual_msrs, VirtualMsrs),
    (10, "drs", debug_regs, set_debug_regs, DebugRegisters),
    (11, "tsc", tsc, set_tsc, Tsc),
    (12, "cet", cet, set_cet, Cet),
    (13, "cet_ss", cet_ss, set_cet_ss, CetSs),
    (14, "tsc_aux", tsc_aux, set_tsc_aux, TscAux),
    // The deadline must be restored after APIC mode and guest TSC.
    (15, "tsc_deadline", tsc_deadline, set_tsc_deadline, TscDeadline),

Beyond IA32_SPEC_CTRL (tracked separately in P0-urgent/run-1-part-1-spec-ctrl), several non-mitigation guest-writable MSRs are neither included in VpSavedState nor carried by VirtualMsrs, causing fresh destination vCPUs to revert to reset/binder defaults after restore:

  • IA32_UMWAIT_CONTROL (0xe1, run-1-part-2): Exposed on profiles advertising WAITPKG (CPUID.(7,0):ECX[5], e.g., Alder Lake). Guest-configured UMWAIT/TPAUSE time limits are lost on restore.
  • Machine-check configuration and bank status (MCG_STATUS, MCi_CTL/STATUS/ADDR/MISC/CTL2, run-1-part-3): Restoring the LAPIC CMCI LVT entry does not restore MCA bank controls or pending error telemetry.
  • Miscellaneous feature/power MSRs (IA32_MISC_ENABLE, MISC_FEATURES_ENABLES, POWER_CTL, AMD VM_HSAVE_PA, run-1-part-4): KvmProcessorBinder::bind reinitializes IA32_MISC_ENABLE to hv1_emulator::x86::MISC_ENABLE on fresh vCPUs, discarding any guest modifications permitted by KVM.
  • IA32_DEBUGCTL (0x1d9, run-1-part-5): Architectural DR0–DR7 registers are saved in DebugRegisters, but IA32_DEBUGCTL (branch-step, LBR, RTM debug controls where supported) is omitted.
  • IA32_BNDCFGS (0xd90, run-1-part-6): Exposed on Skylake-SP (CPUID.(7,0):EBX[14]); Xsave saves MPX bounds registers, but the IA32_BNDCFGS supervisor configuration MSR is omitted.
  • AMD legacy PMU MSRs (PerfEvtSel0–3, PerfCtr0–3, run-1-part-7): Although architectural PMU leaf 0xA is cleared in built-in profiles, AMD legacy PMU MSRs remain accessible unless filtered by KVM and are not saved across restore.

Proposed Fix

For each MSR family above, either:

  1. Add capability-gated optional state elements to VpSavedState (with a defined fallback when restoring older snapshots that lack those fields), or
  2. Filter/disallow guest writes (or hide the corresponding CPUID feature bits, such as WAITPKG or MPX, on snapshot-supported profiles) when OpenVMM intentionally keeps those registers fixed.

Activity

  1. ppenna commented on Nov 17, 2026

    @ppenna
    Contributor

    Correction, from #1182's audit: two of the reasons that #1638 recorded for state without an element, in Guest-Writable Processor State and OpenVMM's Guide, do not hold.

    • IA32_BNDCFGS. It said that KVM has not presented MPX since Linux 5.6, because the kernel does not enable its state components. Linux 5.6 dropped MPX for the kernel's own processes, but still enables MPX's state components for KVM's guests: XFEATURE_MASK_USER_SUPPORTED keeps BNDREGS and BNDCSR, KVM_SUPPORTED_XCR0 includes them, vmx_set_cpu_caps() sets X86_FEATURE_MPX where kvm_mpx_supported() finds them in XCR0, and vmx_set_msr() takes a guest's write of the MSR where its CPUID has MPX. So a guest of the Skylake-SP profile can write it, on every backend that supports the profile.
    • MSR_VM_HSAVE_PA. It said that no guest runs the instructions that use it, since no CPU profile enumerates SVM. That holds for the instructions, but KVM on an AMD host keeps a guest's writes of VM_CR and VM_HSAVE_PA whatever its CPUID says (svm_set_msr()), and keeps the OSVW MSRs for every guest whose CPUID enumerates OSVW, as every AMD profile does, so a restore lost them.

    #1732, with microsoft/openvmm#5134, adds elements for these MSRs (bndcfgs, svm_msrs, and osvw, fields 21-23 of the VP saved state, with no version change), has WHP and MSHV hold the umwait_control element that this issue gave KVM, and corrects the design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions