Repository navigation
[DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots - #5134
Merged
Pedro Henrique Penna (ppenna) merged 4 commits intoNov 17, 2026
Conversation
A guest whose CPUID enumerates WAITPKG writes IA32_UMWAIT_CONTROL, and Linux does so on every CPU at boot, but on WHP and MSHV no element of the VP state held it: both backends cleared the umwait_control capability that from_cpuid derives from CPUID.(7,0):ECX[5], and returned UnsupportedState from its accessors. A restore therefore gave the guest a new VP's value, 0, instead of the one that it had written: its limit on how long UMWAIT and TPAUSE may wait, and whether they may enter C0.2, were gone, while the guest kernel still believed them set. The KVM backend has held the element since microsoft/nvx#1160; the Alder Lake and Granite Rapids CPU profiles enumerate WAITPKG, and the Alder Lake profile comes from WHP alone. The Microsoft hypervisor keeps the MSR as a VP register, as it keeps IA32_SPEC_CTRL, so read and write it through WHvX64RegisterUnwaitControl on WHP and HvX64RegisterName::UmwaitControl on MSHV, and keep the capability that CPUID gives. MSHV clears it for an SNP guest, whose registers the root cannot read, as it does SPEC_CTRL's. The other elements that KVM holds, IA32_DEBUGCTL, the feature and power MSRs, and the machine-check state, stay absent on both: the Microsoft hypervisor drops a guest's writes of IA32_DEBUGCTL and IA32_MISC_ENABLE, raises #GP for IA32_POWER_CTL and MSR_MISC_FEATURES_ENABLES, and gives the guest no machine-check banks, so a guest has nothing there for a snapshot to keep. No version is involved: the element, field 17, exists already. A snapshot that WHP or MSHV captured before this change has no such field, so its restore leaves the MSR at 0, as before; an earlier build restores a later snapshot without it, in the same way. Evidence: on a bare-metal Intel Core i9-12900H (Alder Lake) under WHP (Windows 11, build 26300), with the intel.alderlake.v1 profile, a microVM whose Linux guest read 0x186a0 from the MSR, wrote 0x30d41, and was then captured and restored read 0 with the OpenVMM of c7122cd and reads 0x30d41 with this change. The same driver found no other MSR whose value changes across a snapshot there, nor on nvx-mshv-test (MSHV, Intel Xeon Platinum 8573C, intel.emeraldrapids.v1, which hides WAITPKG): a guest's writes of IA32_DEBUGCTL (each of LBR, BTF, BUS_LOCK_DETECT, the freeze bits, and RTM_DEBUG) and IA32_MISC_ENABLE (each of twelve bits) read back unchanged, IA32_POWER_CTL, MSR_MISC_FEATURES_ENABLES, and the bank registers raise #GP, and IA32_MCG_CAP reports no bank. Tests: the contract test umwait_control_survives_a_save_and_restore, which saves the VP, clears the MSR, and restores the saved state, now runs for WHP on a host with WAITPKG and passes there; before, the backend claimed no capability, so the case reported ignored. No MSHV host with WAITPKG was available; the MSHV path is the same register access as SPEC_CTRL's. Refs microsoft/nvx#1182 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A guest whose CPUID enumerates MPX can write IA32_BNDCFGS, MPX's supervisor-mode configuration: whether bound checks run at CPL 0, whether BNDPRESERVE keeps the bound registers across a branch, and the base of the bound directory. The bound registers and the user-mode configuration are XSAVE state, which a snapshot holds, but no element of the VP state held IA32_BNDCFGS, so a restore gave the guest a new VP's value, 0, with MPX off at CPL 0 whatever the guest had set. The Skylake-SP CPU profile enumerates MPX, and its XCR0 enables MPX's two state components, so its guests can have the MSR on every backend that supports the profile: KVM, MSHV, and WHP all contributed MPX to it. microsoft/nvx#1160 left the MSR out on the ground that KVM has not exposed MPX since Linux 5.6, which is not so. Linux 5.6 removed MPX for the kernel's own processes but kept the XSAVE support of its state components so that KVM guests can use them: XFEATURE_MASK_USER_SUPPORTED still holds BNDREGS and BNDCSR, xstate.c enables them where the CPU has X86_FEATURE_MPX, KVM_SUPPORTED_XCR0 holds them, and vmx_set_cpu_caps sets X86_FEATURE_MPX where kvm_mpx_supported() finds them in XCR0. vmx_set_msr then takes a guest's write of the MSR whenever the guest's CPUID has MPX. Add the MSR as an optional VP state element, Bndcfgs, in field 21, with X86PartitionCapabilities::bndcfgs, which from_cpuid derives from CPUID.(7,0):EBX[14]. KVM, MSHV, and WHP read and write it as a register: KVM as MSR 0xd90, which it lists in KVM_GET_MSR_INDEX_LIST exactly where it exposes MPX, so the capability follows the list, as UMWAIT_CONTROL's does, and is clear for an SNP guest; MSHV as HvX64RegisterName::Bndcfgs, clear for an SNP guest too; and WHP as WHvX64RegisterBndcfgs. OpenHCL does not save the MSR: its capability stays clear and its accessors return Unimplemented. The element needs no version, as the other optional elements need none: a saved state from a build without it has no field 21, so its restore leaves the MSR at 0, as before, and a build without it ignores the field. Evidence: no host available has MPX (an Alder Lake WHP host, Emerald Rapids KVM and MSHV hosts, and the Ice Lake-SP CI runners), so this rests on the Linux sources and on the profile, which the support check of every backend admits only where the backend presents MPX. Tests: the bit that enumerates MPX; the capability for CPUID leaves with and without it, and for every pinned profile, where only intel.skylake-sp.v1 has it; the wire bytes of field 21 and both directions of a build without it; KVM's gating by the host's list and for SNP; KVM's own MSR, where the host's list has it; a contract test, bndcfgs_survives_a_save_and_restore, which runs on a host with MPX; and refuses_unsupported_cpu_state, which now also restores a crafted state with the element into a partition without it. Refs microsoft/nvx#1182 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
requested a review
from a team
as a code owner
November 17, 2026 16:24
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
This was referenced Nov 17, 2026
On an AMD host, KVM keeps four guest-writable MSRs in the vCPU that no element of the VP state held, so a restore gave the guest a new VP's values instead of the ones that it wrote: - VM_CR and VM_HSAVE_PA, SVM's control and host save area MSRs. svm.c takes a guest's writes of both whatever its CPUID says: VM_CR's five emulated bits, with LOCK and SVMDIS, and any page address that the guest has for VM_HSAVE_PA. No CPU profile enumerates SVM, but every guest on an AMD host can write them. - OSVW_ID_LENGTH and OSVW_STATUS, which say which errata a CPU has. KVM takes a guest's writes where the vCPU has OSVW, which every AMD CPU profile enumerates, and gives a new vCPU values that it derives from the host's. Add two optional VP state elements: SvmMsrs, in field 22, with X86PartitionCapabilities::svm_msrs, which from_cpuid leaves clear since no CPUID bit decides it; and Osvw, in field 23, with X86PartitionCapabilities::osvw, which from_cpuid derives from CPUID.0x80000001:ECX[9]. For KVM: - KVM lists VM_HSAVE_PA in KVM_GET_MSR_INDEX_LIST, and not VM_CR, which it emulates beside it, but it lists VM_HSAVE_PA wherever the host can read it: an Intel host under Hyper-V can, and its KVM has no VM_CR. So svm_msrs needs KVM to run on SVM, as the vendor of KVM_GET_SUPPORTED_CPUID shows, as well as VM_HSAVE_PA in the list. KVM takes any VM_HSAVE_PA from the host and keeps only its page address, so a restore refuses one that is not a page address within the guest's physical address width, which no guest can write, before it writes either MSR. It refuses a VM_CR that sets SVMDIS while the restored EFER.SVME is set too, which no guest can hold, and which KVM refuses only after it has set SVMDIS and LOCK, which it then keeps. It writes VM_CR first, whose invalid bits KVM refuses, so that a refused state writes nothing. - KVM lists neither OSVW MSR, and a kernel that keeps a vCPU's features within KVM's own refuses them for a vCPU without OSVW, so osvw also needs KVM_GET_SUPPORTED_CPUID to enumerate OSVW, as it does on an AMD host. A VM reset puts back the values that KVM gave the vCPU at bind, as a reboot's firmware would write them again, rather than the architectural reset value, 0. - Both are clear for an SNP guest. Once a guest sets VM_CR's SVMDIS, KVM keeps SVMDIS and LOCK for the life of the vCPU, so a reset leaves them, as it did before the element, and SvmMsrs does not ask a debug build to compare a reset or restored VP with the state that it set. A restore into a new VP sets both bits as the guest saved them. MSHV, WHP, and OpenHCL hold neither element: on WHP, OpenVMM serves the OSVW MSRs itself, reading 0 and refusing writes, and none of them reads the SVM MSRs, so they clear both capabilities. Neither element needs a version: a saved state from an earlier build holds neither field, so its restore leaves the MSRs as bind set them, and an earlier build ignores both fields. Evidence: the Linux sources (svm_get_msr, svm_set_msr, svm_set_vm_cr, svm_init_osvw, the OSVW cases of kvm_set_msr_common and kvm_get_msr_common, kvm_vcpu_after_set_cpuid, and the CPUID_8000_0001_ECX capabilities), and nvx-kvm-test (an Intel Xeon Platinum 8573C under Hyper-V, Linux 6.18.57), whose KVM lists VM_HSAVE_PA but refuses a read of VM_CR. No AMD host was available, so the AMD paths have not run on hardware here. Tests: the bit that enumerates OSVW, its leaf gating, and the pinned profiles that have it, which are the AMD ones; the wire bytes of fields 22 and 23 and both directions of a build without them; KVM's gating of both, the vendor and OSVW that KVM's supported CPUID gives, and its checks of VM_HSAVE_PA and VM_CR; KVM tests that the host's KVM keeps VM_CR and VM_HSAVE_PA exactly where the capability says, and of both MSR pairs where KVM runs on SVM or supports OSVW; contract tests, svm_msrs_survive_a_save_and_restore, osvw_survives_a_save_and_restore, and a_reset_returns_osvw_to_what_a_new_vp_holds, which run on an AMD host; and refuses_unsupported_cpu_state, which now also restores crafted states with these elements into a partition without them. Refs microsoft/nvx#1182 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The activity of a VP on MSHV or WHP comes from its Hyper-V registers, whose pending interruption can be a software interrupt (INT n), a privileged software exception (INT1), or a software exception (INT3 or INTO) while the hypervisor delivers one: an intercept during the delivery, such as one for the page of the stack that it pushes the frame to, leaves the event pending, with the length of the instruction that raised it. vp::Activity has no variant for these, and its HvRegisterState::set_values panicked on them, as on a pending event other than an exception or an extint and on a vector beyond 8 bits, so a capture at such a moment crashed the VMM instead of failing. Add vp::Activity::from_hv_values, which converts the four registers' values or names what the activity cannot hold, and use it in the activity getters of MSHV and WHP, both of WHP's APIC kinds, which now fail with UnsupportedState instead. A save, and with it a capture, refuses such a VP before it publishes anything: a guest-requested capture rolls back, so the guest continues, takes the event, and can request the capture again. set_values keeps its panic for the other callers, through from_hv_values. KVM is unaffected: it never reports a soft interrupt or exception as injected, and has the guest run the instruction again. The saved-state format does not change. Tests: the registers of every representable interruption and event read back as written; each software type, the types that Hyper-V does not define, the other event types, and vectors beyond 8 bits are refused, and an interruption that is not pending is not. Refs microsoft/nvx#1182 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
copilot/nvx-issue-1182-processor-state
branch
from
November 17, 2026 16:46
59a5970 to
c9755a9
Compare
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 17, 2026
A snapshot did not save every guest-visible piece of processor state that a guest can write, and a capture could crash instead of refusing state that it cannot represent (#1182): - On WHP and MSHV, a guest's IA32_UMWAIT_CONTROL, which Linux writes on every CPU at boot where the CPU profile enumerates WAITPKG, as the Alder Lake and Granite Rapids profiles do. - On every backend, IA32_BNDCFGS, which a guest of the Skylake-SP profile, whose CPUID enumerates MPX, can write: #1160 left it out on the ground that KVM no longer exposes MPX, which Linux's sources contradict. - On KVM on an AMD host, VM_CR and VM_HSAVE_PA, which KVM keeps for any guest, and the OSVW MSRs, which it keeps for every guest of an AMD profile. - On MSHV and WHP, a capture of a VP that the hypervisor stopped while it delivered a software interrupt or exception panicked in the conversion of its pending interruption. Pin c9755a905618feeae5072d74c206a65736406aab, the head of microsoft/openvmm#5134, four commits on 2807bafcf, the head of feature/microvm/dev, where #1392's fix landed: - 07dcf07d7 virt_whp, virt_mshv: save and restore a guest's IA32_UMWAIT_CONTROL - 3587c0111 virt: save and restore a guest's IA32_BNDCFGS - 3c40841df virt, virt_kvm: save and restore AMD's VM_CR, VM_HSAVE_PA, and OSVW - c9755a905 virt_mshv, virt_whp: refuse to save a pending software interrupt WHP and MSHV now read and write IA32_UMWAIT_CONTROL as a VP register, with the existing umwait_control element. The VP state gains three optional elements, each in a field that a saved state from an earlier build lacks and that an earlier build ignores, so no version of the saved state, the snapshot manifest, the time ABI, or any other contract changes, and none is created or bumped: bndcfgs (field 21), where the guest's CPUID enumerates MPX and the backend can access the MSR; svm_msrs (field 22), on KVM where it runs on SVM, as on an AMD host; and osvw (field 23), on KVM where the guest's CPUID and KVM's supported CPUID enumerate OSVW. A restore refuses an element that the partition lacks, as it already did, and a VM_HSAVE_PA that no guest could write, or a VM_CR that sets SVMDIS while EFER.SVME is set, before it writes either SVM MSR. A save on MSHV or WHP refuses a pending software interrupt or exception, a pending event other than an exception or an extint, and a vector beyond 8 bits, with an error, so a guest-requested capture rolls back and the guest continues. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 17, 2026
The NVX pin moves to c9755a905, position 306, the head of microsoft/openvmm#5134, whose four commits, positions 303-306, follow 2807bafcf, the head of feature/microvm/dev, which microsoft/openvmm#5132 merged. The pin is 180 commits past the mapped tip, and no PR maps positions 303-306 yet. Position 303 has WHP and MSHV save and restore IA32_UMWAIT_CONTROL, as a register of the VP, completing position 277's umwait_control element, which KVM alone held. Position 304 adds the bndcfgs element, field 21, for IA32_BNDCFGS where the guest's CPUID enumerates MPX, on every backend, and rewrites position 279's Guide remark that no guest can write the MSR. Position 305 adds the svm_msrs and osvw elements, fields 22 and 23, for VM_CR and VM_HSAVE_PA, which KVM on SVM keeps for every guest, and the OSVW MSRs, which it keeps for a guest whose CPUID enumerates OSVW where it supports OSVW, has a VM reset put back the OSVW values that KVM gave the vCPU, and refuses a VM_HSAVE_PA that no guest can write. Position 306 has MSHV and WHP refuse to save a VP that holds a pending software interrupt or exception, or another interruption or event that the saved activity cannot hold, where the conversion of the activity's registers, which upstream main has, panicked. None of them changes a version. They change no line that a mapped PR adds, though they change files that 13 mapped PRs also change. Replaying each on upstream main, c0694c4da, conflicts. Positions 303-305 edit or add beside what positions 272, 277-279, 283, 295, and 296 add, and each beside its predecessors among them, and position 305 also beside what positions 128 and 248 add, so they follow those positions upstream. Position 306 rewrites only lines that upstream main has, but its conversion builds the fields of the activity that positions 287 and 291 add, and its Guide paragraphs sit beside what positions 071, 285, 289, 292, and 301 write, so it follows those positions upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
merged commit Nov 17, 2026
c9755a9
into
feature/microvm/dev
165 of 173 checks passed
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs microsoft/nvx#1182.
NVX pins the head of this branch in microsoft/nvx#1732, so this is a prerequisite of that pull request, and it lands by fast-forwarding
feature/microvm/devto that exact commit, not with the merge button, which squashes.What and why
microsoft/nvx#1182's audit of what a snapshot keeps of a VP found four gaps: guest-writable processor state that no element of the VP state held, so a restore gave the guest a new VP's value, and a capture that panicked instead of refusing state that the saved activity cannot hold.
IA32_UMWAIT_CONTROLon WHP and MSHV (07dcf07d7). Linux writes it on every CPU at boot where the CPU profile enumerates WAITPKG (Alder Lake, Granite Rapids). KVM has held theumwait_controlelement since openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features nvx#1160, but both of the Microsoft hypervisor's backends cleared its capability, so a restore set 0. The hypervisor keeps the MSR as a VP register, as it keepsIA32_SPEC_CTRL, so WHP and MSHV now read and write it through the register state, keeping the capability that CPUID gives (clear for an SNP guest on MSHV).IA32_BNDCFGS(3587c0111). A guest whose CPUID enumerates MPX (Skylake-SP) can write it on every backend. openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features nvx#1160 left it out on the ground that KVM has not exposed MPX since Linux 5.6, which the Linux sources contradict: Linux 5.6 dropped MPX for the kernel's own processes but still enables MPX's state components for KVM's guests, andvmx_set_msrtakes a guest's write where its CPUID has MPX. New optional elementbndcfgs, field 21; KVM follows its MSR list, as forumwait_control; MSHV and WHP use the register; OpenHCL leaves it unimplemented.VM_CR,VM_HSAVE_PA, and OSVW MSRs on KVM (3c40841df).svm.ckeeps a guest'sVM_CRandVM_HSAVE_PAwhatever its CPUID says, and KVM keeps the OSVW MSRs for a vCPU whose CPUID enumerates OSVW, as every AMD profile does. New optional elementssvm_msrs, field 22, andosvw, field 23.svm_msrsneeds KVM to run on SVM, as the vendor of its supported CPUID shows, as well asVM_HSAVE_PAinKVM_GET_MSR_INDEX_LIST: an Intel host under Hyper-V listsVM_HSAVE_PAbut has noVM_CR.osvwneeds KVM's supported CPUID to enumerate OSVW, since a recent kernel keeps a vCPU's features within KVM's own. A restore refuses aVM_HSAVE_PAthat no guest could write, or aVM_CRthat setsSVMDISwhile the restoredEFER.SVMEis set, which KVM refuses only after it has setSVMDISandLOCK, before it writes either MSR, and a VM reset puts back the OSVW values that KVM gave the vCPU at bind. MSHV, WHP, and OpenHCL clear both capabilities.c9755a905). While an intercept interrupts the delivery ofINT n,INT1,INT3, orINTO, the hypervisor holds the event pending with the length of its instruction, which the saved activity cannot hold, and the activity'sHvRegisterState::set_valuespanicked on it (and on an event other than an exception or an extint, or a vector beyond 8 bits).Activity::from_hv_valuesnow names what it cannot hold, and the activity getters of MSHV and WHP fail the save withUnsupportedState, so a guest-requested capture rolls back. KVM never reports such an event: it has the guest run the instruction again.No saved-state format, snapshot manifest, CPU profile, or other version changes. The new state lives in optional fields 21-23 of
virt.x86.VpSavedState, the next free numbers below 100 after the chain's 16-20 (upstream main uses 1-15, 100-104, and 200): a saved state from an earlier build lacks them, so its restore leaves the MSRs asbindset them, as before, and an earlier build ignores them. The fields are in the per-VP saved state, not inopenvmm.snapshot.SnapshotManifest, whose field 18 (snapshot_id) and reserved fields 19 and 20 they cannot collide with.Known limits, documented in the Guide: once a guest sets
VM_CR.SVMDIS, KVM keepsSVMDISandLOCKfor the life of the vCPU, so a VM reset leaves them set, as before (a restore into a new VP sets them exactly). AMD's legacy performance counters still have no element, since a time ABI guest has no PMU.Validation
At this head,
c9755a905, four commits on2807bafcf, the head offeature/microvm/dev, with Rust 1.97.1. Its last change, after review, added only theVM_CRcheck to KVM'sset_svm_msrs, so the KVM unit and contract tests ran again at this head (virt_kvm: 154 passed, includinga_restore_refuses_svmdis_while_efer_enables_svm; contract tests: 32 passed), and the other results are from59a5970e6, which differs from it only there:cargo nextest run -p virt -p x86defson Windows (177 passed);-p virt -p virt_kvm --run-ignored allagainst/dev/kvmon an Intel Xeon Platinum 8573C KVM host, Linux 6.18.57 (303 passed, including the newkvm_testsfor BNDCFGS, the SVM MSRs, and OSVW, which skip what that host lacks);-p virt -p virt_mshvon an MSHV host with the same CPU (233 passed).cargo nextest run -p vmm_tests --test virt_backend_tests: WHP on an Intel Core i9-12900H (Alder Lake, WAITPKG), 26 passed and 10 skipped, includingumwait_control_survives_a_save_and_restore, which reported ignored before; KVM, 32 passed and 44 skipped; MSHV, 15 passed and 61 skipped.refuses_unsupported_cpu_statenow also restores crafted states with fields 21-23 into partitions that lack them, on all three.cargo xtask fmt(twice, stable);cargo clippy --all-targets -- -D warningsforvirt,x86defs,virt_whp, andvmm_testson Windows andvirt_kvm,virt_mshv, andvmm_testsforx86_64-unknown-linux-musl;cargo checkofvirt_mshv_vtlfor musl;cargo docwith-D warnings; every commit builds on its own.IA32_UMWAIT_CONTROLread 0 after a snapshot and restore withc7122cd20, and reads 0x30d41 with this branch; every other MSR that the guest can access reads back unchanged. NVX's microVM snapshot scenarios (snapshot-core,smp-snapshot,restore-processors,snapshot-rollback,snapshot-tiers,console-snapshot) pass on WHP, KVM, and MSHV.Not tested here
x64-linux-amd-kvmjob runs the newsvm_msrsandosvwcontract tests, which need an AMD host.bndcfgscontract test runs only on one.IA32_SPEC_CTRL.