Repository navigation
openvmm: pin the processor state that snapshots lost - #1732
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 17, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Processor-state persistence changes span three hypervisors, with AMD, MPX, and MSHV WAITPKG validation gaps requiring maintainer review before landing.
No open issues 🎉
What changed in this PR
This PR updates NVX’s OpenVMM pin to address remaining processor-state snapshot gaps tracked by #1182 and documents the resulting backend behavior without changing snapshot or ABI versions.
Changes:
- Documents preservation of user-wait, MPX, and AMD processor registers across snapshots.
- Describes capture refusal and rollback for software interrupts or events that MSHV and WHP cannot represent.
- Maps the four OpenVMM commits at positions 303–306 and updates the recorded pin.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Updates the pin and maps the four commits and their upstream dependencies. |
doc/design/2.2-cpu-and-topology.md |
Clarifies register preservation by backend and corrects prior MPX and SVM claims. |
doc/design/1.5-compatibility-and-versioning-model.md |
Expands the saved-state inventory and documents unsupported-event capture refusal. |
Review effort: Balanced
A snapshot did not save every guest-visible piece of processor state that a guest can write, and a capture could crash instead of refusing state that it cannot represent (#1182): - On WHP and MSHV, a guest's IA32_UMWAIT_CONTROL, which Linux writes on every CPU at boot where the CPU profile enumerates WAITPKG, as the Alder Lake and Granite Rapids profiles do. - On every backend, IA32_BNDCFGS, which a guest of the Skylake-SP profile, whose CPUID enumerates MPX, can write: #1160 left it out on the ground that KVM no longer exposes MPX, which Linux's sources contradict. - On KVM on an AMD host, VM_CR and VM_HSAVE_PA, which KVM keeps for any guest, and the OSVW MSRs, which it keeps for every guest of an AMD profile. - On MSHV and WHP, a capture of a VP that the hypervisor stopped while it delivered a software interrupt or exception panicked in the conversion of its pending interruption. Pin c9755a905618feeae5072d74c206a65736406aab, the head of microsoft/openvmm#5134, four commits on 2807bafcf, the head of feature/microvm/dev, where #1392's fix landed: - 07dcf07d7 virt_whp, virt_mshv: save and restore a guest's IA32_UMWAIT_CONTROL - 3587c0111 virt: save and restore a guest's IA32_BNDCFGS - 3c40841df virt, virt_kvm: save and restore AMD's VM_CR, VM_HSAVE_PA, and OSVW - c9755a905 virt_mshv, virt_whp: refuse to save a pending software interrupt WHP and MSHV now read and write IA32_UMWAIT_CONTROL as a VP register, with the existing umwait_control element. The VP state gains three optional elements, each in a field that a saved state from an earlier build lacks and that an earlier build ignores, so no version of the saved state, the snapshot manifest, the time ABI, or any other contract changes, and none is created or bumped: bndcfgs (field 21), where the guest's CPUID enumerates MPX and the backend can access the MSR; svm_msrs (field 22), on KVM where it runs on SVM, as on an AMD host; and osvw (field 23), on KVM where the guest's CPUID and KVM's supported CPUID enumerate OSVW. A restore refuses an element that the partition lacks, as it already did, and a VM_HSAVE_PA that no guest could write, or a VM_CR that sets SVMDIS while EFER.SVME is set, before it writes either SVM MSR. A save on MSHV or WHP refuses a pending software interrupt or exception, a pending event other than an exception or an extint, and a vector beyond 8 bits, with an error, so a guest-requested capture rolls back and the guest continues. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Guest-Writable Processor State (2.2.1.4) said that only KVM holds the guest-writable MSR elements and that MSHV and WHP hold none, and gave IA32_BNDCFGS and MSR_VM_HSAVE_PA as state that no guest can write: KVM had not presented MPX since Linux 5.6, it said, and no profile enumerates SVM. #1160 recorded both, and neither held: Linux 5.6 dropped MPX for the kernel's own processes but still enables MPX's state components for KVM's guests, so a KVM guest of the Skylake-SP profile writes IA32_BNDCFGS, and KVM on an AMD host keeps a guest's VM_CR and VM_HSAVE_PA whatever its CPUID says, and its OSVW MSRs for every guest of an AMD profile. The OpenVMM pin now saves and restores IA32_UMWAIT_CONTROL and IA32_BNDCFGS on MSHV and WHP, IA32_BNDCFGS on KVM, and AMD's VM_CR, VM_HSAVE_PA, OSVW_ID_LENGTH, and OSVW_STATUS on KVM on an AMD host. Say which backend holds each element and why the Microsoft hypervisor's backends hold no others, that a restore refuses a VM_HSAVE_PA or VM_CR that no guest can hold, and that a VM reset gives a processor back KVM's OSVW values. Name the new elements in Processor State Elements (1.5.4), and say there that MSHV and WHP refuse to capture a processor that the hypervisor stopped while it delivered a software interrupt or exception, which the saved activity cannot hold, and that a guest-requested capture then rolls back. Refs #1182 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c9755a905, position 306, the head of microsoft/openvmm#5134, whose four commits, positions 303-306, follow 2807bafcf, the head of feature/microvm/dev, which microsoft/openvmm#5132 merged. The pin is 180 commits past the mapped tip, and no PR maps positions 303-306 yet. Position 303 has WHP and MSHV save and restore IA32_UMWAIT_CONTROL, as a register of the VP, completing position 277's umwait_control element, which KVM alone held. Position 304 adds the bndcfgs element, field 21, for IA32_BNDCFGS where the guest's CPUID enumerates MPX, on every backend, and rewrites position 279's Guide remark that no guest can write the MSR. Position 305 adds the svm_msrs and osvw elements, fields 22 and 23, for VM_CR and VM_HSAVE_PA, which KVM on SVM keeps for every guest, and the OSVW MSRs, which it keeps for a guest whose CPUID enumerates OSVW where it supports OSVW, has a VM reset put back the OSVW values that KVM gave the vCPU, and refuses a VM_HSAVE_PA that no guest can write. Position 306 has MSHV and WHP refuse to save a VP that holds a pending software interrupt or exception, or another interruption or event that the saved activity cannot hold, where the conversion of the activity's registers, which upstream main has, panicked. None of them changes a version. They change no line that a mapped PR adds, though they change files that 13 mapped PRs also change. Replaying each on upstream main, c0694c4da, conflicts. Positions 303-305 edit or add beside what positions 272, 277-279, 283, 295, and 296 add, and each beside its predecessors among them, and position 305 also beside what positions 128 and 248 add, so they follow those positions upstream. Position 306 rewrites only lines that upstream main has, but its conversion builds the fields of the activity that positions 287 and 291 add, and its Guide paragraphs sit beside what positions 071, 285, 289, 292, and 301 write, so it follows those positions upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/issue-1182-processor-state
branch
from
November 17, 2026 16:49
a836ef9 to
b37d8b1
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-hypervisor snapshot-state changes lack MPX and MSHV WAITPKG runtime validation, while AMD and exact-head CI evidence remains pending, requiring maintainer review of the remaining platform risk.
No open issues 🎉
Review effort: Balanced
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1182-processor-state
branch
November 17, 2026 20:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1182
What does this pull request change?
#1182's 24 children are closed: 23 as completed, with their fixes on
devand in the OpenVMM pin, and #1389 as not planned, since its cause is in Linux (KVM reports a fixed MXCSR in the XSAVE area on hosts without XSAVEC or XSAVES, and no KVM interface reports the guest's; see its closing comment). Checking the umbrella's objective against the pin found four gaps, which microsoft/openvmm#5134 closes. This pull request pins it, updates the design, and maps it in the OpenVMM roadmap. Start with the first commit's message and microsoft/openvmm#5134's description.openvmm: pin the processor state that snapshots lostmoves theopenvmmgitlink, and nothing else, from2807bafcf, the head offeature/microvm/dev, toc9755a905, the head of [DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots openvmm#5134, four commits on it:07dcf07d7: WHP and MSHV save and restore a guest'sIA32_UMWAIT_CONTROL, which Linux writes on every CPU at boot under the Alder Lake and Granite Rapids profiles, as a register of the VP, in the existingumwait_controlelement. Both backends had cleared its capability, so a restore set 0.3587c0111: a new element,bndcfgs, holdsIA32_BNDCFGSwhere the guest's CPUID enumerates MPX (Skylake-SP), on KVM where the host lists the MSR, and on MSHV and WHP.3c40841df: new elements on KVM:svm_msrsholds AMD'sVM_CRandVM_HSAVE_PA, which KVM on SVM keeps for every guest, andosvwholds the OSVW MSRs, which it keeps for a guest whose CPUID enumerates OSVW (every AMD profile). A restore refuses aVM_HSAVE_PAthat no guest could write, or aVM_CRthat setsSVMDISwhile the restoredEFER.SVMEis set, before it writes either MSR, and a VM reset puts back the OSVW values that KVM gave the vCPU.c9755a905: MSHV and WHP refuse to save a VP that holds a pending software interrupt or exception (INT n,INT1,INT3,INTO), or another interruption or event that the saved activity cannot hold, instead of panicking; a guest-requested capture rolls back.doc: describe the processor state that snapshots keep on each backendupdates Guest-Writable Processor State (2.2.1.4) and Processor State Elements (1.5.4), and corrects two claims that openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160 recorded there: that KVM has not presented MPX since Linux 5.6, so no guest can writeIA32_BNDCFGS, and that no guest runs the instructions that useMSR_VM_HSAVE_PA.doc: map the processor state of snapshots in the OpenVMM roadmapmaps positions 303-306.No snapshot, CPU-profile, or ABI version changed: no manifest version or magic, time ABI, restore packet, machine contract, or saved-state schema version. The new state uses optional fields 21-23 of
virt.x86.VpSavedState, the per-VP saved state thatstate.bincarries: a snapshot from an earlier build lacks them, so its restore leaves those MSRs as before, and an earlier build ignores them. They cannot collide with #1210's snapshot-ID field map, which is in another message,openvmm.snapshot.SnapshotManifest(field 18snapshot_id, fields 19 and 20 reserved for upstream), and protobuf numbers a field within its message. NVX decodes no part ofVpSavedStateand reserves no range in it; there, upstream main uses fields 1-15, 100-104, and 200, and the chain's earlier positions 16-20, so 21-23 are the next free numbers below 100, as the comment above the element list prescribes.The children
dev(merge commit of the closing pull request)c73872117ed0b1029a5c37f7bb375818f5157027149c4bfc5f887fa3b8f1b54b5asysenter_eipholds SYSENTER_ESPae35cbd896ef7f4cde335cset_xsavekeeps old PKRU or MXCSR after a resetac846abb2c69e41ebd10bead43631299b0b792e825e7999cc213aa852da532f5e3b67d11964c7237085f36aa9dc3dd7060641119c749d626fa08b45646600c5cfe9cb66b544edf432508f76abccli; hltruns past the HLT on WHP5a955324fc9Every merge commit is an ancestor of
origin/dev. The v7.0 reorganization rewrote the OpenVMM chain after most of these pins, so their OpenVMM commits are not ancestors of the pin by SHA; I matched each to the chain commit that carries it now: 37 have the same patch ID, six differ only in the rustfmt formatting of test code (git range-diff), and the commits of #1408 and #1474 are in the pin as they are.Left out
VM_CRorVM_HSAVE_PA. Whether the Microsoft hypervisor keeps a guest's writes of them is unverified, since no AMD host was available. On WHP, OpenVMM serves the OSVW MSRs itself, reading 0 and refusing writes.VM_CRon KVM. Once a guest setsSVMDIS, KVM keepsSVMDISandLOCKfor the life of the vCPU, so a VM reset leaves them set, as it did before; a restore into a new VP sets them as saved. The Guide says so.Why is this change needed?
#1182's objective is that a capture saves every guest-visible piece of processor state or refuses what it cannot represent, a restore writes all of it, and a restore rejects unsupported or malformed state with an error before it changes the partition. Against the pin, four gaps broke it:
IA32_UMWAIT_CONTROL. Position 277 of the OpenVMM chain (openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160) gave KVM the element and had both of the Microsoft hypervisor's backends clear it. On a bare-metal Alder Lake WHP host underintel.alderlake.v1, a guest that wrote 0x30d41 read 0 after a snapshot and restore.IA32_BNDCFGS. openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160 left it out on the ground that KVM has not presented MPX since Linux 5.6, which Linux's sources contradict: 5.6 dropped MPX for the kernel's own processes but still enables MPX's state components for KVM guests (XFEATURE_MASK_USER_SUPPORTED,KVM_SUPPORTED_XCR0,kvm_mpx_supported()), andvmx_set_msrtakes a guest's write where its CPUID has MPX. The Skylake-SP profile's CPUID and XCR0 enable MPX on all three backends.svm_set_msrtakes a guest'sVM_CRandVM_HSAVE_PAwithout a CPUID check, andkvm_set_msr_commontakes the OSVW MSRs for a vCPU with OSVW, which every AMD profile enumerates.Alternatives that I rejected:
svm_msrson KVM's MSR list alone. KVM listsVM_HSAVE_PAwherever the host can read it, which an Intel host under Hyper-V can, and its KVM has noVM_CR: on the KVM test host, that gate made every save and reset fail. The capability also needs KVM to run on SVM, as the vendor of its supported CPUID shows.VM_CRwhoseSVMDISKVM keeps. It would make a VM reset fail for a guest that set the bit, where it now behaves as before; the element instead asks no debug comparison ofVM_CRafter a reset.IA32_MISC_ENABLE. It would change what every AMD KVM guest sees at boot, so a reset puts back KVM's values instead.How was this change validated?
Checks and results:
OpenVMM, at
c9755a905, with Rust 1.97.1 (details in microsoft/openvmm#5134). The last change to it, after review, added only aVM_CRcheck to KVM'sset_svm_msrs, so the KVM unit and contract tests ran again atc9755a905, with the same results, and the other results below are from59a5970e6, which differs only there:cargo nextest run -p virt -p x86defson Windows: 177 passed.cargo nextest run -p virt -p virt_kvm --run-ignored allon the KVM host: 303 passed, including the new tests against/dev/kvm, which check that the host keepsVM_CRandVM_HSAVE_PAexactly where the capability says.cargo nextest run -p virt -p virt_mshvon the MSHV host: 233 passed.cargo nextest run -p vmm_tests --test virt_backend_tests: WHP 26 passed and 10 skipped, includingumwait_control_survives_a_save_and_restore, which reported ignored before; KVM 32 passed and 44 skipped; MSHV 15 passed and 61 skipped.refuses_unsupported_cpu_staterestores crafted states with fields 21-23 into partitions that lack them on all three.cargo xtask fmt,cargo clippy --all-targets -- -D warnings(Windows andx86_64-unknown-linux-musl),cargo check -p virt_mshv_vtlfor musl, andcargo docwith-D warningspass, and every commit builds on its own./dev/cpu/0/msr, captures, and restores in a new OpenVMM process:IA32_UMWAIT_CONTROLread 0 after the restore withc7122cd20and reads 0x30d41 withc9755a905; every other MSR that the guest can access reads back unchanged.python3 scripts/nvx.py test-microvm --backend {whp,kvm,mshv} --scenario snapshot-core --scenario smp-snapshot --scenario restore-processors --scenario snapshot-rollback --scenario snapshot-tiers --scenario console-snapshot: passed on all three, with this branch's pin and the OpenVMM built from it.NVX, at this branch's head:
python scripts/nvx.py verify: the source tree and submodule metadata are consistent.cargo test --locked --manifest-path nvx/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored: 4 passed;--test openvmm_cpu_profiles -- --ignored: 1 passed. No vector, excerpt, version, or CPU profile thatnvxcopies changed.python scripts/openvmm_pin.py check-branch --topic --target origin/devandcheck-roadmap --worktree: passed.python scripts/check_markdown_links.py: passed.python scripts/local_gates.py --base origin/dev --run:test-changes,markdown-links,test-layout,openvmm-vectors, andopenvmm-cpu-profilespassed.nvx-clireported one error, intest_private_network_ingress_script_checks_policy_and_traffic, while four gate runs shared the machine;python -m unittest scripts.test_microvm_tests.MicrovmTests.test_private_network_ingress_script_checks_policy_and_trafficpasses alone.openvmm-ciwaited for [DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots openvmm#5134's CI, below.Backend, host OS, host type, and guest:
intel.alderlake.v1, Alpine.CI:
c9755a905passed every job butrun vmm-tests [x64-windows-amd-snp], which fails on every pull request into the branch, OpenVMM/microVM umbrella: network egress, ingress, and identity #1392's at2807bafcfincluded, and which the merge does not wait for. Itsx64-linux-amd-kvmjob, on an AMD host, which no host here has, passedsvm_msrs_survive_a_save_and_restore,osvw_survives_a_save_and_restore,a_reset_returns_osvw_to_what_a_new_vp_holds, andrefuses_unsupported_cpu_state(144 passed, 67 skipped). Two jobs passed on their one rerun:aarch64-windows, whose first attempt could not download its artifacts (HTTP 503: Egress is over the account limit), andx64-windows-intel-tdx, whose first attempt timed out inmemstat::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_memory_validation_debug_small, an OpenHCL test that this change does not reach, since OpenHCL clears the new capabilities.b37d8b1fb2epassed.nvx / ubuntu-latestpassed on its one rerun; its first attempt failed inhost::vm::tests::spawn_resolves_relative_runtime_and_block_inputs_outside_the_vm_directorywithText file busy, the fixture flake that nvx: make Linux copy/exec and deadline fixtures deterministic #1319 tracks, innvx/, which this change does not touch.Not run anywhere:
bndcfgs_survives_a_save_and_restore, which needs a host with MPX, and MSHV with WAITPKG.Checklist
python3 scripts/check_test_layout.pychecks.python3 scripts/check_test_changes.py --base origin/devlists no changeto an existing test, or a Changes to existing tests section below
explains each one.
.github/copilot-instructions.mdand the skills when a convention orworkflow changes.
nvxupdatesnvx-rs/public-api/withpython3 scripts/public_api.py --update.description says how the change versions or migrates them.
OpenVMM pin
c9755a905618feeae5072d74c206a65736406aab, with a good signature (python scripts/openvmm_pin.py source).2807bafcf, the head offeature/microvm/dev, where [DO NOT REVIEW] microVM: refuse a decoded network identity other than the derived one openvmm#5132 (OpenVMM/microVM umbrella: network egress, ingress, and identity #1392) merged; no other prerequisite.feature/microvm/devfast-forwarded from2807bafcftoc9755a905, not with the merge button, which squashes. This pull request merges next.