Skip to content

openvmm: pin the processor state that snapshots lost - #1732

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1182-processor-state
Nov 17, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1182-processor-state

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1182

What does this pull request change?

#1182's 24 children are closed: 23 as completed, with their fixes on dev and in the OpenVMM pin, and #1389 as not planned, since its cause is in Linux (KVM reports a fixed MXCSR in the XSAVE area on hosts without XSAVEC or XSAVES, and no KVM interface reports the guest's; see its closing comment). Checking the umbrella's objective against the pin found four gaps, which microsoft/openvmm#5134 closes. This pull request pins it, updates the design, and maps it in the OpenVMM roadmap. Start with the first commit's message and microsoft/openvmm#5134's description.

  • openvmm: pin the processor state that snapshots lost moves the openvmm gitlink, and nothing else, from 2807bafcf, the head of feature/microvm/dev, to c9755a905, the head of [DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots openvmm#5134, four commits on it:
    • 07dcf07d7: WHP and MSHV save and restore a guest's IA32_UMWAIT_CONTROL, which Linux writes on every CPU at boot under the Alder Lake and Granite Rapids profiles, as a register of the VP, in the existing umwait_control element. Both backends had cleared its capability, so a restore set 0.
    • 3587c0111: a new element, bndcfgs, holds IA32_BNDCFGS where the guest's CPUID enumerates MPX (Skylake-SP), on KVM where the host lists the MSR, and on MSHV and WHP.
    • 3c40841df: new elements on KVM: svm_msrs holds AMD's VM_CR and VM_HSAVE_PA, which KVM on SVM keeps for every guest, and osvw holds the OSVW MSRs, which it keeps for a guest whose CPUID enumerates OSVW (every AMD profile). A restore refuses a VM_HSAVE_PA that no guest could write, or a VM_CR that sets SVMDIS while the restored EFER.SVME is set, before it writes either MSR, and a VM reset puts back the OSVW values that KVM gave the vCPU.
    • c9755a905: MSHV and WHP refuse to save a VP that holds a pending software interrupt or exception (INT n, INT1, INT3, INTO), or another interruption or event that the saved activity cannot hold, instead of panicking; a guest-requested capture rolls back.
  • doc: describe the processor state that snapshots keep on each backend updates Guest-Writable Processor State (2.2.1.4) and Processor State Elements (1.5.4), and corrects two claims that openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160 recorded there: that KVM has not presented MPX since Linux 5.6, so no guest can write IA32_BNDCFGS, and that no guest runs the instructions that use MSR_VM_HSAVE_PA.
  • doc: map the processor state of snapshots in the OpenVMM roadmap maps positions 303-306.

No snapshot, CPU-profile, or ABI version changed: no manifest version or magic, time ABI, restore packet, machine contract, or saved-state schema version. The new state uses optional fields 21-23 of virt.x86.VpSavedState, the per-VP saved state that state.bin carries: a snapshot from an earlier build lacks them, so its restore leaves those MSRs as before, and an earlier build ignores them. They cannot collide with #1210's snapshot-ID field map, which is in another message, openvmm.snapshot.SnapshotManifest (field 18 snapshot_id, fields 19 and 20 reserved for upstream), and protobuf numbers a field within its message. NVX decodes no part of VpSavedState and reserves no range in it; there, upstream main uses fields 1-15, 100-104, and 200, and the chain's earlier positions 16-20, so 21-23 are the next free numbers below 100, as the comment above the element list prescribes.

The children

Child Closed Fix on dev (merge commit of the closing pull request)
#1357, a host profile can give the guest VMX or SVM completed, #1585 c73872117ed
#1382, a restored profile's XSAVE layout overflows length math completed, #1586 0b1029a5c37
#1358, a snapshot's EFER.SVME gives SVM to a KVM guest completed, #1587 f7bb375818f
#1141, KVM restore loses SPEC_CTRL completed, #1637 5157027149c
#1160, KVM VP snapshots omit guest-writable MSRs completed, #1638 4bfc5f887fa
#1368, an Intel profile's guest writes AMD HWCR completed, #1639 3b8f1b54b5a
#1367, sysenter_eip holds SYSENTER_ESP completed, #1588 ae35cbd896e
#1359, PKRU lost with task PKRU 0 completed, #1640 f7f4cde335c
#1360, set_xsave keeps old PKRU or MXCSR after a reset completed, #1641 ac846abb2c6
#1389, MXCSR lost on hosts without XSAVEC or XSAVES not planned (Linux) none
#1380, an XSAVE component outside the caps completed, #1589 9e41ebd10be
#1381, a short or malformed XSAVE area panics restore completed, #1642 ad43631299b
#1365, a restored non-Fixed APIC timer completed, #1590 0b792e825e7
#1171, reserved REMOTE_READ delivery completed, #1643 999cc213aa8
#1158, pending SMI and SMM state lost completed, #1644 52da532f5e3
#1408, a guest's SMI kills the VM completed, #1698 b67d11964c7
#1391, vGIF lets a guest without SVM clear GIF completed, #1592 237085f36aa
#1161, two pending NMIs collapse into one completed, #1645 9dc3dd70606
#1169, pending exceptions dropped, debug panics completed, #1646 41119c749d6
#1385, an injected #BP or #OF completed, #1594 26fa08b4564
#1369, the interrupt-shadow kind completed, #1647 6600c5cfe9c
#1150, capture mixes state from before and after INIT or SIPI completed, #1595 b66b544edf4
#1168, unsupported state panics restore after changing the partition completed, #1648 32508f76abc
#1474, a VP saved in cli; hlt runs past the HLT on WHP completed, #1682 5a955324fc9

Every merge commit is an ancestor of origin/dev. The v7.0 reorganization rewrote the OpenVMM chain after most of these pins, so their OpenVMM commits are not ancestors of the pin by SHA; I matched each to the chain commit that carries it now: 37 have the same patch ID, six differ only in the rustfmt formatting of test code (git range-diff), and the commits of #1408 and #1474 are in the pin as they are.

Left out

  • WHP and MSHV on AMD. Neither holds VM_CR or VM_HSAVE_PA. Whether the Microsoft hypervisor keeps a guest's writes of them is unverified, since no AMD host was available. On WHP, OpenVMM serves the OSVW MSRs itself, reading 0 and refusing writes.
  • VM_CR on KVM. Once a guest sets SVMDIS, KVM keeps SVMDIS and LOCK for the life of the vCPU, so a VM reset leaves them set, as it did before; a restore into a new VP sets them as saved. The Guide says so.
  • AMD's legacy performance counters still have no element, since a time ABI guest has no PMU and the VMM raises #GP for them.
  • openvmm: KVM snapshots lose MXCSR on hosts without XSAVEC or XSAVES #1389 remains a Linux limitation.

Why is this change needed?

#1182's objective is that a capture saves every guest-visible piece of processor state or refuses what it cannot represent, a restore writes all of it, and a restore rejects unsupported or malformed state with an error before it changes the partition. Against the pin, four gaps broke it:

  • WHP and MSHV dropped IA32_UMWAIT_CONTROL. Position 277 of the OpenVMM chain (openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160) gave KVM the element and had both of the Microsoft hypervisor's backends clear it. On a bare-metal Alder Lake WHP host under intel.alderlake.v1, a guest that wrote 0x30d41 read 0 after a snapshot and restore.
  • No element held IA32_BNDCFGS. openvmm: KVM VP snapshots omit guest-writable MSRs across several CPU features #1160 left it out on the ground that KVM has not presented MPX since Linux 5.6, which Linux's sources contradict: 5.6 dropped MPX for the kernel's own processes but still enables MPX's state components for KVM guests (XFEATURE_MASK_USER_SUPPORTED, KVM_SUPPORTED_XCR0, kvm_mpx_supported()), and vmx_set_msr takes a guest's write where its CPUID has MPX. The Skylake-SP profile's CPUID and XCR0 enable MPX on all three backends.
  • KVM on an AMD host kept four guest-writable MSRs that no element held. svm_set_msr takes a guest's VM_CR and VM_HSAVE_PA without a CPUID check, and kvm_set_msr_common takes the OSVW MSRs for a vCPU with OSVW, which every AMD profile enumerates.
  • A capture on MSHV or WHP could panic. The activity's conversion from Hyper-V registers, which upstream main has, panicked on a pending software interrupt or exception, which the hypervisor holds with its instruction length while an intercept interrupts the delivery, and on any other interruption or event that the activity has no variant for.

Alternatives that I rejected:

  • Saving the software interrupt. The activity has no variant for it; adding one would give earlier builds a saved state that they cannot decode, which is a format change. Refusing keeps the format, and a guest-requested capture rolls back and can be requested again. KVM has the guest run the instruction again, which MSHV and WHP cannot.
  • Gating svm_msrs on KVM's MSR list alone. KVM lists VM_HSAVE_PA wherever the host can read it, which an Intel host under Hyper-V can, and its KVM has no VM_CR: on the KVM test host, that gate made every save and reset fail. The capability also needs KVM to run on SVM, as the vendor of its supported CPUID shows.
  • Refusing to reset or restore a VM_CR whose SVMDIS KVM keeps. It would make a VM reset fail for a guest that set the bit, where it now behaves as before; the element instead asks no debug comparison of VM_CR after a reset.
  • Writing OSVW's reset value, 0, at bind, as the VMM does for IA32_MISC_ENABLE. It would change what every AMD KVM guest sees at boot, so a reset puts back KVM's values instead.

How was this change validated?

Checks and results:

OpenVMM, at c9755a905, with Rust 1.97.1 (details in microsoft/openvmm#5134). The last change to it, after review, added only a VM_CR check to KVM's set_svm_msrs, so the KVM unit and contract tests ran again at c9755a905, with the same results, and the other results below are from 59a5970e6, which differs only there:

  • cargo nextest run -p virt -p x86defs on Windows: 177 passed. cargo nextest run -p virt -p virt_kvm --run-ignored all on the KVM host: 303 passed, including the new tests against /dev/kvm, which check that the host keeps VM_CR and VM_HSAVE_PA exactly where the capability says. cargo nextest run -p virt -p virt_mshv on the MSHV host: 233 passed.
  • cargo nextest run -p vmm_tests --test virt_backend_tests: WHP 26 passed and 10 skipped, including umwait_control_survives_a_save_and_restore, which reported ignored before; KVM 32 passed and 44 skipped; MSHV 15 passed and 61 skipped. refuses_unsupported_cpu_state restores crafted states with fields 21-23 into partitions that lack them on all three.
  • cargo xtask fmt, cargo clippy --all-targets -- -D warnings (Windows and x86_64-unknown-linux-musl), cargo check -p virt_mshv_vtl for musl, and cargo doc with -D warnings pass, and every commit builds on its own.
  • On the WHP host, a scratch driver that boots the Alpine guest, writes MSRs through /dev/cpu/0/msr, captures, and restores in a new OpenVMM process: IA32_UMWAIT_CONTROL read 0 after the restore with c7122cd20 and reads 0x30d41 with c9755a905; every other MSR that the guest can access reads back unchanged.
  • python3 scripts/nvx.py test-microvm --backend {whp,kvm,mshv} --scenario snapshot-core --scenario smp-snapshot --scenario restore-processors --scenario snapshot-rollback --scenario snapshot-tiers --scenario console-snapshot: passed on all three, with this branch's pin and the OpenVMM built from it.

NVX, at this branch's head:

  • python scripts/nvx.py verify: the source tree and submodule metadata are consistent.
  • cargo test --locked --manifest-path nvx/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored: 4 passed; --test openvmm_cpu_profiles -- --ignored: 1 passed. No vector, excerpt, version, or CPU profile that nvx copies changed.
  • python scripts/openvmm_pin.py check-branch --topic --target origin/dev and check-roadmap --worktree: passed. python scripts/check_markdown_links.py: passed.
  • python scripts/local_gates.py --base origin/dev --run: test-changes, markdown-links, test-layout, openvmm-vectors, and openvmm-cpu-profiles passed. nvx-cli reported one error, in test_private_network_ingress_script_checks_policy_and_traffic, while four gate runs shared the machine; python -m unittest scripts.test_microvm_tests.MicrovmTests.test_private_network_ingress_script_checks_policy_and_traffic passes alone. openvmm-ci waited for [DO NOT REVIEW] virt: keep UMWAIT_CONTROL, BNDCFGS, and AMD's SVM and OSVW MSRs across snapshots openvmm#5134's CI, below.

Backend, host OS, host type, and guest:

  • WHP: Windows 11 (build 26300), bare metal, Intel Core i9-12900H (Alder Lake), intel.alderlake.v1, Alpine.
  • KVM: Azure Linux 3.0 with Linux 6.18.57, virtual machine, Intel Xeon Platinum 8573C, Alpine.
  • MSHV: Azure Linux 3.0 with the MSHV kernel, virtual machine, Intel Xeon Platinum 8573C, Alpine.

CI:

  • OpenVMM's CI of c9755a905 passed every job but run vmm-tests [x64-windows-amd-snp], which fails on every pull request into the branch, OpenVMM/microVM umbrella: network egress, ingress, and identity #1392's at 2807bafcf included, and which the merge does not wait for. Its x64-linux-amd-kvm job, on an AMD host, which no host here has, passed svm_msrs_survive_a_save_and_restore, osvw_survives_a_save_and_restore, a_reset_returns_osvw_to_what_a_new_vp_holds, and refuses_unsupported_cpu_state (144 passed, 67 skipped). Two jobs passed on their one rerun: aarch64-windows, whose first attempt could not download its artifacts (HTTP 503: Egress is over the account limit), and x64-windows-intel-tdx, whose first attempt timed out in memstat::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_memory_validation_debug_small, an OpenHCL test that this change does not reach, since OpenHCL clears the new capabilities.
  • This pull request's CI of b37d8b1fb2e passed. nvx / ubuntu-latest passed on its one rerun; its first attempt failed in host::vm::tests::spawn_resolves_relative_runtime_and_block_inputs_outside_the_vm_directory with Text file busy, the fixture flake that nvx: make Linux copy/exec and deadline fixtures deterministic #1319 tracks, in nvx/, which this change does not touch.

Not run anywhere: bndcfgs_survives_a_save_and_restore, which needs a host with MPX, and MSHV with WAITPKG.

Checklist

  • Tests cover the new or fixed behavior, in the layout that
    python3 scripts/check_test_layout.py checks.
  • python3 scripts/check_test_changes.py --base origin/dev lists no change
    to an existing test, or a Changes to existing tests section below
    explains each one.
  • The documentation that the change affects is updated, including
    .github/copilot-instructions.md and the skills when a convention or
    workflow changes.
  • A change to the public API of nvx updates nvx-rs/public-api/ with
    python3 scripts/public_api.py --update.
  • Snapshots, protocols, and on-disk formats stay compatible, or the
    description says how the change versions or migrates them.

OpenVMM pin

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Processor-state persistence changes span three hypervisors, with AMD, MPX, and MSHV WAITPKG validation gaps requiring maintainer review before landing.

No open issues 🎉

What changed in this PR

This PR updates NVX’s OpenVMM pin to address remaining processor-state snapshot gaps tracked by #1182 and documents the resulting backend behavior without changing snapshot or ABI versions.

Changes:

  • Documents preservation of user-wait, MPX, and AMD processor registers across snapshots.
  • Describes capture refusal and rollback for software interrupts or events that MSHV and WHP cannot represent.
  • Maps the four OpenVMM commits at positions 303–306 and updates the recorded pin.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin and maps the four commits and their upstream dependencies.
doc/​design/​2.2-cpu-and-topology.md Clarifies register preservation by backend and corrects prior MPX and SVM claims.
doc/​design/​1.5-compatibility-and-versioning-model.md Expands the saved-state inventory and documents unsupported-event capture refusal.

Review effort: Balanced

A snapshot did not save every guest-visible piece of processor state
that a guest can write, and a capture could crash instead of refusing
state that it cannot represent (#1182):

- On WHP and MSHV, a guest's IA32_UMWAIT_CONTROL, which Linux writes on
  every CPU at boot where the CPU profile enumerates WAITPKG, as the
  Alder Lake and Granite Rapids profiles do.
- On every backend, IA32_BNDCFGS, which a guest of the Skylake-SP
  profile, whose CPUID enumerates MPX, can write: #1160 left it out on
  the ground that KVM no longer exposes MPX, which Linux's sources
  contradict.
- On KVM on an AMD host, VM_CR and VM_HSAVE_PA, which KVM keeps for any
  guest, and the OSVW MSRs, which it keeps for every guest of an AMD
  profile.
- On MSHV and WHP, a capture of a VP that the hypervisor stopped while
  it delivered a software interrupt or exception panicked in the
  conversion of its pending interruption.

Pin c9755a905618feeae5072d74c206a65736406aab, the head of
microsoft/openvmm#5134, four commits on 2807bafcf, the head of
feature/microvm/dev, where #1392's fix landed:

- 07dcf07d7 virt_whp, virt_mshv: save and restore a guest's
  IA32_UMWAIT_CONTROL
- 3587c0111 virt: save and restore a guest's IA32_BNDCFGS
- 3c40841df virt, virt_kvm: save and restore AMD's VM_CR, VM_HSAVE_PA,
  and OSVW
- c9755a905 virt_mshv, virt_whp: refuse to save a pending software
  interrupt

WHP and MSHV now read and write IA32_UMWAIT_CONTROL as a VP register,
with the existing umwait_control element. The VP state gains three
optional elements, each in a field that a saved state from an earlier
build lacks and that an earlier build ignores, so no version of the
saved state, the snapshot manifest, the time ABI, or any other contract
changes, and none is created or bumped: bndcfgs (field 21), where the
guest's CPUID enumerates MPX and the backend can access the MSR;
svm_msrs (field 22), on KVM where it runs on SVM, as on an AMD host; and
osvw (field 23), on KVM where the guest's CPUID and KVM's supported
CPUID enumerate OSVW. A restore refuses an element that the partition
lacks, as it already did, and a VM_HSAVE_PA that no guest could write,
or a VM_CR that sets SVMDIS while EFER.SVME is set, before it writes
either SVM MSR. A save on MSHV or WHP refuses a pending software
interrupt or exception, a pending event other than an exception or an
extint, and a vector beyond 8 bits, with an error, so a guest-requested
capture rolls back and the guest continues.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Guest-Writable Processor State (2.2.1.4) said that only KVM holds the
guest-writable MSR elements and that MSHV and WHP hold none, and gave
IA32_BNDCFGS and MSR_VM_HSAVE_PA as state that no guest can write: KVM
had not presented MPX since Linux 5.6, it said, and no profile
enumerates SVM. #1160 recorded both, and neither held: Linux 5.6 dropped
MPX for the kernel's own processes but still enables MPX's state
components for KVM's guests, so a KVM guest of the Skylake-SP profile
writes IA32_BNDCFGS, and KVM on an AMD host keeps a guest's VM_CR and
VM_HSAVE_PA whatever its CPUID says, and its OSVW MSRs for every guest
of an AMD profile.

The OpenVMM pin now saves and restores IA32_UMWAIT_CONTROL and
IA32_BNDCFGS on MSHV and WHP, IA32_BNDCFGS on KVM, and AMD's VM_CR,
VM_HSAVE_PA, OSVW_ID_LENGTH, and OSVW_STATUS on KVM on an AMD host. Say
which backend holds each element and why the Microsoft hypervisor's
backends hold no others, that a restore refuses a VM_HSAVE_PA or VM_CR
that no guest can hold, and that a VM reset gives a processor back KVM's
OSVW values. Name the new elements in Processor State Elements (1.5.4),
and say there that MSHV and WHP refuse to capture a processor that the
hypervisor stopped while it delivered a software interrupt or exception,
which the saved activity cannot hold, and that a guest-requested capture
then rolls back.

Refs #1182

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c9755a905, position 306, the head of
microsoft/openvmm#5134, whose four commits, positions 303-306, follow
2807bafcf, the head of feature/microvm/dev, which microsoft/openvmm#5132
merged. The pin is 180 commits past the mapped tip, and no PR maps
positions 303-306 yet.

Position 303 has WHP and MSHV save and restore IA32_UMWAIT_CONTROL, as a
register of the VP, completing position 277's umwait_control element,
which KVM alone held. Position 304 adds the bndcfgs element, field 21,
for IA32_BNDCFGS where the guest's CPUID enumerates MPX, on every
backend, and rewrites position 279's Guide remark that no guest can
write the MSR. Position 305 adds the svm_msrs and osvw elements, fields
22 and 23, for VM_CR and VM_HSAVE_PA, which KVM on SVM keeps for every
guest, and the OSVW MSRs, which it keeps for a guest whose CPUID
enumerates OSVW where it supports OSVW, has a VM reset put back the OSVW
values that KVM gave the vCPU, and refuses a VM_HSAVE_PA that no guest
can write. Position 306 has MSHV and WHP refuse to save a VP that holds
a pending software interrupt or exception, or another interruption or
event that the saved activity cannot hold, where the conversion of the
activity's registers, which upstream main has, panicked. None of them
changes a version.

They change no line that a mapped PR adds, though they change files that
13 mapped PRs also change. Replaying each on upstream main, c0694c4da,
conflicts. Positions 303-305 edit or add beside what positions 272,
277-279, 283, 295, and 296 add, and each beside its predecessors among
them, and position 305 also beside what positions 128 and 248 add, so
they follow those positions upstream. Position 306 rewrites only lines
that upstream main has, but its conversion builds the fields of the
activity that positions 287 and 291 add, and its Guide paragraphs sit
beside what positions 071, 285, 289, 292, and 301 write, so it follows
those positions upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-1182-processor-state branch from a836ef9 to b37d8b1 Compare November 17, 2026 16:49
Copilot AI balanced review requested due to automatic review settings November 17, 2026 16:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-hypervisor snapshot-state changes lack MPX and MSHV WAITPKG runtime validation, while AMD and exact-head CI evidence remains pending, requiring maintainer review of the remaining platform risk.

No open issues 🎉

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 17, 2026
Merged via the queue into dev with commit 6429fa4 Nov 17, 2026
96 of 98 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1182-processor-state branch November 17, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenVMM/microVM umbrella: processor state across snapshots

2 participants