Repository navigation
openvmm: pin the SYSENTER MSRs held in the fields that name them - #1588
Merged
Pedro Henrique Penna (ppenna) merged 2 commits intoNov 14, 2026
Merged
Conversation
On KVM, MSHV, and WHP, a VP's saved VirtualMsrs held IA32_SYSENTER_ESP in sysenter_eip and IA32_SYSENTER_EIP in sysenter_esp, since get_values and set_values paired the registers with the fields in the opposite order of names(). A save and a restore round-tripped, because the two swaps cancel, but anything that read the fields by name saw the wrong register, and OpenHCL's TDX and SNP processors, which fill the fields by name, disagreed with these backends (#1367). Pin c5509cde20d624fa49b90722c34ad292ab7672cb, the head of microsoft/openvmm#5100, one commit on 5d2a71027, the head of feature/microvm/dev: - c5509cde2 virt: hold each SYSENTER MSR in the field that names it get_values, set_values, and at_reset now pair the registers with the fields in the order that names() lists them, and the two fields swap their declaration order and their numbers, so that field 3 holds IA32_SYSENTER_ESP and field 4 holds IA32_SYSENTER_EIP, as KVM, MSHV, and WHP have always saved them. The bytes that a VirtualMsrs encodes do not change, so every snapshot of the current format restores with the right SYSENTER_EIP and SYSENTER_ESP, and a snapshot that the pin saves restores the same way in a build from before it. No OpenHCL saved state holds VirtualMsrs, so none is misread. No version is created or bumped, and no snapshot or wire format changes. Tests: the revision's three tests of virt fail at 5d2a71027 and pass with it, and its 108 tests of virt pass on Windows and on Azure Linux 3.0. On nvx-kvm-test its 77 tests of virt_kvm and kvm, the ignored /dev/kvm ones included, pass as well. A guest on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test read the IA32_SYSENTER_ESP and IA32_SYSENTER_EIP of each of its one or two CPUs and requested a snapshot: OpenVMM built from 157866605 and OpenVMM built from the pin's tree saved the same bytes, and a snapshot that either build captured, restored with the other build, kept each CPU's two MSRs. With OpenVMM built from the pin's tree and the artifacts of v0.1.0-dev.0b1029a5c372, the microVM scenarios smp, smp-snapshot, snapshot-core, snapshot-rollback, restore-downtime, restore-processors, snapshot-tiers, and time-abi-conformance pass at 1 and 2 processors on all three hosts. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's, and nothing of NVX names the fields of VirtualMsrs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c5509cde2, position 280, 156 commits past the mapped tip. Position 280, the one commit of microsoft/openvmm#5100 on 5d2a71027, the head of feature/microvm/dev, is not yet assigned to a PR. Position 280 makes VirtualMsrs, the virtual MSRs of a VP, hold IA32_SYSENTER_ESP in sysenter_esp and IA32_SYSENTER_EIP in sysenter_eip, which KVM, MSHV, and WHP held in the other's field, since get_values and set_values paired the registers with the fields in the opposite order of names. It swaps the fields and their numbers, so that the type encodes the same bytes as before, and documents on HvRegisterState that its three methods list the same registers in the same order. It changes code that upstream main has and no commit of the chain changes, adds three tests to the tests module that position 132 adds and position 251 extends, and changes nothing that a mapped PR adds. Replaying it on upstream main, 7d90c4104, conflicts only where its tests extend that module, which upstream main lacks, so they follow positions 132 and 251 upstream, and the rest of the commit applies there without conflicts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Changes to persisted VP-state field mappings require maintainer verification of compatibility and merge gates while exact-pin CI is running and OpenVMM #5100 remains unmerged.
No open issues 🎉
What changed in this PR
This PR updates NVX’s OpenVMM pin to fix the SYSENTER field-name mismatch described in #1367 while preserving existing snapshot bytes and compatibility.
Changes:
- Records revision
c5509cde2at position 280 and updates chain counts and ranges. - Documents the register-field correction, saved-state compatibility, and upstream test dependencies.
| File | Description |
|---|---|
| doc/openvmm-upstream-roadmap.md | Maps the new pin and explains the SYSENTER fix and its upstream dependencies. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1367-sysenter-msr-order
branch
November 14, 2026 13:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1367
What and why
This pins microsoft/openvmm#5100 at
c5509cde20d624fa49b90722c34ad292ab7672cb. With the pin,VirtualMsrs, a VP's virtual MSRs, holds IA32_SYSENTER_ESP (0x175) insysenter_espand IA32_SYSENTER_EIP (0x176) insysenter_eipon KVM, MSHV, and WHP, and the bytes that it saves do not change. Before, each of the two fields held the other's register there, so anything that read a snapshot's VP state by name saw the wrong one: inspect output, saved-state tooling, and OpenHCL's CVM processors, which fill the fields by name, disagreed with the generic backends about protobuf fields 3 and 4.At NVX's previous pin,
5d2a71027:vmm_core/virt/src/x86/vp.rs,names()ofHvRegisterState for VirtualMsrslistsSysenterEspbeforeSysenterEip, butget_valuesandset_valuespaired the registers with the fields in the other order, sosysenter_eipheld IA32_SYSENTER_ESP andsysenter_espheld IA32_SYSENTER_EIP. KVM (throughget_msrs_stateandset_msrs_state), MSHV, and WHP save and restore the type through those methods, so the two swaps cancel: a save and a restore round-trip exactly, and a guest sees no difference.SysenterEipandSysenterEspholding 0x1111 and 0x2222 passed throughnames()toset_values, givessysenter_eip == 0x2222andsysenter_esp == 0x1111; the pull request's threevirttests fail at5d2a71027for that reason. On the three test hosts, a Linux guest that reads each CPU's IA32_SYSENTER_ESP and IA32_SYSENTER_EIP through/dev/cpu/N/msr(the EIP is the address ofentry_SYSENTER_compat) and requests a snapshot gives astate.binwhose VPVirtualMsrsholds the guest's ESP in the field namedsysenter_eip(field 3) and its EIP insysenter_esp(field 4), on KVM, MSHV, and WHP alike.The fix, and why it needs no version
The issue offers three fixes:
names(). That swaps the meaning of fields 3 and 4 for every saved state of KVM, MSHV, and WHP, so a snapshot taken before the fix restores with EIP and ESP swapped unless the restore can tell it apart with a new field or a version mark. NVX creates no such mark.#[mesh(3)]and#[mesh(4)]tags and theget_valuesorder, so that old generic snapshots keep their meaning, at the cost, the issue says, that OpenHCL CVM saved states are misread instead.The pin takes the second, and swaps the two fields' declaration order along with their tags. mesh's encoder writes a message's fields in declaration order and its decoder finds them by number, so the encoded bytes do not change, and neither does the meaning of any saved state: KVM, MSHV, and WHP have always saved the ESP in field 3 and the EIP in field 4, since the type's first commit. The issue's cost does not arise either: no OpenHCL saved state holds
VirtualMsrs. The TDX, SNP, and CCA processors returnSaveError::NotSupportedfrom their save, and the hypervisor-backed processor saves a struct of its own, so no saved state of OpenHCL's CVM processors exists for the swap to misread, and their fill by name was already right. Of the 19 implementors ofHvRegisterState, no other pairs its registers with its fields in an order that itsnamesdoes not list.What the OpenVMM pull request changes
microsoft/openvmm#5100 adds one signed commit on
5d2a71027, the head offeature/microvm/dev. It has a good signature (openvmm_pin.py source).c5509cde2get_values,set_values, andat_resetofVirtualMsrspair the registers with the fields in the order thatnames()lists them, andsysenter_espandsysenter_eipswap their declaration order and their numbers, 3 and 4, so that each holds the register that its name says and the saved layout is the one that KVM, MSHV, and WHP always wrote.HvRegisterState's three methods document the contract thatVirtualMsrsbroke:names,get_values, andset_valueslist the same registers in the same order. Three tests invirt:virtual_msrs_write_each_register_from_its_own_fieldandvirtual_msrs_read_each_register_into_its_own_field(the issue's reproduction) check what the registers carry, andvirtual_msrs_keep_the_layout_of_earlier_saved_statesdecodes a state that holds the ESP in field 3 and the EIP in field 4, as earlier releases saved it, and checks that the type encodes one that does. They fail at5d2a71027and pass with the commit.The issue attaches no patch.
What this pull request changes in NVX
c362295a514moves theopenvmmgitlink to the pin, and nothing else.4c6f1a161c9maps position 280 in the OpenVMM roadmap.No documentation or
nvx-rschange: nothing of NVX names the fields or the layout ofVirtualMsrs. No version is created or bumped, and no snapshot, saved-state, or wire format changes. In particular the snapshot manifest's version, whose mismatch refuses earlier formats (#1417), is unchanged.Compatibility of existing snapshots
virt.x86schema, where field 3 is nowsysenter_espand field 4sysenter_eip.Pin details
7d90c4104, has and no commit of the chain changes, and replaying it there conflicts only where its tests extend thetestsmodule of the VP state module, which upstream main lacks and position 132 adds, so the tests follow positions 132 and 251 upstream and the rest of the commit applies without conflicts.5d2a71027, is NVX's previous pin and the head offeature/microvm/dev.nvx-rscopies from OpenVMM match the revision's (openvmm_vectorsandopenvmm_cpu_profiles, below).Landing order
microsoft/openvmm#5100 lands first, by fast-forwarding
feature/microvm/devfrom5d2a71027toc5509cde20d624fa49b90722c34ad292ab7672cb, whichpython scripts/openvmm_pin.py land c5509cde20d624fa49b90722c34ad292ab7672cb --remote REMOTE --pushdoes. This pull request merges after that, through the merge queue. OpenVMM's merge button squashes, which would leave the pinned revision off the branch.Validation
NVX
Run on Windows:
python scripts/openvmm_pin.py source c5509cde20d624fa49b90722c34ad292ab7672cb --pr 5100: passed. The revision is the pull request's head, with a good signature.python scripts/openvmm_pin.py check-roadmap --worktree: passed, with 280 positions and 291 references.python scripts/openvmm_pin.py check-branch --topic: passed, with two signed commits and no coordinated files.python scripts/check_markdown_links.pyandgit diff --check: passed.python scripts/nvx.py verify: passed.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored(4 passed) and--test openvmm_cpu_profiles -- --ignored(1 passed).python scripts/check_test_changes.py --base origin/devreports no change to an existing test, so the pull request needs notests-modifiedlabel.python scripts/local_gates.py --base origin/dev --run --keep-going:test-changes,markdown-links,test-layout,openvmm-vectors, andopenvmm-cpu-profilespassed. Fornvx-cliI ran its commands without thepip installstep, since this machine installs no packages:compileallpassed, 2199scripts/test_*.pytests ran with one failure, the 84 adversarial tests passed, and everynvx.pyhelp command exits 0. The failure,test_public_acceptance_preserves_test_and_cleanup_failures, keeps a diagnostic under 4608 characters and counted my longTEMPpath; it passes with a shorter one.openvmm-cineeds OpenVMM's required jobs to have passed, which take hours, so the pull request's ownopenvmm-cijob decides it.OpenVMM
With Rust 1.97.1 and cargo-nextest 0.9.133, on the commit as pinned:
cargo nextest run --profile agent -p virtfails the three new tests at5d2a71027with only the tests applied, and passes them with the commit.cargo nextest run --profile agent -p virtpassed 108 tests (1 skipped),cargo clippy --all-targetsandcargo doc --no-depsonvirthave no warnings, andcargo xtask fmt --fixchanges nothing.virttests, clippy, and rustdoc pass, andcargo checkpasses forvirt_mshv_vtl,virt_mshv,virt_kvm, andvmm_core, which useVirtualMsrs.cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvmpassed 77 tests, the ignored/dev/kvmones included.Hosts
All three are virtual machines. On each, a Linux microVM of one or two CPUs reads every CPU's IA32_SYSENTER_ESP and IA32_SYSENTER_EIP through
/dev/cpu/N/msr(the EIP is the address ofentry_SYSENTER_compat) and requests a snapshot. It was captured and restored in each of the four combinations of OpenVMM as released inv0.1.0-dev.0b1029a5c372, which is157866605and lacks only the EFER.SVME restore check of5d2a71027(before the fix), and OpenVMM built from the pin's tree (with the pin), at one and at two CPUs: eight captures and restores per host.VirtualMsrsholds the guest's SYSENTER_ESP in field 3, which the type namessysenter_eip, and its SYSENTER_EIP in field 4, which it namessysenter_esp.sysenter_esp, holds the ESP, and field 4, nowsysenter_eip, the EIP.In all 24 captures and restores, the ones whose capture and restore builds differ included, each CPU's SYSENTER_ESP and SYSENTER_EIP after the restore equal the guest's before the capture.
The checkouts were worktrees of NVX at
f7bb375818fwith the commit's diff applied to OpenVMM5d2a71027: the diff is byte-identical to the pin's.nvx-rs downloadinstalledv0.1.0-dev.0b1029a5c372, and OpenVMM was then built from each worktree in a release build.With that build and those artifacts,
python scripts/nvx.py test-microvm --scenario smp --scenario smp-snapshot --scenario snapshot-core --scenario snapshot-rollback --scenario restore-downtime --scenario restore-processors --scenario time-abi-conformance --scenario snapshot-tiers --processors 1 2passed on each of the three hosts, with--backend kvm,mshv, andwhp.The KVM
restore-downtimescenario can fail under heavy host load, and it affects the base as well. Its 8-vCPU case takes a TSC warp probe, which reportsNVX-WARP-PROBE-FAILwhen other jobs saturate the host. On nvx-kvm-test, while the host's load average was 30 to 44, the scenario failed in the same way with OpenVMM built from before this change as with the pin's build, and it passed with the pin's build when the load fell to about 5. In the final runs above, at a load average under 2, all of the scenarios passed on KVM, and they passed in this pull request's CI (below); had that scenario failed there, the cause would be that load flake, which affects the base too, and CI decides.CI
I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI, which passed:
4c6f1a161c9passed, withRequired status check: the microVM scenarios on Linux KVM, Linux MSHV, and Windows WHP, the NVX CLI gates on Linux and Windows, the runtime adapter, the platform and performance gates, andOpenVMM CI of the pinned revision. No scenario failed, so the KVMrestore-downtimeload flake did not occur there.c5509cde20d624fa49b90722c34ad292ab7672cbpassed the six jobs thatopenvmm-cirequires:unit tests [x64-linux]andunit tests [x64-windows], which ran the three newvirttests, andrun vmm-testsonx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd.run vmm-tests [x64-linux-amd-kvm]ran on an AMD EPYC 9V45 and passed all 110 tests.x64-windows-amd-snpfailed.run vmm-tests [aarch64-windows]failed one test,multiarch::hyperv_openhcl_uefi_aarch64_ubuntu_2404_server_aarch64_boot_heavy, which timed out after 663 s in an aarch64 OpenHCL Ubuntu boot while the other 97 passed. That job passed in the four earlier runs of the branch, and the change touches only x86 code and a doc comment, so I did not rerun it;openvmm-cidoes not require it.