Skip to content

openvmm: pin the SYSENTER MSRs held in the fields that name them - #1588

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1367-sysenter-msr-order
Nov 14, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1367-sysenter-msr-order

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1367

What and why

This pins microsoft/openvmm#5100 at c5509cde20d624fa49b90722c34ad292ab7672cb. With the pin, VirtualMsrs, a VP's virtual MSRs, holds IA32_SYSENTER_ESP (0x175) in sysenter_esp and IA32_SYSENTER_EIP (0x176) in sysenter_eip on KVM, MSHV, and WHP, and the bytes that it saves do not change. Before, each of the two fields held the other's register there, so anything that read a snapshot's VP state by name saw the wrong one: inspect output, saved-state tooling, and OpenHCL's CVM processors, which fill the fields by name, disagreed with the generic backends about protobuf fields 3 and 4.

At NVX's previous pin, 5d2a71027:

  • Cause. In vmm_core/virt/src/x86/vp.rs, names() of HvRegisterState for VirtualMsrs lists SysenterEsp before SysenterEip, but get_values and set_values paired the registers with the fields in the other order, so sysenter_eip held IA32_SYSENTER_ESP and sysenter_esp held IA32_SYSENTER_EIP. KVM (through get_msrs_state and set_msrs_state), MSHV, and WHP save and restore the type through those methods, so the two swaps cancel: a save and a restore round-trip exactly, and a guest sees no difference.
  • Reproduction. The issue's reproduction, the registers SysenterEip and SysenterEsp holding 0x1111 and 0x2222 passed through names() to set_values, gives sysenter_eip == 0x2222 and sysenter_esp == 0x1111; the pull request's three virt tests fail at 5d2a71027 for that reason. On the three test hosts, a Linux guest that reads each CPU's IA32_SYSENTER_ESP and IA32_SYSENTER_EIP through /dev/cpu/N/msr (the EIP is the address of entry_SYSENTER_compat) and requests a snapshot gives a state.bin whose VP VirtualMsrs holds the guest's ESP in the field named sysenter_eip (field 3) and its EIP in sysenter_esp (field 4), on KVM, MSHV, and WHP alike.

The fix, and why it needs no version

The issue offers three fixes:

  1. Reorder names(). That swaps the meaning of fields 3 and 4 for every saved state of KVM, MSHV, and WHP, so a snapshot taken before the fix restores with EIP and ESP swapped unless the restore can tell it apart with a new field or a version mark. NVX creates no such mark.
  2. Swap the #[mesh(3)] and #[mesh(4)] tags and the get_values order, so that old generic snapshots keep their meaning, at the cost, the issue says, that OpenHCL CVM saved states are misread instead.
  3. Keep the wire format and only document the fields as swapped for the generic backends, which leaves every reader by name wrong.

The pin takes the second, and swaps the two fields' declaration order along with their tags. mesh's encoder writes a message's fields in declaration order and its decoder finds them by number, so the encoded bytes do not change, and neither does the meaning of any saved state: KVM, MSHV, and WHP have always saved the ESP in field 3 and the EIP in field 4, since the type's first commit. The issue's cost does not arise either: no OpenHCL saved state holds VirtualMsrs. The TDX, SNP, and CCA processors return SaveError::NotSupported from their save, and the hypervisor-backed processor saves a struct of its own, so no saved state of OpenHCL's CVM processors exists for the swap to misread, and their fill by name was already right. Of the 19 implementors of HvRegisterState, no other pairs its registers with its fields in an order that its names does not list.

What the OpenVMM pull request changes

microsoft/openvmm#5100 adds one signed commit on 5d2a71027, the head of feature/microvm/dev. It has a good signature (openvmm_pin.py source).

Position Commit Change
280 c5509cde2 get_values, set_values, and at_reset of VirtualMsrs pair the registers with the fields in the order that names() lists them, and sysenter_esp and sysenter_eip swap their declaration order and their numbers, 3 and 4, so that each holds the register that its name says and the saved layout is the one that KVM, MSHV, and WHP always wrote. HvRegisterState's three methods document the contract that VirtualMsrs broke: names, get_values, and set_values list the same registers in the same order. Three tests in virt: virtual_msrs_write_each_register_from_its_own_field and virtual_msrs_read_each_register_into_its_own_field (the issue's reproduction) check what the registers carry, and virtual_msrs_keep_the_layout_of_earlier_saved_states decodes a state that holds the ESP in field 3 and the EIP in field 4, as earlier releases saved it, and checks that the type encodes one that does. They fail at 5d2a71027 and pass with the commit.

The issue attaches no patch.

What this pull request changes in NVX

  • c362295a514 moves the openvmm gitlink to the pin, and nothing else.
  • 4c6f1a161c9 maps position 280 in the OpenVMM roadmap.

No documentation or nvx-rs change: nothing of NVX names the fields or the layout of VirtualMsrs. No version is created or bumped, and no snapshot, saved-state, or wire format changes. In particular the snapshot manifest's version, whose mismatch refuses earlier formats (#1417), is unchanged.

Compatibility of existing snapshots

  • A snapshot of the current format that KVM, MSHV, or WHP saved before the pin restores with the right SYSENTER_EIP and SYSENTER_ESP after it, and one that the pin saves restores in a build from before it, since the bytes are the same. The real-guest runs below check both directions on all three backends.
  • The guest sees the same registers as before the pin. Only the names that the two saved fields carry change, in inspect output and in the generated virt.x86 schema, where field 3 is now sysenter_esp and field 4 sysenter_eip.

Pin details

  • Roadmap. The pin is 156 commits past the mapped tip. Position 280 is not yet assigned to a PR and changes nothing that a mapped PR adds. It changes code that upstream main, 7d90c4104, has and no commit of the chain changes, and replaying it there conflicts only where its tests extend the tests module of the VP state module, which upstream main lacks and position 132 adds, so the tests follow positions 132 and 251 upstream and the rest of the commit applies without conflicts.
  • Prerequisites. None. The pull request's base, 5d2a71027, is NVX's previous pin and the head of feature/microvm/dev.
  • Equivalence. None claimed; this pin replaces no reviewed revision.
  • Coordinated changes. None. The control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's (openvmm_vectors and openvmm_cpu_profiles, below).

Landing order

microsoft/openvmm#5100 lands first, by fast-forwarding feature/microvm/dev from 5d2a71027 to c5509cde20d624fa49b90722c34ad292ab7672cb, which python scripts/openvmm_pin.py land c5509cde20d624fa49b90722c34ad292ab7672cb --remote REMOTE --push does. This pull request merges after that, through the merge queue. OpenVMM's merge button squashes, which would leave the pinned revision off the branch.

Validation

NVX

Run on Windows:

  • python scripts/openvmm_pin.py source c5509cde20d624fa49b90722c34ad292ab7672cb --pr 5100: passed. The revision is the pull request's head, with a good signature.
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed, with 280 positions and 291 references.
  • python scripts/openvmm_pin.py check-branch --topic: passed, with two signed commits and no coordinated files.
  • python scripts/check_markdown_links.py and git diff --check: passed.
  • python scripts/nvx.py verify: passed.
  • cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored (4 passed) and --test openvmm_cpu_profiles -- --ignored (1 passed).
  • python scripts/check_test_changes.py --base origin/dev reports no change to an existing test, so the pull request needs no tests-modified label.
  • python scripts/local_gates.py --base origin/dev --run --keep-going: test-changes, markdown-links, test-layout, openvmm-vectors, and openvmm-cpu-profiles passed. For nvx-cli I ran its commands without the pip install step, since this machine installs no packages: compileall passed, 2199 scripts/test_*.py tests ran with one failure, the 84 adversarial tests passed, and every nvx.py help command exits 0. The failure, test_public_acceptance_preserves_test_and_cleanup_failures, keeps a diagnostic under 4608 characters and counted my long TEMP path; it passes with a shorter one. openvmm-ci needs OpenVMM's required jobs to have passed, which take hours, so the pull request's own openvmm-ci job decides it.

OpenVMM

With Rust 1.97.1 and cargo-nextest 0.9.133, on the commit as pinned:

  • Reproduction. cargo nextest run --profile agent -p virt fails the three new tests at 5d2a71027 with only the tests applied, and passes them with the commit.
  • Windows x64: cargo nextest run --profile agent -p virt passed 108 tests (1 skipped), cargo clippy --all-targets and cargo doc --no-deps on virt have no warnings, and cargo xtask fmt --fix changes nothing.
  • Linux x64, on nvx-mshv-test (Azure Linux 3.0): the same virt tests, clippy, and rustdoc pass, and cargo check passes for virt_mshv_vtl, virt_mshv, virt_kvm, and vmm_core, which use VirtualMsrs.
  • KVM, on nvx-kvm-test: cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm passed 77 tests, the ignored /dev/kvm ones included.

Hosts

All three are virtual machines. On each, a Linux microVM of one or two CPUs reads every CPU's IA32_SYSENTER_ESP and IA32_SYSENTER_EIP through /dev/cpu/N/msr (the EIP is the address of entry_SYSENTER_compat) and requests a snapshot. It was captured and restored in each of the four combinations of OpenVMM as released in v0.1.0-dev.0b1029a5c372, which is 157866605 and lacks only the EFER.SVME restore check of 5d2a71027 (before the fix), and OpenVMM built from the pin's tree (with the pin), at one and at two CPUs: eight captures and restores per host.

Host Platform Host type Before the fix With the pin
nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3) linux-kvm virtual-machine Each VP's VirtualMsrs holds the guest's SYSENTER_ESP in field 3, which the type names sysenter_eip, and its SYSENTER_EIP in field 4, which it names sysenter_esp. The same bytes: field 3, now sysenter_esp, holds the ESP, and field 4, now sysenter_eip, the EIP.
nvx-mshv-test (Intel Xeon Platinum 8573C, Linux 6.18.34.mshv3-1.azl3) linux-mshv virtual-machine The same. The same bytes.
nvx-whp-test (Intel Xeon Platinum 8370C, Windows Server 2025) windows-whp virtual-machine The same. The same bytes.

In all 24 captures and restores, the ones whose capture and restore builds differ included, each CPU's SYSENTER_ESP and SYSENTER_EIP after the restore equal the guest's before the capture.

The checkouts were worktrees of NVX at f7bb375818f with the commit's diff applied to OpenVMM 5d2a71027: the diff is byte-identical to the pin's. nvx-rs download installed v0.1.0-dev.0b1029a5c372, and OpenVMM was then built from each worktree in a release build.

With that build and those artifacts, python scripts/nvx.py test-microvm --scenario smp --scenario smp-snapshot --scenario snapshot-core --scenario snapshot-rollback --scenario restore-downtime --scenario restore-processors --scenario time-abi-conformance --scenario snapshot-tiers --processors 1 2 passed on each of the three hosts, with --backend kvm, mshv, and whp.

The KVM restore-downtime scenario can fail under heavy host load, and it affects the base as well. Its 8-vCPU case takes a TSC warp probe, which reports NVX-WARP-PROBE-FAIL when other jobs saturate the host. On nvx-kvm-test, while the host's load average was 30 to 44, the scenario failed in the same way with OpenVMM built from before this change as with the pin's build, and it passed with the pin's build when the load fell to about 5. In the final runs above, at a load average under 2, all of the scenarios passed on KVM, and they passed in this pull request's CI (below); had that scenario failed there, the cause would be that load flake, which affects the base too, and CI decides.

CI

I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI, which passed:

  • This pull request. Run 43596026345 of the head 4c6f1a161c9 passed, with Required status check: the microVM scenarios on Linux KVM, Linux MSHV, and Windows WHP, the NVX CLI gates on Linux and Windows, the runtime adapter, the platform and performance gates, and OpenVMM CI of the pinned revision. No scenario failed, so the KVM restore-downtime load flake did not occur there.
  • OpenVMM. Run 43594226323 of [DO NOT REVIEW] virt: hold each SYSENTER MSR in the field that names it openvmm#5100 at c5509cde20d624fa49b90722c34ad292ab7672cb passed the six jobs that openvmm-ci requires: unit tests [x64-linux] and unit tests [x64-windows], which ran the three new virt tests, and run vmm-tests on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd. run vmm-tests [x64-linux-amd-kvm] ran on an AMD EPYC 9V45 and passed all 110 tests.
  • Not required. The binary-size jobs of OpenVMM's run passed, and the SEV-SNP job was not waited for: x64-windows-amd-snp failed. run vmm-tests [aarch64-windows] failed one test, multiarch::hyperv_openhcl_uefi_aarch64_ubuntu_2404_server_aarch64_boot_heavy, which timed out after 663 s in an aarch64 OpenHCL Ubuntu boot while the other 97 passed. That job passed in the four earlier runs of the branch, and the change touches only x86 code and a doc comment, so I did not rerun it; openvmm-ci does not require it.

On KVM, MSHV, and WHP, a VP's saved VirtualMsrs held IA32_SYSENTER_ESP
in sysenter_eip and IA32_SYSENTER_EIP in sysenter_esp, since get_values
and set_values paired the registers with the fields in the opposite
order of names(). A save and a restore round-tripped, because the two
swaps cancel, but anything that read the fields by name saw the wrong
register, and OpenHCL's TDX and SNP processors, which fill the fields by
name, disagreed with these backends (#1367).

Pin c5509cde20d624fa49b90722c34ad292ab7672cb, the head of
microsoft/openvmm#5100, one commit on 5d2a71027, the head of
feature/microvm/dev:

- c5509cde2 virt: hold each SYSENTER MSR in the field that names it

get_values, set_values, and at_reset now pair the registers with the
fields in the order that names() lists them, and the two fields swap
their declaration order and their numbers, so that field 3 holds
IA32_SYSENTER_ESP and field 4 holds IA32_SYSENTER_EIP, as KVM, MSHV, and
WHP have always saved them. The bytes that a VirtualMsrs encodes do not
change, so every snapshot of the current format restores with the right
SYSENTER_EIP and SYSENTER_ESP, and a snapshot that the pin saves
restores the same way in a build from before it. No OpenHCL saved state
holds VirtualMsrs, so none is misread. No version is created or bumped,
and no snapshot or wire format changes.

Tests: the revision's three tests of virt fail at 5d2a71027 and pass
with it, and its 108 tests of virt pass on Windows and on Azure Linux
3.0. On nvx-kvm-test its 77 tests of virt_kvm and kvm, the ignored
/dev/kvm ones included, pass as well. A guest on nvx-kvm-test,
nvx-mshv-test, and nvx-whp-test read the IA32_SYSENTER_ESP and
IA32_SYSENTER_EIP of each of its one or two CPUs and requested a
snapshot: OpenVMM built from 157866605 and OpenVMM built from the pin's
tree saved the same bytes, and a snapshot that either build captured,
restored with the other build, kept each CPU's two MSRs. With OpenVMM
built from the pin's tree and the artifacts of v0.1.0-dev.0b1029a5c372,
the microVM scenarios smp, smp-snapshot, snapshot-core,
snapshot-rollback, restore-downtime, restore-processors, snapshot-tiers,
and time-abi-conformance pass at 1 and 2 processors on all three hosts.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx-rs copies from OpenVMM match the revision's, and
nothing of NVX names the fields of VirtualMsrs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c5509cde2, position 280, 156 commits past the
mapped tip. Position 280, the one commit of microsoft/openvmm#5100 on
5d2a71027, the head of feature/microvm/dev, is not yet assigned to a PR.

Position 280 makes VirtualMsrs, the virtual MSRs of a VP, hold
IA32_SYSENTER_ESP in sysenter_esp and IA32_SYSENTER_EIP in sysenter_eip,
which KVM, MSHV, and WHP held in the other's field, since get_values and
set_values paired the registers with the fields in the opposite order of
names. It swaps the fields and their numbers, so that the type encodes
the same bytes as before, and documents on HvRegisterState that its
three methods list the same registers in the same order. It changes
code that upstream main has and no commit of the chain changes, adds
three tests to the tests module that position 132 adds and position 251
extends, and changes nothing that a mapped PR adds. Replaying it on
upstream main, 7d90c4104, conflicts only where its tests extend that
module, which upstream main lacks, so they follow positions 132 and 251
upstream, and the rest of the commit applies there without conflicts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Changes to persisted VP-state field mappings require maintainer verification of compatibility and merge gates while exact-pin CI is running and OpenVMM #5100 remains unmerged.

No open issues 🎉

What changed in this PR

This PR updates NVX’s OpenVMM pin to fix the SYSENTER field-name mismatch described in #1367 while preserving existing snapshot bytes and compatibility.

Changes:

  • Records revision c5509cde2 at position 280 and updates chain counts and ranges.
  • Documents the register-field correction, saved-state compatibility, and upstream test dependencies.
File Description
doc/​openvmm-upstream-roadmap.md Maps the new pin and explains the SYSENTER fix and its upstream dependencies.

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 14, 2026
Merged via the queue into dev with commit ae35cbd Nov 14, 2026
48 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1367-sysenter-msr-order branch November 14, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: VirtualMsrs.sysenter_eip holds SYSENTER_ESP on KVM, MSHV and WHP

2 participants