Repository navigation
openvmm: pin the restore's check of the APIC timer's delivery mode - #1590
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 14, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The clock-and-time documentation overstates both masked timer acceptance and the scope of the no-change guarantee during restore.
2 open issues
What changed in this PR
This PR pins OpenVMM’s KVM restore validation for unmasked LAPIC timers with non-Fixed delivery modes, addressing #1365 while preserving legitimate snapshots.
Changes:
- Advances the OpenVMM pin to
30b755406. - Documents the delivery-mode restriction in the clock-and-time design.
- Maps position 282 and its upstream dependencies in the roadmap.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Updates the pin, chain counts, and position 282’s upstream mapping. |
doc/design/3.2-clock-and-time.md |
Describes KVM’s timer delivery-mode validation and masked-entry behavior. |
Review effort: Balanced
Restoring a VP whose saved APIC page held an unmasked LVT timer entry with a delivery mode other than Fixed succeeded on KVM, and KVM then fired the timer in that mode once it had expired, as a one-shot timer whose current count is 0 has: inside KVM_SET_LAPIC when APICv is active, and at the next KVM_RUN otherwise. The restored VP took an INIT, a startup IPI, an NMI, or an SMI, or woke from halt, none of which the snapshot holds (#1365). Pin 30b7554069182a5262099f902c4a203d54c8bb88, the head of microsoft/openvmm#5102, one commit on b723cbed3, the head of feature/microvm/dev: - 30b755406 virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed set_apic now rejects such a page with KvmError::InvalidState as its first step, before it forgets the timer that the VP stopped with or changes the APIC base or the page, so the restore fails with the apic element's StateError, and set_apic itself leaves the VP's APIC as it was. The elements that restore_all restores before the APIC, the registers among them, have run: KVM's write of CR8 can change the LAPIC's task priority. It refuses every unmasked entry with a mode other than Fixed, lowest priority and ExtINT included, and accepts a masked entry. A guest cannot write the entry's delivery mode, since KVM masks its writes, so a save never holds such a page, a snapshot that a VP saved restores as before, and only a crafted snapshot is refused. MSHV and WHP are unchanged. No version is created or bumped, and no snapshot or wire format changes. Tests: on nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57, nested KVM without APICv), the revision's contract test restore_refuses_a_non_fixed_apic_timer fails at b723cbed3 with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]" and passes at the pin, as does its unit test an_unmasked_timer_that_is_not_fixed_is_refused. A scratch test that gave KVM_SET_LAPIC the same page showed what KVM does with it at KVM_RUN: INIT resets the VP, NMI injects an NMI, SMI kills the VM, and lowest priority is taken as a fixed interrupt. Mutants of the check fail the tests. At the pin, the 84 tests of virt_kvm and kvm pass with cargo nextest, the ignored ones included. With OpenVMM built from the pin's tree, which replaced the packaged binary (SHA-256 193d1063 to 04915a93), and the artifacts of v0.1.0-dev.9e41ebd10be8, the microVM scenarios guest-boot (1 processor), interrupt-map, smp, smp-snapshot, snapshot-core, snapshot-rollback, restore-downtime, restore-processors, snapshot-tiers, time-abi-conformance, and time-host-correction pass at 1 and 2 processors, and smp-lapic at 1, 2, 4, and 8, on nvx-kvm-test. The change is KVM-only, so nvx-mshv-test and nvx-whp-test were not run. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3.2.5 specified how a restore runs a one-shot LAPIC timer, and what KVM's LAPIC state write does to one that has expired, but not that KVM delivers the expiry of an unmasked timer in the delivery mode of its LVT entry, which a saved state that no capture holds can set to something other than a fixed interrupt (#1365). It now says that on KVM a restore rejects a LAPIC state whose unmasked timer entry has a delivery mode other than fixed, and that the step that restores the LAPIC state does so before it writes the processor's APIC base or LAPIC page; the steps before it, such as the registers', have run. Five of the seven other modes would give the restored processor an INIT, a startup IPI, an NMI, or an SMI, or wake it from halt, none of which the saved state holds; lowest priority and ExtINT, which KVM delivers as a fixed interrupt or ignores, are rejected too. The guest cannot write the delivery mode, so no capture holds a state with any of them. The check accepts a masked entry, which delivers nothing, and the rules above, such as the rejection of an armed periodic or TSC-deadline timer, still apply to its state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 30b755406, position 282, 158 commits past the mapped tip. Position 282, the one commit of microsoft/openvmm#5102 on b723cbed3, the head of feature/microvm/dev, is not yet assigned to a PR. Position 282 makes the KVM backend's set_apic reject a LAPIC page whose unmasked timer entry has a delivery mode other than Fixed, which KVM would fire as an INIT, a startup IPI, an NMI, or an SMI, or as a wake of a halted VP, none of which the saved state holds, in code that upstream main has. It adds check_restorable_apic_timer, which set_apic calls first, a unit test, a KVM-only contract test of virt_backend_tests in a module of its own, and a paragraph and a sentence of the Guide. It changes files that PR01's position 002, PR05's positions 007 and 008, PR10's positions 022, 271, and 273, and positions 140, 248, 250, 251, 253, 279, and 281 also change, and adds to no line that a mapped PR adds. Replaying it on upstream main, 7d90c4104, conflicts in vp_state.rs, at the start of set_apic, whose timer reset position 253 adds, and from the end of the AccessVpState implementation on, which positions 250, 251, 253, and 279 add; in the Guide's KVM timekeeping section, which positions 250 and 253 extend; in the Guide's page of the contract tests, whose kvm_only sentence is position 279's; and in the contract tests' main.rs, beside the attribute that position 008 drops. So it follows positions 008, 250, 251, 253, and 279 upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/issue-1365-apic-timer-mode
branch
from
November 14, 2026 17:06
ea60a0d to
89343de
Compare
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1365-apic-timer-mode
branch
November 14, 2026 17:59
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #1365
What and why
This pins microsoft/openvmm#5102 at
30b7554069182a5262099f902c4a203d54c8bb88. With the pin, a KVM restore refuses a VP's LAPIC page whose unmasked LVT timer entry has a delivery mode other than Fixed:set_apicfails withKvmError::InvalidState, whichrestore_allreports as theStateErrorof the apic element, before it changes the VP's APIC base, its LAPIC page, or the timer that it stopped with. The elements thatrestore_allrestores before the APIC, the registers among them, have run by then: KVM's write of CR8 can change the LAPIC's task priority. Before, the restore succeeded, and KVM then delivered the timer's expiry in that mode, which gives the restored VP an INIT, an NMI, or an SMI, or wakes it from halt, none of which the snapshot holds.At NVX's previous pin,
b723cbed3:Cause.
KvmVpStateAccess::set_apicinvmm_core/virt_kvm/src/arch/x86_64/vp_state.rsgaveKVM_SET_LAPICthe saved page unchecked. KVM starts the timer again from the page's counts and delivers each expiry through the delivery mode of the LVT timer entry (kvm_apic_local_deliver). A timer that has expired, as a one-shot timer whose current count is 0 has, fires at once: insideKVM_SET_LAPICwhen APICv is active, and at the nextKVM_RUNotherwise. The restore appears to succeed, sincerestore_alldoes not read KVM's LAPIC state back (Apic::can_compareis false while the backend cannot freeze time). A guest cannot write the entry's delivery mode, because KVM masks its writes to the entry (apic_lvt_maskinkvm_lapic_reg_write), so a save never holds such a page, and only a crafted snapshot reaches this. MSHV and WHP hand the page to their hypervisor, whose APIC is not OpenVMM's, and are unchanged and not examined.Reproduction. On nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3, nested KVM without APICv), the revision's contract test sets an expired one-shot timer in each delivery mode from 1 to 7 on a real KVM VP through the public
virtinterfaces. Atb723cbed3it fails with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]"; with the pin it passes. A scratch test that is not part of the change gaveKVM_SET_LAPICthe same page in each mode and ran a real-mode BSP that spins with interrupts enabled for 200 ms:KVM_RUNKVM_RUNfailed, and every later ioctl on the VM returned EIO.0xfff0.The host has no APICv, so delivery inside the ioctl was not observed. The contract test and the scratch test drive
set_apicandKVM_SET_LAPICthemselves, since a save never holds such a page: reaching it throughnvx-rstakes a snapshot whose LAPIC page was edited.The fix, and the decisions that the issue left open
The issue's
fix.patch, written against OpenVMM3cf58a5234e, addscheck_restorable_apic_timer, whichset_apiccalls first. The pin keeps its rule and adapts it to the currentvirt_kvm:set_apicrefuses an entry that is unmasked and has a delivery mode other than Fixed, as the patch does, whatever the timer's mode and whether or not the timer has expired: a timer that has not expired fires in that mode when it does, a periodic timer fires in it again, and no save holds such an entry. Lowest priority and ExtINT are refused too, though KVM delivers them as a fixed interrupt or ignores them. A masked entry is accepted: it delivers nothing, and a guest write that unmasks it rewrites its delivery mode too.set_apic, before the reset of the timer that the VP stopped with, which openvmm: keep a restore from firing a taken one-shot LAPIC timer again #1514's pin added, and before the writes of the APIC base and the page. A refusedset_apictherefore changes nothing. It reads the saved page's entry, whichkvm_lapic_pagedoes not change.KvmError::InvalidStatewith a static message, as the patch has it, and no new variant or type.virt_backend_teststhat restores into a real KVM VP, and that registers with thekvm_onlyform ofbackend_test!that [DO NOT REVIEW] virt_kvm: refuse a restored EFER.SVME for a guest whose CPUID has no SVM openvmm#5097 added (position 279), since only KVM owes it. No NVX test is added: none builds a snapshot with an edited LAPIC page, and the contract test covers the restore at OpenVMM'svirtinterface.Why the check cannot refuse a legitimate restore
A legitimate restore holds a page that
KVM_GET_LAPICreturned, or a reset's. KVM masks a guest's write to the timer's entry, so a saved entry has the delivery mode Fixed, masked or not, and the reset page's entry is masked. Every page that OpenVMM saves passes the check, so the restore of a saved snapshot behaves as before.What the OpenVMM pull request changes
microsoft/openvmm#5102 adds one signed commit on
b723cbed3, the head offeature/microvm/dev. It has a good signature (openvmm_pin.py source).30b755406set_apiccalls the newcheck_restorable_apic_timerfirst, which refuses an unmasked LVT timer entry whose delivery mode is not Fixed withKvmError::InvalidState. The unit testan_unmasked_timer_that_is_not_fixed_is_refusedcovers the delivery modes 1 to 7 of a one-shot, a periodic, and a TSC-deadline timer, and the masked entries, the Fixed mode, and the reset value that restore. The contract testapic::kvm::restore_refuses_a_non_fixed_apic_timerrequires on a real VP thatset_apicrefuses an expired one-shot timer in every mode from 1 to 7 and leaves the APIC state unchanged, and that it accepts an expired Fixed timer and masked timers in every mode. With the check weakened, the unit test or the contract test fails, as below.The issue's patch is data: its rule is the commit's, and its test is the model of the unit test.
What this pull request changes in NVX
f737fc9141fmoves theopenvmmgitlink to the pin, and nothing else.ae2d56ee537states in 3.2.5 ofdoc/design/3.2-clock-and-time.mdthat a KVM restore rejects a LAPIC state whose unmasked timer entry has a delivery mode other than fixed, before the step that restores the LAPIC state writes anything, and that the check accepts a masked entry without changing the other rules for the timer.89343defb1amaps position 282 in the OpenVMM roadmap.No
nvx-rschange: the pin changes only what the KVM backend accepts. No version is created or bumped, and no snapshot, saved-state, or wire format changes.Compatibility of existing snapshots
The check accepts every snapshot that a VP saved, since its timer entry's delivery mode is Fixed, so such a restore behaves as before. Only a page with an unmasked entry in another mode, which a save never holds, is refused, and it never restored faithfully.
Pin details
7d90c4104, has, and replaying it there conflicts invp_state.rs, in the Guide's two pages, and in the contract tests'main.rs, so it follows positions 008, 250, 251, 253, and 279 upstream.b723cbed3, is NVX's previous pin and the head offeature/microvm/dev.nvx-rscopies from OpenVMM match the revision's (openvmm_vectorsandopenvmm_cpu_profiles, below).Landing order
microsoft/openvmm#5102 lands first, by fast-forwarding
feature/microvm/devfromb723cbed3to30b7554069182a5262099f902c4a203d54c8bb88, whichpython scripts/openvmm_pin.py land 30b7554069182a5262099f902c4a203d54c8bb88 --remote REMOTE --pushdoes. This pull request merges after that, through the merge queue. OpenVMM's merge button squashes, which would leave the pinned revision off the branch.Validation
NVX
Run on Windows:
python scripts/openvmm_pin.py source 30b7554069182a5262099f902c4a203d54c8bb88 --pr 5102: passed. The revision is the pull request's head, with a good signature.python scripts/openvmm_pin.py check-roadmap --worktree: passed, with 282 positions and 293 references.python scripts/openvmm_pin.py check-branch --topic: passed, with three signed commits and no coordinated files.python scripts/check_markdown_links.pyandgit diff --check: passed.python scripts/nvx.py verify: passed.cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored(4 passed) and--test openvmm_cpu_profiles -- --ignored(1 passed).python scripts/check_test_changes.py --base origin/devreports no change to an existing test, so the pull request needs notests-modifiedlabel.python scripts/local_gates.py --base origin/dev --run --keep-going:test-changes,markdown-links,test-layout,openvmm-vectors, andopenvmm-cpu-profilespassed. I leftnvx-cli, which the change affects only through two documents, andopenvmm-ci, which waits for OpenVMM's own jobs, to this pull request's CI.OpenVMM
With Rust 1.97.1 and cargo-nextest 0.9.133, on nvx-kvm-test, for the tree of the pinned commit (the five files match the commit's blobs):
b723cbed3with only the new test files,cargo nextest run -p vmm_tests --test virt_backend_tests -E 'test(/restore_refuses/)'fails with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]"; with the commit it passes.cargo check,cargo clippy --all-targetsfor the gnu and musl targets, andcargo doc --no-depsonvirt_kvmhave no warnings, andcargo clippy --all-targetsandcargo doc --no-depsonvmm_testshave none.cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvmpasses 84 tests, the ignored/dev/kvmones included.cargo nextest run -p vmm_tests --test virt_backend_testsruns the two contract tests that KVM runs, and both pass.cargo xtask fmt --fix, run on Windows, changes nothing.set_apicthat ignores the refusal fails the contract test.vmm_testsare left to OpenVMM's CI of the pull request.Hosts
On nvx-kvm-test (a virtual machine with an Intel Xeon Platinum 8573C and Linux 6.18.57.1-1.azl3), OpenVMM was built from the pin's tree in a release build, which replaced the packaged binary of the release
v0.1.0-dev.9e41ebd10be8(SHA-256193d1063…to04915a93…), andpython scripts/nvx.py test-microvm --backend kvm --timeout 180 --scenario NAME --processors COUNTSpassed on its first attempt forguest-boot(1),interrupt-map,smp,smp-snapshot,snapshot-core,snapshot-rollback,restore-downtime,restore-processors,snapshot-tiers,time-abi-conformance, andtime-host-correction(1 and 2), andsmp-lapic(1, 2, 4, and 8). An earlier run, under a host load of 25 to 45 from other jobs, failed the warp probes ofsmp-snapshot,smp-lapic, andrestore-downtimeequally with and without the change, so I repeated it on a quiet host.set_apicfails withInvalidStateand changes nothing; the scenarios pass.The change is KVM-only, so I did not run nvx-mshv-test or nvx-whp-test.
CI
I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux and Windows, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI, which passed:
89343defb1apassed, withRequired status check: the microVM scenarios (two shards, and guests and sandboxes) on Linux KVM, Linux MSHV (with its debug kernel), and Windows WHP (with its runtime adapter), the NVX CLI gates on Linux and Windows,nvx-rsand the binaries ofnvx-rsand OpenVMM on Linux and Windows, the package, platform, and performance regression gates, andOpenVMM CI of the pinned revision. It superseded run 43618232226 of the headea60a0d3106, which the push of the review fixes cancelled; that run had no failure.30b7554069182a5262099f902c4a203d54c8bb88passed the six jobs thatopenvmm-cirequires:unit tests [x64-linux]andunit tests [x64-windows], andrun vmm-testsonx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd. The log ofrun vmm-tests [x64-linux-amd-kvm]showsvmm_tests::virt_backend_tests apic::kvm::restore_refuses_a_non_fixed_apic_timerpassing, so the contract test runs in CI.run vmm-tests [x64-windows-amd-snp]is the run's only failed job, and I did not wait for or rerun it:openvmm-cidoes not require it, and a Windows SEV-SNP job does not runvirt_kvm, the only crate that the commit changes. The ignored/dev/kvmtests ofvirt_kvmrun in no OpenVMM CI job, so the KVM evidence for the unit-level behavior is the run on nvx-kvm-test above.89343defb1aand recommended approval, with no open issue, after two threads on the first head,ea60a0d3106, which I fixed inae2d56ee537: the guarantee of the check is now scoped to the step that restores the LAPIC state, and the masked-entry sentence says that the check accepts it while the other rules for the timer still apply.python scripts/openvmm_pin.py land 30b7554069182a5262099f902c4a203d54c8bb88 --remote origin --pushfast-forwardedfeature/microvm/devfromb723cbed3to the pin, and [DO NOT REVIEW] virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed openvmm#5102 shows MERGED at that commit. The merge queue's group run, 43622374805, passed, and the pull request merged as0b792e825e7.