Skip to content

openvmm: pin the restore's check of the APIC timer's delivery mode - #1590

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1365-apic-timer-mode
Nov 14, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1365-apic-timer-mode

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1365

What and why

This pins microsoft/openvmm#5102 at 30b7554069182a5262099f902c4a203d54c8bb88. With the pin, a KVM restore refuses a VP's LAPIC page whose unmasked LVT timer entry has a delivery mode other than Fixed: set_apic fails with KvmError::InvalidState, which restore_all reports as the StateError of the apic element, before it changes the VP's APIC base, its LAPIC page, or the timer that it stopped with. The elements that restore_all restores before the APIC, the registers among them, have run by then: KVM's write of CR8 can change the LAPIC's task priority. Before, the restore succeeded, and KVM then delivered the timer's expiry in that mode, which gives the restored VP an INIT, an NMI, or an SMI, or wakes it from halt, none of which the snapshot holds.

At NVX's previous pin, b723cbed3:

  • Cause. KvmVpStateAccess::set_apic in vmm_core/virt_kvm/src/arch/x86_64/vp_state.rs gave KVM_SET_LAPIC the saved page unchecked. KVM starts the timer again from the page's counts and delivers each expiry through the delivery mode of the LVT timer entry (kvm_apic_local_deliver). A timer that has expired, as a one-shot timer whose current count is 0 has, fires at once: inside KVM_SET_LAPIC when APICv is active, and at the next KVM_RUN otherwise. The restore appears to succeed, since restore_all does not read KVM's LAPIC state back (Apic::can_compare is false while the backend cannot freeze time). A guest cannot write the entry's delivery mode, because KVM masks its writes to the entry (apic_lvt_mask in kvm_lapic_reg_write), so a save never holds such a page, and only a crafted snapshot reaches this. MSHV and WHP hand the page to their hypervisor, whose APIC is not OpenVMM's, and are unchanged and not examined.

  • Reproduction. On nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3, nested KVM without APICv), the revision's contract test sets an expired one-shot timer in each delivery mode from 1 to 7 on a real KVM VP through the public virt interfaces. At b723cbed3 it fails with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]"; with the pin it passes. A scratch test that is not part of the change gave KVM_SET_LAPIC the same page in each mode and ran a real-mode BSP that spins with interrupts enabled for 200 ms:

    Delivery mode What the VP did at KVM_RUN
    Fixed (0) The guest took the interrupt.
    Lowest priority (1) The guest took it as a fixed interrupt.
    SMI (2) KVM_RUN failed, and every later ioctl on the VM returned EIO.
    Remote read (3) No visible effect on a running BSP; it wakes a halted VP.
    NMI (4) An NMI was injected.
    INIT (5) The VP reset: an emulation failure at rip 0xfff0.
    Startup (6), ExtINT (7) No visible effect on a running BSP.

    The host has no APICv, so delivery inside the ioctl was not observed. The contract test and the scratch test drive set_apic and KVM_SET_LAPIC themselves, since a save never holds such a page: reaching it through nvx-rs takes a snapshot whose LAPIC page was edited.

The fix, and the decisions that the issue left open

The issue's fix.patch, written against OpenVMM 3cf58a5234e, adds check_restorable_apic_timer, which set_apic calls first. The pin keeps its rule and adapts it to the current virt_kvm:

  1. The rule. set_apic refuses an entry that is unmasked and has a delivery mode other than Fixed, as the patch does, whatever the timer's mode and whether or not the timer has expired: a timer that has not expired fires in that mode when it does, a periodic timer fires in it again, and no save holds such an entry. Lowest priority and ExtINT are refused too, though KVM delivers them as a fixed interrupt or ignores them. A masked entry is accepted: it delivers nothing, and a guest write that unmasks it rewrites its delivery mode too.
  2. Where it runs. It is the first statement of set_apic, before the reset of the timer that the VP stopped with, which openvmm: keep a restore from firing a taken one-shot LAPIC timer again #1514's pin added, and before the writes of the APIC base and the page. A refused set_apic therefore changes nothing. It reads the saved page's entry, which kvm_lapic_page does not change.
  3. The existing error. KvmError::InvalidState with a static message, as the patch has it, and no new variant or type.
  4. The timer entry only. The check reads the entry's mask and delivery mode. It checks neither the other fields of the entry nor the other LVT entries, and the refire of an expired one-shot Fixed timer, which the issue names as a related limit, is openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390's and stays as it is.
  5. Tests. The patch's unit test is the model of the OpenVMM commit's, which also checks the timer modes. The patch has no KVM-backed test, so the commit adds a contract test of virt_backend_tests that restores into a real KVM VP, and that registers with the kvm_only form of backend_test! that [DO NOT REVIEW] virt_kvm: refuse a restored EFER.SVME for a guest whose CPUID has no SVM openvmm#5097 added (position 279), since only KVM owes it. No NVX test is added: none builds a snapshot with an edited LAPIC page, and the contract test covers the restore at OpenVMM's virt interface.
  6. Docs. The Guide's KVM timekeeping section describes the check, and the page of the backend contract tests names the refusal among the contracts that only KVM owes. 3.2.5 of NVX's clock and time design says that a KVM restore rejects such a state.

Why the check cannot refuse a legitimate restore

A legitimate restore holds a page that KVM_GET_LAPIC returned, or a reset's. KVM masks a guest's write to the timer's entry, so a saved entry has the delivery mode Fixed, masked or not, and the reset page's entry is masked. Every page that OpenVMM saves passes the check, so the restore of a saved snapshot behaves as before.

What the OpenVMM pull request changes

microsoft/openvmm#5102 adds one signed commit on b723cbed3, the head of feature/microvm/dev. It has a good signature (openvmm_pin.py source).

Position Commit Change
282 30b755406 set_apic calls the new check_restorable_apic_timer first, which refuses an unmasked LVT timer entry whose delivery mode is not Fixed with KvmError::InvalidState. The unit test an_unmasked_timer_that_is_not_fixed_is_refused covers the delivery modes 1 to 7 of a one-shot, a periodic, and a TSC-deadline timer, and the masked entries, the Fixed mode, and the reset value that restore. The contract test apic::kvm::restore_refuses_a_non_fixed_apic_timer requires on a real VP that set_apic refuses an expired one-shot timer in every mode from 1 to 7 and leaves the APIC state unchanged, and that it accepts an expired Fixed timer and masked timers in every mode. With the check weakened, the unit test or the contract test fails, as below.

The issue's patch is data: its rule is the commit's, and its test is the model of the unit test.

What this pull request changes in NVX

  • f737fc9141f moves the openvmm gitlink to the pin, and nothing else.
  • ae2d56ee537 states in 3.2.5 of doc/design/3.2-clock-and-time.md that a KVM restore rejects a LAPIC state whose unmasked timer entry has a delivery mode other than fixed, before the step that restores the LAPIC state writes anything, and that the check accepts a masked entry without changing the other rules for the timer.
  • 89343defb1a maps position 282 in the OpenVMM roadmap.

No nvx-rs change: the pin changes only what the KVM backend accepts. No version is created or bumped, and no snapshot, saved-state, or wire format changes.

Compatibility of existing snapshots

The check accepts every snapshot that a VP saved, since its timer entry's delivery mode is Fixed, so such a restore behaves as before. Only a page with an unmasked entry in another mode, which a save never holds, is refused, and it never restored faithfully.

Pin details

  • Roadmap. The pin is 158 commits past the mapped tip. Position 282 is not yet assigned to a PR, and adds to no line that a mapped PR adds. It changes code that upstream main, 7d90c4104, has, and replaying it there conflicts in vp_state.rs, in the Guide's two pages, and in the contract tests' main.rs, so it follows positions 008, 250, 251, 253, and 279 upstream.
  • Prerequisites. None. The pull request's base, b723cbed3, is NVX's previous pin and the head of feature/microvm/dev.
  • Equivalence. None claimed; this pin replaces no reviewed revision.
  • Coordinated changes. None. The control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's (openvmm_vectors and openvmm_cpu_profiles, below).

Landing order

microsoft/openvmm#5102 lands first, by fast-forwarding feature/microvm/dev from b723cbed3 to 30b7554069182a5262099f902c4a203d54c8bb88, which python scripts/openvmm_pin.py land 30b7554069182a5262099f902c4a203d54c8bb88 --remote REMOTE --push does. This pull request merges after that, through the merge queue. OpenVMM's merge button squashes, which would leave the pinned revision off the branch.

Validation

NVX

Run on Windows:

  • python scripts/openvmm_pin.py source 30b7554069182a5262099f902c4a203d54c8bb88 --pr 5102: passed. The revision is the pull request's head, with a good signature.
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed, with 282 positions and 293 references.
  • python scripts/openvmm_pin.py check-branch --topic: passed, with three signed commits and no coordinated files.
  • python scripts/check_markdown_links.py and git diff --check: passed.
  • python scripts/nvx.py verify: passed.
  • cargo test --locked --manifest-path nvx-rs/Cargo.toml --no-default-features --test openvmm_vectors -- --ignored (4 passed) and --test openvmm_cpu_profiles -- --ignored (1 passed).
  • python scripts/check_test_changes.py --base origin/dev reports no change to an existing test, so the pull request needs no tests-modified label.
  • python scripts/local_gates.py --base origin/dev --run --keep-going: test-changes, markdown-links, test-layout, openvmm-vectors, and openvmm-cpu-profiles passed. I left nvx-cli, which the change affects only through two documents, and openvmm-ci, which waits for OpenVMM's own jobs, to this pull request's CI.

OpenVMM

With Rust 1.97.1 and cargo-nextest 0.9.133, on nvx-kvm-test, for the tree of the pinned commit (the five files match the commit's blobs):

  • Reproduction. The table above: at b723cbed3 with only the new test files, cargo nextest run -p vmm_tests --test virt_backend_tests -E 'test(/restore_refuses/)' fails with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]"; with the commit it passes.
  • Linux x64: cargo check, cargo clippy --all-targets for the gnu and musl targets, and cargo doc --no-deps on virt_kvm have no warnings, and cargo clippy --all-targets and cargo doc --no-deps on vmm_tests have none. cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm passes 84 tests, the ignored /dev/kvm ones included. cargo nextest run -p vmm_tests --test virt_backend_tests runs the two contract tests that KVM runs, and both pass. cargo xtask fmt --fix, run on Windows, changes nothing.
  • Mutations. A check that refuses nothing, or that refuses a masked entry too, fails both new tests; a check that accepts lowest priority and ExtINT fails the unit test; and a set_apic that ignores the refusal fails the contract test.
  • Not run. The Windows and MSHV builds of vmm_tests are left to OpenVMM's CI of the pull request.

Hosts

On nvx-kvm-test (a virtual machine with an Intel Xeon Platinum 8573C and Linux 6.18.57.1-1.azl3), OpenVMM was built from the pin's tree in a release build, which replaced the packaged binary of the release v0.1.0-dev.9e41ebd10be8 (SHA-256 193d1063… to 04915a93…), and python scripts/nvx.py test-microvm --backend kvm --timeout 180 --scenario NAME --processors COUNTS passed on its first attempt for guest-boot (1), interrupt-map, smp, smp-snapshot, snapshot-core, snapshot-rollback, restore-downtime, restore-processors, snapshot-tiers, time-abi-conformance, and time-host-correction (1 and 2), and smp-lapic (1, 2, 4, and 8). An earlier run, under a host load of 25 to 45 from other jobs, failed the warp probes of smp-snapshot, smp-lapic, and restore-downtime equally with and without the change, so I repeated it on a quiet host.

Host Platform Host type Before the fix With the pin
nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3) linux-kvm virtual-machine The reproduction above: the restore succeeds, and KVM delivers the expired timer in its mode. set_apic fails with InvalidState and changes nothing; the scenarios pass.

The change is KVM-only, so I did not run nvx-mshv-test or nvx-whp-test.

CI

I left the rest of the KVM, MSHV, and WHP microVM suites, the NVX CLI gates on Linux and Windows, the runtime adapter, the platform and performance gates, and OpenVMM's own CI of the pin to CI, which passed:

  • This pull request. Run 43619266466 of the head 89343defb1a passed, with Required status check: the microVM scenarios (two shards, and guests and sandboxes) on Linux KVM, Linux MSHV (with its debug kernel), and Windows WHP (with its runtime adapter), the NVX CLI gates on Linux and Windows, nvx-rs and the binaries of nvx-rs and OpenVMM on Linux and Windows, the package, platform, and performance regression gates, and OpenVMM CI of the pinned revision. It superseded run 43618232226 of the head ea60a0d3106, which the push of the review fixes cancelled; that run had no failure.
  • OpenVMM. Run 43616184876 of [DO NOT REVIEW] virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed openvmm#5102 at 30b7554069182a5262099f902c4a203d54c8bb88 passed the six jobs that openvmm-ci requires: unit tests [x64-linux] and unit tests [x64-windows], and run vmm-tests on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd. The log of run vmm-tests [x64-linux-amd-kvm] shows vmm_tests::virt_backend_tests apic::kvm::restore_refuses_a_non_fixed_apic_timer passing, so the contract test runs in CI.
  • Not required. run vmm-tests [x64-windows-amd-snp] is the run's only failed job, and I did not wait for or rerun it: openvmm-ci does not require it, and a Windows SEV-SNP job does not run virt_kvm, the only crate that the commit changes. The ignored /dev/kvm tests of virt_kvm run in no OpenVMM CI job, so the KVM evidence for the unit-level behavior is the run on nvx-kvm-test above.
  • Review. Copilot reviewed the head 89343defb1a and recommended approval, with no open issue, after two threads on the first head, ea60a0d3106, which I fixed in ae2d56ee537: the guarantee of the check is now scoped to the step that restores the LAPIC state, and the masked-entry sentence says that the check accepts it while the other rules for the timer still apply.
  • Landing. python scripts/openvmm_pin.py land 30b7554069182a5262099f902c4a203d54c8bb88 --remote origin --push fast-forwarded feature/microvm/dev from b723cbed3 to the pin, and [DO NOT REVIEW] virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed openvmm#5102 shows MERGED at that commit. The merge queue's group run, 43622374805, passed, and the pull request merged as 0b792e825e7.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The clock-and-time documentation overstates both masked timer acceptance and the scope of the no-change guarantee during restore.

2 open issues
What changed in this PR

This PR pins OpenVMM’s KVM restore validation for unmasked LAPIC timers with non-Fixed delivery modes, addressing #1365 while preserving legitimate snapshots.

Changes:

  • Advances the OpenVMM pin to 30b755406.
  • Documents the delivery-mode restriction in the clock-and-time design.
  • Maps position 282 and its upstream dependencies in the roadmap.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin, chain counts, and position 282’s upstream mapping.
doc/​design/​3.2-clock-and-time.md Describes KVM’s timer delivery-mode validation and masked-entry behavior.

Review effort: Balanced

Comment thread doc/design/3.2-clock-and-time.md Resolved
Comment thread doc/design/3.2-clock-and-time.md Resolved
Restoring a VP whose saved APIC page held an unmasked LVT timer entry
with a delivery mode other than Fixed succeeded on KVM, and KVM then
fired the timer in that mode once it had expired, as a one-shot timer
whose current count is 0 has: inside KVM_SET_LAPIC when APICv is
active, and at the next KVM_RUN otherwise. The restored VP took an
INIT, a startup IPI, an NMI, or an SMI, or woke from halt, none of
which the snapshot holds (#1365).

Pin 30b7554069182a5262099f902c4a203d54c8bb88, the head of
microsoft/openvmm#5102, one commit on b723cbed3, the head of
feature/microvm/dev:

- 30b755406 virt_kvm: reject a restored APIC timer whose delivery mode
  is not Fixed

set_apic now rejects such a page with KvmError::InvalidState as its
first step, before it forgets the timer that the VP stopped with or
changes the APIC base or the page, so the restore fails with the apic
element's StateError, and set_apic itself leaves the VP's APIC as it
was. The elements that restore_all restores before the APIC, the
registers among them, have run: KVM's write of CR8 can change the
LAPIC's task priority. It refuses every unmasked entry with a mode other
than Fixed, lowest priority and ExtINT included, and accepts a masked
entry. A guest cannot write the entry's delivery mode, since KVM masks
its writes, so a save never holds such a page, a snapshot that a VP
saved restores as before, and only a crafted snapshot is refused. MSHV
and WHP are unchanged. No version is created or bumped, and no snapshot
or wire format changes.

Tests: on nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57,
nested KVM without APICv), the revision's contract test
restore_refuses_a_non_fixed_apic_timer fails at b723cbed3 with "the
restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5,
6, 7]" and passes at the pin, as does its unit test
an_unmasked_timer_that_is_not_fixed_is_refused. A scratch test that gave
KVM_SET_LAPIC the same page showed what KVM does with it at KVM_RUN:
INIT resets the VP, NMI injects an NMI, SMI kills the VM, and lowest
priority is taken as a fixed interrupt. Mutants of the check fail the
tests. At the pin, the 84 tests of virt_kvm and kvm pass with cargo
nextest, the ignored ones included. With OpenVMM built from the pin's
tree, which replaced the packaged binary (SHA-256 193d1063 to 04915a93),
and the artifacts of v0.1.0-dev.9e41ebd10be8, the microVM scenarios
guest-boot (1 processor), interrupt-map, smp, smp-snapshot,
snapshot-core, snapshot-rollback, restore-downtime, restore-processors,
snapshot-tiers, time-abi-conformance, and time-host-correction pass at 1
and 2 processors, and smp-lapic at 1, 2, 4, and 8, on nvx-kvm-test. The
change is KVM-only, so nvx-mshv-test and nvx-whp-test were not run.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3.2.5 specified how a restore runs a one-shot LAPIC timer, and what
KVM's LAPIC state write does to one that has expired, but not that KVM
delivers the expiry of an unmasked timer in the delivery mode of its LVT
entry, which a saved state that no capture holds can set to something
other than a fixed interrupt (#1365).

It now says that on KVM a restore rejects a LAPIC state whose unmasked
timer entry has a delivery mode other than fixed, and that the step that
restores the LAPIC state does so before it writes the processor's APIC
base or LAPIC page; the steps before it, such as the registers', have
run. Five of the seven other modes would give the restored processor an
INIT, a startup IPI, an NMI, or an SMI, or wake it from halt, none of
which the saved state holds; lowest priority and ExtINT, which KVM
delivers as a fixed interrupt or ignores, are rejected too. The guest
cannot write the delivery mode, so no capture holds a state with any of
them. The check accepts a masked entry, which delivers nothing, and the
rules above, such as the rejection of an armed periodic or TSC-deadline
timer, still apply to its state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 30b755406, position 282, 158 commits past the
mapped tip. Position 282, the one commit of microsoft/openvmm#5102 on
b723cbed3, the head of feature/microvm/dev, is not yet assigned to a PR.

Position 282 makes the KVM backend's set_apic reject a LAPIC page whose
unmasked timer entry has a delivery mode other than Fixed, which KVM
would fire as an INIT, a startup IPI, an NMI, or an SMI, or as a wake of
a halted VP, none of which the saved state holds, in code that upstream
main has. It adds check_restorable_apic_timer, which set_apic calls
first, a unit test, a KVM-only contract test of virt_backend_tests in a
module of its own, and a paragraph and a sentence of the Guide. It
changes files that PR01's position 002, PR05's positions 007 and 008,
PR10's positions 022, 271, and 273, and positions 140, 248, 250, 251,
253, 279, and 281 also change, and adds to no line that a mapped PR
adds. Replaying it on upstream main, 7d90c4104, conflicts in
vp_state.rs, at the start of set_apic, whose timer reset position 253
adds, and from the end of the AccessVpState implementation on, which
positions 250, 251, 253, and 279 add; in the Guide's KVM timekeeping
section, which positions 250 and 253 extend; in the Guide's page of the
contract tests, whose kvm_only sentence is position 279's; and in the
contract tests' main.rs, beside the attribute that position 008 drops.
So it follows positions 008, 250, 251, 253, and 279 upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-1365-apic-timer-mode branch from ea60a0d to 89343de Compare November 14, 2026 17:06
Copilot AI balanced review requested due to automatic review settings November 14, 2026 17:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the OpenVMM pin and documentation changes.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 14, 2026
Merged via the queue into dev with commit 0b792e8 Nov 14, 2026
49 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1365-apic-timer-mode branch November 14, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: a restored non-Fixed APIC timer makes KVM inject INIT, SIPI or NMI

2 participants