Skip to content

[DO NOT REVIEW] virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed - #5102

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1365-apic-timer-mode
Nov 14, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1365-apic-timer-mode

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Keeps a restore from giving KVM an APIC timer that delivers in a mode other than Fixed, for microsoft/nvx#1365.

One commit on b723cbed3, the head of feature/microvm/dev. microsoft/nvx#1590 pins it for microsoft/nvx#1365; this pull request lands first, by fast-forwarding feature/microvm/dev to 30b755406.

Reject a restored APIC timer whose delivery mode is not Fixed (30b755406)

  • Cause. KvmVpStateAccess::set_apic gave KVM_SET_LAPIC the saved LAPIC page unchanged. KVM starts the timer again from the page's counts and delivers each expiry through the delivery mode of the LVT timer entry (kvm_apic_local_deliver). A timer that has expired, as a one-shot timer whose current count is 0 has, fires at once: inside KVM_SET_LAPIC when APICv is active, and at the next KVM_RUN otherwise. A page whose entry is unmasked and has a mode other than Fixed therefore gives the restored VP an INIT, a startup IPI, an NMI, or an SMI, or wakes it from halt (remote read), none of which the saved state holds. The restore appears to succeed, since restore_all does not read KVM's LAPIC state back: Apic::can_compare is false while the backend cannot freeze time. A guest cannot set the entry's delivery mode, because KVM masks its writes to the entry (apic_lvt_mask in kvm_lapic_reg_write), so a save never holds such a page, and only a damaged or crafted saved state does. MSHV and WHP hand the page to their hypervisor, whose APIC is not OpenVMM's, and are unchanged and not examined.
  • Change. set_apic calls check_restorable_apic_timer as its first step, before it forgets the timer that the VP stopped with or changes the APIC base or the page. The check fails with KvmError::InvalidState for an unmasked entry whose delivery mode is not Fixed, so restore_all fails with the StateError of the apic element, as it does for a pending ExtINT, and set_apic leaves the VP's APIC as it was. The elements that restore_all restores before the APIC, the registers among them, have run: KVM's write of CR8 can change the LAPIC's task priority. It refuses every such mode, whatever the timer's mode and whether or not the timer has expired, lowest priority and ExtINT included, which KVM delivers as a fixed interrupt or ignores, since no save holds them either.
  • A legitimate restore passes. A masked entry is accepted: it delivers nothing, and a guest write that unmasks it rewrites its delivery mode too. Every page that OpenVMM saves passes the check, and so does the reset page, whose timer is masked.
  • Tests. The unit test an_unmasked_timer_that_is_not_fixed_is_refused checks the refusal for the delivery modes 1 to 7 of a one-shot, a periodic, and a TSC-deadline timer, and that a masked entry, the Fixed mode, and the reset value are accepted. The backend contract test apic::kvm::restore_refuses_a_non_fixed_apic_timer registers with the kvm_only form of backend_test! that [DO NOT REVIEW] virt_kvm: refuse a restored EFER.SVME for a guest whose CPUID has no SVM #5097 added, sets an expired one-shot timer in each of those modes on a real KVM VP through the public virt interfaces, and requires that set_apic refuses each one and leaves the APIC state unchanged, and that it accepts an expired Fixed timer and masked timers in every mode. The virt_backend_tests executable runs in run vmm-tests [x64-linux-amd-kvm], whereas no CI job runs the ignored /dev/kvm tests of virt_kvm.
  • Docs. The Guide's KVM timekeeping section describes the check, and the page of the backend contract tests lists the refusal among the contracts that only KVM owes.

No version is created or bumped, and no snapshot, saved-state, or wire format changes.

Validation

With Rust 1.97.1 and cargo-nextest 0.9.133, on a KVM virtual machine (Intel Xeon Platinum 8573C, Linux 6.18.57.1-1.azl3, nested KVM without APICv or the VMX preemption timer):

  • Reproduction. At b723cbed3 with only the new test files, cargo nextest run -p vmm_tests --test virt_backend_tests -E 'test(/restore_refuses/)' fails with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]"; with this change it passes. A scratch test, which is not part of this change, gave KVM_SET_LAPIC the same page for each delivery mode and ran a real-mode BSP that spins with interrupts enabled. INIT resets the VP (an emulation failure at rip 0xfff0), NMI injects an NMI, SMI makes KVM_RUN fail and every later ioctl on the VM return EIO, and lowest priority takes the vector as a fixed interrupt; remote read, startup IPI, and ExtINT have no visible effect on a running BSP. The host has no APICv, so delivery inside the ioctl was not observed.
  • Checks, on this commit. cargo check, cargo clippy --all-targets for the gnu and musl targets, and cargo doc --no-deps on virt_kvm have no warnings, and cargo clippy --all-targets and cargo doc --no-deps on vmm_tests have none. cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm passes 84 tests. cargo nextest run -p vmm_tests --test virt_backend_tests runs the two contract tests that KVM runs, and both pass. cargo xtask fmt --fix, run on Windows, changes nothing.
  • Mutations. A check that refuses nothing, or that refuses a masked entry too, fails both new tests; a check that accepts lowest priority and ExtINT fails the unit test; and a set_apic that ignores the refusal fails the contract test.
  • Not run. The Windows and MSHV builds of vmm_tests are left to this pull request's CI.

KvmVpStateAccess::set_apic gave KVM_SET_LAPIC the saved LAPIC page as
it was. KVM starts the timer again from the page's counts and delivers
an expiry through the delivery mode of the LVT timer entry, in
kvm_apic_local_deliver. A page whose entry is unmasked and has a mode
other than Fixed therefore makes KVM interrupt the restored VP in that
mode once the timer has expired, as a one-shot timer whose current
count is 0 has: inside KVM_SET_LAPIC when APICv is active, and at the
next KVM_RUN otherwise. Depending on the mode, the VP takes an INIT, a
startup IPI, an NMI, or an SMI, or wakes from halt (remote read), none
of which the saved state holds. The restore appeared to succeed, since
restore_all does not read KVM's LAPIC state back: Apic::can_compare is
false while the backend cannot freeze time.

A guest cannot set the entry's delivery mode, because KVM masks its
writes to the entry (apic_lvt_mask in kvm_lapic_reg_write), so a save
never holds such a page. Only a damaged or crafted saved state does.

set_apic now rejects such a page with KvmError::InvalidState as its
first step, before it forgets the timer that the VP stopped with or
changes the APIC base or the page, so a rejected restore leaves the
APIC as it was. It refuses every unmasked entry with a mode other than
Fixed, whatever the timer's mode, lowest priority and ExtINT included,
which KVM delivers as a fixed interrupt or ignores, since no save holds
them either. A masked entry is accepted: it delivers nothing, and a
guest write that unmasks it rewrites its delivery mode too. Every page
that OpenVMM saves restores as before, and so does the reset page,
whose timer is masked. The Guide's KVM timekeeping section describes
the check.

Tests: an_unmasked_timer_that_is_not_fixed_is_refused checks the
refusal for the delivery modes 1 to 7 of a one-shot, a periodic, and a
TSC-deadline timer, and that a masked entry, the Fixed mode, and the
reset value are accepted. The backend contract test
restore_refuses_a_non_fixed_apic_timer, which only KVM owes and which
registers with the `kvm_only` form of backend_test!, sets an expired
one-shot timer in each of those modes on a real KVM VP, and requires
that every restore is refused and leaves the APIC state unchanged, and
that an expired Fixed timer and masked timers in every mode restore.
The Guide's page of the backend contract tests lists the refusal among
the contracts that only KVM owes.

At feature/microvm/dev b723cbe, on nvx-kvm-test (Linux 6.18.57, nested
KVM on an Intel Xeon Platinum 8573C without APICv), the contract test
fails with "the restore accepted an expired timer in the delivery modes
[1, 2, 3, 4, 5, 6, 7]". A scratch test that gave KVM_SET_LAPIC the same
page and ran a real-mode BSP that spins with interrupts enabled showed
what KVM then does at KVM_RUN: INIT resets the VP (an emulation failure
at rip 0xfff0), NMI injects an NMI, SMI makes KVM_RUN fail and every
later ioctl on the VM return EIO, and lowest priority takes the vector
as a fixed interrupt; remote read, startup IPI, and ExtINT have no
visible effect on a running BSP. With this change both tests pass.
Mutants of the check fail them: refusing nothing, or refusing a masked
entry too, fails both; accepting lowest priority and ExtINT fails the
unit test; and a set_apic that ignores the refusal fails the contract
test. On nvx-kvm-test, cargo check, clippy --all-targets (gnu and
musl), doc, and nextest pass for virt_kvm, as do clippy and doc for
vmm_tests, all 84 tests of virt_kvm and kvm with those that need
/dev/kvm, and the two contract tests that KVM runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. labels Nov 14, 2026
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 14, 2026
Restoring a VP whose saved APIC page held an unmasked LVT timer entry
with a delivery mode other than Fixed succeeded on KVM, and KVM then
fired the timer in that mode once it had expired, as a one-shot timer
whose current count is 0 has: inside KVM_SET_LAPIC when APICv is
active, and at the next KVM_RUN otherwise. The restored VP took an
INIT, a startup IPI, an NMI, or an SMI, or woke from halt, none of
which the snapshot holds (#1365).

Pin 30b7554069182a5262099f902c4a203d54c8bb88, the head of
microsoft/openvmm#5102, one commit on b723cbed3, the head of
feature/microvm/dev:

- 30b755406 virt_kvm: reject a restored APIC timer whose delivery mode
  is not Fixed

set_apic now rejects such a page with KvmError::InvalidState as its
first step, before it forgets the timer that the VP stopped with or
changes the APIC base or the page, so the restore fails with the apic
element's StateError, and set_apic itself leaves the VP's APIC as it
was. The elements that restore_all restores before the APIC, the
registers among them, have run: KVM's write of CR8 can change the
LAPIC's task priority. It refuses every unmasked entry with a mode other
than Fixed, lowest priority and ExtINT included, and accepts a masked
entry. A guest cannot write the entry's delivery mode, since KVM masks
its writes, so a save never holds such a page, a snapshot that a VP
saved restores as before, and only a crafted snapshot is refused. MSHV
and WHP are unchanged. No version is created or bumped, and no snapshot
or wire format changes.

Tests: on nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57,
nested KVM without APICv), the revision's contract test
restore_refuses_a_non_fixed_apic_timer fails at b723cbed3 with "the
restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5,
6, 7]" and passes at the pin, as does its unit test
an_unmasked_timer_that_is_not_fixed_is_refused. A scratch test that gave
KVM_SET_LAPIC the same page showed what KVM does with it at KVM_RUN:
INIT resets the VP, NMI injects an NMI, SMI kills the VM, and lowest
priority is taken as a fixed interrupt. Mutants of the check fail the
tests. At the pin, the 84 tests of virt_kvm and kvm pass with cargo
nextest, the ignored ones included. With OpenVMM built from the pin's
tree, which replaced the packaged binary (SHA-256 193d1063 to 04915a93),
and the artifacts of v0.1.0-dev.9e41ebd10be8, the microVM scenarios
guest-boot (1 processor), interrupt-map, smp, smp-snapshot,
snapshot-core, snapshot-rollback, restore-downtime, restore-processors,
snapshot-tiers, time-abi-conformance, and time-host-correction pass at 1
and 2 processors, and smp-lapic at 1, 2, 4, and 8, on nvx-kvm-test. The
change is KVM-only, so nvx-mshv-test and nvx-whp-test were not run.

NVX needs no coordinated change: the control-session vectors, the source
excerpts, the contract versions, the activation bound, and the CPU
profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 14, 2026
The NVX pin moves to 30b755406, position 282, 158 commits past the
mapped tip. Position 282, the one commit of microsoft/openvmm#5102 on
b723cbed3, the head of feature/microvm/dev, is not yet assigned to a PR.

Position 282 makes the KVM backend's set_apic reject a LAPIC page whose
unmasked timer entry has a delivery mode other than Fixed, which KVM
would fire as an INIT, a startup IPI, an NMI, or an SMI, or as a wake of
a halted VP, none of which the saved state holds, in code that upstream
main has. It adds check_restorable_apic_timer, which set_apic calls
first, a unit test, a KVM-only contract test of virt_backend_tests in a
module of its own, and a paragraph and a sentence of the Guide. It
changes files that PR01's position 002, PR05's positions 007 and 008,
PR10's positions 022, 271, and 273, and positions 140, 248, 250, 251,
253, 279, and 281 also change, and adds to no line that a mapped PR
adds. Replaying it on upstream main, 7d90c4104, conflicts in
vp_state.rs, at the start of set_apic, whose timer reset position 253
adds, and from the end of the AccessVpState implementation on, which
positions 250, 251, 253, and 279 add; in the Guide's KVM timekeeping
section, which positions 250 and 253 extend; in the Guide's page of the
contract tests, whose kvm_only sentence is position 279's; and in the
contract tests' main.rs, beside the attribute that position 008 drops.
So it follows positions 008, 250, 251, 253, and 279 upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) merged commit 30b7554 into feature/microvm/dev Nov 14, 2026
73 of 74 checks passed
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant