Skip to content

[DO NOT REVIEW] vmm_core, virt_kvm: restore one-shot LAPIC timers without drift or a downtime refire - #5053

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
feature/microvm/devfrom
copilot/nvx-issue-1366-kvm-oneshot-lapic-restore
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
feature/microvm/devfrom
copilot/nvx-issue-1366-kvm-oneshot-lapic-restore

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Restores one-shot LAPIC timers across an NVX time ABI snapshot without a duplicated interrupt or drift, for microsoft/nvx#1366. On KVM, a timer that expired during the downtime reached a VP restored runnable with interrupts enabled twice, since KVM_SET_LAPIC fires an expired one-shot timer again after the restore queued its interrupt. And a running timer fired early, by milliseconds in the VM worker: the capture saves each VP's timer count after its anchor, while KVM's timers count with the VPs stopped, and the restore let KVM run the saved timer from the processor restore, before the restore anchor, and then subtracted the whole downtime from the count that it read back.

Three commits on 90f87319d, the head of feature/microvm/dev. microsoft/nvx#1505 pins this head.

Share the guest RAM of the real-mode KVM tests (2a0401c43)

  • Change. GuestRam, the guest RAM in memory slot 0 that the ExtINT tests of virt_kvm map for their real-mode guest, moves unchanged to a test-only module of the x86-64 backend, test_guest, which the LAPIC timer tests of the next commits share. No production code changes. The unsafe code that the bot flags in this pull request is all test-only: GuestRam, moved unchanged, and the read_u64 that the third commit adds to it.

Do not fire an expired LAPIC timer whose interrupt is pending (21a76f878)

  • Change. KvmVpStateAccess::set_apic passes kvm_lapic_page(registers) to KVM_SET_LAPIC, which gives KVM an initial count of 0 for an expired one-shot timer, whose current count is 0 and initial count is not, when its unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR. KVM starts no timer with an initial count of 0, and IRR then holds the one interrupt that the expiry owes, so this never loses one, whatever produced the page. The guest reads 0 from the initial count register. MSHV and WHP do not fire such a timer, and are unchanged. The Guide's timekeeping page says so.
  • Why only then. KVM_GET_LAPIC saves a timer that expired while its vCPU was outside KVM_RUN, whose expiry KVM has not injected yet (in lapic_timer.pending, or in a VMX preemption timer, which does not expire outside guest mode), with the same page as one whose interrupt the guest took. OpenVMM stops the VPs at a snapshot boundary before the quiesce and saves them after it, so a timer that the guest armed for its next tick often expires in between. An initial count of 0 for that page loses its interrupt, so such a timer keeps its counts and KVM fires it again; see "Not fixed here".
  • Tests. an_expired_timer_whose_interrupt_is_pending_is_not_started_again gives an initial count of 0 only to such a timer, and passes a counting timer, an expired one whose vector is not pending, and a masked, periodic, NMI-mode, or invalid-vector one through. The KVM-backed tests, ignored without /dev/kvm, run a real-mode guest that counts its timer interrupts through the sequence of a time ABI restore: in a_timer_that_expires_in_the_downtime_fires_once, a timer armed for 1 s and restored after a downtime of 2 s fires twice without the change and once with it, and a_timer_whose_expiry_kvm_had_not_delivered_fires_once checks that a timer that expired while its VP was stopped, which KVM saves with its vector neither pending nor in service, still fires once; an initial count of 0 for its page would give it no interrupt at all.

Run a restored LAPIC timer from the time ABI anchors (024ecee62)

  • Change. PartitionUnit::omit_saved_tsc becomes prepare_time_abi_restore: a time ABI restore drops each VP's saved TSC, as before, and takes the one-shot timer out of its saved APIC state with VpSavedState::take_one_shot_timer, which leaves the initial and current counts 0, so that the processor restore runs no timer on any backend. The partition unit keeps each timer with the VP's saved TSC, which a save reads after the APIC state, so that it dates the saved count. After the synchronized TSC set, advance_lapic_timers takes the restore's LapicRestoreClock, and Apic::restore_one_shot_timer sets each timer again from the count that it had at the capture anchor, the saved count plus the ticks of the saved TSC past T_c at the declared rate and at most the initial count, less the ticks of the downtime D by the time ABI's rule, and less the ticks of the VP's TSC past the set's target at the native rate. The timer so runs as the TSC does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set, from which 90f87319d also runs VM time. A count that reaches 0 queues the interrupt unless the LVT is masked, and on KVM the previous commit then keeps KVM from firing the timer again. A TSC behind its reference, as a VP's can be by the skew between VPs, counts no ticks. A timer that expired before the capture keeps its counts, and a VP whose saved state holds no APIC state advances by the downtime, as before. On MSHV and WHP, which freeze partition time from the quiesce until the set, the saved TSC is the anchor's, so their timers advance as before. The snapshot guide says so.
  • No version. No saved state, contract, format, or version changes; a snapshot taken before this change gets the same correction.
  • Tests. time_abi_restore_counts_the_timer_from_the_anchors checks that a timer saved at the capture anchor and set at the restore anchor advances by the downtime alone, and loses the ticks of both TSC intervals, at the declared and the native rate, with any divide configuration; that a TSC behind its reference counts no ticks, and one that wrapped past 2^64 counts the cycles since it; and that the count at the capture anchor is at most the initial count. time_abi_restore_expires_the_timer_once checks that the downtime, or the time past the restore anchor, expires a timer once and queues its interrupt unless the LVT is masked, and that an expired or idle timer keeps its counts. time_abi_restore_rejects_periodic_and_deadline_timers, time_abi_restore_takes_the_timer_out_of_the_restored_state, take_one_shot_timer_dates_the_count_with_the_saved_tsc, and lapic_ticks_in_tsc_cycles cover the rest. The KVM-backed a_restored_timer_fires_on_the_guest_tsc arms a 100 ms timer, captures it 20 ms later with 20 ms from the capture anchor to the save, and restores it after a downtime of 50 ms with 20 ms from the processor restore to the restore anchor: it must fire 100 ms of guest TSC time after it was armed, within -100 µs and +20 ms.
  • What remains. The timer is late by the time that a save takes from the APIC state to the TSC, about 13 µs on the KVM test VM, and by the time from the advance's TSC read to its LAPIC write. The capture's TSC counts at the declared rate only within the rate tolerance, which can make the timer early by at most 250 ppm of the time from the capture anchor to the save.

Not fixed here

A one-shot timer that expired before the capture, and whose interrupt the guest took, still fires once more on KVM. KVM saves it with the same LAPIC state as a timer whose expiry it had not injected yet, the common state at the capture, and KVM_SET_LAPIC fires such a timer again: at once from Linux 6.3, a full period later before. Telling the two apart needs more of KVM, microsoft/nvx#1390, whose fix builds on this pull request.

Validation

At this head, 024ecee62, with Rust 1.97.1, for virt, vmm_core, openvmm_core, and virt_kvm:

  • Windows: cargo clippy --all-targets and cargo doc --no-deps, without warnings from the packages; cargo nextest run --profile agent, 282 passed and 1 skipped; cargo xtask fmt --fix, clean.
  • Linux, on the Azure Linux 3 KVM test VM (Linux 6.6, nested, without APICv or the VMX preemption timer): cargo clippy --all-targets and cargo doc --no-deps, without warnings; cargo nextest run --profile agent, 324 passed and 16 skipped; the 9 KVM-backed tests of virt_kvm passed.
  • Measured with the KVM-backed tests there, under a load average of 9: the timer that expires in the downtime fires twice without 21a76f878's rule and once with it. The 100 ms timer fires 29.3 to 29.5 ms early in the earlier restore sequence, and 161 to 333 µs late with this head in 15 runs, against 176 to 234 µs in 13 of 15 runs of the same timer without a snapshot.
  • End to end, through NVX's microVM scenarios, with OpenVMM built from this head: smp-lapic (1, 2, 4, and 8 VPs), smp-snapshot, restore-processors, snapshot-core, snapshot-rollback, and time-abi-conformance passed on the KVM and MSHV test VMs. In the MSHV restores, the TSCs restarted 45 to 134 µs after the restore anchor, and the LAPIC advance ran after that. With the same code on fb4f34126, before the rebase, those scenarios, restore-downtime, and snapshot-tiers also passed on the KVM, MSHV, and WHP test VMs.

See microsoft/nvx#1505.

The extint tests run a real-mode guest in a KVM VM, whose RAM their
GuestRam maps into memory slot 0. The LAPIC timer tests that follow run
such a guest too, so GuestRam moves, unchanged, to a test-only module of
the x86-64 backend that both use.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KVM_SET_LAPIC starts a counting LAPIC timer from the page's current
count, so it fires a one-shot timer that has expired, whose current
count is 0 and initial count is not, once more: at once from Linux 6.3,
a full period later before. A time ABI restore that expires a one-shot
timer over its downtime sets its count to 0 and queues its vector in
IRR, so a VP restored runnable with interrupts enabled took the
interrupt twice (microsoft/nvx#1366), since KVM's re-fire reaches IRR
only after the VP accepts the queued vector. A timer whose interrupt a
saved state holds in IRR, delivered but not yet accepted, fired twice
the same way after any restore.

KvmVpStateAccess::set_apic now gives KVM an initial count of 0 for an
expired, unmasked, Fixed-mode one-shot timer whose valid vector is
pending in IRR, which KVM does not start. IRR then holds the one
interrupt that the expiry owes, so this never loses one; the guest reads
0 from the initial count register. MSHV and WHP do not fire such a
timer, and are unchanged.

An expired timer whose vector is not pending keeps its initial count,
so KVM fires it again. KVM_GET_LAPIC saves a timer that expired while
its vCPU was outside KVM_RUN, whose expiry KVM has not injected yet (in
lapic_timer.pending, or in a VMX preemption timer, which does not
expire outside guest mode), with the same page as one whose interrupt
the guest took. OpenVMM stops the VPs at a snapshot boundary and saves
them only after the quiesce, so a timer that the guest armed for its
next tick usually expires in between: firing it again delivers its
interrupt, which an initial count of 0 would lose. A timer whose
interrupt the guest took before the capture gets a spurious interrupt,
a limit of KVM's interface (microsoft/nvx#1390).

Tests: kvm_lapic_page gives an initial count of 0 only to such a timer,
and passes a counting timer, an expired one whose vector is not
pending, and a masked, periodic, NMI-mode, or invalid-vector one
through. With KVM, on a real-mode guest that counts its timer
interrupts, in the sequence of a time ABI restore: a timer armed for
1 s and restored after a downtime of 2 s fires once, and a timer that
expired while its VP was stopped, which KVM saves expired with its
vector neither pending nor in service, fires once after the restore.
On Linux 6.6 (nvx-kvm-test), without the fix the first fired twice, and
the second fires not at all when KVM gets an initial count of 0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A time ABI restore advances each counting one-shot LAPIC timer by the
downtime D, the host time from the capture anchor to the restore
anchor, so that the timer keeps step with the guest TSC. But a capture
saves each VP's APIC state after its anchor, and on KVM, whose timers
count while the VPs are stopped, the saved count had already lost the
ticks from the anchor to the save. The restore then set the saved state
in the processor restore, where KVM started the timer before the
restore anchor, and the LAPIC advance subtracted D from the count that
it read back. So a restored timer fired early by both windows
(microsoft/nvx#1366), milliseconds in the VM worker. MSHV and WHP freeze
partition time from the quiesce and until the synchronized TSC set, so
neither window ran their timers.

A time ABI restore now takes each VP's LAPIC timer out of its saved
APIC state, so that the processor restore leaves it stopped, and keeps
it with the VP's saved TSC, which a save reads after the APIC state.
After the synchronized TSC set, the LAPIC advance sets the timer again
from the count that it had at the capture anchor, the saved count plus
the ticks of the saved TSC past T_c at the declared rate, less the
ticks of D, by the time ABI's rule, and less the ticks of the VP's TSC
past the set's target at the destination's native rate, so that the
timer runs as the TSC does: from the restore anchor on KVM, and on
MSHV and WHP from the resumption of partition time that ends the set.
A TSC behind its reference, as a VP's can be by the skew between VPs,
counts no ticks. On MSHV and WHP the saved TSC is the anchor's, so
their timers advance as before. A timer that expired before the capture
keeps its counts. The saved state is unchanged, so a snapshot taken
before this change gets the same correction.
PartitionUnit::omit_saved_tsc becomes prepare_time_abi_restore, and
advance_lapic_timers takes the restore's anchors and rates with its
downtime.

What remains makes the timer late by the time that a save takes from
the APIC state to the TSC, about 13 us on nvx-kvm-test, and by the time
from the advance's TSC read to its LAPIC write. The capture's TSC,
which counts at the declared rate only within the rate tolerance, can
make it early by at most 250 ppm of the time from the capture anchor to
the save.

Tests: the restored count is the advanced count for a timer saved at
the capture anchor and set at the restore anchor, and loses the ticks
of both TSC intervals, at the declared and the native rate, with any
divide configuration; a TSC behind its reference counts no ticks, and
one that wrapped past 2^64 counts the cycles since it; the count at the
capture anchor is at most the initial count; the downtime, or the time
past the restore anchor, expires a timer once, and queues its interrupt
unless the LVT is masked; an expired or idle timer keeps its counts;
and the checks reject a periodic or TSC-deadline timer by its saved
counts. VpSavedState::take_one_shot_timer takes the timer with the
saved TSC. With KVM, a real-mode guest's timer armed for 100 ms,
captured 20 ms later with 20 ms from the capture anchor to the save,
and restored after a downtime of 50 ms with 20 ms from the processor
restore to the restore anchor fires 100 ms of guest TSC time after it
was armed, within -100 us and +20 ms. On Linux 6.6 (nvx-kvm-test), under
a load average of 20, it fired 133 to 154 us late in 9 of 10 runs and
498 us late in one, against 102 to 125 us for a timer that ran without
a snapshot, and 22.9 to 29.6 ms early with the old sequence, which
expired it at the restore.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@ppenna
Pedro Henrique Penna (ppenna) merged commit 024ecee into feature/microvm/dev Nov 12, 2026
71 of 74 checks passed
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 12, 2026
The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at
position 261, 130 commits past the mapped tip. Positions 259-261,
727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62,
the head of feature/microvm/dev, which microsoft/openvmm#5053 merged.

They have the KVM backend tell a one-shot LAPIC timer whose interrupt
the guest took apart from one whose expiry KVM holds when it saves a VP,
and change nothing that a mapped commit adds, so they stay unassigned.
Position 259 adds a kvm crate call that has KVM deliver the interrupts
that it holds without entering the guest; it applies to upstream main,
ba105dd7f, without conflicts, and the kvm crate type-checks there, so it
can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer
module and rewrites the Guide paragraph of position 257; replaying it on
upstream main conflicts in that paragraph, the x86-64 module list, and
the VP state module, and its KVM tests use positions 256, 257, and 259,
so it follows them upstream. Position 261 builds position 260's KVM
tests for musl, and changes nothing else, so it folds into position 260.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 12, 2026
On KVM, a time ABI restore could give a guest one extra LAPIC timer
interrupt, and fired a running one-shot timer early (microsoft#1366).
KVM_SET_LAPIC fires a one-shot timer whose current count is 0 again, so
a timer that expired during the downtime, whose vector the restore
queues in IRR, reached a VP restored runnable with interrupts enabled
twice. And the capture saved each VP's timer count after its anchor,
while the restore let KVM run the saved timer from the processor
restore, before the restore anchor, and then subtracted the whole
downtime, so a restored timer fired early by both windows, milliseconds
in the VM worker. MSHV and WHP freeze partition time over both windows.

Pin 024ecee62d7c9b70456b63a5b4f94a6488f835cd, the head of
microsoft/openvmm#5053, three commits on 90f87319d, the head of
feature/microvm/dev:

- 2a0401c43 virt_kvm: share the guest RAM of the real-mode KVM tests
- 21a76f878 virt_kvm: do not fire an expired LAPIC timer whose interrupt
  is pending
- 024ecee62 vmm_core, virt: run a restored LAPIC timer from the time ABI
  anchors

The KVM backend now gives KVM an initial count of 0 for an expired
one-shot timer whose unmasked LVT entry delivers a fixed interrupt to a
valid vector that is pending in IRR, so that it fires once. A
time ABI restore takes each VP's timer out of its saved APIC state and
sets it only after the synchronized TSC set, from its count at the
capture anchor, which the saved TSC dates, advanced by the downtime and
by the ticks of the TSC past the set's target, so that it runs as the
TSC does, as the previous pin's VM time does: from the restore anchor on
KVM, and on MSHV and WHP from the resumption of partition time that ends
the set. No saved-state, manifest, or time ABI format changes; a
snapshot taken before this pin restores with the same correction.

A one-shot timer that expired before the capture, and whose interrupt
the guest took, still fires once more on KVM. KVM saves it with the same
LAPIC state as a timer that expired while its vCPU was stopped and whose
interrupt KVM has not injected yet, which is the common state at the
capture, since the snapshot boundary stops the VPs before the quiesce.
Giving KVM an initial count of 0 for such a page loses that interrupt,
so the restore keeps KVM's re-fire, and only a KVM change that tells the
two apart can remove the spurious one (microsoft#1390).

Evidence: KVM-backed tests run a real-mode guest that counts its timer
interrupts through the restore sequence, on nvx-kvm-test (Linux 6.6,
nested, without APICv or the VMX preemption timer). A timer armed for
1 s and restored after a 2 s downtime with interrupts enabled fires
twice without the KVM backend's initial count of 0, and once at the
pin. A timer that expired while its VP was stopped is saved expired,
with its vector neither pending nor in service, the same page as a
timer delivered before the capture, and fires once at the pin, while an
initial count of 0 for that page gives it no interrupt at all. A 100 ms
timer, with 20 ms from the capture anchor to the save and 20 ms from
the processor restore to the restore anchor, fires 29.3 to 29.5 ms
early with the earlier restore sequence, and 161 to 333 us late at the
pin in 15 runs under a load average of 9, against 176 to 234 us in 13
of 15 runs of the same timer without a snapshot, and 429 us and 3.3 ms
in the others. On MSHV, whose frozen write restarts the TSCs 45 to
134 us after the restore anchor in the scenarios' restores, the LAPIC
advance runs after that restart and counts each timer from its VP's
TSC, so the timers run from where the previous pin runs VM time. At the
pin, the smp-lapic, smp-snapshot, restore-processors, snapshot-core,
snapshot-rollback, and time-abi-conformance microVM scenarios pass on
KVM (nvx-kvm-test) and MSHV (nvx-mshv-test), with OpenVMM built from
the pin. The restore-downtime scenario, which now simulates a year,
needs guest artifacts with the 365-day bound, and is left to CI with
WHP.

NVX needs no other coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 12, 2026
Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at
position 258, which with positions 256 and 257, 2a0401c43 and
21a76f878, follows position 255, 90f87319d, the head of
feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is
127 commits past the mapped tip.

They change how a restore runs a one-shot LAPIC timer and change
nothing that a mapped commit adds, so they stay unassigned. Position 256
moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which
upstream main has, into a test module that position 257's tests share.
Position 257 changes the KVM backend's set_apic, which upstream main
has, and the Guide's timekeeping reference, which position 255 also
changes. Position 258 changes the time ABI restore of the LAPIC timers
that position 139 adds, which positions 146, 147, 155, 158, and 255
also change, and sets each timer again after the synchronized TSC set,
so that it runs as the TSCs do, as position 255's VM time does.

Replaying position 256 on upstream main, ba105dd7f, conflicts only in
virt_kvm's x86-64 module list, beside the modules of positions 133 and
138. Position 257 applies there without conflicts, but its KVM tests
use position 256's guest RAM, position 139's timer advance, the TSC
helpers of positions 132 and 160, and the kvm crate's TSC accessors of
PR10's position 026, so it follows them upstream. Position 258 conflicts
in all 12 of its files, seven of which upstream main lacks, so it
follows the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant