Repository navigation
[DO NOT REVIEW] vmm_core, virt_kvm: restore one-shot LAPIC timers without drift or a downtime refire - #5053
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 12, 2026
Conversation
The extint tests run a real-mode guest in a KVM VM, whose RAM their GuestRam maps into memory slot 0. The LAPIC timer tests that follow run such a guest too, so GuestRam moves, unchanged, to a test-only module of the x86-64 backend that both use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KVM_SET_LAPIC starts a counting LAPIC timer from the page's current count, so it fires a one-shot timer that has expired, whose current count is 0 and initial count is not, once more: at once from Linux 6.3, a full period later before. A time ABI restore that expires a one-shot timer over its downtime sets its count to 0 and queues its vector in IRR, so a VP restored runnable with interrupts enabled took the interrupt twice (microsoft/nvx#1366), since KVM's re-fire reaches IRR only after the VP accepts the queued vector. A timer whose interrupt a saved state holds in IRR, delivered but not yet accepted, fired twice the same way after any restore. KvmVpStateAccess::set_apic now gives KVM an initial count of 0 for an expired, unmasked, Fixed-mode one-shot timer whose valid vector is pending in IRR, which KVM does not start. IRR then holds the one interrupt that the expiry owes, so this never loses one; the guest reads 0 from the initial count register. MSHV and WHP do not fire such a timer, and are unchanged. An expired timer whose vector is not pending keeps its initial count, so KVM fires it again. KVM_GET_LAPIC saves a timer that expired while its vCPU was outside KVM_RUN, whose expiry KVM has not injected yet (in lapic_timer.pending, or in a VMX preemption timer, which does not expire outside guest mode), with the same page as one whose interrupt the guest took. OpenVMM stops the VPs at a snapshot boundary and saves them only after the quiesce, so a timer that the guest armed for its next tick usually expires in between: firing it again delivers its interrupt, which an initial count of 0 would lose. A timer whose interrupt the guest took before the capture gets a spurious interrupt, a limit of KVM's interface (microsoft/nvx#1390). Tests: kvm_lapic_page gives an initial count of 0 only to such a timer, and passes a counting timer, an expired one whose vector is not pending, and a masked, periodic, NMI-mode, or invalid-vector one through. With KVM, on a real-mode guest that counts its timer interrupts, in the sequence of a time ABI restore: a timer armed for 1 s and restored after a downtime of 2 s fires once, and a timer that expired while its VP was stopped, which KVM saves expired with its vector neither pending nor in service, fires once after the restore. On Linux 6.6 (nvx-kvm-test), without the fix the first fired twice, and the second fires not at all when KVM gets an initial count of 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A time ABI restore advances each counting one-shot LAPIC timer by the downtime D, the host time from the capture anchor to the restore anchor, so that the timer keeps step with the guest TSC. But a capture saves each VP's APIC state after its anchor, and on KVM, whose timers count while the VPs are stopped, the saved count had already lost the ticks from the anchor to the save. The restore then set the saved state in the processor restore, where KVM started the timer before the restore anchor, and the LAPIC advance subtracted D from the count that it read back. So a restored timer fired early by both windows (microsoft/nvx#1366), milliseconds in the VM worker. MSHV and WHP freeze partition time from the quiesce and until the synchronized TSC set, so neither window ran their timers. A time ABI restore now takes each VP's LAPIC timer out of its saved APIC state, so that the processor restore leaves it stopped, and keeps it with the VP's saved TSC, which a save reads after the APIC state. After the synchronized TSC set, the LAPIC advance sets the timer again from the count that it had at the capture anchor, the saved count plus the ticks of the saved TSC past T_c at the declared rate, less the ticks of D, by the time ABI's rule, and less the ticks of the VP's TSC past the set's target at the destination's native rate, so that the timer runs as the TSC does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set. A TSC behind its reference, as a VP's can be by the skew between VPs, counts no ticks. On MSHV and WHP the saved TSC is the anchor's, so their timers advance as before. A timer that expired before the capture keeps its counts. The saved state is unchanged, so a snapshot taken before this change gets the same correction. PartitionUnit::omit_saved_tsc becomes prepare_time_abi_restore, and advance_lapic_timers takes the restore's anchors and rates with its downtime. What remains makes the timer late by the time that a save takes from the APIC state to the TSC, about 13 us on nvx-kvm-test, and by the time from the advance's TSC read to its LAPIC write. The capture's TSC, which counts at the declared rate only within the rate tolerance, can make it early by at most 250 ppm of the time from the capture anchor to the save. Tests: the restored count is the advanced count for a timer saved at the capture anchor and set at the restore anchor, and loses the ticks of both TSC intervals, at the declared and the native rate, with any divide configuration; a TSC behind its reference counts no ticks, and one that wrapped past 2^64 counts the cycles since it; the count at the capture anchor is at most the initial count; the downtime, or the time past the restore anchor, expires a timer once, and queues its interrupt unless the LVT is masked; an expired or idle timer keeps its counts; and the checks reject a periodic or TSC-deadline timer by its saved counts. VpSavedState::take_one_shot_timer takes the timer with the saved TSC. With KVM, a real-mode guest's timer armed for 100 ms, captured 20 ms later with 20 ms from the capture anchor to the save, and restored after a downtime of 50 ms with 20 ms from the processor restore to the restore anchor fires 100 ms of guest TSC time after it was armed, within -100 us and +20 ms. On Linux 6.6 (nvx-kvm-test), under a load average of 20, it fired 133 to 154 us late in 9 of 10 runs and 498 us late in one, against 102 to 125 us for a timer that ran without a snapshot, and 22.9 to 29.6 ms early with the old sequence, which expired it at the restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
requested a review
from a team
as a code owner
November 12, 2026 00:32
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
Pedro Henrique Penna (ppenna)
merged commit Nov 12, 2026
024ecee
into
feature/microvm/dev
71 of 74 checks passed
This was referenced Nov 12, 2026
Merged
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 12, 2026
The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at position 261, 130 commits past the mapped tip. Positions 259-261, 727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62, the head of feature/microvm/dev, which microsoft/openvmm#5053 merged. They have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped commit adds, so they stay unassigned. Position 259 adds a kvm crate call that has KVM deliver the interrupts that it holds without entering the guest; it applies to upstream main, ba105dd7f, without conflicts, and the kvm crate type-checks there, so it can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer module and rewrites the Guide paragraph of position 257; replaying it on upstream main conflicts in that paragraph, the x86-64 module list, and the VP state module, and its KVM tests use positions 256, 257, and 259, so it follows them upstream. Position 261 builds position 260's KVM tests for musl, and changes nothing else, so it folds into position 260. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali
pushed a commit
to syeedali/nvx
that referenced
this pull request
Nov 12, 2026
On KVM, a time ABI restore could give a guest one extra LAPIC timer interrupt, and fired a running one-shot timer early (microsoft#1366). KVM_SET_LAPIC fires a one-shot timer whose current count is 0 again, so a timer that expired during the downtime, whose vector the restore queues in IRR, reached a VP restored runnable with interrupts enabled twice. And the capture saved each VP's timer count after its anchor, while the restore let KVM run the saved timer from the processor restore, before the restore anchor, and then subtracted the whole downtime, so a restored timer fired early by both windows, milliseconds in the VM worker. MSHV and WHP freeze partition time over both windows. Pin 024ecee62d7c9b70456b63a5b4f94a6488f835cd, the head of microsoft/openvmm#5053, three commits on 90f87319d, the head of feature/microvm/dev: - 2a0401c43 virt_kvm: share the guest RAM of the real-mode KVM tests - 21a76f878 virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending - 024ecee62 vmm_core, virt: run a restored LAPIC timer from the time ABI anchors The KVM backend now gives KVM an initial count of 0 for an expired one-shot timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR, so that it fires once. A time ABI restore takes each VP's timer out of its saved APIC state and sets it only after the synchronized TSC set, from its count at the capture anchor, which the saved TSC dates, advanced by the downtime and by the ticks of the TSC past the set's target, so that it runs as the TSC does, as the previous pin's VM time does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores with the same correction. A one-shot timer that expired before the capture, and whose interrupt the guest took, still fires once more on KVM. KVM saves it with the same LAPIC state as a timer that expired while its vCPU was stopped and whose interrupt KVM has not injected yet, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce. Giving KVM an initial count of 0 for such a page loses that interrupt, so the restore keeps KVM's re-fire, and only a KVM change that tells the two apart can remove the spurious one (microsoft#1390). Evidence: KVM-backed tests run a real-mode guest that counts its timer interrupts through the restore sequence, on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer). A timer armed for 1 s and restored after a 2 s downtime with interrupts enabled fires twice without the KVM backend's initial count of 0, and once at the pin. A timer that expired while its VP was stopped is saved expired, with its vector neither pending nor in service, the same page as a timer delivered before the capture, and fires once at the pin, while an initial count of 0 for that page gives it no interrupt at all. A 100 ms timer, with 20 ms from the capture anchor to the save and 20 ms from the processor restore to the restore anchor, fires 29.3 to 29.5 ms early with the earlier restore sequence, and 161 to 333 us late at the pin in 15 runs under a load average of 9, against 176 to 234 us in 13 of 15 runs of the same timer without a snapshot, and 429 us and 3.3 ms in the others. On MSHV, whose frozen write restarts the TSCs 45 to 134 us after the restore anchor in the scenarios' restores, the LAPIC advance runs after that restart and counts each timer from its VP's TSC, so the timers run from where the previous pin runs VM time. At the pin, the smp-lapic, smp-snapshot, restore-processors, snapshot-core, snapshot-rollback, and time-abi-conformance microVM scenarios pass on KVM (nvx-kvm-test) and MSHV (nvx-mshv-test), with OpenVMM built from the pin. The restore-downtime scenario, which now simulates a year, needs guest artifacts with the 365-day bound, and is left to CI with WHP. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali
pushed a commit
to syeedali/nvx
that referenced
this pull request
Nov 12, 2026
Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at position 258, which with positions 256 and 257, 2a0401c43 and 21a76f878, follows position 255, 90f87319d, the head of feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is 127 commits past the mapped tip. They change how a restore runs a one-shot LAPIC timer and change nothing that a mapped commit adds, so they stay unassigned. Position 256 moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which upstream main has, into a test module that position 257's tests share. Position 257 changes the KVM backend's set_apic, which upstream main has, and the Guide's timekeeping reference, which position 255 also changes. Position 258 changes the time ABI restore of the LAPIC timers that position 139 adds, which positions 146, 147, 155, 158, and 255 also change, and sets each timer again after the synchronized TSC set, so that it runs as the TSCs do, as position 255's VM time does. Replaying position 256 on upstream main, ba105dd7f, conflicts only in virt_kvm's x86-64 module list, beside the modules of positions 133 and 138. Position 257 applies there without conflicts, but its KVM tests use position 256's guest RAM, position 139's timer advance, the TSC helpers of positions 132 and 160, and the kvm crate's TSC accessors of PR10's position 026, so it follows them upstream. Position 258 conflicts in all 12 of its files, seven of which upstream main lacks, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores one-shot LAPIC timers across an NVX time ABI snapshot without a duplicated interrupt or drift, for microsoft/nvx#1366. On KVM, a timer that expired during the downtime reached a VP restored runnable with interrupts enabled twice, since
KVM_SET_LAPICfires an expired one-shot timer again after the restore queued its interrupt. And a running timer fired early, by milliseconds in the VM worker: the capture saves each VP's timer count after its anchor, while KVM's timers count with the VPs stopped, and the restore let KVM run the saved timer from the processor restore, before the restore anchor, and then subtracted the whole downtime from the count that it read back.Three commits on
90f87319d, the head offeature/microvm/dev. microsoft/nvx#1505 pins this head.Share the guest RAM of the real-mode KVM tests (
2a0401c43)GuestRam, the guest RAM in memory slot 0 that the ExtINT tests ofvirt_kvmmap for their real-mode guest, moves unchanged to a test-only module of the x86-64 backend,test_guest, which the LAPIC timer tests of the next commits share. No production code changes. Theunsafecode that the bot flags in this pull request is all test-only:GuestRam, moved unchanged, and theread_u64that the third commit adds to it.Do not fire an expired LAPIC timer whose interrupt is pending (
21a76f878)KvmVpStateAccess::set_apicpasseskvm_lapic_page(registers)toKVM_SET_LAPIC, which gives KVM an initial count of 0 for an expired one-shot timer, whose current count is 0 and initial count is not, when its unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR. KVM starts no timer with an initial count of 0, and IRR then holds the one interrupt that the expiry owes, so this never loses one, whatever produced the page. The guest reads 0 from the initial count register. MSHV and WHP do not fire such a timer, and are unchanged. The Guide's timekeeping page says so.KVM_GET_LAPICsaves a timer that expired while its vCPU was outsideKVM_RUN, whose expiry KVM has not injected yet (inlapic_timer.pending, or in a VMX preemption timer, which does not expire outside guest mode), with the same page as one whose interrupt the guest took. OpenVMM stops the VPs at a snapshot boundary before the quiesce and saves them after it, so a timer that the guest armed for its next tick often expires in between. An initial count of 0 for that page loses its interrupt, so such a timer keeps its counts and KVM fires it again; see "Not fixed here".an_expired_timer_whose_interrupt_is_pending_is_not_started_againgives an initial count of 0 only to such a timer, and passes a counting timer, an expired one whose vector is not pending, and a masked, periodic, NMI-mode, or invalid-vector one through. The KVM-backed tests, ignored without/dev/kvm, run a real-mode guest that counts its timer interrupts through the sequence of a time ABI restore: ina_timer_that_expires_in_the_downtime_fires_once, a timer armed for 1 s and restored after a downtime of 2 s fires twice without the change and once with it, anda_timer_whose_expiry_kvm_had_not_delivered_fires_oncechecks that a timer that expired while its VP was stopped, which KVM saves with its vector neither pending nor in service, still fires once; an initial count of 0 for its page would give it no interrupt at all.Run a restored LAPIC timer from the time ABI anchors (
024ecee62)PartitionUnit::omit_saved_tscbecomesprepare_time_abi_restore: a time ABI restore drops each VP's saved TSC, as before, and takes the one-shot timer out of its saved APIC state withVpSavedState::take_one_shot_timer, which leaves the initial and current counts 0, so that the processor restore runs no timer on any backend. The partition unit keeps each timer with the VP's saved TSC, which a save reads after the APIC state, so that it dates the saved count. After the synchronized TSC set,advance_lapic_timerstakes the restore'sLapicRestoreClock, andApic::restore_one_shot_timersets each timer again from the count that it had at the capture anchor, the saved count plus the ticks of the saved TSC pastT_cat the declared rate and at most the initial count, less the ticks of the downtimeDby the time ABI's rule, and less the ticks of the VP's TSC past the set's target at the native rate. The timer so runs as the TSC does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set, from which90f87319dalso runs VM time. A count that reaches 0 queues the interrupt unless the LVT is masked, and on KVM the previous commit then keeps KVM from firing the timer again. A TSC behind its reference, as a VP's can be by the skew between VPs, counts no ticks. A timer that expired before the capture keeps its counts, and a VP whose saved state holds no APIC state advances by the downtime, as before. On MSHV and WHP, which freeze partition time from the quiesce until the set, the saved TSC is the anchor's, so their timers advance as before. The snapshot guide says so.time_abi_restore_counts_the_timer_from_the_anchorschecks that a timer saved at the capture anchor and set at the restore anchor advances by the downtime alone, and loses the ticks of both TSC intervals, at the declared and the native rate, with any divide configuration; that a TSC behind its reference counts no ticks, and one that wrapped past 2^64 counts the cycles since it; and that the count at the capture anchor is at most the initial count.time_abi_restore_expires_the_timer_oncechecks that the downtime, or the time past the restore anchor, expires a timer once and queues its interrupt unless the LVT is masked, and that an expired or idle timer keeps its counts.time_abi_restore_rejects_periodic_and_deadline_timers,time_abi_restore_takes_the_timer_out_of_the_restored_state,take_one_shot_timer_dates_the_count_with_the_saved_tsc, andlapic_ticks_in_tsc_cyclescover the rest. The KVM-backeda_restored_timer_fires_on_the_guest_tscarms a 100 ms timer, captures it 20 ms later with 20 ms from the capture anchor to the save, and restores it after a downtime of 50 ms with 20 ms from the processor restore to the restore anchor: it must fire 100 ms of guest TSC time after it was armed, within -100 µs and +20 ms.Not fixed here
A one-shot timer that expired before the capture, and whose interrupt the guest took, still fires once more on KVM. KVM saves it with the same LAPIC state as a timer whose expiry it had not injected yet, the common state at the capture, and
KVM_SET_LAPICfires such a timer again: at once from Linux 6.3, a full period later before. Telling the two apart needs more of KVM, microsoft/nvx#1390, whose fix builds on this pull request.Validation
At this head,
024ecee62, with Rust 1.97.1, forvirt,vmm_core,openvmm_core, andvirt_kvm:cargo clippy --all-targetsandcargo doc --no-deps, without warnings from the packages;cargo nextest run --profile agent, 282 passed and 1 skipped;cargo xtask fmt --fix, clean.cargo clippy --all-targetsandcargo doc --no-deps, without warnings;cargo nextest run --profile agent, 324 passed and 16 skipped; the 9 KVM-backed tests ofvirt_kvmpassed.21a76f878's rule and once with it. The 100 ms timer fires 29.3 to 29.5 ms early in the earlier restore sequence, and 161 to 333 µs late with this head in 15 runs, against 176 to 234 µs in 13 of 15 runs of the same timer without a snapshot.smp-lapic(1, 2, 4, and 8 VPs),smp-snapshot,restore-processors,snapshot-core,snapshot-rollback, andtime-abi-conformancepassed on the KVM and MSHV test VMs. In the MSHV restores, the TSCs restarted 45 to 134 µs after the restore anchor, and the LAPIC advance ran after that. With the same code onfb4f34126, before the rebase, those scenarios,restore-downtime, andsnapshot-tiersalso passed on the KVM, MSHV, and WHP test VMs.See microsoft/nvx#1505.