Repository navigation
[DO NOT REVIEW] kvm, virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again - #5055
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 12, 2026
Conversation
KVM records a LAPIC timer that expires while its vCPU is outside KVM_RUN in lapic_timer.pending, and sets the timer's vector in IRR only in the run loop of the next KVM_RUN, so until then KVM_GET_LAPIC reports the timer as expired, without its interrupt. VpRunner::complete_exit cannot deliver it: its immediate_exit makes KVM_RUN return before that loop. VpRunner::deliver_pending_interrupts completes the current exit, if any, and runs that loop once without entering the guest: it blocks the thread's cancellation signal, queues one for the thread, and unblocks it in KVM_RUN's signal mask (KVM_SET_SIGNAL_MASK), so KVM handles the vCPU's pending requests, injects its pending timer interrupts into IRR, and returns EINTR instead of entering the guest. A pending request that is an exit, such as the EOI of a userspace I/O APIC, is returned for the caller to handle, as from VpRunner::run, after which the caller calls it again. The call clears request_interrupt_window for the run, which KVM would otherwise report as an open interrupt window, then puts back a null signal mask, takes the one signal that it queued, and restores the thread's mask, so a cancellation that arrived meanwhile reaches the handler as before. KVM records no expiry of the VMX preemption timer, which it can use for the LAPIC timer of a running vCPU, while the vCPU is outside guest mode, so this delivers none. VpRunner::run and deliver_pending_interrupts share the mapping of KVM_RUN's result (run_result) and the decoding of its exit (exit), which are unchanged. Tests: the virt_kvm LAPIC timer tests in the next change check, on KVM, that a delivery runs no guest instruction, leaves the thread's signal mask unchanged, and sets the vector of a held timer expiry in IRR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KVM_GET_LAPIC reports a one-shot LAPIC timer that expired, and whose interrupt the guest took, with the same page as one whose expiry KVM holds but has not injected: an initial count, a current count of 0, and the vector clear in IRR. KVM_SET_LAPIC fires such a timer again, at once from Linux 6.3 and a full period later from 5.8 to 6.2, which the second needs, so a restore of the first gave the guest a timer interrupt that it never programmed (microsoft/nvx#1390). OpenVMM stops a VP with immediate_exit, which leaves KVM holding the expiry of a timer that expires afterwards, and saves the VP only after the quiesce and the capture anchor, so both pages are common, and an initial count of 0 for every such page would lose the second's interrupt. The backend now tells them apart when the VP stops. run_vp reads the LAPIC timer's current count (lapic_timer::TimerAtStop), and then has KVM deliver the interrupts that it holds, with VpRunner::deliver_pending_interrupts, handling the exits that the delivery reports as for a run, so that KVM then holds only an expiry after the read. KvmVpStateAccess::apic then reports an expired, unmasked, Fixed-mode one-shot timer whose valid vector is clear in IRR: - with its vector pending in IRR, and clear in TMR, if the timer was counting at the read: it expired after the stop, and KVM holds its interrupt. set_apic restores it with an initial count of 0, which KVM does not start (microsoft/nvx#1366), so the guest takes it once; - with an initial count of 0, if it had expired at the read: the guest took its interrupt. KVM_SET_LAPIC does not start it, and the guest reads 0 from the initial count register, which Linux never reads. KVM records an expiry when its host timer fires, which can be a moment after the count reads 0, and longer on a loaded host: on nvx-kvm-test (Linux 6.6, nested under Hyper-V), with every CPU busy, a delivery still found no expiry up to 4 ms after the count read 0, in 59,000 tries. So if the timer had expired at the read, the VP has KVM deliver again, after pauses that grow from 50 us to 1 ms, until the vector is set in IRR or 20 ms after the read, and only a delivery after that reports the interrupt as taken. The VP measures those 20 ms with Instant, or with the pauses that it slept when they add up to more, as a pause never ends early: if the host's monotonic clock steps back, Instant measures no time since the read until the clock is back where it was, which would otherwise stretch the wait, and stall the stop, and with it a capture, by the step. Only such a timer delays the stop, by up to 20 ms: one whose interrupt the guest took but whose timer it has not yet armed again or stopped, which a Linux guest does within microseconds, or one whose expiry KVM records late. An expiry that KVM records more than 20 ms after the read is saved as taken, and a restore does not deliver it. KVM records no expiry of the VMX preemption timer while the vCPU is outside guest mode, so a delivery cannot settle a timer that had expired at the read on an Intel host whose kvm_intel may use it, which the partition reads from /sys/module/kvm_intel/parameters/preemption_timer, and assumes when it cannot read that. There the backend reports such a timer as KVM does, and a restore fires it again, as before. AMD hosts have no preemption timer, and the nested KVM of nvx-kvm-test, an Azure VM, uses none. Saved states keep their format: one saved before this change restores as before. set_apic forgets the timer read at the stop, which the restored timer replaces. The KVM tests restore with vp_state::kvm_lapic_page, which becomes visible to the x86-64 backend for that. The Guide's KVM timekeeping section describes the capture. Tests: TimerAtStop::settle queues the vector of a timer that expired after the stop, zeroes the initial count of one taken before it, and leaves as KVM reports them a counting, masked, periodic, TSC-deadline, or NMI-mode timer, one with an invalid vector or already pending, and one of a disabled APIC. The wait for an expiry that KVM never records ends once the clock measures 20 ms after the read, or, if the clock steps back before or during the wait, once the pauses, which grow as with a working clock, add up to 20 ms. On KVM, with a real-mode guest that counts its timer interrupts, stopped, saved, and restored as the backend does: a stop runs no guest instruction and leaves the thread's signal mask unchanged; a timer whose interrupt the guest took, halted or spinning, is not taken again after a restore; a timer whose expiry KVM holds, from before or after the stop, is taken once; an expiry that KVM records only after the first delivery is saved pending, over 500 stops at the moment the count reads 0; and 200 stops at random times around the expiry, saved at once or up to 3 ms later, give each expiry exactly one interrupt. On Linux 6.6 (nvx-kvm-test), without settle the taken timer fired again after every restore, and without the delivery the held one was lost. A C program that saves such a timer at random times without OpenVMM gave 2 interrupts in 206 of 400 saves as KVM reports it, none in 94 of 400 with an initial count of 0 for every expired timer, and exactly 1 in all 400 with the delivery and this classification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
requested a review
from a team
as a code owner
November 12, 2026 02:35
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
The KVM-backed tests of the LAPIC timer module force their VP out of KVM_RUN with a POSIX timer that signals the test's thread, which they asked for with libc::SIGEV_THREAD_ID. The libc crate defines that constant for glibc but not for musl, so cargo clippy --all-targets failed to build the virt_kvm tests for x86_64-unknown-linux-musl, in the clippy and unit-test job of #5055. The tests now use Linux's value, 4, as the libc crate does for glibc. Nothing else changes, and no production code. Tests: cargo clippy --all-targets for virt_kvm, for x86_64-unknown-linux-gnu and x86_64-unknown-linux-musl, and the KVM-backed tests on the KVM test VM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 12, 2026
On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the guest had taken before the capture (#1390). KVM's LAPIC state shows such a timer, with its initial count, a current count of 0, and its vector clear in IRR, like one whose expiry KVM holds but has not injected, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce, and KVM_SET_LAPIC fires both again, which only the second needs: at once from Linux 6.3, and a full period later from 5.8 to 6.2. Pin c72fc56a3716fafaf1a9b5473c2de646c2a74815, the head of microsoft/openvmm#5055, three commits on 024ecee62, the head of feature/microvm/dev, which #1366 pinned: - 727cad63f kvm: deliver the interrupts that KVM holds without running the VP - 9f14b54fd virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again - c72fc56a3 virt_kvm: build the LAPIC timer KVM tests for musl When the KVM backend stops a VP, it now reads the LAPIC timer's count, and then has KVM deliver the interrupts that it holds, with a KVM_RUN that a pending signal, unblocked only in KVM_RUN, makes return before it enters the guest. The saved state then has the vector of a timer that expired after the read pending in IRR, which the restore of #1366 delivers once, and an initial count of 0 for a timer whose interrupt the guest took, which KVM does not start; the restored guest reads 0 from the initial count register, which Linux never reads. KVM records an expiry only once its host timer fires, up to 4 ms late on a busy nvx-kvm-test, so if the timer had expired at the read, the backend delivers again for up to 20 ms, which a step back of the host's monotonic clock cannot stretch, before it saves the interrupt as taken. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores as before. Two limits remain, which 3.2.5 states. An expiry that KVM records more than 20 ms late is saved as taken, and a restore does not deliver it. On a host whose kvm_intel runs the timer on the VMX preemption timer, which records no expiry outside guest mode, a timer that had expired at the stop is saved as KVM reports it, and fires once more after a restore. AMD hosts have no preemption timer, and the nested KVM of nvx-kvm-test, an Azure VM, uses none. MSHV and WHP are unchanged. Evidence: on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer), KVM-backed tests run a real-mode guest that counts its timer interrupts, stopped, saved, and restored as the backend does. A timer whose interrupt the guest took fired again after every restore before, and does not at the pin; a timer whose expiry KVM holds, from before or after the stop, fires once; 500 stops at the moment the count reads 0 save the expiry pending; and 200 stops at random times around a 2 ms timer give each expiry exactly one interrupt, also while eight other KVM processes run on the host's eight CPUs; at the pin, they also build for musl. With OpenVMM built from 9f14b54fd, whose code the pin's third commit, which changes only these tests, leaves as it is, the smp-lapic, smp-snapshot, and restore-processors microVM scenarios pass on KVM with 1, 2, 4, and 8 vCPUs, and snapshot-core, snapshot-rollback, and time-abi-conformance in their own configurations. With the first two commits on an earlier base, restore-downtime also passed in its own configurations, and smp, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, run-console-restore, and lifecycle with 1, 2, 4, and 8 vCPUs. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 12, 2026
The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at position 261, 130 commits past the mapped tip. Positions 259-261, 727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62, the head of feature/microvm/dev, which microsoft/openvmm#5053 merged. They have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped commit adds, so they stay unassigned. Position 259 adds a kvm crate call that has KVM deliver the interrupts that it holds without entering the guest; it applies to upstream main, ba105dd7f, without conflicts, and the kvm crate type-checks there, so it can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer module and rewrites the Guide paragraph of position 257; replaying it on upstream main conflicts in that paragraph, the x86-64 module list, and the VP state module, and its KVM tests use positions 256, 257, and 259, so it follows them upstream. Position 261 builds position 260's KVM tests for musl, and changes nothing else, so it folds into position 260. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
merged commit Nov 12, 2026
c72fc56
into
feature/microvm/dev
73 of 74 checks passed
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 12, 2026
A time ABI restore logged how long its phases took by subtracting one reading of an Instant::elapsed from another, in the "time ABI restore clock" event and in the synchronized TSC sets of MSHV and WHP. Instant::elapsed saturates when the host's monotonic clock steps back, which a faulty host can do, but the Duration subtraction panics with "overflow when subtracting durations", in release builds too, so a step back between two readings, far enough that the later reading was the smaller one, aborted the restore after the TSCs, LAPICs, and VM time were set, on any backend, at OpenVMM's default info log level (#1375). Pin c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, the head of microsoft/openvmm#5056, one commit on c72fc56a3, the head of feature/microvm/dev: - c2c21ddb2 microVM: saturate the time ABI restore's logged phase durations The revision adds virt::time_abi::phase_us, which gives a phase's microseconds from its two readings, 0 when the later reading is the smaller one, and computes with it every phase that the three events took as the difference of two readings. Evidence: on nvx-kvm-test, an LD_PRELOAD shim stepped the host's CLOCK_MONOTONIC back by 250 ms right after the "time ABI synchronized TSC set" event of the restores of the restore-processors scenario. The OpenVMM built without the fix panicked in Duration's subtraction at time_abi.rs:1304, in time_abi_restore, and the scenario failed; with the fix, it passed and logged lapic_us=0x0 vm_time_us=0x0. Without the shim, the restore-processors, snapshot-core, and time-abi-conformance scenarios pass on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test, and restores with the backends' events enabled log the MSHV and WHP phases that the fix computes. The new virt test panics with "overflow when subtracting durations" against the base's subtraction and passes at the pinned revision; the tests of virt, virt_whp, and openvmm_core pass on Windows, and those of virt, virt_mshv, and openvmm_core on Linux. These runs had the fix on an earlier revision of microsoft/openvmm#5055, which differs from c72fc56a3 only in the KVM backend's LAPIC timer capture and its tests; at the pinned revision, virt and openvmm_core build and the new test passes on Windows. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
added a commit
to microsoft/nvx
that referenced
this pull request
Nov 12, 2026
The NVX pin moves to c2c21ddb2, position 262, 131 commits past the mapped tip. Position 262, which microsoft/openvmm#5056 proposes for feature/microvm/dev on its head, c72fc56a3, is unassigned. The Scope also says that microsoft/openvmm#5055 merged positions 259-261. It computes the phases of the time ABI restore's clock event, which position 144 adds and position 146 gives its concurrent LAPIC and VM-time phases, and of the synchronized TSC sets of the MSHV and WHP backends of positions 134 and 140, that are the difference of two readings with a helper that it adds to virt's time ABI module of position 132, which gives 0 where the host's monotonic clock stepped back past the phase's start and the subtraction panicked. It changes nothing that a mapped PR adds. Replaying it on upstream main conflicts in all four of its files, since upstream main lacks the time ABI, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 12, 2026
Gautam Ahuja (ahuja-gautam)
pushed a commit
to ahuja-gautam/openvmm
that referenced
this pull request
Nov 13, 2026
The KVM-backed tests of the LAPIC timer module force their VP out of KVM_RUN with a POSIX timer that signals the test's thread, which they asked for with libc::SIGEV_THREAD_ID. The libc crate defines that constant for glibc but not for musl, so cargo clippy --all-targets failed to build the virt_kvm tests for x86_64-unknown-linux-musl, in the clippy and unit-test job of microsoft#5055. The tests now use Linux's value, 4, as the libc crate does for glibc. Nothing else changes, and no production code. Tests: cargo clippy --all-targets for virt_kvm, for x86_64-unknown-linux-gnu and x86_64-unknown-linux-musl, and the KVM-backed tests on the KVM test VM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keeps a restore on KVM from firing a one-shot LAPIC timer again whose interrupt the guest had already taken, for microsoft/nvx#1390, the case of microsoft/nvx#1366 that #5053 left open.
KVM_GET_LAPICsaves such a timer with the same LAPIC state as one whose expiry KVM recorded but has not injected yet: an initial count, a current count of 0, and the vector clear in IRR.KVM_SET_LAPICfires both again, at once from Linux 6.3 and a full period later from 5.8 to 6.2, which only the second needs, so the restored guest took a timer interrupt that it never programmed. The second state is common at a capture: OpenVMM stops a VP withimmediate_exit, which never moves KVM'slapic_timer.pendinginto IRR, and saves it only after the quiesce. So an initial count of 0 for every such page would lose interrupts instead; this tells the two states apart when the VP stops.Three commits on
024ecee62, the head offeature/microvm/dev. microsoft/nvx#1514 pins this head.Deliver the interrupts that KVM holds without running the VP (
727cad63f)VpRunner::deliver_pending_interruptscompletes the current exit, if any, and runsKVM_RUN's loop once without entering the guest: it blocks the thread's VP cancellation signal, queues one for the thread, and unblocks it only inKVM_RUN's signal mask (KVM_SET_SIGNAL_MASK), so KVM handles the vCPU's pending requests, injects its pending timer interrupts into IRR, and returnsEINTRbefore it enters the guest. A pending request that is an exit, such as the EOI of a userspace I/O APIC, is returned for the caller to handle, as fromVpRunner::run. The call clearsrequest_interrupt_windowfor that run, which KVM would otherwise report as an open interrupt window, then puts back a null signal mask, takes the one signal that it queued, and restores the thread's mask, so a cancellation that arrived meanwhile, a real-time signal that queues separately, reaches the handler as before.runand the new call share the mapping ofKVM_RUN's result and the decoding of its exit, which are unchanged. The newunsafeblocks are the signal-mask calls and the ioctl, each with itsSAFETYcomment.main(ba105dd7f) without conflicts, and thekvmcrate type-checks there.Keep a restore from firing a taken one-shot LAPIC timer again (
9f14b54fd)run_vpreads its LAPIC timer's current count (lapic_timer::TimerAtStop), then has KVM deliver the interrupts that it holds, handling the exits that the delivery reports as for a run, so that KVM then holds only an expiry from after the read.KvmVpStateAccess::apicthen reports an expired, unmasked, Fixed-mode one-shot timer whose valid vector is clear in IRR:set_apic'skvm_lapic_pagefrom [DO NOT REVIEW] vmm_core, virt_kvm: restore one-shot LAPIC timers without drift or a downtime refire #5053 restores it with an initial count of 0, so the guest takes it once;KVM_SET_LAPICdoes not start it again, and the guest reads 0 from the initial count register, which Linux never reads.Instant, or with the pauses that it slept when they add up to more, since a pause never ends early, so a step back of the host's monotonic clock, after whichInstantmeasures nothing until the clock is back, cannot stretch the wait, and with it the stop and a capture. Only such a timer delays the stop: one whose interrupt the guest took but whose timer it has not yet armed again or stopped, which a Linux guest does within microseconds, or one whose expiry KVM records late.kvm_intelmay run the timer on the VMX preemption timer, which the partition reads from/sys/module/kvm_intel/parameters/preemption_timerand assumes when it cannot read that, a delivery cannot settle a timer that had expired at the read, so the backend reports it as KVM does, and a restore fires it again, as before. AMD hosts have no preemption timer.set_apicforgets the timer read at the stop. The Guide's KVM timekeeping section describes the capture.TimerAtStop::settlequeues the vector of a timer that expired after the stop, zeroes the initial count of one taken before it, and leaves a counting, masked, periodic, TSC-deadline, or NMI-mode timer, one with an invalid vector or already pending, and one of a disabled APIC as KVM reports them; the wait for an expiry that KVM never records ends once the clock measures 20 ms, and, when the clock steps back before or during the wait, once the pauses, which grow as with a working clock, add up to 20 ms. The KVM-backed tests, ignored without/dev/kvm, run a real-mode guest that counts its timer interrupts and stop, save, and restore its VP as the backend does:stopping_runs_no_guest_instruction(and leaves the thread's signal mask unchanged),a_taken_timer_interrupt_is_not_taken_again_after_a_restore(halted or spinning),a_timer_interrupt_that_kvm_holds_is_taken_once_after_a_restore(from before or after the stop),an_expiry_that_kvm_records_after_the_stop_is_saved_as_pending(500 stops as the count reads 0), anda_restore_gives_each_timer_expiry_one_interrupt_whenever_the_vp_stops(200 stops at random times around a 2 ms expiry, saved at once or up to 3 ms later).Build the LAPIC timer KVM tests for musl (
c72fc56a3)KVM_RUNwith a POSIX timer that signals the test's thread, which they asked for withlibc::SIGEV_THREAD_ID. Thelibccrate defines that constant for glibc but not for musl, socargo clippy --all-targetsfailed to build thevirt_kvmtests forx86_64-unknown-linux-musl, in this pull request'sclippy [x64-linux-musl, misc nostd], unit tests [x64-linux-musl]job at9f14b54fd. The tests now use Linux's value, 4, as thelibccrate does for glibc. Only that test module changes.Not fixed here
The exact fix needs KVM: the
KVM_X86_QUIRK_LAPIC_ONESHOT_REFIREsketch in microsoft/nvx#1390, which upstream KVM has not taken, so this pull request codes against no quirk. Two limits remain until then:Validation
At this head,
c72fc56a3, with Rust 1.97.1, forkvmandvirt_kvm:cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agentpass; both packages are Linux-only, so they build nothing there.cargo xtask fmt --fixis clean, andrustfmt --checkovervirt_kvm's x86-64 backend, which the full pass does not reach throughcfg_select!, and overkvm, too.cargo clippy --all-targetswithout warnings forx86_64-unknown-linux-gnuandx86_64-unknown-linux-musl, where9f14b54fdfails as in CI;cargo doc --no-deps; andcargo nextest run --profile agent, 52 passed. The 13 KVM-backed tests ofvirt_kvm, 5 of them new, pass twice, once with eight other KVM processes busy on the VM's eight CPUs.At
9f14b54fd, whose code the third commit changes only in those tests:run vmm-tests [x64-linux-amd-kvm],cargo hack (linux), and every clippy and unit-test job but the musl one.9f14b54fd, on the KVM test VM:smp-lapic,smp-snapshot, andrestore-processorswith 1, 2, 4, and 8 VPs, andsnapshot-core,snapshot-rollback, andtime-abi-conformance, pass. With the same two commits on an earlier base,restore-downtime,smp,snapshot-tiers,console-snapshot,filesystem-snapshot,network-snapshot,run-console-restore, andlifecyclealso passed. MSHV and WHP are unchanged: onlykvmandvirt_kvmchange.settle, the taken timer fired again after every restore, and without the delivery, the held one was lost. A C program that saves such a timer at random times through the raw uAPI, without OpenVMM, gave 2 interrupts in 206 of 400 saves as KVM reports it, none in 94 of 400 with an initial count of 0 for every expired timer, and exactly 1 in all 400 with the delivery and this classification.See microsoft/nvx#1514.