Skip to content

openvmm: keep a restore from firing a taken one-shot LAPIC timer again - #1514

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1390-kvm-set-lapic-refire
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1390-kvm-set-lapic-refire

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1390. Closes #1366.

What and why

On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the guest had already taken before the capture: case 1 of #1366, which #1505 documented as a KVM limit, and the subject of #1390. KVM_GET_LAPIC saves such a timer with the same page as one whose expiry KVM recorded but has not injected yet: an initial count, a current count of 0, and the vector clear in IRR and ISR. KVM_SET_LAPIC fires both again, at once from Linux 6.3 and a full period later from 5.8 to 6.2, which only the second needs. The second state is common at OpenVMM's capture: the snapshot boundary stops the VPs with immediate_exit, which never moves KVM's held expiry (lapic_timer.pending) into IRR, and saves them after the quiesce. So an initial count of 0 for every such page, #1366's first option, would lose interrupts instead.

This change tells the two states apart when the KVM backend stops a VP, without any KVM change:

  • The backend reads the timer's current count, then has KVM deliver the interrupts that it holds, with a KVM_RUN that a pending signal, unblocked only in KVM_RUN's signal mask, makes return before it enters the guest. KVM then holds only an expiry from after the read.
  • Capture then saves a timer that expired after the read with its vector pending in IRR, which openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366's restore rule gives KVM with an initial count of 0, so that it fires once, and a timer whose interrupt the guest took with an initial count of 0, which KVM does not start. The restored guest reads 0 from that timer's initial count register, which Linux never reads.
  • KVM records an expiry only when its host timer fires, which on the busy nested KVM test VM took up to 4 ms after the count read 0, in 59,000 tries. So if the timer had already expired at the read, the backend delivers again, with pauses that grow from 50 µs to 1 ms, for up to 20 ms before it saves the interrupt as taken. Only a timer whose interrupt the guest took but has not armed again, which Linux does within microseconds, or whose expiry KVM records late, delays the stop. A step back of the host's monotonic clock cannot stretch that wait, since the backend also counts the pauses that it slept (the failure class of openvmm: time-ABI restore panics in a log line if the host clock steps back #1375).

Why Part of #1390. The exact fix needs KVM's opt-in quirk, the KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE sketch in #1390, which upstream KVM has not assigned, so this change codes against no quirk bit. Two limits remain on KVM until then, which 3.2.5 and the OpenVMM Guide state:

  1. An expiry that KVM records more than 20 ms after the timer's count read 0 is saved as taken, and the restore does not deliver it.
  2. On a host whose KVM may run the LAPIC timer on the VMX preemption timer, an Intel host whose kvm_intel preemption_timer parameter is not N or cannot be read, as on bare-metal Intel hosts, KVM records no expiry while the VP is outside guest mode. There capture saves a timer that had expired when its VP stopped as KVM reports it, and KVM's re-fire is kept: the restored timer fires once more, which is due if KVM had not delivered its interrupt and spurious if the guest had taken it. AMD hosts have no preemption timer, and the nested KVM of the CI runners and the KVM test VM uses none.

Why Closes #1366. #1505 fixed the issue's cases 2 and 3, and this change fixes case 1 as far as OpenVMM can: everything that OpenVMM can do for the three cases is done. What remains of case 1 is the two limits above, which only KVM's uAPI can lift, and which #1390, the KVM half of #1366, tracks and keeps open.

The pin, microsoft/openvmm#5055 at c72fc56a3716fafaf1a9b5473c2de646c2a74815, adds three commits to feature/microvm/dev, 024ecee62, #1366's pin:

  • 727cad63f kvm: deliver the interrupts that KVM holds without running the VP. VpRunner::deliver_pending_interrupts completes the current exit and runs KVM_RUN's loop once with the thread's VP cancellation signal pending and unblocked only in KVM_RUN (KVM_SET_SIGNAL_MASK), so KVM handles the vCPU's pending requests, injects its pending timer interrupts into IRR, and returns EINTR before it enters the guest. An exit that a pending request causes, such as a userspace I/O APIC's EOI, is returned to the caller as from run. The thread's signal mask, and a cancellation that arrives meanwhile, are kept.
  • 9f14b54fd virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again. The lapic_timer module, the stop sequence of run_vp, which reads the timer and delivers, again for up to 20 ms if needed, and the classification in KvmVpStateAccess::apic; set_apic forgets the timer read at the stop. The Guide's KVM timekeeping section describes the capture.
  • c72fc56a3 virt_kvm: build the LAPIC timer KVM tests for musl. Test-only: the KVM-backed tests asked for their stop timer's signal with libc::SIGEV_THREAD_ID, which the libc crate defines for glibc but not for musl, so OpenVMM's musl clippy job failed at 9f14b54fd; they now use Linux's value, 4.

The NVX commits:

  • bc3b6d96882 openvmm: pin captures that keep a taken LAPIC timer from firing again. It changes only the gitlink, from 024ecee62 to c72fc56a3.
  • 626e0931db5 doc: specify how capture tells a taken LAPIC timer apart on KVM. Section 3.2.5 rewrites openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366's case-1 paragraph and states the two limits.
  • 6c19fb9b879 doc: map the LAPIC timer capture fix in the OpenVMM roadmap. Positions 259 to 261.

No saved state, manifest, time ABI, contract, or format version changes. A snapshot taken before this change restores as before, with KVM's re-fire for a timer that had expired before its capture.

Validation

Local checks at this head (Windows):

  • python scripts/openvmm_pin.py check-branch --topic: passed (3 signed commits, no coordinated files).
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed (261 positions, 270 references).
  • python scripts/openvmm_pin.py source c72fc56a3716fafaf1a9b5473c2de646c2a74815 --pr 5055: good signature, and the revision is the pull request's head.
  • python scripts/nvx.py verify: passed.
  • python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run: passed. That is check_test_changes.py (0 changes to existing tests), check_markdown_links.py, check_test_layout.py, and the ignored openvmm_vectors (4) and openvmm_cpu_profiles (1) tests of nvx-rs.
  • git diff --check origin/dev HEAD: clean.
  • The nvx-cli gate, python -m unittest discover over scripts/: not run locally. This change does not touch scripts/, and the gate takes over two hours on this shared machine. It is left to CI's NVX CLI / Linux and NVX CLI / Windows jobs, which must pass on this head.

OpenVMM package checks at c72fc56a3, for kvm and virt_kvm:

  • Windows: cargo clippy --all-targets, cargo doc --no-deps, and cargo nextest run --profile agent passed; both packages build only for Linux. cargo xtask fmt --fix: clean, and rustfmt --check over virt_kvm's x86-64 backend, which the full pass does not reach through cfg_select!, and over kvm: clean.
  • nvx-kvm-test (Azure Linux 3, Linux 6.6, nested KVM without APICv or the VMX preemption timer): cargo clippy --all-targets without warnings for x86_64-unknown-linux-gnu and x86_64-unknown-linux-musl, where 9f14b54fd fails as in OpenVMM's CI; cargo doc --no-deps; and cargo nextest run --profile agent, 52 passed. The 13 KVM-backed tests of virt_kvm, 5 of them new, passed twice, once with eight other KVM processes busy on the VM's eight CPUs, and three times more at 9f14b54fd.
  • OpenVMM's CI at 9f14b54fd passed run vmm-tests [x64-linux-amd-kvm], cargo hack (linux), and every clippy and unit-test job but the musl one, which the third commit fixes.

Remote runs on nvx-kvm-test (kvm, virtual-machine). "Before" is OpenVMM before this change, or, for the KVM-backed tests, the change without the part that the test covers:

Profile Backend Host type Check Before After
nvx-kvm-test kvm virtual-machine C program on the raw uAPI, 400 saves of a 2 ms one-shot timer at random times 2 interrupts in 206 saves as KVM reports them; 0 in 94 with an initial count of 0 for every expired timer exactly 1 in all 400, with the delivery and this classification
nvx-kvm-test kvm virtual-machine KVM test, a timer whose interrupt the guest took, halted or spinning, then a restore fired again after every restore (without settle) not taken again
nvx-kvm-test kvm virtual-machine KVM test, a timer whose expiry KVM holds, from before or after the stop lost (without the delivery) taken once
nvx-kvm-test kvm virtual-machine KVM tests, 500 stops as the count reads 0, and 200 stops at random times around a 2 ms expiry, saved at once or up to 3 ms later - each expiry exactly one interrupt, also under load
nvx-kvm-test kvm virtual-machine unit test, a clock that steps back during the wait the wait does not end (bounded by the clock alone) ends once the pauses add up to 20 ms
nvx-kvm-test kvm virtual-machine microVM smp-lapic, smp-snapshot, and restore-processors (1, 2, 4, 8 VPs), snapshot-core, snapshot-rollback, time-abi-conformance, with OpenVMM built from 9f14b54fd, whose code the pin's third commit changes only in tests, over the v0.1.0-dev.c52fc3f2fd33 artifacts (binary hash 4bed6b09 to 180fa50e) not run passed
nvx-kvm-test kvm virtual-machine the first two commits on an earlier base: restore-downtime, and smp, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, run-console-restore, lifecycle (1, 2, 4, 8 VPs) not run passed

MSHV and WHP: not run on the test VMs. Only the kvm and virt_kvm crates change, and both build only for Linux with KVM; CI's microVM jobs run on all three backends.

CI at this head, 6c19fb9b879, in run 43149003122: passed.

  • OpenVMM CI of the pinned revision: passed. OpenVMM's required jobs passed for c72fc56a3 in its run 43148867189: unit tests [x64-linux], unit tests [x64-windows], and run vmm-tests on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd. Every other OpenVMM job passed too, the x64-linux-musl clippy and unit tests among them, but run vmm-tests [x64-windows-amd-snp], still queued, which NVX does not require.
  • NVX CLI / Linux and NVX CLI / Windows: passed.
  • The microVM tests on the KVM, MSHV, and WHP runners, including the WHP runtime adapter: passed.
  • The platform benchmarks and the performance regression gate: passed.

The first run of this pull request, at 985a6072f34, was superseded by this head after OpenVMM's clippy [x64-linux-musl, misc nostd], unit tests [x64-linux-musl] job failed at 9f14b54fd to build the new KVM tests for musl; c72fc56a3 fixes that.

No existing test changes, so no tests-modified label.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain in the documentation updates covering LAPIC timer restore behavior and the OpenVMM pin.

No open issues 🎉

What changed in this PR

Aligns NVX’s documentation with the OpenVMM fix for already-delivered one-shot LAPIC timers firing again after KVM restores, while documenting the remaining limitations tracked by #1390.

Changes:

  • Explain capture-time timer classification, the 20 ms wait, and the VMX preemption-timer fallback.
  • Update the documented OpenVMM pin and map its two new commits at positions 259–260.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin, commit mapping, and upstream dependency notes.
doc/​design/​3.2-clock-and-time.md Describes KVM timer capture behavior and its remaining limits.

Review effort: Balanced

On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the
guest had taken before the capture (#1390). KVM's LAPIC state shows such
a timer, with its initial count, a current count of 0, and its vector
clear in IRR, like one whose expiry KVM holds but has not injected,
which is the common state at the capture, since the snapshot boundary
stops the VPs before the quiesce, and KVM_SET_LAPIC fires both again,
which only the second needs: at once from Linux 6.3, and a full period
later from 5.8 to 6.2.

Pin c72fc56a3716fafaf1a9b5473c2de646c2a74815, the head of
microsoft/openvmm#5055, three commits on 024ecee62, the head of
feature/microvm/dev, which #1366 pinned:

- 727cad63f kvm: deliver the interrupts that KVM holds without running
  the VP
- 9f14b54fd virt_kvm: keep a restore from firing a taken one-shot LAPIC
  timer again
- c72fc56a3 virt_kvm: build the LAPIC timer KVM tests for musl

When the KVM backend stops a VP, it now reads the LAPIC timer's count,
and then has KVM deliver the interrupts that it holds, with a KVM_RUN
that a pending signal, unblocked only in KVM_RUN, makes return before it
enters the guest. The saved state then has the vector of a timer that
expired after the read pending in IRR, which the restore of #1366
delivers once, and an initial count of 0 for a timer whose interrupt the
guest took, which KVM does not start; the restored guest reads 0 from
the initial count register, which Linux never reads. KVM records an
expiry only once its host timer fires, up to 4 ms late on a busy
nvx-kvm-test, so if the timer had expired at the read, the backend
delivers again for up to 20 ms, which a step back of the host's
monotonic clock cannot stretch, before it saves the interrupt as taken.
No saved-state, manifest, or time ABI format changes; a snapshot taken
before this pin restores as before.

Two limits remain, which 3.2.5 states. An expiry that KVM records more
than 20 ms late is saved as taken, and a restore does not deliver it.
On a host whose kvm_intel runs the timer on the VMX preemption timer,
which records no expiry outside guest mode, a timer that had expired at
the stop is saved as KVM reports it, and fires once more after a
restore. AMD hosts have no preemption timer, and the nested KVM of
nvx-kvm-test, an Azure VM, uses none. MSHV and WHP are unchanged.

Evidence: on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX
preemption timer), KVM-backed tests run a real-mode guest that counts
its timer interrupts, stopped, saved, and restored as the backend does.
A timer whose interrupt the guest took fired again after every restore
before, and does not at the pin; a timer whose expiry KVM holds, from
before or after the stop, fires once; 500 stops at the moment the count
reads 0 save the expiry pending; and 200 stops at random times around a
2 ms timer give each expiry exactly one interrupt, also while eight
other KVM processes run on the host's eight CPUs; at the pin, they also
build for musl. With OpenVMM built from 9f14b54fd, whose code the pin's
third commit, which changes only these tests, leaves as it is, the
smp-lapic, smp-snapshot, and restore-processors microVM scenarios pass
on KVM with 1, 2, 4, and 8 vCPUs, and snapshot-core, snapshot-rollback,
and time-abi-conformance in their own configurations. With the first two
commits on an earlier base, restore-downtime also passed in its own
configurations, and smp, snapshot-tiers, console-snapshot,
filesystem-snapshot, network-snapshot, run-console-restore, and
lifecycle with 1, 2, 4, and 8 vCPUs.

NVX needs no other coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that nvx-rs
copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3.2.5 said that on KVM a one-shot LAPIC timer that expired before the
capture fires once more after the restore, spuriously if the guest had
taken its interrupt (#1390). It now says how the KVM backend tells that
timer apart from one whose interrupt KVM still holds: when it stops a
processor, it reads the timer's count, has KVM deliver the interrupts
that it holds without running the processor, and delivers again for up
to 20 ms if the timer had already expired, since KVM records an expiry
only once its host timer fires. Of the expired timers whose unmasked LVT
entry delivers a fixed interrupt to a valid vector, capture saves one
that expired after the read with its vector pending in IRR, which the
restore gives KVM with an initial count of zero, as for a timer that
expires in the downtime, and one whose interrupt the processor took with
an initial count of zero, which the restored guest reads instead of the
count that it wrote.

It also states the two limits that remain on KVM: an expiry that KVM
records more than 20 ms late is saved as taken, and not delivered after
a restore, and a host whose KVM runs the timer on the VMX preemption
timer saves a timer that had expired at the stop as KVM reports it, so
that it fires once more.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at
position 261, 130 commits past the mapped tip. Positions 259-261,
727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62,
the head of feature/microvm/dev, which microsoft/openvmm#5053 merged.

They have the KVM backend tell a one-shot LAPIC timer whose interrupt
the guest took apart from one whose expiry KVM holds when it saves a VP,
and change nothing that a mapped commit adds, so they stay unassigned.
Position 259 adds a kvm crate call that has KVM deliver the interrupts
that it holds without entering the guest; it applies to upstream main,
ba105dd7f, without conflicts, and the kvm crate type-checks there, so it
can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer
module and rewrites the Guide paragraph of position 257; replaying it on
upstream main conflicts in that paragraph, the x86-64 module list, and
the VP state module, and its KVM tests use positions 256, 257, and 259,
so it follows them upstream. Position 261 builds position 260's KVM
tests for musl, and changes nothing else, so it folds into position 260.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the agents/issue-1390-kvm-set-lapic-refire branch from 985a607 to 6c19fb9 Compare November 12, 2026 03:33
Copilot AI balanced review requested due to automatic review settings November 12, 2026 03:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The KVM LAPIC timer-capture documentation and roadmap updates have no active review issues.

No open issues 🎉

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 12, 2026
Merged via the queue into dev with commit f7720fb Nov 12, 2026
50 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1390-kvm-set-lapic-refire branch November 12, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift

2 participants