Repository navigation
openvmm: keep a restore from firing a taken one-shot LAPIC timer again - #1514
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 12, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain in the documentation updates covering LAPIC timer restore behavior and the OpenVMM pin.
No open issues 🎉
What changed in this PR
Aligns NVX’s documentation with the OpenVMM fix for already-delivered one-shot LAPIC timers firing again after KVM restores, while documenting the remaining limitations tracked by #1390.
Changes:
- Explain capture-time timer classification, the 20 ms wait, and the VMX preemption-timer fallback.
- Update the documented OpenVMM pin and map its two new commits at positions 259–260.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Updates the pin, commit mapping, and upstream dependency notes. |
doc/design/3.2-clock-and-time.md |
Describes KVM timer capture behavior and its remaining limits. |
Review effort: Balanced
On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the guest had taken before the capture (#1390). KVM's LAPIC state shows such a timer, with its initial count, a current count of 0, and its vector clear in IRR, like one whose expiry KVM holds but has not injected, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce, and KVM_SET_LAPIC fires both again, which only the second needs: at once from Linux 6.3, and a full period later from 5.8 to 6.2. Pin c72fc56a3716fafaf1a9b5473c2de646c2a74815, the head of microsoft/openvmm#5055, three commits on 024ecee62, the head of feature/microvm/dev, which #1366 pinned: - 727cad63f kvm: deliver the interrupts that KVM holds without running the VP - 9f14b54fd virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again - c72fc56a3 virt_kvm: build the LAPIC timer KVM tests for musl When the KVM backend stops a VP, it now reads the LAPIC timer's count, and then has KVM deliver the interrupts that it holds, with a KVM_RUN that a pending signal, unblocked only in KVM_RUN, makes return before it enters the guest. The saved state then has the vector of a timer that expired after the read pending in IRR, which the restore of #1366 delivers once, and an initial count of 0 for a timer whose interrupt the guest took, which KVM does not start; the restored guest reads 0 from the initial count register, which Linux never reads. KVM records an expiry only once its host timer fires, up to 4 ms late on a busy nvx-kvm-test, so if the timer had expired at the read, the backend delivers again for up to 20 ms, which a step back of the host's monotonic clock cannot stretch, before it saves the interrupt as taken. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores as before. Two limits remain, which 3.2.5 states. An expiry that KVM records more than 20 ms late is saved as taken, and a restore does not deliver it. On a host whose kvm_intel runs the timer on the VMX preemption timer, which records no expiry outside guest mode, a timer that had expired at the stop is saved as KVM reports it, and fires once more after a restore. AMD hosts have no preemption timer, and the nested KVM of nvx-kvm-test, an Azure VM, uses none. MSHV and WHP are unchanged. Evidence: on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer), KVM-backed tests run a real-mode guest that counts its timer interrupts, stopped, saved, and restored as the backend does. A timer whose interrupt the guest took fired again after every restore before, and does not at the pin; a timer whose expiry KVM holds, from before or after the stop, fires once; 500 stops at the moment the count reads 0 save the expiry pending; and 200 stops at random times around a 2 ms timer give each expiry exactly one interrupt, also while eight other KVM processes run on the host's eight CPUs; at the pin, they also build for musl. With OpenVMM built from 9f14b54fd, whose code the pin's third commit, which changes only these tests, leaves as it is, the smp-lapic, smp-snapshot, and restore-processors microVM scenarios pass on KVM with 1, 2, 4, and 8 vCPUs, and snapshot-core, snapshot-rollback, and time-abi-conformance in their own configurations. With the first two commits on an earlier base, restore-downtime also passed in its own configurations, and smp, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, run-console-restore, and lifecycle with 1, 2, 4, and 8 vCPUs. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
3.2.5 said that on KVM a one-shot LAPIC timer that expired before the capture fires once more after the restore, spuriously if the guest had taken its interrupt (#1390). It now says how the KVM backend tells that timer apart from one whose interrupt KVM still holds: when it stops a processor, it reads the timer's count, has KVM deliver the interrupts that it holds without running the processor, and delivers again for up to 20 ms if the timer had already expired, since KVM records an expiry only once its host timer fires. Of the expired timers whose unmasked LVT entry delivers a fixed interrupt to a valid vector, capture saves one that expired after the read with its vector pending in IRR, which the restore gives KVM with an initial count of zero, as for a timer that expires in the downtime, and one whose interrupt the processor took with an initial count of zero, which the restored guest reads instead of the count that it wrote. It also states the two limits that remain on KVM: an expiry that KVM records more than 20 ms late is saved as taken, and not delivered after a restore, and a host whose KVM runs the timer on the VMX preemption timer saves a timer that had expired at the stop as KVM reports it, so that it fires once more. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at position 261, 130 commits past the mapped tip. Positions 259-261, 727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62, the head of feature/microvm/dev, which microsoft/openvmm#5053 merged. They have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped commit adds, so they stay unassigned. Position 259 adds a kvm crate call that has KVM deliver the interrupts that it holds without entering the guest; it applies to upstream main, ba105dd7f, without conflicts, and the kvm crate type-checks there, so it can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer module and rewrites the Guide paragraph of position 257; replaying it on upstream main conflicts in that paragraph, the x86-64 module list, and the VP state module, and its KVM tests use positions 256, 257, and 259, so it follows them upstream. Position 261 builds position 260's KVM tests for musl, and changes nothing else, so it folds into position 260. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
force-pushed
the
agents/issue-1390-kvm-set-lapic-refire
branch
from
November 12, 2026 03:33
985a607 to
6c19fb9
Compare
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1390-kvm-set-lapic-refire
branch
November 12, 2026 05:07
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1390. Closes #1366.
What and why
On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the guest had already taken before the capture: case 1 of #1366, which #1505 documented as a KVM limit, and the subject of #1390.
KVM_GET_LAPICsaves such a timer with the same page as one whose expiry KVM recorded but has not injected yet: an initial count, a current count of 0, and the vector clear in IRR and ISR.KVM_SET_LAPICfires both again, at once from Linux 6.3 and a full period later from 5.8 to 6.2, which only the second needs. The second state is common at OpenVMM's capture: the snapshot boundary stops the VPs withimmediate_exit, which never moves KVM's held expiry (lapic_timer.pending) into IRR, and saves them after the quiesce. So an initial count of 0 for every such page, #1366's first option, would lose interrupts instead.This change tells the two states apart when the KVM backend stops a VP, without any KVM change:
KVM_RUNthat a pending signal, unblocked only inKVM_RUN's signal mask, makes return before it enters the guest. KVM then holds only an expiry from after the read.Why
Part of #1390. The exact fix needs KVM's opt-in quirk, theKVM_X86_QUIRK_LAPIC_ONESHOT_REFIREsketch in #1390, which upstream KVM has not assigned, so this change codes against no quirk bit. Two limits remain on KVM until then, which 3.2.5 and the OpenVMM Guide state:kvm_intelpreemption_timerparameter is notNor cannot be read, as on bare-metal Intel hosts, KVM records no expiry while the VP is outside guest mode. There capture saves a timer that had expired when its VP stopped as KVM reports it, and KVM's re-fire is kept: the restored timer fires once more, which is due if KVM had not delivered its interrupt and spurious if the guest had taken it. AMD hosts have no preemption timer, and the nested KVM of the CI runners and the KVM test VM uses none.Why
Closes #1366. #1505 fixed the issue's cases 2 and 3, and this change fixes case 1 as far as OpenVMM can: everything that OpenVMM can do for the three cases is done. What remains of case 1 is the two limits above, which only KVM's uAPI can lift, and which #1390, the KVM half of #1366, tracks and keeps open.The pin, microsoft/openvmm#5055 at
c72fc56a3716fafaf1a9b5473c2de646c2a74815, adds three commits tofeature/microvm/dev,024ecee62, #1366's pin:727cad63fkvm: deliver the interrupts that KVM holds without running the VP.VpRunner::deliver_pending_interruptscompletes the current exit and runsKVM_RUN's loop once with the thread's VP cancellation signal pending and unblocked only inKVM_RUN(KVM_SET_SIGNAL_MASK), so KVM handles the vCPU's pending requests, injects its pending timer interrupts into IRR, and returnsEINTRbefore it enters the guest. An exit that a pending request causes, such as a userspace I/O APIC's EOI, is returned to the caller as fromrun. The thread's signal mask, and a cancellation that arrives meanwhile, are kept.9f14b54fdvirt_kvm: keep a restore from firing a taken one-shot LAPIC timer again. Thelapic_timermodule, the stop sequence ofrun_vp, which reads the timer and delivers, again for up to 20 ms if needed, and the classification inKvmVpStateAccess::apic;set_apicforgets the timer read at the stop. The Guide's KVM timekeeping section describes the capture.c72fc56a3virt_kvm: build the LAPIC timer KVM tests for musl. Test-only: the KVM-backed tests asked for their stop timer's signal withlibc::SIGEV_THREAD_ID, which thelibccrate defines for glibc but not for musl, so OpenVMM's musl clippy job failed at9f14b54fd; they now use Linux's value, 4.The NVX commits:
bc3b6d96882openvmm: pin captures that keep a taken LAPIC timer from firing again. It changes only the gitlink, from024ecee62toc72fc56a3.626e0931db5doc: specify how capture tells a taken LAPIC timer apart on KVM. Section 3.2.5 rewrites openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366's case-1 paragraph and states the two limits.6c19fb9b879doc: map the LAPIC timer capture fix in the OpenVMM roadmap. Positions 259 to 261.No saved state, manifest, time ABI, contract, or format version changes. A snapshot taken before this change restores as before, with KVM's re-fire for a timer that had expired before its capture.
Validation
Local checks at this head (Windows):
python scripts/openvmm_pin.py check-branch --topic: passed (3 signed commits, no coordinated files).python scripts/openvmm_pin.py check-roadmap --worktree: passed (261 positions, 270 references).python scripts/openvmm_pin.py source c72fc56a3716fafaf1a9b5473c2de646c2a74815 --pr 5055: good signature, and the revision is the pull request's head.python scripts/nvx.py verify: passed.python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run: passed. That ischeck_test_changes.py(0 changes to existing tests),check_markdown_links.py,check_test_layout.py, and the ignoredopenvmm_vectors(4) andopenvmm_cpu_profiles(1) tests ofnvx-rs.git diff --check origin/dev HEAD: clean.nvx-cligate,python -m unittest discoveroverscripts/: not run locally. This change does not touchscripts/, and the gate takes over two hours on this shared machine. It is left to CI'sNVX CLI / LinuxandNVX CLI / Windowsjobs, which must pass on this head.OpenVMM package checks at
c72fc56a3, forkvmandvirt_kvm:cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agentpassed; both packages build only for Linux.cargo xtask fmt --fix: clean, andrustfmt --checkovervirt_kvm's x86-64 backend, which the full pass does not reach throughcfg_select!, and overkvm: clean.cargo clippy --all-targetswithout warnings forx86_64-unknown-linux-gnuandx86_64-unknown-linux-musl, where9f14b54fdfails as in OpenVMM's CI;cargo doc --no-deps; andcargo nextest run --profile agent, 52 passed. The 13 KVM-backed tests ofvirt_kvm, 5 of them new, passed twice, once with eight other KVM processes busy on the VM's eight CPUs, and three times more at9f14b54fd.9f14b54fdpassedrun vmm-tests [x64-linux-amd-kvm],cargo hack (linux), and every clippy and unit-test job but the musl one, which the third commit fixes.Remote runs on nvx-kvm-test (kvm, virtual-machine). "Before" is OpenVMM before this change, or, for the KVM-backed tests, the change without the part that the test covers:
settle)smp-lapic,smp-snapshot, andrestore-processors(1, 2, 4, 8 VPs),snapshot-core,snapshot-rollback,time-abi-conformance, with OpenVMM built from9f14b54fd, whose code the pin's third commit changes only in tests, over thev0.1.0-dev.c52fc3f2fd33artifacts (binary hash4bed6b09to180fa50e)restore-downtime, andsmp,snapshot-tiers,console-snapshot,filesystem-snapshot,network-snapshot,run-console-restore,lifecycle(1, 2, 4, 8 VPs)MSHV and WHP: not run on the test VMs. Only the
kvmandvirt_kvmcrates change, and both build only for Linux with KVM; CI's microVM jobs run on all three backends.CI at this head,
6c19fb9b879, in run 43149003122: passed.OpenVMM CI of the pinned revision: passed. OpenVMM's required jobs passed forc72fc56a3in its run 43148867189:unit tests [x64-linux],unit tests [x64-windows], andrun vmm-testsonx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd. Every other OpenVMM job passed too, thex64-linux-muslclippy and unit tests among them, butrun vmm-tests [x64-windows-amd-snp], still queued, which NVX does not require.NVX CLI / LinuxandNVX CLI / Windows: passed.The first run of this pull request, at
985a6072f34, was superseded by this head after OpenVMM'sclippy [x64-linux-musl, misc nostd], unit tests [x64-linux-musl]job failed at9f14b54fdto build the new KVM tests for musl;c72fc56a3fixes that.No existing test changes, so no
tests-modifiedlabel.c72fc56a3716fafaf1a9b5473c2de646c2a74815.024ecee62, the head offeature/microvm/devafter [DO NOT REVIEW] vmm_core, virt_kvm: restore one-shot LAPIC timers without drift or a downtime refire openvmm#5053 (openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366). There are no other prerequisites.d5e9acee, that the package checks and KVM-backed tests above validated; the microVM runs used9f14b54fd, treeb36fecea, which differs from it only in the test module thatc72fc56a3changes. Against the first revisions, on an earlier version of openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366's commits,git range-diffshows727cad63fpatch-identical toa5ff2654d;9f14b54fddiffers fromfc5e966c3in the Guide paragraph, merged with the wording that openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366's21a76f878landed with, and in the bound of the 20 ms wait against a clock that steps back, from openvmm: time-ABI restore panics in a log line if the host clock steps back #1375's audit.feature/microvm/devfrom024ecee62to exactlyc72fc56a3716fafaf1a9b5473c2de646c2a74815. OpenVMM's merge button would squash the commits and leave the pin off the branch. This pull request then merges through the queue.