From bc3b6d9688253c60da4166bcb3ecfe3ea063c43a Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 10:30:49 -0800 Subject: [PATCH 1/3] openvmm: pin captures that keep a taken LAPIC timer from firing again On KVM, a restore fired a one-shot LAPIC timer again whose interrupt the guest had taken before the capture (#1390). KVM's LAPIC state shows such a timer, with its initial count, a current count of 0, and its vector clear in IRR, like one whose expiry KVM holds but has not injected, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce, and KVM_SET_LAPIC fires both again, which only the second needs: at once from Linux 6.3, and a full period later from 5.8 to 6.2. Pin c72fc56a3716fafaf1a9b5473c2de646c2a74815, the head of microsoft/openvmm#5055, three commits on 024ecee62, the head of feature/microvm/dev, which #1366 pinned: - 727cad63f kvm: deliver the interrupts that KVM holds without running the VP - 9f14b54fd virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again - c72fc56a3 virt_kvm: build the LAPIC timer KVM tests for musl When the KVM backend stops a VP, it now reads the LAPIC timer's count, and then has KVM deliver the interrupts that it holds, with a KVM_RUN that a pending signal, unblocked only in KVM_RUN, makes return before it enters the guest. The saved state then has the vector of a timer that expired after the read pending in IRR, which the restore of #1366 delivers once, and an initial count of 0 for a timer whose interrupt the guest took, which KVM does not start; the restored guest reads 0 from the initial count register, which Linux never reads. KVM records an expiry only once its host timer fires, up to 4 ms late on a busy nvx-kvm-test, so if the timer had expired at the read, the backend delivers again for up to 20 ms, which a step back of the host's monotonic clock cannot stretch, before it saves the interrupt as taken. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores as before. Two limits remain, which 3.2.5 states. An expiry that KVM records more than 20 ms late is saved as taken, and a restore does not deliver it. On a host whose kvm_intel runs the timer on the VMX preemption timer, which records no expiry outside guest mode, a timer that had expired at the stop is saved as KVM reports it, and fires once more after a restore. AMD hosts have no preemption timer, and the nested KVM of nvx-kvm-test, an Azure VM, uses none. MSHV and WHP are unchanged. Evidence: on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer), KVM-backed tests run a real-mode guest that counts its timer interrupts, stopped, saved, and restored as the backend does. A timer whose interrupt the guest took fired again after every restore before, and does not at the pin; a timer whose expiry KVM holds, from before or after the stop, fires once; 500 stops at the moment the count reads 0 save the expiry pending; and 200 stops at random times around a 2 ms timer give each expiry exactly one interrupt, also while eight other KVM processes run on the host's eight CPUs; at the pin, they also build for musl. With OpenVMM built from 9f14b54fd, whose code the pin's third commit, which changes only these tests, leaves as it is, the smp-lapic, smp-snapshot, and restore-processors microVM scenarios pass on KVM with 1, 2, 4, and 8 vCPUs, and snapshot-core, snapshot-rollback, and time-abi-conformance in their own configurations. With the first two commits on an earlier base, restore-downtime also passed in its own configurations, and smp, snapshot-tiers, console-snapshot, filesystem-snapshot, network-snapshot, run-console-restore, and lifecycle with 1, 2, 4, and 8 vCPUs. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 024ecee62..c72fc56a3 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 024ecee62d7c9b70456b63a5b4f94a6488f835cd +Subproject commit c72fc56a3716fafaf1a9b5473c2de646c2a74815 From 626e0931db54cb154093c8c9b7597550a9fc5b72 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 07:40:05 -0800 Subject: [PATCH 2/3] doc: specify how capture tells a taken LAPIC timer apart on KVM 3.2.5 said that on KVM a one-shot LAPIC timer that expired before the capture fires once more after the restore, spuriously if the guest had taken its interrupt (#1390). It now says how the KVM backend tells that timer apart from one whose interrupt KVM still holds: when it stops a processor, it reads the timer's count, has KVM deliver the interrupts that it holds without running the processor, and delivers again for up to 20 ms if the timer had already expired, since KVM records an expiry only once its host timer fires. Of the expired timers whose unmasked LVT entry delivers a fixed interrupt to a valid vector, capture saves one that expired after the read with its vector pending in IRR, which the restore gives KVM with an initial count of zero, as for a timer that expires in the downtime, and one whose interrupt the processor took with an initial count of zero, which the restored guest reads instead of the count that it wrote. It also states the two limits that remain on KVM: an expiry that KVM records more than 20 ms late is saved as taken, and not delivered after a restore, and a host whose KVM runs the timer on the VMX preemption timer saves a timer that had expired at the stop as KVM reports it, so that it fires once more. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/3.2-clock-and-time.md | 37 ++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 036d48207..9c39121d8 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -241,15 +241,34 @@ so the saved TSC dates the saved count. The restore runs no timer until it advances the count, by `ticks` and by the ticks for which the TSC has run from the target that the restore wrote to it. -A one-shot timer that expired before the capture keeps its counts. KVM's saved -LAPIC state does not show whether KVM had delivered the timer's interrupt, so -on KVM the restored timer fires once more: the interrupt is due if KVM had not -delivered it, and spurious if the guest had already taken it. KVM fires it at -once from Linux 6.3, and a full timer period later before. On KVM, an expired -timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector -that is pending in the interrupt request register (IRR), which already holds -the interrupt, as after a downtime that expires the timer, reads an initial -count of zero after the restore, so that KVM does not fire it again. +A one-shot timer that expired before the capture keeps its counts, except +that on KVM, capture saves one whose interrupt the processor took with an +initial count of zero, which the restored guest reads instead of the count +that it wrote; Linux never reads it. KVM's LAPIC state shows such a timer like +one whose interrupt is still due, since KVM holds the expiry of a timer that +expires while its processor is outside the guest until the processor next +runs, and KVM's LAPIC state write fires a timer whose current count is zero +again, at once from Linux 6.3 and a full timer period later before. So when +the KVM backend stops a processor, it reads the timer's count and then has KVM +deliver the interrupts that it holds, without running the processor. KVM +records an expiry only once its host timer fires, which a loaded host can +delay, so if the timer had already expired at that read, the backend has KVM +deliver again until the interrupt appears, for up to 20 ms. Of the expired +timers whose unmasked LVT entry delivers a fixed interrupt to a valid vector, +capture then saves one that expired after the read with its vector pending in +the interrupt request register (IRR), and takes one whose vector is still +clear then for one whose interrupt the processor took. On KVM, such a timer +whose vector is pending in IRR, which already holds the interrupt, as after a +downtime that expires the timer, reads an initial count of zero after the +restore, so that KVM does not fire it again. + +Two limits remain on KVM. If KVM records an expiry more than 20 ms late, +capture saves its interrupt as taken, and the restore does not deliver it. A +KVM that runs the timer on the VMX preemption timer, as on bare-metal Intel +hosts, records no expiry while the processor is outside guest mode, so on such +a host capture saves a timer that had expired when its processor stopped as +KVM reports it, and the restored timer fires once more: the interrupt is due +if KVM had not delivered it, and spurious if the guest had already taken it. ## 3.2.6 Backend Obligations From 6c19fb9b879fc5b552465f9aeec68bda153a52e6 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 18:37:29 -0800 Subject: [PATCH 3/3] doc: map the LAPIC timer capture fix in the OpenVMM roadmap The NVX pin moves to c72fc56a3, the head of microsoft/openvmm#5055, at position 261, 130 commits past the mapped tip. Positions 259-261, 727cad63f, 9f14b54fd, and c72fc56a3, follow position 258, 024ecee62, the head of feature/microvm/dev, which microsoft/openvmm#5053 merged. They have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped commit adds, so they stay unassigned. Position 259 adds a kvm crate call that has KVM deliver the interrupts that it holds without entering the guest; it applies to upstream main, ba105dd7f, without conflicts, and the kvm crate type-checks there, so it can go upstream on its own. Position 260 adds virt_kvm's LAPIC timer module and rewrites the Guide paragraph of position 257; replaying it on upstream main conflicts in that paragraph, the x86-64 module list, and the VP state module, and its KVM tests use positions 256, 257, and 259, so it follows them upstream. Position 261 builds position 260's KVM tests for musl, and changes nothing else, so it folds into position 260. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 39 ++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index dccf0a18e..39fda910a 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -10,9 +10,10 @@ summary, and commit mapping below cover positions 001-131, the PR09 fixups at positions 241-243, the PR13 fixup at position 247, the PR18 fixup at position 248, the PR20 fixup at position 249, and the PR19 fixup at position 250. -- NVX's pinned OpenVMM revision: `024ecee62d7c9b70456b63a5b4f94a6488f835cd`, position 258, the - head of microsoft/openvmm#5053, whose three commits follow position 255, `90f87319d`, the head - of `feature/microvm/dev`, which microsoft/openvmm#5052 merged with position 254 after position +- NVX's pinned OpenVMM revision: `c72fc56a3716fafaf1a9b5473c2de646c2a74815`, position 261, the + head of microsoft/openvmm#5055, whose three commits follow position 258, `024ecee62`, the head of + `feature/microvm/dev`, which microsoft/openvmm#5053 merged with positions 256 and 257 after + position 255, `90f87319d`, which microsoft/openvmm#5052 merged with position 254 after position 253, `3cd9f149b`, which microsoft/openvmm#5046 merged after position 252, `9f10ce7a8`, which microsoft/openvmm#5044 merged after position 251, `174da3a0d`, which microsoft/openvmm#5042 merged after position 250, `23136b647`, @@ -45,8 +46,8 @@ position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1 positions 149-220 are positions 148-219, and the tree of position 220 is the tree of - `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 127 commits past - the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-258 are not + `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 130 commits past + the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-261 are not yet assigned to PRs, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, and position 250 is a PR19 fixup; see [Commits after the mapped tip](#commits-after-the-mapped-tip). @@ -1029,7 +1030,8 @@ ba105dd7f -> 365-day maximum snapshot downtime [253] -> restored VM time run from the TSCs' restart [254-255] -> LAPIC timer restores without drift or a downtime refire [256-258] - -> 024ecee62 (NVX pin) + -> LAPIC timer captures that keep a taken timer from refiring [259-261] + -> c72fc56a3 (NVX pin) ``` For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main, @@ -1066,7 +1068,7 @@ helper that the case uses. ## Commits after the mapped tip -The NVX pin extends the chain past position 131 without merges. Of the 127 later commits, +The NVX pin extends the chain past position 131 without merges. Of the 130 later commits, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, position 250 is a PR19 fixup, and the others are not yet classified or assigned to PRs. @@ -1074,7 +1076,7 @@ Each of positions 132-186 combines the v2.4 commits that the table lists, as v3. and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0 -reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-258 followed the +reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-261 followed the v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions 222 and 223. Each commit has the position that it had in v5.1 through position 147, and one less from v5.1 position 149 on; see @@ -1210,6 +1212,9 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4 | 256 | `2a0401c43` | - | - | - | - | virt_kvm: share the guest RAM of the real-mode KVM tests (microsoft/openvmm#5053) | | 257 | `21a76f878` | - | - | - | - | virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending (microsoft/openvmm#5053) | | 258 | `024ecee62` | - | - | - | - | vmm_core, virt: run a restored LAPIC timer from the time ABI anchors (microsoft/openvmm#5053) | +| 259 | `727cad63f` | - | - | - | - | kvm: deliver the interrupts that KVM holds without running the VP (microsoft/openvmm#5055) | +| 260 | `9f14b54fd` | - | - | - | - | virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again (microsoft/openvmm#5055) | +| 261 | `c72fc56a3` | - | - | - | - | virt_kvm: build the LAPIC timer KVM tests for musl (microsoft/openvmm#5055); fold into 260 | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -1847,6 +1852,24 @@ files, seven of which upstream main lacks, those that PR10's positions 024 and 0 position 036, and positions 132, 135, 139, and 256 add, so it follows the time ABI series upstream. +Positions 259-261 have the KVM backend tell a one-shot LAPIC timer whose interrupt the guest took +apart from one whose expiry KVM holds when it saves a VP, and change nothing that a mapped PR +adds. Position 259 adds a call to the `kvm` crate's VP runner that has KVM deliver the interrupts +that it holds without entering the guest, in a file that upstream main has and that PR10's +position 026 and positions 133, 138, 147, and 148 also change, none of them in the runner's +`KVM_RUN` and exit decoding, which it changes and upstream main has. Replaying it on upstream main +applies without conflicts, and the `kvm` crate type-checks there, so it can go upstream on its own. + +Position 260 adds a LAPIC timer module to `virt_kvm`'s x86-64 backend, which the VP run loop, the +partition, and the VP state's `apic` and `set_apic` use, in files that upstream main has and that +PR10's positions 023 and 026, PR17's position 077, and positions 133, 138, 144, 147, 149, 150, +255, 256, 257, and 258 also change, and rewrites the paragraph of the Guide's timekeeping reference +that position 257 adds. Replaying it on upstream main conflicts in that paragraph, in the module +list of the x86-64 module, and in the VP state module, whose KVM LAPIC page from position 257 its +KVM tests use, with position 256's guest RAM and position 259's delivery, so it follows positions +256, 257, and 259 upstream. Position 261 changes only position 260's KVM tests, which it builds for +musl, for which the `libc` crate defines no `SIGEV_THREAD_ID`, so it folds into position 260. + ## PR summary Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0