From 9eb1cb2901dfd6953125b5a1deb4924816461a0f Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 09:48:26 -0800 Subject: [PATCH 1/3] openvmm: pin LAPIC timer restores without drift or a downtime refire On KVM, a time ABI restore could give a guest one extra LAPIC timer interrupt, and fired a running one-shot timer early (#1366). KVM_SET_LAPIC fires a one-shot timer whose current count is 0 again, so a timer that expired during the downtime, whose vector the restore queues in IRR, reached a VP restored runnable with interrupts enabled twice. And the capture saved each VP's timer count after its anchor, while the restore let KVM run the saved timer from the processor restore, before the restore anchor, and then subtracted the whole downtime, so a restored timer fired early by both windows, milliseconds in the VM worker. MSHV and WHP freeze partition time over both windows. Pin 024ecee62d7c9b70456b63a5b4f94a6488f835cd, the head of microsoft/openvmm#5053, three commits on 90f87319d, the head of feature/microvm/dev: - 2a0401c43 virt_kvm: share the guest RAM of the real-mode KVM tests - 21a76f878 virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending - 024ecee62 vmm_core, virt: run a restored LAPIC timer from the time ABI anchors The KVM backend now gives KVM an initial count of 0 for an expired one-shot timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR, so that it fires once. A time ABI restore takes each VP's timer out of its saved APIC state and sets it only after the synchronized TSC set, from its count at the capture anchor, which the saved TSC dates, advanced by the downtime and by the ticks of the TSC past the set's target, so that it runs as the TSC does, as the previous pin's VM time does: from the restore anchor on KVM, and on MSHV and WHP from the resumption of partition time that ends the set. No saved-state, manifest, or time ABI format changes; a snapshot taken before this pin restores with the same correction. A one-shot timer that expired before the capture, and whose interrupt the guest took, still fires once more on KVM. KVM saves it with the same LAPIC state as a timer that expired while its vCPU was stopped and whose interrupt KVM has not injected yet, which is the common state at the capture, since the snapshot boundary stops the VPs before the quiesce. Giving KVM an initial count of 0 for such a page loses that interrupt, so the restore keeps KVM's re-fire, and only a KVM change that tells the two apart can remove the spurious one (#1390). Evidence: KVM-backed tests run a real-mode guest that counts its timer interrupts through the restore sequence, on nvx-kvm-test (Linux 6.6, nested, without APICv or the VMX preemption timer). A timer armed for 1 s and restored after a 2 s downtime with interrupts enabled fires twice without the KVM backend's initial count of 0, and once at the pin. A timer that expired while its VP was stopped is saved expired, with its vector neither pending nor in service, the same page as a timer delivered before the capture, and fires once at the pin, while an initial count of 0 for that page gives it no interrupt at all. A 100 ms timer, with 20 ms from the capture anchor to the save and 20 ms from the processor restore to the restore anchor, fires 29.3 to 29.5 ms early with the earlier restore sequence, and 161 to 333 us late at the pin in 15 runs under a load average of 9, against 176 to 234 us in 13 of 15 runs of the same timer without a snapshot, and 429 us and 3.3 ms in the others. On MSHV, whose frozen write restarts the TSCs 45 to 134 us after the restore anchor in the scenarios' restores, the LAPIC advance runs after that restart and counts each timer from its VP's TSC, so the timers run from where the previous pin runs VM time. At the pin, the smp-lapic, smp-snapshot, restore-processors, snapshot-core, snapshot-rollback, and time-abi-conformance microVM scenarios pass on KVM (nvx-kvm-test) and MSHV (nvx-mshv-test), with OpenVMM built from the pin. The restore-downtime scenario, which now simulates a year, needs guest artifacts with the 365-day bound, and is left to CI with WHP. NVX needs no other coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 90f87319d..024ecee62 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 90f87319d30d8c35bc3526f12fc7b6529c882cf0 +Subproject commit 024ecee62d7c9b70456b63a5b4f94a6488f835cd From b6cf1642f20624a48628a3247e773d1a5884c2e1 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 00:06:01 -0800 Subject: [PATCH 2/3] doc: specify how a restore runs a one-shot LAPIC timer The time ABI's timer rule (3.2.5) advanced a counting one-shot LAPIC timer by the downtime's ticks without saying from which count, or from when the restored timer runs, and the restore protocol (5.5) restored every processor with its timer running before the restore anchor. On KVM, whose timers count while the VPs are stopped, a restored timer therefore fired early, and a timer that the downtime expired fired twice (#1366). 3.2.5 now says that the count that advances is the timer's count at the capture anchor, which the saved TSC dates, since a save reads each processor's TSC after its LAPIC state, and that the restored timer runs as the TSC does: the restore runs no timer until it advances the count by the downtime's ticks and by the ticks for which the TSC has run from the target that the restore wrote to it, from the restore anchor, or, on a backend that writes the TSCs with partition time frozen, from the resumption of partition time (3.2.8). It also states what KVM's LAPIC state leaves open: a one-shot timer that expired before the capture keeps its counts and fires once more on KVM, which is due if KVM had not delivered its interrupt and spurious if the guest had taken it (#1390), and on KVM an expired timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector that is pending in IRR reads an initial count of zero after the restore, so that KVM does not fire it again. Steps 14 and 18 of the restore protocol now restore every processor with its LAPIC timer stopped, and set the timers again after the TSC readback, beside step 18's VM time, which runs from the instant from which the TSCs run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/3.2-clock-and-time.md | 17 +++++++++++++++++ doc/design/5.5-restore-protocol.md | 12 +++++++----- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 848e5a99e..036d48207 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -234,6 +234,23 @@ the LAPIC register that configures it (its LVT entry) is masked; otherwise, the count decreases by `ticks`. A queued or pending vector must reach its processor when the processor is released, even if it is halted. +The count that advances is the timer's count at the capture anchor, and the +restored timer runs as the TSC does, so that the timer keeps step with the TSC +across the snapshot. A save reads each processor's TSC after its LAPIC state, +so the saved TSC dates the saved count. The restore runs no timer until it +advances the count, by `ticks` and by the ticks for which the TSC has run from +the target that the restore wrote to it. + +A one-shot timer that expired before the capture keeps its counts. KVM's saved +LAPIC state does not show whether KVM had delivered the timer's interrupt, so +on KVM the restored timer fires once more: the interrupt is due if KVM had not +delivered it, and spurious if the guest had already taken it. KVM fires it at +once from Linux 6.3, and a full timer period later before. On KVM, an expired +timer whose unmasked LVT entry delivers a fixed interrupt to a valid vector +that is pending in the interrupt request register (IRR), which already holds +the interrupt, as after a downtime that expires the timer, reads an initial +count of zero after the restore, so that KVM does not fire it again. + ## 3.2.6 Backend Obligations Each backend implements the same obligations behind one VMM interface. The diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index e800a04f7..7db5452a0 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -62,7 +62,8 @@ In the worker, with every processor stopped, restore: 13. Reads the native TSC and LAPIC rates and applies the [rate policy](3.2-clock-and-time.md#324-rates). 14. Restores every state unit while stopped: VM time, chipset and virtio - devices, the time platform, the partition, and every processor. + devices, the time platform, the partition, and every processor, with its + LAPIC timer stopped. 15. Freezes the set of instantiated processors. 16. Takes the restore anchor, computes `D`, and writes the same target to every instantiated processor's TSC at one host instant, never applying @@ -75,10 +76,11 @@ In the worker, with every processor stopped, restore: 17. Reads back every processor's TSC, after which guest time runs from the restore anchor, or, on a backend that writes the TSCs with partition time frozen, from the instant at which partition time resumes. -18. Advances every counting one-shot LAPIC timer by `D`, and VM time and the - RTC by `D + Q`, while the idle PIT catches up from its saved cursor. VM - time runs from the instant from which the TSCs run, once the state units - start. +18. Sets every one-shot LAPIC timer again, advancing a counting one from its + count at the capture anchor by `D` and by the ticks for which the TSC has + run from the target, and advances VM time and the RTC by `D + Q`, while + the idle PIT catches up from its saved cursor. VM time runs from the + instant from which the TSCs run, once the state units start. 19. Seals the time fields of the restore packet. 20. Starts the state units, with host input gated when the restore requires an acknowledgement, and publishes the readiness event. A device that From a9ada35d79b0eccb059e5d2698fbdb28d44f8fce Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 16:35:19 -0800 Subject: [PATCH 3/3] doc: map the LAPIC timer restore fixes in the OpenVMM roadmap Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at position 258, which with positions 256 and 257, 2a0401c43 and 21a76f878, follows position 255, 90f87319d, the head of feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is 127 commits past the mapped tip. They change how a restore runs a one-shot LAPIC timer and change nothing that a mapped commit adds, so they stay unassigned. Position 256 moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which upstream main has, into a test module that position 257's tests share. Position 257 changes the KVM backend's set_apic, which upstream main has, and the Guide's timekeeping reference, which position 255 also changes. Position 258 changes the time ABI restore of the LAPIC timers that position 139 adds, which positions 146, 147, 155, 158, and 255 also change, and sets each timer again after the synchronized TSC set, so that it runs as the TSCs do, as position 255's VM time does. Replaying position 256 on upstream main, ba105dd7f, conflicts only in virt_kvm's x86-64 module list, beside the modules of positions 133 and 138. Position 257 applies there without conflicts, but its KVM tests use position 256's guest RAM, position 139's timer advance, the TSC helpers of positions 132 and 160, and the kvm crate's TSC accessors of PR10's position 026, so it follows them upstream. Position 258 conflicts in all 12 of its files, seven of which upstream main lacks, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 46 +++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 8 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index bf19d652e..dccf0a18e 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -10,9 +10,10 @@ summary, and commit mapping below cover positions 001-131, the PR09 fixups at positions 241-243, the PR13 fixup at position 247, the PR18 fixup at position 248, the PR20 fixup at position 249, and the PR19 fixup at position 250. -- NVX's pinned OpenVMM revision: `90f87319d30d8c35bc3526f12fc7b6529c882cf0`, position 255, the - head of microsoft/openvmm#5052, whose two commits follow position 253, `3cd9f149b`, the head of - `feature/microvm/dev`, which microsoft/openvmm#5046 merged after position 252, `9f10ce7a8`, +- NVX's pinned OpenVMM revision: `024ecee62d7c9b70456b63a5b4f94a6488f835cd`, position 258, the + head of microsoft/openvmm#5053, whose three commits follow position 255, `90f87319d`, the head + of `feature/microvm/dev`, which microsoft/openvmm#5052 merged with position 254 after position + 253, `3cd9f149b`, which microsoft/openvmm#5046 merged after position 252, `9f10ce7a8`, which microsoft/openvmm#5044 merged after position 251, `174da3a0d`, which microsoft/openvmm#5042 merged after position 250, `23136b647`, which microsoft/openvmm#5041 merged after position 249, `fb4f34126`, @@ -44,8 +45,8 @@ position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1 positions 149-220 are positions 148-219, and the tree of position 220 is the tree of - `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 124 commits past - the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-255 are not + `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 127 commits past + the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-258 are not yet assigned to PRs, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, and position 250 is a PR19 fixup; see [Commits after the mapped tip](#commits-after-the-mapped-tip). @@ -1027,7 +1028,8 @@ ba105dd7f -> clock tokens split at Linux's whitespace [252] -> 365-day maximum snapshot downtime [253] -> restored VM time run from the TSCs' restart [254-255] - -> 90f87319d (NVX pin) + -> LAPIC timer restores without drift or a downtime refire [256-258] + -> 024ecee62 (NVX pin) ``` For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main, @@ -1064,7 +1066,7 @@ helper that the case uses. ## Commits after the mapped tip -The NVX pin extends the chain past position 131 without merges. Of the 124 later commits, +The NVX pin extends the chain past position 131 without merges. Of the 127 later commits, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, position 250 is a PR19 fixup, and the others are not yet classified or assigned to PRs. @@ -1072,7 +1074,7 @@ Each of positions 132-186 combines the v2.4 commits that the table lists, as v3. and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0 -reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-255 followed the +reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-258 followed the v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions 222 and 223. Each commit has the position that it had in v5.1 through position 147, and one less from v5.1 position 149 on; see @@ -1205,6 +1207,9 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4 | 253 | `3cd9f149b` | - | - | - | - | virt: accept a snapshot downtime of up to 365 days (microsoft/openvmm#5046) | | 254 | `bce9b3673` | - | - | - | - | virt: report when the synchronized TSC set lets the TSCs run (microsoft/openvmm#5052) | | 255 | `90f87319d` | - | - | - | - | openvmm: run restored VM time from the instant the TSCs resume (microsoft/openvmm#5052) | +| 256 | `2a0401c43` | - | - | - | - | virt_kvm: share the guest RAM of the real-mode KVM tests (microsoft/openvmm#5053) | +| 257 | `21a76f878` | - | - | - | - | virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending (microsoft/openvmm#5053) | +| 258 | `024ecee62` | - | - | - | - | vmm_core, virt: run a restored LAPIC timer from the time ABI anchors (microsoft/openvmm#5053) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -1817,6 +1822,31 @@ upstream main lacks the time ABI of positions 132-140, the time advance of posit of positions 241, 244, and 245, and the snapshot guide's paragraph on the VM-time cut that positions 147, 156, and 201 write, so they follow PR10 and the time ABI series upstream. +Positions 256-258 change how a restore runs a one-shot LAPIC timer, and change nothing that a +mapped PR adds. Position 256 moves the guest RAM of the real-mode KVM tests of `virt_kvm`'s ExtINT +module, which upstream main has and position 138 also changes, into a test module that position +257's tests share. Replaying it on upstream main conflicts only in the module list of `virt_kvm`'s +x86-64 module, which PR10's positions 023 and 026, PR17's position 077, and positions 133, 138, +144, 147, 149, and 150 also change, where its test module joins the modules of positions 133 and +138; its new module and its change of the ExtINT tests apply there. + +Position 257 changes the KVM backend's `set_apic`, in a file that upstream main has and PR10's +position 026 and position 147 also change, and the Guide's timekeeping reference, which position +255 also changes. Replaying it on upstream main applies without conflicts, but its KVM tests use +position 256's guest RAM, position 139's LAPIC timer advance, position 132's TSC target, position +160's host TSC bracket, and the TSC accessors that PR10's position 026 adds to the `kvm` crate, +which upstream main lacks, so it follows them upstream. + +Position 258 changes the time ABI restore of the LAPIC timers that position 139 adds to `virt`'s +VP snapshot module, the VP set, the partition unit, and the VM worker, which positions 146, 147, +155, 158, and 255 also change, and the snapshot guide's sentence on it that position 147 adds. It +sets each timer again after the synchronized TSC set, so that the timer runs as the TSCs do, as +position 255's VM time does. It adds a LAPIC tick conversion to the rate module of position 132 +and extends position 257's KVM tests. Replaying it on upstream main conflicts in all 12 of its +files, seven of which upstream main lacks, those that PR10's positions 024 and 029, PR12's +position 036, and positions 132, 135, 139, and 256 add, so it follows the time ABI series +upstream. + ## PR summary Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0