From f737fc9141fed8da9c07c13c18e9f237072cc29c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 14 Nov 2026 08:42:59 -0800 Subject: [PATCH 1/3] openvmm: pin the restore's check of the APIC timer's delivery mode Restoring a VP whose saved APIC page held an unmasked LVT timer entry with a delivery mode other than Fixed succeeded on KVM, and KVM then fired the timer in that mode once it had expired, as a one-shot timer whose current count is 0 has: inside KVM_SET_LAPIC when APICv is active, and at the next KVM_RUN otherwise. The restored VP took an INIT, a startup IPI, an NMI, or an SMI, or woke from halt, none of which the snapshot holds (#1365). Pin 30b7554069182a5262099f902c4a203d54c8bb88, the head of microsoft/openvmm#5102, one commit on b723cbed3, the head of feature/microvm/dev: - 30b755406 virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed set_apic now rejects such a page with KvmError::InvalidState as its first step, before it forgets the timer that the VP stopped with or changes the APIC base or the page, so the restore fails with the apic element's StateError, and set_apic itself leaves the VP's APIC as it was. The elements that restore_all restores before the APIC, the registers among them, have run: KVM's write of CR8 can change the LAPIC's task priority. It refuses every unmasked entry with a mode other than Fixed, lowest priority and ExtINT included, and accepts a masked entry. A guest cannot write the entry's delivery mode, since KVM masks its writes, so a save never holds such a page, a snapshot that a VP saved restores as before, and only a crafted snapshot is refused. MSHV and WHP are unchanged. No version is created or bumped, and no snapshot or wire format changes. Tests: on nvx-kvm-test (Intel Xeon Platinum 8573C, Linux 6.18.57, nested KVM without APICv), the revision's contract test restore_refuses_a_non_fixed_apic_timer fails at b723cbed3 with "the restore accepted an expired timer in the delivery modes [1, 2, 3, 4, 5, 6, 7]" and passes at the pin, as does its unit test an_unmasked_timer_that_is_not_fixed_is_refused. A scratch test that gave KVM_SET_LAPIC the same page showed what KVM does with it at KVM_RUN: INIT resets the VP, NMI injects an NMI, SMI kills the VM, and lowest priority is taken as a fixed interrupt. Mutants of the check fail the tests. At the pin, the 84 tests of virt_kvm and kvm pass with cargo nextest, the ignored ones included. With OpenVMM built from the pin's tree, which replaced the packaged binary (SHA-256 193d1063 to 04915a93), and the artifacts of v0.1.0-dev.9e41ebd10be8, the microVM scenarios guest-boot (1 processor), interrupt-map, smp, smp-snapshot, snapshot-core, snapshot-rollback, restore-downtime, restore-processors, snapshot-tiers, time-abi-conformance, and time-host-correction pass at 1 and 2 processors, and smp-lapic at 1, 2, 4, and 8, on nvx-kvm-test. The change is KVM-only, so nvx-mshv-test and nvx-whp-test were not run. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, the activation bound, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index b723cbed3..30b755406 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit b723cbed35f253a7b97e9e7f32f3d7d3502cae36 +Subproject commit 30b7554069182a5262099f902c4a203d54c8bb88 From ae2d56ee53755507e582f8556a5e74a03f3e1098 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 14 Nov 2026 08:42:59 -0800 Subject: [PATCH 2/3] doc: specify that a KVM restore rejects a non-fixed LAPIC timer 3.2.5 specified how a restore runs a one-shot LAPIC timer, and what KVM's LAPIC state write does to one that has expired, but not that KVM delivers the expiry of an unmasked timer in the delivery mode of its LVT entry, which a saved state that no capture holds can set to something other than a fixed interrupt (#1365). It now says that on KVM a restore rejects a LAPIC state whose unmasked timer entry has a delivery mode other than fixed, and that the step that restores the LAPIC state does so before it writes the processor's APIC base or LAPIC page; the steps before it, such as the registers', have run. Five of the seven other modes would give the restored processor an INIT, a startup IPI, an NMI, or an SMI, or wake it from halt, none of which the saved state holds; lowest priority and ExtINT, which KVM delivers as a fixed interrupt or ignores, are rejected too. The guest cannot write the delivery mode, so no capture holds a state with any of them. The check accepts a masked entry, which delivers nothing, and the rules above, such as the rejection of an armed periodic or TSC-deadline timer, still apply to its state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/3.2-clock-and-time.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 371208462..89f62b8ef 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -234,6 +234,18 @@ the LAPIC register that configures it (its LVT entry) is masked; otherwise, the count decreases by `ticks`. A queued or pending vector must reach its processor when the processor is released, even if it is halted. +On KVM, a restore also rejects a LAPIC state whose unmasked timer entry has a +delivery mode other than fixed, and the step that restores the LAPIC state +does so before it writes the processor's APIC base or LAPIC page. KVM delivers +the expiry of an unmasked timer in its entry's delivery mode, so five of the +seven other modes would give the restored processor an INIT, a startup IPI, an +NMI, or an SMI, or wake it from halt, none of which the saved state holds; +lowest priority and ExtINT, which KVM delivers as a fixed interrupt or +ignores, are rejected too. The guest cannot write the delivery mode, so no +capture holds a state with any of them. This check accepts a masked entry, +which delivers nothing, and the rules above, such as the rejection of an armed +periodic or TSC-deadline timer, still apply to its state. + The count that advances is the timer's count at the capture anchor, and the restored timer runs as the TSC does, so that the timer keeps step with the TSC across the snapshot. A save reads each processor's TSC after its LAPIC state, From 89343defb1aa0311e24d2bf3b0587eaef1579efc Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Sat, 14 Nov 2026 08:43:17 -0800 Subject: [PATCH 3/3] doc: map the APIC timer's checked delivery mode in the OpenVMM roadmap The NVX pin moves to 30b755406, position 282, 158 commits past the mapped tip. Position 282, the one commit of microsoft/openvmm#5102 on b723cbed3, the head of feature/microvm/dev, is not yet assigned to a PR. Position 282 makes the KVM backend's set_apic reject a LAPIC page whose unmasked timer entry has a delivery mode other than Fixed, which KVM would fire as an INIT, a startup IPI, an NMI, or an SMI, or as a wake of a halted VP, none of which the saved state holds, in code that upstream main has. It adds check_restorable_apic_timer, which set_apic calls first, a unit test, a KVM-only contract test of virt_backend_tests in a module of its own, and a paragraph and a sentence of the Guide. It changes files that PR01's position 002, PR05's positions 007 and 008, PR10's positions 022, 271, and 273, and positions 140, 248, 250, 251, 253, 279, and 281 also change, and adds to no line that a mapped PR adds. Replaying it on upstream main, 7d90c4104, conflicts in vp_state.rs, at the start of set_apic, whose timer reset position 253 adds, and from the end of the AccessVpState implementation on, which positions 250, 251, 253, and 279 add; in the Guide's KVM timekeeping section, which positions 250 and 253 extend; in the Guide's page of the contract tests, whose kvm_only sentence is position 279's; and in the contract tests' main.rs, beside the attribute that position 008 drops. So it follows positions 008, 250, 251, 253, and 279 upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 37 ++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index a13f4efe3..e465bd736 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -11,10 +11,11 @@ summary, and commit mapping below cover positions 001-124, the PR09 fixups at positions 234-236, the PR13 fixup at position 240, the PR18 fixup at position 241, the PR20 fixup at position 242, the PR19 fixup at position 243, and the PR10 fixups at positions 271 and 273. -- NVX's pinned OpenVMM revision: `b723cbed35f253a7b97e9e7f32f3d7d3502cae36`, position 281, the - head of microsoft/openvmm#5101, whose one commit follows position 280, `c5509cde2`, the head of - `feature/microvm/dev`, to which microsoft/openvmm#5100 added position 280 after position 279, - `5d2a71027`, microsoft/openvmm#5097 added position 279 after position 278, `157866605`, +- NVX's pinned OpenVMM revision: `30b7554069182a5262099f902c4a203d54c8bb88`, position 282, the + head of microsoft/openvmm#5102, whose one commit follows position 281, `b723cbed3`, the head of + `feature/microvm/dev`, to which microsoft/openvmm#5101 added position 281 after position 280, + `c5509cde2`, microsoft/openvmm#5100 added position 280 after position 279, `5d2a71027`, + microsoft/openvmm#5097 added position 279 after position 278, `157866605`, microsoft/openvmm#5095 added positions 277-278 after position 276, `7b205c0a0`, microsoft/openvmm#5094 added position 276 after position 275, `efe6bc267`, microsoft/openvmm#5087 added position 275 after position 274, `e2458a6ce`, @@ -27,9 +28,9 @@ changes upstream had landed, and added eight fixups at its tip, positions 262-269. A commit's position is its v6.0 position less the number of dropped commits before it: v6.0 positions 003-007 are positions 002-006, 010-013 are 007-010, 015-050 are 011-046, and 054-268 are - 047-261; see [Conflicts of the v6.1 rebase](#conflicts-of-the-v61-rebase). The pin is 157 + 047-261; see [Conflicts of the v6.1 rebase](#conflicts-of-the-v61-rebase). The pin is 158 commits past the mapped tip on the same first-parent chain. Positions 125-233, 237-239, - 244-270, 272, and 274-281 are not yet assigned to PRs, positions 234-236 are PR09 fixups, + 244-270, 272, and 274-282 are not yet assigned to PRs, positions 234-236 are PR09 fixups, position 240 is a PR13 fixup, position 241 is a PR18 fixup, position 242 is a PR20 fixup, position 243 is a PR19 fixup, and positions 271 and 273 are PR10 fixups; see [Commits after the mapped tip](#commits-after-the-mapped-tip). @@ -1172,7 +1173,8 @@ The physical branch is one first-parent chain: -> restored EFER's SVME checked against the guest's CPUID [279] -> SYSENTER MSRs held in the fields that name them [280] -> restored XSAVE component checked against the caps [281] - -> b723cbed3 (NVX pin) + -> restored APIC timer's delivery mode checked to be Fixed [282] + -> 30b755406 (NVX pin) ``` For upstream review, PR01-PR03, PR05, PR06, and PR14 can be rebased independently onto current @@ -1223,7 +1225,7 @@ restore and time advance set in the RTC's clock source, which overflowed near ei ## Commits after the mapped tip -The NVX pin extends the chain past position 124 without merges. Of the 157 later commits, +The NVX pin extends the chain past position 124 without merges. Of the 158 later commits, positions 234-236 are PR09 fixups, position 240 is a PR13 fixup, position 241 is a PR18 fixup, position 242 is a PR20 fixup, position 243 is a PR19 fixup, positions 262-269 are the fixups of the v6.1 rebase, positions 271 and 273 are PR10 fixups, and the others are not yet classified or @@ -1233,7 +1235,7 @@ and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that s commits. Positions 180-184 followed the v3.0 reorganization, positions 185-198 are the ABI restart of the v4.0 reorganization, positions 199-204 followed it, positions 205-211 followed the v5.0 reorganization, positions 212 and 213 followed the v5.1 rebase, positions 214-261 followed the -v6.0 reorganization, positions 262-269 are the v6.1 rebase's fixups, and positions 270-281 +v6.0 reorganization, positions 262-269 are the v6.1 rebase's fixups, and positions 270-282 followed the v6.1 rebase, so they combine none, apart from position 213, which combines v5.1 positions 222 and 223. Each commit's position is its v6.0 position less seven, the number of commits that the v6.1 rebase dropped before it, and its v6.0 position is its v5.1 position @@ -1403,10 +1405,11 @@ through 147 and one less from v5.1 position 149 on; see | 279 | `5d2a71027` | - | - | - | - | - | virt_kvm: refuse a restored EFER.SVME for a guest whose CPUID has no SVM (microsoft/openvmm#5097) | | 280 | `c5509cde2` | - | - | - | - | - | virt: hold each SYSENTER MSR in the field that names it (microsoft/openvmm#5100) | | 281 | `b723cbed3` | - | - | - | - | - | virt_kvm: refuse a restored XSAVE component outside the caps (microsoft/openvmm#5101) | +| 282 | `30b755406` | - | - | - | - | - | virt_kvm: reject a restored APIC timer whose delivery mode is not Fixed (microsoft/openvmm#5102) | The remarks that say whether commits apply to upstream main record what was found on upstream main when each commit was mapped: the v3.0 reorganization's base, `b2676222b`, for the commits -that it regrouped, and later fork bases, up to `7d90c4104` for positions 270-281, for later +that it regrouped, and later fork bases, up to `7d90c4104` for positions 270-282, for later commits. They name the positions that the commits have now, but the v6.1 rebase did not check the earlier ones again on `7d90c4104`. @@ -2278,6 +2281,20 @@ ignored test that restores each component into a real VP. It changes files that Replaying it on upstream main applies without conflicts, and `virt_kvm` type-checks there with its tests, which pass, the ignored one included, so it can go upstream on its own. +Position 282 makes the KVM backend's `set_apic` reject a LAPIC page whose unmasked timer entry has +a delivery mode other than Fixed, which KVM would fire as an INIT, a startup IPI, an NMI, or an SMI, +or as a wake of a halted VP, none of which the saved state holds, in code that upstream main has. +It adds `check_restorable_apic_timer`, which `set_apic` calls first, a unit test, a KVM-only +contract test of `virt_backend_tests` in a module of its own, and a paragraph and a sentence of the +Guide. It changes files that PR01's position 002, PR05's positions 007 and 008, PR10's positions +022, 271, and 273, and positions 140, 248, 250, 251, 253, 279, and 281 also change, and adds to no +line that a mapped PR adds. Replaying it on upstream main conflicts in `vp_state.rs`, at the start +of `set_apic`, whose timer reset position 253 adds, and from the end of the `AccessVpState` +implementation on, which positions 250, 251, 253, and 279 add; in the Guide's KVM timekeeping +section, which positions 250 and 253 extend; in the Guide's page of the contract tests, whose +`kvm_only` sentence is position 279's; and in the contract tests' `main.rs`, beside the attribute +that position 008 drops. So it follows positions 008, 250, 251, 253, and 279 upstream. + ## PR summary Each count is of v6.1 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0