From 779556c0b928a6825179013db77286768d8196bb Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 08:54:07 -0800 Subject: [PATCH 1/3] openvmm: pin restored VM time that runs from the TSCs' restart After a time ABI restore, the guest's TSC led VM time, the clock of OpenVMM's emulated device timers, by the time from the restart of the TSCs to the start of the state units (#1371). The restore set the TSCs at the anchor, from which they ran on KVM, and on MSHV and WHP once partition time resumed, and advanced stopped VM time by the downtime to that anchor, but VM time stayed stopped until the units started: 0.28 to 0.35 ms later on KVM, 0.63 to 0.91 ms on MSHV, and 0.78 to 1.22 ms on WHP in the microVM restore scenarios, and as long as a management client holds a restore paused. Pin 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the head of microsoft/openvmm#5052, two commits on 3cd9f149b, the head of feature/microvm/dev: - bce9b3673 virt: report when the synchronized TSC set lets the TSCs run - 90f87319d openvmm: run restored VM time from the instant the TSCs resume The revision has the synchronized TSC set report the host instant from which the TSCs run: the restore anchor on KVM, and on MSHV and WHP, which hold the TSCs at the target until partition time resumes, the instant at which the hypervisor restarted them, which a read of VP 0's TSC places inside the call that resumes partition time. The restore's time advance carries that instant to the state units, and VM time runs from it once the units start, so the time that the restore takes to start the units no longer shifts the emulated devices' timers relative to the TSC. No saved state, contract, or format changes. Evidence: a temporary probe logged TSC/F minus VM time at the restore anchor and once the units started, in the snapshot-core and restore-downtime scenarios, which pass. On nested KVM, MSHV, and WHP hosts, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and +0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 us, -0.10 to +0.09 us, and -0.71 to +4.45 us with the fix. OpenVMM's new restore test moves the relation by 31.0 ms without the fix; with it, the package-scoped tests, Clippy, and rustdoc pass on Windows and on the KVM and MSHV hosts, as do the KVM and MSHV hardware tests. On MSHV and WHP, VM time still leads the TSC after a restore by the time that the stopped partition held the TSCs frozen before the capture anchor, 1.22 to 1.68 ms in the snapshot-core captures, because the restore advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before the fix to 1.22 to 1.29 ms with it. That capture-side defect predates this change and is left to a separate issue. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- openvmm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openvmm b/openvmm index 3cd9f149b..90f87319d 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit 3cd9f149b89759810ff0f9ab3b96e561d7656cb5 +Subproject commit 90f87319d30d8c35bc3526f12fc7b6529c882cf0 From 16dda04c6bf72d79400079fe4f90ba45ec75130c Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 14:59:55 -0800 Subject: [PATCH 2/3] doc: map VM time that restarts with the TSCs in the OpenVMM roadmap Maps NVX's new pin, 90f87319d, the head of microsoft/openvmm#5052, at position 255, which with position 254 follows position 253, 3cd9f149b, the head of feature/microvm/dev, which microsoft/openvmm#5046 merged. The pin is 124 commits past the mapped tip. Position 254 has the synchronized TSC set of the time ABI backends of positions 133, 134, and 140 report the host instant from which the TSCs run, and position 255 has the time ABI restore of position 139 pass it to the time advance that PR10's position 019 adds, from which VmTimeKeeper's next start runs VM time. Neither fixes a commit that a roadmap PR maps, so both stay unassigned; the remarks record that position 255 changes PR10's time advance, its chipset state unit, and the tests of positions 241, 244, and 245. Replaying them on upstream main conflicts in 15 of their 16 files, all but the Guide's timekeeping page: upstream main lacks the time ABI of positions 132-140 and the time advance of position 019, so they follow PR10 and the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/openvmm-upstream-roadmap.md | 35 +++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index ce5c6b451..bf19d652e 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -10,9 +10,10 @@ summary, and commit mapping below cover positions 001-131, the PR09 fixups at positions 241-243, the PR13 fixup at position 247, the PR18 fixup at position 248, the PR20 fixup at position 249, and the PR19 fixup at position 250. -- NVX's pinned OpenVMM revision: `3cd9f149b89759810ff0f9ab3b96e561d7656cb5`, position 253, the - head of microsoft/openvmm#5046, whose commit follows position 252, `9f10ce7a8`, the head of - `feature/microvm/dev`, which microsoft/openvmm#5044 merged after position 251, `174da3a0d`, +- NVX's pinned OpenVMM revision: `90f87319d30d8c35bc3526f12fc7b6529c882cf0`, position 255, the + head of microsoft/openvmm#5052, whose two commits follow position 253, `3cd9f149b`, the head of + `feature/microvm/dev`, which microsoft/openvmm#5046 merged after position 252, `9f10ce7a8`, + which microsoft/openvmm#5044 merged after position 251, `174da3a0d`, which microsoft/openvmm#5042 merged after position 250, `23136b647`, which microsoft/openvmm#5041 merged after position 249, `fb4f34126`, which microsoft/openvmm#5038 merged after position 248, `58d3c3b1d`, @@ -43,8 +44,8 @@ position 221 into positions 035 and 036, and combined v5.1 positions 222 and 223, the two commits of microsoft/openvmm#4984, into position 220. Positions 001-147 keep their positions, v5.1 positions 149-220 are positions 148-219, and the tree of position 220 is the tree of - `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 122 commits past - the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-253 are not + `3f2ab4c2f`; see [The v6.0 reorganization](#the-v60-reorganization). The pin is 124 commits past + the mapped tip on the same first-parent chain. Positions 132-240, 244-246, and 251-255 are not yet assigned to PRs, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, and position 250 is a PR19 fixup; see [Commits after the mapped tip](#commits-after-the-mapped-tip). @@ -1025,7 +1026,8 @@ ba105dd7f -> direct-claimed scratch attached in place [251] -> clock tokens split at Linux's whitespace [252] -> 365-day maximum snapshot downtime [253] - -> 3cd9f149b (NVX pin) + -> restored VM time run from the TSCs' restart [254-255] + -> 90f87319d (NVX pin) ``` For upstream review, PR01-PR06 and PR14 can be rebased independently onto current upstream main, @@ -1062,7 +1064,7 @@ helper that the case uses. ## Commits after the mapped tip -The NVX pin extends the chain past position 131 without merges. Of the 122 later commits, +The NVX pin extends the chain past position 131 without merges. Of the 124 later commits, positions 241-243 are PR09 fixups, position 247 is a PR13 fixup, position 248 is a PR18 fixup, position 249 is a PR20 fixup, position 250 is a PR19 fixup, and the others are not yet classified or assigned to PRs. @@ -1070,7 +1072,7 @@ Each of positions 132-186 combines the v2.4 commits that the table lists, as v3. and 162-200 did, which developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 187-191 followed the v3.0 reorganization, positions 192-205 are the ABI restart of the v4.0 reorganization, positions 206-211 followed it, positions 212-218 followed the v5.0 -reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-253 followed the +reorganization, positions 219 and 220 followed the v5.1 rebase, and positions 221-255 followed the v6.0 reorganization, so they combine none, apart from position 220, which combines v5.1 positions 222 and 223. Each commit has the position that it had in v5.1 through position 147, and one less from v5.1 position 149 on; see @@ -1201,6 +1203,8 @@ at `e1ca353eb`, whose first 210 commits are those of `feature/microvm/release/v4 | 251 | `174da3a0d` | - | - | - | - | microVM: attach direct-claimed scratch in place instead of linking it (microsoft/openvmm#5042) | | 252 | `9f10ce7a8` | - | - | - | - | virt: split clock-token command lines at every Linux whitespace byte (microsoft/openvmm#5044) | | 253 | `3cd9f149b` | - | - | - | - | virt: accept a snapshot downtime of up to 365 days (microsoft/openvmm#5046) | +| 254 | `bce9b3673` | - | - | - | - | virt: report when the synchronized TSC set lets the TSCs run (microsoft/openvmm#5052) | +| 255 | `90f87319d` | - | - | - | - | openvmm: run restored VM time from the instant the TSCs resume (microsoft/openvmm#5052) | The remarks that say whether commits apply to upstream main record what the v3.0 reorganization found on its base, `b2676222b`, at the positions that the commits have now. @@ -1408,6 +1412,13 @@ position 166's link helper and the path-based opener of a given length that posi only that helper used, replaces position 166's test of the link, extends position 237's test of the openers, adds tests beside those that position 147 adds to the microVM restore module, and changes the CLI reference's sentence on direct-claimed scratch, which position 166 adds. +Position 254 has the synchronized TSC set of the time ABI backends of positions 133, 134, and 140 +report the host instant from which the TSCs run, which the time ABI restore of position 139 passes, +with position 255, to the time advance that PR10's position 019 adds: `StateUnits::advance_time`, +`StateUnit::advance_time`, and `VmTimeKeeper::advance` take that instant, from which the keeper's +next start runs VM time, and the chipset's state unit, whose time advance position 019 adds, takes +it too. Position 255 also has the tests of positions 241, 244, and 245 pass an instant, and extends +the snapshot guide's paragraph on the VM-time cut that positions 156 and 201 write. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Position 177, by contrast, changes only `disk_blob`, which upstream main and the fork base @@ -1798,6 +1809,14 @@ in `virt` and in the snapshot time module of `openvmm_helpers`, which positions Replaying it on upstream main conflicts in all five of its files: upstream main lacks the time ABI, so it follows the time ABI series upstream. +Positions 254 and 255 change the synchronized TSC set of the time ABI backends of positions 133, +134, and 140, the time ABI restore of position 139, and the time advance of PR10's position 019, +with its callers and tests. Replaying them on upstream main conflicts in 15 of their 16 files, all +but the Guide's timekeeping page, whose description of `VmTimeKeeper` upstream main has alike: +upstream main lacks the time ABI of positions 132-140, the time advance of position 019, the tests +of positions 241, 244, and 245, and the snapshot guide's paragraph on the VM-time cut that +positions 147, 156, and 201 write, so they follow PR10 and the time ABI series upstream. + ## PR summary Each count is of v6.0 commits, which include the fixups that the v3.0, v4.0, v5.0, and v6.0 From 83198281c358ef05cc33988d05f82e1537f5be87 Mon Sep 17 00:00:00 2001 From: Pedro Henrique Penna Date: Wed, 11 Nov 2026 00:20:47 -0800 Subject: [PATCH 3/3] doc: say that restored VM time runs from the TSCs' restart The time design said that guest time runs from the restore anchor on every backend, and left VM time to the start of the state units. With the pinned OpenVMM revision, VM time runs from the instant from which the TSCs run, the restore anchor, or, on MSHV and WHP, which write the TSCs with partition time frozen, the instant at which partition time resumes, rather than from the later start of the units (#1371), so the time that the restore takes to release the processors no longer shifts VM time relative to the TSC. Section 3.2.8 and steps 17 and 18 of the restore protocol now say so. They leave alone the claims that the TSC runs through the capture's cut-to-anchor interval and that no backend stops guest time for a capture, which MSHV and WHP contradict by freezing the TSCs of a stopped partition: that capture-side defect is left to a separate issue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- doc/design/3.2-clock-and-time.md | 12 ++++++++---- doc/design/5.5-restore-protocol.md | 7 +++++-- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/doc/design/3.2-clock-and-time.md b/doc/design/3.2-clock-and-time.md index 659a9c0c4..848e5a99e 100644 --- a/doc/design/3.2-clock-and-time.md +++ b/doc/design/3.2-clock-and-time.md @@ -358,10 +358,14 @@ can oops before its guest rejects the packet, which hung about one such restore in twenty in qualification. Snapshots captured before the change therefore keep their 30-day window; only a recapture gets the longer one. -The restore anchor is the instant of the synchronized TSC set, and guest time -runs from that instant on every backend; no backend defers it to the first -processor run. No backend stops guest time for a capture either, so a capture -that rolls back leaves the guest's clocks continuous, and the guest sees the +The restore anchor is the instant of the synchronized TSC set. Guest time runs +from that instant, or, on a backend that writes the TSCs with partition time +frozen, from the instant at which partition time resumes after the write, and +no backend defers it to the first processor run. VM time runs from the same +instant rather than from the later start of the state units, so the time that +the restore takes to release the processors does not shift VM time relative to +the TSC. No backend stops guest time for a capture either, so a capture that +rolls back leaves the guest's clocks continuous, and the guest sees the quiesce as elapsed time, as it would a long preemption. ## 3.2.9 Time Samples diff --git a/doc/design/5.5-restore-protocol.md b/doc/design/5.5-restore-protocol.md index e7d5544cc..e800a04f7 100644 --- a/doc/design/5.5-restore-protocol.md +++ b/doc/design/5.5-restore-protocol.md @@ -73,9 +73,12 @@ In the worker, with every processor stopped, restore: ``` 17. Reads back every processor's TSC, after which guest time runs from the - restore anchor. + restore anchor, or, on a backend that writes the TSCs with partition time + frozen, from the instant at which partition time resumes. 18. Advances every counting one-shot LAPIC timer by `D`, and VM time and the - RTC by `D + Q`, while the idle PIT catches up from its saved cursor. + RTC by `D + Q`, while the idle PIT catches up from its saved cursor. VM + time runs from the instant from which the TSCs run, once the state units + start. 19. Seals the time fields of the restore packet. 20. Starts the state units, with host input gated when the restore requires an acknowledgement, and publishes the readiness event. A device that