Repository navigation
openvmm: run restored VM time from the instant the TSCs resume - #1498
Merged
Pedro Henrique Penna (ppenna) merged 3 commits intoNov 12, 2026
Merged
Conversation
After a time ABI restore, the guest's TSC led VM time, the clock of OpenVMM's emulated device timers, by the time from the restart of the TSCs to the start of the state units (#1371). The restore set the TSCs at the anchor, from which they ran on KVM, and on MSHV and WHP once partition time resumed, and advanced stopped VM time by the downtime to that anchor, but VM time stayed stopped until the units started: 0.28 to 0.35 ms later on KVM, 0.63 to 0.91 ms on MSHV, and 0.78 to 1.22 ms on WHP in the microVM restore scenarios, and as long as a management client holds a restore paused. Pin 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the head of microsoft/openvmm#5052, two commits on 3cd9f149b, the head of feature/microvm/dev: - bce9b3673 virt: report when the synchronized TSC set lets the TSCs run - 90f87319d openvmm: run restored VM time from the instant the TSCs resume The revision has the synchronized TSC set report the host instant from which the TSCs run: the restore anchor on KVM, and on MSHV and WHP, which hold the TSCs at the target until partition time resumes, the instant at which the hypervisor restarted them, which a read of VP 0's TSC places inside the call that resumes partition time. The restore's time advance carries that instant to the state units, and VM time runs from it once the units start, so the time that the restore takes to start the units no longer shifts the emulated devices' timers relative to the TSC. No saved state, contract, or format changes. Evidence: a temporary probe logged TSC/F minus VM time at the restore anchor and once the units started, in the snapshot-core and restore-downtime scenarios, which pass. On nested KVM, MSHV, and WHP hosts, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and +0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 us, -0.10 to +0.09 us, and -0.71 to +4.45 us with the fix. OpenVMM's new restore test moves the relation by 31.0 ms without the fix; with it, the package-scoped tests, Clippy, and rustdoc pass on Windows and on the KVM and MSHV hosts, as do the KVM and MSHV hardware tests. On MSHV and WHP, VM time still leads the TSC after a restore by the time that the stopped partition held the TSCs frozen before the capture anchor, 1.22 to 1.68 ms in the snapshot-core captures, because the restore advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before the fix to 1.22 to 1.29 ms with it. That capture-side defect predates this change and is left to a separate issue. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maps NVX's new pin, 90f87319d, the head of microsoft/openvmm#5052, at position 255, which with position 254 follows position 253, 3cd9f149b, the head of feature/microvm/dev, which microsoft/openvmm#5046 merged. The pin is 124 commits past the mapped tip. Position 254 has the synchronized TSC set of the time ABI backends of positions 133, 134, and 140 report the host instant from which the TSCs run, and position 255 has the time ABI restore of position 139 pass it to the time advance that PR10's position 019 adds, from which VmTimeKeeper's next start runs VM time. Neither fixes a commit that a roadmap PR maps, so both stay unassigned; the remarks record that position 255 changes PR10's time advance, its chipset state unit, and the tests of positions 241, 244, and 245. Replaying them on upstream main conflicts in 15 of their 16 files, all but the Guide's timekeeping page: upstream main lacks the time ABI of positions 132-140 and the time advance of position 019, so they follow PR10 and the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The time design said that guest time runs from the restore anchor on every backend, and left VM time to the start of the state units. With the pinned OpenVMM revision, VM time runs from the instant from which the TSCs run, the restore anchor, or, on MSHV and WHP, which write the TSCs with partition time frozen, the instant at which partition time resumes, rather than from the later start of the units (#1371), so the time that the restore takes to release the processors no longer shifts VM time relative to the TSC. Section 3.2.8 and steps 17 and 18 of the restore protocol now say so. They leave alone the claims that the TSC runs through the capture's cut-to-anchor interval and that no backend stops guest time for a capture, which MSHV and WHP contradict by freezing the TSCs of a stopped partition: that capture-side defect is left to a separate issue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-hypervisor clock changes require maintainer assessment because exact-pin WHP and simulated-year restore qualification remain outstanding, while a documented capture defect increases MSHV/WHP time skew.
No open issues 🎉
What changed in this PR
This PR addresses #1371 by advancing the OpenVMM pin to a revision that starts restored VM time from the instant the guest TSCs resume, preventing restore delays from adding clock skew.
Changes:
- Records the new OpenVMM revision and maps commits 254–255 in the upstream roadmap.
- Updates the architecture and restore-protocol documentation to explain backend-specific TSC restart timing and VM-time alignment.
| File | Description |
|---|---|
doc/openvmm-upstream-roadmap.md |
Records the new pin, commit mappings, and upstream dependencies. |
doc/design/5.5-restore-protocol.md |
Clarifies when TSCs resume and how VM time follows that instant. |
doc/design/3.2-clock-and-time.md |
Explains why restore delays no longer shift VM time relative to the TSC. |
Review effort: Balanced
Contributor
Author
|
On the note in Copilot's review of
|
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1371-restore-vm-time-skew
branch
November 12, 2026 00:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1371
What and why
After a time ABI restore, the guest's TSC led VM time, the clock of OpenVMM's
emulated device timers, by the time from the TSCs' restart to the start of the
state units, so every restore added that skew between the guest's TSC and its
device clocks, as #1371 reports. A temporary probe that logged TSC/F minus VM
time at the restore anchor and once the units started measured it in the
snapshot-coreandrestore-downtimescenarios: 0.28 to 0.35 ms on KVM, 0.63to 0.91 ms on MSHV, and 0.78 to 1.22 ms on WHP, and it grows with any delay in
the restore, such as a management client that holds a restore paused.
Root cause:
LoadedVm::time_abi_restoresets every processor's TSC at therestore anchor and advances stopped VM time by the downtime to that anchor, but
VM time stayed stopped until
VmTimeKeeper::startanchored it at a laterInstant::now(), after the LAPIC and VM-time advances, the rest of therestore, and the start of the state units. The TSCs run from the anchor on KVM,
whose common offset sets them, and, on MSHV and WHP, which write them with
partition time frozen, from the instant at which the hypervisor resumes
partition time inside the set, 48 to 430 µs after the anchor. The capture
compensates for its own window, from the VM-time cut to the capture anchor, but
nothing compensated for the restore's.
microsoft/openvmm#5052 runs restored VM time from the instant at which the TSCs
run, the issue's first option:
virt: report when the synchronized TSC set lets the TSCs runadds thatinstant to the synchronized TSC set's report: the anchor on KVM, and on MSHV
and WHP the TSCs' restart inside the call that resumes partition time, which
a read of VP 0's TSC against host time after the call places, across a wrap
of the TSC's 64 bits, between host samples taken around the call.
openvmm: run restored VM time from the instant the TSCs resumehas a stateunit's time advance carry that instant beside its duration, and
VmTimeKeeper's next start run VM time from it rather than from the start,unless a restore or reset comes first. The first start consumes it, so a
later start, including the rollback of a failed start, runs VM time from
itself, as before, and no start runs VM time from an instant after itself.
The restore converts the instant from the time ABI's host monotonic clock to
VM time's clock with the tightest of up to 16 back-to-back pairings.
Commits:
openvmm: pin restored VM time that runs from the TSCs' restartmoves thegitlink, and nothing else, from
3cd9f149b, the head offeature/microvm/dev, to90f87319d30d8c35bc3526f12fc7b6529c882cf0, thehead of [DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume openvmm#5052, two commits on it. The control-session
vectors, source excerpts, contract versions, and CPU profiles that nvx-rs
copies from OpenVMM match the revision's.
doc: map VM time that restarts with the TSCs in the OpenVMM roadmapmapspositions 254 and 255, 124 commits past the mapped tip, as not assigned to
a PR; the remarks record the earlier positions whose code they change.
doc: say that restored VM time runs from the TSCs' restartsays, in 3.2.8of the architecture guide and steps 17 and 18 of the restore protocol, from
which instant guest time and VM time run after a restore.
Decisions:
The issue's second option would pair the advanced time with a later host
instant. Running from the earlier instant keeps the keeper's start protocol,
and the device timers due since then fire once VM time starts. The third
option, documenting the skew, would accept an error that grows without
bound while a client holds a restore paused.
target until partition time resumes, so VM time run from the anchor would
lead them by that interval; OpenVMM's restore test moves by -10.5 ms with
the anchor after a 10 ms frozen write, and within 6 µs with the restart.
instant travels only in the worker's state unit requests.
say from which instant guest time and VM time run, and where they said that
guest time runs from the anchor on every backend, which MSHV's and WHP's
frozen writes contradict. The new text claims only that the time that a
restore takes to release the processors no longer shifts VM time relative
to the TSC. The claims that the defect below contradicts are left to it.
A separate defect that this pull request exposes but does not fix
On MSHV and WHP, VM time still leads the TSC after a restore, by a capture-side
defect that predates this change. Their stopped partitions freeze the TSCs:
the partition unit's stop freezes partition time before the VM-time cut, and
the TSCs stay frozen through the capture anchor. The restore nevertheless
advances VM time and the RTC by
D + Q, as if the TSCs had run throughQ,the interval from the VM-time cut to the anchor, as 3.2.8 assumes. VM time
therefore leads the TSC by the time that they stayed frozen before the capture
anchor. Before this change, the restore window, in which the TSCs ran while VM
time stayed stopped, partly offset that lead, so the net lead on MSHV and WHP
grows with this change. In the probe's
snapshot-coreruns, measured againstthe relation at the capture:
3.2.8's statements that the guest TSC "kept running through
Q" and that nobackend stops guest time for a capture do not hold on MSHV and WHP. This pull
request leaves them, and the capture path, to a separate issue.
Validation
Head submitted to CI:
83198281c358ef05cc33988d05f82e1537f5be87, onb9948459ce5, the head ofdev.OpenVMM, Rust 1.97.1, at
90f87319d, forvirt,virt_kvm,virt_mshv,virt_whp(Windows),vmcore,vmm_core,state_unit,vmotherboard, andopenvmm_core:cargo clippy --all-targetsandcargo doc --no-deps, withoutwarnings from the packages;
cargo nextest run --profile agent, 404 passedand 19 skipped;
cargo xtask fmt --fix, clean.nvx-kvm-testandnvx-mshv-test):cargo clippy --all-targetsandcargo doc --no-deps, without warnings;cargo nextest run --profile agent, 484 passed and 26 skipped on each; the hardware tests,virt_kvm's12 and
virt_mshv's 13, passed, the latter placing the TSCs' restart within72 ns of the reported instant.
restore_keeps_the_relation_of_vm_time_to_the_tscfails by 31.8 ms and
test_advance_runs_from_the_resumed_instantfails; withthe anchor in place of the restart, the former fails by -10.5 ms; with a
saturating TSC distance, the wrap case of
a_tsc_restart_is_where_the_read_places_it_inside_the_resumefails.NVX, at this head:
python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run, onWindows: passed, with the 4 vector and 1 CPU-profile comparisons passing
against
90f87319d.python scripts/openvmm_pin.py source 90f87319d30d8c35bc3526f12fc7b6529c882cf0 --pr 5052:a good signature, the head of [DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume openvmm#5052, based on
feature/microvm/dev.check-roadmap(255 positions, 264 references) andcheck-branch --topic: passed.python scripts/nvx.py verify: passed.End to end, on the dedicated test VMs, all of host type
virtual-machine."Before" is OpenVMM
fb4f34126, NVX's pin when the work started; the probe's"after" is this change on it, before review's wrap fix and wording:
nvx-kvm-test90f87319d,snapshot-core,restore-processors, andtime-abi-conformancepassnvx-mshv-test90f87319d, the same three scenarios passnvx-whp-testWith this change on
fb4f34126, the tree of review's wrap fix but for arewording of comments and Guide text,
snapshot-core,restore-downtime,snapshot-tiers,smp-snapshot,snapshot-rollback,restore-processors,and
time-abi-conformancepassed on all three test VMs, before #516 extendedrestore-downtimewith a simulated year. The runs with90f87319dused theartifacts of
v0.1.0-dev.c52fc3f2fd33, whose guest predates #516, so they leftout
restore-downtime.Left to CI:
NVX CLI / LinuxandNVX CLI / Windows, since this changetouches no script; the microVM shards, among them
restore-downtimewith#516's simulated year and guest, which CI builds; the WHP scenarios at this
pin; debug-kernel jobs; platforms and packages; the performance gate; and
OpenVMM CI of the pinned revision, which needs OpenVMM's CI of the pin.CI run 43116959322
on
83198281c35passed all of these, with the performance gate. In it,restore-downtimerestored each of its snapshots, with #516's guest, aftersimulated downtimes of 31,535,802 to 31,535,835 s on KVM, MSHV, and WHP, at 1
and 8 vCPUs, from both downtime sources: the expired timer fired from 1.86 s
before to 0.03 s after the clock step, and the frozen workload gained no CPU
time. Its
OpenVMM CI of the pinned revisionfound OpenVMM's six requiredjobs passed for
90f87319din microsoft/openvmm run43115771162,
whose SEV-SNP VMM tests, which NVX does not require, were still queued.
Changes to existing tests
None:
python scripts/check_test_changes.py --base origin/devlists no changeto existing tests. OpenVMM's changed tests are in microsoft/openvmm#5052.
OpenVMM pin and landing order
90f87319d30d8c35bc3526f12fc7b6529c882cf0, two commits on3cd9f149b, thehead of
feature/microvm/devand NVX's previous pin, which[DO NOT REVIEW] virt: accept a snapshot downtime of up to 365 days openvmm#5046 landed for openvmm: extend maximum snapshot downtime to one year #516. It has no other prerequisite.
Its patches are those that the earlier host runs used on
fb4f34126, plusreview's wrap fix and wording: its rebase onto
3cd9f149bchanged none.mapped tip.
feature/microvm/devto90f87319d30d8c35bc3526f12fc7b6529c882cf0, since OpenVMM's merge buttonsquashes and would leave the pin off the branch.