Skip to content

openvmm: run restored VM time from the instant the TSCs resume - #1498

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1371-restore-vm-time-skew
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-1371-restore-vm-time-skew

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1371

What and why

After a time ABI restore, the guest's TSC led VM time, the clock of OpenVMM's
emulated device timers, by the time from the TSCs' restart to the start of the
state units, so every restore added that skew between the guest's TSC and its
device clocks, as #1371 reports. A temporary probe that logged TSC/F minus VM
time at the restore anchor and once the units started measured it in the
snapshot-core and restore-downtime scenarios: 0.28 to 0.35 ms on KVM, 0.63
to 0.91 ms on MSHV, and 0.78 to 1.22 ms on WHP, and it grows with any delay in
the restore, such as a management client that holds a restore paused.

Root cause: LoadedVm::time_abi_restore sets every processor's TSC at the
restore anchor and advances stopped VM time by the downtime to that anchor, but
VM time stayed stopped until VmTimeKeeper::start anchored it at a later
Instant::now(), after the LAPIC and VM-time advances, the rest of the
restore, and the start of the state units. The TSCs run from the anchor on KVM,
whose common offset sets them, and, on MSHV and WHP, which write them with
partition time frozen, from the instant at which the hypervisor resumes
partition time inside the set, 48 to 430 µs after the anchor. The capture
compensates for its own window, from the VM-time cut to the capture anchor, but
nothing compensated for the restore's.

microsoft/openvmm#5052 runs restored VM time from the instant at which the TSCs
run, the issue's first option:

  • virt: report when the synchronized TSC set lets the TSCs run adds that
    instant to the synchronized TSC set's report: the anchor on KVM, and on MSHV
    and WHP the TSCs' restart inside the call that resumes partition time, which
    a read of VP 0's TSC against host time after the call places, across a wrap
    of the TSC's 64 bits, between host samples taken around the call.
  • openvmm: run restored VM time from the instant the TSCs resume has a state
    unit's time advance carry that instant beside its duration, and
    VmTimeKeeper's next start run VM time from it rather than from the start,
    unless a restore or reset comes first. The first start consumes it, so a
    later start, including the rollback of a failed start, runs VM time from
    itself, as before, and no start runs VM time from an instant after itself.
    The restore converts the instant from the time ABI's host monotonic clock to
    VM time's clock with the tightest of up to 16 back-to-back pairings.

Commits:

  1. openvmm: pin restored VM time that runs from the TSCs' restart moves the
    gitlink, and nothing else, from 3cd9f149b, the head of
    feature/microvm/dev, to 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the
    head of [DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume openvmm#5052, two commits on it. The control-session
    vectors, source excerpts, contract versions, and CPU profiles that nvx-rs
    copies from OpenVMM match the revision's.
  2. doc: map VM time that restarts with the TSCs in the OpenVMM roadmap maps
    positions 254 and 255, 124 commits past the mapped tip, as not assigned to
    a PR; the remarks record the earlier positions whose code they change.
  3. doc: say that restored VM time runs from the TSCs' restart says, in 3.2.8
    of the architecture guide and steps 17 and 18 of the restore protocol, from
    which instant guest time and VM time run after a restore.

Decisions:

  • Run VM time from that instant rather than advance it again at the start.
    The issue's second option would pair the advanced time with a later host
    instant. Running from the earlier instant keeps the keeper's start protocol,
    and the device timers due since then fire once VM time starts. The third
    option, documenting the skew, would accept an error that grows without
    bound while a client holds a restore paused.
  • The TSCs' restart, not the anchor. On MSHV and WHP, the TSCs hold the
    target until partition time resumes, so VM time run from the anchor would
    lead them by that interval; OpenVMM's restore test moves by -10.5 ms with
    the anchor after a 10 ms frozen write, and within 6 µs with the restart.
  • No version. No saved state, contract, format, or version changes; the
    instant travels only in the worker's state unit requests.
  • Documentation. 3.2.8 and the restore protocol change only where they
    say from which instant guest time and VM time run, and where they said that
    guest time runs from the anchor on every backend, which MSHV's and WHP's
    frozen writes contradict. The new text claims only that the time that a
    restore takes to release the processors no longer shifts VM time relative
    to the TSC. The claims that the defect below contradicts are left to it.

A separate defect that this pull request exposes but does not fix

On MSHV and WHP, VM time still leads the TSC after a restore, by a capture-side
defect that predates this change. Their stopped partitions freeze the TSCs:
the partition unit's stop freezes partition time before the VM-time cut, and
the TSCs stay frozen through the capture anchor. The restore nevertheless
advances VM time and the RTC by D + Q, as if the TSCs had run through Q,
the interval from the VM-time cut to the anchor, as 3.2.8 assumes. VM time
therefore leads the TSC by the time that they stayed frozen before the capture
anchor. Before this change, the restore window, in which the TSCs ran while VM
time stayed stopped, partly offset that lead, so the net lead on MSHV and WHP
grows with this change. In the probe's snapshot-core runs, measured against
the relation at the capture:

Backend Lead at the restore anchor, before / after Net VM-time lead at the units' start, before → after
KVM none / none -0.29 to -0.30 ms (the TSC led) → 0.12 to 0.14 µs
MSHV 1.27 / 1.22 ms 0.64 to 0.65 ms → 1.22 ms
WHP 1.68 / 1.29 ms 0.84 to 0.89 ms → 1.28 to 1.29 ms

3.2.8's statements that the guest TSC "kept running through Q" and that no
backend stops guest time for a capture do not hold on MSHV and WHP. This pull
request leaves them, and the capture path, to a separate issue.

Validation

Head submitted to CI: 83198281c358ef05cc33988d05f82e1537f5be87, on
b9948459ce5, the head of dev.

OpenVMM, Rust 1.97.1, at 90f87319d, for virt, virt_kvm, virt_mshv,
virt_whp (Windows), vmcore, vmm_core, state_unit, vmotherboard, and
openvmm_core:

  • Windows: cargo clippy --all-targets and cargo doc --no-deps, without
    warnings from the packages; cargo nextest run --profile agent, 404 passed
    and 19 skipped; cargo xtask fmt --fix, clean.
  • Linux (nvx-kvm-test and nvx-mshv-test): cargo clippy --all-targets and
    cargo doc --no-deps, without warnings; cargo nextest run --profile agent, 484 passed and 26 skipped on each; the hardware tests, virt_kvm's
    12 and virt_mshv's 13, passed, the latter placing the TSCs' restart within
    72 ns of the reported instant.
  • Without the keeper's change, restore_keeps_the_relation_of_vm_time_to_the_tsc
    fails by 31.8 ms and test_advance_runs_from_the_resumed_instant fails; with
    the anchor in place of the restart, the former fails by -10.5 ms; with a
    saturating TSC distance, the wrap case of
    a_tsc_restart_is_where_the_read_places_it_inside_the_resume fails.

NVX, at this head:

  • python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run, on
    Windows: passed, with the 4 vector and 1 CPU-profile comparisons passing
    against 90f87319d.
  • python scripts/openvmm_pin.py source 90f87319d30d8c35bc3526f12fc7b6529c882cf0 --pr 5052:
    a good signature, the head of [DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume openvmm#5052, based on
    feature/microvm/dev. check-roadmap (255 positions, 264 references) and
    check-branch --topic: passed. python scripts/nvx.py verify: passed.

End to end, on the dedicated test VMs, all of host type virtual-machine.
"Before" is OpenVMM fb4f34126, NVX's pin when the work started; the probe's
"after" is this change on it, before review's wrap fix and wording:

Profile Backend Before After
nvx-kvm-test KVM TSC/F minus VM time moved +0.28 to +0.35 ms from the restore anchor to the units' start +0.14 to +0.37 µs. With OpenVMM built from 90f87319d, snapshot-core, restore-processors, and time-abi-conformance pass
nvx-mshv-test MSHV +0.63 to +0.91 ms -0.10 to +0.09 µs. With OpenVMM built from 90f87319d, the same three scenarios pass
nvx-whp-test WHP +0.78 to +1.22 ms -0.71 to +4.45 µs

With this change on fb4f34126, the tree of review's wrap fix but for a
rewording of comments and Guide text, snapshot-core, restore-downtime,
snapshot-tiers, smp-snapshot, snapshot-rollback, restore-processors,
and time-abi-conformance passed on all three test VMs, before #516 extended
restore-downtime with a simulated year. The runs with 90f87319d used the
artifacts of v0.1.0-dev.c52fc3f2fd33, whose guest predates #516, so they left
out restore-downtime.

Left to CI: NVX CLI / Linux and NVX CLI / Windows, since this change
touches no script; the microVM shards, among them restore-downtime with
#516's simulated year and guest, which CI builds; the WHP scenarios at this
pin; debug-kernel jobs; platforms and packages; the performance gate; and
OpenVMM CI of the pinned revision, which needs OpenVMM's CI of the pin.

CI run 43116959322
on 83198281c35 passed all of these, with the performance gate. In it,
restore-downtime restored each of its snapshots, with #516's guest, after
simulated downtimes of 31,535,802 to 31,535,835 s on KVM, MSHV, and WHP, at 1
and 8 vCPUs, from both downtime sources: the expired timer fired from 1.86 s
before to 0.03 s after the clock step, and the frozen workload gained no CPU
time. Its OpenVMM CI of the pinned revision found OpenVMM's six required
jobs passed for 90f87319d in microsoft/openvmm run
43115771162,
whose SEV-SNP VMM tests, which NVX does not require, were still queued.

Changes to existing tests

None: python scripts/check_test_changes.py --base origin/dev lists no change
to existing tests. OpenVMM's changed tests are in microsoft/openvmm#5052.

OpenVMM pin and landing order

After a time ABI restore, the guest's TSC led VM time, the clock of
OpenVMM's emulated device timers, by the time from the restart of the
TSCs to the start of the state units (#1371). The restore set the TSCs
at the anchor, from which they ran on KVM, and on MSHV and WHP once
partition time resumed, and advanced stopped VM time by the downtime
to that anchor, but VM time stayed stopped until the units started:
0.28 to 0.35 ms later on KVM, 0.63 to 0.91 ms on MSHV, and 0.78 to
1.22 ms on WHP in the microVM restore scenarios, and as long as a
management client holds a restore paused.

Pin 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the head of
microsoft/openvmm#5052, two commits on 3cd9f149b, the head of
feature/microvm/dev:

- bce9b3673 virt: report when the synchronized TSC set lets the TSCs
  run
- 90f87319d openvmm: run restored VM time from the instant the TSCs
  resume

The revision has the synchronized TSC set report the host instant from
which the TSCs run: the restore anchor on KVM, and on MSHV and WHP,
which hold the TSCs at the target until partition time resumes, the
instant at which the hypervisor restarted them, which a read of VP 0's
TSC places inside the call that resumes partition time. The restore's
time advance carries that instant to the state units, and VM time runs
from it once the units start, so the time that the restore takes to
start the units no longer shifts the emulated devices' timers relative
to the TSC. No saved state, contract, or format changes.

Evidence: a temporary probe logged TSC/F minus VM time at the restore
anchor and once the units started, in the snapshot-core and
restore-downtime scenarios, which pass. On nested KVM, MSHV, and WHP
hosts, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and
+0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 us, -0.10 to
+0.09 us, and -0.71 to +4.45 us with the fix. OpenVMM's new restore
test moves the relation by 31.0 ms without the fix; with it, the
package-scoped tests, Clippy, and rustdoc pass on Windows and on the
KVM and MSHV hosts, as do the KVM and MSHV hardware tests.

On MSHV and WHP, VM time still leads the TSC after a restore by the
time that the stopped partition held the TSCs frozen before the
capture anchor, 1.22 to 1.68 ms in the snapshot-core captures, because
the restore advances VM time across the capture's cut-to-anchor
interval as if the TSCs had run through it. The restore window had
partly offset that lead, so the net lead grows from 0.64 to 0.89 ms
before the fix to 1.22 to 1.29 ms with it. That capture-side defect
predates this change and is left to a separate issue.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Maps NVX's new pin, 90f87319d, the head of microsoft/openvmm#5052, at
position 255, which with position 254 follows position 253, 3cd9f149b,
the head of feature/microvm/dev, which microsoft/openvmm#5046 merged.
The pin is 124 commits past the mapped tip.

Position 254 has the synchronized TSC set of the time ABI backends of
positions 133, 134, and 140 report the host instant from which the
TSCs run, and position 255 has the time ABI restore of position 139
pass it to the time advance that PR10's position 019 adds, from which
VmTimeKeeper's next start runs VM time. Neither fixes a commit that a
roadmap PR maps, so both stay unassigned; the remarks record that
position 255 changes PR10's time advance, its chipset state unit, and
the tests of positions 241, 244, and 245. Replaying them on upstream
main conflicts in 15 of their 16 files, all but the Guide's
timekeeping page: upstream main lacks the time ABI of positions
132-140 and the time advance of position 019, so they follow PR10 and
the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The time design said that guest time runs from the restore anchor on
every backend, and left VM time to the start of the state units. With
the pinned OpenVMM revision, VM time runs from the instant from which
the TSCs run, the restore anchor, or, on MSHV and WHP, which write the
TSCs with partition time frozen, the instant at which partition time
resumes, rather than from the later start of the units (#1371), so
the time that the restore takes to release the processors no longer
shifts VM time relative to the TSC.

Section 3.2.8 and steps 17 and 18 of the restore protocol now say so.
They leave alone the claims that the TSC runs through the capture's
cut-to-anchor interval and that no backend stops guest time for a
capture, which MSHV and WHP contradict by freezing the TSCs of a
stopped partition: that capture-side defect is left to a separate
issue.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-hypervisor clock changes require maintainer assessment because exact-pin WHP and simulated-year restore qualification remain outstanding, while a documented capture defect increases MSHV/WHP time skew.

No open issues 🎉

What changed in this PR

This PR addresses #1371 by advancing the OpenVMM pin to a revision that starts restored VM time from the instant the guest TSCs resume, preventing restore delays from adding clock skew.

Changes:

  • Records the new OpenVMM revision and maps commits 254–255 in the upstream roadmap.
  • Updates the architecture and restore-protocol documentation to explain backend-specific TSC restart timing and VM-time alignment.
File Description
doc/​openvmm-upstream-roadmap.md Records the new pin, commit mappings, and upstream dependencies.
doc/​design/​5.5-restore-protocol.md Clarifies when TSCs resume and how VM time follows that instant.
doc/​design/​3.2-clock-and-time.md Explains why restore delays no longer shift VM time relative to the TSC.

Review effort: Balanced

@ppenna

Copy link
Copy Markdown
Contributor Author

On the note in Copilot's review of 83198281c35 that WHP qualification at the exact pin and the simulated-year restore remained outstanding, and that the documented capture-side defect increases the skew on MSHV and WHP:

  • The exact pin, on WHP and the other backends. CI run 43116959322, on this head, built OpenVMM from 90f87319d and passed the microVM shards, guests and sandboxes, platform, and package jobs of KVM, MSHV, and WHP, WHP's runtime adapter, MSHV's debug-kernel job, and the performance gate. Its OpenVMM CI of the pinned revision found OpenVMM's six required jobs passed for the pin.
  • The simulated year. In that run, restore-downtime, with openvmm: extend maximum snapshot downtime to one year #516's guest, restored each of its snapshots after simulated downtimes of 31,535,802 to 31,535,835 s on all three backends, at 1 and 8 vCPUs, from both downtime sources, and passed: the expired timer fired from 1.86 s before to 0.03 s after the clock step, the frozen workload gained no CPU time, and the warp probes and the time ABI's restore checks passed. This change does not alter how far a restore advances VM time, only the host instant from which VM time then runs.
  • The capture-side defect. It predates this change and does not depend on it. MSHV and WHP freeze the TSCs of a stopped partition before the VM-time cut, so the restore's advance of VM time by D + Q leads the TSC by the time that they stayed frozen before the capture anchor, 1.22 to 1.68 ms in the probe's snapshot-core captures. Before this change, the restore window offset part of that lead by accident, at the price of a skew that grew with every delay in the restore, without bound while a client held a restore paused. With it, the lead left is the capture's, fixed for each snapshot. Closing it needs a change to the capture path, which openvmm: after a microVM restore, the TSC leads VM time by the restore's length #1371 does not cover, so it is left to a separate issue, as the description says.

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 12, 2026
Merged via the queue into dev with commit fc59c90 Nov 12, 2026
48 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1371-restore-vm-time-skew branch November 12, 2026 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: after a microVM restore, the TSC leads VM time by the restore's length

2 participants