Skip to content

[DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume - #5052

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
feature/microvm/devfrom
copilot/nvx-issue-1371-restore-vm-time-skew
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
feature/microvm/devfrom
copilot/nvx-issue-1371-restore-vm-time-skew

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Runs the VM time of a restored microVM from the instant at which the restore lets the guest's TSCs run, rather than from the later start of the state units, for microsoft/nvx#1371. After a time ABI restore, the guest's TSC led VM time, the clock of the emulated devices' timers, by the time from the TSCs' restart to the start of the state units: time_abi_restore set the TSCs and advanced stopped VM time by the downtime to the restore anchor, but VmTimeKeeper::start anchored VM time at a later Instant::now(), after the LAPIC and VM-time advances, the rest of the restore, and the start of the units, and as long after as a client held a restore paused. The capture compensates for its own window, from the VM-time cut to the capture anchor, but nothing compensated for the restore's.

Two commits on 3cd9f149b, the head of feature/microvm/dev. microsoft/nvx#1498 pins this head.

Report when the synchronized TSC set lets the TSCs run (bce9b3673)

  • Change. TscSetReport gains resumed, a host sample of the instant from which every VP's TSC runs from the target. KVM's common offset makes them run from the restore anchor itself. MSHV and WHP write the target while partition time is frozen, read it back, and resume partition time inside the set, so their TSCs hold the target until the hypervisor restarts them somewhere inside the call that resumes partition time: 48 to 430 µs after the anchor in the microVM restores measured. That call takes 14 to 26 µs on MSHV and 24 to 125 µs on WHP, and the restart falls 3 to 5 µs after its start on an idle partition and 14 to 52 µs after it in a restore, so the backends place it from VP 0's TSC, read against host time right after the call: virt::time_abi::tsc_restart takes the TSC's distance past the target, across a wrap of its 64 bits, at the native rate, back from the read, and keeps the result between host samples taken around the call. A failed read-back still takes precedence over a failure to take a sample.
  • Tests. a_tsc_restart_is_where_the_read_places_it_inside_the_resume places the restart where the read puts it, also after the TSC wraps from a target near the top of its range, and inside the call; the wrap case fails with a saturating distance. MSHV's hardware test of the backend primitives, time_abi_backend_on_hardware, checks that VP 0's TSC, read against host time after the set, ran from the target from the reported instant, within the read's width and 20 µs, and reports how long after the anchor the TSCs resume.

Run restored VM time from the instant the TSCs resume (90f87319d)

  • Change. A state unit's time advance names the host instant at which the downtime ended beside its duration: StateUnits::advance_time and StateUnit::advance_time take it, StateRequest::AdvanceTime carries both in TimeAdvance, and VmTimeKeeper::advance keeps the instant, so that the keeper's next start runs VM time from it rather than from the start, unless a restore or reset comes first. The first start consumes it, so a later start, including the rollback of a failed start, runs VM time from itself, as before, and a start never runs VM time from an instant after itself. The chipset devices that keep time of their own, the RTC's calendar, keep it on host clocks that run while they are stopped, and still take only the duration. time_abi_restore names the set's resumed instant, which it converts from the time ABI's host monotonic clock to the clock that VM time runs on with the tightest of up to 16 back-to-back pairings of both, so VM time runs from the restore anchor on KVM and, on MSHV and WHP, from where the call that resumed partition time restarted the TSCs, as the TSCs do. The restore logs that instant's distance from the anchor. The timekeeping page and the snapshot guide say so.
  • Why not advance VM time once more at the start. Running VM time from that instant keeps the keeper's start protocol: the start pairs the advanced time with an earlier host instant, and the device timers due since then fire once VM time starts.
  • No version. No saved state, contract, format, or version changes.
  • Tests. restore_keeps_the_relation_of_vm_time_to_the_tsc captures a VM whose VM time runs in a VmTimeKeeper behind state units, with a fake backend whose TSC counts host monotonic time through the capture, as KVM's does, and restores it after a downtime, starting the units 30 ms after the set: TSC/F minus VM time stays within 2 ms of its value before the capture, with the TSCs running from the anchor and after a 10 ms frozen write. It moved by at most 5.6 µs in six runs; without the keeper's change it moves by 31 ms, and with the anchor in place of resumed, by -10.5 ms after the frozen write. test_advance_runs_from_the_resumed_instant checks that the keeper runs advanced time from the instant that the advance named, a later start from itself, never from an instant after its start, and drops the instant on a restore; it fails without the keeper's change. advance_time_reaches_only_units_that_advance_time checks that a unit receives both values, and host_instant_keeps_the_tightest_clock_pairing the conversion. The state unit tests of terminal and lost units pass an instant.

Not fixed here

On MSHV and WHP, whose stopped partitions freeze the TSCs, the restore still advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it, so VM time leads the TSC after a restore by the time that they stayed frozen before the capture anchor: 1.22 to 1.68 ms in the snapshot-core captures measured. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before this change to 1.22 to 1.29 ms with it. That capture-side defect predates this change, which leaves it to a separate fix.

Validation

At this head, 90f87319d, with Rust 1.97.1, for virt, virt_kvm, virt_mshv, virt_whp (Windows only), vmcore, vmm_core, state_unit, vmotherboard, and openvmm_core:

  • Windows: cargo clippy --all-targets and cargo doc --no-deps, without warnings from the packages; cargo nextest run --profile agent, 404 passed and 19 skipped; cargo xtask fmt --fix, clean. Without the keeper's change, restore_keeps_the_relation_of_vm_time_to_the_tsc (by 31.8 ms) and test_advance_runs_from_the_resumed_instant fail.
  • Linux, on Azure Linux 3 KVM and MSHV test VMs: cargo clippy --all-targets and cargo doc --no-deps, without warnings; cargo nextest run --profile agent, 484 passed and 26 skipped on each; the hardware tests, virt_kvm's 12 and virt_mshv's 13, passed, the latter placing the TSCs' restart within 72 ns of the reported instant.
  • End to end, through NVX's microVM scenarios, with OpenVMM built from this head: snapshot-core, restore-processors, and time-abi-conformance passed on the KVM and MSHV test VMs. With this change on fb4f34126, before its rebase, which changed no patch, and a rewording of its comments and Guide text, snapshot-core, restore-downtime, snapshot-tiers, smp-snapshot, snapshot-rollback, restore-processors, and time-abi-conformance passed on the KVM, MSHV, and WHP test VMs.
  • A temporary probe logged TSC/F minus VM time at the restore anchor and once the units started, in the snapshot-core and restore-downtime scenarios. On the KVM, MSHV, and WHP test VMs, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and +0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 µs, -0.10 to +0.09 µs, and -0.71 to +4.45 µs with this change on it, before review's wrap fix and wording.

See microsoft/nvx#1498.

A time ABI restore sets every VP's TSC to one target at the restore
anchor. KVM's common offset makes every TSC run from the target at the
anchor itself, but MSHV and WHP write the target while partition time
is frozen, read it back, and only then resume partition time, so their
TSCs hold the target until the resume, 48 to 430 us after the anchor
in the microVM restores measured. The set's report named only the
anchor, so the VMM could not resume its own clocks with the TSCs
(microsoft/nvx#1371).

TscSetReport now also carries a host sample of the instant from which
the TSCs run from the target: the anchor on KVM, and on MSHV and WHP
the instant at which the hypervisor restarted them inside the call
that resumes partition time. That call takes 14 to 26 us on MSHV and
24 to 125 us on WHP, and the restart falls anywhere in it: 3 to 5 us
after its start on an idle partition, and 14 to 52 us after it in a
restore. The backends therefore place it from VP 0's TSC, read against
host time right after the call: tsc_restart takes the TSC's distance
past the target, across a wrap of its 64 bits, at the native rate,
back from the read, and keeps the result between host samples taken
around the call. A failed read-back still takes precedence over a
failure to take a sample.

Tests: tsc_restart places the restart where the read puts it, also
after the TSC wraps, inside the call. MSHV's hardware test of the
backend primitives checks that VP 0's TSC, read against host time
after the set, ran from the target from the reported instant, within
the read's width and 20 us, and reports how long after the anchor the
TSCs resume.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After a time ABI restore, the guest's TSC led VM time, the clock of
the emulated devices' timers, by the time from the restart of the TSCs
to the start of the state units (microsoft/nvx#1371). time_abi_restore
sets the TSCs at the restore anchor, from which they run on KVM, and on
MSHV and WHP once partition time resumes, and advances stopped VM time
by the downtime from the snapshot's VM-time cut to that anchor, but VM
time stayed stopped until VmTimeKeeper::start anchored it at a later
Instant::now(), after the LAPIC and VM-time advances, the rest of the
restore, and the start of the units: milliseconds in the worker, and
as long as a client waits to resume a restore that it holds paused.
The capture compensates for its own window, from the cut to the
capture anchor, but nothing compensated for the restore's.

A state unit's time advance now names the host instant at which the
downtime ended beside its duration: StateUnits::advance_time and
StateUnit::advance_time take it, and VmTimeKeeper::advance keeps it,
so that the keeper's next start runs VM time from that instant rather
than from the start, unless a restore or reset comes first. The
chipset devices that keep time of their own, the RTC's calendar, keep
it on host clocks that run while they are stopped, and still take
only the duration. time_abi_restore names the instant at which the
synchronized TSC set resumed the TSCs, which it converts from the time
ABI's host monotonic clock to the clock that VM time runs on with the
tightest of up to 16 pairs of readings of both, taken back to back, so
VM time runs from the anchor on KVM and, on MSHV and WHP, from where
the call that resumed partition time restarted the TSCs, as the TSCs
do. The restore logs that instant's distance from the anchor.

Running VM time from that instant, rather than advancing it once more
at the start, keeps the keeper's start protocol: the start pairs the
advanced time with an earlier host instant, and the device timers due
since then fire once VM time starts. The first start consumes the
instant, so a start after a stop, including the rollback of a failed
start, runs VM time from itself, as before: MSHV and WHP freeze the
TSCs while the partition is stopped, and KVM does not. No saved state,
contract, or format changes.

Tests: a fake backend whose TSC counts host monotonic time, as KVM's
does, captures a VM whose VM time runs in a VmTimeKeeper behind state
units, and restores it after a downtime, starting the units 30 ms
after the set: TSC/F minus VM time stays within 2 ms of its value
before the capture, with the TSCs running from the anchor and after a
10 ms frozen write. It moved by at most 5.6 us in six runs; without
the keeper's fix it moves by 31.0 ms, and with the anchor in place of
the TSCs' resumption, by -10.5 ms after the frozen write. The keeper
runs advanced time from the instant that the advance named, a later
start from itself, never from an instant after its start, and drops
the instant on a restore; a state unit's advance carries both values;
the conversion keeps the tightest pairing of the two clocks.

In the microVM snapshot-core and restore-downtime scenarios on nested
KVM, MSHV, and WHP hosts, TSC/F minus VM time moved from the restore
anchor to the start of the units by 0.28 to 0.35 ms, 0.63 to 0.91 ms,
and 0.78 to 1.22 ms before the fix, and by at most 0.4 us, 0.1 us, and
4.5 us with it.

On MSHV and WHP, whose stopped partitions freeze the TSCs, the restore
still advances VM time across the capture's cut-to-anchor interval as
if the TSCs had run through it, so VM time leads the TSC after a
restore by the time that they stayed frozen before the capture anchor:
1.22 to 1.68 ms in the snapshot-core captures measured. The restore
window had partly offset that lead, so the net lead grows from 0.64 to
0.89 ms before the fix to 1.22 to 1.29 ms with it. That capture-side
defect predates this change, which leaves it to a separate fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code labels Nov 11, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@ppenna
Pedro Henrique Penna (ppenna) merged commit 90f8731 into feature/microvm/dev Nov 12, 2026
73 of 74 checks passed
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 12, 2026
After a time ABI restore, the guest's TSC led VM time, the clock of
OpenVMM's emulated device timers, by the time from the restart of the
TSCs to the start of the state units (microsoft#1371). The restore set the TSCs
at the anchor, from which they ran on KVM, and on MSHV and WHP once
partition time resumed, and advanced stopped VM time by the downtime
to that anchor, but VM time stayed stopped until the units started:
0.28 to 0.35 ms later on KVM, 0.63 to 0.91 ms on MSHV, and 0.78 to
1.22 ms on WHP in the microVM restore scenarios, and as long as a
management client holds a restore paused.

Pin 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the head of
microsoft/openvmm#5052, two commits on 3cd9f149b, the head of
feature/microvm/dev:

- bce9b3673 virt: report when the synchronized TSC set lets the TSCs
  run
- 90f87319d openvmm: run restored VM time from the instant the TSCs
  resume

The revision has the synchronized TSC set report the host instant from
which the TSCs run: the restore anchor on KVM, and on MSHV and WHP,
which hold the TSCs at the target until partition time resumes, the
instant at which the hypervisor restarted them, which a read of VP 0's
TSC places inside the call that resumes partition time. The restore's
time advance carries that instant to the state units, and VM time runs
from it once the units start, so the time that the restore takes to
start the units no longer shifts the emulated devices' timers relative
to the TSC. No saved state, contract, or format changes.

Evidence: a temporary probe logged TSC/F minus VM time at the restore
anchor and once the units started, in the snapshot-core and
restore-downtime scenarios, which pass. On nested KVM, MSHV, and WHP
hosts, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and
+0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 us, -0.10 to
+0.09 us, and -0.71 to +4.45 us with the fix. OpenVMM's new restore
test moves the relation by 31.0 ms without the fix; with it, the
package-scoped tests, Clippy, and rustdoc pass on Windows and on the
KVM and MSHV hosts, as do the KVM and MSHV hardware tests.

On MSHV and WHP, VM time still leads the TSC after a restore by the
time that the stopped partition held the TSCs frozen before the
capture anchor, 1.22 to 1.68 ms in the snapshot-core captures, because
the restore advances VM time across the capture's cut-to-anchor
interval as if the TSCs had run through it. The restore window had
partly offset that lead, so the net lead grows from 0.64 to 0.89 ms
before the fix to 1.22 to 1.29 ms with it. That capture-side defect
predates this change and is left to a separate issue.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 12, 2026
Maps NVX's new pin, 90f87319d, the head of microsoft/openvmm#5052, at
position 255, which with position 254 follows position 253, 3cd9f149b,
the head of feature/microvm/dev, which microsoft/openvmm#5046 merged.
The pin is 124 commits past the mapped tip.

Position 254 has the synchronized TSC set of the time ABI backends of
positions 133, 134, and 140 report the host instant from which the
TSCs run, and position 255 has the time ABI restore of position 139
pass it to the time advance that PR10's position 019 adds, from which
VmTimeKeeper's next start runs VM time. Neither fixes a commit that a
roadmap PR maps, so both stay unassigned; the remarks record that
position 255 changes PR10's time advance, its chipset state unit, and
the tests of positions 241, 244, and 245. Replaying them on upstream
main conflicts in 15 of their 16 files, all but the Guide's
timekeeping page: upstream main lacks the time ABI of positions
132-140 and the time advance of position 019, so they follow PR10 and
the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali pushed a commit to syeedali/nvx that referenced this pull request Nov 12, 2026
Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at
position 258, which with positions 256 and 257, 2a0401c43 and
21a76f878, follows position 255, 90f87319d, the head of
feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is
127 commits past the mapped tip.

They change how a restore runs a one-shot LAPIC timer and change
nothing that a mapped commit adds, so they stay unassigned. Position 256
moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which
upstream main has, into a test module that position 257's tests share.
Position 257 changes the KVM backend's set_apic, which upstream main
has, and the Guide's timekeeping reference, which position 255 also
changes. Position 258 changes the time ABI restore of the LAPIC timers
that position 139 adds, which positions 146, 147, 155, 158, and 255
also change, and sets each timer again after the synchronized TSC set,
so that it runs as the TSCs do, as position 255's VM time does.

Replaying position 256 on upstream main, ba105dd7f, conflicts only in
virt_kvm's x86-64 module list, beside the modules of positions 133 and
138. Position 257 applies there without conflicts, but its KVM tests
use position 256's guest RAM, position 139's timer advance, the TSC
helpers of positions 132 and 160, and the kvm crate's TSC accessors of
PR10's position 026, so it follows them upstream. Position 258 conflicts
in all 12 of its files, seven of which upstream main lacks, so it
follows the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Guide release-nvx Targets a branch under feature/microvm/. unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant