Repository navigation
[DO NOT REVIEW] openvmm: run restored VM time from the instant the TSCs resume - #5052
Merged
Pedro Henrique Penna (ppenna) merged 2 commits intoNov 12, 2026
Conversation
A time ABI restore sets every VP's TSC to one target at the restore anchor. KVM's common offset makes every TSC run from the target at the anchor itself, but MSHV and WHP write the target while partition time is frozen, read it back, and only then resume partition time, so their TSCs hold the target until the resume, 48 to 430 us after the anchor in the microVM restores measured. The set's report named only the anchor, so the VMM could not resume its own clocks with the TSCs (microsoft/nvx#1371). TscSetReport now also carries a host sample of the instant from which the TSCs run from the target: the anchor on KVM, and on MSHV and WHP the instant at which the hypervisor restarted them inside the call that resumes partition time. That call takes 14 to 26 us on MSHV and 24 to 125 us on WHP, and the restart falls anywhere in it: 3 to 5 us after its start on an idle partition, and 14 to 52 us after it in a restore. The backends therefore place it from VP 0's TSC, read against host time right after the call: tsc_restart takes the TSC's distance past the target, across a wrap of its 64 bits, at the native rate, back from the read, and keeps the result between host samples taken around the call. A failed read-back still takes precedence over a failure to take a sample. Tests: tsc_restart places the restart where the read puts it, also after the TSC wraps, inside the call. MSHV's hardware test of the backend primitives checks that VP 0's TSC, read against host time after the set, ran from the target from the reported instant, within the read's width and 20 us, and reports how long after the anchor the TSCs resume. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After a time ABI restore, the guest's TSC led VM time, the clock of the emulated devices' timers, by the time from the restart of the TSCs to the start of the state units (microsoft/nvx#1371). time_abi_restore sets the TSCs at the restore anchor, from which they run on KVM, and on MSHV and WHP once partition time resumes, and advances stopped VM time by the downtime from the snapshot's VM-time cut to that anchor, but VM time stayed stopped until VmTimeKeeper::start anchored it at a later Instant::now(), after the LAPIC and VM-time advances, the rest of the restore, and the start of the units: milliseconds in the worker, and as long as a client waits to resume a restore that it holds paused. The capture compensates for its own window, from the cut to the capture anchor, but nothing compensated for the restore's. A state unit's time advance now names the host instant at which the downtime ended beside its duration: StateUnits::advance_time and StateUnit::advance_time take it, and VmTimeKeeper::advance keeps it, so that the keeper's next start runs VM time from that instant rather than from the start, unless a restore or reset comes first. The chipset devices that keep time of their own, the RTC's calendar, keep it on host clocks that run while they are stopped, and still take only the duration. time_abi_restore names the instant at which the synchronized TSC set resumed the TSCs, which it converts from the time ABI's host monotonic clock to the clock that VM time runs on with the tightest of up to 16 pairs of readings of both, taken back to back, so VM time runs from the anchor on KVM and, on MSHV and WHP, from where the call that resumed partition time restarted the TSCs, as the TSCs do. The restore logs that instant's distance from the anchor. Running VM time from that instant, rather than advancing it once more at the start, keeps the keeper's start protocol: the start pairs the advanced time with an earlier host instant, and the device timers due since then fire once VM time starts. The first start consumes the instant, so a start after a stop, including the rollback of a failed start, runs VM time from itself, as before: MSHV and WHP freeze the TSCs while the partition is stopped, and KVM does not. No saved state, contract, or format changes. Tests: a fake backend whose TSC counts host monotonic time, as KVM's does, captures a VM whose VM time runs in a VmTimeKeeper behind state units, and restores it after a downtime, starting the units 30 ms after the set: TSC/F minus VM time stays within 2 ms of its value before the capture, with the TSCs running from the anchor and after a 10 ms frozen write. It moved by at most 5.6 us in six runs; without the keeper's fix it moves by 31.0 ms, and with the anchor in place of the TSCs' resumption, by -10.5 ms after the frozen write. The keeper runs advanced time from the instant that the advance named, a later start from itself, never from an instant after its start, and drops the instant on a restore; a state unit's advance carries both values; the conversion keeps the tightest pairing of the two clocks. In the microVM snapshot-core and restore-downtime scenarios on nested KVM, MSHV, and WHP hosts, TSC/F minus VM time moved from the restore anchor to the start of the units by 0.28 to 0.35 ms, 0.63 to 0.91 ms, and 0.78 to 1.22 ms before the fix, and by at most 0.4 us, 0.1 us, and 4.5 us with it. On MSHV and WHP, whose stopped partitions freeze the TSCs, the restore still advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it, so VM time leads the TSC after a restore by the time that they stayed frozen before the capture anchor: 1.22 to 1.68 ms in the snapshot-core captures measured. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before the fix to 1.22 to 1.29 ms with it. That capture-side defect predates this change, which leaves it to a separate fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna)
requested a review
from a team
as a code owner
November 11, 2026 22:55
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
Pedro Henrique Penna (ppenna)
merged commit Nov 12, 2026
90f8731
into
feature/microvm/dev
73 of 74 checks passed
syeedali
pushed a commit
to syeedali/nvx
that referenced
this pull request
Nov 12, 2026
After a time ABI restore, the guest's TSC led VM time, the clock of OpenVMM's emulated device timers, by the time from the restart of the TSCs to the start of the state units (microsoft#1371). The restore set the TSCs at the anchor, from which they ran on KVM, and on MSHV and WHP once partition time resumed, and advanced stopped VM time by the downtime to that anchor, but VM time stayed stopped until the units started: 0.28 to 0.35 ms later on KVM, 0.63 to 0.91 ms on MSHV, and 0.78 to 1.22 ms on WHP in the microVM restore scenarios, and as long as a management client holds a restore paused. Pin 90f87319d30d8c35bc3526f12fc7b6529c882cf0, the head of microsoft/openvmm#5052, two commits on 3cd9f149b, the head of feature/microvm/dev: - bce9b3673 virt: report when the synchronized TSC set lets the TSCs run - 90f87319d openvmm: run restored VM time from the instant the TSCs resume The revision has the synchronized TSC set report the host instant from which the TSCs run: the restore anchor on KVM, and on MSHV and WHP, which hold the TSCs at the target until partition time resumes, the instant at which the hypervisor restarted them, which a read of VP 0's TSC places inside the call that resumes partition time. The restore's time advance carries that instant to the state units, and VM time runs from it once the units start, so the time that the restore takes to start the units no longer shifts the emulated devices' timers relative to the TSC. No saved state, contract, or format changes. Evidence: a temporary probe logged TSC/F minus VM time at the restore anchor and once the units started, in the snapshot-core and restore-downtime scenarios, which pass. On nested KVM, MSHV, and WHP hosts, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and +0.78 to +1.22 ms at fb4f34126, and by +0.14 to +0.37 us, -0.10 to +0.09 us, and -0.71 to +4.45 us with the fix. OpenVMM's new restore test moves the relation by 31.0 ms without the fix; with it, the package-scoped tests, Clippy, and rustdoc pass on Windows and on the KVM and MSHV hosts, as do the KVM and MSHV hardware tests. On MSHV and WHP, VM time still leads the TSC after a restore by the time that the stopped partition held the TSCs frozen before the capture anchor, 1.22 to 1.68 ms in the snapshot-core captures, because the restore advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before the fix to 1.22 to 1.29 ms with it. That capture-side defect predates this change and is left to a separate issue. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali
pushed a commit
to syeedali/nvx
that referenced
this pull request
Nov 12, 2026
Maps NVX's new pin, 90f87319d, the head of microsoft/openvmm#5052, at position 255, which with position 254 follows position 253, 3cd9f149b, the head of feature/microvm/dev, which microsoft/openvmm#5046 merged. The pin is 124 commits past the mapped tip. Position 254 has the synchronized TSC set of the time ABI backends of positions 133, 134, and 140 report the host instant from which the TSCs run, and position 255 has the time ABI restore of position 139 pass it to the time advance that PR10's position 019 adds, from which VmTimeKeeper's next start runs VM time. Neither fixes a commit that a roadmap PR maps, so both stay unassigned; the remarks record that position 255 changes PR10's time advance, its chipset state unit, and the tests of positions 241, 244, and 245. Replaying them on upstream main conflicts in 15 of their 16 files, all but the Guide's timekeeping page: upstream main lacks the time ABI of positions 132-140 and the time advance of position 019, so they follow PR10 and the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syeedali
pushed a commit
to syeedali/nvx
that referenced
this pull request
Nov 12, 2026
Maps NVX's new pin, 024ecee62, the head of microsoft/openvmm#5053, at position 258, which with positions 256 and 257, 2a0401c43 and 21a76f878, follows position 255, 90f87319d, the head of feature/microvm/dev, which microsoft/openvmm#5052 merged. The pin is 127 commits past the mapped tip. They change how a restore runs a one-shot LAPIC timer and change nothing that a mapped commit adds, so they stay unassigned. Position 256 moves the guest RAM of the real-mode ExtINT tests of virt_kvm, which upstream main has, into a test module that position 257's tests share. Position 257 changes the KVM backend's set_apic, which upstream main has, and the Guide's timekeeping reference, which position 255 also changes. Position 258 changes the time ABI restore of the LAPIC timers that position 139 adds, which positions 146, 147, 155, 158, and 255 also change, and sets each timer again after the synchronized TSC set, so that it runs as the TSCs do, as position 255's VM time does. Replaying position 256 on upstream main, ba105dd7f, conflicts only in virt_kvm's x86-64 module list, beside the modules of positions 133 and 138. Position 257 applies there without conflicts, but its KVM tests use position 256's guest RAM, position 139's timer advance, the TSC helpers of positions 132 and 160, and the kvm crate's TSC accessors of PR10's position 026, so it follows them upstream. Position 258 conflicts in all 12 of its files, seven of which upstream main lacks, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the VM time of a restored microVM from the instant at which the restore lets the guest's TSCs run, rather than from the later start of the state units, for microsoft/nvx#1371. After a time ABI restore, the guest's TSC led VM time, the clock of the emulated devices' timers, by the time from the TSCs' restart to the start of the state units:
time_abi_restoreset the TSCs and advanced stopped VM time by the downtime to the restore anchor, butVmTimeKeeper::startanchored VM time at a laterInstant::now(), after the LAPIC and VM-time advances, the rest of the restore, and the start of the units, and as long after as a client held a restore paused. The capture compensates for its own window, from the VM-time cut to the capture anchor, but nothing compensated for the restore's.Two commits on
3cd9f149b, the head offeature/microvm/dev. microsoft/nvx#1498 pins this head.Report when the synchronized TSC set lets the TSCs run (
bce9b3673)TscSetReportgainsresumed, a host sample of the instant from which every VP's TSC runs from the target. KVM's common offset makes them run from the restore anchor itself. MSHV and WHP write the target while partition time is frozen, read it back, and resume partition time inside the set, so their TSCs hold the target until the hypervisor restarts them somewhere inside the call that resumes partition time: 48 to 430 µs after the anchor in the microVM restores measured. That call takes 14 to 26 µs on MSHV and 24 to 125 µs on WHP, and the restart falls 3 to 5 µs after its start on an idle partition and 14 to 52 µs after it in a restore, so the backends place it from VP 0's TSC, read against host time right after the call:virt::time_abi::tsc_restarttakes the TSC's distance past the target, across a wrap of its 64 bits, at the native rate, back from the read, and keeps the result between host samples taken around the call. A failed read-back still takes precedence over a failure to take a sample.a_tsc_restart_is_where_the_read_places_it_inside_the_resumeplaces the restart where the read puts it, also after the TSC wraps from a target near the top of its range, and inside the call; the wrap case fails with a saturating distance. MSHV's hardware test of the backend primitives,time_abi_backend_on_hardware, checks that VP 0's TSC, read against host time after the set, ran from the target from the reported instant, within the read's width and 20 µs, and reports how long after the anchor the TSCs resume.Run restored VM time from the instant the TSCs resume (
90f87319d)StateUnits::advance_timeandStateUnit::advance_timetake it,StateRequest::AdvanceTimecarries both inTimeAdvance, andVmTimeKeeper::advancekeeps the instant, so that the keeper's next start runs VM time from it rather than from the start, unless a restore or reset comes first. The first start consumes it, so a later start, including the rollback of a failed start, runs VM time from itself, as before, and a start never runs VM time from an instant after itself. The chipset devices that keep time of their own, the RTC's calendar, keep it on host clocks that run while they are stopped, and still take only the duration.time_abi_restorenames the set'sresumedinstant, which it converts from the time ABI's host monotonic clock to the clock that VM time runs on with the tightest of up to 16 back-to-back pairings of both, so VM time runs from the restore anchor on KVM and, on MSHV and WHP, from where the call that resumed partition time restarted the TSCs, as the TSCs do. The restore logs that instant's distance from the anchor. The timekeeping page and the snapshot guide say so.restore_keeps_the_relation_of_vm_time_to_the_tsccaptures a VM whose VM time runs in aVmTimeKeeperbehind state units, with a fake backend whose TSC counts host monotonic time through the capture, as KVM's does, and restores it after a downtime, starting the units 30 ms after the set: TSC/F minus VM time stays within 2 ms of its value before the capture, with the TSCs running from the anchor and after a 10 ms frozen write. It moved by at most 5.6 µs in six runs; without the keeper's change it moves by 31 ms, and with the anchor in place ofresumed, by -10.5 ms after the frozen write.test_advance_runs_from_the_resumed_instantchecks that the keeper runs advanced time from the instant that the advance named, a later start from itself, never from an instant after its start, and drops the instant on a restore; it fails without the keeper's change.advance_time_reaches_only_units_that_advance_timechecks that a unit receives both values, andhost_instant_keeps_the_tightest_clock_pairingthe conversion. The state unit tests of terminal and lost units pass an instant.Not fixed here
On MSHV and WHP, whose stopped partitions freeze the TSCs, the restore still advances VM time across the capture's cut-to-anchor interval as if the TSCs had run through it, so VM time leads the TSC after a restore by the time that they stayed frozen before the capture anchor: 1.22 to 1.68 ms in the snapshot-core captures measured. The restore window had partly offset that lead, so the net lead grows from 0.64 to 0.89 ms before this change to 1.22 to 1.29 ms with it. That capture-side defect predates this change, which leaves it to a separate fix.
Validation
At this head,
90f87319d, with Rust 1.97.1, forvirt,virt_kvm,virt_mshv,virt_whp(Windows only),vmcore,vmm_core,state_unit,vmotherboard, andopenvmm_core:cargo clippy --all-targetsandcargo doc --no-deps, without warnings from the packages;cargo nextest run --profile agent, 404 passed and 19 skipped;cargo xtask fmt --fix, clean. Without the keeper's change,restore_keeps_the_relation_of_vm_time_to_the_tsc(by 31.8 ms) andtest_advance_runs_from_the_resumed_instantfail.cargo clippy --all-targetsandcargo doc --no-deps, without warnings;cargo nextest run --profile agent, 484 passed and 26 skipped on each; the hardware tests,virt_kvm's 12 andvirt_mshv's 13, passed, the latter placing the TSCs' restart within 72 ns of the reported instant.snapshot-core,restore-processors, andtime-abi-conformancepassed on the KVM and MSHV test VMs. With this change onfb4f34126, before its rebase, which changed no patch, and a rewording of its comments and Guide text,snapshot-core,restore-downtime,snapshot-tiers,smp-snapshot,snapshot-rollback,restore-processors, andtime-abi-conformancepassed on the KVM, MSHV, and WHP test VMs.snapshot-coreandrestore-downtimescenarios. On the KVM, MSHV, and WHP test VMs, the relation moved by +0.28 to +0.35 ms, +0.63 to +0.91 ms, and +0.78 to +1.22 ms atfb4f34126, and by +0.14 to +0.37 µs, -0.10 to +0.09 µs, and -0.71 to +4.45 µs with this change on it, before review's wrap fix and wording.See microsoft/nvx#1498.