Skip to content

[DO NOT REVIEW] microVM: saturate the time ABI restore's logged phase durations - #5056

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1375-restore-log-clock-step
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1375-restore-log-clock-step

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Keeps a faulty host clock from aborting a time ABI restore over its log, for microsoft/nvx#1375. The time ABI restore logs how long its phases took by subtracting one reading of an Instant::elapsed from another. Instant::elapsed saturates when the host's monotonic clock steps back, which a faulty host can do, but a Duration subtraction panics with "overflow when subtracting durations", in release builds too. A step back between two readings, far enough that the later reading was the smaller one, so aborted the restore after the TSCs, LAPICs, and VM time were set, on any backend, at the default info log level.

One commit on c72fc56a3, the head of feature/microvm/dev. microsoft/nvx#1521 pins this head.

Saturate the time ABI restore's logged phase durations (c2c21ddb2)

  • Cause. The "time ABI restore clock" event of LoadedVm::time_abi_restore computed lapic_us and vm_time_us as (lapic_advanced - tsc_set) and (vm_time_advanced - tsc_set), each the difference of two readings of started.elapsed(). The synchronized TSC sets of the MSHV and WHP backends, which only time_abi_restore calls, computed write_us, and on WHP readback_us, the same way.
  • Change. virt::time_abi::phase_us(start, end) gives a phase's microseconds from its two readings, end.saturating_sub(start), so 0 when the later reading is the smaller one, and the three events compute with it every phase that they took as the difference of two readings. Only the logged values change: no field, version, or format changes. The time ABI's other logged differences of two clock readings already saturate: check_cpu_profile_with subtracts Instants, which saturates as Instant::elapsed does, and the synchronized TSC set event saturates resumed_after_anchor_ns.
  • Alternatives. An inline saturating_sub at each of the five sites, as the issue's patch proposed for the two fields of the restore clock event, would fix the same class; one helper gives the class one documented and tested definition, which matters since the restore takes no clock that a test could step back.
  • Tests. a_phase_lasts_zero_when_the_host_clock_steps_back_past_its_start: phase_us gives 2,500 µs from 1,500 µs to 4,000 µs, and 0 from 1,500 µs to 0 and to 1,000 µs. With phase_us subtracting as the events did, (end - start), the test fails with the issue's panic, "overflow when subtracting durations".

Not in this change: VM time's stop computes the stopped VM time with the subtraction of pal_async::timer::Instant, Instant::now() - start_time.os_time() in PrimaryKeeper::run (vm/vmcore/src/vmtime.rs), which panics on the same kind of step during a capture. On the KVM test VM, a 30 s step back of CLOCK_MONOTONIC after the guest booted made the capture panic there, with "supplied instant ... is later than ...". It sets the stopped VM time rather than a log, in code that upstream main has, so it is a follow-up outside microsoft/nvx#1375.

Validation

At this head, c2c21ddb2, with Rust 1.97.1, on Windows: cargo check -p virt -p openvmm_core and the new test pass.

With the same change on an earlier revision of #5055's second commit, which differs from c72fc56a3 only in the KVM backend's LAPIC timer capture and its tests:

  • Windows: cargo check, cargo clippy --all-targets, cargo doc --no-deps, and cargo nextest run --profile agent for virt, virt_whp, and openvmm_core: 278 passed; cargo xtask fmt --fix: clean. The new test fails against the subtraction as the Tests item says.
  • Linux, on the Azure Linux 3 MSHV test VM: the same commands for virt, virt_mshv, and openvmm_core: 317 passed; cargo xtask fmt --fix: clean.
  • End to end through NVX, on the Azure Linux 3 KVM test VM: an LD_PRELOAD shim stepped CLOCK_MONOTONIC, which Instant reads, back by 250 ms right after the "time ABI synchronized TSC set" event of the restores of NVX's restore-processors scenario. OpenVMM built without this change panicked in Duration::sub, called at time_abi.rs:1304 in time_abi_restore, and the restore failed; with it, both restores completed and logged lapic_us=0x0 vm_time_us=0x0.
  • Without the shim, NVX's restore-processors, snapshot-core, and time-abi-conformance scenarios pass with 1 and 2 VPs on the KVM, MSHV, and WHP test VMs, and restores with the backends' info events enabled log the MSHV and WHP phases that this change computes.

See microsoft/nvx#1521.

The time ABI restore logs how long each of its phases took from
readings of one Instant::elapsed, subtracting the reading that ends
the earlier phase from the reading that ends the later one: the "time
ABI restore clock" event of time_abi_restore subtracts the end of the
synchronized TSC set from the ends of the LAPIC and VM-time advances,
and the synchronized TSC sets of MSHV and WHP subtract the end of the
freeze or suspend from the end of the TSC writes, and WHP that from the
end of the read-back. Instant::elapsed saturates when the host's
monotonic clock steps back, which a faulty host can do, but a Duration
subtraction panics with "overflow when subtracting durations", in
release builds too. A step back between two readings, far enough that
the later reading was the smaller one, so aborted the restore after
the TSCs, LAPICs, and VM time were set, on any backend, at the default
info log level (microsoft/nvx#1375).

virt::time_abi::phase_us gives a phase's microseconds from its two
readings, 0 when the later reading is the smaller one, and the three
events compute with it every phase that they took as the difference of
two readings. The time ABI's other logged differences of two clock
readings already saturate: check_cpu_profile_with subtracts Instants,
which saturates as Instant::elapsed does, and the synchronized TSC set
event saturates resumed_after_anchor_ns.

Tests: phase_us gives 2500 us from 1500 us to 4000 us, and 0 from
1500 us to 0 us and to 1000 us. Without the fix, its subtraction panics
with "overflow when subtracting durations".

Evidence: on nvx-kvm-test, an LD_PRELOAD shim stepped the host's
CLOCK_MONOTONIC back by 250 ms right after the "time ABI synchronized
TSC set" event of the restores of NVX's restore-processors scenario.
The OpenVMM built without this commit panicked there in Duration's
subtraction, called at time_abi.rs:1304 in time_abi_restore, and the
restore failed; with this commit, both restores completed and logged
lapic_us=0x0 vm_time_us=0x0. Without the shim, the restore-processors,
snapshot-core, and time-abi-conformance scenarios pass on KVM, MSHV,
and WHP, and restores with the backends' events enabled log the MSHV
and WHP phases that the change computes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added unsafe Related to unsafe code and removed unsafe Related to unsafe code labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@github-actions github-actions Bot added the release-nvx Targets a branch under feature/microvm/. label Nov 12, 2026
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 12, 2026
A time ABI restore logged how long its phases took by subtracting one
reading of an Instant::elapsed from another, in the "time ABI restore
clock" event and in the synchronized TSC sets of MSHV and WHP.
Instant::elapsed saturates when the host's monotonic clock steps back,
which a faulty host can do, but the Duration subtraction panics with
"overflow when subtracting durations", in release builds too, so a step
back between two readings, far enough that the later reading was the
smaller one, aborted the restore after the TSCs, LAPICs, and VM time
were set, on any backend, at OpenVMM's default info log level (#1375).

Pin c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, the head of
microsoft/openvmm#5056, one commit on c72fc56a3, the head of
feature/microvm/dev:

- c2c21ddb2 microVM: saturate the time ABI restore's logged phase
  durations

The revision adds virt::time_abi::phase_us, which gives a phase's
microseconds from its two readings, 0 when the later reading is the
smaller one, and computes with it every phase that the three events
took as the difference of two readings.

Evidence: on nvx-kvm-test, an LD_PRELOAD shim stepped the host's
CLOCK_MONOTONIC back by 250 ms right after the "time ABI synchronized
TSC set" event of the restores of the restore-processors scenario. The
OpenVMM built without the fix panicked in Duration's subtraction at
time_abi.rs:1304, in time_abi_restore, and the scenario failed; with
the fix, it passed and logged lapic_us=0x0 vm_time_us=0x0. Without the
shim, the restore-processors, snapshot-core, and time-abi-conformance
scenarios pass on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test, and
restores with the backends' events enabled log the MSHV and WHP phases
that the fix computes. The new virt test panics with "overflow when
subtracting durations" against the base's subtraction and passes at the
pinned revision; the tests of virt, virt_whp, and openvmm_core pass on
Windows, and those of virt, virt_mshv, and openvmm_core on Linux. These
runs had the fix on an earlier revision of microsoft/openvmm#5055, which
differs from c72fc56a3 only in the KVM backend's LAPIC timer capture
and its tests; at the pinned revision, virt and openvmm_core build and
the new test passes on Windows.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 12, 2026
The NVX pin moves to c2c21ddb2, position 262, 131 commits past the
mapped tip. Position 262, which microsoft/openvmm#5056 proposes for
feature/microvm/dev on its head, c72fc56a3, is unassigned. The Scope
also says that microsoft/openvmm#5055 merged positions 259-261.

It computes the phases of the time ABI restore's clock event, which
position 144 adds and position 146 gives its concurrent LAPIC and
VM-time phases, and of the synchronized TSC sets of the MSHV and WHP
backends of positions 134 and 140, that are the difference of two
readings with a helper that it adds to virt's time ABI module of
position 132, which gives 0 where the host's monotonic clock stepped
back past the phase's start and the subtraction panicked. It changes
nothing that a mapped PR adds. Replaying it on upstream main conflicts
in all four of its files, since upstream main lacks the time ABI, so it
follows the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) merged commit c2c21dd into feature/microvm/dev Nov 12, 2026
73 of 74 checks passed
@github-actions

Copy link
Copy Markdown

Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 12, 2026
The NVX pin moves to 5d656d854, position 263, 132 commits past the
mapped tip. Position 263, which microsoft/openvmm#5061 proposes for
feature/microvm/dev on its head, c2c21ddb2, is unassigned. The Scope
also says that microsoft/openvmm#5056 merged position 262.

It makes the PIT's count writes in the low-then-high access mode take
each count's low byte first, in Timer::write, code that upstream main
has, and adds three tests, in a file that PR10's position 022 and
position 139 also change, neither of them in the code that it changes.
It changes nothing that a mapped PR adds. Replaying it on upstream main
applies without conflicts, and chipset type-checks there with its
tests, which pass, so it can go upstream on its own.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-nvx Targets a branch under feature/microvm/.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant