Skip to content

openvmm: saturate the time ABI restore's logged phase durations - #1521

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1375-restore-log-clock-step
Nov 12, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1375-restore-log-clock-step

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1375

What and why

A time ABI restore logged how long its phases took by subtracting one reading of an Instant::elapsed from another. Instant::elapsed saturates when the host's monotonic clock steps back, which a faulty host can do, but a Duration subtraction panics with "overflow when subtracting durations", in release builds too. A step back between two readings, far enough that the later one was the smaller, so aborted the restore after the TSCs, LAPICs, and VM time were set, on any backend, at OpenVMM's default info log level.

Root cause, in OpenVMM: the "time ABI restore clock" event of LoadedVm::time_abi_restore computed lapic_us and vm_time_us as (lapic_advanced - tsc_set) and (vm_time_advanced - tsc_set). The synchronized TSC sets of the MSHV and WHP backends, which only time_abi_restore calls, computed write_us, and on WHP readback_us, the same way.

The pin, microsoft/openvmm#5056 at c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, adds one commit to feature/microvm/dev, c72fc56a3, #1390's pin:

  • c2c21ddb2 microVM: saturate the time ABI restore's logged phase durations. virt::time_abi::phase_us(start, end) gives a phase's microseconds from its two readings, 0 when the later reading is the smaller one, with a unit test, and the three events compute with it every phase that they took as the difference of two readings.

The NVX commits:

  • 8a63f699ef3 openvmm: pin the saturated time ABI restore phase logs. It changes only the gitlink, from c72fc56a3 to c2c21ddb2.
  • 401c7b7e8be doc: map the saturated restore phase logs in the OpenVMM roadmap. Position 262, unassigned.

Decisions:

  • One helper in virt::time_abi rather than an inline saturating_sub at each of the five sites in three crates, as the issue's patch proposed for its two fields: one documented, tested definition covers the class, and the restore takes no clock that a test could step back. The test fails with the issue's panic when the helper subtracts as the events did.
  • The class, not only the issue's two fields: the MSHV and WHP TSC sets' phase logs panicked the same way. The time ABI's other logged differences of two clock readings already saturate: check_cpu_profile_with subtracts Instants, which saturates as Instant::elapsed does, and the synchronized TSC set event saturates resumed_after_anchor_ns.
  • Only the logged values change: no log field, and no saved state, manifest, time ABI, contract, or format version.

Follow-ups from the audit of the time ABI's capture and restore, not fixed here:

  1. openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390's settle poll. When a KVM VP stops, the backend waits up to 20 ms for KVM to record a LAPIC timer's expiry. That wait measured its time with Instant::elapsed alone, so a backward clock step could stretch it, and the stop, by the step's length; it could not panic. openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390 already fixed it in 9f14b54fd, which also counts the pauses that the wait slept. Nothing remains.
  2. A capture panic that predates these issues. VM time's stop computes the stopped VM time as Instant::now() - start_time.os_time() with pal_async::timer::Instant's subtraction, which panics with "supplied instant ... is later than ..." when the host's monotonic clock stepped back past the instant from which VM time started (vm/vmcore/src/vmtime.rs:751, PrimaryKeeper::run, code that upstream OpenVMM has). On nvx-kvm-test, a 30 s step back of CLOCK_MONOTONIC after the guest booted made the capture panic there. It sets the stopped VM time rather than a log, so it needs its own issue. A possible fix computes it with TimeState::now, which already takes a clock behind VM time's start as that start, or with Instant::saturating_sub.

Validation

Local checks at this head (Windows):

  • python scripts/openvmm_pin.py check-branch --topic: passed (2 signed commits, no coordinated files).
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed (262 positions, 271 references).
  • python scripts/openvmm_pin.py source c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5 --pr 5056: good signature, and the revision is the pull request's head.
  • python scripts/nvx.py verify: passed.
  • python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run: passed. That is check_test_changes.py (0 changes to existing tests), check_markdown_links.py, check_test_layout.py, and the ignored openvmm_vectors (4) and openvmm_cpu_profiles (1) tests of nvx-rs.
  • The nvx-cli gate, python -m unittest discover over scripts/: not run locally. This change does not touch scripts/, and the gate takes over two hours on this shared machine. It is left to CI's NVX CLI / Linux and NVX CLI / Windows jobs, which must pass on this head.

OpenVMM package checks:

  • At c2c21ddb2, on Windows: cargo check -p virt -p openvmm_core and the new test, a_phase_lasts_zero_when_the_host_clock_steps_back_past_its_start, passed.
  • With the same change on an earlier revision of [DO NOT REVIEW] kvm, virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again openvmm#5055, which differs from c72fc56a3 only in the KVM backend's LAPIC timer capture and its tests: on Windows, cargo check, cargo clippy --all-targets, cargo doc --no-deps, and cargo nextest run --profile agent for virt, virt_whp, and openvmm_core passed (278 tests), and cargo xtask fmt --fix was clean; on nvx-mshv-test (Linux), the same commands for virt, virt_mshv, and openvmm_core passed (317 tests), and cargo xtask fmt --fix was clean. The new test fails with "overflow when subtracting durations" when phase_us subtracts as the events did.

Remote runs, with the change on that earlier revision of microsoft/openvmm#5055, OpenVMM built over the v0.1.0-dev.f04f1de7b2e5 artifacts with nvx.py build-openvmm, whose binary hash changed each time. "Before" is OpenVMM without this change:

Profile Backend Host type Check Before After
nvx-kvm-test kvm virtual-machine test-microvm --scenario restore-processors --processors 1, with an LD_PRELOAD shim that steps CLOCK_MONOTONIC, which Instant reads, back 250 ms right after the "time ABI synchronized TSC set" event (binary f611db64 before, 2e749846 after) panic "overflow when subtracting durations" in Duration::sub, called at time_abi.rs:1304 in time_abi_restore; the restore failed both restores completed, lapic_us=0x0 vm_time_us=0x0; passed
nvx-kvm-test kvm virtual-machine restore-processors, snapshot-core, and time-abi-conformance (1, 2 VPs) not run passed
nvx-mshv-test mshv virtual-machine restore-processors, snapshot-core, and time-abi-conformance (1, 2 VPs) not run passed
nvx-whp-test whp virtual-machine restore-processors, snapshot-core, and time-abi-conformance (1, 2 VPs) not run passed
nvx-mshv-test, nvx-whp-test mshv, whp virtual-machine restore-processors (1, 2 VPs) with the backends' info events enabled, which log the TSC sets' phases not run passed; write_us, and on WHP readback_us, logged

restore-downtime was not run on the test VMs: it needs #516's guest artifacts, which no release had yet. It is left to CI's microVM jobs, with the other scenarios, on all three backends. On fb4f34126, before #516 changed that scenario, it passed on the three test VMs with this change.

CI at this head, 401c7b7e8be, in run 43160211326: passed.

  • OpenVMM CI of the pinned revision: passed. OpenVMM's required jobs passed for c2c21ddb2 in its run 43159706129: unit tests [x64-linux], unit tests [x64-windows], and run vmm-tests on x64-linux-amd-kvm, x64-linux-intel-mshv, x64-windows-intel, and x64-windows-amd. Of the OpenVMM jobs that NVX does not require, run vmm-tests [x64-windows-amd-snp] failed: two OpenHCL TPM tests of Hyper-V SNP guests, tpm138::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_prepped_snp_cvm_guest and ..._snp_seal_signer, timed out after 600 s, in code that this change does not reach.
  • NVX CLI / Linux and NVX CLI / Windows: passed.
  • The microVM tests on the KVM, MSHV, and WHP runners, including the MSHV debug kernel and the WHP runtime adapter: passed.
  • The platform benchmarks on KVM, MSHV, and WHP and the performance regression gate: passed.

No existing test changes, so no tests-modified label.

A time ABI restore logged how long its phases took by subtracting one
reading of an Instant::elapsed from another, in the "time ABI restore
clock" event and in the synchronized TSC sets of MSHV and WHP.
Instant::elapsed saturates when the host's monotonic clock steps back,
which a faulty host can do, but the Duration subtraction panics with
"overflow when subtracting durations", in release builds too, so a step
back between two readings, far enough that the later reading was the
smaller one, aborted the restore after the TSCs, LAPICs, and VM time
were set, on any backend, at OpenVMM's default info log level (#1375).

Pin c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, the head of
microsoft/openvmm#5056, one commit on c72fc56a3, the head of
feature/microvm/dev:

- c2c21ddb2 microVM: saturate the time ABI restore's logged phase
  durations

The revision adds virt::time_abi::phase_us, which gives a phase's
microseconds from its two readings, 0 when the later reading is the
smaller one, and computes with it every phase that the three events
took as the difference of two readings.

Evidence: on nvx-kvm-test, an LD_PRELOAD shim stepped the host's
CLOCK_MONOTONIC back by 250 ms right after the "time ABI synchronized
TSC set" event of the restores of the restore-processors scenario. The
OpenVMM built without the fix panicked in Duration's subtraction at
time_abi.rs:1304, in time_abi_restore, and the scenario failed; with
the fix, it passed and logged lapic_us=0x0 vm_time_us=0x0. Without the
shim, the restore-processors, snapshot-core, and time-abi-conformance
scenarios pass on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test, and
restores with the backends' events enabled log the MSHV and WHP phases
that the fix computes. The new virt test panics with "overflow when
subtracting durations" against the base's subtraction and passes at the
pinned revision; the tests of virt, virt_whp, and openvmm_core pass on
Windows, and those of virt, virt_mshv, and openvmm_core on Linux. These
runs had the fix on an earlier revision of microsoft/openvmm#5055, which
differs from c72fc56a3 only in the KVM backend's LAPIC timer capture
and its tests; at the pinned revision, virt and openvmm_core build and
the new test passes on Windows.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, and the CPU profiles that
nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c2c21ddb2, position 262, 131 commits past the
mapped tip. Position 262, which microsoft/openvmm#5056 proposes for
feature/microvm/dev on its head, c72fc56a3, is unassigned. The Scope
also says that microsoft/openvmm#5055 merged positions 259-261.

It computes the phases of the time ABI restore's clock event, which
position 144 adds and position 146 gives its concurrent LAPIC and
VM-time phases, and of the synchronized TSC sets of the MSHV and WHP
backends of positions 134 and 140, that are the difference of two
readings with a helper that it adds to virt's time ABI module of
position 132, which gives 0 where the host's monotonic clock stepped
back past the phase's start and the subtraction panicked. It changes
nothing that a mapped PR adds. Replaying it on upstream main conflicts
in all four of its files, since upstream main lacks the time ABI, so it
follows the time ABI series upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified in the roadmap update tracking the restore-phase logging fix.

No open issues 🎉

What changed in this PR

Updates the OpenVMM roadmap to track NVX’s pin for the restore-phase logging fix, which prevents duration-subtraction panics when the host clock steps backward.

Changes:

  • Updates the documented pin to c2c21ddb2 and the chain totals to position 262.
  • Records the new commit’s scope and its dependency on the time ABI series for upstreaming.
File Description
doc/​openvmm-upstream-roadmap.md Updates the pin, commit mapping, chain counts, and upstreaming notes for the restore-phase logging fix.

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 12, 2026
Merged via the queue into dev with commit 83764f5 Nov 12, 2026
48 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1375-restore-log-clock-step branch November 12, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: time-ABI restore panics in a log line if the host clock steps back

2 participants