Repository navigation
openvmm: saturate the time ABI restore's logged phase durations - #1521
Merged
Pedro Henrique Penna (ppenna) merged 2 commits intoNov 12, 2026
Merged
Conversation
A time ABI restore logged how long its phases took by subtracting one reading of an Instant::elapsed from another, in the "time ABI restore clock" event and in the synchronized TSC sets of MSHV and WHP. Instant::elapsed saturates when the host's monotonic clock steps back, which a faulty host can do, but the Duration subtraction panics with "overflow when subtracting durations", in release builds too, so a step back between two readings, far enough that the later reading was the smaller one, aborted the restore after the TSCs, LAPICs, and VM time were set, on any backend, at OpenVMM's default info log level (#1375). Pin c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, the head of microsoft/openvmm#5056, one commit on c72fc56a3, the head of feature/microvm/dev: - c2c21ddb2 microVM: saturate the time ABI restore's logged phase durations The revision adds virt::time_abi::phase_us, which gives a phase's microseconds from its two readings, 0 when the later reading is the smaller one, and computes with it every phase that the three events took as the difference of two readings. Evidence: on nvx-kvm-test, an LD_PRELOAD shim stepped the host's CLOCK_MONOTONIC back by 250 ms right after the "time ABI synchronized TSC set" event of the restores of the restore-processors scenario. The OpenVMM built without the fix panicked in Duration's subtraction at time_abi.rs:1304, in time_abi_restore, and the scenario failed; with the fix, it passed and logged lapic_us=0x0 vm_time_us=0x0. Without the shim, the restore-processors, snapshot-core, and time-abi-conformance scenarios pass on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test, and restores with the backends' events enabled log the MSHV and WHP phases that the fix computes. The new virt test panics with "overflow when subtracting durations" against the base's subtraction and passes at the pinned revision; the tests of virt, virt_whp, and openvmm_core pass on Windows, and those of virt, virt_mshv, and openvmm_core on Linux. These runs had the fix on an earlier revision of microsoft/openvmm#5055, which differs from c72fc56a3 only in the KVM backend's LAPIC timer capture and its tests; at the pinned revision, virt and openvmm_core build and the new test passes on Windows. NVX needs no coordinated change: the control-session vectors, the source excerpts, the contract versions, and the CPU profiles that nvx-rs copies from OpenVMM match the revision's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to c2c21ddb2, position 262, 131 commits past the mapped tip. Position 262, which microsoft/openvmm#5056 proposes for feature/microvm/dev on its head, c72fc56a3, is unassigned. The Scope also says that microsoft/openvmm#5055 merged positions 259-261. It computes the phases of the time ABI restore's clock event, which position 144 adds and position 146 gives its concurrent LAPIC and VM-time phases, and of the synchronized TSC sets of the MSHV and WHP backends of positions 134 and 140, that are the difference of two readings with a helper that it adds to virt's time ABI module of position 132, which gives 0 where the host's monotonic clock stepped back past the phase's start and the subtraction panicked. It changes nothing that a mapped PR adds. Replaying it on upstream main conflicts in all four of its files, since upstream main lacks the time ABI, so it follows the time ABI series upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified in the roadmap update tracking the restore-phase logging fix.
No open issues 🎉
What changed in this PR
Updates the OpenVMM roadmap to track NVX’s pin for the restore-phase logging fix, which prevents duration-subtraction panics when the host clock steps backward.
Changes:
- Updates the documented pin to
c2c21ddb2and the chain totals to position 262. - Records the new commit’s scope and its dependency on the time ABI series for upstreaming.
| File | Description |
|---|---|
| doc/openvmm-upstream-roadmap.md | Updates the pin, commit mapping, chain counts, and upstreaming notes for the restore-phase logging fix. |
Review effort: Balanced
Pedro Henrique Penna (ppenna)
deleted the
agents/issue-1375-restore-log-clock-step
branch
November 12, 2026 06:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1375
What and why
A time ABI restore logged how long its phases took by subtracting one reading of an
Instant::elapsedfrom another.Instant::elapsedsaturates when the host's monotonic clock steps back, which a faulty host can do, but aDurationsubtraction panics with "overflow when subtracting durations", in release builds too. A step back between two readings, far enough that the later one was the smaller, so aborted the restore after the TSCs, LAPICs, and VM time were set, on any backend, at OpenVMM's defaultinfolog level.Root cause, in OpenVMM: the "time ABI restore clock" event of
LoadedVm::time_abi_restorecomputedlapic_usandvm_time_usas(lapic_advanced - tsc_set)and(vm_time_advanced - tsc_set). The synchronized TSC sets of the MSHV and WHP backends, which onlytime_abi_restorecalls, computedwrite_us, and on WHPreadback_us, the same way.The pin, microsoft/openvmm#5056 at
c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5, adds one commit tofeature/microvm/dev,c72fc56a3, #1390's pin:c2c21ddb2microVM: saturate the time ABI restore's logged phase durations.virt::time_abi::phase_us(start, end)gives a phase's microseconds from its two readings, 0 when the later reading is the smaller one, with a unit test, and the three events compute with it every phase that they took as the difference of two readings.The NVX commits:
8a63f699ef3openvmm: pin the saturated time ABI restore phase logs. It changes only the gitlink, fromc72fc56a3toc2c21ddb2.401c7b7e8bedoc: map the saturated restore phase logs in the OpenVMM roadmap. Position 262, unassigned.Decisions:
virt::time_abirather than an inlinesaturating_subat each of the five sites in three crates, as the issue's patch proposed for its two fields: one documented, tested definition covers the class, and the restore takes no clock that a test could step back. The test fails with the issue's panic when the helper subtracts as the events did.check_cpu_profile_withsubtractsInstants, which saturates asInstant::elapseddoes, and the synchronized TSC set event saturatesresumed_after_anchor_ns.Follow-ups from the audit of the time ABI's capture and restore, not fixed here:
Instant::elapsedalone, so a backward clock step could stretch it, and the stop, by the step's length; it could not panic. openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390 already fixed it in9f14b54fd, which also counts the pauses that the wait slept. Nothing remains.Instant::now() - start_time.os_time()withpal_async::timer::Instant's subtraction, which panics with "supplied instant ... is later than ..." when the host's monotonic clock stepped back past the instant from which VM time started (vm/vmcore/src/vmtime.rs:751,PrimaryKeeper::run, code that upstream OpenVMM has). On nvx-kvm-test, a 30 s step back ofCLOCK_MONOTONICafter the guest booted made the capture panic there. It sets the stopped VM time rather than a log, so it needs its own issue. A possible fix computes it withTimeState::now, which already takes a clock behind VM time's start as that start, or withInstant::saturating_sub.Validation
Local checks at this head (Windows):
python scripts/openvmm_pin.py check-branch --topic: passed (2 signed commits, no coordinated files).python scripts/openvmm_pin.py check-roadmap --worktree: passed (262 positions, 271 references).python scripts/openvmm_pin.py source c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5 --pr 5056: good signature, and the revision is the pull request's head.python scripts/nvx.py verify: passed.python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run: passed. That ischeck_test_changes.py(0 changes to existing tests),check_markdown_links.py,check_test_layout.py, and the ignoredopenvmm_vectors(4) andopenvmm_cpu_profiles(1) tests ofnvx-rs.nvx-cligate,python -m unittest discoveroverscripts/: not run locally. This change does not touchscripts/, and the gate takes over two hours on this shared machine. It is left to CI'sNVX CLI / LinuxandNVX CLI / Windowsjobs, which must pass on this head.OpenVMM package checks:
c2c21ddb2, on Windows:cargo check -p virt -p openvmm_coreand the new test,a_phase_lasts_zero_when_the_host_clock_steps_back_past_its_start, passed.c72fc56a3only in the KVM backend's LAPIC timer capture and its tests: on Windows,cargo check,cargo clippy --all-targets,cargo doc --no-deps, andcargo nextest run --profile agentforvirt,virt_whp, andopenvmm_corepassed (278 tests), andcargo xtask fmt --fixwas clean; on nvx-mshv-test (Linux), the same commands forvirt,virt_mshv, andopenvmm_corepassed (317 tests), andcargo xtask fmt --fixwas clean. The new test fails with "overflow when subtracting durations" whenphase_ussubtracts as the events did.Remote runs, with the change on that earlier revision of microsoft/openvmm#5055, OpenVMM built over the
v0.1.0-dev.f04f1de7b2e5artifacts withnvx.py build-openvmm, whose binary hash changed each time. "Before" is OpenVMM without this change:test-microvm --scenario restore-processors --processors 1, with anLD_PRELOADshim that stepsCLOCK_MONOTONIC, whichInstantreads, back 250 ms right after the "time ABI synchronized TSC set" event (binaryf611db64before,2e749846after)Duration::sub, called attime_abi.rs:1304intime_abi_restore; the restore failedlapic_us=0x0 vm_time_us=0x0; passedrestore-processors,snapshot-core, andtime-abi-conformance(1, 2 VPs)restore-processors,snapshot-core, andtime-abi-conformance(1, 2 VPs)restore-processors,snapshot-core, andtime-abi-conformance(1, 2 VPs)restore-processors(1, 2 VPs) with the backends'infoevents enabled, which log the TSC sets' phaseswrite_us, and on WHPreadback_us, loggedrestore-downtimewas not run on the test VMs: it needs #516's guest artifacts, which no release had yet. It is left to CI's microVM jobs, with the other scenarios, on all three backends. Onfb4f34126, before #516 changed that scenario, it passed on the three test VMs with this change.CI at this head,
401c7b7e8be, in run 43160211326: passed.OpenVMM CI of the pinned revision: passed. OpenVMM's required jobs passed forc2c21ddb2in its run 43159706129:unit tests [x64-linux],unit tests [x64-windows], andrun vmm-testsonx64-linux-amd-kvm,x64-linux-intel-mshv,x64-windows-intel, andx64-windows-amd. Of the OpenVMM jobs that NVX does not require,run vmm-tests [x64-windows-amd-snp]failed: two OpenHCL TPM tests of Hyper-V SNP guests,tpm138::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_prepped_snp_cvm_guestand..._snp_seal_signer, timed out after 600 s, in code that this change does not reach.NVX CLI / LinuxandNVX CLI / Windows: passed.No existing test changes, so no
tests-modifiedlabel.c2c21ddb25822e13bd1cf1b33ed693545cc7bbb5.c72fc56a3, the head offeature/microvm/devafter [DO NOT REVIEW] kvm, virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again openvmm#5055 (openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390). There are no other prerequisites.92426efc. Its change is patch-identical (git patch-id --stable) to the commit that the package checks and remote runs above validated on the earlier revision of [DO NOT REVIEW] kvm, virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again openvmm#5055, whose tree differs fromc72fc56a3's only invmm_core/virt_kvm/src/arch/x86_64/lapic_timer.rs, which this change does not touch.feature/microvm/devfromc72fc56a3to exactlyc2c21ddb25822e13bd1cf1b33ed693545cc7bbb5. OpenVMM's merge button would squash the commit and leave the pin off the branch. This pull request then merges through the queue.