Skip to content

[DO NOT REVIEW] vmcore: skip a periodic timer's missed periods at once - #5073

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1164-periodic-timer-catch-up
Nov 13, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
feature/microvm/devfrom
copilot/nvx-issue-1164-periodic-timer-catch-up

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Makes VmTimerPeriodic skip a periodic timer's missed periods at once, for microsoft/nvx#1164. Its poll moved the deadline one period per iteration, so after VM time jumped far past the deadline, as it does across a snapshot's downtime, the RTC's 8192 Hz periodic interrupt kept a restored VM from running for minutes.

One commit on 60e160369, the head of feature/microvm/dev.

Skip a periodic timer's missed periods at once (5ca20ee08)

  • Cause. VmTimerPeriodic::poll_timeout looped while VmTimeAccess::poll_timeout was ready, moving the deadline by one period each time until it passed the current VM time, and returned ready once for all the deadlines that it passed. The RTC uses it for its periodic interrupt, at up to 8192 Hz with status A's rate selector 3, and for its update-ended interrupt, at 1 Hz. After a restore whose downtime advance moves VM time 30 days past the deadline, the loop runs some 21 billion times, and the RTC's state unit polls the device, under its lock, as the restored VM starts. A period shorter than VM time's 100 ns resolution counted as zero, so once the timer fired, the loop never moved the deadline.
  • Change. A running timer's poll moves its timeout to the first of its deadlines after now in one step, from the number of whole periods since the timeout, and registers its waker for that deadline, in a new TimerState::poll_periodic_timeout, under the same lock and clock reading as the check that found the timeout due. A ready poll so accounts for every deadline up to now, and leaves the timer waiting for a deadline after the time that it read, without polling again or waking its own task. The deadlines stay a whole number of periods after the timer's start, as before, and a deadline at exactly now counts with the ones before it. start counts a period shorter than 100 ns as 100 ns, and caps a period at i64::MAX units, the longest interval that VM time's circular order puts after a time. VmTimerPeriodic drops last_timeout: its VmTimeAccess's timeout now holds the deadline alone.
  • Alternatives. The patch attached to openvmm: RTC periodic catch-up stalls restore with billions of iterations nvx#1164 computes the next deadline from the poll's now, then polls again to register it, and wakes its own task when that deadline has passed already. With periods near 100 ns, that self-wake repeats, and ArcMutexChipsetDeviceUnit::run polls a device again at once when its event is signaled, so the unit's task would never yield. Computing and registering the deadline under the timer state's lock leaves no such window. Like this change, the patch counts a deadline at exactly now with the ones before it.
  • Tests. In a new periodic module of vmtime's tests, whose cases that spun run under a 10 s watchdog: test_skips_missed_periods_at_once moves stopped VM time 30 days past an 8192 Hz timer, and requires one ready poll, the first deadline after now, a pending poll, and the next deadline once VM time passes that one; test_rounds_short_periods_up requires a 50 ns timer's deadlines 100 ns apart, and, while VM time runs, a deadline after the time of each of 100 fires; test_catch_up_across_the_wrap catches up across VM time's wrap onto a deadline. With 60e160369's code, the first two do not complete within 10 s, and the third leaves the timeout at now, VmTime(170) rather than VmTime(180).

Not in this change, found while reviewing it:

  • pal_async's thread-pool timer, Timer::set_deadline in windows/tp.rs, subtracts Instant::now() from the deadline without saturating, so it panics for a deadline that has passed. VM time's timers can pass it one, as TimerState::poll_timeout already does with a timeout at exactly now, but OpenVMM runs them on VmTaskDrivers over DefaultDriver, whose IOCP and epoll timers accept past deadlines.
  • A periodic timer's phase across a restore: the RTC starts its timers afresh when it restores, as before.

Validation

With Rust 1.97.1, on Windows, at 5ca20ee08: for vmcore, cargo check, cargo clippy --all-targets (clean apart from the environment's LNK4199 linker notices), cargo doc --no-deps, and cargo nextest run --profile agent (40 passed); cargo nextest run --profile agent -p chipset -p chipset_legacy (115 passed); then cargo xtask fmt --fix (complete, no changes). The three new tests passed 200 stress iterations, and fail with 60e160369's code as described above.

Replayed on upstream main, 7d90c4104, the commit applies without conflicts, and vmcore type-checks there with all its targets and its 35 tests pass.

In NVX's Alpine microVM, a guest enabled the RTC's periodic interrupt at 8192 Hz through /dev/port (status A 0x23 and status B's periodic interrupt enable) and requested a capture, and each restore of the snapshot timed the guest's first line after the capture. With OpenVMM built from 60e160369:

Backend Host Real downtime 1 simulated day 30 simulated days
KVM Azure Linux 3 VM 0.09 s 30.8 s 921.7 s
MSHV Azure Linux 3 VM 0.04 s 30.7 s 917.3 s
WHP Windows VM 0.15 s 27.2 s 809.8 s

The simulated downtimes use OpenVMM's --x-time-abi-test-hook downtime-add-s=N. With OpenVMM built from 5ca20ee08, the same restores took 0.07 s, 0.09 s, and 0.11 s on KVM, 0.04 s, 0.04 s, and 0.06 s on MSHV, and 0.59 s, 0.18 s, and 0.20 s on WHP, and after each restore the guest found the RTC's periodic interrupt flag set, and set again 10 ms after its read had cleared it.

On Linux, on the Azure Linux 3 KVM test VM, at 5ca20ee08: cargo nextest run --profile agent -p vmcore -p chipset -p chipset_legacy (155 passed), cargo test -p vmcore --lib (40 passed), and 200 stress iterations of the three new tests.

microsoft/nvx#1565 pins this head and records those runs, with NVX's microVM scenarios, which pass with it on KVM, MSHV, and WHP.

OpenVMM CI of this head, run 43368520199, passed every job that NVX requires. Its run vmm-tests [x64-windows-amd-snp] failed in four of its 114 tests: multiarch::vmgs::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_prepped_snp_vmgstool_copy_igvmfile, whose Hyper-V VM never started because the runner's Hyper-V failed to load the CVM firmware vmfirmwarecvm.dll (0x80070002), as at 60e160369 in #5072's run 43352068114, and multiarch::tpm::…_snp_ak_cert_retry, multiarch::tpm::…_snp_hw_ak_stable, and multiarch::vmbus_relay::…_snp_vmbus_relay, which timed out after about 12 minutes each. The same job fails with timeouts in the same TPM and VMBus relay tests in the same hours on pull requests that lack this change, such as run 43330262358, run 43334289464, and run 43342660705, so it predates this change.

VmTimerPeriodic::poll_timeout moved its deadline one period per
iteration until the deadline passed the current VM time. After VM time
jumped past the deadline, as it does across a snapshot's downtime, the
poll iterated once for every missed period, although it returned ready
only once for all of them. An RTC with periodic interrupts at 8192 Hz,
status A's rate selector 3, misses some 21 billion periods in a 30-day
downtime (microsoft/nvx#1164), and the RTC's state unit polls the
device as the restored VM starts, so the VM ran only once the loop had
ended. The RTC's update-ended interrupts, at 1 Hz, use the same timer.
A period shorter than VM time's 100 ns resolution was worse: VM time
counted it as zero, so once the timer fired, the loop never moved the
deadline and spun forever.

A running timer's poll now moves its timeout to the first of its
deadlines after now in one step, from the number of whole periods since
the timeout, and registers its waker for that deadline, under the same
lock and clock reading as the check that found the timeout due. A
ready poll so accounts for every deadline up to now at once, and leaves
the timer waiting for a deadline after the time that it read, without
polling again or waking its own task. The deadlines stay a whole number
of periods after the timer's start, as before, and a deadline at
exactly now counts with the ones before it. start counts a period
shorter than 100 ns as 100 ns, and one longer than VM time's circular
order can place after a time as the longest that it can.

Tests: the new tests of VmTimerPeriodic move stopped VM time 30 days
past an 8192 Hz timer, and require one ready poll, the first deadline
after now, and the next one once VM time passes it; require a 50 ns
timer's deadlines 100 ns apart and, while VM time runs, a deadline
after the time of each of 100 fires; and catch up across VM time's
wrap onto a deadline. At 60e1603, the first two do not complete
within 10 s, and the third leaves the timeout at now rather than a
period after it. With cargo nextest on Windows x64, they pass with this
commit, with vmcore's other 37 tests and the 115 tests of chipset and
chipset_legacy, whose RTC is the timer's only user.

Evidence: in NVX's Alpine microVM, a guest enabled the RTC's periodic
interrupt at 8192 Hz through /dev/port and requested a capture, and
each restore timed the guest's first line after the capture. With
OpenVMM built from 60e1603, that line came within 0.2 s of a restore
after the real downtime on KVM, MSHV, and WHP; after a simulated day,
downtime-add-s=86400, it took 30.8 s on KVM, 30.7 s on MSHV, and 27.2 s
on WHP; and after 30 days, 921.7 s, 917.3 s, and 809.8 s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the release-nvx Targets a branch under feature/microvm/. label Nov 13, 2026
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 13, 2026
OpenVMM's VmTimerPeriodic moved its deadline one period per iteration
until it passed the current VM time, so after a restore whose downtime
moved VM time far past the deadline, the poll iterated once for every
missed period (#1164). The RTC uses it for its periodic interrupt, at
up to 8192 Hz, and for its update-ended interrupt, at 1 Hz, and the
RTC's state unit polls the device as the restored VM starts, so a guest
that had enabled the RTC's periodic interrupt at 8192 Hz ran again only
some 15 minutes after a restore from a 30-day downtime.

Pin 5ca20ee081c255bed107e58b6dc5b5b769b6101d, the head of
microsoft/openvmm#5073, one commit on 60e160369, the head of
feature/microvm/dev:

- 5ca20ee08 vmcore: skip a periodic timer's missed periods at once

A running timer's poll now moves its timeout to the first of its
deadlines after now in one step, and registers its waker for that
deadline under the same lock and clock reading as the check that found
the timeout due, so that it neither loops nor wakes its own task.

Evidence: in NVX's Alpine microVM, a guest enabled the RTC's periodic
interrupt at 8192 Hz through /dev/port and requested a capture, and
each restore timed the guest's first line after the capture: after the
real downtime, and, with OpenVMM's downtime-add-s hook, after a
simulated day and after 30 simulated days. With OpenVMM 60e160369, they
took 0.09 s, 30.8 s, and 921.7 s on nvx-kvm-test, 0.04 s, 30.7 s, and
917.3 s on nvx-mshv-test, and 0.15 s, 27.2 s, and 809.8 s on
nvx-whp-test. With OpenVMM built from the pin, they took 0.07 s,
0.09 s, and 0.11 s on nvx-kvm-test, 0.04 s, 0.04 s, and 0.06 s on
nvx-mshv-test, and 0.59 s, 0.18 s, and 0.20 s on nvx-whp-test, and
after each restore the guest found the RTC's periodic interrupt flag
set, and set again 10 ms after its read had cleared it.

Tests: with OpenVMM built from the pin, the microVM scenarios
guest-boot, lifecycle, interrupt-map, snapshot-core,
restore-processors, smp-snapshot, restore-downtime,
time-abi-conformance, and time-host-correction pass with 1 and 2
processors on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test. The
revision's vmcore, chipset, and chipset_legacy tests, 155 in all, pass
on Windows and on Linux, on nvx-kvm-test.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, the activation bound, and the
CPU profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Nov 13, 2026
The NVX pin moves to 5ca20ee08, position 268, 137 commits past the
mapped tip. Position 268, which microsoft/openvmm#5073 proposes for
feature/microvm/dev on its head, 60e160369, is unassigned. The Scope
also says that microsoft/openvmm#5072 merged position 267.

Position 268 makes VmTimerPeriodic skip at once the deadlines that VM
time passed, and register its next deadline under the timer state's
lock, in code that upstream main has, and adds tests, in a file that
PR10's position 019 and positions 156, 157, and 255 also change, none
of them in the code that it changes. It changes nothing that a mapped
PR adds. Replaying it on upstream main, 7d90c4104, applies without
conflicts, and vmcore type-checks there with all its targets and its
35 tests pass, so it can go upstream on its own.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) merged commit 5ca20ee into feature/microvm/dev Nov 13, 2026
73 of 74 checks passed
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-nvx Targets a branch under feature/microvm/.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant