Skip to content

openvmm: skip a periodic timer's missed periods at once - #1565

Merged
Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1164-periodic-timer-catch-up
Nov 13, 2026
Merged

Pedro Henrique Penna (ppenna) merged 2 commits into
devfrom
agents/issue-1164-periodic-timer-catch-up

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1164

What and why

A restore after a long downtime stalled when the guest had enabled the CMOS RTC's periodic interrupt. At 8192 Hz, status A's rate selector 3, the restored guest ran again only some 15 minutes after a restore from a 30-day downtime, on KVM, MSHV, and WHP alike, and some 30 s after a 1-day one.

Root cause, in OpenVMM: VmTimerPeriodic::poll_timeout in vm/vmcore/src/vmtime.rs moved its deadline one period per loop iteration until it passed the current VM time, and returned ready once for all the deadlines that it passed. A restore's downtime advance moves VM time past the deadline by the whole downtime, some 21 billion periods of 122 µs in 30 days, and the RTC's state unit polls the device, under its lock, as the restored VM starts. The RTC's update-ended interrupt, at 1 Hz, uses the same timer, and a period below VM time's 100 ns resolution counted as zero, so the loop would have spun forever once such a timer fired; the RTC never asks for one.

NVX's own guests leave the RTC's periodic interrupt off: their kernel has no RTC driver (CONFIG_RTC_CLASS is not set), which is why the restore-downtime scenario, which restores after a simulated year, never met the stall. A guest that programs the RTC through /dev/port or ioperm does.

This builds on #1563, which has merged; its pin, 60e160369, is the head of feature/microvm/dev.

The pin, microsoft/openvmm#5073 at 5ca20ee081c255bed107e58b6dc5b5b769b6101d, adds one commit on 60e160369:

  • 5ca20ee08 vmcore: skip a periodic timer's missed periods at once. A running timer's poll moves its timeout to the first of its deadlines after now in one step, from the number of whole periods since the timeout, and registers its waker for that deadline in a new TimerState::poll_periodic_timeout, under the timer state's lock and with the same clock reading as the check that found the timeout due. A ready poll so accounts for every deadline up to now, and the deadlines stay a whole number of periods after the timer's start, as before. start counts a period shorter than 100 ns as 100 ns, and caps one at i64::MAX units, the longest interval that VM time's circular order puts after a time. Its tests, in a new periodic module of vmtime's tests, catch an 8192 Hz timer up across 30 days of VM time, run a 50 ns timer with VM time stopped and running, and catch up across VM time's wrap onto a deadline.

The NVX commits:

  • c13e733cf7f openvmm: pin the periodic timer's one-step catch-up. It changes only the gitlink, from 60e160369 to 5ca20ee08.
  • 7287f794836 doc: map the periodic timer's one-step catch-up in the OpenVMM roadmap. Position 268, unassigned.

Decisions:

  • A deadline at exactly now counts with the ones before it, as in the patch attached to the issue, so that the timer never leaves a deadline due once its poll returns. Catch-up still delivers one periodic interrupt for all the missed periods, as before: the RTC latches one flag.
  • The patch attached to the issue registers the new deadline with a second poll, and wakes its own task when that deadline has passed already. With periods near 100 ns that self-wake repeats, and ArcMutexChipsetDeviceUnit::run polls a device again at once when its event is signaled, so the unit's task would never yield. Registering the deadline under the timer state's lock leaves no such window, so the poll neither loops nor wakes its own task.
  • The tests move VM time with VmTimeKeeper::restore rather than the chain's advance, which reset VM time the same way, so that the commit applies to upstream main on its own.
  • No saved-state field, snapshot format, contract, or version changes.

Not in this change, found while working on it:

  1. A periodic timer's phase across a restore. The RTC starts its timers afresh when it restores, as before; the issue leaves this aside too.
  2. pal_async's thread-pool timer. Timer::set_deadline in support/pal/pal_async/src/windows/tp.rs panics for a deadline that has passed, which VM time's timers can hand it, as TimerState::poll_timeout already does with a timeout at exactly now. OpenVMM runs VM time on VmTaskDrivers over DefaultDriver, whose IOCP and epoll timers accept such deadlines, so this is not reachable there.
  3. scripts/nvx_hosts.py run from a Windows controller. SubprocessRunner writes the script through a text-mode pipe, which turns its line feeds into CRLF, so a bash script fails on a Linux host (set: pipefail\r: invalid option name). The Linux runs below therefore passed their bash scripts through nvx_hosts.py exec, as base64 decoded into a temporary file.

Validation

Local checks at this head, 7287f794836 (Windows):

  • python scripts/openvmm_pin.py check-branch --topic --target origin/agents/issue-1151-rtc-wide-reads: passed (2 signed commits, no coordinated files).
  • python scripts/openvmm_pin.py check-roadmap --worktree: passed (268 positions, 277 references).
  • python scripts/openvmm_pin.py source 5ca20ee081c255bed107e58b6dc5b5b769b6101d --pr 5073: good signature, and the revision is the pull request's head.
  • python scripts/nvx.py verify: passed.
  • python scripts/local_gates.py --base origin/dev --only test-changes markdown-links test-layout openvmm-vectors openvmm-cpu-profiles --run: passed. That is check_test_changes.py (0 changes to existing tests), check_markdown_links.py, check_test_layout.py, and the ignored openvmm_vectors (4) and openvmm_cpu_profiles (1) tests of nvx-rs.
  • The nvx-cli gate, its commands after pip install, whose one requirement, PyYAML, is installed, with NVX_RS set to a release nvx-rs built here and Git's sh on PATH: passed. The unit tests ran 2198 tests (88 skipped) and test_adversarial.py 84 (11 skipped).
  • The openvmm-ci gate, python scripts/nvx.py check-openvmm-ci --revision 5ca20ee081c255bed107e58b6dc5b5b769b6101d: OpenVMM's CI of the pin, run 43368520199, passed every job that NVX requires; see the CI paragraph below.

OpenVMM package checks at 5ca20ee08, with Rust 1.97.1: on Windows, for vmcore, cargo check, cargo clippy --all-targets (clean apart from the environment's LNK4199 linker notices), cargo doc --no-deps, and cargo nextest run --profile agent (40 passed), and cargo nextest run --profile agent -p chipset -p chipset_legacy (115 passed), then cargo xtask fmt --fix (no changes); on Linux, on nvx-kvm-test, cargo nextest run --profile agent -p vmcore -p chipset -p chipset_legacy (155 passed) and cargo test -p vmcore --lib (40 passed). The three new tests passed 200 stress iterations on each. With 60e160369's code, test_skips_missed_periods_at_once and test_rounds_short_periods_up do not complete within their 10 s watchdog, and test_catch_up_across_the_wrap leaves the timeout at now, VmTime(170) where VmTime(180) is expected. Replayed on upstream main, 7d90c4104, the commit applies without conflicts, and vmcore type-checks there with all its targets and its 35 tests pass.

Reproduction, in NVX's Alpine microVM: the guest enabled the RTC's periodic interrupt at 8192 Hz through /dev/port, writing status A 0x23 and setting status B's periodic interrupt enable, and requested a capture. Each restore of that snapshot timed the guest's first line after the capture: after the real downtime, then with --x-time-abi-test-hook downtime-add-s=86400 and downtime-add-s=2592000. Every restore then found status C's periodic and IRQ flags set, 0xc0, and set again 10 ms after the read that cleared them:

Profile Backend Host type OpenVMM Real downtime 1 simulated day 30 simulated days
nvx-kvm-test kvm virtual-machine before: 60e160369 0.09 s 30.8 s 921.7 s
nvx-mshv-test mshv virtual-machine before: 60e160369 0.04 s 30.7 s 917.3 s
nvx-whp-test whp virtual-machine before: 60e160369 0.15 s 27.2 s 809.8 s
nvx-kvm-test kvm virtual-machine after: 5ca20ee08 0.07 s 0.09 s 0.11 s
nvx-mshv-test mshv virtual-machine after: 5ca20ee08 0.04 s 0.04 s 0.06 s
nvx-whp-test whp virtual-machine after: 5ca20ee08 0.59 s 0.18 s 0.20 s

The "before" runs used a temporary worktree of 0e159a5808e, #1563's head, on each host, with the guest artifacts of the release v0.1.0-dev.8e9e9eb74cf2, which nvx-rs download installed, and OpenVMM built from 60e160369 (clean) by nvx.py build-openvmm --backend BACKEND: 35af18e3 on KVM, 1d54e162 on MSHV (musl), and 89ffd0af on WHP. The "after" runs used a temporary worktree of 52ca738f654, whose tree is this head's, with the same release's guest artifacts and OpenVMM built from 5ca20ee08 (clean) the same way: f260557d on KVM, 1237bbd3 on MSHV (musl), and 1c68b25e on WHP.

The guest script, which the driver typed into the guest's console with the harness's OpenvmmProcess
set -eu
reg() {
    printf "\\$(printf '%03o' "$1")" | dd of=/dev/port bs=1 seek=112 count=1 conv=notrunc 2>/dev/null
    dd if=/dev/port bs=1 skip=113 count=1 2>/dev/null | od -An -tu1 | tr -d '[:space:]'
}
set_reg() {
    printf "\\$(printf '%03o' "$1")" | dd of=/dev/port bs=1 seek=112 count=1 conv=notrunc 2>/dev/null
    printf "\\$(printf '%03o' "$2")" | dd of=/dev/port bs=1 seek=113 count=1 conv=notrunc 2>/dev/null
}
set_reg 10 35
set_reg 11 $(($(reg 11) | 64))
reg 12 >/dev/null
sleep 0.01
echo "NVX-1164-VALUES phase=capture a=$(reg 10) b=$(reg 11) c=$(reg 12)"
echo NVX-1164-ARMED
nvx-snapshot
echo NVX-1164-RESTORED
c1=$(reg 12)
sleep 0.01
c2=$(reg 12)
echo "NVX-1164-VALUES phase=restore a=$(reg 10) b=$(reg 11) c1=$c1 c2=$c2"
echo NVX-1164-DONE
nvx-exit 0

Remote runs at the head's tree, in the temporary worktree of 52ca738f654, with OpenVMM built from 5ca20ee08. The scenarios are guest-boot, lifecycle, which reads the RTC's status B, interrupt-map, snapshot-core, restore-processors, smp-snapshot, restore-downtime, time-abi-conformance, and time-host-correction, with --processors 1 2:

Profile Backend Host type Check Result
nvx-kvm-test kvm virtual-machine the scenarios, OpenVMM binary f260557d passed, in 238 s
nvx-mshv-test mshv virtual-machine the scenarios, OpenVMM binary 1237bbd3 (musl) passed, in 227 s
nvx-whp-test whp virtual-machine the scenarios, OpenVMM binary 1c68b25e passed, in about 244 s

No bare-metal profile is configured; the change is algorithmic and was reproduced and fixed on all three backends, so bare-metal hosts and the remaining scenarios are left to CI.

CI at this head, 7287f794836: passed, run 43371731765, with OpenVMM CI of the pinned revision, which found that OpenVMM's CI of 5ca20ee08, run 43368520199, passed every job that NVX requires. That run's run vmm-tests [x64-windows-amd-snp], which NVX does not require, failed in four of its 114 tests. multiarch::vmgs::hyperv_openhcl_uefi_x64_windows_datacenter_core_2025_x64_prepped_snp_vmgstool_copy_igvmfile failed as it did at 60e160369, in microsoft/openvmm#5072's run 43352068114: the runner's Hyper-V failed to load the CVM firmware vmfirmwarecvm.dll (0x80070002). multiarch::tpm::…_snp_ak_cert_retry, multiarch::tpm::…_snp_hw_ak_stable, and multiarch::vmbus_relay::…_snp_vmbus_relay timed out after about 12 minutes each. The same job fails with timeouts in the same TPM and VMBus relay tests in the same hours on OpenVMM pull requests that lack this change, such as run 43330262358 (vmbus_relay::…_snp_vmbus_relay, 757 s), run 43334289464 (tpm::…_snp_ak_cert_cache, 719 s), and run 43342660705 (tpm::…_snp_cvm_guest, 792 s), so it predates this change.

No existing test changes, so no tests-modified label.

OpenVMM's VmTimerPeriodic moved its deadline one period per iteration
until it passed the current VM time, so after a restore whose downtime
moved VM time far past the deadline, the poll iterated once for every
missed period (#1164). The RTC uses it for its periodic interrupt, at
up to 8192 Hz, and for its update-ended interrupt, at 1 Hz, and the
RTC's state unit polls the device as the restored VM starts, so a guest
that had enabled the RTC's periodic interrupt at 8192 Hz ran again only
some 15 minutes after a restore from a 30-day downtime.

Pin 5ca20ee081c255bed107e58b6dc5b5b769b6101d, the head of
microsoft/openvmm#5073, one commit on 60e160369, the head of
feature/microvm/dev:

- 5ca20ee08 vmcore: skip a periodic timer's missed periods at once

A running timer's poll now moves its timeout to the first of its
deadlines after now in one step, and registers its waker for that
deadline under the same lock and clock reading as the check that found
the timeout due, so that it neither loops nor wakes its own task.

Evidence: in NVX's Alpine microVM, a guest enabled the RTC's periodic
interrupt at 8192 Hz through /dev/port and requested a capture, and
each restore timed the guest's first line after the capture: after the
real downtime, and, with OpenVMM's downtime-add-s hook, after a
simulated day and after 30 simulated days. With OpenVMM 60e160369, they
took 0.09 s, 30.8 s, and 921.7 s on nvx-kvm-test, 0.04 s, 30.7 s, and
917.3 s on nvx-mshv-test, and 0.15 s, 27.2 s, and 809.8 s on
nvx-whp-test. With OpenVMM built from the pin, they took 0.07 s,
0.09 s, and 0.11 s on nvx-kvm-test, 0.04 s, 0.04 s, and 0.06 s on
nvx-mshv-test, and 0.59 s, 0.18 s, and 0.20 s on nvx-whp-test, and
after each restore the guest found the RTC's periodic interrupt flag
set, and set again 10 ms after its read had cleared it.

Tests: with OpenVMM built from the pin, the microVM scenarios
guest-boot, lifecycle, interrupt-map, snapshot-core,
restore-processors, smp-snapshot, restore-downtime,
time-abi-conformance, and time-host-correction pass with 1 and 2
processors on nvx-kvm-test, nvx-mshv-test, and nvx-whp-test. The
revision's vmcore, chipset, and chipset_legacy tests, 155 in all, pass
on Windows and on Linux, on nvx-kvm-test.

NVX needs no coordinated change: the control-session vectors, the
source excerpts, the contract versions, the activation bound, and the
CPU profiles that nvx-rs copies from OpenVMM match the revision's.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX pin moves to 5ca20ee08, position 268, 137 commits past the
mapped tip. Position 268, which microsoft/openvmm#5073 proposes for
feature/microvm/dev on its head, 60e160369, is unassigned. The Scope
also says that microsoft/openvmm#5072 merged position 267.

Position 268 makes VmTimerPeriodic skip at once the deadlines that VM
time passed, and register its next deadline under the timer state's
lock, in code that upstream main has, and adds tests, in a file that
PR10's position 019 and positions 156, 157, and 255 also change, none
of them in the code that it changes. It changes nothing that a mapped
PR adds. Replaying it on upstream main, 7d90c4104, applies without
conflicts, and vmcore type-checks there with all its targets and its
35 tests pass, so it can go upstream on its own.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The OpenVMM pin and roadmap updates have no active review issues.

No open issues 🎉

What changed in this PR

This PR addresses #1164 by pinning an OpenVMM fix that skips missed periodic timer deadlines in one step, avoiding long restore stalls when RTC periodic interrupts are enabled.

Changes:

  • Advances the OpenVMM pin to 5ca20ee08.
  • Maps the fix to roadmap position 268 and updates the chain counts.
  • Documents the fix’s scope and independent upstream applicability.
File Description
doc/​openvmm-upstream-roadmap.md Records the new pin, position 268, updated commit counts, and upstream applicability.

Review effort: Balanced

@ppenna
Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 13, 2026
Merged via the queue into dev with commit 8bb7d53 Nov 13, 2026
49 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the agents/issue-1164-periodic-timer-catch-up branch November 13, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openvmm: RTC periodic catch-up stalls restore with billions of iterations

2 participants