Skip to content

openvmm: RTC periodic catch-up stalls restore with billions of iterations #1164

Closed

Description

@tangruize

An RTC with periodic interrupts enabled can spend excessive time catching up after snapshot downtime. At 8192 Hz, 30 days of advancement leave roughly 21 billion missed periods, delaying restored execution while the device processes them individually.

In vm/vmcore/src/vmtime.rs, VmTimerPeriodic::poll_timeout advances the deadline only one period per iteration:

                let mut res = Poll::Pending;
                while let Poll::Ready(now) = self.vmtime.poll_timeout(cx) {
                    res = Poll::Ready(now);

                    let time = last_timeout.wrapping_add(period);
                    self.vmtime.set_timeout(time);
                    *last_timeout = time;
                }
                res

Every replacement deadline remains overdue until the loop traverses the entire backlog. The RTC only latches one periodic-interrupt flag, so visiting each missed deadline adds work without delivering additional useful interrupts.

Trigger

Enable periodic interrupts with RTC status A's rate selector 3, advance stopped VM time by a long snapshot downtime, and poll the restored RTC. The same helper also backs update-ended interrupts, although their one-second period produces a smaller backlog.

Fix

fix.patch skips whole periods arithmetically while preserving the existing 100 ns deadline grid and wraparound ordering. It registers the next wake without an unbounded loop when a short period expires during polling, and rounds sub-resolution periods up to 100 ns.

Catch-up still produces one pending interrupt rather than replaying every missed tick. The patch does not change the saved-state format or resolve the separate question of preserving timer phase across restore.

Activity

  1. tangruize commented on Nov 1, 2026

    @tangruize
    ContributorAuthor

    Proposed patch

    Patch against OpenVMM 998b9f2402050cd16d1faa897bc831ef1434be48. Save the contents below as fix.patch and run git apply fix.patch.

    fix.patch
    diff --git a/vm/devices/chipset/src/cmos_rtc.rs b/vm/devices/chipset/src/cmos_rtc.rs
    index ae091aec8..856568c0a 100644
    --- a/vm/devices/chipset/src/cmos_rtc.rs
    +++ b/vm/devices/chipset/src/cmos_rtc.rs
    @@ -1024,6 +1024,27 @@ mod tests {
         use local_clock::MockLocalClockAccessor;
         use test_with_tracing::test;
     
    +    #[test]
    +    fn rtc_periodic_catch_up_coalesces_interrupts_after_thirty_days() {
    +        let (mut pool, mut keeper, _clock, mut rtc) = new_test_rtc();
    +        let status_a =
    +            StatusRegA::from(rtc.state.cmos[CmosReg::STATUS_A]).with_periodic_timer_rate(3);
    +        rtc.set_cmos_byte(CmosReg::STATUS_A.0, status_a.into());
    +        let status_b =
    +            StatusRegB::from(rtc.state.cmos[CmosReg::STATUS_B]).with_irq_enable_periodic(true);
    +        rtc.set_cmos_byte(CmosReg::STATUS_B.0, status_b.into());
    +        pool.run_until(keeper.advance(Duration::from_secs(30 * 24 * 60 * 60)));
    +
    +        let mut cx = std::task::Context::from_waker(std::task::Waker::noop());
    +        rtc.poll_device(&mut cx);
    +        let status_c = StatusRegC::from(rtc.get_cmos_byte(CmosReg::STATUS_C.0));
    +        assert!(status_c.irq_periodic());
    +        assert!(status_c.irq_combined());
    +        rtc.poll_device(&mut cx);
    +        let status_c = StatusRegC::from(rtc.get_cmos_byte(CmosReg::STATUS_C.0));
    +        assert!(!status_c.irq_periodic());
    +    }
    +
         fn new_test_rtc() -> (
             pal_async::DefaultPool,
             vmcore::vmtime::VmTimeKeeper,
    diff --git a/vm/vmcore/src/vmtime.rs b/vm/vmcore/src/vmtime.rs
    index 7a5e82b6c..d2f42cbd7 100644
    --- a/vm/vmcore/src/vmtime.rs
    +++ b/vm/vmcore/src/vmtime.rs
    @@ -1002,8 +1002,10 @@ pub fn cancel(&mut self) {
         ///
         /// If the timer is currently running, the timer will be cancelled +
         /// restarted.
    +    /// Periods shorter than VM time's 100 ns resolution are rounded up.
         pub fn start(&mut self, period: Duration) {
             self.cancel();
    +        let period = period.max(Duration::from_nanos(100));
     
             let time = self.vmtime.now().wrapping_add(period);
             self.vmtime.set_timeout(time);
    @@ -1034,13 +1036,25 @@ pub fn poll_timeout(&mut self, cx: &mut Context<'_>) -> Poll<VmTime> {
                     ref mut last_timeout,
                     period,
                 } => {
    -                let mut res = Poll::Pending;
    -                while let Poll::Ready(now) = self.vmtime.poll_timeout(cx) {
    -                    res = Poll::Ready(now);
    -
    -                    let time = last_timeout.wrapping_add(period);
    +                let res = self.vmtime.poll_timeout(cx);
    +                if let Poll::Ready(now) = res {
    +                    // Coalesce missed periods without moving the timer's phase.
    +                    let period_100ns = period.as_nanos() / 100;
    +                    let elapsed_100ns = now.as_100ns().wrapping_sub(last_timeout.as_100ns());
    +                    let periods = u128::from(elapsed_100ns) / period_100ns + 1;
    +                    let time = VmTime::from_100ns(
    +                        last_timeout
    +                            .as_100ns()
    +                            .wrapping_add((periods * period_100ns) as u64),
    +                    );
                         self.vmtime.set_timeout(time);
                         *last_timeout = time;
    +                    // Register the next wake without spinning if a short period
    +                    // expires again while this poll is being handled.
    +                    if self.vmtime.poll_timeout(cx).is_ready() {
    +                        self.vmtime.set_timeout(time);
    +                        cx.waker().wake_by_ref();
    +                    }
                     }
                     res
                 }
    @@ -1062,6 +1076,77 @@ mod tests {
         use std::future::poll_fn;
         use std::time::Duration;
     
    +    #[async_test]
    +    async fn periodic_timer_subresolution_period_does_not_spin(driver: DefaultDriver) {
    +        let mut keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
    +        let source = keeper.builder().build(&driver).await.unwrap();
    +        let mut timer = super::VmTimerPeriodic::new(source.access("periodic"));
    +        timer.start(Duration::ZERO);
    +        keeper.advance(Duration::from_nanos(200)).await;
    +        assert_eq!(
    +            poll_fn(|cx| timer.poll_timeout(cx)).now_or_never(),
    +            Some(VmTime::from_100ns(2))
    +        );
    +        assert!(
    +            poll_fn(|cx| timer.poll_timeout(cx))
    +                .now_or_never()
    +                .is_none()
    +        );
    +    }
    +
    +    #[async_test]
    +    async fn periodic_timer_skips_thirty_days_without_changing_phase(driver: DefaultDriver) {
    +        let mut keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
    +        let source = keeper.builder().build(&driver).await.unwrap();
    +        let mut timer = super::VmTimerPeriodic::new(source.access("periodic"));
    +        let period = Duration::from_nanos(122_070);
    +        timer.start(period);
    +        keeper.advance(Duration::from_secs(30 * 24 * 60 * 60)).await;
    +
    +        let now = timer.vmtime.now();
    +        assert_eq!(
    +            poll_fn(|cx| timer.poll_timeout(cx)).now_or_never(),
    +            Some(now)
    +        );
    +        let next = timer.vmtime.get_timeout().unwrap();
    +        let period_100ns = (period.as_nanos() / 100) as u64;
    +        assert!(next.is_after(now));
    +        assert_eq!(next.as_100ns() % period_100ns, 0);
    +        assert_eq!(
    +            next.as_100ns() - now.as_100ns(),
    +            period_100ns - now.as_100ns() % period_100ns
    +        );
    +        assert!(
    +            poll_fn(|cx| timer.poll_timeout(cx))
    +                .now_or_never()
    +                .is_none()
    +        );
    +    }
    +
    +    #[async_test]
    +    async fn periodic_timer_catch_up_wraps_and_preserves_strict_deadline(driver: DefaultDriver) {
    +        let start = VmTime::from_100ns(u64::MAX - 30);
    +        let mut keeper = VmTimeKeeper::new(&driver, start);
    +        let source = keeper.builder().build(&driver).await.unwrap();
    +        let mut timer = super::VmTimerPeriodic::new(source.access("periodic"));
    +        timer.start(Duration::from_nanos(1000));
    +        keeper.advance(Duration::from_nanos(20_000)).await;
    +        let now = timer.vmtime.now();
    +        assert_eq!(
    +            poll_fn(|cx| timer.poll_timeout(cx)).now_or_never(),
    +            Some(now)
    +        );
    +        assert_eq!(
    +            timer.vmtime.get_timeout(),
    +            Some(now.wrapping_add(Duration::from_nanos(1000)))
    +        );
    +        assert!(
    +            poll_fn(|cx| timer.poll_timeout(cx))
    +                .now_or_never()
    +                .is_none()
    +        );
    +    }
    +
         #[async_test]
         async fn test_vmtime(driver: DefaultDriver) {
             let mut keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
  2. q5438722 commented on Nov 3, 2026

    @q5438722

    An RTC with periodic interrupts enabled can spend excessive time catching up after snapshot downtime. At 8192 Hz, 30 days of advancement leave roughly 21 billion missed periods, delaying restored execution while the device processes them individually.

    In vm/vmcore/src/vmtime.rs, VmTimerPeriodic::poll_timeout advances the deadline only one period per iteration:

                let mut res = Poll::Pending;
                while let Poll::Ready(now) = self.vmtime.poll_timeout(cx) {
                    res = Poll::Ready(now);
    
                    let time = last_timeout.wrapping_add(period);
                    self.vmtime.set_timeout(time);
                    *last_timeout = time;
                }
                res
    

    Every replacement deadline remains overdue until the loop traverses the entire backlog. The RTC only latches one periodic-interrupt flag, so visiting each missed deadline adds work without delivering additional useful interrupts.

    Trigger

    Enable periodic interrupts with RTC status A's rate selector 3, advance stopped VM time by a long snapshot downtime, and poll the restored RTC. The same helper also backs update-ended interrupts, although their one-second period produces a smaller backlog.

    Fix

    fix.patch skips whole periods arithmetically while preserving the existing 100 ns deadline grid and wraparound ordering. It registers the next wake without an unbounded loop when a short period expires during polling, and rounds sub-resolution periods up to 100 ns.

    Catch-up still produces one pending interrupt rather than replaying every missed tick. The patch does not change the saved-state format or resolve the separate question of preserving timer phase across restore.

    just add one reproduction test case:

    use pal_async::DefaultPool;
    use std::future::poll_fn;
    use std::sync::mpsc;
    use std::time::Duration;
    use vmcore::vmtime::SavedState;
    use vmcore::vmtime::VmTime;
    use vmcore::vmtime::VmTimeKeeper;
    use vmcore::vmtime::VmTimerPeriodic;
    
    /// Runs `f` on its own thread and fails if it does not finish within 10s:
    /// a spinning `poll_timeout` blocks its executor thread for good.
    fn within_10s(what: &'static str, f: impl FnOnce() + Send + 'static) {
        let (done, finished) = mpsc::channel();
        std::thread::spawn(move || {
            f();
            let _ = done.send(());
        });
        finished
            .recv_timeout(Duration::from_secs(10))
            .unwrap_or_else(|_| {
                panic!("{what}: VmTimerPeriodic::poll_timeout did not return within 10s")
            });
    }
    
    /// Starts VM time at 0 and a periodic timer with `period`, then waits for
    /// the timer to fire twice.
    fn fires_twice(period: Duration) {
        DefaultPool::run_with(async |driver| {
            let mut keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
            let source = keeper.builder().build(&driver).await.unwrap();
            let mut timer = VmTimerPeriodic::new(source.access("periodic"));
            keeper.start().await;
            timer.start(period);
            poll_fn(|cx| timer.poll_timeout(cx)).await;
            poll_fn(|cx| timer.poll_timeout(cx)).await;
            keeper.stop().await;
        });
    }
    
    /// Control: an ordinary period.
    #[test]
    fn periodic_timer_fires() {
        within_10s("1ms period", || fires_twice(Duration::from_millis(1)));
    }
    
    /// A period shorter than the 100ns resolution of VM time.
    #[test]
    fn sub_resolution_period_does_not_spin() {
        within_10s("50ns period", || fires_twice(Duration::from_nanos(50)));
    }
    
    /// VM time moves far ahead of the timer's last deadline, as a restore or a
    /// downtime advance does: the missed periods must be skipped at once.
    #[test]
    fn missed_periods_are_skipped_at_once() {
        within_10s("catch-up after 2^50 units", || {
            DefaultPool::run_with(async |driver| {
                let mut keeper = VmTimeKeeper::new(&driver, VmTime::from_100ns(0));
                let source = keeper.builder().build(&driver).await.unwrap();
                let mut timer = VmTimerPeriodic::new(source.access("periodic"));
                keeper.start().await;
                timer.start(Duration::from_millis(1));
                poll_fn(|cx| timer.poll_timeout(cx)).await;
                keeper.stop().await;
    
                // ~3.6 years later, that is ~10^11 periods of 1ms.
                keeper
                    .restore(SavedState::from_vmtime(VmTime::from_100ns(1 << 50)))
                    .await;
                keeper.start().await;
                let now = poll_fn(|cx| timer.poll_timeout(cx)).await;
                assert!(!now.is_before(VmTime::from_100ns(1 << 50)));
    
                // And it keeps firing.
                poll_fn(|cx| timer.poll_timeout(cx)).await;
                keeper.stop().await;
            });
        });
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions