Skip to content

openvmm: on KVM, a restored one-shot LAPIC timer can fire an extra time or drift #1366

Closed

Description

@tangruize

On KVM with the NVX time ABI, restoring a VP whose one-shot LAPIC timer has expired can give the guest one extra timer interrupt, and a running one-shot timer fires up to milliseconds early. Any guest that uses a one-shot LAPIC timer can see this; no special configuration is needed. Linux tolerates spurious clockevent interrupts, so the effect is low, but the restored VM is not the one that was captured. There are three cases:

  1. A timer that expired, and whose interrupt was delivered, before the capture. Its current count is 0 and its initial count is nonzero. KVM_SET_LAPIC re-arms it from that count, so it fires at once, a second time.
  2. A timer that expires during the downtime. Apic::advance_one_shot_timer (vmm_core/virt/src/x86/vp/snapshot.rs) sets the count to 0 and queues the vector in IRR. KVM_SET_LAPIC then re-arms the timer as in case 1, so a VP restored runnable with interrupts enabled takes the interrupt twice. A halted VP, or one with interrupts disabled, takes it once, because KVM merges both in IRR.
  3. Drift. The saved count is read after the capture anchor, and on restore KVM runs the timer from the VP restore's set_apic, before LapicAdvance::apply (vmm_core/src/partition_unit/vp_set/time_abi.rs) reads it back and subtracts the full downtime. Those windows are counted on top of the downtime, milliseconds at most, so the timer fires that much early.

Reproduction

Cases 1 and 2 were reproduced on KVM with a test (not attached) in which real guest code counts its timer interrupts after the real restore sequence. A one-shot timer that fired and was taken before the capture fired once more after the restore. One that expired during the downtime was delivered twice to a VP restored runnable with interrupts enabled. Case 3 is not reproduced: it would show as a one-shot timer, armed well ahead of the capture, firing earlier in guest TSC time than it was programmed to, by the capture's anchor-to-save window plus the restore's set_apic-to-anchor window.

Cause

KVM_SET_LAPIC derives a new deadline from the page's counts, and cannot express "expired, already delivered". OpenVMM's saved APIC page cannot tell an expired-and-delivered one-shot timer apart from one whose interrupt is still pending in KVM, and Apic::advance_one_shot_timer leaves TMICT set when the count is or becomes 0 (cases 1 and 2):

// vmm_core/virt/src/x86/vp/snapshot.rs
if registers.timer_ccr == 0 {
    return Ok(());
}
// ...
if ticks >= u64::from(registers.timer_ccr) {
    registers.timer_ccr = 0;
    self.queue_timer_interrupt(&mut registers);

LapicAdvance::apply subtracts the full downtime from the page read back from KVM, whose timer has been running since the VP restore set it (case 3):

// vmm_core/src/partition_unit/vp_set/time_abi.rs
let mut access = vp.access_state(Vtl::Vtl0);
let mut apic = access.apic().context("failed to read stopped LAPIC")?;
apic.advance_one_shot_timer(self.downtime_ns, self.apic_hz)?;
access
    .set_apic(&apic)

Fix

No patch is attached, because each option changes either guest-visible state or the restore order:

  • For cases 1 and 2, write the page with TMICT 0 when the count is 0, so that KVM does not re-arm the timer. The guest then reads TMICT as 0, not the value it wrote.
  • For case 3, account for the time KVM's timer runs outside the downtime: from the capture anchor to the save's read, and from the restore's first set_apic to the restore anchor; or set the timer only after the anchor.
  • Accept and document the behaviour (a known limit of the fix for restored non-Fixed APIC timers).

Related: #1365.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions