Repository navigation
Description
On KVM with the NVX time ABI, restoring a VP whose one-shot LAPIC timer has expired can give the guest one extra timer interrupt, and a running one-shot timer fires up to milliseconds early. Any guest that uses a one-shot LAPIC timer can see this; no special configuration is needed. Linux tolerates spurious clockevent interrupts, so the effect is low, but the restored VM is not the one that was captured. There are three cases:
- A timer that expired, and whose interrupt was delivered, before the capture. Its current count is 0 and its initial count is nonzero.
KVM_SET_LAPICre-arms it from that count, so it fires at once, a second time. - A timer that expires during the downtime.
Apic::advance_one_shot_timer(vmm_core/virt/src/x86/vp/snapshot.rs) sets the count to 0 and queues the vector in IRR.KVM_SET_LAPICthen re-arms the timer as in case 1, so a VP restored runnable with interrupts enabled takes the interrupt twice. A halted VP, or one with interrupts disabled, takes it once, because KVM merges both in IRR. - Drift. The saved count is read after the capture anchor, and on restore KVM runs the timer from the VP restore's
set_apic, beforeLapicAdvance::apply(vmm_core/src/partition_unit/vp_set/time_abi.rs) reads it back and subtracts the full downtime. Those windows are counted on top of the downtime, milliseconds at most, so the timer fires that much early.
Reproduction
Cases 1 and 2 were reproduced on KVM with a test (not attached) in which real guest code counts its timer interrupts after the real restore sequence. A one-shot timer that fired and was taken before the capture fired once more after the restore. One that expired during the downtime was delivered twice to a VP restored runnable with interrupts enabled. Case 3 is not reproduced: it would show as a one-shot timer, armed well ahead of the capture, firing earlier in guest TSC time than it was programmed to, by the capture's anchor-to-save window plus the restore's set_apic-to-anchor window.
Cause
KVM_SET_LAPIC derives a new deadline from the page's counts, and cannot express "expired, already delivered". OpenVMM's saved APIC page cannot tell an expired-and-delivered one-shot timer apart from one whose interrupt is still pending in KVM, and Apic::advance_one_shot_timer leaves TMICT set when the count is or becomes 0 (cases 1 and 2):
// vmm_core/virt/src/x86/vp/snapshot.rs
if registers.timer_ccr == 0 {
return Ok(());
}
// ...
if ticks >= u64::from(registers.timer_ccr) {
registers.timer_ccr = 0;
self.queue_timer_interrupt(&mut registers);LapicAdvance::apply subtracts the full downtime from the page read back from KVM, whose timer has been running since the VP restore set it (case 3):
// vmm_core/src/partition_unit/vp_set/time_abi.rs
let mut access = vp.access_state(Vtl::Vtl0);
let mut apic = access.apic().context("failed to read stopped LAPIC")?;
apic.advance_one_shot_timer(self.downtime_ns, self.apic_hz)?;
access
.set_apic(&apic)Fix
No patch is attached, because each option changes either guest-visible state or the restore order:
- For cases 1 and 2, write the page with TMICT 0 when the count is 0, so that KVM does not re-arm the timer. The guest then reads TMICT as 0, not the value it wrote.
- For case 3, account for the time KVM's timer runs outside the downtime: from the capture anchor to the save's read, and from the restore's first
set_apicto the restore anchor; or set the timer only after the anchor. - Accept and document the behaviour (a known limit of the fix for restored non-Fixed APIC timers).
Related: #1365.
Activity
virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending
vmm_core, virt: run a restored LAPIC timer from the time ABI anchors
vmm_core, virt: run a restored LAPIC timer from the time ABI anchors
virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending
vmm_core, virt: run a restored LAPIC timer from the time ABI anchors
openvmm: pin captures that keep a taken LAPIC timer from firing again
openvmm: pin LAPIC timer restores without drift or a downtime refire
doc: specify how a restore runs a one-shot LAPIC timer
virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending
vmm_core, virt: run a restored LAPIC timer from the time ABI anchors
virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again
virt_kvm: do not fire an expired LAPIC timer whose interrupt is pending
vmm_core, virt: run a restored LAPIC timer from the time ABI anchors
virt_kvm: keep a restore from firing a taken one-shot LAPIC timer again