Skip to content

openvmm: KVM_SET_LAPIC fires an expired, already delivered one-shot timer again #1390

Open

Description

@tangruize

KVM_GET_LAPIC saves a one-shot LAPIC timer that expired and whose interrupt the guest already took with the same bytes as one that expired while KVM had not yet injected its interrupt: TMICT nonzero, current count 0, the vector clear in IRR and ISR. Since Linux 6.3, KVM_SET_LAPIC fires such a timer at once, which the second case needs, so in the first case the restored guest takes a timer interrupt it never programmed. This is a limit of the uAPI in every current kernel, not a one-line bug: an exact restore needs an opt-in change. It is the KVM half of #1366.

Affected host kernels

Host kernel KVM_SET_LAPIC of a one-shot timer with TMICT nonzero and current count 0
6.3 and later (8e6ed96cdd50), 7.3 included fires it at once: an extra interrupt if it was already delivered
5.8 to 6.2 restarts it for a full TMICT period: an extra, late interrupt

TSC-deadline mode is exact, since MSR_IA32_TSC_DEADLINE reads 0 once the interrupt has been injected. Periodic mode coalesces expiries by design.

Reproduction

Arm a one-shot timer in a guest, let it take and EOI the interrupt, stop the vCPU, and save and restore its LAPIC with KVM_GET_LAPIC and KVM_SET_LAPIC. Expected: no further timer interrupt until the guest reprograms the timer. Actual: one more. #1366 shows this through OpenVMM's restore on KVM.

Cause

kvm_apic_set_state cancels the timer and restarts it from the saved current count (__start_apic_timer(apic, APIC_TMCCT)), and a one-shot timer whose count is 0 gets a deadline of now:

// arch/x86/kvm/lapic.c (Linux 7.3, set_target_expiration)
	if (apic_lvtt_period(apic) || apic_lvtt_oneshot(apic)) {
		if (unlikely(count_reg != APIC_TMICT)) {
			deadline = tmict_to_ns(apic,
				     kvm_lapic_get_reg(apic, count_reg));
			if (unlikely(deadline <= 0)) {
				if (apic_lvtt_period(apic))
					deadline = apic->lapic_timer.period;
				else
					deadline = 0;
			}

KVM's only marker of a delivered timer interrupt is lapic_timer.target_expiration, which it zeroes on injection and does not export. KVM_GET_LAPIC also does not move an expiry that KVM recorded but has not injected into the exported IRR: one in lapic_timer.pending, or a VMX preemption timer that expired while the vCPU was stopped. So KVM_SET_LAPIC cannot tell the two states apart, and firing at once is the choice that loses no interrupt. A KVM_SET_LAPIC that stopped firing such a timer would lose the interrupt of every snapshot taken in the second state by a kernel that does not report it in the IRR, which is every kernel today; so an exact restore has to be opt-in.

Fix

KVM has no fix yet. An RFC sketch is attached below; it has not been sent upstream. It adds a quirk, KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE (bit 12 of KVM_CAP_DISABLE_QUIRKS2), enabled by default, which keeps today's behaviour. A VMM that disables it gets an exact restore: KVM_GET_LAPIC sets the timer vector in the exported IRR for an expired one-shot timer whose interrupt is not yet injected, and KVM_SET_LAPIC leaves a one-shot timer with count 0 expired instead of firing it. OpenVMM would disable the quirk where KVM offers it. The sketch changes 4 files (+70/−2) and applies to kvm-x86 next and v7.3; it is compile-tested only.

Related: #1366, #1365.

Activity

  1. tangruize commented on Nov 8, 2026

    @tangruize
    ContributorAuthor

    RFC sketch for KVM

    An RFC for discussion, not a patch to apply as is: it adds KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE. Against kvm-x86 next dea4f6380f4ec96e8f3b0aaf2343764bd0d25ef4; it also applies to v7.3. Compile-tested only, and not yet sent upstream.

    linux.patch
    diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
    index ddc61ea36ac5..61df7093f54a 100644
    --- a/Documentation/virt/kvm/api.rst
    +++ b/Documentation/virt/kvm/api.rst
    @@ -8783,6 +8783,25 @@ KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT        By default, KVM for nested SVM guests
                                                disabled for L2, L1 and L2 continue
                                                to share the IA32_PAT MSR regardless
                                                of the quirk setting.
    +
    +KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE         By default, KVM_SET_LAPIC fires a
    +                                           one-shot APIC timer whose current
    +                                           count (TMCCT) is 0 while its initial
    +                                           count (TMICT) is not, because
    +                                           KVM_GET_LAPIC does not tell whether
    +                                           the interrupt of such an expired
    +                                           timer has been injected. A guest
    +                                           may thus take a spurious timer
    +                                           interrupt after a restore. When
    +                                           this quirk is disabled,
    +                                           KVM_GET_LAPIC reports the interrupt
    +                                           of an expired one-shot timer that
    +                                           KVM has not injected yet as pending
    +                                           in the IRR, and KVM_SET_LAPIC leaves
    +                                           a one-shot timer whose TMCCT is 0
    +                                           expired. Userspace must disable the
    +                                           quirk both when saving and when
    +                                           restoring LAPIC state.
     ========================================   ================================================
     
     7.32 KVM_CAP_MAX_VCPU_ID
    diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
    index 1585ec804066..a14229b78b2f 100644
    --- a/arch/x86/include/uapi/asm/kvm.h
    +++ b/arch/x86/include/uapi/asm/kvm.h
    @@ -478,6 +478,7 @@ struct kvm_sync_regs {
     #define KVM_X86_QUIRK_IGNORE_GUEST_PAT		(1 << 9)
     #define KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM (1 << 10)
     #define KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT	(1 << 11)
    +#define KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE	(1 << 12)
     
     #define KVM_STATE_NESTED_FORMAT_VMX	0
     #define KVM_STATE_NESTED_FORMAT_SVM	1
    diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
    index ecf611845fb1..a4dd338cf6ba 100644
    --- a/arch/x86/kvm/lapic.c
    +++ b/arch/x86/kvm/lapic.c
    @@ -2151,6 +2151,23 @@ static void update_target_expiration(struct kvm_lapic *apic, uint32_t old_diviso
     	apic->lapic_timer.target_expiration = ktime_add_ns(now, ns_remaining_new);
     }
     
    +/*
    + * Leave a restored one-shot timer whose current count is zero expired, i.e.
    + * don't fire it again, unless the quirk says otherwise.  With the quirk
    + * disabled, KVM_GET_LAPIC reports the interrupt of an expired timer in the
    + * IRR if the interrupt has yet to be injected.
    + */
    +static bool apic_restore_expired_oneshot(struct kvm_lapic *apic)
    +{
    +	if (kvm_check_has_quirk(apic->vcpu->kvm,
    +				KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE))
    +		return false;
    +
    +	apic->lapic_timer.tscdeadline = 0;
    +	apic->lapic_timer.target_expiration = 0;
    +	return true;
    +}
    +
     static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg)
     {
     	ktime_t now;
    @@ -2176,6 +2193,8 @@ static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg)
     			if (unlikely(deadline <= 0)) {
     				if (apic_lvtt_period(apic))
     					deadline = apic->lapic_timer.period;
    +				else if (apic_restore_expired_oneshot(apic))
    +					return false;
     				else
     					deadline = 0;
     			}
    @@ -3299,15 +3318,43 @@ static int kvm_apic_state_fixup(struct kvm_vcpu *vcpu,
     	return 0;
     }
     
    +/*
    + * Unless the quirk says otherwise, report the interrupt of an expired one-shot
    + * timer that has yet to be injected as pending in the IRR, as the APIC accepts
    + * the interrupt when the count reaches zero.  The expiry may still be recorded
    + * only in lapic_timer.pending, or be in a hypervisor timer, which runs only in
    + * guest mode.  A timer whose count already reads zero, i.e. that expires in
    + * less than one tick, is treated as expired.
    + */
    +static void kvm_apic_get_expired_oneshot(struct kvm_lapic *apic, char *regs,
    +					 u32 tmcct)
    +{
    +	u32 lvtt = kvm_lapic_get_reg(apic, APIC_LVTT);
    +
    +	if (kvm_check_has_quirk(apic->vcpu->kvm,
    +				KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE))
    +		return;
    +
    +	if (!apic_lvtt_oneshot(apic) || !kvm_lapic_get_reg(apic, APIC_TMICT) ||
    +	    !apic->lapic_timer.target_expiration || tmcct)
    +		return;
    +
    +	if (kvm_apic_hw_enabled(apic) && !(lvtt & APIC_LVT_MASKED))
    +		apic_set_vector(lvtt & APIC_VECTOR_MASK, regs + APIC_IRR);
    +}
    +
     int kvm_apic_get_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s)
     {
    +	u32 tmcct = __apic_read(vcpu->arch.apic, APIC_TMCCT);
    +
     	memcpy(s->regs, vcpu->arch.apic->regs, sizeof(*s));
     
     	/*
     	 * Get calculated timer current count for remaining timer period (if
     	 * any) and store it in the returned register set.
     	 */
    -	apic_set_reg(s->regs, APIC_TMCCT, __apic_read(vcpu->arch.apic, APIC_TMCCT));
    +	apic_set_reg(s->regs, APIC_TMCCT, tmcct);
    +	kvm_apic_get_expired_oneshot(vcpu->arch.apic, s->regs, tmcct);
     
     	return kvm_apic_state_fixup(vcpu, s, false);
     }
    diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
    index 32563a91fba9..738a87a806e5 100644
    --- a/arch/x86/kvm/x86.c
    +++ b/arch/x86/kvm/x86.c
    @@ -146,7 +146,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host);
     	 KVM_X86_QUIRK_STUFF_FEATURE_MSRS |	\
     	 KVM_X86_QUIRK_IGNORE_GUEST_PAT |	\
     	 KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM |	\
    -	 KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT)
    +	 KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT |	\
    +	 KVM_X86_QUIRK_LAPIC_ONESHOT_REFIRE)
     
     #define KVM_X86_CONDITIONAL_QUIRKS		\
     	(KVM_X86_QUIRK_CD_NW_CLEARED |		\
  2. ppenna commented on Nov 12, 2026

    @ppenna
    Contributor

    Status after #1514, merged into dev as f7720fb6e8c and pinning microsoft/openvmm#5055 (c72fc56a3). Together with #1505, which closed #1366, it does everything that OpenVMM can do here without a KVM change:

    • Delivering held expiries. At a stop, OpenVMM puts an expiry that KVM still holds into IRR. It does this with one KVM_RUN that a signal ends before the guest is entered.
    • Telling the two states apart. Each one-shot timer that has expired then waits up to 20 ms for a late expiry. A timer whose interrupt the guest took is saved with TMICT 0, and one whose interrupt is still pending is saved with its vector in IRR, so KVM_SET_LAPIC fires neither again.

    This issue stays open for the two limits that only KVM's opt-in quirk, as sketched above, can lift:

    1. An expiry that KVM records more than 20 ms after the timer's count reads 0 is saved as taken, and the restore does not deliver it.
    2. On hosts whose KVM may run the LAPIC timer on the VMX preemption timer, a taken timer still fires once more after a restore. These are Intel hosts whose kvm_intel preemption_timer parameter is not N or cannot be read, such as bare-metal Intel hosts. AMD hosts, and the nested KVM of the CI runners and the test VM, are not affected.

    Section 3.2.5 of the design documents both.

  3. ppenna commented on Nov 14, 2026

    @ppenna
    Contributor

    Status check on dev (6e7a1320047, OpenVMM efe6bc26734). Nothing in OpenVMM or NVX can lift the two limits, so this issue stays open for KVM's quirk.

    • No kernel offers the quirk. As of 2026-11-13, Linus's tree and kvm-x86 next still end KVM_X86_QUIRK_* at bit 11 (KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT), and I found no posting of the RFC (lore.kernel.org was not reachable from here). On nvx-kvm-test (Linux 6.18.57, nested KVM on a Xeon Platinum 8573C), KVM_CHECK_EXTENSION(KVM_CAP_DISABLE_QUIRKS2) returns 0x7ff, bits 0 to 10. OpenVMM has no bit to disable, and must not code against an unallocated one.
    • KVM still refires. On that kernel, KVM_GET_LAPIC and KVM_SET_LAPIC alone give the guest one more timer interrupt after a restore of a one-shot timer that it already took, halted or spinning, as this issue describes. With the stop sequence of openvmm: keep a restore from firing a taken one-shot LAPIC timer again #1514, the tests of virt_kvm and kvm pass there, /dev/kvm ones included: cargo nextest run --profile agent --run-ignored all -p virt_kvm -p kvm, 72 passed.
    • Limit 1, an expiry that KVM records after 20 ms. There is margin on that kernel. Over two runs of 200,000 expiries, on a host whose load average was 9 to 17 on 8 vCPUs, the delivery that found the vector in IRR came a median 8 µs after the count first read 0, within 0.13 ms and 3 ms for 99.99% of them, and within 7.2 ms for all of them, a figure that includes the measuring thread's own preemption. Telling a late expiry from a taken one exactly needs KVM's lapic_timer.target_expiration or pending, which no uAPI exports, and a longer wait would only stall every capture whose guest left an expired timer, for an expiry that this host never recorded late.
    • Limit 2, the VMX preemption timer. Neither the test VM nor the CI runners can show it, since their nested KVM uses none (kvm_intel preemption_timer is N here), so any change for it would go unvalidated, and a wrong guess loses an interrupt where today's choice only adds a spurious one, which Linux ignores.

    When a kernel offers the quirk, OpenVMM can enable it at VM creation, which covers both the save and the restore, and then drop the stop-time delivery and the 20 ms wait on that kernel. The restored guest would also read its timer's exact initial count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area: openvmmInitial title-keyword triage suggestion

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions