Skip to content

OpenVMM: the microVM control console's interrupt sometimes never reaches the guest on MSHV, so interrupt-map fails with NVX-INTERRUPT-MAP-QUIET (flaky NVX microVM tests / Linux / MSHV) #1400

Closed

Description

Summary

The interrupt-map scenario sometimes fails on MSHV because the control console never raises its interrupt. The control console is virtio5, at window 0xd0007000 on IOAPIC input 23. In two merge groups, the guest wrote 16 one-byte buffers to /dev/hvc2, waiting up to 0.1 s after each and a second after the last, and IRQ 23 still never fired (count=0). OpenVMM exited with status 37 (NVX-INTERRUPT-MAP-QUIET). In both runs the boot console had interrupted on its first write.

Since #1191 added the scenario on 2026-11-03, NVX microVM tests / Linux / MSHV / Shard 1 of 2, which runs it, has completed 290 times, nearly all of them with the scenario. It failed only these 2 times, both on runner azure-mshv-3:

Runner Passed Failed
azure-mshv-1 80 0
azure-mshv-2 72 0
azure-mshv-3 67 2
azure-mshv-5 69 0

No KVM or WHP job has failed this way: I scanned all 601 CI runs since 2026-11-01, including the earlier attempts of re-run runs. The tree of the second incident passed the same job, with the same guest kernel and the same host kernel, in its pull request's run.

Incidents

  1. Run 41572000089, attempt 1, 2026-11-03.
  2. Run 42492123438, attempt 1, 2026-11-08.

Both guests reported the same thing (run 42492123438):

NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=5691 unhandled=5581
NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
NVX-INTERRUPT-MAP-CONSOLE
NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
NVX-INTERRUPT-MAP-FAIL code=37

Run 41572000089 printed the same lines, with count=4717 unhandled=4626 for virtio2. In a passing run, virtio5 also reports count=0 at boot, then count=1 unhandled=1 after the traffic step.

Analysis

  • The device completed every write. Linux 6.18's put_chars() spins in __send_to_port() until the device returns the buffer, so each printf . >/dev/hvc2 returned only after the control console's broker had completed its descriptor (BrokerWorker::poll_guest_input() and then VirtioQueue::complete()). Nothing else uses the control console before the traffic step: init opens hvc2 only in the managed lifecycle, and the broker has no output for the guest.
  • Skipped signals are an unlikely explanation. The queues negotiate VIRTIO_F_EVENT_IDX on the split ring, so a completion skips its interrupt when the guest takes the buffer back and advances used_event before the device reads it. That is why the scenario writes up to 16 times. The boot console shows how rarely this race goes the guest's way: during the same boot it interrupted 5,691 times, and /proc/irq/18/spurious counted 5,581 recent interrupts that found no work because the guest had already reclaimed the buffer. So the device signalled nearly every completion. For the device to lose that race 16 times in a row is implausible.
  • One lost rising edge would silence the device until reset.
    • The virtio-mmio transport keeps INTERRUPT_STATUS and drives its LineInterrupt at that level (InterruptState::update()).
    • The microVM's IOAPIC inputs are edge-triggered (type=edge). IrqEntry::evaluate() delivers an edge-triggered input only on a rising edge. It drops an edge that arrives while the entry is masked, without a message, and never delivers it on unmask.
    • The guest reads and acknowledges INTERRUPT_STATUS only in vm_interrupt().
    • So if the first used-buffer interrupt's edge is lost while the status bit stays set, the line stays high, and every later completion raises no edge. That fits count=0 through 16 writes better than 16 independent skips.
    • The scenario runs OpenVMM with OPENVMM_LOG=off, so the log cannot show whether MSHV's routing warned. The candidates are failed to request msi, when request_virtual_interrupt fails, and irq for masked interrupt, when IRQ 23 has no route.
  • A related hazard, which needs an earlier interrupt on the same device and so does not explain these incidents:
    • On Linux hosts the transport takes INTERRUPT_ACK writes through an accelerated doorbell and applies them later (acknowledge_used_buffer()).
    • When a completion lands between the guest's status read and that deferred acknowledgement, the generation check rightly keeps USED_BUFFER set.
    • With an edge-triggered input, that leaves the line high with no further edge: the same failure shape.
  • Not related to the CPU profile or to openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399. The first incident predates openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399. Both runners selected the unchanged intel.emeraldrapids.v1 (sha256:73c08478…). openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399 changes no virtio, interrupt, or MSHV code.

Reproduction attempts

I ran the scenario on nvx-mshv-test, a Xeon Platinum 8573C VM with MSHV and Azure Linux 3 6.18.34.mshv3-1.azl3, the same host kernel as the runners. The setup matched CI:

python3 scripts/nvx.py test-microvm --backend mshv --scenario interrupt-map ran 611 times at dev and 610 times at #1399's head, four runs in parallel, alternating the revisions, over 24 minutes. Every run passed. At the job's overall CI rate (2 of 290), about four failures would have been expected. That fits failures that cluster on azure-mshv-3; the other runners have passed 221 times in a row.

Next steps

  • When the scenario reports QUIET, have it record what tells the two causes apart:
    • the device's INTERRUPT_STATUS at 0xd0007060, which a read does not acknowledge, if the guest's /dev/mem allows it. A set bit means a lost edge on a high line; a clear bit means the device never signalled.
    • /proc/interrupts (ERR, MIS, SPU) and /proc/irq/23/spurious;
    • OpenVMM's warnings, by running the scenario with a warning-level OPENVMM_LOG filter rather than off.
  • Run the scenario repeatedly on azure-mshv-3 while it is drained.
  • If a lost edge is confirmed, make the transport's line safe on edge-triggered inputs. For example, raise a new edge for a used-buffer notification that arrives while the status is already set, as VirtioMmioInterruptMode::SharedStatus pulses each zero-to-nonzero transition.

Activity

  1. ppenna commented on Nov 8, 2026

    @ppenna
    ContributorAuthor

    More evidence: this failure also hits KVM and WHP, on GitHub-hosted runners with AMD CPUs, and also at 90f9e6180, the OpenVMM revision that dev pins before #1401. It is specific neither to azure-mshv-3 nor to MSHV.

    While qualifying the AMD CPU profiles for #409 (#1401), a temporary workflow ran the default test-microvm suite with --cpu-profile auto on GitHub-hosted runners: Azure VMs with 4 vCPUs and an AMD EPYC 7763 (Milan), 9V74 (Genoa), or 9V45 (Turin), with the guest artifacts of release v0.1.0-dev.733a7031e655. interrupt-map ran in 55 suites and failed in 5, each time with this issue's signature: the control console virtio5, at 0xd0007000 on IOAPIC input 23 (edge), never interrupted, and OpenVMM exited with status 37 (NVX-INTERRUPT-MAP-FAIL code=37).

    The five failures

    # Run and job Backend Runner image CPU OpenVMM CPU profile
    1 42506171991, WHP windows-latest 24 WHP Windows Server 2025 24H2, build 26100.33674, image win25-vs2026 20261103.298.1 EPYC 9V74 (25/17/1) 90f9e6180, from the release amd.genoa.v2
    2 42506171991, WHP windows-latest 10 WHP Windows Server 2025 24H2, build 26100.33674, image win25-vs2026 20261103.298.1 EPYC 7763 (25/1/1) 90f9e6180, from the release amd.milan.v1
    3 42508924142, KVM ubuntu-24.04 19 KVM Ubuntu 24.04, Linux 7.0.0-1017-azure, image ubuntu24 20261101.360.1 EPYC 9V45 (26/2/1) adb7c8bc8 amd.turin.v2
    4 42508924142, WHP windows-latest 4 WHP Windows Server 2025 24H2, build 26100.33674, image win25-vs2026 20261103.298.1 EPYC 9V74 (25/17/1) adb7c8bc8 amd.genoa.v2
    5 42526588675, KVM ubuntu-22.04 4 KVM Ubuntu 22.04, Linux 6.8.0-1070-azure, image ubuntu22 20261101.347.1 EPYC 9V45 (26/2/1) adb7c8bc8 amd.turin.v2

    adb7c8bc8 is microsoft/openvmm#4987, now the head of feature/microvm/dev. It is 90f9e6180 plus a second Turin CPU profile, and changes nothing else, so all five failures run the same interrupt, virtio, and console code. The jobs show as successful because the workflow's steps continue on error: the log of each job's Run the default microVM suite step shows the failure, and the run's artifact for the job, <backend>-<generation>-<image>-<sample> (whp-genoa-windows-latest-24 for the first), holds its microvm.log until 2026-11-15.

    Each guest reported what this issue's two incidents reported. Failure 1's boot report, whose other devices look the same in all five:

    NVX-INTERRUPT-MAP step=boot base=0xd0000000 device=virtio0 irq=16 chip=IO-APIC hwirq=16 type=edge actions=virtio0 count=2 unhandled=0
    NVX-INTERRUPT-MAP step=boot base=0xd0001000 device=virtio1 irq=17 chip=IO-APIC hwirq=17 type=edge actions=virtio1 count=1 unhandled=0
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=6200 unhandled=6069
    NVX-INTERRUPT-MAP step=boot base=0xd0003000 device=virtio3 irq=19 chip=IO-APIC hwirq=19 type=edge actions=virtio3 count=1 unhandled=0
    NVX-INTERRUPT-MAP step=boot base=0xd0006000 device=virtio4 irq=22 chip=IO-APIC hwirq=22 type=edge actions=virtio4 count=1 unhandled=0
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    

    The boot console's and the control console's lines of the five:

    1 (WHP, EPYC 9V74, 90f9e6180):
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=6200 unhandled=6069
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    
    2 (WHP, EPYC 7763, 90f9e6180):
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=6193 unhandled=6101
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    
    3 (KVM, EPYC 9V45, adb7c8bc8):
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=5499 unhandled=5386
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    
    4 (WHP, EPYC 9V74, adb7c8bc8):
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=6024 unhandled=5929
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    
    5 (KVM, EPYC 9V45, adb7c8bc8):
    NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=6484 unhandled=6370
    NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
    NVX-INTERRUPT-MAP-CONSOLE
    NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
    NVX-INTERRUPT-MAP-FAIL code=37
    

    As in this issue's incidents, the boot console interrupted thousands of times during boot, nearly all of them spurious, and interrupted on the first NVX-INTERRUPT-MAP-CONSOLE write, or on the third in failure 4. The control console then stayed at count=0 through its writes.

    Where it ran and failed

    interrupt-map ran once in each of the 55 suites, one suite per runner:

    Backend CPU Runs Failures
    KVM EPYC 7763 10 0
    KVM EPYC 9V74 3 0
    KVM EPYC 9V45 9 2
    WHP EPYC 7763 11 1
    WHP EPYC 9V74 17 2
    WHP EPYC 9V45 5 0
    All 55 5

    By revision, 2 of 17 at 90f9e6180, all on WHP in run 42506171991, and 3 of 38 at adb7c8bc8, in runs 42508924142 and 42526588675. The KVM runs were on Ubuntu 24.04 (Linux 7.0) and 22.04 (Linux 6.8), and the WHP runs on Windows Server 2025. The other 50 suites passed in full.

    How this changes the picture

    • Not azure-mshv-3, not MSHV, not Intel. The same failure hit KVM in 2 of 22 runs and WHP in 3 of 33, on Linux and Windows hosts, on three AMD generations, and on three CPU profiles, at 90f9e6180 and at a revision that only adds a profile to it. The table above in this issue, where only azure-mshv-3 ever failed, reflects NVX CI's runners, which are all Intel.
    • Much more frequent on these runners. 5 of 55 runs, about 9%, against 2 of about 290 on NVX's Intel MSHV runners and none on its Intel KVM and WHP runners. At that rate, a few dozen runs of python3 scripts/nvx.py test-microvm --backend kvm --scenario interrupt-map on GitHub-hosted runners that draw an AMD CPU should reproduce it, with the diagnostics that the next steps above list.
    • The cause is likely in the code that every backend shares. The transport's level-driven LineInterrupt and the IOAPIC that sees its edges are OpenVMM's own on all three backends: OpenVMM runs KVM with a split irqchip, so chipset's IOAPIC (IrqEntry::evaluate(), which drops a masked edge and delivers only rising edges) serves KVM and WHP as it serves MSHV, and only the delivery of the routed interrupt to the vCPUs is the backend's. That fits the lost rising edge on a line that stays high, which the analysis above proposes, and makes MSHV's routing warnings (failed to request msi, irq for masked interrupt) unlikely causes. Three of the failures ran on Windows hosts, so the deferred INTERRUPT_ACK of the accelerated doorbell, which the analysis describes on Linux hosts, is not needed for the failure either.
    • It matters for AMD runners in CI (Add AMD CI runners with their own performance series #410): at this rate, about one in eleven runs of the microVM shard that holds interrupt-map would fail on an AMD runner.
  2. ppenna commented on Nov 9, 2026

    @ppenna
    ContributorAuthor

    Root cause, corrected: no interrupt was lost; the control console's device never sent one, as virtio allows. #1410 fixes the scenario.

    • Not a lost edge. The microVM's virtio slots deliver their status through the shared interrupt-status page (2.4.4), not through INTERRUPT_STATUS: the transport ORs its bits into the slot's word and pulses its IOAPIC input only when the word was zero. So neither the level-driven LineInterrupt of the legacy mode nor its deferred INTERRUPT_ACK doorbell, which OpenVMM enables only in that mode, is involved. With an instrumented OpenVMM (run 42550388573), each of the 12 failing boots that it logged showed the control console completing all 16 writes while the transport never updated the console's word and the IOAPIC never saw input 23 asserted, and the guest read the word as zero after QUIET.
    • The race. Linux's console driver waits for each write by polling the used ring of the console's output with interrupts off (put_chars() spins in __send_to_port()), and takes the buffer back, moving its used event past it, as soon as the device returns it. With VIRTIO_F_EVENT_IDX, the device notifies only if the used event still names the buffer when it reads the event after publishing the buffer (virtio 1.2, 2.7.10), as OpenVMM's split queue does, so a write interrupts only when the device reads the event first. A second instrumented run (run 42552321090) recorded the event that the device read at each completion: every skipped notification of either console, 841 of the control console's 1,159 writes among them, found the event already moved to the next buffer. How often the driver wins varies by boot: of 320 boots on AMD runners under KVM, 22 used up the scenario's 16 writes, and in 2 none of 43 writes interrupted. The boot console's interrupts that found no work count only the notifications that the device sent, not those that it skipped, which is why they suggested that the device signals nearly every completion.
    • Input does not race. The driver takes a console's input buffers back from its interrupt, and the hvc thread polls an open console only when kicked, so the device reads the input's used event before the driver can move it, and its first completion notifies.

    So the control console now interrupts with input: the guest sends the console's control-session broker the bootstrap GuestAttach record and waits for the interrupt of its Reset answer, with /dev/hvc2 held open, raw, and without echo. The check stays as strict, and OpenVMM is unchanged. On GitHub-hosted runners, the previous scenario failed 64 of 640 runs (KVM 31 of 320, WHP 33 of 320, on AMD EPYC 7763, 9V74, and 9V45 and Intel Xeon runners) and the new one 1 of 640 (run 42555071036), then none of 1,920 more with an instrumented OpenVMM (run 42556047328). That one failure, on WHP with an Xeon 8573C runner, left only the guest's console, so its cause is unknown; should it recur, the control console's shared status word (0x3001c) and OPENVMM_LOG=warn tell a signalled interrupt that never arrived from a skipped notification. The pull request has the details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions