Repository navigation
Description
Summary
The interrupt-map scenario sometimes fails on MSHV because the control console never raises its interrupt. The control console is virtio5, at window 0xd0007000 on IOAPIC input 23. In two merge groups, the guest wrote 16 one-byte buffers to /dev/hvc2, waiting up to 0.1 s after each and a second after the last, and IRQ 23 still never fired (count=0). OpenVMM exited with status 37 (NVX-INTERRUPT-MAP-QUIET). In both runs the boot console had interrupted on its first write.
Since #1191 added the scenario on 2026-11-03, NVX microVM tests / Linux / MSHV / Shard 1 of 2, which runs it, has completed 290 times, nearly all of them with the scenario. It failed only these 2 times, both on runner azure-mshv-3:
| Runner | Passed | Failed |
|---|---|---|
azure-mshv-1 |
80 | 0 |
azure-mshv-2 |
72 | 0 |
azure-mshv-3 |
67 | 2 |
azure-mshv-5 |
69 | 0 |
No KVM or WHP job has failed this way: I scanned all 601 CI runs since 2026-11-01, including the earlier attempts of re-run runs. The tree of the second incident passed the same job, with the same guest kernel and the same host kernel, in its pull request's run.
Incidents
- Run 41572000089, attempt 1, 2026-11-03.
- The merge group of openvmm: give each microVM virtio slot its own IOAPIC input #1191, head
f859c1b07, with OpenVMM35e76265f. - Job
NVX microVM tests / Linux / MSHV / Shard 1 of 2onazure-mshv-3, host kernel6.18.34.mshv3-1.azl3,intel.emeraldrapids.v1. - openvmm: give each microVM virtio slot its own IOAPIC input #1191 was queued again and merged 48 minutes later.
- The merge group of openvmm: give each microVM virtio slot its own IOAPIC input #1191, head
- Run 42492123438, attempt 1, 2026-11-08.
- The merge group of openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399, head
71181400915, treea7ea9db9, with OpenVMM90f9e6180. - Job
NVX microVM tests / Linux / MSHV / Shard 1 of 2onazure-mshv-3, with the same host kernel and the same profile. - PR run 42489736500 ran the identical tree with the same guest kernel artifact (
sha256:09445cd1…). Its same job passed onazure-mshv-5.
- The merge group of openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399, head
Both guests reported the same thing (run 42492123438):
NVX-INTERRUPT-MAP step=boot base=0xd0002000 device=virtio2 irq=18 chip=IO-APIC hwirq=18 type=edge actions=virtio2 count=5691 unhandled=5581
NVX-INTERRUPT-MAP step=boot base=0xd0007000 device=virtio5 irq=23 chip=IO-APIC hwirq=23 type=edge actions=virtio5 count=0 unhandled=0
NVX-INTERRUPT-MAP-CONSOLE
NVX-INTERRUPT-MAP-QUIET base=0xd0007000 device=virtio5 irq=23 count=0
NVX-INTERRUPT-MAP-FAIL code=37
Run 41572000089 printed the same lines, with count=4717 unhandled=4626 for virtio2. In a passing run, virtio5 also reports count=0 at boot, then count=1 unhandled=1 after the traffic step.
Analysis
- The device completed every write. Linux 6.18's
put_chars()spins in__send_to_port()until the device returns the buffer, so eachprintf . >/dev/hvc2returned only after the control console's broker had completed its descriptor (BrokerWorker::poll_guest_input()and thenVirtioQueue::complete()). Nothing else uses the control console before the traffic step: init openshvc2only in the managed lifecycle, and the broker has no output for the guest. - Skipped signals are an unlikely explanation. The queues negotiate
VIRTIO_F_EVENT_IDXon the split ring, so a completion skips its interrupt when the guest takes the buffer back and advancesused_eventbefore the device reads it. That is why the scenario writes up to 16 times. The boot console shows how rarely this race goes the guest's way: during the same boot it interrupted 5,691 times, and/proc/irq/18/spuriouscounted 5,581 recent interrupts that found no work because the guest had already reclaimed the buffer. So the device signalled nearly every completion. For the device to lose that race 16 times in a row is implausible. - One lost rising edge would silence the device until reset.
- The virtio-mmio transport keeps
INTERRUPT_STATUSand drives itsLineInterruptat that level (InterruptState::update()). - The microVM's IOAPIC inputs are edge-triggered (
type=edge).IrqEntry::evaluate()delivers an edge-triggered input only on a rising edge. It drops an edge that arrives while the entry is masked, without a message, and never delivers it on unmask. - The guest reads and acknowledges
INTERRUPT_STATUSonly invm_interrupt(). - So if the first used-buffer interrupt's edge is lost while the status bit stays set, the line stays high, and every later completion raises no edge. That fits
count=0through 16 writes better than 16 independent skips. - The scenario runs OpenVMM with
OPENVMM_LOG=off, so the log cannot show whether MSHV's routing warned. The candidates arefailed to request msi, whenrequest_virtual_interruptfails, andirq for masked interrupt, when IRQ 23 has no route.
- The virtio-mmio transport keeps
- A related hazard, which needs an earlier interrupt on the same device and so does not explain these incidents:
- On Linux hosts the transport takes
INTERRUPT_ACKwrites through an accelerated doorbell and applies them later (acknowledge_used_buffer()). - When a completion lands between the guest's status read and that deferred acknowledgement, the generation check rightly keeps
USED_BUFFERset. - With an edge-triggered input, that leaves the line high with no further edge: the same failure shape.
- On Linux hosts the transport takes
- Not related to the CPU profile or to openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399. The first incident predates openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399. Both runners selected the unchanged
intel.emeraldrapids.v1(sha256:73c08478…). openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399 changes no virtio, interrupt, or MSHV code.
Reproduction attempts
I ran the scenario on nvx-mshv-test, a Xeon Platinum 8573C VM with MSHV and Azure Linux 3 6.18.34.mshv3-1.azl3, the same host kernel as the runners. The setup matched CI:
- the guest kernel and Alpine initramfs artifacts of run 42492123438;
- OpenVMM built as CI builds
openvmm-linux-gnu(nvx.py build-openvmm --backend kvm), atcf6c03977(dev144d096f4) and at90f9e6180(openvmm: pin the Granite Rapids CPU profile, without AMX and AVX10 #1399's head2f58232e2).
python3 scripts/nvx.py test-microvm --backend mshv --scenario interrupt-map ran 611 times at dev and 610 times at #1399's head, four runs in parallel, alternating the revisions, over 24 minutes. Every run passed. At the job's overall CI rate (2 of 290), about four failures would have been expected. That fits failures that cluster on azure-mshv-3; the other runners have passed 221 times in a row.
Next steps
- When the scenario reports
QUIET, have it record what tells the two causes apart:- the device's
INTERRUPT_STATUSat0xd0007060, which a read does not acknowledge, if the guest's/dev/memallows it. A set bit means a lost edge on a high line; a clear bit means the device never signalled. /proc/interrupts(ERR,MIS,SPU) and/proc/irq/23/spurious;- OpenVMM's warnings, by running the scenario with a warning-level
OPENVMM_LOGfilter rather thanoff.
- the device's
- Run the scenario repeatedly on
azure-mshv-3while it is drained. - If a lost edge is confirmed, make the transport's line safe on edge-triggered inputs. For example, raise a new edge for a used-buffer notification that arrives while the status is already set, as
VirtioMmioInterruptMode::SharedStatuspulses each zero-to-nonzero transition.
Activity
openvmm: pin amd.turin.v2 for Turin hosts without the TSA immunities
doc: describe the second Turin CPU profile
ppenna commentedon Nov 8, 2026
More evidence: this failure also hits KVM and WHP, on GitHub-hosted runners with AMD CPUs, and also at
90f9e6180, the OpenVMM revision thatdevpins before #1401. It is specific neither toazure-mshv-3nor to MSHV.While qualifying the AMD CPU profiles for #409 (#1401), a temporary workflow ran the default
test-microvmsuite with--cpu-profile autoon GitHub-hosted runners: Azure VMs with 4 vCPUs and an AMD EPYC 7763 (Milan), 9V74 (Genoa), or 9V45 (Turin), with the guest artifacts of releasev0.1.0-dev.733a7031e655.interrupt-mapran in 55 suites and failed in 5, each time with this issue's signature: the control consolevirtio5, at0xd0007000on IOAPIC input 23 (edge), never interrupted, and OpenVMM exited with status 37 (NVX-INTERRUPT-MAP-FAIL code=37).The five failures
WHP windows-latest 24win25-vs202620261103.298.190f9e6180, from the releaseamd.genoa.v2WHP windows-latest 10win25-vs202620261103.298.190f9e6180, from the releaseamd.milan.v1KVM ubuntu-24.04 19ubuntu2420261101.360.1adb7c8bc8amd.turin.v2WHP windows-latest 4win25-vs202620261103.298.1adb7c8bc8amd.genoa.v2KVM ubuntu-22.04 4ubuntu2220261101.347.1adb7c8bc8amd.turin.v2adb7c8bc8is microsoft/openvmm#4987, now the head offeature/microvm/dev. It is90f9e6180plus a second Turin CPU profile, and changes nothing else, so all five failures run the same interrupt, virtio, and console code. The jobs show as successful because the workflow's steps continue on error: the log of each job'sRun the default microVM suitestep shows the failure, and the run's artifact for the job,<backend>-<generation>-<image>-<sample>(whp-genoa-windows-latest-24for the first), holds itsmicrovm.loguntil 2026-11-15.Each guest reported what this issue's two incidents reported. Failure 1's boot report, whose other devices look the same in all five:
The boot console's and the control console's lines of the five:
As in this issue's incidents, the boot console interrupted thousands of times during boot, nearly all of them spurious, and interrupted on the first
NVX-INTERRUPT-MAP-CONSOLEwrite, or on the third in failure 4. The control console then stayed atcount=0through its writes.Where it ran and failed
interrupt-mapran once in each of the 55 suites, one suite per runner:By revision, 2 of 17 at
90f9e6180, all on WHP in run 42506171991, and 3 of 38 atadb7c8bc8, in runs 42508924142 and 42526588675. The KVM runs were on Ubuntu 24.04 (Linux 7.0) and 22.04 (Linux 6.8), and the WHP runs on Windows Server 2025. The other 50 suites passed in full.How this changes the picture
azure-mshv-3, not MSHV, not Intel. The same failure hit KVM in 2 of 22 runs and WHP in 3 of 33, on Linux and Windows hosts, on three AMD generations, and on three CPU profiles, at90f9e6180and at a revision that only adds a profile to it. The table above in this issue, where onlyazure-mshv-3ever failed, reflects NVX CI's runners, which are all Intel.python3 scripts/nvx.py test-microvm --backend kvm --scenario interrupt-mapon GitHub-hosted runners that draw an AMD CPU should reproduce it, with the diagnostics that the next steps above list.LineInterruptand the IOAPIC that sees its edges are OpenVMM's own on all three backends: OpenVMM runs KVM with a split irqchip, sochipset's IOAPIC (IrqEntry::evaluate(), which drops a masked edge and delivers only rising edges) serves KVM and WHP as it serves MSHV, and only the delivery of the routed interrupt to the vCPUs is the backend's. That fits the lost rising edge on a line that stays high, which the analysis above proposes, and makes MSHV's routing warnings (failed to request msi,irq for masked interrupt) unlikely causes. Three of the failures ran on Windows hosts, so the deferredINTERRUPT_ACKof the accelerated doorbell, which the analysis describes on Linux hosts, is not needed for the failure either.interrupt-mapwould fail on an AMD runner.scratch: record the used event of each console completion (#1400)
scratch: compare interrupt-map before and after the fix (#1400)
scratch: compare interrupt-map before and after the fix (#1400)
scratch: diagnose the residual interrupt-map failure (#1400)
ppenna commentedon Nov 9, 2026
Root cause, corrected: no interrupt was lost; the control console's device never sent one, as virtio allows. #1410 fixes the scenario.
INTERRUPT_STATUS: the transport ORs its bits into the slot's word and pulses its IOAPIC input only when the word was zero. So neither the level-drivenLineInterruptof the legacy mode nor its deferredINTERRUPT_ACKdoorbell, which OpenVMM enables only in that mode, is involved. With an instrumented OpenVMM (run 42550388573), each of the 12 failing boots that it logged showed the control console completing all 16 writes while the transport never updated the console's word and the IOAPIC never saw input 23 asserted, and the guest read the word as zero afterQUIET.put_chars()spins in__send_to_port()), and takes the buffer back, moving its used event past it, as soon as the device returns it. WithVIRTIO_F_EVENT_IDX, the device notifies only if the used event still names the buffer when it reads the event after publishing the buffer (virtio 1.2, 2.7.10), as OpenVMM's split queue does, so a write interrupts only when the device reads the event first. A second instrumented run (run 42552321090) recorded the event that the device read at each completion: every skipped notification of either console, 841 of the control console's 1,159 writes among them, found the event already moved to the next buffer. How often the driver wins varies by boot: of 320 boots on AMD runners under KVM, 22 used up the scenario's 16 writes, and in 2 none of 43 writes interrupted. The boot console's interrupts that found no work count only the notifications that the device sent, not those that it skipped, which is why they suggested that the device signals nearly every completion.So the control console now interrupts with input: the guest sends the console's control-session broker the bootstrap
GuestAttachrecord and waits for the interrupt of itsResetanswer, with/dev/hvc2held open, raw, and without echo. The check stays as strict, and OpenVMM is unchanged. On GitHub-hosted runners, the previous scenario failed 64 of 640 runs (KVM 31 of 320, WHP 33 of 320, on AMD EPYC 7763, 9V74, and 9V45 and Intel Xeon runners) and the new one 1 of 640 (run 42555071036), then none of 1,920 more with an instrumented OpenVMM (run 42556047328). That one failure, on WHP with an Xeon 8573C runner, left only the guest's console, so its cause is unknown; should it recur, the control console's shared status word (0x3001c) andOPENVMM_LOG=warntell a signalled interrupt that never arrived from a skipped notification. The pull request has the details.