Repository navigation
Description
Nightly failure report: #1289
Summary
test-microvm's lifecycle scenario failed on Windows/WHP in the nightly full run with guest exited without completion marker 'NVX-LIFECYCLE-OK', although the guest passed every check, printed the marker, and exited with status 0. Its script writes six bytes to the byte channel (port 0xe9), the last one a newline, and then prints the marker on the boot console. Since #1256, OpenVMM relays the byte channel to its stderr and the boot console to its stdout, each from a thread of its own, with no order between them. run_guest_script reads both as one stream and completes only on a line equal to the marker. In this run, the byte channel's newline reached the harness after the marker, which therefore shared a line with the byte channel's previous bytes.
The failure is a race in the scenario, not a defect of the guest or of OpenVMM, which promises no order between its stdout and its stderr.
Incident
- Workflow: CI run 41973357177, attempt 1, the nightly full run (
schedule) ofdev. - NVX head:
7e1e2fc99684e7473ca8e706f90f8421127e531f.devhas since gained only performance baselines and the module diagrams, so the race remains. - OpenVMM gitlink:
9cec2b89245f41f2126f9a787c5e43a0d84b7154. - Failed job:
NVX microVM tests / Windows / WHP / Shard 2 of 2onazure-windows-4(self-hosted,windows,whp,virtual-machine,general), an Intel Xeon Platinum 8573C with theintel.emeraldrapids.v1profile. - Failed step:
Run NVX microVM tests on Windows, which ranpython scripts\nvx.py test-microvm --backend "whp" --shard 2/2 --output-dir "build/test-results/microvm-whp". The shard's earlier scenarios passed, and it stopped atlifecycle, so its later scenarios did not run. - Every other job passed or was skipped, except
Required status check, which fails whenever another job does. - Artifact:
nvx-microvm-tests-whp-shard-2. Itsmicrovm-whp/lifecycle.logholds OpenVMM's raw output.
Evidence
The end of the job's error:
error: guest exited without completion marker 'NVX-LIFECYCLE-OK'
--- OpenVMM output ---
...
/ # echo NVX-GUEST-SCRIPT-"START"; . /tmp/nvx-guest-script
NVX-GUEST-SCRIPT-START
\ufffdNVX-LIFECYCLE-OK
The last bytes of lifecycle.log show the order in which the two streams reached the harness:
NVX-GUEST-SCRIPT-START\r\n boot console (stdout)
\x00\r\n\x7f\xff byte channel, bytes 1 to 5 (stderr)
NVX-LIFECYCLE-OK\r\n boot console (stdout)
\n byte channel, byte 6 (stderr)
The guest wrote all six bytes, 000 015 012 177 377 012, before it printed the marker. The marker's line became \x7f\xffNVX-LIFECYCLE-OK, which the job log shows as \ufffdNVX-LIFECYCLE-OK. run_guest_script raises this error only once OpenVMM has exited with status 0, and the script requests status 0 only with its last line, after every check.
Cause
lifecycle.shends its bytes to the byte channel with a newline "so that the marker starts" a line, then prints the marker at once.workload_boot_commandattaches the boot console by default: the guest's shell writes tohvc1, on OpenVMM's stdout, and the byte channel writes to OpenVMM's stderr.- In OpenVMM,
setup_portbconnects the byte channel to stderr when the boot console is on the terminal, andsetup_host_consolegives each endpoint a relay thread of its own, to stdout for the boot console and to stderr for the byte channel (spawn_output). - A byte that the guest writes to port
0xe9only joins the device's buffer (io_write). The device writes it to its endpoint later (poll_tx), and the relay copies it to stderr after that. Nothing orders it against the boot console's output, which takes its own path to stdout. InteractiveProcessmerges OpenVMM's stderr into the console stream, in one pipe on Windows and one pseudo-terminal on Linux, andrun_guest_scriptcompletes only on a line equal to the marker (contains_output_line).
OpenVMM's exit drain, which #1255 pinned, writes the byte channel's pending output before the boot console's, but only what is still pending when the guest exits. Here the boot console's relay had already written the marker.
Linux/KVM and Linux/MSHV run the scenario on the same merged stream, so they are exposed too, though no failure has been seen there. No other scenario expects the byte channel's output and a boot console marker in one stream.
History
- scripts: keep the guest console's echo out of microVM test markers #991 (
0dab30ee4) added the trailing newline.lifecyclethen ran without the boot console, so the byte channel was the guest's console, and its bytes and the marker reached stdout through the device's one buffer, in order. - kernel: let the boot console replace the xe9 early console #1256 (
3c4935565, ondevsince 2026-11-04 04:25 UTC) movedlifecycleto the boot console, so that its early console check covershvc1. That put the bytes and the marker on different streams. - scripts: give benchmark and test-microvm guests the boot console #1246 (
ae236d6a4) made the boot console the default of the command builders, and its description notes that the byte channel's output moves to stderr.
Relation to #317
#317 failed the same way: OpenVMM's stderr split the guest's NVX-SNAPSHOT-DISPATCHED line on the shared pseudo-terminal. #330 fixed it for capture_snapshot() and measure_once(), with InteractiveProcess(separate_stderr=True) and SeparatedOutput, which matches guest markers only on the console and keeps a transcript of both streams, interleaved by whole lines. run_guest_script still reads one merged stream.
Reproduction
With the scripts of 7e1e2fc99, run from the repository root against the artifact's log, contains_output_line rejects the observed transcript and accepts the same bytes with only the byte channel's last newline moved before the marker:
import sys
sys.path.insert(0, "scripts")
from nvx_tools.benchmark import contains_output_line
from nvx_tools.microvm_tests import LIFECYCLE_COMPLETION_MARKER as marker
raw = open("microvm-whp/lifecycle.log", "rb").read()
tail = b"\x7f\xffNVX-LIFECYCLE-OK\r\n\n"
assert raw.endswith(tail)
print(contains_output_line(raw, marker)) # False
print(contains_output_line(raw[: -len(tail)] + b"\x7f\xff\nNVX-LIFECYCLE-OK\r\n", marker)) # TrueA live reproduction on WHP was not attempted; at the observed rate, it would take a loop of the scenario.
Frequency
In the 110 CI runs since #1256 reached dev, the jobs of the shards that hold lifecycle, KVM shard 1, MSHV shard 2, and WHP shard 2, passed 66, 64, and 63 times and failed once, in this run; the others were cancelled. A scan of the failed CI runs since 2026-10-28, and of the earlier attempts of rerun runs, found this error in no other run, although a few of its API reads failed.
Proposed fix
Keep the byte channel out of the stream in which lifecycle finds its marker, as #330 does for benchmarks:
- Let
run_guest_scriptstart OpenVMM withseparate_stderr=Trueand read it throughSeparatedOutput, at least forlifecycle, so that it finds the boot and completion markers only on the console. - Keep the byte channel in the transcript that
check_quiet_consolereads, since the early console prints there, and in the scenario's log. - Check that stderr holds the byte channel's bytes,
00 0D 0A 7F FF 0A, in order. The scenario writes them but checks nothing about them today. - Add a unit test whose byte channel output lands inside the marker's line, as in this run.
These would not fix it: matching the marker as a substring, since the echo of the staged script holds echo NVX-LIFECYCLE-OK, which #991's whole-line match guards against; or writing the bytes earlier or waiting after them, which only narrows the window.
Acceptance
lifecyclefinds its marker whatever the order in which the byte channel's and the boot console's output reach the harness, on KVM, MSHV, and WHP.lifecyclechecks the byte channel's bytes, andcheck_quiet_consolestill reads the early console's output.- A regression test covers this run's interleaving.
Activity
scripts: let guest script runs read OpenVMM's stderr apart
test-microvm: find lifecycle's marker on the boot console alone