Skip to content

test-microvm: lifecycle intermittently misses its marker when the byte channel's output interleaves with the boot console's #1290

Closed

Description

Nightly failure report: #1289

Summary

test-microvm's lifecycle scenario failed on Windows/WHP in the nightly full run with guest exited without completion marker 'NVX-LIFECYCLE-OK', although the guest passed every check, printed the marker, and exited with status 0. Its script writes six bytes to the byte channel (port 0xe9), the last one a newline, and then prints the marker on the boot console. Since #1256, OpenVMM relays the byte channel to its stderr and the boot console to its stdout, each from a thread of its own, with no order between them. run_guest_script reads both as one stream and completes only on a line equal to the marker. In this run, the byte channel's newline reached the harness after the marker, which therefore shared a line with the byte channel's previous bytes.

The failure is a race in the scenario, not a defect of the guest or of OpenVMM, which promises no order between its stdout and its stderr.

Incident

  • Workflow: CI run 41973357177, attempt 1, the nightly full run (schedule) of dev.
  • NVX head: 7e1e2fc99684e7473ca8e706f90f8421127e531f. dev has since gained only performance baselines and the module diagrams, so the race remains.
  • OpenVMM gitlink: 9cec2b89245f41f2126f9a787c5e43a0d84b7154.
  • Failed job: NVX microVM tests / Windows / WHP / Shard 2 of 2 on azure-windows-4 (self-hosted, windows, whp, virtual-machine, general), an Intel Xeon Platinum 8573C with the intel.emeraldrapids.v1 profile.
  • Failed step: Run NVX microVM tests on Windows, which ran python scripts\nvx.py test-microvm --backend "whp" --shard 2/2 --output-dir "build/test-results/microvm-whp". The shard's earlier scenarios passed, and it stopped at lifecycle, so its later scenarios did not run.
  • Every other job passed or was skipped, except Required status check, which fails whenever another job does.
  • Artifact: nvx-microvm-tests-whp-shard-2. Its microvm-whp/lifecycle.log holds OpenVMM's raw output.

Evidence

The end of the job's error:

error: guest exited without completion marker 'NVX-LIFECYCLE-OK'
--- OpenVMM output ---
...
/ # echo NVX-GUEST-SCRIPT-"START"; . /tmp/nvx-guest-script
NVX-GUEST-SCRIPT-START

\ufffdNVX-LIFECYCLE-OK

The last bytes of lifecycle.log show the order in which the two streams reached the harness:

NVX-GUEST-SCRIPT-START\r\n   boot console (stdout)
\x00\r\n\x7f\xff             byte channel, bytes 1 to 5 (stderr)
NVX-LIFECYCLE-OK\r\n         boot console (stdout)
\n                           byte channel, byte 6 (stderr)

The guest wrote all six bytes, 000 015 012 177 377 012, before it printed the marker. The marker's line became \x7f\xffNVX-LIFECYCLE-OK, which the job log shows as \ufffdNVX-LIFECYCLE-OK. run_guest_script raises this error only once OpenVMM has exited with status 0, and the script requests status 0 only with its last line, after every check.

Cause

  • lifecycle.sh ends its bytes to the byte channel with a newline "so that the marker starts" a line, then prints the marker at once.
  • workload_boot_command attaches the boot console by default: the guest's shell writes to hvc1, on OpenVMM's stdout, and the byte channel writes to OpenVMM's stderr.
  • In OpenVMM, setup_portb connects the byte channel to stderr when the boot console is on the terminal, and setup_host_console gives each endpoint a relay thread of its own, to stdout for the boot console and to stderr for the byte channel (spawn_output).
  • A byte that the guest writes to port 0xe9 only joins the device's buffer (io_write). The device writes it to its endpoint later (poll_tx), and the relay copies it to stderr after that. Nothing orders it against the boot console's output, which takes its own path to stdout.
  • InteractiveProcess merges OpenVMM's stderr into the console stream, in one pipe on Windows and one pseudo-terminal on Linux, and run_guest_script completes only on a line equal to the marker (contains_output_line).

OpenVMM's exit drain, which #1255 pinned, writes the byte channel's pending output before the boot console's, but only what is still pending when the guest exits. Here the boot console's relay had already written the marker.

Linux/KVM and Linux/MSHV run the scenario on the same merged stream, so they are exposed too, though no failure has been seen there. No other scenario expects the byte channel's output and a boot console marker in one stream.

History

Relation to #317

#317 failed the same way: OpenVMM's stderr split the guest's NVX-SNAPSHOT-DISPATCHED line on the shared pseudo-terminal. #330 fixed it for capture_snapshot() and measure_once(), with InteractiveProcess(separate_stderr=True) and SeparatedOutput, which matches guest markers only on the console and keeps a transcript of both streams, interleaved by whole lines. run_guest_script still reads one merged stream.

Reproduction

With the scripts of 7e1e2fc99, run from the repository root against the artifact's log, contains_output_line rejects the observed transcript and accepts the same bytes with only the byte channel's last newline moved before the marker:

import sys
sys.path.insert(0, "scripts")
from nvx_tools.benchmark import contains_output_line
from nvx_tools.microvm_tests import LIFECYCLE_COMPLETION_MARKER as marker

raw = open("microvm-whp/lifecycle.log", "rb").read()
tail = b"\x7f\xffNVX-LIFECYCLE-OK\r\n\n"
assert raw.endswith(tail)
print(contains_output_line(raw, marker))  # False
print(contains_output_line(raw[: -len(tail)] + b"\x7f\xff\nNVX-LIFECYCLE-OK\r\n", marker))  # True

A live reproduction on WHP was not attempted; at the observed rate, it would take a loop of the scenario.

Frequency

In the 110 CI runs since #1256 reached dev, the jobs of the shards that hold lifecycle, KVM shard 1, MSHV shard 2, and WHP shard 2, passed 66, 64, and 63 times and failed once, in this run; the others were cancelled. A scan of the failed CI runs since 2026-10-28, and of the earlier attempts of rerun runs, found this error in no other run, although a few of its API reads failed.

Proposed fix

Keep the byte channel out of the stream in which lifecycle finds its marker, as #330 does for benchmarks:

  • Let run_guest_script start OpenVMM with separate_stderr=True and read it through SeparatedOutput, at least for lifecycle, so that it finds the boot and completion markers only on the console.
  • Keep the byte channel in the transcript that check_quiet_console reads, since the early console prints there, and in the scenario's log.
  • Check that stderr holds the byte channel's bytes, 00 0D 0A 7F FF 0A, in order. The scenario writes them but checks nothing about them today.
  • Add a unit test whose byte channel output lands inside the marker's line, as in this run.

These would not fix it: matching the marker as a substring, since the echo of the staged script holds echo NVX-LIFECYCLE-OK, which #991's whole-line match guards against; or writing the bytes earlier or waiting after them, which only narrows the window.

Acceptance

  • lifecycle finds its marker whatever the order in which the byte channel's and the boot console's output reach the harness, on KVM, MSHV, and WHP.
  • lifecycle checks the byte channel's bytes, and check_quiet_console still reads the early console's output.
  • A regression test covers this run's interleaving.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions