Repository navigation
Description
Summary
The dev push run for 4434b1a (#1039) failed in nvx-rs / windows-latest because host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol panicked. The thread that plays the guest in that test got BrokenPipe from its last send, and the test's join of that thread then failed. The guest answers an exec with 18 output records of 60,000 bytes and then an exit record. As the test expects, the host fails the exec on the 18th record, past the 1 MiB output limit. The test then drops the session, which closes the host's end of the in-memory stream, and only afterwards joins the guest. If the guest's thread has not sent its exit record by then, that send fails, and the guest unwraps the error.
#1039 did not cause this: it changes neither the session nor its tests, its own CI run passed both nvx-rs jobs, and the race has existed since #596 added the test. The code is unchanged on the current dev head. A 200 ms pause before the guest's last send fails the test every time with the CI signature, and joining the guest before dropping the session makes it pass. This is a flaky test, not a product defect.
Incident
- Workflow: CI run 41008579429, attempt 1,
pushtodev. - Head:
4434b1af0a186446b9a324bd0dbeab0e4c5f56aa, the merge of nvx-rs: read OpenVMM's fatal error before a chain of any length, and name repeated CPUID record entries #1039. Push runs ondevqueue rather than cancel each other, so this run was not superseded. nvx-rs: read OpenVMM's fatal error before a chain of any length, and name repeated CPUID record entries #1039's own CI run, 40998797162 for0853bdc, whose tree differs from4434b1aonly indata/*.csv, passed bothnvx-rsjobs, and so did the nextdevrun, 41060782829 for80d4821, which contains4434b1a. - Failed job:
nvx-rs / windows-latest, stepCheck nvx-rs, in the composite action'sTeststep:cargo "+1.95.0" test --locked --all-features. Of the library's 1,132 tests, 1,125 passed, 6 were ignored, and this one failed, 4 s after they started. - Runner: GitHub-hosted
windows-latest, Windows Server 2025, imagewindows-2025-vs202620261021.278.1, runner 2.338.0, Rust 1.95.0. - Downstream:
Required status checkfailed, andPerformance regression gate,Persist performance baseline,Publish development release,Package nvx crate, andPublish nvx cratewere skipped. The same run'sOpenVMM binary / Windows MSVCjob also failed, for an unrelated reason: its self-hosted runner,azure-windows-10, lost communication with GitHub (comment on #1046, setup: accept enforced firewall rules with inactive profiles #1048). This issue tracks only the test.
Decisive output:
2026-10-30T05:30:38.0558022Z running 1132 tests
…
2026-10-30T05:30:42.2232385Z test host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol ... FAILED
…
---- host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol stdout ----
thread '<unnamed>' (844) panicked at src\host\cli\session\tests.rs:153:46:
called `Result::unwrap()` on an `Err` value: Kind(BrokenPipe)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol' (4704) panicked at src\host\cli\session\tests.rs:825:19:
called `Result::unwrap()` on an `Err` value: Any { .. }
…
test result: FAILED. 1125 passed; 1 failed; 6 ignored; 0 measured; 0 filtered out; finished in 46.94s
The first panic is the guest thread's, in Guest::send; the second is the test's join of that thread. The test's assertion, that the exec fails with the output-limit error, held, since the test reached the join.
History
I checked the logs of every failed nvx-rs / * job in all 1,248 CI runs created from 2026-10-19, the day that #596 merged with this test, through 2026-10-30 17:00Z, including earlier attempts of rerun runs and runs that ended cancelled or green: 32 failed jobs. This is the only failure of this test, or of any host::cli::session test, among them. None of the 96 nvx-rs mutants runs in the same window failed in it either.
The test is unchanged since #812 (2026-10-23), which changed only its assertion helper, and Guest::send and stream_pair are unchanged since #596 added them. #1022, the last change to the session's code before this run, changed only how the session connects.
Root cause
Permalinks point to 4434b1a. The session's code and tests are byte-identical on the current dev head, 78c626d.
- The guest sends past the point where the host gives up. The test's guest answers the exec with 18
APP_STDOUTrecords of 60,000 bytes and then anAPP_EXITrecord (tests.rs#L806-L826). Their 1,080,000 bytes exceedMAX_OUTPUT_BYTES, 1 MiB (exec.rs#L31), so the host'sexecfails on the 18th record with the output-limit error and never reads the exit record (session.rs#L519-L532). - The session stays open until the test drops it, before joining the guest. Only a failed transfer closes the session (session.rs#L349-L353); a protocol violation returns its error and keeps the transport. The host's end of the stream therefore closes when the test drops the session (tests.rs#L824), and only then does the test join the guest's thread (#L825).
- The guest never waits for the host, and unwraps every send.
session_withruns the guest overstream_pair(64)(tests.rs#L158-L169), whose guest-to-host channel holds 64 records (#L32-L47), so the guest sends its 20 records, itsREADYincluded, without waiting for the host to read them. Without a timeout, a send can fail only withBrokenPipe, once the host's end is gone (#L76-L80), andGuest::sendunwraps it (#L149-L154). - So the test races the guest's thread. The guest usually sends its exit record long before the host has decoded 18 records of 60,000 bytes. If its thread is descheduled after the 18th output record for as long as the host takes to read up to that record, fail, check the error, and drop the session, the exit record meets a closed stream. On the runner, the test ran among the library's other tests, 4 s after they started.
No other test in tests.rs has this exposure. The others that drop the session before joining the guest, exec_rejects_unbounded_or_relative_arguments, exec_rejects_invalid_execution_fields_before_sending, exec_rejects_invalid_response_timeout_before_sending, session_reset_is_not_a_closed_endpoint, and response_that_does_not_arrive_times_out, have guests that send nothing once the host has given up.
Reproduction
I used a 20-thread Windows 11 Enterprise workstation (10.0.26300) with Rust 1.95.0 at 78c626d, whose nvx-rs differs from 4434b1a only in src/guest/sys/rooted.rs. No hypervisor is needed.
- As is. The test alone (
--exact,--test-threads 1) passed 20,400 of 20,400 runs, 40–120 processes at a time: the window is narrow. - With a wider window. With
thread::sleep(Duration::from_millis(200));before the guest's exit record (tests.rs#L815), the test failed 5 of 5 runs with the CI signature:Kind(BrokenPipe)attests.rs:153:46, then the join's panic. - With the fix below. With the same pause, and the guest joined before the session is dropped, the test passed 5 of 5 runs, and the module's 25 tests passed.
# PowerShell 7, from nvx-rs/ at 78c626d, with the pause added.
$env:NVX_BUNDLE = 'skip'
cargo +1.95.0 test --locked --all-features --lib -- --exact host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocolFix
Join the guest before the session ends: swap the test's last two statements, or remove its drop(session), so that the session ends with the test. The guest never waits for the host, so the join cannot hang. The assertion stays as it is, and Guest::send keeps its unwrap, which still fails the test for a send that fails for any other reason.
Activity
nvx-rs: join the output-limit test's guest before its session ends