Skip to content

nvx-rs: the output-limit session test can close the stream before its guest's last send, so output_beyond_the_guest_limit_breaks_the_protocol panics with BrokenPipe (flaky nvx-rs / windows-latest) #1051

Closed

Description

Summary

The dev push run for 4434b1a (#1039) failed in nvx-rs / windows-latest because host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol panicked. The thread that plays the guest in that test got BrokenPipe from its last send, and the test's join of that thread then failed. The guest answers an exec with 18 output records of 60,000 bytes and then an exit record. As the test expects, the host fails the exec on the 18th record, past the 1 MiB output limit. The test then drops the session, which closes the host's end of the in-memory stream, and only afterwards joins the guest. If the guest's thread has not sent its exit record by then, that send fails, and the guest unwraps the error.

#1039 did not cause this: it changes neither the session nor its tests, its own CI run passed both nvx-rs jobs, and the race has existed since #596 added the test. The code is unchanged on the current dev head. A 200 ms pause before the guest's last send fails the test every time with the CI signature, and joining the guest before dropping the session makes it pass. This is a flaky test, not a product defect.

Incident

Decisive output:

2026-10-30T05:30:38.0558022Z running 1132 tests
…
2026-10-30T05:30:42.2232385Z test host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol ... FAILED
…
---- host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol stdout ----

thread '<unnamed>' (844) panicked at src\host\cli\session\tests.rs:153:46:
called `Result::unwrap()` on an `Err` value: Kind(BrokenPipe)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol' (4704) panicked at src\host\cli\session\tests.rs:825:19:
called `Result::unwrap()` on an `Err` value: Any { .. }
…
test result: FAILED. 1125 passed; 1 failed; 6 ignored; 0 measured; 0 filtered out; finished in 46.94s

The first panic is the guest thread's, in Guest::send; the second is the test's join of that thread. The test's assertion, that the exec fails with the output-limit error, held, since the test reached the join.

History

I checked the logs of every failed nvx-rs / * job in all 1,248 CI runs created from 2026-10-19, the day that #596 merged with this test, through 2026-10-30 17:00Z, including earlier attempts of rerun runs and runs that ended cancelled or green: 32 failed jobs. This is the only failure of this test, or of any host::cli::session test, among them. None of the 96 nvx-rs mutants runs in the same window failed in it either.

The test is unchanged since #812 (2026-10-23), which changed only its assertion helper, and Guest::send and stream_pair are unchanged since #596 added them. #1022, the last change to the session's code before this run, changed only how the session connects.

Root cause

Permalinks point to 4434b1a. The session's code and tests are byte-identical on the current dev head, 78c626d.

  1. The guest sends past the point where the host gives up. The test's guest answers the exec with 18 APP_STDOUT records of 60,000 bytes and then an APP_EXIT record (tests.rs#L806-L826). Their 1,080,000 bytes exceed MAX_OUTPUT_BYTES, 1 MiB (exec.rs#L31), so the host's exec fails on the 18th record with the output-limit error and never reads the exit record (session.rs#L519-L532).
  2. The session stays open until the test drops it, before joining the guest. Only a failed transfer closes the session (session.rs#L349-L353); a protocol violation returns its error and keeps the transport. The host's end of the stream therefore closes when the test drops the session (tests.rs#L824), and only then does the test join the guest's thread (#L825).
  3. The guest never waits for the host, and unwraps every send. session_with runs the guest over stream_pair(64) (tests.rs#L158-L169), whose guest-to-host channel holds 64 records (#L32-L47), so the guest sends its 20 records, its READY included, without waiting for the host to read them. Without a timeout, a send can fail only with BrokenPipe, once the host's end is gone (#L76-L80), and Guest::send unwraps it (#L149-L154).
  4. So the test races the guest's thread. The guest usually sends its exit record long before the host has decoded 18 records of 60,000 bytes. If its thread is descheduled after the 18th output record for as long as the host takes to read up to that record, fail, check the error, and drop the session, the exit record meets a closed stream. On the runner, the test ran among the library's other tests, 4 s after they started.

No other test in tests.rs has this exposure. The others that drop the session before joining the guest, exec_rejects_unbounded_or_relative_arguments, exec_rejects_invalid_execution_fields_before_sending, exec_rejects_invalid_response_timeout_before_sending, session_reset_is_not_a_closed_endpoint, and response_that_does_not_arrive_times_out, have guests that send nothing once the host has given up.

Reproduction

I used a 20-thread Windows 11 Enterprise workstation (10.0.26300) with Rust 1.95.0 at 78c626d, whose nvx-rs differs from 4434b1a only in src/guest/sys/rooted.rs. No hypervisor is needed.

  • As is. The test alone (--exact, --test-threads 1) passed 20,400 of 20,400 runs, 40–120 processes at a time: the window is narrow.
  • With a wider window. With thread::sleep(Duration::from_millis(200)); before the guest's exit record (tests.rs#L815), the test failed 5 of 5 runs with the CI signature: Kind(BrokenPipe) at tests.rs:153:46, then the join's panic.
  • With the fix below. With the same pause, and the guest joined before the session is dropped, the test passed 5 of 5 runs, and the module's 25 tests passed.
# PowerShell 7, from nvx-rs/ at 78c626d, with the pause added.
$env:NVX_BUNDLE = 'skip'
cargo +1.95.0 test --locked --all-features --lib -- --exact host::cli::session::tests::output_beyond_the_guest_limit_breaks_the_protocol

Fix

Join the guest before the session ends: swap the test's last two statements, or remove its drop(session), so that the session ends with the test. The guest never waits for the host, so the join cannot hang. The assertion stays as it is, and Guest::send keeps its unwrap, which still fails the test for a send that fails for any other reason.

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions