runtime-adapter: a guest listener's port in Linux's ephemeral range can keep OpenVMM from publishing it, so one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address fails with listener 124 (flaky NVX microVM tests / Linux / KVM / Guests and sandboxes) #1318

Summary

The merge group of #1192 failed in NVX microVM tests / Linux / KVM / Guests and sandboxes because the runtime adapter scenario vm_clone one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address never reached a listener of a clone's guest that the clone's policy publishes on host loopback: the guest's nc -l waited its 30 s and timed out (listener 124). The scenario failed the same way twice before on Linux/KVM, on other heads. #1192 did not cause it: it changes only the Ubuntu guest's build scripts, their tests, and docs, and the replacement merge group, of the same tree, passed the scenario.

The likely cause is the test's choice of port. It draws the guest listener's port from 49152–65535, most of which lies in Linux's default ephemeral range, 32768–60999, and nothing holds the port until OpenVMM publishes the listener there. While nothing listens, reach_guest's retries to 127.0.0.1:PORT can open a connection to themselves, once the kernel picks the port itself as a retry's source port. Closing that self-connection leaves a TIME_WAIT at 127.0.0.1:PORT for 60 s, which makes OpenVMM's bind of the publication fail with EADDRINUSE past the listener's 30 s. On a Linux kernel, such retries self-connected at even ports and never at odd ones, and the three Linux failures' ports were all even and all inside the range. The failing clones' VMM logs, which would show the failed binds, are not kept.

Incident

• Workflow: CI run 42289202400, attempt 1, merge_group for ubuntu: build runtimes with Python and architecture-independent packages #1192.
• Head: d29f8ccfb6dd9d328e48c7a92c34277c5d60d451, dev at 8b6257ac merged with ubuntu: build runtimes with Python and architecture-independent packages #1192's head 404fee6f, with OpenVMM f1e1d8545 as its openvmm gitlink. The files linked below are the same at 8b6257ac.
• Failed job: NVX microVM tests / Linux / KVM / Guests and sandboxes, step Run runtime adapter scenarios on Linux (python3 scripts/nvx.py test-runtime-adapter --backend kvm). The scenario was the only failure among the suite's 156; vm_clone passed 34 of its 35 tests.
• Runner: self-hosted azure-kvm-2 (self-hosted, linux, kvm, virtual-machine, general), Linux 6.6.159.1-1.azl3.
• Downstream: Required status check failed, and the merge queue removed ubuntu: build runtimes with Python and architecture-independent packages #1192 at 20:53:05Z (failed_checks).
• Replacement: run 42292619757, head 77fa2677, whose tree is d29f8ccf's. Its KVM guests job passed the scenario on azure-kvm-4. The run failed instead in NVX microVM tests / Windows / WHP / Runtime adapter on azure-windows-4, with ci: a cold boot's time sample exceeded 100 µs, and a clone's repair failed with 195, on azure-windows-1 (flaky NVX microVM tests / Windows / WHP / Runtime adapter) #1309's signature (G_CONFORMANCE_C12, time sample uncertainty 142270 ns exceeds 100000 ns), and the queue removed ubuntu: build runtimes with Python and architecture-independent packages #1192 again at 21:18:57Z.

Decisive output:

test one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address ... FAILED
thread 'one_template_serves_the_clones_of_every_network_policy_at_its_guest_s_address' (2000121) panicked at tests/vm_clone.rs:1633:9:
assertion `left == right` failed: 127.0.0.1:49514: {"epoch": "4", "exec.1.cause": "none", "exec.1.launch": "started", "exec.1.settled": "reusable", "exec.1.status": "exit 0", "exec.1.stderr": "", "exec.1.stdout": "listener 124\\n", "exec.1.truncated": "none", "instance": "48769676a08ca5d83b63b383e840ae96", "pid": "2000660", "reattached": "live"}
  left: false
 right: true

assert_listener runs nc -l in the clone's guest at HOST_SIDE.listener, 49514 here, for 30 s, while reach_guest retries 127.0.0.1:49514 every 100 ms. left: false says that the host never exchanged a line with the listener, and listener 124 that nc timed out without a client. Epoch 4 is the clone's fourth session, after its start, the network view, and the reach check, so this was the check on host loopback of the HostLoopback or the Forward clone, the only two of the scenario's seven networks that publish the guest's listeners there.

History

Across the 400 CI runs from 2026-11-01 to 2026-11-06 20:57Z, the runtime adapter's step:

• Linux/KVM: 233 passed, 3 failed, all with this signature, and 3 were cancelled.
• Linux/MSHV, which runs the same scenario: 236 passed, none failed, and 5 were cancelled.
• Windows/WHP's Runtime adapter job: 160 passed, 3 failed, and 1 was cancelled. One failure has the same symptom in vm_layered; the other two are ci: a cold boot's time sample exceeded 100 µs, and a clone's repair failed with 195, on azure-windows-1 (flaky NVX microVM tests / Windows / WHP / Runtime adapter) #1309.

The vm_layered failure on WHP came from the positive control that runs before that scenario: a cold-booted VM of Network::Open, whose host-loopback ingress publishes the guest's listeners, at check_guest_listener's assertion. Windows picks source ports differently, so whether it shares this cause is open.

No earlier occurrence was found: the runtime adapter's failures from 2026-10-29, when #978 merged the scenario, to 2026-11-01 had other signatures.

Root cause

Likely the test's choice of port, not NVX or OpenVMM:

1. HostSide::bind draws HOST_SIDE.listener once per test binary with free_port, from PORTS, 49152–65535. free_port binds the port to check it and releases it at once, so nothing holds the port until OpenVMM publishes a listener there. 72% of PORTS, 49152–60999, lies in Linux's default ephemeral range, 32768–60999, from which connect() picks source ports.
2. Each of the scenario's checks of a listener runs reach_guest, which retries TcpStream::connect(127.0.0.1:PORT) every 100 ms until the check ends: for its 5 s where the clone's policy must not publish the listener, under the Egress, AllowRule, Outbound, Proxy, and Ingress networks, and until OpenVMM publishes the listener otherwise. That is roughly 230 connections to a port at which nothing listens.
3. When connect() picks the destination port itself as a connection's source port, the connection opens to itself, as a TCP simultaneous open. Linux's connect() tries the even ports of the default range first and leaves odd ones to bind(0) (__inet_hash_connect() in net/ipv4/inet_hashtables.c), so only an even port is likely to self-connect. reach_guest then reads nothing until its 5 s read timeout, and closes the connection, which stays in TIME_WAIT at 127.0.0.1:PORT for 60 s.
4. On Linux, OpenVMM publishes a listener on host loopback by binding 127.0.0.1:PORT with SO_REUSEADDR (ingress.rs#L77-L118). A TIME_WAIT whose socket did not set SO_REUSEADDR, which TcpStream::connect's does not, fails that bind with EADDRINUSE. OpenVMM logs failed to publish a guest port and retries with each announcement (ingress.rs#L240-L276), which the guest repeats every 2 s, until the TIME_WAIT ends, 60 s after the close. A self-connection in the 35 s or so before the published check, or in that check before the publication, therefore outlasts the listener's 30 s.

Evidence:

• The three Linux failures' ports, 49892, 52656, and 49514, are all even, and all inside 32768–60999. 36% of draws are both, so all three would be with probability 5% by chance.
• On Linux 6.18 (WSL2), with the default ephemeral range and a replica of free_port and of reach_guest's connections (below), 18 of 1,000 runs of 230 attempts at a free even port of 49152–60999 self-connected, and none of 1,000 at an odd port. The self-connection read nothing for 5 s; once closed, it stayed in TIME-WAIT 127.0.0.1:P 127.0.0.1:P, and SO_REUSEADDR binds of 127.0.0.1:P every 2 s, as OpenVMM retries them, failed with EADDRINUSE for 62 s.
• That predicts a failure in up to 0.36 × 1.8% ≈ 0.65% of Linux runs, fewer where the self-connection comes too early to matter. Linux/KVM and Linux/MSHV together failed 3 of 472 (0.64%).
• Durations fit as well. The scenario passed in 80.3–80.6 s in 7 jobs, KVM and MSHV, but failed after 98.3, 100.9, and 102.4 s. A self-connection in an earlier unpublished check adds up to 5 s to that check, the read timeout that outlives its listener.

What would settle it is the failing clone's VMM log, with failed to publish a guest port and Address already in use, or the host's sockets at the port. Neither survives a failure. The harness prints the end of the VMM's log only when a command of the host program exits with another status, ends before it pauses, or misses its deadline (common/mod.rs#L278-L293, #L342-L373), not when a scenario's assertion fails, and dropping the sandbox removes its directory (#L457-L468). The job's artifact holds only test-microvm's results.

The alternative considered, that OpenVMM publishes the port but the host's connections never reach the guest's nc, would also fail the scenario and spread its durations, but would not explain why all three ports are even.

Reproduction

The mechanism, on any Linux host, without a hypervisor: python3 self_connect.py runs in about 70 s.

#!/usr/bin/env python3
"""Checks, on Linux, whether a loopback client that retries a closed port of the
ephemeral range connects to itself, as vm_clone's reach_guest retries
127.0.0.1:HOST_SIDE.listener, and whether the TIME_WAIT that the
self-connection leaves blocks OpenVMM's later SO_REUSEADDR bind of the port."""
import random
import socket
import subprocess
import time

LOW, HIGH = map(int, open("/proc/sys/net/ipv4/ip_local_port_range").read().split())
FIRST = 49152  # The first of the test's PORTS.


def free(port):
    """Checks the port as free_port does: TCP and UDP, every address, then loopback."""
    for address in ("0.0.0.0", "127.0.0.1"):
        for kind in (socket.SOCK_STREAM, socket.SOCK_DGRAM):
            s = socket.socket(socket.AF_INET, kind)
            try:
                if kind == socket.SOCK_STREAM:
                    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
                s.bind((address, port))
            except OSError:
                return False
            finally:
                s.close()
    return True


def attempt(port):
    """Connects to 127.0.0.1:port as reach_guest does; returns the socket if it connected to itself."""
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    try:
        s.connect(("127.0.0.1", port))
    except ConnectionRefusedError:
        s.close()
        return None
    if s.getsockname() != s.getpeername():
        s.close()
        raise RuntimeError(f"something listens at {port}")
    return s


def draw(parity):
    while True:
        port = random.randrange(FIRST + parity, HIGH + 1, 2)
        if free(port):
            return port


osrelease = open("/proc/sys/kernel/osrelease").read().strip()
print(f"kernel {osrelease}, ephemeral range {LOW}-{HIGH}")

# 1. How often a run of attempts at one closed port connects to itself, by parity.
TRIALS, ATTEMPTS = 1000, 230
for parity, name in ((0, "even"), (1, "odd")):
    hits = 0
    start = time.monotonic()
    for _ in range(TRIALS):
        port = draw(parity)
        for _ in range(ATTEMPTS):
            s = attempt(port)
            if s:
                hits += 1
                s.close()
                break
    print(f"{name} ports {FIRST}-{HIGH}: {hits}/{TRIALS} runs of {ATTEMPTS} attempts self-connected ({time.monotonic() - start:.1f} s)")

# 2. What a self-connection does to a later publication of its port.
port = draw(0)
attempts = 0
while True:
    attempts += 1
    s = attempt(port)
    if s:
        break
print(f"port {port}: attempt {attempts} self-connected: {s.getsockname()} -> {s.getpeername()}")
s.settimeout(5)
start = time.monotonic()
try:
    print(f"read {s.recv(1)!r}")
except socket.timeout:
    print(f"read nothing for {time.monotonic() - start:.1f} s, as reach_guest's 5 s read timeout")
s.close()
closed = time.monotonic()
print("after close: " + subprocess.run(["ss", "-Htan", f"( sport = :{port} )"], capture_output=True, text=True).stdout.strip())
failures, first_error = 0, None
while True:
    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    try:
        listener.bind(("127.0.0.1", port))
        listener.listen()
    except OSError as error:
        failures += 1
        first_error = first_error or error
        listener.close()
        if time.monotonic() - closed > 90:
            print(f"gave up after {failures} failed binds: {first_error}")
            break
        time.sleep(2)
        continue
    listener.close()
    print(f"{failures} SO_REUSEADDR binds of 127.0.0.1:{port}, 2 s apart, failed with {first_error}; the next succeeded {time.monotonic() - closed:.1f} s after the close")
    break

kernel 6.18.33.2-microsoft-standard-WSL2, ephemeral range 32768-60999
even ports 49152-60999: 18/1000 runs of 230 attempts self-connected (1.8 s)
odd ports 49152-60999: 0/1000 runs of 230 attempts self-connected (1.7 s)
port 60792: attempt 2223 self-connected: ('127.0.0.1', 60792) -> ('127.0.0.1', 60792)
read nothing for 5.0 s, as reach_guest's 5 s read timeout
after close: TIME-WAIT 0      0      127.0.0.1:60792 127.0.0.1:60792
31 SO_REUSEADDR binds of 127.0.0.1:60792, 2 s apart, failed with [Errno 98] Address already in use; the next succeeded 62.0 s after the close

The scenario, deterministically, on a Linux/KVM host: just before the HostLoopback clone's check on host loopback, bind a socket to 127.0.0.1:HOST_SIDE.listener, connect it to that same address, which opens a connection to itself, and close it. The check should then fail with listener 124, and the clone's VMM log should show failed to publish a guest port. On Linux 6.18, such a socket self-connected, and the TIME_WAIT that it left failed an SO_REUSEADDR bind of its port with EADDRINUSE. This has not been run on a KVM host.

Impact

• Required status check fails. This took ubuntu: build runtimes with Python and architecture-independent packages #1192 out of the merge queue, failed the dev push run of ci: check nvx's public API against committed snapshots #1261 together with an unrelated nvx-rs / ubuntu-latest failure, and failed host: make a capture one value, which names its snapshots #1275's pull-request run.
• About 1 in 80 Linux/KVM runs failed. Linux/MSHV runs the same scenario and is as exposed.
• No product defect was found: OpenVMM refuses to publish a port that a host socket holds, and retries it with the next announcement, as designed.
• vm_layered draws its guest listener's ports from the same range, so it is exposed too, though less: each scenario draws ports of its own, and its positive control publishes them after few attempts.

Suggested fix

1. Draw the ports of a guest's listener and of a forward outside the host's ephemeral range, in vm_clone.rs and vm_layered.rs, so that no connect() takes them: on Linux, above the end of /proc/sys/net/ipv4/ip_local_port_range, which leaves 61000–65535 of PORTS by default. Windows's ephemeral range is its whole dynamic range, 49152–65535, so whether it needs the same, and which ports its exclusions leave (ci: host-loopback port-pair retry exhausts its attempts inside a Windows TCP port exclusion (flaky NVX CLI / Windows) #473, test-microvm: draw host-loopback port pairs independently #477), is open.
2. Have reach_guest drop a connection that opened to itself, whose local and peer addresses are the same, at once, with SO_LINGER 0, so that it leaves no TIME_WAIT. This alone still leaves the port to the host's other loopback connections, such as OpenVMM's to the host's services on behalf of the clones, which can take it as their source port too.
3. When a check of a listener fails, print the end of the VMM's log and the host's sockets at the port (ss -tan), so that a recurrence shows what kept the listener from the host.

Related work

• nvx-rs: per-clone network policies on deny-all templates #978 added the scenario.
• runtime-adapter: reach_guest gives up on connections that Windows still opens, so allowed_host_loopback_publishes_the_listeners fails (flaky NVX microVM tests / Windows / WHP) #1033 and runtime-adapter: let each connection to a guest listener open or fail #1034: an earlier flake of the same assertion on Windows/WHP, with listener 0, from connections that Windows still opened after reach_guest gave up on them. It was fixed on 2026-10-30, before every occurrence above.
• ci: a cold boot's time sample exceeded 100 µs, and a clone's repair failed with 195, on azure-windows-1 (flaky NVX microVM tests / Windows / WHP / Runtime adapter) #1309: the time-ABI flake that failed this merge group's replacement run, and whose history lists run 42095876865 as unexplained.
• runtime-adapter: a KVM clone whose restore brings 2 or more CPUs online can stop answering once its sandbox starts, so the vm_clone processor-target scenarios time out (flaky NVX microVM tests / Linux / KVM / Guests and sandboxes) #1050: another flake of the same job, in vm_clone's processor-count scenarios.