Skip to content

fix: stop disowning a healthy client under output bursts - #80

Merged
wfaderhold21 merged 1 commit into
mainfrom
topic/relay-stall-detach
Aug 19, 2026
Merged

fix: stop disowning a healthy client under output bursts#80
wfaderhold21 merged 1 commit into
mainfrom
topic/relay-stall-detach

Conversation

@wfaderhold21

Copy link
Copy Markdown
Owner

Reported as: on macOS, running omp would randomly detach the viewing session.

It is neither random nor omp-specific — it is the relay stall detector firing on a healthy client, and macOS makes it easy to trip.

Mechanism

Frames reach an attached client over the reattach socket with a 5s write timeout. Three consecutive timeouts were read as "this client is wedged" and answered with a forced AppEvent::Detach (src/main.rs:385-405). Nothing was wedged:

  • macOS gives a unix socket an 8 KiB buffer (net.local.stream.sendspace) where Linux gives ~208 KiB — small enough that a single ratatui frame on a wide terminal can fill it.
  • The server's BufWriter was the default 8 KiB and the client read in 4 KiB chunks, so one frame crossed the socket as many small blocking writes, each able to stall separately.
  • omp is simply a chatty session: it drives redraws faster than the emulator drains. Linux's large buffer absorbs the burst; macOS's does not.

Separately, a SO_SNDTIMEO expiry surfaces as EAGAIN on Linux but can come back as ETIMEDOUT on BSD/macOS, which is_transient_terminal_error did not match. On that path a single timeout skipped the retry budget entirely and detached on the spot.

Changes

  • reattach::widen_socket_buffers — request 512 KiB SO_SNDBUF/SO_RCVBUF on both ends (best-effort; the kernel clamps to kern.ipc.maxsockbuf).
  • Server writes through a 64 KiB BufWriter; client drains in 64 KiB reads.
  • ETIMEDOUT/TimedOut counted as transient slowness rather than a dead client.
  • Stall budget 3RELAY_STALL_LIMIT = 12 (~60s). Safe: a client that has actually gone away is caught immediately by EOF on the relay reader, so this path only governs the stalled-but-connected case.

Testing

cargo clippy clean, cargo fmt applied, 402 tests pass.

Caveat worth a reviewer's attention: this was diagnosed from the code and not reproduced on macOS. The ETIMEDOUT gap is unambiguously a bug; the buffer sizing is the part that explains the "random" character. If detaches persist, the thing to check is whether they correlate with output bursts at all — if they happen while idle, the cause is elsewhere and the relay reader dropping the connection is the next suspect.

🤖 Generated with Claude Code

Frames reach an attached client over the reattach socket with a 5s write
timeout, and three consecutive timeouts were read as "this client is
wedged" and answered with a forced detach. Nothing was wedged. macOS
gives a unix socket an 8 KiB buffer where Linux gives ~208 KiB — small
enough that a single frame can fill it — so a burst from a chatty session
outran the emulator's drain and the server disowned a terminal that was
merely behind. Hence the reports of random detaches on a Mac while
running omp, and none on Linux.

A `SO_SNDTIMEO` expiry also comes back as EAGAIN on Linux but can surface
as ETIMEDOUT on BSD/macOS, which `is_transient_terminal_error` did not
match. On that path a single timeout skipped the retry budget entirely
and detached on the spot.

Both ends now request a 512 KiB socket buffer, the server buffers a whole
frame before writing and the client drains in 64 KiB reads, ETIMEDOUT is
recognized as slowness, and the stall budget is a minute rather than
fifteen seconds. Being patient costs nothing here: a client that has
actually gone away is still caught immediately by EOF on the relay
reader, so this path only ever governs the stalled-but-connected case.

Diagnosed from the code — not reproduced on macOS.

Co-Authored-By: Claude <noreply@anthropic.com>
@wfaderhold21
wfaderhold21 merged commit 1ca56ab into main Aug 19, 2026
2 checks passed
@wfaderhold21
wfaderhold21 deleted the topic/relay-stall-detach branch August 19, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant