Fish 3.7 on Linux: output and NMSh markers from Fish sometimes never reach NMSh (seen 3 times in CI)

Seen once in CI, not reproduced, cause unknown. Found while investigating CI for #442 (part of #431).

What happened

CI run 39178446961, job Tests (ubuntu-24.04, 2/2), tests/absolutePathLive.test.ts, live fish: absolute executable paths run in the shell with their arguments; NMSh commands still open:

timed out waiting for /ADB\[devices\][\s\S]*ADB\[-l\]/u

The fixture: adb is a two-line /bin/sh script, printf "ADB[%s]\n" "$@", run by absolute path as …/adb devices -l.
What the screen showed since the command was typed (872 characters, complete): the command line, Running …/adb devices -l · <1 ms, Enter queues the next command · Ctrl+S sends to adb, ✔ Completed · 58 ms, then repeated prompts and the idle animation.
The gap: the command completed with exit 0, but its output never reached the frontend at all. This is not a wait that matched too early; the text was never there.

Environment

Runner: Ubuntu 24.04, Fish 3.7.0 (apt).
Code: the PR's change is test-only, in other files, so this ran dev's code (a8b383c2).

What was checked

macOS, Fish 4.9.3: the same test passes 3/3 on dev.
The 30 earlier failed CI runs: this failure does not appear in any of them.
The test is not timing-sensitive in this step: it uses Frontend.run, which also waits for NMSh's journal.

Behind the test: NMSh learns that Fish's own read has the terminal from the fish_read event marker.
What the screen showed: that marker never arrived within 10 s. The activity row stayed Running read …, and the typed answer stayed held (Can't tell whether read is reading your typing · held, not shown). Held safely, but never delivered.

Three Ubuntu Fish 3.7 failures this week, each with Fish output missing:

Input ownership: keys wait for the look, however long a loaded machine makes it (#426) #433's first run 39156227005: a queued printf 'AFTER-%s\n' printed write: Interrupted system call and exited 1.
The adb output above.
This reader marker.

None reproduces with Fish 4.9.3 on macOS.

Hypothesis (unverified): NMSh resizes the shell's terminal when its layout changes, for example when the activity row appears as a command starts. That sends SIGWINCH to the foreground group. If Fish 3.7 does not retry a write the signal interrupts, the output or marker is lost; failure 1 shows exactly that error.

To verify: in a Linux container with Fish 3.7, run a loop of fast commands and reads while resizing the PTY, with and without NMSh, and trace write with strace -f -e trace=write -e signal=SIGWINCH. If it is confirmed, the fix belongs in NMSh, which can avoid resizing during a command's first moments or coalesce resizes, rather than in the tests.

v0.22 ledger: integration candidate verified, crash recovery, test-re…

Diagnostic for #443: 30 iterations and a process watchdog (state, wch…

Cause found, with a system-call trace: Fish 3.7 loses a terminal write that SIGWINCH interrupts, and NMSh sends SIGWINCH on every change to its own layout. The behaviour is the same in v0.20.0, so this is not a regression.

How it was found. A bounded GitHub-hosted loop (branch diag/443-fish37, Ubuntu 24.04, Fish 3.7.0, never master or dev) ran the Fish cases of absolutePathLive, inputOwnershipLive, commandQueueLive and typingAfterRead repeatedly, on the v0.22 code. Fish was wrapped in strace for half of the runs.

Run 39221968682: 1 of 15 plain iterations failed, the run 'a b' output case. 0 of 15 under strace.
Run 39223268583: 3 of 30 plain iterations failed, and 2 of 30 under strace.

The trace of a failure (queue test, strace iteration 17). This is Fish writing one of NMSh's shell markers to the terminal:

08:37:02.647734 write(1, "\33]777;nmsh;caf4a9b1283fc9c6a4103"..., 51) = ? ERESTARTSYS
08:37:02.647895 --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
08:37:02.648113 write(2, "w", 1) ... write(2, "Interrupted system call\n", 24)

The write was waiting in the kernel when a window-size change arrived.
Fish 3.7 does not retry the write: it prints write: Interrupted system call, and the marker is lost.
A plain-run failure (iteration 10) printed the same error, as did Input ownership: keys wait for the look, however long a loaded machine makes it (#426) #433's first CI failure.
Flow control is not involved: the terminal's c_iflag has no IXON.

Why SIGWINCH is so frequent. The shell's terminal height is NMSh's transcript region (plan.ptyRows), and it is applied on every render (TerminalApp: if (this.lastPtyRows !== plan.ptyRows …) this.session.resize(…)). Every change to NMSh's own layout therefore resizes the terminal and signals the foreground job: the activity row, queue rows, notices, suggestions or the composer's height. The interactive Fish of the failing test got 14 such signals in about 20 s, with the test window never resized. All the Fish processes of that iteration together got about 230.

Scope

Linux, Fish 3.x: shown with 3.7.0.
Not reproduced with Fish 4 on macOS.
zsh and Bash were not affected in these runs.
What is lost is any write that is blocked when the signal arrives: a command's output, or one of NMSh's markers. A lost marker can leave NMSh waiting for a prompt or for a read event.

Assessment for v0.22.0: a documented limitation of Linux (beta) with Fish 3, not a release blocker, because:

it is present in v0.20.0 (the same resize code) and v0.21;
it is intermittent: about 1 iteration in 10–15 under CI load, each running several Fish tests;
the fix changes how NMSh sizes the shell's terminal, which also affects full-screen programs and the multi-row progress region, so it is too risky to make during the release freeze.

Fix options (later). Each needs this loop to show zero failures before this issue is closed.

Resize the shell's terminal only when the real window size changes, or coalesce layout-driven resizes while a command runs.
In Fish's integration, retry a marker write that fails.

Diagnostic for #443: keep every interrupted marker write and what Fis…

Release v0.22.0 candidate: security gate review, limitation wording, …

With #445 (Fish markers written again once on a failed write), on the v0.22.0 release head: run 39229412842 had 0 failed iterations in 30 plain and 0 in 30 under strace. Before #445, the same loops failed 6 times in 75 iterations (runs 39221968682 and 39223268583).

This is encouraging, but it is not proof:

No interrupted marker write was captured in this run, so the retry itself was not observed working.
Output from the person's own commands is unchanged and can still be lost this way.

The issue stays open. The changelog documents the remaining limitation and its consequences.

Fish integration: on Fish 3, markers go through cat, so an interrupte…

Release v0.22.0 candidate: post-QA stabilization (F01–F04, #426, #443)…

v0.22: markers fixed on Fish 3, output loss from Fish's own builtins remains (candidate fa7ec665).

Deterministic evidence instead of timing. tests/fishMarkerWrite.test.ts preloads a shim that makes a terminal write() fail with EINTR a set number of times (as SIGWINCH does; a pipe write of a few bytes never waits, so it isn't failed). It runs in a real PTY on CI's Ubuntu Fish 3.7.0.

The shim is faithful: Fish 3 drops its own interrupted echo output and prints write: Interrupted system call, as in the strace above. Fish 4 retries.
Fish integration: write a marker again once when the write fails (#443) #445's single retry saved a marker interrupted once, but still printed the error into the transcript: Fish 3 reports a failed builtin write from its own output layer, so the builtin's 2>/dev/null never silenced it. It lost a marker interrupted twice. NMSh's resizes come in bursts, so that matters.

Fix (#449): on Fish 3 a marker is printf'd into a pipe and written to the terminal by cat. cat ignores SIGWINCH and coreutils retries EINTR. The marker travels on stdin, never in arguments. Fish 4 keeps the builtin. On Ubuntu Fish 3.7 a marker now arrives exactly once, with no error shown, after 1, 2 or 50 interruptions.

Live evidence: the diagnostic loop (run 39252443113, branch diag/443-cat, never merged) on the new helper went 60/60 (30 plain, 30 under strace), with no interrupted marker write recorded. Earlier: 3/30 and 2/30 failures before #445, and 0/60 with two interrupted-and-retried writes on #445.

Still open: output written by Fish itself (echo, printf, string, functions using them) can still lose a write when a resize lands while it waits, on Fish 3 only. Programs Fish starts are unaffected. The root cause is NMSh resizing the shell's PTY on every change to its own layout. The proper fix (resize only for real window changes, or coalesce while a command runs) changes how full-screen programs and progress regions see the terminal, so it isn't a release-freeze change. Changelog Known limitations updated to say exactly this. Keeping the issue open for it.

Tests: Fish output integrity churns NMSh's composer while Fish writes (…

Diagnostic for #443: Fish 3.7 loop, release candidate vs stable PTY s…

Tests: live commands typed for the prompt wait for it; Dock Top and p…

Docs: v0.22 hardening ledger, exit-time typing limitation (#426, #443)

Diagnostic for #443: Fish 3.7 loop, release candidate vs stable PTY s…

Docs: v0.22 hardening evidence (CI, Fish 3.7 loop, signal counts) and…

Merge #457: v0.22 hardening: input ownership by lifecycle (#426), sta…

Fixed at the cause (#457, release/v0.22.0 = e8dd44e4). The shell's terminal rows now come from a reference layout for the window and settings. While a command runs, only a real window resize changes them.

Deterministic: an ordinary sequence sent 6 UI-only resizes on fa7ec665, and sends 0 now.
Live, SIGWINCH counted by a real foreground program: NMSh's chrome sent 2, 2 and 3 signals (zsh, Bash, Fish 4) on fa7ec665, and sends 0 now. Both builds send exactly 1 per window resize.
Fish 3.7 on Ubuntu (run 39290923773): 11 Fish tests × 20 iterations, 0 failures.
Caveat: the new stress test (Fish builtin output while the composer churns) did not reproduce output loss on the baseline either. So no loss-rate improvement is claimed; the claim is that the cause is removed.
Limit: a real window resize while Fish 3 writes its own output can still lose part of it. That is Fish 3's handling of the signal, the same in any terminal.

Stays open for physical QA (R7).

Navigation Menu

Repository navigation

Fish 3.7 on Linux: output and NMSh markers from Fish sometimes never reach NMSh (seen 3 times in CI) #443

What happened

Environment

What was checked

Next

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Fish 3.7 on Linux: output and NMSh markers from Fish sometimes never reach NMSh (seen 3 times in CI) #443

Description

What happened

Environment

What was checked

Next

Activity

raiseCatError commented on Oct 19, 2026

raiseCatError commented on Oct 19, 2026

raiseCatError commented on Oct 19, 2026

raiseCatError commented on Oct 19, 2026

raiseCatError commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions