diff --git a/docs/superpowers/plans/2026-06-08-terminal-catchup-stream-safety.md b/docs/superpowers/plans/2026-06-08-terminal-catchup-stream-safety.md new file mode 100644 index 000000000..407922a7b --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-terminal-catchup-stream-safety.md @@ -0,0 +1,2849 @@ +# Terminal Catch-Up Stream Safety Implementation Plan + +> **For agentic workers:** Execute this plan in one implementation worktree and one final PR. Local commits and internal task checkpoints are fine, but do not publish a PR until every task and final local proof gate passes. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make long-hidden terminal catch-up fast, loss-explicit, and safe across server replay batching, xterm parser semantics, attach races, side-effect parsing, and WebSocket backpressure. + +**Proof status:** Implementation can proceed from this plan. The evidence dossier is committed at `docs/superpowers/proofs/2026-06-08-terminal-catchup-evidence-dossier.md`. It resolves the prior open architecture questions with reproducible probes and source inspection. The browser catch-up acceptance gate for the single implementation PR is the local production-style visible-first audit plus a local browser process-suspend stop/resume positive control on an isolated test server. A real Windows Chrome long-background soak remains useful release/user-acceptance evidence, but it is not a prerequisite for opening the single PR. + +**Architecture:** Keep server-side replay batching as the primary performance fix, but turn it into a protocol-aware stream system. The server owns replay retention, batching, serialized byte budgets, gaps, and backpressure; the client owns xterm surface identity, attach generation safety, parser-applied acknowledgements, and side-effect gating. Paint is a UX signal only, not a replay safety boundary. + +**Tech Stack:** TypeScript, Node.js ESM, React 18, xterm 6.0.0 behavior probes with exact dependency pinning or CI-probed dependency policy, ws WebSockets, Zod client-to-server protocol schemas, TypeScript server-to-client message unions, Vitest, Testing Library, Playwright visible-first audit tooling, structured JSONL logs. + +--- + +## Findings That Led To This Plan + +### The Original Symptom + +A long-running Codex terminal tab stayed hidden for hours. When the tab became visible again, Freshell replayed/caught up very slowly, at roughly a few times realtime. The user-visible problem was terminal catch-up latency after a long-hidden tab. + +### The Incomplete Former Fixes + +PR #396, `fix-terminal-catchup`, attacked the problem on the client: + +- It classified replay frames in `src/components/TerminalView.tsx`. +- It added replay mode and replay write coalescing in `src/components/terminal/terminal-write-queue.ts`. +- It originally used a larger 32 ms replay drain budget. + +That was incomplete and partly misdirected: + +- It coalesced after every WebSocket message, JSON parse, message dispatch, sequence-state update, and side-effect parser pass had already happened. +- Its tests used mocked xterm writes with synchronous callbacks, so they did not model xterm parse/render work. +- The 32 ms replay budget was not supported by later measurement. After server batching, it did not materially improve the 1,200-line audit-scale case and worsened RAF gaps on larger stress. +- It did not generation-tag queued writes or callbacks, so delayed writes from old attach generations could still mutate current terminal state. + +PR #397, `fix-replay-server-batching`, moved the primary optimization to the server: + +- It coalesced contiguous replay frames in `server/terminal-stream/replay-ring.ts`. +- It reduced replay WebSocket message count dramatically. A probe showed 1,200 replay lines dropped from 1,200 `terminal.output` messages to 2, and 12,000 lines dropped from 12,000 to 14. + +That direction is correct, but the implementation is not complete: + +- Batching budgets are based on raw UTF-8 `data` bytes, not serialized JSON/WebSocket payload bytes. A 16 KiB raw escape-heavy payload can serialize to about 98 KiB. +- Server replay batching currently concatenates strings and stores frames in arrays that evict with `shift()`, which is unsafe for many tiny retained frames. +- Foreground replay can still create avoidable buffered backpressure before normal pacing checks. +- Batching can change parser side effects unless replay/live context and parser-sensitive boundaries are explicit. + +### Load-Bearing Validation Results + +These facts are now established and must shape the architecture: + +- xterm `Terminal.write(data, cb)` callback is a parser/buffer acknowledgement, not a paint/render acknowledgement. It fires before DOM paint. +- xterm write callbacks are FIFO in the probes, but callback order does not make them a visual render boundary. +- Concatenating contiguous terminal chunks preserves final xterm terminal state for tested terminal sequences, including split SGR, CSI, OSC title, alternate screen, and split UTF-8 input. +- Parser side effects are not universally chunk-boundary invariant. Turn-complete tracking has chunk-local semantics; startup probes, OSC52, and request-mode paths need explicit handling. +- Hidden in-app tabs keep mounted xterm DOM under Freshell's `visibility:hidden` layout, and rAF drains while the page remains active. Full browser-background throttling remains an unverified risk. +- A single sequence number is not enough to prove terminal-visible state across resize/wrapping, alternate-screen state, scrollback, geometry, or paint. +- Existing attach message guards do not protect queued client writes/callbacks. Stale queued writes from old attach generations can still write into the current xterm surface or advance cursors. +- The right performance lever is server message-count reduction, not restoring PR #396's replay drain budget. +- The current client rejects overlapping sequence ranges, so an oversized payload must not be split into multiple `terminal.output` messages with the same `seqStart`/`seqEnd`. Split oversized PTY output before sequence assignment, then give every fragment its own sequence range. +- JavaScript string slicing can split UTF-16 surrogate pairs. Any server splitter must split on Unicode code point boundaries and must test that no emitted chunk contains lone surrogates. +- A `terminal.output.gap` is not an xterm parser acknowledgement. Gap handling must not advance `parserAppliedSeq`; it must track lost ranges separately and invalidate or recreate the surface unless the gap is explicitly parser-safe. +- The persisted cursor is currently keyed only by `terminalId`. Persisted replay checkpoints must include stream/server identity, otherwise a restarted server or replaced stream can make stale local cursors look valid. +- Some parser side effects bypass `handleTerminalOutput`: request-mode replies are emitted from an xterm parser hook, OSC52 `always` can write to the clipboard directly, and title-change callbacks mutate Redux state. Side-effect suppression needs terminal-instance write scope that survives xterm's asynchronous parsing, not only a helper called before enqueueing output. +- Current server-to-client protocol is a TypeScript union, not a runtime Zod schema. Batch protocol work must either add an explicit server-message schema intentionally or test behavior/types directly; it cannot rely on a non-existent `ServerMessageSchema`. +- Current hello capabilities only advertise `uiScreenshotV1`. Batch output requires `terminalOutputBatchV1` negotiation inside protocol v6, plus a non-batch `terminal.output` fallback for v6 clients that omit the capability. +- Serialized byte budgets are exact for the application JSON payload passed to `ws.send`. They are not exact compressed on-wire byte counts. `ws.bufferedAmount` is useful server-side transport pressure, not a browser parser or paint acknowledgement. + +### Second Load-Bearing Pass Results + +The revised plan was load-bearing checked again. These additional facts change the implementation plan: + +- xterm 6 processes writes asynchronously. Wrapping only the synchronous `term.write(data, cb)` call does not expose replay/live context to parser callbacks; CSI, OSC/title, DCS, and write callbacks can run after `term.write` returns and after a stack-scoped context is cleared. +- Already-submitted xterm writes cannot be made safe by callback filtering alone. Same-surface clear/replay hydrate must wait for the write queue to drain, or the client must replace the xterm surface and fence every callback by terminal-instance and attach-generation tokens. +- Fresh xterm plus replay from `sinceSeq=0` is not a universal full hydrate. It matched under the same geometry in probes, but differed under different current geometry and under a resize-history mismatch. Byte replay is a safe hydrate only when geometry history is compatible or replay includes the relevant resize/snapshot history. +- Unsafe gaps can leave xterm's parser desynchronized. Probes with missing OSC, DCS, and CSI boundaries showed later text being swallowed or interpreted incorrectly. A gap marker plus continued output on the same parser is not safe for parser-unsafe gaps. +- A stateless barrier classifier is not enough. Barrier classification must be stream-stateful across raw chunks and fragments, including pending ESC, CSI, OSC, DCS, APC, C1, BEL, startup-probe, request-mode, and turn-complete states. +- Current `node-pty` string mode loses invalid UTF-8 bytes and 8-bit C1 controls. This plan must either explicitly keep the terminal stream contract as UTF-8 string output with replacement semantics, or add a byte-preserving PTY/output protocol. The implementation path below keeps UTF-8 string output for this catch-up fix and treats replacement/control uncertainty as a conservative barrier; a byte-preserving terminal protocol is a separate architecture project. +- Fragmentation can safely happen after raw-output observers if it remains inside `server/terminal-stream`; current Codex/Claude trackers observe `terminal.output.raw` before the broker path. +- Current server attach staging and live-queue ownership are race-free under the existing synchronous broker attach critical section. The implementation must not add `await` points between attach id/mode reset, replay snapshot selection, staging drain, and `mode = 'live'`. +- `ws.bufferedAmount` is confirmed useful for server-side transport/sender pressure in installed `ws` 8.19.0, with the caveat that bytes accepted into the OS socket buffer can be invisible. +- Batch capability plumbing is feasible through hello/client state/broker attachment state, but batch frames must not mix replay/live source or parser-side-effect barriers. Non-batch fallback must emit safe `terminal.output` segments, not flatten arbitrary batches. +- Existing visible-first audit metrics do not yet capture replay message count, serialized replay bytes, parser-applied lag, gaps, full-hydrate fallback, or stale-generation rejection. Observability work must create those metrics before the browser audit can be acceptance evidence. +- xterm probes validate the installed 6.0.0 package, not the whole `^6.0.0` dependency range. Pin xterm exactly or add CI probes that run against every allowed resolved version. + +### Third Load-Bearing Pass Results + +Fresh Eyes and a third load-bearing pass found these additional constraints: + +- xterm `dispose()` does not cancel pending `write` callbacks in installed `@xterm/xterm@6.0.0`. Runtime probes showed post-dispose write callbacks after both small and large writes; surface replacement is safe only when every write callback and async parser continuation checks terminal-instance and attach-generation fences before mutating state. +- Serial callback-chained xterm writes are acceptable on the tested desktop Chromium surface. A 50,000-write, 3.25 MB benchmark completed in 267 ms with fast first render and low callback latency, so one submitted write per terminal surface remains viable for live output on that class of machine. Replay should still coalesce; a single giant write hurt first-render latency. +- Side-effect gating by a named allow list was incomplete. Write callbacks that persist/advance checkpoints or complete attaches, link/action callbacks, local terminal notices, parser callbacks, clipboard writes, title updates, startup replies, request-mode replies, and turn-complete mutations all belong under a deny-by-default side-effect adapter. +- Local `term.writeln` notices are real today and are not currently tied to surface invalidation. The plan now requires out-of-band React notices where possible, or explicit surface invalidation before any future warm delta replay. +- `streamId` cannot be a loose optional field. It must be a server-owned output-stream identity that changes on new PTY/session stream, Codex PTY recovery replacement under the same `terminalId`, incompatible retention loss, and server restart without compatible persisted retention. +- Checkpoint compatibility needs geometry authority/history, scrollback, and xterm version in addition to terminal/server/stream/surface identity. Multi-client resize with unknown authority must reject warm delta replay unless the server provides compatible geometry history. +- Replay windows cannot reconstruct stream-stateful barrier scanner state from arbitrary prefixes. Retained frames must store barrier classification and scanner state snapshots at ingestion time. +- Broker output is centralized for current browser attach paths, but broker direct `ws.send(JSON.stringify(...))` lacks the handler send callback, large-payload instrumentation, and shared payload limits. Terminal broker sends must use a shared WebSocket sender. +- Batch protocol and capability negotiation do not exist today, and pre-v6 `terminal.output` lacks source/stream/segment metadata. Non-batch fallback is safe only as individual modern `terminal.output` frames with `seqStart`, `seqEnd`, `streamId`, `attachRequestId`, and segment `data`; it must not use the old registry direct-output shape. +- Full Chrome background/freeze behavior remains hard to reproduce deterministically in this environment. CDP `Page.setWebLifecycleState({ state: 'frozen' })` and Xvfb tab-background probes were tried and disproven as valid local proof because timers, RAF, and WebSocket delivery continued while the probes claimed to be frozen/backgrounded. A process-suspend probe proved the failure mechanic: WebSocket frames accumulate while browser execution is stopped and deliver as a burst after resume. The PR gate therefore uses that stop/resume positive control plus visible-first audit metrics; a real Windows Chrome soak remains post-PR acceptance evidence. + +### Proof Dossier Results + +The proof worktree produced durable evidence artifacts that are now committed with this plan: + +- `docs/superpowers/proofs/2026-06-08-terminal-catchup-evidence-dossier.md` +- `docs/superpowers/proofs/artifacts/terminal-catchup-pty-metrics.json` +- `docs/superpowers/proofs/artifacts/terminal-json-serialization.json` +- `docs/superpowers/proofs/artifacts/xterm-write-dispose.json` +- `docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json` +- `docs/superpowers/proofs/artifacts/browser-background-visibility.json` +- `docs/superpowers/proofs/artifacts/browser-process-suspend.json` +- `scripts/proofs/terminal-catchup-pty-metrics.ts` +- `scripts/proofs/terminal-json-serialization-probe.ts` +- `scripts/proofs/xterm-write-dispose-probe.ts` +- `scripts/proofs/browser-freeze-lifecycle-probe.ts` +- `scripts/proofs/browser-background-visibility-probe.ts` +- `scripts/proofs/browser-process-suspend-probe.ts` + +Decisive proof results: + +- The PTY metrics harness exercised the real Freshell path: `TerminalRegistry` spawning a PTY, `TerminalStreamBroker` attaching, raw `terminal.output.raw` capture, and serialized broker sends. It did not rely on stdout-only logs. +- The `agent-burst-12000` stress trace produced 776,745 bytes in 3,239 raw PTY chunks. Current broker replay compressed that to 33 replay frames, and the conservative scanner would emit 170 batches while respecting control barriers. +- A real Codex turn produced 2,000 bytes in 15 raw chunks. `codex-help` produced 4,686 bytes in 7 raw chunks. +- Serialized JSON budgeting is mandatory. A raw 16 KiB ESC-heavy terminal output serialized to 98,423 bytes; a raw 16 KiB ANSI SGR payload serialized to 30,158 bytes. +- Installed `@xterm/xterm@6.0.0` has asynchronous writes and write callbacks can fire after `dispose()`. Callback fencing by terminal instance, surface epoch, attach generation, and write scope is mandatory. +- Local CDP freeze and Xvfb background probes are invalid acceptance proof here. The process-suspend probe proved the catch-up burst mechanic by stopping the Chromium process tree, sending 40 WebSocket frames, and observing all 40 arrive immediately after resume. +- Current 8 MiB coding-agent replay retention covers 8 hours only at about 291 B/s. A 32 MiB hot memory cap covers 8 hours at about 1,165 B/s; a 256 MiB disk cap covers 8 hours at about 9,320 B/s; a 1 GiB hard cap covers 8 hours at about 37,283 B/s. +- Multi-client geometry, rollout compatibility, sender parity, legacy direct registry output, warm replay validity, and replay side effects are resolved into concrete implementation rules in this plan and the dossier. + +## Design Summary + +### New Safety Vocabulary + +Replace the bare rendered high-water cursor with a parser-applied checkpoint plus separate replay/loss cursors: + +```ts +type TerminalSurfaceCheckpoint = { + terminalId: string + streamId: string | null + serverInstanceId: string + serverBootId?: string + surfaceEpoch: number + attachRequestId: string + parserAppliedSeq: number + cols: number + rows: number + geometryEpoch: number + geometryAuthority: 'single_client' | 'server_stream' | 'multi_client_unknown' + scrollback: number + xtermVersion: string + bufferType: 'normal' | 'alternate' | 'unknown' + parserIdle: boolean +} + +type TerminalReplayCursor = { + parserAppliedSeq: number + highestObservedSeq: number + replayRequestSeq: number + knownLostRanges: Array<{ fromSeq: number; toSeq: number }> +} +``` + +Rules: + +- `parserAppliedSeq` advances only from the active attach generation's xterm write callback. +- It means xterm has parsed/applied the bytes to its model. It does not mean paint happened. +- `knownLostRanges` and `highestObservedSeq` never make a surface safe. They are loss accounting and replay bookkeeping only. +- `replayRequestSeq` is derived from safe parser-applied state. It must not jump over a gap that xterm did not parse. +- UI may wait one visible `requestAnimationFrame` before clearing "recovering" state, but protocol safety must not depend on paint. +- A warm delta replay is valid only when terminal id, stream id, server identity, surface epoch, geometry, geometry authority, scrollback, xterm version, parser state, and attach generation remain compatible. +- Retained replay from `sinceSeq=0` is a replay hydrate, not a universal full hydrate. It is trusted only when replay covers a compatible geometry history. Otherwise the old surface is quarantined or replaced; the client must not continue live I/O on a parser whose state was rebuilt from an untrusted hydrate. +- Explicit refresh, terminal replacement, unsafe geometry change, scrollback-setting change, parser-unsafe gap, stale persisted checkpoint, and stale in-flight writes use drain-then-hydrate, fresh-surface, quarantined-loss, or future snapshot paths. They must not advance a safe replay cursor. +- Local client notices such as gaps, reconnecting, launch errors, blocked input, and status banners should be rendered out-of-band in React UI rather than written into xterm. Any remaining local `term.write`/`term.writeln` path must bump `surfaceEpoch`, mark the surface local-mutated, and make the checkpoint ineligible for warm delta replay. + +### Server Batching Rules + +Batching belongs in `server/terminal-stream`, not the client write queue. + +A batch may coalesce frames only when all of these are true: + +- Same terminal id and stream id. +- Same attach request id. +- Same source context, either `live` or `replay`. +- Contiguous sequence ranges. +- No parser side-effect barrier between frames. +- Serialized application JSON payload bytes stay under the configured budget. +- The stream-stateful barrier scanner is in ground state before and after the coalesced span. + +The server must not silently coalesce across these barriers: + +- Gaps. +- Attach control. +- Terminal exit. +- Resize or geometry epoch changes. +- OSC52-sensitive spans. +- Request-mode query/reply paths. +- Startup probe phases. +- Client-authoritative turn-complete paths. +- ESC, OSC, DCS, APC, C1, BEL, and other control spans until classified safely. +- Incomplete/pending control spans from prior raw chunks or fragments. + +The terminal stream contract for this plan remains UTF-8 string output. Invalid UTF-8 and raw 8-bit C1 bytes are not preserved by the current `node-pty` string-mode path, so the server must conservatively treat replacement characters and uncertain control bytes as barriers. A byte-preserving PTY protocol must be designed separately before claiming byte-stream-perfect terminal replay. + +### Protocol Direction + +The single implementation branch makes protocol v6 the compatibility boundary because safe post-attach terminal traffic requires server-owned `streamId` on output, batch, gap, attach-ready, and stream-change messages. Protocol v5 peers must be rejected during `hello` rather than accepted into a stream that would silently drop untagged output. Within protocol v6, explicit batches remain gated by `terminalOutputBatchV1`; v6 clients that omit that capability receive segmented `terminal.output` frames with sequence and stream metadata. + +```ts +type TerminalOutputBatch = { + type: 'terminal.output.batch' + terminalId: string + streamId: string + attachRequestId: string + source: 'live' | 'replay' + seqStart: number + seqEnd: number + data: string + serializedBytes: number + segments: Array<{ + seqStart: number + seqEnd: number + endOffset: number + data?: string + rawFrameCount: number + barrier?: 'control' | 'startup_probe' | 'osc52' | 'request_mode' | 'turn_complete' | 'gap' | 'geometry' + }> +} +``` + +For this plan, `streamId` is server-owned output-stream identity. It is minted when a terminal output stream is created, remains stable across attach/detach/replay for that stream, and changes when the server replaces the stream identity, loses retention across restart, or intentionally starts a new PTY/session stream. Protocol v6 makes `streamId` mandatory on terminal stream messages; any missing or mismatched stream identity after attach is fail-closed by the client. + +`segments[].endOffset` is a UTF-16 code-unit offset into the batch `data` string, matching JavaScript `String.prototype.slice` semantics. It must always fall on a code-point boundary; the batch builder must test emoji/surrogate-pair segment boundaries. If that contract becomes too fragile during implementation, replace offsets with required per-segment `data` and drop top-level slicing rather than leaving offset units implicit. + +Protocol v6 clients that do not advertise `terminalOutputBatchV1` continue to receive compatible `terminal.output` messages, but the fallback must serialize safe batch segments as individual frames. It must not flatten an arbitrary multi-segment batch into one `terminal.output` if that batch crosses replay/live source, parser-barrier, stream-id, attach-id, or budget boundaries. Server-to-client runtime validation is not currently present; if this work adds it, create the schema explicitly and test it as a new behavior. + +The client processes segments in order and runs side-effect parsers with explicit context: + +```ts +type TerminalOutputSideEffectContext = { + source: 'live' | 'replay' + attachRequestId: string + segment: { + seqStart: number + seqEnd: number + barrier?: string + } +} +``` + +Replay context suppresses external side effects such as clipboard prompts, request-mode replies, title updates, and client-minted turn-complete notifications. Because xterm write parsing is asynchronous, context must be terminal-instance scoped and remain associated with the submitted write until its xterm write callback fires. The write queue must allow at most one submitted xterm write per terminal surface unless it can prove parser callbacks are unambiguous for all in-flight writes. Parser callbacks and local terminal notices use a deny-by-default side-effect adapter: any new xterm parser callback, browser side effect, Redux mutation, PTY reply, clipboard action, or local xterm write must declare an effect type and be explicitly allowed for the active terminal-instance write scope. + +## Evidence-Backed Single-Branch Execution Order + +Execute the detailed tasks below in one implementation worktree and publish one PR only after the full local proof gate passes. The phases below are local continuation gates, not PR boundaries. Local commits after each task are still useful for review and recovery, but they all remain on the same implementation branch. + +Before opening the final PR: + +- All task-level red-green-refactor gates must pass locally. +- The focused client, server, parser side-effect, e2e, visible-first, and full coordinated checks in Final Verification must pass locally. +- The implementation must be proven against an isolated local test server on a unique port. Do not stop or restart the self-hosted dev server. +- The local browser proof gate must pass on an isolated local test server: production-style visible-first audit plus process-suspend stop/resume positive-control evidence. A real Windows Chrome long-background soak is recommended release/user-acceptance evidence, not a blocker for opening this PR. + +### Phase 1: Sender, Metrics, And Protocol-Neutral Safety + +Purpose: remove unsafe terminal send paths before adding richer replay behavior. + +Use these work packages: + +- Add `server/ws-send.ts` from the server file-structure section. +- Apply the shared sender portions of Task 8 and Task 10. +- Route broker and registry direct terminal sends through the shared sender. +- Keep output protocol as segmented `terminal.output` until Phase 5. + +Local gate before continuing: + +- Broker, registry direct terminal output, and `ws-handler` use the same serialization, ready-state, payload-measurement, backpressure, send-callback, and structured logging behavior. +- No browser-visible terminal path can emit unsequenced old-shape `terminal.output` without passing through the shared sender. +- Structured logs include severity, terminal id, stream id when known, attach id when known, seq range when applicable, serialized bytes, buffered amount, and rejection reason. + +### Phase 2: Server Stream Identity, Scanner, And Retention Coverage + +Purpose: make server replay safe and measurable before the client trusts warm replay. + +Use these work packages: + +- Task 5: serialized payload budgeting and pre-sequence fragmentation. +- Task 6: stream-stateful barrier-aware batching. +- Task 7: replay deque and retained scanner metadata. +- Add stream identity and retention coverage reporting from the server file-structure section. + +Local gate before continuing: + +- `streamId` is server-minted, stable across attach/detach, and changes on new PTY/session stream, Codex recovery PTY replacement, incompatible retention loss, and restart without compatible persisted retention. +- Oversized output is fragmented before sequence assignment and never splits Unicode surrogate pairs. +- Batch candidates never cross stream, attach, source, geometry, gap, scanner barrier, or serialized payload budget boundaries. +- Retention reports exact coverage for every attach; missing coverage emits a gap/quarantine reason and never advances parser-applied state. +- Memory and optional disk retention defaults reflect the dossier sizing: 32 MiB hot memory for coding-agent terminals, optional 256 MiB disk spool default, and 1 GiB configurable hard cap. + +### Phase 3: Client Checkpoint, Write Queue Fencing, And Side Effects + +Purpose: make the client safe to consume server replay without stale callbacks or replay side effects corrupting state. + +Use these work packages: + +- Task 1: client write generation safety fence. +- Task 2: parser-applied surface checkpoint. +- Task 3: TerminalView parser-applied cursor and attach generations. +- Task 4: async xterm write scope and side-effect suppression. + +Local gate before continuing: + +- Every queued/submitted xterm write and callback is fenced by terminal instance, surface epoch, attach generation, and write scope. +- `parserAppliedSeq` advances only from the active fenced xterm write callback. +- Gap receipt never advances `parserAppliedSeq`. +- Replay suppresses PTY replies, request-mode replies, OSC52 writes/prompts, title updates, client-minted turn completion, link opens, and local terminal writes by default. +- Any remaining local xterm write bumps `surfaceEpoch` and invalidates warm delta replay. + +### Phase 4: Geometry Authority And Warm Replay Policy + +Purpose: make warm replay opt-in based on known-compatible terminal geometry and surface identity. + +Use these work packages: + +- Geometry parts of Task 2 and Task 3. +- Server geometry epoch/history work from the server file-structure section. +- Attach policy updates from the client file-structure section. + +Local gate before continuing: + +- Warm replay is accepted only when terminal id, stream id, server identity, attach generation, surface epoch, geometry, geometry authority/history, scrollback, xterm version, parser-applied checkpoint, retention coverage, and sequence continuity are compatible. +- `geometryAuthority='multi_client_unknown'` quarantines or rebuilds instead of warm-replaying. +- `terminal.resize` updates geometry epoch/history and invalidates incompatible checkpoints. + +### Phase 5: Batch Capability And Non-Batch Fallback + +Purpose: add explicit batch protocol after both sides are safe and backwards compatible. + +Use these work packages: + +- Task 9: batch protocol. +- Server batch builder pieces from Task 6 that were not needed by segmented `terminal.output`. +- Client batch segment parsing in `TerminalView` and `terminal-attach-seq-state`. + +Local gate before continuing: + +- `terminal.output.batch` is sent only to clients advertising `terminalOutputBatchV1`. +- Batch-capable protocol v6 clients still accept segmented `terminal.output`. +- Protocol v5 clients are rejected at `hello`; protocol v6 clients without batch capability receive only segmented `terminal.output` with seq and stream metadata, never `terminal.output.batch`. +- Non-batch fallback emits the same safe segments as batch mode and never flattens across barriers or budgets. + +### Phase 6: Browser Acceptance And Local Proof + +Purpose: prove the implemented system handles the user-visible catch-up scenario and preserves the safety invariants under a browser that actually stops or throttles page execution. + +Use these work packages: + +- Task 10 observability that is not already complete. +- Task 11 browser-level verification. +- Final verification. + +Local pre-PR gate: + +- Visible-first audit records replay message count, serialized replay bytes, parser-applied lag, gap count/ranges, warm replay accepted/rejected reason, stale callback rejection count, side-effect suppression count, retention coverage, and browser lifecycle state. +- Local process-suspend or equivalent positive-control testing proves catch-up burst handling when browser execution is stopped. +- The local process-suspend stop/resume positive-control gate passes on an isolated local test server. CDP freeze or Xvfb tab switching cannot substitute for this gate in this environment unless their counters prove browser execution actually stopped or throttled. +- Only after these gates pass should the branch be pushed and a single implementation PR opened. + +## File Structure + +### Client + +- Modify `src/components/terminal/terminal-write-queue.ts` + - Own queued and submitted write generation metadata. + - Own terminal-instance metadata for submitted xterm writes. + - Submit at most one xterm write per terminal surface at a time unless a later implementation proves parallel submitted writes are context-safe. + - Drop stale queued writes. + - Suppress stale callbacks. + - Report in-flight submitted writes that cannot be canceled. + +- Create `src/lib/terminal-surface-checkpoint.ts` + - Define `TerminalSurfaceCheckpoint`. + - Validate whether an existing xterm surface can be used for delta replay. + - Name semantics as parser-applied, not rendered. + - Include stream/server identity, geometry authority, scrollback, and xterm version so persisted checkpoints cannot survive incompatible server restarts, stream replacements, resize history, scrollback changes, or xterm upgrades. + +- Modify `src/lib/terminal-attach-policy.ts` + - Take a checkpoint instead of a bare rendered sequence. + - Return replay hydrate with an untrusted resulting surface, fresh-surface replacement, or future snapshot recovery when geometry, stream id, surface epoch, or parser state is unsafe. + +- Modify `src/lib/terminal-attach-seq-state.ts` + - Continue to handle sequence ranges, but do not imply sequence range equals full surface validity. + - Keep parser-applied cursor advancement separate from gaps and known lost ranges. + - Accept batch segment ranges in the batch protocol task on this branch. + +- Modify `src/lib/terminal-cursor.ts` + - Persist parser-applied checkpoints with stream/server identity. + - Reject checkpoint reads when stream or server identity is missing or incompatible. + +- Modify `src/components/TerminalView.tsx` + - Rename rendered high-water state to parser-applied high-water. + - Track `surfaceEpoch`, `streamId`, geometry, geometry authority, scrollback, xterm version, and attach generation. + - Pass generation metadata into write queue. + - Drain submitted writes before same-surface clear/replay hydrate, or replace the xterm surface and fence callbacks by terminal-instance token. + - Associate replay/live write context with the submitted xterm write until its callback fires, not only while `term.write` is on the JavaScript stack. + - Move local gap/status/error notices to an out-of-band React overlay where possible; any remaining local xterm write invalidates the surface for warm delta replay. + +- Create `src/lib/terminal-output-side-effects.ts` + - Centralize terminal-output side-effect decisions. + - Deny by default for replay or unknown write scope. + - Require every xterm parser callback, xterm write callback that advances checkpoint/attach state, clipboard write, PTY reply, title update, turn-complete mutation, startup reply, link/action callback, and local xterm notice write to declare an effect type before it can run. + +- Modify `src/components/terminal/request-mode-bypass.ts` + - Consult terminal-instance write scope before sending request-mode replies. + +- Modify `src/lib/terminal-osc52.ts` + - Suppress both prompted and `always` clipboard writes during replay. + +- Test `test/unit/client/components/terminal/terminal-write-queue.test.ts` + - Generation tagging, stale queued write dropping, stale callback suppression, in-flight tracking. + +- Test `test/unit/client/lib/terminal-surface-checkpoint.test.ts` + - Checkpoint compatibility and invalidation on geometry, stream, attach, and surface epoch changes. + - Persisted checkpoint invalidation on server identity changes. + +- Test `test/unit/client/lib/terminal-attach-policy.test.ts` + - Warm delta only when checkpoint is compatible. + +- Test `test/unit/client/lib/terminal-attach-seq-state.test.ts` + - Same-seq duplicates remain rejected. + - Gaps do not advance `parserAppliedSeq`. + +- Test `test/unit/client/components/TerminalView.lifecycle.test.tsx` + - Delayed xterm write callback from an old attach cannot advance current parser-applied cursor or complete current attach. + - Unsafe stale in-flight write forces drain-or-surface-replace behavior. + - Same-surface hydrate waits for xterm write drain; non-draining stale writes replace the xterm surface and bump `surfaceEpoch`. + - Parser-unsafe output gaps quarantine or replace the surface instead of continuing to write later output into a desynchronized parser. + - Replay request-mode replies, OSC52 writes, title updates, stale write-callback checkpoint updates, and attach-completion mutations are suppressed. + - Link/action callbacks are token-fenced to the current terminal surface or explicitly declared outside replay gating. + +### Server + +- Create `server/terminal-stream/output-barrier-scanner.ts` + - Track stream-stateful parser barrier state across raw chunks and fragments. + - Conservative first version treats ESC, BEL, C1, OSC, CSI, DCS, APC, replacement characters, startup-probe spans, request-mode spans, and most control spans as barriers. + - Expose whether the scanner is in ground state so batches only coalesce spans that start and end safely. + +- Create `server/terminal-stream/stream-identity.ts` + - Mint and store server-owned `streamId` values for terminal output streams. + - Change `streamId` on new PTY/session stream, Codex PTY recovery replacement, incompatible retention loss, and server restart without compatible persisted retention. + - Keep `streamId` stable across attach/detach for the same output stream. + +- Create `server/terminal-stream/serialized-budget.ts` + - Compute exact serialized application JSON payload byte size using the same payload shape passed to `ws.send`. + - Provide code point-safe helper functions for finding the largest data segment that fits within a payload budget. + +- Create `server/terminal-stream/output-fragments.ts` + - Fragment oversized raw PTY output before sequence assignment. + - Guarantee emitted chunks do not split surrogate pairs. + - Preserve raw observer ordering by running inside `server/terminal-stream`, after `terminal.output.raw` subscribers have seen the original string event. + - Document that this task does not change the current UTF-8 string terminal contract; byte-preserving PTY output is a separate project. + +- Create `server/terminal-stream/replay-deque.ts` + - Replace many-frame replay retention with a deque or indexed ring that does not evict with `Array.shift()`. + - Retain byte and frame counts. + - Retain per-frame barrier metadata and scanner state before/after the frame so arbitrary `sinceSeq` replay windows do not reconstruct stream-stateful parser state from the wrong prefix. + - Support efficient bounded replay reads. + +- Modify `server/terminal-stream/replay-ring.ts` + - Either wrap `ReplayDeque` for compatibility or migrate callers to the new deque. + - Assign distinct sequence ranges after fragmentation. + - Keep gap semantics. + - Attach stream identity and barrier metadata to retained frames. + +- Create `server/terminal-stream/output-batch.ts` + - Build batches from replay or live frames. + - Preserve segment metadata. + - Enforce serialized payload budget. + - Stop at barriers. + - Treat `segments[].endOffset` as a UTF-16 code-unit offset and prove offsets land on code-point boundaries, or switch to required per-segment `data`. + +- Create `server/ws-send.ts` + - Own JSON serialization, serialized application byte measurement, `ws.send`, `bufferedAmount` reads, send callback accounting, structured send/backpressure logs, and closed-socket handling. + - Export a shared sender used by both `server/ws-handler.ts` and `server/terminal-stream/broker.ts`. + +- Modify `server/terminal-stream/client-output-queue.ts` + - Use the same batch builder for live queued output. + - Avoid divergent replay/live batching semantics. + +- Modify `server/terminal-stream/broker.ts` + - Use batch builder for replay cursor flushes. + - Pace foreground sends before creating avoidable buffered backpressure. + - Keep background pause and catastrophic protection. + - Preserve the synchronous attach critical section; do not add `await` between attach reset, replay snapshot, staging drain, and `mode = 'live'`. + - Use the shared WebSocket sender instead of calling `ws.send` directly. + - Emit structured JSONL logs for replay, batching, gaps, and pressure. + +- Modify `shared/ws-protocol.ts` + - Add required terminal stream identity metadata for protocol v6 terminal stream messages. + - Define `streamId` lifecycle explicitly: server-minted per terminal output stream, stable across attach/detach for that stream, changed on stream replacement, restart without compatible retention, or new PTY stream. + - Add `terminalOutputBatchV1` capability negotiation before emitting `terminal.output.batch`. + - Add `terminal.output.batch` typing and client/server support in this branch. + +- Test `test/unit/server/terminal-stream/output-barrier-scanner.test.ts` + - Transparent text can batch. + - ESC/BEL/OSC/DCS/CSI/APC/control spans stop batches. + - Split `ESC [`, OSC, DCS, APC, and startup-probe spans remain barriers until the terminating state is observed. + - Replacement characters from lossy PTY decoding are barriers. + - Scanner state snapshots before and after a frame can be stored with replay retention and later used without mutating the live scanner. + +- Test `test/unit/server/terminal-stream/serialized-budget.test.ts` + - Escape-heavy data stays within serialized byte budget. + - Oversized raw output is fragmented before sequence assignment. + - Fragments never contain lone surrogates. + +- Test `test/unit/server/terminal-stream/replay-deque.test.ts` + - Eviction is O(1)-style and does not degrade with many tiny frames. + - Replay during eviction emits gaps correctly. + +- Test `test/unit/server/terminal-stream/output-batch.test.ts` + - Batch builder preserves contiguous seq ranges, segment metadata, attach id, source, and budget. + - Batch builder never emits multiple same-seq chunks for one frame. + - Batch builder never combines frames unless the scanner starts and ends in ground state. + - Batch segment `endOffset` values are UTF-16 code-unit offsets and never point into the middle of a surrogate pair. + +- Update `test/unit/server/ws-handler-backpressure.test.ts` + - Foreground replay pauses/yields before avoidable buffered growth. + - Background replay still pauses at the background threshold. + +- Update `test/server/ws-terminal-stream-v2-replay.test.ts` + - Replay batches preserve correctness while respecting barriers and budget. + +### Observability + +- Modify `server/terminal-stream/broker.ts` + - Log structured JSONL events with severity fields for: + - `terminal.replay.gap` + - `terminal.replay.batch` + - `terminal.replay.backpressure_pause` + - `terminal.replay.retention` + - `terminal.replay.cursor_lag` + +- Modify `src/components/TerminalView.tsx` + - Log or emit perf marks for: + - parser-applied lag + - stale generation rejection + - surface replacement fallback after stale in-flight writes + +## Task 1: Client Write Generation Safety Fence + +**Files:** +- Modify: `src/components/terminal/terminal-write-queue.ts` +- Modify: `test/unit/client/components/terminal/terminal-write-queue.test.ts` + +- [ ] **Step 1: Add failing tests for generation invalidation** + +Add these tests to `test/unit/client/components/terminal/terminal-write-queue.test.ts`: + +```ts +it('drops queued writes from stale generations before they reach xterm', () => { + const writes: string[] = [] + const callbacks: string[] = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + write: (chunk, onWritten) => { + writes.push(chunk) + onWritten?.() + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.setActiveGeneration('attach-1') + queue.enqueue('old', () => callbacks.push('old'), { generation: 'attach-1' }) + queue.setActiveGeneration('attach-2', { dropQueuedStaleWrites: true }) + queue.enqueue('new', () => callbacks.push('new'), { generation: 'attach-2' }) + + rafCallbacks.shift()?.(16) + + expect(writes).toEqual(['new']) + expect(callbacks).toEqual(['new']) +}) + +it('suppresses stale write callbacks after generation changes', () => { + const callbacks: string[] = [] + const pendingCallbacks: Array<() => void> = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + write: (_chunk, onWritten) => { + if (onWritten) pendingCallbacks.push(onWritten) + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.setActiveGeneration('attach-1') + queue.enqueue('old', () => callbacks.push('old'), { generation: 'attach-1' }) + rafCallbacks.shift()?.(16) + + expect(queue.hasInFlightWrites()).toBe(true) + queue.setActiveGeneration('attach-2', { dropQueuedStaleWrites: true }) + pendingCallbacks.shift()?.() + + expect(callbacks).toEqual([]) + expect(queue.hasInFlightWrites()).toBe(false) +}) +``` + +- [ ] **Step 2: Run the failing queue tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --run test/unit/client/components/terminal/terminal-write-queue.test.ts +``` + +Expected before implementation: TypeScript or test failures because `setActiveGeneration`, `generation`, and `hasInFlightWrites` do not exist. + +- [ ] **Step 3: Implement generation-aware queue API** + +Update `src/components/terminal/terminal-write-queue.ts` with these API additions: + +```ts +export type TerminalWriteQueue = { + enqueue: (data: string, onWritten?: () => void, options?: TerminalWriteQueueOptions) => void + enqueueTask: (task: () => void, options?: TerminalWriteQueueOptions) => void + setActiveGeneration: ( + generation: string, + options?: { dropQueuedStaleWrites?: boolean }, + ) => void + hasInFlightWrites: (generation?: string) => boolean + clear: () => void +} + +export type TerminalWriteQueueOptions = { + mode?: TerminalWriteQueueMode + generation?: string +} +``` + +Implementation rules: + +- Store `generation` on every queue item. +- Keep `activeGeneration`. +- On `setActiveGeneration(next, { dropQueuedStaleWrites: true })`, remove queued items whose `generation !== next`. +- Before invoking a queued item, drop it if it has a stale generation. +- Increment an in-flight counter before calling `args.write`. +- Wrap `onWritten` so stale callbacks decrement in-flight but do not call user callbacks. +- `clear()` must clear queued work and reset in-flight accounting only for not-yet-submitted work. It cannot cancel callbacks already handed to xterm. + +- [ ] **Step 4: Run queue tests to verify pass** + +Run: + +```bash +timeout 120s npm run test:vitest -- --run test/unit/client/components/terminal/terminal-write-queue.test.ts +``` + +Expected: all queue tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add src/components/terminal/terminal-write-queue.ts test/unit/client/components/terminal/terminal-write-queue.test.ts +git commit -m "Add generation safety to terminal write queue" +``` + +## Task 2: Parser-Applied Surface Checkpoint + +**Files:** +- Create: `src/lib/terminal-surface-checkpoint.ts` +- Create: `test/unit/client/lib/terminal-surface-checkpoint.test.ts` +- Modify: `src/lib/terminal-cursor.ts` +- Modify: `test/unit/client/lib/terminal-cursor.test.ts` +- Modify: `src/lib/terminal-attach-seq-state.ts` +- Modify: `test/unit/client/lib/terminal-attach-seq-state.test.ts` +- Modify: `src/lib/terminal-attach-policy.ts` +- Modify: `test/unit/client/lib/terminal-attach-policy.test.ts` + +- [ ] **Step 1: Write failing checkpoint tests** + +Create `test/unit/client/lib/terminal-surface-checkpoint.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { + createTerminalSurfaceCheckpoint, + canUseCheckpointForDeltaReplay, +} from '@/lib/terminal-surface-checkpoint' + +describe('terminal surface checkpoint', () => { + it('accepts a compatible parser-applied checkpoint', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: true, sinceSeq: 42 }) + }) + + it('rejects a checkpoint after geometry changes', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 100, + rows: 40, + geometryEpoch: 4, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'geometry_changed' }) + }) + + it('rejects a checkpoint while parser work is still in flight', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: false, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'parser_busy' }) + }) + + it('rejects a checkpoint from a different server instance', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-b', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'server_changed' }) + }) +}) +``` + +- [ ] **Step 2: Run failing checkpoint tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --run test/unit/client/lib/terminal-surface-checkpoint.test.ts +``` + +Expected: fail because `src/lib/terminal-surface-checkpoint.ts` does not exist. + +- [ ] **Step 3: Implement checkpoint helper** + +Create `src/lib/terminal-surface-checkpoint.ts`: + +```ts +export type TerminalBufferType = 'normal' | 'alternate' | 'unknown' +export type TerminalGeometryAuthority = 'single_client' | 'server_stream' | 'multi_client_unknown' + +export type TerminalSurfaceCheckpoint = { + terminalId: string + streamId: string | null + serverInstanceId: string + serverBootId?: string + surfaceEpoch: number + attachRequestId: string + parserAppliedSeq: number + cols: number + rows: number + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + scrollback: number + xtermVersion: string + bufferType: TerminalBufferType + parserIdle: boolean +} + +export type CheckpointDeltaReplayInput = { + terminalId: string + streamId: string | null + serverInstanceId: string + serverBootId?: string + surfaceEpoch: number + cols: number + rows: number + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + scrollback: number + xtermVersion: string + requireParserIdle: boolean +} + +export type CheckpointDeltaReplayDecision = + | { ok: true; sinceSeq: number } + | { + ok: false + reason: + | 'missing_checkpoint' + | 'terminal_changed' + | 'stream_changed' + | 'server_changed' + | 'surface_changed' + | 'geometry_changed' + | 'geometry_authority_unknown' + | 'scrollback_changed' + | 'xterm_version_changed' + | 'parser_busy' + | 'no_applied_sequence' + } + +function normalizePositiveInteger(value: number): number { + return Number.isFinite(value) ? Math.max(0, Math.floor(value)) : 0 +} + +export function createTerminalSurfaceCheckpoint( + input: TerminalSurfaceCheckpoint, +): TerminalSurfaceCheckpoint { + return { + ...input, + surfaceEpoch: normalizePositiveInteger(input.surfaceEpoch), + parserAppliedSeq: normalizePositiveInteger(input.parserAppliedSeq), + cols: normalizePositiveInteger(input.cols), + rows: normalizePositiveInteger(input.rows), + geometryEpoch: normalizePositiveInteger(input.geometryEpoch), + scrollback: normalizePositiveInteger(input.scrollback), + } +} + +export function canUseCheckpointForDeltaReplay( + checkpoint: TerminalSurfaceCheckpoint | null | undefined, + input: CheckpointDeltaReplayInput, +): CheckpointDeltaReplayDecision { + if (!checkpoint) return { ok: false, reason: 'missing_checkpoint' } + if (checkpoint.terminalId !== input.terminalId) return { ok: false, reason: 'terminal_changed' } + if (checkpoint.streamId !== input.streamId) return { ok: false, reason: 'stream_changed' } + if (checkpoint.serverInstanceId !== input.serverInstanceId) return { ok: false, reason: 'server_changed' } + if (checkpoint.serverBootId && input.serverBootId && checkpoint.serverBootId !== input.serverBootId) { + return { ok: false, reason: 'server_changed' } + } + if (checkpoint.surfaceEpoch !== input.surfaceEpoch) return { ok: false, reason: 'surface_changed' } + if ( + checkpoint.cols !== input.cols + || checkpoint.rows !== input.rows + || checkpoint.geometryEpoch !== input.geometryEpoch + ) { + return { ok: false, reason: 'geometry_changed' } + } + if ( + checkpoint.geometryAuthority !== input.geometryAuthority + || checkpoint.geometryAuthority === 'multi_client_unknown' + ) { + return { ok: false, reason: 'geometry_authority_unknown' } + } + if (checkpoint.scrollback !== input.scrollback) return { ok: false, reason: 'scrollback_changed' } + if (checkpoint.xtermVersion !== input.xtermVersion) return { ok: false, reason: 'xterm_version_changed' } + if (input.requireParserIdle && !checkpoint.parserIdle) return { ok: false, reason: 'parser_busy' } + if (checkpoint.parserAppliedSeq <= 0) return { ok: false, reason: 'no_applied_sequence' } + return { ok: true, sinceSeq: checkpoint.parserAppliedSeq } +} +``` + +- [ ] **Step 4: Add persisted cursor and gap-separation tests** + +Modify `test/unit/client/lib/terminal-cursor.test.ts` and `test/unit/client/lib/terminal-attach-seq-state.test.ts` so they pin the load-bearing findings: + +```ts +it('does not load a persisted checkpoint for a different server instance', () => { + saveTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 1, + attachRequestId: 'attach-1', + parserAppliedSeq: 25, + cols: 80, + rows: 24, + geometryEpoch: 1, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(loadTerminalSurfaceCheckpoint('term-1', { + streamId: 'stream-1', + serverInstanceId: 'server-b', + })).toBeNull() +}) + +it('records gaps without advancing the parser-applied sequence', () => { + const state = createTerminalAttachSeqState() + const afterFrame = onOutputFrame(state, { seqStart: 1, seqEnd: 1 }) + const afterGap = onOutputGap(afterFrame.state, { fromSeq: 2, toSeq: 10 }) + + expect(afterFrame.state.highestObservedSeq).toBe(1) + expect(afterFrame.state.parserAppliedSeq).toBe(0) + expect(afterGap.state.highestObservedSeq).toBe(10) + expect(afterGap.state.parserAppliedSeq).toBe(0) + expect(afterGap.state.knownLostRanges).toEqual([{ fromSeq: 2, toSeq: 10 }]) + expect(afterGap.surfaceSafeForDeltaReplay).toBe(false) + expect(afterGap.requiresSurfaceQuarantine).toBe(true) +}) +``` + +The exact helper names can follow local conventions, but the behavior is load-bearing: observed server order can advance on output and gaps, `parserAppliedSeq` advances only from a fenced xterm write acknowledgement, gaps do not advance parser-applied state, and persisted checkpoints require compatible stream/server identity. + +- [ ] **Step 5: Implement cursor identity and gap separation** + +Update `src/lib/terminal-cursor.ts` to store/load full `TerminalSurfaceCheckpoint` records instead of a terminal-id-only `{ seq, updatedAt }` cursor. Use a migration path that treats old records as incompatible unless the caller explicitly chooses a full hydrate fallback. + +Update `src/lib/terminal-attach-seq-state.ts` so: + +- `parserAppliedSeq` advances only after output is accepted and later acknowledged by xterm. +- `highestObservedSeq` or equivalent server-order bookkeeping can advance on output/gap. +- `knownLostRanges` records gaps. +- `onOutputGap` does not make `parserAppliedSeq` equal `toSeq`. +- Attach policy can see that an unsafe gap requires quarantine, xterm surface replacement, future snapshot recovery, or explicit loss UI. It must not continue writing later output into a parser that may be stuck inside an OSC/DCS/CSI/control sequence. +- Geometry changes and scrollback setting changes invalidate the checkpoint unless the replay stream includes compatible geometry history. + +- [ ] **Step 6: Update attach policy tests** + +Modify `test/unit/client/lib/terminal-attach-policy.test.ts` so warm reveal uses a checkpoint, not a bare rendered sequence. Add this test: + +```ts +it('falls back to viewport hydrate when the parser-applied checkpoint is unsafe', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'viewport_hydrate', + pendingReason: 'hidden_reveal', + checkpointDecision: { ok: false, reason: 'geometry_changed' }, + })).toMatchObject({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + }) +}) +``` + +Add this geometry-history test: + +```ts +it('does not treat replay from zero as trusted full hydrate without compatible geometry history', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'viewport_hydrate', + pendingReason: 'hidden_reveal', + checkpointDecision: { ok: false, reason: 'geometry_changed' }, + replayHydrateCoversCompatibleGeometryHistory: false, + })).toMatchObject({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, + }) +}) +``` + +- [ ] **Step 7: Modify attach policy** + +Update `src/lib/terminal-attach-policy.ts` so `RevealAttachPolicyInput` uses: + +```ts +import type { CheckpointDeltaReplayDecision } from './terminal-surface-checkpoint' + +export type RevealAttachPolicyInput = { + pendingIntent: TerminalAttachIntent + pendingReason: DeferredAttachReason + checkpointDecision: CheckpointDeltaReplayDecision + replayHydrateCoversCompatibleGeometryHistory?: boolean +} +``` + +Add `trustResultingSurfaceForDeltaReplay?: boolean` to the `RevealAttachPlan` return type. Leave it absent for ordinary plans where the resulting surface remains governed by existing compatibility checks, and set it explicitly to `false` when replay hydrate is chosen after an unsafe checkpoint without compatible geometry history. + +Use `checkpointDecision.ok ? checkpointDecision.sinceSeq : undefined` when choosing delta replay. Replay hydrate from zero remains the default for explicit refresh or unsafe checkpoint, but it must set `trustResultingSurfaceForDeltaReplay: false` unless the server/client can prove compatible geometry history. + +Leave `TerminalView.tsx` call-site migration to Task 3 so Task 2 can finish the policy helper in isolation. Task 3 must construct `checkpointDecision` with `canUseCheckpointForDeltaReplay(...)` before calling `resolveRevealAttachPlan`; intermediate commits are allowed to have a TODO adapter only if the focused tests still pass and the next task removes it before `npm run check`. + +- [ ] **Step 8: Run focused tests** + +Run: + +```bash +timeout 180s npm run test:vitest -- --run test/unit/client/lib/terminal-surface-checkpoint.test.ts test/unit/client/lib/terminal-cursor.test.ts test/unit/client/lib/terminal-attach-seq-state.test.ts test/unit/client/lib/terminal-attach-policy.test.ts +``` + +Expected: pass. + +- [ ] **Step 9: Commit** + +```bash +git add src/lib/terminal-surface-checkpoint.ts test/unit/client/lib/terminal-surface-checkpoint.test.ts src/lib/terminal-cursor.ts test/unit/client/lib/terminal-cursor.test.ts src/lib/terminal-attach-seq-state.ts test/unit/client/lib/terminal-attach-seq-state.test.ts src/lib/terminal-attach-policy.ts test/unit/client/lib/terminal-attach-policy.test.ts +git commit -m "Model terminal catch-up checkpoints explicitly" +``` + +## Task 3: TerminalView Uses Parser-Applied Cursor And Attach Generations + +**Files:** +- Modify: `src/components/TerminalView.tsx` +- Modify: `test/unit/client/components/TerminalView.lifecycle.test.tsx` +- Modify: `test/e2e/terminal-create-attach-ordering.test.tsx` + +- [ ] **Step 1: Add failing delayed callback lifecycle test** + +Add this test inside the `attach sequence v2` describe block in `test/unit/client/components/TerminalView.lifecycle.test.tsx`, next to the existing stale attach tests: + +```ts +it('does not let stale write callbacks advance the current parser-applied cursor', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-stale-write-callback', + serverInstanceId: 'server-a', + streamId: 'stream-1', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + const delayedCallbacks: Array<() => void> = [] + term.write.mockImplementation((_data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push(onWritten) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'old replay text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + expect(delayedCallbacks).toHaveLength(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const secondAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(secondAttach?.attachRequestId).toBeTruthy() + expect(secondAttach?.attachRequestId).not.toBe(firstAttach?.attachRequestId) + + saveTerminalSurfaceCheckpoint({ + terminalId, + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 1, + attachRequestId: secondAttach!.attachRequestId, + parserAppliedSeq: 0, + cols: 80, + rows: 24, + geometryEpoch: 1, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + act(() => { + delayedCallbacks.shift()?.() + }) + + const checkpointAfterStaleCallback = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-1', + serverInstanceId: 'server-a', + }) + expect(checkpointAfterStaleCallback).not.toBeNull() + expect(checkpointAfterStaleCallback?.attachRequestId).toBe(secondAttach?.attachRequestId) + expect(checkpointAfterStaleCallback?.parserAppliedSeq).toBe(0) + + const currentAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(currentAttach?.attachRequestId).toBe(secondAttach?.attachRequestId) +}) +``` + +- [ ] **Step 2: Run the failing lifecycle test** + +Run: + +```bash +timeout 180s npm run test:vitest -- --run test/unit/client/components/TerminalView.lifecycle.test.tsx -t "stale write callbacks" +``` + +Expected before implementation: fail because stale write callbacks are not generation checked. + +- [ ] **Step 3: Rename cursor semantics in TerminalView** + +In `src/components/TerminalView.tsx`: + +- Rename local "rendered" cursor refs and functions to parser-applied names. +- Replace user-facing comments that imply paint/render acknowledgement. +- Keep behavior unchanged until generation checks are added. + +Use names: + +```ts +const parserAppliedSeqRef = useRef(0) +const markParserAppliedSeq = useCallback((terminalId: string | undefined, seq: number, context: { + streamId: string | null + serverInstanceId: string + attachRequestId: string + cols: number + rows: number + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + scrollback: number + xtermVersion: string +}) => { + if (!terminalId || !Number.isFinite(seq)) return + const parserAppliedSeq = Math.max(0, Math.floor(seq)) + if (parserAppliedSeq <= parserAppliedSeqRef.current) return + parserAppliedSeqRef.current = parserAppliedSeq + saveTerminalSurfaceCheckpoint({ + terminalId, + streamId: context.streamId, + serverInstanceId: context.serverInstanceId, + surfaceEpoch: surfaceEpochRef.current, + attachRequestId: context.attachRequestId, + parserAppliedSeq, + cols: context.cols, + rows: context.rows, + geometryEpoch: context.geometryEpoch, + geometryAuthority: context.geometryAuthority, + scrollback: context.scrollback, + xtermVersion: context.xtermVersion, + bufferType: currentBufferTypeRef.current ?? 'unknown', + parserIdle: !writeQueueRef.current?.hasInFlightWrites(context.attachRequestId), + }) +}, []) +``` + +- [ ] **Step 4: Pass active attach generation into queued writes** + +When calling `handleTerminalOutput`, pass the current attach generation into `enqueueTerminalWrite` through `TerminalWriteQueueOptions`: + +```ts +handleTerminalOutput( + raw, + mode, + tid, + !frameOverlapsReplay, + completeParserAppliedFrame, + { + mode: frameOverlapsReplay ? 'replay' : 'live', + generation: currentAttachRef.current?.attachRequestId ?? 'no-attach', + }, +) +``` + +Before beginning a new attach, call: + +```ts +writeQueueRef.current?.setActiveGeneration(nextAttachRequestId, { + dropQueuedStaleWrites: true, +}) +``` + +If `writeQueueRef.current?.hasInFlightWrites()` is true and the new attach would clear or full-hydrate the current surface, do not clear the existing xterm in place. Prefer a bounded drain. If the drain does not complete, recreate the xterm surface and increment `surfaceEpoch`. + +- [ ] **Step 5: Gate parser-applied callbacks by generation** + +Change frame completion callbacks to check attach generation before mutating state: + +```ts +const completeParserAppliedFrame = () => { + const activeAttach = currentAttachRef.current + if (!activeAttach || activeAttach.attachRequestId !== frameAttachRequestId) return + markParserAppliedSeq(tid, frameDecision.state.lastSeq, { + streamId: frameDecision.state.streamId, + serverInstanceId: frameDecision.state.serverInstanceId, + attachRequestId: activeAttach.attachRequestId, + cols: frameDecision.state.cols, + rows: frameDecision.state.rows, + geometryEpoch: frameDecision.state.geometryEpoch, + geometryAuthority: frameDecision.state.geometryAuthority, + scrollback: terminalScrollback, + xtermVersion: XTERM_VERSION, + }) + if (completedAttachOnFrame) { + setIsAttaching(false) + markAttachComplete() + } +} +``` + +- [ ] **Step 6: Run client lifecycle tests** + +Run: + +```bash +timeout 240s npm run test:vitest -- --run test/unit/client/components/TerminalView.lifecycle.test.tsx test/e2e/terminal-create-attach-ordering.test.tsx test/e2e/terminal-flaky-network-responsiveness.test.tsx +``` + +Expected: pass. + +- [ ] **Step 7: Commit** + +```bash +git add src/components/TerminalView.tsx test/unit/client/components/TerminalView.lifecycle.test.tsx test/e2e/terminal-create-attach-ordering.test.tsx +git commit -m "Fence terminal catch-up by attach generation" +``` + +## Task 4: Async Xterm Write Scope And Side-Effect Suppression + +**Files:** +- Create: `src/lib/terminal-output-write-scope.ts` +- Create: `src/lib/terminal-output-side-effects.ts` +- Create: `test/unit/client/lib/terminal-output-write-scope.test.ts` +- Create: `test/unit/client/lib/terminal-output-side-effects.test.ts` +- Modify: `src/components/TerminalView.tsx` +- Modify: `src/components/terminal/terminal-write-queue.ts` +- Modify: `src/components/terminal/request-mode-bypass.ts` +- Modify: `src/lib/terminal-osc52.ts` +- Modify: `test/unit/client/components/terminal/terminal-write-queue.test.ts` +- Modify: `test/unit/client/lib/terminal-osc52.test.ts` +- Modify: `test/unit/shared/turn-complete-signal.test.ts` +- Modify: `test/unit/client/components/TerminalView.lifecycle.test.tsx` + +- [ ] **Step 1: Add async write-scope tests** + +Create `test/unit/client/lib/terminal-output-write-scope.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { + beginTerminalOutputWriteScope, + getTerminalOutputWriteScope, + shouldAllowTerminalOutputSideEffect, +} from '@/lib/terminal-output-write-scope' + +describe('terminal output write scope', () => { + it('keeps replay context visible until the submitted write completes', () => { + const scope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-1', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + + expect(getTerminalOutputWriteScope('surface-1')?.source).toBe('replay') + scope.complete() + expect(getTerminalOutputWriteScope('surface-1')).toBeNull() + }) + + it('suppresses external side effects during replay writes', () => { + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-1', + source: 'replay', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'osc52_clipboard_write', + mode: 'shell', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'title_update', + mode: 'shell', + })).toBe(false) + }) +}) +``` + +Add focused regression tests in existing suites: + +- A probe-style unit test models xterm's async behavior: `term.write` returns before parser callbacks, and the write scope still suppresses replay side effects until the callback completes. +- `request-mode-bypass` does not call `sendInput` while the terminal instance's submitted write scope is replay. +- OSC52 policy `always` does not write to the clipboard while the terminal instance's submitted write scope is replay. +- `TerminalView` ignores or defers xterm title callbacks fired during replay-scope writes. +- xterm write callbacks from stale or replay-suppressed scope do not advance parser-applied checkpoints, persist cursors, or complete attaches. +- local notices are rendered out-of-band or explicitly mark the terminal surface incompatible for warm delta replay. +- link/action callbacks are fenced to the current terminal-instance token or declared outside replay parsing. +- Startup probes and client-minted turn-complete signals are still allowed for live output and suppressed for replay output. + +- [ ] **Step 2: Run failing scope and side-effect tests** + +Run: + +```bash +timeout 180s npm run test:vitest -- --run test/unit/client/lib/terminal-output-write-scope.test.ts test/unit/client/lib/terminal-output-side-effects.test.ts test/unit/client/lib/terminal-osc52.test.ts test/unit/shared/turn-complete-signal.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx +``` + +Expected: fail because submitted-write scope does not exist and replay parser callbacks are not guarded. + +- [ ] **Step 3: Implement terminal-instance write scope** + +Create `src/lib/terminal-output-write-scope.ts`: + +```ts +export type TerminalOutputSource = 'live' | 'replay' + +export type TerminalOutputSideEffect = + | 'startup_reply' + | 'osc52_prompt' + | 'osc52_clipboard_write' + | 'request_mode_reply' + | 'title_update' + | 'turn_complete' + | 'parser_applied_checkpoint' + | 'attach_completion' + | 'cursor_persist' + | 'link_action' + | 'terminal_action' + | 'local_xterm_notice' + +export type TerminalOutputWriteContext = { + terminalInstanceId: string + source: TerminalOutputSource + attachRequestId: string | undefined + generation: string + suppressExternalSideEffects: boolean +} + +const activeScopes = new Map() + +export function getTerminalOutputWriteScope( + terminalInstanceId: string | undefined, +): TerminalOutputWriteContext | null { + if (!terminalInstanceId) return null + return activeScopes.get(terminalInstanceId) ?? null +} + +export function beginTerminalOutputWriteScope( + context: TerminalOutputWriteContext, +): { complete: () => void } { + activeScopes.set(context.terminalInstanceId, context) + let completed = false + return { + complete: () => { + if (completed) return + completed = true + if (activeScopes.get(context.terminalInstanceId) === context) { + activeScopes.delete(context.terminalInstanceId) + } + }, + } +} +``` + +Create `src/lib/terminal-output-side-effects.ts` and export `shouldAllowTerminalOutputSideEffect(input)` from there. Replay or unknown scope suppresses all external side effects by default. Live output still follows existing mode-specific rules, including server-authoritative turn-complete for Claude and Codex. New side effects must fail closed until tests explicitly allow them for live/current-surface scope. + +- [ ] **Step 4: Make xterm writes serial per terminal surface** + +Update `src/components/terminal/terminal-write-queue.ts` and `src/components/TerminalView.tsx` so the queue submits at most one xterm write at a time for a terminal surface. Start the write scope immediately before submitting the xterm write, and complete it only in the xterm write callback. Do not submit the next queued write until the callback for the current submitted write has run. + +Update existing queue tests at the same time. Any queue mock that accepts an `onWritten` callback must either call it explicitly or keep it in a pending callback list and drive it from the test. The existing live-mode time-slice tests must not accidentally stall just because the production queue now waits for xterm write completion before submitting the next write. If an implementation proves a separate live-only fast path is context-safe, cover that proof with tests; otherwise the tests should model serial completion. + +Use the same source/generation/terminal-instance metadata passed to the write queue: + +```ts +const scope = beginTerminalOutputWriteScope({ + terminalInstanceId, + source: frameOverlapsReplay ? 'replay' : 'live', + attachRequestId: frameAttachRequestId, + generation: frameAttachRequestId ?? 'no-attach', + suppressExternalSideEffects: frameOverlapsReplay, +}) +term.write(data, () => { + try { + onWritten?.() + } finally { + scope.complete() + } +}) +``` + +Do not rely on stack-scoped context around `term.write`; xterm parsing is asynchronous. Do not rely on the old `allowReplies` boolean as a complete safety boundary. + +- [ ] **Step 5: Gate parser side-effect paths** + +Update the concrete side-effect paths found by load-bearing: + +- `src/components/terminal/request-mode-bypass.ts`: check the terminal-instance write scope before `sendInput(response)`. +- `src/lib/terminal-osc52.ts`: check the terminal-instance write scope before prompts and before direct `always` clipboard writes. +- `src/components/TerminalView.tsx`: suppress xterm title-change Redux updates when the terminal-instance write scope is replay, or queue the latest live title only after replay completes. +- Startup probe replies and client-minted turn-complete dispatches use `shouldAllowTerminalOutputSideEffect`. + +- [ ] **Step 6: Run parser and lifecycle tests** + +Run: + +```bash +timeout 300s npm run test:vitest -- --run test/unit/client/components/terminal/terminal-write-queue.test.ts test/unit/client/lib/terminal-output-write-scope.test.ts test/unit/client/lib/terminal-startup-probes.test.ts test/unit/client/lib/terminal-osc52.test.ts test/unit/shared/turn-complete-signal.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx test/e2e/codex-startup-probes.test.tsx test/e2e/opencode-startup-probes.test.tsx test/e2e/terminal-osc52-policy-flow.test.tsx +``` + +Expected: pass. + +- [ ] **Step 7: Commit** + +```bash +git add src/lib/terminal-output-write-scope.ts src/lib/terminal-output-side-effects.ts test/unit/client/lib/terminal-output-write-scope.test.ts test/unit/client/lib/terminal-output-side-effects.test.ts src/components/terminal/terminal-write-queue.ts test/unit/client/components/terminal/terminal-write-queue.test.ts src/components/TerminalView.tsx src/components/terminal/request-mode-bypass.ts src/lib/terminal-osc52.ts test/unit/client/lib/terminal-osc52.test.ts test/unit/shared/turn-complete-signal.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx +git commit -m "Gate terminal output side effects by write scope" +``` + +## Task 5: Serialized Payload Budgeting And Pre-Sequence Fragmentation + +**Files:** +- Create: `server/terminal-stream/stream-identity.ts` +- Create: `server/terminal-stream/serialized-budget.ts` +- Create: `server/terminal-stream/output-fragments.ts` +- Create: `test/unit/server/terminal-stream/stream-identity.test.ts` +- Create: `test/unit/server/terminal-stream/serialized-budget.test.ts` +- Create: `test/unit/server/terminal-stream/output-fragments.test.ts` +- Modify: `server/terminal-registry.ts` +- Modify: `server/terminal-stream/broker.ts` +- Modify: `server/terminal-stream/client-output-queue.ts` +- Modify: `server/terminal-stream/replay-ring.ts` + +- [ ] **Step 1: Add failing serialized budget and fragmentation tests** + +Create `test/unit/server/terminal-stream/serialized-budget.test.ts` and `test/unit/server/terminal-stream/output-fragments.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { measureTerminalOutputPayloadBytes } from '../../../../server/terminal-stream/serialized-budget' +import { + containsLoneSurrogate, + fragmentTerminalOutputForPayloadBudget, +} from '../../../../server/terminal-stream/output-fragments' + +describe('terminal stream serialized budget', () => { + it('measures escaped JSON bytes instead of raw data bytes', () => { + const data = '\u001b'.repeat(16 * 1024) + const bytes = measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-1', + data, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }) + + expect(bytes).toBeGreaterThan(16 * 1024) + }) + + it('fragments escaped output before sequence assignment so every payload fits the budget', () => { + const data = '\u001b'.repeat(16 * 1024) + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 16 * 1024, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + expect(chunks.length).toBeGreaterThan(1) + for (const chunk of chunks) { + expect(measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + })).toBeLessThanOrEqual(16 * 1024) + } + expect(chunks.join('')).toBe(data) + }) + + it('does not split surrogate pairs', () => { + const data = `prefix-${'😀'.repeat(2048)}-suffix` + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 2048, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + expect(chunks.join('')).toBe(data) + expect(chunks.every((chunk) => !containsLoneSurrogate(chunk))).toBe(true) + }) + + it('preserves replacement characters emitted by current string-mode PTY decoding', () => { + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 2048, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data: `prefix-\ufffd-suffix`, + }) + + expect(chunks.join('')).toBe('prefix-\ufffd-suffix') + }) +}) +``` + +Do not add replay-ring read-path coalescing assertions in this task. Task 5 proves the fragmentation helper and pre-sequence sequence-space expansion. Task 6 owns the replay read-path proof after the barrier scanner and serialized-budget batch builder exist; otherwise the existing raw-byte coalescing behavior can defeat this task's append-time fragmentation. + +Create `test/unit/server/terminal-stream/stream-identity.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { createTerminalStreamIdentityTracker } from '../../../../server/terminal-stream/stream-identity' + +describe('terminal stream identity', () => { + it('keeps stream id stable across attach and detach for the same output stream', () => { + const tracker = createTerminalStreamIdentityTracker() + const initial = tracker.ensureStream('term-1') + + expect(tracker.ensureStream('term-1')).toBe(initial) + tracker.recordAttach('term-1', 'attach-1') + tracker.recordDetach('term-1', 'attach-1') + + expect(tracker.ensureStream('term-1')).toBe(initial) + }) + + it('changes stream id on pty replacement and incompatible retention loss', () => { + const tracker = createTerminalStreamIdentityTracker() + const initial = tracker.ensureStream('term-1') + + const afterRecovery = tracker.replaceStream('term-1', 'codex_pty_recovery') + const afterRetentionLoss = tracker.replaceStream('term-1', 'retention_lost') + + expect(afterRecovery).not.toBe(initial) + expect(afterRetentionLoss).not.toBe(afterRecovery) + }) +}) +``` + +- [ ] **Step 2: Run failing serialized budget and fragmentation tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/stream-identity.test.ts test/unit/server/terminal-stream/serialized-budget.test.ts test/unit/server/terminal-stream/output-fragments.test.ts +``` + +Expected: fail because the helpers and pre-sequence fragmentation path do not exist. + +- [ ] **Step 3: Implement serialized budget and Unicode-safe fragmentation helpers** + +Create `server/terminal-stream/stream-identity.ts` and wire it into terminal stream ingestion. `streamId` changes on new PTY/session stream, Codex PTY recovery replacement, incompatible retention loss, and server restart without compatible persisted retention. It does not change on attach/detach. Codex recovery paths in `server/terminal-registry.ts` must emit enough lifecycle signal for the terminal stream broker to call `replaceStream('codex_pty_recovery')` when `record.pty` is replaced under the same `terminalId`. + +Create `server/terminal-stream/serialized-budget.ts`: + +```ts +export type JsonPayload = Record + +export function measureSerializedJsonBytes(payload: JsonPayload): number { + return Buffer.byteLength(JSON.stringify(payload), 'utf8') +} + +export function measureTerminalOutputPayloadBytes(payload: JsonPayload): number { + return measureSerializedJsonBytes(payload) +} +``` + +Create `server/terminal-stream/output-fragments.ts`: + +```ts +import { measureSerializedJsonBytes, type JsonPayload } from './serialized-budget.js' + +export function containsLoneSurrogate(data: string): boolean { + for (let i = 0; i < data.length; i += 1) { + const code = data.charCodeAt(i) + if (code >= 0xd800 && code <= 0xdbff) { + const next = data.charCodeAt(i + 1) + if (!(next >= 0xdc00 && next <= 0xdfff)) return true + i += 1 + continue + } + if (code >= 0xdc00 && code <= 0xdfff) return true + } + return false +} + +export function fragmentTerminalOutputForPayloadBudget(input: { + maxSerializedBytes: number + data: string + payloadForData: (data: string) => JsonPayload +}): string[] { + // Oversized terminal chunks are rare, but this binary search reserializes + // candidate JSON payloads. Keep the helper covered by a stress test or + // replace it with incremental measurement if it appears in hot-path logs. + const maxSerializedBytes = Math.max(1, Math.floor(input.maxSerializedBytes)) + if (measureSerializedJsonBytes(input.payloadForData(input.data)) <= maxSerializedBytes) { + return [input.data] + } + + const chunks: string[] = [] + const codePoints = Array.from(input.data) + let offset = 0 + + while (offset < codePoints.length) { + let low = 1 + let high = codePoints.length - offset + let best = 0 + + while (low <= high) { + const mid = Math.floor((low + high) / 2) + const candidate = codePoints.slice(offset, offset + mid).join('') + const bytes = measureSerializedJsonBytes(input.payloadForData(candidate)) + if (bytes <= maxSerializedBytes) { + best = mid + low = mid + 1 + } else { + high = mid - 1 + } + } + + if (best <= 0) { + throw new Error('terminal output payload budget is too small for one code point') + } + + chunks.push(codePoints.slice(offset, offset + best).join('')) + offset += best + } + + return chunks +} +``` + +- [ ] **Step 4: Fragment after raw observers and before assigning replay/live sequence numbers** + +Update the terminal-stream broker ingestion path so `fragmentTerminalOutputForPayloadBudget` runs after `terminal.output.raw` observers receive the original string event, and before assigning `seqStart`/`seqEnd` for replay/live frames. The budget-measurement payload must still include representative `seqStart` and `seqEnd` fields, because those fields are present in the actual `terminal.output` JSON passed to `ws.send`. The resulting fragments become normal frames with distinct sequence ranges. Do not split a `ReplayFrame` after it already has a sequence number, because the current client treats repeated or overlapping sequence ranges as invalid. + +Use the same fragmentation helper for live queue inputs and replay retention so live and replay semantics do not diverge. Keep raw byte counts for retention accounting only. + +This task intentionally preserves the current UTF-8 string terminal contract. Add comments and tests that document current `node-pty` string-mode behavior for invalid UTF-8 and 8-bit C1 bytes as replacement characters. Do not claim byte-stream-perfect replay in this task. + +- [ ] **Step 5: Use serialized application JSON bytes in server stream batching** + +Replace raw `Buffer.byteLength(data, 'utf8')` batch limit checks for outgoing WebSocket payloads with `measureTerminalOutputPayloadBytes`. Name this budget "serialized application JSON bytes" in code and logs; do not call it exact wire bytes because per-message compression can change on-wire size. + +- [ ] **Step 6: Run focused server stream tests** + +Run: + +```bash +timeout 300s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/stream-identity.test.ts test/unit/server/terminal-stream/serialized-budget.test.ts test/unit/server/terminal-stream/output-fragments.test.ts test/unit/server/terminal-stream/replay-ring.test.ts test/unit/server/terminal-stream/client-output-queue.test.ts test/unit/server/ws-handler-backpressure.test.ts test/server/ws-terminal-stream-v2-replay.test.ts +``` + +Expected: pass. + +- [ ] **Step 7: Commit** + +```bash +git add server/terminal-stream/stream-identity.ts test/unit/server/terminal-stream/stream-identity.test.ts server/terminal-stream/serialized-budget.ts server/terminal-stream/output-fragments.ts test/unit/server/terminal-stream/serialized-budget.test.ts test/unit/server/terminal-stream/output-fragments.test.ts server/terminal-registry.ts server/terminal-stream/broker.ts server/terminal-stream/client-output-queue.ts server/terminal-stream/replay-ring.ts +git commit -m "Fragment terminal output before sequence assignment" +``` + +## Task 6: Barrier-Aware Server Batching + +**Files:** +- Create: `server/terminal-stream/output-barrier-scanner.ts` +- Create: `server/terminal-stream/output-batch.ts` +- Create: `test/unit/server/terminal-stream/output-barrier-scanner.test.ts` +- Create: `test/unit/server/terminal-stream/output-batch.test.ts` +- Modify: `server/terminal-stream/replay-ring.ts` +- Modify: `server/terminal-stream/client-output-queue.ts` +- Modify: `server/terminal-stream/broker.ts` + +- [ ] **Step 1: Add stateful barrier scanner tests** + +Create `test/unit/server/terminal-stream/output-barrier-scanner.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { createTerminalOutputBarrierScanner } from '../../../../server/terminal-stream/output-barrier-scanner' + +describe('terminal output barrier scanner', () => { + it('treats plain printable text and newlines as transparent', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('hello\nworld\r\n')).toMatchObject({ barrier: false, ground: true }) + }) + + it('treats escape sequences as barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\u001b[31mred')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + }) + }) + + it('treats BEL as a turn-complete-sensitive barrier', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\u0007')).toMatchObject({ + barrier: true, + reason: 'turn_complete', + ground: true, + }) + }) + + it('treats OSC sequences as OSC52-sensitive barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\u001b]52;c;SGVsbG8=\u0007')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: true, + }) + }) + + it('carries pending CSI state across fragments', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\u001b[')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + }) + expect(scanner.scan('6n')).toMatchObject({ + barrier: true, + reason: 'request_mode', + ground: true, + }) + }) + + it('carries pending OSC state across fragments', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\u001b]52;c;')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: false, + }) + expect(scanner.scan('SGVsbG8=\u0007')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: true, + }) + }) + + it('treats replacement characters from lossy PTY decoding as barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + expect(scanner.scan('\ufffd')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + }) + }) + + it('returns scanner state snapshots that can be stored on retained frames', () => { + const scanner = createTerminalOutputBarrierScanner() + const first = scanner.scan('\u001b[') + const second = scanner.scan('6n') + + expect(first.stateBefore.mode).toBe('ground') + expect(first.stateAfter.mode).toBe('csi') + expect(second.stateBefore.mode).toBe('csi') + expect(second.stateAfter.mode).toBe('ground') + }) +}) +``` + +- [ ] **Step 2: Add batch builder tests** + +Create `test/unit/server/terminal-stream/output-batch.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { buildTerminalOutputBatches } from '../../../../server/terminal-stream/output-batch' + +describe('terminal output batch builder', () => { + it('coalesces contiguous transparent frames under the serialized budget', () => { + const batches = buildTerminalOutputBatches({ + terminalId: 'term-1', + attachRequestId: 'attach-1', + source: 'replay', + maxSerializedBytes: 16 * 1024, + frames: [ + { seqStart: 1, seqEnd: 1, data: 'a', bytes: 1, at: 1 }, + { seqStart: 2, seqEnd: 2, data: 'b', bytes: 1, at: 2 }, + ], + }) + + expect(batches).toHaveLength(1) + expect(batches[0]).toMatchObject({ + seqStart: 1, + seqEnd: 2, + data: 'ab', + source: 'replay', + }) + expect(batches[0].segments).toEqual([ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1 }, + ]) + }) + + it('does not coalesce across parser barriers', () => { + const batches = buildTerminalOutputBatches({ + terminalId: 'term-1', + attachRequestId: 'attach-1', + source: 'replay', + maxSerializedBytes: 16 * 1024, + frames: [ + { seqStart: 1, seqEnd: 1, data: 'a', bytes: 1, at: 1 }, + { seqStart: 2, seqEnd: 2, data: '\u0007', bytes: 1, at: 2 }, + { seqStart: 3, seqEnd: 3, data: 'b', bytes: 1, at: 3 }, + ], + }) + + expect(batches.map((batch) => batch.data)).toEqual(['a', '\u0007', 'b']) + }) + + it('uses UTF-16 code-unit segment offsets on code-point boundaries', () => { + const batches = buildTerminalOutputBatches({ + terminalId: 'term-1', + attachRequestId: 'attach-1', + source: 'replay', + maxSerializedBytes: 16 * 1024, + frames: [ + { seqStart: 1, seqEnd: 1, data: '😀', bytes: 4, at: 1 }, + { seqStart: 2, seqEnd: 2, data: 'b', bytes: 1, at: 2 }, + ], + }) + + expect(batches[0].data).toBe('😀b') + expect(batches[0].segments).toMatchObject([ + { seqStart: 1, seqEnd: 1, endOffset: 2 }, + { seqStart: 2, seqEnd: 2, endOffset: 3 }, + ]) + }) + + it('does not re-coalesce serialized-budget fragments across control barriers', () => { + const frames = Array.from({ length: 8 }, (_unused, index) => ({ + seqStart: index + 1, + seqEnd: index + 1, + data: '\u001b'.repeat(2048), + bytes: 2048, + at: index + 1, + barrier: true, + barrierReason: 'control', + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'ground' }, + })) + + const batches = buildTerminalOutputBatches({ + terminalId: 'term-1', + attachRequestId: 'attach-1', + source: 'replay', + maxSerializedBytes: 16 * 1024, + frames, + }) + + expect(batches.length).toBeGreaterThan(1) + expect(new Set(batches.map((batch) => `${batch.seqStart}:${batch.seqEnd}`)).size) + .toBe(batches.length) + expect(batches.every((batch) => batch.serializedBytes <= 16 * 1024)).toBe(true) + }) +}) +``` + +Also update `test/unit/server/terminal-stream/replay-ring.test.ts` in this task where existing read-path coalescing assertions conflict with scanner-aware semantics. Transparent text may still coalesce through `buildTerminalOutputBatches`; control-barrier fragments and serialized-budget fragments must not be re-coalesced into an oversized serialized payload. This is the task that reconciles the old raw-byte coalescing tests with the new barrier-aware replay contract. + +- [ ] **Step 3: Run failing batch tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/output-barrier-scanner.test.ts test/unit/server/terminal-stream/output-batch.test.ts +``` + +Expected: fail because the helpers do not exist. + +- [ ] **Step 4: Implement conservative stateful barrier scanner** + +Create `server/terminal-stream/output-barrier-scanner.ts`: + +```ts +export type TerminalOutputBarrierReason = + | 'control' + | 'osc52' + | 'request_mode' + | 'turn_complete' + | 'startup_probe' + +export type TerminalOutputScannerMode = 'ground' | 'esc' | 'csi' | 'osc' | 'dcs' | 'apc' + +export type TerminalOutputScannerState = { + mode: TerminalOutputScannerMode +} + +export type TerminalOutputBarrierClassification = + | { + barrier: false + ground: boolean + stateBefore: TerminalOutputScannerState + stateAfter: TerminalOutputScannerState + } + | { + barrier: true + reason: TerminalOutputBarrierReason + ground: boolean + stateBefore: TerminalOutputScannerState + stateAfter: TerminalOutputScannerState + } + +export type TerminalOutputBarrierScanner = { + scan: (data: string) => TerminalOutputBarrierClassification + isGround: () => boolean +} + +const ESC = '\u001b' +const BEL = '\u0007' + +export function createTerminalOutputBarrierScanner(): TerminalOutputBarrierScanner { + // Implement a small conservative VT state machine, not a stateless substring scan. + // Track at least: ground, esc, csi, osc, dcs, apc. Pending non-ground + // state is itself a barrier and prevents coalescing until a terminator/final byte. + // Treat U+FFFD as a barrier because current node-pty string mode may have + // already lost invalid UTF-8 or 8-bit C1 control bytes. + throw new Error('implement stateful scanner') +} +``` + +The implementation must be stream-stateful. It must remember pending control/string states across raw chunks and across serialized-budget fragments. It must return scanner-state snapshots that can be stored on retained frames at ingestion time; replay batching for arbitrary `sinceSeq` windows must consume stored metadata instead of mutating or reconstructing the live scanner from an unsafe prefix. A later byte-preserving terminal protocol may replace this scanner; this task must not rely on byte-perfect input. + +- [ ] **Step 5: Implement batch builder** + +Create `server/terminal-stream/output-batch.ts` using `createTerminalOutputBarrierScanner` and `measureTerminalOutputPayloadBytes`. The builder must: + +- Preserve `seqStart`, `seqEnd`, `attachRequestId`, and source. +- Stop before barriers. +- Emit a barrier frame as its own batch. +- Stop before serialized budget overflow. +- Preserve segment metadata. +- Only coalesce transparent spans when the scanner is in ground state before and after the span. +- Consume stored frame barrier metadata (`barrier`, `barrierReason`, `scannerStateBefore`, `scannerStateAfter`) instead of re-scanning retained replay from an arbitrary window. +- Keep live scanner state with the terminal stream so live and replay batching see the same barrier decisions. +- Emit `segments[].endOffset` as a UTF-16 code-unit offset on code-point boundaries. + +- [ ] **Step 6: Wire server replay and live queues to the batch builder** + +Use `buildTerminalOutputBatches` in: + +- `server/terminal-stream/replay-ring.ts` for replay batch reads. +- `server/terminal-stream/client-output-queue.ts` for live queued batch reads. +- `server/terminal-stream/broker.ts` for sending batches. + +- [ ] **Step 7: Run server stream tests** + +Run: + +```bash +timeout 300s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/output-barrier-scanner.test.ts test/unit/server/terminal-stream/output-batch.test.ts test/unit/server/terminal-stream/replay-ring.test.ts test/unit/server/terminal-stream/client-output-queue.test.ts test/unit/server/ws-handler-backpressure.test.ts test/server/ws-terminal-stream-v2-replay.test.ts test/server/ws-edge-cases.test.ts +``` + +Expected: pass. + +- [ ] **Step 8: Commit** + +```bash +git add server/terminal-stream/output-barrier-scanner.ts server/terminal-stream/output-batch.ts test/unit/server/terminal-stream/output-barrier-scanner.test.ts test/unit/server/terminal-stream/output-batch.test.ts test/unit/server/terminal-stream/replay-ring.test.ts server/terminal-stream/replay-ring.ts server/terminal-stream/client-output-queue.ts server/terminal-stream/broker.ts +git commit -m "Make terminal replay batching barrier aware" +``` + +## Task 7: Replace ReplayRing Eviction With A Deque + +**Files:** +- Create: `server/terminal-stream/replay-deque.ts` +- Create: `test/unit/server/terminal-stream/replay-deque.test.ts` +- Modify: `server/terminal-stream/replay-ring.ts` +- Modify: `test/unit/server/terminal-stream/replay-ring.test.ts` + +- [ ] **Step 1: Add deque stress tests** + +Create `test/unit/server/terminal-stream/replay-deque.test.ts`: + +```ts +import { describe, expect, it } from 'vitest' +import { ReplayDeque } from '../../../../server/terminal-stream/replay-deque' + +describe('ReplayDeque', () => { + it('evicts many tiny frames without shifting the backing array per frame', () => { + const deque = new ReplayDeque(1024) + + for (let i = 0; i < 4096; i += 1) { + deque.append('x') + } + + expect(deque.totalBytes()).toBeLessThanOrEqual(1024) + expect(deque.headSeq()).toBe(4096) + expect(deque.tailSeq()).toBeGreaterThan(1) + }) + + it('reports a gap after eviction while preserving retained frames', () => { + const deque = new ReplayDeque(3) + deque.append('a') + deque.append('b') + deque.append('c') + deque.append('d') + + const replay = deque.replayBatchSince(0, 1024, 4) + + expect(replay.missedFromSeq).toBe(1) + expect(replay.frames.map((frame) => frame.data).join('')).toBe('bcd') + }) + + it('preserves barrier metadata for arbitrary replay windows', () => { + const deque = new ReplayDeque(1024) + deque.append({ + data: '\u001b[', + barrier: true, + barrierReason: 'control', + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'csi' }, + }) + deque.append({ + data: '6n', + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: { mode: 'csi' }, + scannerStateAfter: { mode: 'ground' }, + }) + + const replay = deque.replayBatchSince(1, 1024) + + expect(replay.frames[0]).toMatchObject({ + data: '6n', + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: { mode: 'csi' }, + scannerStateAfter: { mode: 'ground' }, + }) + }) +}) +``` + +- [ ] **Step 2: Run failing deque tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/replay-deque.test.ts +``` + +Expected: fail because `ReplayDeque` does not exist. + +- [ ] **Step 3: Implement ReplayDeque** + +Create `server/terminal-stream/replay-deque.ts` with: + +```ts +import type { ReplayFrame } from './replay-ring.js' +import type { + TerminalOutputBarrierReason, + TerminalOutputScannerState, +} from './output-barrier-scanner.js' + +export type ReplayDequeFrame = ReplayFrame & { + barrier?: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore?: TerminalOutputScannerState + scannerStateAfter?: TerminalOutputScannerState +} + +export type ReplayDequeAppendInput = + | string + | { + data: string + barrier?: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState + } + +export class ReplayDeque { + private frames: ReplayDequeFrame[] = [] + private start = 0 + private bytes = 0 + private nextSeq = 1 + private head = 0 + + constructor(private readonly maxBytes: number) {} + + append(input: ReplayDequeAppendInput): ReplayDequeFrame { + const data = typeof input === 'string' ? input : input.data + const frame: ReplayDequeFrame = { + seqStart: this.nextSeq, + seqEnd: this.nextSeq, + data, + bytes: Buffer.byteLength(data, 'utf8'), + at: Date.now(), + ...(typeof input === 'string' ? {} : { + barrier: input.barrier, + barrierReason: input.barrierReason, + scannerStateBefore: input.scannerStateBefore, + scannerStateAfter: input.scannerStateAfter, + }), + } + this.nextSeq += 1 + this.head = frame.seqEnd + this.frames.push(frame) + this.bytes += frame.bytes + this.evictIfNeeded() + return frame + } + + totalBytes(): number { + return this.bytes + } + + headSeq(): number { + return this.head + } + + tailSeq(): number { + const first = this.frames[this.start] + return first ? first.seqStart : this.head + 1 + } + + replayBatchSince(sinceSeq: number, maxBytes: number, toSeq = Number.POSITIVE_INFINITY): { + frames: ReplayDequeFrame[] + missedFromSeq?: number + } { + const tail = this.tailSeq() + const missedFromSeq = sinceSeq < tail - 1 ? sinceSeq + 1 : undefined + const frames: ReplayDequeFrame[] = [] + let budget = Math.max(0, Math.floor(maxBytes)) + + for (let i = this.start; i < this.frames.length; i += 1) { + const frame = this.frames[i] + if (!frame || frame.seqStart > toSeq) break + if (frame.seqEnd <= sinceSeq) continue + if (frame.bytes > budget && frames.length > 0) break + frames.push({ ...frame }) + budget -= frame.bytes + if (budget <= 0) break + } + + return { frames, missedFromSeq } + } + + private evictIfNeeded(): void { + while (this.bytes > this.maxBytes && this.start < this.frames.length) { + const frame = this.frames[this.start] + this.start += 1 + if (frame) this.bytes -= frame.bytes + } + if (this.start > 4096 && this.start * 2 > this.frames.length) { + this.frames = this.frames.slice(this.start) + this.start = 0 + } + } +} +``` + +Make `ReplayRing` delegate to `ReplayDeque` so existing imports remain stable while the internal storage changes. Update the affected `replay-ring.test.ts` assertions where the new scanner/barrier and no-implicit-coalescing semantics intentionally replace the old coalesced replay behavior. + +- [ ] **Step 4: Run replay tests** + +Run: + +```bash +timeout 240s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/replay-deque.test.ts test/unit/server/terminal-stream/replay-ring.test.ts test/unit/server/ws-handler-backpressure.test.ts test/server/ws-terminal-stream-v2-replay.test.ts +``` + +Expected: pass. + +- [ ] **Step 5: Commit** + +```bash +git add server/terminal-stream/replay-deque.ts test/unit/server/terminal-stream/replay-deque.test.ts server/terminal-stream/replay-ring.ts test/unit/server/terminal-stream/replay-ring.test.ts +git commit -m "Use deque storage for terminal replay retention" +``` + +## Task 8: Foreground Replay Pacing + +**Files:** +- Create: `server/ws-send.ts` +- Create: `test/unit/server/ws-send.test.ts` +- Modify: `server/ws-handler.ts` +- Modify: `server/terminal-stream/broker.ts` +- Modify: `test/unit/server/ws-handler-backpressure.test.ts` + +- [ ] **Step 1: Add failing foreground pacing test** + +Add this test to `test/unit/server/ws-handler-backpressure.test.ts`: + +```ts +it('pauses foreground replay before avoidable buffered growth exceeds the pacing threshold', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-foreground-paced') + const pacingThresholdBytes = 512 * 1024 + const allowedBatchOvershootBytes = 64 * 1024 + + for (let i = 1; i <= 1400; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-foreground-paced', + data: `line-${i};${'x'.repeat(2048)}`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 0 }) + wsReplay.send.mockImplementation((raw: string) => { + wsReplay.bufferedAmount += Buffer.byteLength(raw, 'utf8') + }) + + await broker.attach( + wsReplay as any, + 'term-foreground-paced', + 'transport_reconnect', + 80, + 24, + 0, + 'foreground-attach', + undefined, + 'foreground', + ) + + vi.advanceTimersByTime(5) + + expect(wsReplay.bufferedAmount).toBeLessThanOrEqual( + pacingThresholdBytes + allowedBatchOvershootBytes, + ) + + broker.close() +}) +``` + +- [ ] **Step 2: Run failing pacing test** + +Run: + +```bash +timeout 180s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/ws-handler-backpressure.test.ts -t "foreground replay" +``` + +Expected before implementation: fail. The seeded backlog is intentionally larger than 2 MiB and the mock socket never drains, so unpaced replay sends the full backlog and exceeds `pacingThresholdBytes + allowedBatchOvershootBytes`. + +- [ ] **Step 3: Implement normal foreground pacing** + +Create `server/ws-send.ts` and route both `server/ws-handler.ts` and `server/terminal-stream/broker.ts` through it. The shared sender must own: + +- JSON serialization and serialized application byte measurement. +- `ws.send` callback handling. +- `ws.bufferedAmount` reads before and after sends. +- closed-socket handling. +- `ws_send_large` and replay/backpressure structured JSONL instrumentation. +- a single max serialized message budget shared by normal handler sends and terminal broker sends. + +Then in `server/terminal-stream/broker.ts`: + +- Keep catastrophic threshold behavior. +- Add a normal foreground replay pacing threshold lower than catastrophic, using serialized bytes and `ws.bufferedAmount`. +- After each replay send, re-read `ws.bufferedAmount`. +- If above threshold, schedule the next flush instead of continuing the same flush. +- Background threshold remains stricter than foreground threshold. +- Do not call `ws.send(JSON.stringify(...))` directly. + +- [ ] **Step 4: Run backpressure tests** + +Run: + +```bash +timeout 240s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/ws-send.test.ts test/unit/server/ws-handler-backpressure.test.ts test/server/ws-edge-cases.test.ts +``` + +Expected: pass. + +- [ ] **Step 5: Commit** + +```bash +git add server/ws-send.ts test/unit/server/ws-send.test.ts server/ws-handler.ts server/terminal-stream/broker.ts test/unit/server/ws-handler-backpressure.test.ts +git commit -m "Pace foreground terminal replay under socket pressure" +``` + +## Task 9: Add Batch Protocol + +**Files:** +- Modify: `shared/ws-protocol.ts` +- Modify: `src/lib/ws-client.ts` +- Modify: `server/ws-handler.ts` +- Modify: `server/terminal-stream/broker.ts` +- Modify: `src/components/TerminalView.tsx` +- Modify: `src/lib/terminal-attach-seq-state.ts` +- Modify: `test/server/ws-protocol.test.ts` +- Modify: `test/unit/client/lib/ws-client.test.ts` +- Modify: `test/unit/client/lib/terminal-attach-seq-state.test.ts` +- Modify: `test/unit/client/components/TerminalView.lifecycle.test.tsx` +- Modify: `test/server/ws-terminal-stream-v2-replay.test.ts` + +- [ ] **Step 1: Add capability and protocol tests** + +Add tests in `test/server/ws-protocol.test.ts` and `test/unit/client/lib/ws-client.test.ts` that validate: + +- The client hello advertises `capabilities.terminalOutputBatchV1: true` only after the client can parse batches. +- The server records the capability on the WebSocket/client attachment state. +- A batch-capable client can receive the batch shape below. +- A protocol v6 client that omits the batch capability still receives compatible `terminal.output` messages. +- Non-batch fallback serializes safe batch segments as individual modern `terminal.output` frames that include `seqStart`, `seqEnd`, `streamId`, and `attachRequestId`; it must not use the old registry `{ type, terminalId, data }` shape. +- Non-batch fallback does not flatten arbitrary batches across parser barriers, stream id, attach id, budget, or replay/live source boundaries. +- Protocol v5 clients are rejected during `hello` so they cannot silently receive or drop unsafe untagged terminal stream messages. +- A batch-capable client rejects or splits any batch whose segments cannot all be accepted before bytes are written to xterm. + +Batch shape: + +```ts +{ + type: 'terminal.output.batch', + terminalId: 'term-1', + streamId: 'stream-1', + attachRequestId: 'attach-1', + source: 'replay', + seqStart: 1, + seqEnd: 2, + data: 'ab', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, data: 'a', rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, data: 'b', rawFrameCount: 1 }, + ], +} +``` + +`endOffset` is a UTF-16 code-unit offset into top-level `data`; segment `data` is optional redundancy for debugging and non-batch fallback. If `data` is present, the client and server tests must assert it equals the slice implied by the previous segment offset and `endOffset`. + +Do not write a test against a non-existent `ServerMessageSchema`. If this task adds server-to-client runtime validation, add the schema intentionally in this task and test that new API. Otherwise, use type-level tests and behavior tests around client/server message handling. + +- [ ] **Step 2: Run failing capability/protocol tests** + +Run: + +```bash +timeout 240s npm run test:vitest -- --run test/unit/client/lib/ws-client.test.ts +timeout 240s npm run test:vitest -- --config vitest.server.config.ts --run test/server/ws-protocol.test.ts test/server/ws-terminal-stream-v2-replay.test.ts -t "terminal.output.batch|terminalOutputBatchV1|legacy" +``` + +Expected: fail because the capability and batch behavior do not exist. + +- [ ] **Step 3: Add protocol capability and message types** + +In `shared/ws-protocol.ts`: + +- Add `terminalOutputBatchV1?: boolean` to hello capabilities. +- Add `TerminalOutputBatchMessage` to the server-to-client TypeScript union. +- Keep existing `TerminalOutputMessage` support. +- If adding runtime validation for server-to-client messages, add an explicit `TerminalOutputBatchMessageSchema` and a tested server-message schema. Do not pretend the current TS-only union already validates server messages at runtime. + +- [ ] **Step 4: Advertise and persist client capability** + +In `src/lib/ws-client.ts`, advertise `terminalOutputBatchV1: true` only in the same branch that implements client parsing. + +In `server/ws-handler.ts`, read the capability from the hello payload and pass it into terminal stream attachment state. Keep default false for clients that omit the capability. + +- [ ] **Step 5: Emit batch messages only when supported** + +In `server/terminal-stream/broker.ts`, emit `terminal.output.batch` only for clients whose attachment state says `terminalOutputBatchV1` is true. For protocol v6 clients without that capability, send `terminal.output` messages using the same server-side batch builder internally if useful, but serialize each safe segment as its own compatible output frame with `seqStart`, `seqEnd`, `streamId`, `attachRequestId`, and `data`. + +Do not flatten an arbitrary batch into one fallback `terminal.output`. Fallback frames have less segment metadata than `terminal.output.batch`, so they must not cross replay/live source, attach id, stream id, parser-barrier, or serialized-budget boundaries. Do not route terminal stream fallback through the old registry direct-output shape, because current clients ignore output without sequence ranges. + +- [ ] **Step 6: Process batch messages client-side** + +In `src/components/TerminalView.tsx`, process `terminal.output.batch` by prevalidating every segment's sequence range before writing any batch bytes to xterm. If any segment is stale, overlapping, from an incompatible stream/attach/source, or parser-barrier-sensitive, do not partially write the combined batch. + +Efficient writes are allowed only for a homogeneous accepted span: + +- same source (`live` or `replay`); +- same attach request id and stream id; +- all segments accepted by sequence state; +- no parser-side-effect barrier inside the combined write; +- one terminal-instance write scope can safely cover the whole submitted write. + +For barrier segments, mixed replay/live spans, or spans requiring different side-effect context, split the client write per segment. Advance `parserAppliedSeq` only after the corresponding xterm write callback, and never for segments that were not actually submitted. + +- [ ] **Step 7: Run protocol and lifecycle tests** + +Run: + +```bash +timeout 300s npm run test:vitest -- --run test/unit/client/lib/ws-client.test.ts test/unit/client/lib/terminal-attach-seq-state.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx +timeout 300s npm run test:vitest -- --config vitest.server.config.ts --run test/server/ws-protocol.test.ts test/server/ws-terminal-stream-v2-replay.test.ts test/unit/server/ws-handler-backpressure.test.ts +``` + +Expected: pass. + +- [ ] **Step 8: Commit** + +```bash +git add shared/ws-protocol.ts src/lib/ws-client.ts server/ws-handler.ts server/terminal-stream/broker.ts src/components/TerminalView.tsx src/lib/terminal-attach-seq-state.ts test/server/ws-protocol.test.ts test/unit/client/lib/ws-client.test.ts test/unit/client/lib/terminal-attach-seq-state.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx test/server/ws-terminal-stream-v2-replay.test.ts +git commit -m "Add protocol-aware terminal output batches" +``` + +## Task 10: Structured Observability And Retention SLO + +**Files:** +- Modify: `server/terminal-stream/broker.ts` +- Modify: `server/terminal-stream/replay-ring.ts` +- Modify: `src/components/TerminalView.tsx` +- Modify: `test/unit/server/ws-handler-backpressure.test.ts` +- Modify: `test/unit/client/components/TerminalView.lifecycle.test.tsx` + +- [ ] **Step 1: Add structured log assertions** + +In `test/unit/server/ws-handler-backpressure.test.ts`, add assertions that the broker logger receives structured fields: + +```ts +expect(log).toHaveBeenCalledWith(expect.objectContaining({ + event: 'terminal.replay.batch', + severity: 'debug', + terminalId: 'term-replay-coalesced', + seqStart: 1, + seqEnd: 1000, + serializedBytes: expect.any(Number), + rawFrameCount: expect.any(Number), +}), 'debug') +``` + +Add similar assertions for: + +- `terminal.replay.gap` +- `terminal.replay.backpressure_pause` +- `terminal.replay.retention` + +- [ ] **Step 2: Run failing log tests** + +Run: + +```bash +timeout 180s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/ws-handler-backpressure.test.ts -t "terminal.replay" +``` + +Expected before implementation: fail because structured event names or fields are missing. + +- [ ] **Step 3: Add server structured events** + +In `server/terminal-stream/broker.ts`, emit JSON-friendly structured log payloads with at least: + +```ts +{ + event: 'terminal.replay.batch', + severity: 'debug', + terminalId, + attachRequestId, + source, + seqStart, + seqEnd, + rawFrameCount, + dataBytes, + serializedBytes, + bufferedAmount, +} +``` + +For warnings: + +```ts +{ + event: 'terminal.replay.gap', + severity: 'warn', + terminalId, + attachRequestId, + fromSeq, + toSeq, + reason, +} +``` + +- [ ] **Step 4: Add client perf marks** + +In `src/components/TerminalView.tsx`, add perf marks for: + +- `terminal.parser_applied` +- `terminal.attach_generation_stale_rejected` +- `terminal.catchup.full_hydrate_fallback` +- `terminal.catchup.surface_quarantined` + +Use existing perf bridge patterns already present in `TerminalView.tsx`. + +These marks must be promoted into the visible-first audit artifact in Task 11. Debug-only console logs are not enough. + +- [ ] **Step 5: Run focused observability tests** + +Run: + +```bash +timeout 240s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/ws-handler-backpressure.test.ts +timeout 240s npm run test:vitest -- --run test/unit/client/components/TerminalView.lifecycle.test.tsx +``` + +Expected: pass. + +- [ ] **Step 6: Commit** + +```bash +git add server/terminal-stream/broker.ts server/terminal-stream/replay-ring.ts src/components/TerminalView.tsx test/unit/server/ws-handler-backpressure.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx +git commit -m "Instrument terminal catch-up replay safety" +``` + +## Task 11: Browser-Level Verification + +**Files:** +- Create: `test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts` +- Modify: `test/e2e-browser/perf/run-sample.ts` +- Modify: `test/e2e-browser/perf/scenarios.ts` +- Modify: `test/unit/lib/visible-first-audit-scenarios.test.ts` +- Modify: `test/unit/lib/visible-first-audit-derived-metrics.test.ts` +- Modify: `test/unit/lib/visible-first-audit-gate.test.ts` + +- [ ] **Step 1: Add terminal catch-up acceptance metrics** + +Extend the existing `terminal-reconnect-backlog` scenario to record: + +- replay message count +- total serialized replay bytes +- parser-applied lag +- gaps count +- full hydrate fallback count +- surface quarantine count +- stale generation rejection count +- focused ready time +- terminal input to first output +- max RAF gap +- stopped/backgrounded duration covered by retention +- WebSocket state after process suspend/resume or real browser background/resume +- replay gaps or surface quarantine after suspend/background resume +- batch protocol coverage when `terminalOutputBatchV1` is enabled + +When Phase 5 enables `terminal.output.batch`, update the `terminal-reconnect-backlog` audit scenario's `allowedWsTypesBeforeReady` to include both `terminal.output` and `terminal.output.batch`. Otherwise the audit can reject expected batch replay traffic before the first-output milestone for the wrong reason. + +- [ ] **Step 2: Add unit tests for metrics contract** + +Add a `requiredMetricIds` field to the visible-first audit scenario definitions, including `terminal-reconnect-backlog`, so scenarios can declare required derived metrics directly. + +In `test/unit/lib/visible-first-audit-scenarios.test.ts`, `test/unit/lib/visible-first-audit-derived-metrics.test.ts`, and `test/unit/lib/visible-first-audit-gate.test.ts`, assert the scenario and derived metrics include terminal catch-up metrics: + +```ts +expect(scenarioMap.get('terminal-reconnect-backlog')?.requiredMetricIds).toEqual(expect.arrayContaining([ + 'terminalReplayMessageCount', + 'terminalReplaySerializedBytes', + 'terminalParserAppliedLagMs', + 'terminalReplayGapCount', + 'terminalFullHydrateFallbackCount', + 'terminalSurfaceQuarantineCount', + 'terminalStaleGenerationRejectionCount', + 'terminalStoppedRetentionCoveredMs', + 'terminalStopResumeGapCount', +])) +``` + +- [ ] **Step 3: Run failing audit contract tests** + +Run: + +```bash +timeout 120s npm run test:vitest -- --run test/unit/lib/visible-first-audit-scenarios.test.ts test/unit/lib/visible-first-audit-derived-metrics.test.ts test/unit/lib/visible-first-audit-gate.test.ts +``` + +Expected: fail until metric contract is added. + +- [ ] **Step 4: Implement metric capture** + +In `test/e2e-browser/perf/run-sample.ts`, collect metrics from: + +- browser perf marks +- WebSocket recorder +- server structured logs +- terminal helper output checks + +Do not rely on CDP `Network.webSocketFrameReceived.payloadData` as compressed wire-byte evidence. Treat its `payloadData` length as serialized application payload evidence only. Prefer server structured logs for serialized replay bytes and replay frame counts, and client perf marks for parser-applied lag and stale-generation rejection. + +The acceptance target for the 1,200-line backlog case: + +- Replay messages stay in the same order of magnitude as #397, not #396. +- No stale-generation cursor advancement. +- No replay-triggered OSC52 or startup replies. +- No replay gaps in the seeded audit scenario. +- No full hydrate fallback or surface quarantine in the compatible warm surface path. + +- [ ] **Step 5: Add browser stop/resume positive-control probe** + +Create `test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts`. It must: + +- Open an isolated Freshell page through the existing Playwright server fixture. +- Seed terminal output while the page is active and establish a parser-applied checkpoint. +- Use a positive-control browser-execution stop mechanism that proves page work actually stopped or was throttled. Acceptable local proof is process-tree suspend/resume or another mechanism that records timer, RAF, and WebSocket counters before, during, and after the stopped period. +- Generate enough terminal output while browser execution is stopped to exercise server retention but stay within the configured retention budget. +- Resume browser execution. +- Assert the WebSocket behavior observed during stop/resume: still open and stalled, closed/reconnected, or buffered/resumed. The test must record which path happened. +- Assert catch-up either has no gaps and no quarantine for the covered retention window, or reports explicit gaps/quarantine when retention is exceeded. Silent parser-applied cursor jumps are failures. + +This positive-control probe is required implementation evidence for the PR. CDP `Page.setWebLifecycleState({ state: 'frozen' })` and Xvfb tab switching were disproven as valid proof in `docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json` and `docs/superpowers/proofs/artifacts/browser-background-visibility.json` because page work continued at active rates. + +- [ ] **Step 6: Run browser perf audit and positive-control probe for the terminal scenario** + +Run: + +```bash +timeout 1200s tsx scripts/visible-first-audit.ts --scenario terminal-reconnect-backlog --profile desktop_local --output /tmp/freshell-terminal-catchup-audit.json +timeout 1200s npm run test:e2e:chromium -- test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts +``` + +Expected: audit completes and writes `/tmp/freshell-terminal-catchup-audit.json`; the stop/resume spec passes, proves the page execution stop with timer/RAF/WebSocket counters, and records WebSocket state plus retention coverage. + +- [ ] **Step 7: Record real Windows Chrome acceptance follow-up** + +This is not a prerequisite for opening the single implementation PR. It is recommended release/user-acceptance evidence when a real Windows Chrome environment can be observed for a long background soak. If run, retain the artifact and summarize it in the PR or follow-up issue. + +Required gate: + +1. Start an isolated Freshell server from the feature worktree on a unique port. Do not touch the self-hosted dev server. +2. Open real Windows Chrome, not headless Chromium or Xvfb. +3. Create a terminal running the deterministic generator and at least one real Codex turn. +4. Confirm `document.visibilityState === 'hidden'` or record an OS freeze/suspend event while the Freshell tab is backgrounded, minimized, or otherwise stopped by the OS/browser. +5. Keep it backgrounded for a 4h soak at a calibrated 1 KiB/s stream and one shorter burst of at least 750 KiB. +6. Refocus and assert: + - no unsafe warm replay after retention loss; + - no unsequenced `terminal.output`; + - no parser-unsafe gap continues on the same parser; + - no replay-triggered OSC52/request-mode/title/turn side effect; + - catch-up to server head completes under the configured UX budget for covered retention; + - all terminal catch-up metrics are present in structured JSONL logs. +7. If disk retention is part of a later implementation, repeat one 8h overnight soak before enabling that retention mode by default. + +- [ ] **Step 8: Commit** + +```bash +git add test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts test/e2e-browser/perf/run-sample.ts test/e2e-browser/perf/scenarios.ts test/unit/lib/visible-first-audit-scenarios.test.ts test/unit/lib/visible-first-audit-derived-metrics.test.ts test/unit/lib/visible-first-audit-gate.test.ts +git commit -m "Audit terminal catch-up replay performance" +``` + +## Final Verification + +- [ ] **Step 1: Run focused client suite** + +```bash +timeout 600s npm run test:vitest -- --run test/unit/client/components/terminal/terminal-write-queue.test.ts test/unit/client/lib/terminal-surface-checkpoint.test.ts test/unit/client/lib/terminal-cursor.test.ts test/unit/client/lib/terminal-attach-seq-state.test.ts test/unit/client/lib/terminal-attach-policy.test.ts test/unit/client/lib/ws-client.test.ts test/unit/client/lib/terminal-output-side-effects.test.ts test/unit/client/components/TerminalView.lifecycle.test.tsx test/e2e/terminal-create-attach-ordering.test.tsx test/e2e/terminal-flaky-network-responsiveness.test.tsx +``` + +Expected: pass. + +- [ ] **Step 2: Run focused server suite** + +```bash +timeout 600s npm run test:vitest -- --config vitest.server.config.ts --run test/unit/server/terminal-stream/stream-identity.test.ts test/unit/server/terminal-stream/output-barrier-scanner.test.ts test/unit/server/terminal-stream/output-batch.test.ts test/unit/server/terminal-stream/serialized-budget.test.ts test/unit/server/terminal-stream/output-fragments.test.ts test/unit/server/terminal-stream/replay-deque.test.ts test/unit/server/terminal-stream/replay-ring.test.ts test/unit/server/terminal-stream/client-output-queue.test.ts test/unit/server/ws-send.test.ts test/unit/server/ws-handler-backpressure.test.ts test/server/ws-terminal-stream-v2-replay.test.ts test/server/ws-edge-cases.test.ts test/server/ws-protocol.test.ts +``` + +Expected: pass. + +- [ ] **Step 3: Run parser side-effect suite** + +```bash +timeout 600s npm run test:vitest -- --run test/unit/client/lib/terminal-output-write-scope.test.ts test/unit/client/lib/terminal-output-side-effects.test.ts test/unit/client/lib/terminal-startup-probes.test.ts test/unit/client/lib/terminal-osc52.test.ts test/unit/shared/turn-complete-signal.test.ts test/e2e/codex-startup-probes.test.tsx test/e2e/opencode-startup-probes.test.tsx test/e2e/terminal-osc52-policy-flow.test.tsx +``` + +Expected: pass. + +- [ ] **Step 4: Run terminal visible-first audit and local stop/resume proof** + +```bash +timeout 1200s tsx scripts/visible-first-audit.ts --scenario terminal-reconnect-backlog --profile desktop_local --output /tmp/freshell-terminal-catchup-audit.json +timeout 1200s npm run test:e2e:chromium -- test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts +``` + +Expected: pass and show no replay gaps, no stale cursor advancement, no unexpected surface quarantine, #397-class replay message count, explicit stop/resume retention coverage, and recorded timer/RAF/WebSocket counters proving the browser page was actually stopped or throttled during the local probe. + +- [ ] **Step 5: Confirm browser acceptance scope** + +The required local acceptance scope is the visible-first audit plus process-suspend stop/resume proof above. Record that real Windows Chrome long-background soak remains recommended user-acceptance evidence, not a blocker for this PR. + +- [ ] **Step 6: Verify xterm dependency policy** + +Either pin `@xterm/xterm` exactly to the probed version or run committed xterm parser probes in CI against every allowed resolved version. + +```bash +node -e "const p=require('./package.json'); if (p.dependencies['@xterm/xterm'] !== '6.0.0') process.exit(1)" +``` + +Expected: pass if the implementation chooses exact pinning. If it chooses CI probes instead, replace this command with the committed probe command and explain that choice in the PR. + +- [ ] **Step 7: Run repo-supported full check** + +```bash +FRESHELL_TEST_SUMMARY="terminal catch-up stream safety" timeout 1800s npm run check +``` + +Expected: full coordinated check passes. + +## Residual Risks And Cheapest Validations + +- Stream-stateful barrier scanner may be too conservative and reduce batching for ANSI-heavy output. Cheapest validation: log batch reasons and compare real coding-agent sessions. +- Multi-client geometry remains inherently constrained by one PTY size. Cheapest validation: visible-client resize authority test plus logs for geometry epoch mismatches. +- Retained byte replay is not a complete snapshot system, especially across geometry history. Cheapest validation: observe retained age, retained bytes, output rate, gap frequency, and geometry changes before designing snapshots. +- Protocol v5 clients cannot safely interoperate with mandatory stream identity. Cheapest validation: keep the protocol v6 rejection test and deploy client/server together. +- Protocol v6 clients may omit `terminal.output.batch`. Cheapest validation: keep additive fallback until support policy says non-batch v6 clients can be dropped. +- Stream/server identity rollout may need a compatibility bridge for existing local cursors. Cheapest validation: tests that old cursor records force full hydrate instead of warm delta replay. +- Current terminal stream remains UTF-8 string based and is not byte-perfect for invalid UTF-8 or raw 8-bit C1 controls. Cheapest validation: decide whether coding-agent terminals need byte-perfect replay before starting a separate byte-protocol project. +- Real Windows Chrome long-background behavior remains valuable release/user-acceptance evidence. Cheapest validation: run the Task 11 Windows Chrome follow-up when a suitable environment is available; local CDP freeze and Xvfb background probes are not acceptable substitutes unless their counters prove page execution actually stopped or throttled. + +## Self-Review + +Spec coverage: + +- The plan keeps #397's server-side batching benefit. +- The plan rejects #396's unsupported 32 ms replay budget. +- The plan fixes stale queued writes/callbacks with attach generation safety. +- The plan requires terminal-instance write scoping and drain-or-replace behavior for already-submitted xterm writes. +- The plan treats xterm disposal as non-canceling for pending write callbacks and requires token fences on post-dispose continuations. +- The plan replaces path-specific side-effect allow lists with a deny-by-default terminal output side-effect adapter. +- The plan replaces raw-byte send budgets with serialized application JSON payload budgets. +- The plan fragments oversized PTY output before sequence assignment, preserving distinct sequence ranges. +- The plan requires Unicode-safe fragmentation that does not split surrogate pairs. +- The plan defines server-owned stream identity and changes it on PTY/session replacement, Codex recovery replacement, retention loss, and incompatible restart. +- The plan explicitly preserves current UTF-8 string output semantics and does not claim byte-perfect terminal replay. +- The plan separates parser-applied sequence, observed sequence, replay request sequence, and known lost ranges. +- The plan quarantines parser-unsafe gaps instead of continuing to write into a potentially desynchronized parser. +- The plan replaces unsafe replay retention data structures. +- The plan treats parser side effects with an async terminal-instance xterm write scope, covering request-mode replies, OSC52 `always`, title updates, startup replies, write-callback checkpoint mutations, attach-completion mutations, local notices, link/action callbacks, and turn-complete. +- The plan replaces stateless barrier classification with a stream-stateful barrier scanner. +- The plan stores barrier scanner metadata with retained replay frames so arbitrary replay windows do not reconstruct unsafe parser state. +- The plan routes terminal broker WebSocket sends through a shared sender with callbacks, payload budgets, and instrumentation. +- The plan uses protocol v6 for mandatory stream identity, gates `terminal.output.batch` behind `terminalOutputBatchV1`, and keeps a non-batch v6 fallback. +- The plan requires safe non-batch fallback segmentation instead of flattening arbitrary batches. +- The plan adds observability for retention, lag, gaps, serialized bytes, and backpressure. +- The plan requires visible-first derived metrics and a local stop/resume positive-control probe before using browser audit results as PR acceptance evidence; real Windows Chrome long-background testing remains recommended release/user-acceptance follow-up. + +Placeholder scan: + +- Reviewer-facing placeholder tokens were scanned and none remain. Intentional test-first implementation stubs such as `throw new Error('implement stateful scanner')` remain only where the plan explicitly instructs workers to write failing tests before implementation. +- Every task has exact files, failing tests, commands, expected outcomes, and commits. + +Type consistency: + +- Cursor terminology is consistently `parserAppliedSeq`. +- Surface validity is consistently represented as `TerminalSurfaceCheckpoint`. +- Replay/loss bookkeeping is separate from parser-applied checkpoint validity. +- Output source is consistently `'live' | 'replay'`. +- Batch metadata consistently uses `segments`, `serializedBytes`, `seqStart`, and `seqEnd`. + +## Execution Handoff + +Plan complete and saved to `docs/superpowers/plans/2026-06-08-terminal-catchup-stream-safety.md`. + +Execution model: + +1. **Single-Branch Execution (required)** - implement every task in one worktree branch, run every task gate and final local proof gate, then open one PR. Local commits can track task boundaries, but do not publish partial PRs. + +2. **Agent Coordination** - scoped subagents can help with disjoint tasks or reviews inside that same branch, but integration remains local until the full end-to-end proof passes. diff --git a/docs/superpowers/proofs/2026-06-08-terminal-catchup-evidence-dossier.md b/docs/superpowers/proofs/2026-06-08-terminal-catchup-evidence-dossier.md new file mode 100644 index 000000000..4c3143d35 --- /dev/null +++ b/docs/superpowers/proofs/2026-06-08-terminal-catchup-evidence-dossier.md @@ -0,0 +1,416 @@ +# Terminal Catch-Up Evidence Dossier + +Created in proof worktree: `/home/dan/code/freshell/.worktrees/proof-terminal-catchup-architecture` + +Verdict: **implementation can proceed now**, with one explicit do-not-merge gate for real browser background/OS freeze behavior. All other implementation-shaping ambiguity is either proven in this worktree or converted into concrete architecture rules below. + +## Evidence Artifacts + +- `scripts/proofs/terminal-catchup-pty-metrics.ts` +- `scripts/proofs/terminal-catchup-agent-output-generator.mjs` +- `scripts/proofs/terminal-json-serialization-probe.ts` +- `scripts/proofs/xterm-write-dispose-probe.ts` +- `scripts/proofs/browser-freeze-lifecycle-probe.ts` +- `scripts/proofs/browser-background-visibility-probe.ts` +- `scripts/proofs/browser-process-suspend-probe.ts` +- `docs/superpowers/proofs/artifacts/terminal-catchup-pty-metrics.json` +- `docs/superpowers/proofs/artifacts/terminal-json-serialization.json` +- `docs/superpowers/proofs/artifacts/xterm-write-dispose.json` +- `docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json` +- `docs/superpowers/proofs/artifacts/browser-background-visibility.json` +- `docs/superpowers/proofs/artifacts/browser-process-suspend.json` + +## Assumptions Ledger + +| Status | Claim | Why it matters | Proof method | Evidence location | Confidence | Implementation implication | +|---|---|---|---|---|---:|---| +| PROVEN | Freshell PTY output enters the broker through `node-pty -> TerminalRegistry -> terminal.output.raw -> ReplayRing -> TerminalStreamBroker`. | Metrics must exercise the real terminal path, not stdout-only logs. | Harness spawned real PTYs through `TerminalRegistry`, attached via `TerminalStreamBroker`, captured raw events and broker sends. | `terminal-catchup-pty-metrics.ts`; `server/terminal-registry.ts:1523`; `server/terminal-stream/broker.ts:385` | 0.98 | Performance work belongs server-side in `server/terminal-stream`, with client safety around xterm application. | +| PROVEN | Real Codex CLI and stress traces have small chunks but bursty delivery; stress output reached 776,745 bytes in one burst and replay compressed 3,239 raw PTY chunks into 33 replay frames. | Sizes retention and batching; proves message-count reduction. | PTY harness scenarios: `codex-version`, `codex-help`, `codex-real-turn`, `agent-burst-12000`, `control-barrier`. | `terminal-catchup-pty-metrics.json` | 0.95 | Keep server batching; do not rely on client write coalescing as the primary fix. | +| PROVEN | A conservative stream scanner still batches heavily: 3,239 stress chunks become 170 conservative batches while respecting control barriers. | Shows safety barriers do not eliminate batching gains. | Stateful scanner in harness treats OSC/DCS/request/replacement/pending states as barriers. | `terminal-catchup-pty-metrics.json`, `agent-burst-12000.scanner` | 0.90 | Implement scanner snapshots at ingestion; batch only when scanner ground-state and no side-effect barrier. | +| PROVEN | Raw 16 KiB budget is not serialized JSON budget. ESC-only 16 KiB data serialized as 98,423 bytes; ANSI SGR 16 KiB serialized as 30,158 bytes. | Current budget can overshoot WebSocket payload/backpressure limits. | JSON serialization probe around current `terminal.output` shape. | `terminal-json-serialization.json`; `server/terminal-stream/broker.ts:593`; `server/terminal-stream/broker.ts:661` | 0.99 | Batch budgets must be serialized application payload bytes, measured before send. | +| DISPROVEN | CDP `Page.setWebLifecycleState({state:'frozen'})` is a valid local proof of frozen-tab behavior in this environment. | Would have closed browser-background ambiguity. | Headless and headed/Xvfb probes kept timers, RAF, and WS at active rates while command reported success. | `browser-freeze-lifecycle.json` | 0.99 | Do not cite CDP lifecycle freeze as acceptance evidence here. | +| DISPROVEN | Xvfb `bringToFront()` produces real background-tab `document.hidden` behavior locally. | Would have proven OS-tab background throttling. | Two-page headed/Xvfb probe showed `visibilityState:"visible"` and active-rate timers/RAF/WS for the covered page. | `browser-background-visibility.json` | 0.99 | Real OS/browser background remains a do-not-merge acceptance gate. | +| PROVEN | When browser execution is actually stopped, WS data accumulates and delivers as a catch-up burst on resume. | Proves the failure mechanics even though OS background is gated. | Suspended exact Chromium process tree with `SIGSTOP`; server sent 40 WS frames while stopped; after resume immediate page delta had 1 interval tick, 3 RAF ticks, and 40 WS messages. | `browser-process-suspend.json` | 0.95 | Retention and replay must tolerate frozen clients that receive a burst on resume. | +| GATED | Real Windows/Chrome/OS background/freeze behavior for Freshell tabs is not proven in this environment. | The original bug involved a browser tab backgrounded for hours. | Feasible local probes failed to enter real hidden/frozen browser state. | `browser-freeze-lifecycle.json`; `browser-background-visibility.json` | 0.70 | Coding can start; merge requires the acceptance gate below. | +| PROVEN | Current multi-client resize authority is partial: broker attach resizes on `viewport_hydrate` always and `transport_reconnect` only when no other socket or same socket, but standalone `terminal.resize` resizes unconditionally. | Geometry changes alter wrapping and invalidate byte replay. | Source inspection. | `server/terminal-stream/broker.ts:109`; `server/ws-handler.ts:3203`; `server/terminal-registry.ts:3160` | 0.98 | Warm replay is allowed only under compatible geometry authority/history; unknown multi-client geometry quarantines/rebuilds. | +| PLAN-FIXED | Geometry authority must be explicit: `single_client`, `server_stream`, or `multi_client_unknown`. | Without this, a client can replay bytes into a surface wrapped at a different size. | Current source lacks geometry epochs/history and only stores current `cols/rows`. | `server/terminal-registry.ts:3164`; `src/lib/terminal-attach-policy.ts:31` | 0.93 | Add geometry epoch/history to server stream metadata and checkpoint validation. | +| PROVEN | Current hello capabilities only include `uiScreenshotV1`; server stores only that flag. | Batch protocol must not break old clients. | Source inspection and runtime `HelloSchema.safeParse` with added `terminalOutputBatchV1`: accepted and stripped. | `shared/ws-protocol.ts:225`; `src/lib/ws-client.ts:333`; `server/ws-handler.ts:2131`; command output from `tsx -e HelloSchema.safeParse(...)` | 0.99 | Additive capability rollout is safe if new clients still accept legacy `terminal.output`. | +| PROVEN | Server-to-client messages are TypeScript-only; no runtime server-message schema exists. | Tests must not pretend a `ServerMessageSchema` already validates batches. | Source comment and union inspection. | `shared/ws-protocol.ts:1`; `shared/ws-protocol.ts:981` | 0.99 | Either add explicit server-message schema intentionally or test sender output directly. | +| PLAN-FIXED | Warm replay validity needs terminal id, stream id, server identity, attach generation, geometry authority/history, scrollback, xterm version, parser-applied checkpoint, local surface epoch, retention coverage, and seq continuity. | A `seq` alone does not prove equivalent terminal surface state. | Current state has terminalId/renderedSeq/serverInstanceId/bootId/attachRequestId, but no streamId/surfaceEpoch/geometryEpoch/parserApplied/retention coverage. | `src/components/TerminalView.tsx:535`; `shared/ws-protocol.ts:626`; `src/components/TerminalView.tsx:1666` | 0.95 | Implement checkpoint validation; if any key is absent/incompatible, quarantine or full rebuild. | +| PROVEN | xterm writes are asynchronous in this install and callbacks survive `dispose()`. | Stale callbacks can mutate a new surface unless fenced. | Installed `@xterm/xterm@6.0.0` probe: small write callbacks after write returns; large callback fired 310 ms after dispose; 500 queued callbacks all fired after dispose, FIFO. | `xterm-write-dispose.json`; `package.json:92` | 0.99 | Every xterm callback and parser side effect must check surface epoch + attach generation before mutating state. | +| PROVEN | Broker send and handler send are not equivalent. | Terminal stream sends need same backpressure, payload, and instrumentation behavior as other WS sends. | Source inspection: handler send logs large payloads and callback timing; broker `safeSend` only `JSON.stringify` + `ws.send`. | `server/ws-handler.ts:1492`; `server/ws-handler.ts:1511`; `server/terminal-stream/broker.ts:661` | 0.99 | Introduce shared sender and route broker/direct terminal sends through it. | +| PROVEN | Legacy direct registry output path is still reachable for any `registry.attach` call without `suppressOutput`. Browser attach uses suppressOutput via broker, but direct path emits unsequenced `terminal.output`. | Rollout must not leave a path that bypasses seq/batch/stream safety. | Source inspection. | Browser path: `server/ws-handler.ts:3027`; suppress: `server/terminal-stream/broker.ts:103`; direct path: `server/terminal-registry.ts:3045`; unsequenced send: `server/terminal-registry.ts:3463` | 0.98 | During rollout, migrate or fence direct path; legacy fallback must emit modern segmented frames with seq metadata. | +| PROVEN | Current client rejects untagged/overlapping output during active attach and ignores missing sequence ranges. | Old direct frames can be dropped, and new batching must preserve seq continuity. | Source inspection. | `src/components/TerminalView.tsx:1666`; `src/components/TerminalView.tsx:2119`; `src/components/TerminalView.tsx:2128` | 0.98 | Do not emit old-shape `terminal.output` from any browser-visible path. | +| PROVEN | Replay-sensitive side effects exist before xterm write and inside xterm callbacks/handlers: startup replies, OSC52, turn completion, link handling, title updates, local `writeln` notices, perf marks, checkpoint persistence. | Replay must not re-send PTY replies, clipboard writes, Redux updates, or mutate the local surface invisibly. | Source inspection. | `src/components/TerminalView.tsx:1078`; `src/components/TerminalView.tsx:1035`; `src/components/TerminalView.tsx:1096`; `src/components/terminal/request-mode-bypass.ts:237`; `src/components/TerminalView.tsx:1268`; `src/components/TerminalView.tsx:1597`; local writes at `src/components/TerminalView.tsx:1985`, `2243`, `2518`, `2550`, `2646` | 0.98 | Add deny-by-default side-effect adapter; local terminal writes invalidate warm replay unless moved out-of-band. | +| PROVEN | Current client write queue coalesces replay writes but has no attach/surface generation metadata. | Delayed callbacks can advance stale checkpoints or complete stale attach generations. | Source inspection. | `src/components/terminal/terminal-write-queue.ts:22`; `src/components/terminal/terminal-write-queue.ts:54`; `src/components/terminal/terminal-write-queue.ts:85` | 0.98 | Write queue must own generation-tagged queued/submitted writes and callback fences. | +| PROVEN | Current replay ring uses array `shift()` eviction and coalesces by raw bytes, not scanner/serialized budgets. | Many tiny frames can make eviction O(n); raw budget is unsafe for JSON. | Source inspection. | `server/terminal-stream/replay-ring.ts:57`; `server/terminal-stream/replay-ring.ts:81`; `server/terminal-stream/replay-ring.ts:143` | 0.98 | Replace with indexed deque/ring; store scanner state snapshots and serialized byte accounting. | +| PROVEN | Current coding CLI replay retention floor is 8 MiB; this covers 8 hours only at about 291 B/s. | The bug is hours-long hidden catch-up; 8 MiB is not enough for sustained multi-KiB/s output. | Source default and measured rate math. | `server/terminal-stream/broker.ts:19`; `terminal-catchup-pty-metrics.json` | 0.95 | Add explicit memory/disk retention budgets and coverage gates; missing coverage invalidates warm replay. | + +No row is `BLOCKED`. The only remaining uncertainty is intentionally `GATED`. + +## Decisive Proofs + +### PTY Metrics + +The PTY harness exercises the real Freshell terminal path. It spawns a shell PTY through `TerminalRegistry`, attaches through `TerminalStreamBroker`, sends commands via `registry.input`, captures `terminal.output.raw`, and captures the broker's serialized WebSocket messages. + +Key results from `terminal-catchup-pty-metrics.json`: + +- `codex-real-turn`: 2,000 raw bytes, 15 raw chunks, 120 B/s over the measured turn; preview contains `proof-line-1` through `proof-line-40`. +- `codex-help`: 4,686 raw bytes, 7 raw chunks, 7,025 B/s during the help burst. +- `agent-burst-12000`: 776,745 raw bytes, 3,239 raw chunks, 622,392 B/s during the local burst; broker replay emitted 33 output frames. +- `control-barrier`: scanner found 7 barrier frames, 6 side-effect barriers, 1 replacement frame, and 2 pending-state frames. + +The stress scenario is intentionally a burst upper bound, not a claim that real agents sustain 622 KiB/s for hours. + +### Browser Lifecycle + +Local browser lifecycle proof is split into three artifacts: + +- `browser-freeze-lifecycle.json`: CDP `Page.setWebLifecycleState({state:'frozen'})` did not freeze page work. During the supposed frozen period, page counters advanced 40 intervals, 120 RAFs, and 40 WS messages, exactly matching the server's 40 sends. +- `browser-background-visibility.json`: Xvfb `bringToFront()` did not make the prior tab hidden. `document.hidden` stayed false and all counters ran at active rates. +- `browser-process-suspend.json`: stopping the full Chromium process tree did stop page work. The server sent 40 WS frames while stopped. Immediately after resume, page counters advanced by 1 interval and 3 RAFs but received all 40 queued WS messages. + +Conclusion: local probes prove the catch-up mechanics but do not prove real OS tab background behavior. Merge must be gated on the acceptance test below. + +### Retention Sizing + +Measured rates converted to retained bytes: + +| Rate source | Rate | 4h retained | 8h retained | +|---|---:|---:|---:| +| Actual small Codex turn | 120 B/s | 1.65 MiB | 3.30 MiB | +| Current 8 MiB coding CLI floor coverage | 291 B/s for 8h | 8 MiB covers 4h at 582 B/s | 8 MiB covers 8h at 291 B/s | +| `codex-help` burst if sustained | 7,025 B/s | 96.5 MiB | 193 MiB | +| 32 MiB memory cap coverage | 1,165 B/s for 8h | 32 MiB covers 4h at 2,330 B/s | 32 MiB covers 8h at 1,165 B/s | +| 256 MiB disk cap coverage | 9,320 B/s for 8h | 256 MiB covers 4h at 18,641 B/s | 256 MiB covers 8h at 9,320 B/s | +| 1 GiB disk cap coverage | 37,283 B/s for 8h | 1 GiB covers 4h at 74,565 B/s | 1 GiB covers 8h at 37,283 B/s | +| Local stress burst if sustained | 622,392 B/s | 8.35 GiB | 16.70 GiB | + +Retention rule: + +- Keep a hot memory replay ring with a default coding-agent cap of 32 MiB per terminal. +- Add an optional disk replay spool defaulting to 256 MiB per coding-agent terminal, with a 1 GiB configurable hard cap. +- Warm replay is valid only when retention covers every byte since `parserAppliedSeq + 1`. +- If coverage is missing, emit a gap and quarantine/rebuild. Do not silently continue warm replay. + +### Geometry Authority + +Current source proves that resize authority is not fully tracked: + +- Attach path resizes on `viewport_hydrate`, and on `transport_reconnect` only when no other attached socket exists or the same socket is reconnecting: `server/terminal-stream/broker.ts:109`. +- Any `terminal.resize` message calls `registry.resize` unconditionally: `server/ws-handler.ts:3203`. +- `TerminalRegistry.resize` stores only current `cols`/`rows` and calls `pty.resize`: `server/terminal-registry.ts:3160`. + +Architecture rule: + +- Warm replay is allowed only under `geometryAuthority='single_client'` or `geometryAuthority='server_stream'` with compatible history from checkpoint to head. +- If any other client can resize without compatible server-side geometry history, set `geometryAuthority='multi_client_unknown'` and quarantine/rebuild instead of warm replay. + +### Rollout Matrix + +Use an additive capability, not a protocol bump, for `terminalOutputBatchV1`. + +| Client | Server | Proven behavior / required behavior | +|---|---|---| +| old | old | Existing legacy `terminal.output` path works. | +| new | old | Proven compatible: current `HelloSchema` accepts and strips unknown `terminalOutputBatchV1`; old server sends legacy output; new client must support legacy. | +| old | new | New server sees no batch capability and must send only legacy segmented `terminal.output` with seq metadata. Do-not-merge if batch is sent without capability. | +| new | new | New server may send `terminal.output.batch` only when capability is present; unsafe/barrier segments may still use legacy `terminal.output`. | + +### Warm Replay Validity Keys + +| Key | Current state | Required rule | +|---|---|---| +| Terminal id | Present in output and pane content. | Must match checkpoint. | +| Stream identity | No terminal output stream id; unrelated opencode tracker stream ids exist and are not terminal byte-stream ids. | Server mints `streamId`; changes on PTY/session replacement, Codex recovery PTY replacement, incompatible retention loss, and incompatible restart. | +| Server identity | `ready` includes `serverInstanceId` and `bootId`. | Checkpoint must include both; boot/server mismatch invalid unless persisted replay retention proves compatibility. | +| Attach generation | Current `attachRequestId` tags stream messages. | Every queued/submitted write and callback must also carry attach generation. | +| Geometry authority/history | Current `cols/rows`; no history/epoch. | Include `geometryEpoch`, authority, and resize history/snapshots needed for replay. | +| Scrollback | xterm and server settings have scrollback, but checkpoints do not include it. | Include scrollback in checkpoint; change invalidates warm replay. | +| xterm version | Package range is `^6.0.0`; installed is 6.0.0. | Pin exact xterm or run CI probe for resolved version; include version in checkpoint. | +| Parser-applied checkpoint | Current `renderedSeq` is advanced from write callback naming, but stored as terminal cursor only. | Rename to `parserAppliedSeq`; advance only from fenced xterm write callback. | +| Local mutation epoch | Local `term.writeln` paths exist. | Out-of-band React notices preferred; remaining local writes bump `surfaceEpoch` and invalidate warm delta replay. | +| Retention coverage | Current ring can evict and reports gaps. | Warm replay requires retained coverage from checkpoint to target; missing coverage quarantines/rebuilds. | +| Seq continuity | Client rejects overlap and missing seq. | Batch/legacy fallback must preserve unique contiguous seq ranges; gaps never advance parser-applied state. | + +### xterm Fence Requirement + +The installed package behavior is decisive: + +- Small write callbacks run after `term.write` returns. +- A large write callback fired after `term.dispose()`. +- 500 queued write callbacks all fired after dispose and remained FIFO. + +Required fence: + +- Every write callback receives `{ terminalInstanceId, surfaceEpoch, attachRequestId, seqEnd, source }`. +- Callback mutates checkpoint/attach state only if all identity fields still match active state. +- Parser callbacks and side effects must use the same active write scope. Stack-scoped context around `term.write()` is not enough. + +### Shared Sender + +Required shared sender behavior: + +- Check `readyState`. +- Check and close on configured backpressure threshold. +- Serialize exactly once. +- Measure serialized payload bytes. +- Enforce payload budget before `ws.send`. +- Use send callback for large-payload timing and error logging. +- Return boolean delivery acceptance to callers. +- Emit structured JSONL events with severity and stable event names. + +Broker and registry direct output must use this sender. + +### Side-Effect Adapter + +Deny-by-default effect contract: + +```ts +type TerminalSideEffectContext = { + terminalId: string + streamId: string + terminalInstanceId: string + surfaceEpoch: number + attachRequestId: string + source: 'live' | 'replay' + parserAppliedSeq?: number + segmentSeqStart?: number + segmentSeqEnd?: number +} + +type TerminalSideEffect = + | { type: 'pty.reply.startup_probe'; data: string } + | { type: 'pty.reply.request_mode'; data: string } + | { type: 'clipboard.osc52.write'; text: string } + | { type: 'clipboard.osc52.prompt'; event: unknown } + | { type: 'redux.turn_complete.record' } + | { type: 'redux.title.update'; title: string } + | { type: 'pane.link.open'; uri: string } + | { type: 'terminal.local_notice.write'; text: string } + | { type: 'checkpoint.persist'; seq: number } + | { type: 'attach.complete'; seq: number } + | { type: 'perf.mark'; name: string } +``` + +Default policy: + +- Replay denies PTY replies, clipboard writes/prompts, title changes, link opens, client-minted turn completion, and local terminal writes. +- Live allows declared effects only if the context matches the active surface. +- Unknown effect type is a test failure and runtime suppression. + +## Implementation Plan + +### PR 1: Sender, Metrics, And Protocol-Neutral Safety + +Expected files touched: + +- `server/ws-handler.ts` +- `server/terminal-stream/broker.ts` +- `server/terminal-registry.ts` +- `server/terminal-stream/constants.ts` +- `server/perf-logger.ts` +- New `server/ws-sender.ts` + +Work: + +- Extract shared WebSocket sender. +- Route broker sends and registry terminal sends through it. +- Add serialized byte measurements and structured JSONL logs: + - `terminal_catchup.replay_hit` + - `terminal_catchup.replay_miss` + - `terminal_catchup.batch_sent` + - `terminal_catchup.gap` + - `terminal_catchup.sender_backpressure` +- Keep protocol output as legacy `terminal.output`. + +Tests: + +- Unit shared sender: readyState, backpressure, payload byte warning, send callback errors. +- Regression: broker and handler send same serialized payload and same close behavior. + +### PR 2: Server Stream Identity, Scanner, And Retention Coverage + +Expected files touched: + +- `server/terminal-stream/replay-ring.ts` +- New `server/terminal-stream/control-sequence-scanner.ts` +- New `server/terminal-stream/retention-store.ts` +- `server/terminal-stream/types.ts` +- `server/terminal-stream/broker.ts` +- `server/terminal-registry.ts` + +Work: + +- Add terminal byte-stream `streamId`. +- Replace array/`shift()` ring with indexed deque. +- Store scanner state snapshots at ingestion. +- Split frames on code-point boundaries before seq assignment when needed. +- Add serialized-byte budgeting. +- Add memory retention cap and optional disk spool cap. +- Emit explicit coverage result for every attach. + +Tests: + +- Scanner state across split ESC/CSI/OSC/DCS/APC/BEL/C1/replacement. +- No lone surrogate chunks. +- Batch never crosses source/stream/attach/geometry/barrier/budget. +- Retention miss emits gap and never advances parser-applied seq. +- Ring eviction avoids O(n) shift behavior. + +### PR 3: Client Checkpoint, Write Queue Fencing, And Side Effects + +Expected files touched: + +- `src/components/TerminalView.tsx` +- `src/components/terminal/terminal-write-queue.ts` +- `src/components/terminal/request-mode-bypass.ts` +- New `src/lib/terminal-surface-checkpoint.ts` +- New `src/lib/terminal-side-effects.ts` +- `src/lib/terminal-attach-policy.ts` +- Cursor persistence helper currently used by `TerminalView` + +Work: + +- Replace bare rendered cursor with `TerminalSurfaceCheckpoint`. +- Rename semantics to `parserAppliedSeq`. +- Add `surfaceEpoch`, `terminalInstanceId`, `xtermVersion`, `scrollback`, geometry, stream/server ids. +- Submit at most one xterm write per surface unless later tests prove parallel scoped parser callbacks. +- Fence callbacks after dispose/stale attach. +- Move local terminal notices out-of-band where possible; otherwise bump surface mutation epoch. +- Add deny-by-default side-effect adapter. + +Tests: + +- xterm dispose callback regression from artifact. +- Stale callback cannot mark attach complete or persist cursor. +- Replay suppresses OSC52, request-mode replies, title changes, turn completion, link opens. +- Local notice invalidates warm replay. + +### PR 4: Geometry Authority And Warm Replay Policy + +Expected files touched: + +- `server/terminal-stream/broker.ts` +- `server/terminal-registry.ts` +- `shared/ws-protocol.ts` +- `src/lib/terminal-attach-policy.ts` +- `src/components/TerminalView.tsx` + +Work: + +- Add `geometryEpoch`, authority, and resize history. +- Make warm replay validator require compatible geometry. +- Quarantine/rebuild on multi-client unknown authority. +- Ensure `terminal.resize` updates server geometry epoch and invalidates affected checkpoints. + +Tests: + +- Single client warm delta allowed. +- Same socket reconnect allowed if geometry unchanged. +- Other client resize marks unknown/incompatible. +- Replay after incompatible geometry refuses warm delta and uses rebuild/quarantine. + +### PR 5: Batch Capability And Legacy Fallback + +Expected files touched: + +- `shared/ws-protocol.ts` +- `src/lib/ws-client.ts` +- `server/ws-handler.ts` +- `server/terminal-stream/broker.ts` +- `src/components/TerminalView.tsx` + +Work: + +- Add `terminalOutputBatchV1` hello capability. +- Add `terminal.output.batch` type and tests. +- Server sends batch only for capable client. +- Legacy fallback emits individual modern `terminal.output` segments with seq metadata; never old-shape unsequenced output. + +Tests: + +- Old client/new server receives no batch. +- New client/old server works with legacy output. +- New/new receives batch for safe segments. +- Unsafe/barrier batch falls back or splits without changing semantics. + +### PR 6: Browser Acceptance And Tuning + +Expected files touched: + +- `test/e2e-browser/...` +- `scripts/visible-first-audit.ts` +- `scripts/assert-visible-first-audit-gate.ts` +- `docs/index.html` only if user-facing behavior changes are visible. + +Work: + +- Add visible-first metrics: + - replay message count + - serialized replay bytes + - parser-applied lag + - gap count/ranges + - warm replay accepted/rejected reason + - stale callback rejection count + - side-effect suppression count +- Add real browser background acceptance gate. + +Exact do-not-merge browser gate: + +1. Start an isolated Freshell server from the feature worktree on a unique port; do not touch the self-hosted dev server. +2. Open real Windows Chrome, not headless/Xvfb. +3. Create a terminal running the deterministic generator and at least one real Codex turn. +4. Confirm `document.visibilityState === 'hidden'` or an OS freeze/suspend event while the tab is backgrounded/minimized. +5. Keep it backgrounded for a 4h soak at a calibrated 1 KiB/s stream and one shorter burst at >=750 KiB total. +6. Refocus and assert: + - no unsafe warm replay after retention loss; + - no unsequenced `terminal.output`; + - no parser-unsafe gap continues on same parser; + - no replay-triggered OSC52/request-mode/title/turn side effect; + - catch-up to server head completes under configured UX budget for covered retention; + - all metrics above are present in JSONL logs. +7. Repeat one 8h overnight soak before merge if disk retention is part of the PR. + +## Invariants + +- Sequence ranges are unique, contiguous within a stream, and never overlap. +- Split oversized output before seq assignment, on Unicode code-point boundaries. +- `parserAppliedSeq` advances only from active fenced xterm write callback. +- Gap receipt never advances `parserAppliedSeq`. +- Warm replay requires every validity key to match. +- Batch never crosses stream, attach, source, geometry, parser barrier, gap, or serialized budget boundary. +- Legacy fallback emits the same safe segments as batch mode. +- Side effects are denied by default. +- Local terminal writes invalidate warm replay unless rendered out-of-band. +- Shared sender is the only terminal WebSocket send path. + +## Kill Switches And Rollback + +- `TERMINAL_CATCHUP_ENABLED=false`: disable warm delta replay and use viewport hydrate/quarantine path. +- `TERMINAL_OUTPUT_BATCH_V1=false`: force legacy segmented `terminal.output`. +- `TERMINAL_REPLAY_DISK_SPOOL=false`: memory-only retention. +- `TERMINAL_REPLAY_MAX_MEMORY_BYTES`: hard memory cap. +- `TERMINAL_REPLAY_MAX_DISK_BYTES`: hard disk cap. +- `TERMINAL_REPLAY_WARM_GEOMETRY=false`: quarantine on any resize history mismatch. +- Runtime setting/logged config snapshot must include every switch value. + +Rollback path: + +- Disable batch capability first. +- Disable warm replay second. +- Keep shared sender and logging; they are safety improvements and should remain unless they are the failure source. + +## Do-Not-Merge Criteria + +- Any batch is sent to a client that did not advertise `terminalOutputBatchV1`. +- Any terminal output path can emit old-shape unsequenced `terminal.output` to browser clients. +- Any xterm callback can mutate state without surface + attach-generation fence. +- Any gap advances `parserAppliedSeq`. +- Any batch crosses a scanner barrier, source boundary, stream id, attach id, geometry epoch, or serialized byte budget. +- Any local `term.write`/`term.writeln` does not either render out-of-band or invalidate warm replay. +- Any replay side effect is allowed by default. +- xterm remains a loose semver range without CI behavior probes for the resolved version. +- Browser background acceptance gate is missing or fails. +- Structured JSONL observability is missing severity, terminal id, stream id, attach id, seq range, and rejection reason. diff --git a/docs/superpowers/proofs/artifacts/browser-background-visibility.json b/docs/superpowers/proofs/artifacts/browser-background-visibility.json new file mode 100644 index 000000000..f574cc5dc --- /dev/null +++ b/docs/superpowers/proofs/artifacts/browser-background-visibility.json @@ -0,0 +1,94 @@ +{ + "generatedAt": "2026-06-09T01:00:28.173Z", + "chromiumVersion": "145.0.7632.6", + "headed": true, + "wsSendIntervalMs": 50, + "active": { + "before": { + "at": 1087.699999988079, + "visibilityState": "visible", + "hidden": false, + "intervalTicks": 21, + "rafTicks": 57, + "wsReceived": 21, + "lastIntervalAt": 1059.4000000059605, + "lastRafAt": 1086.2999999821186, + "lastWsAt": 1064.9000000059605 + }, + "after": { + "at": 2091.2999999821186, + "visibilityState": "visible", + "hidden": false, + "intervalTicks": 41, + "rafTicks": 117, + "wsReceived": 41, + "lastIntervalAt": 2059.2999999821186, + "lastRafAt": 2087.0999999940395, + "lastWsAt": 2069.4000000059605 + }, + "pageDelta": { + "elapsedMs": 1003.5999999940395, + "visibilityStateBefore": "visible", + "visibilityStateAfter": "visible", + "intervalTicks": 20, + "rafTicks": 60, + "wsReceived": 20 + }, + "serverSentDelta": 20 + }, + "backgrounded": { + "before": { + "at": 2349.0999999940395, + "visibilityState": "visible", + "hidden": false, + "intervalTicks": 46, + "rafTicks": 132, + "wsReceived": 46, + "lastIntervalAt": 2309.199999988079, + "lastRafAt": 2336.4000000059605, + "lastWsAt": 2322.5999999940395 + }, + "after": { + "at": 5351.5, + "visibilityState": "visible", + "hidden": false, + "intervalTicks": 106, + "rafTicks": 312, + "wsReceived": 106, + "lastIntervalAt": 5309.199999988079, + "lastRafAt": 5336.5999999940395, + "lastWsAt": 5327.9000000059605 + }, + "pageDelta": { + "elapsedMs": 3002.4000000059605, + "visibilityStateBefore": "visible", + "visibilityStateAfter": "visible", + "intervalTicks": 60, + "rafTicks": 180, + "wsReceived": 60 + }, + "serverSentDelta": 60 + }, + "resumed": { + "after": { + "at": 6362, + "visibilityState": "visible", + "hidden": false, + "intervalTicks": 127, + "rafTicks": 373, + "wsReceived": 126, + "lastIntervalAt": 6359.199999988079, + "lastRafAt": 6352.699999988079, + "lastWsAt": 6331.199999988079 + }, + "pageDeltaFromHiddenEnd": { + "elapsedMs": 1010.5, + "visibilityStateBefore": "visible", + "visibilityStateAfter": "visible", + "intervalTicks": 21, + "rafTicks": 61, + "wsReceived": 20 + }, + "serverSentDelta": 20 + } +} diff --git a/docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json b/docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json new file mode 100644 index 000000000..65b52c91a --- /dev/null +++ b/docs/superpowers/proofs/artifacts/browser-freeze-lifecycle.json @@ -0,0 +1,103 @@ +{ + "generatedAt": "2026-06-09T00:59:10.228Z", + "chromiumVersion": "145.0.7632.6", + "headed": true, + "wsSendIntervalMs": 50, + "active": { + "before": { + "at": 1024.800000011921, + "visibilityState": "visible", + "intervalTicks": 20, + "rafTicks": 50, + "wsReceived": 20, + "lastIntervalAt": 1012.1000000238419, + "lastRafAt": 1024, + "lastWsAt": 993.3000000119209 + }, + "after": { + "at": 2027, + "visibilityState": "visible", + "intervalTicks": 40, + "rafTicks": 110, + "wsReceived": 40, + "lastIntervalAt": 2012.1000000238419, + "lastRafAt": 2024.4000000059605, + "lastWsAt": 1995.5 + }, + "pageDelta": { + "elapsedMs": 1002.1999999880791, + "intervalTicks": 20, + "rafTicks": 60, + "wsReceived": 20 + }, + "serverSentDelta": 20 + }, + "frozen": { + "stateCommand": "Page.setWebLifecycleState({ state: 'frozen' })", + "durationMs": 3008, + "before": { + "at": 2027, + "visibilityState": "visible", + "intervalTicks": 40, + "rafTicks": 110, + "wsReceived": 40, + "lastIntervalAt": 2012.1000000238419, + "lastRafAt": 2024.4000000059605, + "lastWsAt": 1995.5 + }, + "evalStatus": "returned", + "evalSnapshot": { + "at": 4029.9000000059605, + "visibilityState": "visible", + "intervalTicks": 80, + "rafTicks": 230, + "wsReceived": 80, + "lastIntervalAt": 4012.100000023842, + "lastRafAt": 4024.100000023842, + "lastWsAt": 4002.7000000178814 + }, + "pageDeltaIfEvalReturned": { + "elapsedMs": 2002.9000000059605, + "intervalTicks": 40, + "rafTicks": 120, + "wsReceived": 40 + }, + "serverSentDelta": 40 + }, + "resumed": { + "immediate": { + "at": 4032, + "visibilityState": "visible", + "intervalTicks": 80, + "rafTicks": 230, + "wsReceived": 80, + "lastIntervalAt": 4012.100000023842, + "lastRafAt": 4024.100000023842, + "lastWsAt": 4002.7000000178814 + }, + "after": { + "at": 5034.800000011921, + "visibilityState": "visible", + "intervalTicks": 100, + "rafTicks": 290, + "wsReceived": 100, + "lastIntervalAt": 5012.100000023842, + "lastRafAt": 5024.100000023842, + "lastWsAt": 5003.800000011921 + }, + "immediateDeltaFromFreezeStart": { + "elapsedMs": 2005, + "intervalTicks": 40, + "rafTicks": 120, + "wsReceived": 40 + }, + "pageDeltaAfterResume": { + "elapsedMs": 1002.8000000119209, + "intervalTicks": 20, + "rafTicks": 60, + "wsReceived": 20 + }, + "serverSentDeltaImmediate": 40, + "serverSentDeltaAfterResume": 20 + } +} diff --git a/docs/superpowers/proofs/artifacts/browser-process-suspend.json b/docs/superpowers/proofs/artifacts/browser-process-suspend.json new file mode 100644 index 000000000..6f7ae5de4 --- /dev/null +++ b/docs/superpowers/proofs/artifacts/browser-process-suspend.json @@ -0,0 +1,89 @@ +{ + "generatedAt": "2026-06-09T01:04:42.470Z", + "chromiumVersion": "145.0.7632.6", + "browserPid": 1211057, + "wsSendIntervalMs": 50, + "active": { + "before": { + "at": 1020.2000000178814, + "intervalTicks": 20, + "rafTicks": 61, + "wsReceived": 20, + "lastIntervalAt": 1006.9000000059605, + "lastRafAt": 1009.1000000238419, + "lastWsAt": 1013.2000000178814 + }, + "after": { + "at": 2024.300000011921, + "intervalTicks": 40, + "rafTicks": 121, + "wsReceived": 40, + "lastIntervalAt": 2007, + "lastRafAt": 2009, + "lastWsAt": 2017.2000000178814 + }, + "pageDelta": { + "elapsedMs": 1004.0999999940395, + "intervalTicks": 20, + "rafTicks": 60, + "wsReceived": 20 + }, + "serverSentDelta": 20 + }, + "stopped": { + "signal": "SIGSTOP", + "stoppedPids": [ + 1211057, + 1211092, + 1211140, + 1211093, + 1211229, + 1211145 + ], + "durationMs": 2002, + "before": { + "at": 2024.300000011921, + "intervalTicks": 40, + "rafTicks": 121, + "wsReceived": 40, + "lastIntervalAt": 2007, + "lastRafAt": 2009, + "lastWsAt": 2017.2000000178814 + }, + "afterResumeImmediate": { + "at": 4053, + "intervalTicks": 41, + "rafTicks": 124, + "wsReceived": 80, + "lastIntervalAt": 4050.5, + "lastRafAt": 4050.600000023842, + "lastWsAt": 4052.800000011921 + }, + "pageDeltaAfterResumeImmediate": { + "elapsedMs": 2028.699999988079, + "intervalTicks": 1, + "rafTicks": 3, + "wsReceived": 40 + }, + "serverSentWhileStopped": 40, + "serverSentByResumeImmediate": 40 + }, + "resumed": { + "after": { + "at": 5057.600000023842, + "intervalTicks": 62, + "rafTicks": 184, + "wsReceived": 100, + "lastIntervalAt": 5057, + "lastRafAt": 5042.300000011921, + "lastWsAt": 5026.5 + }, + "pageDeltaAfterResume": { + "elapsedMs": 1004.6000000238419, + "intervalTicks": 21, + "rafTicks": 60, + "wsReceived": 20 + }, + "serverSentDeltaAfterResume": 20 + } +} diff --git a/docs/superpowers/proofs/artifacts/terminal-catchup-pty-metrics.json b/docs/superpowers/proofs/artifacts/terminal-catchup-pty-metrics.json new file mode 100644 index 000000000..7cc877b59 --- /dev/null +++ b/docs/superpowers/proofs/artifacts/terminal-catchup-pty-metrics.json @@ -0,0 +1,718 @@ +{ + "generatedAt": "2026-06-09T00:56:47.470Z", + "repoRoot": "/home/dan/code/freshell/.worktrees/proof-terminal-catchup-architecture", + "nodeVersion": "v22.21.1", + "terminalStreamBatchMaxBytes": 16384, + "dependencies": { + "@xterm/xterm": "6.0.0", + "ws": "8.19.0", + "node-pty": "1.2.0-beta.11" + }, + "results": [ + { + "scenario": "codex-version", + "command": "codex --version; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-version' '1'", + "terminalId": "FXO8Qf8fVAUZToVgC8dQe", + "durationMs": 658, + "raw": { + "chunks": 6, + "bytes": 613, + "bytesPerSecond": 932, + "previewStartJsonEscaped": "codex --version; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-version' '1'\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ codex --version; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-version' '1'\r\n\u001b[?2004l\rcodex-cli 0.137.0\r\n\r\nFRESHELL_PROOF_DONE:codex-version:1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "previewEndJsonEscaped": "codex --version; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-version' '1'\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ codex --version; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-version' '1'\r\n\u001b[?2004l\rcodex-cli 0.137.0\r\n\r\nFRESHELL_PROOF_DONE:codex-version:1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "chunkBytes": { + "min": 19, + "p50": 77, + "p90": 196, + "p95": 196, + "p99": 196, + "max": 196 + }, + "interarrivalMs": { + "min": 0, + "p50": 8, + "p90": 347, + "p95": 347, + "p99": 347, + "max": 347 + }, + "burstGroupsUnder10ms": { + "count": 3, + "chunks": { + "min": 1, + "p50": 2, + "p90": 3, + "p95": 3, + "p99": 3, + "max": 3 + }, + "bytes": { + "min": 77, + "p50": 254, + "p90": 282, + "p95": 282, + "p99": 282, + "max": 282 + }, + "durationMs": { + "min": 0, + "p50": 1, + "p90": 8, + "p95": 8, + "p99": 8, + "max": 8 + } + } + }, + "broker": { + "sentMessages": 8, + "outputMessages": 5, + "replayOutputMessages": 1, + "outputSerializedBytes": { + "min": 159, + "p50": 412, + "p90": 841, + "p95": 841, + "p99": 841, + "max": 841 + }, + "maxSerializedBytes": 841, + "maxRawDataBytesInOutputMessage": 613, + "sequenceRanges": [ + { + "attachRequestId": "codex-version-attach", + "seqStart": 1, + "seqEnd": 1, + "dataBytes": 77 + }, + { + "attachRequestId": "codex-version-attach", + "seqStart": 2, + "seqEnd": 3, + "dataBytes": 282 + }, + { + "attachRequestId": "codex-version-attach", + "seqStart": 4, + "seqEnd": 4, + "dataBytes": 19 + }, + { + "attachRequestId": "codex-version-attach", + "seqStart": 5, + "seqEnd": 6, + "dataBytes": 235 + }, + { + "attachRequestId": "codex-version-replay", + "seqStart": 1, + "seqEnd": 6, + "dataBytes": 613 + } + ] + }, + "scanner": { + "scannerFrameCount": 6, + "conservativeBatchCount": 3, + "barrierFrameCount": 2, + "sideEffectBarrierFrameCount": 2, + "replacementFrameCount": 0, + "pendingEndStateCount": 0, + "finalState": "ground" + }, + "ratios": { + "rawChunksPerLiveOutputMessage": 1.2, + "rawChunksPerReplayOutputMessage": 6, + "rawChunksPerConservativeScannerBatch": 2 + } + }, + { + "scenario": "codex-help", + "command": "codex exec --help; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-help' '1'", + "terminalId": "cptMLKlrWJZwJt4Zr1IVL", + "durationMs": 667, + "raw": { + "chunks": 7, + "bytes": 4686, + "bytesPerSecond": 7025, + "previewStartJsonEscaped": "codex exec --help; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-help' '1'\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ codex exec --help; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-help' '1'\r\n\u001b[?2004l\rRun Codex non-interactively\r\n\r\n\u001b[1m\u001b[4mUsage:\u001b[0m codex exec [OPTIONS] [PROMPT]\r\n codex exec [OPTIONS] [ARGS]\r\n\r\n\u001b[1m\u001b[4mCommands:\u001b[0m\r\n \u001b[1mresume\u001b[0m Resume a previous session by id or pick the most recent with --last\r\n \u001b[1mreview\u001b[0m Run a code review against the current repository\r\n \u001b[1mhelp\u001b[0m Print this message or the help of the given subcommand(s)\r\n\r\n\u001b[1m\u001b[4mArguments:\u001b[0m\r\n [PROMPT]\r\n Initial instructions for the agent. If not provided as an argument (or if `-` is used), instructions are read\r\n from stdin. If stdin is piped and a prompt is also provided, stdin is appended as a `` block\r\n\r\n\u001b[1m\u001b[4mOptions:\u001b[0m\r\n \u001b[1m-c\u001b[0m, \u001b[1m--config\u001b[0m \r\n Override a configuration value that would otherwise be loaded from `~/.codex/config.toml`. Use a dotted path\r\n (`foo.bar.baz`) to override nested values. The `value` portion is parsed as TOML. If it fails to parse as\r\n TOML, the raw string is used as a literal.\r\n \r\n Examples: - `-c model=\"o3\"` - `-c 'sandbox_permissions=[\"disk-full-read-access\"]'` - `-c\r\n shell_environment_policy.inherit=all`\r\n\r\n \u001b[1m--enable\u001b[0m \r\n Enable a feature (repea", + "previewEndJsonEscaped": "vocation. DANGEROUS. Intended only for\r\n automation that already vets hook sources\r\n\r\n \u001b[1m-C\u001b[0m, \u001b[1m--cd\u001b[0m \r\n Tell the agent to use the specified directory as its working root\r\n\r\n \u001b[1m--add-dir\u001b[0m \r\n Additional directories that should be writable alongside the primary workspace\r\n\r\n \u001b[1m--skip-git-repo-check\u001b[0m\r\n Allow running Codex outside a Git repository\r\n\r\n \u001b[1m--ephemeral\u001b[0m\r\n Run without persisting session files to disk\r\n\r\n \u001b[1m--ignore-user-config\u001b[0m\r\n Do not load `$CODEX_HOME/config.toml`; auth still uses `CODEX_HOME`\r\n\r\n \u001b[1m--ignore-rules\u001b[0m\r\n Do not load user or project execpolicy `.rules` files\r\n\r\n \u001b[1m--output-schema\u001b[0m \r\n Path to a JSON Schema file describing the model's final response shape\r\n\r\n \u001b[1m--color\u001b[0m \r\n Specifies color settings for use in the output\r\n \r\n [default: auto]\r\n [possible values: always, never, auto]\r\n\r\n \u001b[1m--json\u001b[0m\r\n Print events to stdout as JSONL\r\n\r\n \u001b[1m-o\u001b[0m, \u001b[1m--output-last-message\u001b[0m \r\n Specifies file where the last message from the agent should be written\r\n\r\n \u001b[1m-h\u001b[0m, \u001b[1m--help\u001b[0m\r\n Print help (see a summary with '-h')\r\n\r\n \u001b[1m-V\u001b[0m, \u001b[1m--version\u001b[0m\r\n Print version\r\n\r\nFRESHELL_PROOF_DONE:codex-help:1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "chunkBytes": { + "min": 36, + "p50": 196, + "p90": 3823, + "p95": 3823, + "p99": 3823, + "max": 3823 + }, + "interarrivalMs": { + "min": 0, + "p50": 0, + "p90": 361, + "p95": 361, + "p99": 361, + "max": 361 + }, + "burstGroupsUnder10ms": { + "count": 3, + "chunks": { + "min": 1, + "p50": 2, + "p90": 4, + "p95": 4, + "p99": 4, + "max": 4 + }, + "bytes": { + "min": 76, + "p50": 281, + "p90": 4329, + "p95": 4329, + "p99": 4329, + "max": 4329 + }, + "durationMs": { + "min": 0, + "p50": 0, + "p90": 8, + "p95": 8, + "p99": 8, + "max": 8 + } + } + }, + "broker": { + "sentMessages": 8, + "outputMessages": 5, + "replayOutputMessages": 1, + "outputSerializedBytes": { + "min": 215, + "p50": 461, + "p90": 5533, + "p95": 5533, + "p99": 5533, + "max": 5533 + }, + "maxSerializedBytes": 5533, + "maxRawDataBytesInOutputMessage": 4686, + "sequenceRanges": [ + { + "attachRequestId": "codex-help-attach", + "seqStart": 1, + "seqEnd": 1, + "dataBytes": 76 + }, + { + "attachRequestId": "codex-help-attach", + "seqStart": 2, + "seqEnd": 3, + "dataBytes": 281 + }, + { + "attachRequestId": "codex-help-attach", + "seqStart": 4, + "seqEnd": 5, + "dataBytes": 4097 + }, + { + "attachRequestId": "codex-help-attach", + "seqStart": 6, + "seqEnd": 7, + "dataBytes": 232 + }, + { + "attachRequestId": "codex-help-replay", + "seqStart": 1, + "seqEnd": 7, + "dataBytes": 4686 + } + ] + }, + "scanner": { + "scannerFrameCount": 7, + "conservativeBatchCount": 3, + "barrierFrameCount": 2, + "sideEffectBarrierFrameCount": 2, + "replacementFrameCount": 0, + "pendingEndStateCount": 0, + "finalState": "ground" + }, + "ratios": { + "rawChunksPerLiveOutputMessage": 1.4, + "rawChunksPerReplayOutputMessage": 7, + "rawChunksPerConservativeScannerBatch": 2.333 + } + }, + { + "scenario": "codex-real-turn", + "command": "codex exec --ephemeral --color always --sandbox read-only --cd /tmp --skip-git-repo-check \"Print exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\"; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-real-turn' '40'", + "terminalId": "UyLO2yOWLvGow2BLbne9Q", + "durationMs": 16727, + "raw": { + "chunks": 15, + "bytes": 2000, + "bytesPerSecond": 120, + "previewStartJsonEscaped": "codex exec --ephemeral --color always --sandbox read-only --cd /tmp --skip-git-repo-check \"Print exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\"; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-real-turn' '40'\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ codex exec --ephemeral --color always --sandbox read-only --cd /tmp --skip-git-repo-check \"Print exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\"; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-real-turn' '40'\r\n\u001b[?2004l\rOpenAI Codex v0.137.0\r\n--------\r\n\u001b[1mworkdir:\u001b[0m /tmp\r\n\u001b[1mmodel:\u001b[0m gpt-5.5\r\n\u001b[1mprovider:\u001b[0m openai\r\n\u001b[1mapproval:\u001b[0m never\r\n\u001b[1msandbox:\u001b[0m read-only\r\n\u001b[1mreasoning effort:\u001b[0m xhigh\r\n\u001b[1mreasoning summaries:\u001b[0m none\r\n\u001b[1msession id:\u001b[0m 019ea9e1-720f-7aa2-b6f9-1f624ab4403f\r\n--------\r\n\u001b[36muser\u001b[0m\r\nPrint exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\r\n\u001b[35m\u001b[3mcodex\u001b[0m\u001b[0m\r\nproof-line-1\r\nproof-line-2\r\nproof-line-3\r\nproof-line-4\r\nproof-line-5\r\nproof-line-6\r\nproof-line-7\r\nproof-line-8\r\nproof-line-9\r\nproof-line-10\r\nproof-line-11\r\nproof-line-12\r\nproof-line-13\r\nproof-line-14\r\nproof-line-15\r\nproof-line-16\r\nproof-line-17\r\nproof-line-18\r\nproof-line-19\r\nproof-line-20\r\nproof-line-21\r\nproof-line-22\r\nproof-line-23\r\nproof-line-24\r\nproof-line-25\r\nproof-line-26\r\nproof-line-27\r\nproof-line-28\r\nproof-line-29\r\nproof-line-30\r\nproof-line-31\r\np", + "previewEndJsonEscaped": "ktrees/proof-terminal-catchup-architecture\u001b[00m$ codex exec --ephemeral --color always --sandbox read-only --cd /tmp --skip-git-repo-check \"Print exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\"; printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' 'codex-real-turn' '40'\r\n\u001b[?2004l\rOpenAI Codex v0.137.0\r\n--------\r\n\u001b[1mworkdir:\u001b[0m /tmp\r\n\u001b[1mmodel:\u001b[0m gpt-5.5\r\n\u001b[1mprovider:\u001b[0m openai\r\n\u001b[1mapproval:\u001b[0m never\r\n\u001b[1msandbox:\u001b[0m read-only\r\n\u001b[1mreasoning effort:\u001b[0m xhigh\r\n\u001b[1mreasoning summaries:\u001b[0m none\r\n\u001b[1msession id:\u001b[0m 019ea9e1-720f-7aa2-b6f9-1f624ab4403f\r\n--------\r\n\u001b[36muser\u001b[0m\r\nPrint exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands.\r\n\u001b[35m\u001b[3mcodex\u001b[0m\u001b[0m\r\nproof-line-1\r\nproof-line-2\r\nproof-line-3\r\nproof-line-4\r\nproof-line-5\r\nproof-line-6\r\nproof-line-7\r\nproof-line-8\r\nproof-line-9\r\nproof-line-10\r\nproof-line-11\r\nproof-line-12\r\nproof-line-13\r\nproof-line-14\r\nproof-line-15\r\nproof-line-16\r\nproof-line-17\r\nproof-line-18\r\nproof-line-19\r\nproof-line-20\r\nproof-line-21\r\nproof-line-22\r\nproof-line-23\r\nproof-line-24\r\nproof-line-25\r\nproof-line-26\r\nproof-line-27\r\nproof-line-28\r\nproof-line-29\r\nproof-line-30\r\nproof-line-31\r\nproof-line-32\r\nproof-line-33\r\nproof-line-34\r\nproof-line-35\r\nproof-line-36\r\nproof-line-37\r\nproof-line-38\r\nproof-line-39\r\nproof-line-40\r\n\u001b[2mtokens used\u001b[0m\r\n4,139\r\n\r\nFRESHELL_PROOF_DONE:codex-real-turn:40\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "chunkBytes": { + "min": 2, + "p50": 52, + "p90": 262, + "p95": 615, + "p99": 615, + "max": 615 + }, + "interarrivalMs": { + "min": 0, + "p50": 0, + "p90": 856, + "p95": 14978, + "p99": 14978, + "max": 14978 + }, + "burstGroupsUnder10ms": { + "count": 6, + "chunks": { + "min": 1, + "p50": 1, + "p90": 8, + "p95": 8, + "p99": 8, + "max": 8 + }, + "bytes": { + "min": 28, + "p50": 253, + "p90": 615, + "p95": 615, + "p99": 615, + "max": 615 + }, + "durationMs": { + "min": 0, + "p50": 0, + "p90": 1, + "p95": 1, + "p99": 1, + "max": 1 + } + } + }, + "broker": { + "sentMessages": 10, + "outputMessages": 7, + "replayOutputMessages": 1, + "outputSerializedBytes": { + "min": 184, + "p50": 645, + "p90": 2465, + "p95": 2465, + "p99": 2465, + "max": 2465 + }, + "maxSerializedBytes": 2465, + "maxRawDataBytesInOutputMessage": 2000, + "sequenceRanges": [ + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 1, + "seqEnd": 1, + "dataBytes": 253 + }, + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 2, + "seqEnd": 3, + "dataBytes": 458 + }, + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 4, + "seqEnd": 11, + "dataBytes": 408 + }, + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 12, + "seqEnd": 12, + "dataBytes": 615 + }, + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 13, + "seqEnd": 13, + "dataBytes": 28 + }, + { + "attachRequestId": "codex-real-turn-attach", + "seqStart": 14, + "seqEnd": 15, + "dataBytes": 238 + }, + { + "attachRequestId": "codex-real-turn-replay", + "seqStart": 1, + "seqEnd": 15, + "dataBytes": 2000 + } + ] + }, + "scanner": { + "scannerFrameCount": 15, + "conservativeBatchCount": 8, + "barrierFrameCount": 7, + "sideEffectBarrierFrameCount": 2, + "replacementFrameCount": 0, + "pendingEndStateCount": 3, + "finalState": "ground" + }, + "ratios": { + "rawChunksPerLiveOutputMessage": 2.143, + "rawChunksPerReplayOutputMessage": 15, + "rawChunksPerConservativeScannerBatch": 1.875 + } + }, + { + "scenario": "agent-burst-12000", + "command": "node scripts/proofs/terminal-catchup-agent-output-generator.mjs agent-burst 12000", + "terminalId": "hXOtvh0cTEEHXpaJK_vf2", + "durationMs": 1248, + "raw": { + "chunks": 3239, + "bytes": 776745, + "bytesPerSecond": 622392, + "previewStartJsonEscaped": "node scripts/proofs/terminal-catchup-agent-output-generator.mjs agent-burst 12000\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ node scripts/proofs/terminal-catchup-agent-output-generator.mjs agent-burst 12000\r\n\u001b[?2004l\r\u001b]0;Freshell proof agent burst\u0007\u001b[?25l\u001b[38;5;39mthinking\u001b[0m 00001 scanning repository\r\u001b[2K\r[\u001b[32mok\u001b[0m] 00001 edit src/example-1.ts chunk=1 tokens=1001\r\n[\u001b[32mok\u001b[0m] 00002 edit src/example-2.ts chunk=2 tokens=1002\r\n[\u001b[32mok\u001b[0m] 00003 edit src/example-3.ts chunk=3 tokens=1003\r\n[\u001b[32mok\u001b[0m] 00004 edit src/example-4.ts chunk=4 tokens=1004\r\n[\u001b[32mok\u001b[0m] 00005 edit src/example-5.ts chunk=5 tokens=1005\r\n[\u001b[32mok\u001b[0m] 00006 edit src/example-6.ts chunk=6 tokens=1006\r\n[\u001b[32mok\u001b[0m] 00007 edit src/example-7.ts chunk=7 tokens=1007\r\n[\u001b[32mok\u001b[0m] 00008 edit src/example-8.ts chunk=8 tokens=1008\r\n[\u001b[32mok\u001b[0m] 00009 edit src/example-9.ts chunk=9 tokens=1009\r\n[\u001b[32mok\u001b[0m] 00010 edit src/example-10.ts chunk=10 tokens=1010\r\n[\u001b[32mok\u001b[0m] 00011 edit src/example-11.ts chunk=0 tokens=1011\r\n[\u001b[32mok\u001b[0m] 00012 edit src/example-12.ts chunk=1 tokens=1012\r\n[\u001b[32mok\u001b[0m] 00013 edit src/example-13.ts chunk=2 tokens=1013\r\n[\u001b[32mok\u001b[0m] 00014 edit src/example-14.ts chunk=3 tokens=1014\r\n[\u001b[32mok\u001b[0m] 00015 edit src/example-15.ts chunk=4 tokens=1015\r\n[\u001b[32mok\u001b[0m] 00016 edit src/example-16.ts chunk=5 tokens=1016\r\n[\u001b[33mwarn\u001b[0m] 00017 edit src/example-17.ts chunk=6 tokens=1017\r\n[\u001b[32mok\u001b[0m] 00018 edit src/example-18.ts chunk=7 token", + "previewEndJsonEscaped": "[\u001b[32mok\u001b[0m] 11980 edit src/example-20.ts chunk=1 tokens=12980\r\n[\u001b[32mok\u001b[0m] 11981 edit src/example-21.ts chunk=2 tokens=12981\r\n[\u001b[32mok\u001b[0m] 11982 edit src/example-22.ts chunk=3 tokens=12982\r\n[\u001b[32mok\u001b[0m] 11983 edit src/example-0.ts chunk=4 tokens=12983\r\n[\u001b[32mok\u001b[0m] 11984 edit src/example-1.ts chunk=5 tokens=12984\r\n[\u001b[33mwarn\u001b[0m] 11985 edit src/example-2.ts chunk=6 tokens=12985\r\n[\u001b[32mok\u001b[0m] 11986 edit src/example-3.ts chunk=7 tokens=12986\r\n[\u001b[32mok\u001b[0m] 11987 edit src/example-4.ts chunk=8 tokens=12987\r\n[\u001b[32mok\u001b[0m] 11988 edit src/example-5.ts chunk=9 tokens=12988\r\n[\u001b[32mok\u001b[0m] 11989 edit src/example-6.ts chunk=10 tokens=12989\r\n[\u001b[32mok\u001b[0m] 11990 edit src/example-7.ts chunk=0 tokens=12990\r\n[\u001b[32mok\u001b[0m] 11991 edit src/example-8.ts chunk=1 tokens=12991\r\n[\u001b[32mok\u001b[0m] 11992 edit src/example-9.ts chunk=2 tokens=12992\r\n[\u001b[32mok\u001b[0m] 11993 edit src/example-10.ts chunk=3 tokens=12993\r\n[\u001b[32mok\u001b[0m] 11994 edit src/example-11.ts chunk=4 tokens=12994\r\n[\u001b[32mok\u001b[0m] 11995 edit src/example-12.ts chunk=5 tokens=12995\r\n[\u001b[32mok\u001b[0m] 11996 edit src/example-13.ts chunk=6 tokens=12996\r\n[\u001b[32mok\u001b[0m] 11997 edit src/example-14.ts chunk=7 tokens=12997\r\n[\u001b[32mok\u001b[0m] 11998 edit src/example-15.ts chunk=8 tokens=12998\r\n[\u001b[32mok\u001b[0m] 11999 edit src/example-16.ts chunk=9 tokens=12999\r\n[\u001b[32mok\u001b[0m] 12000 edit src/example-17.ts chunk=10 tokens=13000\r\n\u001b[?25hFRESHELL_PROOF_DONE:agent-burst:12000\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "chunkBytes": { + "min": 2, + "p50": 130, + "p90": 513, + "p95": 766, + "p99": 1403, + "max": 6999 + }, + "interarrivalMs": { + "min": 0, + "p50": 0, + "p90": 0, + "p95": 1, + "p99": 4, + "max": 327 + }, + "burstGroupsUnder10ms": { + "count": 3, + "chunks": { + "min": 1, + "p50": 2, + "p90": 3236, + "p95": 3236, + "p99": 3236, + "max": 3236 + }, + "bytes": { + "min": 83, + "p50": 288, + "p90": 776374, + "p95": 776374, + "p99": 776374, + "max": 776374 + }, + "durationMs": { + "min": 0, + "p50": 0, + "p90": 645, + "p95": 645, + "p99": 645, + "max": 645 + } + } + }, + "broker": { + "sentMessages": 294, + "outputMessages": 290, + "replayOutputMessages": 33, + "outputSerializedBytes": { + "min": 208, + "p50": 3784, + "p90": 18366, + "p95": 19430, + "p99": 19614, + "max": 19615 + }, + "maxSerializedBytes": 19615, + "maxRawDataBytesInOutputMessage": 16384, + "sequenceRanges": [ + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 1, + "seqEnd": 1, + "dataBytes": 83 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 2, + "seqEnd": 3, + "dataBytes": 288 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 4, + "seqEnd": 5, + "dataBytes": 86 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 6, + "seqEnd": 26, + "dataBytes": 1912 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 27, + "seqEnd": 55, + "dataBytes": 2430 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 56, + "seqEnd": 73, + "dataBytes": 1995 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 74, + "seqEnd": 75, + "dataBytes": 1850 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 76, + "seqEnd": 85, + "dataBytes": 1338 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 86, + "seqEnd": 101, + "dataBytes": 3332 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 102, + "seqEnd": 117, + "dataBytes": 3601 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 118, + "seqEnd": 139, + "dataBytes": 3789 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 140, + "seqEnd": 162, + "dataBytes": 3634 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 163, + "seqEnd": 168, + "dataBytes": 767 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 169, + "seqEnd": 175, + "dataBytes": 1094 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 176, + "seqEnd": 192, + "dataBytes": 4027 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 193, + "seqEnd": 205, + "dataBytes": 3174 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 206, + "seqEnd": 213, + "dataBytes": 1088 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 214, + "seqEnd": 237, + "dataBytes": 2932 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 238, + "seqEnd": 250, + "dataBytes": 1671 + }, + { + "attachRequestId": "agent-burst-12000-attach", + "seqStart": 251, + "seqEnd": 257, + "dataBytes": 575 + } + ] + }, + "scanner": { + "scannerFrameCount": 3239, + "conservativeBatchCount": 170, + "barrierFrameCount": 169, + "sideEffectBarrierFrameCount": 169, + "replacementFrameCount": 0, + "pendingEndStateCount": 0, + "finalState": "ground" + }, + "ratios": { + "rawChunksPerLiveOutputMessage": 11.169, + "rawChunksPerReplayOutputMessage": 98.152, + "rawChunksPerConservativeScannerBatch": 19.053 + } + }, + { + "scenario": "control-barrier", + "command": "node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1", + "terminalId": "iVqY191Z6FYkF3dMqytuz", + "durationMs": 681, + "raw": { + "chunks": 10, + "bytes": 775, + "bytesPerSecond": 1138, + "previewStartJsonEscaped": "node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1\r\n\u001b[?2004l\r\u001b]0;Freshell proof split OSC\u0007after-title\r\n\u001b[38;5;196msplit-sgr-red\u001b[0m\r\n\u001bPqdcs-payload\u001b\\after-dcs\r\n\u001b[?2026;1$y\u001b]52;c;cHJvb2YgY2xpcGJvYXJk\u0007� replacement-byte-sentinel\r\nFRESHELL_PROOF_DONE:control-barrier:1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "previewEndJsonEscaped": "node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1\r\n\u001b[?2004l\r\u001b]0;Freshell proof split OSC\u0007after-title\r\n\u001b[38;5;196msplit-sgr-red\u001b[0m\r\n\u001bPqdcs-payload\u001b\\after-dcs\r\n\u001b[?2026;1$y\u001b]52;c;cHJvb2YgY2xpcGJvYXJk\u0007� replacement-byte-sentinel\r\nFRESHELL_PROOF_DONE:control-barrier:1\r\n\u001b[?2004h\u001b]0;dan@DANDESKTOP: ~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u0007\u001b[01;32mdan@DANDESKTOP\u001b[00m:\u001b[01;34m~/code/freshell/.worktrees/proof-terminal-catchup-architecture\u001b[00m$ ", + "chunkBytes": { + "min": 22, + "p50": 39, + "p90": 196, + "p95": 196, + "p99": 196, + "max": 196 + }, + "interarrivalMs": { + "min": 0, + "p50": 5, + "p90": 381, + "p95": 381, + "p99": 381, + "max": 381 + }, + "burstGroupsUnder10ms": { + "count": 3, + "chunks": { + "min": 1, + "p50": 2, + "p90": 7, + "p95": 7, + "p99": 7, + "max": 7 + }, + "bytes": { + "min": 83, + "p50": 288, + "p90": 404, + "p95": 404, + "p99": 404, + "max": 404 + }, + "durationMs": { + "min": 0, + "p50": 0, + "p90": 21, + "p95": 21, + "p99": 21, + "max": 21 + } + } + }, + "broker": { + "sentMessages": 11, + "outputMessages": 8, + "replayOutputMessages": 1, + "outputSerializedBytes": { + "min": 173, + "p50": 225, + "p90": 1052, + "p95": 1052, + "p99": 1052, + "max": 1052 + }, + "maxSerializedBytes": 1052, + "maxRawDataBytesInOutputMessage": 775, + "sequenceRanges": [ + { + "attachRequestId": "control-barrier-attach", + "seqStart": 1, + "seqEnd": 1, + "dataBytes": 83 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 2, + "seqEnd": 3, + "dataBytes": 288 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 4, + "seqEnd": 4, + "dataBytes": 28 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 5, + "seqEnd": 5, + "dataBytes": 25 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 6, + "seqEnd": 6, + "dataBytes": 22 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 7, + "seqEnd": 9, + "dataBytes": 133 + }, + { + "attachRequestId": "control-barrier-attach", + "seqStart": 10, + "seqEnd": 10, + "dataBytes": 196 + }, + { + "attachRequestId": "control-barrier-replay", + "seqStart": 1, + "seqEnd": 10, + "dataBytes": 775 + } + ] + }, + "scanner": { + "scannerFrameCount": 10, + "conservativeBatchCount": 8, + "barrierFrameCount": 7, + "sideEffectBarrierFrameCount": 6, + "replacementFrameCount": 1, + "pendingEndStateCount": 2, + "finalState": "ground" + }, + "ratios": { + "rawChunksPerLiveOutputMessage": 1.25, + "rawChunksPerReplayOutputMessage": 10, + "rawChunksPerConservativeScannerBatch": 1.25 + } + } + ] +} diff --git a/docs/superpowers/proofs/artifacts/terminal-json-serialization.json b/docs/superpowers/proofs/artifacts/terminal-json-serialization.json new file mode 100644 index 000000000..8468f57b0 --- /dev/null +++ b/docs/superpowers/proofs/artifacts/terminal-json-serialization.json @@ -0,0 +1,48 @@ +{ + "generatedAt": "2026-06-09T01:06:22.197Z", + "targetRawBytes": 16384, + "samples": [ + { + "name": "plain-ascii", + "rawBytes": 16384, + "serializedBytes": 16503, + "jsonOverheadBytes": 119, + "expansionRatio": 1.007 + }, + { + "name": "ansi-sgr-repeat", + "rawBytes": 16384, + "serializedBytes": 30158, + "jsonOverheadBytes": 13774, + "expansionRatio": 1.841 + }, + { + "name": "esc-only-control", + "rawBytes": 16384, + "serializedBytes": 98423, + "jsonOverheadBytes": 82039, + "expansionRatio": 6.007 + }, + { + "name": "newline-heavy", + "rawBytes": 16384, + "serializedBytes": 24695, + "jsonOverheadBytes": 8311, + "expansionRatio": 1.507 + }, + { + "name": "quote-heavy", + "rawBytes": 16384, + "serializedBytes": 32887, + "jsonOverheadBytes": 16503, + "expansionRatio": 2.007 + }, + { + "name": "backslash-heavy", + "rawBytes": 16384, + "serializedBytes": 32887, + "jsonOverheadBytes": 16503, + "expansionRatio": 2.007 + } + ] +} diff --git a/docs/superpowers/proofs/artifacts/xterm-write-dispose.json b/docs/superpowers/proofs/artifacts/xterm-write-dispose.json new file mode 100644 index 000000000..b0ab4b109 --- /dev/null +++ b/docs/superpowers/proofs/artifacts/xterm-write-dispose.json @@ -0,0 +1,178 @@ +{ + "generatedAt": "2026-06-09T01:05:38.465Z", + "nodeVersion": "v22.21.1", + "dependencies": { + "@xterm/xterm": "6.0.0" + }, + "smallWriteOrder": [ + { + "event": "after-write-alpha-returned", + "at": 0.155, + "cursorX": 0, + "cursorY": 0 + }, + { + "event": "after-write-beta-returned", + "at": 0.293, + "cursorX": 0, + "cursorY": 0 + }, + { + "event": "callback-alpha", + "at": 2.479, + "cursorX": 5, + "cursorY": 0 + }, + { + "event": "callback-beta", + "at": 2.775, + "cursorX": 9, + "cursorY": 1 + }, + { + "event": "after-50ms", + "at": 50.123, + "cursorX": 9, + "cursorY": 1 + } + ], + "disposeAfterLargeWrite": { + "events": [ + { + "event": "after-large-write-returned", + "at": 0.129, + "cursorX": 0, + "cursorY": 0 + }, + { + "event": "after-dispose", + "at": 0.225, + "cursorX": 0, + "cursorY": 0 + }, + { + "event": "large-write-callback", + "at": 310.032, + "cursorX": 80, + "cursorY": 29 + }, + { + "event": "after-250ms", + "at": 310.193, + "cursorX": 80, + "cursorY": 29 + } + ], + "callbackAfterDispose": true + }, + "disposeAfterQueuedWrites": { + "disposeAt": 0.382, + "callbackCount": 500, + "callbacksAfterDispose": 500, + "firstCallbacks": [ + { + "index": 0, + "at": 1.738, + "afterDispose": true + }, + { + "index": 1, + "at": 1.757, + "afterDispose": true + }, + { + "index": 2, + "at": 1.763, + "afterDispose": true + }, + { + "index": 3, + "at": 1.77, + "afterDispose": true + }, + { + "index": 4, + "at": 1.774, + "afterDispose": true + }, + { + "index": 5, + "at": 1.778, + "afterDispose": true + }, + { + "index": 6, + "at": 1.789, + "afterDispose": true + }, + { + "index": 7, + "at": 1.793, + "afterDispose": true + }, + { + "index": 8, + "at": 1.796, + "afterDispose": true + }, + { + "index": 9, + "at": 1.802, + "afterDispose": true + } + ], + "lastCallbacks": [ + { + "index": 490, + "at": 3.19, + "afterDispose": true + }, + { + "index": 491, + "at": 3.201, + "afterDispose": true + }, + { + "index": 492, + "at": 3.205, + "afterDispose": true + }, + { + "index": 493, + "at": 3.209, + "afterDispose": true + }, + { + "index": 494, + "at": 3.212, + "afterDispose": true + }, + { + "index": 495, + "at": 3.213, + "afterDispose": true + }, + { + "index": 496, + "at": 3.215, + "afterDispose": true + }, + { + "index": 497, + "at": 3.22, + "afterDispose": true + }, + { + "index": 498, + "at": 3.222, + "afterDispose": true + }, + { + "index": 499, + "at": 3.224, + "afterDispose": true + } + ], + "fifo": true + } +} diff --git a/package-lock.json b/package-lock.json index 475f77427..a82b904cc 100644 --- a/package-lock.json +++ b/package-lock.json @@ -20,7 +20,7 @@ "@xterm/addon-fit": "^0.11.0", "@xterm/addon-search": "^0.16.0", "@xterm/addon-webgl": "^0.19.0", - "@xterm/xterm": "^6.0.0", + "@xterm/xterm": "6.0.0", "ai": "^6.0.86", "chokidar": "^3.6.0", "cookie-parser": "^1.4.7", diff --git a/package.json b/package.json index 989265fe0..9129bd2f8 100644 --- a/package.json +++ b/package.json @@ -89,7 +89,7 @@ "@xterm/addon-fit": "^0.11.0", "@xterm/addon-search": "^0.16.0", "@xterm/addon-webgl": "^0.19.0", - "@xterm/xterm": "^6.0.0", + "@xterm/xterm": "6.0.0", "ai": "^6.0.86", "chokidar": "^3.6.0", "cookie-parser": "^1.4.7", diff --git a/scripts/assert-visible-first-audit-gate.ts b/scripts/assert-visible-first-audit-gate.ts index f28808266..f6f0f6505 100644 --- a/scripts/assert-visible-first-audit-gate.ts +++ b/scripts/assert-visible-first-audit-gate.ts @@ -27,8 +27,9 @@ async function main(): Promise { if (!result.ok) { process.exitCode = 1 } - } catch { - writeResult({ ok: false, violations: [] }) + } catch (error) { + const message = error instanceof Error ? error.message : String(error) + writeResult({ ok: false, validationErrors: [message], violations: [] }) process.exitCode = 1 } } diff --git a/scripts/proofs/browser-background-visibility-probe.ts b/scripts/proofs/browser-background-visibility-probe.ts new file mode 100644 index 000000000..a3fd9b759 --- /dev/null +++ b/scripts/proofs/browser-background-visibility-probe.ts @@ -0,0 +1,176 @@ +import fs from 'fs/promises' +import http from 'http' +import path from 'path' +import { fileURLToPath } from 'url' +import { chromium } from '@playwright/test' +import { WebSocketServer } from 'ws' + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +type Snapshot = { + at: number + visibilityState: string + hidden: boolean + intervalTicks: number + rafTicks: number + wsReceived: number + lastIntervalAt: number + lastRafAt: number + lastWsAt: number +} + +function delta(a: Snapshot, b: Snapshot) { + return { + elapsedMs: b.at - a.at, + visibilityStateBefore: a.visibilityState, + visibilityStateAfter: b.visibilityState, + intervalTicks: b.intervalTicks - a.intervalTicks, + rafTicks: b.rafTicks - a.rafTicks, + wsReceived: b.wsReceived - a.wsReceived, + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + let serverSent = 0 + + const httpServer = http.createServer((_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain' }) + res.end('ok') + }) + const wss = new WebSocketServer({ server: httpServer }) + const sendTimer = setInterval(() => { + serverSent += 1 + const payload = JSON.stringify({ n: serverSent, at: Date.now() }) + for (const client of wss.clients) { + if (client.readyState === client.OPEN) client.send(payload) + } + }, 50) + + const port = await new Promise((resolve) => { + httpServer.listen(0, '127.0.0.1', () => { + const address = httpServer.address() + if (!address || typeof address === 'string') throw new Error('No listen port') + resolve(address.port) + }) + }) + + const headed = process.env.FRESHELL_PROOF_HEADED_BROWSER === '1' + const browser = await chromium.launch({ headless: !headed }) + const context = await browser.newContext() + const page = await context.newPage() + const coverPage = await context.newPage() + const wsUrl = `ws://127.0.0.1:${port}` + const html = ` + + + + +` + + try { + await page.goto(`data:text/html,${encodeURIComponent(html)}`) + await coverPage.goto('data:text/html,covercover') + await page.bringToFront() + await page.evaluate(() => (window as any).__wsReady) + + await sleep(1000) + const activeA = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentActiveA = serverSent + await sleep(1000) + const activeB = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentActiveB = serverSent + + await coverPage.bringToFront() + await sleep(250) + const hiddenA = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentHiddenA = serverSent + await sleep(3000) + const hiddenB = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentHiddenB = serverSent + + await page.bringToFront() + await sleep(1000) + const resumed = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentResumed = serverSent + + const artifact = { + generatedAt: new Date().toISOString(), + chromiumVersion: browser.version(), + headed, + wsSendIntervalMs: 50, + active: { + before: activeA, + after: activeB, + pageDelta: delta(activeA, activeB), + serverSentDelta: serverSentActiveB - serverSentActiveA, + }, + backgrounded: { + before: hiddenA, + after: hiddenB, + pageDelta: delta(hiddenA, hiddenB), + serverSentDelta: serverSentHiddenB - serverSentHiddenA, + }, + resumed: { + after: resumed, + pageDeltaFromHiddenEnd: delta(hiddenB, resumed), + serverSentDelta: serverSentResumed - serverSentHiddenB, + }, + } + + const outPath = path.resolve(artifactDir, 'browser-background-visibility.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath }, null, 2)) + } finally { + await browser.close().catch(() => undefined) + clearInterval(sendTimer) + await new Promise((resolve) => wss.close(() => resolve())) + await new Promise((resolve) => httpServer.close(() => resolve())) + } +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/scripts/proofs/browser-freeze-lifecycle-probe.ts b/scripts/proofs/browser-freeze-lifecycle-probe.ts new file mode 100644 index 000000000..e0f40d7ea --- /dev/null +++ b/scripts/proofs/browser-freeze-lifecycle-probe.ts @@ -0,0 +1,198 @@ +import fs from 'fs/promises' +import http from 'http' +import path from 'path' +import { fileURLToPath } from 'url' +import { chromium } from '@playwright/test' +import { WebSocketServer } from 'ws' + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +type Snapshot = { + at: number + visibilityState: string + intervalTicks: number + rafTicks: number + wsReceived: number + lastIntervalAt: number + lastRafAt: number + lastWsAt: number +} + +function delta(a: Snapshot, b: Snapshot) { + return { + elapsedMs: b.at - a.at, + intervalTicks: b.intervalTicks - a.intervalTicks, + rafTicks: b.rafTicks - a.rafTicks, + wsReceived: b.wsReceived - a.wsReceived, + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + + let serverSent = 0 + const httpServer = http.createServer((_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain' }) + res.end('ok') + }) + const wss = new WebSocketServer({ server: httpServer }) + const sendTimer = setInterval(() => { + serverSent += 1 + const payload = JSON.stringify({ n: serverSent, at: Date.now() }) + for (const client of wss.clients) { + if (client.readyState === client.OPEN) client.send(payload) + } + }, 50) + + const port = await new Promise((resolve) => { + httpServer.listen(0, '127.0.0.1', () => { + const address = httpServer.address() + if (!address || typeof address === 'string') throw new Error('No listen port') + resolve(address.port) + }) + }) + + const headed = process.env.FRESHELL_PROOF_HEADED_BROWSER === '1' + const browser = await chromium.launch({ headless: !headed }) + const page = await browser.newPage() + const cdp = await page.context().newCDPSession(page) + const wsUrl = `ws://127.0.0.1:${port}` + const html = ` + + + + +` + + try { + await page.goto(`data:text/html,${encodeURIComponent(html)}`) + await page.evaluate(() => (window as any).__wsReady) + await sleep(1000) + const activeA = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentAtActiveA = serverSent + await sleep(1000) + const activeB = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentAtActiveB = serverSent + + await cdp.send('Page.setWebLifecycleState', { state: 'frozen' }) + const freezeStart = Date.now() + const frozenA = activeB + const serverSentAtFreezeStart = serverSent + await sleep(2000) + const serverSentAtFreezeEnd = serverSent + + let frozenEvalStatus: 'returned' | 'timed_out' | 'error' = 'timed_out' + let frozenEvalSnapshot: Snapshot | null = null + let frozenEvalError: string | undefined + try { + const evaluated = await Promise.race([ + cdp.send('Runtime.evaluate', { + expression: 'window.__snapshot()', + returnByValue: true, + }), + sleep(1000).then(() => null), + ]) + if (evaluated) { + frozenEvalStatus = 'returned' + frozenEvalSnapshot = (evaluated as any).result.value as Snapshot + } + } catch (error) { + frozenEvalStatus = 'error' + frozenEvalError = error instanceof Error ? error.message : String(error) + } + + await cdp.send('Page.setWebLifecycleState', { state: 'active' }) + const resumedImmediate = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentAtResumeImmediate = serverSent + await sleep(1000) + const resumedB = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentAtResumeB = serverSent + + const artifact = { + generatedAt: new Date().toISOString(), + chromiumVersion: browser.version(), + headed, + wsSendIntervalMs: 50, + active: { + before: activeA, + after: activeB, + pageDelta: delta(activeA, activeB), + serverSentDelta: serverSentAtActiveB - serverSentAtActiveA, + }, + frozen: { + stateCommand: "Page.setWebLifecycleState({ state: 'frozen' })", + durationMs: Date.now() - freezeStart, + before: frozenA, + evalStatus: frozenEvalStatus, + evalSnapshot: frozenEvalSnapshot, + evalError: frozenEvalError, + pageDeltaIfEvalReturned: frozenEvalSnapshot ? delta(frozenA, frozenEvalSnapshot) : null, + serverSentDelta: serverSentAtFreezeEnd - serverSentAtFreezeStart, + }, + resumed: { + immediate: resumedImmediate, + after: resumedB, + immediateDeltaFromFreezeStart: delta(frozenA, resumedImmediate), + pageDeltaAfterResume: delta(resumedImmediate, resumedB), + serverSentDeltaImmediate: serverSentAtResumeImmediate - serverSentAtFreezeStart, + serverSentDeltaAfterResume: serverSentAtResumeB - serverSentAtResumeImmediate, + }, + } + + const outPath = path.resolve(artifactDir, 'browser-freeze-lifecycle.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath }, null, 2)) + } finally { + await browser.close().catch(() => undefined) + clearInterval(sendTimer) + await new Promise((resolve) => wss.close(() => resolve())) + await new Promise((resolve) => httpServer.close(() => resolve())) + } +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/scripts/proofs/browser-process-suspend-probe.ts b/scripts/proofs/browser-process-suspend-probe.ts new file mode 100644 index 000000000..98a881519 --- /dev/null +++ b/scripts/proofs/browser-process-suspend-probe.ts @@ -0,0 +1,205 @@ +import fs from 'fs/promises' +import http from 'http' +import path from 'path' +import { fileURLToPath } from 'url' +import { execFileSync } from 'child_process' +import { chromium } from '@playwright/test' +import { WebSocketServer } from 'ws' + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +type Snapshot = { + at: number + intervalTicks: number + rafTicks: number + wsReceived: number + lastIntervalAt: number + lastRafAt: number + lastWsAt: number +} + +function delta(a: Snapshot, b: Snapshot) { + return { + elapsedMs: b.at - a.at, + intervalTicks: b.intervalTicks - a.intervalTicks, + rafTicks: b.rafTicks - a.rafTicks, + wsReceived: b.wsReceived - a.wsReceived, + } +} + +function chromiumProcessTree(rootPid: number): number[] { + const output = execFileSync('ps', ['-eo', 'pid=,ppid=,comm='], { encoding: 'utf8' }) + const children = new Map() + for (const line of output.split('\n')) { + const match = line.trim().match(/^(\d+)\s+(\d+)\s+(.+)$/) + if (!match) continue + const pid = Number(match[1]) + const ppid = Number(match[2]) + if (!children.has(ppid)) children.set(ppid, []) + children.get(ppid)?.push(pid) + } + + const tree: number[] = [] + const visit = (pid: number) => { + tree.push(pid) + for (const child of children.get(pid) ?? []) visit(child) + } + visit(rootPid) + return tree +} + +function signalPids(pids: number[], signal: NodeJS.Signals): void { + for (const pid of pids) { + try { + process.kill(pid, signal) + } catch { + // Process may have exited during shutdown. + } + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + let serverSent = 0 + const httpServer = http.createServer((_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain' }) + res.end('ok') + }) + const wss = new WebSocketServer({ server: httpServer }) + const sendTimer = setInterval(() => { + serverSent += 1 + const payload = JSON.stringify({ n: serverSent, at: Date.now() }) + for (const client of wss.clients) { + if (client.readyState === client.OPEN) client.send(payload) + } + }, 50) + + const port = await new Promise((resolve) => { + httpServer.listen(0, '127.0.0.1', () => { + const address = httpServer.address() + if (!address || typeof address === 'string') throw new Error('No listen port') + resolve(address.port) + }) + }) + + const browserServer = await chromium.launchServer({ headless: true }) + const browserProcess = browserServer.process() + if (!browserProcess?.pid) throw new Error('Browser process pid unavailable') + const browser = await chromium.connect(browserServer.wsEndpoint()) + const page = await browser.newPage() + const wsUrl = `ws://127.0.0.1:${port}` + const html = ` + + + + +` + + try { + await page.goto(`data:text/html,${encodeURIComponent(html)}`) + await page.evaluate(() => (window as any).__wsReady) + await sleep(1000) + const activeA = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentActiveA = serverSent + await sleep(1000) + const activeB = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentActiveB = serverSent + + const stoppedBefore = activeB + const pidsToStop = chromiumProcessTree(browserProcess.pid) + const serverSentStopStart = serverSent + const stopStartedAt = Date.now() + signalPids([...pidsToStop].sort((a, b) => b - a), 'SIGSTOP') + await sleep(2000) + const serverSentStopEnd = serverSent + signalPids([...pidsToStop].sort((a, b) => a - b), 'SIGCONT') + const stopEndedAt = Date.now() + const resumedImmediate = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentResumeImmediate = serverSent + await sleep(1000) + const resumedAfter = await page.evaluate(() => (window as any).__snapshot()) as Snapshot + const serverSentResumeAfter = serverSent + + const artifact = { + generatedAt: new Date().toISOString(), + chromiumVersion: browser.version(), + browserPid: browserProcess.pid, + wsSendIntervalMs: 50, + active: { + before: activeA, + after: activeB, + pageDelta: delta(activeA, activeB), + serverSentDelta: serverSentActiveB - serverSentActiveA, + }, + stopped: { + signal: 'SIGSTOP', + stoppedPids: pidsToStop, + durationMs: stopEndedAt - stopStartedAt, + before: stoppedBefore, + afterResumeImmediate: resumedImmediate, + pageDeltaAfterResumeImmediate: delta(stoppedBefore, resumedImmediate), + serverSentWhileStopped: serverSentStopEnd - serverSentStopStart, + serverSentByResumeImmediate: serverSentResumeImmediate - serverSentStopStart, + }, + resumed: { + after: resumedAfter, + pageDeltaAfterResume: delta(resumedImmediate, resumedAfter), + serverSentDeltaAfterResume: serverSentResumeAfter - serverSentResumeImmediate, + }, + } + + const outPath = path.resolve(artifactDir, 'browser-process-suspend.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath }, null, 2)) + } finally { + await browser.close().catch(() => undefined) + await browserServer.close().catch(() => undefined) + clearInterval(sendTimer) + await new Promise((resolve) => wss.close(() => resolve())) + await new Promise((resolve) => httpServer.close(() => resolve())) + } +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/scripts/proofs/terminal-catchup-agent-output-generator.mjs b/scripts/proofs/terminal-catchup-agent-output-generator.mjs new file mode 100644 index 000000000..928bfef23 --- /dev/null +++ b/scripts/proofs/terminal-catchup-agent-output-generator.mjs @@ -0,0 +1,58 @@ +#!/usr/bin/env node + +const scenario = process.argv[2] || 'agent-burst' +const lineCount = Number(process.argv[3] || 1200) + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)) +const ESC = '\u001b' +const BEL = '\u0007' +const ST = `${ESC}\\` + +function write(data) { + process.stdout.write(data) +} + +async function agentBurst() { + write(`${ESC}]0;Freshell proof agent burst${BEL}`) + write(`${ESC}[?25l`) + for (let i = 1; i <= lineCount; i += 1) { + if (i % 200 === 1) { + write(`${ESC}[38;5;39mthinking${ESC}[0m ${String(i).padStart(5, '0')} scanning repository\r`) + await sleep(2) + write(`${ESC}[2K\r`) + } + if (i % 97 === 0) { + write(`${ESC}]52;c;${Buffer.from(`clipboard-${i}`).toString('base64')}${BEL}`) + } + if (i % 251 === 0) { + write(`${ESC}P1;2;3|proof-dcs-${i}${ST}`) + } + const severity = i % 17 === 0 ? `${ESC}[33mwarn${ESC}[0m` : `${ESC}[32mok${ESC}[0m` + write(`[${severity}] ${String(i).padStart(5, '0')} edit src/example-${i % 23}.ts chunk=${i % 11} tokens=${1000 + i}\n`) + if (i % 128 === 0) await sleep(4) + } + write(`${ESC}[?25h`) + write(`FRESHELL_PROOF_DONE:${scenario}:${lineCount}\n`) +} + +async function controlBarrier() { + write(`${ESC}]0;Freshell proof split OSC`) + await sleep(5) + write(`${BEL}after-title\n`) + write(`${ESC}[38;5;196m`) + await sleep(5) + write(`split-sgr-red${ESC}[0m\n`) + write(`${ESC}Pq`) + await sleep(5) + write(`dcs-payload${ST}after-dcs\n`) + write(`${ESC}[?2026;1$y`) + write(`${ESC}]52;c;${Buffer.from('proof clipboard').toString('base64')}${BEL}`) + write('\uFFFD replacement-byte-sentinel\n') + write(`FRESHELL_PROOF_DONE:${scenario}:${lineCount}\n`) +} + +if (scenario === 'control-barrier') { + await controlBarrier() +} else { + await agentBurst() +} diff --git a/scripts/proofs/terminal-catchup-pty-metrics.ts b/scripts/proofs/terminal-catchup-pty-metrics.ts new file mode 100644 index 000000000..3e1cf667e --- /dev/null +++ b/scripts/proofs/terminal-catchup-pty-metrics.ts @@ -0,0 +1,450 @@ +import fs from 'fs/promises' +import path from 'path' +import { fileURLToPath } from 'url' +import WebSocket from 'ws' +import { TerminalRegistry } from '../../server/terminal-registry.js' +import { TerminalStreamBroker } from '../../server/terminal-stream/broker.js' +import { TERMINAL_STREAM_BATCH_MAX_BYTES } from '../../server/terminal-stream/constants.js' + +type RawChunk = { + at: number + bytes: number + chars: number + data: string +} + +type SentMessage = { + at: number + bytes: number + type?: string + message: any +} + +type Scenario = { + name: string + command: string + timeoutMs: number + marker: string +} + +type ScannerState = 'ground' | 'esc' | 'csi' | 'osc' | 'dcs' | 'apc' | 'pm' | 'sos' | 'stringEsc' + +type ScannerFrame = { + startState: ScannerState + endState: ScannerState + hasControl: boolean + hasSideEffectBarrier: boolean + hasReplacement: boolean + conservativeBarrier: boolean +} + +type MockSocket = WebSocket & { + sent: SentMessage[] +} + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +function percentile(values: number[], pct: number): number { + if (values.length === 0) return 0 + const sorted = [...values].sort((a, b) => a - b) + const index = Math.min(sorted.length - 1, Math.max(0, Math.ceil((pct / 100) * sorted.length) - 1)) + return sorted[index] +} + +function summarize(values: number[]) { + return { + min: values.length ? Math.min(...values) : 0, + p50: percentile(values, 50), + p90: percentile(values, 90), + p95: percentile(values, 95), + p99: percentile(values, 99), + max: values.length ? Math.max(...values) : 0, + } +} + +function createMockSocket(connectionId: string): MockSocket { + const sent: SentMessage[] = [] + return { + readyState: WebSocket.OPEN, + bufferedAmount: 0, + connectionId, + sent, + send(data: string, cb?: (err?: Error) => void) { + const parsed = JSON.parse(data) + sent.push({ + at: Date.now(), + bytes: Buffer.byteLength(data, 'utf8'), + type: typeof parsed?.type === 'string' ? parsed.type : undefined, + message: parsed, + }) + cb?.() + }, + close() { + this.readyState = WebSocket.CLOSED + }, + } as unknown as MockSocket +} + +function scanFrame(data: string, startState: ScannerState): ScannerFrame { + let state = startState + let stringReturnState: ScannerState = 'osc' + let hasControl = false + let hasSideEffectBarrier = false + let hasReplacement = false + + const enterString = (next: ScannerState) => { + state = next + stringReturnState = next + hasControl = true + hasSideEffectBarrier = true + } + + for (let i = 0; i < data.length; i += 1) { + const ch = data[i] + const code = ch.codePointAt(0) ?? 0 + if (code > 0xffff) i += 1 + if (ch === '\uFFFD') hasReplacement = true + + if (state === 'ground') { + if (ch === '\u001b') { + state = 'esc' + hasControl = true + continue + } + if (code === 0x9b) { + state = 'csi' + hasControl = true + continue + } + if (code === 0x9d) { + enterString('osc') + continue + } + if (code === 0x90) { + enterString('dcs') + continue + } + if (code === 0x9f) { + enterString('apc') + continue + } + if (code < 0x20 && ch !== '\n' && ch !== '\r' && ch !== '\t') { + hasControl = true + if (ch === '\u0007') hasSideEffectBarrier = true + } + continue + } + + if (state === 'esc') { + if (ch === '[') { + state = 'csi' + } else if (ch === ']') { + enterString('osc') + } else if (ch === 'P') { + enterString('dcs') + } else if (ch === '_') { + enterString('apc') + } else if (ch === '^') { + enterString('pm') + } else if (ch === 'X') { + enterString('sos') + } else { + state = 'ground' + } + continue + } + + if (state === 'csi') { + if (code >= 0x40 && code <= 0x7e) { + if (data.slice(Math.max(0, i - 16), i + 1).includes('$')) { + hasSideEffectBarrier = true + } + state = 'ground' + } + continue + } + + if (state === 'osc' || state === 'dcs' || state === 'apc' || state === 'pm' || state === 'sos') { + if (ch === '\u0007') { + state = 'ground' + } else if (ch === '\u001b') { + stringReturnState = state + state = 'stringEsc' + } + continue + } + + if (state === 'stringEsc') { + state = ch === '\\' ? 'ground' : stringReturnState + } + } + + return { + startState, + endState: state, + hasControl, + hasSideEffectBarrier, + hasReplacement, + conservativeBarrier: startState !== 'ground' || state !== 'ground' || hasSideEffectBarrier || hasReplacement, + } +} + +function conservativeScannerBatches(chunks: RawChunk[], budgetBytes: number) { + const frames: ScannerFrame[] = [] + let state: ScannerState = 'ground' + let groups = 0 + let currentBytes = 0 + + for (const chunk of chunks) { + const frame = scanFrame(chunk.data, state) + frames.push(frame) + state = frame.endState + const mustSplit = frame.conservativeBarrier || currentBytes + chunk.bytes > budgetBytes + if (groups === 0 || mustSplit) { + groups += 1 + currentBytes = chunk.bytes + } else { + currentBytes += chunk.bytes + } + if (frame.conservativeBarrier) { + currentBytes = 0 + } + } + + return { + scannerFrameCount: frames.length, + conservativeBatchCount: groups, + barrierFrameCount: frames.filter((frame) => frame.conservativeBarrier).length, + sideEffectBarrierFrameCount: frames.filter((frame) => frame.hasSideEffectBarrier).length, + replacementFrameCount: frames.filter((frame) => frame.hasReplacement).length, + pendingEndStateCount: frames.filter((frame) => frame.endState !== 'ground').length, + finalState: state, + } +} + +function burstGroups(chunks: RawChunk[], maxGapMs: number) { + if (chunks.length === 0) return [] + const groups: Array<{ chunks: number; bytes: number; durationMs: number }> = [] + let current = { chunks: 1, bytes: chunks[0].bytes, start: chunks[0].at, end: chunks[0].at } + for (let i = 1; i < chunks.length; i += 1) { + const previous = chunks[i - 1] + const chunk = chunks[i] + if (chunk.at - previous.at <= maxGapMs) { + current.chunks += 1 + current.bytes += chunk.bytes + current.end = chunk.at + } else { + groups.push({ chunks: current.chunks, bytes: current.bytes, durationMs: current.end - current.start }) + current = { chunks: 1, bytes: chunk.bytes, start: chunk.at, end: chunk.at } + } + } + groups.push({ chunks: current.chunks, bytes: current.bytes, durationMs: current.end - current.start }) + return groups +} + +async function waitForMarker(rawChunks: RawChunk[], marker: string, timeoutMs: number): Promise { + const deadline = Date.now() + timeoutMs + while (Date.now() < deadline) { + if (rawChunks.some((chunk) => chunk.data.includes(marker))) return + await sleep(50) + } + throw new Error(`Timed out waiting for ${marker}`) +} + +async function runScenario(scenario: Scenario) { + const rawChunks: RawChunk[] = [] + const registry = new TerminalRegistry() + const broker = new TerminalStreamBroker(registry) + const socket = createMockSocket(`proof-${scenario.name}`) + const startAt = Date.now() + + const onRaw = (event: { terminalId?: string; data?: string; at?: number }) => { + if (typeof event.data !== 'string') return + rawChunks.push({ + at: event.at ?? Date.now(), + bytes: Buffer.byteLength(event.data, 'utf8'), + chars: event.data.length, + data: event.data, + }) + } + + registry.on('terminal.output.raw', onRaw) + const record = registry.create({ + mode: 'shell', + shell: 'system', + cwd: repoRoot, + cols: 120, + rows: 30, + }) + + await broker.attach( + socket, + record.terminalId, + 'viewport_hydrate', + 120, + 30, + 0, + `${scenario.name}-attach`, + ) + + await sleep(250) + registry.input(record.terminalId, `${scenario.command}\r`) + await waitForMarker(rawChunks, scenario.marker, scenario.timeoutMs) + await sleep(300) + + await broker.attach( + socket, + record.terminalId, + 'transport_reconnect', + 120, + 30, + 0, + `${scenario.name}-replay`, + ) + await sleep(300) + + broker.close() + registry.off('terminal.output.raw', onRaw) + await registry.shutdownGracefully(1000) + + const outputMessages = socket.sent.filter((sent) => sent.type === 'terminal.output') + const replayOutputMessages = outputMessages.filter( + (sent) => sent.message.attachRequestId === `${scenario.name}-replay`, + ) + const rawBytes = rawChunks.reduce((sum, chunk) => sum + chunk.bytes, 0) + const rawJoined = rawChunks.map((chunk) => chunk.data).join('') + const durationMs = Math.max(1, Math.max(...rawChunks.map((chunk) => chunk.at), startAt) - startAt) + const chunkBytes = rawChunks.map((chunk) => chunk.bytes) + const interarrivalMs = rawChunks.slice(1).map((chunk, index) => chunk.at - rawChunks[index].at) + const bursts = burstGroups(rawChunks, 10) + const scanner = conservativeScannerBatches(rawChunks, TERMINAL_STREAM_BATCH_MAX_BYTES) + const serializedBytes = outputMessages.map((sent) => sent.bytes) + + return { + scenario: scenario.name, + command: scenario.command, + terminalId: record.terminalId, + durationMs, + raw: { + chunks: rawChunks.length, + bytes: rawBytes, + bytesPerSecond: Math.round((rawBytes * 1000) / durationMs), + previewStartJsonEscaped: rawJoined.slice(0, 1600), + previewEndJsonEscaped: rawJoined.slice(-1600), + chunkBytes: summarize(chunkBytes), + interarrivalMs: summarize(interarrivalMs), + burstGroupsUnder10ms: { + count: bursts.length, + chunks: summarize(bursts.map((burst) => burst.chunks)), + bytes: summarize(bursts.map((burst) => burst.bytes)), + durationMs: summarize(bursts.map((burst) => burst.durationMs)), + }, + }, + broker: { + sentMessages: socket.sent.length, + outputMessages: outputMessages.length, + replayOutputMessages: replayOutputMessages.length, + outputSerializedBytes: summarize(serializedBytes), + maxSerializedBytes: serializedBytes.length ? Math.max(...serializedBytes) : 0, + maxRawDataBytesInOutputMessage: outputMessages.length + ? Math.max(...outputMessages.map((sent) => Buffer.byteLength(sent.message.data ?? '', 'utf8'))) + : 0, + sequenceRanges: outputMessages + .filter((sent) => typeof sent.message.seqStart === 'number') + .map((sent) => ({ + attachRequestId: sent.message.attachRequestId, + seqStart: sent.message.seqStart, + seqEnd: sent.message.seqEnd, + dataBytes: Buffer.byteLength(sent.message.data ?? '', 'utf8'), + })) + .slice(0, 20), + }, + scanner, + ratios: { + rawChunksPerLiveOutputMessage: outputMessages.length ? Number((rawChunks.length / outputMessages.length).toFixed(3)) : null, + rawChunksPerReplayOutputMessage: replayOutputMessages.length ? Number((rawChunks.length / replayOutputMessages.length).toFixed(3)) : null, + rawChunksPerConservativeScannerBatch: scanner.conservativeBatchCount + ? Number((rawChunks.length / scanner.conservativeBatchCount).toFixed(3)) + : null, + }, + } +} + +async function dependencyVersion(packageName: string): Promise { + try { + const packageJsonUrl = await import.meta.resolve(`${packageName}/package.json`) + const parsed = JSON.parse(await fs.readFile(fileURLToPath(packageJsonUrl), 'utf8')) + return parsed.version ?? null + } catch { + return null + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + const marker = (name: string, count: number) => `FRESHELL_PROOF_DONE:${name}:${count}` + const printMarker = (name: string, count: number) => `printf '\\nFRESHELL_PROOF_DONE:%s:%s\\n' '${name}' '${count}'` + const scenarios: Scenario[] = [ + { + name: 'codex-version', + command: `codex --version; ${printMarker('codex-version', 1)}`, + timeoutMs: 30_000, + marker: marker('codex-version', 1), + }, + { + name: 'codex-help', + command: `codex exec --help; ${printMarker('codex-help', 1)}`, + timeoutMs: 30_000, + marker: marker('codex-help', 1), + }, + { + name: 'codex-real-turn', + command: `codex exec --ephemeral --color always --sandbox read-only --cd /tmp --skip-git-repo-check "Print exactly 40 numbered lines, each line beginning proof-line-, and do not run shell commands."; ${printMarker('codex-real-turn', 40)}`, + timeoutMs: 240_000, + marker: marker('codex-real-turn', 40), + }, + { + name: 'agent-burst-12000', + command: `node scripts/proofs/terminal-catchup-agent-output-generator.mjs agent-burst 12000`, + timeoutMs: 180_000, + marker: marker('agent-burst', 12000), + }, + { + name: 'control-barrier', + command: `node scripts/proofs/terminal-catchup-agent-output-generator.mjs control-barrier 1`, + timeoutMs: 30_000, + marker: marker('control-barrier', 1), + }, + ] + + const results = [] + for (const scenario of scenarios) { + results.push(await runScenario(scenario)) + } + + const artifact = { + generatedAt: new Date().toISOString(), + repoRoot, + nodeVersion: process.version, + terminalStreamBatchMaxBytes: TERMINAL_STREAM_BATCH_MAX_BYTES, + dependencies: { + '@xterm/xterm': await dependencyVersion('@xterm/xterm'), + ws: await dependencyVersion('ws'), + 'node-pty': await dependencyVersion('node-pty'), + }, + results, + } + + const outPath = path.resolve(artifactDir, 'terminal-catchup-pty-metrics.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath, scenarioCount: results.length }, null, 2)) +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/scripts/proofs/terminal-json-serialization-probe.ts b/scripts/proofs/terminal-json-serialization-probe.ts new file mode 100644 index 000000000..7e9e64db7 --- /dev/null +++ b/scripts/proofs/terminal-json-serialization-probe.ts @@ -0,0 +1,62 @@ +import fs from 'fs/promises' +import path from 'path' +import { fileURLToPath } from 'url' + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') + +function repeatToBytes(unit: string, targetBytes: number): string { + let out = '' + while (Buffer.byteLength(out, 'utf8') < targetBytes) out += unit + while (Buffer.byteLength(out, 'utf8') > targetBytes) out = out.slice(0, -1) + return out +} + +function measure(name: string, data: string) { + const message = { + type: 'terminal.output', + terminalId: 'term-proof', + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-proof', + data, + } + const serialized = JSON.stringify(message) + const rawBytes = Buffer.byteLength(data, 'utf8') + const serializedBytes = Buffer.byteLength(serialized, 'utf8') + return { + name, + rawBytes, + serializedBytes, + jsonOverheadBytes: serializedBytes - rawBytes, + expansionRatio: Number((serializedBytes / rawBytes).toFixed(3)), + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + const target = 16 * 1024 + const samples = [ + measure('plain-ascii', repeatToBytes('a', target)), + measure('ansi-sgr-repeat', repeatToBytes('\u001b[32mok\u001b[0m ', target)), + measure('esc-only-control', repeatToBytes('\u001b', target)), + measure('newline-heavy', repeatToBytes('x\n', target)), + measure('quote-heavy', repeatToBytes('"', target)), + measure('backslash-heavy', repeatToBytes('\\', target)), + ] + + const artifact = { + generatedAt: new Date().toISOString(), + targetRawBytes: target, + samples, + } + const outPath = path.resolve(artifactDir, 'terminal-json-serialization.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath }, null, 2)) +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/scripts/proofs/xterm-write-dispose-probe.ts b/scripts/proofs/xterm-write-dispose-probe.ts new file mode 100644 index 000000000..3b3321b8c --- /dev/null +++ b/scripts/proofs/xterm-write-dispose-probe.ts @@ -0,0 +1,125 @@ +import fs from 'fs/promises' +import path from 'path' +import { fileURLToPath } from 'url' +import xtermPkg from '@xterm/xterm' + +const { Terminal } = xtermPkg as unknown as { Terminal: typeof import('@xterm/xterm').Terminal } + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const repoRoot = path.resolve(__dirname, '../..') +const artifactDir = path.resolve(repoRoot, 'docs/superpowers/proofs/artifacts') +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +async function dependencyVersion(packageName: string): Promise { + try { + const packageJsonUrl = await import.meta.resolve(`${packageName}/package.json`) + const parsed = JSON.parse(await fs.readFile(fileURLToPath(packageJsonUrl), 'utf8')) + return parsed.version ?? null + } catch { + return null + } +} + +async function smallWriteOrder() { + const term = new Terminal({ allowProposedApi: true, cols: 80, rows: 24 }) + const events: Array<{ event: string; at: number; cursorX?: number; cursorY?: number }> = [] + const startedAt = performance.now() + const mark = (event: string) => { + events.push({ + event, + at: Number((performance.now() - startedAt).toFixed(3)), + cursorX: term.buffer.active.cursorX, + cursorY: term.buffer.active.cursorY, + }) + } + term.write('alpha', () => mark('callback-alpha')) + mark('after-write-alpha-returned') + term.write('\nbeta', () => mark('callback-beta')) + mark('after-write-beta-returned') + await sleep(50) + mark('after-50ms') + term.dispose() + return events +} + +async function disposeAfterLargeWrite() { + const term = new Terminal({ allowProposedApi: true, cols: 120, rows: 30 }) + const startedAt = performance.now() + const events: Array<{ event: string; at: number; cursorX?: number; cursorY?: number }> = [] + const mark = (event: string) => { + events.push({ + event, + at: Number((performance.now() - startedAt).toFixed(3)), + cursorX: term.buffer.active.cursorX, + cursorY: term.buffer.active.cursorY, + }) + } + const data = `${'large-line-0123456789 '.repeat(16)}\n`.repeat(50_000) + let callbackAfterDispose = false + let disposed = false + term.write(data, () => { + callbackAfterDispose = disposed + mark('large-write-callback') + }) + mark('after-large-write-returned') + term.dispose() + disposed = true + mark('after-dispose') + await sleep(250) + mark('after-250ms') + return { + events, + callbackAfterDispose, + } +} + +async function disposeAfterQueuedWrites() { + const term = new Terminal({ allowProposedApi: true, cols: 120, rows: 30 }) + const startedAt = performance.now() + let disposed = false + const callbacks: Array<{ index: number; at: number; afterDispose: boolean }> = [] + for (let i = 0; i < 500; i += 1) { + term.write(`queued-${i}\n`, () => { + callbacks.push({ + index: i, + at: Number((performance.now() - startedAt).toFixed(3)), + afterDispose: disposed, + }) + }) + } + const disposeAt = Number((performance.now() - startedAt).toFixed(3)) + term.dispose() + disposed = true + await sleep(250) + return { + disposeAt, + callbackCount: callbacks.length, + callbacksAfterDispose: callbacks.filter((callback) => callback.afterDispose).length, + firstCallbacks: callbacks.slice(0, 10), + lastCallbacks: callbacks.slice(-10), + fifo: callbacks.every((callback, index) => callback.index === index), + } +} + +async function main() { + await fs.mkdir(artifactDir, { recursive: true }) + const artifact = { + generatedAt: new Date().toISOString(), + nodeVersion: process.version, + dependencies: { + '@xterm/xterm': await dependencyVersion('@xterm/xterm'), + }, + smallWriteOrder: await smallWriteOrder(), + disposeAfterLargeWrite: await disposeAfterLargeWrite(), + disposeAfterQueuedWrites: await disposeAfterQueuedWrites(), + } + + const outPath = path.resolve(artifactDir, 'xterm-write-dispose.json') + await fs.writeFile(outPath, `${JSON.stringify(artifact, null, 2)}\n`) + console.log(JSON.stringify({ outPath }, null, 2)) +} + +main().catch((error) => { + console.error(error) + process.exitCode = 1 +}) diff --git a/server/terminal-registry.ts b/server/terminal-registry.ts index 1ef821d68..0559fea43 100644 --- a/server/terminal-registry.ts +++ b/server/terminal-registry.ts @@ -2857,6 +2857,11 @@ export class TerminalRegistry extends EventEmitter { record.codexSidecarLifecycleUnsubscribe?.() record.codexSidecarLifecycleUnsubscribe = undefined record.pty = candidate.pty + this.emit('terminal.stream.replaced', { + terminalId: record.terminalId, + reason: 'codex_pty_recovery', + at: Date.now(), + }) record.mcpCwd = candidate.mcpCwd record.codexSidecar = plan.sidecar record.codexSidecarLifecyclePublished = true diff --git a/server/terminal-stream/broker.ts b/server/terminal-stream/broker.ts index 3c6067b81..eb5fc20f5 100644 --- a/server/terminal-stream/broker.ts +++ b/server/terminal-stream/broker.ts @@ -4,8 +4,32 @@ import { buildTerminalSessionRef, type TerminalRegistry } from '../terminal-regi import { logger } from '../logger.js' import { logTerminalStreamPerfEvent, type TerminalStreamPerfEvent } from '../perf-logger.js' import type { TerminalOutputRawEvent } from './registry-events.js' -import { ClientOutputQueue, isGapEvent, type GapEvent } from './client-output-queue.js' +import { + ClientOutputQueue, + isGapEvent, + type GapEvent, +} from './client-output-queue.js' import { ReplayRing, type ReplayFrame } from './replay-ring.js' +import type { TerminalOutputBatch } from './output-batch.js' +import { fragmentTerminalOutputForPayloadBudget } from './output-fragments.js' +import { + prepareJsonMessage, + readWebSocketBufferedAmount, + sendJsonMessage, + sendPreparedJsonMessage, + type PreparedJsonMessage, + type SendJsonResult, +} from '../ws-send.js' +import { + isTerminalStreamAttachRequestIdWithinSerializedBudget, + measureTerminalOutputPayloadBytes, + TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + type JsonPayload, +} from './serialized-budget.js' +import { + createTerminalStreamIdentityTracker, + type TerminalStreamReplacementReason, +} from './stream-identity.js' import { TERMINAL_BACKGROUND_BUFFERED_PAUSE_BYTES, TERMINAL_BACKGROUND_RETRY_FLUSH_MS, @@ -15,30 +39,149 @@ import { TERMINAL_WS_CATASTROPHIC_STALL_MS, } from './constants.js' import type { BrokerClientAttachment, BrokerTerminalState } from './types.js' +import type { TerminalGeometryAuthority } from '../../shared/ws-protocol.js' const log = logger.child({ component: 'terminal-stream-broker' }) +const DEFAULT_CODING_CLI_REPLAY_RING_MAX_BYTES = 32 * 1024 * 1024 const CODING_CLI_MIN_REPLAY_RING_MAX_BYTES = Number( - process.env.CODING_CLI_MIN_REPLAY_RING_MAX_BYTES || 8 * 1024 * 1024, + process.env.CODING_CLI_MIN_REPLAY_RING_MAX_BYTES || DEFAULT_CODING_CLI_REPLAY_RING_MAX_BYTES, +) +const TERMINAL_STREAM_BUDGET_SEQ_PLACEHOLDER = Number.MAX_SAFE_INTEGER +const CONFIGURED_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES = Number( + process.env.TERMINAL_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES || 512 * 1024 + 64 * 1024, +) +const TERMINAL_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES = Math.min( + Math.max( + TERMINAL_BACKGROUND_BUFFERED_PAUSE_BYTES + 1024, + Number.isFinite(CONFIGURED_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES) + && CONFIGURED_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES > 0 + ? Math.floor(CONFIGURED_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES) + : 512 * 1024 + 64 * 1024, + ), + Math.max(1024, TERMINAL_WS_CATASTROPHIC_BUFFERED_BYTES - 1), +) +const CONFIGURED_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS = Number( + process.env.TERMINAL_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS || 1000, +) +const TERMINAL_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS = Math.max( + 1, + Number.isFinite(CONFIGURED_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS) + && CONFIGURED_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS > 0 + ? Math.floor(CONFIGURED_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS) + : 1000, +) +const CONFIGURED_REPLAY_RETENTION_LOG_RATE_LIMIT_MS = Number( + process.env.TERMINAL_REPLAY_RETENTION_LOG_RATE_LIMIT_MS || 1000, +) +const TERMINAL_REPLAY_RETENTION_LOG_RATE_LIMIT_MS = Math.max( + 1, + Number.isFinite(CONFIGURED_REPLAY_RETENTION_LOG_RATE_LIMIT_MS) + && CONFIGURED_REPLAY_RETENTION_LOG_RATE_LIMIT_MS > 0 + ? Math.floor(CONFIGURED_REPLAY_RETENTION_LOG_RATE_LIMIT_MS) + : 1000, ) type PerfLevel = 'debug' | 'info' | 'warn' | 'error' type AttachIntent = 'viewport_hydrate' | 'keepalive_delta' | 'transport_reconnect' type AttachPriority = 'foreground' | 'background' +type ReplayGapRange = { + fromSeq: number + toSeq: number +} +type ReplaySendOutcome = + | { status: 'sent'; pauseAfter: boolean; sentSeqEnd: number } + | { status: 'paused' | 'failed' } +type LiveSendOutcome = + | { status: 'sent'; sentFrameCount: number; sentSeqEnd: number } + | { status: 'failed'; sentFrameCount: number; sentSeqEnd?: number; reason?: SendJsonResult['reason'] } type PerfEventLogger = ( event: TerminalStreamPerfEvent, context: Record, level?: PerfLevel, ) => void +type ReplayGapReason = 'replay_window_exceeded' | 'replay_budget_exceeded' | 'queue_overflow' +type ReplayBackpressurePayloadFields = { + seqStart?: number + seqEnd?: number + rawFrameCount: number + dataBytes: number +} + +function jsonNumber(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) ? value : undefined +} + +function jsonString(value: unknown): string | undefined { + return typeof value === 'string' && value.length > 0 ? value : undefined +} + +function jsonNumberField(payload: JsonPayload, field: string): number | undefined { + return jsonNumber(payload[field]) +} + +function jsonStringField(payload: JsonPayload, field: string): string | undefined { + return jsonString(payload[field]) +} + +function payloadRawFrameCount(payload: JsonPayload): number { + const segments = Array.isArray(payload.segments) ? payload.segments : [] + if (segments.length > 0) { + return segments.reduce((sum, segment) => { + const rawFrameCount = jsonNumber((segment as { rawFrameCount?: unknown }).rawFrameCount) + return sum + Math.max(0, Math.floor(rawFrameCount ?? 0)) + }, 0) + } + + const seqStart = jsonNumberField(payload, 'seqStart') + const seqEnd = jsonNumberField(payload, 'seqEnd') + if (seqStart === undefined || seqEnd === undefined || seqEnd < seqStart) return 0 + return Math.max(1, Math.floor(seqEnd - seqStart + 1)) +} + +function batchRawFrameCount(batch: TerminalOutputBatch): number { + return batch.segments.reduce((sum, segment) => { + const frameCount = Math.max(1, Math.floor(segment.seqEnd - segment.seqStart + 1)) + return sum + frameCount + }, 0) +} + +function payloadBackpressureFields(payload: JsonPayload): ReplayBackpressurePayloadFields { + const data = typeof payload.data === 'string' ? payload.data : '' + const seqStart = jsonNumberField(payload, 'seqStart') ?? jsonNumberField(payload, 'fromSeq') + const seqEnd = jsonNumberField(payload, 'seqEnd') ?? jsonNumberField(payload, 'toSeq') + const rawFrameCount = jsonStringField(payload, 'type') === 'terminal.output.gap' + ? 0 + : payloadRawFrameCount(payload) + return { + ...(seqStart !== undefined ? { seqStart } : {}), + ...(seqEnd !== undefined ? { seqEnd } : {}), + rawFrameCount, + dataBytes: Buffer.byteLength(data, 'utf8'), + } +} export class TerminalStreamBroker { private terminals = new Map() private wsToTerminals = new Map>() private terminalLocks = new Map>() + private streamIdentity = createTerminalStreamIdentityTracker() private readonly onRawOutputBound = (event: TerminalOutputRawEvent) => { this.onTerminalOutputRaw(event) } + private readonly onStreamReplacedBound = (payload: { + terminalId?: string + reason?: TerminalStreamReplacementReason + }) => { + const terminalId = payload?.terminalId + if (typeof terminalId !== 'string' || !terminalId) return + this.replaceStreamIdentity( + terminalId, + payload.reason ?? 'server_restart_incompatible_retention', + ) + } + private readonly onTerminalExitBound = (payload: { terminalId?: string }) => { const terminalId = payload?.terminalId if (typeof terminalId === 'string' && terminalId) { @@ -55,6 +198,7 @@ export class TerminalStreamBroker { } if (typeof eventSource.on === 'function') { eventSource.on('terminal.output.raw', this.onRawOutputBound) + eventSource.on('terminal.stream.replaced', this.onStreamReplacedBound) eventSource.on('terminal.exit', this.onTerminalExitBound) } } @@ -65,6 +209,7 @@ export class TerminalStreamBroker { } if (typeof eventSource.off === 'function') { eventSource.off('terminal.output.raw', this.onRawOutputBound) + eventSource.off('terminal.stream.replaced', this.onStreamReplacedBound) eventSource.off('terminal.exit', this.onTerminalExitBound) } for (const state of this.terminals.values()) { @@ -88,9 +233,14 @@ export class TerminalStreamBroker { attachRequestId?: string, maxReplayBytes?: number, priority: AttachPriority = 'foreground', - ): Promise<'attached' | 'duplicate' | 'missing'> { + terminalOutputBatchV1 = false, + ): Promise<'attached' | 'duplicate' | 'missing' | 'invalid_attach_request_id'> { + if (!isTerminalStreamAttachRequestIdWithinSerializedBudget(attachRequestId)) { + return 'invalid_attach_request_id' + } + const normalizedSinceSeq = sinceSeq === undefined || sinceSeq === 0 ? 0 : sinceSeq - let result: 'attached' | 'duplicate' | 'missing' = 'attached' + let result: 'attached' | 'duplicate' | 'missing' | 'invalid_attach_request_id' = 'attached' await this.withTerminalLock(terminalId, async () => { const existingState = this.terminals.get(terminalId) @@ -116,14 +266,25 @@ export class TerminalStreamBroker { && (!hasOtherAttachedSockets || Boolean(existingAttachment)) ) + const terminalState = existingState ?? this.getOrCreateTerminalState(terminalId) if (shouldResize && !this.registry.resize(terminalId, cols, rows)) { this.registry.detach(terminalId, ws) result = 'missing' return } + if (shouldResize) { + this.recordTerminalGeometry( + terminalState, + cols, + rows, + hasOtherAttachedSockets ? 'multi_client_unknown' : 'single_client', + ) + } else if (hasOtherAttachedSockets) { + terminalState.geometryAuthority = 'multi_client_unknown' + } - const terminalState = existingState ?? this.getOrCreateTerminalState(terminalId) const attachment = existingAttachment ?? this.getOrCreateAttachment(terminalState, ws, terminalId) + attachment.terminalOutputBatchV1 = terminalOutputBatchV1 if (attachment.flushTimer) { clearTimeout(attachment.flushTimer) @@ -141,23 +302,37 @@ export class TerminalStreamBroker { if (terminalState.replayRing.headSeq() === 0) { const snapshot = record.buffer.snapshot() if (snapshot) { - terminalState.replayRing.append(snapshot) + this.appendOutputFrames(terminalId, snapshot) } } - const replay = terminalState.replayRing.replaySince(normalizedSinceSeq) + const streamId = this.streamIdentity.recordAttach(terminalId) + const replayResetReason = terminalState.geometryAuthority === 'multi_client_unknown' && normalizedSinceSeq > 0 + ? 'geometry_authority_unknown' as const + : undefined + const effectiveSinceSeq = replayResetReason ? 0 : normalizedSinceSeq + + const replay = terminalState.replayRing.replaySince(effectiveSinceSeq) let replayFrames = replay.frames let effectiveMissedFromSeq = replay.missedFromSeq let budgetTruncated = false const headSeq = terminalState.replayRing.headSeq() - // Truncate replay to tail frames within byte budget + // maxReplayBytes is a legacy protocol name; interpret it as serialized + // application JSON bytes for the terminal.output payloads we will send. if (maxReplayBytes !== undefined && maxReplayBytes > 0 && replayFrames.length > 0) { - let budgetRemaining = maxReplayBytes + const maxReplaySerializedApplicationJsonBytes = Math.floor(maxReplayBytes) + let budgetRemaining = maxReplaySerializedApplicationJsonBytes let keepFromIndex = replayFrames.length for (let i = replayFrames.length - 1; i >= 0; i--) { - if (replayFrames[i].bytes > budgetRemaining) break - budgetRemaining -= replayFrames[i].bytes + const frameSerializedApplicationJsonBytes = this.measureOutputFrameSerializedApplicationJsonBytes( + terminalId, + replayFrames[i], + attachment.activeAttachRequestId, + 'replay', + ) + if (frameSerializedApplicationJsonBytes > budgetRemaining) break + budgetRemaining -= frameSerializedApplicationJsonBytes keepFromIndex = i } if (keepFromIndex > 0) { @@ -171,6 +346,7 @@ export class TerminalStreamBroker { terminalId, connectionId: ws.connectionId, maxReplayBytes, + maxReplaySerializedApplicationJsonBytes, droppedFrames: keepFromIndex, droppedFromSeq: truncatedFromSeq, droppedToSeq: truncatedToSeq, @@ -179,6 +355,10 @@ export class TerminalStreamBroker { } } + const streamFilteredReplay = this.filterReplayFramesForStream(replayFrames, streamId) + replayFrames = streamFilteredReplay.frames + const skippedReplayGaps = streamFilteredReplay.skippedGaps + const replayFromSeq = replayFrames.length > 0 ? replayFrames[0].seqStart : headSeq + 1 const replayToSeq = replayFrames.length > 0 ? replayFrames[replayFrames.length - 1].seqEnd : headSeq @@ -186,7 +366,7 @@ export class TerminalStreamBroker { this.perfEventLogger('terminal_stream_replay_hit', { terminalId, connectionId: ws.connectionId, - sinceSeq: normalizedSinceSeq, + sinceSeq: effectiveSinceSeq, replayFromSeq, replayToSeq, replayFrameCount: replayFrames.length, @@ -197,6 +377,12 @@ export class TerminalStreamBroker { if (!this.safeSend(ws, { type: 'terminal.attach.ready', terminalId, + streamId, + geometryEpoch: terminalState.geometryEpoch, + geometryAuthority: terminalState.geometryAuthority, + requestedSinceSeq: normalizedSinceSeq, + effectiveSinceSeq, + ...(replayResetReason ? { replayResetReason } : {}), headSeq, replayFromSeq, replayToSeq, @@ -213,7 +399,7 @@ export class TerminalStreamBroker { this.perfEventLogger('terminal_stream_replay_miss', { terminalId, connectionId: ws.connectionId, - sinceSeq: normalizedSinceSeq, + sinceSeq: effectiveSinceSeq, missedFromSeq: effectiveMissedFromSeq, missedToSeq, replayFromSeq, @@ -228,27 +414,64 @@ export class TerminalStreamBroker { reason: gapReason, }, 'warn') - if (!this.safeSend(ws, { + const gapPayload = { type: 'terminal.output.gap', terminalId, + streamId, fromSeq: effectiveMissedFromSeq, toSeq: missedToSeq, reason: gapReason, ...(attachment.activeAttachRequestId ? { attachRequestId: attachment.activeAttachRequestId } : {}), - })) { + } + if (!this.safeSend(ws, gapPayload)) { return } + this.logTerminalReplayGap({ + terminalId, + connectionId: ws.connectionId, + attachRequestId: attachment.activeAttachRequestId, + streamId, + fromSeq: effectiveMissedFromSeq, + toSeq: missedToSeq, + reason: gapReason, + source: 'replay', + }) attachment.lastSeq = Math.max(attachment.lastSeq, missedToSeq) } } + const preReplayGapLimit = replayFrames.length > 0 ? replayFromSeq - 1 : headSeq + for (const gap of skippedReplayGaps) { + if (gap.fromSeq > preReplayGapLimit) continue + const fromSeq = Math.max(gap.fromSeq, attachment.lastSeq + 1) + const toSeq = Math.min(gap.toSeq, preReplayGapLimit) + if (toSeq < fromSeq) continue + if (!this.sendReplayGap( + ws, + terminalId, + fromSeq, + toSeq, + streamId, + attachment.activeAttachRequestId, + )) return + attachment.lastSeq = Math.max(attachment.lastSeq, toSeq) + } + const staged = attachment.attachStaging.filter((frame) => frame.seqStart > replayToSeq) attachment.attachStaging = [] attachment.replayCursor = replayFrames.length > 0 - ? { nextSeq: replayFromSeq, toSeq: replayToSeq } + ? { nextSeq: replayFromSeq, toSeq: replayToSeq, streamId } : null for (const frame of staged) { - attachment.queue.enqueue(frame) + attachment.queue.enqueue( + frame, + this.measureOutputFrameSerializedApplicationJsonBytes( + terminalId, + frame, + attachment.activeAttachRequestId, + 'live', + ), + ) } attachment.mode = 'live' @@ -278,6 +501,7 @@ export class TerminalStreamBroker { attachment.flushTimer = null } + this.streamIdentity.recordDetach(terminalId) state.clients.delete(ws) this.unregisterWsTerminal(ws, terminalId) this.registry.detach(terminalId, ws) @@ -297,6 +521,40 @@ export class TerminalStreamBroker { return this.terminals.get(terminalId)?.clients.size || 0 } + recordResize(terminalId: string, ws: LiveWebSocket, cols: number, rows: number): void { + const state = this.terminals.get(terminalId) + if (!state) return + const hasOtherAttachedSockets = [...state.clients.keys()].some((attachedWs) => attachedWs !== ws) + this.recordTerminalGeometry( + state, + cols, + rows, + hasOtherAttachedSockets ? 'multi_client_unknown' : 'single_client', + ) + } + + private recordTerminalGeometry( + state: BrokerTerminalState, + cols: number, + rows: number, + authority: TerminalGeometryAuthority, + ): void { + const normalizedCols = Math.max(2, Math.floor(Number.isFinite(cols) ? cols : 80)) + const normalizedRows = Math.max(2, Math.floor(Number.isFinite(rows) ? rows : 24)) + const hasPreviousGeometry = typeof state.geometryCols === 'number' + && typeof state.geometryRows === 'number' + const geometryChanged = !hasPreviousGeometry + ? false + : state.geometryCols !== normalizedCols || state.geometryRows !== normalizedRows + + if (geometryChanged) { + state.geometryEpoch += 1 + } + state.geometryCols = normalizedCols + state.geometryRows = normalizedRows + state.geometryAuthority = authority + } + private getOrCreateTerminalState(terminalId: string): BrokerTerminalState { const replayRingMaxBytes = this.resolveReplayRingMaxBytes(terminalId) let state = this.terminals.get(terminalId) @@ -304,10 +562,13 @@ export class TerminalStreamBroker { state = { replayRing: new ReplayRing(replayRingMaxBytes), clients: new Map(), + geometryEpoch: 1, + geometryAuthority: 'single_client', } this.terminals.set(terminalId, state) } else { state.replayRing.setMaxBytes(replayRingMaxBytes) + this.handleReplayRetentionLoss(terminalId, state, this.streamIdentity.ensureStream(terminalId)) } return state } @@ -362,6 +623,7 @@ export class TerminalStreamBroker { lastSeq: 0, flushTimer: null, catastrophicClosed: false, + terminalOutputBatchV1: false, } terminalState.clients.set(ws, attachment) this.registerWsTerminal(ws, terminalId) @@ -384,15 +646,65 @@ export class TerminalStreamBroker { private onTerminalOutputRaw(event: TerminalOutputRawEvent): void { const state = this.getOrCreateTerminalState(event.terminalId) - const frame = state.replayRing.append(event.data) + const frames = this.appendOutputFrames(event.terminalId, event.data) for (const attachment of state.clients.values()) { - if (attachment.mode === 'attaching') { - attachment.attachStaging.push(frame) - continue + for (const frame of frames) { + if (attachment.mode === 'attaching') { + attachment.attachStaging.push(frame) + continue + } + attachment.queue.enqueue( + frame, + this.measureOutputFrameSerializedApplicationJsonBytes( + event.terminalId, + frame, + attachment.activeAttachRequestId, + 'live', + ), + ) + } + if (frames.length > 0 && attachment.mode !== 'attaching') { + this.scheduleFlush(event.terminalId, attachment) + } + } + } + + private appendOutputFrames(terminalId: string, data: string): ReplayFrame[] { + const state = this.getOrCreateTerminalState(terminalId) + let streamId = this.streamIdentity.ensureStream(terminalId) + const fragments = fragmentTerminalOutputForPayloadBudget({ + data, + maxSerializedBytes: TERMINAL_STREAM_BATCH_MAX_BYTES, + payloadForData: (chunk) => this.buildTerminalOutputPayload({ + terminalId, + streamId, + seqStart: TERMINAL_STREAM_BUDGET_SEQ_PLACEHOLDER, + seqEnd: TERMINAL_STREAM_BUDGET_SEQ_PLACEHOLDER, + data: chunk, + attachRequestId: TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + source: 'replay', + }), + }) + const frames: ReplayFrame[] = [] + for (const fragment of fragments) { + const fragmentStreamId = streamId + frames.push(state.replayRing.append(fragment, { streamId })) + const retainedStreamId = this.handleReplayRetentionLoss(terminalId, state, fragmentStreamId) + if (retainedStreamId) { + this.retagFrames(frames, fragmentStreamId, retainedStreamId) + streamId = retainedStreamId + } + } + return frames + } + + private retagFrames(frames: ReplayFrame[], fromStreamId: string, toStreamId: string): void { + if (!fromStreamId || !toStreamId || fromStreamId === toStreamId) return + for (const frame of frames) { + if (frame.streamId === fromStreamId) { + frame.streamId = toStreamId } - attachment.queue.enqueue(frame) - this.scheduleFlush(event.terminalId, attachment) } } @@ -427,7 +739,7 @@ export class TerminalStreamBroker { return } - const wsBuffered = ws.bufferedAmount as number | undefined + const wsBuffered = readWebSocketBufferedAmount(ws) if ( attachment.priority === 'background' && typeof wsBuffered === 'number' @@ -444,46 +756,86 @@ export class TerminalStreamBroker { return } - const pendingBytes = attachment.queue.pendingBytes() - if (pendingBytes > TERMINAL_STREAM_BATCH_MAX_BYTES) { + const pendingSerializedApplicationJsonBytes = attachment.queue.pendingBytes() + if (pendingSerializedApplicationJsonBytes > TERMINAL_STREAM_BATCH_MAX_BYTES) { + const droppedSerializedApplicationJsonBytes = attachment.queue.peekDroppedBytes() this.perfEventLogger('terminal_stream_queue_pressure', { terminalId, connectionId: ws.connectionId, - pendingBytes, + pendingSerializedApplicationJsonBytes, + batchMaxSerializedApplicationJsonBytes: TERMINAL_STREAM_BATCH_MAX_BYTES, + pendingBytes: pendingSerializedApplicationJsonBytes, batchMaxBytes: TERMINAL_STREAM_BATCH_MAX_BYTES, bufferedAmount: ws.bufferedAmount, queueDepth: attachment.queue.pendingFrames(), - droppedBytes: attachment.queue.peekDroppedBytes(), + droppedSerializedApplicationJsonBytes, + droppedBytes: droppedSerializedApplicationJsonBytes, }, 'warn') } - const batch = attachment.queue.nextBatch(TERMINAL_STREAM_BATCH_MAX_BYTES) - if (batch.length === 0) return - const attachRequestId = attachment.activeAttachRequestId - for (const item of batch) { + const preparedBatch = attachment.queue.prepareBatch( + TERMINAL_STREAM_BATCH_MAX_BYTES, + (frame) => this.measureOutputFrameSerializedApplicationJsonBytes(terminalId, frame, attachRequestId, 'live'), + { terminalId, attachRequestId, source: 'live' }, + ) + if (preparedBatch.entries.length === 0) return + + let sentGaps = 0 + let sentFrames = 0 + const acknowledgeAndRetry = (reason?: SendJsonResult['reason']) => { + attachment.queue.acknowledgePreparedBatch(preparedBatch, { gaps: sentGaps, frames: sentFrames }) + if (reason === 'closed' || ws.readyState !== WebSocket.OPEN) { + this.detach(terminalId, ws) + return + } + if (this.hasPendingAttachmentOutput(attachment)) { + this.scheduleFlush(terminalId, attachment, TERMINAL_STREAM_RETRY_FLUSH_MS) + } + } + + for (const item of preparedBatch.entries) { if (isGapEvent(item)) { + const gapQueueContext = item.reason === 'queue_overflow' + ? { + queueDepth: attachment.queue.pendingFrames(), + droppedSerializedApplicationJsonBytes: attachment.queue.peekDroppedBytes(), + } + : undefined if (!this.sendGap( ws, terminalId, item, attachRequestId, - item.reason === 'queue_overflow' - ? { - queueDepth: attachment.queue.pendingFrames(), - droppedBytes: attachment.queue.consumeDroppedBytes(), - } - : undefined, - )) return + gapQueueContext, + )) { + acknowledgeAndRetry() + return + } + if (item.reason === 'queue_overflow') { + attachment.queue.consumeDroppedBytes() + } + sentGaps += 1 attachment.lastSeq = Math.max(attachment.lastSeq, item.toSeq) continue } - if (!this.sendFrame(ws, terminalId, item, attachRequestId)) return - attachment.lastSeq = Math.max(attachment.lastSeq, item.seqEnd) + const sendResult = this.sendFrame(ws, terminalId, item, attachRequestId, 'live', attachment.terminalOutputBatchV1) + if (sendResult.sentFrameCount > 0) { + sentFrames += sendResult.sentFrameCount + if (typeof sendResult.sentSeqEnd === 'number') { + attachment.lastSeq = Math.max(attachment.lastSeq, sendResult.sentSeqEnd) + } + } + if (sendResult.status !== 'sent') { + acknowledgeAndRetry(sendResult.reason) + return + } } - if (attachment.queue.pendingBytes() > 0) { + attachment.queue.acknowledgePreparedBatch(preparedBatch, { gaps: sentGaps, frames: sentFrames }) + + if (this.hasPendingAttachmentOutput(attachment)) { this.scheduleFlush(terminalId, attachment) } } @@ -503,6 +855,8 @@ export class TerminalStreamBroker { cursor.nextSeq - 1, TERMINAL_STREAM_BATCH_MAX_BYTES, cursor.toSeq, + (frame) => this.measureOutputFrameSerializedApplicationJsonBytes(terminalId, frame, attachRequestId, 'replay'), + { terminalId, attachRequestId, source: 'replay' }, ) if (replay.missedFromSeq !== undefined) { @@ -511,23 +865,66 @@ export class TerminalStreamBroker { : Math.min(cursor.toSeq + 1, terminalState.replayRing.headSeq() + 1) const missedToSeq = Math.min(cursor.toSeq, replayFromSeq - 1) if (missedToSeq >= replay.missedFromSeq) { - if (!this.sendReplayGap( - attachment.ws, + const gapSend = this.sendReplayGapWithPacing( terminalId, + attachment, replay.missedFromSeq, missedToSeq, + cursor.streamId, attachRequestId, - )) return - attachment.lastSeq = Math.max(attachment.lastSeq, missedToSeq) - cursor.nextSeq = missedToSeq + 1 + ) + if (gapSend.status !== 'sent') return + attachment.lastSeq = Math.max(attachment.lastSeq, gapSend.sentSeqEnd) + cursor.nextSeq = gapSend.sentSeqEnd + 1 + if (gapSend.pauseAfter) return } } + let skippedGap: ReplayGapRange | null = null + const flushSkippedGap = (): 'sent' | 'paused' | 'failed' | 'none' => { + if (!skippedGap) return 'none' + const gap = skippedGap + const gapSend = this.sendReplayGapWithPacing( + terminalId, + attachment, + gap.fromSeq, + gap.toSeq, + cursor.streamId, + attachRequestId, + ) + if (gapSend.status !== 'sent') return gapSend.status + skippedGap = null + attachment.lastSeq = Math.max(attachment.lastSeq, gap.toSeq) + cursor.nextSeq = gap.toSeq + 1 + return gapSend.pauseAfter ? 'paused' : 'sent' + } + for (const frame of replay.frames) { - if (!this.sendFrame(attachment.ws, terminalId, frame, attachRequestId)) return - attachment.lastSeq = Math.max(attachment.lastSeq, frame.seqEnd) - cursor.nextSeq = frame.seqEnd + 1 + if (frame.streamId !== cursor.streamId) { + if (!skippedGap || frame.seqStart > skippedGap.toSeq + 1) { + const gapResult = flushSkippedGap() + if (gapResult === 'paused' || gapResult === 'failed') return + skippedGap = { fromSeq: frame.seqStart, toSeq: frame.seqEnd } + } else { + skippedGap.toSeq = Math.max(skippedGap.toSeq, frame.seqEnd) + } + continue + } + const gapResult = flushSkippedGap() + if (gapResult === 'paused' || gapResult === 'failed') return + const frameSend = this.sendReplayFrameWithPacing( + terminalId, + attachment, + frame, + attachRequestId, + ) + if (frameSend.status !== 'sent') return + attachment.lastSeq = Math.max(attachment.lastSeq, frameSend.sentSeqEnd) + cursor.nextSeq = frameSend.sentSeqEnd + 1 + if (frameSend.pauseAfter) return } + const gapResult = flushSkippedGap() + if (gapResult === 'paused' || gapResult === 'failed') return if (cursor.nextSeq > cursor.toSeq || replay.frames.length === 0) { attachment.replayCursor = null @@ -542,7 +939,31 @@ export class TerminalStreamBroker { } private hasPendingAttachmentOutput(attachment: BrokerClientAttachment): boolean { - return Boolean(attachment.replayCursor) || attachment.queue.pendingBytes() > 0 + return Boolean(attachment.replayCursor) || attachment.queue.hasPendingEntries() + } + + private filterReplayFramesForStream( + frames: ReplayFrame[], + streamId: string, + ): { frames: ReplayFrame[]; skippedGaps: ReplayGapRange[] } { + const keptFrames: ReplayFrame[] = [] + const skippedGaps: ReplayGapRange[] = [] + + for (const frame of frames) { + if (frame.streamId === streamId) { + keptFrames.push(frame) + continue + } + + const lastGap = skippedGaps[skippedGaps.length - 1] + if (!lastGap || frame.seqStart > lastGap.toSeq + 1) { + skippedGaps.push({ fromSeq: frame.seqStart, toSeq: frame.seqEnd }) + } else { + lastGap.toSeq = Math.max(lastGap.toSeq, frame.seqEnd) + } + } + + return { frames: keptFrames, skippedGaps } } private catastrophicBlocked(terminalId: string, attachment: BrokerClientAttachment): boolean { @@ -590,20 +1011,539 @@ export class TerminalStreamBroker { return true } + private logTerminalReplayBatch(input: { + terminalId: string + attachRequestId?: string + source: 'live' | 'replay' + payload: JsonPayload + result: SendJsonResult + batch?: TerminalOutputBatch + envelopeIndex: number + envelopeCount: number + }): void { + if (input.source !== 'replay') return + + const seqStart = jsonNumberField(input.payload, 'seqStart') ?? input.batch?.seqStart + const seqEnd = jsonNumberField(input.payload, 'seqEnd') ?? input.batch?.seqEnd + const streamId = jsonStringField(input.payload, 'streamId') ?? input.batch?.streamId + const payloadType = jsonStringField(input.payload, 'type') ?? input.result.messageType ?? 'unknown' + const rawFrameCount = payloadRawFrameCount(input.payload) + const logicalRawFrameCount = input.batch ? batchRawFrameCount(input.batch) : rawFrameCount + const data = typeof input.payload.data === 'string' ? input.payload.data : '' + const serializedBytes = input.result.serializedApplicationJsonBytes + ?? jsonNumberField(input.payload, 'serializedBytes') + ?? measureTerminalOutputPayloadBytes(input.payload) + const bufferedAmount = input.result.bufferedAfter ?? input.result.bufferedBefore + log.debug({ + event: 'terminal.replay.batch', + severity: 'debug', + terminalId: input.terminalId, + source: input.source, + payloadType, + ...(input.attachRequestId ? { attachRequestId: input.attachRequestId } : {}), + ...(streamId ? { streamId } : {}), + ...(seqStart !== undefined ? { seqStart } : {}), + ...(seqEnd !== undefined ? { seqEnd } : {}), + rawFrameCount, + dataBytes: Buffer.byteLength(data, 'utf8'), + serializedBytes, + ...(typeof bufferedAmount === 'number' ? { bufferedAmount } : {}), + envelopeIndex: input.envelopeIndex, + envelopeCount: input.envelopeCount, + envelopeSplit: input.envelopeCount > 1, + ...(input.batch + ? { + logicalSeqStart: input.batch.seqStart, + logicalSeqEnd: input.batch.seqEnd, + logicalRawFrameCount, + legacyOutputSerializedBytes: input.batch.legacyOutputSerializedBytes, + } + : {}), + }, 'Terminal replay batch payload sent') + } + + private logTerminalReplayGap(input: { + terminalId: string + attachRequestId?: string + streamId?: string + source?: 'live' | 'replay' + fromSeq: number + toSeq: number + reason: ReplayGapReason + connectionId?: string + queueDepth?: number + droppedSerializedApplicationJsonBytes?: number + }): void { + const event = input.source === 'replay' ? 'terminal.replay.gap' : 'terminal.output.gap' + log.warn({ + event, + severity: 'warn', + terminalId: input.terminalId, + fromSeq: input.fromSeq, + toSeq: input.toSeq, + reason: input.reason, + ...(input.attachRequestId ? { attachRequestId: input.attachRequestId } : {}), + ...(input.streamId ? { streamId: input.streamId } : {}), + ...(input.source ? { source: input.source } : {}), + ...(input.connectionId ? { connectionId: input.connectionId } : {}), + ...(typeof input.queueDepth === 'number' ? { queueDepth: input.queueDepth } : {}), + ...(typeof input.droppedSerializedApplicationJsonBytes === 'number' + ? { + droppedSerializedApplicationJsonBytes: input.droppedSerializedApplicationJsonBytes, + droppedBytes: input.droppedSerializedApplicationJsonBytes, + } + : {}), + }, input.source === 'replay' ? 'Terminal replay gap emitted' : 'Terminal output gap emitted') + } + + private logTerminalReplayRetention(input: { + terminalId: string + streamId: string + previousStreamId?: string + attachRequestIds: string[] + attachmentCount: number + reason: TerminalStreamReplacementReason + retainedBytes: number + maxBytes: number + tailSeq: number + headSeq: number + suppressedCount?: number + }): void { + const basePayload = { + event: 'terminal.replay.retention', + severity: 'warn', + terminalId: input.terminalId, + streamId: input.streamId, + ...(input.previousStreamId ? { previousStreamId: input.previousStreamId } : {}), + attachRequestIds: input.attachRequestIds, + attachmentCount: input.attachmentCount, + reason: input.reason, + retainedBytes: input.retainedBytes, + maxBytes: input.maxBytes, + tailSeq: input.tailSeq, + headSeq: input.headSeq, + ...(typeof input.suppressedCount === 'number' && input.suppressedCount > 0 + ? { suppressedCount: input.suppressedCount } + : {}), + } + log.warn(basePayload, 'Terminal replay retention loss changed stream identity') + } + private sendFrame( ws: LiveWebSocket, terminalId: string, - frame: ReplayFrame, + frame: ReplayFrame | TerminalOutputBatch, attachRequestId?: string, - ): boolean { - return this.safeSend(ws, { + source: 'live' | 'replay' = 'live', + terminalOutputBatchV1 = false, + ): LiveSendOutcome { + if (this.isTerminalOutputBatch(frame)) { + if (terminalOutputBatchV1 && attachRequestId) { + const payloads = this.buildTerminalOutputBatchPayloads({ + terminalId, + batch: frame, + attachRequestId, + source, + }) + for (let index = 0; index < payloads.length; index += 1) { + const payload = payloads[index] + const prepared = this.prepareSendPayload(payload) + if (!prepared) return { status: 'failed', sentFrameCount: this.countSentBatchPayloadFrames(payloads, index) } + const result = this.safeSendPrepared(ws, prepared) + if (!result.sent) { + const sentFrameCount = this.countSentBatchPayloadFrames(payloads, index) + return { + status: 'failed', + sentFrameCount, + ...(sentFrameCount > 0 ? { sentSeqEnd: this.payloadSeqEnd(payloads[index - 1]) } : {}), + reason: result.reason, + } + } + this.logTerminalReplayBatch({ + terminalId, + attachRequestId, + source, + payload, + result, + batch: frame, + envelopeIndex: index + 1, + envelopeCount: payloads.length, + }) + } + return { + status: 'sent', + sentFrameCount: this.countSentBatchPayloadFrames(payloads, payloads.length), + sentSeqEnd: frame.seqEnd, + } + } + return this.sendLegacyOutputSegments(ws, terminalId, frame, attachRequestId) + } + + const result = sendJsonMessage(ws, this.buildTerminalOutputPayload({ type: 'terminal.output', terminalId, + streamId: frame.streamId, seqStart: frame.seqStart, seqEnd: frame.seqEnd, data: frame.data, - ...(attachRequestId ? { attachRequestId } : {}), + attachRequestId, + source, + })) + if (!result.sent) { + return { status: 'failed', sentFrameCount: 0, reason: result.reason } + } + return { status: 'sent', sentFrameCount: 1, sentSeqEnd: frame.seqEnd } + } + + private isTerminalOutputBatch(frame: ReplayFrame | TerminalOutputBatch): frame is TerminalOutputBatch { + return Array.isArray((frame as Partial).segments) + } + + private countSentBatchPayloadFrames(payloads: JsonPayload[], sentPayloadCount: number): number { + return payloads + .slice(0, Math.max(0, sentPayloadCount)) + .reduce((sum, payload) => sum + payloadRawFrameCount(payload), 0) + } + + private payloadSeqEnd(payload: JsonPayload | undefined): number | undefined { + return typeof payload?.seqEnd === 'number' ? payload.seqEnd : undefined + } + + private buildTerminalOutputBatchPayloads(input: { + terminalId: string + batch: TerminalOutputBatch + attachRequestId: string + source: 'live' | 'replay' + }): JsonPayload[] { + const fullPayload = this.buildTerminalOutputBatchPayload(input, 0, input.batch.segments.length) + const fullPayloadBytes = typeof fullPayload.serializedBytes === 'number' + ? fullPayload.serializedBytes + : Number.POSITIVE_INFINITY + if (fullPayloadBytes <= TERMINAL_STREAM_BATCH_MAX_BYTES) { + return [fullPayload] + } + if (input.batch.segments.length <= 1) { + return this.buildTerminalOutputBatchSingleSegmentFallbackPayloads(input, 0) + } + + const payloads: JsonPayload[] = [] + let startIndex = 0 + while (startIndex < input.batch.segments.length) { + let endIndex = startIndex + 1 + let currentPayload = this.buildTerminalOutputBatchPayload(input, startIndex, endIndex) + const currentPayloadBytes = typeof currentPayload.serializedBytes === 'number' + ? currentPayload.serializedBytes + : Number.POSITIVE_INFINITY + if (currentPayloadBytes > TERMINAL_STREAM_BATCH_MAX_BYTES) { + payloads.push(...this.buildTerminalOutputBatchSingleSegmentFallbackPayloads(input, startIndex)) + startIndex = endIndex + continue + } + + while (endIndex < input.batch.segments.length) { + const candidate = this.buildTerminalOutputBatchPayload(input, startIndex, endIndex + 1) + const candidateBytes = typeof candidate.serializedBytes === 'number' + ? candidate.serializedBytes + : Number.POSITIVE_INFINITY + if (candidateBytes > TERMINAL_STREAM_BATCH_MAX_BYTES) break + currentPayload = candidate + endIndex += 1 + } + + payloads.push(currentPayload) + startIndex = endIndex + } + + return payloads + } + + private buildTerminalOutputBatchSingleSegmentFallbackPayloads( + input: { + terminalId: string + batch: TerminalOutputBatch + attachRequestId: string + source: 'live' | 'replay' + }, + segmentIndex: number, + ): JsonPayload[] { + const segment = input.batch.segments[segmentIndex] + if (!segment) return [] + const startOffset = segmentIndex === 0 + ? 0 + : input.batch.segments[segmentIndex - 1]?.endOffset ?? 0 + const endOffset = Math.max(startOffset, Math.floor(segment.endOffset)) + return [this.buildTerminalOutputPayload({ + type: 'terminal.output', + terminalId: input.terminalId, + streamId: input.batch.streamId, + seqStart: segment.seqStart, + seqEnd: segment.seqEnd, + data: input.batch.data.slice(startOffset, endOffset), + attachRequestId: input.attachRequestId, + source: input.source, + })] + } + + private buildTerminalOutputBatchPayload( + input: { + terminalId: string + batch: TerminalOutputBatch + attachRequestId: string + source: 'live' | 'replay' + }, + startSegmentIndex: number, + endSegmentIndex: number, + ): JsonPayload { + const firstSegment = input.batch.segments[startSegmentIndex] + const lastSegment = input.batch.segments[endSegmentIndex - 1] + const startOffset = startSegmentIndex === 0 + ? 0 + : input.batch.segments[startSegmentIndex - 1]?.endOffset ?? 0 + const endOffset = lastSegment?.endOffset ?? startOffset + const basePayload = { + type: 'terminal.output.batch', + terminalId: input.terminalId, + streamId: input.batch.streamId, + attachRequestId: input.attachRequestId, + source: input.source, + seqStart: firstSegment?.seqStart ?? input.batch.seqStart, + seqEnd: lastSegment?.seqEnd ?? input.batch.seqEnd, + data: input.batch.data.slice(startOffset, endOffset), + serializedBytes: 0, + segments: this.buildTerminalOutputBatchWireSegments( + input.batch, + startSegmentIndex, + endSegmentIndex, + startOffset, + ), + } + + let serializedBytes = 0 + for (let attempt = 0; attempt < 4; attempt += 1) { + const measured = measureTerminalOutputPayloadBytes({ + ...basePayload, + serializedBytes, + }) + if (measured === serializedBytes) break + serializedBytes = measured + } + + return { + ...basePayload, + serializedBytes, + } + } + + private buildTerminalOutputBatchWireSegments( + batch: TerminalOutputBatch, + startSegmentIndex: number, + endSegmentIndex: number, + baseOffset: number, + ): JsonPayload[] { + let previousEndOffset = 0 + return batch.segments.slice(startSegmentIndex, endSegmentIndex).map((segment) => { + const relativeEndOffset = Math.max(previousEndOffset, Math.floor(segment.endOffset - baseOffset)) + previousEndOffset = relativeEndOffset + return { + seqStart: segment.seqStart, + seqEnd: segment.seqEnd, + endOffset: relativeEndOffset, + rawFrameCount: Math.max(1, segment.seqEnd - segment.seqStart + 1), + ...(segment.barrier && segment.barrierReason ? { barrier: segment.barrierReason } : {}), + } + }) + } + + private buildLegacyOutputSegmentPayloads( + terminalId: string, + batch: TerminalOutputBatch, + attachRequestId?: string, + source?: 'live' | 'replay', + ): JsonPayload[] { + const payloads: JsonPayload[] = [] + let previousEndOffset = 0 + for (const segment of batch.segments) { + const endOffset = Math.max(previousEndOffset, Math.floor(segment.endOffset)) + const data = batch.data.slice(previousEndOffset, endOffset) + previousEndOffset = endOffset + payloads.push(this.buildTerminalOutputPayload({ + type: 'terminal.output', + terminalId, + streamId: batch.streamId, + seqStart: segment.seqStart, + seqEnd: segment.seqEnd, + data, + attachRequestId, + source, + })) + } + return payloads + } + + private sendLegacyOutputSegments( + ws: LiveWebSocket, + terminalId: string, + batch: TerminalOutputBatch, + attachRequestId?: string, + ): LiveSendOutcome { + const source = batch.source === 'replay' ? 'replay' : 'live' + const payloads = this.buildLegacyOutputSegmentPayloads(terminalId, batch, attachRequestId, source) + for (let index = 0; index < payloads.length; index += 1) { + const payload = payloads[index] + const prepared = this.prepareSendPayload(payload) + if (!prepared) { + return { status: 'failed', sentFrameCount: index, ...(index > 0 ? { sentSeqEnd: this.payloadSeqEnd(payloads[index - 1]) } : {}) } + } + const result = this.safeSendPrepared(ws, prepared) + if (!result.sent) { + return { + status: 'failed', + sentFrameCount: index, + ...(index > 0 ? { sentSeqEnd: this.payloadSeqEnd(payloads[index - 1]) } : {}), + reason: result.reason, + } + } + this.logTerminalReplayBatch({ + terminalId, + attachRequestId, + source, + payload, + result, + batch, + envelopeIndex: index + 1, + envelopeCount: payloads.length, + }) + } + return { status: 'sent', sentFrameCount: payloads.length, sentSeqEnd: batch.seqEnd } + } + + private sendLegacyOutputSegmentsWithPacing( + terminalId: string, + attachment: BrokerClientAttachment, + batch: TerminalOutputBatch, + attachRequestId?: string, + ): ReplaySendOutcome { + let sentSeqEnd = attachment.lastSeq + const source = batch.source === 'live' ? 'live' : 'replay' + const payloads = this.buildLegacyOutputSegmentPayloads(terminalId, batch, attachRequestId, source) + for (let index = 0; index < payloads.length; index += 1) { + const payload = payloads[index] + const prepared = this.prepareSendPayload(payload) + if (!prepared) return { status: 'failed' } + const payloadSeqEnd = typeof payload.seqEnd === 'number' ? payload.seqEnd : sentSeqEnd + const result = this.sendPreparedReplayPayloadWithPacing( + terminalId, + attachment, + prepared, + payloadSeqEnd, + { + backpressureFields: payloadBackpressureFields(payload), + onSent: (sendResult) => this.logTerminalReplayBatch({ + terminalId, + attachRequestId, + source, + payload, + result: sendResult, + batch, + envelopeIndex: index + 1, + envelopeCount: payloads.length, + }), + }, + ) + if (result.status !== 'sent') return result + sentSeqEnd = result.sentSeqEnd + if (result.pauseAfter) return result + } + return { + status: 'sent', + pauseAfter: false, + sentSeqEnd, + } + } + + private sendReplayFrameWithPacing( + terminalId: string, + attachment: BrokerClientAttachment, + frame: ReplayFrame | TerminalOutputBatch, + attachRequestId?: string, + ): ReplaySendOutcome { + if (this.isTerminalOutputBatch(frame)) { + if (attachment.terminalOutputBatchV1 && attachRequestId) { + return this.sendBatchPayloadsWithPacing({ + terminalId, + attachment, + batch: frame, + attachRequestId, + source: 'replay', + }) + } + + return this.sendLegacyOutputSegmentsWithPacing(terminalId, attachment, frame, attachRequestId) + } + + const payload = this.buildTerminalOutputPayload({ + type: 'terminal.output', + terminalId, + streamId: frame.streamId, + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + data: frame.data, + attachRequestId, + source: 'replay', }) + const prepared = this.prepareSendPayload(payload) + if (!prepared) return { status: 'failed' } + return this.sendPreparedReplayPayloadWithPacing( + terminalId, + attachment, + prepared, + frame.seqEnd, + { backpressureFields: payloadBackpressureFields(payload) }, + ) + } + + private sendBatchPayloadsWithPacing(input: { + terminalId: string + attachment: BrokerClientAttachment + batch: TerminalOutputBatch + attachRequestId: string + source: 'live' | 'replay' + }): ReplaySendOutcome { + let sentSeqEnd = input.attachment.lastSeq + const payloads = this.buildTerminalOutputBatchPayloads(input) + for (let index = 0; index < payloads.length; index += 1) { + const payload = payloads[index] + const prepared = this.prepareSendPayload(payload) + if (!prepared) return { status: 'failed' } + const payloadSeqEnd = typeof payload.seqEnd === 'number' ? payload.seqEnd : sentSeqEnd + const result = this.sendPreparedReplayPayloadWithPacing( + input.terminalId, + input.attachment, + prepared, + payloadSeqEnd, + { + backpressureFields: payloadBackpressureFields(payload), + onSent: (sendResult) => this.logTerminalReplayBatch({ + terminalId: input.terminalId, + attachRequestId: input.attachRequestId, + source: input.source, + payload, + result: sendResult, + batch: input.batch, + envelopeIndex: index + 1, + envelopeCount: payloads.length, + }), + }, + ) + if (result.status !== 'sent') return result + sentSeqEnd = result.sentSeqEnd + if (result.pauseAfter) return result + } + return { + status: 'sent', + pauseAfter: false, + sentSeqEnd, + } } private sendGap( @@ -611,26 +1551,56 @@ export class TerminalStreamBroker { terminalId: string, gap: GapEvent, attachRequestId?: string, - queueContext?: { queueDepth?: number; droppedBytes?: number }, + queueContext?: { + queueDepth?: number + droppedBytes?: number + droppedSerializedApplicationJsonBytes?: number + }, ): boolean { + const droppedSerializedApplicationJsonBytes = queueContext?.droppedSerializedApplicationJsonBytes + ?? queueContext?.droppedBytes this.perfEventLogger('terminal_stream_gap', { terminalId, connectionId: ws.connectionId, fromSeq: gap.fromSeq, toSeq: gap.toSeq, + streamId: gap.streamId, reason: gap.reason, ...(typeof queueContext?.queueDepth === 'number' ? { queueDepth: queueContext.queueDepth } : {}), - ...(typeof queueContext?.droppedBytes === 'number' ? { droppedBytes: queueContext.droppedBytes } : {}), + ...(typeof droppedSerializedApplicationJsonBytes === 'number' + ? { + droppedSerializedApplicationJsonBytes, + droppedBytes: droppedSerializedApplicationJsonBytes, + } + : {}), }, gap.reason === 'queue_overflow' ? 'warn' : 'info') - return this.safeSend(ws, { + const sent = this.safeSend(ws, { type: 'terminal.output.gap', terminalId, + streamId: gap.streamId, fromSeq: gap.fromSeq, toSeq: gap.toSeq, reason: gap.reason, ...(attachRequestId ? { attachRequestId } : {}), }) + if (sent) { + this.logTerminalReplayGap({ + terminalId, + connectionId: ws.connectionId, + attachRequestId, + streamId: gap.streamId, + source: 'live', + fromSeq: gap.fromSeq, + toSeq: gap.toSeq, + reason: gap.reason, + ...(typeof queueContext?.queueDepth === 'number' ? { queueDepth: queueContext.queueDepth } : {}), + ...(typeof droppedSerializedApplicationJsonBytes === 'number' + ? { droppedSerializedApplicationJsonBytes } + : {}), + }) + } + return sent } private sendReplayGap( @@ -638,6 +1608,7 @@ export class TerminalStreamBroker { terminalId: string, fromSeq: number, toSeq: number, + streamId: string, attachRequestId?: string, ): boolean { this.perfEventLogger('terminal_stream_gap', { @@ -645,38 +1616,433 @@ export class TerminalStreamBroker { connectionId: ws.connectionId, fromSeq, toSeq, + streamId, reason: 'replay_window_exceeded', }, 'warn') - return this.safeSend(ws, { + const sent = this.safeSend(ws, { type: 'terminal.output.gap', terminalId, + streamId, fromSeq, toSeq, reason: 'replay_window_exceeded', ...(attachRequestId ? { attachRequestId } : {}), }) + if (sent) { + this.logTerminalReplayGap({ + terminalId, + connectionId: ws.connectionId, + attachRequestId, + streamId, + fromSeq, + toSeq, + reason: 'replay_window_exceeded', + source: 'replay', + }) + } + return sent } - private safeSend(ws: LiveWebSocket, msg: unknown): boolean { - if (ws.readyState !== WebSocket.OPEN) return false - try { - ws.send(JSON.stringify(msg)) - return true - } catch { + private sendReplayGapWithPacing( + terminalId: string, + attachment: BrokerClientAttachment, + fromSeq: number, + toSeq: number, + streamId: string, + attachRequestId?: string, + ): ReplaySendOutcome { + const payload = { + type: 'terminal.output.gap', + terminalId, + streamId, + fromSeq, + toSeq, + reason: 'replay_window_exceeded', + ...(attachRequestId ? { attachRequestId } : {}), + } + const backpressureFields = payloadBackpressureFields(payload) + const prepared = this.prepareSendPayload(payload) + if (!prepared) return { status: 'failed' } + if (this.shouldPauseReplayBeforeSend(terminalId, attachment, prepared, backpressureFields)) { + return { status: 'paused' } + } + + this.perfEventLogger('terminal_stream_gap', { + terminalId, + connectionId: attachment.ws.connectionId, + fromSeq, + toSeq, + streamId, + reason: 'replay_window_exceeded', + }, 'warn') + + const result = this.safeSendPrepared(attachment.ws, prepared) + if (!result.sent) return { status: 'failed' } + this.logTerminalReplayGap({ + terminalId, + connectionId: attachment.ws.connectionId, + attachRequestId, + streamId, + fromSeq, + toSeq, + reason: 'replay_window_exceeded', + source: 'replay', + }) + return { + status: 'sent', + pauseAfter: this.shouldPauseReplayAfterSend(terminalId, attachment, result, backpressureFields), + sentSeqEnd: toSeq, + } + } + + private sendPreparedReplayPayloadWithPacing( + terminalId: string, + attachment: BrokerClientAttachment, + prepared: PreparedJsonMessage, + sentSeqEnd: number, + options?: { + backpressureFields?: ReplayBackpressurePayloadFields + onSent?: (result: SendJsonResult) => void + }, + ): ReplaySendOutcome { + if (this.shouldPauseReplayBeforeSend(terminalId, attachment, prepared, options?.backpressureFields)) { + return { status: 'paused' } + } + const result = this.safeSendPrepared(attachment.ws, prepared) + if (!result.sent) return { status: 'failed' } + options?.onSent?.(result) + return { + status: 'sent', + pauseAfter: this.shouldPauseReplayAfterSend(terminalId, attachment, result, options?.backpressureFields), + sentSeqEnd, + } + } + + private shouldPauseReplayBeforeSend( + terminalId: string, + attachment: BrokerClientAttachment, + prepared: PreparedJsonMessage, + backpressureFields?: ReplayBackpressurePayloadFields, + ): boolean { + const buffered = readWebSocketBufferedAmount(attachment.ws) + if (typeof buffered !== 'number') return false + const threshold = this.replayBufferedPauseThreshold(attachment) + const projectedBufferedAmount = buffered + prepared.serializedApplicationJsonBytes + if (projectedBufferedAmount <= threshold) { + this.resetReplayBackpressureLogState(attachment) return false } + this.pauseReplayForBackpressure(terminalId, attachment, { + bufferedAmount: buffered, + projectedBufferedAmount, + threshold, + serializedApplicationJsonBytes: prepared.serializedApplicationJsonBytes, + phase: 'before_send', + ...(backpressureFields ?? {}), + }) + return true + } + + private shouldPauseReplayAfterSend( + terminalId: string, + attachment: BrokerClientAttachment, + result: SendJsonResult, + backpressureFields?: ReplayBackpressurePayloadFields, + ): boolean { + const buffered = result.bufferedAfter + if (typeof buffered !== 'number') return false + const threshold = this.replayBufferedPauseThreshold(attachment) + if (buffered <= threshold) { + this.resetReplayBackpressureLogState(attachment) + return false + } + this.pauseReplayForBackpressure(terminalId, attachment, { + bufferedAmount: buffered, + threshold, + serializedApplicationJsonBytes: result.serializedApplicationJsonBytes, + phase: 'after_send', + ...(backpressureFields ?? {}), + }) + return true + } + + private replayBufferedPauseThreshold(attachment: BrokerClientAttachment): number { + return attachment.priority === 'background' + ? TERMINAL_BACKGROUND_BUFFERED_PAUSE_BYTES + : TERMINAL_FOREGROUND_REPLAY_BUFFERED_PAUSE_BYTES + } + + private replayBufferedPauseDelayMs(attachment: BrokerClientAttachment): number { + return attachment.priority === 'background' + ? TERMINAL_BACKGROUND_RETRY_FLUSH_MS + : TERMINAL_STREAM_RETRY_FLUSH_MS + } + + private pauseReplayForBackpressure( + terminalId: string, + attachment: BrokerClientAttachment, + context: { + bufferedAmount: number + threshold: number + serializedApplicationJsonBytes?: number + projectedBufferedAmount?: number + phase: 'before_send' | 'after_send' + seqStart?: number + seqEnd?: number + rawFrameCount?: number + dataBytes?: number + }, + ): void { + const retryMs = this.replayBufferedPauseDelayMs(attachment) + const now = Date.now() + const lastLogAt = attachment.replayBackpressureLogLastAt + if ( + typeof lastLogAt === 'number' + && now - lastLogAt < TERMINAL_REPLAY_BACKPRESSURE_LOG_RATE_LIMIT_MS + ) { + attachment.replayBackpressureLogSuppressed = (attachment.replayBackpressureLogSuppressed ?? 0) + 1 + this.scheduleFlush(terminalId, attachment, retryMs) + return + } + const suppressedCount = attachment.replayBackpressureLogSuppressed ?? 0 + attachment.replayBackpressureLogLastAt = now + attachment.replayBackpressureLogSuppressed = 0 + log.debug({ + event: 'terminal.replay.backpressure_pause', + severity: 'debug', + terminalId, + source: 'replay', + reason: 'websocket_buffered_amount', + connectionId: attachment.ws.connectionId, + priority: attachment.priority, + retryMs, + ...(attachment.activeAttachRequestId ? { attachRequestId: attachment.activeAttachRequestId } : {}), + ...(attachment.replayCursor?.streamId ? { streamId: attachment.replayCursor.streamId } : {}), + ...(suppressedCount > 0 ? { suppressedCount } : {}), + ...(typeof attachment.replayCursor?.nextSeq === 'number' + ? { nextSeq: attachment.replayCursor.nextSeq } + : {}), + ...(typeof attachment.replayCursor?.toSeq === 'number' + ? { toSeq: attachment.replayCursor.toSeq } + : {}), + ...(typeof context.seqStart === 'number' ? { seqStart: context.seqStart } : {}), + ...(typeof context.seqEnd === 'number' ? { seqEnd: context.seqEnd } : {}), + ...(typeof context.rawFrameCount === 'number' ? { rawFrameCount: context.rawFrameCount } : {}), + ...(typeof context.dataBytes === 'number' ? { dataBytes: context.dataBytes } : {}), + bufferedAmount: context.bufferedAmount, + ...(typeof context.projectedBufferedAmount === 'number' + ? { projectedBufferedAmount: context.projectedBufferedAmount } + : {}), + threshold: context.threshold, + ...(typeof context.serializedApplicationJsonBytes === 'number' + ? { + serializedBytes: context.serializedApplicationJsonBytes, + serializedApplicationJsonBytes: context.serializedApplicationJsonBytes, + } + : {}), + phase: context.phase, + }, 'Terminal replay paused for websocket backpressure') + log.debug({ + event: 'terminal_stream_replay_backpressure_pause', + terminalId, + connectionId: attachment.ws.connectionId, + priority: attachment.priority, + retryMs, + ...(suppressedCount > 0 ? { suppressedCount } : {}), + ...context, + }, 'Terminal replay paused for websocket backpressure') + this.scheduleFlush(terminalId, attachment, retryMs) + } + + private resetReplayBackpressureLogState(attachment: BrokerClientAttachment): void { + attachment.replayBackpressureLogLastAt = undefined + attachment.replayBackpressureLogSuppressed = 0 + } + + private prepareSendPayload(payload: unknown): PreparedJsonMessage | null { + try { + return prepareJsonMessage(payload) + } catch (error) { + log.warn({ + err: error instanceof Error ? error : new Error(String(error)), + }, 'WebSocket message serialization failed') + return null + } + } + + private safeSendPrepared(ws: LiveWebSocket, prepared: PreparedJsonMessage): SendJsonResult { + return sendPreparedJsonMessage(ws, prepared) + } + + private safeSend(ws: LiveWebSocket, msg: unknown): boolean { + return sendJsonMessage(ws, msg).sent } private handleTerminalExit(terminalId: string): void { const state = this.terminals.get(terminalId) - if (!state) return + if (!state) { + this.streamIdentity.forgetStream(terminalId) + return + } for (const attachment of state.clients.values()) { if (attachment.flushTimer) clearTimeout(attachment.flushTimer) this.unregisterWsTerminal(attachment.ws, terminalId) } state.clients.clear() this.terminals.delete(terminalId) + this.streamIdentity.forgetStream(terminalId) + } + + private buildTerminalOutputPayload(input: { + type?: 'terminal.output' + terminalId: string + streamId: string + seqStart: number + seqEnd: number + data: string + attachRequestId?: string + source?: 'live' | 'replay' + }): JsonPayload { + return { + type: input.type ?? 'terminal.output', + terminalId: input.terminalId, + streamId: input.streamId, + seqStart: input.seqStart, + seqEnd: input.seqEnd, + data: input.data, + ...(input.attachRequestId ? { attachRequestId: input.attachRequestId } : {}), + ...(input.source ? { source: input.source } : {}), + } + } + + private measureOutputFrameSerializedApplicationJsonBytes( + terminalId: string, + frame: ReplayFrame, + attachRequestId?: string, + source?: 'live' | 'replay', + ): number { + return measureTerminalOutputPayloadBytes(this.buildTerminalOutputPayload({ + terminalId, + streamId: frame.streamId, + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + data: frame.data, + attachRequestId, + source, + })) + } + + private replaceStreamIdentity(terminalId: string, reason: TerminalStreamReplacementReason): string { + const previousStreamId = this.streamIdentity.getStream(terminalId) + const streamId = this.streamIdentity.replaceStream(terminalId, reason) + const state = this.terminals.get(terminalId) + if (state) { + for (const attachment of state.clients.values()) { + if (previousStreamId && reason === 'retention_lost') { + attachment.queue.retagPendingStream(previousStreamId, streamId) + for (const frame of attachment.attachStaging) { + if (frame.streamId === previousStreamId) { + frame.streamId = streamId + } + } + } + this.sendStreamChanged( + attachment.ws, + terminalId, + streamId, + reason, + attachment.activeAttachRequestId, + ) + this.convertReplayCursorToCurrentStreamGap(terminalId, attachment, streamId) + } + } + log.info({ + terminalId, + streamId, + reason, + }, 'Terminal output stream identity replaced') + return streamId + } + + private sendStreamChanged( + ws: LiveWebSocket, + terminalId: string, + streamId: string, + reason: TerminalStreamReplacementReason, + attachRequestId?: string, + ): boolean { + return this.safeSend(ws, { + type: 'terminal.stream.changed', + terminalId, + streamId, + reason, + ...(attachRequestId ? { attachRequestId } : {}), + }) + } + + private convertReplayCursorToCurrentStreamGap( + terminalId: string, + attachment: BrokerClientAttachment, + streamId: string, + ): void { + const cursor = attachment.replayCursor + if (!cursor) return + + attachment.replayCursor = null + const fromSeq = Math.max(cursor.nextSeq, attachment.lastSeq + 1) + const toSeq = cursor.toSeq + if (toSeq < fromSeq) return + + if (this.sendReplayGap( + attachment.ws, + terminalId, + fromSeq, + toSeq, + streamId, + attachment.activeAttachRequestId, + )) { + attachment.lastSeq = Math.max(attachment.lastSeq, toSeq) + } + } + + private handleReplayRetentionLoss( + terminalId: string, + state: BrokerTerminalState, + retainedSuffixStreamId: string, + ): string | undefined { + if (!state.replayRing.consumeRetentionLoss()) return undefined + const previousStreamId = this.streamIdentity.getStream(terminalId) + const streamId = this.replaceStreamIdentity(terminalId, 'retention_lost') + state.replayRing.retagRetainedStreamSuffix(retainedSuffixStreamId, streamId) + const now = Date.now() + const lastLogAt = state.replayRetentionLogLastAt + if ( + typeof lastLogAt === 'number' + && now - lastLogAt < TERMINAL_REPLAY_RETENTION_LOG_RATE_LIMIT_MS + ) { + state.replayRetentionLogSuppressed = (state.replayRetentionLogSuppressed ?? 0) + 1 + return streamId + } + const suppressedCount = state.replayRetentionLogSuppressed ?? 0 + state.replayRetentionLogLastAt = now + state.replayRetentionLogSuppressed = 0 + this.logTerminalReplayRetention({ + terminalId, + streamId, + ...(previousStreamId ? { previousStreamId } : {}), + attachRequestIds: [...state.clients.values()] + .map((attachment) => attachment.activeAttachRequestId) + .filter((attachRequestId): attachRequestId is string => Boolean(attachRequestId)), + attachmentCount: state.clients.size, + reason: 'retention_lost', + retainedBytes: state.replayRing.retainedBytes(), + maxBytes: state.replayRing.retentionMaxBytes(), + tailSeq: state.replayRing.tailSeq(), + headSeq: state.replayRing.headSeq(), + ...(suppressedCount > 0 ? { suppressedCount } : {}), + }) + return streamId } private withTerminalLock(terminalId: string, task: () => Promise): Promise { diff --git a/server/terminal-stream/client-output-queue.ts b/server/terminal-stream/client-output-queue.ts index f7a3eee6a..1a2a2e069 100644 --- a/server/terminal-stream/client-output-queue.ts +++ b/server/terminal-stream/client-output-queue.ts @@ -1,17 +1,36 @@ import type { ReplayFrame } from './replay-ring.js' +import { buildTerminalOutputBatches } from './output-batch.js' + +type QueuedReplayFrame = ReplayFrame & { + queuedBytes: number +} export type GapEvent = { type: 'gap' fromSeq: number toSeq: number + streamId: string reason: 'queue_overflow' } +export type QueuedFrameByteMeasure = (frame: ReplayFrame) => number +export type QueuedBatchContext = { + terminalId?: string + attachRequestId?: string + source?: string +} + +export type PreparedClientOutputBatch = { + entries: Array + gapCount: number + frameCount: number +} + export function isGapEvent(entry: ReplayFrame | GapEvent): entry is GapEvent { return 'type' in entry && entry.type === 'gap' } -export const DEFAULT_TERMINAL_CLIENT_QUEUE_MAX_BYTES = 128 * 1024 +export const DEFAULT_TERMINAL_CLIENT_QUEUE_MAX_BYTES = 32 * 1024 * 1024 function resolveMaxBytes(explicitMaxBytes?: number): number { if (typeof explicitMaxBytes === 'number' && Number.isFinite(explicitMaxBytes) && explicitMaxBytes > 0) { @@ -28,64 +47,82 @@ function resolveMaxBytes(explicitMaxBytes?: number): number { export class ClientOutputQueue { private readonly maxBytes: number - private frames: ReplayFrame[] = [] + private frames: QueuedReplayFrame[] = [] private totalBytes = 0 - private pendingGap: GapEvent | null = null + private pendingGaps: GapEvent[] = [] private droppedBytes = 0 constructor(maxBytes?: number) { this.maxBytes = resolveMaxBytes(maxBytes) } - enqueue(frame: ReplayFrame): void { - this.frames.push({ ...frame }) - this.totalBytes += frame.bytes + enqueue(frame: ReplayFrame, queuedBytes = frame.bytes): void { + const normalizedQueuedBytes = Number.isFinite(queuedBytes) && queuedBytes > 0 + ? Math.floor(queuedBytes) + : 0 + this.frames.push({ ...frame, queuedBytes: normalizedQueuedBytes }) + this.totalBytes += normalizedQueuedBytes this.evictOverflow() } - nextBatch(maxBytes: number): Array { + prepareBatch( + maxBytes: number, + measureFrameBytes?: QueuedFrameByteMeasure, + batchContext?: QueuedBatchContext, + ): PreparedClientOutputBatch { const out: Array = [] - let budget = Number.isFinite(maxBytes) && maxBytes > 0 ? Math.floor(maxBytes) : 0 + const budget = Number.isFinite(maxBytes) && maxBytes > 0 ? Math.floor(maxBytes) : 0 - if (this.pendingGap) { - out.push(this.pendingGap) - this.pendingGap = null + if (this.pendingGaps.length > 0) { + out.push(...this.pendingGaps) } if (budget <= 0) { - return out + return { + entries: out, + gapCount: this.pendingGaps.length, + frameCount: 0, + } } - while (this.frames.length > 0) { - const first = this.frames[0] - if (first.bytes > budget && out.some((item) => !isGapEvent(item))) break + const batches = buildTerminalOutputBatches({ + frames: this.frames, + maxSerializedBytes: budget, + maxTotalSerializedBytes: budget, + measureFrameBytes: (frame) => this.measureFrameForBatch(frame, measureFrameBytes), + terminalId: batchContext?.terminalId, + attachRequestId: batchContext?.attachRequestId, + source: batchContext?.source, + }) + const consumedFrameCount = batches.reduce((sum, batch) => sum + batch.segments.length, 0) + out.push(...batches) + + return { + entries: out, + gapCount: this.pendingGaps.length, + frameCount: consumedFrameCount, + } + } - const frame = this.frames.shift() - if (!frame) break - this.totalBytes -= frame.bytes - budget -= frame.bytes - - const merged: ReplayFrame = { ...frame } - while (this.frames.length > 0) { - const next = this.frames[0] - if (next.seqStart !== merged.seqEnd + 1) break - if (next.bytes > budget) break - - const nextFrame = this.frames.shift() - if (!nextFrame) break - this.totalBytes -= nextFrame.bytes - budget -= nextFrame.bytes - merged.seqEnd = nextFrame.seqEnd - merged.data += nextFrame.data - merged.bytes += nextFrame.bytes - merged.at = nextFrame.at - } + nextBatch( + maxBytes: number, + measureFrameBytes?: QueuedFrameByteMeasure, + batchContext?: QueuedBatchContext, + ): Array { + const prepared = this.prepareBatch(maxBytes, measureFrameBytes, batchContext) + this.acknowledgePreparedBatch(prepared) + return prepared.entries + } - out.push(merged) - if (budget <= 0) break + acknowledgePreparedBatch(prepared: PreparedClientOutputBatch, counts: { gaps?: number; frames?: number } = {}): void { + const gaps = Math.max(0, Math.min( + this.pendingGaps.length, + Math.floor(counts.gaps ?? prepared.gapCount), + )) + if (gaps > 0) { + this.pendingGaps.splice(0, gaps) } - - return out + this.consumeFrames(Math.max(0, Math.floor(counts.frames ?? prepared.frameCount))) } pendingBytes(): number { @@ -96,6 +133,26 @@ export class ClientOutputQueue { return this.frames.length } + hasPendingEntries(): boolean { + return this.pendingGaps.length > 0 || this.frames.length > 0 + } + + retagPendingStream(fromStreamId: string, toStreamId: string): void { + if (!fromStreamId || !toStreamId || fromStreamId === toStreamId) return + + for (const frame of this.frames) { + if (frame.streamId === fromStreamId) { + frame.streamId = toStreamId + } + } + + for (const gap of this.pendingGaps) { + if (gap.streamId === fromStreamId) { + gap.streamId = toStreamId + } + } + } + peekDroppedBytes(): number { return this.droppedBytes } @@ -109,7 +166,7 @@ export class ClientOutputQueue { clear(): void { this.frames = [] this.totalBytes = 0 - this.pendingGap = null + this.pendingGaps = [] this.droppedBytes = 0 } @@ -117,24 +174,55 @@ export class ClientOutputQueue { while (this.totalBytes > this.maxBytes && this.frames.length > 0) { const dropped = this.frames.shift() if (!dropped) break - this.totalBytes -= dropped.bytes - this.droppedBytes += dropped.bytes - this.extendGap(dropped.seqStart, dropped.seqEnd) + this.totalBytes -= dropped.queuedBytes + this.droppedBytes += dropped.queuedBytes + this.extendGap(dropped.streamId, dropped.seqStart, dropped.seqEnd) + } + } + + private measureFrameForBatch(frame: ReplayFrame, measureFrameBytes?: QueuedFrameByteMeasure): number { + if (!measureFrameBytes) return frame.bytes + const measured = measureFrameBytes(this.toReplayFrame(frame)) + return Number.isFinite(measured) && measured > 0 ? Math.floor(measured) : 0 + } + + private consumeFrames(count: number): void { + for (let consumed = 0; consumed < count; consumed += 1) { + const frame = this.frames.shift() + if (!frame) return + this.totalBytes -= frame.queuedBytes } } - private extendGap(fromSeq: number, toSeq: number): void { - if (!this.pendingGap) { - this.pendingGap = { + private toReplayFrame(frame: ReplayFrame): ReplayFrame { + return { + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + data: frame.data, + bytes: frame.bytes, + at: frame.at, + streamId: frame.streamId, + barrier: frame.barrier, + ...(frame.barrier && frame.barrierReason ? { barrierReason: frame.barrierReason } : {}), + scannerStateBefore: frame.scannerStateBefore, + scannerStateAfter: frame.scannerStateAfter, + } + } + + private extendGap(streamId: string, fromSeq: number, toSeq: number): void { + const pendingGap = this.pendingGaps[this.pendingGaps.length - 1] + if (!pendingGap || pendingGap.streamId !== streamId || fromSeq > pendingGap.toSeq + 1) { + this.pendingGaps.push({ type: 'gap', fromSeq, toSeq, + streamId, reason: 'queue_overflow', - } + }) return } - this.pendingGap.fromSeq = Math.min(this.pendingGap.fromSeq, fromSeq) - this.pendingGap.toSeq = Math.max(this.pendingGap.toSeq, toSeq) + pendingGap.fromSeq = Math.min(pendingGap.fromSeq, fromSeq) + pendingGap.toSeq = Math.max(pendingGap.toSeq, toSeq) } } diff --git a/server/terminal-stream/output-barrier-scanner.ts b/server/terminal-stream/output-barrier-scanner.ts new file mode 100644 index 000000000..56555561c --- /dev/null +++ b/server/terminal-stream/output-barrier-scanner.ts @@ -0,0 +1,353 @@ +export type TerminalOutputBarrierReason = + | 'control' + | 'osc52' + | 'request_mode' + | 'turn_complete' + | 'startup_probe' + +export type TerminalOutputScannerMode = 'ground' | 'esc' | 'csi' | 'osc' | 'dcs' | 'apc' + +export type TerminalOutputScannerState = { + mode: TerminalOutputScannerMode +} + +export type TerminalOutputBarrierClassification = + | { + barrier: false + ground: boolean + stateBefore: TerminalOutputScannerState + stateAfter: TerminalOutputScannerState + } + | { + barrier: true + reason: TerminalOutputBarrierReason + ground: boolean + stateBefore: TerminalOutputScannerState + stateAfter: TerminalOutputScannerState + } + +export type TerminalOutputBarrierScanner = { + scan: (data: string) => TerminalOutputBarrierClassification + isGround: () => boolean +} + +const ESC = 0x1b +const BEL = 0x07 +const CSI = 0x9b +const OSC = 0x9d +const DCS = 0x90 +const SOS = 0x98 +const ST = 0x9c +const PM = 0x9e +const APC = 0x9f +const REPLACEMENT_CHARACTER = 0xfffd +const CSI_PAYLOAD_SUFFIX_LIMIT = 64 + +const REASON_PRIORITY: Record = { + control: 1, + turn_complete: 2, + startup_probe: 3, + request_mode: 4, + osc52: 5, +} + +function snapshot(mode: TerminalOutputScannerMode): TerminalOutputScannerState { + return { mode } +} + +function defaultReasonForMode(mode: TerminalOutputScannerMode): TerminalOutputBarrierReason { + return mode === 'osc' ? 'osc52' : 'control' +} + +function isCsiFinalByte(codePoint: number): boolean { + return codePoint >= 0x40 && codePoint <= 0x7e +} + +function isEscIntermediateByte(codePoint: number): boolean { + return codePoint >= 0x20 && codePoint <= 0x2f +} + +function isEscFinalByte(codePoint: number): boolean { + return codePoint >= 0x30 && codePoint <= 0x7e +} + +function isTransparentGroundControl(codePoint: number): boolean { + return codePoint === 0x09 || codePoint === 0x0a || codePoint === 0x0d +} + +function isGroundControlBarrier(codePoint: number): boolean { + if (isTransparentGroundControl(codePoint)) return false + return codePoint < 0x20 || codePoint === 0x7f || (codePoint >= 0x80 && codePoint <= 0x9f) +} + +function classifyCsiFinal(payload: string, finalChar: string): TerminalOutputBarrierReason { + const normalizedPayload = payload.replace(/[ -/]/gu, '') + if (finalChar === 'n' && normalizedPayload.endsWith('6')) { + return 'request_mode' + } + if (finalChar === 'c') { + return 'startup_probe' + } + return 'control' +} + +export function createTerminalOutputBarrierScanner(): TerminalOutputBarrierScanner { + let mode: TerminalOutputScannerMode = 'ground' + let csiPayloadSuffix = '' + let stringEscPending = false + + const enterCsi = () => { + mode = 'csi' + csiPayloadSuffix = '' + stringEscPending = false + } + + const enterStringMode = (nextMode: 'osc' | 'dcs' | 'apc') => { + mode = nextMode + csiPayloadSuffix = '' + stringEscPending = false + } + + const enterEsc = () => { + mode = 'esc' + csiPayloadSuffix = '' + stringEscPending = false + } + + const enterGround = () => { + mode = 'ground' + csiPayloadSuffix = '' + stringEscPending = false + } + + const recordReason = ( + current: TerminalOutputBarrierReason | undefined, + reason: TerminalOutputBarrierReason, + ): TerminalOutputBarrierReason => { + if (!current || REASON_PRIORITY[reason] > REASON_PRIORITY[current]) { + return reason + } + return current + } + + const scanner: TerminalOutputBarrierScanner = { + scan(data: string): TerminalOutputBarrierClassification { + const stateBefore = snapshot(mode) + let barrierReason: TerminalOutputBarrierReason | undefined + + if (mode !== 'ground') { + barrierReason = defaultReasonForMode(mode) + } + + const markBarrier = (reason: TerminalOutputBarrierReason) => { + barrierReason = recordReason(barrierReason, reason) + } + + const appendCsiPayload = (char: string) => { + csiPayloadSuffix += char + if (csiPayloadSuffix.length > CSI_PAYLOAD_SUFFIX_LIMIT) { + csiPayloadSuffix = csiPayloadSuffix.slice(-CSI_PAYLOAD_SUFFIX_LIMIT) + } + } + + const processStringMode = ( + codePoint: number, + stringMode: 'osc' | 'dcs' | 'apc', + ) => { + markBarrier(defaultReasonForMode(stringMode)) + + if (codePoint === REPLACEMENT_CHARACTER) { + markBarrier('control') + } + + if (stringEscPending) { + if (codePoint === 0x5c) { + enterGround() + return + } + if (codePoint === ESC) { + stringEscPending = true + return + } + stringEscPending = false + return + } + + if (codePoint === ESC) { + stringEscPending = true + return + } + if (codePoint === ST) { + enterGround() + return + } + if (stringMode === 'osc' && codePoint === BEL) { + enterGround() + } + } + + for (let index = 0; index < data.length;) { + const codePoint = data.codePointAt(index) + if (codePoint === undefined) break + const char = String.fromCodePoint(codePoint) + index += char.length + + if (mode === 'ground') { + if (codePoint === REPLACEMENT_CHARACTER) { + markBarrier('control') + continue + } + if (codePoint === BEL) { + markBarrier('turn_complete') + continue + } + if (codePoint === ESC) { + markBarrier('control') + enterEsc() + continue + } + if (codePoint === CSI) { + markBarrier('control') + enterCsi() + continue + } + if (codePoint === OSC) { + markBarrier('osc52') + enterStringMode('osc') + continue + } + if (codePoint === DCS) { + markBarrier('control') + enterStringMode('dcs') + continue + } + if (codePoint === SOS || codePoint === PM) { + markBarrier('control') + enterStringMode('apc') + continue + } + if (codePoint === APC) { + markBarrier('control') + enterStringMode('apc') + continue + } + if (isGroundControlBarrier(codePoint)) { + markBarrier('control') + } + continue + } + + if (mode === 'esc') { + markBarrier('control') + if (codePoint === 0x5b) { + enterCsi() + continue + } + if (codePoint === 0x58 || codePoint === 0x5e) { + enterStringMode('apc') + continue + } + if (codePoint === 0x5d) { + markBarrier('osc52') + enterStringMode('osc') + continue + } + if (codePoint === 0x50) { + enterStringMode('dcs') + continue + } + if (codePoint === 0x5f) { + enterStringMode('apc') + continue + } + if (codePoint === ESC) { + enterEsc() + continue + } + if (codePoint === CSI) { + enterCsi() + continue + } + if (codePoint === OSC) { + markBarrier('osc52') + enterStringMode('osc') + continue + } + if (codePoint === DCS) { + enterStringMode('dcs') + continue + } + if (codePoint === SOS || codePoint === PM) { + enterStringMode('apc') + continue + } + if (codePoint === APC) { + enterStringMode('apc') + continue + } + if (isEscIntermediateByte(codePoint)) { + continue + } + if (isEscFinalByte(codePoint)) { + enterGround() + } + continue + } + + if (mode === 'csi') { + markBarrier('control') + if (codePoint === REPLACEMENT_CHARACTER) { + markBarrier('control') + continue + } + if (codePoint === ESC) { + enterEsc() + continue + } + if (codePoint === BEL) { + markBarrier('turn_complete') + continue + } + if (codePoint === ST) { + enterGround() + continue + } + if (isCsiFinalByte(codePoint)) { + markBarrier(classifyCsiFinal(csiPayloadSuffix, char)) + enterGround() + continue + } + appendCsiPayload(char) + continue + } + + processStringMode(codePoint, mode) + } + + const stateAfter = snapshot(mode) + const ground = mode === 'ground' + if (!barrierReason) { + return { + barrier: false, + ground, + stateBefore, + stateAfter, + } + } + + return { + barrier: true, + reason: barrierReason, + ground, + stateBefore, + stateAfter, + } + }, + + isGround(): boolean { + return mode === 'ground' + }, + } + + return scanner +} diff --git a/server/terminal-stream/output-batch.ts b/server/terminal-stream/output-batch.ts new file mode 100644 index 000000000..299abae66 --- /dev/null +++ b/server/terminal-stream/output-batch.ts @@ -0,0 +1,415 @@ +import type { ReplayFrame } from './replay-ring.js' +import { + createTerminalOutputBarrierScanner, + type TerminalOutputBarrierReason, + type TerminalOutputScannerState, +} from './output-barrier-scanner.js' +import { measureTerminalOutputPayloadBytes, type JsonPayload } from './serialized-budget.js' + +type FrameBoundaryMetadata = { + attachRequestId?: string + source?: string +} + +export type TerminalOutputBatchSegment = { + seqStart: number + seqEnd: number + streamId: string + offset: number + endOffset: number + bytes: number + barrier: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState +} + +export type TerminalOutputBatch = ReplayFrame & FrameBoundaryMetadata & { + // Internal batching budget measured as the legacy terminal.output envelope. + // terminal.output.batch wire payloads compute their own serializedBytes. + legacyOutputSerializedBytes: number + segments: TerminalOutputBatchSegment[] + barrier: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState +} + +export type TerminalOutputBatchBuildInput = { + frames: Iterable + maxSerializedBytes: number + maxTotalSerializedBytes?: number + terminalId?: string + attachRequestId?: string + source?: string + payloadForFrame?: (frame: ReplayFrame) => JsonPayload + measureFrameBytes?: (frame: ReplayFrame) => number +} + +type FrameClassification = { + barrier: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState +} + +type AnnotatedReplayFrame = ReplayFrame & Partial & FrameBoundaryMetadata + +type MutableTerminalOutputBatch = FrameBoundaryMetadata & { + seqStart: number + seqEnd: number + chunks: string[] + dataLength: number + dataJsonContentBytes: number + bytes: number + at: number + streamId: string + legacyOutputSerializedBytes: number + segments: TerminalOutputBatchSegment[] + barrier: false + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState +} + +function normalizeBudget(value: number | undefined): number { + if (typeof value !== 'number' || !Number.isFinite(value) || value <= 0) return 0 + return Math.floor(value) +} + +function cloneScannerState(state: TerminalOutputScannerState): TerminalOutputScannerState { + return { mode: state.mode } +} + +function defaultPayloadForFrame( + terminalId: string, + attachRequestId: string | undefined, + frame: ReplayFrame, + source?: string, +): JsonPayload { + return { + type: 'terminal.output', + terminalId, + streamId: frame.streamId, + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + data: frame.data, + ...(attachRequestId ? { attachRequestId } : {}), + ...(source ? { source } : {}), + } +} + +function jsonStringContentBytes(data: string): number { + return Math.max(0, Buffer.byteLength(JSON.stringify(data), 'utf8') - 2) +} + +function classifyFrame( + frame: AnnotatedReplayFrame, + fallbackScanner: ReturnType, +): FrameClassification { + if ( + typeof frame.barrier === 'boolean' + && frame.scannerStateBefore + && frame.scannerStateAfter + ) { + return { + barrier: frame.barrier, + ...(frame.barrier && frame.barrierReason ? { barrierReason: frame.barrierReason } : {}), + scannerStateBefore: cloneScannerState(frame.scannerStateBefore), + scannerStateAfter: cloneScannerState(frame.scannerStateAfter), + } + } + + const scanned = fallbackScanner.scan(frame.data) + return { + barrier: scanned.barrier, + ...(scanned.barrier ? { barrierReason: scanned.reason } : {}), + scannerStateBefore: cloneScannerState(scanned.stateBefore), + scannerStateAfter: cloneScannerState(scanned.stateAfter), + } +} + +function isTransparentGroundFrame(classification: FrameClassification): boolean { + return !classification.barrier + && classification.scannerStateBefore.mode === 'ground' + && classification.scannerStateAfter.mode === 'ground' +} + +function frameAttachRequestId( + frame: AnnotatedReplayFrame, + inputAttachRequestId: string | undefined, +): string | undefined { + return frame.attachRequestId ?? inputAttachRequestId +} + +function frameSource(frame: AnnotatedReplayFrame, inputSource: string | undefined): string | undefined { + return frame.source ?? inputSource +} + +function measureBatch( + input: TerminalOutputBatchBuildInput, + batch: ReplayFrame & FrameBoundaryMetadata, + dataJsonContentBytes?: number, +): number { + if (input.payloadForFrame) { + return measureTerminalOutputPayloadBytes(input.payloadForFrame(batch)) + } + + if (input.terminalId) { + if (dataJsonContentBytes !== undefined) { + const emptyPayloadBytes = measureTerminalOutputPayloadBytes(defaultPayloadForFrame( + input.terminalId, + batch.attachRequestId ?? input.attachRequestId, + { ...batch, data: '' }, + batch.source ?? input.source, + )) + return emptyPayloadBytes - 2 + dataJsonContentBytes + 2 + } + + return measureTerminalOutputPayloadBytes(defaultPayloadForFrame( + input.terminalId, + batch.attachRequestId ?? input.attachRequestId, + batch, + batch.source ?? input.source, + )) + } + + if (input.measureFrameBytes) { + const measured = input.measureFrameBytes(batch) + return Number.isFinite(measured) && measured > 0 ? Math.floor(measured) : 0 + } + + return batch.bytes +} + +function segmentForFrame( + frame: ReplayFrame, + classification: FrameClassification, + offset: number, +): TerminalOutputBatchSegment { + return { + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + streamId: frame.streamId, + offset, + endOffset: offset + frame.data.length, + bytes: frame.bytes, + barrier: classification.barrier, + ...(classification.barrier && classification.barrierReason + ? { barrierReason: classification.barrierReason } + : {}), + scannerStateBefore: cloneScannerState(classification.scannerStateBefore), + scannerStateAfter: cloneScannerState(classification.scannerStateAfter), + } +} + +function buildSingleBatch( + frame: AnnotatedReplayFrame, + classification: FrameClassification, + input: TerminalOutputBatchBuildInput, +): TerminalOutputBatch { + const attachRequestId = frameAttachRequestId(frame, input.attachRequestId) + const source = frameSource(frame, input.source) + const batch: TerminalOutputBatch = { + ...frame, + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + data: frame.data, + bytes: frame.bytes, + at: frame.at, + streamId: frame.streamId, + ...(attachRequestId ? { attachRequestId } : {}), + ...(source ? { source } : {}), + barrier: classification.barrier, + ...(classification.barrier && classification.barrierReason + ? { barrierReason: classification.barrierReason } + : {}), + scannerStateBefore: cloneScannerState(classification.scannerStateBefore), + scannerStateAfter: cloneScannerState(classification.scannerStateAfter), + legacyOutputSerializedBytes: 0, + segments: [segmentForFrame(frame, classification, 0)], + } + batch.legacyOutputSerializedBytes = measureBatch(input, batch, jsonStringContentBytes(batch.data)) + return batch +} + +function canMerge( + current: MutableTerminalOutputBatch, + next: AnnotatedReplayFrame, + nextClassification: FrameClassification, + input: TerminalOutputBatchBuildInput, +): boolean { + if (!isTransparentGroundFrame(nextClassification)) return false + if (current.barrier) return false + if (current.scannerStateBefore.mode !== 'ground' || current.scannerStateAfter.mode !== 'ground') return false + if (next.seqStart !== current.seqEnd + 1) return false + if (next.streamId !== current.streamId) return false + if (frameAttachRequestId(next, input.attachRequestId) !== current.attachRequestId) return false + if (frameSource(next, input.source) !== current.source) return false + return true +} + +function startMutableBatch( + frame: AnnotatedReplayFrame, + classification: FrameClassification, + input: TerminalOutputBatchBuildInput, +): MutableTerminalOutputBatch { + const attachRequestId = frameAttachRequestId(frame, input.attachRequestId) + const source = frameSource(frame, input.source) + const dataJsonContentBytes = jsonStringContentBytes(frame.data) + const batch: MutableTerminalOutputBatch = { + seqStart: frame.seqStart, + seqEnd: frame.seqEnd, + chunks: [frame.data], + dataLength: frame.data.length, + dataJsonContentBytes, + bytes: frame.bytes, + at: frame.at, + streamId: frame.streamId, + ...(attachRequestId ? { attachRequestId } : {}), + ...(source ? { source } : {}), + barrier: false, + scannerStateBefore: cloneScannerState(classification.scannerStateBefore), + scannerStateAfter: cloneScannerState(classification.scannerStateAfter), + segments: [segmentForFrame(frame, classification, 0)], + legacyOutputSerializedBytes: 0, + } + batch.legacyOutputSerializedBytes = measureBatch(input, materializeMutableBatchFrame(batch), dataJsonContentBytes) + return batch +} + +function materializeMutableBatchFrame(batch: MutableTerminalOutputBatch): ReplayFrame & FrameBoundaryMetadata { + return { + seqStart: batch.seqStart, + seqEnd: batch.seqEnd, + data: batch.chunks.join(''), + bytes: batch.bytes, + at: batch.at, + streamId: batch.streamId, + barrier: false, + scannerStateBefore: batch.scannerStateBefore, + scannerStateAfter: batch.scannerStateAfter, + ...(batch.attachRequestId ? { attachRequestId: batch.attachRequestId } : {}), + ...(batch.source ? { source: batch.source } : {}), + } +} + +function flushMutableBatch(batch: MutableTerminalOutputBatch): TerminalOutputBatch { + const frame = materializeMutableBatchFrame(batch) + return { + ...frame, + legacyOutputSerializedBytes: batch.legacyOutputSerializedBytes, + segments: batch.segments, + } +} + +function measureMergedBatch( + current: MutableTerminalOutputBatch, + next: AnnotatedReplayFrame, + nextClassification: FrameClassification, + input: TerminalOutputBatchBuildInput, +): number { + const dataJsonContentBytes = current.dataJsonContentBytes + jsonStringContentBytes(next.data) + const candidate: ReplayFrame & FrameBoundaryMetadata = { + seqStart: current.seqStart, + seqEnd: next.seqEnd, + data: '', + bytes: current.bytes + next.bytes, + at: next.at, + streamId: current.streamId, + barrier: false, + scannerStateBefore: current.scannerStateBefore, + scannerStateAfter: nextClassification.scannerStateAfter, + ...(current.attachRequestId ? { attachRequestId: current.attachRequestId } : {}), + ...(current.source ? { source: current.source } : {}), + } + + if (input.terminalId && !input.payloadForFrame) { + return measureBatch(input, candidate, dataJsonContentBytes) + } + if (!input.payloadForFrame && !input.measureFrameBytes) { + return candidate.bytes + } + + candidate.data = `${current.chunks.join('')}${next.data}` + return measureBatch(input, candidate, dataJsonContentBytes) +} + +function appendMutableBatch( + current: MutableTerminalOutputBatch, + next: AnnotatedReplayFrame, + nextClassification: FrameClassification, + legacyOutputSerializedBytes: number, +): void { + const offset = current.dataLength + current.seqEnd = next.seqEnd + current.chunks.push(next.data) + current.dataLength += next.data.length + current.dataJsonContentBytes += jsonStringContentBytes(next.data) + current.bytes += next.bytes + current.at = next.at + current.scannerStateAfter = cloneScannerState(nextClassification.scannerStateAfter) + current.segments.push(segmentForFrame(next, nextClassification, offset)) + current.legacyOutputSerializedBytes = legacyOutputSerializedBytes +} + +export function buildTerminalOutputBatches( + input: TerminalOutputBatchBuildInput, +): TerminalOutputBatch[] { + const maxSerializedBytes = normalizeBudget(input.maxSerializedBytes) + const maxTotalSerializedBytes = input.maxTotalSerializedBytes === undefined + ? Number.POSITIVE_INFINITY + : normalizeBudget(input.maxTotalSerializedBytes) + if (maxSerializedBytes <= 0 || maxTotalSerializedBytes <= 0) return [] + + const fallbackScanner = createTerminalOutputBarrierScanner() + const batches: TerminalOutputBatch[] = [] + let current: MutableTerminalOutputBatch | null = null + let totalLegacyOutputSerializedBytes = 0 + + const pushBatch = (batch: TerminalOutputBatch): boolean => { + if ( + Number.isFinite(maxTotalSerializedBytes) + && totalLegacyOutputSerializedBytes + batch.legacyOutputSerializedBytes > maxTotalSerializedBytes + && batches.length > 0 + ) { + return false + } + batches.push(batch) + totalLegacyOutputSerializedBytes += batch.legacyOutputSerializedBytes + return true + } + + const pushCurrent = (): boolean => { + if (!current) return true + const batch = flushMutableBatch(current) + current = null + return pushBatch(batch) + } + + for (const rawFrame of input.frames) { + const frame = rawFrame as AnnotatedReplayFrame + const classification = classifyFrame(frame, fallbackScanner) + + if (!isTransparentGroundFrame(classification)) { + if (!pushCurrent()) return batches + const nextBatch = buildSingleBatch(frame, classification, input) + if (!pushBatch(nextBatch)) return batches + continue + } + + if (current && canMerge(current, frame, classification, input)) { + const mergedSerializedBytes = measureMergedBatch(current, frame, classification, input) + if (mergedSerializedBytes <= maxSerializedBytes) { + appendMutableBatch(current, frame, classification, mergedSerializedBytes) + continue + } + } + + if (!pushCurrent()) return batches + current = startMutableBatch(frame, classification, input) + } + + pushCurrent() + + return batches +} diff --git a/server/terminal-stream/output-fragments.ts b/server/terminal-stream/output-fragments.ts new file mode 100644 index 000000000..0c898fe7e --- /dev/null +++ b/server/terminal-stream/output-fragments.ts @@ -0,0 +1,60 @@ +import { measureSerializedJsonBytes, type JsonPayload } from './serialized-budget.js' + +export function containsLoneSurrogate(data: string): boolean { + for (let index = 0; index < data.length; index += 1) { + const code = data.charCodeAt(index) + if (code >= 0xd800 && code <= 0xdbff) { + const next = data.charCodeAt(index + 1) + if (!(next >= 0xdc00 && next <= 0xdfff)) return true + index += 1 + continue + } + if (code >= 0xdc00 && code <= 0xdfff) return true + } + return false +} + +export function fragmentTerminalOutputForPayloadBudget(input: { + maxSerializedBytes: number + data: string + payloadForData: (data: string) => JsonPayload +}): string[] { + const maxSerializedBytes = Math.max(1, Math.floor(input.maxSerializedBytes)) + if (measureSerializedJsonBytes(input.payloadForData(input.data)) <= maxSerializedBytes) { + return [input.data] + } + + // node-pty currently gives Freshell terminal output as JavaScript strings. + // Fragment on code points so Task 5 preserves that string contract without + // splitting surrogate pairs or claiming byte-perfect PTY replay. + const codePoints = Array.from(input.data) + const chunks: string[] = [] + let offset = 0 + + while (offset < codePoints.length) { + let low = 1 + let high = codePoints.length - offset + let best = 0 + + while (low <= high) { + const mid = Math.floor((low + high) / 2) + const candidate = codePoints.slice(offset, offset + mid).join('') + const bytes = measureSerializedJsonBytes(input.payloadForData(candidate)) + if (bytes <= maxSerializedBytes) { + best = mid + low = mid + 1 + } else { + high = mid - 1 + } + } + + if (best <= 0) { + throw new Error('terminal output payload budget is too small for one code point') + } + + chunks.push(codePoints.slice(offset, offset + best).join('')) + offset += best + } + + return chunks +} diff --git a/server/terminal-stream/replay-deque.ts b/server/terminal-stream/replay-deque.ts new file mode 100644 index 000000000..d0c78ef10 --- /dev/null +++ b/server/terminal-stream/replay-deque.ts @@ -0,0 +1,227 @@ +import { buildTerminalOutputBatches } from './output-batch.js' +import type { + TerminalOutputBarrierReason, + TerminalOutputScannerState, +} from './output-barrier-scanner.js' +import type { + ReplayBatchContext, + ReplayFrame, + ReplayFrameByteMeasure, +} from './replay-ring.js' + +const DEFAULT_STREAM_ID = 'stream-1' +const GROUND_SCANNER_STATE: TerminalOutputScannerState = { mode: 'ground' } +const COMPACT_MIN_EVICTED_FRAMES = 1024 + +export type ReplayDequeAppendInput = string | { + data: string + streamId?: string + barrier?: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore?: TerminalOutputScannerState + scannerStateAfter?: TerminalOutputScannerState + at?: number +} + +function normalizeMaxBytes(maxBytes: number): number { + if (Number.isFinite(maxBytes) && maxBytes > 0) { + return Math.floor(maxBytes) + } + return 0 +} + +function cloneScannerState(state: TerminalOutputScannerState): TerminalOutputScannerState { + return { mode: state.mode } +} + +export class ReplayDeque { + private frames: ReplayFrame[] = [] + private startIndex = 0 + private retainedBytes = 0 + private nextSeq = 1 + private head = 0 + private maxBytes: number + private retentionLossPending = false + + constructor(maxBytes: number) { + this.maxBytes = normalizeMaxBytes(maxBytes) + } + + setMaxBytes(nextMaxBytes: number): void { + const normalizedMaxBytes = normalizeMaxBytes(nextMaxBytes) + if (normalizedMaxBytes === this.maxBytes) return + this.maxBytes = normalizedMaxBytes + this.evictIfNeeded() + } + + append(input: ReplayDequeAppendInput): ReplayFrame { + const frameInput = typeof input === 'string' ? { data: input } : input + const seq = this.nextSeq + this.nextSeq += 1 + this.head = seq + + const barrier = frameInput.barrier ?? false + const frame: ReplayFrame = { + seqStart: seq, + seqEnd: seq, + data: frameInput.data, + bytes: Buffer.byteLength(frameInput.data, 'utf8'), + at: frameInput.at ?? Date.now(), + streamId: frameInput.streamId ?? DEFAULT_STREAM_ID, + barrier, + ...(barrier && frameInput.barrierReason ? { barrierReason: frameInput.barrierReason } : {}), + scannerStateBefore: cloneScannerState(frameInput.scannerStateBefore ?? GROUND_SCANNER_STATE), + scannerStateAfter: cloneScannerState(frameInput.scannerStateAfter ?? GROUND_SCANNER_STATE), + } + + this.frames.push(frame) + this.retainedBytes += frame.bytes + this.evictIfNeeded() + return frame + } + + consumeRetentionLoss(): boolean { + const retentionLossPending = this.retentionLossPending + this.retentionLossPending = false + return retentionLossPending + } + + retagRetainedStreamSuffix(fromStreamId: string, toStreamId: string): void { + for (let index = this.frames.length - 1; index >= this.startIndex; index -= 1) { + const frame = this.frames[index] + if (frame.streamId !== fromStreamId) break + frame.streamId = toStreamId + } + } + + replaySince(sinceSeq?: number): { frames: ReplayFrame[]; missedFromSeq?: number } { + const normalizedSinceSeq = this.normalizeSinceSeq(sinceSeq) + const missedFromSeq = this.missedFromSeq(normalizedSinceSeq) + if (this.retainedCount() === 0) { + return missedFromSeq === undefined ? { frames: [] } : { frames: [], missedFromSeq } + } + + const frames = Array.from(this.iterReplayFrames(normalizedSinceSeq, Number.POSITIVE_INFINITY)) + return missedFromSeq === undefined ? { frames } : { frames, missedFromSeq } + } + + replayBatchSince( + sinceSeq: number | undefined, + maxBytes: number, + toSeq?: number, + measureFrameBytes?: ReplayFrameByteMeasure, + batchContext?: ReplayBatchContext, + ): { frames: ReplayFrame[]; missedFromSeq?: number } { + const normalizedSinceSeq = this.normalizeSinceSeq(sinceSeq) + const missedFromSeq = this.missedFromSeq(normalizedSinceSeq) + if (this.retainedCount() === 0) { + return missedFromSeq === undefined ? { frames: [] } : { frames: [], missedFromSeq } + } + + const normalizedMaxBytes = Number.isFinite(maxBytes) && maxBytes > 0 ? Math.floor(maxBytes) : 0 + const normalizedToSeq = typeof toSeq === 'number' && Number.isFinite(toSeq) + ? Math.max(0, Math.floor(toSeq)) + : Number.POSITIVE_INFINITY + + const frames = buildTerminalOutputBatches({ + frames: this.iterReplayFrames(normalizedSinceSeq, normalizedToSeq), + maxSerializedBytes: normalizedMaxBytes, + maxTotalSerializedBytes: normalizedMaxBytes, + measureFrameBytes, + terminalId: batchContext?.terminalId, + attachRequestId: batchContext?.attachRequestId, + source: batchContext?.source, + }) + + return missedFromSeq === undefined ? { frames } : { frames, missedFromSeq } + } + + totalBytes(): number { + return this.retainedBytes + } + + headSeq(): number { + return this.head + } + + tailSeq(): number { + const firstFrame = this.firstFrame() + return firstFrame ? firstFrame.seqStart : this.head + 1 + } + + private normalizeSinceSeq(sinceSeq?: number): number { + return sinceSeq === undefined || sinceSeq === 0 ? 0 : sinceSeq + } + + private missedFromSeq(normalizedSinceSeq: number): number | undefined { + const firstFrame = this.firstFrame() + if (!firstFrame) { + return normalizedSinceSeq < this.head ? normalizedSinceSeq + 1 : undefined + } + + return normalizedSinceSeq < firstFrame.seqStart - 1 + ? normalizedSinceSeq + 1 + : undefined + } + + private evictIfNeeded(): void { + while (this.retainedBytes > this.maxBytes && this.retainedCount() > 0) { + const removed = this.frames[this.startIndex] + if (!removed) break + this.startIndex += 1 + this.retainedBytes -= removed.bytes + this.retentionLossPending = true + } + this.compactIfNeeded() + } + + private compactIfNeeded(): void { + if (this.startIndex === 0) return + const retainedCount = this.retainedCount() + if (retainedCount === 0) { + this.frames = [] + this.startIndex = 0 + return + } + if ( + this.startIndex < COMPACT_MIN_EVICTED_FRAMES + || this.startIndex < retainedCount + ) { + return + } + + this.frames = this.frames.slice(this.startIndex) + this.startIndex = 0 + } + + private retainedCount(): number { + return this.frames.length - this.startIndex + } + + private firstFrame(): ReplayFrame | undefined { + return this.retainedCount() > 0 ? this.frames[this.startIndex] : undefined + } + + private firstFrameIndexAfter(seq: number): number { + let low = this.startIndex + let high = this.frames.length + while (low < high) { + const mid = Math.floor((low + high) / 2) + if (this.frames[mid].seqEnd <= seq) { + low = mid + 1 + } else { + high = mid + } + } + return low + } + + private *iterReplayFrames(sinceSeq: number, toSeq: number): IterableIterator { + const start = this.firstFrameIndexAfter(sinceSeq) + for (let index = start; index < this.frames.length; index += 1) { + const frame = this.frames[index] + if (frame.seqStart > toSeq) break + yield frame + } + } +} diff --git a/server/terminal-stream/replay-ring.ts b/server/terminal-stream/replay-ring.ts index f5ef23049..c8fb29830 100644 --- a/server/terminal-stream/replay-ring.ts +++ b/server/terminal-stream/replay-ring.ts @@ -1,13 +1,33 @@ +import { + createTerminalOutputBarrierScanner, + type TerminalOutputBarrierClassification, + type TerminalOutputBarrierReason, + type TerminalOutputScannerState, +} from './output-barrier-scanner.js' +import { ReplayDeque } from './replay-deque.js' + export type ReplayFrame = { seqStart: number seqEnd: number data: string bytes: number at: number + streamId: string + barrier: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState } export const DEFAULT_TERMINAL_REPLAY_RING_MAX_BYTES = 1024 * 1024 +export type ReplayFrameByteMeasure = (frame: ReplayFrame) => number +export type ReplayBatchContext = { + terminalId?: string + attachRequestId?: string + source?: string +} + function resolveMaxBytes(explicitMaxBytes?: number): number { if (typeof explicitMaxBytes === 'number' && Number.isFinite(explicitMaxBytes) && explicitMaxBytes > 0) { return Math.floor(explicitMaxBytes) @@ -22,144 +42,94 @@ function resolveMaxBytes(explicitMaxBytes?: number): number { } export class ReplayRing { - private frames: ReplayFrame[] = [] - private totalBytes = 0 - private nextSeq = 1 - private head = 0 + private readonly storage: ReplayDeque private maxBytes: number private readonly utf8FatalDecoder = new TextDecoder('utf-8', { fatal: true }) + private readonly barrierScanner = createTerminalOutputBarrierScanner() constructor(maxBytes?: number) { this.maxBytes = resolveMaxBytes(maxBytes) + this.storage = new ReplayDeque(this.maxBytes) } setMaxBytes(nextMaxBytes?: number): void { const resolved = resolveMaxBytes(nextMaxBytes) if (resolved === this.maxBytes) return this.maxBytes = resolved - this.evictIfNeeded() + this.storage.setMaxBytes(this.maxBytes) } - append(data: string): ReplayFrame { - const seq = this.nextSeq - this.nextSeq += 1 - this.head = seq + append(data: string, metadata: { streamId: string }): ReplayFrame { + const streamClassification = this.barrierScanner.scan(data) const normalizedData = this.normalizeFrameData(data) + const wasTruncated = Buffer.byteLength(normalizedData, 'utf8') < Buffer.byteLength(data, 'utf8') + const barrierClassification = wasTruncated + ? this.conservativeTruncatedClassification(streamClassification) + : streamClassification - const frame: ReplayFrame = { - seqStart: seq, - seqEnd: seq, + return this.storage.append({ data: normalizedData, - bytes: Buffer.byteLength(normalizedData, 'utf8'), at: Date.now(), - } - - this.frames.push(frame) - this.totalBytes += frame.bytes - this.evictIfNeeded() - return frame + streamId: metadata.streamId, + barrier: barrierClassification.barrier, + ...(barrierClassification.barrier ? { barrierReason: barrierClassification.reason } : {}), + scannerStateBefore: barrierClassification.stateBefore, + scannerStateAfter: barrierClassification.stateAfter, + }) } - replaySince(sinceSeq?: number): { frames: ReplayFrame[]; missedFromSeq?: number } { - const normalizedSinceSeq = sinceSeq === undefined || sinceSeq === 0 ? 0 : sinceSeq - if (this.frames.length === 0) { - if (normalizedSinceSeq < this.head) { - return { frames: [], missedFromSeq: normalizedSinceSeq + 1 } - } - return { frames: [] } - } + consumeRetentionLoss(): boolean { + return this.storage.consumeRetentionLoss() + } - const tail = this.frames[0].seqStart - const missedFromSeq = normalizedSinceSeq < tail - 1 - ? normalizedSinceSeq + 1 - : undefined + retagRetainedStreamSuffix(fromStreamId: string, toStreamId: string): void { + this.storage.retagRetainedStreamSuffix(fromStreamId, toStreamId) + } - const frames = this.frames.slice(this.firstFrameIndexAfter(normalizedSinceSeq)) - return { frames, missedFromSeq } + replaySince(sinceSeq?: number): { frames: ReplayFrame[]; missedFromSeq?: number } { + return this.storage.replaySince(sinceSeq) } replayBatchSince( sinceSeq: number | undefined, maxBytes: number, toSeq?: number, + measureFrameBytes?: ReplayFrameByteMeasure, + batchContext?: ReplayBatchContext, ): { frames: ReplayFrame[]; missedFromSeq?: number } { - const normalizedSinceSeq = sinceSeq === undefined || sinceSeq === 0 ? 0 : sinceSeq - const normalizedMaxBytes = Number.isFinite(maxBytes) && maxBytes > 0 ? Math.floor(maxBytes) : 0 - const normalizedToSeq = typeof toSeq === 'number' && Number.isFinite(toSeq) - ? Math.max(0, Math.floor(toSeq)) - : Number.POSITIVE_INFINITY - - if (this.frames.length === 0) { - if (normalizedSinceSeq < this.head) { - return { frames: [], missedFromSeq: normalizedSinceSeq + 1 } - } - return { frames: [] } - } - - const tail = this.frames[0].seqStart - const missedFromSeq = normalizedSinceSeq < tail - 1 - ? normalizedSinceSeq + 1 - : undefined - const frames: ReplayFrame[] = [] - let budget = normalizedMaxBytes - - if (budget <= 0) { - return { frames, missedFromSeq } - } - - const startIndex = this.firstFrameIndexAfter(normalizedSinceSeq) - for (let i = startIndex; i < this.frames.length; i += 1) { - const frame = this.frames[i] - if (frame.seqStart > normalizedToSeq) break - if (frame.bytes > budget && frames.length > 0) break - - const previous = frames[frames.length - 1] - if (previous && frame.seqStart === previous.seqEnd + 1) { - previous.seqEnd = frame.seqEnd - previous.data += frame.data - previous.bytes += frame.bytes - previous.at = frame.at - } else { - frames.push({ ...frame }) - } - budget -= frame.bytes - if (budget <= 0) break - } - - return { frames, missedFromSeq } + return this.storage.replayBatchSince(sinceSeq, maxBytes, toSeq, measureFrameBytes, batchContext) } headSeq(): number { - return this.head + return this.storage.headSeq() } tailSeq(): number { - if (this.frames.length === 0) { - return this.head + 1 - } - return this.frames[0].seqStart + return this.storage.tailSeq() } - private evictIfNeeded(): void { - while (this.totalBytes > this.maxBytes && this.frames.length > 0) { - const removed = this.frames.shift() - if (!removed) break - this.totalBytes -= removed.bytes - } + retainedBytes(): number { + return this.storage.totalBytes() + } + + retentionMaxBytes(): number { + return this.maxBytes } - private firstFrameIndexAfter(seq: number): number { - let low = 0 - let high = this.frames.length - while (low < high) { - const mid = Math.floor((low + high) / 2) - if (this.frames[mid].seqEnd <= seq) { - low = mid + 1 - } else { - high = mid - } + private conservativeTruncatedClassification( + classification: TerminalOutputBarrierClassification, + ): { + barrier: true + reason: TerminalOutputBarrierReason + stateBefore: TerminalOutputScannerState + stateAfter: TerminalOutputScannerState + } { + return { + barrier: true, + reason: classification.barrier ? classification.reason : 'control', + stateBefore: classification.stateBefore, + stateAfter: classification.stateAfter, } - return low } private decodeUtf8Fatal(bytes: Uint8Array): string | null { diff --git a/server/terminal-stream/serialized-budget.ts b/server/terminal-stream/serialized-budget.ts new file mode 100644 index 000000000..5dec4ffe0 --- /dev/null +++ b/server/terminal-stream/serialized-budget.ts @@ -0,0 +1,23 @@ +export type JsonPayload = Record + +export const TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE = 'x'.repeat(512) +export const TERMINAL_STREAM_ATTACH_REQUEST_ID_SERIALIZED_BYTES_RESERVE = Buffer.byteLength( + JSON.stringify(TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE), + 'utf8', +) + +export function measureSerializedJsonBytes(payload: JsonPayload): number { + return Buffer.byteLength(JSON.stringify(payload), 'utf8') +} + +export function measureTerminalOutputPayloadBytes(payload: JsonPayload): number { + return measureSerializedJsonBytes(payload) +} + +export function isTerminalStreamAttachRequestIdWithinSerializedBudget( + attachRequestId: string | undefined, +): boolean { + if (attachRequestId === undefined) return true + return Buffer.byteLength(JSON.stringify(attachRequestId), 'utf8') + <= TERMINAL_STREAM_ATTACH_REQUEST_ID_SERIALIZED_BYTES_RESERVE +} diff --git a/server/terminal-stream/stream-identity.ts b/server/terminal-stream/stream-identity.ts new file mode 100644 index 000000000..9ea09ecec --- /dev/null +++ b/server/terminal-stream/stream-identity.ts @@ -0,0 +1,66 @@ +import { randomUUID } from 'node:crypto' + +export type TerminalStreamReplacementReason = + | 'new_pty_session' + | 'codex_pty_recovery' + | 'retention_lost' + | 'server_restart_incompatible_retention' + +export type TerminalStreamIdentityTracker = { + ensureStream: (terminalId: string) => string + getStream: (terminalId: string) => string | undefined + recordAttach: (terminalId: string) => string + recordDetach: (terminalId: string) => string | undefined + replaceStream: (terminalId: string, reason: TerminalStreamReplacementReason) => string + forgetStream: (terminalId: string) => void +} + +type StreamState = { + streamId: string + generation: number +} + +export function createTerminalStreamIdentityTracker(): TerminalStreamIdentityTracker { + const streams = new Map() + + const mintStreamId = () => randomUUID() + + const ensureState = (terminalId: string): StreamState => { + let state = streams.get(terminalId) + if (!state) { + state = { + streamId: mintStreamId(), + generation: 1, + } + streams.set(terminalId, state) + } + return state + } + + return { + ensureStream(terminalId) { + return ensureState(terminalId).streamId + }, + getStream(terminalId) { + return streams.get(terminalId)?.streamId + }, + recordAttach(terminalId) { + const state = ensureState(terminalId) + return state.streamId + }, + recordDetach(terminalId) { + const state = streams.get(terminalId) + if (!state) return undefined + return state.streamId + }, + replaceStream(terminalId, _reason) { + const state = ensureState(terminalId) + state.generation += 1 + state.streamId = mintStreamId() + return state.streamId + }, + forgetStream(terminalId) { + streams.delete(terminalId) + }, + } +} diff --git a/server/terminal-stream/types.ts b/server/terminal-stream/types.ts index c1001653b..1f73b93d0 100644 --- a/server/terminal-stream/types.ts +++ b/server/terminal-stream/types.ts @@ -1,6 +1,7 @@ import type { LiveWebSocket } from '../ws-handler.js' import type { ClientOutputQueue } from './client-output-queue.js' import type { ReplayFrame, ReplayRing } from './replay-ring.js' +import type { TerminalGeometryAuthority } from '../../shared/ws-protocol.js' export type BrokerClientMode = 'attaching' | 'live' export type BrokerClientPriority = 'foreground' | 'background' @@ -8,6 +9,7 @@ export type BrokerClientPriority = 'foreground' | 'background' export type ReplayCursor = { nextSeq: number toSeq: number + streamId: string } export type BrokerClientAttachment = { @@ -22,9 +24,18 @@ export type BrokerClientAttachment = { activeAttachRequestId?: string catastrophicSince?: number catastrophicClosed?: boolean + replayBackpressureLogLastAt?: number + replayBackpressureLogSuppressed?: number + terminalOutputBatchV1: boolean } export type BrokerTerminalState = { replayRing: ReplayRing clients: Map + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + geometryCols?: number + geometryRows?: number + replayRetentionLogLastAt?: number + replayRetentionLogSuppressed?: number } diff --git a/server/terminal-view/mirror.ts b/server/terminal-view/mirror.ts index 938332279..2dae480d1 100644 --- a/server/terminal-view/mirror.ts +++ b/server/terminal-view/mirror.ts @@ -51,7 +51,7 @@ export class TerminalViewMirror { applyOutput(rawOutput: string): ReplayFrame { const normalized = normalizeOutput(rawOutput) - const frame = this.replayRing.append(normalized) + const frame = this.replayRing.append(normalized, { streamId: this.terminalId }) this.lines = appendLines(this.lines, normalized) this.revision += 1 return frame diff --git a/server/ws-handler.ts b/server/ws-handler.ts index 641b33ebd..cf2638eca 100644 --- a/server/ws-handler.ts +++ b/server/ws-handler.ts @@ -4,7 +4,7 @@ import WebSocket, { WebSocketServer } from 'ws' import { z } from 'zod' import { logger } from './logger.js' import { recordSessionLifecycleEvent } from './session-observability.js' -import { getPerfConfig, logPerfEvent, shouldLog, startPerfTimer } from './perf-logger.js' +import { getPerfConfig, startPerfTimer } from './perf-logger.js' import { getRequiredAuthToken, isLoopbackAddress, isOriginAllowed, timingSafeCompare } from './auth.js' import { buildTerminalSessionRef, modeSupportsResume, terminalIdFromCreateError } from './terminal-registry.js' import type { TerminalRecord, TerminalRegistry, TerminalMode } from './terminal-registry.js' @@ -29,6 +29,7 @@ import type { import type { ExtensionManager } from './extension-manager.js' import { allocateLocalhostPort } from './local-port.js' import { TerminalStreamBroker } from './terminal-stream/broker.js' +import { isTerminalStreamAttachRequestIdWithinSerializedBudget } from './terminal-stream/serialized-budget.js' import { buildSidebarOpenSessionKeys, type SidebarSessionLocator } from './sidebar-session-selection.js' import { loadSessionHistory } from './session-history-loader.js' import type { SdkCreatedSession, SdkSessionState } from './sdk-bridge-types.js' @@ -102,6 +103,7 @@ import { planCodexCreateRestoreDecision, resolveCodexCreateRestoreDecision, } from './coding-cli/codex-app-server/restore-decision.js' +import { sendJsonMessage } from './ws-send.js' type WsHandlerConfig = { maxConnections: number @@ -212,6 +214,7 @@ export interface LiveWebSocket extends WebSocket { connectionId?: string connectedAt?: number isMobileClient?: boolean + supportsTerminalOutputBatchV1?: boolean // Generation counter for chunked session updates to prevent interleaving sessionUpdateGeneration?: number } @@ -428,6 +431,7 @@ const TabsSyncClientRetireSchema = z.object({ type ClientState = { authenticated: boolean supportsUiScreenshotV1: boolean + supportsTerminalOutputBatchV1: boolean attachedTerminalIds: Set createdByRequestId: Map terminalCreateTimestamps: number[] @@ -1229,6 +1233,7 @@ export class WsHandler { const state: ClientState = { authenticated: false, supportsUiScreenshotV1: false, + supportsTerminalOutputBatchV1: false, attachedTerminalIds: new Set(), createdByRequestId: new Map(), terminalCreateTimestamps: [], @@ -1453,88 +1458,13 @@ export class WsHandler { } } - private closeForBackpressureIfNeeded(ws: LiveWebSocket, bufferedOverride?: number): boolean { - const buffered = bufferedOverride ?? (ws.bufferedAmount as number | undefined) - if (typeof buffered !== 'number' || buffered <= this.config.maxWsBufferedAmount) return false - - if (perfConfig.enabled && shouldLog(`ws_backpressure_${ws.connectionId || 'unknown'}`, perfConfig.rateLimitMs)) { - logPerfEvent( - 'ws_backpressure_close', - { - connectionId: ws.connectionId, - bufferedBytes: buffered, - limitBytes: this.config.maxWsBufferedAmount, - }, - 'warn', - ) - } - ws.close(CLOSE_CODES.BACKPRESSURE, 'Backpressure') - return true - } - private send(ws: LiveWebSocket, msg: unknown, skipBackpressureCheck = false) { - let messageType: string | undefined - try { - // Backpressure guard (skipped for pre-drained chunked sends). - const buffered = ws.bufferedAmount as number | undefined - if (!skipBackpressureCheck && this.closeForBackpressureIfNeeded(ws, buffered)) return - let serialized = '' - let payloadBytes: number | undefined - let serializeMs: number | undefined - let shouldLogSend = false - - if (perfConfig.enabled) { - if (msg && typeof msg === 'object' && 'type' in msg) { - const typeValue = (msg as { type?: unknown }).type - if (typeof typeValue === 'string') messageType = typeValue - } - - const serializeStart = process.hrtime.bigint() - serialized = JSON.stringify(msg) - const serializeEnd = process.hrtime.bigint() - payloadBytes = Buffer.byteLength(serialized) - - if (payloadBytes >= perfConfig.wsPayloadWarnBytes) { - shouldLogSend = shouldLog( - `ws_send_large_${ws.connectionId || 'unknown'}_${messageType || 'unknown'}`, - perfConfig.rateLimitMs, - ) - if (shouldLogSend) { - serializeMs = Number((Number(serializeEnd - serializeStart) / 1e6).toFixed(2)) - } - } - } else { - serialized = JSON.stringify(msg) - } - - const sendStart = shouldLogSend ? process.hrtime.bigint() : null - ws.send(serialized, (err) => { - if (!shouldLogSend) return - const sendMs = sendStart ? Number((Number(process.hrtime.bigint() - sendStart) / 1e6).toFixed(2)) : undefined - logPerfEvent( - 'ws_send_large', - { - connectionId: ws.connectionId, - messageType, - payloadBytes, - bufferedBytes: buffered, - serializeMs, - sendMs, - error: !!err, - }, - 'warn', - ) - }) - } catch (err) { - log.warn( - { - err: err instanceof Error ? err : new Error(String(err)), - connectionId: ws.connectionId || 'unknown', - messageType: messageType || 'unknown', - }, - 'WebSocket send failed', - ) - } + sendJsonMessage(ws, msg, { + skipBackpressureCheck, + maxBufferedAmount: this.config.maxWsBufferedAmount, + backpressureCloseCode: CLOSE_CODES.BACKPRESSURE, + backpressureCloseReason: 'Backpressure', + }) } private safeSend(ws: LiveWebSocket, msg: unknown, skipBackpressureCheck = false) { @@ -2129,6 +2059,8 @@ export class WsHandler { } state.authenticated = true state.supportsUiScreenshotV1 = !!m.capabilities?.uiScreenshotV1 + state.supportsTerminalOutputBatchV1 = !!m.capabilities?.terminalOutputBatchV1 + ws.supportsTerminalOutputBatchV1 = state.supportsTerminalOutputBatchV1 state.sidebarOpenSessionKeys = buildSidebarOpenSessionKeys( m.sidebarOpenSessions ?? [], this.serverInstanceId, @@ -2992,6 +2924,15 @@ export class WsHandler { } case 'terminal.attach': { + if (!isTerminalStreamAttachRequestIdWithinSerializedBudget(m.attachRequestId)) { + this.sendError(ws, { + code: 'INVALID_MESSAGE', + message: 'attachRequestId exceeds terminal output serialized application JSON byte budget', + terminalId: m.terminalId, + }) + return + } + const record = this.registry.get(m.terminalId) if (!record) { recordSessionLifecycleEvent({ @@ -3034,7 +2975,16 @@ export class WsHandler { m.attachRequestId, m.maxReplayBytes, m.priority ?? 'foreground', + state.supportsTerminalOutputBatchV1, ) + if (attachResult === 'invalid_attach_request_id') { + this.sendError(ws, { + code: 'INVALID_MESSAGE', + message: 'attachRequestId exceeds terminal output serialized application JSON byte budget', + terminalId: m.terminalId, + }) + return + } if (attachResult === 'missing') { const latestRecord = this.registry.get(m.terminalId) if (latestRecord && latestRecord.status !== 'running') { @@ -3212,6 +3162,8 @@ export class WsHandler { }) } this.sendError(ws, { code: 'INVALID_TERMINAL_ID', message: 'Terminal not running', terminalId: m.terminalId }) + } else { + this.terminalStreamBroker.recordResize(m.terminalId, ws, m.cols, m.rows) } return } diff --git a/server/ws-send.ts b/server/ws-send.ts new file mode 100644 index 000000000..fcc63a345 --- /dev/null +++ b/server/ws-send.ts @@ -0,0 +1,252 @@ +import WebSocket from 'ws' +import { logger } from './logger.js' +import { getPerfConfig, logPerfEvent, shouldLog } from './perf-logger.js' + +const log = logger.child({ component: 'ws-send' }) +const perfConfig = getPerfConfig() + +export const MAX_SERIALIZED_APPLICATION_JSON_BYTES = Math.max( + 1024, + readPositiveNumber( + process.env.MAX_SERIALIZED_APPLICATION_JSON_BYTES ?? process.env.WS_MAX_PAYLOAD_BYTES, + 16 * 1024 * 1024, + ), +) + +export type JsonWebSocket = { + readyState: number + bufferedAmount?: number + connectionId?: string + send: (data: string, cb?: (err?: Error) => void) => void + close?: (code?: number, reason?: string) => void +} + +export type PreparedJsonMessage = { + serialized: string + serializedApplicationJsonBytes: number + messageType?: string + serializeMs?: number +} + +export type SendJsonOptions = { + skipBackpressureCheck?: boolean + maxBufferedAmount?: number + backpressureCloseCode?: number + backpressureCloseReason?: string +} + +export type SendJsonResult = { + /** true means ws.send returned without throwing; async callback failures are logged separately. */ + sent: boolean + reason?: 'closed' | 'backpressure' | 'oversized' | 'serialize_error' | 'send_error' + serializedApplicationJsonBytes?: number + bufferedBefore?: number + bufferedAfter?: number + messageType?: string + error?: unknown +} + +export function readWebSocketBufferedAmount(ws: { bufferedAmount?: number }): number | undefined { + const buffered = ws.bufferedAmount + return typeof buffered === 'number' && Number.isFinite(buffered) ? buffered : undefined +} + +export function prepareJsonMessage(message: unknown): PreparedJsonMessage { + const serializeStart = perfConfig.enabled ? process.hrtime.bigint() : null + const serialized = JSON.stringify(message) + const serializeEnd = serializeStart ? process.hrtime.bigint() : null + if (typeof serialized !== 'string') { + throw new Error('WebSocket JSON message is not serializable') + } + + return { + serialized, + serializedApplicationJsonBytes: Buffer.byteLength(serialized, 'utf8'), + messageType: extractMessageType(message), + serializeMs: serializeStart && serializeEnd + ? Number((Number(serializeEnd - serializeStart) / 1e6).toFixed(2)) + : undefined, + } +} + +export function sendJsonMessage( + ws: JsonWebSocket, + message: unknown, + options: SendJsonOptions = {}, +): SendJsonResult { + let prepared: PreparedJsonMessage + try { + prepared = prepareJsonMessage(message) + } catch (error) { + log.warn({ + err: error instanceof Error ? error : new Error(String(error)), + connectionId: ws.connectionId || 'unknown', + messageType: extractMessageType(message) || 'unknown', + }, 'WebSocket message serialization failed') + return { + sent: false, + reason: 'serialize_error', + error, + messageType: extractMessageType(message), + bufferedBefore: readWebSocketBufferedAmount(ws), + bufferedAfter: readWebSocketBufferedAmount(ws), + } + } + + return sendPreparedJsonMessage(ws, prepared, options) +} + +export function sendPreparedJsonMessage( + ws: JsonWebSocket, + prepared: PreparedJsonMessage, + options: SendJsonOptions = {}, +): SendJsonResult { + const bufferedBefore = readWebSocketBufferedAmount(ws) + const baseResult = { + serializedApplicationJsonBytes: prepared.serializedApplicationJsonBytes, + bufferedBefore, + messageType: prepared.messageType, + } + + if (ws.readyState !== WebSocket.OPEN) { + return { + ...baseResult, + sent: false, + reason: 'closed', + bufferedAfter: readWebSocketBufferedAmount(ws), + } + } + + if (prepared.serializedApplicationJsonBytes > MAX_SERIALIZED_APPLICATION_JSON_BYTES) { + log.warn({ + connectionId: ws.connectionId || 'unknown', + messageType: prepared.messageType || 'unknown', + serializedApplicationJsonBytes: prepared.serializedApplicationJsonBytes, + maxSerializedApplicationJsonBytes: MAX_SERIALIZED_APPLICATION_JSON_BYTES, + }, 'WebSocket JSON message exceeds serialized byte budget') + return { + ...baseResult, + sent: false, + reason: 'oversized', + bufferedAfter: readWebSocketBufferedAmount(ws), + } + } + + if ( + !options.skipBackpressureCheck + && typeof options.maxBufferedAmount === 'number' + && typeof bufferedBefore === 'number' + && bufferedBefore > options.maxBufferedAmount + ) { + logBackpressureClose(ws, bufferedBefore, options.maxBufferedAmount) + try { + ws.close?.( + options.backpressureCloseCode, + options.backpressureCloseReason ?? 'Backpressure', + ) + } catch (error) { + log.warn({ + err: error instanceof Error ? error : new Error(String(error)), + connectionId: ws.connectionId || 'unknown', + }, 'WebSocket backpressure close failed') + } + return { + ...baseResult, + sent: false, + reason: 'backpressure', + bufferedAfter: readWebSocketBufferedAmount(ws), + } + } + + const shouldLogSend = shouldLogLargeSend(ws, prepared) + const sendStart = shouldLogSend ? process.hrtime.bigint() : null + try { + ws.send(prepared.serialized, (err) => { + const bufferedAfterCallback = readWebSocketBufferedAmount(ws) + if (err) { + log.warn({ + err, + connectionId: ws.connectionId || 'unknown', + messageType: prepared.messageType || 'unknown', + payloadBytes: prepared.serializedApplicationJsonBytes, + bufferedBytes: bufferedBefore, + bufferedBytesAfter: bufferedAfterCallback, + }, 'WebSocket send callback reported failure') + } + if (shouldLogSend) { + const sendMs = sendStart ? Number((Number(process.hrtime.bigint() - sendStart) / 1e6).toFixed(2)) : undefined + logPerfEvent( + 'ws_send_large', + { + connectionId: ws.connectionId, + messageType: prepared.messageType, + payloadBytes: prepared.serializedApplicationJsonBytes, + bufferedBytes: bufferedBefore, + bufferedBytesAfter: bufferedAfterCallback, + serializeMs: prepared.serializeMs, + sendMs, + error: !!err, + }, + 'warn', + ) + } + }) + } catch (error) { + log.warn({ + err: error instanceof Error ? error : new Error(String(error)), + connectionId: ws.connectionId || 'unknown', + messageType: prepared.messageType || 'unknown', + }, 'WebSocket send failed') + return { + ...baseResult, + sent: false, + reason: 'send_error', + bufferedAfter: readWebSocketBufferedAmount(ws), + error, + } + } + + return { + ...baseResult, + sent: true, + bufferedAfter: readWebSocketBufferedAmount(ws), + } +} + +function extractMessageType(message: unknown): string | undefined { + if (!message || typeof message !== 'object' || !('type' in message)) return undefined + const typeValue = (message as { type?: unknown }).type + return typeof typeValue === 'string' ? typeValue : undefined +} + +function readPositiveNumber(value: string | undefined, fallback: number): number { + const parsed = Number(value) + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback +} + +function shouldLogLargeSend(ws: JsonWebSocket, prepared: PreparedJsonMessage): boolean { + if (!perfConfig.enabled) return false + if (prepared.serializedApplicationJsonBytes < perfConfig.wsPayloadWarnBytes) return false + return shouldLog( + `ws_send_large_${ws.connectionId || 'unknown'}_${prepared.messageType || 'unknown'}`, + perfConfig.rateLimitMs, + ) +} + +function logBackpressureClose( + ws: JsonWebSocket, + bufferedBytes: number, + limitBytes: number, +): void { + if (perfConfig.enabled && shouldLog(`ws_backpressure_${ws.connectionId || 'unknown'}`, perfConfig.rateLimitMs)) { + logPerfEvent( + 'ws_backpressure_close', + { + connectionId: ws.connectionId, + bufferedBytes, + limitBytes, + }, + 'warn', + ) + } +} diff --git a/shared/ws-protocol.ts b/shared/ws-protocol.ts index 828ccf736..03ce21a77 100644 --- a/shared/ws-protocol.ts +++ b/shared/ws-protocol.ts @@ -7,6 +7,7 @@ * Client MUST use `import type` to avoid bundling Zod runtime code. */ import { z } from 'zod' +import { WS_PROTOCOL_VERSION } from './ws-version.js' import type { ClientExtensionEntry } from './extension-types.js' import type { ServerSettings } from './settings.js' import { LiveTerminalHandleSchema, SessionRefSchema, type RestoreError } from './session-contract.js' @@ -34,7 +35,7 @@ export const ErrorCode = z.enum([ export type ErrorCode = z.infer -export const WS_PROTOCOL_VERSION = 5 as const +export { WS_PROTOCOL_VERSION } export const ShellSchema = z.enum(['system', 'cmd', 'powershell', 'wsl']) @@ -228,6 +229,7 @@ export const HelloSchema = z.object({ protocolVersion: z.literal(WS_PROTOCOL_VERSION), capabilities: z.object({ uiScreenshotV1: z.boolean().optional(), + terminalOutputBatchV1: z.boolean().optional(), }).optional(), client: z.object({ mobile: z.boolean().optional(), @@ -659,6 +661,12 @@ export type TerminalCreatedMessage = { export type TerminalAttachReadyMessage = { type: 'terminal.attach.ready' terminalId: string + streamId: string + geometryEpoch?: number + geometryAuthority?: TerminalGeometryAuthority + requestedSinceSeq?: number + effectiveSinceSeq?: number + replayResetReason?: 'geometry_authority_unknown' headSeq: number replayFromSeq: number replayToSeq: number @@ -666,6 +674,16 @@ export type TerminalAttachReadyMessage = { sessionRef?: SessionLocator } +export type TerminalGeometryAuthority = 'single_client' | 'server_stream' | 'multi_client_unknown' + +export type TerminalStreamChangedMessage = { + type: 'terminal.stream.changed' + terminalId: string + streamId: string + reason: 'new_pty_session' | 'codex_pty_recovery' | 'retention_lost' | 'server_restart_incompatible_retention' + attachRequestId?: string +} + export type TerminalDetachedMessage = { type: 'terminal.detached' terminalId: string @@ -688,15 +706,40 @@ export type TerminalStatusMessage = { export type TerminalOutputMessage = { type: 'terminal.output' terminalId: string + streamId: string seqStart: number seqEnd: number data: string attachRequestId?: string + source?: 'live' | 'replay' +} + +export type TerminalOutputBatchSegment = { + seqStart: number + seqEnd: number + endOffset: number + data?: string + rawFrameCount: number + barrier?: 'control' | 'startup_probe' | 'osc52' | 'request_mode' | 'turn_complete' | 'gap' | 'geometry' +} + +export type TerminalOutputBatchMessage = { + type: 'terminal.output.batch' + terminalId: string + streamId: string + attachRequestId: string + source: 'live' | 'replay' + seqStart: number + seqEnd: number + data: string + serializedBytes: number + segments: TerminalOutputBatchSegment[] } export type TerminalOutputGapMessage = { type: 'terminal.output.gap' terminalId: string + streamId: string fromSeq: number toSeq: number reason: 'queue_overflow' | 'replay_window_exceeded' | 'replay_budget_exceeded' @@ -984,10 +1027,12 @@ export type ServerMessage = | ErrorMessage | TerminalCreatedMessage | TerminalAttachReadyMessage + | TerminalStreamChangedMessage | TerminalDetachedMessage | TerminalExitMessage | TerminalStatusMessage | TerminalOutputMessage + | TerminalOutputBatchMessage | TerminalOutputGapMessage | TerminalTitleUpdatedMessage | TerminalSessionAssociatedMessage diff --git a/shared/ws-version.ts b/shared/ws-version.ts new file mode 100644 index 000000000..5c33806d2 --- /dev/null +++ b/shared/ws-version.ts @@ -0,0 +1 @@ +export const WS_PROTOCOL_VERSION = 6 as const diff --git a/src/components/TerminalView.tsx b/src/components/TerminalView.tsx index 619641eb8..88452168f 100644 --- a/src/components/TerminalView.tsx +++ b/src/components/TerminalView.tsx @@ -38,7 +38,15 @@ import { } from '@/lib/terminal-restore' import { isTerminalPasteShortcut } from '@/lib/terminal-input-policy' import { terminalFollowsOscTitle } from '@/lib/terminal-title-policy' -import { clearTerminalCursor, loadTerminalCursor, saveTerminalCursor } from '@/lib/terminal-cursor' +import { + clearTerminalCursor, + loadTerminalSurfaceCheckpoint, + saveTerminalSurfaceCheckpoint, +} from '@/lib/terminal-cursor' +import { + canUseCheckpointForDeltaReplay, + type TerminalGeometryAuthority, +} from '@/lib/terminal-surface-checkpoint' import { resolveRevealAttachPlan, type DeferredAttachReason, @@ -49,10 +57,14 @@ import { getInstalledPerfAuditBridge } from '@/lib/perf-audit-bridge' import { beginAttach, createAttachSeqState, + markOutputRangeUnapplied, + markParserAppliedSeq, onAttachReady, + onOutputBatchSegments, onOutputFrame, onOutputGap, type AttachSeqState, + type OutputBatchAcceptedSegment, } from '@/lib/terminal-attach-seq-state' import { useMobile } from '@/hooks/useMobile' import { useKeyboardInset } from '@/hooks/useKeyboardInset' @@ -69,9 +81,17 @@ import { import { createOsc52ParserState, extractOsc52Events, + shouldAllowOsc52ClipboardWrite, + shouldAllowOsc52Prompt, type Osc52Event, type Osc52Policy, } from '@/lib/terminal-osc52' +import { + beginTerminalOutputWriteScope, + getTerminalOutputWriteScope, + shouldAllowTerminalOutputSideEffect, + type TerminalOutputSource, +} from '@/lib/terminal-output-write-scope' import { createTerminalStartupProbeState, extractTerminalStartupProbes, @@ -115,6 +135,9 @@ import { buildRestoreError, sanitizeSessionRef } from '@shared/session-contract' const log = createLogger('TerminalView') const SESSION_ACTIVITY_THROTTLE_MS = 5000 +const TERMINAL_CHECKPOINT_XTERM_VERSION = '6.0.0' +const QUARANTINE_REPAIR_POLL_MS = 16 +const QUARANTINE_REPAIR_TIMEOUT_MS = 2000 export const RATE_LIMIT_RETRY_MAX_ATTEMPTS = 5 export const RATE_LIMIT_RETRY_BASE_MS = 2000 export const RATE_LIMIT_RETRY_MAX_MS = 12000 @@ -129,6 +152,15 @@ const DEFAULT_MIN_CONTRAST_RATIO = 1 const MAX_LAST_SENT_VIEWPORT_CACHE_ENTRIES = 200 const TRUNCATED_REPLAY_BYTES = 128 * 1024 const INPUT_BLOCKED_NOTICE_THROTTLE_MS = 2000 +const TERMINAL_OUTPUT_BATCH_BARRIER_REASONS = new Set([ + 'control', + 'startup_probe', + 'osc52', + 'request_mode', + 'turn_complete', + 'gap', + 'geometry', +]) function viewportHydrateReplayOptions(content?: TerminalPaneContent | null): { maxReplayBytes: number } | undefined { return content?.mode === 'opencode' @@ -190,10 +222,25 @@ type StartupProbeReplayDiscardState = { resumeState: TerminalStartupProbeState | null } +type TerminalOutputSubmission = { + submittedWrite: boolean + submittedBytesEqualInput: boolean +} + function resolveMinimumContrastRatio(theme?: { isDark?: boolean } | null): number { return theme?.isDark === false ? LIGHT_THEME_MIN_CONTRAST_RATIO : DEFAULT_MIN_CONTRAST_RATIO } +function isUtf16SurrogateSplitOffset(data: string, offset: number): boolean { + if (offset <= 0 || offset >= data.length) return false + const previous = data.charCodeAt(offset - 1) + const next = data.charCodeAt(offset) + return previous >= 0xD800 + && previous <= 0xDBFF + && next >= 0xDC00 + && next <= 0xDFFF +} + function consumeStartupProbeReplayDiscard( raw: string, state: StartupProbeReplayDiscardState, @@ -293,12 +340,34 @@ type PendingDurableReplacement = { reason: 'opencode_replay_window_exceeded' } +type AttachTerminalOptions = { + clearViewportFirst?: boolean + suppressNextMatchingResize?: boolean + skipPreAttachFit?: boolean + maxReplayBytes?: number + priority?: TerminalAttachPriority + sinceSeq?: number +} + type SentViewport = { terminalId: string cols: number rows: number } +function parseTerminalGeometryAuthority(value: unknown): TerminalGeometryAuthority | null { + return value === 'single_client' || value === 'server_stream' || value === 'multi_client_unknown' + ? value + : null +} + +function normalizeGeometryEpoch(value: unknown, fallback: number): number { + if (typeof value === 'number' && Number.isFinite(value)) { + return Math.max(0, Math.floor(value)) + } + return Math.max(0, Math.floor(Number.isFinite(fallback) ? fallback : 1)) +} + const lastSentViewportByTerminal = new Map() function rememberSentViewport(terminalId: string, cols: number, rows: number): void { @@ -415,6 +484,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const containerRef = useRef(null) const wrapperRef = useRef(null) const termRef = useRef(null) + const terminalInstanceIdRef = useRef(`terminal-surface:${nanoid()}`) const runtimeRef = useRef(null) const writeQueueRef = useRef(null) const layoutSchedulerRef = useRef | null>(null) @@ -442,6 +512,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) resumeState: null, }) const osc52ParserRef = useRef(createOsc52ParserState()) + const settingsRef = useRef(settings) const resolvedThemeRef = useRef(getTerminalTheme(settings.terminal.theme, settings.theme)) const osc52PolicyRef = useRef(settings.terminal.osc52Clipboard) const pendingOsc52EventRef = useRef(null) @@ -491,8 +562,20 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const requestIdRef = useRef(terminalContent?.createRequestId || '') const terminalIdRef = useRef(terminalContent?.terminalId) const seqStateRef = useRef(createAttachSeqState()) - const renderedSeqRef = useRef(0) - const hasTrustedSurfaceRef = useRef(false) + const parserAppliedSeqRef = useRef(0) + const surfaceEpochRef = useRef(0) + const geometryEpochRef = useRef(1) + const geometryAuthorityRef = useRef('single_client') + const attachTerminalRef = useRef<((tid: string, intent: AttachIntent, opts?: AttachTerminalOptions) => void) | null>(null) + const quarantineRepairRef = useRef<{ + terminalId: string + attachRequestId: string + queue: TerminalWriteQueue + startedAt: number + timedOut: boolean + timer: ReturnType | null + } | null>(null) + const abandonedAttachRequestIdsRef = useRef(new Set()) const attachCounterRef = useRef(0) const currentAttachRef = useRef<{ requestId: string @@ -501,6 +584,13 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) sinceSeq: number cols: number rows: number + surfaceQuarantined: boolean + streamId?: string | null + expectedStreamId?: string | null + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + expectedGeometryEpoch: number + expectedGeometryAuthority: TerminalGeometryAuthority } | null>(null) const launchAttemptRef = useRef(null) const suppressNextMatchingResizeRef = useRef<{ @@ -532,25 +622,275 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) seqStateRef.current = nextState }, []) - const resetRenderedSurface = useCallback((seq = 0) => { - renderedSeqRef.current = Math.max(0, Math.floor(Number.isFinite(seq) ? seq : 0)) - hasTrustedSurfaceRef.current = false + const getTerminalCheckpointStreamId = useCallback((): string | null => { + const streamId = contentRef.current?.streamId + return typeof streamId === 'string' && streamId.length > 0 + ? streamId + : null }, []) - const markRenderedSeq = useCallback((terminalId: string | undefined, seq: number) => { - if (!terminalId || !Number.isFinite(seq)) return - const renderedSeq = Math.max(0, Math.floor(seq)) - if (renderedSeq <= renderedSeqRef.current) { - if (renderedSeq > 0) { - hasTrustedSurfaceRef.current = true + const getTerminalCheckpointServerInstanceId = useCallback((): string | null => { + const contentServerInstanceId = contentRef.current?.serverInstanceId + if (typeof contentServerInstanceId === 'string' && contentServerInstanceId.length > 0) { + return contentServerInstanceId + } + return typeof serverInstanceIdRef.current === 'string' && serverInstanceIdRef.current.length > 0 + ? serverInstanceIdRef.current + : null + }, []) + + const buildCheckpointReplayInput = useCallback((terminalId: string, dimensions?: { cols?: number; rows?: number }) => { + const serverInstanceId = getTerminalCheckpointServerInstanceId() + if (!terminalId || !serverInstanceId) return null + const term = termRef.current + const normalizeDimension = (value: number | undefined, fallback: number) => { + const resolved = typeof value === 'number' && Number.isFinite(value) ? value : fallback + return Math.max(2, Math.floor(resolved)) + } + const normalizeScrollback = (value: number) => ( + Math.max(0, Math.floor(Number.isFinite(value) ? value : 0)) + ) + return { + terminalId, + streamId: getTerminalCheckpointStreamId(), + serverInstanceId, + surfaceEpoch: surfaceEpochRef.current, + cols: normalizeDimension(dimensions?.cols, term?.cols ?? 80), + rows: normalizeDimension(dimensions?.rows, term?.rows ?? 24), + geometryEpoch: geometryEpochRef.current, + geometryAuthority: geometryAuthorityRef.current, + scrollback: normalizeScrollback(settingsRef.current.terminal.scrollback), + xtermVersion: TERMINAL_CHECKPOINT_XTERM_VERSION, + requireParserIdle: true, + } + }, [ + getTerminalCheckpointServerInstanceId, + getTerminalCheckpointStreamId, + ]) + + const getCheckpointDeltaReplayDecision = useCallback((terminalId: string, dimensions?: { cols?: number; rows?: number }) => { + const checkpointInput = buildCheckpointReplayInput(terminalId, dimensions) + if (!checkpointInput) { + return { ok: false as const, reason: 'missing_checkpoint' as const } + } + const checkpoint = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: checkpointInput.streamId, + serverInstanceId: checkpointInput.serverInstanceId, + }) + return canUseCheckpointForDeltaReplay(checkpoint, checkpointInput) + }, [buildCheckpointReplayInput]) + + const resetParserAppliedSurface = useCallback((seq = 0, opts?: { incrementEpoch?: boolean }) => { + parserAppliedSeqRef.current = Math.max(0, Math.floor(Number.isFinite(seq) ? seq : 0)) + if (opts?.incrementEpoch !== false) { + surfaceEpochRef.current += 1 + } + }, []) + + const syncGeometryEpochForViewport = useCallback((terminalId: string, cols: number, rows: number) => { + const lastSentViewport = lastSentViewportRef.current + if ( + lastSentViewport + && lastSentViewport.terminalId === terminalId + && (lastSentViewport.cols !== cols || lastSentViewport.rows !== rows) + ) { + geometryEpochRef.current += 1 + } + }, []) + + const clearQuarantineRepair = useCallback((attachRequestId?: string) => { + const pending = quarantineRepairRef.current + if (!pending) { + if (attachRequestId) { + abandonedAttachRequestIdsRef.current.delete(attachRequestId) } return } - renderedSeqRef.current = renderedSeq - hasTrustedSurfaceRef.current = true - saveTerminalCursor(terminalId, renderedSeq) + if (attachRequestId && pending.attachRequestId !== attachRequestId) return + if (pending.timer) { + clearTimeout(pending.timer) + } + abandonedAttachRequestIdsRef.current.delete(pending.attachRequestId) + quarantineRepairRef.current = null }, []) + const recordTerminalPerfAuditEvent = useCallback((event: string, data: Record = {}) => { + const payload = Object.fromEntries(Object.entries({ + event, + timestamp: typeof performance !== 'undefined' ? performance.now() : Date.now(), + tabId, + paneId: paneIdRef.current, + ...data, + }).filter(([, value]) => value !== undefined)) + getInstalledPerfAuditBridge()?.addPerfEvent(payload) + }, [tabId]) + + const scheduleQuarantineRepair = useCallback((terminalId: string, attachRequestId: string) => { + clearQuarantineRepair() + const queue = writeQueueRef.current + if (!queue) return + const startedAt = Date.now() + const poll = () => { + const pending = quarantineRepairRef.current + if (!pending || pending.attachRequestId !== attachRequestId) return + const activeAttach = currentAttachRef.current + if ( + !mountedRef.current + || pending.terminalId !== terminalId + || terminalIdRef.current !== terminalId + || !activeAttach + || activeAttach.terminalId !== terminalId + || activeAttach.requestId !== attachRequestId + || writeQueueRef.current !== pending.queue + ) { + clearQuarantineRepair(attachRequestId) + return + } + if (!pending.queue.hasInFlightWrites()) { + clearQuarantineRepair(attachRequestId) + attachTerminalRef.current?.(terminalId, 'viewport_hydrate', { + clearViewportFirst: true, + ...viewportHydrateReplayOptions(contentRef.current), + }) + return + } + if (!pending.timedOut && Date.now() - pending.startedAt >= QUARANTINE_REPAIR_TIMEOUT_MS) { + log.warn('Terminal quarantine repair timed out while writes remained in flight', { + paneId: paneIdRef.current, + terminalId, + attachRequestId, + }) + abandonedAttachRequestIdsRef.current.add(attachRequestId) + pending.timedOut = true + pending.queue.setActiveGeneration(`${attachRequestId}:quarantine-timeout`, { + dropQueuedStaleWrites: true, + }) + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantine_timeout', { + terminalId, + attachRequestId, + parserAppliedSeq: parserAppliedSeqRef.current, + reason: 'in_flight_writes_timeout', + }) + } + pending.timer = setTimeout(poll, QUARANTINE_REPAIR_POLL_MS) + } + quarantineRepairRef.current = { + terminalId, + attachRequestId, + queue, + startedAt, + timedOut: false, + timer: setTimeout(poll, QUARANTINE_REPAIR_POLL_MS), + } + }, [clearQuarantineRepair, recordTerminalPerfAuditEvent]) + + const markParserAppliedFrame = useCallback((terminalId: string | undefined, seq: number, attachContext?: { + requestId: string + terminalId: string + cols: number + rows: number + surfaceQuarantined?: boolean + streamId?: string | null + }) => { + if (!terminalId || !Number.isFinite(seq)) return + const parserAppliedSeq = Math.max(0, Math.floor(seq)) + const attach = attachContext ?? currentAttachRef.current + const surfaceQuarantined = attach?.surfaceQuarantined === true + if (parserAppliedSeq <= parserAppliedSeqRef.current) { + return + } + const previousParserAppliedSeq = parserAppliedSeqRef.current + parserAppliedSeqRef.current = parserAppliedSeq + recordTerminalPerfAuditEvent('terminal.parser_applied', { + terminalId, + attachRequestId: attach?.requestId, + activeAttachRequestId: currentAttachRef.current?.requestId, + streamId: attach?.streamId ?? getTerminalCheckpointStreamId(), + parserAppliedSeq, + previousParserAppliedSeq, + surfaceEpoch: surfaceEpochRef.current, + surfaceQuarantined, + }) + + if (surfaceQuarantined) return + if (!attach || attach.terminalId !== terminalId) return + if (typeof attach.streamId !== 'string' || attach.streamId.length === 0) return + const checkpointInput = buildCheckpointReplayInput(terminalId, { + cols: attach.cols, + rows: attach.rows, + }) + if (!checkpointInput) return + + saveTerminalSurfaceCheckpoint({ + terminalId: checkpointInput.terminalId, + streamId: checkpointInput.streamId, + serverInstanceId: checkpointInput.serverInstanceId, + surfaceEpoch: checkpointInput.surfaceEpoch, + attachRequestId: attach.requestId, + parserAppliedSeq, + cols: checkpointInput.cols, + rows: checkpointInput.rows, + geometryEpoch: checkpointInput.geometryEpoch, + geometryAuthority: checkpointInput.geometryAuthority, + scrollback: checkpointInput.scrollback, + xtermVersion: checkpointInput.xtermVersion, + // Task 3 cannot yet prove normal vs alternate buffer. Keep checkpoints conservative + // until the geometry/buffer authority work supplies this context. + bufferType: 'unknown', + parserIdle: true, + }) + }, [buildCheckpointReplayInput, getTerminalCheckpointStreamId, recordTerminalPerfAuditEvent]) + + const writeLocalXtermNotice = useCallback((term: Terminal, data: string) => { + const terminalInstanceId = terminalInstanceIdRef.current + if (!shouldAllowTerminalOutputSideEffect({ + terminalInstanceId, + source: 'live', + effect: 'local_xterm_notice', + mode: contentRef.current?.mode, + })) { + return + } + const invalidateAppliedSurface = () => { + resetParserAppliedSurface(parserAppliedSeqRef.current) + } + const generation = currentAttachRef.current?.requestId + const queue = writeQueueRef.current + if (queue) { + queue.enqueue(data, invalidateAppliedSurface, { mode: 'live', generation }) + return + } + const scope = beginTerminalOutputWriteScope({ + terminalInstanceId, + source: 'live', + attachRequestId: generation, + generation: generation ?? 'local-notice', + suppressExternalSideEffects: false, + }) + let didComplete = false + const complete = () => { + if (didComplete) return + didComplete = true + scope.complete() + } + try { + term.write(data, () => { + try { + invalidateAppliedSurface() + } finally { + complete() + } + }) + } catch { + // disposed + complete() + } + }, [resetParserAppliedSurface]) + + useEffect(() => () => { + clearQuarantineRepair() + }, [clearQuarantineRepair]) + // Keep refs in sync with props useEffect(() => { if (terminalContent) { @@ -566,7 +906,10 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } terminalIdRef.current = terminalContent.terminalId if (terminalContent.terminalId !== prevTerminalId) { - resetRenderedSurface() + resetParserAppliedSurface() + geometryEpochRef.current = 1 + geometryAuthorityRef.current = 'single_client' + clearQuarantineRepair() forgetSentViewport(prevTerminalId) const cachedViewport = terminalContent.terminalId ? lastSentViewportByTerminal.get(terminalContent.terminalId) @@ -574,15 +917,12 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) lastSentViewportRef.current = terminalContent.terminalId && cachedViewport ? { terminalId: terminalContent.terminalId, cols: cachedViewport.cols, rows: cachedViewport.rows } : null - const initialSeq = terminalContent.terminalId - ? loadTerminalCursor(terminalContent.terminalId) - : 0 - applySeqState(createAttachSeqState({ lastSeq: initialSeq })) + applySeqState(createAttachSeqState()) } requestIdRef.current = terminalContent.createRequestId contentRef.current = terminalContent } - }, [terminalContent, paneId, applySeqState, resetRenderedSurface]) + }, [terminalContent, paneId, applySeqState, clearQuarantineRepair, resetParserAppliedSurface]) // Register terminal buffer accessor with test harness (for E2E tests). // Uses xterm.js Terminal.buffer.active API which works with all renderers @@ -643,6 +983,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) // Sync during render (not in useEffect) so refs always have latest values paneLastInputAtRef.current = paneLastInputAt + settingsRef.current = settings debugRef.current = !!settings.logging?.debug refreshRequestRef.current = refreshRequest providerBehaviorRef.current = providerBehavior @@ -980,6 +1321,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (matchesSuppressedViewport) { suppressNextMatchingResizeRef.current = null } else if (!matchesLastSentViewport && !suppressNetworkEffects) { + syncGeometryEpochForViewport(tid, term.cols, term.rows) ws.send({ type: 'terminal.resize', terminalId: tid, cols: term.cols, rows: term.rows }) rememberSentViewport(tid, term.cols, term.rows) lastSentViewportRef.current = { terminalId: tid, cols: term.cols, rows: term.rows } @@ -994,21 +1336,39 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (shouldFocus) { term.focus() } - }, [suppressNetworkEffects, ws]) + }, [suppressNetworkEffects, syncGeometryEpochForViewport, ws]) - const enqueueTerminalWrite = useCallback((data: string, onWritten?: () => void, options?: TerminalWriteQueueOptions) => { - if (!data) return + const enqueueTerminalWrite = useCallback((data: string, onWritten?: () => void, options?: TerminalWriteQueueOptions): boolean => { + if (!data) return false const queue = writeQueueRef.current if (queue) { queue.enqueue(data, onWritten, options) - return + return true } const term = termRef.current - if (!term) return + if (!term) return false + const mode = options?.mode ?? 'live' + const generation = options?.generation ?? 'no-attach' + const scope = beginTerminalOutputWriteScope({ + terminalInstanceId: terminalInstanceIdRef.current, + source: mode, + attachRequestId: options?.generation, + generation, + suppressExternalSideEffects: mode === 'replay', + }) try { - term.write(data, onWritten) + term.write(data, () => { + try { + onWritten?.() + } finally { + scope.complete() + } + }) + return true } catch { // disposed + scope.complete() + return false } }, []) @@ -1032,15 +1392,28 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) setPendingOsc52Event(null) }, []) - const handleOsc52Event = useCallback((event: Osc52Event) => { + const handleOsc52Event = useCallback((event: Osc52Event, source: TerminalOutputSource, mode: TerminalPaneContent['mode']) => { const policy = osc52PolicyRef.current if (policy === 'always') { - attemptOsc52ClipboardWrite(event.text) + if (shouldAllowOsc52ClipboardWrite({ + terminalInstanceId: terminalInstanceIdRef.current, + source, + mode, + })) { + attemptOsc52ClipboardWrite(event.text) + } return } if (policy === 'never') { return } + if (!shouldAllowOsc52Prompt({ + terminalInstanceId: terminalInstanceIdRef.current, + source, + mode, + })) { + return + } if (pendingOsc52EventRef.current) { osc52QueueRef.current.push(event) return @@ -1072,16 +1445,22 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) mode: TerminalPaneContent['mode'], tid: string | undefined, allowReplies: boolean, - onRendered?: () => void, + onParserApplied?: () => void, writeOptions?: TerminalWriteQueueOptions, - ) => { + ): TerminalOutputSubmission => { + const outputSource = writeOptions?.mode ?? 'live' const startup = extractTerminalStartupProbes(raw, startupProbeStateRef.current, { foreground: resolvedThemeRef.current.foreground, background: resolvedThemeRef.current.background, cursor: resolvedThemeRef.current.cursor, }) - if (allowReplies) { + if (allowReplies && shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: terminalInstanceIdRef.current, + source: outputSource, + effect: 'startup_reply', + mode, + })) { for (const reply of startup.replies) { sendInput(reply) } @@ -1093,7 +1472,12 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) // claude and codex turn-completion are server-authoritative (terminal.turn.complete // broadcast). The client must not mint a completion from output (live or replayed) // for those modes — only opencode/other modes still use the client BEL path. - if (count > 0 && tid && mode !== 'claude' && mode !== 'codex') { + if (count > 0 && tid && shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: terminalInstanceIdRef.current, + source: outputSource, + effect: 'turn_complete', + mode, + })) { dispatch(recordTurnComplete({ tabId, paneId: paneIdRef.current, @@ -1102,15 +1486,19 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) })) } - if (cleaned) { - enqueueTerminalWrite(cleaned, onRendered, writeOptions) - } else { - onRendered?.() - } + const submittedBytesEqualInput = cleaned === raw + const submittedWrite = cleaned + ? enqueueTerminalWrite( + cleaned, + submittedBytesEqualInput ? onParserApplied : undefined, + writeOptions, + ) + : false for (const event of osc.events) { - handleOsc52Event(event) + handleOsc52Event(event, outputSource, mode) } + return { submittedWrite, submittedBytesEqualInput: submittedWrite && submittedBytesEqualInput } }, [dispatch, enqueueTerminalWrite, handleOsc52Event, sendInput, tabId]) const findNext = useCallback((value: string = searchQuery) => { @@ -1255,6 +1643,18 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const resolvedTheme = getTerminalTheme(settings.terminal.theme, settings.theme) resolvedThemeRef.current = resolvedTheme + const terminalInstanceId = `terminal-surface:${nanoid()}` + terminalInstanceIdRef.current = terminalInstanceId + const allowCurrentLinkAction = () => ( + terminalInstanceIdRef.current === terminalInstanceId + && shouldAllowTerminalOutputSideEffect({ + terminalInstanceId, + source: 'live', + effect: 'link_action', + mode: contentRef.current?.mode, + }) + ) + const term = new Terminal({ allowProposedApi: true, convertEol: true, @@ -1267,6 +1667,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) minimumContrastRatio: resolveMinimumContrastRatio(resolvedTheme), linkHandler: { activate: (event: MouseEvent, uri: string) => { + if (!allowCurrentLinkAction()) return if (event.button !== 0) return // Only open http/https URLs. Block javascript:, data:, and other // potentially dangerous schemes from OSC 8 links. @@ -1278,12 +1679,14 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } }, hover: (_event: MouseEvent, text: string, _range: import('@xterm/xterm').IBufferRange) => { + if (!allowCurrentLinkAction()) return setHoveredUrl(paneId, text) if (wrapperRef.current) { wrapperRef.current.dataset.hoveredUrl = text } }, leave: () => { + if (!allowCurrentLinkAction()) return clearHoveredUrl(paneId) if (wrapperRef.current) { delete wrapperRef.current.dataset.hoveredUrl @@ -1306,11 +1709,13 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) termRef.current = term runtimeRef.current = runtime const writeQueue = createTerminalWriteQueue({ + terminalInstanceId, write: (data, onWritten) => { try { term.write(data, onWritten) } catch { // disposed + onWritten?.() } }, }) @@ -1319,7 +1724,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) layoutSchedulerRef.current = layoutScheduler term.open(containerRef.current) - const requestModeBypass = registerTerminalRequestModeBypass(term, sendInput) + const requestModeBypass = registerTerminalRequestModeBypass(term, sendInput, { terminalInstanceId }) term.attachCustomWheelEventHandler((event) => { const lines = event.deltaY < 0 ? -1 : event.deltaY > 0 ? 1 : 0 if (!translateScrollLinesToInput(term, lines)) { @@ -1332,6 +1737,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) }) // Register custom link provider for clickable local file paths + const linkProviderTerminalInstanceId = terminalInstanceId const filePathLinkDisposable = typeof term.registerLinkProvider === 'function' ? term.registerLinkProvider({ provideLinks(bufferLineNumber: number, callback: (links: import('@xterm/xterm').ILink[] | undefined) => void) { @@ -1348,6 +1754,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) text: m.path, activate: (event: MouseEvent) => { if (event && event.button !== 0) return + if (terminalInstanceIdRef.current !== linkProviderTerminalInstanceId) return queuePaneSplit({ kind: 'editor', filePath: m.path, @@ -1380,6 +1787,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) text: m.url, activate: (event: MouseEvent) => { if (event && event.button !== 0) return + if (terminalInstanceIdRef.current !== linkProviderTerminalInstanceId) return if (warnExternalLinksRef.current !== false) { setPendingLinkUriRef.current(m.url) } else { @@ -1387,12 +1795,14 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } }, hover: () => { + if (terminalInstanceIdRef.current !== linkProviderTerminalInstanceId) return setHoveredUrl(paneId, m.url) if (wrapperRef.current) { wrapperRef.current.dataset.hoveredUrl = m.url } }, leave: () => { + if (terminalInstanceIdRef.current !== linkProviderTerminalInstanceId) return clearHoveredUrl(paneId) if (wrapperRef.current) { delete wrapperRef.current.dataset.hoveredUrl @@ -1403,24 +1813,51 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) }) : { dispose: () => {} } + const allowCurrentTerminalAction = () => ( + terminalInstanceIdRef.current === terminalInstanceId + && shouldAllowTerminalOutputSideEffect({ + terminalInstanceId, + source: 'live', + effect: 'terminal_action', + mode: contentRef.current?.mode, + }) + ) const unregisterActions = registerTerminalActions(paneId, { copySelection: async () => { + if (!allowCurrentTerminalAction()) return const selection = term.getSelection() - if (selection) { + if (selection && allowCurrentTerminalAction()) { await copyText(selection) } }, paste: async () => { + if (!allowCurrentTerminalAction()) return const text = await readText() + if (!allowCurrentTerminalAction()) return if (!text) return term.paste(text) }, - selectAll: () => term.selectAll(), - clearScrollback: () => term.clear(), - reset: () => term.reset(), - scrollToBottom: () => { try { term.scrollToBottom() } catch { /* disposed */ } }, - hasSelection: () => term.getSelection().length > 0, - openSearch: () => setSearchOpen(true), + selectAll: () => { + if (!allowCurrentTerminalAction()) return + term.selectAll() + }, + clearScrollback: () => { + if (!allowCurrentTerminalAction()) return + term.clear() + }, + reset: () => { + if (!allowCurrentTerminalAction()) return + term.reset() + }, + scrollToBottom: () => { + if (!allowCurrentTerminalAction()) return + try { term.scrollToBottom() } catch { /* disposed */ } + }, + hasSelection: () => allowCurrentTerminalAction() && term.getSelection().length > 0, + openSearch: () => { + if (!allowCurrentTerminalAction()) return + setSearchOpen(true) + }, }) const unregisterCaptureHandler = registerTerminalCaptureHandler(paneId, { suspendWebgl: () => runtimeRef.current?.suspendWebgl?.() ?? false, @@ -1539,6 +1976,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const wrapperEl = wrapperRef.current return () => { + clearQuarantineRepair() requestModeBypass.dispose() filePathLinkDisposable?.dispose() urlLinkDisposable?.dispose() @@ -1564,10 +2002,14 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) focus: false, } if (termRef.current === term) { + if (terminalInstanceIdRef.current === terminalInstanceId) { + terminalInstanceIdRef.current = `terminal-surface:disposed:${nanoid()}` + } runtime.dispose() runtimeRef.current = null term.dispose() termRef.current = null + mountedRef.current = false } } // eslint-disable-next-line react-hooks/exhaustive-deps @@ -1599,12 +2041,22 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (!isTerminal) return const term = termRef.current if (!term) return + const titleTerminalInstanceId = terminalInstanceIdRef.current const disposable = term.onTitleChange((rawTitle: string) => { // Only shell terminals follow the program's OSC window title. Coding-agent // terminals are named from the server session (working dir / first message // / Gemini) and must stay stable, so they ignore OSC titles entirely. if (!terminalFollowsOscTitle(contentRef.current?.mode)) return + if (terminalInstanceIdRef.current !== titleTerminalInstanceId) return + if (!shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: titleTerminalInstanceId, + source: getTerminalOutputWriteScope(titleTerminalInstanceId) ? undefined : 'live', + effect: 'title_update', + mode: contentRef.current?.mode, + })) { + return + } // Strip prefix noise (spinners, status chars) - everything before first letter const match = rawTitle.match(/[a-zA-Z]/) if (!match) return // No letters = all noise, ignore @@ -1652,6 +2104,67 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } }, [paneId, tabId]) + const markTerminalOutputRangeLost = useCallback((input: { + terminalId: string + messageType: string + attachRequestId?: string + streamId?: unknown + fromSeq?: unknown + toSeq?: unknown + reason: string + invalidReason?: string + }) => { + const previousSeqState = seqStateRef.current + const explicitFromSeq = input.fromSeq + const explicitToSeq = input.toSeq + const hasExplicitRange = typeof explicitFromSeq === 'number' + && typeof explicitToSeq === 'number' + && Number.isFinite(explicitFromSeq) + && Number.isFinite(explicitToSeq) + && Number.isInteger(explicitFromSeq) + && Number.isInteger(explicitToSeq) + && explicitFromSeq >= 0 + && explicitToSeq >= explicitFromSeq + && explicitToSeq > 0 + const fromSeq = hasExplicitRange + ? Math.max(0, Math.floor(explicitFromSeq)) + : previousSeqState.highestObservedSeq + 1 + const toSeq = hasExplicitRange + ? Math.max(fromSeq, Math.floor(explicitToSeq)) + : fromSeq + const gapDecision = onOutputGap(previousSeqState, { fromSeq, toSeq }) + const nextSeqState = gapDecision.state + applySeqState(nextSeqState) + resetParserAppliedSurface(parserAppliedSeqRef.current) + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantined', { + terminalId: input.terminalId, + messageType: input.messageType, + attachRequestId: input.attachRequestId, + activeAttachRequestId: currentAttachRef.current?.requestId, + streamId: typeof input.streamId === 'string' ? input.streamId : undefined, + fromSeq, + toSeq, + syntheticLostRange: !hasExplicitRange, + parserAppliedSeq: parserAppliedSeqRef.current, + highestObservedSeq: nextSeqState.highestObservedSeq, + reason: input.reason, + invalidReason: input.invalidReason, + }) + const completedAttachOnGap = !nextSeqState.pendingReplay + && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) + if (completedAttachOnGap) { + resetStartupProbeParser({ discardReplayRemainder: Boolean(previousSeqState.pendingReplay) }) + setIsAttaching(false) + markAttachComplete() + } + }, [ + applySeqState, + markAttachComplete, + recordTerminalPerfAuditEvent, + resetParserAppliedSurface, + resetStartupProbeParser, + ]) + const registerForBackgroundHydration = useCallback((options?: { queueIfStarted?: boolean }) => { if (hydrationRegisteredRef.current) return hydrationRegisteredRef.current = true @@ -1671,6 +2184,12 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const current = currentAttachRef.current if (!current) return true if (!msg.attachRequestId) { + recordTerminalPerfAuditEvent('terminal.attach_generation_stale_rejected', { + terminalId: msg.terminalId, + messageType: msg.type, + activeAttachRequestId: current.requestId, + reason: 'missing_attach_request_id', + }) if (debugRef.current) { log.debug('Ignoring untagged stream message for active attach generation', { paneId: paneIdRef.current, @@ -1681,20 +2200,132 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } return false } - return msg.attachRequestId === current.requestId - }, []) + if (abandonedAttachRequestIdsRef.current.has(msg.attachRequestId)) { + recordTerminalPerfAuditEvent('terminal.attach_generation_stale_rejected', { + terminalId: msg.terminalId, + messageType: msg.type, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: current.requestId, + reason: 'abandoned_attach_request_id', + }) + if (debugRef.current) { + log.debug('Ignoring abandoned attach generation message', { + paneId: paneIdRef.current, + terminalId: msg.terminalId, + type: msg.type, + attachRequestId: msg.attachRequestId, + currentAttachRequestId: current.requestId, + }) + } + return false + } + const isCurrent = msg.attachRequestId === current.requestId + if (!isCurrent) { + recordTerminalPerfAuditEvent('terminal.attach_generation_stale_rejected', { + terminalId: msg.terminalId, + messageType: msg.type, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: current.requestId, + reason: 'stale_attach_request_id', + }) + } + return isCurrent + }, [recordTerminalPerfAuditEvent]) + + const isCurrentAttachStreamMessage = useCallback((msg: { + type: string + terminalId: string + attachRequestId?: string + streamId?: unknown + seqStart?: unknown + seqEnd?: unknown + fromSeq?: unknown + toSeq?: unknown + }) => { + if (!isCurrentAttachMessage(msg)) return false + + const current = currentAttachRef.current + const activeStreamId = current?.streamId + const messageStreamId = typeof msg.streamId === 'string' && msg.streamId.length > 0 + ? msg.streamId + : null + if (activeStreamId === undefined) { + return true + } + if (typeof activeStreamId === 'string' && messageStreamId === activeStreamId) { + return true + } + + const fromSeq = typeof msg.seqStart === 'number' + ? msg.seqStart + : (typeof msg.fromSeq === 'number' ? msg.fromSeq : undefined) + const toSeq = typeof msg.seqEnd === 'number' + ? msg.seqEnd + : (typeof msg.toSeq === 'number' ? msg.toSeq : undefined) + if (typeof fromSeq === 'number' && typeof toSeq === 'number') { + const previousSeqState = seqStateRef.current + const gapDecision = onOutputGap(previousSeqState, { fromSeq, toSeq }) + const nextSeqState = gapDecision.state + applySeqState(nextSeqState) + resetParserAppliedSurface(parserAppliedSeqRef.current) + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantined', { + terminalId: msg.terminalId, + messageType: msg.type, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: current?.requestId, + activeStreamId, + streamId: messageStreamId, + fromSeq, + toSeq, + parserAppliedSeq: parserAppliedSeqRef.current, + highestObservedSeq: nextSeqState.highestObservedSeq, + reason: 'stream_identity_mismatch', + }) + const completedAttachOnGap = !nextSeqState.pendingReplay + && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) + if (completedAttachOnGap) { + resetStartupProbeParser({ discardReplayRemainder: Boolean(previousSeqState.pendingReplay) }) + setIsAttaching(false) + markAttachComplete() + } + } else { + resetParserAppliedSurface(parserAppliedSeqRef.current) + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantined', { + terminalId: msg.terminalId, + messageType: msg.type, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: current?.requestId, + activeStreamId, + streamId: messageStreamId, + parserAppliedSeq: parserAppliedSeqRef.current, + reason: 'stream_identity_mismatch', + }) + } + + log.warn('Ignoring terminal stream message with mismatched stream identity', { + paneId: paneIdRef.current, + terminalId: msg.terminalId, + type: msg.type, + attachRequestId: msg.attachRequestId, + activeStreamId, + messageStreamId, + fromSeq, + toSeq, + }) + return false + }, [ + applySeqState, + isCurrentAttachMessage, + markAttachComplete, + recordTerminalPerfAuditEvent, + resetParserAppliedSurface, + resetStartupProbeParser, + ]) const attachTerminal = useCallback(( tid: string, intent: AttachIntent, - opts?: { - clearViewportFirst?: boolean - suppressNextMatchingResize?: boolean - skipPreAttachFit?: boolean - maxReplayBytes?: number - priority?: TerminalAttachPriority - sinceSeq?: number - }, + opts?: AttachTerminalOptions, ) => { if (suppressNetworkEffects) return const term = termRef.current @@ -1709,46 +2340,117 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } const cols = Math.max(2, term.cols || 80) const rows = Math.max(2, term.rows || 24) + syncGeometryEpochForViewport(tid, cols, rows) + const attachRequestId = `${paneIdRef.current}:${++attachCounterRef.current}:${nanoid(6)}` + const writeQueue = writeQueueRef.current + const hasInFlightWrites = writeQueue?.hasInFlightWrites() === true + const expectedStreamId = getTerminalCheckpointStreamId() + const checkpointInput = buildCheckpointReplayInput(tid, { cols, rows }) + const checkpointDecision = getCheckpointDeltaReplayDecision(tid, { cols, rows }) + const expectedGeometryEpoch = checkpointInput?.geometryEpoch ?? geometryEpochRef.current + const expectedGeometryAuthority = checkpointInput?.geometryAuthority ?? geometryAuthorityRef.current + const explicitSinceSeq = typeof opts?.sinceSeq === 'number' + ? Math.max(0, Math.floor(opts.sinceSeq)) + : undefined + let effectiveIntent = intent + let clearViewportFirst = opts?.clearViewportFirst === true + let fullHydrateFallbackReason: string | null = null + if (hasInFlightWrites && effectiveIntent !== 'viewport_hydrate') { + effectiveIntent = 'viewport_hydrate' + clearViewportFirst = true + fullHydrateFallbackReason = 'in_flight_writes' + } else if (effectiveIntent !== 'viewport_hydrate' && explicitSinceSeq === undefined && !checkpointDecision.ok) { + effectiveIntent = 'viewport_hydrate' + clearViewportFirst = true + fullHydrateFallbackReason = checkpointDecision.reason + } + const deltaSeq = Math.max(0, Math.floor(explicitSinceSeq ?? (checkpointDecision.ok ? checkpointDecision.sinceSeq : 0))) + const sinceSeq = effectiveIntent === 'viewport_hydrate' ? 0 : deltaSeq + const surfaceQuarantined = hasInFlightWrites + writeQueue?.setActiveGeneration(attachRequestId, { dropQueuedStaleWrites: true }) + if (!surfaceQuarantined) { + clearQuarantineRepair() + } + if (surfaceQuarantined) { + log.warn('Quarantining terminal attach while writes are still in flight', { + paneId: paneIdRef.current, + terminalId: tid, + attachRequestId, + intent: effectiveIntent, + requestedIntent: intent, + sinceSeq, + clearViewportFirst, + }) + } + setIsAttaching(true) setTruncatedHistoryGap(null) - const renderedSeq = hasTrustedSurfaceRef.current ? renderedSeqRef.current : 0 - const deltaSeq = Math.max(0, Math.floor(opts?.sinceSeq ?? renderedSeq)) - const sinceSeq = intent === 'viewport_hydrate' ? 0 : deltaSeq - // Startup probes must not leak across attach generations. resetStartupProbeParser() - if (intent === 'viewport_hydrate') { - resetRenderedSurface() - if (opts?.clearViewportFirst) { + if (effectiveIntent === 'viewport_hydrate') { + resetParserAppliedSurface() + if (clearViewportFirst && !surfaceQuarantined) { try { termRef.current?.clear() } catch { // disposed } } - // Keep persisted cursor untouched so transport reconnect can still use high-water. applySeqState(beginAttach(createAttachSeqState({ lastSeq: 0 }))) } else { - applySeqState(beginAttach(createAttachSeqState({ lastSeq: deltaSeq }))) + applySeqState(beginAttach(createAttachSeqState({ + lastSeq: deltaSeq, + parserAppliedSeq: deltaSeq, + }))) } deferredAttachStateRef.current = { mode: 'attaching', - pendingIntent: intent, + pendingIntent: effectiveIntent, pendingSinceSeq: sinceSeq, pendingReason: opts?.priority === 'background' ? 'background_catchup' : 'initial_hydrate', } - const attachRequestId = `${paneIdRef.current}:${++attachCounterRef.current}:${nanoid(6)}` currentAttachRef.current = { requestId: attachRequestId, - intent, + intent: effectiveIntent, terminalId: tid, sinceSeq, cols, rows, + surfaceQuarantined, + expectedStreamId, + geometryEpoch: expectedGeometryEpoch, + geometryAuthority: expectedGeometryAuthority, + expectedGeometryEpoch, + expectedGeometryAuthority, + } + if (fullHydrateFallbackReason) { + recordTerminalPerfAuditEvent('terminal.catchup.full_hydrate_fallback', { + terminalId: tid, + attachRequestId, + requestedIntent: intent, + intent: effectiveIntent, + sinceSeq, + deltaSeq, + streamId: expectedStreamId, + reason: fullHydrateFallbackReason, + hasInFlightWrites, + }) + } + if (surfaceQuarantined) { + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantined', { + terminalId: tid, + attachRequestId, + requestedIntent: intent, + intent: effectiveIntent, + sinceSeq, + streamId: expectedStreamId, + parserAppliedSeq: parserAppliedSeqRef.current, + reason: 'in_flight_writes', + }) } suppressNextMatchingResizeRef.current = opts?.suppressNextMatchingResize ? { terminalId: tid, cols, rows } @@ -1757,7 +2459,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) ws.send({ type: 'terminal.attach', terminalId: tid, - intent, + intent: effectiveIntent, cols, rows, sinceSeq, @@ -1767,7 +2469,24 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) }) rememberSentViewport(tid, cols, rows) lastSentViewportRef.current = { terminalId: tid, cols, rows } - }, [suppressNetworkEffects, ws, applySeqState, resetRenderedSurface, resetStartupProbeParser]) + if (surfaceQuarantined) { + scheduleQuarantineRepair(tid, attachRequestId) + } + }, [ + suppressNetworkEffects, + ws, + applySeqState, + buildCheckpointReplayInput, + clearQuarantineRepair, + getCheckpointDeltaReplayDecision, + getTerminalCheckpointStreamId, + recordTerminalPerfAuditEvent, + resetParserAppliedSurface, + scheduleQuarantineRepair, + resetStartupProbeParser, + syncGeometryEpochForViewport, + ]) + attachTerminalRef.current = attachTerminal const runRefreshAttach = useCallback((request: PaneRefreshRequest | null | undefined) => { if (suppressNetworkEffects) return false @@ -1840,11 +2559,11 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) hydrationRegisteredRef.current = false } getHydrationQueue().onActiveTabChanged(tabId, tabOrderRef.current) + const checkpointDecision = getCheckpointDeltaReplayDecision(tid) const revealPlan = resolveRevealAttachPlan({ pendingIntent: deferred.pendingIntent, pendingReason: deferred.pendingReason, - hasTrustedSurface: hasTrustedSurfaceRef.current, - renderedSeq: renderedSeqRef.current, + checkpointDecision, }) attachTerminal(tid, revealPlan.intent, { clearViewportFirst: revealPlan.clearViewportFirst, @@ -1860,7 +2579,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } requestTerminalLayout({ fit: true, resize: true }) } - }, [hidden, isTerminal, paneId, requestTerminalLayout, tabId, attachTerminal]) + }, [hidden, isTerminal, paneId, requestTerminalLayout, tabId, attachTerminal, getCheckpointDeltaReplayDecision]) // Background hydration: triggered by the hydration queue for hidden tabs useEffect(() => { @@ -1868,11 +2587,12 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) setBackgroundHydrationTriggered(false) const tid = terminalIdRef.current if (!tid || !hiddenRef.current) return - if (hasTrustedSurfaceRef.current && renderedSeqRef.current > 0) { + const checkpointDecision = getCheckpointDeltaReplayDecision(tid) + if (checkpointDecision.ok) { attachTerminal(tid, 'keepalive_delta', { clearViewportFirst: false, priority: 'background', - sinceSeq: renderedSeqRef.current, + sinceSeq: checkpointDecision.sinceSeq, }) return } @@ -1881,7 +2601,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) priority: 'background', ...viewportHydrateReplayOptions(contentRef.current), }) - }, [backgroundHydrationTriggered, attachTerminal]) + }, [backgroundHydrationTriggered, attachTerminal, getCheckpointDeltaReplayDecision]) // Create or attach to backend terminal useEffect(() => { @@ -1982,7 +2702,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (requestIdRef.current !== requestId) return sendCreate(requestId) }, delayMs) - term.writeln(`\r\n[Rate limited - retrying in ${(delayMs / 1000).toFixed(0)}s]\r\n`) + writeLocalXtermNotice(term, `\r\n[Rate limited - retrying in ${(delayMs / 1000).toFixed(0)}s]\r\n`) return true } @@ -1995,6 +2715,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) requestIdRef.current = pending.requestId terminalIdRef.current = undefined launchAttemptRef.current = null + clearQuarantineRepair() currentAttachRef.current = null deferredAttachStateRef.current = { mode: 'none', @@ -2009,6 +2730,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) updateContent({ terminalId: undefined, serverInstanceId: undefined, + streamId: undefined, createRequestId: pending.requestId, status: 'creating', restoreError: undefined, @@ -2042,6 +2764,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) reason: 'opencode_replay_window_exceeded', } clearRateLimitRetry() + clearQuarantineRepair() currentAttachRef.current = null launchAttemptRef.current = null deferredAttachStateRef.current = { @@ -2054,14 +2777,11 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) setTruncatedHistoryGap(null) dispatch(clearPaneRuntimeActivity({ paneId: paneIdRef.current })) clearTerminalCursor(terminalId) + resetParserAppliedSurface() forgetSentViewport(terminalId) lastSentViewportRef.current = null applySeqState(createAttachSeqState()) - try { - term.writeln('\r\n[Restarting OpenCode session because the saved terminal replay is no longer available]\r\n') - } catch { - // disposed - } + writeLocalXtermNotice(term, '\r\n[Restarting OpenCode session because the saved terminal replay is no longer available]\r\n') ws.send({ type: 'terminal.kill', terminalId }) return true } @@ -2072,6 +2792,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const failLaunch = (message: string, restore: boolean, terminalId?: string) => { clearRateLimitRetry() + clearQuarantineRepair() setIsAttaching(false) currentAttachRef.current = null deferredAttachStateRef.current = { @@ -2083,33 +2804,433 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) dispatch(clearPaneRuntimeActivity({ paneId: paneIdRef.current })) if (terminalId) { clearTerminalCursor(terminalId) + resetParserAppliedSurface() forgetSentViewport(terminalId) } lastSentViewportRef.current = null terminalIdRef.current = undefined launchAttemptRef.current = null applySeqState(createAttachSeqState()) - updateContent({ terminalId: undefined, status: 'error' }) + updateContent({ terminalId: undefined, streamId: undefined, status: 'error' }) const currentTab = tabRef.current if (currentTab) { dispatch(updateTab({ id: currentTab.id, updates: { status: 'error' } })) } const prefix = restore ? '[Restore failed]' : '[Launch failed]' - term.writeln(`\r\n${prefix} ${message}\r\n`) + writeLocalXtermNotice(term, `\r\n${prefix} ${message}\r\n`) } unsub = ws.onMessage((msg) => { const tid = terminalIdRef.current const reqId = requestIdRef.current + const markFirstOutputIfNeeded = (raw: string) => { + if ( + raw.length > 0 + && !terminalFirstOutputMarkedRef.current + && activeTabId === tabId + && activePaneId === paneId + && !hiddenRef.current + ) { + getInstalledPerfAuditBridge()?.mark('terminal.first_output', { + tabId, + paneId, + terminalId: tid, + }) + terminalFirstOutputMarkedRef.current = true + } + } + + const completeParserAppliedFrame = (input: { + attachRequestId?: string + mode: TerminalPaneContent['mode'] + terminalInstanceId: string + parserAppliedSeq: number + completedAttach: boolean + }) => { + const activeAttach = currentAttachRef.current + if (!activeAttach || activeAttach.requestId !== input.attachRequestId) return + if (terminalInstanceIdRef.current !== input.terminalInstanceId) return + if (!shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: input.terminalInstanceId, + effect: 'parser_applied_checkpoint', + mode: input.mode, + generation: input.attachRequestId, + })) { + return + } + const nextSeqState = markParserAppliedSeq(seqStateRef.current, input.parserAppliedSeq) + applySeqState(nextSeqState) + markParserAppliedFrame(tid, nextSeqState.parserAppliedSeq, activeAttach) + if (input.completedAttach) { + completeAttachGeneration({ + attachRequestId: input.attachRequestId, + mode: input.mode, + terminalInstanceId: input.terminalInstanceId, + terminalId: tid, + allowWithoutWriteScope: false, + }) + } + } + + const completeAttachGeneration = (input: { + attachRequestId?: string + mode: TerminalPaneContent['mode'] + terminalInstanceId: string + terminalId?: string + allowWithoutWriteScope: boolean + }) => { + const activeAttach = currentAttachRef.current + if (!activeAttach || activeAttach.requestId !== input.attachRequestId) return false + if (input.terminalId && activeAttach.terminalId !== input.terminalId) return false + if (terminalInstanceIdRef.current !== input.terminalInstanceId) return false + const allowedByWriteScope = shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: input.terminalInstanceId, + effect: 'attach_completion', + mode: input.mode, + generation: input.attachRequestId, + }) + if (!allowedByWriteScope) { + const activeScope = getTerminalOutputWriteScope(input.terminalInstanceId) + if (!input.allowWithoutWriteScope || activeScope) return false + } + setIsAttaching(false) + markAttachComplete() + return true + } + + const submitAcceptedOutput = (input: { + raw: string + seqStart: number + seqEnd: number + attachRequestId?: string + mode: TerminalPaneContent['mode'] + previousSeqState: AttachSeqState + outputSource: TerminalOutputSource + parserAppliedSeq: number + completedAttach: boolean + disableWriteCoalescing?: boolean + }) => { + let raw = input.raw + const frameOverlapsReplay = input.outputSource === 'replay' || Boolean( + input.previousSeqState.pendingReplay + && input.seqEnd >= input.previousSeqState.pendingReplay.fromSeq + && input.seqStart <= input.previousSeqState.pendingReplay.toSeq, + ) + const enteringFreshLiveOutput = input.outputSource === 'live' + && !frameOverlapsReplay + && (Boolean(input.previousSeqState.pendingReplay) || input.previousSeqState.awaitingFreshSequence) + if ( + enteringFreshLiveOutput + && !startupProbeReplayDiscardStateRef.current.remainder + && !startupProbeReplayDiscardStateRef.current.buffered + ) { + resetStartupProbeParser({ discardReplayRemainder: Boolean(input.previousSeqState.pendingReplay) }) + } + const replayDiscard = consumeStartupProbeReplayDiscard(raw, startupProbeReplayDiscardStateRef.current) + if (replayDiscard.resumeState) { + startupProbeStateRef.current = replayDiscard.resumeState + } + raw = replayDiscard.raw + const inputBytesEqualSubmission = raw === input.raw + const outputTerminalInstanceId = terminalInstanceIdRef.current + const completeNoWriteReplayAttach = () => { + completeAttachGeneration({ + attachRequestId: input.attachRequestId, + mode: input.mode, + terminalInstanceId: outputTerminalInstanceId, + terminalId: tid, + allowWithoutWriteScope: true, + }) + } + const queueNoWriteReplayAttachCompletion = () => { + const queue = writeQueueRef.current + if (queue) { + queue.enqueueTask(completeNoWriteReplayAttach, { + mode: input.outputSource, + generation: input.attachRequestId, + }) + return + } + completeNoWriteReplayAttach() + } + const submission = handleTerminalOutput( + raw, + input.mode, + tid, + input.outputSource === 'live', + inputBytesEqualSubmission + ? () => completeParserAppliedFrame({ + attachRequestId: input.attachRequestId, + mode: input.mode, + terminalInstanceId: outputTerminalInstanceId, + parserAppliedSeq: input.parserAppliedSeq, + completedAttach: input.completedAttach, + }) + : undefined, + { + mode: input.outputSource, + generation: input.attachRequestId, + coalesce: input.disableWriteCoalescing ? false : undefined, + }, + ) + if ( + !submission.submittedWrite + || !inputBytesEqualSubmission + || !submission.submittedBytesEqualInput + ) { + applySeqState(markOutputRangeUnapplied(seqStateRef.current, { + fromSeq: input.seqStart, + toSeq: input.seqEnd, + })) + if (input.completedAttach && frameOverlapsReplay) { + queueNoWriteReplayAttachCompletion() + } + } + if (input.completedAttach && frameOverlapsReplay) { + resetStartupProbeParser({ discardReplayRemainder: true }) + } + markFirstOutputIfNeeded(raw) + } + + if (msg.type === 'terminal.output.batch' && msg.terminalId === tid) { + if (!isCurrentAttachStreamMessage(msg)) { + if (debugRef.current) { + log.debug('Ignoring stale attach generation message', { + paneId: paneIdRef.current, + terminalId: msg.terminalId, + attachRequestId: msg.attachRequestId, + currentAttachRequestId: currentAttachRef.current?.requestId, + currentAttachStreamId: currentAttachRef.current?.streamId, + streamId: msg.streamId, + type: msg.type, + }) + } + return + } + + const outputSource = msg.source === 'live' || msg.source === 'replay' + ? msg.source + : null + const batchDataInput = msg.data + const batchData = typeof batchDataInput === 'string' ? batchDataInput : '' + const rawSegmentsInput = Array.isArray(msg.segments) ? msg.segments : [] + const batchSeqStart = msg.seqStart + const batchSeqEnd = msg.seqEnd + const batchSerializedBytes = msg.serializedBytes + const batchSegments: Array<{ + seqStart: number + seqEnd: number + data: string + barrier: boolean + }> = [] + let previousEndOffset = 0 + let previousSeqEnd: number | null = null + let invalidBatchReason: string | null = null + + if (!outputSource) { + invalidBatchReason = 'invalid_source' + } else if (typeof batchDataInput !== 'string') { + invalidBatchReason = 'invalid_batch_data' + } else if (rawSegmentsInput.length === 0) { + invalidBatchReason = 'missing_segments' + } else if ( + typeof batchSeqStart !== 'number' + || typeof batchSeqEnd !== 'number' + || !Number.isFinite(batchSeqStart) + || !Number.isFinite(batchSeqEnd) + || !Number.isInteger(batchSeqStart) + || !Number.isInteger(batchSeqEnd) + || batchSeqStart < 0 + || batchSeqEnd < batchSeqStart + ) { + invalidBatchReason = 'invalid_batch_range' + } else if ( + typeof batchSerializedBytes !== 'number' + || !Number.isFinite(batchSerializedBytes) + || !Number.isInteger(batchSerializedBytes) + || batchSerializedBytes < 0 + ) { + invalidBatchReason = 'invalid_batch_serialized_bytes' + } + + for (const rawSegment of rawSegmentsInput) { + if (invalidBatchReason) break + const seqStart = rawSegment?.seqStart + const seqEnd = rawSegment?.seqEnd + const endOffset = rawSegment?.endOffset + const rawFrameCount = rawSegment?.rawFrameCount + if ( + typeof seqStart !== 'number' + || typeof seqEnd !== 'number' + || typeof endOffset !== 'number' + || typeof rawFrameCount !== 'number' + || !Number.isFinite(seqStart) + || !Number.isFinite(seqEnd) + || !Number.isFinite(endOffset) + || !Number.isFinite(rawFrameCount) + || !Number.isInteger(seqStart) + || !Number.isInteger(seqEnd) + || !Number.isInteger(endOffset) + || !Number.isInteger(rawFrameCount) + || seqStart < 0 + || seqEnd < seqStart + || endOffset < 0 + || rawFrameCount <= 0 + || rawFrameCount !== seqEnd - seqStart + 1 + ) { + invalidBatchReason = 'invalid_segment_range' + break + } + const barrier = rawSegment?.barrier + if ( + barrier !== undefined + && ( + typeof barrier !== 'string' + || !TERMINAL_OUTPUT_BATCH_BARRIER_REASONS.has(barrier) + ) + ) { + invalidBatchReason = 'invalid_segment_barrier' + break + } + if (previousSeqEnd !== null && seqStart !== previousSeqEnd + 1) { + invalidBatchReason = 'non_contiguous_segment_range' + break + } + const normalizedEndOffset = endOffset + if ( + normalizedEndOffset < previousEndOffset + || normalizedEndOffset > batchData.length + ) { + invalidBatchReason = 'invalid_segment_offset' + break + } + if ( + isUtf16SurrogateSplitOffset(batchData, previousEndOffset) + || isUtf16SurrogateSplitOffset(batchData, normalizedEndOffset) + ) { + invalidBatchReason = 'invalid_segment_offset' + break + } + const segmentData = batchData.slice(previousEndOffset, normalizedEndOffset) + if (typeof rawSegment.data === 'string' && rawSegment.data !== segmentData) { + invalidBatchReason = 'segment_data_mismatch' + break + } + batchSegments.push({ + seqStart, + seqEnd, + data: segmentData, + barrier: typeof barrier === 'string' && barrier.length > 0, + }) + previousEndOffset = normalizedEndOffset + previousSeqEnd = seqEnd + } + + if (!invalidBatchReason && previousEndOffset !== batchData.length) { + invalidBatchReason = 'trailing_batch_data' + } + if ( + !invalidBatchReason + && ( + batchSegments[0]?.seqStart !== batchSeqStart + || batchSegments[batchSegments.length - 1]?.seqEnd !== batchSeqEnd + ) + ) { + invalidBatchReason = 'batch_range_mismatch' + } + + if (invalidBatchReason) { + markTerminalOutputRangeLost({ + terminalId: tid, + messageType: msg.type, + attachRequestId: msg.attachRequestId, + streamId: msg.streamId, + fromSeq: batchSeqStart, + toSeq: batchSeqEnd, + reason: 'invalid_terminal_output_batch', + invalidReason: invalidBatchReason, + }) + if (import.meta.env.DEV) { + log.warn('Ignoring invalid terminal.output.batch', { + paneId: paneIdRef.current, + terminalId: tid, + reason: invalidBatchReason, + }) + } + return + } + if (!outputSource) return + + const previousSeqState = seqStateRef.current + const batchDecision = onOutputBatchSegments(previousSeqState, batchSegments) + if (!batchDecision.accept) { + if (import.meta.env.DEV) { + log.warn('Ignoring overlapping terminal.output.batch sequence range', { + paneId: paneIdRef.current, + terminalId: tid, + rejectedSegment: batchDecision.rejectedSegment, + lastSeq: previousSeqState.lastSeq, + }) + } + return + } + + if (tid && batchDecision.freshReset) { + clearTerminalCursor(tid) + resetParserAppliedSurface() + } + + const mode = contentRef.current?.mode || 'shell' + const completedAttachOnBatch = !batchDecision.state.pendingReplay + && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) + applySeqState(batchDecision.state) + + const containsBarrier = batchSegments.some((segment) => segment.barrier) + if (!containsBarrier) { + submitAcceptedOutput({ + raw: batchData, + seqStart: msg.seqStart, + seqEnd: msg.seqEnd, + attachRequestId: msg.attachRequestId, + mode, + previousSeqState, + outputSource, + parserAppliedSeq: batchDecision.state.highestObservedSeq, + completedAttach: completedAttachOnBatch, + }) + return + } + + batchSegments.forEach((segment, index) => { + const acceptedSegment: OutputBatchAcceptedSegment | undefined = batchDecision.segments[index] + if (!acceptedSegment) return + submitAcceptedOutput({ + raw: segment.data, + seqStart: segment.seqStart, + seqEnd: segment.seqEnd, + attachRequestId: msg.attachRequestId, + mode, + previousSeqState: acceptedSegment.previousState, + outputSource, + parserAppliedSeq: acceptedSegment.parserAppliedSeq, + completedAttach: completedAttachOnBatch && index === batchSegments.length - 1, + disableWriteCoalescing: true, + }) + }) + return + } + if (msg.type === 'terminal.output' && msg.terminalId === tid) { - if (!isCurrentAttachMessage(msg)) { + if (!isCurrentAttachStreamMessage(msg)) { if (debugRef.current) { log.debug('Ignoring stale attach generation message', { paneId: paneIdRef.current, terminalId: msg.terminalId, attachRequestId: msg.attachRequestId, currentAttachRequestId: currentAttachRef.current?.requestId, + currentAttachStreamId: currentAttachRef.current?.streamId, + streamId: msg.streamId, type: msg.type, }) } @@ -2145,74 +3266,40 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (tid && frameDecision.freshReset) { clearTerminalCursor(tid) - resetRenderedSurface() + resetParserAppliedSurface() } - let raw = msg.data || '' const mode = contentRef.current?.mode || 'shell' const frameOverlapsReplay = Boolean( previousSeqState.pendingReplay && msg.seqEnd >= previousSeqState.pendingReplay.fromSeq && msg.seqStart <= previousSeqState.pendingReplay.toSeq, ) - const enteringFreshLiveOutput = !frameOverlapsReplay - && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) - if ( - enteringFreshLiveOutput - && !startupProbeReplayDiscardStateRef.current.remainder - && !startupProbeReplayDiscardStateRef.current.buffered - ) { - resetStartupProbeParser({ discardReplayRemainder: Boolean(previousSeqState.pendingReplay) }) - } - const replayDiscard = consumeStartupProbeReplayDiscard(raw, startupProbeReplayDiscardStateRef.current) - if (replayDiscard.resumeState) { - startupProbeStateRef.current = replayDiscard.resumeState - } - raw = replayDiscard.raw const completedAttachOnFrame = !frameDecision.state.pendingReplay && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) - const completeRenderedFrame = () => { - markRenderedSeq(tid, frameDecision.state.lastSeq) - if (completedAttachOnFrame) { - setIsAttaching(false) - markAttachComplete() - } - } - handleTerminalOutput( - raw, - mode, - tid, - !frameOverlapsReplay, - completeRenderedFrame, - { mode: frameOverlapsReplay ? 'replay' : 'live' }, - ) - if (completedAttachOnFrame && frameOverlapsReplay) { - resetStartupProbeParser({ discardReplayRemainder: true }) - } - if ( - raw.length > 0 - && !terminalFirstOutputMarkedRef.current - && activeTabId === tabId - && activePaneId === paneId - && !hiddenRef.current - ) { - getInstalledPerfAuditBridge()?.mark('terminal.first_output', { - tabId, - paneId, - terminalId: tid, - }) - terminalFirstOutputMarkedRef.current = true - } applySeqState(frameDecision.state) + submitAcceptedOutput({ + raw: msg.data || '', + seqStart: msg.seqStart, + seqEnd: msg.seqEnd, + attachRequestId: msg.attachRequestId, + mode, + previousSeqState, + outputSource: frameOverlapsReplay ? 'replay' : 'live', + parserAppliedSeq: frameDecision.state.highestObservedSeq, + completedAttach: completedAttachOnFrame, + }) } if (msg.type === 'terminal.output.gap' && msg.terminalId === tid) { - if (!isCurrentAttachMessage(msg)) { + if (!isCurrentAttachStreamMessage(msg)) { if (debugRef.current) { log.debug('Ignoring stale attach generation message', { paneId: paneIdRef.current, terminalId: msg.terminalId, attachRequestId: msg.attachRequestId, currentAttachRequestId: currentAttachRef.current?.requestId, + currentAttachStreamId: currentAttachRef.current?.streamId, + streamId: msg.streamId, type: msg.type, }) } @@ -2239,16 +3326,26 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const reason = msg.reason === 'replay_window_exceeded' ? 'reconnect window exceeded' : 'slow link backlog' - try { - term.writeln(`\r\n[Output gap ${msg.fromSeq}-${msg.toSeq}: ${reason}]\r\n`) - } catch { - // disposed - } + writeLocalXtermNotice(term, `\r\n[Output gap ${msg.fromSeq}-${msg.toSeq}: ${reason}]\r\n`) } const previousSeqState = seqStateRef.current - const nextSeqState = onOutputGap(previousSeqState, { fromSeq: msg.fromSeq, toSeq: msg.toSeq }) + const gapDecision = onOutputGap(previousSeqState, { fromSeq: msg.fromSeq, toSeq: msg.toSeq }) + const nextSeqState = gapDecision.state applySeqState(nextSeqState) - markRenderedSeq(tid, nextSeqState.lastSeq) + resetParserAppliedSurface(parserAppliedSeqRef.current) + if (gapDecision.requiresSurfaceQuarantine) { + recordTerminalPerfAuditEvent('terminal.catchup.surface_quarantined', { + terminalId: tid, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: currentAttachRef.current?.requestId, + streamId: msg.streamId, + fromSeq: msg.fromSeq, + toSeq: msg.toSeq, + parserAppliedSeq: parserAppliedSeqRef.current, + highestObservedSeq: nextSeqState.highestObservedSeq, + reason: msg.reason ?? 'output_gap', + }) + } const completedAttachOnGap = !nextSeqState.pendingReplay && (Boolean(previousSeqState.pendingReplay) || previousSeqState.awaitingFreshSequence) if (completedAttachOnGap) { @@ -2258,6 +3355,41 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } } + if (msg.type === 'terminal.stream.changed' && msg.terminalId === tid) { + if (!isCurrentAttachMessage(msg)) { + if (debugRef.current) { + log.debug('Ignoring stale attach generation stream change', { + paneId: paneIdRef.current, + terminalId: msg.terminalId, + attachRequestId: msg.attachRequestId, + currentAttachRequestId: currentAttachRef.current?.requestId, + type: msg.type, + }) + } + return + } + + const nextStreamId = typeof msg.streamId === 'string' && msg.streamId.length > 0 + ? msg.streamId + : null + const previousStreamId = getTerminalCheckpointStreamId() + const activeAttach = currentAttachRef.current + if (activeAttach?.terminalId === tid && activeAttach.requestId === msg.attachRequestId) { + currentAttachRef.current = { + ...activeAttach, + streamId: nextStreamId, + } + } + resetParserAppliedSurface(parserAppliedSeqRef.current) + if (nextStreamId) { + if (previousStreamId !== nextStreamId) { + updateContent({ streamId: nextStreamId }) + } + } else if (previousStreamId) { + updateContent({ streamId: undefined }) + } + } + if (msg.type === 'terminal.attach.ready' && msg.terminalId === tid) { if (!isCurrentAttachMessage(msg)) { if (debugRef.current) { @@ -2272,6 +3404,120 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) return } + const readyStreamId = typeof msg.streamId === 'string' && msg.streamId.length > 0 + ? msg.streamId + : null + const readyGeometryAuthority = parseTerminalGeometryAuthority( + (msg as { geometryAuthority?: unknown }).geometryAuthority, + ) ?? geometryAuthorityRef.current + const readyGeometryEpoch = normalizeGeometryEpoch( + (msg as { geometryEpoch?: unknown }).geometryEpoch, + geometryEpochRef.current, + ) + const previousStreamId = getTerminalCheckpointStreamId() + const activeAttach = currentAttachRef.current + const expectedStreamId = activeAttach?.expectedStreamId ?? previousStreamId + const expectedGeometryAuthority = activeAttach?.expectedGeometryAuthority ?? geometryAuthorityRef.current + const expectedGeometryEpoch = activeAttach?.expectedGeometryEpoch ?? geometryEpochRef.current + const incompatibleDeltaStream = activeAttach?.terminalId === tid + && activeAttach.requestId === msg.attachRequestId + && activeAttach.sinceSeq > 0 + && typeof expectedStreamId === 'string' + && expectedStreamId.length > 0 + && readyStreamId !== expectedStreamId + const incompatibleDeltaGeometry = activeAttach?.terminalId === tid + && activeAttach.requestId === msg.attachRequestId + && activeAttach.sinceSeq > 0 + && ( + readyGeometryAuthority === 'multi_client_unknown' + || readyGeometryAuthority !== expectedGeometryAuthority + || readyGeometryEpoch !== expectedGeometryEpoch + ) + geometryAuthorityRef.current = readyGeometryAuthority + geometryEpochRef.current = readyGeometryEpoch + if (incompatibleDeltaStream) { + log.warn('Rejecting warm-delta terminal attach after stream identity changed', { + paneId: paneIdRef.current, + terminalId: tid, + attachRequestId: msg.attachRequestId, + expectedStreamId, + readyStreamId, + sinceSeq: activeAttach.sinceSeq, + }) + recordTerminalPerfAuditEvent('terminal.catchup.full_hydrate_fallback', { + terminalId: tid, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: activeAttach.requestId, + streamId: readyStreamId, + expectedStreamId, + sinceSeq: activeAttach.sinceSeq, + reason: 'stream_identity_changed', + }) + resetParserAppliedSurface(parserAppliedSeqRef.current) + updateContent({ streamId: undefined }) + attachTerminal(tid, 'viewport_hydrate', { + clearViewportFirst: true, + ...viewportHydrateReplayOptions(contentRef.current), + }) + return + } + if (incompatibleDeltaGeometry) { + const reason = readyGeometryAuthority === 'multi_client_unknown' + ? 'geometry_authority_unknown' + : 'geometry_changed' + log.warn('Rejecting warm-delta terminal attach after geometry authority changed', { + paneId: paneIdRef.current, + terminalId: tid, + attachRequestId: msg.attachRequestId, + expectedGeometryAuthority, + expectedGeometryEpoch, + geometryAuthority: readyGeometryAuthority, + geometryEpoch: readyGeometryEpoch, + sinceSeq: activeAttach.sinceSeq, + reason, + }) + recordTerminalPerfAuditEvent('terminal.catchup.full_hydrate_fallback', { + terminalId: tid, + attachRequestId: msg.attachRequestId, + activeAttachRequestId: activeAttach.requestId, + streamId: readyStreamId, + expectedStreamId, + geometryAuthority: readyGeometryAuthority, + geometryEpoch: readyGeometryEpoch, + expectedGeometryAuthority, + expectedGeometryEpoch, + sinceSeq: activeAttach.sinceSeq, + reason, + }) + resetParserAppliedSurface(parserAppliedSeqRef.current) + attachTerminal(tid, 'viewport_hydrate', { + clearViewportFirst: true, + ...viewportHydrateReplayOptions(contentRef.current), + }) + return + } + if (activeAttach?.terminalId === tid && activeAttach.requestId === msg.attachRequestId) { + currentAttachRef.current = { + ...activeAttach, + streamId: readyStreamId, + geometryAuthority: readyGeometryAuthority, + geometryEpoch: readyGeometryEpoch, + } + } + if (readyStreamId) { + if (previousStreamId !== readyStreamId) { + if (previousStreamId) { + resetParserAppliedSurface(parserAppliedSeqRef.current) + } + updateContent({ streamId: readyStreamId }) + } + } else { + resetParserAppliedSurface(parserAppliedSeqRef.current) + if (previousStreamId) { + updateContent({ streamId: undefined }) + } + } + if (launchAttemptRef.current?.terminalId === tid) { launchAttemptRef.current = { ...launchAttemptRef.current, @@ -2332,6 +3578,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) : {}), attachReady: false, } + clearQuarantineRepair() currentAttachRef.current = null if (debugRef.current) log.debug('[TRACE resumeSessionId] terminal.created received', { paneId: paneIdRef.current, @@ -2345,6 +3592,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) updateContent({ terminalId: newId, serverInstanceId: serverInstanceIdRef.current, + streamId: undefined, status: 'running', ...(createdSessionUpdates ?? {}), ...(msg.clearCodexDurability ? { codexDurability: undefined } : {}), @@ -2420,6 +3668,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } launchAttemptRef.current = null + clearQuarantineRepair() currentAttachRef.current = null deferredAttachStateRef.current = { mode: 'none', @@ -2429,6 +3678,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } dispatch(clearPaneRuntimeActivity({ paneId: paneIdRef.current })) clearTerminalCursor(tid) + resetParserAppliedSurface() forgetSentViewport(tid) lastSentViewportRef.current = null // Clear terminalIdRef AND the stored terminalId to prevent any subsequent @@ -2438,7 +3688,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) // would otherwise reset the ref from the Redux state on re-render. terminalIdRef.current = undefined applySeqState(createAttachSeqState()) - updateContent({ terminalId: undefined, status: 'exited' }) + updateContent({ terminalId: undefined, streamId: undefined, status: 'exited' }) const exitTab = tabRef.current if (exitTab) { const code = typeof msg.exitCode === 'number' ? msg.exitCode : undefined @@ -2515,7 +3765,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const previous = lastInputBlockedNoticeRef.current if (!previous || previous.reason !== reason || now - previous.at >= INPUT_BLOCKED_NOTICE_THROTTLE_MS) { lastInputBlockedNoticeRef.current = { reason, at: now } - term.writeln(`\r\n[${terminalInputBlockedNotice(reason)}]\r\n`) + writeLocalXtermNotice(term, `\r\n[${terminalInputBlockedNotice(reason)}]\r\n`) } return } @@ -2536,6 +3786,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) dispatch(clearPaneRuntimeActivity({ paneId: paneIdRef.current })) updateContent({ status: 'error', + streamId: undefined, ...(launchAttempt?.recoveryIntent ? { restoreError: buildRestoreError('dead_live_handle') } : {}), @@ -2547,10 +3798,24 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) const prefix = launchAttempt ? (launchAttempt.restore ? '[Restore failed]' : '[Launch failed]') : '[Error]' - term.writeln(`\r\n${prefix} ${msg.message || msg.code || 'Unknown error'}\r\n`) + writeLocalXtermNotice(term, `\r\n${prefix} ${msg.message || msg.code || 'Unknown error'}\r\n`) } - if (msg.type === 'error' && msg.code === 'INVALID_TERMINAL_ID' && !msg.requestId) { + const activeAttachForInvalidTerminalError = currentAttachRef.current + const currentAttachInvalidTerminalError = Boolean( + msg.type === 'error' + && msg.code === 'INVALID_TERMINAL_ID' + && typeof msg.requestId === 'string' + && typeof msg.terminalId === 'string' + && activeAttachForInvalidTerminalError !== null + && activeAttachForInvalidTerminalError.requestId === msg.requestId + && activeAttachForInvalidTerminalError.terminalId === msg.terminalId + ) + if ( + msg.type === 'error' + && msg.code === 'INVALID_TERMINAL_ID' + && (!msg.requestId || currentAttachInvalidTerminalError) + ) { const currentTerminalId = terminalIdRef.current const current = contentRef.current const launchAttempt = launchAttemptRef.current @@ -2558,6 +3823,8 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (debugRef.current) log.debug('[TRACE resumeSessionId] INVALID_TERMINAL_ID received', { paneId: paneIdRef.current, msgTerminalId: msg.terminalId, + requestId: msg.requestId, + currentAttachRequestId: activeAttachForInvalidTerminalError?.requestId, currentTerminalId, currentResumeSessionId: current?.resumeSessionId, currentStatus: current?.status, @@ -2573,7 +3840,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) // Show feedback if the terminal already exited (the ID was cleared by // the exit handler, so msg.terminalId no longer matches the ref) if (current?.status === 'exited') { - term.writeln('\r\n[Terminal exited - use the + button or split to start a new session]\r\n') + writeLocalXtermNotice(term, '\r\n[Terminal exited - use the + button or split to start a new session]\r\n') } return } @@ -2610,9 +3877,11 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) type: 'client.diagnostic', ...restoreDiagnostic, }) - term.writeln('\r\n[Starting a new terminal because the previous live terminal is gone and no durable session identity was saved]\r\n') + writeLocalXtermNotice(term, '\r\n[Starting a new terminal because the previous live terminal is gone and no durable session identity was saved]\r\n') const newRequestId = nanoid() launchAttemptRef.current = null + clearQuarantineRepair() + currentAttachRef.current = null clearRateLimitRetry() setIsAttaching(false) dispatch(clearPaneRuntimeActivity({ paneId: paneIdRef.current })) @@ -2620,6 +3889,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) addTerminalFreshRecoveryRequestId(newRequestId, 'fresh_after_restore_unavailable') requestIdRef.current = newRequestId clearTerminalCursor(currentTerminalId) + resetParserAppliedSurface() forgetSentViewport(currentTerminalId) lastSentViewportRef.current = null terminalIdRef.current = undefined @@ -2633,6 +3903,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) updateContent({ terminalId: undefined, serverInstanceId: undefined, + streamId: undefined, createRequestId: newRequestId, status: 'creating', restoreError: undefined, @@ -2643,7 +3914,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) } return } - term.writeln('\r\n[Reconnecting...]\r\n') + writeLocalXtermNotice(term, '\r\n[Reconnecting...]\r\n') const newRequestId = nanoid() if (debugRef.current) log.debug('[TRACE resumeSessionId] INVALID_TERMINAL_ID reconnecting', { paneId: paneIdRef.current, @@ -2660,7 +3931,10 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) consumeTerminalRestoreRequestId(requestIdRef.current) addTerminalRestoreRequestId(newRequestId) requestIdRef.current = newRequestId + clearQuarantineRepair() + currentAttachRef.current = null clearTerminalCursor(currentTerminalId) + resetParserAppliedSurface() forgetSentViewport(currentTerminalId) lastSentViewportRef.current = null terminalIdRef.current = undefined @@ -2674,6 +3948,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) updateContent({ terminalId: undefined, serverInstanceId: undefined, + streamId: undefined, createRequestId: newRequestId, status: 'creating', }) @@ -2682,7 +3957,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) dispatch(updateTab({ id: currentTab.id, updates: { status: 'creating' } })) } } else if (current?.status === 'exited') { - term.writeln('\r\n[Terminal exited - use the + button or split to start a new session]\r\n') + writeLocalXtermNotice(term, '\r\n[Terminal exited - use the + button or split to start a new session]\r\n') } } }) @@ -2696,12 +3971,13 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) }) if (!tid) return if (hiddenRef.current) { - const canResumeFromRenderedSurface = hasTrustedSurfaceRef.current && renderedSeqRef.current > 0 - deferredAttachStateRef.current = deferredAttachStateRef.current.mode === 'live' || canResumeFromRenderedSurface + const checkpointDecision = getCheckpointDeltaReplayDecision(tid) + const canResumeFromParserAppliedSurface = checkpointDecision.ok + deferredAttachStateRef.current = deferredAttachStateRef.current.mode === 'live' || canResumeFromParserAppliedSurface ? { mode: 'waiting_for_geometry', pendingIntent: 'transport_reconnect', - pendingSinceSeq: renderedSeqRef.current, + pendingSinceSeq: canResumeFromParserAppliedSurface ? checkpointDecision.sinceSeq : 0, pendingReason: 'transport_reconnect', } : { @@ -2710,7 +3986,7 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) pendingSinceSeq: 0, pendingReason: 'hidden_reveal', } - registerForBackgroundHydration({ queueIfStarted: canResumeFromRenderedSurface }) + registerForBackgroundHydration({ queueIfStarted: canResumeFromParserAppliedSurface }) return } attachTerminal(tid, 'transport_reconnect') @@ -2740,12 +4016,13 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) if (currentTerminalId) { if (hiddenRef.current) { - const canResumeFromRenderedSurface = hasTrustedSurfaceRef.current && renderedSeqRef.current > 0 - deferredAttachStateRef.current = deferredAttachStateRef.current.mode === 'live' || canResumeFromRenderedSurface + const checkpointDecision = getCheckpointDeltaReplayDecision(currentTerminalId) + const canResumeFromParserAppliedSurface = checkpointDecision.ok + deferredAttachStateRef.current = deferredAttachStateRef.current.mode === 'live' || canResumeFromParserAppliedSurface ? { mode: 'waiting_for_geometry', pendingIntent: 'transport_reconnect', - pendingSinceSeq: renderedSeqRef.current, + pendingSinceSeq: canResumeFromParserAppliedSurface ? checkpointDecision.sinceSeq : 0, pendingReason: 'transport_reconnect', } : { @@ -2815,13 +4092,21 @@ function TerminalView({ tabId, paneId, paneContent, hidden }: TerminalViewProps) dispatch, handleTerminalOutput, attachTerminal, - markRenderedSeq, + clearQuarantineRepair, + getCheckpointDeltaReplayDecision, + getTerminalCheckpointStreamId, + isCurrentAttachMessage, + isCurrentAttachStreamMessage, markAttachComplete, + markParserAppliedFrame, + markTerminalOutputRangeLost, + recordTerminalPerfAuditEvent, registerForBackgroundHydration, - resetRenderedSurface, + resetParserAppliedSurface, resetStartupProbeParser, runRefreshAttach, syncContentRefWithSessionAssociation, + writeLocalXtermNotice, ]) useEffect(() => { diff --git a/src/components/terminal/request-mode-bypass.ts b/src/components/terminal/request-mode-bypass.ts index 6517a577b..0e1d9718b 100644 --- a/src/components/terminal/request-mode-bypass.ts +++ b/src/components/terminal/request-mode-bypass.ts @@ -1,4 +1,5 @@ import type { Terminal } from '@xterm/xterm' +import { shouldAllowTerminalOutputSideEffect } from '@/lib/terminal-output-side-effects' type RequestModeStatus = 0 | 1 | 2 | 3 | 4 @@ -220,6 +221,7 @@ export function buildTerminalRequestModeResponse( export function registerTerminalRequestModeBypass( term: TerminalWithRequestModeAccess | undefined, sendInput: (data: string) => void, + options?: { terminalInstanceId?: string }, ): CsiHandlerRegistration { if (!term) { return { @@ -243,6 +245,14 @@ export function registerTerminalRequestModeBypass( : undefined if (mode === undefined) return false const response = buildTerminalRequestModeResponse(mode, ansi, snapshotTerminalRequestModes(term)) + if (!shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: options?.terminalInstanceId, + source: options?.terminalInstanceId ? undefined : 'live', + effect: 'request_mode_reply', + mode: 'shell', + })) { + return true + } sendInput(response) return true } diff --git a/src/components/terminal/terminal-write-queue.ts b/src/components/terminal/terminal-write-queue.ts index b66e75555..cffd78ffa 100644 --- a/src/components/terminal/terminal-write-queue.ts +++ b/src/components/terminal/terminal-write-queue.ts @@ -1,6 +1,13 @@ +import { beginTerminalOutputWriteScope } from '@/lib/terminal-output-write-scope' + export type TerminalWriteQueue = { enqueue: (data: string, onWritten?: () => void, options?: TerminalWriteQueueOptions) => void enqueueTask: (task: () => void, options?: TerminalWriteQueueOptions) => void + setActiveGeneration: ( + generation: string, + options?: { dropQueuedStaleWrites?: boolean }, + ) => void + hasInFlightWrites: (generation?: string) => boolean clear: () => void } @@ -8,9 +15,12 @@ export type TerminalWriteQueueMode = 'live' | 'replay' export type TerminalWriteQueueOptions = { mode?: TerminalWriteQueueMode + generation?: string + coalesce?: boolean } type TerminalWriteQueueArgs = { + terminalInstanceId: string write: (data: string, onWritten?: () => void) => void onDrain?: () => void budgetMs?: number @@ -22,6 +32,8 @@ type TerminalWriteQueueArgs = { type WriteQueueItem = { kind: 'write' mode: TerminalWriteQueueMode + generation: string | undefined + coalescible: boolean data: string callbacks: Array<() => void> } @@ -29,12 +41,13 @@ type WriteQueueItem = { type TaskQueueItem = { kind: 'task' mode: TerminalWriteQueueMode + generation: string | undefined task: () => void } type QueueItem = WriteQueueItem | TaskQueueItem -const MAX_COALESCED_REPLAY_WRITE_LENGTH = 256 * 1024 +const MAX_COALESCED_TERMINAL_WRITE_LENGTH = 256 * 1024 export function createTerminalWriteQueue(args: TerminalWriteQueueArgs): TerminalWriteQueue { const queue: QueueItem[] = [] @@ -44,26 +57,117 @@ export function createTerminalWriteQueue(args: TerminalWriteQueueArgs): Terminal const cancelFrame = args.cancelFrame ?? ((id) => cancelAnimationFrame(id)) let rafId: number | null = null let scheduled = false + let activeGeneration: string | undefined + let inFlightWrites = 0 + let submittedWriteInFlight = false + let flushing = false + const inFlightWritesByGeneration = new Map() + + const resolveGeneration = (options?: TerminalWriteQueueOptions) => options?.generation ?? activeGeneration + + const isStaleGeneration = (generation: string | undefined) => ( + activeGeneration !== undefined && generation !== activeGeneration + ) + + const dropQueuedWritesOutsideGeneration = (generation: string) => { + for (let index = queue.length - 1; index >= 0; index -= 1) { + if (queue[index]?.generation !== generation) { + queue.splice(index, 1) + } + } + } + + const incrementInFlightWrites = (generation: string | undefined) => { + inFlightWrites += 1 + inFlightWritesByGeneration.set( + generation, + (inFlightWritesByGeneration.get(generation) ?? 0) + 1, + ) + } + + const decrementInFlightWrites = (generation: string | undefined) => { + if (inFlightWrites > 0) { + inFlightWrites -= 1 + } + const generationCount = inFlightWritesByGeneration.get(generation) ?? 0 + if (generationCount <= 1) { + inFlightWritesByGeneration.delete(generation) + return + } + inFlightWritesByGeneration.set(generation, generationCount - 1) + } + + const continueAfterWriteCompletion = () => { + if (flushing) return + if (queue.length > 0) { + scheduleFlush() + return + } + args.onDrain?.() + } const runItem = (item: QueueItem) => { + if (isStaleGeneration(item.generation)) { + return + } + if (item.kind === 'task') { item.task() return } - const onWritten = item.callbacks.length > 0 - ? () => { + incrementInFlightWrites(item.generation) + submittedWriteInFlight = true + let didWriteComplete = false + const scope = beginTerminalOutputWriteScope({ + terminalInstanceId: args.terminalInstanceId, + source: item.mode, + attachRequestId: item.generation, + generation: item.generation ?? 'no-attach', + suppressExternalSideEffects: item.mode === 'replay', + }) + const onWritten = () => { + if (didWriteComplete) return + didWriteComplete = true + try { + if (!isStaleGeneration(item.generation)) { for (const callback of item.callbacks) callback() } - : undefined - args.write(item.data, onWritten) + } finally { + scope.complete() + decrementInFlightWrites(item.generation) + submittedWriteInFlight = false + continueAfterWriteCompletion() + } + } + + try { + args.write(item.data, onWritten) + } catch (error) { + if (!didWriteComplete) { + didWriteComplete = true + scope.complete() + decrementInFlightWrites(item.generation) + submittedWriteInFlight = false + } + throw error + } } const flush = () => { + if (submittedWriteInFlight) return const deadline = now() + budgetMs - while (queue.length > 0 && now() <= deadline) { - const next = queue.shift() - if (next) runItem(next) + flushing = true + try { + while (queue.length > 0 && now() <= deadline && !submittedWriteInFlight) { + const next = queue.shift() + if (next) runItem(next) + } + } finally { + flushing = false + } + if (submittedWriteInFlight) { + return } if (queue.length > 0) { scheduleFlush() @@ -86,25 +190,46 @@ export function createTerminalWriteQueue(args: TerminalWriteQueueArgs): Terminal enqueue(data, onWritten, options) { if (!data) return const mode = options?.mode ?? 'live' + const generation = resolveGeneration(options) + const coalescible = options?.coalesce !== false const callbacks = onWritten ? [onWritten] : [] const previous = queue[queue.length - 1] if ( - mode === 'replay' + coalescible && previous?.kind === 'write' - && previous.mode === 'replay' - && previous.data.length + data.length <= MAX_COALESCED_REPLAY_WRITE_LENGTH + && previous.coalescible + && previous.mode === mode + && previous.generation === generation + && previous.data.length + data.length <= MAX_COALESCED_TERMINAL_WRITE_LENGTH ) { previous.data += data previous.callbacks.push(...callbacks) } else { - queue.push({ kind: 'write', mode, data, callbacks }) + queue.push({ kind: 'write', mode, generation, coalescible, data, callbacks }) } scheduleFlush() }, enqueueTask(task, options) { - queue.push({ kind: 'task', mode: options?.mode ?? 'live', task }) + queue.push({ + kind: 'task', + mode: options?.mode ?? 'live', + generation: resolveGeneration(options), + task, + }) scheduleFlush() }, + setActiveGeneration(generation, options) { + activeGeneration = generation + if (options?.dropQueuedStaleWrites) { + dropQueuedWritesOutsideGeneration(generation) + } + }, + hasInFlightWrites(generation) { + if (generation === undefined) { + return inFlightWrites > 0 + } + return (inFlightWritesByGeneration.get(generation) ?? 0) > 0 + }, clear() { queue.length = 0 if (scheduled && rafId !== null) { diff --git a/src/lib/pane-action-registry.ts b/src/lib/pane-action-registry.ts index 68db3749d..92c1c0454 100644 --- a/src/lib/pane-action-registry.ts +++ b/src/lib/pane-action-registry.ts @@ -37,7 +37,11 @@ const browserRegistry = new Map() export function registerTerminalActions(paneId: string, actions: TerminalActions): () => void { terminalRegistry.set(paneId, actions) - return () => terminalRegistry.delete(paneId) + return () => { + if (terminalRegistry.get(paneId) === actions) { + terminalRegistry.delete(paneId) + } + } } export function getTerminalActions(paneId: string): TerminalActions | undefined { diff --git a/src/lib/terminal-attach-policy.ts b/src/lib/terminal-attach-policy.ts index aa85524de..ff5769b7b 100644 --- a/src/lib/terminal-attach-policy.ts +++ b/src/lib/terminal-attach-policy.ts @@ -1,3 +1,5 @@ +import type { CheckpointDeltaReplayDecision } from './terminal-surface-checkpoint' + export type TerminalAttachIntent = 'viewport_hydrate' | 'keepalive_delta' | 'transport_reconnect' export type TerminalAttachPriority = 'foreground' | 'background' @@ -10,10 +12,18 @@ export type DeferredAttachReason = | 'background_catchup' export type RevealAttachPolicyInput = { + pendingIntent: TerminalAttachIntent + pendingReason: DeferredAttachReason + checkpointDecision: CheckpointDeltaReplayDecision + replayHydrateCoversCompatibleGeometryHistory?: boolean +} + +type LegacyRevealAttachPolicyInput = { pendingIntent: TerminalAttachIntent pendingReason: DeferredAttachReason hasTrustedSurface: boolean renderedSeq: number + replayHydrateCoversCompatibleGeometryHistory?: boolean } export type RevealAttachPlan = { @@ -21,41 +31,70 @@ export type RevealAttachPlan = { clearViewportFirst: boolean priority: TerminalAttachPriority sinceSeq?: number + trustResultingSurfaceForDeltaReplay?: boolean } -function normalizeSeq(seq: number): number { - if (!Number.isFinite(seq)) return 0 - return Math.max(0, Math.floor(seq)) +function resolveCheckpointDecision( + input: RevealAttachPolicyInput | LegacyRevealAttachPolicyInput, +): CheckpointDeltaReplayDecision { + if ('checkpointDecision' in input) return input.checkpointDecision + + return { ok: false, reason: 'missing_checkpoint' } +} + +function replayHydrateTrust( + input: RevealAttachPolicyInput | LegacyRevealAttachPolicyInput, + checkpointDecision: CheckpointDeltaReplayDecision, +): Pick { + const replayHydrateNeedsProvenGeometry = !checkpointDecision.ok + || input.pendingReason === 'explicit_refresh' + if (!replayHydrateNeedsProvenGeometry) return {} + if (input.replayHydrateCoversCompatibleGeometryHistory === true) return {} + return { trustResultingSurfaceForDeltaReplay: false } +} + +function fullViewportHydratePlan( + input: RevealAttachPolicyInput | LegacyRevealAttachPolicyInput, + checkpointDecision: CheckpointDeltaReplayDecision, +): RevealAttachPlan { + return { + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + ...replayHydrateTrust(input, checkpointDecision), + } } -export function resolveRevealAttachPlan(input: RevealAttachPolicyInput): RevealAttachPlan { - const renderedSeq = normalizeSeq(input.renderedSeq) +export function resolveRevealAttachPlan( + input: RevealAttachPolicyInput | LegacyRevealAttachPolicyInput, +): RevealAttachPlan { + const checkpointDecision = resolveCheckpointDecision(input) + const sinceSeq = checkpointDecision.ok ? checkpointDecision.sinceSeq : undefined if (input.pendingIntent !== 'viewport_hydrate') { + if (!checkpointDecision.ok) { + return fullViewportHydratePlan(input, checkpointDecision) + } + return { intent: input.pendingIntent, clearViewportFirst: false, priority: 'foreground', - ...(input.hasTrustedSurface && renderedSeq > 0 ? { sinceSeq: renderedSeq } : {}), + sinceSeq, } } if ( input.pendingReason !== 'explicit_refresh' - && input.hasTrustedSurface - && renderedSeq > 0 + && sinceSeq ) { return { intent: 'transport_reconnect', clearViewportFirst: false, priority: 'foreground', - sinceSeq: renderedSeq, + sinceSeq, } } - return { - intent: 'viewport_hydrate', - clearViewportFirst: true, - priority: 'foreground', - } + return fullViewportHydratePlan(input, checkpointDecision) } diff --git a/src/lib/terminal-attach-seq-state.ts b/src/lib/terminal-attach-seq-state.ts index 15632b80f..8f14f63c9 100644 --- a/src/lib/terminal-attach-seq-state.ts +++ b/src/lib/terminal-attach-seq-state.ts @@ -1,31 +1,128 @@ export type PendingReplay = { fromSeq: number; toSeq: number } | null +export type LostSeqRange = { fromSeq: number; toSeq: number } export type OutputFrameDecision = | { accept: true; freshReset: boolean; state: AttachSeqState } | { accept: false; reason: 'overlap' } +export type OutputGapDecision = { + state: AttachSeqState + surfaceSafeForDeltaReplay: boolean + requiresSurfaceQuarantine: boolean +} + +export type OutputBatchAcceptedSegment = { + seqStart: number + seqEnd: number + freshReset: boolean + parserAppliedSeq: number + previousState: AttachSeqState + state: AttachSeqState +} + +export type OutputBatchDecision = + | { + accept: true + freshReset: boolean + state: AttachSeqState + segments: OutputBatchAcceptedSegment[] + } + | { + accept: false + reason: 'overlap' + rejectedSegment: { seqStart: number; seqEnd: number } + state: AttachSeqState + } + export type AttachSeqState = { + /** + * Backward-compatible alias for highestObservedSeq until TerminalView migrates + * to the explicit parser-applied checkpoint model. + */ lastSeq: number + highestObservedSeq: number + parserAppliedSeq: number awaitingFreshSequence: boolean pendingReplay: PendingReplay + knownLostRanges: LostSeqRange[] + surfaceSafeForDeltaReplay: boolean + requiresSurfaceQuarantine: boolean } -export function createAttachSeqState(input?: Partial): AttachSeqState { +function normalizeSeq(seq: unknown): number { + return typeof seq === 'number' && Number.isFinite(seq) + ? Math.max(0, Math.floor(seq)) + : 0 +} + +function normalizeLostRanges(ranges: LostSeqRange[] | undefined): LostSeqRange[] { + if (!ranges?.length) return [] + return mergeLostRanges(ranges.map((range) => { + const fromSeq = normalizeSeq(range.fromSeq) + const toSeq = Math.max(fromSeq, normalizeSeq(range.toSeq)) + return { fromSeq, toSeq } + }).filter((range) => range.toSeq > 0)) +} + +function mergeLostRanges(ranges: LostSeqRange[]): LostSeqRange[] { + if (ranges.length === 0) return [] + + const sorted = [...ranges].sort((a, b) => a.fromSeq - b.fromSeq) + const merged: LostSeqRange[] = [] + for (const range of sorted) { + const previous = merged[merged.length - 1] + if (!previous || range.fromSeq > previous.toSeq + 1) { + merged.push({ ...range }) + continue + } + previous.toSeq = Math.max(previous.toSeq, range.toSeq) + } + return merged +} + +function buildState(input: Partial): AttachSeqState { + const knownLostRanges = normalizeLostRanges(input.knownLostRanges) + const parserAppliedSeq = normalizeSeq(input.parserAppliedSeq) + const highestObservedSeq = Math.max( + normalizeSeq(input.highestObservedSeq ?? input.lastSeq), + parserAppliedSeq, + ) + const surfaceSafeForDeltaReplay = input.surfaceSafeForDeltaReplay ?? knownLostRanges.length === 0 + const requiresSurfaceQuarantine = input.requiresSurfaceQuarantine ?? !surfaceSafeForDeltaReplay + return { - lastSeq: Math.max(0, Math.floor(input?.lastSeq ?? 0)), - awaitingFreshSequence: Boolean(input?.awaitingFreshSequence), - pendingReplay: input?.pendingReplay ?? null, + lastSeq: highestObservedSeq, + highestObservedSeq, + parserAppliedSeq, + awaitingFreshSequence: Boolean(input.awaitingFreshSequence), + pendingReplay: input.pendingReplay ?? null, + knownLostRanges, + surfaceSafeForDeltaReplay, + requiresSurfaceQuarantine, } } +function toGapDecision(state: AttachSeqState): OutputGapDecision { + return { + state, + surfaceSafeForDeltaReplay: state.surfaceSafeForDeltaReplay, + requiresSurfaceQuarantine: state.requiresSurfaceQuarantine, + } +} + +export function createAttachSeqState(input?: Partial): AttachSeqState { + return buildState(input ?? {}) +} + export function beginAttach(state: AttachSeqState): AttachSeqState { - return { ...state, awaitingFreshSequence: true } + return { ...createAttachSeqState(state), awaitingFreshSequence: true } } export function onAttachReady( state: AttachSeqState, ready: { headSeq: number; replayFromSeq: number; replayToSeq: number }, ): AttachSeqState { + const current = createAttachSeqState(state) const hasReplayWindow = ready.replayFromSeq > 0 && ready.replayFromSeq <= ready.replayToSeq @@ -34,71 +131,96 @@ export function onAttachReady( // If we're still awaiting fresh attach data and the replay starts at/before // our cursor, rewind to replayFromSeq-1 so those replay frames are accepted. const shouldRewindCursorForReplay = hasReplayWindow - && state.awaitingFreshSequence - && ready.replayFromSeq <= state.lastSeq + && current.awaitingFreshSequence + && ready.replayFromSeq <= current.highestObservedSeq const replayBaseline = shouldRewindCursorForReplay ? Math.max(0, ready.replayFromSeq - 1) - : state.lastSeq + : current.highestObservedSeq const replayAlreadyCovered = hasReplayWindow && ready.replayToSeq <= replayBaseline if (hasReplayWindow && !replayAlreadyCovered) { // Keep awaitingFreshSequence true until replay/live output is actually accepted. // attach.ready arrives before replay frames, so clearing it here is premature. - return { - ...state, + return buildState({ + ...current, lastSeq: replayBaseline, + highestObservedSeq: replayBaseline, + parserAppliedSeq: Math.min(current.parserAppliedSeq, replayBaseline), pendingReplay: { fromSeq: ready.replayFromSeq, toSeq: ready.replayToSeq }, - } + }) } - return { - ...state, + return buildState({ + ...current, lastSeq: Math.max(replayBaseline, ready.headSeq), + highestObservedSeq: Math.max(replayBaseline, ready.headSeq), awaitingFreshSequence: false, pendingReplay: null, - } + }) } export function onOutputGap( state: AttachSeqState, gap: { fromSeq: number; toSeq: number }, -): AttachSeqState { - const fromSeq = Math.max(0, Math.floor(gap.fromSeq)) - const toSeq = Math.max(fromSeq, Math.floor(gap.toSeq)) - const nextLastSeq = Math.max(state.lastSeq, toSeq) - const shouldClearReplay = state.pendingReplay - ? toSeq >= state.pendingReplay.toSeq +): OutputGapDecision { + const current = createAttachSeqState(state) + const fromSeq = normalizeSeq(gap.fromSeq) + const toSeq = Math.max(fromSeq, normalizeSeq(gap.toSeq)) + const hasLostRange = toSeq > 0 + const nextHighestObservedSeq = Math.max(current.highestObservedSeq, toSeq) + const shouldClearReplay = current.pendingReplay + ? toSeq >= current.pendingReplay.toSeq : false - return { - ...state, - lastSeq: nextLastSeq, + const knownLostRanges = hasLostRange + ? mergeLostRanges([...current.knownLostRanges, { fromSeq, toSeq }]) + : current.knownLostRanges + + return toGapDecision(buildState({ + ...current, + lastSeq: nextHighestObservedSeq, + highestObservedSeq: nextHighestObservedSeq, awaitingFreshSequence: false, - pendingReplay: shouldClearReplay ? null : state.pendingReplay, - } + pendingReplay: shouldClearReplay ? null : current.pendingReplay, + knownLostRanges, + surfaceSafeForDeltaReplay: hasLostRange ? false : current.surfaceSafeForDeltaReplay, + requiresSurfaceQuarantine: hasLostRange || current.requiresSurfaceQuarantine, + })) } export function onOutputFrame( state: AttachSeqState, frame: { seqStart: number; seqEnd: number }, ): OutputFrameDecision { + const current = createAttachSeqState(state) + const seqStart = normalizeSeq(frame.seqStart) + const seqEnd = Math.max(seqStart, normalizeSeq(frame.seqEnd)) const shouldFreshReset = - state.awaitingFreshSequence - && frame.seqStart === 1 - && state.lastSeq > 0 + current.awaitingFreshSequence + && seqStart === 1 + && current.highestObservedSeq > 0 const effectiveState = shouldFreshReset - ? { ...state, lastSeq: 0, pendingReplay: null } - : state + ? buildState({ + ...current, + lastSeq: 0, + highestObservedSeq: 0, + parserAppliedSeq: 0, + pendingReplay: null, + knownLostRanges: [], + surfaceSafeForDeltaReplay: true, + requiresSurfaceQuarantine: false, + }) + : current - const overlapsExisting = frame.seqStart <= effectiveState.lastSeq - const offersNewData = frame.seqEnd > effectiveState.lastSeq + const overlapsExisting = seqStart <= effectiveState.highestObservedSeq + const offersNewData = seqEnd > effectiveState.highestObservedSeq // We treat any overlap with pendingReplay as replay-context data. Server stream-v2 // currently emits per-sequence frames, so partial-range replays that would duplicate // already-rendered bytes are not expected in practice. This assumption is load-bearing // for overlap acceptance inside pending replay windows. const inPendingReplay = Boolean( effectiveState.pendingReplay - && frame.seqEnd >= effectiveState.pendingReplay.fromSeq - && frame.seqStart <= effectiveState.pendingReplay.toSeq, + && seqEnd >= effectiveState.pendingReplay.fromSeq + && seqStart <= effectiveState.pendingReplay.toSeq, ) const allowsReplayAdvance = inPendingReplay && offersNewData const isDuplicateOrStaleOverlap = overlapsExisting && !allowsReplayAdvance @@ -109,19 +231,102 @@ export function onOutputFrame( return { accept: false, reason: 'overlap' } } - const nextLastSeq = Math.max(effectiveState.lastSeq, frame.seqEnd) - const pendingReplay = effectiveState.pendingReplay && frame.seqEnd >= effectiveState.pendingReplay.toSeq + const nextHighestObservedSeq = Math.max(effectiveState.highestObservedSeq, seqEnd) + const pendingReplay = effectiveState.pendingReplay && seqEnd >= effectiveState.pendingReplay.toSeq ? null : effectiveState.pendingReplay return { accept: true, freshReset: shouldFreshReset, - state: { + state: buildState({ ...effectiveState, - lastSeq: nextLastSeq, + lastSeq: nextHighestObservedSeq, + highestObservedSeq: nextHighestObservedSeq, pendingReplay, awaitingFreshSequence: false, - }, + }), + } +} + +export function onOutputBatchSegments( + state: AttachSeqState, + segments: Array<{ seqStart: number; seqEnd: number }>, +): OutputBatchDecision { + const initialState = createAttachSeqState(state) + let current = initialState + let freshReset = false + const acceptedSegments: OutputBatchAcceptedSegment[] = [] + + for (const segment of segments) { + const previousState = current + const decision = onOutputFrame(current, segment) + if (!decision.accept) { + return { + accept: false, + reason: decision.reason, + rejectedSegment: { + seqStart: normalizeSeq(segment.seqStart), + seqEnd: Math.max(normalizeSeq(segment.seqStart), normalizeSeq(segment.seqEnd)), + }, + state: initialState, + } + } + freshReset = freshReset || decision.freshReset + current = decision.state + acceptedSegments.push({ + seqStart: normalizeSeq(segment.seqStart), + seqEnd: Math.max(normalizeSeq(segment.seqStart), normalizeSeq(segment.seqEnd)), + freshReset: decision.freshReset, + parserAppliedSeq: decision.state.highestObservedSeq, + previousState, + state: decision.state, + }) + } + + return { + accept: true, + freshReset, + state: current, + segments: acceptedSegments, } } + +export function markParserAppliedSeq(state: AttachSeqState, seq: number): AttachSeqState { + const current = createAttachSeqState(state) + let acknowledgedSeq = Math.min(normalizeSeq(seq), current.highestObservedSeq) + for (const range of current.knownLostRanges) { + if (range.toSeq <= current.parserAppliedSeq || acknowledgedSeq < range.fromSeq) { + continue + } + if (range.fromSeq <= current.parserAppliedSeq) { + acknowledgedSeq = current.parserAppliedSeq + break + } + if (acknowledgedSeq >= range.fromSeq) { + acknowledgedSeq = range.fromSeq - 1 + break + } + } + if (acknowledgedSeq <= current.parserAppliedSeq) return current + return buildState({ + ...current, + parserAppliedSeq: acknowledgedSeq, + }) +} + +export function markOutputRangeUnapplied( + state: AttachSeqState, + range: { fromSeq: number; toSeq: number }, +): AttachSeqState { + const current = createAttachSeqState(state) + const fromSeq = normalizeSeq(range.fromSeq) + const toSeq = Math.max(fromSeq, normalizeSeq(range.toSeq)) + if (toSeq <= 0) return current + return buildState({ + ...current, + knownLostRanges: mergeLostRanges([...current.knownLostRanges, { fromSeq, toSeq }]), + surfaceSafeForDeltaReplay: false, + requiresSurfaceQuarantine: true, + }) +} diff --git a/src/lib/terminal-cursor.ts b/src/lib/terminal-cursor.ts index 5a7cc560e..812619272 100644 --- a/src/lib/terminal-cursor.ts +++ b/src/lib/terminal-cursor.ts @@ -1,14 +1,28 @@ import { createLogger } from '@/lib/client-logger' +import { + createTerminalSurfaceCheckpoint, + type TerminalSurfaceCheckpoint, + type TerminalGeometryAuthority, + type TerminalBufferType, +} from '@/lib/terminal-surface-checkpoint' import { TERMINAL_CURSOR_STORAGE_KEY } from '@/store/storage-keys' const log = createLogger('TerminalCursor') -export type CursorEntry = { - seq: number +export type CheckpointEntry = { + checkpoint: TerminalSurfaceCheckpoint updatedAt: number } -type CursorMap = Record +export type CursorEntry = CheckpointEntry + +export type TerminalSurfaceCheckpointIdentity = { + streamId: string | null + serverInstanceId: string + serverBootId?: string +} + +type CursorMap = Record const MAX_ENTRIES = 500 const MAX_AGE_MS = 14 * 24 * 60 * 60 * 1000 @@ -36,6 +50,58 @@ function normalizeTimestamp(value: unknown): number { return at >= 0 ? at : 0 } +function stringOrNull(value: unknown): string | null { + return typeof value === 'string' ? value : null +} + +function optionalString(value: unknown): string | undefined { + return typeof value === 'string' && value.length > 0 ? value : undefined +} + +function isGeometryAuthority(value: unknown): value is TerminalGeometryAuthority { + return value === 'single_client' + || value === 'server_stream' + || value === 'multi_client_unknown' +} + +function isBufferType(value: unknown): value is TerminalBufferType { + return value === 'normal' || value === 'alternate' || value === 'unknown' +} + +function sanitizeCheckpoint( + terminalId: string, + raw: unknown, +): TerminalSurfaceCheckpoint | null { + if (!raw || typeof raw !== 'object') return null + const candidate = raw as Record + if (candidate.terminalId !== terminalId) return null + if (typeof candidate.serverInstanceId !== 'string' || candidate.serverInstanceId.length === 0) return null + if (typeof candidate.attachRequestId !== 'string' || candidate.attachRequestId.length === 0) return null + if (typeof candidate.xtermVersion !== 'string' || candidate.xtermVersion.length === 0) return null + if (!isGeometryAuthority(candidate.geometryAuthority)) return null + + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId, + streamId: stringOrNull(candidate.streamId), + serverInstanceId: candidate.serverInstanceId, + serverBootId: optionalString(candidate.serverBootId), + surfaceEpoch: normalizeSeq(candidate.surfaceEpoch), + attachRequestId: candidate.attachRequestId, + parserAppliedSeq: normalizeSeq(candidate.parserAppliedSeq), + cols: normalizeSeq(candidate.cols), + rows: normalizeSeq(candidate.rows), + geometryEpoch: normalizeSeq(candidate.geometryEpoch), + geometryAuthority: candidate.geometryAuthority, + scrollback: normalizeSeq(candidate.scrollback), + xtermVersion: candidate.xtermVersion, + bufferType: isBufferType(candidate.bufferType) ? candidate.bufferType : 'unknown', + parserIdle: candidate.parserIdle === true, + }) + + if (checkpoint.parserAppliedSeq <= 0) return null + return checkpoint +} + function sanitizeMap(raw: unknown): CursorMap { if (!raw || typeof raw !== 'object') return {} const input = raw as Record @@ -46,11 +112,11 @@ function sanitizeMap(raw: unknown): CursorMap { if (!value || typeof value !== 'object') continue const candidate = value as Record - const seq = normalizeSeq(candidate.seq) + const checkpoint = sanitizeCheckpoint(terminalId, candidate.checkpoint) const updatedAt = normalizeTimestamp(candidate.updatedAt) - if (seq <= 0 || updatedAt <= 0) continue + if (!checkpoint || updatedAt <= 0) continue - out[terminalId] = { seq, updatedAt } + out[terminalId] = { checkpoint, updatedAt } } return out @@ -85,7 +151,8 @@ function areCursorMapsEqual(a: CursorMap, b: CursorMap): boolean { const aEntry = a[key] const bEntry = b[key] if (!aEntry || !bEntry) return false - if (aEntry.seq !== bEntry.seq || aEntry.updatedAt !== bEntry.updatedAt) return false + if (aEntry.updatedAt !== bEntry.updatedAt) return false + if (JSON.stringify(aEntry.checkpoint) !== JSON.stringify(bEntry.checkpoint)) return false } return true @@ -151,22 +218,42 @@ function ensureLoaded(): CursorMap { return cache } -export function loadTerminalCursor(terminalId: string): number { - if (!terminalId) return 0 - const map = ensureLoaded() - return map[terminalId]?.seq ?? 0 +function sameCheckpointSurface( + a: TerminalSurfaceCheckpoint, + b: TerminalSurfaceCheckpoint, +): boolean { + return a.terminalId === b.terminalId + && a.streamId === b.streamId + && a.serverInstanceId === b.serverInstanceId + && a.serverBootId === b.serverBootId + && a.surfaceEpoch === b.surfaceEpoch + && a.cols === b.cols + && a.rows === b.rows + && a.geometryEpoch === b.geometryEpoch + && a.geometryAuthority === b.geometryAuthority + && a.scrollback === b.scrollback + && a.xtermVersion === b.xtermVersion + && a.bufferType === b.bufferType } -export function saveTerminalCursor(terminalId: string, seq: number): void { - if (!terminalId) return - const normalizedSeq = normalizeSeq(seq) - if (normalizedSeq <= 0) return +function chooseCheckpoint( + existing: TerminalSurfaceCheckpoint | undefined, + next: TerminalSurfaceCheckpoint, +): TerminalSurfaceCheckpoint { + if (!existing) return next + if (!sameCheckpointSurface(existing, next)) return next + if (existing.parserAppliedSeq > next.parserAppliedSeq) return existing + return next +} + +function saveCheckpointEntry(checkpoint: TerminalSurfaceCheckpoint): void { + if (!checkpoint.terminalId || checkpoint.parserAppliedSeq <= 0) return const map = ensureLoaded() const now = Date.now() - const existing = map[terminalId] - const nextSeq = Math.max(existing?.seq ?? 0, normalizedSeq) - map[terminalId] = { seq: nextSeq, updatedAt: now } + const existing = map[checkpoint.terminalId] + const nextCheckpoint = chooseCheckpoint(existing?.checkpoint, checkpoint) + map[checkpoint.terminalId] = { checkpoint: nextCheckpoint, updatedAt: now } const shouldPrune = Object.keys(map).length > MAX_ENTRIES || now - lastPruneAt >= PRUNE_INTERVAL_MS @@ -181,6 +268,37 @@ export function saveTerminalCursor(terminalId: string, seq: number): void { schedulePersist() } +export function loadTerminalSurfaceCheckpoint( + terminalId: string, + identity: TerminalSurfaceCheckpointIdentity, +): TerminalSurfaceCheckpoint | null { + if (!terminalId) return null + const entry = ensureLoaded()[terminalId] + if (!entry) return null + + const checkpoint = entry.checkpoint + if (checkpoint.terminalId !== terminalId) return null + if (checkpoint.streamId !== (identity.streamId ?? null)) return null + if (checkpoint.serverInstanceId !== identity.serverInstanceId) return null + if ((checkpoint.serverBootId ?? null) !== (identity.serverBootId ?? null)) return null + + return { ...checkpoint } +} + +export function saveTerminalSurfaceCheckpoint(input: TerminalSurfaceCheckpoint): void { + saveCheckpointEntry(createTerminalSurfaceCheckpoint(input)) +} + +export function loadTerminalCursor(terminalId: string): number { + void terminalId + return 0 +} + +export function saveTerminalCursor(terminalId: string, seq: number): void { + void terminalId + void seq +} + export function clearTerminalCursor(terminalId: string): void { if (!terminalId) return const map = ensureLoaded() diff --git a/src/lib/terminal-osc52.ts b/src/lib/terminal-osc52.ts index 0c34a48da..20c2a1f0e 100644 --- a/src/lib/terminal-osc52.ts +++ b/src/lib/terminal-osc52.ts @@ -1,3 +1,6 @@ +import { shouldAllowTerminalOutputSideEffect } from './terminal-output-side-effects.js' +import type { TerminalOutputSource } from './terminal-output-write-scope.js' + const ESC = '\u001b' const BEL = '\u0007' const C1_ST = '\u009c' @@ -118,3 +121,27 @@ export function extractOsc52Events( return { cleaned, events } } + +type Osc52SideEffectInput = { + terminalInstanceId?: string + source?: TerminalOutputSource + mode?: string +} + +export function shouldAllowOsc52Prompt(input: Osc52SideEffectInput): boolean { + return shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: input.terminalInstanceId, + source: input.source, + effect: 'osc52_prompt', + mode: input.mode, + }) +} + +export function shouldAllowOsc52ClipboardWrite(input: Osc52SideEffectInput): boolean { + return shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: input.terminalInstanceId, + source: input.source, + effect: 'osc52_clipboard_write', + mode: input.mode, + }) +} diff --git a/src/lib/terminal-output-side-effects.ts b/src/lib/terminal-output-side-effects.ts new file mode 100644 index 000000000..e761304c2 --- /dev/null +++ b/src/lib/terminal-output-side-effects.ts @@ -0,0 +1,92 @@ +import { + getTerminalOutputWriteScope, + type TerminalOutputSideEffect, + type TerminalOutputSource, +} from './terminal-output-write-scope.js' + +export type TerminalOutputSideEffectMode = 'shell' | 'claude' | 'codex' | 'opencode' | string + +export type ShouldAllowTerminalOutputSideEffectInput = { + terminalInstanceId?: string + source?: TerminalOutputSource + effect: TerminalOutputSideEffect + mode?: TerminalOutputSideEffectMode + generation?: string +} + +const INTERNAL_WRITE_CALLBACK_EFFECTS = new Set([ + 'parser_applied_checkpoint', + 'attach_completion', + 'cursor_persist', +]) + +const LIVE_EXTERNAL_EFFECTS = new Set([ + 'startup_reply', + 'osc52_prompt', + 'osc52_clipboard_write', + 'request_mode_reply', + 'title_update', + 'turn_complete', + 'link_action', + 'terminal_action', + 'local_xterm_notice', +]) + +function isServerAuthoritativeTurnCompleteMode(mode: string | undefined): boolean { + return mode === 'claude' || mode === 'codex' +} + +export function shouldAllowTerminalOutputSideEffect( + input: ShouldAllowTerminalOutputSideEffectInput, +): boolean { + if (input.source) { + if (input.source === 'replay') { + return INTERNAL_WRITE_CALLBACK_EFFECTS.has(input.effect) + && Boolean(getTerminalOutputWriteScope(input.terminalInstanceId)) + } + + if (input.effect === 'turn_complete') { + return !isServerAuthoritativeTurnCompleteMode(input.mode) + } + + if (INTERNAL_WRITE_CALLBACK_EFFECTS.has(input.effect)) { + return true + } + + if (LIVE_EXTERNAL_EFFECTS.has(input.effect)) { + return true + } + + return false + } + + const scope = getTerminalOutputWriteScope(input.terminalInstanceId) + if (input.generation && scope && scope.generation !== input.generation) { + return false + } + + if (scope?.suppressExternalSideEffects === true && !INTERNAL_WRITE_CALLBACK_EFFECTS.has(input.effect)) { + return false + } + + const source = scope?.source + if (!source) return false + + if (source === 'replay') { + return INTERNAL_WRITE_CALLBACK_EFFECTS.has(input.effect) && Boolean(scope) + } + + if (input.effect === 'turn_complete') { + return !isServerAuthoritativeTurnCompleteMode(input.mode) + } + + if (INTERNAL_WRITE_CALLBACK_EFFECTS.has(input.effect)) { + return true + } + + if (LIVE_EXTERNAL_EFFECTS.has(input.effect)) { + return true + } + + return false +} diff --git a/src/lib/terminal-output-write-scope.ts b/src/lib/terminal-output-write-scope.ts new file mode 100644 index 000000000..e9eb9736b --- /dev/null +++ b/src/lib/terminal-output-write-scope.ts @@ -0,0 +1,50 @@ +export type TerminalOutputSource = 'live' | 'replay' + +export type TerminalOutputSideEffect = + | 'startup_reply' + | 'osc52_prompt' + | 'osc52_clipboard_write' + | 'request_mode_reply' + | 'title_update' + | 'turn_complete' + | 'parser_applied_checkpoint' + | 'attach_completion' + | 'cursor_persist' + | 'link_action' + | 'terminal_action' + | 'local_xterm_notice' + +export type TerminalOutputWriteContext = { + terminalInstanceId: string + source: TerminalOutputSource + attachRequestId: string | undefined + generation: string + suppressExternalSideEffects: boolean +} + +const activeScopes = new Map() + +export function getTerminalOutputWriteScope( + terminalInstanceId: string | undefined, +): TerminalOutputWriteContext | null { + if (!terminalInstanceId) return null + return activeScopes.get(terminalInstanceId) ?? null +} + +export function beginTerminalOutputWriteScope( + context: TerminalOutputWriteContext, +): { complete: () => void } { + activeScopes.set(context.terminalInstanceId, context) + let completed = false + return { + complete: () => { + if (completed) return + completed = true + if (activeScopes.get(context.terminalInstanceId) === context) { + activeScopes.delete(context.terminalInstanceId) + } + }, + } +} + +export { shouldAllowTerminalOutputSideEffect } from './terminal-output-side-effects.js' diff --git a/src/lib/terminal-surface-checkpoint.ts b/src/lib/terminal-surface-checkpoint.ts new file mode 100644 index 000000000..86b530a49 --- /dev/null +++ b/src/lib/terminal-surface-checkpoint.ts @@ -0,0 +1,148 @@ +export type TerminalBufferType = 'normal' | 'alternate' | 'unknown' +export type TerminalGeometryAuthority = 'single_client' | 'server_stream' | 'multi_client_unknown' + +export type TerminalSurfaceCheckpoint = { + terminalId: string + streamId: string | null + serverInstanceId: string + serverBootId?: string + surfaceEpoch: number + attachRequestId: string + parserAppliedSeq: number + cols: number + rows: number + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + scrollback: number + xtermVersion: string + bufferType: TerminalBufferType + parserIdle: boolean +} + +export type CheckpointDeltaReplayInput = { + terminalId: string + streamId: string | null + serverInstanceId: string + serverBootId?: string + surfaceEpoch: number + cols: number + rows: number + geometryEpoch: number + geometryAuthority: TerminalGeometryAuthority + scrollback: number + xtermVersion: string + requireParserIdle: boolean +} + +export type CheckpointDeltaReplayDecision = + | { ok: true; sinceSeq: number } + | { + ok: false + reason: + | 'missing_checkpoint' + | 'terminal_changed' + | 'stream_changed' + | 'server_changed' + | 'surface_changed' + | 'geometry_changed' + | 'geometry_authority_unknown' + | 'scrollback_changed' + | 'xterm_version_changed' + | 'parser_busy' + | 'no_applied_sequence' + } + +function normalizeNonNegativeInteger(value: number): number { + if (!Number.isFinite(value)) return 0 + return Math.max(0, Math.floor(value)) +} + +function normalizeCheckpoint(input: TerminalSurfaceCheckpoint): TerminalSurfaceCheckpoint { + return { + ...input, + streamId: input.streamId ?? null, + surfaceEpoch: normalizeNonNegativeInteger(input.surfaceEpoch), + parserAppliedSeq: normalizeNonNegativeInteger(input.parserAppliedSeq), + cols: normalizeNonNegativeInteger(input.cols), + rows: normalizeNonNegativeInteger(input.rows), + geometryEpoch: normalizeNonNegativeInteger(input.geometryEpoch), + scrollback: normalizeNonNegativeInteger(input.scrollback), + } +} + +function normalizeReplayInput(input: CheckpointDeltaReplayInput): CheckpointDeltaReplayInput { + return { + ...input, + streamId: input.streamId ?? null, + surfaceEpoch: normalizeNonNegativeInteger(input.surfaceEpoch), + cols: normalizeNonNegativeInteger(input.cols), + rows: normalizeNonNegativeInteger(input.rows), + geometryEpoch: normalizeNonNegativeInteger(input.geometryEpoch), + scrollback: normalizeNonNegativeInteger(input.scrollback), + } +} + +export function createTerminalSurfaceCheckpoint( + input: TerminalSurfaceCheckpoint, +): TerminalSurfaceCheckpoint { + return normalizeCheckpoint(input) +} + +export function canUseCheckpointForDeltaReplay( + checkpoint: TerminalSurfaceCheckpoint | null | undefined, + input: CheckpointDeltaReplayInput, +): CheckpointDeltaReplayDecision { + if (!checkpoint) return { ok: false, reason: 'missing_checkpoint' } + + const current = normalizeReplayInput(input) + const saved = normalizeCheckpoint(checkpoint) + + if (saved.terminalId !== current.terminalId) { + return { ok: false, reason: 'terminal_changed' } + } + // Stream identity is required for v2 delta replay. A missing/null stream id + // is a protocol failure, not a compatible legacy identity. + if (!saved.streamId || !current.streamId) { + return { ok: false, reason: 'stream_changed' } + } + if (saved.streamId !== current.streamId) { + return { ok: false, reason: 'stream_changed' } + } + if ( + saved.serverInstanceId !== current.serverInstanceId + || (saved.serverBootId ?? null) !== (current.serverBootId ?? null) + ) { + return { ok: false, reason: 'server_changed' } + } + if (saved.surfaceEpoch !== current.surfaceEpoch) { + return { ok: false, reason: 'surface_changed' } + } + if ( + saved.cols !== current.cols + || saved.rows !== current.rows + || saved.geometryEpoch !== current.geometryEpoch + ) { + return { ok: false, reason: 'geometry_changed' } + } + if ( + saved.geometryAuthority === 'multi_client_unknown' + || current.geometryAuthority === 'multi_client_unknown' + || saved.geometryAuthority !== current.geometryAuthority + ) { + return { ok: false, reason: 'geometry_authority_unknown' } + } + if (saved.scrollback !== current.scrollback) { + return { ok: false, reason: 'scrollback_changed' } + } + if (saved.xtermVersion !== current.xtermVersion) { + return { ok: false, reason: 'xterm_version_changed' } + } + if (current.requireParserIdle && !saved.parserIdle) { + return { ok: false, reason: 'parser_busy' } + } + if (saved.parserAppliedSeq <= 0) { + return { ok: false, reason: 'no_applied_sequence' } + } + + return { ok: true, sinceSeq: saved.parserAppliedSeq } +} diff --git a/src/lib/ws-client.ts b/src/lib/ws-client.ts index d82cf4eba..427d7c3da 100644 --- a/src/lib/ws-client.ts +++ b/src/lib/ws-client.ts @@ -6,6 +6,7 @@ import { } from '@/lib/perf-logger' import { getAuthToken } from '@/lib/auth' import { sanitizeSessionLocators } from '@/lib/session-utils' +import { WS_PROTOCOL_VERSION } from '@shared/ws-version' import type { ServerMessage, SessionLocator } from '@shared/ws-protocol' import { createLogger } from '@/lib/client-logger' @@ -74,7 +75,6 @@ type InFlightCreate = { } const CONNECTION_TIMEOUT_MS = 10_000 -const WS_PROTOCOL_VERSION = 5 const perfConfig = getClientPerfConfig() function isTerminalInputMessage(msg: unknown): msg is TerminalInputClientMessage { @@ -206,7 +206,10 @@ export class WsClient { } } - if (msg.type === 'terminal.output' && typeof msg.terminalId === 'string') { + if ( + (msg.type === 'terminal.output' || msg.type === 'terminal.output.batch') + && typeof msg.terminalId === 'string' + ) { markTerminalOutputSeen(msg.terminalId) } @@ -334,7 +337,7 @@ export class WsClient { type: 'hello', token, protocolVersion: WS_PROTOCOL_VERSION, - capabilities: { uiScreenshotV1: true }, + capabilities: { uiScreenshotV1: true, terminalOutputBatchV1: true }, ...helloExtensions, }) } diff --git a/src/store/paneTypes.ts b/src/store/paneTypes.ts index 01a1198a5..607272e0e 100644 --- a/src/store/paneTypes.ts +++ b/src/store/paneTypes.ts @@ -66,6 +66,8 @@ export type TerminalPaneContent = { codexDurability?: CodexDurabilityRef /** Runtime-only server locality for same-server matching; never part of canonical durable identity. */ serverInstanceId?: string + /** Runtime output stream identity from terminal.attach.ready; invalidates delta replay after stream replacement. */ + streamId?: string /** Explicit restore failure when no canonical durable target exists. */ restoreError?: RestoreError /** Initial working directory */ diff --git a/src/store/panesSlice.ts b/src/store/panesSlice.ts index e6067e2d6..d74c8532a 100644 --- a/src/store/panesSlice.ts +++ b/src/store/panesSlice.ts @@ -67,6 +67,7 @@ function normalizePaneContent( ...(sessionRef ? { sessionRef } : {}), ...(codexDurability ? { codexDurability } : {}), serverInstanceId: typeof input.serverInstanceId === 'string' ? input.serverInstanceId : undefined, + streamId: typeof input.streamId === 'string' && input.streamId.length > 0 ? input.streamId : undefined, ...(restoreError.success ? { restoreError: restoreError.data } : {}), initialCwd: typeof input.initialCwd === 'string' ? input.initialCwd : undefined, } diff --git a/src/store/storage-keys.ts b/src/store/storage-keys.ts index 8cfc8ff09..4c64e8778 100644 --- a/src/store/storage-keys.ts +++ b/src/store/storage-keys.ts @@ -3,7 +3,7 @@ export const STORAGE_KEYS = { tabs: 'freshell.tabs.v2', panes: 'freshell.panes.v2', sessionActivity: 'freshell.sessionActivity.v2', - terminalCursor: 'freshell.terminal-cursors.v1', + terminalCursor: 'freshell.terminal-cursors.v2', browserPreferences: 'freshell.browser-preferences.v1', tabRecency: 'freshell.tab-recency.v1', turnCompletion: 'freshell.turn-completion.v1', diff --git a/test/e2e-browser/perf/audit-aggregator.ts b/test/e2e-browser/perf/audit-aggregator.ts index 1bd691dfe..658160c53 100644 --- a/test/e2e-browser/perf/audit-aggregator.ts +++ b/test/e2e-browser/perf/audit-aggregator.ts @@ -9,6 +9,7 @@ export type VisibleFirstScenarioSummary = Partial> + terminalReplayMessageCount?: number + terminalReplaySerializedBytes?: number + terminalParserAppliedLagMs?: number + terminalReplayGapCount?: number + terminalFullHydrateFallbackCount?: number + terminalSurfaceQuarantineCount?: number + terminalStaleGenerationRejectionCount?: number + terminalStoppedRetentionCoveredMs?: number + terminalStopResumeGapCount?: number +} & Record>> function summarizeSample(sample: VisibleFirstAuditSample) { const derived = sample.derived as Record - return { + const summary: Record = { status: sample.status, durationMs: sample.durationMs, - focusedReadyMs: derived.focusedReadyMs, - wsReadyMs: derived.wsReadyMs, - terminalInputToFirstOutputMs: derived.terminalInputToFirstOutputMs, - httpRequestsBeforeReady: derived.httpRequestsBeforeReady, - httpBytesBeforeReady: derived.httpBytesBeforeReady, - wsFramesBeforeReady: derived.wsFramesBeforeReady, - wsBytesBeforeReady: derived.wsBytesBeforeReady, - offscreenHttpRequestsBeforeReady: derived.offscreenHttpRequestsBeforeReady, - offscreenHttpBytesBeforeReady: derived.offscreenHttpBytesBeforeReady, - offscreenWsFramesBeforeReady: derived.offscreenWsFramesBeforeReady, - offscreenWsBytesBeforeReady: derived.offscreenWsBytesBeforeReady, } + + for (const [metric, value] of Object.entries(derived)) { + if (typeof value === 'number' && Number.isFinite(value)) { + summary[metric] = value + } + } + + return summary } export function summarizeScenarioSamples( diff --git a/test/e2e-browser/perf/audit-contract.ts b/test/e2e-browser/perf/audit-contract.ts index a13e958ae..8b55238e8 100644 --- a/test/e2e-browser/perf/audit-contract.ts +++ b/test/e2e-browser/perf/audit-contract.ts @@ -37,6 +37,7 @@ const VisibleFirstProfileSchema = z.object({ const VisibleFirstSummaryMetricSchema = z.object({ focusedReadyMs: z.number().nonnegative().optional(), wsReadyMs: z.number().nonnegative().optional(), + maxRafGapMs: z.number().nonnegative().optional(), terminalInputToFirstOutputMs: z.number().nonnegative().optional(), httpRequestsBeforeReady: z.number().nonnegative().optional(), httpBytesBeforeReady: z.number().nonnegative().optional(), @@ -46,6 +47,15 @@ const VisibleFirstSummaryMetricSchema = z.object({ offscreenHttpBytesBeforeReady: z.number().nonnegative().optional(), offscreenWsFramesBeforeReady: z.number().nonnegative().optional(), offscreenWsBytesBeforeReady: z.number().nonnegative().optional(), + terminalReplayMessageCount: z.number().nonnegative().optional(), + terminalReplaySerializedBytes: z.number().nonnegative().optional(), + terminalParserAppliedLagMs: z.number().nonnegative().optional(), + terminalReplayGapCount: z.number().nonnegative().optional(), + terminalFullHydrateFallbackCount: z.number().nonnegative().optional(), + terminalSurfaceQuarantineCount: z.number().nonnegative().optional(), + terminalStaleGenerationRejectionCount: z.number().nonnegative().optional(), + terminalStoppedRetentionCoveredMs: z.number().nonnegative().optional(), + terminalStopResumeGapCount: z.number().nonnegative().optional(), }).strict().catchall(z.unknown()) export const VisibleFirstAuditSampleSchema = z.object({ diff --git a/test/e2e-browser/perf/derive-visible-first-metrics.ts b/test/e2e-browser/perf/derive-visible-first-metrics.ts index 59a9d3f86..9f95abdc9 100644 --- a/test/e2e-browser/perf/derive-visible-first-metrics.ts +++ b/test/e2e-browser/perf/derive-visible-first-metrics.ts @@ -27,11 +27,15 @@ export type DerivedMetricsInput = { http?: { requests: VisibleFirstHttpObservation[] } ws?: { frames: VisibleFirstWsObservation[] } } + server?: { + terminalReplayEvents?: Array> + } } export type VisibleFirstDerivedMetrics = { focusedReadyMs: number wsReadyMs?: number + maxRafGapMs?: number terminalInputToFirstOutputMs?: number httpRequestsBeforeReady: number httpBytesBeforeReady: number @@ -41,6 +45,15 @@ export type VisibleFirstDerivedMetrics = { offscreenHttpBytesBeforeReady: number offscreenWsFramesBeforeReady: number offscreenWsBytesBeforeReady: number + terminalReplayMessageCount: number + terminalReplaySerializedBytes: number + terminalParserAppliedLagMs?: number + terminalReplayGapCount: number + terminalFullHydrateFallbackCount: number + terminalSurfaceQuarantineCount: number + terminalStaleGenerationRejectionCount: number + terminalStoppedRetentionCoveredMs?: number + terminalStopResumeGapCount?: number } const IGNORED_ROUTE_IDS = new Set(['/api/health', '/api/logs/client']) @@ -113,6 +126,237 @@ function resolveWsReadyMs(input: DerivedMetricsInput): number | undefined { return typeof durationMs === 'number' && Number.isFinite(durationMs) ? durationMs : undefined } +function finiteNumber(value: unknown): number | undefined { + return typeof value === 'number' && Number.isFinite(value) ? value : undefined +} + +function nonnegativeMetric(value: unknown): number | undefined { + const numberValue = finiteNumber(value) + return numberValue === undefined ? undefined : Math.max(0, numberValue) +} + +function parsePayload(payload?: string): Record | null { + if (!payload) return null + try { + const parsed = JSON.parse(payload) as unknown + return parsed && typeof parsed === 'object' && !Array.isArray(parsed) + ? parsed as Record + : null + } catch { + return null + } +} + +function eventName(entry: Record): string | null { + return typeof entry.event === 'string' ? entry.event : null +} + +function sumMetric(values: Iterable): number { + let sum = 0 + for (const value of values) { + const numberValue = nonnegativeMetric(value) + if (numberValue !== undefined) { + sum += numberValue + } + } + return sum +} + +function countPerfEvents(input: DerivedMetricsInput, name: string): number { + return (input.browser.perfEvents ?? []).filter((entry) => eventName(entry) === name).length +} + +function finiteNonnegativeValues(values: Iterable): number[] { + return Array.from(values) + .map(nonnegativeMetric) + .filter((value): value is number => value !== undefined) +} + +function resolveMaxRafGapMs(input: DerivedMetricsInput): number | undefined { + const values = finiteNonnegativeValues((input.browser.perfEvents ?? []) + .filter((entry) => eventName(entry) === 'visible_first.audit.max_raf_gap') + .flatMap((entry) => [entry.maxGapMs, entry.durationMs])) + return values.length > 0 ? Math.max(...values) : undefined +} + +function isReplayBatchEvent(entry: Record): boolean { + return entry.event === 'terminal.replay.batch' && entry.source === 'replay' +} + +function isReplayGapEvent(entry: Record): boolean { + return entry.event === 'terminal.replay.gap' || ( + entry.event === 'terminal.output.gap' + && entry.source === 'replay' + ) +} + +function isReceivedTerminalOutputFrame(frame: VisibleFirstWsObservation): boolean { + return (frame as { direction?: unknown }).direction === undefined + || (frame as { direction?: unknown }).direction === 'received' +} + +function isReplayOutputPayload(payload: Record | null): boolean { + if (!payload) return false + if (payload.type === 'terminal.output.batch') { + return payload.source === 'replay' + } + if (payload.type === 'terminal.output') { + return payload.source === 'replay' + } + return false +} + +function replayWsFramesBeforeReady(input: DerivedMetricsInput, focusedReadyMs: number): VisibleFirstWsObservation[] { + return (input.transport.ws?.frames ?? []).filter((frame) => { + if (frame.timestamp > focusedReadyMs || !isReceivedTerminalOutputFrame(frame)) return false + const frameType = frame.type ?? classifyWsFrameType(frame.payload ?? '') + if (frameType !== 'terminal.output' && frameType !== 'terminal.output.batch') return false + return isReplayOutputPayload(parsePayload(frame.payload)) + }) +} + +function replayWsGapFramesBeforeReady(input: DerivedMetricsInput, focusedReadyMs: number): VisibleFirstWsObservation[] { + return (input.transport.ws?.frames ?? []).filter((frame) => { + if (frame.timestamp > focusedReadyMs || !isReceivedTerminalOutputFrame(frame)) return false + const frameType = frame.type ?? classifyWsFrameType(frame.payload ?? '') + return frameType === 'terminal.output.gap' + }) +} + +function isSentTerminalRecoveryInputFrame(frame: VisibleFirstWsObservation): boolean { + if ((frame as { direction?: unknown }).direction !== 'sent') return false + const frameType = frame.type ?? classifyWsFrameType(frame.payload ?? '') + return frameType === 'terminal.input' || frameType === 'terminal.attach' +} + +function resolveTerminalInputToFirstOutputMs( + input: DerivedMetricsInput, + focusedReadyMs: number, +): number | undefined { + const explicitSample = input.browser.terminalLatencySamplesMs?.[0] + if (typeof explicitSample === 'number' && Number.isFinite(explicitSample)) { + return explicitSample + } + + const frames = input.transport.ws?.frames ?? [] + const inputFrame = frames + .filter((frame) => frame.timestamp <= focusedReadyMs && isSentTerminalRecoveryInputFrame(frame)) + .sort((a, b) => a.timestamp - b.timestamp)[0] + if (!inputFrame) return undefined + + const outputFrame = frames + .filter((frame) => { + if (frame.timestamp < inputFrame.timestamp || frame.timestamp > focusedReadyMs) return false + if (!isReceivedTerminalOutputFrame(frame)) return false + const frameType = frame.type ?? classifyWsFrameType(frame.payload ?? '') + return frameType === 'terminal.output' || frameType === 'terminal.output.batch' + }) + .sort((a, b) => a.timestamp - b.timestamp)[0] + if (!outputFrame) return undefined + + return Math.max(0, outputFrame.timestamp - inputFrame.timestamp) +} + +function resolveReplayMessageCount(input: DerivedMetricsInput, replayFrames: VisibleFirstWsObservation[]): number { + const serverReplayBatchEvents = (input.server?.terminalReplayEvents ?? []).filter(isReplayBatchEvent) + return serverReplayBatchEvents.length > 0 ? serverReplayBatchEvents.length : replayFrames.length +} + +function resolveReplaySerializedBytes(input: DerivedMetricsInput, replayFrames: VisibleFirstWsObservation[]): number { + const serverReplayBatchEvents = (input.server?.terminalReplayEvents ?? []).filter(isReplayBatchEvent) + if (serverReplayBatchEvents.length > 0) { + return sumMetric(serverReplayBatchEvents.map((entry) => entry.serializedBytes)) + } + return replayFrames.reduce((sum, frame) => sum + resolveObservationBytes(frame), 0) +} + +function resolveReplayGapCount(input: DerivedMetricsInput, focusedReadyMs: number): number { + const serverReplayGapEvents = (input.server?.terminalReplayEvents ?? []).filter(isReplayGapEvent) + return serverReplayGapEvents.length > 0 + ? serverReplayGapEvents.length + : replayWsGapFramesBeforeReady(input, focusedReadyMs).length +} + +function payloadSeqEnd(frame: VisibleFirstWsObservation): number | undefined { + const payload = parsePayload(frame.payload) + const seqEnd = payload?.seqEnd + return typeof seqEnd === 'number' && Number.isFinite(seqEnd) ? seqEnd : undefined +} + +function resolveParserAppliedLagMs( + input: DerivedMetricsInput, + replayFrames: VisibleFirstWsObservation[], +): number | undefined { + const frameMilestones = replayFrames + .map((frame) => { + const seqEnd = payloadSeqEnd(frame) + return seqEnd === undefined ? null : { seqEnd, timestamp: frame.timestamp } + }) + .filter((entry): entry is { seqEnd: number; timestamp: number } => entry !== null) + + if (frameMilestones.length === 0) return undefined + + let maxLagMs = 0 + let observedParserAppliedEvidence = false + const parserAppliedEvents = (input.browser.perfEvents ?? []) + .filter((entry) => eventName(entry) === 'terminal.parser_applied') + for (const event of parserAppliedEvents) { + const timestamp = finiteNumber(event.timestamp) + const parserAppliedSeq = finiteNumber(event.parserAppliedSeq) + if (timestamp === undefined || parserAppliedSeq === undefined) continue + const coveredFrames = frameMilestones.filter((frame) => frame.seqEnd <= parserAppliedSeq) + if (coveredFrames.length === 0) continue + observedParserAppliedEvidence = true + const lastFrameTimestamp = Math.max(...coveredFrames.map((frame) => frame.timestamp)) + maxLagMs = Math.max(maxLagMs, Math.max(0, timestamp - lastFrameTimestamp)) + } + + return observedParserAppliedEvidence ? maxLagMs : undefined +} + +function resolveStopResumeMetrics(input: DerivedMetricsInput): { + terminalStoppedRetentionCoveredMs?: number + terminalStopResumeGapCount?: number +} { + const events = (input.browser.perfEvents ?? []) + .filter((entry) => eventName(entry) === 'terminal.catchup.stop_resume') + .filter((entry) => { + const stoppedDurationMs = finiteNumber(entry.stoppedDurationMs) + const outputStartedAfterStopMs = finiteNumber(entry.outputStartedAfterStopMs) + const outputStartedBeforeResumeMs = finiteNumber(entry.outputStartedBeforeResumeMs) + const cdpCatchupOutputMessageCount = finiteNumber(entry.cdpCatchupOutputMessageCount) + const catchupOutputMessageCount = finiteNumber(entry.catchupOutputMessageCount) + return entry.source === 'visible_first_audit_process_suspend' + && entry.browserExecutionStopped === true + && stoppedDurationMs !== undefined + && stoppedDurationMs > 0 + && outputStartedAfterStopMs !== undefined + && outputStartedAfterStopMs >= 0 + && outputStartedBeforeResumeMs !== undefined + && outputStartedBeforeResumeMs >= 0 + && ( + (cdpCatchupOutputMessageCount !== undefined && cdpCatchupOutputMessageCount > 0) + || (catchupOutputMessageCount !== undefined && catchupOutputMessageCount > 0) + ) + }) + const retentionCoveredValues = events + .map((entry) => entry.retentionCoveredMs) + .map(nonnegativeMetric) + .filter((value): value is number => value !== undefined) + const gapCountValues = events + .map((entry) => nonnegativeMetric(entry.gapCount)) + .filter((value): value is number => value !== undefined) + + return { + ...(retentionCoveredValues.length > 0 + ? { terminalStoppedRetentionCoveredMs: Math.max(...retentionCoveredValues) } + : {}), + ...(gapCountValues.length > 0 + ? { terminalStopResumeGapCount: gapCountValues.reduce((sum, value) => sum + value, 0) } + : {}), + } +} + function resolveObservationBytes(observation: { encodedDataLength?: number | null; bytes?: number | null; payloadLength?: number | null }): number { if (typeof observation.encodedDataLength === 'number' && Number.isFinite(observation.encodedDataLength)) { return Math.max(0, observation.encodedDataLength) @@ -130,6 +374,11 @@ export function deriveVisibleFirstMetrics(input: DerivedMetricsInput): VisibleFi const focusedReadyMs = input.browser.milestones[input.focusedReadyMilestone] const allowedApiRoutes = new Set(input.allowedApiRouteIdsBeforeReady) const allowedWsTypes = new Set(input.allowedWsTypesBeforeReady) + const replayFrames = replayWsFramesBeforeReady(input, focusedReadyMs) + const stopResumeMetrics = resolveStopResumeMetrics(input) + const terminalInputToFirstOutputMs = resolveTerminalInputToFirstOutputMs(input, focusedReadyMs) + const maxRafGapMs = resolveMaxRafGapMs(input) + const terminalParserAppliedLagMs = resolveParserAppliedLagMs(input, replayFrames) let httpRequestsBeforeReady = 0 let httpBytesBeforeReady = 0 @@ -172,8 +421,9 @@ export function deriveVisibleFirstMetrics(input: DerivedMetricsInput): VisibleFi return { focusedReadyMs, ...(resolveWsReadyMs(input) !== undefined ? { wsReadyMs: resolveWsReadyMs(input) } : {}), - ...(typeof input.browser.terminalLatencySamplesMs?.[0] === 'number' - ? { terminalInputToFirstOutputMs: input.browser.terminalLatencySamplesMs[0] } + ...(maxRafGapMs !== undefined ? { maxRafGapMs } : {}), + ...(terminalInputToFirstOutputMs !== undefined + ? { terminalInputToFirstOutputMs } : {}), httpRequestsBeforeReady, httpBytesBeforeReady, @@ -183,5 +433,13 @@ export function deriveVisibleFirstMetrics(input: DerivedMetricsInput): VisibleFi offscreenHttpBytesBeforeReady, offscreenWsFramesBeforeReady, offscreenWsBytesBeforeReady, + terminalReplayMessageCount: resolveReplayMessageCount(input, replayFrames), + terminalReplaySerializedBytes: resolveReplaySerializedBytes(input, replayFrames), + ...(terminalParserAppliedLagMs !== undefined ? { terminalParserAppliedLagMs } : {}), + terminalReplayGapCount: resolveReplayGapCount(input, focusedReadyMs), + terminalFullHydrateFallbackCount: countPerfEvents(input, 'terminal.catchup.full_hydrate_fallback'), + terminalSurfaceQuarantineCount: countPerfEvents(input, 'terminal.catchup.surface_quarantined'), + terminalStaleGenerationRejectionCount: countPerfEvents(input, 'terminal.attach_generation_stale_rejected'), + ...stopResumeMetrics, } } diff --git a/test/e2e-browser/perf/parse-server-logs.ts b/test/e2e-browser/perf/parse-server-logs.ts index 895edf088..ec2d7db48 100644 --- a/test/e2e-browser/perf/parse-server-logs.ts +++ b/test/e2e-browser/perf/parse-server-logs.ts @@ -1,15 +1,24 @@ import fs from 'fs/promises' +const TERMINAL_REPLAY_AUDIT_EVENTS = new Set([ + 'terminal.replay.batch', + 'terminal.replay.gap', +]) + export async function parseVisibleFirstServerLogs(debugLogPath: string): Promise<{ httpRequests: unknown[] perfEvents: unknown[] perfSystemSamples: unknown[] + terminalReplayEvents: Array> + terminalOutputEvents: Array> parserDiagnostics: string[] }> { const content = await fs.readFile(debugLogPath, 'utf8') const httpRequests: unknown[] = [] const perfEvents: unknown[] = [] const perfSystemSamples: unknown[] = [] + const terminalReplayEvents: Array> = [] + const terminalOutputEvents: Array> = [] const parserDiagnostics: string[] = [] for (const [index, line] of content.split(/\r?\n/).entries()) { @@ -28,6 +37,16 @@ export async function parseVisibleFirstServerLogs(debugLogPath: string): Promise if (parsed.component === 'perf' || (typeof parsed.event === 'string' && parsed.event.startsWith('perf'))) { perfEvents.push(parsed) } + if (typeof parsed.event === 'string' && TERMINAL_REPLAY_AUDIT_EVENTS.has(parsed.event)) { + terminalReplayEvents.push(parsed as Record) + } + if ( + parsed.event === 'terminal.output' + || parsed.event === 'terminal.output.batch' + || parsed.event === 'terminal.output.gap' + ) { + terminalOutputEvents.push(parsed as Record) + } } catch (error) { parserDiagnostics.push(`line ${index + 1}: ${(error as Error).message}`) } @@ -37,6 +56,8 @@ export async function parseVisibleFirstServerLogs(debugLogPath: string): Promise httpRequests, perfEvents, perfSystemSamples, + terminalReplayEvents, + terminalOutputEvents, parserDiagnostics, } } diff --git a/test/e2e-browser/perf/run-sample.ts b/test/e2e-browser/perf/run-sample.ts index 5e0cc98e6..23a3a6ae2 100644 --- a/test/e2e-browser/perf/run-sample.ts +++ b/test/e2e-browser/perf/run-sample.ts @@ -1,4 +1,7 @@ +import { execFile } from 'child_process' import fs from 'fs/promises' +import path from 'path' +import { promisify } from 'util' import { chromium, type Browser, type BrowserContext, type Page } from '@playwright/test' import WebSocket from 'ws' import { TestHarness } from '../helpers/test-harness.js' @@ -14,14 +17,14 @@ import { applyProfileNetworkConditions, buildAuditContextOptions, } from './create-audit-context.js' -import { deriveVisibleFirstMetrics } from './derive-visible-first-metrics.js' +import { classifyWsFrameType, deriveVisibleFirstMetrics } from './derive-visible-first-metrics.js' import { createNetworkRecorder, summarizeNetworkCapture, type NetworkCapture, } from './network-recorder.js' import { parseVisibleFirstServerLogs } from './parse-server-logs.js' -import { AUDIT_SCENARIOS } from './scenarios.js' +import { AUDIT_SCENARIOS, type AuditScenarioDefinition } from './scenarios.js' import { buildAgentChatBrowserStorageSeed, buildOffscreenTabBrowserStorageSeed, @@ -58,8 +61,21 @@ type ReconnectBootstrapResult = { browserStorageSeed: Record } +type ReceivedWsFrame = { + observedAtMs: number + type: string + payloadLength: number +} + +type ReconnectStopResumeEvidence = Record + +const execFileAsync = promisify(execFile) const SAMPLE_TIMEOUT_MS = 30_000 const TERMINAL_RECONNECT_CREATE_REQUEST_ID = 'visible-first-reconnect-create' +const TERMINAL_RECONNECT_REPLAY_MESSAGE_BUDGET = 30 +const TERMINAL_RECONNECT_STOPPED_PROBE_MS = 1_200 +const TERMINAL_RECONNECT_STOPPED_OUTPUT_DELAY_MS = 300 +const TERMINAL_RECONNECT_STOPPED_OUTPUT_LINE_COUNT = 60 function getScenarioDefinition(scenarioId: VisibleFirstScenarioId) { const scenario = AUDIT_SCENARIOS.find((entry) => entry.id === scenarioId) @@ -86,11 +102,317 @@ function emptyCollectors(): SampleCollectors { httpRequests: [], perfEvents: [], perfSystemSamples: [], + terminalReplayEvents: [], + terminalOutputEvents: [], parserDiagnostics: [], }, } } +async function installRafGapSampler(page: Page): Promise { + await page.addInitScript(() => { + const state = { + maxGapMs: 0, + lastRafAt: 0, + sampleCount: 0, + } + ;(window as Window & { + __FRESHELL_VISIBLE_FIRST_RAF_GAP__?: typeof state + }).__FRESHELL_VISIBLE_FIRST_RAF_GAP__ = state + + const tick = (now: number) => { + if (state.lastRafAt > 0) { + state.maxGapMs = Math.max(state.maxGapMs, now - state.lastRafAt) + } + state.lastRafAt = now + state.sampleCount += 1 + window.requestAnimationFrame(tick) + } + window.requestAnimationFrame(tick) + }) +} + +async function readRafGapSummary(page: Page): Promise | null> { + return page.evaluate(() => { + const state = (window as Window & { + __FRESHELL_VISIBLE_FIRST_RAF_GAP__?: { + maxGapMs: number + lastRafAt: number + sampleCount: number + } + }).__FRESHELL_VISIBLE_FIRST_RAF_GAP__ + if (!state) return null + return { + event: 'visible_first.audit.max_raf_gap', + timestamp: performance.now(), + maxGapMs: state.maxGapMs, + sampleCount: state.sampleCount, + } + }) +} + +async function waitForRafSampler(page: Page): Promise { + await page.evaluate(() => new Promise((resolve) => { + const auditWindow = window as Window & { + __FRESHELL_VISIBLE_FIRST_RAF_GAP__?: { + maxGapMs: number + lastRafAt: number + sampleCount: number + } + } + window.requestAnimationFrame((firstNow) => { + const firstState = auditWindow.__FRESHELL_VISIBLE_FIRST_RAF_GAP__ + if (firstState) { + if (firstState.lastRafAt > 0) { + firstState.maxGapMs = Math.max(firstState.maxGapMs, firstNow - firstState.lastRafAt) + } + firstState.lastRafAt = firstNow + firstState.sampleCount += 1 + } + window.requestAnimationFrame((secondNow) => { + const secondState = auditWindow.__FRESHELL_VISIBLE_FIRST_RAF_GAP__ + if (secondState) { + if (secondState.lastRafAt > 0) { + secondState.maxGapMs = Math.max(secondState.maxGapMs, secondNow - secondState.lastRafAt) + } + secondState.lastRafAt = secondNow + secondState.sampleCount += 1 + } + resolve() + }) + }) + })) +} + +function assertRequiredMetricsPresent( + scenario: AuditScenarioDefinition, + derived: Record, +): void { + const missing = (scenario.requiredMetricIds ?? []).filter((metricId) => { + const value = derived[metricId] + return typeof value !== 'number' || !Number.isFinite(value) + }) + if (missing.length > 0) { + throw new Error(`Missing required audit metrics for ${scenario.id}: ${missing.join(', ')}`) + } +} + +function assertTerminalReconnectTargets(derived: Record): void { + const replayMessageCount = typeof derived.terminalReplayMessageCount === 'number' + ? derived.terminalReplayMessageCount + : Number.NaN + if (!Number.isFinite(replayMessageCount) || replayMessageCount <= 0) { + throw new Error('terminal-reconnect-backlog did not record replay message evidence') + } + if (replayMessageCount > TERMINAL_RECONNECT_REPLAY_MESSAGE_BUDGET) { + throw new Error( + `terminal-reconnect-backlog replay message count ${replayMessageCount} exceeded budget ${TERMINAL_RECONNECT_REPLAY_MESSAGE_BUDGET}`, + ) + } + + const zeroMetrics = [ + 'terminalReplayGapCount', + 'terminalFullHydrateFallbackCount', + 'terminalSurfaceQuarantineCount', + 'terminalStaleGenerationRejectionCount', + 'terminalStopResumeGapCount', + ] + for (const metric of zeroMetrics) { + const value = derived[metric] + if (typeof value !== 'number' || value !== 0) { + throw new Error(`terminal-reconnect-backlog expected ${metric}=0, got ${String(value)}`) + } + } +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => { + setTimeout(resolve, ms) + }) +} + +async function waitForFile(filePath: string, timeoutMs = SAMPLE_TIMEOUT_MS): Promise { + const deadline = Date.now() + timeoutMs + while (Date.now() < deadline) { + try { + await fs.stat(filePath) + return + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { + throw error + } + } + await sleep(25) + } + throw new Error(`Timed out waiting for audit proof file ${filePath}`) +} + +function shellQuote(value: string): string { + return `'${value.replace(/'/g, `'\\''`)}'` +} + +async function collectProcessForestPids(rootPids: number[]): Promise { + if (process.platform === 'win32') { + throw new Error('POSIX SIGSTOP/SIGCONT process suspend proof is not available on native Windows') + } + + const { stdout } = await execFileAsync('ps', ['-eo', 'pid=,ppid=']) + const childrenByParent = new Map() + for (const line of stdout.split(/\r?\n/)) { + const [pidRaw, ppidRaw] = line.trim().split(/\s+/) + const pid = Number(pidRaw) + const ppid = Number(ppidRaw) + if (!Number.isFinite(pid) || !Number.isFinite(ppid)) continue + const children = childrenByParent.get(ppid) ?? [] + children.push(pid) + childrenByParent.set(ppid, children) + } + + const pids = new Set() + const queue = [...rootPids] + while (queue.length > 0) { + const pid = queue.shift() + if (!pid || pids.has(pid)) continue + pids.add(pid) + queue.push(...(childrenByParent.get(pid) ?? [])) + } + return [...pids] +} + +async function collectBrowserExecutionPids(browser: Browser): Promise { + const browserWithProcess = browser as unknown as { process?: () => { pid?: number } | null } + const childProcess = browserWithProcess.process?.() + if (!childProcess?.pid) { + const browserWithCdp = browser as unknown as { + newBrowserCDPSession?: () => Promise<{ + send: (method: string) => Promise + detach: () => Promise + }> + } + if (typeof browserWithCdp.newBrowserCDPSession !== 'function') { + throw new Error('Playwright did not expose a browser process or CDP session for the reconnect suspend proof') + } + const cdpSession = await browserWithCdp.newBrowserCDPSession() + try { + const processInfo = await cdpSession.send('SystemInfo.getProcessInfo') as { + processInfo?: Array<{ id?: number }> + } + const pids = (processInfo.processInfo ?? []) + .map((entry) => Number(entry.id)) + .filter((pid) => Number.isInteger(pid) && pid > 0) + if (pids.length > 0) { + return collectProcessForestPids(pids) + } + } finally { + await cdpSession.detach().catch(() => {}) + } + throw new Error('Could not resolve Chromium process IDs for the reconnect suspend proof') + } + return collectProcessForestPids([childProcess.pid]) +} + +function signalPids(pids: number[], signal: NodeJS.Signals): void { + for (const pid of pids) { + try { + process.kill(pid, signal) + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ESRCH') { + throw error + } + } + } +} + +function isTerminalOutputFrame(frame: ReceivedWsFrame): boolean { + return frame.type === 'terminal.output' || frame.type === 'terminal.output.batch' +} + +async function runReconnectStopResumeProof(input: { + browser: Browser + page: Page + serverInfo: TestServerInfo + harness: TestHarness + terminal: TerminalHelper + receivedWsFrames: ReceivedWsFrame[] +}): Promise { + const terminalId = getActiveTerminalId(await input.harness.getState()) + if (!terminalId) { + throw new Error('terminal-reconnect-backlog stop/resume proof did not find an active terminal') + } + + const stoppedPids = await collectBrowserExecutionPids(input.browser) + const markerId = `${Date.now()}-${Math.random().toString(36).slice(2)}` + const outputScriptPath = path.join(input.serverInfo.homeDir, `visible-first-stop-resume-${markerId}.cjs`) + const scheduledMarkerPath = path.join(input.serverInfo.homeDir, `visible-first-stop-resume-scheduled-${markerId}.json`) + const startedMarkerPath = path.join(input.serverInfo.homeDir, `visible-first-stop-resume-started-${markerId}.json`) + const finalLine = `visible-first-stop-resume-${TERMINAL_RECONNECT_STOPPED_OUTPUT_LINE_COUNT}` + const outputScript = [ + 'const { writeFileSync } = require("fs");', + `const scheduledMarkerPath = ${JSON.stringify(scheduledMarkerPath)};`, + `const startedMarkerPath = ${JSON.stringify(startedMarkerPath)};`, + `const prefix = ${JSON.stringify('visible-first-stop-resume-')};`, + 'writeFileSync(scheduledMarkerPath, JSON.stringify({ scheduledAt: Date.now() }));', + `setTimeout(() => {`, + ' writeFileSync(startedMarkerPath, JSON.stringify({ startedAt: Date.now() }));', + ` for (let i = 1; i <= ${TERMINAL_RECONNECT_STOPPED_OUTPUT_LINE_COUNT}; i += 1) console.log(prefix + i);`, + `}, ${TERMINAL_RECONNECT_STOPPED_OUTPUT_DELAY_MS});`, + `setTimeout(() => process.exit(0), ${TERMINAL_RECONNECT_STOPPED_OUTPUT_DELAY_MS + 500});`, + ].join('\n') + + await fs.writeFile(outputScriptPath, `${outputScript}\n`, 'utf8') + await input.terminal.executeCommand(`node ${shellQuote(outputScriptPath)}`) + await waitForFile(scheduledMarkerPath) + + const wsFrameBaseline = input.receivedWsFrames.length + const stopStartedAt = Date.now() + try { + signalPids([...stoppedPids].sort((a, b) => b - a), 'SIGSTOP') + await sleep(TERMINAL_RECONNECT_STOPPED_PROBE_MS) + } finally { + signalPids([...stoppedPids].sort((a, b) => a - b), 'SIGCONT') + } + const stopEndedAt = Date.now() + + await waitForFile(startedMarkerPath) + const outputStartedAt = JSON.parse(await fs.readFile(startedMarkerPath, 'utf8')).startedAt as number + if (outputStartedAt < stopStartedAt || outputStartedAt > stopEndedAt) { + throw new Error( + `terminal-reconnect-backlog stop/resume proof output started outside stopped window: ${outputStartedAt}`, + ) + } + + await input.terminal.waitForOutput(finalLine, { terminalId, timeout: SAMPLE_TIMEOUT_MS }) + await input.page.waitForTimeout(150) + + const catchupFrames = input.receivedWsFrames.slice(wsFrameBaseline) + const outputMessageCount = catchupFrames.filter(isTerminalOutputFrame).length + if (outputMessageCount <= 0) { + throw new Error('terminal-reconnect-backlog stop/resume proof did not observe post-resume terminal output frames') + } + + const gapCount = catchupFrames.filter((frame) => frame.type === 'terminal.output.gap').length + const timestamp = await input.page.evaluate(() => performance.now()) + const stoppedDurationMs = Math.max(0, stopEndedAt - stopStartedAt) + const retentionCoveredMs = Math.max(0, stopEndedAt - outputStartedAt) + + return { + event: 'terminal.catchup.stop_resume', + source: 'visible_first_audit_process_suspend', + scenarioId: 'terminal-reconnect-backlog', + timestamp, + terminalId, + browserExecutionStopped: true, + stoppedDurationMs, + stoppedOutputDelayMs: TERMINAL_RECONNECT_STOPPED_OUTPUT_DELAY_MS, + outputStartedAfterStopMs: outputStartedAt - stopStartedAt, + outputStartedBeforeResumeMs: stopEndedAt - outputStartedAt, + retentionCoveredMs, + gapCount, + cdpCatchupOutputMessageCount: outputMessageCount, + cdpCatchupMessageCount: catchupFrames.length, + } +} + function normalizeTransportCapture( capture: NetworkCapture, browserTimeOriginMs: number, @@ -429,15 +751,25 @@ async function executeSampleDefault( profileId: input.profileId, })) const page = await context.newPage() + await installRafGapSampler(page) const cdpSession = await context.newCDPSession(page) await applyProfileNetworkConditions(cdpSession, input.profileId) const recorder = createNetworkRecorder() + const receivedWsFrames: ReceivedWsFrame[] = [] cdpSession.on('Network.requestWillBeSent', (event) => recorder.onRequestWillBeSent(event)) cdpSession.on('Network.responseReceived', (event) => recorder.onResponseReceived(event)) cdpSession.on('Network.loadingFinished', (event) => recorder.onLoadingFinished(event)) cdpSession.on('Network.webSocketFrameSent', (event) => recorder.onWebSocketFrameSent(event)) - cdpSession.on('Network.webSocketFrameReceived', (event) => recorder.onWebSocketFrameReceived(event)) + cdpSession.on('Network.webSocketFrameReceived', (event) => { + recorder.onWebSocketFrameReceived(event) + const payload = event.response?.payloadData ?? '' + receivedWsFrames.push({ + observedAtMs: Date.now(), + type: classifyWsFrameType(payload), + payloadLength: Buffer.byteLength(payload), + }) + }) const browserStorageSeed = await resolveBrowserStorageSeed({ scenarioId: input.scenarioId, @@ -473,10 +805,26 @@ async function executeSampleDefault( await waitForAuditMilestone(page, harness, scenario.focusedReadyMilestone) + await waitForRafSampler(page) const browserSnapshot = await harness.getPerfAuditSnapshot() if (!browserSnapshot) { throw new Error('Perf audit snapshot was not available from the test harness') } + const rafGapSummary = await readRafGapSummary(page) + if (rafGapSummary) { + browserSnapshot.perfEvents.push(rafGapSummary) + } + + if (input.scenarioId === 'terminal-reconnect-backlog') { + browserSnapshot.perfEvents.push(await runReconnectStopResumeProof({ + browser, + page, + serverInfo, + harness, + terminal, + receivedWsFrames, + })) + } const browserTimeOriginMs = await page.evaluate(() => performance.timeOrigin) const rawCapture = recorder.snapshot() @@ -551,7 +899,14 @@ export async function runVisibleFirstAuditSample( allowedWsTypesBeforeReady: scenario.allowedWsTypesBeforeReady, browser: collectors.browser, transport: collectors.transport, + server: { + terminalReplayEvents: collectors.server.terminalReplayEvents, + }, }) + assertRequiredMetricsPresent(scenario, derived) + if (scenario.id === 'terminal-reconnect-backlog') { + assertTerminalReconnectTargets(derived) + } } catch (error) { const message = error instanceof Error ? error.message : String(error) errors.push(message) diff --git a/test/e2e-browser/perf/scenarios.ts b/test/e2e-browser/perf/scenarios.ts index 157cf49ee..709988476 100644 --- a/test/e2e-browser/perf/scenarios.ts +++ b/test/e2e-browser/perf/scenarios.ts @@ -8,6 +8,20 @@ export type AuditScenarioId = | 'terminal-reconnect-backlog' | 'offscreen-tab-selection' +export type AuditRequiredMetricId = + | 'focusedReadyMs' + | 'maxRafGapMs' + | 'terminalInputToFirstOutputMs' + | 'terminalReplayMessageCount' + | 'terminalReplaySerializedBytes' + | 'terminalParserAppliedLagMs' + | 'terminalReplayGapCount' + | 'terminalFullHydrateFallbackCount' + | 'terminalSurfaceQuarantineCount' + | 'terminalStaleGenerationRejectionCount' + | 'terminalStoppedRetentionCoveredMs' + | 'terminalStopResumeGapCount' + export type AuditScenarioContext = { token?: string profileId: AuditProfileId @@ -19,6 +33,7 @@ export type AuditScenarioDefinition = { focusedReadyMilestone: string allowedApiRouteIdsBeforeReady: readonly string[] allowedWsTypesBeforeReady: readonly string[] + requiredMetricIds?: readonly AuditRequiredMetricId[] buildUrl: (context: AuditScenarioContext) => string seedServerHome?: () => Promise seedBrowserStorage?: () => Record @@ -33,6 +48,25 @@ function buildRootUrl(token?: string): string { return `/?${params.toString()}` } +export const TERMINAL_CATCHUP_REQUIRED_METRIC_IDS = [ + 'terminalReplayMessageCount', + 'terminalReplaySerializedBytes', + 'terminalParserAppliedLagMs', + 'terminalReplayGapCount', + 'terminalFullHydrateFallbackCount', + 'terminalSurfaceQuarantineCount', + 'terminalStaleGenerationRejectionCount', + 'terminalStoppedRetentionCoveredMs', + 'terminalStopResumeGapCount', +] as const satisfies readonly AuditRequiredMetricId[] + +const TERMINAL_RECONNECT_BACKLOG_REQUIRED_METRIC_IDS = [ + 'focusedReadyMs', + 'terminalInputToFirstOutputMs', + 'maxRafGapMs', + ...TERMINAL_CATCHUP_REQUIRED_METRIC_IDS, +] as const satisfies readonly AuditRequiredMetricId[] + export const AUDIT_SCENARIOS: readonly AuditScenarioDefinition[] = [ { id: 'auth-required-cold-boot', @@ -86,10 +120,12 @@ export const AUDIT_SCENARIOS: readonly AuditScenarioDefinition[] = [ 'terminal.attach', 'terminal.attach.ready', 'terminal.output', + 'terminal.output.batch', 'terminal.output.gap', 'terminals.changed', ], + requiredMetricIds: TERMINAL_RECONNECT_BACKLOG_REQUIRED_METRIC_IDS, buildUrl: ({ token }) => buildRootUrl(token), }, { diff --git a/test/e2e-browser/perf/seed-server-home.ts b/test/e2e-browser/perf/seed-server-home.ts index 7c50ce870..61d69683a 100644 --- a/test/e2e-browser/perf/seed-server-home.ts +++ b/test/e2e-browser/perf/seed-server-home.ts @@ -145,6 +145,7 @@ async function writeSessionFile(filePath: string, content: string, mtimeMs: numb export async function seedVisibleFirstAuditServerHome(homeDir: string): Promise { const claudeProjectsDir = path.join(homeDir, '.claude', 'projects') await fs.mkdir(claudeProjectsDir, { recursive: true }) + await fs.writeFile(path.join(homeDir, '.hushlogin'), '', 'utf8') const projectPaths: string[] = [] let sessionSequence = 1 diff --git a/test/e2e-browser/perf/visible-first-audit-gate.ts b/test/e2e-browser/perf/visible-first-audit-gate.ts index cc36f4250..bc5cac9c6 100644 --- a/test/e2e-browser/perf/visible-first-audit-gate.ts +++ b/test/e2e-browser/perf/visible-first-audit-gate.ts @@ -6,9 +6,11 @@ import { type VisibleFirstProfileId, type VisibleFirstScenarioId, } from './audit-contract.js' +import { AUDIT_SCENARIOS, type AuditRequiredMetricId } from './scenarios.js' export type VisibleFirstAuditGateResult = { ok: boolean + validationErrors?: string[] violations: Array<{ scenarioId: string profileId: string @@ -39,6 +41,10 @@ const OFFSCREEN_METRICS: GateMetric[] = [ 'offscreenWsBytesBeforeReady', ] +const SCENARIO_REQUIRED_METRICS = new Map( + AUDIT_SCENARIOS.map((scenario) => [scenario.id, scenario.requiredMetricIds ?? []] as const), +) + function assertFullAuditMatrix(artifact: VisibleFirstAuditArtifact, label: string): void { const profileIds = new Set(artifact.profiles.map((profile) => profile.id)) for (const profileId of AUDIT_PROFILE_IDS) { @@ -86,6 +92,58 @@ function getMetricValue( return typeof value === 'number' ? value : 0 } +function isFiniteMetric(value: unknown): value is number { + return typeof value === 'number' && Number.isFinite(value) +} + +function assertMetricPresent(input: { + artifact: VisibleFirstAuditArtifact + label: string + scenarioId: VisibleFirstScenarioId + profileId: VisibleFirstProfileId + metricId: AuditRequiredMetricId +}): void { + const scenario = getScenario(input.artifact, input.scenarioId) + const sample = scenario.samples.find((entry) => entry.profileId === input.profileId) + if (!sample) { + throw new Error(`${input.label} missing scenario/profile pair ${input.scenarioId}/${input.profileId}`) + } + if (!isFiniteMetric((sample.derived as Record)[input.metricId])) { + throw new Error( + `${input.label} missing required metric ${input.scenarioId}/${input.profileId}:${input.metricId}`, + ) + } + + const summary = scenario.summaryByProfile[input.profileId] as Record | undefined + if (!summary || !isFiniteMetric(summary[input.metricId])) { + throw new Error( + `${input.label} missing required summary metric ${input.scenarioId}/${input.profileId}:${input.metricId}`, + ) + } +} + +function assertRequiredMetricIdsPresent( + artifact: VisibleFirstAuditArtifact, + label: string, +): void { + for (const scenarioId of AUDIT_SCENARIO_IDS) { + const requiredMetricIds = SCENARIO_REQUIRED_METRICS.get(scenarioId) ?? [] + if (requiredMetricIds.length === 0) continue + const scenario = getScenario(artifact, scenarioId) + for (const sample of scenario.samples) { + for (const metricId of requiredMetricIds) { + assertMetricPresent({ + artifact, + label, + scenarioId, + profileId: sample.profileId, + metricId, + }) + } + } + } +} + function createViolation( base: VisibleFirstAuditArtifact, candidate: VisibleFirstAuditArtifact, @@ -119,6 +177,7 @@ export function evaluateVisibleFirstAuditGate( assertVisibleFirstAuditTrusted(candidate) assertFullAuditMatrix(base, 'base') assertFullAuditMatrix(candidate, 'candidate') + assertRequiredMetricIdsPresent(candidate, 'candidate') const violations: VisibleFirstAuditGateResult['violations'] = [] diff --git a/test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts b/test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts new file mode 100644 index 000000000..0fb2a7408 --- /dev/null +++ b/test/e2e-browser/specs/terminal-background-freeze-catchup.spec.ts @@ -0,0 +1,522 @@ +import { execFile } from 'child_process' +import fs from 'fs/promises' +import path from 'path' +import { promisify } from 'util' +import type { CDPSession, Page } from '@playwright/test' +import { test, expect } from '../helpers/fixtures.js' +import type { TestHarness } from '../helpers/test-harness.js' +import type { TerminalHelper } from '../helpers/terminal-helpers.js' + +const execFileAsync = promisify(execFile) +const ACTIVE_PROBE_MS = 300 +const STOPPED_PROBE_MS = 2_500 +const RESUMED_PROBE_MS = 500 +const STOPPED_OUTPUT_DELAY_MS = 1_000 +const STOPPED_OUTPUT_LINE_COUNT = 240 + +type StopProbeSnapshot = { + timerTicks: number + rafTicks: number + wsMessageCount: number + wsOpenCount: number + wsCloseCount: number + wsErrorCount: number + lastWsReadyState: number | null + maxTimerGapMs: number + maxRafGapMs: number + messageTypes: Record +} + +type ReceivedWsFrame = { + observedAtMs: number + type: string + payloadLength: number +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => { + setTimeout(resolve, ms) + }) +} + +async function waitForFile(filePath: string, timeoutMs = 15_000): Promise { + const deadline = Date.now() + timeoutMs + while (Date.now() < deadline) { + try { + await fs.stat(filePath) + return + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') { + throw error + } + } + await sleep(25) + } + throw new Error(`Timed out waiting for file ${filePath}`) +} + +function classifyWsPayload(payload: string): string { + try { + const parsed = JSON.parse(payload) as { type?: unknown } + return typeof parsed.type === 'string' ? parsed.type : 'unknown' + } catch { + return 'unknown' + } +} + +function shellQuote(value: string): string { + return `'${value.replace(/'/g, `'\\''`)}'` +} + +async function selectShellFromPicker(page: Page): Promise { + if (await page.locator('.xterm').first().isVisible().catch(() => false)) return + await page.waitForTimeout(500) + if (await page.locator('.xterm').first().isVisible().catch(() => false)) return + + for (const name of ['Shell', 'WSL', 'CMD', 'PowerShell', 'Bash']) { + const button = page.getByRole('button', { name: new RegExp(`^${name}$`, 'i') }) + if (await button.first().isVisible().catch(() => false)) { + await button.first().click() + await page.locator('.xterm').first().waitFor({ state: 'visible', timeout: 30_000 }) + return + } + } + + throw new Error('Could not create a terminal from the pane picker') +} + +async function getActiveTerminalId(harness: TestHarness): Promise { + const state = await harness.getState() + const activeTabId = state?.tabs?.activeTabId + const layout = activeTabId ? state?.panes?.layouts?.[activeTabId] : null + const queue = layout ? [layout] : [] + while (queue.length > 0) { + const node = queue.shift() + if (node?.type === 'leaf' && node.content?.kind === 'terminal' && typeof node.content.terminalId === 'string') { + return node.content.terminalId + } + if (Array.isArray(node?.children)) { + queue.push(...node.children) + } + } + throw new Error('No active terminal id found') +} + +async function waitForParserAppliedCheckpoint(page: Page): Promise { + await page.waitForFunction( + () => { + const events = window.__FRESHELL_TEST_HARNESS__?.getPerfAuditSnapshot()?.perfEvents ?? [] + return events.some((event) => event.event === 'terminal.parser_applied' + && typeof event.parserAppliedSeq === 'number' + && event.parserAppliedSeq > 0) + }, + { timeout: 15_000 }, + ) + return page.evaluate(() => { + const events = window.__FRESHELL_TEST_HARNESS__?.getPerfAuditSnapshot()?.perfEvents ?? [] + return events + .filter((event) => event.event === 'terminal.parser_applied' && typeof event.parserAppliedSeq === 'number') + .reduce((max, event) => Math.max(max, Number(event.parserAppliedSeq)), 0) + }) +} + +async function installStopProbe(page: Page): Promise { + await page.addInitScript(() => { + const state = { + timerTicks: 0, + rafTicks: 0, + wsMessageCount: 0, + wsOpenCount: 0, + wsCloseCount: 0, + wsErrorCount: 0, + lastWsReadyState: null as number | null, + lastTimerAt: performance.now(), + lastRafAt: 0, + maxTimerGapMs: 0, + maxRafGapMs: 0, + messageTypes: {} as Record, + } + + window.setInterval(() => { + const now = performance.now() + state.maxTimerGapMs = Math.max(state.maxTimerGapMs, now - state.lastTimerAt) + state.lastTimerAt = now + state.timerTicks += 1 + }, 50) + + const rafTick = (now: number) => { + if (state.lastRafAt > 0) { + state.maxRafGapMs = Math.max(state.maxRafGapMs, now - state.lastRafAt) + } + state.lastRafAt = now + state.rafTicks += 1 + window.requestAnimationFrame(rafTick) + } + window.requestAnimationFrame(rafTick) + + const NativeWebSocket = window.WebSocket + window.WebSocket = class InstrumentedWebSocket extends NativeWebSocket { + constructor(url: string | URL, protocols?: string | string[]) { + super(url, protocols) + state.lastWsReadyState = this.readyState + this.addEventListener('open', () => { + state.wsOpenCount += 1 + state.lastWsReadyState = this.readyState + }) + this.addEventListener('close', () => { + state.wsCloseCount += 1 + state.lastWsReadyState = this.readyState + }) + this.addEventListener('error', () => { + state.wsErrorCount += 1 + state.lastWsReadyState = this.readyState + }) + this.addEventListener('message', (event) => { + state.wsMessageCount += 1 + state.lastWsReadyState = this.readyState + if (typeof event.data === 'string') { + try { + const parsed = JSON.parse(event.data) as { type?: unknown } + if (typeof parsed.type === 'string') { + state.messageTypes[parsed.type] = (state.messageTypes[parsed.type] ?? 0) + 1 + } + } catch { + state.messageTypes.unknown = (state.messageTypes.unknown ?? 0) + 1 + } + } + }) + } + } + + ;(window as Window & { + __FRESHELL_STOP_RESUME_PROBE__?: { snapshot: () => StopProbeSnapshot } + }).__FRESHELL_STOP_RESUME_PROBE__ = { + snapshot: () => ({ + timerTicks: state.timerTicks, + rafTicks: state.rafTicks, + wsMessageCount: state.wsMessageCount, + wsOpenCount: state.wsOpenCount, + wsCloseCount: state.wsCloseCount, + wsErrorCount: state.wsErrorCount, + lastWsReadyState: state.lastWsReadyState, + maxTimerGapMs: state.maxTimerGapMs, + maxRafGapMs: state.maxRafGapMs, + messageTypes: { ...state.messageTypes }, + }), + } + }) +} + +async function stopProbeSnapshot(page: Page): Promise { + return page.evaluate(() => { + const probe = (window as Window & { + __FRESHELL_STOP_RESUME_PROBE__?: { snapshot: () => StopProbeSnapshot } + }).__FRESHELL_STOP_RESUME_PROBE__ + if (!probe) { + throw new Error('Stop/resume probe was not installed') + } + return probe.snapshot() + }) +} + +function deltaSnapshots(before: StopProbeSnapshot, after: StopProbeSnapshot): StopProbeSnapshot { + const messageTypes: Record = {} + for (const [type, count] of Object.entries(after.messageTypes)) { + messageTypes[type] = count - (before.messageTypes[type] ?? 0) + } + return { + timerTicks: after.timerTicks - before.timerTicks, + rafTicks: after.rafTicks - before.rafTicks, + wsMessageCount: after.wsMessageCount - before.wsMessageCount, + wsOpenCount: after.wsOpenCount - before.wsOpenCount, + wsCloseCount: after.wsCloseCount - before.wsCloseCount, + wsErrorCount: after.wsErrorCount - before.wsErrorCount, + lastWsReadyState: after.lastWsReadyState, + maxTimerGapMs: after.maxTimerGapMs, + maxRafGapMs: after.maxRafGapMs, + messageTypes, + } +} + +async function collectProcessForestPids(rootPids: number[]): Promise { + if (process.platform === 'win32') { + throw new Error('POSIX SIGSTOP/SIGCONT process suspend probe is not available on native Windows') + } + + const { stdout } = await execFileAsync('ps', ['-eo', 'pid=,ppid=']) + const childrenByParent = new Map() + for (const line of stdout.split(/\r?\n/)) { + const [pidRaw, ppidRaw] = line.trim().split(/\s+/) + const pid = Number(pidRaw) + const ppid = Number(ppidRaw) + if (!Number.isFinite(pid) || !Number.isFinite(ppid)) continue + const children = childrenByParent.get(ppid) ?? [] + children.push(pid) + childrenByParent.set(ppid, children) + } + + const pids = new Set() + const queue = [...rootPids] + while (queue.length > 0) { + const pid = queue.shift() + if (!pid || pids.has(pid)) continue + pids.add(pid) + queue.push(...(childrenByParent.get(pid) ?? [])) + } + return [...pids] +} + +async function collectBrowserExecutionPids(page: Page): Promise { + const browser = page.context().browser() as unknown as { + process?: unknown + newBrowserCDPSession?: unknown + } | null + if (browser && typeof browser.process === 'function') { + const childProcess = browser.process() as { pid?: number } | null + if (childProcess?.pid) { + return collectProcessForestPids([childProcess.pid]) + } + } + + if (!browser || typeof browser.newBrowserCDPSession !== 'function') { + throw new Error('Playwright did not expose browser process or browser-level CDP session for the suspend probe') + } + + const cdpSession = await browser.newBrowserCDPSession() as { + send: (method: string) => Promise + detach: () => Promise + } + try { + const processInfo = await cdpSession.send('SystemInfo.getProcessInfo') as { + processInfo?: Array<{ id?: number; type?: string }> + } + const pids = (processInfo.processInfo ?? []) + .map((entry) => Number(entry.id)) + .filter((pid) => Number.isInteger(pid) && pid > 0) + if (pids.length > 0) { + return collectProcessForestPids(pids) + } + } finally { + await cdpSession.detach().catch(() => {}) + } + + throw new Error('Could not resolve Chromium process IDs for the suspend probe') +} + +function signalPids(pids: number[], signal: NodeJS.Signals): void { + for (const pid of pids) { + try { + process.kill(pid, signal) + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ESRCH') { + throw error + } + } + } +} + +function resolveWsBehavior(input: { + before: StopProbeSnapshot + afterResumeImmediate: StopProbeSnapshot + afterCatchup: StopProbeSnapshot +}): 'still_open_stalled' | 'closed_reconnected' | 'buffered_resumed' { + const stoppedDelta = deltaSnapshots(input.before, input.afterResumeImmediate) + const catchupDelta = deltaSnapshots(input.afterResumeImmediate, input.afterCatchup) + if (stoppedDelta.wsCloseCount > 0 || stoppedDelta.wsOpenCount > 0 || catchupDelta.wsOpenCount > 0) { + return 'closed_reconnected' + } + if (stoppedDelta.wsMessageCount > 0 || catchupDelta.wsMessageCount > 0) { + return 'buffered_resumed' + } + return 'still_open_stalled' +} + +function countPerfEvents(events: Array>, eventName: string): number { + return events.filter((event) => event.event === eventName).length +} + +function countTerminalOutputMessages(snapshot: StopProbeSnapshot): number { + return (snapshot.messageTypes['terminal.output'] ?? 0) + + (snapshot.messageTypes['terminal.output.batch'] ?? 0) +} + +function isTerminalOutputWsFrame(frame: ReceivedWsFrame): boolean { + return frame.type === 'terminal.output' || frame.type === 'terminal.output.batch' +} + +test.describe('terminal background freeze catch-up', () => { + test.skip(({ browserName }) => browserName !== 'chromium', 'Chromium-only CDP/process-suspend proof') + test.skip(process.platform === 'win32', 'POSIX SIGSTOP/SIGCONT process suspend proof is not available on native Windows') + + test('process suspend catches up terminal output without silent gaps or quarantine', async ({ + page, + serverInfo, + harness, + terminal, + }, testInfo) => { + test.slow() + testInfo.annotations.push({ + type: 'terminal-catchup-proof', + description: 'Local POSIX process-suspend positive control; this does not replace the real Windows Chrome background gate.', + }) + + let stoppedPids: number[] = [] + let cdpSession: CDPSession | null = null + const receivedWsFrames: ReceivedWsFrame[] = [] + try { + await installStopProbe(page) + cdpSession = await page.context().newCDPSession(page) + await cdpSession.send('Network.enable') + cdpSession.on('Network.webSocketFrameReceived', (event: { response?: { payloadData?: string } }) => { + const payload = event.response?.payloadData ?? '' + receivedWsFrames.push({ + observedAtMs: Date.now(), + type: classifyWsPayload(payload), + payloadLength: Buffer.byteLength(payload), + }) + }) + await page.goto(`${serverInfo.baseUrl}/?token=${serverInfo.token}&e2e=1&perfAudit=1`) + await harness.waitForHarness() + await harness.waitForConnection() + await selectShellFromPicker(page) + await terminal.waitForTerminal() + await terminal.waitForPrompt({ timeout: 30_000 }) + const terminalId = await getActiveTerminalId(harness) + + await terminal.executeCommand('echo active-freeze-checkpoint') + await terminal.waitForOutput('active-freeze-checkpoint', { terminalId, timeout: 15_000 }) + const parserAppliedCheckpoint = await waitForParserAppliedCheckpoint(page) + expect(parserAppliedCheckpoint).toBeGreaterThan(0) + + const activeBefore = await stopProbeSnapshot(page) + await page.waitForTimeout(ACTIVE_PROBE_MS) + const activeAfter = await stopProbeSnapshot(page) + const activeDelta = deltaSnapshots(activeBefore, activeAfter) + expect(activeDelta.timerTicks).toBeGreaterThan(0) + expect(activeDelta.rafTicks).toBeGreaterThan(0) + + stoppedPids = await collectBrowserExecutionPids(page) + + const scheduledLine = 'freeze-catchup-scheduled' + const startedLine = 'freeze-catchup-started' + const finalLine = `freeze-catchup-${STOPPED_OUTPUT_LINE_COUNT}` + const markerId = `${Date.now()}-${Math.random().toString(36).slice(2)}` + const outputScriptPath = path.join(serverInfo.homeDir, `freeze-catchup-output-${markerId}.cjs`) + const scheduledMarkerPath = path.join(serverInfo.homeDir, `freeze-catchup-scheduled-${markerId}.json`) + const startedMarkerPath = path.join(serverInfo.homeDir, `freeze-catchup-started-${markerId}.json`) + const outputScript = [ + 'const { writeFileSync } = require("fs");', + `const scheduled = ${JSON.stringify(scheduledLine)};`, + `const started = ${JSON.stringify(startedLine)};`, + `const prefix = ${JSON.stringify('freeze-catchup-')};`, + `const scheduledMarkerPath = ${JSON.stringify(scheduledMarkerPath)};`, + `const startedMarkerPath = ${JSON.stringify(startedMarkerPath)};`, + 'writeFileSync(scheduledMarkerPath, JSON.stringify({ scheduledAt: Date.now() }));', + 'console.log(scheduled);', + 'setTimeout(() => {', + ' writeFileSync(startedMarkerPath, JSON.stringify({ startedAt: Date.now() }));', + ' console.log(started);', + ` for (let i = 1; i <= ${STOPPED_OUTPUT_LINE_COUNT}; i += 1) console.log(prefix + i);`, + `}, ${STOPPED_OUTPUT_DELAY_MS});`, + `setTimeout(() => process.exit(0), ${STOPPED_OUTPUT_DELAY_MS + 500});`, + ].join('\n') + await fs.writeFile(outputScriptPath, `${outputScript}\n`, 'utf8') + await terminal.executeCommand(`node ${shellQuote(outputScriptPath)}`) + await waitForFile(scheduledMarkerPath) + + const scheduledMarkerObservedAt = Date.now() + const beforeStop = await stopProbeSnapshot(page) + const wsFrameBaseline = receivedWsFrames.length + const stopStartedAt = Date.now() + expect(stopStartedAt - scheduledMarkerObservedAt).toBeLessThan(STOPPED_OUTPUT_DELAY_MS) + signalPids([...stoppedPids].sort((a, b) => b - a), 'SIGSTOP') + await sleep(STOPPED_PROBE_MS) + signalPids([...stoppedPids].sort((a, b) => a - b), 'SIGCONT') + const stopEndedAt = Date.now() + const stoppedDurationMs = stopEndedAt - stopStartedAt + expect(stoppedDurationMs).toBeGreaterThan(STOPPED_OUTPUT_DELAY_MS) + const afterResumeImmediate = await stopProbeSnapshot(page) + const stoppedDelta = deltaSnapshots(beforeStop, afterResumeImmediate) + await waitForFile(startedMarkerPath) + const outputStartedAt = JSON.parse(await fs.readFile(startedMarkerPath, 'utf8')).startedAt as number + expect(outputStartedAt).toBeGreaterThanOrEqual(stopStartedAt) + expect(outputStartedAt).toBeLessThanOrEqual(stopEndedAt) + + expect(stoppedDelta.timerTicks).toBeLessThan(5) + expect(stoppedDelta.rafTicks).toBeLessThan(5) + + await terminal.waitForOutput(finalLine, { terminalId, timeout: 30_000 }) + await page.waitForTimeout(RESUMED_PROBE_MS) + const afterCatchup = await stopProbeSnapshot(page) + const resumedDelta = deltaSnapshots(afterResumeImmediate, afterCatchup) + expect(resumedDelta.timerTicks).toBeGreaterThan(0) + expect(resumedDelta.rafTicks).toBeGreaterThan(0) + + const buffer = await terminal.getVisibleText(terminalId) + expect(buffer).toContain(startedLine) + expect(buffer).toContain('freeze-catchup-1') + expect(buffer).toContain(finalLine) + + const perfEvents = (await harness.getPerfAuditSnapshot())?.perfEvents ?? [] + const pageCatchupDelta = deltaSnapshots(beforeStop, afterCatchup) + const pageCatchupOutputMessageCount = countTerminalOutputMessages(pageCatchupDelta) + const cdpCatchupFrames = receivedWsFrames.slice(wsFrameBaseline) + const cdpCatchupOutputMessageCount = cdpCatchupFrames.filter(isTerminalOutputWsFrame).length + const outputGapCount = afterCatchup.messageTypes['terminal.output.gap'] ?? 0 + const quarantineCount = countPerfEvents(perfEvents, 'terminal.catchup.surface_quarantined') + const hydrateFallbackCount = countPerfEvents(perfEvents, 'terminal.catchup.full_hydrate_fallback') + const staleGenerationRejectionCount = countPerfEvents(perfEvents, 'terminal.attach_generation_stale_rejected') + const wsBehavior = resolveWsBehavior({ + before: beforeStop, + afterResumeImmediate, + afterCatchup, + }) + const proofBody = { + note: 'Local POSIX process-suspend positive control; not a substitute for the real Windows Chrome gate.', + stoppedPids, + stoppedDurationMs, + stoppedOutputDelayMs: STOPPED_OUTPUT_DELAY_MS, + scheduledToStopMs: stopStartedAt - scheduledMarkerObservedAt, + outputStartedAt, + outputStartedAfterStopMs: outputStartedAt - stopStartedAt, + outputStartedBeforeResumeMs: stopEndedAt - outputStartedAt, + parserAppliedCheckpoint, + wsBehavior, + cdpCatchupOutputMessageCount, + pageCatchupOutputMessageCount, + activeDelta, + stoppedDelta, + resumedDelta, + pageCatchupDelta, + cdpCatchupFrames, + beforeStop, + afterResumeImmediate, + afterCatchup, + outputGapCount, + quarantineCount, + hydrateFallbackCount, + staleGenerationRejectionCount, + } + + const proofPath = testInfo.outputPath('terminal-background-freeze-catchup.json') + await fs.writeFile(proofPath, `${JSON.stringify(proofBody, null, 2)}\n`, 'utf8') + await testInfo.attach('terminal-background-freeze-catchup', { + contentType: 'application/json', + path: proofPath, + }) + + expect(['still_open_stalled', 'closed_reconnected', 'buffered_resumed']).toContain(wsBehavior) + expect(cdpCatchupOutputMessageCount).toBeGreaterThan(0) + expect(outputGapCount).toBe(0) + expect(quarantineCount).toBe(0) + expect(hydrateFallbackCount).toBe(0) + expect(staleGenerationRejectionCount).toBe(0) + } finally { + if (stoppedPids.length > 0) { + signalPids([...stoppedPids].sort((a, b) => a - b), 'SIGCONT') + } + await cdpSession?.detach().catch(() => {}) + await harness.killAllTerminals(serverInfo).catch(() => {}) + } + }) +}) diff --git a/test/e2e/codex-startup-probes.test.tsx b/test/e2e/codex-startup-probes.test.tsx index f2efbabec..54ce126f2 100644 --- a/test/e2e/codex-startup-probes.test.tsx +++ b/test/e2e/codex-startup-probes.test.tsx @@ -30,16 +30,22 @@ const wsHarness = vi.hoisted(() => { const reconnectHandlers = new Set<() => void>() const latestAttachRequestIdByTerminal = new Map() - const withCurrentAttachRequestId = (msg: any) => { + const withCurrentAttachMetadata = (msg: any) => { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + const streamId = typeof msg.streamId === 'string' && msg.streamId.length > 0 + ? msg.streamId + : `stream-${msg.terminalId}` + return { + ...msg, + ...(attachRequestId && !msg.attachRequestId ? { attachRequestId } : {}), + streamId, + } } return { @@ -54,7 +60,7 @@ const wsHarness = vi.hoisted(() => { return () => reconnectHandlers.delete(handler) }), emit(msg: any) { - const normalized = withCurrentAttachRequestId(msg) + const normalized = withCurrentAttachMetadata(msg) for (const handler of messageHandlers) handler(normalized) }, rememberAttach(msg: any) { diff --git a/test/e2e/opencode-startup-probes.test.tsx b/test/e2e/opencode-startup-probes.test.tsx index 174afed64..f5ef1b9d5 100644 --- a/test/e2e/opencode-startup-probes.test.tsx +++ b/test/e2e/opencode-startup-probes.test.tsx @@ -31,16 +31,22 @@ const wsHarness = vi.hoisted(() => { const reconnectHandlers = new Set<() => void>() const latestAttachRequestIdByTerminal = new Map() - const withCurrentAttachRequestId = (msg: any) => { + const withCurrentAttachMetadata = (msg: any) => { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + const streamId = typeof msg.streamId === 'string' && msg.streamId.length > 0 + ? msg.streamId + : `stream-${msg.terminalId}` + return { + ...msg, + ...(attachRequestId && !msg.attachRequestId ? { attachRequestId } : {}), + streamId, + } } return { @@ -55,7 +61,7 @@ const wsHarness = vi.hoisted(() => { return () => reconnectHandlers.delete(handler) }), emit(msg: any) { - const normalized = withCurrentAttachRequestId(msg) + const normalized = withCurrentAttachMetadata(msg) for (const handler of messageHandlers) handler(normalized) }, rememberAttach(msg: any) { diff --git a/test/e2e/terminal-console-violations-regression.test.tsx b/test/e2e/terminal-console-violations-regression.test.tsx index 33bcd5e21..96962abd9 100644 --- a/test/e2e/terminal-console-violations-regression.test.tsx +++ b/test/e2e/terminal-console-violations-regression.test.tsx @@ -12,17 +12,30 @@ import TerminalView from '@/components/TerminalView' const wsHarness = vi.hoisted(() => { const handlers = new Set<(msg: any) => void>() const latestAttachRequestIdByTerminal = new Map() + const latestStreamIdByTerminal = new Map() - const withCurrentAttachRequestId = (msg: any) => { + const withCurrentAttachMetadata = (msg: any) => { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } - const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + let normalized = msg + if (!normalized.attachRequestId) { + const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) + if (attachRequestId) normalized = { ...normalized, attachRequestId } + } + if (normalized.type === 'terminal.attach.ready' && typeof normalized.streamId !== 'string') { + normalized = { ...normalized, streamId: `${msg.terminalId}:stream` } + } else if (typeof normalized.streamId !== 'string') { + const streamId = latestStreamIdByTerminal.get(msg.terminalId) + if (streamId) normalized = { ...normalized, streamId } + } + if (normalized.type === 'terminal.attach.ready' && typeof normalized.streamId === 'string') { + latestStreamIdByTerminal.set(msg.terminalId, normalized.streamId) + } + return normalized } return { @@ -34,7 +47,7 @@ const wsHarness = vi.hoisted(() => { return () => handlers.delete(handler) }), emit(msg: any) { - const normalized = withCurrentAttachRequestId(msg) + const normalized = withCurrentAttachMetadata(msg) for (const handler of handlers) { handler(normalized) } @@ -42,6 +55,7 @@ const wsHarness = vi.hoisted(() => { reset() { handlers.clear() latestAttachRequestIdByTerminal.clear() + latestStreamIdByTerminal.clear() }, rememberAttach(msg: any) { if ( diff --git a/test/e2e/terminal-create-attach-ordering.test.tsx b/test/e2e/terminal-create-attach-ordering.test.tsx index 1490f198e..44b63b3a7 100644 --- a/test/e2e/terminal-create-attach-ordering.test.tsx +++ b/test/e2e/terminal-create-attach-ordering.test.tsx @@ -173,7 +173,7 @@ function createStore(options: { paneTitles: {}, }, settings: { settings: defaultSettings, status: 'loaded' }, - connection: { status: 'ready', error: null }, + connection: { status: 'ready', error: null, serverInstanceId: 'srv-order' }, }, }) } @@ -258,6 +258,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-order-create', + streamId: 'stream-order-create', headSeq: 1, replayFromSeq: 1, replayToSeq: 1, @@ -266,6 +267,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-create', + streamId: 'stream-order-create', seqStart: 1, seqEnd: 1, data: 'create-replay-1', @@ -327,6 +329,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-order-hidden', + streamId: 'stream-order-hidden', headSeq: 8, replayFromSeq: 6, replayToSeq: 8, @@ -335,6 +338,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-hidden', + streamId: 'stream-order-hidden', seqStart: 6, seqEnd: 6, data: 'hidden-r6', @@ -343,6 +347,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-hidden', + streamId: 'stream-order-hidden', seqStart: 8, seqEnd: 8, data: 'hidden-r8', @@ -390,6 +395,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-order-coalesce', + streamId: 'stream-order-coalesce', headSeq: 3, replayFromSeq: 1, replayToSeq: 3, @@ -398,6 +404,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-coalesce', + streamId: 'stream-order-coalesce', seqStart: 1, seqEnd: 1, data: 'replay-1', @@ -406,6 +413,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-coalesce', + streamId: 'stream-order-coalesce', seqStart: 2, seqEnd: 2, data: 'replay-2', @@ -414,6 +422,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-coalesce', + streamId: 'stream-order-coalesce', seqStart: 3, seqEnd: 3, data: 'replay-3', @@ -463,6 +472,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-order-reconnect', + streamId: 'stream-order-reconnect', headSeq: 1, replayFromSeq: 2, replayToSeq: 1, @@ -471,6 +481,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-reconnect', + streamId: 'stream-order-reconnect', seqStart: 2, seqEnd: 2, data: 'before-reconnect', @@ -496,6 +507,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-reconnect', + streamId: 'stream-order-reconnect', seqStart: 3, seqEnd: 3, data: 'stale-after-reconnect', @@ -504,6 +516,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-order-reconnect', + streamId: 'stream-order-reconnect', headSeq: 2, replayFromSeq: 3, replayToSeq: 2, @@ -512,6 +525,7 @@ describe('terminal create/attach ordering (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-order-reconnect', + streamId: 'stream-order-reconnect', seqStart: 3, seqEnd: 3, data: 'fresh-after-reconnect', diff --git a/test/e2e/terminal-flaky-network-responsiveness.test.tsx b/test/e2e/terminal-flaky-network-responsiveness.test.tsx index 853d8aa80..e1eff1798 100644 --- a/test/e2e/terminal-flaky-network-responsiveness.test.tsx +++ b/test/e2e/terminal-flaky-network-responsiveness.test.tsx @@ -238,6 +238,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-flaky', + streamId: 'stream-flaky-gap', headSeq: 3, replayFromSeq: 1, replayToSeq: 3, @@ -245,6 +246,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output.gap', terminalId: 'term-flaky', + streamId: 'stream-flaky-gap', fromSeq: 4, toSeq: 8, reason: 'queue_overflow', @@ -252,6 +254,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-flaky', + streamId: 'stream-flaky-gap', seqStart: 9, seqEnd: 9, data: 'after-gap', @@ -259,8 +262,8 @@ describe('terminal flaky-network responsiveness (e2e)', () => { flushRafQueue(rafCallbacks) - expect(terminalInstances[0].writeln).toHaveBeenCalledWith(expect.stringContaining('[Output gap 4-8')) - expect(terminalInstances[0].write).toHaveBeenCalled() + const writes = terminalInstances[0].write.mock.calls.map(([data]) => String(data)) + expect(writes.some((line) => line.includes('[Output gap 4-8'))).toBe(true) await waitFor(() => { expect(screen.queryByText('Recovering terminal output...')).not.toBeInTheDocument() }) @@ -301,6 +304,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.attach.ready', terminalId: 'term-flaky', + streamId: 'stream-flaky-replay', headSeq: 9, replayFromSeq: 7, replayToSeq: 9, @@ -308,6 +312,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-flaky', + streamId: 'stream-flaky-replay', seqStart: 7, seqEnd: 7, data: 'replay-7', @@ -315,6 +320,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-flaky', + streamId: 'stream-flaky-replay', seqStart: 8, seqEnd: 8, data: 'replay-8', @@ -322,6 +328,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-flaky', + streamId: 'stream-flaky-replay', seqStart: 9, seqEnd: 9, data: 'replay-9', @@ -329,6 +336,7 @@ describe('terminal flaky-network responsiveness (e2e)', () => { wsHarness.emit({ type: 'terminal.output', terminalId: 'term-flaky', + streamId: 'stream-flaky-replay', seqStart: 10, seqEnd: 10, data: 'live-10', diff --git a/test/e2e/terminal-settings-remount-scrollback.test.tsx b/test/e2e/terminal-settings-remount-scrollback.test.tsx index 5e4a9e4dc..257a4731f 100644 --- a/test/e2e/terminal-settings-remount-scrollback.test.tsx +++ b/test/e2e/terminal-settings-remount-scrollback.test.tsx @@ -15,16 +15,29 @@ import TerminalView from '@/components/TerminalView' const wsHarness = vi.hoisted(() => { const handlers = new Set<(msg: any) => void>() const latestAttachRequestIdByTerminal = new Map() - const withCurrentAttachRequestId = (msg: any) => { + const latestStreamIdByTerminal = new Map() + const withCurrentAttachMetadata = (msg: any) => { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } - const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + let normalized = msg + if (!normalized.attachRequestId) { + const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) + if (attachRequestId) normalized = { ...normalized, attachRequestId } + } + if (normalized.type === 'terminal.attach.ready' && typeof normalized.streamId !== 'string') { + normalized = { ...normalized, streamId: `${msg.terminalId}:stream` } + } else if (typeof normalized.streamId !== 'string') { + const streamId = latestStreamIdByTerminal.get(msg.terminalId) + if (streamId) normalized = { ...normalized, streamId } + } + if (normalized.type === 'terminal.attach.ready' && typeof normalized.streamId === 'string') { + latestStreamIdByTerminal.set(msg.terminalId, normalized.streamId) + } + return normalized } return { send: vi.fn(), @@ -35,7 +48,7 @@ const wsHarness = vi.hoisted(() => { return () => handlers.delete(handler) }), emit(msg: any) { - const normalized = withCurrentAttachRequestId(msg) + const normalized = withCurrentAttachMetadata(msg) for (const handler of handlers) { handler(normalized) } @@ -43,6 +56,7 @@ const wsHarness = vi.hoisted(() => { reset() { handlers.clear() latestAttachRequestIdByTerminal.clear() + latestStreamIdByTerminal.clear() }, rememberAttach(msg: any) { if ( @@ -350,7 +364,7 @@ describe('settings remount scrollback hydration (e2e)', () => { const allWrites = terminalInstances.flatMap((instance) => instance.write.mock.calls.map(([data]) => data)) expect(allWrites).toContain('hidden-replayed-after-settings') // Gap messages are written to the terminal for all gap types including replay_window_exceeded - const allGapLines = terminalInstances.flatMap((instance) => instance.writeln.mock.calls.map(([data]) => String(data))) + const allGapLines = allWrites.map((data) => String(data)) expect(allGapLines.some((line) => line.includes('reconnect window exceeded'))).toBe(true) }) @@ -462,8 +476,7 @@ describe('settings remount scrollback hydration (e2e)', () => { expect(allWrites).toContain('hidden-r6') expect(allWrites).toContain('hidden-r8') expect(allWrites).toContain('hidden-live') - const allGapLines = terminalInstances.flatMap((instance) => instance.writeln.mock.calls.map(([data]) => String(data))) - expect(allGapLines.some((line) => line.includes('reconnect window exceeded'))).toBe(false) + expect(allWrites.some((line) => line.includes('reconnect window exceeded'))).toBe(false) }) it('hidden remount restore sends zero attach while hidden and one viewport attach on visibility', async () => { diff --git a/test/helpers/visible-first/terminal-mirror-fixture.ts b/test/helpers/visible-first/terminal-mirror-fixture.ts index 6ec71c9dd..b08dbced4 100644 --- a/test/helpers/visible-first/terminal-mirror-fixture.ts +++ b/test/helpers/visible-first/terminal-mirror-fixture.ts @@ -29,6 +29,7 @@ type TerminalMirrorFixtureOptions = { } const ANSI_ESCAPE_PATTERN = /\u001B\[[0-9;?]*[ -/]*[@-~]/gu +const FIXTURE_STREAM_ID = 'visible-first-terminal-mirror-fixture' function normalizeOutput(value: string): string { return value @@ -54,7 +55,7 @@ export function createTerminalMirrorFixture(options: TerminalMirrorFixtureOption return { applyOutput(rawOutput: string): ReplayFrame { const normalized = normalizeOutput(rawOutput) - const frame = replayRing.append(normalized) + const frame = replayRing.append(normalized, { streamId: FIXTURE_STREAM_ID }) appendText(normalized) return frame }, diff --git a/test/server/ws-edge-cases.test.ts b/test/server/ws-edge-cases.test.ts index a9bbb354d..efef2405d 100644 --- a/test/server/ws-edge-cases.test.ts +++ b/test/server/ws-edge-cases.test.ts @@ -1112,8 +1112,23 @@ describe('WebSocket edge cases', () => { (m) => m.type === 'terminal.output' && m.terminalId === terminalId, ], 5000) - expect(ready.headSeq).toBeGreaterThanOrEqual(replay.seqEnd) - expect(replay.data.length).toBeGreaterThan(60_000) + const replayFrames = [replay] + let nextSeq = (replay.seqEnd as number) + 1 + while (replayFrames[replayFrames.length - 1].seqEnd < ready.headSeq) { + const frame = await waitForMessage( + ws2, + (m) => m.type === 'terminal.output' + && m.terminalId === terminalId + && m.seqStart === nextSeq, + 5000, + ) + replayFrames.push(frame) + nextSeq = (frame.seqEnd as number) + 1 + } + + expect(ready.headSeq).toBeGreaterThanOrEqual(replayFrames[replayFrames.length - 1].seqEnd) + expect(replayFrames.length).toBeGreaterThan(1) + expect(replayFrames.map((frame) => String(frame.data)).join('').length).toBeGreaterThan(50_000) const extras = await collectMessages(ws2, 150) const legacyFrames = extras.filter((m) => typeof m.type === 'string' && m.type.startsWith('terminal.attached')) @@ -1411,11 +1426,22 @@ describe('WebSocket edge cases', () => { const terminalId = await createTerminal(ws, 'large-chunk') // Single large chunk - const outputPromise = waitForMessage(ws, (m) => m.type === 'terminal.output') registry.simulateOutput(terminalId, 'x'.repeat(50_000)) - const output = await outputPromise - expect(output.data.length).toBe(50_000) + let outputData = '' + let nextSeq: number | undefined + while (outputData.length < 50_000) { + const output = await waitForMessage( + ws, + (m) => m.type === 'terminal.output' + && m.terminalId === terminalId + && (nextSeq === undefined || m.seqStart === nextSeq), + 5000, + ) + outputData += String(output.data) + nextSeq = (output.seqEnd as number) + 1 + } + expect(outputData).toHaveLength(50_000) close() }) diff --git a/test/server/ws-protocol.test.ts b/test/server/ws-protocol.test.ts index f84b00417..7c95b1959 100644 --- a/test/server/ws-protocol.test.ts +++ b/test/server/ws-protocol.test.ts @@ -7,6 +7,7 @@ import { ClaudeActivityListResponseSchema, ClaudeActivityUpdatedSchema, ClaudeActivityListSchema, + HelloSchema, TerminalTurnCompleteSchema, } from '../../shared/ws-protocol.js' import { @@ -414,6 +415,38 @@ describe('ws protocol', () => { await closeWebSocket(ws) }) + it('accepts hello with terminalOutputBatchV1 capability', async () => { + const parsed = HelloSchema.safeParse({ + type: 'hello', + token: 'testtoken-testtoken', + protocolVersion: WS_PROTOCOL_VERSION, + capabilities: { + uiScreenshotV1: true, + terminalOutputBatchV1: true, + }, + }) + expect(parsed.success).toBe(true) + + const ws = new WebSocket(`ws://127.0.0.1:${port}/ws`) + await new Promise((resolve) => ws.on('open', () => resolve())) + + ws.send(JSON.stringify({ + type: 'hello', + token: 'testtoken-testtoken', + protocolVersion: WS_PROTOCOL_VERSION, + capabilities: { terminalOutputBatchV1: true }, + })) + + const ready = await new Promise((resolve) => { + ws.on('message', (data) => { + const msg = JSON.parse(data.toString()) + if (msg.type === 'ready') resolve(msg) + }) + }) + expect(ready.type).toBe('ready') + await closeWebSocket(ws) + }) + it('respects hello client.mobile override when classifying the connection', async () => { const ws = new WebSocket(`ws://127.0.0.1:${port}/ws`, { headers: { diff --git a/test/server/ws-tabs-registry.test.ts b/test/server/ws-tabs-registry.test.ts index eddeee2fa..7c7c41425 100644 --- a/test/server/ws-tabs-registry.test.ts +++ b/test/server/ws-tabs-registry.test.ts @@ -127,19 +127,19 @@ describe('ws tabs registry protocol', () => { delete process.env.MAX_REGULAR_WS_MESSAGE_BYTES }) - it('uses protocol version 5 and rejects version 4 clients with reload-required mismatch', async () => { - expect(WS_PROTOCOL_VERSION).toBe(5) + it('uses protocol version 6 and rejects version 5 clients with reload-required mismatch', async () => { + expect(WS_PROTOCOL_VERSION).toBe(6) await startServer({ tabsRegistryStore: await createTabsRegistryStore(tempDir, { now: () => NOW }) }) const ws = new WebSocket(`ws://127.0.0.1:${port}/ws`) await new Promise((resolve) => ws.on('open', () => resolve())) - ws.send(JSON.stringify({ type: 'hello', token: 'tabs-sync-token', protocolVersion: 4 })) + ws.send(JSON.stringify({ type: 'hello', token: 'tabs-sync-token', protocolVersion: 5 })) const error = await waitForMessage(ws, (msg) => msg.type === 'error' && msg.code === 'PROTOCOL_MISMATCH') - expect(error.message).toMatch(/expected protocol version 5/i) + expect(error.message).toMatch(/expected protocol version 6/i) expect(error.message).toMatch(/reload/i) ws.close() }) - it('accepts v5 push/query, returns same-device/devices, and rejects invalid retention', async () => { + it('accepts v6 push/query, returns same-device/devices, and rejects invalid retention', async () => { await startServer({ tabsRegistryStore: await createTabsRegistryStore(tempDir, { now: () => NOW }) }) const ws = await connect() diff --git a/test/server/ws-terminal-stream-v2-replay.test.ts b/test/server/ws-terminal-stream-v2-replay.test.ts index 18701a32f..845c0d222 100644 --- a/test/server/ws-terminal-stream-v2-replay.test.ts +++ b/test/server/ws-terminal-stream-v2-replay.test.ts @@ -243,7 +243,10 @@ function collectMessages(ws: WebSocket, durationMs: number): Promise { }) } -async function createAuthenticatedConnection(port: number): Promise<{ ws: WebSocket; close: () => Promise }> { +async function createAuthenticatedConnection( + port: number, + opts?: { terminalOutputBatchV1?: boolean }, +): Promise<{ ws: WebSocket; close: () => Promise }> { const ws = new WebSocket(`ws://127.0.0.1:${port}/ws`) await new Promise((resolve) => ws.on('open', () => resolve())) @@ -252,6 +255,9 @@ async function createAuthenticatedConnection(port: number): Promise<{ ws: WebSoc type: 'hello', token: 'testtoken-testtoken', protocolVersion: WS_PROTOCOL_VERSION, + ...(opts?.terminalOutputBatchV1 + ? { capabilities: { terminalOutputBatchV1: true } } + : {}), })) await readyPromise @@ -408,9 +414,11 @@ describe('terminal stream v2 replay', () => { expect(ready.replayFromSeq).toBe(3) expect(ready.replayToSeq).toBe(4) expect(ready.attachRequestId).toBe(attachRequestId) - expect(replayed.length).toBe(1) + expect(replayed.length).toBe(2) expect(replayed[0]?.seqStart).toBe(3) - expect(replayed[0]?.seqEnd).toBe(4) + expect(replayed[0]?.seqEnd).toBe(3) + expect(replayed[1]?.seqStart).toBe(4) + expect(replayed[1]?.seqEnd).toBe(4) expect(replayed.map((frame) => frame.data).join('')).toBe('threefour') expect(replayed.every((frame) => frame.seqStart > 2)).toBe(true) expect(replayed.every((frame) => frame.attachRequestId === attachRequestId)).toBe(true) @@ -418,6 +426,101 @@ describe('terminal stream v2 replay', () => { await close2() }) + it('sends terminal.output.batch to batch-capable replay attachments', async () => { + const { ws: ws1, close: close1 } = await createAuthenticatedConnection(port) + const { terminalId } = await createTerminal(ws1, 'stream-batch-create') + + for (const chunk of ['one', 'two', 'three', 'four']) { + registry.simulateOutput(terminalId, chunk) + } + await waitForMessage( + ws1, + (msg) => msg.type === 'terminal.output' && msg.terminalId === terminalId && msg.seqEnd >= 4, + ) + await close1() + + const { ws: ws2, close: close2 } = await createAuthenticatedConnection(port, { + terminalOutputBatchV1: true, + }) + const attachRequestId = 'attach-batch-replay-1' + const batchPromise = waitForMessage( + ws2, + (msg) => msg.type === 'terminal.output.batch' && msg.terminalId === terminalId && msg.seqEnd === 4, + ) + + sendAttach(ws2, terminalId, { sinceSeq: 1, attachRequestId }) + + const batch = await batchPromise + expect(batch).toMatchObject({ + type: 'terminal.output.batch', + terminalId, + attachRequestId, + source: 'replay', + seqStart: 2, + seqEnd: 4, + data: 'twothreefour', + serializedBytes: expect.any(Number), + segments: [ + { seqStart: 2, seqEnd: 2, endOffset: 3, rawFrameCount: 1 }, + { seqStart: 3, seqEnd: 3, endOffset: 8, rawFrameCount: 1 }, + { seqStart: 4, seqEnd: 4, endOffset: 12, rawFrameCount: 1 }, + ], + }) + expect(batch.streamId).toEqual(expect.any(String)) + expect(batch.serializedBytes).toBeGreaterThan(0) + expect(batch.segments.every((segment: any) => segment.streamId === undefined)).toBe(true) + expect(batch.segments.every((segment: any) => segment.attachRequestId === undefined)).toBe(true) + + await close2() + }) + + it('keeps legacy replay fallback as per-segment modern terminal.output frames across barriers', async () => { + const { ws: ws1, close: close1 } = await createAuthenticatedConnection(port) + const { terminalId } = await createTerminal(ws1, 'stream-legacy-barrier-create') + + for (const chunk of ['before', '\u0007', 'after']) { + registry.simulateOutput(terminalId, chunk) + } + await waitForMessage( + ws1, + (msg) => msg.type === 'terminal.output' && msg.terminalId === terminalId && msg.seqEnd >= 3, + ) + await close1() + + const { ws: ws2, close: close2 } = await createAuthenticatedConnection(port) + const attachRequestId = 'attach-legacy-barrier-1' + const replayed: any[] = [] + const onMessage = (data: WebSocket.Data) => { + const msg = JSON.parse(data.toString()) + if (msg.type === 'terminal.output' && msg.terminalId === terminalId) { + replayed.push(msg) + } + } + ws2.on('message', onMessage) + + const tailPromise = waitForMessage( + ws2, + (msg) => msg.type === 'terminal.output' && msg.terminalId === terminalId && msg.seqEnd === 3, + ) + sendAttach(ws2, terminalId, { sinceSeq: 0, attachRequestId }) + await tailPromise + ws2.off('message', onMessage) + + expect(replayed).toHaveLength(3) + expect(replayed.map((frame) => frame.type)).toEqual([ + 'terminal.output', + 'terminal.output', + 'terminal.output', + ]) + expect(replayed.map((frame) => frame.data)).toEqual(['before', '\u0007', 'after']) + expect(replayed.map((frame) => [frame.seqStart, frame.seqEnd])).toEqual([[1, 1], [2, 2], [3, 3]]) + expect(replayed.every((frame) => typeof frame.streamId === 'string' && frame.streamId.length > 0)).toBe(true) + expect(replayed.every((frame) => frame.attachRequestId === attachRequestId)).toBe(true) + expect(replayed.every((frame) => frame.source === 'replay')).toBe(true) + + await close2() + }) + it('terminal.create returns created only until explicit attach', async () => { const { ws, close } = await createAuthenticatedConnection(port) const observed: any[] = [] @@ -495,6 +598,42 @@ describe('terminal stream v2 replay', () => { } }) + it('rejects attachRequestId values too large for terminal.output serialized payload budgets', async () => { + const { ws, close } = await createAuthenticatedConnection(port) + const { terminalId } = await createTerminal(ws, 'stream-long-attach-id-create') + const oversizedAttachRequestId = `long-${'x'.repeat(20 * 1024)}` + + ws.send(JSON.stringify({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + cols: 120, + rows: 40, + attachRequestId: oversizedAttachRequestId, + })) + + const error = await waitForMessage( + ws, + (msg) => msg.type === 'error' && msg.code === 'INVALID_MESSAGE' && msg.terminalId === terminalId, + ) + expect(error.message).toMatch(/attachRequestId/i) + + const messages = await collectMessages(ws, 150) + expect(messages.some((msg) => + msg.type === 'terminal.attach.ready' + && msg.terminalId === terminalId + && msg.attachRequestId === oversizedAttachRequestId, + )).toBe(false) + expect(messages.some((msg) => + msg.type === 'terminal.output' + && msg.terminalId === terminalId + && msg.attachRequestId === oversizedAttachRequestId, + )).toBe(false) + + await close() + }) + it('attach replay from sinceSeq emits ready first and replays an exact range above sequence 1', async () => { const { ws: ws1, close: close1 } = await createAuthenticatedConnection(port) const { terminalId } = await createTerminal(ws1, 'stream-range-create') @@ -537,8 +676,10 @@ describe('terminal stream v2 replay', () => { expect(received[0]?.type).toBe('terminal.attach.ready') const replayed = received.filter((msg) => msg.type === 'terminal.output') - expect(replayed).toHaveLength(1) + expect(replayed).toHaveLength(7) expect(replayed[0]?.seqStart).toBe(6) + expect(replayed[0]?.seqEnd).toBe(6) + expect(replayed[replayed.length - 1]?.seqStart).toBe(12) expect(replayed[replayed.length - 1]?.seqEnd).toBe(12) expect(replayed.map((msg) => msg.data ?? '').join('')).toBe('f6|f7|f8|f9|f10|f11|f12|') expect(received.some((msg) => msg.type === 'terminals.changed')).toBe(false) diff --git a/test/unit/client/components/TerminalView.keyboard.test.tsx b/test/unit/client/components/TerminalView.keyboard.test.tsx index beb3c9f3e..7b9a1eb53 100644 --- a/test/unit/client/components/TerminalView.keyboard.test.tsx +++ b/test/unit/client/components/TerminalView.keyboard.test.tsx @@ -831,6 +831,46 @@ describe('TerminalView keyboard handling', () => { data: 'pasted content', }) }) + + it('stale terminal actions captured before remount do not mutate the old terminal instance', async () => { + clipboardMocks.readText.mockResolvedValue('pasted content') + const { store, tabId, paneId, paneContent } = createTestStore('term-1') + + const first = render( + + + + ) + + await waitFor(() => { + expect(capturedTerminal).not.toBeNull() + }) + + const staleTerminal = capturedTerminal! + const staleActions = getTerminalActions(paneId)! + first.unmount() + + render( + + + + ) + + await waitFor(() => { + expect(capturedTerminal).not.toBe(staleTerminal) + }) + + await staleActions.paste() + staleActions.selectAll() + staleActions.clearScrollback() + staleActions.reset() + staleActions.scrollToBottom() + + expect(staleTerminal.paste).not.toHaveBeenCalled() + expect(staleTerminal.selectAll).not.toHaveBeenCalled() + expect(staleTerminal.clear).not.toHaveBeenCalled() + expect(staleTerminal.reset).not.toHaveBeenCalled() + }) }) describe('other keys', () => { diff --git a/test/unit/client/components/TerminalView.lifecycle.test.tsx b/test/unit/client/components/TerminalView.lifecycle.test.tsx index 70ed93e69..361ba9470 100644 --- a/test/unit/client/components/TerminalView.lifecycle.test.tsx +++ b/test/unit/client/components/TerminalView.lifecycle.test.tsx @@ -5,7 +5,7 @@ import { Provider } from 'react-redux' import tabsReducer, { setActiveTab } from '@/store/tabsSlice' import panesReducer, { requestPaneRefresh } from '@/store/panesSlice' import settingsReducer, { defaultSettings, updateSettingsLocal } from '@/store/settingsSlice' -import connectionReducer from '@/store/connectionSlice' +import connectionReducer, { setStatus as setConnectionStatus } from '@/store/connectionSlice' import turnCompletionReducer from '@/store/turnCompletionSlice' import paneRuntimeActivityReducer from '@/store/paneRuntimeActivitySlice' import { persistMiddleware, resetPersistedLayoutCacheForTests, resetPersistFlushListenersForTests } from '@/store/persistMiddleware' @@ -14,7 +14,11 @@ import { LAYOUT_STORAGE_KEY } from '@/store/storage-keys' import { flushPersistedLayoutNow } from '@/store/persistControl' import { useAppSelector } from '@/store/hooks' import type { PaneNode, TerminalPaneContent } from '@/store/paneTypes' -import { __resetTerminalCursorCacheForTests } from '@/lib/terminal-cursor' +import { + __resetTerminalCursorCacheForTests, + loadTerminalSurfaceCheckpoint, + saveTerminalSurfaceCheckpoint, +} from '@/lib/terminal-cursor' import { resetHydrationQueueForTests } from '@/lib/hydration-queue' import { createPerfAuditBridge, installPerfAuditBridge } from '@/lib/perf-audit-bridge' import { TERMINAL_CURSOR_STORAGE_KEY } from '@/store/storage-keys' @@ -72,6 +76,7 @@ vi.mock('lucide-react', () => ({ const terminalInstances: any[] = [] const latestAttachRequestIdByTerminal = new Map() +const latestStreamIdByTerminal = new Map() vi.mock('@xterm/xterm', () => { class MockTerminal { @@ -249,17 +254,60 @@ function createSettingsState(overrides: Record = {}) { } } +function terminalWriteStrings(term: { write: { mock: { calls: Array<[unknown]> } } }): string[] { + return term.write.mock.calls.map(([data]) => String(data)) +} + +function expectTerminalWriteContaining(term: { write: { mock: { calls: Array<[unknown]> } } }, text: string) { + expect(terminalWriteStrings(term).some((entry) => entry.includes(text))).toBe(true) +} + function withCurrentAttachRequestId( - msg: T & { __preserveMissingAttachRequestId?: boolean }, + msg: T & { __preserveMissingAttachRequestId?: boolean; __preserveMissingStreamId?: boolean }, ): T { - if (msg.__preserveMissingAttachRequestId) return msg - if (msg.attachRequestId || typeof msg.terminalId !== 'string') return msg - if (msg.type !== 'terminal.attach.ready' && msg.type !== 'terminal.output' && msg.type !== 'terminal.output.gap') { + const isStreamPayload = msg.type === 'terminal.attach.ready' + || msg.type === 'terminal.stream.changed' + || msg.type === 'terminal.output' + || msg.type === 'terminal.output.batch' + || msg.type === 'terminal.output.gap' + if (!isStreamPayload || typeof msg.terminalId !== 'string') { return msg } - const attachRequestId = latestAttachRequestIdForTerminal(msg.terminalId) - if (!attachRequestId) return msg - return { ...msg, attachRequestId } + + let next: T & { __preserveMissingAttachRequestId?: boolean; __preserveMissingStreamId?: boolean } = msg + if (!msg.__preserveMissingAttachRequestId && !msg.attachRequestId) { + const attachRequestId = latestAttachRequestIdForTerminal(msg.terminalId) + if (attachRequestId) { + next = { ...next, attachRequestId } + } + } + + if (!msg.__preserveMissingStreamId) { + if (msg.type === 'terminal.attach.ready') { + const streamId = typeof (next as { streamId?: unknown }).streamId === 'string' + ? (next as { streamId: string }).streamId + : (latestStreamIdByTerminal.get(msg.terminalId) ?? `test-stream:${msg.terminalId}`) + next = { ...next, streamId } as typeof next + latestStreamIdByTerminal.set(msg.terminalId, streamId) + } else if (msg.type === 'terminal.output' || msg.type === 'terminal.output.batch' || msg.type === 'terminal.output.gap') { + const messageStreamId = (next as { streamId?: unknown }).streamId + const streamId = typeof messageStreamId === 'string' && messageStreamId.length > 0 + ? messageStreamId + : latestStreamIdByTerminal.get(msg.terminalId) + if (streamId) { + next = { ...next, streamId } as typeof next + } + } + } + + if (msg.type === 'terminal.stream.changed') { + const streamId = (next as { streamId?: unknown }).streamId + if (typeof streamId === 'string' && streamId.length > 0) { + latestStreamIdByTerminal.set(msg.terminalId, streamId) + } + } + + return next } function sentMessages() { @@ -280,6 +328,7 @@ describe('TerminalView lifecycle updates', () => { resetPersistedLayoutCacheForTests() resetPersistFlushListenersForTests() latestAttachRequestIdByTerminal.clear() + latestStreamIdByTerminal.clear() wsMocks.send.mockClear() wsMocks.send.mockImplementation((msg: any) => { if ( @@ -2455,8 +2504,8 @@ describe('TerminalView lifecycle updates', () => { const tab = store.getState().tabs.tabs.find((entry) => entry.id === tabId) expect(tab?.status).toBe('error') - expect(term.writeln).toHaveBeenCalledWith(expect.stringContaining('[Restore failed]')) - expect(term.writeln).toHaveBeenCalledWith(expect.stringContaining('execvp(3) failed.: No such file or directory')) + expectTerminalWriteContaining(term, '[Restore failed]') + expectTerminalWriteContaining(term, 'execvp(3) failed.: No such file or directory') }) it('marks startup exit before first attach as a launch failure', async () => { @@ -2542,9 +2591,7 @@ describe('TerminalView lifecycle updates', () => { expect(wsMocks.send.mock.calls.filter(([msg]) => msg?.type === 'terminal.create')).toHaveLength(0) expect(store.getState().tabs.tabs.find((entry) => entry.id === tabId)?.status).toBe('error') - expect(term.writeln).toHaveBeenCalledWith( - expect.stringContaining('[Launch failed] The terminal exited before it finished starting (exit 2).'), - ) + expectTerminalWriteContaining(term, '[Launch failed] The terminal exited before it finished starting (exit 2).') }) it('marks restored terminal.create requests', async () => { @@ -2782,8 +2829,7 @@ describe('TerminalView lifecycle updates', () => { // Verify user-facing feedback was shown const term = terminalInstances[0] - const writelnCalls = term.writeln.mock.calls.map(([s]: [string]) => s) - expect(writelnCalls.some((s: string) => s.includes('Terminal exited'))).toBe(true) + expectTerminalWriteContaining(term, 'Terminal exited') }) it('shows feedback when Codex input is blocked by the restore identity gate', async () => { @@ -2813,9 +2859,7 @@ describe('TerminalView lifecycle updates', () => { }) const term = terminalInstances[0] - expect(term.writeln).toHaveBeenCalledWith( - expect.stringContaining('Input not sent: Codex is still saving restore state. Try again in a moment.'), - ) + expectTerminalWriteContaining(term, 'Input not sent: Codex is still saving restore state. Try again in a moment.') }) it('shows feedback when Codex input is blocked by lifecycle-loss proof', async () => { @@ -2845,9 +2889,7 @@ describe('TerminalView lifecycle updates', () => { }) const term = terminalInstances[0] - expect(term.writeln).toHaveBeenCalledWith( - expect.stringContaining('Input not sent: Codex is resolving a worker disconnect. Try again in a moment.'), - ) + expectTerminalWriteContaining(term, 'Input not sent: Codex is resolving a worker disconnect. Try again in a moment.') }) it('shows feedback when Codex input is blocked by clean-exit state resolution', async () => { @@ -2877,9 +2919,7 @@ describe('TerminalView lifecycle updates', () => { }) const term = terminalInstances[0] - expect(term.writeln).toHaveBeenCalledWith( - expect.stringContaining('Input not sent: Codex is checking whether the session is still active. Try again in a moment.'), - ) + expectTerminalWriteContaining(term, 'Input not sent: Codex is checking whether the session is still active. Try again in a moment.') }) it('mirrors canonical durable identity to pane and tab on terminal.session.associated', async () => { @@ -3497,6 +3537,8 @@ describe('TerminalView lifecycle updates', () => { ackInitialAttach?: boolean refreshOnMount?: boolean sessionRef?: TerminalPaneContent['sessionRef'] + serverInstanceId?: string + streamId?: string }) { const tabId = 'tab-v2-stream' const paneId = 'pane-v2-stream' @@ -3504,6 +3546,9 @@ describe('TerminalView lifecycle updates', () => { const initialStatus = opts?.status ?? 'running' const terminalId = opts?.terminalId const mode = opts?.mode ?? 'shell' + if (terminalId && opts?.streamId) { + latestStreamIdByTerminal.set(terminalId, opts.streamId) + } const paneContent: TerminalPaneContent = { kind: 'terminal', @@ -3513,6 +3558,7 @@ describe('TerminalView lifecycle updates', () => { shell: 'system', ...(terminalId ? { terminalId } : {}), ...(opts?.sessionRef ? { sessionRef: opts.sessionRef } : {}), + ...(opts?.streamId ? { streamId: opts.streamId } : {}), } const root: PaneNode = { type: 'leaf', id: paneId, content: paneContent } @@ -3559,7 +3605,7 @@ describe('TerminalView lifecycle updates', () => { : {}, }, settings: createSettingsState(), - connection: { status: 'connected', error: null }, + connection: { status: 'connected', error: null, serverInstanceId: opts?.serverInstanceId ?? 'srv-v2-stream' }, }, }) @@ -3813,7 +3859,7 @@ describe('TerminalView lifecycle updates', () => { }) }) - it('reconnect and future creates stay on the explicit attach lifecycle', async () => { + it('reconnect without a parser-applied checkpoint stays on the explicit hydrate lifecycle', async () => { const first = await renderTerminalHarness({ status: 'creating', hidden: false, @@ -3848,7 +3894,8 @@ describe('TerminalView lifecycle updates', () => { terminalId: 'term-latched-1', cols: expect.any(Number), rows: expect.any(Number), - intent: 'transport_reconnect', + intent: 'viewport_hydrate', + sinceSeq: 0, })) first.unmount() @@ -3929,6 +3976,8 @@ describe('TerminalView lifecycle updates', () => { }) it('drops stale and untagged terminal.output from non-current attach generations', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) const { terminalId, term } = await renderTerminalHarness({ status: 'running', terminalId: 'term-attach-gen', @@ -3950,211 +3999,2995 @@ describe('TerminalView lifecycle updates', () => { expect(secondAttach?.attachRequestId).toBeTruthy() expect(secondAttach?.attachRequestId).not.toBe(firstAttach?.attachRequestId) - messageHandler!({ - type: 'terminal.output', - terminalId, - seqStart: 1, - seqEnd: 1, - data: 'STALE', - attachRequestId: firstAttach!.attachRequestId, - } as any) - messageHandler!({ - type: 'terminal.output', - terminalId, - seqStart: 2, - seqEnd: 2, - data: 'FRESH', - attachRequestId: secondAttach!.attachRequestId, - } as any) - messageHandler!({ - type: 'terminal.output', - terminalId, - seqStart: 3, - seqEnd: 3, - data: 'UNTAGGED', - __preserveMissingAttachRequestId: true, - } as any) + let now = 200 + const performanceNowSpy = vi.spyOn(performance, 'now').mockImplementation(() => { + now += 0.01 + return now + }) + try { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'STALE', + attachRequestId: firstAttach!.attachRequestId, + } as any) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'FRESH', + attachRequestId: secondAttach!.attachRequestId, + } as any) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 3, + seqEnd: 3, + data: 'UNTAGGED', + __preserveMissingAttachRequestId: true, + } as any) + } finally { + performanceNowSpy.mockRestore() + } const writes = term.write.mock.calls.map(([d]) => String(d)).join('') expect(writes).toContain('FRESH') expect(writes).not.toContain('STALE') expect(writes).not.toContain('UNTAGGED') - }) - - it('keeps queued viewport_hydrate intent when reconnect fires before the first hidden attach completes', async () => { - const { requestId, rerender, store, tabId, paneId } = await renderTerminalHarness({ - status: 'creating', - hidden: true, - requestId: 'req-v2-hidden-created-before-reconnect', - }) - - wsMocks.send.mockClear() - messageHandler!({ - type: 'terminal.created', - requestId, - terminalId: 'term-hidden-created-before-reconnect', - createdAt: Date.now(), - }) - - reconnectHandler?.() - - rerender( - - , - ) - - await waitFor(() => { - expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', - terminalId: 'term-hidden-created-before-reconnect', - sinceSeq: 0, - cols: expect.any(Number), - rows: expect.any(Number), - })) - }) - }) - - it('uses the highest rendered sequence in reconnect attach requests', async () => { - const { terminalId, term } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-reconnect' }) - - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 2, data: 'ab' }) - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 3, seqEnd: 3, data: 'c' }) - - const writes = term.write.mock.calls.map(([data]: [string]) => data) - expect(writes).toContain('ab') - expect(writes).toContain('c') - - wsMocks.send.mockClear() - reconnectHandler?.() - - expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', + const staleRejectedEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.attach_generation_stale_rejected') + expect(staleRejectedEvents).toHaveLength(2) + expect(staleRejectedEvents[0]).toEqual(expect.objectContaining({ + event: 'terminal.attach_generation_stale_rejected', + timestamp: expect.any(Number), terminalId, - sinceSeq: 3, - attachRequestId: expect.any(String), + messageType: 'terminal.output', + attachRequestId: firstAttach!.attachRequestId, + activeAttachRequestId: secondAttach!.attachRequestId, + reason: 'stale_attach_request_id', + })) + expect(staleRejectedEvents[1]).toEqual(expect.objectContaining({ + event: 'terminal.attach_generation_stale_rejected', + timestamp: expect.any(Number), + terminalId, + messageType: 'terminal.output', + activeAttachRequestId: secondAttach!.attachRequestId, + reason: 'missing_attach_request_id', })) + expect(staleRejectedEvents[1]).not.toHaveProperty('attachRequestId') + expect(Number(staleRejectedEvents[0].timestamp)).toBeLessThan(Number(staleRejectedEvents[1].timestamp)) + expect(bridge.snapshot().metadata['terminal.attach_generation_stale_rejected']).toBeUndefined() + expect(bridge.snapshot().milestones['terminal.attach_generation_stale_rejected']).toBeUndefined() }) - it('reattaches with latest rendered sequence after terminal view remount', async () => { - const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-remount' }) - - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) - unmount() - wsMocks.send.mockClear() + it('persists attach-ready stream id into pane content and checkpoint identity', async () => { + const { store, tabId, terminalId } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-attach-stream-checkpoint', + serverInstanceId: 'server-attach-stream', + ackInitialAttach: false, + clearSends: false, + }) - render( - - - - ) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() - await waitFor(() => { - expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', terminalId, - sinceSeq: 0, - attachRequestId: expect.any(String), - })) + streamId: 'stream-from-ready', + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-from-ready', + seqStart: 1, + seqEnd: 1, + data: 'checkpointed on stream', + attachRequestId: attach!.attachRequestId, + }) }) - }) - - it('does not attach a remounted hidden pane until it becomes visible', async () => { - const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-hidden-remount' }) - - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) - unmount() - wsMocks.send.mockClear() - - render( - - - ) - await waitFor(() => { - expect(messageHandler).not.toBeNull() + const layout = store.getState().panes.layouts[tabId] + expect(layout.type).toBe('leaf') + expect(layout.content.kind).toBe('terminal') + expect(layout.content.streamId).toBe('stream-from-ready') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-from-ready', + serverInstanceId: 'server-attach-stream', + })?.parserAppliedSeq).toBe(1) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: null, + serverInstanceId: 'server-attach-stream', + })).toBeNull() + }) + + it('accepts live output after a terminal.stream.changed control message without trusting the old stream', async () => { + const { store, tabId, terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-active-stream-change-client', + serverInstanceId: 'server-active-stream-change', + ackInitialAttach: false, + clearSends: false, }) - expect(wsMocks.send).not.toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', - terminalId, - })) - }) - - it('performs one deferred viewport hydration attach when a remounted hidden pane becomes visible', async () => { - const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-deferred-hydrate' }) - - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) - unmount() - wsMocks.send.mockClear() - const view = render( - - - ) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() - await waitFor(() => { - expect(messageHandler).not.toBeNull() + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-before-change', + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-before-change', + seqStart: 1, + seqEnd: 1, + data: 'BEFORE STREAM CHANGE', + attachRequestId: attach!.attachRequestId, + }) }) - expect(wsMocks.send).not.toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', - terminalId, - })) - wsMocks.send.mockClear() - view.rerender( - - - ) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-before-change', + serverInstanceId: 'server-active-stream-change', + })?.parserAppliedSeq).toBe(1) - await waitFor(() => { - expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ - type: 'terminal.attach', + act(() => { + messageHandler!({ + type: 'terminal.stream.changed', terminalId, - sinceSeq: 0, - intent: 'viewport_hydrate', - attachRequestId: expect.any(String), - })) + streamId: 'stream-after-change', + reason: 'codex_pty_recovery', + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-after-change', + seqStart: 2, + seqEnd: 2, + data: 'AFTER STREAM CHANGE', + attachRequestId: attach!.attachRequestId, + }) }) - expect(wsMocks.send.mock.calls - .map(([msg]) => msg) - .filter((msg) => msg?.type === 'terminal.resize' && msg?.terminalId === terminalId)).toHaveLength(0) - }) - it('uses keepalive_delta when a live terminal re-runs the attach effect above the rendered high-water mark', async () => { - const { rerender, store, tabId, paneId, terminalId, term } = await renderTerminalHarness({ + const writes = terminalWriteStrings(term).join('') + expect(writes).toContain('BEFORE STREAM CHANGE') + expect(writes).toContain('AFTER STREAM CHANGE') + + const layout = store.getState().panes.layouts[tabId] + expect(layout.type).toBe('leaf') + expect(layout.content.kind).toBe('terminal') + expect(layout.content.streamId).toBe('stream-after-change') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-before-change', + serverInstanceId: 'server-active-stream-change', + })).toBeNull() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-after-change', + serverInstanceId: 'server-active-stream-change', + })?.parserAppliedSeq).toBe(2) + }) + + it('treats mismatched replay after a stream change as a completing lost range', async () => { + const { store, terminalId, term, queryByText } = await renderTerminalHarness({ status: 'running', - terminalId: 'term-v2-keepalive-intent', + terminalId: 'term-stale-replay-stream-change-client', + serverInstanceId: 'server-stale-replay-stream-change', + ackInitialAttach: false, clearSends: false, }) + act(() => { + store.dispatch(setConnectionStatus('ready')) + }) - const initialAttachRequestId = latestAttachRequestIdForTerminal(terminalId) - expect(initialAttachRequestId).toBeTruthy() + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() act(() => { messageHandler!({ type: 'terminal.attach.ready', terminalId, + streamId: 'stream-before-change', headSeq: 0, replayFromSeq: 1, replayToSeq: 0, - attachRequestId: initialAttachRequestId, + attachRequestId: attach!.attachRequestId, }) - messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) }) - const writes = term.write.mock.calls.map(([data]: [string]) => data) - expect(writes).toContain('abc') - wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + const replayAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(replayAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-before-change', + headSeq: 2, + replayFromSeq: 1, + replayToSeq: 2, + attachRequestId: replayAttach!.attachRequestId, + }) + }) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + + act(() => { + messageHandler!({ + type: 'terminal.stream.changed', + terminalId, + streamId: 'stream-after-change', + reason: 'codex_pty_recovery', + attachRequestId: replayAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-before-change', + seqStart: 1, + seqEnd: 2, + data: 'STALE REPLAY SHOULD NOT RENDER', + attachRequestId: replayAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-after-change', + seqStart: 3, + seqEnd: 3, + data: 'LIVE AFTER STREAM CHANGE', + attachRequestId: replayAttach!.attachRequestId, + }) + }) + + const writes = terminalWriteStrings(term).join('') + expect(writes).not.toContain('STALE REPLAY SHOULD NOT RENDER') + expect(writes).toContain('LIVE AFTER STREAM CHANGE') + expect(queryByText('Recovering terminal output...')).toBeNull() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-after-change', + serverInstanceId: 'server-stale-replay-stream-change', + })).toBeNull() + }) + + it('rejects a warm-delta attach when attach-ready reports a different stream id', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { store, tabId, terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-stream-rotation-client', + serverInstanceId: 'server-stream-rotation', + ackInitialAttach: false, + clearSends: false, + }) + + const initialAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(initialAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-before-rotation', + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: initialAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-before-rotation', + seqStart: 1, + seqEnd: 1, + data: 'before rotation', + attachRequestId: initialAttach!.attachRequestId, + }) + }) + + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-before-rotation', + serverInstanceId: 'server-stream-rotation', + })?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + const warmDeltaAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(warmDeltaAttach).toMatchObject({ + intent: 'transport_reconnect', + sinceSeq: 1, + }) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-after-rotation', + headSeq: 1, + replayFromSeq: 2, + replayToSeq: 1, + attachRequestId: warmDeltaAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-after-rotation', + seqStart: 2, + seqEnd: 2, + data: 'STREAM B SHOULD NOT RENDER ON STREAM A SURFACE', + attachRequestId: warmDeltaAttach!.attachRequestId, + }) + }) + + const layout = store.getState().panes.layouts[tabId] + expect(layout.type).toBe('leaf') + expect(layout.content.kind).toBe('terminal') + expect(layout.content.streamId).toBeUndefined() + expect(terminalWriteStrings(term).join('')).not.toContain('STREAM B SHOULD NOT RENDER') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-after-rotation', + serverInstanceId: 'server-stream-rotation', + })).toBeNull() + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + const repairAttach = sentMessages() + .filter((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + .at(-1) + expect(repairAttach?.attachRequestId).not.toBe(warmDeltaAttach!.attachRequestId) + const fallbackEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.full_hydrate_fallback') + expect(fallbackEvents).toEqual([ + expect.objectContaining({ + event: 'terminal.catchup.full_hydrate_fallback', + timestamp: expect.any(Number), + terminalId, + attachRequestId: warmDeltaAttach!.attachRequestId, + reason: 'stream_identity_changed', + expectedStreamId: 'stream-before-rotation', + streamId: 'stream-after-rotation', + sinceSeq: 1, + }), + ]) + expect(bridge.snapshot().milestones['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().metadata['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + }) + + it('rejects a warm-delta attach when attach-ready reports unknown geometry authority', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-geometry-authority-client', + serverInstanceId: 'server-geometry-authority', + ackInitialAttach: false, + clearSends: false, + }) + + const initialAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(initialAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-geometry', + geometryEpoch: 1, + geometryAuthority: 'single_client', + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: initialAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-geometry', + seqStart: 1, + seqEnd: 1, + data: 'before geometry conflict', + attachRequestId: initialAttach!.attachRequestId, + }) + }) + + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-geometry', + serverInstanceId: 'server-geometry-authority', + })?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + const warmDeltaAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(warmDeltaAttach).toMatchObject({ + intent: 'transport_reconnect', + sinceSeq: 1, + }) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-geometry', + geometryEpoch: 2, + geometryAuthority: 'multi_client_unknown', + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: warmDeltaAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-geometry', + seqStart: 2, + seqEnd: 2, + data: 'GEOMETRY DELTA SHOULD NOT RENDER', + attachRequestId: warmDeltaAttach!.attachRequestId, + }) + }) + + expect(terminalWriteStrings(term).join('')).not.toContain('GEOMETRY DELTA SHOULD NOT RENDER') + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + const repairAttach = sentMessages() + .filter((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + .at(-1) + expect(repairAttach?.attachRequestId).not.toBe(warmDeltaAttach!.attachRequestId) + const fallbackEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.full_hydrate_fallback') + expect(fallbackEvents).toEqual([ + expect.objectContaining({ + event: 'terminal.catchup.full_hydrate_fallback', + timestamp: expect.any(Number), + terminalId, + attachRequestId: warmDeltaAttach!.attachRequestId, + reason: 'geometry_authority_unknown', + geometryAuthority: 'multi_client_unknown', + geometryEpoch: 2, + expectedGeometryAuthority: 'single_client', + expectedGeometryEpoch: 1, + sinceSeq: 1, + }), + ]) + expect(bridge.snapshot().milestones['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().metadata['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + }) + + it('does not render or checkpoint terminal.output from a mismatched stream id', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-stream-mismatch-client', + serverInstanceId: 'server-stream-mismatch', + ackInitialAttach: false, + clearSends: false, + }) + + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-active', + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-stale', + seqStart: 1, + seqEnd: 1, + data: 'STALE STREAM', + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-active', + seqStart: 2, + seqEnd: 2, + data: 'ACTIVE STREAM', + attachRequestId: attach!.attachRequestId, + }) + }) + + const writes = term.write.mock.calls.map(([data]) => String(data)).join('') + expect(writes).not.toContain('STALE STREAM') + expect(writes).toContain('ACTIVE STREAM') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-active', + serverInstanceId: 'server-stream-mismatch', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('writes a homogeneous terminal.output.batch once and advances the parser-applied cursor after acknowledgement', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-combined', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 3, + data: 'abc', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1 }, + { seqStart: 3, seqEnd: 3, endOffset: 3, rawFrameCount: 1 }, + ], + }) + }) + + expect(terminalWriteStrings(term)).toContain('abc') + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 3, + })) + const parserAppliedEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.parser_applied') + expect(parserAppliedEvents).toEqual([ + expect.objectContaining({ + event: 'terminal.parser_applied', + timestamp: expect.any(Number), + terminalId, + attachRequestId, + parserAppliedSeq: 3, + previousParserAppliedSeq: 0, + surfaceQuarantined: false, + }), + ]) + expect(bridge.snapshot().milestones['terminal.parser_applied']).toBeUndefined() + }) + + it('records each parser-applied acknowledgement as a separate audit event', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-parser-applied-events', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + let now = 100 + const performanceNowSpy = vi.spyOn(performance, 'now').mockImplementation(() => { + now += 0.01 + return now + }) + try { + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: 'first parser applied', + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 2, + seqEnd: 2, + data: 'second parser applied', + }) + }) + + const parserAppliedEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.parser_applied') + expect(parserAppliedEvents).toHaveLength(2) + expect(parserAppliedEvents[0]).toEqual(expect.objectContaining({ + timestamp: expect.any(Number), + terminalId, + attachRequestId, + streamId, + parserAppliedSeq: 1, + previousParserAppliedSeq: 0, + })) + expect(parserAppliedEvents[1]).toEqual(expect.objectContaining({ + timestamp: expect.any(Number), + terminalId, + attachRequestId, + streamId, + parserAppliedSeq: 2, + previousParserAppliedSeq: 1, + })) + expect(Number(parserAppliedEvents[0].timestamp)).toBeLessThan(Number(parserAppliedEvents[1].timestamp)) + expect(bridge.snapshot().metadata['terminal.parser_applied']).toBeUndefined() + } finally { + performanceNowSpy.mockRestore() + } + }) + + it('rejects an overlapping terminal.output.batch before writing partial bytes', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-overlap', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: 'already-rendered', + }) + }) + expect(terminalWriteStrings(term)).toContain('already-rendered') + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 2, + data: 'ab', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1 }, + ], + }) + }) + + expect(term.write).not.toHaveBeenCalled() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 2, + seqEnd: 2, + data: 'accepted-after-reject', + }) + }) + expect(terminalWriteStrings(term)).toContain('accepted-after-reject') + }) + + it('rejects terminal.output.batch with non-contiguous segment ranges before writing', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-hole', + serverInstanceId: 'server-output-batch-hole', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 3, + data: 'ac', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 3, seqEnd: 3, endOffset: 2, rawFrameCount: 1 }, + ], + }) + }) + + expect(term.write).not.toHaveBeenCalled() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 4, + seqEnd: 4, + data: 'accepted-after-hole', + }) + }) + expect(terminalWriteStrings(term)).toContain('accepted-after-hole') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-hole', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + })) + }) + + it('rejects terminal.output.batch with malformed fields before writing or checkpointing', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-malformed-numbers', + serverInstanceId: 'server-output-batch-malformed-numbers', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + const validSegment = { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 } + const malformedBatches: Array> = [ + { seqStart: '1', seqEnd: 1, segments: [validSegment] }, + { seqStart: 1, seqEnd: null, segments: [validSegment] }, + { seqStart: 1, seqEnd: 1, serializedBytes: '128', segments: [validSegment] }, + { seqStart: 1, seqEnd: 1, serializedBytes: null, segments: [validSegment] }, + { seqStart: 1, seqEnd: 1, serializedBytes: -1, segments: [validSegment] }, + { seqStart: 1, seqEnd: 1, serializedBytes: 128.5, segments: [validSegment] }, + { seqStart: 1, seqEnd: 1, data: null, segments: [{ seqStart: 1, seqEnd: 1, endOffset: 0, rawFrameCount: 1 }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, seqStart: '1' }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, seqEnd: null }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, endOffset: true }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, endOffset: Number.POSITIVE_INFINITY }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: '1' }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: null }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: 0 }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: -1 }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: 1.5 }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, rawFrameCount: 2 }] }, + { seqStart: 1, seqEnd: 2, segments: [{ seqStart: 1, seqEnd: 2, endOffset: 1, rawFrameCount: 1 }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, barrier: null }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, barrier: '' }] }, + { seqStart: 1, seqEnd: 1, segments: [{ ...validSegment, barrier: 'unknown' }] }, + ] + + term.write.mockClear() + act(() => { + for (const malformed of malformedBatches) { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + data: 'x', + serializedBytes: 128, + ...malformed, + }) + } + }) + + expect(term.write).not.toHaveBeenCalled() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-malformed-numbers', + })).toBeNull() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 3, + seqEnd: 3, + data: 'accepted-after-malformed-batch', + }) + }) + expect(terminalWriteStrings(term)).toContain('accepted-after-malformed-batch') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-malformed-numbers', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + })) + }) + + it('rejects terminal.output.batch when an endOffset splits a UTF-16 surrogate pair', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-surrogate-split', + serverInstanceId: 'server-output-batch-surrogate-split', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 2, + data: '\ud83d\ude00', + serializedBytes: 128, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, data: '\ud83d', rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, data: '\ude00', rawFrameCount: 1 }, + ], + }) + }) + + expect(term.write).not.toHaveBeenCalled() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-surrogate-split', + })).toBeNull() + }) + + it('rejects terminal.output.batch when segment data disagrees with offsets', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-data-mismatch', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 2, + data: 'ab', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, data: 'a', rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, data: 'not-b', rawFrameCount: 1 }, + ], + }) + }) + + expect(term.write).not.toHaveBeenCalled() + }) + + it('fails closed after an invalid terminal.output.batch instead of checkpointing later output across the lost range', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-invalid-fail-closed', + serverInstanceId: 'server-output-batch-invalid-fail-closed', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 2, + data: 'ab', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, data: 'a', rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, data: 'not-b', rawFrameCount: 1 }, + ], + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 3, + seqEnd: 3, + data: 'c', + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['c']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-invalid-fail-closed', + })).toBeNull() + expect(bridge.snapshot().perfEvents).toContainEqual(expect.objectContaining({ + event: 'terminal.catchup.surface_quarantined', + terminalId, + reason: 'invalid_terminal_output_batch', + invalidReason: 'segment_data_mismatch', + fromSeq: 1, + toSeq: 2, + })) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + })) + }) + + it('does not checkpoint through the unapplied tail of an invalid terminal.output.batch that overlaps the current cursor', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-invalid-overlap-tail', + serverInstanceId: 'server-output-batch-invalid-overlap-tail', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 10, + data: 'abcdefghij', + }) + }) + + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-invalid-overlap-tail', + })?.parserAppliedSeq).toBe(10) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 9, + seqEnd: 11, + data: 'ijk', + serializedBytes: 256, + segments: [ + { seqStart: 9, seqEnd: 9, endOffset: 1, data: 'i', rawFrameCount: 1 }, + { seqStart: 10, seqEnd: 10, endOffset: 2, data: 'j', rawFrameCount: 1 }, + { seqStart: 11, seqEnd: 11, endOffset: 3, data: 'not-k', rawFrameCount: 1 }, + ], + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 12, + seqEnd: 12, + data: 'l', + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['l']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-invalid-overlap-tail', + })?.parserAppliedSeq).toBe(10) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + })) + }) + + it('splits terminal.output.batch writes around parser barrier segments', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-barrier', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'replay', + seqStart: 1, + seqEnd: 3, + data: 'aBc', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1, barrier: 'control' }, + { seqStart: 3, seqEnd: 3, endOffset: 3, rawFrameCount: 1 }, + ], + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['a', 'B', 'c']) + }) + + it('does not checkpoint a stripped terminal.output.batch BEL segment as parser-applied', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-stripped-bel', + mode: 'codex', + serverInstanceId: 'server-output-batch-stripped-bel', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 2, + data: 'A\x07', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1, barrier: 'turn_complete' }, + ], + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['A']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-stripped-bel', + })?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 1, + })) + }) + + it('completes attach when replay ends in a stripped terminal.output.batch BEL segment without checkpointing it', async () => { + const { terminalId, term, queryByText, store } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-replay-stripped-complete', + mode: 'codex', + serverInstanceId: 'server-output-batch-replay-stripped-complete', + ackInitialAttach: false, + clearSends: false, + }) + act(() => { + store.dispatch(setConnectionStatus('ready')) + }) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + const attachRequestId = attach?.attachRequestId + const streamId = 'stream-output-batch-replay-stripped-complete' + expect(attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId, + }) + }) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'replay', + seqStart: 1, + seqEnd: 1, + data: '\x07', + serializedBytes: 128, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1, barrier: 'turn_complete' }, + ], + }) + }) + + expect(term.write).not.toHaveBeenCalled() + await waitFor(() => { + expect(queryByText('Recovering terminal output...')).toBeNull() + }) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-replay-stripped-complete', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + })) + }) + + it('queues stripped terminal.output.batch replay completion behind earlier replay write callbacks', async () => { + const { terminalId, term, queryByText, store } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-replay-stripped-tail', + mode: 'codex', + serverInstanceId: 'server-output-batch-replay-stripped-tail', + ackInitialAttach: false, + clearSends: false, + }) + act(() => { + store.dispatch(setConnectionStatus('ready')) + }) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + const attachRequestId = attach?.attachRequestId + const streamId = 'stream-output-batch-replay-stripped-tail' + expect(attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId, + headSeq: 2, + replayFromSeq: 1, + replayToSeq: 2, + attachRequestId, + }) + }) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockClear() + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'replay', + seqStart: 1, + seqEnd: 2, + data: 'A\x07', + serializedBytes: 128, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 1, rawFrameCount: 1 }, + { seqStart: 2, seqEnd: 2, endOffset: 2, rawFrameCount: 1, barrier: 'turn_complete' }, + ], + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['A']) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-replay-stripped-tail', + })).toBeNull() + + act(() => { + delayedCallbacks[0]?.callback() + }) + + await waitFor(() => { + expect(queryByText('Recovering terminal output...')).toBeNull() + }) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-replay-stripped-tail', + })?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 1, + })) + }) + + it('does not checkpoint a mixed renderable and stripped terminal.output.batch segment as parser-applied', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-batch-mixed-stripped-bel', + mode: 'codex', + serverInstanceId: 'server-output-batch-mixed-stripped-bel', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 1, + data: 'A\x07', + serializedBytes: 256, + segments: [ + { seqStart: 1, seqEnd: 1, endOffset: 2, rawFrameCount: 1, barrier: 'turn_complete' }, + ], + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['A']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-batch-mixed-stripped-bel', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + })) + }) + + it('does not checkpoint a stripped legacy terminal.output BEL frame as parser-applied', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-legacy-stripped-bel', + mode: 'codex', + serverInstanceId: 'server-output-legacy-stripped-bel', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: '\x07', + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 2, + seqEnd: 2, + data: 'B', + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['B']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-legacy-stripped-bel', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + })) + }) + + it('completes attach when replay ends in a stripped legacy terminal.output BEL frame without checkpointing it', async () => { + const { terminalId, term, queryByText, store } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-legacy-replay-stripped-complete', + mode: 'codex', + serverInstanceId: 'server-output-legacy-replay-stripped-complete', + ackInitialAttach: false, + clearSends: false, + }) + act(() => { + store.dispatch(setConnectionStatus('ready')) + }) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + const attachRequestId = attach?.attachRequestId + const streamId = 'stream-output-legacy-replay-stripped-complete' + expect(attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId, + }) + }) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: '\x07', + }) + }) + + expect(term.write).not.toHaveBeenCalled() + await waitFor(() => { + expect(queryByText('Recovering terminal output...')).toBeNull() + }) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-legacy-replay-stripped-complete', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + })) + }) + + it('queues stripped legacy terminal.output replay completion behind earlier replay write callbacks', async () => { + const { terminalId, term, queryByText, store } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-legacy-replay-stripped-tail', + mode: 'codex', + serverInstanceId: 'server-output-legacy-replay-stripped-tail', + ackInitialAttach: false, + clearSends: false, + }) + act(() => { + store.dispatch(setConnectionStatus('ready')) + }) + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + const attachRequestId = attach?.attachRequestId + const streamId = 'stream-output-legacy-replay-stripped-tail' + expect(attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId, + headSeq: 2, + replayFromSeq: 1, + replayToSeq: 2, + attachRequestId, + }) + }) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockClear() + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: 'A', + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 2, + seqEnd: 2, + data: '\x07', + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['A']) + expect(queryByText('Recovering terminal output...')).not.toBeNull() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-legacy-replay-stripped-tail', + })).toBeNull() + + act(() => { + delayedCallbacks[0]?.callback() + }) + + await waitFor(() => { + expect(queryByText('Recovering terminal output...')).toBeNull() + }) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-legacy-replay-stripped-tail', + })?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 1, + })) + }) + + it('does not checkpoint a mixed renderable and stripped legacy terminal.output frame as parser-applied', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-output-legacy-mixed-stripped-bel', + mode: 'codex', + serverInstanceId: 'server-output-legacy-mixed-stripped-bel', + }) + const attachRequestId = latestAttachRequestIdForTerminal(terminalId) + const streamId = latestStreamIdByTerminal.get(terminalId) + expect(attachRequestId).toBeTruthy() + expect(streamId).toBeTruthy() + + term.write.mockClear() + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data: 'A\x07', + }) + }) + + expect(terminalWriteStrings(term)).toEqual(['A']) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId, + serverInstanceId: 'server-output-legacy-mixed-stripped-bel', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + })) + }) + + it('does not render or checkpoint terminal.output missing stream id after attach-ready', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-missing-output-stream-client', + serverInstanceId: 'server-missing-output-stream', + ackInitialAttach: false, + clearSends: false, + }) + + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-active', + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'MISSING STREAM', + attachRequestId: attach!.attachRequestId, + __preserveMissingStreamId: true, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-active', + seqStart: 2, + seqEnd: 2, + data: 'ACTIVE AFTER MISSING', + attachRequestId: attach!.attachRequestId, + }) + }) + + const writes = term.write.mock.calls.map(([data]) => String(data)).join('') + expect(writes).not.toContain('MISSING STREAM') + expect(writes).toContain('ACTIVE AFTER MISSING') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-active', + serverInstanceId: 'server-missing-output-stream', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('does not render or checkpoint terminal.output.gap missing stream id after attach-ready', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-missing-gap-stream-client', + serverInstanceId: 'server-missing-gap-stream', + ackInitialAttach: false, + clearSends: false, + }) + + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + streamId: 'stream-active', + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output.gap', + terminalId, + fromSeq: 1, + toSeq: 5, + reason: 'queue_overflow', + attachRequestId: attach!.attachRequestId, + __preserveMissingStreamId: true, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + streamId: 'stream-active', + seqStart: 6, + seqEnd: 6, + data: 'ACTIVE AFTER MISSING GAP', + attachRequestId: attach!.attachRequestId, + }) + }) + + const writes = term.write.mock.calls.map(([data]) => String(data)).join('') + expect(writes).not.toContain('Output gap 1-5') + expect(writes).toContain('ACTIVE AFTER MISSING GAP') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-active', + serverInstanceId: 'server-missing-gap-stream', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('keeps the legacy missing-stream output path only before attach-ready establishes stream identity', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-legacy-pre-ready-stream', + ackInitialAttach: false, + clearSends: false, + }) + + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'LEGACY BEFORE READY', + attachRequestId: attach!.attachRequestId, + }) + }) + + const writes = term.write.mock.calls.map(([data]) => String(data)).join('') + expect(writes).toContain('LEGACY BEFORE READY') + }) + + it('clears stale stored stream id when attach-ready omits stream id and rejects untagged output and gaps', async () => { + const { store, tabId, terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-legacy-ready-without-stream', + serverInstanceId: 'server-missing-ready-stream', + streamId: 'stored-stale-stream', + ackInitialAttach: false, + clearSends: false, + }) + + const attach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: attach!.attachRequestId, + __preserveMissingStreamId: true, + } as any) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'UNTAGGED AFTER BAD READY', + attachRequestId: attach!.attachRequestId, + __preserveMissingStreamId: true, + } as any) + messageHandler!({ + type: 'terminal.output.gap', + terminalId, + fromSeq: 2, + toSeq: 3, + reason: 'queue_overflow', + attachRequestId: attach!.attachRequestId, + __preserveMissingStreamId: true, + } as any) + }) + + const layout = store.getState().panes.layouts[tabId] + expect(layout.type).toBe('leaf') + expect(layout.content.kind).toBe('terminal') + expect(layout.content.streamId).toBeUndefined() + + const writes = terminalWriteStrings(term).join('') + expect(writes).not.toContain('UNTAGGED AFTER BAD READY') + expect(writes).not.toContain('Output gap 2-3') + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stored-stale-stream', + serverInstanceId: 'server-missing-ready-stream', + })).toBeNull() + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: null, + serverInstanceId: 'server-missing-ready-stream', + })).toBeNull() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('does not use a null-stream checkpoint for warm delta after attach-ready omits stream id', async () => { + const terminalId = 'term-null-stream-checkpoint' + const serverInstanceId = 'server-null-stream-checkpoint' + saveTerminalSurfaceCheckpoint({ + terminalId, + streamId: null, + serverInstanceId, + surfaceEpoch: 0, + attachRequestId: 'seed-null-stream-checkpoint', + parserAppliedSeq: 17, + cols: 80, + rows: 24, + geometryEpoch: 1, + geometryAuthority: 'single_client', + scrollback: 10000, + xtermVersion: '6.0.0', + bufferType: 'unknown', + parserIdle: true, + }) + expect(loadTerminalSurfaceCheckpoint(terminalId, { + streamId: null, + serverInstanceId, + })?.parserAppliedSeq).toBe(17) + + await renderTerminalHarness({ + status: 'running', + terminalId, + serverInstanceId, + ackInitialAttach: false, + clearSends: false, + }) + + const initialAttach = sentMessages() + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(initialAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + }) + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: initialAttach!.attachRequestId, + __preserveMissingStreamId: true, + } as any) + }) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('ignores xterm title callbacks fired while replay writes are scoped', async () => { + const { terminalId, term, store, tabId, paneId } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-replay-title', + ackInitialAttach: false, + clearSends: false, + }) + + await waitFor(() => { + expect(term.onTitleChange).toHaveBeenCalled() + }) + const titleHandler = term.onTitleChange.mock.calls[0]?.[0] + expect(typeof titleHandler).toBe('function') + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + const attach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'replay title frame', + attachRequestId: attach!.attachRequestId, + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['replay title frame']) + + act(() => { + titleHandler('Replay Title') + }) + + expect(store.getState().tabs.tabs.find((tab) => tab.id === tabId)?.title).toBe('Shell') + expect(store.getState().panes.paneTitles[tabId]?.[paneId]).not.toBe('Replay Title') + + act(() => { + delayedCallbacks[0]?.callback() + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'live title frame', + attachRequestId: attach!.attachRequestId, + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual([ + 'replay title frame', + 'live title frame', + ]) + + act(() => { + titleHandler('Live Title') + }) + + expect(store.getState().tabs.tabs.find((tab) => tab.id === tabId)?.title).toBe('Live Title') + expect(store.getState().panes.paneTitles[tabId]?.[paneId]).toBe('Live Title') + }) + + it('does not let stale write callbacks advance the current parser-applied cursor', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-stale-write-callback', + serverInstanceId: 'server-a', + streamId: 'stream-1', + ackInitialAttach: false, + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: firstAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'first replay text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const initialCheckpoint = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-1', + serverInstanceId: 'server-a', + }) + expect(initialCheckpoint?.attachRequestId).toBe(firstAttach?.attachRequestId) + expect(initialCheckpoint?.parserAppliedSeq).toBe(1) + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'old replay text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + expect(delayedCallbacks).toHaveLength(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const secondAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(secondAttach?.attachRequestId).toBeTruthy() + expect(secondAttach?.attachRequestId).not.toBe(firstAttach?.attachRequestId) + expect(secondAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + }) + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 4, + replayFromSeq: 2, + replayToSeq: 4, + attachRequestId: secondAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 4, + data: 'current replay text', + attachRequestId: secondAttach!.attachRequestId, + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['old replay text']) + + act(() => { + delayedCallbacks.find(({ data }) => data === 'old replay text')?.callback() + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual([ + 'old replay text', + 'current replay text', + ]) + + const checkpointAfterStaleCallback = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-1', + serverInstanceId: 'server-a', + }) + expect(checkpointAfterStaleCallback).not.toBeNull() + expect(checkpointAfterStaleCallback?.attachRequestId).toBe(firstAttach?.attachRequestId) + expect(checkpointAfterStaleCallback?.parserAppliedSeq).toBe(1) + + const currentAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(currentAttach?.attachRequestId).toBe(secondAttach?.attachRequestId) + + wsMocks.send.mockClear() + term.clear.mockClear() + act(() => { + delayedCallbacks.find(({ data }) => data === 'current replay text')?.callback() + }) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + expect(term.clear).toHaveBeenCalledTimes(1) + + const checkpointAfterCurrentCallback = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-1', + serverInstanceId: 'server-a', + }) + expect(checkpointAfterCurrentCallback?.attachRequestId).toBe(firstAttach?.attachRequestId) + expect(checkpointAfterCurrentCallback?.parserAppliedSeq).toBe(1) + }) + + it('fails closed from delta attach when writes are in flight and repairs quarantine after drain', async () => { + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-in-flight-delta', + serverInstanceId: 'server-a', + streamId: 'stream-delta', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'checkpointed text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const initialCheckpoint = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-delta', + serverInstanceId: 'server-a', + }) + expect(initialCheckpoint?.attachRequestId).toBe(firstAttach?.attachRequestId) + expect(initialCheckpoint?.parserAppliedSeq).toBe(1) + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'old in-flight delta text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks).toHaveLength(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const secondAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(secondAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + }) + const fallbackEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.full_hydrate_fallback') + expect(fallbackEvents).toEqual([ + expect.objectContaining({ + event: 'terminal.catchup.full_hydrate_fallback', + timestamp: expect.any(Number), + terminalId, + attachRequestId: secondAttach!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + hasInFlightWrites: true, + }), + ]) + const quarantineEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.surface_quarantined') + expect(quarantineEvents).toEqual([ + expect.objectContaining({ + event: 'terminal.catchup.surface_quarantined', + timestamp: expect.any(Number), + terminalId, + attachRequestId: secondAttach!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + }), + ]) + expect(bridge.snapshot().milestones['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().metadata['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().milestones['terminal.catchup.surface_quarantined']).toBeUndefined() + expect(bridge.snapshot().metadata['terminal.catchup.surface_quarantined']).toBeUndefined() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 3, + replayFromSeq: 2, + replayToSeq: 3, + attachRequestId: secondAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 3, + data: 'quarantined replay text', + attachRequestId: secondAttach!.attachRequestId, + }) + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['old in-flight delta text']) + + act(() => { + delayedCallbacks.find(({ data }) => data === 'old in-flight delta text')?.callback() + }) + + expect(delayedCallbacks.map(({ data }) => data)).toEqual([ + 'old in-flight delta text', + 'quarantined replay text', + ]) + + wsMocks.send.mockClear() + term.clear.mockClear() + act(() => { + delayedCallbacks.find(({ data }) => data === 'quarantined replay text')?.callback() + }) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + expect(term.clear).toHaveBeenCalledTimes(1) + + const checkpointAfterCallbacks = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-delta', + serverInstanceId: 'server-a', + }) + expect(checkpointAfterCallbacks?.attachRequestId).toBe(firstAttach?.attachRequestId) + expect(checkpointAfterCallbacks?.parserAppliedSeq).toBe(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('drops quarantined replay and forces a clearing hydrate when quarantine repair times out before writes drain', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-in-flight-quarantine-timeout', + serverInstanceId: 'server-a', + streamId: 'stream-timeout', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'checkpointed text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const delayedCallbacks: Array<{ data: string; callback: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push({ data, callback: onWritten }) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'old in-flight timeout text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['old in-flight timeout text']) + + vi.useFakeTimers() + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const quarantinedAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(quarantinedAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + }) + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 3, + replayFromSeq: 2, + replayToSeq: 3, + attachRequestId: quarantinedAttach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 3, + data: 'quarantined replay timeout text', + attachRequestId: quarantinedAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks.map(({ data }) => data)).toEqual(['old in-flight timeout text']) + + act(() => { + vi.advanceTimersByTime(2_100) + }) + + wsMocks.send.mockClear() + term.clear.mockClear() + act(() => { + delayedCallbacks.find(({ data }) => data === 'old in-flight timeout text')?.callback() + }) + + expect(terminalWriteStrings(term)).not.toContain('quarantined replay timeout text') + + act(() => { + vi.advanceTimersByTime(100) + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + expect(term.clear).toHaveBeenCalledTimes(1) + const repairAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(repairAttach?.attachRequestId).not.toBe(quarantinedAttach?.attachRequestId) + }) + + it('records repeated in-flight full-hydrate fallback and quarantine audit events separately', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-repeat-fallback-quarantine', + serverInstanceId: 'server-a', + streamId: 'stream-repeat', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'checkpoint before repeat fallback', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const delayedCallbacks: Array<() => void> = [] + term.write.mockImplementation((_data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push(onWritten) + }) + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'held in-flight write', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks).toHaveLength(1) + + const bridge = createPerfAuditBridge() + installPerfAuditBridge(bridge) + let now = 200 + const performanceNowSpy = vi.spyOn(performance, 'now').mockImplementation(() => { + now += 0.01 + return now + }) + try { + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + reconnectHandler?.() + }) + + const reconnectAttaches = sentMessages() + .filter((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(reconnectAttaches).toHaveLength(2) + expect(reconnectAttaches[0]?.attachRequestId).not.toBe(reconnectAttaches[1]?.attachRequestId) + + const fallbackEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.full_hydrate_fallback') + expect(fallbackEvents).toHaveLength(2) + expect(fallbackEvents[0]).toEqual(expect.objectContaining({ + event: 'terminal.catchup.full_hydrate_fallback', + timestamp: expect.any(Number), + terminalId, + attachRequestId: reconnectAttaches[0]!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + hasInFlightWrites: true, + })) + expect(fallbackEvents[1]).toEqual(expect.objectContaining({ + event: 'terminal.catchup.full_hydrate_fallback', + timestamp: expect.any(Number), + terminalId, + attachRequestId: reconnectAttaches[1]!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + hasInFlightWrites: true, + })) + expect(Number(fallbackEvents[0]!.timestamp)).toBeLessThan(Number(fallbackEvents[1]!.timestamp)) + + const quarantineEvents = bridge.snapshot().perfEvents + .filter((event) => event.event === 'terminal.catchup.surface_quarantined') + expect(quarantineEvents).toHaveLength(2) + expect(quarantineEvents[0]).toEqual(expect.objectContaining({ + event: 'terminal.catchup.surface_quarantined', + timestamp: expect.any(Number), + terminalId, + attachRequestId: reconnectAttaches[0]!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + })) + expect(quarantineEvents[1]).toEqual(expect.objectContaining({ + event: 'terminal.catchup.surface_quarantined', + timestamp: expect.any(Number), + terminalId, + attachRequestId: reconnectAttaches[1]!.attachRequestId, + requestedIntent: 'transport_reconnect', + intent: 'viewport_hydrate', + reason: 'in_flight_writes', + })) + expect(Number(quarantineEvents[0]!.timestamp)).toBeLessThan(Number(quarantineEvents[1]!.timestamp)) + + expect(bridge.snapshot().metadata['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().milestones['terminal.catchup.full_hydrate_fallback']).toBeUndefined() + expect(bridge.snapshot().metadata['terminal.catchup.surface_quarantined']).toBeUndefined() + expect(bridge.snapshot().milestones['terminal.catchup.surface_quarantined']).toBeUndefined() + } finally { + performanceNowSpy.mockRestore() + } + }) + + it('does not clear the old surface when full hydrate starts with in-flight writes', async () => { + const { store, tabId, paneId, terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-in-flight-full-hydrate', + serverInstanceId: 'server-a', + streamId: 'stream-in-flight', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'trusted text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const trustedCheckpoint = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-in-flight', + serverInstanceId: 'server-a', + }) + expect(trustedCheckpoint?.parserAppliedSeq).toBe(1) + + const delayedCallbacks: Array<() => void> = [] + term.write.mockImplementation((_data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push(onWritten) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'in-flight text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks).toHaveLength(1) + + term.clear.mockClear() + wsMocks.send.mockClear() + act(() => { + store.dispatch(requestPaneRefresh({ tabId, paneId })) + }) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + expect(term.clear).not.toHaveBeenCalled() + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('cancels quarantined repair after invalid-terminal replacement before writes drain', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-quarantine-invalid', + serverInstanceId: 'server-a', + streamId: 'stream-invalid', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'trusted text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const delayedCallbacks: Array<() => void> = [] + term.write.mockImplementation((_data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push(onWritten) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'old in-flight text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks).toHaveLength(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const quarantineAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(quarantineAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + }) + + act(() => { + messageHandler!({ + type: 'error', + code: 'INVALID_TERMINAL_ID', + terminalId, + message: 'gone', + }) + }) + + wsMocks.send.mockClear() + await act(async () => { + delayedCallbacks.forEach((callback) => callback()) + await new Promise((resolve) => setTimeout(resolve, 50)) + }) + + expect(wsMocks.send.mock.calls + .map(([msg]) => msg) + .filter((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId)).toHaveLength(0) + }) + + it('handles tagged invalid-terminal errors from quarantine repair attaches', async () => { + const { terminalId, term, store, tabId, requestId } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-quarantine-repair-invalid', + serverInstanceId: 'server-a', + streamId: 'stream-repair-invalid', + clearSends: false, + }) + + const firstAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(firstAttach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'trusted text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + + const delayedCallbacks: Array<() => void> = [] + term.write.mockImplementation((_data: string, onWritten?: () => void) => { + if (onWritten) delayedCallbacks.push(onWritten) + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: 'old in-flight text', + attachRequestId: firstAttach!.attachRequestId, + }) + }) + expect(delayedCallbacks).toHaveLength(1) + + wsMocks.send.mockClear() + act(() => { + reconnectHandler?.() + }) + + const quarantineAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(quarantineAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + }) + expect(quarantineAttach?.attachRequestId).toBeTruthy() + + wsMocks.send.mockClear() + await act(async () => { + delayedCallbacks.forEach((callback) => callback()) + await new Promise((resolve) => setTimeout(resolve, 50)) + }) + + const repairAttach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(repairAttach).toMatchObject({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, + }) + expect(repairAttach?.attachRequestId).toBeTruthy() + expect(repairAttach?.attachRequestId).not.toBe(quarantineAttach?.attachRequestId) + + act(() => { + messageHandler!({ + type: 'error', + code: 'INVALID_TERMINAL_ID', + terminalId, + requestId: repairAttach!.attachRequestId, + message: 'Terminal not running', + }) + }) + + await waitFor(() => { + const layout = store.getState().panes.layouts[tabId] as { type: 'leaf'; content: any } + expect(layout.content.terminalId).toBeUndefined() + expect(layout.content.status).toBe('creating') + expect(layout.content.createRequestId).not.toBe(requestId) + }) + + const layout = store.getState().panes.layouts[tabId] as { type: 'leaf'; content: any } + expect(restoreMocks.addTerminalFreshRecoveryRequestId).toHaveBeenCalledWith( + layout.content.createRequestId, + 'fresh_after_restore_unavailable', + ) + + wsMocks.send.mockClear() + await act(async () => { + await new Promise((resolve) => setTimeout(resolve, 50)) + }) + + expect(wsMocks.send.mock.calls + .map(([msg]) => msg) + .filter((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId)).toHaveLength(0) + }) + + it('keeps queued viewport_hydrate intent when reconnect fires before the first hidden attach completes', async () => { + const { requestId, rerender, store, tabId, paneId } = await renderTerminalHarness({ + status: 'creating', + hidden: true, + requestId: 'req-v2-hidden-created-before-reconnect', + }) + + wsMocks.send.mockClear() + messageHandler!({ + type: 'terminal.created', + requestId, + terminalId: 'term-hidden-created-before-reconnect', + createdAt: Date.now(), + }) + + reconnectHandler?.() + + rerender( + + , + ) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId: 'term-hidden-created-before-reconnect', + sinceSeq: 0, + cols: expect.any(Number), + rows: expect.any(Number), + })) + }) + }) + + it('uses the highest rendered sequence in reconnect attach requests', async () => { + const { terminalId, term } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-reconnect' }) + + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 2, data: 'ab' }) + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 3, seqEnd: 3, data: 'c' }) + + const writes = term.write.mock.calls.map(([data]: [string]) => data) + expect(writes).toContain('ab') + expect(writes).toContain('c') + + wsMocks.send.mockClear() + reconnectHandler?.() + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 3, + attachRequestId: expect.any(String), + })) + }) + + it('reattaches with latest rendered sequence after terminal view remount', async () => { + const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-remount' }) + + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) + unmount() + wsMocks.send.mockClear() + + render( + + + + ) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + }) + + it('does not attach a remounted hidden pane until it becomes visible', async () => { + const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-hidden-remount' }) + + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) + unmount() + wsMocks.send.mockClear() + + render( + + + ) + + await waitFor(() => { + expect(messageHandler).not.toBeNull() + }) + expect(wsMocks.send).not.toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + })) + }) + + it('performs one deferred viewport hydration attach when a remounted hidden pane becomes visible', async () => { + const { store, tabId, paneId, terminalId, unmount } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-deferred-hydrate' }) + + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) + unmount() + wsMocks.send.mockClear() + + const view = render( + + + ) + + await waitFor(() => { + expect(messageHandler).not.toBeNull() + }) + expect(wsMocks.send).not.toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + })) + + wsMocks.send.mockClear() + view.rerender( + + + ) + + await waitFor(() => { + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + sinceSeq: 0, + intent: 'viewport_hydrate', + attachRequestId: expect.any(String), + })) + }) + expect(wsMocks.send.mock.calls + .map(([msg]) => msg) + .filter((msg) => msg?.type === 'terminal.resize' && msg?.terminalId === terminalId)).toHaveLength(0) + }) + + it('uses keepalive_delta when a live terminal re-runs the attach effect above the rendered high-water mark', async () => { + const { rerender, store, tabId, paneId, terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-v2-keepalive-intent', + clearSends: false, + }) + + const initialAttachRequestId = latestAttachRequestIdForTerminal(terminalId) + expect(initialAttachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 0, + replayFromSeq: 1, + replayToSeq: 0, + attachRequestId: initialAttachRequestId, + }) + messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 3, data: 'abc' }) + }) + + const writes = term.write.mock.calls.map(([data]: [string]) => data) + expect(writes).toContain('abc') + + wsMocks.send.mockClear() + + const readPaneContent = () => { + const layout = store.getState().panes.layouts[tabId] + return layout && layout.type === 'leaf' && layout.content.kind === 'terminal' ? layout.content : null + } - const readPaneContent = () => { - const layout = store.getState().panes.layouts[tabId] - return layout && layout.type === 'leaf' && layout.content.kind === 'terminal' ? layout.content : null - } - await act(async () => { rerender( @@ -4364,7 +7197,7 @@ describe('TerminalView lifecycle updates', () => { .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) expect(attach?.attachRequestId).toBeTruthy() - term.writeln.mockClear() + term.write.mockClear() messageHandler!({ type: 'terminal.output.gap', terminalId, @@ -4374,14 +7207,15 @@ describe('TerminalView lifecycle updates', () => { attachRequestId: attach!.attachRequestId, } as any) - expect(term.writeln).toHaveBeenCalledWith(expect.stringContaining('Output gap 1-50: reconnect window exceeded')) + expectTerminalWriteContaining(term, 'Output gap 1-50: reconnect window exceeded') wsMocks.send.mockClear() reconnectHandler?.() expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ type: 'terminal.attach', terminalId, - sinceSeq: 50, + intent: 'viewport_hydrate', + sinceSeq: 0, attachRequestId: expect.any(String), })) }) @@ -4906,11 +7740,11 @@ describe('TerminalView lifecycle updates', () => { expect(__getLastSentViewportCacheSizeForTests()).toBe(200) }) - it('renders terminal.output.gap marker and advances sinceSeq for subsequent attach', async () => { + it('renders terminal.output.gap marker and fails closed for subsequent attach', async () => { const { terminalId, term } = await renderTerminalHarness({ status: 'running', terminalId: 'term-v2-gap' }) messageHandler!({ type: 'terminal.output', terminalId, seqStart: 1, seqEnd: 1, data: 'ok' }) - term.writeln.mockClear() + term.write.mockClear() wsMocks.send.mockClear() messageHandler!({ @@ -4921,13 +7755,167 @@ describe('TerminalView lifecycle updates', () => { reason: 'queue_overflow', }) - expect(term.writeln).toHaveBeenCalledWith(expect.stringContaining('Output gap 2-5: slow link backlog')) + expectTerminalWriteContaining(term, 'Output gap 2-5: slow link backlog') reconnectHandler?.() expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ type: 'terminal.attach', terminalId, - sinceSeq: 5, + intent: 'viewport_hydrate', + sinceSeq: 0, + attachRequestId: expect.any(String), + })) + }) + + it('queues local gap notices behind a pending replay write', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-v2-gap-notice-queued', + ackInitialAttach: false, + clearSends: false, + }) + + const submittedWrites: Array<{ data: string; onWritten?: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + submittedWrites.push({ data, onWritten }) + }) + + const attach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'REPLAY', + attachRequestId: attach!.attachRequestId, + }) + }) + + expect(submittedWrites.map((entry) => entry.data)).toEqual(['REPLAY']) + + act(() => { + messageHandler!({ + type: 'terminal.output.gap', + terminalId, + fromSeq: 2, + toSeq: 5, + reason: 'queue_overflow', + attachRequestId: attach!.attachRequestId, + }) + }) + + expect(term.writeln).not.toHaveBeenCalled() + expect(submittedWrites.map((entry) => entry.data)).toEqual(['REPLAY']) + + act(() => { + submittedWrites[0].onWritten?.() + }) + + await waitFor(() => { + expect(submittedWrites.map((entry) => entry.data)).toContainEqual( + expect.stringContaining('Output gap 2-5: slow link backlog'), + ) + }) + }) + + it('invalidates warm delta eligibility only after a queued local notice applies', async () => { + const { terminalId, term } = await renderTerminalHarness({ + status: 'running', + terminalId: 'term-v2-local-notice-invalidates', + mode: 'codex', + serverInstanceId: 'server-local-notice', + streamId: 'stream-local-notice', + ackInitialAttach: false, + clearSends: false, + }) + + const submittedWrites: Array<{ data: string; onWritten?: () => void }> = [] + term.write.mockImplementation((data: string, onWritten?: () => void) => { + submittedWrites.push({ data, onWritten }) + }) + + const attach = wsMocks.send.mock.calls + .map(([msg]) => msg) + .find((msg) => msg?.type === 'terminal.attach' && msg?.terminalId === terminalId) + expect(attach?.attachRequestId).toBeTruthy() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 1, + replayFromSeq: 1, + replayToSeq: 1, + attachRequestId: attach!.attachRequestId, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'REPLAY', + attachRequestId: attach!.attachRequestId, + }) + }) + + expect(submittedWrites.map((entry) => entry.data)).toEqual(['REPLAY']) + + act(() => { + messageHandler!({ + type: 'terminal.input.blocked', + terminalId, + reason: 'codex_identity_pending', + }) + }) + + expect(submittedWrites.map((entry) => entry.data)).toEqual(['REPLAY']) + + act(() => { + submittedWrites[0].onWritten?.() + }) + + await waitFor(() => { + expect(submittedWrites.map((entry) => entry.data)).toContainEqual( + expect.stringContaining('Input not sent: Codex is still saving restore state.'), + ) + }) + + const checkpointAfterReplay = loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-local-notice', + serverInstanceId: 'server-local-notice', + }) + expect(checkpointAfterReplay?.attachRequestId).toBe(attach?.attachRequestId) + expect(checkpointAfterReplay?.parserAppliedSeq).toBe(1) + + const noticeWrite = submittedWrites.find((entry) => entry.data.includes('Input not sent')) + expect(noticeWrite?.onWritten).toBeTypeOf('function') + + wsMocks.send.mockClear() + act(() => { + noticeWrite?.onWritten?.() + }) + act(() => { + reconnectHandler?.() + }) + + expect(wsMocks.send).toHaveBeenCalledWith(expect.objectContaining({ + type: 'terminal.attach', + terminalId, + intent: 'viewport_hydrate', + sinceSeq: 0, attachRequestId: expect.any(String), })) }) @@ -4979,7 +7967,7 @@ describe('TerminalView lifecycle updates', () => { messageHandler!({ type: 'terminal.output', terminalId, seqStart: 13, seqEnd: 13, data: 'LIVE' }) }) - expect(term.writeln).toHaveBeenCalledWith(expect.stringContaining('Output gap 1-8: reconnect window exceeded')) + expectTerminalWriteContaining(term, 'Output gap 1-8: reconnect window exceeded') const writes = term.write.mock.calls.map(([data]: [string]) => String(data)).join('') expect(writes).toContain('TAIL') expect(writes).toContain('LIVE') diff --git a/test/unit/client/components/TerminalView.osc52.test.tsx b/test/unit/client/components/TerminalView.osc52.test.tsx index 5e89300b5..81f0e93fc 100644 --- a/test/unit/client/components/TerminalView.osc52.test.tsx +++ b/test/unit/client/components/TerminalView.osc52.test.tsx @@ -85,18 +85,34 @@ vi.mock('lucide-react', () => ({ const terminalInstances: any[] = [] const latestAttachRequestIdByTerminal = new Map() +const latestStreamIdByTerminal = new Map() const ioEvents: Array<{ kind: 'send' | 'write', type?: string, data: string }> = [] function withCurrentAttachRequestId(msg: any) { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } - const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + let next = msg + if (!next.attachRequestId) { + const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) + if (attachRequestId) next = { ...next, attachRequestId } + } + if (msg.type === 'terminal.attach.ready') { + const streamId = typeof next.streamId === 'string' && next.streamId.length > 0 + ? next.streamId + : `test-stream:${msg.terminalId}` + latestStreamIdByTerminal.set(msg.terminalId, streamId) + next = { ...next, streamId } + } else { + const streamId = typeof next.streamId === 'string' && next.streamId.length > 0 + ? next.streamId + : latestStreamIdByTerminal.get(msg.terminalId) + if (streamId) next = { ...next, streamId } + } + return next } vi.mock('@xterm/xterm', () => { @@ -217,6 +233,7 @@ describe('TerminalView OSC52 policy handling', () => { beforeEach(() => { terminalInstances.length = 0 latestAttachRequestIdByTerminal.clear() + latestStreamIdByTerminal.clear() ioEvents.length = 0 wsMocks.send.mockClear() wsMocks.send.mockImplementation((msg: any) => { @@ -404,6 +421,59 @@ describe('TerminalView OSC52 policy handling', () => { expect(screen.queryByRole('dialog', { name: 'Clipboard access request' })).not.toBeInTheDocument() }) + it('allows live startup replies and OSC52 writes while an earlier replay write callback is still pending', async () => { + const { terminalId } = await renderView('always') + clipboardMocks.copyText.mockClear() + wsMocks.send.mockClear() + + act(() => { + messageHandler!({ + type: 'terminal.attach.ready', + terminalId, + headSeq: 2, + replayFromSeq: 1, + replayToSeq: 1, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 1, + seqEnd: 1, + data: 'pending replay write', + }) + }) + + await waitFor(() => { + expect(writeEvents().map((event) => event.data)).toContain('pending replay write') + }) + + act(() => { + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 2, + seqEnd: 2, + data: OPEN_CODE_STARTUP_PROBE_FRAME, + }) + messageHandler!({ + type: 'terminal.output', + terminalId, + seqStart: 3, + seqEnd: 3, + data: `live${OSC52_COPY}`, + }) + }) + + expect(wsMocks.send.mock.calls.map(([msg]) => msg).filter((msg) => msg?.type === 'terminal.input')).toEqual( + OPEN_CODE_STARTUP_EXPECTED_REPLIES.map((data) => ({ + type: 'terminal.input', + terminalId, + data, + })), + ) + expect(clipboardMocks.copyText).toHaveBeenCalledWith('copy') + }) + it('ask + Yes copies once and keeps ask policy', async () => { const { store, terminalId } = await renderView('ask') act(() => { diff --git a/test/unit/client/components/TerminalView.renderer.test.tsx b/test/unit/client/components/TerminalView.renderer.test.tsx index ce26f6373..62447c993 100644 --- a/test/unit/client/components/TerminalView.renderer.test.tsx +++ b/test/unit/client/components/TerminalView.renderer.test.tsx @@ -92,17 +92,33 @@ vi.mock('@/components/terminal/terminal-runtime', () => ({ const terminalInstances: any[] = [] let messageHandler: ((msg: any) => void) | null = null const latestAttachRequestIdByTerminal = new Map() +const latestStreamIdByTerminal = new Map() function withCurrentAttachRequestId(msg: any) { if ( - msg?.attachRequestId - || typeof msg?.terminalId !== 'string' + typeof msg?.terminalId !== 'string' || (msg?.type !== 'terminal.attach.ready' && msg?.type !== 'terminal.output' && msg?.type !== 'terminal.output.gap') ) { return msg } - const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) - return attachRequestId ? { ...msg, attachRequestId } : msg + let next = msg + if (!next.attachRequestId) { + const attachRequestId = latestAttachRequestIdByTerminal.get(msg.terminalId) + if (attachRequestId) next = { ...next, attachRequestId } + } + if (msg.type === 'terminal.attach.ready') { + const streamId = typeof next.streamId === 'string' && next.streamId.length > 0 + ? next.streamId + : `test-stream:${msg.terminalId}` + latestStreamIdByTerminal.set(msg.terminalId, streamId) + next = { ...next, streamId } + } else { + const streamId = typeof next.streamId === 'string' && next.streamId.length > 0 + ? next.streamId + : latestStreamIdByTerminal.get(msg.terminalId) + if (streamId) next = { ...next, streamId } + } + return next } vi.mock('@xterm/xterm', () => { @@ -218,6 +234,7 @@ describe('TerminalView renderer mode', () => { beforeEach(() => { terminalInstances.length = 0 latestAttachRequestIdByTerminal.clear() + latestStreamIdByTerminal.clear() runtimeMockState.throwOnAttach = false runtimeMockState.lastEnableWebgl = null runtimeMockState.lastRuntime = null diff --git a/test/unit/client/components/TerminalView.titleFreeze.test.tsx b/test/unit/client/components/TerminalView.titleFreeze.test.tsx index 2152b3895..e6efd792c 100644 --- a/test/unit/client/components/TerminalView.titleFreeze.test.tsx +++ b/test/unit/client/components/TerminalView.titleFreeze.test.tsx @@ -126,7 +126,7 @@ function createStore(mode: TerminalPaneContent['mode']) { } async function renderAndGetTitleCb(store: ReturnType['store'], paneContent: TerminalPaneContent, tabId: string, paneId: string) { - render( + const view = render( , @@ -135,7 +135,7 @@ async function renderAndGetTitleCb(store: ReturnType['store' expect(terminalInstances.length).toBeGreaterThan(0) expect(terminalInstances[terminalInstances.length - 1].titleCb).toBeTypeOf('function') }) - return terminalInstances[terminalInstances.length - 1].titleCb! + return { titleCb: terminalInstances[terminalInstances.length - 1].titleCb!, view } } describe('TerminalView OSC title scope', () => { @@ -153,7 +153,7 @@ describe('TerminalView OSC title scope', () => { it('a shell terminal still follows OSC titles (program tracking)', async () => { const { store, paneContent, tabId, paneId } = createStore('shell') - const fire = await renderAndGetTitleCb(store, paneContent, tabId, paneId) + const { titleCb: fire } = await renderAndGetTitleCb(store, paneContent, tabId, paneId) act(() => fire('vim README.md')) @@ -163,11 +163,41 @@ describe('TerminalView OSC title scope', () => { it('a coding-agent (claude) terminal ignores OSC titles (stays its working-dir name)', async () => { const { store, paneContent, tabId, paneId } = createStore('claude') - const fire = await renderAndGetTitleCb(store, paneContent, tabId, paneId) + const { titleCb: fire } = await renderAndGetTitleCb(store, paneContent, tabId, paneId) act(() => fire('Building project...')) expect(store.getState().tabs.tabs[0].title).toBe('freshell') expect(store.getState().panes.paneTitles[tabId][paneId]).toBe('freshell') }) + + it('ignores a stale title callback after its terminal instance is disposed', async () => { + const { store, paneContent, tabId, paneId } = createStore('shell') + const { titleCb: staleFire, view } = await renderAndGetTitleCb(store, paneContent, tabId, paneId) + + view.unmount() + + const current = render( + + + , + ) + await waitFor(() => { + expect(terminalInstances.length).toBeGreaterThan(1) + expect(terminalInstances[terminalInstances.length - 1].titleCb).toBeTypeOf('function') + }) + const currentFire = terminalInstances[terminalInstances.length - 1].titleCb! + + act(() => staleFire('stale old title')) + + expect(store.getState().tabs.tabs[0].title).toBe('freshell') + expect(store.getState().panes.paneTitles[tabId][paneId]).toBe('freshell') + + act(() => currentFire('current title')) + + expect(store.getState().tabs.tabs[0].title).toBe('current title') + expect(store.getState().panes.paneTitles[tabId][paneId]).toBe('current title') + + current.unmount() + }) }) diff --git a/test/unit/client/components/TerminalView.urlClick.test.tsx b/test/unit/client/components/TerminalView.urlClick.test.tsx index 5803c0c7d..e49b9bf23 100644 --- a/test/unit/client/components/TerminalView.urlClick.test.tsx +++ b/test/unit/client/components/TerminalView.urlClick.test.tsx @@ -214,6 +214,56 @@ describe('TerminalView URL click behavior', () => { expect(windowOpenSpy).not.toHaveBeenCalled() }) + it('ignores stale OSC 8 linkHandler callbacks after terminal instance replacement', async () => { + const store = createStore({ terminal: { ...defaultSettings.terminal, warnExternalLinks: false } }) + + const first = render( + + + ) + + await waitFor(() => { + expect(terminalInstances).toHaveLength(1) + }) + + const staleHandler = getLinkHandler() + first.unmount() + + const current = render( + + + ) + + await waitFor(() => { + expect(terminalInstances).toHaveLength(2) + }) + + const mockRange = { start: { x: 1, y: 1 }, end: { x: 20, y: 1 } } + act(() => { + staleHandler.hover?.(new MouseEvent('mouseover'), 'https://stale.example.com', mockRange) + staleHandler.activate(new MouseEvent('click'), 'https://stale.example.com') + staleHandler.leave?.(new MouseEvent('mouseout'), 'https://stale.example.com', mockRange) + }) + await act(async () => { + await Promise.resolve() + }) + + expect(getHoveredUrl('pane-1')).toBeUndefined() + expect(store.getState().panes.layouts['tab-1'].type).toBe('leaf') + + const currentHandler = getLinkHandler() + act(() => { + currentHandler.activate(new MouseEvent('click'), 'https://current.example.com') + }) + await waitFor(() => { + expect(store.getState().panes.layouts['tab-1'].type).toBe('split') + }) + + current.unmount() + }) + it('OSC 8 linkHandler.activate with warnExternalLinks=true shows modal, confirm opens browser pane', async () => { const store = createStore() diff --git a/test/unit/client/components/terminal/terminal-write-queue.test.ts b/test/unit/client/components/terminal/terminal-write-queue.test.ts index 99045c946..6c91da3c9 100644 --- a/test/unit/client/components/terminal/terminal-write-queue.test.ts +++ b/test/unit/client/components/terminal/terminal-write-queue.test.ts @@ -1,5 +1,9 @@ import { describe, it, expect, vi } from 'vitest' import { createTerminalWriteQueue } from '@/components/terminal/terminal-write-queue' +import { + getTerminalOutputWriteScope, + shouldAllowTerminalOutputSideEffect, +} from '@/lib/terminal-output-write-scope' describe('createTerminalWriteQueue', () => { it('processes queued writes in time slices and preserves order', () => { @@ -8,9 +12,11 @@ describe('createTerminalWriteQueue', () => { let nowMs = 0 const queue = createTerminalWriteQueue({ - write: (chunk) => { + terminalInstanceId: 'surface-timeslice', + write: (chunk, onWritten) => { writes.push(chunk) nowMs += 5 + onWritten?.() }, requestFrame: (cb) => { rafCallbacks.push(cb) @@ -21,9 +27,9 @@ describe('createTerminalWriteQueue', () => { budgetMs: 4, }) - queue.enqueue('A') - queue.enqueue('B') - queue.enqueue('C') + queue.enqueue('A', undefined, { coalesce: false }) + queue.enqueue('B', undefined, { coalesce: false }) + queue.enqueue('C', undefined, { coalesce: false }) expect(writes).toEqual([]) @@ -43,6 +49,7 @@ describe('createTerminalWriteQueue', () => { const write = vi.fn() const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-clear', write, requestFrame: (cb) => { rafCallbacks.push(cb) @@ -51,8 +58,8 @@ describe('createTerminalWriteQueue', () => { cancelFrame, }) - queue.enqueue('A') - queue.enqueue('B') + queue.enqueue('A', undefined, { coalesce: false }) + queue.enqueue('B', undefined, { coalesce: false }) queue.clear() expect(cancelFrame).toHaveBeenCalledTimes(1) @@ -65,9 +72,11 @@ describe('createTerminalWriteQueue', () => { let nowMs = 0 const queue = createTerminalWriteQueue({ - write: (chunk) => { + terminalInstanceId: 'surface-continuation', + write: (chunk, onWritten) => { writes.push(chunk) nowMs += 5 + onWritten?.() }, requestFrame: (cb) => { rafCallbacks.push(cb) @@ -78,8 +87,8 @@ describe('createTerminalWriteQueue', () => { budgetMs: 4, }) - queue.enqueue('A') - queue.enqueue('B') + queue.enqueue('A', undefined, { coalesce: false }) + queue.enqueue('B', undefined, { coalesce: false }) expect(rafCallbacks).toHaveLength(1) @@ -87,7 +96,7 @@ describe('createTerminalWriteQueue', () => { expect(writes).toEqual(['A']) expect(rafCallbacks).toHaveLength(1) - queue.enqueue('C') + queue.enqueue('C', undefined, { coalesce: false }) expect(rafCallbacks).toHaveLength(1) rafCallbacks.shift()?.(32) @@ -105,6 +114,7 @@ describe('createTerminalWriteQueue', () => { const rafCallbacks: FrameRequestCallback[] = [] const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-coalesce', write: (chunk, onWritten) => { writes.push(chunk) onWritten?.() @@ -126,13 +136,160 @@ describe('createTerminalWriteQueue', () => { expect(callbacks).toEqual(['A', 'B']) }) + it('coalesces adjacent live writes and preserves write callbacks', () => { + const writes: string[] = [] + const callbacks: string[] = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-live-coalesce', + write: (chunk, onWritten) => { + writes.push(chunk) + onWritten?.() + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.enqueue('A', () => callbacks.push('A'), { mode: 'live' }) + queue.enqueue('B', () => callbacks.push('B'), { mode: 'live' }) + queue.enqueue('C', undefined, { mode: 'live' }) + + rafCallbacks.shift()?.(16) + + expect(writes).toEqual(['ABC']) + expect(callbacks).toEqual(['A', 'B']) + }) + + it('does not coalesce across explicit output barriers', () => { + const writes: string[] = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-live-barriers', + write: (chunk, onWritten) => { + writes.push(chunk) + onWritten?.() + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.enqueue('A', undefined, { mode: 'live' }) + queue.enqueue('B', undefined, { mode: 'live', coalesce: false }) + queue.enqueue('C', undefined, { mode: 'live' }) + + rafCallbacks.shift()?.(16) + + expect(writes).toEqual(['A', 'B', 'C']) + }) + + it('keeps a four-hour hidden-tab live backlog bounded to large coalesced writes', () => { + const writes: string[] = [] + const callbacks: number[] = [] + const rafCallbacks: FrameRequestCallback[] = [] + const line = `${'B'.repeat(1023)}\n` + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-live-four-hour-backlog', + write: (chunk, onWritten) => { + writes.push(chunk) + onWritten?.() + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + for (let index = 0; index < 14_400; index += 1) { + queue.enqueue(line, () => callbacks.push(index), { mode: 'live' }) + } + + rafCallbacks.shift()?.(16) + + expect(writes.length).toBeLessThanOrEqual(57) + expect(writes.reduce((total, write) => total + write.length, 0)).toBe(line.length * 14_400) + expect(callbacks).toHaveLength(14_400) + expect(callbacks[0]).toBe(0) + expect(callbacks.at(-1)).toBe(14_399) + }) + + it('drops queued writes from stale generations before they reach xterm', () => { + const writes: string[] = [] + const callbacks: string[] = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-stale-queued', + write: (chunk, onWritten) => { + writes.push(chunk) + onWritten?.() + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.setActiveGeneration('attach-1') + queue.enqueue('old', () => callbacks.push('old'), { generation: 'attach-1' }) + queue.setActiveGeneration('attach-2', { dropQueuedStaleWrites: true }) + queue.enqueue('new', () => callbacks.push('new'), { generation: 'attach-2' }) + + rafCallbacks.shift()?.(16) + + expect(writes).toEqual(['new']) + expect(callbacks).toEqual(['new']) + }) + + it('suppresses stale write callbacks after generation changes', () => { + const callbacks: string[] = [] + const pendingCallbacks: Array<() => void> = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-stale-callback', + write: (_chunk, onWritten) => { + if (onWritten) pendingCallbacks.push(onWritten) + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.setActiveGeneration('attach-1') + queue.enqueue('old', () => callbacks.push('old'), { generation: 'attach-1' }) + rafCallbacks.shift()?.(16) + + expect(queue.hasInFlightWrites()).toBe(true) + queue.setActiveGeneration('attach-2', { dropQueuedStaleWrites: true }) + pendingCallbacks.shift()?.() + + expect(callbacks).toEqual([]) + expect(queue.hasInFlightWrites()).toBe(false) + }) + it('keeps replay work on the normal frame budget', () => { const tasks: string[] = [] const rafCallbacks: FrameRequestCallback[] = [] let nowMs = 0 const queue = createTerminalWriteQueue({ - write: () => {}, + terminalInstanceId: 'surface-replay-budget', + write: (_chunk, onWritten) => { + onWritten?.() + }, requestFrame: (cb) => { rafCallbacks.push(cb) return rafCallbacks.length @@ -170,4 +327,54 @@ describe('createTerminalWriteQueue', () => { expect(tasks).toEqual(['A', 'B', 'C']) expect(rafCallbacks).toHaveLength(0) }) + + it('keeps submitted write scope active across async parser callbacks and serializes writes', () => { + const writes: string[] = [] + const pendingCallbacks: Array<() => void> = [] + const rafCallbacks: FrameRequestCallback[] = [] + + const queue = createTerminalWriteQueue({ + terminalInstanceId: 'surface-async-scope', + write: (chunk, onWritten) => { + writes.push(chunk) + if (onWritten) pendingCallbacks.push(onWritten) + }, + requestFrame: (cb) => { + rafCallbacks.push(cb) + return rafCallbacks.length + }, + cancelFrame: () => {}, + }) + + queue.enqueue('replay', undefined, { mode: 'replay', generation: 'attach-1' }) + queue.enqueue('live', undefined, { mode: 'live', generation: 'attach-1' }) + + rafCallbacks.shift()?.(16) + + expect(writes).toEqual(['replay']) + expect(getTerminalOutputWriteScope('surface-async-scope')?.source).toBe('replay') + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-async-scope', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(false) + expect(pendingCallbacks).toHaveLength(1) + expect(queue.hasInFlightWrites()).toBe(true) + + pendingCallbacks.shift()?.() + + expect(getTerminalOutputWriteScope('surface-async-scope')).toBeNull() + expect(writes).toEqual(['replay']) + expect(rafCallbacks).toHaveLength(1) + + rafCallbacks.shift()?.(32) + + expect(writes).toEqual(['replay', 'live']) + expect(getTerminalOutputWriteScope('surface-async-scope')?.source).toBe('live') + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-async-scope', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(true) + }) }) diff --git a/test/unit/client/lib/pane-action-registry.test.ts b/test/unit/client/lib/pane-action-registry.test.ts new file mode 100644 index 000000000..8590a15e0 --- /dev/null +++ b/test/unit/client/lib/pane-action-registry.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, it, vi } from 'vitest' +import { + getTerminalActions, + registerTerminalActions, + type TerminalActions, +} from '@/lib/pane-action-registry' + +function createTerminalActions(): TerminalActions { + return { + copySelection: vi.fn(), + paste: vi.fn(), + selectAll: vi.fn(), + clearScrollback: vi.fn(), + reset: vi.fn(), + scrollToBottom: vi.fn(), + hasSelection: vi.fn(() => false), + openSearch: vi.fn(), + } +} + +describe('pane action registry', () => { + it('does not let an older terminal unregister remove newer pane actions', () => { + const paneId = 'pane-terminal-actions' + const staleActions = createTerminalActions() + const currentActions = createTerminalActions() + const unregisterStale = registerTerminalActions(paneId, staleActions) + const unregisterCurrent = registerTerminalActions(paneId, currentActions) + + try { + unregisterStale() + expect(getTerminalActions(paneId)).toBe(currentActions) + } finally { + unregisterCurrent() + unregisterStale() + } + + expect(getTerminalActions(paneId)).toBeUndefined() + }) +}) diff --git a/test/unit/client/lib/terminal-attach-policy.test.ts b/test/unit/client/lib/terminal-attach-policy.test.ts index de4b4012d..95c004ef7 100644 --- a/test/unit/client/lib/terminal-attach-policy.test.ts +++ b/test/unit/client/lib/terminal-attach-policy.test.ts @@ -2,12 +2,11 @@ import { describe, expect, it } from 'vitest' import { resolveRevealAttachPlan } from '@/lib/terminal-attach-policy' describe('terminal attach policy', () => { - it('promotes viewport hydrate to delta reconnect from a trusted rendered high-water mark', () => { + it('promotes viewport hydrate to delta reconnect from a compatible checkpoint', () => { expect(resolveRevealAttachPlan({ pendingIntent: 'viewport_hydrate', pendingReason: 'hidden_reveal', - hasTrustedSurface: true, - renderedSeq: 41, + checkpointDecision: { ok: true, sinceSeq: 41 }, })).toEqual({ intent: 'transport_reconnect', clearViewportFirst: false, @@ -20,12 +19,12 @@ describe('terminal attach policy', () => { expect(resolveRevealAttachPlan({ pendingIntent: 'viewport_hydrate', pendingReason: 'hidden_reveal', - hasTrustedSurface: false, - renderedSeq: 41, + checkpointDecision: { ok: false, reason: 'missing_checkpoint' }, })).toEqual({ intent: 'viewport_hydrate', clearViewportFirst: true, priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, }) }) @@ -33,12 +32,12 @@ describe('terminal attach policy', () => { expect(resolveRevealAttachPlan({ pendingIntent: 'viewport_hydrate', pendingReason: 'explicit_refresh', - hasTrustedSurface: true, - renderedSeq: 41, + checkpointDecision: { ok: true, sinceSeq: 41 }, })).toEqual({ intent: 'viewport_hydrate', clearViewportFirst: true, priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, }) }) @@ -46,8 +45,7 @@ describe('terminal attach policy', () => { expect(resolveRevealAttachPlan({ pendingIntent: 'transport_reconnect', pendingReason: 'transport_reconnect', - hasTrustedSurface: true, - renderedSeq: 41, + checkpointDecision: { ok: true, sinceSeq: 41 }, })).toEqual({ intent: 'transport_reconnect', clearViewportFirst: false, @@ -55,4 +53,70 @@ describe('terminal attach policy', () => { sinceSeq: 41, }) }) + + it('falls back to clearing viewport hydrate for unsafe transport reconnect', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'transport_reconnect', + pendingReason: 'hidden_reveal', + checkpointDecision: { ok: false, reason: 'parser_busy' }, + })).toEqual({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, + }) + }) + + it('falls back to clearing viewport hydrate for unsafe keepalive delta', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'keepalive_delta', + pendingReason: 'background_catchup', + checkpointDecision: { ok: false, reason: 'geometry_changed' }, + })).toEqual({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, + }) + }) + + it('does not trust legacy rendered high-water input during checkpoint migration', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'viewport_hydrate', + pendingReason: 'hidden_reveal', + hasTrustedSurface: true, + renderedSeq: 41, + })).toEqual({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, + }) + }) + + it('falls back to viewport hydrate when the parser-applied checkpoint is unsafe', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'viewport_hydrate', + pendingReason: 'hidden_reveal', + checkpointDecision: { ok: false, reason: 'geometry_changed' }, + })).toMatchObject({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + }) + }) + + it('does not treat replay from zero as trusted full hydrate without compatible geometry history', () => { + expect(resolveRevealAttachPlan({ + pendingIntent: 'viewport_hydrate', + pendingReason: 'hidden_reveal', + checkpointDecision: { ok: false, reason: 'geometry_changed' }, + replayHydrateCoversCompatibleGeometryHistory: false, + })).toMatchObject({ + intent: 'viewport_hydrate', + clearViewportFirst: true, + priority: 'foreground', + trustResultingSurfaceForDeltaReplay: false, + }) + }) }) diff --git a/test/unit/client/lib/terminal-attach-seq-state.test.ts b/test/unit/client/lib/terminal-attach-seq-state.test.ts index 5cedcbf71..85b36c219 100644 --- a/test/unit/client/lib/terminal-attach-seq-state.test.ts +++ b/test/unit/client/lib/terminal-attach-seq-state.test.ts @@ -2,7 +2,10 @@ import { describe, expect, it } from 'vitest' import { createAttachSeqState, beginAttach, + markOutputRangeUnapplied, + markParserAppliedSeq, onAttachReady, + onOutputBatchSegments, onOutputFrame, onOutputGap, } from '@/lib/terminal-attach-seq-state' @@ -92,6 +95,39 @@ describe('terminal-attach-seq-state', () => { } }) + it('accepts all batch segments as one preflight decision', () => { + let state = beginAttach(createAttachSeqState({ lastSeq: 0 })) + state = onAttachReady(state, { headSeq: 8, replayFromSeq: 6, replayToSeq: 8 }) + + const decision = onOutputBatchSegments(state, [ + { seqStart: 6, seqEnd: 6 }, + { seqStart: 7, seqEnd: 7 }, + { seqStart: 8, seqEnd: 8 }, + ]) + + expect(decision.accept).toBe(true) + if (!decision.accept) throw new Error('expected accepted batch decision') + expect(decision.state.lastSeq).toBe(8) + expect(decision.state.pendingReplay).toBeNull() + expect(decision.segments.map((segment) => segment.parserAppliedSeq)).toEqual([6, 7, 8]) + }) + + it('rejects an entire batch when any segment overlaps already accepted output', () => { + const state = createAttachSeqState({ lastSeq: 2 }) + + const decision = onOutputBatchSegments(state, [ + { seqStart: 3, seqEnd: 3 }, + { seqStart: 2, seqEnd: 2 }, + { seqStart: 4, seqEnd: 4 }, + ]) + + expect(decision.accept).toBe(false) + if (decision.accept) throw new Error('expected rejected batch decision') + expect(decision.reason).toBe('overlap') + expect(decision.rejectedSegment).toEqual({ seqStart: 2, seqEnd: 2 }) + expect(decision.state).toEqual(state) + }) + it('drops overlap outside pending replay window', () => { const state = createAttachSeqState({ lastSeq: 8 }) const decision = onOutputFrame(state, { seqStart: 7, seqEnd: 8 }) @@ -102,7 +138,7 @@ describe('terminal-attach-seq-state', () => { it('advances through replay_window_exceeded gap and preserves forward progress', () => { let state = beginAttach(createAttachSeqState({ lastSeq: 0 })) state = onAttachReady(state, { headSeq: 8, replayFromSeq: 6, replayToSeq: 8 }) - state = onOutputGap(state, { fromSeq: 1, toSeq: 5 }) + state = onOutputGap(state, { fromSeq: 1, toSeq: 5 }).state const frame = expectAcceptedFrame(onOutputFrame(state, { seqStart: 6, seqEnd: 8 })) expect(frame.freshReset).toBe(false) expect(frame.state.lastSeq).toBe(8) @@ -112,7 +148,7 @@ describe('terminal-attach-seq-state', () => { it('clears pending replay when a gap covers replay tail', () => { let state = beginAttach(createAttachSeqState({ lastSeq: 0 })) state = onAttachReady(state, { headSeq: 8, replayFromSeq: 6, replayToSeq: 8 }) - state = onOutputGap(state, { fromSeq: 1, toSeq: 8 }) + state = onOutputGap(state, { fromSeq: 1, toSeq: 8 }).state expect(state.lastSeq).toBe(8) expect(state.pendingReplay).toBeNull() expect(state.awaitingFreshSequence).toBe(false) @@ -124,12 +160,91 @@ describe('terminal-attach-seq-state', () => { pendingReplay: { fromSeq: 4, toSeq: 6 }, awaitingFreshSequence: true, }) - const next = onOutputGap(state, { fromSeq: -5, toSeq: -1 }) + const next = onOutputGap(state, { fromSeq: -5, toSeq: -1 }).state expect(next.lastSeq).toBe(3) expect(next.pendingReplay).toEqual({ fromSeq: 4, toSeq: 6 }) expect(next.awaitingFreshSequence).toBe(false) }) + it('records gaps without advancing the parser-applied sequence', () => { + const state = createAttachSeqState() + const afterFrame = onOutputFrame(state, { seqStart: 1, seqEnd: 1 }) + expect(afterFrame.accept).toBe(true) + if (!afterFrame.accept) throw new Error('expected accepted frame decision') + + const afterGap = onOutputGap(afterFrame.state, { fromSeq: 2, toSeq: 10 }) + + expect(afterFrame.state.highestObservedSeq).toBe(1) + expect(afterFrame.state.parserAppliedSeq).toBe(0) + expect(afterGap.state.highestObservedSeq).toBe(10) + expect(afterGap.state.parserAppliedSeq).toBe(0) + expect(afterGap.state.knownLostRanges).toEqual([{ fromSeq: 2, toSeq: 10 }]) + expect(afterGap.surfaceSafeForDeltaReplay).toBe(false) + expect(afterGap.requiresSurfaceQuarantine).toBe(true) + expect('lastSeq' in afterGap).toBe(false) + expect('parserAppliedSeq' in afterGap).toBe(false) + }) + + it('marks parser-applied output only after xterm acknowledgement', () => { + const frame = expectAcceptedFrame(onOutputFrame(createAttachSeqState(), { + seqStart: 1, + seqEnd: 3, + })) + + const appliedToTwo = markParserAppliedSeq(frame.state, 2) + expect(appliedToTwo.parserAppliedSeq).toBe(2) + expect(appliedToTwo.highestObservedSeq).toBe(3) + + const notDecreased = markParserAppliedSeq(appliedToTwo, 1) + expect(notDecreased.parserAppliedSeq).toBe(2) + + const cappedAtObserved = markParserAppliedSeq(appliedToTwo, 9) + expect(cappedAtObserved.parserAppliedSeq).toBe(3) + }) + + it('does not mark parser-applied output across a known lost range', () => { + const frame = expectAcceptedFrame(onOutputFrame(createAttachSeqState(), { + seqStart: 1, + seqEnd: 1, + })) + const applied = markParserAppliedSeq(frame.state, 1) + const gap = onOutputGap(applied, { fromSeq: 2, toSeq: 10 }) + + expect(markParserAppliedSeq(gap.state, 10).parserAppliedSeq).toBe(1) + }) + + it('does not mark parser-applied output through the tail of an overlapping known lost range', () => { + const frame = expectAcceptedFrame(onOutputFrame(createAttachSeqState(), { + seqStart: 1, + seqEnd: 12, + })) + const applied = markParserAppliedSeq(frame.state, 10) + const gap = onOutputGap(applied, { fromSeq: 9, toSeq: 11 }) + + const afterTail = markParserAppliedSeq(gap.state, 12) + + expect(gap.state.knownLostRanges).toEqual([{ fromSeq: 9, toSeq: 11 }]) + expect(afterTail.parserAppliedSeq).toBe(10) + expect(afterTail.highestObservedSeq).toBe(12) + }) + + it('does not mark parser-applied output across an unapplied output range', () => { + const accepted = expectAcceptedFrame(onOutputFrame(createAttachSeqState(), { + seqStart: 1, + seqEnd: 3, + })) + const withUnappliedRange = markOutputRangeUnapplied(accepted.state, { + fromSeq: 2, + toSeq: 2, + }) + + const applied = markParserAppliedSeq(withUnappliedRange, 3) + + expect(withUnappliedRange.knownLostRanges).toEqual([{ fromSeq: 2, toSeq: 2 }]) + expect(withUnappliedRange.surfaceSafeForDeltaReplay).toBe(false) + expect(applied.parserAppliedSeq).toBe(1) + }) + it('allows single fresh restart at seq=1 while awaitingFreshSequence', () => { let state = createAttachSeqState({ lastSeq: 22, awaitingFreshSequence: true }) const first = expectAcceptedFrame(onOutputFrame(state, { seqStart: 1, seqEnd: 1 })) diff --git a/test/unit/client/lib/terminal-cursor.test.ts b/test/unit/client/lib/terminal-cursor.test.ts index 7a33d6806..e6f567e04 100644 --- a/test/unit/client/lib/terminal-cursor.test.ts +++ b/test/unit/client/lib/terminal-cursor.test.ts @@ -4,10 +4,42 @@ import { clearTerminalCursor, getCursorMapSize, loadTerminalCursor, + loadTerminalSurfaceCheckpoint, saveTerminalCursor, + saveTerminalSurfaceCheckpoint, } from '@/lib/terminal-cursor' +import type { TerminalSurfaceCheckpoint } from '@/lib/terminal-surface-checkpoint' import { TERMINAL_CURSOR_STORAGE_KEY } from '@/store/storage-keys' +function createCheckpoint( + overrides: Partial = {}, +): TerminalSurfaceCheckpoint { + return { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 1, + attachRequestId: 'attach-1', + parserAppliedSeq: 1, + cols: 80, + rows: 24, + geometryEpoch: 1, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + ...overrides, + } +} + +function loadCheckpointSeq(terminalId: string): number { + return loadTerminalSurfaceCheckpoint(terminalId, { + streamId: 'stream-1', + serverInstanceId: 'server-a', + })?.parserAppliedSeq ?? 0 +} + describe('terminal-cursor', () => { beforeEach(() => { vi.useRealTimers() @@ -19,53 +51,72 @@ describe('terminal-cursor', () => { vi.useRealTimers() }) - it('loads and saves terminal cursor sequence values', () => { - expect(loadTerminalCursor('term-1')).toBe(0) + it('loads and saves terminal surface checkpoint sequence values', () => { + expect(loadCheckpointSeq('term-1')).toBe(0) + + saveTerminalSurfaceCheckpoint(createCheckpoint({ parserAppliedSeq: 4 })) + expect(loadCheckpointSeq('term-1')).toBe(4) - saveTerminalCursor('term-1', 4) - expect(loadTerminalCursor('term-1')).toBe(4) + saveTerminalSurfaceCheckpoint(createCheckpoint({ parserAppliedSeq: 2 })) + expect(loadCheckpointSeq('term-1')).toBe(4) - saveTerminalCursor('term-1', 2) - expect(loadTerminalCursor('term-1')).toBe(4) + saveTerminalSurfaceCheckpoint(createCheckpoint({ parserAppliedSeq: 8 })) + expect(loadCheckpointSeq('term-1')).toBe(8) + }) - saveTerminalCursor('term-1', 8) - expect(loadTerminalCursor('term-1')).toBe(8) + it('uses the incompatible v2 storage namespace for checkpoint records', () => { + expect(TERMINAL_CURSOR_STORAGE_KEY).toBe('freshell.terminal-cursors.v2') }) it('clears an entry when terminal exits', () => { - saveTerminalCursor('term-2', 11) - expect(loadTerminalCursor('term-2')).toBe(11) + saveTerminalSurfaceCheckpoint(createCheckpoint({ + terminalId: 'term-2', + parserAppliedSeq: 11, + })) + expect(loadCheckpointSeq('term-2')).toBe(11) clearTerminalCursor('term-2') - expect(loadTerminalCursor('term-2')).toBe(0) + expect(loadCheckpointSeq('term-2')).toBe(0) }) it('drops expired entries when loading from storage', () => { const now = Date.now() const fifteenDaysMs = 15 * 24 * 60 * 60 * 1000 localStorage.setItem(TERMINAL_CURSOR_STORAGE_KEY, JSON.stringify({ - stale: { seq: 5, updatedAt: now - fifteenDaysMs }, - fresh: { seq: 9, updatedAt: now }, + stale: { + checkpoint: createCheckpoint({ terminalId: 'stale', parserAppliedSeq: 5 }), + updatedAt: now - fifteenDaysMs, + }, + fresh: { + checkpoint: createCheckpoint({ terminalId: 'fresh', parserAppliedSeq: 9 }), + updatedAt: now, + }, })) __resetTerminalCursorCacheForTests() - expect(loadTerminalCursor('stale')).toBe(0) - expect(loadTerminalCursor('fresh')).toBe(9) + expect(loadCheckpointSeq('stale')).toBe(0) + expect(loadCheckpointSeq('fresh')).toBe(9) expect(getCursorMapSize()).toBe(1) }) it('enforces max entry count by keeping most recently updated entries', () => { const now = Date.now() - const payload: Record = {} + const payload: Record = {} for (let i = 0; i < 520; i += 1) { - payload[`term-${i}`] = { seq: i + 1, updatedAt: now - i } + payload[`term-${i}`] = { + checkpoint: createCheckpoint({ + terminalId: `term-${i}`, + parserAppliedSeq: i + 1, + }), + updatedAt: now - i, + } } localStorage.setItem(TERMINAL_CURSOR_STORAGE_KEY, JSON.stringify(payload)) __resetTerminalCursorCacheForTests() expect(getCursorMapSize()).toBeLessThanOrEqual(500) - expect(loadTerminalCursor('term-0')).toBe(1) - expect(loadTerminalCursor('term-519')).toBe(0) + expect(loadCheckpointSeq('term-0')).toBe(1) + expect(loadCheckpointSeq('term-519')).toBe(0) }) it('remains resilient when stored payload is malformed', () => { @@ -76,15 +127,100 @@ describe('terminal-cursor', () => { expect(getCursorMapSize()).toBe(0) }) - it('debounces localStorage persistence for rapid cursor updates', () => { + it('treats legacy persisted cursor records as incompatible by default', () => { + localStorage.setItem(TERMINAL_CURSOR_STORAGE_KEY, JSON.stringify({ + 'term-legacy': { seq: 12, updatedAt: Date.now() }, + })) + __resetTerminalCursorCacheForTests() + + expect(loadTerminalCursor('term-legacy')).toBe(0) + expect(loadTerminalSurfaceCheckpoint('term-legacy', { + streamId: 'stream-1', + serverInstanceId: 'server-a', + })).toBeNull() + }) + + it('does not let legacy cursor writes mutate a trusted checkpoint', () => { + saveTerminalSurfaceCheckpoint(createCheckpoint({ + terminalId: 'term-legacy-write', + parserAppliedSeq: 25, + })) + + saveTerminalCursor('term-legacy-write', 100) + + expect(loadTerminalCursor('term-legacy-write')).toBe(0) + expect(loadTerminalSurfaceCheckpoint('term-legacy-write', { + streamId: 'stream-1', + serverInstanceId: 'server-a', + })?.parserAppliedSeq).toBe(25) + }) + + it('does not create a trusted cursor from legacy cursor writes', () => { + saveTerminalCursor('term-only-legacy', 100) + + expect(loadTerminalCursor('term-only-legacy')).toBe(0) + expect(loadTerminalSurfaceCheckpoint('term-only-legacy', { + streamId: null, + serverInstanceId: 'legacy-cursor', + })).toBeNull() + }) + + it('does not load a persisted checkpoint for a different server instance', () => { + saveTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 1, + attachRequestId: 'attach-1', + parserAppliedSeq: 25, + cols: 80, + rows: 24, + geometryEpoch: 1, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(loadTerminalSurfaceCheckpoint('term-1', { + streamId: 'stream-1', + serverInstanceId: 'server-b', + })).toBeNull() + }) + + it('does not load a persisted checkpoint when either side lacks a boot id', () => { + saveTerminalSurfaceCheckpoint(createCheckpoint({ + terminalId: 'term-boot', + serverBootId: 'boot-a', + parserAppliedSeq: 25, + })) + + expect(loadTerminalSurfaceCheckpoint('term-boot', { + streamId: 'stream-1', + serverInstanceId: 'server-a', + })).toBeNull() + expect(loadTerminalSurfaceCheckpoint('term-boot', { + streamId: 'stream-1', + serverInstanceId: 'server-a', + serverBootId: 'boot-b', + })).toBeNull() + expect(loadTerminalSurfaceCheckpoint('term-boot', { + streamId: 'stream-1', + serverInstanceId: 'server-a', + serverBootId: 'boot-a', + })?.parserAppliedSeq).toBe(25) + }) + + it('debounces localStorage persistence for rapid checkpoint updates', () => { vi.useFakeTimers() const setItemSpy = vi.spyOn(Storage.prototype, 'setItem') - saveTerminalCursor('term-rapid', 1) - saveTerminalCursor('term-rapid', 2) - saveTerminalCursor('term-rapid', 3) + saveTerminalSurfaceCheckpoint(createCheckpoint({ terminalId: 'term-rapid', parserAppliedSeq: 1 })) + saveTerminalSurfaceCheckpoint(createCheckpoint({ terminalId: 'term-rapid', parserAppliedSeq: 2 })) + saveTerminalSurfaceCheckpoint(createCheckpoint({ terminalId: 'term-rapid', parserAppliedSeq: 3 })) - expect(loadTerminalCursor('term-rapid')).toBe(3) + expect(loadCheckpointSeq('term-rapid')).toBe(3) expect(setItemSpy).not.toHaveBeenCalled() vi.advanceTimersByTime(250) @@ -97,12 +233,12 @@ describe('terminal-cursor', () => { vi.useFakeTimers() const setItemSpy = vi.spyOn(Storage.prototype, 'setItem') - saveTerminalCursor('term-clear', 7) + saveTerminalSurfaceCheckpoint(createCheckpoint({ terminalId: 'term-clear', parserAppliedSeq: 7 })) expect(setItemSpy).not.toHaveBeenCalled() clearTerminalCursor('term-clear') expect(setItemSpy).toHaveBeenCalledTimes(1) - expect(loadTerminalCursor('term-clear')).toBe(0) + expect(loadCheckpointSeq('term-clear')).toBe(0) vi.advanceTimersByTime(250) expect(setItemSpy).toHaveBeenCalledTimes(1) diff --git a/test/unit/client/lib/terminal-osc52.test.ts b/test/unit/client/lib/terminal-osc52.test.ts index 1eaabccfa..111ccd8bd 100644 --- a/test/unit/client/lib/terminal-osc52.test.ts +++ b/test/unit/client/lib/terminal-osc52.test.ts @@ -1,5 +1,11 @@ import { describe, expect, it } from 'vitest' -import { createOsc52ParserState, extractOsc52Events } from '@/lib/terminal-osc52' +import { + createOsc52ParserState, + extractOsc52Events, + shouldAllowOsc52ClipboardWrite, + shouldAllowOsc52Prompt, +} from '@/lib/terminal-osc52' +import { beginTerminalOutputWriteScope } from '@/lib/terminal-output-write-scope' describe('terminal-osc52', () => { it('extracts OSC52 payload and returns cleaned output', () => { @@ -48,4 +54,42 @@ describe('terminal-osc52', () => { expect(result.cleaned).toBe('ab') expect(result.events).toEqual([]) }) + + it('suppresses always-policy clipboard writes while the submitted write scope is replay', () => { + const replayScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-osc52', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + + expect(shouldAllowOsc52ClipboardWrite({ + terminalInstanceId: 'surface-osc52', + mode: 'shell', + })).toBe(false) + expect(shouldAllowOsc52Prompt({ + terminalInstanceId: 'surface-osc52', + mode: 'shell', + })).toBe(false) + replayScope.complete() + + const liveScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-osc52', + source: 'live', + attachRequestId: 'attach-2', + generation: 'attach-2', + suppressExternalSideEffects: false, + }) + + expect(shouldAllowOsc52ClipboardWrite({ + terminalInstanceId: 'surface-osc52', + mode: 'shell', + })).toBe(true) + expect(shouldAllowOsc52Prompt({ + terminalInstanceId: 'surface-osc52', + mode: 'shell', + })).toBe(true) + liveScope.complete() + }) }) diff --git a/test/unit/client/lib/terminal-output-side-effects.test.ts b/test/unit/client/lib/terminal-output-side-effects.test.ts new file mode 100644 index 000000000..b34b8a26d --- /dev/null +++ b/test/unit/client/lib/terminal-output-side-effects.test.ts @@ -0,0 +1,162 @@ +import { describe, expect, it, vi } from 'vitest' + +import { registerTerminalRequestModeBypass } from '@/components/terminal/request-mode-bypass' +import { + beginTerminalOutputWriteScope, + shouldAllowTerminalOutputSideEffect, + type TerminalOutputSideEffect, +} from '@/lib/terminal-output-write-scope' + +describe('terminal output side-effect policy', () => { + it('fails closed for unknown output scope', () => { + const effects: TerminalOutputSideEffect[] = [ + 'startup_reply', + 'osc52_prompt', + 'osc52_clipboard_write', + 'request_mode_reply', + 'title_update', + 'turn_complete', + 'parser_applied_checkpoint', + 'attach_completion', + 'cursor_persist', + 'link_action', + 'terminal_action', + 'local_xterm_notice', + ] + + for (const effect of effects) { + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'unknown-surface', + effect, + mode: 'shell', + })).toBe(false) + } + }) + + it('allows declared live external side effects and suppresses replay side effects', () => { + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'startup_reply', + mode: 'shell', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'startup_reply', + mode: 'shell', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'local_xterm_notice', + mode: 'shell', + })).toBe(false) + }) + + it('uses explicit live frame context instead of an unrelated active replay write scope', () => { + const replayScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-live-frame', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + + try { + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-live-frame', + source: 'live', + effect: 'startup_reply', + mode: 'opencode', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-live-frame', + source: 'live', + effect: 'osc52_clipboard_write', + mode: 'opencode', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-live-frame', + source: 'live', + effect: 'turn_complete', + mode: 'opencode', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-live-frame', + source: 'replay', + effect: 'osc52_clipboard_write', + mode: 'opencode', + })).toBe(false) + } finally { + replayScope.complete() + } + }) + + it('keeps server-authoritative turn completion for Claude and Codex', () => { + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'opencode', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'claude', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'codex', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'turn_complete', + mode: 'opencode', + })).toBe(false) + }) + + it('does not send request-mode bypass replies while the submitted write scope is replay', () => { + const disposers = [{ dispose: vi.fn() }, { dispose: vi.fn() }] + const registerCsiHandler = vi + .fn() + .mockReturnValueOnce(disposers[0]) + .mockReturnValueOnce(disposers[1]) + const sendInput = vi.fn() + const term = { + parser: { registerCsiHandler }, + modes: { bracketedPasteMode: true }, + options: {}, + buffer: { active: { type: 'normal' as const } }, + } + + registerTerminalRequestModeBypass(term as any, sendInput, { + terminalInstanceId: 'surface-request-mode', + }) + const privateHandler = registerCsiHandler.mock.calls[1]?.[1] + + const replayScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-request-mode', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + expect(privateHandler([2004])).toBe(true) + expect(sendInput).not.toHaveBeenCalled() + replayScope.complete() + + const liveScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-request-mode', + source: 'live', + attachRequestId: 'attach-2', + generation: 'attach-2', + suppressExternalSideEffects: false, + }) + expect(privateHandler([2004])).toBe(true) + expect(sendInput).toHaveBeenCalledWith('\u001b[?2004;1$y') + liveScope.complete() + }) +}) diff --git a/test/unit/client/lib/terminal-output-write-scope.test.ts b/test/unit/client/lib/terminal-output-write-scope.test.ts new file mode 100644 index 000000000..6e509e551 --- /dev/null +++ b/test/unit/client/lib/terminal-output-write-scope.test.ts @@ -0,0 +1,71 @@ +import { describe, expect, it } from 'vitest' + +import { + beginTerminalOutputWriteScope, + getTerminalOutputWriteScope, + shouldAllowTerminalOutputSideEffect, +} from '@/lib/terminal-output-write-scope' + +describe('terminal output write scope', () => { + it('keeps replay context visible until the submitted write completes', () => { + const scope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-1', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + + expect(getTerminalOutputWriteScope('surface-1')?.source).toBe('replay') + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-1', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(false) + + scope.complete() + expect(getTerminalOutputWriteScope('surface-1')).toBeNull() + }) + + it('only completes the submitted scope that is still current for the terminal instance', () => { + const replayScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-1', + source: 'replay', + attachRequestId: 'attach-1', + generation: 'attach-1', + suppressExternalSideEffects: true, + }) + const liveScope = beginTerminalOutputWriteScope({ + terminalInstanceId: 'surface-1', + source: 'live', + attachRequestId: 'attach-2', + generation: 'attach-2', + suppressExternalSideEffects: false, + }) + + replayScope.complete() + expect(getTerminalOutputWriteScope('surface-1')?.source).toBe('live') + + liveScope.complete() + expect(getTerminalOutputWriteScope('surface-1')).toBeNull() + }) + + it('suppresses external side effects during replay writes', () => { + expect(shouldAllowTerminalOutputSideEffect({ + terminalInstanceId: 'surface-1', + source: 'replay', + effect: 'request_mode_reply', + mode: 'shell', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'osc52_clipboard_write', + mode: 'shell', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'title_update', + mode: 'shell', + })).toBe(false) + }) +}) diff --git a/test/unit/client/lib/terminal-surface-checkpoint.test.ts b/test/unit/client/lib/terminal-surface-checkpoint.test.ts new file mode 100644 index 000000000..b664d2adb --- /dev/null +++ b/test/unit/client/lib/terminal-surface-checkpoint.test.ts @@ -0,0 +1,240 @@ +import { describe, expect, it } from 'vitest' +import { + createTerminalSurfaceCheckpoint, + canUseCheckpointForDeltaReplay, +} from '@/lib/terminal-surface-checkpoint' + +describe('terminal surface checkpoint', () => { + it('accepts a compatible parser-applied checkpoint', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: true, sinceSeq: 42 }) + }) + + it('rejects warm delta when the current stream identity is missing even if the checkpoint is null-stream', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: null, + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: null, + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'stream_changed' }) + }) + + it('rejects a checkpoint after geometry changes', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 100, + rows: 40, + geometryEpoch: 4, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'geometry_changed' }) + }) + + it('rejects a checkpoint while parser work is still in flight', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: false, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'parser_busy' }) + }) + + it('rejects a checkpoint from a different server instance', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-b', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'server_changed' }) + }) + + it('rejects a checkpoint when only the checkpoint has a server boot id', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + serverBootId: 'boot-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'server_changed' }) + }) + + it('rejects a checkpoint when only the current server has a boot id', () => { + const checkpoint = createTerminalSurfaceCheckpoint({ + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + surfaceEpoch: 2, + attachRequestId: 'attach-2', + parserAppliedSeq: 42, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + bufferType: 'normal', + parserIdle: true, + }) + + expect(canUseCheckpointForDeltaReplay(checkpoint, { + terminalId: 'term-1', + streamId: 'stream-1', + serverInstanceId: 'server-a', + serverBootId: 'boot-a', + surfaceEpoch: 2, + cols: 120, + rows: 40, + geometryEpoch: 3, + geometryAuthority: 'single_client', + scrollback: 5000, + xtermVersion: '6.0.0', + requireParserIdle: true, + })).toMatchObject({ ok: false, reason: 'server_changed' }) + }) +}) diff --git a/test/unit/client/lib/ws-client.test.ts b/test/unit/client/lib/ws-client.test.ts index f056083f1..180b99284 100644 --- a/test/unit/client/lib/ws-client.test.ts +++ b/test/unit/client/lib/ws-client.test.ts @@ -79,7 +79,7 @@ describe('WsClient.connect', () => { expect(resolved).toBe(true) }) - it('sends protocol version in hello and only advertises surviving capabilities', async () => { + it('sends protocol version in hello and only advertises parse-supported capabilities', async () => { const c = new WsClient('ws://example/ws') const p = c.connect() expect(MockWebSocket.instances).toHaveLength(1) @@ -88,7 +88,10 @@ describe('WsClient.connect', () => { const hello = JSON.parse(MockWebSocket.instances[0].sent[0]) expect(hello.type).toBe('hello') expect(hello.protocolVersion).toBe(WS_PROTOCOL_VERSION) - expect(hello.capabilities).toEqual({ uiScreenshotV1: true }) + expect(hello.capabilities).toEqual({ + uiScreenshotV1: true, + terminalOutputBatchV1: true, + }) MockWebSocket.instances[0]._message({ type: 'ready' }) await p diff --git a/test/unit/lib/visible-first-audit-derived-metrics.test.ts b/test/unit/lib/visible-first-audit-derived-metrics.test.ts index be9a48938..f73258b2a 100644 --- a/test/unit/lib/visible-first-audit-derived-metrics.test.ts +++ b/test/unit/lib/visible-first-audit-derived-metrics.test.ts @@ -52,4 +52,283 @@ describe('deriveVisibleFirstMetrics', () => { expect(result.offscreenWsFramesBeforeReady).toBe(2) expect(result.terminalInputToFirstOutputMs).toBe(45) }) + + it('derives terminal catch-up replay metrics from structured logs, websocket evidence, and client perf events', () => { + const result = deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: ['/api/bootstrap', '/api/terminals/:terminalId/viewport'], + allowedWsTypesBeforeReady: ['hello', 'ready', 'terminal.output', 'terminal.output.batch', 'terminal.output.gap'], + browser: { + milestones: { + 'terminal.first_output': 150, + }, + perfEvents: [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 31 }, + { event: 'terminal.parser_applied', timestamp: 118, parserAppliedSeq: 6 }, + { event: 'terminal.parser_applied', timestamp: 140, parserAppliedSeq: 8 }, + { event: 'terminal.catchup.full_hydrate_fallback', timestamp: 141 }, + { event: 'terminal.catchup.surface_quarantined', timestamp: 142 }, + { event: 'terminal.attach_generation_stale_rejected', timestamp: 143 }, + { + event: 'terminal.catchup.stop_resume', + timestamp: 144, + source: 'visible_first_audit_process_suspend', + browserExecutionStopped: true, + retentionCoveredMs: 2_500, + stoppedDurationMs: 8_000, + outputStartedAfterStopMs: 1_000, + outputStartedBeforeResumeMs: 2_500, + cdpCatchupOutputMessageCount: 3, + gapCount: 1, + }, + ], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 90, + direction: 'sent', + type: 'terminal.attach', + payload: JSON.stringify({ + type: 'terminal.attach', + terminalId: 'term-reconnect', + }), + payloadLength: 80, + }, + { + timestamp: 100, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'replay', + seqStart: 1, + seqEnd: 6, + serializedBytes: 400, + }), + payloadLength: 400, + }, + { + timestamp: 130, + direction: 'received', + type: 'terminal.output', + payload: JSON.stringify({ + type: 'terminal.output', + source: 'replay', + seqStart: 7, + seqEnd: 8, + }), + payloadLength: 120, + }, + ], + }, + }, + server: { + terminalReplayEvents: [ + { + event: 'terminal.replay.batch', + source: 'replay', + payloadType: 'terminal.output.batch', + seqStart: 1, + seqEnd: 6, + serializedBytes: 400, + }, + { + event: 'terminal.replay.batch', + source: 'replay', + payloadType: 'terminal.output', + seqStart: 7, + seqEnd: 8, + serializedBytes: 120, + }, + { + event: 'terminal.replay.gap', + source: 'replay', + fromSeq: 9, + toSeq: 9, + }, + ], + }, + }) + + expect(result.maxRafGapMs).toBe(31) + expect(result.terminalInputToFirstOutputMs).toBe(10) + expect(result.terminalReplayMessageCount).toBe(2) + expect(result.terminalReplaySerializedBytes).toBe(520) + expect(result.terminalParserAppliedLagMs).toBe(18) + expect(result.terminalReplayGapCount).toBe(1) + expect(result.terminalFullHydrateFallbackCount).toBe(1) + expect(result.terminalSurfaceQuarantineCount).toBe(1) + expect(result.terminalStaleGenerationRejectionCount).toBe(1) + expect(result.terminalStoppedRetentionCoveredMs).toBe(2_500) + expect(result.terminalStopResumeGapCount).toBe(1) + }) + + it('omits source-dependent required metrics when RAF or parser-applied evidence is absent', () => { + const withoutRaf = deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: [], + allowedWsTypesBeforeReady: ['terminal.output.batch'], + browser: { + milestones: { + 'terminal.first_output': 100, + }, + perfEvents: [ + { event: 'terminal.parser_applied', timestamp: 40, parserAppliedSeq: 1 }, + ], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 30, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'replay', + seqStart: 1, + seqEnd: 1, + }), + payloadLength: 120, + }, + ], + }, + }, + }) + expect(withoutRaf).not.toHaveProperty('maxRafGapMs') + expect(withoutRaf.terminalParserAppliedLagMs).toBe(10) + + const withoutParserApplied = deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: [], + allowedWsTypesBeforeReady: ['terminal.output.batch'], + browser: { + milestones: { + 'terminal.first_output': 100, + }, + perfEvents: [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 16 }, + ], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 30, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'replay', + seqStart: 1, + seqEnd: 1, + }), + payloadLength: 120, + }, + ], + }, + }, + }) + expect(withoutParserApplied.maxRafGapMs).toBe(16) + expect(withoutParserApplied).not.toHaveProperty('terminalParserAppliedLagMs') + }) + + it('does not count untagged or live terminal output as replay websocket evidence', () => { + const result = deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: [], + allowedWsTypesBeforeReady: ['terminal.output', 'terminal.output.batch'], + browser: { + milestones: { + 'terminal.first_output': 100, + }, + perfEvents: [ + { event: 'terminal.parser_applied', timestamp: 90, parserAppliedSeq: 2 }, + ], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 40, + direction: 'received', + type: 'terminal.output', + payload: JSON.stringify({ + type: 'terminal.output', + seqStart: 1, + seqEnd: 1, + }), + payloadLength: 120, + }, + { + timestamp: 50, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'live', + seqStart: 2, + seqEnd: 2, + }), + payloadLength: 140, + }, + ], + }, + }, + }) + + expect(result.terminalReplayMessageCount).toBe(0) + expect(result.terminalReplaySerializedBytes).toBe(0) + expect(result).not.toHaveProperty('terminalParserAppliedLagMs') + }) + + it('omits stop/resume metrics when process-suspend proof evidence is absent or synthetic', () => { + const result = deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: [], + allowedWsTypesBeforeReady: [], + browser: { + milestones: { + 'terminal.first_output': 50, + }, + perfEvents: [ + { + event: 'terminal.catchup.stop_resume', + source: 'unit_reconnect_fixture', + browserExecutionStopped: false, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 5, + gapCount: 0, + }, + { + event: 'terminal.catchup.stop_resume', + source: 'visible_first_audit_process_suspend', + browserExecutionStopped: true, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 0, + gapCount: 0, + }, + ], + }, + transport: { + http: { requests: [] }, + ws: { frames: [] }, + }, + }) + + expect(result).not.toHaveProperty('terminalStoppedRetentionCoveredMs') + expect(result).not.toHaveProperty('terminalStopResumeGapCount') + }) }) diff --git a/test/unit/lib/visible-first-audit-gate.test.ts b/test/unit/lib/visible-first-audit-gate.test.ts index 0a83990b5..27b7fca28 100644 --- a/test/unit/lib/visible-first-audit-gate.test.ts +++ b/test/unit/lib/visible-first-audit-gate.test.ts @@ -18,12 +18,91 @@ import { evaluateVisibleFirstAuditGate, type VisibleFirstAuditGateResult, } from '@test/e2e-browser/perf/visible-first-audit-gate' +import { deriveVisibleFirstMetrics } from '@test/e2e-browser/perf/derive-visible-first-metrics' const execFileAsync = promisify(execFile) const require = createRequire(import.meta.url) type GateMetric = VisibleFirstAuditGateResult['violations'][number]['metric'] +const TERMINAL_RECONNECT_REQUIRED_METRICS = { + focusedReadyMs: 100, + maxRafGapMs: 16, + terminalReplayMessageCount: 2, + terminalReplaySerializedBytes: 12_000, + terminalParserAppliedLagMs: 24, + terminalReplayGapCount: 0, + terminalFullHydrateFallbackCount: 0, + terminalSurfaceQuarantineCount: 0, + terminalStaleGenerationRejectionCount: 0, + terminalStoppedRetentionCoveredMs: 0, + terminalStopResumeGapCount: 0, +} + +function reconnectRequiredMetrics(profileId: VisibleFirstProfileId) { + return { + terminalInputToFirstOutputMs: profileId === 'mobile_restricted' ? 35 : 25, + ...TERMINAL_RECONNECT_REQUIRED_METRICS, + } +} + +function deriveReconnectMetricsWithStopResumeEvent(stopResumeEvent: Record) { + return deriveVisibleFirstMetrics({ + focusedReadyMilestone: 'terminal.first_output', + allowedApiRouteIdsBeforeReady: ['/api/bootstrap', '/api/terminals/:terminalId/viewport'], + allowedWsTypesBeforeReady: ['hello', 'ready', 'terminal.attach', 'terminal.output', 'terminal.output.batch'], + browser: { + milestones: { + 'terminal.first_output': 100, + }, + perfEvents: [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 16 }, + { event: 'terminal.parser_applied', timestamp: 40, parserAppliedSeq: 1 }, + stopResumeEvent, + ], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 10, + direction: 'sent', + type: 'terminal.attach', + payload: JSON.stringify({ type: 'terminal.attach', terminalId: 'term-reconnect' }), + payloadLength: 80, + }, + { + timestamp: 30, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'replay', + terminalId: 'term-reconnect', + seqStart: 1, + seqEnd: 1, + serializedBytes: 120, + }), + payloadLength: 120, + }, + ], + }, + }, + server: { + terminalReplayEvents: [ + { + event: 'terminal.replay.batch', + source: 'replay', + seqStart: 1, + seqEnd: 1, + serializedBytes: 120, + }, + ], + }, + }) +} + function createArtifact(): VisibleFirstAuditArtifact { return VisibleFirstAuditSchema.parse({ schemaVersion: 1, @@ -52,7 +131,9 @@ function createArtifact(): VisibleFirstAuditArtifact { browser: {}, transport: {}, server: {}, - derived: {}, + derived: scenarioId === 'terminal-reconnect-backlog' + ? reconnectRequiredMetrics(profileId) + : {}, errors: [], })), summaryByProfile: { @@ -63,6 +144,7 @@ function createArtifact(): VisibleFirstAuditArtifact { offscreenHttpBytesBeforeReady: 0, offscreenWsFramesBeforeReady: 0, offscreenWsBytesBeforeReady: 0, + ...(scenarioId === 'terminal-reconnect-backlog' ? reconnectRequiredMetrics('desktop_local') : {}), }, mobile_restricted: { focusedReadyMs: 150, @@ -71,6 +153,7 @@ function createArtifact(): VisibleFirstAuditArtifact { offscreenHttpBytesBeforeReady: 0, offscreenWsFramesBeforeReady: 0, offscreenWsBytesBeforeReady: 0, + ...(scenarioId === 'terminal-reconnect-backlog' ? reconnectRequiredMetrics('mobile_restricted') : {}), }, }, })), @@ -100,6 +183,20 @@ function setMetric( } } +function replaceReconnectDerivedMetrics( + artifact: VisibleFirstAuditArtifact, + profileId: VisibleFirstProfileId, + derived: Record, +): void { + const scenario = getScenario(artifact, 'terminal-reconnect-backlog') + const sample = scenario.samples.find((entry) => entry.profileId === profileId) + if (!sample) { + throw new Error(`Sample not found in test fixture: terminal-reconnect-backlog/${profileId}`) + } + sample.derived = derived + scenario.summaryByProfile[profileId] = derived +} + function removeSample( artifact: VisibleFirstAuditArtifact, scenarioId: VisibleFirstScenarioId, @@ -148,6 +245,69 @@ describe('evaluateVisibleFirstAuditGate', () => { expect(() => evaluateVisibleFirstAuditGate(base, candidate)).toThrow(/terminal-cold-boot\/mobile_restricted/i) }) + it('fails when a candidate sample is missing a scenario-required terminal catch-up metric', () => { + const base = createArtifact() + const candidate = createArtifact() + const scenario = getScenario(candidate, 'terminal-reconnect-backlog') + delete (scenario.samples[0].derived as Record).terminalReplayGapCount + delete (scenario.summaryByProfile.desktop_local as Record).terminalReplayGapCount + + expect(() => evaluateVisibleFirstAuditGate(base, candidate)).toThrow( + /terminal-reconnect-backlog\/desktop_local.*terminalReplayGapCount/i, + ) + }) + + it('fails when synthetic stop/resume proof does not derive source-backed required metrics', () => { + const base = createArtifact() + const candidate = createArtifact() + const derived = deriveReconnectMetricsWithStopResumeEvent({ + event: 'terminal.catchup.stop_resume', + source: 'unit_reconnect_fixture', + browserExecutionStopped: false, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 5, + gapCount: 0, + }) + + expect(derived).not.toHaveProperty('terminalStoppedRetentionCoveredMs') + expect(derived).not.toHaveProperty('terminalStopResumeGapCount') + + replaceReconnectDerivedMetrics(candidate, 'desktop_local', derived) + + expect(() => evaluateVisibleFirstAuditGate(base, candidate)).toThrow( + /terminal-reconnect-backlog\/desktop_local:terminalStoppedRetentionCoveredMs/i, + ) + }) + + it('accepts required stop/resume metrics derived from validated process-suspend proof', () => { + const base = createArtifact() + const candidate = createArtifact() + const derived = deriveReconnectMetricsWithStopResumeEvent({ + event: 'terminal.catchup.stop_resume', + source: 'visible_first_audit_process_suspend', + browserExecutionStopped: true, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 5, + gapCount: 0, + }) + + expect(derived.terminalStoppedRetentionCoveredMs).toBe(900) + expect(derived.terminalStopResumeGapCount).toBe(0) + + replaceReconnectDerivedMetrics(candidate, 'desktop_local', derived) + + expect(evaluateVisibleFirstAuditGate(base, candidate)).toEqual({ + ok: true, + violations: [], + }) + }) + it('fails on a positive mobile_restricted focusedReadyMs delta', () => { const base = createArtifact() const candidate = createArtifact() @@ -265,4 +425,40 @@ describe('evaluateVisibleFirstAuditGate', () => { ], }) }) + + it('prints validation errors when the gate CLI rejects missing required metrics', async () => { + const base = createArtifact() + const candidate = createArtifact() + const scenario = getScenario(candidate, 'terminal-reconnect-backlog') + delete (scenario.samples[0].derived as Record).terminalParserAppliedLagMs + delete (scenario.summaryByProfile.desktop_local as Record).terminalParserAppliedLagMs + + const { tempDir, basePath, candidatePath } = await writeArtifacts(base, candidate) + tempDirs.add(tempDir) + + const result = await execFileAsync( + process.execPath, + [ + require.resolve('tsx/cli'), + path.resolve(process.cwd(), 'scripts/assert-visible-first-audit-gate.ts'), + '--base', + basePath, + '--candidate', + candidatePath, + ], + { + cwd: process.cwd(), + }, + ).catch((error: any) => error) + + expect(result.code).toBe(1) + expect(result.stderr).toBe('') + expect(JSON.parse(result.stdout)).toEqual({ + ok: false, + validationErrors: [ + expect.stringMatching(/terminal-reconnect-backlog\/desktop_local:terminalParserAppliedLagMs/), + ], + violations: [], + }) + }) }) diff --git a/test/unit/lib/visible-first-audit-run-sample.test.ts b/test/unit/lib/visible-first-audit-run-sample.test.ts index 4da808cb8..be36bbcc3 100644 --- a/test/unit/lib/visible-first-audit-run-sample.test.ts +++ b/test/unit/lib/visible-first-audit-run-sample.test.ts @@ -2,6 +2,82 @@ import { describe, expect, it } from 'vitest' import { runVisibleFirstAuditSample } from '@test/e2e-browser/perf/run-sample' +function validStopResumeEvent() { + return { + event: 'terminal.catchup.stop_resume', + timestamp: 90, + source: 'visible_first_audit_process_suspend', + browserExecutionStopped: true, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 5, + gapCount: 0, + } +} + +function createReconnectCollectors(input: { + perfEvents?: Array> +} = {}) { + return { + browser: { + milestones: { 'terminal.first_output': 100 }, + perfEvents: input.perfEvents ?? [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 16 }, + { event: 'terminal.parser_applied', timestamp: 40, parserAppliedSeq: 1 }, + validStopResumeEvent(), + ], + terminalLatencySamplesMs: [], + }, + transport: { + http: { requests: [] }, + ws: { + frames: [ + { + timestamp: 10, + direction: 'sent', + type: 'terminal.attach', + payload: JSON.stringify({ type: 'terminal.attach', terminalId: 'term-reconnect' }), + payloadLength: 80, + }, + { + timestamp: 30, + direction: 'received', + type: 'terminal.output.batch', + payload: JSON.stringify({ + type: 'terminal.output.batch', + source: 'replay', + terminalId: 'term-reconnect', + seqStart: 1, + seqEnd: 1, + serializedBytes: 120, + }), + payloadLength: 120, + }, + ], + }, + summary: { http: { byRoute: {} }, ws: { byType: {} } }, + }, + server: { + httpRequests: [], + perfEvents: [], + perfSystemSamples: [], + terminalReplayEvents: [ + { + event: 'terminal.replay.batch', + source: 'replay', + seqStart: 1, + seqEnd: 1, + serializedBytes: 120, + }, + ], + terminalOutputEvents: [], + parserDiagnostics: [], + }, + } +} + describe('runVisibleFirstAuditSample', () => { it('returns one schema-shaped sample with browser, transport, server, and derived data', async () => { const sample = await runVisibleFirstAuditSample({ @@ -23,6 +99,8 @@ describe('runVisibleFirstAuditSample', () => { httpRequests: [], perfEvents: [], perfSystemSamples: [], + terminalReplayEvents: [], + terminalOutputEvents: [], parserDiagnostics: [], }, }), @@ -35,4 +113,96 @@ describe('runVisibleFirstAuditSample', () => { expect(sample.server).toBeDefined() expect(sample.derived.focusedReadyMs).toBeTypeOf('number') }) + + it('populates reconnect backlog required metrics from websocket and replay evidence', async () => { + const sample = await runVisibleFirstAuditSample({ + scenarioId: 'terminal-reconnect-backlog', + profileId: 'desktop_local', + deps: { + executeSample: async () => createReconnectCollectors(), + }, + }) + + expect(sample.status).toBe('ok') + expect(sample.errors).toEqual([]) + expect(sample.derived).toEqual(expect.objectContaining({ + terminalInputToFirstOutputMs: 20, + terminalReplayMessageCount: 1, + terminalReplaySerializedBytes: 120, + terminalParserAppliedLagMs: 10, + terminalReplayGapCount: 0, + terminalFullHydrateFallbackCount: 0, + terminalSurfaceQuarantineCount: 0, + terminalStaleGenerationRejectionCount: 0, + terminalStoppedRetentionCoveredMs: 900, + terminalStopResumeGapCount: 0, + })) + }) + + it('fails reconnect backlog samples when stop/resume evidence is synthetic', async () => { + const sample = await runVisibleFirstAuditSample({ + scenarioId: 'terminal-reconnect-backlog', + profileId: 'desktop_local', + deps: { + executeSample: async () => createReconnectCollectors({ + perfEvents: [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 16 }, + { event: 'terminal.parser_applied', timestamp: 40, parserAppliedSeq: 1 }, + { + event: 'terminal.catchup.stop_resume', + timestamp: 90, + source: 'unit_reconnect_fixture', + browserExecutionStopped: false, + retentionCoveredMs: 900, + stoppedDurationMs: 1_200, + outputStartedAfterStopMs: 300, + outputStartedBeforeResumeMs: 900, + cdpCatchupOutputMessageCount: 5, + gapCount: 0, + }, + ], + }), + }, + }) + + expect(sample.status).toBe('error') + expect(sample.errors.join('\n')).toMatch(/terminalStoppedRetentionCoveredMs/) + expect(sample.errors.join('\n')).toMatch(/terminalStopResumeGapCount/) + }) + + it('fails reconnect backlog samples when RAF sampler evidence is missing', async () => { + const sample = await runVisibleFirstAuditSample({ + scenarioId: 'terminal-reconnect-backlog', + profileId: 'desktop_local', + deps: { + executeSample: async () => createReconnectCollectors({ + perfEvents: [ + { event: 'terminal.parser_applied', timestamp: 40, parserAppliedSeq: 1 }, + validStopResumeEvent(), + ], + }), + }, + }) + + expect(sample.status).toBe('error') + expect(sample.errors.join('\n')).toMatch(/maxRafGapMs/) + }) + + it('fails reconnect backlog samples when parser-applied evidence is missing', async () => { + const sample = await runVisibleFirstAuditSample({ + scenarioId: 'terminal-reconnect-backlog', + profileId: 'desktop_local', + deps: { + executeSample: async () => createReconnectCollectors({ + perfEvents: [ + { event: 'visible_first.audit.max_raf_gap', maxGapMs: 16 }, + validStopResumeEvent(), + ], + }), + }, + }) + + expect(sample.status).toBe('error') + expect(sample.errors.join('\n')).toMatch(/terminalParserAppliedLagMs/) + }) }) diff --git a/test/unit/lib/visible-first-audit-scenarios.test.ts b/test/unit/lib/visible-first-audit-scenarios.test.ts index 065f0f459..c0540e503 100644 --- a/test/unit/lib/visible-first-audit-scenarios.test.ts +++ b/test/unit/lib/visible-first-audit-scenarios.test.ts @@ -2,6 +2,18 @@ import { describe, expect, it } from 'vitest' import { AUDIT_SCENARIOS } from '@test/e2e-browser/perf/scenarios' +const TERMINAL_CATCHUP_REQUIRED_METRICS = [ + 'terminalReplayMessageCount', + 'terminalReplaySerializedBytes', + 'terminalParserAppliedLagMs', + 'terminalReplayGapCount', + 'terminalFullHydrateFallbackCount', + 'terminalSurfaceQuarantineCount', + 'terminalStaleGenerationRejectionCount', + 'terminalStoppedRetentionCoveredMs', + 'terminalStopResumeGapCount', +] + describe('visible-first audit scenarios', () => { it('defines the six accepted scenarios in stable order', () => { expect(AUDIT_SCENARIOS.map((scenario) => scenario.id)).toEqual([ @@ -53,4 +65,20 @@ describe('visible-first audit scenarios', () => { expect(scenario.allowedWsTypesBeforeReady.some((type) => forbiddenWsTypes.has(type))).toBe(false) } }) + + it('declares the terminal catch-up replay metrics required by the reconnect-backlog scenario', () => { + const scenarioMap = new Map(AUDIT_SCENARIOS.map((scenario) => [scenario.id, scenario])) + const scenario = scenarioMap.get('terminal-reconnect-backlog') + + expect(scenario?.allowedWsTypesBeforeReady).toEqual(expect.arrayContaining([ + 'terminal.output', + 'terminal.output.batch', + ])) + expect(scenario?.requiredMetricIds).toEqual(expect.arrayContaining([ + 'focusedReadyMs', + 'terminalInputToFirstOutputMs', + 'maxRafGapMs', + ...TERMINAL_CATCHUP_REQUIRED_METRICS, + ])) + }) }) diff --git a/test/unit/lib/visible-first-audit-server-log-parser.test.ts b/test/unit/lib/visible-first-audit-server-log-parser.test.ts index ab1263663..c4a2e08ff 100644 --- a/test/unit/lib/visible-first-audit-server-log-parser.test.ts +++ b/test/unit/lib/visible-first-audit-server-log-parser.test.ts @@ -22,6 +22,9 @@ describe('parseVisibleFirstServerLogs', () => { JSON.stringify({ event: 'http_request', path: '/api/settings' }), JSON.stringify({ event: 'perf_system', rssBytes: 123 }), JSON.stringify({ component: 'perf', event: 'http_request_slow', durationMs: 50 }), + JSON.stringify({ event: 'terminal.replay.batch', source: 'replay', serializedBytes: 456 }), + JSON.stringify({ event: 'terminal.replay.diagnostic', source: 'replay', serializedBytes: 999 }), + JSON.stringify({ event: 'terminal.output.gap', source: 'replay', fromSeq: 1, toSeq: 2 }), '{not-json', ].join('\n'), 'utf8', @@ -31,6 +34,12 @@ describe('parseVisibleFirstServerLogs', () => { expect(result.httpRequests.length).toBeGreaterThan(0) expect(result.perfEvents.length).toBeGreaterThan(0) expect(result.perfSystemSamples.length).toBeGreaterThan(0) + expect(result.terminalReplayEvents).toEqual([ + expect.objectContaining({ event: 'terminal.replay.batch', serializedBytes: 456 }), + ]) + expect(result.terminalOutputEvents).toEqual([ + expect.objectContaining({ event: 'terminal.output.gap', fromSeq: 1, toSeq: 2 }), + ]) expect(result.parserDiagnostics).toHaveLength(1) }) }) diff --git a/test/unit/server/terminal-stream/client-output-queue.test.ts b/test/unit/server/terminal-stream/client-output-queue.test.ts index 915047a61..f769a1723 100644 --- a/test/unit/server/terminal-stream/client-output-queue.test.ts +++ b/test/unit/server/terminal-stream/client-output-queue.test.ts @@ -1,14 +1,22 @@ import { describe, expect, it } from 'vitest' -import { ClientOutputQueue } from '../../../../server/terminal-stream/client-output-queue' +import { ClientOutputQueue, isGapEvent } from '../../../../server/terminal-stream/client-output-queue' +import { createTerminalOutputBarrierScanner } from '../../../../server/terminal-stream/output-barrier-scanner' import type { ReplayFrame } from '../../../../server/terminal-stream/replay-ring' -function frame(seq: number, data: string): ReplayFrame { +function frame(seq: number, data: string, streamId = 'stream-1'): ReplayFrame { + const scanner = createTerminalOutputBarrierScanner() + const classification = scanner.scan(data) return { seqStart: seq, seqEnd: seq, data, bytes: Buffer.byteLength(data, 'utf8'), at: seq, + streamId, + barrier: classification.barrier, + ...(classification.barrier ? { barrierReason: classification.reason } : {}), + scannerStateBefore: classification.stateBefore, + scannerStateAfter: classification.stateAfter, } } @@ -22,6 +30,22 @@ describe('ClientOutputQueue', () => { expect(queue.pendingBytes()).toBeLessThanOrEqual(5) }) + it('keeps a four-hour one-kib-per-second hidden-tab backlog by default', () => { + const queue = new ClientOutputQueue() + const frameCount = 4 * 60 * 60 + const queuedBytesPerSecond = 1024 + + for (let seq = 1; seq <= frameCount; seq += 1) { + queue.enqueue(frame(seq, `line-${seq}\n`), queuedBytesPerSecond) + } + + const prepared = queue.prepareBatch(64 * 1024) + + expect(queue.pendingFrames()).toBe(frameCount) + expect(queue.peekDroppedBytes()).toBe(0) + expect(prepared.entries.some(isGapEvent)).toBe(false) + }) + it('coalesces adjacent frames when queued', () => { const queue = new ClientOutputQueue(1024) queue.enqueue(frame(1, 'hello ')) @@ -36,6 +60,100 @@ describe('ClientOutputQueue', () => { }) }) + it('does not consume prepared frames until acknowledged', () => { + const queue = new ClientOutputQueue(1024) + queue.enqueue(frame(1, 'hello ')) + queue.enqueue(frame(2, 'world')) + + const prepared = queue.prepareBatch(1024) + + expect(prepared.frameCount).toBe(2) + expect(queue.pendingFrames()).toBe(2) + expect(queue.pendingBytes()).toBe(Buffer.byteLength('hello world', 'utf8')) + + queue.acknowledgePreparedBatch(prepared) + + expect(queue.pendingFrames()).toBe(0) + expect(queue.pendingBytes()).toBe(0) + }) + + it('keeps an unsent prepared suffix after partial acknowledgement', () => { + const queue = new ClientOutputQueue(1024) + queue.enqueue(frame(1, 'one')) + queue.enqueue(frame(2, 'two')) + queue.enqueue(frame(3, 'three')) + + const prepared = queue.prepareBatch(1024) + queue.acknowledgePreparedBatch(prepared, { frames: 2 }) + + expect(queue.pendingFrames()).toBe(1) + const retry = queue.nextBatch(1024) + const dataFrames = retry.filter((entry): entry is ReplayFrame => entry.type !== 'gap') + expect(dataFrames).toHaveLength(1) + expect(dataFrames[0]).toMatchObject({ + seqStart: 3, + seqEnd: 3, + data: 'three', + }) + }) + + it('keeps overflow gaps pending until acknowledged', () => { + const queue = new ClientOutputQueue(2) + queue.enqueue(frame(1, '1')) + queue.enqueue(frame(2, '2')) + queue.enqueue(frame(3, '3')) + + const prepared = queue.prepareBatch(64) + expect(prepared.entries[0]).toMatchObject({ type: 'gap', fromSeq: 1, toSeq: 1 }) + + const retryBeforeAck = queue.prepareBatch(64) + expect(retryBeforeAck.entries[0]).toMatchObject({ type: 'gap', fromSeq: 1, toSeq: 1 }) + + queue.acknowledgePreparedBatch(prepared, { gaps: 1, frames: 0 }) + const retryAfterGapAck = queue.nextBatch(64) + expect(retryAfterGapAck.some(isGapEvent)).toBe(false) + }) + + it('does not coalesce adjacent frames from different stream ids', () => { + const queue = new ClientOutputQueue(1024) + queue.enqueue(frame(1, 'old', 'stream-old')) + queue.enqueue(frame(2, 'new', 'stream-new')) + + const batch = queue.nextBatch(1024) + const dataFrames = batch.filter((entry): entry is ReplayFrame => entry.type !== 'gap') + + expect(dataFrames).toHaveLength(2) + expect(dataFrames[0]).toMatchObject({ + seqStart: 1, + seqEnd: 1, + data: 'old', + streamId: 'stream-old', + }) + expect(dataFrames[1]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + data: 'new', + streamId: 'stream-new', + }) + expect(queue.pendingBytes()).toBe(0) + }) + + it('does not coalesce adjacent frames across barrier metadata', () => { + const queue = new ClientOutputQueue(1024) + queue.enqueue(frame(1, 'before')) + queue.enqueue(frame(2, '\u001b[31m')) + queue.enqueue(frame(3, 'after')) + + const batch = queue.nextBatch(1024) + const dataFrames = batch.filter((entry): entry is ReplayFrame => entry.type !== 'gap') + + expect(dataFrames.map((entry) => entry.data)).toEqual([ + 'before', + '\u001b[31m', + 'after', + ]) + }) + it('drops oldest frames when queue overflows', () => { const queue = new ClientOutputQueue(2) queue.enqueue(frame(1, '1')) @@ -65,6 +183,7 @@ describe('ClientOutputQueue', () => { type: 'gap', fromSeq: 1, toSeq: 3, + streamId: 'stream-1', reason: 'queue_overflow', }) const dataFrames = batch.filter((entry): entry is ReplayFrame => entry.type !== 'gap') @@ -87,4 +206,69 @@ describe('ClientOutputQueue', () => { expect(queue.consumeDroppedBytes()).toBe(1) expect(queue.peekDroppedBytes()).toBe(0) }) + + it('splits overflow gaps at stream id boundaries', () => { + const queue = new ClientOutputQueue(1) + queue.enqueue(frame(1, '1', 'stream-old')) + queue.enqueue(frame(2, '2', 'stream-old')) + queue.enqueue(frame(3, '3', 'stream-new')) + queue.enqueue(frame(4, '4', 'stream-new')) + + const batch = queue.nextBatch(64) + const gaps = batch.filter(isGapEvent) + const dataFrames = batch.filter((entry): entry is ReplayFrame => entry.type !== 'gap') + + expect(gaps).toEqual([ + { + type: 'gap', + fromSeq: 1, + toSeq: 2, + streamId: 'stream-old', + reason: 'queue_overflow', + }, + { + type: 'gap', + fromSeq: 3, + toSeq: 3, + streamId: 'stream-new', + reason: 'queue_overflow', + }, + ]) + expect(dataFrames).toHaveLength(1) + expect(dataFrames[0]).toMatchObject({ + seqStart: 4, + seqEnd: 4, + streamId: 'stream-new', + }) + }) + + it('retags queued frames and pending gaps when stream identity rotates before flush', () => { + const queue = new ClientOutputQueue(2) + queue.enqueue(frame(1, '1', 'stream-old')) + queue.enqueue(frame(2, '2', 'stream-old')) + queue.enqueue(frame(3, '3', 'stream-old')) + + queue.retagPendingStream('stream-old', 'stream-new') + + const batch = queue.nextBatch(64) + const gaps = batch.filter(isGapEvent) + const dataFrames = batch.filter((entry): entry is ReplayFrame => entry.type !== 'gap') + + expect(gaps).toEqual([ + { + type: 'gap', + fromSeq: 1, + toSeq: 1, + streamId: 'stream-new', + reason: 'queue_overflow', + }, + ]) + expect(dataFrames).toHaveLength(1) + expect(dataFrames[0]).toMatchObject({ + seqStart: 2, + seqEnd: 3, + data: '23', + streamId: 'stream-new', + }) + }) }) diff --git a/test/unit/server/terminal-stream/output-barrier-scanner.test.ts b/test/unit/server/terminal-stream/output-barrier-scanner.test.ts new file mode 100644 index 000000000..478e16d6a --- /dev/null +++ b/test/unit/server/terminal-stream/output-barrier-scanner.test.ts @@ -0,0 +1,190 @@ +import { describe, expect, it } from 'vitest' +import { createTerminalOutputBarrierScanner } from '../../../../server/terminal-stream/output-barrier-scanner' + +describe('terminal output barrier scanner', () => { + it('treats plain printable text and newlines as transparent', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('hello\nworld\r\n')).toMatchObject({ + barrier: false, + ground: true, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('treats escape and control sequences as barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b[31mred')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('treats BEL as a turn-complete-sensitive barrier', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u0007')).toMatchObject({ + barrier: true, + reason: 'turn_complete', + ground: true, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('treats OSC sequences as OSC52-sensitive barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b]52;c;SGVsbG8=\u0007')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: true, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('carries pending CSI state across fragments', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b[')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'csi' }, + }) + expect(scanner.scan('6n')).toMatchObject({ + barrier: true, + reason: 'request_mode', + ground: true, + stateBefore: { mode: 'csi' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('keeps ESC intermediate sequences pending across fragments', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b(')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'esc' }, + }) + expect(scanner.scan('0')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + stateBefore: { mode: 'esc' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('keeps SOS and PM string controls pending until ST', () => { + const sosScanner = createTerminalOutputBarrierScanner() + expect(sosScanner.scan('\u001bXopaque')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'apc' }, + }) + expect(sosScanner.scan('\u001b\\')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + stateBefore: { mode: 'apc' }, + stateAfter: { mode: 'ground' }, + }) + + const pmScanner = createTerminalOutputBarrierScanner() + expect(pmScanner.scan('\u009epending')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'apc' }, + }) + expect(pmScanner.scan('\u009c')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + stateBefore: { mode: 'apc' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('keeps large unterminated CSI streams bounded and fail-closed', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b[')).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateAfter: { mode: 'csi' }, + }) + expect(scanner.scan('1'.repeat(100_000))).toMatchObject({ + barrier: true, + reason: 'control', + ground: false, + stateBefore: { mode: 'csi' }, + stateAfter: { mode: 'csi' }, + }) + expect(scanner.scan('6n')).toMatchObject({ + barrier: true, + reason: 'request_mode', + ground: true, + stateBefore: { mode: 'csi' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('carries pending OSC state across fragments', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\u001b]52;c;')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: false, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'osc' }, + }) + expect(scanner.scan('SGVsbG8=\u0007')).toMatchObject({ + barrier: true, + reason: 'osc52', + ground: true, + stateBefore: { mode: 'osc' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('treats replacement characters from lossy PTY decoding as barriers', () => { + const scanner = createTerminalOutputBarrierScanner() + + expect(scanner.scan('\ufffd')).toMatchObject({ + barrier: true, + reason: 'control', + ground: true, + stateBefore: { mode: 'ground' }, + stateAfter: { mode: 'ground' }, + }) + }) + + it('returns scanner state snapshots that can be stored on retained frames', () => { + const scanner = createTerminalOutputBarrierScanner() + const first = scanner.scan('\u001b[') + const second = scanner.scan('6n') + + expect(first.stateBefore).toEqual({ mode: 'ground' }) + expect(first.stateAfter).toEqual({ mode: 'csi' }) + expect(second.stateBefore).toEqual({ mode: 'csi' }) + expect(second.stateAfter).toEqual({ mode: 'ground' }) + }) +}) diff --git a/test/unit/server/terminal-stream/output-batch.test.ts b/test/unit/server/terminal-stream/output-batch.test.ts new file mode 100644 index 000000000..7cbeda886 --- /dev/null +++ b/test/unit/server/terminal-stream/output-batch.test.ts @@ -0,0 +1,229 @@ +import { describe, expect, it } from 'vitest' +import { buildTerminalOutputBatches } from '../../../../server/terminal-stream/output-batch' +import type { + TerminalOutputBarrierReason, + TerminalOutputScannerState, +} from '../../../../server/terminal-stream/output-barrier-scanner' +import { measureTerminalOutputPayloadBytes } from '../../../../server/terminal-stream/serialized-budget' +import type { ReplayFrame } from '../../../../server/terminal-stream/replay-ring' + +type TestFrame = ReplayFrame & { + barrier: boolean + barrierReason?: TerminalOutputBarrierReason + scannerStateBefore: TerminalOutputScannerState + scannerStateAfter: TerminalOutputScannerState +} + +const GROUND: TerminalOutputScannerState = { mode: 'ground' } + +function transparentFrame(seq: number, data: string, streamId = 'stream-1'): TestFrame { + return { + seqStart: seq, + seqEnd: seq, + data, + bytes: Buffer.byteLength(data, 'utf8'), + at: seq, + streamId, + barrier: false, + scannerStateBefore: GROUND, + scannerStateAfter: GROUND, + } +} + +function barrierFrame( + seq: number, + data: string, + barrierReason: TerminalOutputBarrierReason, + streamId = 'stream-1', +): TestFrame { + return { + ...transparentFrame(seq, data, streamId), + barrier: true, + barrierReason, + } +} + +function build(frames: TestFrame[], maxSerializedBytes = 16 * 1024) { + return buildTerminalOutputBatches({ + terminalId: 'term-1', + attachRequestId: 'attach-1', + source: 'replay', + maxSerializedBytes, + frames, + }) +} + +describe('terminal output batch builder', () => { + it('coalesces contiguous transparent frames under the serialized budget and emits segment metadata', () => { + const batches = build([ + transparentFrame(1, 'a'), + transparentFrame(2, 'b'), + ]) + + expect(batches).toHaveLength(1) + expect(batches[0]).toMatchObject({ + seqStart: 1, + seqEnd: 2, + data: 'ab', + streamId: 'stream-1', + attachRequestId: 'attach-1', + source: 'replay', + barrier: false, + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'ground' }, + }) + expect(batches[0].segments).toEqual([ + { + seqStart: 1, + seqEnd: 1, + streamId: 'stream-1', + offset: 0, + endOffset: 1, + bytes: 1, + barrier: false, + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'ground' }, + }, + { + seqStart: 2, + seqEnd: 2, + streamId: 'stream-1', + offset: 1, + endOffset: 2, + bytes: 1, + barrier: false, + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'ground' }, + }, + ]) + }) + + it('does not coalesce across parser barriers', () => { + const batches = build([ + transparentFrame(1, 'a'), + barrierFrame(2, '\u0007', 'turn_complete'), + transparentFrame(3, 'b'), + ]) + + expect(batches.map((batch) => batch.data)).toEqual(['a', '\u0007', 'b']) + expect(batches.map((batch) => batch.seqStart)).toEqual([1, 2, 3]) + expect(batches[1]).toMatchObject({ + barrier: true, + barrierReason: 'turn_complete', + segments: [ + { + seqStart: 2, + seqEnd: 2, + streamId: 'stream-1', + offset: 0, + endOffset: 1, + barrier: true, + barrierReason: 'turn_complete', + }, + ], + }) + }) + + it('does not coalesce across stream boundaries', () => { + const batches = build([ + transparentFrame(1, 'old', 'stream-old'), + transparentFrame(2, 'new', 'stream-new'), + ]) + + expect(batches).toHaveLength(2) + expect(batches[0]).toMatchObject({ + seqStart: 1, + seqEnd: 1, + data: 'old', + streamId: 'stream-old', + }) + expect(batches[1]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + data: 'new', + streamId: 'stream-new', + }) + }) + + it('uses UTF-16 code-unit segment offsets on code-point boundaries', () => { + const batches = build([ + transparentFrame(1, '😀'), + transparentFrame(2, 'b'), + ]) + + expect(batches).toHaveLength(1) + expect(batches[0].data).toBe('😀b') + expect(batches[0].segments).toMatchObject([ + { seqStart: 1, seqEnd: 1, offset: 0, endOffset: 2 }, + { seqStart: 2, seqEnd: 2, offset: 2, endOffset: 3 }, + ]) + }) + + it('uses stored scanner metadata instead of rescanning retained windows from ground', () => { + const batches = build([ + { + ...transparentFrame(2, '6n'), + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: { mode: 'csi' }, + scannerStateAfter: { mode: 'ground' }, + }, + transparentFrame(3, 'after'), + ]) + + expect(batches.map((batch) => batch.data)).toEqual(['6n', 'after']) + expect(batches[0]).toMatchObject({ + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: { mode: 'csi' }, + scannerStateAfter: { mode: 'ground' }, + }) + }) + + it('does not re-coalesce serialized-budget fragments across control barriers and keeps batches within budget', () => { + const frames = Array.from({ length: 8 }, (_unused, index) => barrierFrame( + index + 1, + '\u001b'.repeat(2048), + 'control', + )) + const maxSerializedBytes = 16 * 1024 + + const batches = build(frames, maxSerializedBytes) + + expect(batches).toHaveLength(frames.length) + expect(new Set(batches.map((batch) => `${batch.seqStart}:${batch.seqEnd}`)).size) + .toBe(batches.length) + expect(batches.every((batch) => batch.legacyOutputSerializedBytes <= maxSerializedBytes)).toBe(true) + expect(batches.every((batch) => + measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-1', + streamId: batch.streamId, + seqStart: batch.seqStart, + seqEnd: batch.seqEnd, + data: batch.data, + attachRequestId: 'attach-1', + }) <= maxSerializedBytes, + )).toBe(true) + }) + + it('coalesces many small transparent frames without changing segment offsets', () => { + const frames = Array.from({ length: 4096 }, (_unused, index) => transparentFrame(index + 1, 'x')) + + const batches = build(frames) + + expect(batches).toHaveLength(1) + expect(batches[0]).toMatchObject({ + seqStart: 1, + seqEnd: 4096, + }) + expect(batches[0].data).toHaveLength(4096) + expect(batches[0].segments).toHaveLength(4096) + expect(batches[0].segments[4095]).toMatchObject({ + seqStart: 4096, + seqEnd: 4096, + offset: 4095, + endOffset: 4096, + }) + }) +}) diff --git a/test/unit/server/terminal-stream/output-fragments.test.ts b/test/unit/server/terminal-stream/output-fragments.test.ts new file mode 100644 index 000000000..dfb7dd94c --- /dev/null +++ b/test/unit/server/terminal-stream/output-fragments.test.ts @@ -0,0 +1,111 @@ +import { describe, expect, it } from 'vitest' +import { measureTerminalOutputPayloadBytes } from '../../../../server/terminal-stream/serialized-budget' +import { + containsLoneSurrogate, + fragmentTerminalOutputForPayloadBudget, +} from '../../../../server/terminal-stream/output-fragments' + +describe('terminal output fragmentation', () => { + it('fragments escaped output before sequence assignment so every payload fits the budget', () => { + const data = '\u001b'.repeat(16 * 1024) + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 16 * 1024, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + expect(chunks.length).toBeGreaterThan(1) + for (const chunk of chunks) { + expect(measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + })).toBeLessThanOrEqual(16 * 1024) + } + expect(chunks.join('')).toBe(data) + }) + + it('does not split surrogate pairs', () => { + const data = `prefix-${'😀'.repeat(2048)}-suffix` + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 2048, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + expect(chunks.join('')).toBe(data) + expect(chunks.every((chunk) => !containsLoneSurrogate(chunk))).toBe(true) + }) + + it('preserves lone surrogates already present in string-mode output', () => { + const data = `prefix-\uD800-middle-\uDC00-suffix` + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 128, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + expect(chunks.join('')).toBe(data) + expect(chunks.some((chunk) => containsLoneSurrogate(chunk))).toBe(true) + }) + + it('throws when the budget is too small for one code point', () => { + expect(() => fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 1, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + }), + data: 'x', + })).toThrow(/too small for one code point/) + }) + + it('preserves replacement characters emitted by current string-mode PTY decoding', () => { + const data = `prefix-\uFFFD-\uFFFD-suffix` + const chunks = fragmentTerminalOutputForPayloadBudget({ + maxSerializedBytes: 2048, + payloadForData: (chunk) => ({ + type: 'terminal.output', + terminalId: 'term-1', + data: chunk, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }), + data, + }) + + // node-pty currently delivers terminal output as JavaScript strings. Invalid + // UTF-8 and raw 8-bit C1 bytes are already represented as replacement + // characters by this point, so Task 5 preserves them without claiming + // byte-perfect replay. + expect(chunks.join('')).toBe(data) + }) +}) diff --git a/test/unit/server/terminal-stream/replay-deque.test.ts b/test/unit/server/terminal-stream/replay-deque.test.ts new file mode 100644 index 000000000..6bb6d9820 --- /dev/null +++ b/test/unit/server/terminal-stream/replay-deque.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it, vi } from 'vitest' +import { ReplayDeque } from '../../../../server/terminal-stream/replay-deque' + +const STREAM_ID = 'stream-1' +const GROUND = { mode: 'ground' } as const +const CSI = { mode: 'csi' } as const + +describe('ReplayDeque', () => { + it('evicts many tiny frames without shifting the backing array per frame', () => { + const deque = new ReplayDeque(1024) + const shiftSpy = vi.spyOn(Array.prototype, 'shift') + let shiftCalls = 0 + + try { + for (let index = 0; index < 4096; index += 1) { + deque.append('x') + } + shiftCalls = shiftSpy.mock.calls.length + } finally { + shiftSpy.mockRestore() + } + + expect(shiftCalls).toBe(0) + expect(deque.totalBytes()).toBeLessThanOrEqual(1024) + expect(deque.headSeq()).toBe(4096) + expect(deque.tailSeq()).toBeGreaterThan(1) + }) + + it('reports a gap after eviction while preserving retained frames', () => { + const deque = new ReplayDeque(3) + deque.append('a') + deque.append('b') + deque.append('c') + deque.append('d') + + const replay = deque.replayBatchSince(0, 1024, 4) + + expect(replay.missedFromSeq).toBe(1) + expect(replay.frames.map((frame) => frame.data).join('')).toBe('bcd') + expect(replay.frames.at(-1)?.seqEnd).toBe(4) + }) + + it('preserves barrier metadata for arbitrary replay windows', () => { + const deque = new ReplayDeque(1024) + deque.append({ + data: '\u001b[', + streamId: STREAM_ID, + barrier: true, + barrierReason: 'control', + scannerStateBefore: GROUND, + scannerStateAfter: CSI, + }) + deque.append({ + data: '6n', + streamId: STREAM_ID, + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: CSI, + scannerStateAfter: GROUND, + }) + + const replay = deque.replayBatchSince(1, 1024, 2) + + expect(replay.frames).toHaveLength(1) + expect(replay.frames[0]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + data: '6n', + streamId: STREAM_ID, + barrier: true, + barrierReason: 'request_mode', + scannerStateBefore: CSI, + scannerStateAfter: GROUND, + }) + }) +}) diff --git a/test/unit/server/terminal-stream/replay-ring.test.ts b/test/unit/server/terminal-stream/replay-ring.test.ts index d2486a472..cf5c2e266 100644 --- a/test/unit/server/terminal-stream/replay-ring.test.ts +++ b/test/unit/server/terminal-stream/replay-ring.test.ts @@ -4,6 +4,12 @@ import { ReplayRing, } from '../../../../server/terminal-stream/replay-ring' +const STREAM_ID = 'stream-1' + +function append(ring: ReplayRing, data: string, streamId = STREAM_ID) { + return ring.append(data, { streamId }) +} + describe('ReplayRing', () => { const originalMaxBytes = process.env.TERMINAL_REPLAY_RING_MAX_BYTES @@ -17,9 +23,9 @@ describe('ReplayRing', () => { it('assigns monotonic sequence numbers starting at 1', () => { const ring = new ReplayRing(1024) - const one = ring.append('a') - const two = ring.append('b') - const three = ring.append('c') + const one = append(ring, 'a') + const two = append(ring, 'b') + const three = append(ring, 'c') expect(one.seqStart).toBe(1) expect(one.seqEnd).toBe(1) @@ -31,9 +37,9 @@ describe('ReplayRing', () => { it('evicts oldest frames to enforce byte budget', () => { const ring = new ReplayRing(5) - ring.append('abc') // 3 - ring.append('de') // 2 (total 5) - ring.append('f') // 1 (evict seq 1) + append(ring, 'abc') // 3 + append(ring, 'de') // 2 (total 5) + append(ring, 'f') // 1 (evict seq 1) expect(ring.headSeq()).toBe(3) expect(ring.tailSeq()).toBe(2) @@ -43,9 +49,9 @@ describe('ReplayRing', () => { it('replays only frames newer than sinceSeq', () => { const ring = new ReplayRing(1024) - ring.append('a') - ring.append('b') - ring.append('c') + append(ring, 'a') + append(ring, 'b') + append(ring, 'c') const replay = ring.replaySince(1) expect(replay.frames.map((f) => f.data)).toEqual(['b', 'c']) @@ -55,10 +61,10 @@ describe('ReplayRing', () => { it('returns coalesced bounded replay batches without materializing the full replay window', () => { const ring = new ReplayRing(1024) - ring.append('aa') - ring.append('bb') - ring.append('cc') - ring.append('dd') + append(ring, 'aa') + append(ring, 'bb') + append(ring, 'cc') + append(ring, 'dd') const firstBatch = ring.replayBatchSince(0, 4, 4) expect(firstBatch.frames).toHaveLength(1) @@ -82,9 +88,9 @@ describe('ReplayRing', () => { it('splits coalesced replay batches at the byte budget', () => { const ring = new ReplayRing(1024) - ring.append('aaa') - ring.append('bbb') - ring.append('ccc') + append(ring, 'aaa') + append(ring, 'bbb') + append(ring, 'ccc') const firstBatch = ring.replayBatchSince(0, 6, 3) expect(firstBatch.frames).toHaveLength(1) @@ -105,13 +111,91 @@ describe('ReplayRing', () => { }) }) + it('does not coalesce replay batches across retained parser barriers', () => { + const ring = new ReplayRing(1024) + append(ring, 'before') + append(ring, '\u001b[31m') + append(ring, 'after') + + const batch = ring.replayBatchSince(0, 1024, 3) + + expect(batch.frames.map((frame) => frame.data)).toEqual([ + 'before', + '\u001b[31m', + 'after', + ]) + expect(batch.frames[1]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + barrier: true, + barrierReason: 'control', + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'ground' }, + }) + }) + + it('does not re-coalesce retained fragments into oversized serialized payloads', () => { + const ring = new ReplayRing(1024) + append(ring, 'a'.repeat(60)) + append(ring, 'b'.repeat(60)) + + const measureSerializedPayload = (frame: { data: string }) => 100 + frame.data.length + const firstBatch = ring.replayBatchSince(0, 170, 2, measureSerializedPayload) + + expect(firstBatch.frames).toHaveLength(1) + expect(firstBatch.frames[0]).toMatchObject({ + seqStart: 1, + seqEnd: 1, + data: 'a'.repeat(60), + }) + + const secondBatch = ring.replayBatchSince(firstBatch.frames[0].seqEnd, 170, 2, measureSerializedPayload) + expect(secondBatch.frames).toHaveLength(1) + expect(secondBatch.frames[0]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + data: 'b'.repeat(60), + }) + }) + + it('does not coalesce adjacent replay frames from different stream ids', () => { + const ring = new ReplayRing(1024) + append(ring, 'old', 'stream-old') + append(ring, 'new', 'stream-new') + + const limitedBatch = ring.replayBatchSince(0, 3, 2) + expect(limitedBatch.frames).toHaveLength(1) + expect(limitedBatch.frames[0]).toMatchObject({ + seqStart: 1, + seqEnd: 1, + data: 'old', + streamId: 'stream-old', + }) + + const batch = ring.replayBatchSince(0, 1024, 2) + + expect(batch.frames).toHaveLength(2) + expect(batch.frames[0]).toMatchObject({ + seqStart: 1, + seqEnd: 1, + data: 'old', + streamId: 'stream-old', + }) + expect(batch.frames[1]).toMatchObject({ + seqStart: 2, + seqEnd: 2, + data: 'new', + streamId: 'stream-new', + }) + }) + it('reports replay miss when requested sequence is older than tail', () => { const ring = new ReplayRing(2) - ring.append('1') - ring.append('2') - ring.append('3') - ring.append('4') - ring.append('5') + append(ring, '1') + append(ring, '2') + append(ring, '3') + append(ring, '4') + append(ring, '5') expect(ring.headSeq()).toBe(5) expect(ring.tailSeq()).toBe(4) @@ -126,20 +210,20 @@ describe('ReplayRing', () => { const ring = new ReplayRing() const half = 'x'.repeat(DEFAULT_TERMINAL_REPLAY_RING_MAX_BYTES / 2) - ring.append(half) - ring.append(half) + append(ring, half) + append(ring, half) expect(ring.tailSeq()).toBe(1) - ring.append('y') + append(ring, 'y') expect(ring.headSeq()).toBe(3) expect(ring.tailSeq()).toBe(2) }) it('supports runtime max-byte resize and re-evicts to the new budget', () => { const ring = new ReplayRing(1024) - ring.append('x'.repeat(300)) - ring.append('y'.repeat(300)) - ring.append('z'.repeat(300)) + append(ring, 'x'.repeat(300)) + append(ring, 'y'.repeat(300)) + append(ring, 'z'.repeat(300)) ring.setMaxBytes(400) @@ -150,7 +234,7 @@ describe('ReplayRing', () => { it('retains truncated tail bytes when a single append exceeds maxBytes', () => { const ring = new ReplayRing(8) - ring.append('0123456789') + append(ring, '0123456789') const replay = ring.replaySince(0) expect(replay.frames).toHaveLength(1) @@ -159,9 +243,48 @@ describe('ReplayRing', () => { expect(replay.missedFromSeq).toBeUndefined() }) + it('marks a retained tail truncated inside OSC as fail-closed barrier metadata', () => { + const ring = new ReplayRing(8) + const frame = append(ring, `\u001b]52;c;${'A'.repeat(32)}`) + + expect(frame.data).toBe('A'.repeat(8)) + expect(frame).toMatchObject({ + barrier: true, + barrierReason: 'osc52', + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'osc' }, + }) + }) + + it('keeps retained tails truncated inside CSI from batching as transparent text', () => { + const ring = new ReplayRing(8) + append(ring, `\u001b[${'1'.repeat(32)}`) + ring.setMaxBytes(1024) + append(ring, 'after') + + const replay = ring.replaySince(0) + expect(replay.frames[0]).toMatchObject({ + data: '1'.repeat(8), + barrier: true, + barrierReason: 'control', + scannerStateBefore: { mode: 'ground' }, + scannerStateAfter: { mode: 'csi' }, + }) + expect(replay.frames[1]).toMatchObject({ + data: 'after', + barrier: true, + barrierReason: 'control', + scannerStateBefore: { mode: 'csi' }, + scannerStateAfter: { mode: 'ground' }, + }) + + const batch = ring.replayBatchSince(0, 1024, 2) + expect(batch.frames.map((frame) => frame.data)).toEqual(['1'.repeat(8), 'after']) + }) + it('truncates oversized multi-byte frames on UTF-8 boundaries', () => { const ring = new ReplayRing(7) - ring.append('🙂🙂🙂') + append(ring, '🙂🙂🙂') const replay = ring.replaySince(0) expect(replay.frames).toHaveLength(1) @@ -171,7 +294,7 @@ describe('ReplayRing', () => { it('preserves literal U+FFFD characters emitted by the source output', () => { const ring = new ReplayRing(4) - ring.append(`A\uFFFDB`) + append(ring, `A\uFFFDB`) const replay = ring.replaySince(0) expect(replay.frames).toHaveLength(1) @@ -181,10 +304,10 @@ describe('ReplayRing', () => { it('keeps the current head anchor stable after older frames overflow out of the replay window', () => { const ring = new ReplayRing(3) - ring.append('a') - ring.append('b') - ring.append('c') - ring.append('d') + append(ring, 'a') + append(ring, 'b') + append(ring, 'c') + append(ring, 'd') expect(ring.headSeq()).toBe(4) expect(ring.tailSeq()).toBe(2) diff --git a/test/unit/server/terminal-stream/serialized-budget.test.ts b/test/unit/server/terminal-stream/serialized-budget.test.ts new file mode 100644 index 000000000..94b4fb327 --- /dev/null +++ b/test/unit/server/terminal-stream/serialized-budget.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from 'vitest' +import { measureTerminalOutputPayloadBytes } from '../../../../server/terminal-stream/serialized-budget' + +describe('terminal stream serialized budget', () => { + it('measures escaped JSON bytes instead of raw data bytes', () => { + const data = '\u001b'.repeat(16 * 1024) + const bytes = measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-1', + data, + seqStart: 1, + seqEnd: 1, + attachRequestId: 'attach-1', + }) + + expect(Buffer.byteLength(data, 'utf8')).toBe(16 * 1024) + expect(bytes).toBeGreaterThan(16 * 1024) + }) +}) diff --git a/test/unit/server/terminal-stream/stream-identity.test.ts b/test/unit/server/terminal-stream/stream-identity.test.ts new file mode 100644 index 000000000..4e975973a --- /dev/null +++ b/test/unit/server/terminal-stream/stream-identity.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from 'vitest' +import { createTerminalStreamIdentityTracker } from '../../../../server/terminal-stream/stream-identity' + +describe('terminal stream identity', () => { + it('keeps stream id stable across attach and detach for the same output stream', () => { + const tracker = createTerminalStreamIdentityTracker() + const initial = tracker.ensureStream('term-1') + + expect(tracker.ensureStream('term-1')).toBe(initial) + tracker.recordAttach('term-1') + tracker.recordDetach('term-1') + + expect(tracker.ensureStream('term-1')).toBe(initial) + }) + + it('changes stream id on pty replacement and incompatible retention loss', () => { + const tracker = createTerminalStreamIdentityTracker() + const initial = tracker.ensureStream('term-1') + + const afterRecovery = tracker.replaceStream('term-1', 'codex_pty_recovery') + const afterRetentionLoss = tracker.replaceStream('term-1', 'retention_lost') + + expect(afterRecovery).not.toBe(initial) + expect(afterRetentionLoss).not.toBe(afterRecovery) + }) + + it('uses fixed-length opaque stream ids across replacements', () => { + const tracker = createTerminalStreamIdentityTracker() + const ids = [ + tracker.ensureStream('term-variable-length-name'), + tracker.replaceStream('term-variable-length-name', 'codex_pty_recovery'), + tracker.replaceStream('term-variable-length-name', 'retention_lost'), + ] + + expect(new Set(ids).size).toBe(ids.length) + expect(new Set(ids.map((id) => id.length)).size).toBe(1) + }) +}) diff --git a/test/unit/server/ws-handler-backpressure.test.ts b/test/unit/server/ws-handler-backpressure.test.ts index 3bfd0c8cb..cc2549e3f 100644 --- a/test/unit/server/ws-handler-backpressure.test.ts +++ b/test/unit/server/ws-handler-backpressure.test.ts @@ -6,8 +6,33 @@ import WebSocket from 'ws' import { WsHandler } from '../../../server/ws-handler' import { TerminalRegistry } from '../../../server/terminal-registry' import { TerminalStreamBroker } from '../../../server/terminal-stream/broker' +import { + measureTerminalOutputPayloadBytes, + TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, +} from '../../../server/terminal-stream/serialized-budget' import { MAX_REALTIME_MESSAGE_BYTES } from '../../../shared/read-models.js' +const loggerMocks = vi.hoisted(() => { + const logger = { + child: vi.fn(), + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + } + logger.child.mockReturnValue(logger) + return { logger } +}) + +vi.mock('../../../server/logger', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + logger: loggerMocks.logger, + sessionLifecycleLogger: loggerMocks.logger, + } +}) + vi.mock('node-pty', () => ({ spawn: vi.fn(), })) @@ -32,6 +57,16 @@ function createMockWs(overrides: Record = {}) { return ws } +function structuredLogs(level: 'debug' | 'info' | 'warn' | 'error', event: string) { + return loggerMocks.logger[level].mock.calls + .map(([payload]) => payload) + .filter((payload): payload is Record => ( + !!payload + && typeof payload === 'object' + && (payload as { event?: unknown }).event === event + )) +} + class FakeBrokerRegistry extends EventEmitter { private records = new Map string } }>() private replayRingMaxChars: number | undefined @@ -70,20 +105,35 @@ class FakeBrokerRegistry extends EventEmitter { } let originalAuthToken: string | undefined +let originalTerminalClientQueueMaxBytes: string | undefined beforeEach(() => { originalAuthToken = process.env.AUTH_TOKEN + originalTerminalClientQueueMaxBytes = process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES process.env.AUTH_TOKEN = TEST_AUTH_TOKEN + loggerMocks.logger.debug.mockClear() + loggerMocks.logger.info.mockClear() + loggerMocks.logger.warn.mockClear() + loggerMocks.logger.error.mockClear() }) afterEach(() => { if (originalAuthToken === undefined) { delete process.env.AUTH_TOKEN - return + } else { + process.env.AUTH_TOKEN = originalAuthToken + } + if (originalTerminalClientQueueMaxBytes === undefined) { + delete process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES + } else { + process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES = originalTerminalClientQueueMaxBytes } - process.env.AUTH_TOKEN = originalAuthToken }) +function forceSmallTerminalClientQueueForOverflowTest(): void { + process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES = String(128 * 1024) +} + describe('WsHandler backpressure', () => { let server: http.Server let handler: WsHandler @@ -241,7 +291,9 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { // Recover below threshold and allow queued frame to flush. ws.bufferedAmount = 0 vi.advanceTimersByTime(100) - expect(ws.send).toHaveBeenCalledWith(expect.stringContaining('"type":"terminal.output"')) + expect(ws.send.mock.calls.some(([raw]) => + typeof raw === 'string' && raw.includes('"type":"terminal.output"') + )).toBe(true) expect(perfSpy).not.toHaveBeenCalledWith('terminal_stream_catastrophic_close', expect.any(Object), expect.anything()) broker.close() @@ -315,7 +367,380 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { } }) + it('emits structured terminal.replay.batch logs for replay batch sends', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-structured-batch') + + for (let i = 1; i <= 8; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-structured-batch', + data: `batch-${i};`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 0 }) + await broker.attach( + wsReplay as any, + 'term-structured-batch', + 'transport_reconnect', + 80, + 24, + 0, + 'structured-batch-attach', + undefined, + 'foreground', + true, + ) + vi.advanceTimersByTime(5) + + expect(structuredLogs('debug', 'terminal.replay.batch')).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + event: 'terminal.replay.batch', + severity: 'debug', + terminalId: 'term-structured-batch', + attachRequestId: 'structured-batch-attach', + source: 'replay', + streamId: expect.any(String), + seqStart: 1, + seqEnd: 8, + rawFrameCount: 8, + dataBytes: expect.any(Number), + serializedBytes: expect.any(Number), + bufferedAmount: expect.any(Number), + }), + ]), + ) + + broker.close() + }) + + it('does not emit terminal.replay.batch logs for live output batches', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-batch-observability') + + const ws = createMockWs() + await broker.attach( + ws as any, + 'term-live-batch-observability', + 'viewport_hydrate', + 80, + 24, + 0, + 'live-batch-attach', + undefined, + 'foreground', + true, + ) + ws.send.mockClear() + loggerMocks.logger.debug.mockClear() + + registry.emit('terminal.output.raw', { + terminalId: 'term-live-batch-observability', + data: 'live batch payload', + at: Date.now(), + }) + vi.advanceTimersByTime(5) + + const liveBatches = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .filter((payload) => payload?.type === 'terminal.output.batch') + expect(liveBatches).toEqual([ + expect.objectContaining({ + terminalId: 'term-live-batch-observability', + attachRequestId: 'live-batch-attach', + source: 'live', + }), + ]) + expect(structuredLogs('debug', 'terminal.replay.batch') + .filter((payload) => payload.terminalId === 'term-live-batch-observability')).toHaveLength(0) + + broker.close() + }) + + it('retains unsent live output after a partial legacy batch send failure', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-partial-send') + + const ws = createMockWs() + await broker.attach( + ws as any, + 'term-live-partial-send', + 'viewport_hydrate', + 80, + 24, + 0, + 'live-partial-send-attach', + ) + + ws.send.mockClear() + const acceptedOutputPayloads: Array> = [] + let outputSendAttempts = 0 + ws.send.mockImplementation((raw: string, cb?: (err?: Error) => void) => { + const payload = JSON.parse(raw) + if (payload?.type === 'terminal.output') { + outputSendAttempts += 1 + if (outputSendAttempts === 2) { + throw new Error('simulated partial send failure') + } + acceptedOutputPayloads.push(payload) + } + cb?.() + }) + + registry.emit('terminal.output.raw', { terminalId: 'term-live-partial-send', data: 'one', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-live-partial-send', data: 'two', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-live-partial-send', data: 'three', at: Date.now() }) + + vi.advanceTimersByTime(1) + + expect(outputSendAttempts).toBe(2) + expect(acceptedOutputPayloads.map((payload) => payload.data)).toEqual(['one']) + expect(ws.close).not.toHaveBeenCalled() + + vi.advanceTimersByTime(50) + + expect(acceptedOutputPayloads.map((payload) => payload.data)).toEqual(['one', 'two', 'three']) + expect(acceptedOutputPayloads.every((payload) => payload.source === 'live')).toBe(true) + expect(ws.close).not.toHaveBeenCalled() + + broker.close() + }) + + it('emits structured terminal.replay.gap logs for replay gaps', async () => { + const originalRingMax = process.env.TERMINAL_REPLAY_RING_MAX_BYTES + process.env.TERMINAL_REPLAY_RING_MAX_BYTES = '8' + try { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-structured-gap') + + const wsSeed = createMockWs() + await broker.attach(wsSeed as any, 'term-structured-gap', 'viewport_hydrate', 80, 24, 0) + + registry.emit('terminal.output.raw', { terminalId: 'term-structured-gap', data: 'aaaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-structured-gap', data: 'bbbb', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-structured-gap', data: 'cccc', at: Date.now() }) + + const wsReplay = createMockWs() + await broker.attach( + wsReplay as any, + 'term-structured-gap', + 'viewport_hydrate', + 80, + 24, + 0, + 'structured-gap-attach', + ) + + expect(structuredLogs('warn', 'terminal.replay.gap')).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + event: 'terminal.replay.gap', + severity: 'warn', + terminalId: 'term-structured-gap', + attachRequestId: 'structured-gap-attach', + streamId: expect.any(String), + source: 'replay', + fromSeq: 1, + toSeq: 1, + reason: 'replay_window_exceeded', + }), + ]), + ) + + broker.close() + } finally { + if (originalRingMax === undefined) delete process.env.TERMINAL_REPLAY_RING_MAX_BYTES + else process.env.TERMINAL_REPLAY_RING_MAX_BYTES = originalRingMax + } + }) + + it('emits structured terminal.replay.backpressure_pause logs for replay pacing', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(4 * 1024 * 1024) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-structured-backpressure') + + for (let i = 1; i <= 10; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-structured-backpressure', + data: `paused-${i};${'x'.repeat(2 * 1024)}`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 768 * 1024 }) + await broker.attach( + wsReplay as any, + 'term-structured-backpressure', + 'transport_reconnect', + 80, + 24, + 0, + 'structured-backpressure-attach', + undefined, + 'foreground', + ) + vi.advanceTimersByTime(50) + + const pauseLog = structuredLogs('debug', 'terminal.replay.backpressure_pause') + .find((payload) => payload.terminalId === 'term-structured-backpressure') + expect(pauseLog).toEqual(expect.objectContaining({ + event: 'terminal.replay.backpressure_pause', + severity: 'debug', + terminalId: 'term-structured-backpressure', + attachRequestId: 'structured-backpressure-attach', + source: 'replay', + seqStart: 1, + seqEnd: 1, + rawFrameCount: 1, + dataBytes: expect.any(Number), + bufferedAmount: expect.any(Number), + threshold: expect.any(Number), + retryMs: expect.any(Number), + reason: 'websocket_buffered_amount', + })) + expect(pauseLog?.dataBytes).toBeGreaterThan(0) + + broker.close() + }) + + it('emits structured terminal.replay.retention logs when retention loss rotates stream identity', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-structured-retention') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-structured-retention', 'viewport_hydrate', 80, 24, 0, 'structured-retention-attach') + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + + registry.emit('terminal.output.raw', { terminalId: 'term-structured-retention', data: 'aaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-structured-retention', data: 'bbb', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-structured-retention', data: 'ccc', at: Date.now() }) + + const retentionLogs = structuredLogs('warn', 'terminal.replay.retention') + .filter((payload) => payload.terminalId === 'term-structured-retention') + expect(retentionLogs).toHaveLength(1) + expect(retentionLogs[0]).toEqual(expect.objectContaining({ + event: 'terminal.replay.retention', + severity: 'warn', + terminalId: 'term-structured-retention', + attachRequestIds: ['structured-retention-attach'], + attachmentCount: 1, + previousStreamId: ready.streamId, + streamId: expect.any(String), + reason: 'retention_lost', + retainedBytes: expect.any(Number), + maxBytes: 6, + tailSeq: expect.any(Number), + headSeq: expect.any(Number), + })) + expect(retentionLogs[0]?.attachRequestId).toBeUndefined() + + broker.close() + }) + + it('emits one aggregate terminal.replay.retention log for multiple attached clients', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-retention-multi-client') + + const wsA = createMockWs() + const wsB = createMockWs() + await broker.attach(wsA as any, 'term-retention-multi-client', 'viewport_hydrate', 80, 24, 0, 'retention-attach-a') + await broker.attach(wsB as any, 'term-retention-multi-client', 'viewport_hydrate', 80, 24, 0, 'retention-attach-b') + + registry.emit('terminal.output.raw', { terminalId: 'term-retention-multi-client', data: 'aaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-multi-client', data: 'bbb', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-multi-client', data: 'ccc', at: Date.now() }) + + const retentionLogs = structuredLogs('warn', 'terminal.replay.retention') + .filter((payload) => payload.terminalId === 'term-retention-multi-client') + expect(retentionLogs).toHaveLength(1) + expect(retentionLogs[0]).toEqual(expect.objectContaining({ + event: 'terminal.replay.retention', + severity: 'warn', + terminalId: 'term-retention-multi-client', + attachRequestIds: expect.arrayContaining(['retention-attach-a', 'retention-attach-b']), + attachmentCount: 2, + reason: 'retention_lost', + retainedBytes: expect.any(Number), + maxBytes: 6, + tailSeq: expect.any(Number), + headSeq: expect.any(Number), + })) + expect(retentionLogs[0]?.attachRequestIds).toHaveLength(2) + expect(retentionLogs[0]?.attachRequestId).toBeUndefined() + + broker.close() + }) + + it('rate limits structured terminal.replay.retention logs and reports suppressed losses', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-retention-rate-limit') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-retention-rate-limit', 'viewport_hydrate', 80, 24, 0, 'retention-rate-attach') + + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'aaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'bbb', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'ccc', at: Date.now() }) + + let retentionLogs = structuredLogs('warn', 'terminal.replay.retention') + .filter((payload) => payload.terminalId === 'term-retention-rate-limit') + expect(retentionLogs).toHaveLength(1) + expect(retentionLogs[0]).toEqual(expect.objectContaining({ + event: 'terminal.replay.retention', + severity: 'warn', + terminalId: 'term-retention-rate-limit', + attachRequestIds: ['retention-rate-attach'], + attachmentCount: 1, + reason: 'retention_lost', + })) + expect(retentionLogs[0]?.attachRequestId).toBeUndefined() + expect(retentionLogs[0]?.suppressedCount).toBeUndefined() + + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'ddd', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'eee', at: Date.now() }) + + retentionLogs = structuredLogs('warn', 'terminal.replay.retention') + .filter((payload) => payload.terminalId === 'term-retention-rate-limit') + expect(retentionLogs).toHaveLength(1) + + vi.advanceTimersByTime(1000) + registry.emit('terminal.output.raw', { terminalId: 'term-retention-rate-limit', data: 'fff', at: Date.now() }) + + retentionLogs = structuredLogs('warn', 'terminal.replay.retention') + .filter((payload) => payload.terminalId === 'term-retention-rate-limit') + expect(retentionLogs).toHaveLength(2) + expect(retentionLogs[1]).toEqual(expect.objectContaining({ + event: 'terminal.replay.retention', + severity: 'warn', + terminalId: 'term-retention-rate-limit', + attachRequestIds: ['retention-rate-attach'], + attachmentCount: 1, + reason: 'retention_lost', + suppressedCount: 2, + })) + expect(retentionLogs[1]?.attachRequestId).toBeUndefined() + + broker.close() + }) + it('echoes attachRequestId on attach.ready, output, and output.gap for a client attachment', async () => { + forceSmallTerminalClientQueueForOverflowTest() const registry = new FakeBrokerRegistry() const broker = new TerminalStreamBroker(registry as any, vi.fn()) registry.createTerminal('term-attach-id') @@ -375,6 +800,54 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { broker.close() }) + it('reports unknown geometry authority and ignores warm delta when another client is attached', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-geometry-authority') + + const wsA = createMockWs() + await broker.attach(wsA as any, 'term-geometry-authority', 'viewport_hydrate', 80, 24, 0, 'geometry-a-1') + registry.emit('terminal.output.raw', { + terminalId: 'term-geometry-authority', + data: 'geometry-seed', + at: Date.now(), + }) + vi.advanceTimersByTime(1) + + const wsB = createMockWs() + await broker.attach(wsB as any, 'term-geometry-authority', 'viewport_hydrate', 100, 30, 0, 'geometry-b-1') + const readyB = wsB.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(readyB).toMatchObject({ + terminalId: 'term-geometry-authority', + attachRequestId: 'geometry-b-1', + geometryAuthority: 'multi_client_unknown', + geometryEpoch: expect.any(Number), + }) + + wsA.send.mockClear() + await broker.attach(wsA as any, 'term-geometry-authority', 'transport_reconnect', 80, 24, 1, 'geometry-a-2') + const readyA2 = wsA.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + + expect(readyA2).toMatchObject({ + terminalId: 'term-geometry-authority', + attachRequestId: 'geometry-a-2', + geometryAuthority: 'multi_client_unknown', + geometryEpoch: expect.any(Number), + requestedSinceSeq: 1, + effectiveSinceSeq: 0, + replayResetReason: 'geometry_authority_unknown', + replayFromSeq: 1, + replayToSeq: 1, + }) + expect(readyA2.geometryEpoch).toBeGreaterThan(readyB.geometryEpoch) + + broker.close() + }) + it('keeps each live terminal.output frame within the shared realtime byte budget', async () => { const registry = new FakeBrokerRegistry() const broker = new TerminalStreamBroker(registry as any, vi.fn()) @@ -403,6 +876,625 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { broker.close() }) + it('keeps actual live and replay terminal.output JSON payloads within the serialized budget', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-serialized-budget') + + const wsLive = createMockWs() + await broker.attach( + wsLive as any, + 'term-serialized-budget', + 'viewport_hydrate', + 80, + 24, + 0, + TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + ) + + registry.emit('terminal.output.raw', { + terminalId: 'term-serialized-budget', + data: '\u001b'.repeat(20 * 1024), + at: Date.now(), + }) + for (let i = 0; i < 20; i += 1) { + vi.advanceTimersByTime(1) + } + + const liveOutputFrames = wsLive.send.mock.calls + .map(([raw]) => raw) + .filter((raw): raw is string => { + if (typeof raw !== 'string') return false + const payload = JSON.parse(raw) + return payload?.type === 'terminal.output' + }) + expect(liveOutputFrames.length).toBeGreaterThan(1) + + for (const raw of liveOutputFrames) { + const payload = JSON.parse(raw) + expect(Buffer.byteLength(raw, 'utf8')).toBeLessThanOrEqual(MAX_REALTIME_MESSAGE_BYTES) + expect(payload).toEqual(expect.objectContaining({ + type: 'terminal.output', + terminalId: 'term-serialized-budget', + attachRequestId: TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + streamId: expect.any(String), + seqStart: expect.any(Number), + seqEnd: expect.any(Number), + })) + } + + const wsReplay = createMockWs() + await broker.attach( + wsReplay as any, + 'term-serialized-budget', + 'transport_reconnect', + 80, + 24, + 0, + TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + ) + for (let i = 0; i < 20; i += 1) { + vi.advanceTimersByTime(1) + } + + const replayOutputFrames = wsReplay.send.mock.calls + .map(([raw]) => raw) + .filter((raw): raw is string => { + if (typeof raw !== 'string') return false + const payload = JSON.parse(raw) + return payload?.type === 'terminal.output' + }) + expect(replayOutputFrames.length).toBeGreaterThan(1) + + for (const raw of replayOutputFrames) { + const payload = JSON.parse(raw) + expect(Buffer.byteLength(raw, 'utf8')).toBeLessThanOrEqual(MAX_REALTIME_MESSAGE_BYTES) + expect(payload).toEqual(expect.objectContaining({ + type: 'terminal.output', + terminalId: 'term-serialized-budget', + attachRequestId: TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE, + streamId: expect.any(String), + seqStart: expect.any(Number), + seqEnd: expect.any(Number), + })) + } + + broker.close() + }) + + it('uses serialized terminal.output JSON bytes for maxReplayBytes truncation', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-replay-serialized-truncation') + + const wsSeed = createMockWs() + await broker.attach( + wsSeed as any, + 'term-replay-serialized-truncation', + 'viewport_hydrate', + 80, + 24, + 0, + 'seed-serialized-truncation', + ) + const seedReady = wsSeed.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(seedReady?.streamId).toEqual(expect.any(String)) + + const replayAttachRequestId = TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE + const chunks = [ + `A${'\u001b'.repeat(100)}`, + `B${'\u001b'.repeat(100)}`, + `C${'\u001b'.repeat(100)}`, + ] + for (const chunk of chunks) { + registry.emit('terminal.output.raw', { + terminalId: 'term-replay-serialized-truncation', + data: chunk, + at: Date.now(), + }) + } + + const oneSerializedPayloadBudget = measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId: 'term-replay-serialized-truncation', + streamId: seedReady.streamId, + seqStart: 3, + seqEnd: 3, + data: chunks[2], + attachRequestId: replayAttachRequestId, + source: 'replay', + }) + expect(chunks.reduce((sum, chunk) => sum + Buffer.byteLength(chunk, 'utf8'), 0)) + .toBeLessThan(oneSerializedPayloadBudget) + + const wsReplay = createMockWs() + await broker.attach( + wsReplay as any, + 'term-replay-serialized-truncation', + 'viewport_hydrate', + 80, + 24, + 0, + replayAttachRequestId, + oneSerializedPayloadBudget, + ) + vi.advanceTimersByTime(1) + + const replayMessages = wsReplay.send.mock.calls + .map(([raw]) => ({ + raw, + payload: typeof raw === 'string' ? JSON.parse(raw) : raw, + })) + + const gap = replayMessages.find(({ payload }) => payload?.type === 'terminal.output.gap')?.payload + expect(gap).toMatchObject({ + fromSeq: 1, + toSeq: 2, + reason: 'replay_budget_exceeded', + attachRequestId: replayAttachRequestId, + streamId: seedReady.streamId, + }) + + const outputFrames = replayMessages + .filter(({ payload }) => payload?.type === 'terminal.output') + expect(outputFrames.map(({ payload }) => payload.data)).toEqual([chunks[2]]) + expect(outputFrames[0]?.payload).toMatchObject({ + seqStart: 3, + seqEnd: 3, + attachRequestId: replayAttachRequestId, + streamId: seedReady.streamId, + source: 'replay', + }) + expect(Buffer.byteLength(String(outputFrames[0]?.raw ?? ''), 'utf8')) + .toBeLessThanOrEqual(oneSerializedPayloadBudget) + + broker.close() + }) + + it('emits separate queue overflow gaps for different stream ids', async () => { + const originalClientQueueMaxBytes = process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES + process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES = '2048' + + try { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-gap-stream') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-gap-stream', 'viewport_hydrate', 80, 24, 0, 'gap-attach') + const attachReady = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(attachReady?.streamId).toEqual(expect.any(String)) + + for (let i = 0; i < 3; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-gap-stream', + data: `old-${i}-${'o'.repeat(900)}`, + at: Date.now(), + }) + } + registry.emit('terminal.stream.replaced', { + terminalId: 'term-gap-stream', + reason: 'codex_pty_recovery', + }) + for (let i = 0; i < 3; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-gap-stream', + data: `new-${i}-${'n'.repeat(900)}`, + at: Date.now(), + }) + } + vi.advanceTimersByTime(5) + + const gaps = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .filter((payload) => payload?.type === 'terminal.output.gap') + + expect(gaps.length).toBeGreaterThanOrEqual(2) + expect(gaps[0]).toEqual(expect.objectContaining({ + streamId: attachReady.streamId, + reason: 'queue_overflow', + })) + expect(new Set(gaps.map((gap) => gap.streamId)).size).toBeGreaterThanOrEqual(2) + + broker.close() + } finally { + if (originalClientQueueMaxBytes === undefined) { + delete process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES + } else { + process.env.TERMINAL_CLIENT_QUEUE_MAX_BYTES = originalClientQueueMaxBytes + } + } + }) + + it('notifies active clients before live output switches to a replacement stream id', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-stream-change') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-live-stream-change', 'viewport_hydrate', 80, 24, 0, 'live-change-attach') + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + + registry.emit('terminal.output.raw', { terminalId: 'term-live-stream-change', data: 'before-change', at: Date.now() }) + vi.advanceTimersByTime(1) + + registry.emit('terminal.stream.replaced', { + terminalId: 'term-live-stream-change', + reason: 'codex_pty_recovery', + }) + registry.emit('terminal.output.raw', { terminalId: 'term-live-stream-change', data: 'after-change', at: Date.now() }) + vi.advanceTimersByTime(1) + + const payloads = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const streamChangedIndex = payloads.findIndex((payload) => payload?.type === 'terminal.stream.changed') + const afterOutputIndex = payloads.findIndex((payload) => + payload?.type === 'terminal.output' && payload.data === 'after-change' + ) + const streamChanged = payloads[streamChangedIndex] + const afterOutput = payloads[afterOutputIndex] + + expect(streamChanged).toMatchObject({ + terminalId: 'term-live-stream-change', + reason: 'codex_pty_recovery', + attachRequestId: 'live-change-attach', + streamId: expect.any(String), + }) + expect(streamChanged.streamId).not.toBe(ready.streamId) + expect(afterOutput).toMatchObject({ + terminalId: 'term-live-stream-change', + data: 'after-change', + streamId: streamChanged.streamId, + attachRequestId: 'live-change-attach', + }) + expect(streamChangedIndex).toBeGreaterThan(-1) + expect(afterOutputIndex).toBeGreaterThan(streamChangedIndex) + + broker.close() + }) + + it('converts a stale replay cursor to a current-stream gap before replacement live output', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-replay-stream-change') + + const seedWs = createMockWs() + await broker.attach(seedWs as any, 'term-replay-stream-change', 'viewport_hydrate', 80, 24, 0, 'seed-attach') + const initialReady = seedWs.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(initialReady?.streamId).toEqual(expect.any(String)) + + registry.emit('terminal.output.raw', { terminalId: 'term-replay-stream-change', data: 'old-a', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-replay-stream-change', data: 'old-b', at: Date.now() }) + + const replayWs = createMockWs() + await broker.attach( + replayWs as any, + 'term-replay-stream-change', + 'transport_reconnect', + 80, + 24, + 0, + 'replay-attach', + ) + + registry.emit('terminal.stream.replaced', { + terminalId: 'term-replay-stream-change', + reason: 'codex_pty_recovery', + }) + registry.emit('terminal.output.raw', { terminalId: 'term-replay-stream-change', data: 'new-live', at: Date.now() }) + vi.advanceTimersByTime(1) + + const payloads = replayWs.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const ready = payloads.find((payload) => payload?.type === 'terminal.attach.ready') + const streamChangedIndex = payloads.findIndex((payload) => payload?.type === 'terminal.stream.changed') + const gapIndex = payloads.findIndex((payload) => payload?.type === 'terminal.output.gap') + const newOutputIndex = payloads.findIndex((payload) => + payload?.type === 'terminal.output' && payload.data === 'new-live' + ) + const replayOutputs = payloads + .filter((payload) => payload?.type === 'terminal.output') + .map((payload) => payload.data) + + expect(ready).toMatchObject({ + terminalId: 'term-replay-stream-change', + streamId: initialReady.streamId, + replayFromSeq: 1, + replayToSeq: 2, + attachRequestId: 'replay-attach', + }) + expect(payloads[streamChangedIndex]).toMatchObject({ + terminalId: 'term-replay-stream-change', + reason: 'codex_pty_recovery', + attachRequestId: 'replay-attach', + streamId: expect.any(String), + }) + expect(payloads[streamChangedIndex].streamId).not.toBe(initialReady.streamId) + expect(payloads[gapIndex]).toMatchObject({ + terminalId: 'term-replay-stream-change', + streamId: payloads[streamChangedIndex].streamId, + fromSeq: 1, + toSeq: 2, + reason: 'replay_window_exceeded', + attachRequestId: 'replay-attach', + }) + expect(payloads[newOutputIndex]).toMatchObject({ + terminalId: 'term-replay-stream-change', + streamId: payloads[streamChangedIndex].streamId, + seqStart: 3, + seqEnd: 3, + data: 'new-live', + attachRequestId: 'replay-attach', + }) + expect(streamChangedIndex).toBeGreaterThan(-1) + expect(gapIndex).toBeGreaterThan(streamChangedIndex) + expect(newOutputIndex).toBeGreaterThan(gapIndex) + expect(replayOutputs).toEqual(['new-live']) + + broker.close() + }) + + it('notifies active clients when retention loss rotates live stream identity', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-retention-change') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-live-retention-change', 'viewport_hydrate', 80, 24, 0, 'live-retention-attach') + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-change', data: 'aaa', at: Date.now() }) + vi.advanceTimersByTime(1) + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-change', data: 'bbb', at: Date.now() }) + vi.advanceTimersByTime(1) + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-change', data: 'ccc', at: Date.now() }) + vi.advanceTimersByTime(1) + + const payloads = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const streamChangedIndex = payloads.findIndex((payload) => + payload?.type === 'terminal.stream.changed' && payload.reason === 'retention_lost' + ) + const cccOutputIndex = payloads.findIndex((payload) => + payload?.type === 'terminal.output' && payload.data === 'ccc' + ) + const streamChanged = payloads[streamChangedIndex] + const cccOutput = payloads[cccOutputIndex] + + expect(streamChanged).toMatchObject({ + terminalId: 'term-live-retention-change', + reason: 'retention_lost', + attachRequestId: 'live-retention-attach', + streamId: expect.any(String), + }) + expect(streamChanged.streamId).not.toBe(ready.streamId) + expect(cccOutput).toMatchObject({ + terminalId: 'term-live-retention-change', + data: 'ccc', + streamId: streamChanged.streamId, + attachRequestId: 'live-retention-attach', + }) + expect(cccOutputIndex).toBeGreaterThan(streamChangedIndex) + + broker.close() + }) + + it('retags queued live output when retention loss rotates stream identity before flush', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-retention-queued') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-live-retention-queued', 'viewport_hydrate', 80, 24, 0, 'live-retention-queued-attach') + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + ws.send.mockClear() + + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-queued', data: 'aaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-queued', data: 'bbb', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-live-retention-queued', data: 'ccc', at: Date.now() }) + vi.advanceTimersByTime(1) + + const payloads = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const streamChangedIndex = payloads.findIndex((payload) => + payload?.type === 'terminal.stream.changed' && payload.reason === 'retention_lost' + ) + const streamChanged = payloads[streamChangedIndex] + const outputs = payloads.filter((payload) => payload?.type === 'terminal.output') + + expect(streamChanged).toMatchObject({ + terminalId: 'term-live-retention-queued', + reason: 'retention_lost', + attachRequestId: 'live-retention-queued-attach', + streamId: expect.any(String), + }) + expect(streamChanged.streamId).not.toBe(ready.streamId) + expect(outputs.map((payload) => payload.data)).toEqual(['aaa', 'bbb', 'ccc']) + expect(outputs.every((payload) => payload.streamId === streamChanged.streamId)).toBe(true) + expect(outputs.every((payload) => payload.streamId !== ready.streamId)).toBe(true) + for (const output of outputs) { + expect(payloads.indexOf(output)).toBeGreaterThan(streamChangedIndex) + expect(output.attachRequestId).toBe('live-retention-queued-attach') + } + + broker.close() + }) + + it('retags returned live fragments when retention loss rotates stream identity before enqueue', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(64 * 1024) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-live-retention-fragments') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-live-retention-fragments', 'viewport_hydrate', 80, 24, 0, 'live-retention-fragments-attach') + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + ws.send.mockClear() + + registry.emit('terminal.output.raw', { + terminalId: 'term-live-retention-fragments', + data: 'x'.repeat(200 * 1024), + at: Date.now(), + }) + for (let i = 0; i < 100; i += 1) { + vi.advanceTimersByTime(1) + } + + const payloads = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const streamChanges = payloads.filter((payload) => + payload?.type === 'terminal.stream.changed' && payload.reason === 'retention_lost' + ) + const outputs = payloads.filter((payload) => payload?.type === 'terminal.output') + const finalStreamId = streamChanges.at(-1)?.streamId + + expect(streamChanges.length).toBeGreaterThan(0) + expect(finalStreamId).toEqual(expect.any(String)) + expect(finalStreamId).not.toBe(ready.streamId) + expect(outputs.length).toBeGreaterThan(0) + expect(outputs.every((payload) => payload.streamId === finalStreamId)).toBe(true) + expect(outputs.every((payload) => payload.streamId !== ready.streamId)).toBe(true) + expect(outputs.map((payload) => payload.data).join('')).toHaveLength(200 * 1024) + + broker.close() + }) + + it('retags retained replay frames when retention loss rotates stream identity', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(6) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-retention-stream') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-retention-stream', 'viewport_hydrate', 80, 24, 0, 'retention-attach') + const initialReady = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + const initialStreamId = initialReady?.streamId + expect(initialStreamId).toEqual(expect.any(String)) + + for (const data of ['aaa', 'bbb', 'ccc']) { + registry.emit('terminal.output.raw', { terminalId: 'term-retention-stream', data, at: Date.now() }) + vi.advanceTimersByTime(1) + } + + const wsAfterLoss = createMockWs() + await broker.attach( + wsAfterLoss as any, + 'term-retention-stream', + 'transport_reconnect', + 80, + 24, + 0, + 'after-retention-loss', + ) + vi.advanceTimersByTime(1) + + const payloadsAfterLoss = wsAfterLoss.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const readyAfterLoss = payloadsAfterLoss + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(readyAfterLoss?.streamId).toEqual(expect.any(String)) + expect(readyAfterLoss.streamId).not.toBe(initialStreamId) + + const gapAfterLoss = payloadsAfterLoss + .find((payload) => payload?.type === 'terminal.output.gap') + expect(gapAfterLoss).toMatchObject({ + streamId: readyAfterLoss.streamId, + fromSeq: 1, + toSeq: 1, + reason: 'replay_window_exceeded', + }) + + const replayOutputsAfterLoss = payloadsAfterLoss + .filter((payload) => payload?.type === 'terminal.output') + expect(replayOutputsAfterLoss.map((payload) => String(payload.data)).join('')).toBe('bbbccc') + expect(replayOutputsAfterLoss.every((payload) => payload.streamId === readyAfterLoss.streamId)).toBe(true) + expect(replayOutputsAfterLoss.every((payload) => payload.streamId !== initialStreamId)).toBe(true) + + broker.close() + }) + + it('gaps old-stream retained replay and only sends output for the attach-ready stream', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(9) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-retained-boundary') + + const ws = createMockWs() + await broker.attach(ws as any, 'term-retained-boundary', 'viewport_hydrate', 80, 24, 0, 'boundary-seed') + const initialReady = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(initialReady?.streamId).toEqual(expect.any(String)) + + registry.emit('terminal.output.raw', { terminalId: 'term-retained-boundary', data: 'aaa', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retained-boundary', data: 'bbb', at: Date.now() }) + registry.emit('terminal.stream.replaced', { + terminalId: 'term-retained-boundary', + reason: 'codex_pty_recovery', + }) + registry.emit('terminal.output.raw', { terminalId: 'term-retained-boundary', data: 'ccc', at: Date.now() }) + registry.emit('terminal.output.raw', { terminalId: 'term-retained-boundary', data: 'ddd', at: Date.now() }) + + const wsAfterLoss = createMockWs() + await broker.attach( + wsAfterLoss as any, + 'term-retained-boundary', + 'transport_reconnect', + 80, + 24, + 0, + 'boundary-after-loss', + ) + vi.advanceTimersByTime(1) + + const payloadsAfterLoss = wsAfterLoss.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const readyAfterLoss = payloadsAfterLoss + .find((payload) => payload?.type === 'terminal.attach.ready') + const replayOutputsAfterLoss = payloadsAfterLoss + .filter((payload) => payload?.type === 'terminal.output') + const replayGapsAfterLoss = payloadsAfterLoss + .filter((payload) => payload?.type === 'terminal.output.gap') + + expect(readyAfterLoss?.streamId).toEqual(expect.any(String)) + expect(readyAfterLoss.streamId).not.toBe(initialReady.streamId) + expect(replayGapsAfterLoss).toEqual([ + expect.objectContaining({ + streamId: readyAfterLoss.streamId, + fromSeq: 1, + toSeq: 2, + reason: 'replay_window_exceeded', + }), + ]) + expect(replayOutputsAfterLoss.map((payload) => String(payload.data))).toEqual(['ccc', 'ddd']) + expect(replayOutputsAfterLoss.every((payload) => payload.streamId === readyAfterLoss.streamId)).toBe(true) + expect(replayOutputsAfterLoss.every((payload) => payload.streamId !== initialReady.streamId)).toBe(true) + + broker.close() + }) + it('superseding attach on same socket clears stale queued frames and avoids duplicate old-frame delivery', async () => { const registry = new FakeBrokerRegistry() const broker = new TerminalStreamBroker(registry as any, vi.fn()) @@ -461,7 +1553,7 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { broker.close() }) - it('coalesces contiguous replay frames before sending terminal.output payloads', async () => { + it('coalesces contiguous replay frames into terminal.output.batch for batch-capable clients', async () => { const registry = new FakeBrokerRegistry() const broker = new TerminalStreamBroker(registry as any, vi.fn()) registry.createTerminal('term-replay-coalesced') @@ -475,22 +1567,150 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { } const wsReplay = createMockWs() - await broker.attach(wsReplay as any, 'term-replay-coalesced', 'transport_reconnect', 80, 24, 0, 'replay-attach') + await broker.attach( + wsReplay as any, + 'term-replay-coalesced', + 'transport_reconnect', + 80, + 24, + 0, + 'replay-attach', + undefined, + 'foreground', + true, + ) vi.advanceTimersByTime(5) const outputs = wsReplay.send.mock.calls .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) - .filter((payload) => payload?.type === 'terminal.output') + .filter((payload) => payload?.type === 'terminal.output.batch') - expect(outputs).toHaveLength(1) + expect(outputs.length).toBeGreaterThan(0) + expect(outputs.length).toBeLessThan(1000) expect(outputs[0]).toMatchObject({ + type: 'terminal.output.batch', attachRequestId: 'replay-attach', + source: 'replay', seqStart: 1, + segments: expect.any(Array), + }) + expect(outputs[outputs.length - 1]).toMatchObject({ + attachRequestId: 'replay-attach', + source: 'replay', seqEnd: 1000, }) - expect(outputs[0].data).toContain('f1;') - expect(outputs[0].data).toContain('f1000;') - expect(Buffer.byteLength(outputs[0].data, 'utf8')).toBeLessThanOrEqual(MAX_REALTIME_MESSAGE_BYTES) + expect(outputs.every((payload) => payload.serializedBytes <= MAX_REALTIME_MESSAGE_BYTES)).toBe(true) + expect(outputs.reduce((sum, payload) => sum + payload.segments.length, 0)).toBe(1000) + const joinedData = outputs.map((payload) => String(payload.data)).join('') + expect(joinedData).toContain('f1;') + expect(joinedData).toContain('f1000;') + + broker.close() + }) + + it('falls back to budget-safe terminal.output when one batch segment exceeds the batch envelope budget', async () => { + const registry = new FakeBrokerRegistry() + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + const terminalId = 'term-single-batch-budget' + const attachRequestId = TERMINAL_STREAM_ATTACH_REQUEST_ID_RESERVE_VALUE + registry.createTerminal(terminalId) + + const ws = createMockWs() + await broker.attach( + ws as any, + terminalId, + 'viewport_hydrate', + 80, + 24, + 0, + attachRequestId, + undefined, + 'foreground', + true, + ) + const ready = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .find((payload) => payload?.type === 'terminal.attach.ready') + expect(ready?.streamId).toEqual(expect.any(String)) + const streamId = String(ready.streamId) + + const legacyBudgetBytes = (data: string) => measureTerminalOutputPayloadBytes({ + type: 'terminal.output', + terminalId, + streamId, + seqStart: Number.MAX_SAFE_INTEGER, + seqEnd: Number.MAX_SAFE_INTEGER, + data, + attachRequestId, + }) + const batchBudgetBytes = (data: string) => { + let serializedBytes = 0 + for (let attempt = 0; attempt < 4; attempt += 1) { + const measured = measureTerminalOutputPayloadBytes({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 1, + data, + serializedBytes, + segments: [{ seqStart: 1, seqEnd: 1, endOffset: data.length, rawFrameCount: 1 }], + }) + if (measured === serializedBytes) return measured + serializedBytes = measured + } + return measureTerminalOutputPayloadBytes({ + type: 'terminal.output.batch', + terminalId, + streamId, + attachRequestId, + source: 'live', + seqStart: 1, + seqEnd: 1, + data, + serializedBytes, + segments: [{ seqStart: 1, seqEnd: 1, endOffset: data.length, rawFrameCount: 1 }], + }) + } + + let data = '' + for (let length = 1; length <= MAX_REALTIME_MESSAGE_BYTES; length += 1) { + const candidate = 'x'.repeat(length) + if ( + legacyBudgetBytes(candidate) <= MAX_REALTIME_MESSAGE_BYTES + && batchBudgetBytes(candidate) > MAX_REALTIME_MESSAGE_BYTES + ) { + data = candidate + break + } + } + expect(data.length).toBeGreaterThan(0) + expect(legacyBudgetBytes(data)).toBeLessThanOrEqual(MAX_REALTIME_MESSAGE_BYTES) + expect(batchBudgetBytes(data)).toBeGreaterThan(MAX_REALTIME_MESSAGE_BYTES) + + ws.send.mockClear() + registry.emit('terminal.output.raw', { terminalId, data, at: Date.now() }) + vi.advanceTimersByTime(5) + + const payloads = ws.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + const batchOutputs = payloads.filter((payload) => payload?.type === 'terminal.output.batch') + const outputFallbacks = payloads.filter((payload) => payload?.type === 'terminal.output') + + expect(batchOutputs.every((payload) => payload.serializedBytes <= MAX_REALTIME_MESSAGE_BYTES)).toBe(true) + expect(outputFallbacks).toHaveLength(1) + expect(outputFallbacks[0]).toMatchObject({ + type: 'terminal.output', + terminalId, + streamId, + attachRequestId, + seqStart: 1, + seqEnd: 1, + data, + }) + expect(measureTerminalOutputPayloadBytes(outputFallbacks[0])).toBeLessThanOrEqual(MAX_REALTIME_MESSAGE_BYTES) broker.close() }) @@ -526,6 +1746,143 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { broker.close() }) + it('paces foreground replay when socket bufferedAmount grows under replay pressure', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(4 * 1024 * 1024) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-foreground-paced') + + const wsSeed = createMockWs() + await broker.attach(wsSeed as any, 'term-foreground-paced', 'viewport_hydrate', 80, 24, 0, 'seed-attach') + + const chunk = 'x'.repeat(2 * 1024) + for (let i = 1; i <= 1400; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-foreground-paced', + data: `foreground-paced-${i};${chunk}`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 0 }) + wsReplay.send.mockImplementation((raw: string) => { + wsReplay.bufferedAmount += Buffer.byteLength(raw, 'utf8') + }) + + await broker.attach( + wsReplay as any, + 'term-foreground-paced', + 'transport_reconnect', + 80, + 24, + 0, + 'foreground-paced-attach', + undefined, + 'foreground', + ) + for (let i = 0; i < 220; i += 1) { + vi.runOnlyPendingTimers() + } + + expect(wsReplay.bufferedAmount).toBeLessThanOrEqual(512 * 1024 + 64 * 1024) + + broker.close() + }) + + it('resumes foreground replay after bufferedAmount drains and completes the retained backlog', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(4 * 1024 * 1024) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-foreground-resume') + + const chunk = 'x'.repeat(2 * 1024) + for (let i = 1; i <= 800; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-foreground-resume', + data: `foreground-resume-${i};${chunk}`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 0 }) + wsReplay.send.mockImplementation((raw: string) => { + wsReplay.bufferedAmount += Buffer.byteLength(raw, 'utf8') + }) + + await broker.attach( + wsReplay as any, + 'term-foreground-resume', + 'transport_reconnect', + 80, + 24, + 0, + 'foreground-resume-attach', + undefined, + 'foreground', + ) + + let outputs: any[] = [] + for (let cycle = 0; cycle < 20; cycle += 1) { + for (let i = 0; i < 220; i += 1) { + vi.runOnlyPendingTimers() + } + outputs = wsReplay.send.mock.calls + .map(([raw]) => (typeof raw === 'string' ? JSON.parse(raw) : raw)) + .filter((payload) => payload?.type === 'terminal.output') + if (outputs.map((payload) => String(payload.data)).join('').includes('foreground-resume-800;')) { + break + } + wsReplay.bufferedAmount = 0 + } + + expect(outputs.length).toBeGreaterThan(0) + expect(outputs.every((payload) => payload.attachRequestId === 'foreground-resume-attach')).toBe(true) + expect(outputs.map((payload) => String(payload.data)).join('')).toContain('foreground-resume-800;') + + broker.close() + }) + + it('rate limits foreground replay backpressure pause logs while the socket remains blocked', async () => { + const registry = new FakeBrokerRegistry() + registry.setReplayRingMaxBytes(4 * 1024 * 1024) + const broker = new TerminalStreamBroker(registry as any, vi.fn()) + registry.createTerminal('term-foreground-log-limited') + + for (let i = 1; i <= 40; i += 1) { + registry.emit('terminal.output.raw', { + terminalId: 'term-foreground-log-limited', + data: `foreground-log-limited-${i};${'x'.repeat(2 * 1024)}`, + at: Date.now(), + }) + } + + const wsReplay = createMockWs({ bufferedAmount: 768 * 1024 }) + await broker.attach( + wsReplay as any, + 'term-foreground-log-limited', + 'transport_reconnect', + 80, + 24, + 0, + 'foreground-log-limited-attach', + undefined, + 'foreground', + ) + + for (let i = 0; i < 10; i += 1) { + vi.advanceTimersByTime(50) + } + + const pauseLogs = loggerMocks.logger.debug.mock.calls.filter(([payload]) => + payload + && typeof payload === 'object' + && (payload as { event?: unknown }).event === 'terminal_stream_replay_backpressure_pause' + ) + expect(pauseLogs).toHaveLength(1) + + broker.close() + }) + it('pauses background replay when socket bufferedAmount is above the background threshold without closing', async () => { const registry = new FakeBrokerRegistry() const broker = new TerminalStreamBroker(registry as any, vi.fn()) @@ -591,7 +1948,7 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { }) } - const ws = createMockWs({ bufferedAmount: 768 * 1024 }) + const ws = createMockWs({ bufferedAmount: 512 * 1024 + 16 * 1024 }) await broker.attach(ws as any, 'term-promoted', 'keepalive_delta', 80, 24, 0, 'background-attach', undefined, 'background') vi.advanceTimersByTime(100) expect(ws.send.mock.calls @@ -612,6 +1969,7 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { }) it('emits terminal_stream_replay_hit, terminal_stream_queue_pressure, and terminal_stream_gap on overflow', async () => { + forceSmallTerminalClientQueueForOverflowTest() const registry = new FakeBrokerRegistry() const perfSpy = vi.fn() const broker = new TerminalStreamBroker(registry as any, perfSpy) @@ -621,6 +1979,7 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { await broker.attach(wsSeed as any, 'term-overflow', 'viewport_hydrate', 80, 24, 0) registry.emit('terminal.output.raw', { terminalId: 'term-overflow', data: 'seed-1', at: Date.now() }) registry.emit('terminal.output.raw', { terminalId: 'term-overflow', data: 'seed-2', at: Date.now() }) + broker.detach('term-overflow', wsSeed as any) const wsReplay = createMockWs() await broker.attach(wsReplay as any, 'term-overflow', 'viewport_hydrate', 80, 24, 1) @@ -655,6 +2014,28 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { payload.droppedBytes > 0 && level === 'warn', )).toBe(true) + expect(structuredLogs('warn', 'terminal.replay.gap') + .filter((payload) => ( + payload.terminalId === 'term-overflow' + && payload.reason === 'queue_overflow' + ))).toHaveLength(0) + expect(structuredLogs('warn', 'terminal.output.gap')).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + event: 'terminal.output.gap', + severity: 'warn', + terminalId: 'term-overflow', + source: 'live', + reason: 'queue_overflow', + fromSeq: expect.any(Number), + toSeq: expect.any(Number), + streamId: expect.any(String), + queueDepth: expect.any(Number), + droppedBytes: expect.any(Number), + droppedSerializedApplicationJsonBytes: expect.any(Number), + }), + ]), + ) broker.close() }) @@ -691,7 +2072,7 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { broker.close() }) - it('enforces a larger replay floor for coding-cli terminals to reduce history loss on attach', async () => { + it('enforces the 32 MiB replay floor for coding-cli terminals to reduce history loss on attach', async () => { const registry = new FakeBrokerRegistry() registry.setReplayRingMaxBytes(8) const perfSpy = vi.fn() @@ -700,6 +2081,9 @@ describe('TerminalStreamBroker catastrophic bufferedAmount handling', () => { const wsSeed = createMockWs() await broker.attach(wsSeed as any, 'term-coding-floor', 'viewport_hydrate', 80, 24, 0) + const terminalState = (broker as any).terminals.get('term-coding-floor') + expect(terminalState?.replayRing.retentionMaxBytes()).toBe(32 * 1024 * 1024) + registry.emit('terminal.output.raw', { terminalId: 'term-coding-floor', data: 'x'.repeat(96 * 1024), diff --git a/test/unit/server/ws-send.test.ts b/test/unit/server/ws-send.test.ts new file mode 100644 index 000000000..149b9a68d --- /dev/null +++ b/test/unit/server/ws-send.test.ts @@ -0,0 +1,231 @@ +// @vitest-environment node +import { describe, it, expect, beforeEach, vi } from 'vitest' +import WebSocket from 'ws' + +const perfMocks = vi.hoisted(() => ({ + config: { + enabled: true, + wsPayloadWarnBytes: 16, + rateLimitMs: 0, + }, + logPerfEvent: vi.fn(), + shouldLog: vi.fn(() => true), +})) + +const loggerMocks = vi.hoisted(() => { + const logger = { + child: vi.fn(), + debug: vi.fn(), + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + } + logger.child.mockReturnValue(logger) + return { logger } +}) + +vi.mock('../../../server/perf-logger', () => ({ + getPerfConfig: () => perfMocks.config, + logPerfEvent: perfMocks.logPerfEvent, + shouldLog: perfMocks.shouldLog, +})) + +vi.mock('../../../server/logger', () => ({ logger: loggerMocks.logger })) + +import { + prepareJsonMessage, + readWebSocketBufferedAmount, + sendJsonMessage, + sendPreparedJsonMessage, +} from '../../../server/ws-send' + +function createMockWs(overrides: Record = {}) { + const ws = { + readyState: WebSocket.OPEN, + bufferedAmount: 0, + connectionId: 'conn-test', + send: vi.fn(), + close: vi.fn(), + ...overrides, + } + return ws as typeof ws & { + readyState: number + bufferedAmount: number + connectionId?: string + send: ReturnType + close: ReturnType + } +} + +describe('ws-send', () => { + beforeEach(() => { + perfMocks.config.enabled = true + perfMocks.config.wsPayloadWarnBytes = 16 + perfMocks.config.rateLimitMs = 0 + perfMocks.logPerfEvent.mockClear() + perfMocks.shouldLog.mockClear() + perfMocks.shouldLog.mockReturnValue(true) + loggerMocks.logger.warn.mockClear() + }) + + it('serializes JSON once, measures serialized bytes, and reports bufferedAmount before and after send', () => { + const ws = createMockWs() + ws.send.mockImplementation((raw: string, cb?: (err?: Error) => void) => { + ws.bufferedAmount += Buffer.byteLength(raw, 'utf8') + cb?.() + }) + + const prepared = prepareJsonMessage({ type: 'unit.test', value: 'ok' }) + const result = sendPreparedJsonMessage(ws, prepared) + + expect(result.sent).toBe(true) + expect(result.serializedApplicationJsonBytes).toBe(Buffer.byteLength(prepared.serialized, 'utf8')) + expect(result.bufferedBefore).toBe(0) + expect(result.bufferedAfter).toBe(Buffer.byteLength(prepared.serialized, 'utf8')) + expect(ws.send).toHaveBeenCalledWith(prepared.serialized, expect.any(Function)) + }) + + it('does not send to a closed socket', () => { + const ws = createMockWs({ readyState: WebSocket.CLOSED }) + const result = sendJsonMessage(ws, { type: 'closed.test' }) + + expect(result.sent).toBe(false) + expect(result.reason).toBe('closed') + expect(ws.send).not.toHaveBeenCalled() + }) + + it('closes before sending when bufferedAmount exceeds the configured backpressure limit', () => { + const ws = createMockWs({ bufferedAmount: 2 * 1024 * 1024 + 1 }) + const result = sendJsonMessage(ws, { type: 'pressure.test' }, { + maxBufferedAmount: 2 * 1024 * 1024, + backpressureCloseCode: 4008, + backpressureCloseReason: 'Backpressure', + }) + + expect(result.sent).toBe(false) + expect(result.reason).toBe('backpressure') + expect(ws.close).toHaveBeenCalledWith(4008, 'Backpressure') + expect(ws.send).not.toHaveBeenCalled() + expect(perfMocks.logPerfEvent).toHaveBeenCalledWith( + 'ws_backpressure_close', + expect.objectContaining({ + connectionId: 'conn-test', + bufferedBytes: 2 * 1024 * 1024 + 1, + limitBytes: 2 * 1024 * 1024, + }), + 'warn', + ) + }) + + it('logs ws_send_large from the ws.send callback with bufferedAmount measurements', () => { + const ws = createMockWs({ bufferedAmount: 100 }) + ws.send.mockImplementation((raw: string, cb?: (err?: Error) => void) => { + ws.bufferedAmount += Buffer.byteLength(raw, 'utf8') + cb?.() + }) + + const result = sendJsonMessage(ws, { type: 'large.test', data: 'x'.repeat(32) }) + + expect(result.sent).toBe(true) + expect(perfMocks.logPerfEvent).toHaveBeenCalledWith( + 'ws_send_large', + expect.objectContaining({ + connectionId: 'conn-test', + messageType: 'large.test', + payloadBytes: result.serializedApplicationJsonBytes, + bufferedBytes: 100, + bufferedBytesAfter: result.bufferedAfter, + error: false, + }), + 'warn', + ) + }) + + it('logs callback errors for small sends when perf logging is disabled', () => { + perfMocks.config.enabled = false + const ws = createMockWs() + const error = new Error('small send failed') + ws.send.mockImplementation((_raw: string, cb?: (err?: Error) => void) => { + cb?.(error) + }) + + const result = sendJsonMessage(ws, { type: 'small.test', data: 'ok' }) + + expect(result.sent).toBe(true) + expect(loggerMocks.logger.warn).toHaveBeenCalledWith( + expect.objectContaining({ + err: error, + connectionId: 'conn-test', + messageType: 'small.test', + }), + 'WebSocket send callback reported failure', + ) + expect(perfMocks.logPerfEvent).not.toHaveBeenCalledWith( + 'ws_send_large', + expect.anything(), + expect.anything(), + ) + }) + + it('logs callback errors even when large-send perf logging is rate limited', () => { + perfMocks.config.enabled = true + perfMocks.config.wsPayloadWarnBytes = 1 + perfMocks.shouldLog.mockReturnValue(false) + const ws = createMockWs() + const error = new Error('rate limited send failed') + ws.send.mockImplementation((_raw: string, cb?: (err?: Error) => void) => { + cb?.(error) + }) + + const result = sendJsonMessage(ws, { type: 'limited.test', data: 'x'.repeat(32) }) + + expect(result.sent).toBe(true) + expect(loggerMocks.logger.warn).toHaveBeenCalledWith( + expect.objectContaining({ + err: error, + connectionId: 'conn-test', + messageType: 'limited.test', + }), + 'WebSocket send callback reported failure', + ) + expect(perfMocks.logPerfEvent).not.toHaveBeenCalledWith( + 'ws_send_large', + expect.anything(), + expect.anything(), + ) + }) + + it('normalizes unavailable bufferedAmount reads to undefined', () => { + expect(readWebSocketBufferedAmount({ bufferedAmount: undefined })).toBeUndefined() + expect(readWebSocketBufferedAmount({ bufferedAmount: Number.NaN })).toBeUndefined() + }) + + it('uses one shared serialized JSON budget even when WS_MAX_PAYLOAD_BYTES differs', async () => { + const originalMaxSerialized = process.env.MAX_SERIALIZED_APPLICATION_JSON_BYTES + const originalWsMaxPayload = process.env.WS_MAX_PAYLOAD_BYTES + process.env.MAX_SERIALIZED_APPLICATION_JSON_BYTES = '128' + process.env.WS_MAX_PAYLOAD_BYTES = '4096' + vi.resetModules() + try { + const { sendJsonMessage: sendWithReloadedBudget } = await import('../../../server/ws-send') + const ws = createMockWs() + const result = sendWithReloadedBudget(ws, { type: 'budget.test', data: 'x'.repeat(1500) }) + + expect(result.sent).toBe(false) + expect(result.reason).toBe('oversized') + expect(ws.send).not.toHaveBeenCalled() + } finally { + if (originalMaxSerialized === undefined) { + delete process.env.MAX_SERIALIZED_APPLICATION_JSON_BYTES + } else { + process.env.MAX_SERIALIZED_APPLICATION_JSON_BYTES = originalMaxSerialized + } + if (originalWsMaxPayload === undefined) { + delete process.env.WS_MAX_PAYLOAD_BYTES + } else { + process.env.WS_MAX_PAYLOAD_BYTES = originalWsMaxPayload + } + vi.resetModules() + } + }) +}) diff --git a/test/unit/shared/turn-complete-signal.test.ts b/test/unit/shared/turn-complete-signal.test.ts index e8073dd82..f4470b04a 100644 --- a/test/unit/shared/turn-complete-signal.test.ts +++ b/test/unit/shared/turn-complete-signal.test.ts @@ -6,6 +6,7 @@ import { extractTurnCompleteSignals, isSubmitInput, } from '../../../shared/turn-complete-signal' +import { shouldAllowTerminalOutputSideEffect } from '@/lib/terminal-output-write-scope' describe('shared turn-complete signal parser', () => { it('counts BEL in Codex output and strips it from cleaned output', () => { @@ -98,3 +99,33 @@ describe('countTrackerTurnCompleteSignals', () => { expect(countTrackerTurnCompleteSignals('\x1b]0;title\x07', state)).toBe(0) }) }) + +describe('turn-complete side-effect gating', () => { + it('allows client-minted live turn completion only for non-server-authoritative modes', () => { + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'opencode', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'shell', + })).toBe(true) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'claude', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'live', + effect: 'turn_complete', + mode: 'codex', + })).toBe(false) + expect(shouldAllowTerminalOutputSideEffect({ + source: 'replay', + effect: 'turn_complete', + mode: 'opencode', + })).toBe(false) + }) +}) diff --git a/test/unit/visible-first/terminal-mirror-fixture.test.ts b/test/unit/visible-first/terminal-mirror-fixture.test.ts index 454a4d10a..90a0f4864 100644 --- a/test/unit/visible-first/terminal-mirror-fixture.test.ts +++ b/test/unit/visible-first/terminal-mirror-fixture.test.ts @@ -9,7 +9,9 @@ describe('createTerminalMirrorFixture', () => { replayMaxBytes: 12, }) - mirror.applyOutput('\u001b[31mred\u001b[0m\nplain') + const firstFrame = mirror.applyOutput('\u001b[31mred\u001b[0m\nplain') + expect(firstFrame.streamId).toEqual(expect.any(String)) + expect(firstFrame.streamId.length).toBeGreaterThan(0) const viewport = mirror.serializeViewport() expect(viewport.lines).toEqual(['red', 'plain']) @@ -29,5 +31,6 @@ describe('createTerminalMirrorFixture', () => { const replay = mirror.replaySince(0) expect(replay.missedFromSeq).toBeGreaterThan(0) expect(replay.frames.length).toBeGreaterThan(0) + expect(replay.frames.every((frame) => frame.streamId === firstFrame.streamId)).toBe(true) }) })