Harden rollout writer recovery and surface persistence health

[shiny-code-bot]

Objective

Contain the failure mode seen on July 18, 2026: a live resumed session's 4.31 GB rollout stopped advancing, the writer's original storage error was lost, and more than 1,100 channel closed diagnostics followed while the process remained alive and disk usage was 98%.

Make the existing V1 writer supervised, bounded, recoverable, and observable before changing the storage format.

Finish Line

Rollout writer failures preserve root cause, stay bounded, recover safely, and surface one actionable session-health signal.

Current Status

State: Active
Next action: Preserve structured writer failures end-to-end and add a supervised, rate-limited recovery state model around the current recorder.
Blocked by: None
Waiting for: None
Last verified: July 18, 2026 against the stopped 4.31 GB rollout and current recorder recovery source.

Scope

• In: V1 recorder/task supervision, typed persistence failures, byte-bounded pending data, classified backoff, durable barriers, state DB ordering, non-persisted health notifications, TUI/app-server status.
• Out: segmented V2 storage, automatic user-history deletion, cross-volume copying, compression policy, unrelated session UX.

Acceptance Criteria

• The first causal error retains operation, error kind, raw OS code, first/last-seen times, and retry state.
• Writer panic, cancellation, terminal exit, and channel closure remain distinguishable.
• Pending data is bounded by serialized bytes and applies typed backpressure at capacity.
• flush/shutdown succeed only after the committed prefix is durably synced; state DB metadata never advances first.
• Disk-full/quota failures back off without retrying on every appended event and recover in order after space returns.
• 1,100 failed appends produce one immediate diagnostic, bounded suppression summaries, and one recovery message.
• Session saving health is delivered on a non-persisted control path and visibly clears after recovery.
• Existing V1 rollouts remain readable and normal healthy-path latency stays within an agreed budget.

Relationships

This will be attached as a sub-issue of the agent reliability parent. Segmented/checkpointed storage depends on the stable persistence contract established here.

Validation

• Fault-injection tests for ENOSPC, quota, EIO, permissions, panic/cancel, partial lines, retry order, and causal error preservation.
• Session tests for bounded backpressure, non-recursive health events, and state DB ordering.
• TUI/app-server snapshots for paused, blocked, recovered, and terminal saving states.
• Logical long-run soak with a fake quota writer; real-disk smoke only in a controlled environment.

Decisions

• Storage thresholds are advisory; the actual OS write error is authoritative.
• Health transitions are edge-triggered and payload/path redacted.
• Do not auto-delete user history to recover disk space.

Open Questions

• Choose the byte cap and healthy-path batching budget from measurements rather than a new user-facing flag.