Stabilise riskoff for release: eliminate unexplained errors

[onsails]

Destination

A release-ready codebase with no unexplained errors in the riskoff PC logs: every ERROR/WARN class diagnosed and fixed-or-accepted, verified by a clean production run. The cron→report pipeline (cron run → Telegram report) works end-to-end with zero silent drops.

Notes

• Scope: cron→delivery→report pipeline + prefilter + claude_health + the stderr-drain noise (both worker and cron copies). Provider composition and other subsystems are out of scope.
• Repo conventions: AGENTS.md (Rust standards, verification cadence, architecture split). Plan first; do not fix until tickets resolve and the approach is proven empirically.
• Skills per ticket: grilling + domain-modeling for decisions; research for the shared wait_with_output hang; prototype for the live probe.
• Log source: ~/.right/logs/riskoff.log.2026-08-*, PC logs via ~/.right/run/state.json + PC API, aggregator log right-mcp-server.

Decisions so far

• Diagnose delivery subprocess exit-1 and 120s timeout — transport never relays guest exit to host wait_with_output; fix: break-on-result + kill.
• Diagnose prefilter 30s timeout on json output path — same transport hang, 30s budget.
• Suppress stderr-drain timeout noise on success paths — downgrade to debug in worker+cron, fix stale "ssh" comment.
• claude_health false-fails — rename to honest MCP-init signal, log full exit/stderr on failure.
• cron reflection exit-1 cascade — keep reflection, fix its transport, skip for deterministic failures.
• Probe: does stream-json exit where json-mode hangs? — directionally right; production logs now prove it directly. Transport never relays exit to host. Fix = break-on-result + kill.
• Release-readiness gate: define 'no unexplained errors' — 24h prod soak + curated allowlist; zero unexplained = ready.
• Fix delivery and prefilter json-mode hang (break-on-result + kill) — root cause CONFIRMED from prod logs: result envelope lands ~45s after spawn, ExecEvent::Exited never fires; break-on-result + kill at async_delivery.rs:1145 and learning_prefilter.rs:559. Ready to implement.
• Land claude_health honest-signal rename and failure diagnostics — DONE: ClaudeHealth→McpInitHealth, logs claude_health:→right_mcp_init:; probe stderr piped + bounded 2 KiB tail excerpt in failure error alongside exit code; still non-billable.
• Skip cron reflection for deterministic failures — DONE: failure arm calls skip_reflection_decision; 529/429/5xx/turn-limit classified results skip the futile --resume turn and report the classifier's user message; unclassifiable failures still reflect.
• Delivery relay leaks "send_message not available" note into content — follow-up: relay model attempts mcp__right__send_message (correctly disallowed) and leaks a "Delivery note" into user-facing content. Blocked by Fix delivery and prefilter json-mode hang (break-on-result + kill) #189.

Not yet specified

All execution tickets have landed (#183–#187, #189, #190–#194; #189/#191 closed). The frontier is the release gate itself: run the 24h production soak and curate the error allowlist per #188.

Out of scope

• Provider composition/credential subsystem (right_openshell::providers, dashboard) — separate ownership.
• Historical ssrf_denied network-policy errors — pre-fix era, not reproducing.
• Telegram API delete_my_commands channel rejection — accepted noise.
• Historical delivery exit-1 fast-fail (auth/ssrf, SSH era) — distinct root cause, gone since microsandbox cutover.

[onsails]

Map complete: all decisions resolved, the way to a stable release is clear. Hand off to execution.

[onsails]

Root-cause pass (2026-08-21) — evidence-corrected

Ran against production logs (riskoff.log + sandbox exec.log). Four distinct root causes, not one shared mechanism. The map's claim that delivery+prefilter share a single transport hang is partly wrong.

1. Delivery hang (#189) — PROVEN, transport + semantic stack

• Delivery relay = haiku, --resume <main session>, stdin-piped, json+schema.
• Result envelope lands on stdout 45s after spawn (12:45:15), host times out at 12:46:30. 24/24 timeouts today.
• Envelope stop_reason=tool_use, num_turns=2, duration_api_ms=15070 — the turn finished its API work in ~15s; the guest then fails to emit ExecEvent::Exited, so wait_with_output hangs.
• Fix premise holds: a terminal result exists to break on, ~75s before the deadline. break-on-result+kill is correct.

2. Prefilter timeout (#184) — NOT the same mechanism; root cause UNPROVEN

• Prefilter = haiku, argv-prompt (-- <prompt>, NOT stdin), fresh session, --tools "" + --max-turns 5, json+schema.
• 13 timeouts today. command.output() = same wait_with_output transport.
• Contradiction: the Probe: does stream-json exit where json-mode hangs? #190 probe proved fresh argv-prompt json+schema exits cleanly in ~8s. The prefilter is the same shape yet times out at 30s.
• Distinguishing factors (only differences from the Probe: does stream-json exit where json-mode hangs? #190 probe): --tools "" (empty string) and --max-turns 5.
• No clean end_turn haiku envelope was found in any prefilter timeout window — the tool_use envelopes near those times are other concurrent sessions (the prefilter cannot use tools). So there is no evidence the prefilter emits a terminal result before 30s.
• Implication: break-on-result+kill may NOT fix the prefilter — if haiku never produces a result in 30s, there is nothing to break on. Needs a targeted probe: json+schema --tools "" --max-turns 5 vs json+schema without those flags, measuring result-arrival time.

3. Reflection exit-1 cascade (#187) — DETERMINISTIC failure, distinct signature, already gone

• 239× "reflection failed: reflection CC exited with code 1" across 07-30..08-17, but 0 since 08-18 (1 on 08-20).
• Signature: is_error:true, duration_api_ms:0, num_turns:1, stop_reason:stop_sequence, total_cost_usd:0, all-zero usage. The reflection CC call never reached the API — deterministic spawn/config failure, not a transport hang, not a slow turn.
• Matches cron reflection exit-1 cascade (391 warnings) #187's "skip for deterministic failures" — the fix shape is right, and the symptom has already stopped on its own (likely the microsandbox cutover removed the trigger).

4. claude_health "exit code 1" (#186) — intermittent, all reason=startup

• 120× "failed: exit code: 1" across all days; every one is reason="startup". The hourly steady-state probe is 15/15 "ok" on 08-21.
• The startup probe and the hourly probe differ: startup fires before the sandbox/aggregator is fully warm. "exit code 1" with no stderr logged is the whole diagnosis — cause not established, but it is startup-only, not a steady-state ERROR.

5. stderr-drain timeout (#185) — symptom only, cause still a guess

• Real (exit-0 turns, bytes_so_far=0, ~2s). The "pipe write-end held by another process (ssh master forwarding?)" text is a stale hypothesis from the SSH era, never verified against the microsandbox transport. Downgrading to debug treats it as benign without knowing why it fires.

Shared vs distinct — corrected

• Shared: Diagnose delivery subprocess exit-1 and 120s timeout #183/Diagnose prefilter 30s timeout on json output path #184/Fix delivery and prefilter json-mode hang (break-on-result + kill) #189 all route through wait_with_output, which awaits ExecEvent::Exited. That function is the single point that makes a "finished" turn look hung.
• Distinct trigger: Fix delivery and prefilter json-mode hang (break-on-result + kill) #189 (stdin-piped --resume) vs Diagnose prefilter 30s timeout on json output path #184 (argv-prompt fresh --tools "") — and Probe: does stream-json exit where json-mode hangs? #190 evidence says the latter should exit. Diagnose prefilter 30s timeout on json output path #184's true trigger is unknown.
• Distinct causes entirely: cron reflection exit-1 cascade (391 warnings) #187 (deterministic zero-API failure, gone), claude_health false-fails (197 failed / 7 probe timeouts) #186 (startup-only exit-1), Suppress stderr-drain timeout noise on success paths #185 (stderr FD, unknown), Delivery relay leaks 'send_message not available' note into content #191 (relay tries send_message, leaks note).

Open questions (one probe settles #184)

1. Does claude -p --output-format json --json-schema X --tools "" --max-turns 5 -- <prompt> emit its result envelope at all, and in how long? Compare against the same minus --tools "".
2. Does the sandbox transport ever deliver ExecEvent::Exited for a --resume turn? (still a code-comment claim, never independently observed at gRPC level).

Until #1 is answered, do NOT ship #184 on faith of the #189 fix.