docs: whole-codebase logic bug audit ahead of larger-scale deployment

[passcod]

Adds docs/logic-bug-audit-2026-07.md: a full logic-bug audit of every crate and subsystem (protocol, core defs/OI/runtime/system, daemon, ctl, web), carried out ahead of larger-scale deployment. Report only — no fixes are applied in this PR.

Contents

• 138 findings, each with location, concrete failure scenario, severity justification, fix-testability assessment, and a reviewer confidence rating.
• Categorised on two axes as requested: severity/impact (1 critical, 22 high, 54 medium, 61 low) and testability of the fix (72 easy / 58 moderate / 8 hard).
• A priority list of the critical + high findings, eight cross-cutting themes (recurring failure patterns spanning subsystems), and a suggested fix sequencing.
• Eight companion pattern analyses under docs/logic-bug-audit-2026-07/, one per cross-cutting theme, each assessing if and how a high-level pattern or change would fix and prevent the class: the failure pattern, affected findings, a verdict, a concrete proposed design grounded in the real code, what it does and does not prevent, a migration path, and enforcement (tests, spec items, CI checks). Cross-linked from the main report's themes section.

Headline items

• Critical: a failed script evaluation in /apps/update still swaps the partially-evaluated AppDef into the registry and then runs the destructive post-reload steps — holding (relocating) live volume data, wiping scaling decisions, and tearing down forwards — from a single script typo. Independently found by two reviewers and manually verified.
• High findings cluster around self-healing machinery that turns transient errors into permanent silent failures (image pull exhaustion, Job killed on a failed observation, dead start-limit detection), barrier/replay correctness (rt.stop on scaled deployments, unenforced chained-barrier deadlines, rt.signal dedup), TLS serving/issuance, pod network prefix collisions, and event-stream clients that hang forever on server error responses.

Theme verdicts (short form)

1. Stream error handling — yes: one open_subscription helper in crates/protocol eliminates the class; four call sites re-implement the same handshake today.
2. Partial state on failure — partially: one invariant, three boundary-specific disciplines (commit-on-success reload, known-set invariant in the reconciler, durable-first ordering).
3. Observation uncertainty — yes, strongest type-driven candidate: an Observed/Failed enum at the observer boundary; needs no persistence change.
4. Fault lifecycle — yes, if faults are split into condition/event/latched shapes with a first-class (app, kind, subject) key; converge-to-current-set for condition faults.
5. Reserved namespaces — yes: "identity is granted once and recorded; every later match is against the record", with no renames of deployed units/volumes required.
6. BSL strict validation — yes, most mechanical theme: a shared throwing take_* helper module plus a scoped CI grep; must land after the critical reload fix.
7. Retry/backoff — yes, as two things: a shared per-key RetryGate (extracted from the scheduler's existing tested back-off) for tick-driven retries, plus a loop discipline with transient/fatal classification for relays.
8. Restart/replay — partially: two mechanical rules (positional replay matching, no INSERT OR REPLACE on rows with independently-owned columns) held together by a RestartWorld test harness, the one element that would have caught all four findings.

Method

Seventeen independent subsystem reviews, each reading its scope in full and cross-checking against callers, docs/spec/, and existing tests; spec- or test-asserted behaviour was treated as intentional. Line numbers reference e14aa26 (v0.4.5). Duplicated findings across reviewers were merged (corroboration noted inline).

🤖 Generated with Claude Code

https://claude.ai/code/session_01A6G4NQfLjX6J9fB5wNKjvo

[github-code-quality]

Code Coverage Overview

Languages: TypeScript, Rust

TypeScript / code-coverage/vitest

The overall coverage in the claude/seedling-logi... branch remains at 65%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.

File	main e14aa26	claude/seedling-logi... 841df86	+/-
src/routes/Apps.tsx	74%	78%	+4%

Rust / code-coverage/rust

The overall coverage remains at 55%, unchanged from the main branch.

---

_{Updated July 17, 2026 01:35 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}