feat(billing): recover Stripe reconciliation dead letters

[seonghobae]

Buyer/operability objective

Refs #488. This Draft child of #574 adds a bounded operator recovery path for Stripe reconciliation work that exhausted its automatic retry budget. It preserves append-only attempt history and does not reset the automatic retry budget: each explicitly authorized recovery creates exactly one new finite lease/attempt, re-fetches current provider state through the existing authoritative reconciliation service, and returns to dead_letter immediately if that manual attempt fails.

Exact current stack

• protected develop: 2c328875e00e86537df3e965170be80532571cad;
• exact live parent/base feat(billing): consume Stripe reconciliation triggers with leases #574: feat/stripe-reconciliation-worker-488@18bec1d606b7e6762bf11ff6cf429d024ba729bc;
• exact contributor head: 24b1fe3d7f447748aff943bb7f5395ac90cce8ad;
• branch: feat/stripe-reconciliation-dead-letter-recovery-488;
• state: Draft and mechanically mergeable at the latest refetch;
• current submitted reviews: none;
• current inline review threads: none.

Any parent, contributor-head, protected-base, review, or check movement invalidates ancestry- and head-sensitive evidence until freshly revalidated.

Recovery contract

• dead-letter inspection is tenant-scoped through normalized Subscription → Customer → organization authority;
• only owner/admin API callers can inspect or retry the tenant backlog;
• one bounded control-free evidenceReference is durable idempotency authority for one Event recovery;
• the same Event/evidence pair replays the existing receipt without provider I/O;
• a new explicit evidence reference creates a new attempt number while preserving prior attempts;
• lease plaintext exists only in-process; persisted worker state retains only the SHA-256 lease digest;
• recovery authority and worker attempt/result mutation are committed together under nested SQLite savepoints;
• arbitrary provider exception text is collapsed to bounded machine-readable failure codes;
• the API exposes no provider secret, raw webhook body, lease token, or caller-selected Subscription identity; and
• successful recovery can advance durable claim evidence but does not rewrite orgs.plan, membership/RBAC, or browser session authority.

TDD and bounded request repair

The original RED contracts cover tenant isolation, immutable attempt history, retry success/failure, same-evidence replay, new-evidence retry, validation, in-progress replay, transactional audit rollback, API auth/RBAC, tenant non-disclosure, provider reconciliation, idempotent replay, backlog clearing, and durable actor/evidence attribution.

Fresh current-source inspection found an additional ScopeWeave-owned resource-boundary defect: the privileged retry route called c.req.json() before enforcing any request-body byte ceiling. A tenant administrator could therefore cause the process to buffer an arbitrarily large JSON body even though the durable evidenceReference is itself bounded.

Repair chronology:

• RED commit f2d41bc403a0d3d7f8afa67adac1a9adf66757a4 adds realistic API regressions for both an oversized declared Content-Length body and an oversized streamed body with no declared length; both must fail with HTTP 413 and a stable non-secret error;
• a first custom bounded-reader implementation was discarded after self-review because this route module is exact-c8-owned and the custom reader unnecessarily expanded the production branch surface;
• current production commit 24b1fe3d7f447748aff943bb7f5395ac90cce8ad uses Hono's built-in bodyLimit middleware at a 4 KiB ceiling, preserving the existing authenticated route and JSON contract while enforcing both declared and streamed byte limits before JSON parsing.

This keeps the root-cause fix small and deterministic while leaving the provider, entitlement, audit, and authorization boundaries unchanged.

Coverage and scope

server/stripe_reconciliation_recovery.mjs and server/stripe_reconciliation_recovery_routes.mjs remain registered in canonical c8 instrumentation. The API recovery regression remains registered in test:api and therefore in canonical coverage cases. No threshold, production file, security gate, or dependency check was removed to make the repair green.

Current exact-head evidence

Fresh repository-native runs for exact head 24b1fe3d7f447748aff943bb7f5395ac90cce8ad exist but are not passing evidence yet:

• Server Tests 32452371660 — queued;
• Security Scan 32452371676 — queued;
• SAST Semgrep 32452371677 — queued;
• Dependency Review 32452371662 — queued; and
• OSV Scanner 32452372112 — queued.

CodeQL Required is absent from this stacked exact-head enumeration because the repository-wide stacked/exact-head CodeQL control remains in #523 rather than protected develop; absence is non-passing, not implicit success. There is no qualifying independent current-head approval. Pending, queued, absent, skipped-required, cancelled, stale, predecessor, synthetic, status-only, model-only, rate-limited, or infrastructure-only evidence is not merge authority.

Stack and merge boundary

Child #579 was content-forwarded after this parent moved so its current source retains the same bounded recovery-body regression and middleware instead of semantically reverting the repaired parent. Because that child was originally forked from predecessor c314af00c5949c728909f56bc1aff7c3ee439a50, its ancestry still needs normal stack reconciliation before integration; no force-push or destructive rebase was used.

Remain Draft. Do not integrate independently of #574/#573/#570/#569/#568 and the preceding #488 stack. Do not merge or enable auto-merge until the unchanged exact head is reconciled to the then-live prerequisite stack and every applicable exact-head repository/organization deterministic/browser/coverage/CodeQL/SAST/security/dependency/supply-chain/package/provenance/review gate is substantively terminal-passing under live governance.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d74e0573-9ab9-459c-968c-51bc19249b28

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}