Skip to content

fix(exact-review): handle terminal admissions and tuple failures#840

Merged
brokemac79 merged 5 commits into
mainfrom
codex/terminal-admission-tuple-failure
Jul 25, 2026
Merged

fix(exact-review): handle terminal admissions and tuple failures#840
brokemac79 merged 5 commits into
mainfrom
codex/terminal-admission-tuple-failure

Conversation

@brokemac79

@brokemac79 brokemac79 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Resolve ordinary exact-review queue items before dispatch when their target is already closed, genuinely missing, or gone.
  • Preserve a command-requested re-review through the worker so its existing acknowledgement can be completed even if the target closed first.
  • Bound live admission checks without turning GitHub target access failures or rate limits into poison-item retries.
  • Preserve permanent publisher failures, including tuple_protocol_invalid, in the batch outcome so the coordinator cannot relabel them as generic retryable failures.

Problem

Two independent paths could keep work moving when it should stop:

  1. An exact review could remain queued after its target item closed, then be dispatched unnecessarily. A simple 404 is not enough proof: it can also mean the bot cannot see the repository.
  2. The publisher could classify a tuple failure as permanent, but fail before writing its batch outcome. The outer batch wrapper then treated the missing output as retryable_failure/unknown_failure, causing the terminal failure to retry.

The initial terminal-admission implementation also needed two operational safeguards: preserve the visible completion path for command-requested reviews, and rate-bound real GitHub reads without allowing a rate-limited 403 to park valid items.

Implementation

  • Admission makes target-repository read-only checks for up to four non-publication candidates. A full pass, or a terminal result that exposes more ready review work, records a five-second durable admission interval; new review work respects it, while publication work and active-lease/reconciliation wake-ups remain eligible.
  • A target confirmed closed, genuinely missing (404 with the repository still visible to the same token), or gone (410) is removed as completed work without dispatch. A live open target follows the normal dispatch path.
  • Command-context terminal items are dispatched only to complete their existing command-status lifecycle. Ordinary terminal items remain local no-ops.
  • Target-specific access failures are bounded per item; shared network, server, and rate-limit failures apply dispatcher backoff without charging every item. A 403 is item-specific only when it is not identified as a GitHub rate-limit response (429, retry/rate headers, or standard rate-limit/abuse text).
  • Lease reclamation and publication-expiry recovery are saved before the awaited live read. The post-read state check requires the same pending revision, and dispatch-error handling re-reads the batch reservation so it cannot restore a reservation that another worker cleared.
  • Publisher exceptions now write a structured batch mutation result before rethrowing, retaining either the permanent or retryable disposition for the batch coordinator.

Causal links and related work

  • #459 made an invalid record tuple a permanent tuple_protocol_invalid result.
  • #781 added the batch fallback that treats a missing per-item result file as a generic retry; that fallback could hide the permanent result above when the publisher did not write its batch mutation output.
  • #733 is the observed tuple-failure incident; its batch is in run 30122306387. #818 is the observed terminal-item queue incident.
  • Already in main: #836 fixes publication-head latching; #837 changes recovery ordering; #838 supplies the bounded review-retry model used here; and #839 supplies the dispatch-failure classification extended here. They are related infrastructure, not duplicate fixes.

Validation

  • node --test test/dashboard-worker.test.ts — 203 passing.
  • node --test test/exact-review-publication-batches.test.ts — 44 passing.
  • pnpm run build:dashboard — passed.
  • pnpm exec oxfmt --check dashboard/exact-review-queue.ts src/repair/publish-event-result.ts test/dashboard-worker.test.ts test/exact-review-publication-batches.test.ts — passed for changed surfaces.
  • git diff --check — passed.
  • Linux Docker/Crabbox proof (cbx_4ce91d0f8380, Playwright Noble / Node 24.15.0) ran the same queue suite 203/203 and publication-batch suite 44/44 against this final code; it used only local storage and fake GitHub responses. The lease stopped automatically.

Isolated before/after runtime probes for the two original failures also passed:

  • Terminal admission: origin/main dispatched one already-closed item; this PR’s terminal path dispatched zero and completed the queue item locally.
  • Publisher tuple failure: origin/main produced no batch mutation output after an impossible tuple; this PR produced permanent_failure with tuple_protocol_invalid and an error fingerprint.

pnpm run format:check remains baseline noise across unrelated repository files; the changed files pass the focused formatter check above.

Review closeout

  • ClawSweeper review found no discrete code defect but requested runtime behavior proof. The Docker evidence above supplies that proof without production mutation.
  • Structured auto-review identified the command-status and admission-rate gaps during development; both were accepted, fixed, and covered by focused regression tests.
  • codex review --uncommitted — final dirty-patch review clean after accepted command-status, rate-budget, partial-pass, and rate-limited-403 findings were fixed and focused proof rerun.
  • codex review --base origin/main — final branch review clean: no actionable regressions relative to the merge base.

Risk and rollout

The admission path adds read-only GitHub checks and a small durable five-second throttle only when a full admission burst is reached or terminal completion reveals more ready review work. It intentionally does not bypass command-status completion. No production queue dispatch, item comment, workflow dispatch, gate change, or state mutation was performed as validation. No changelog or workflow change is included.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Jul 24, 2026
@clawsweeper

clawsweeper Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 24, 2026, 6:38 PM ET / 22:38 UTC.

ClawSweeper review

What this changes

This PR checks exact-review targets before dispatch so closed or conclusively gone items are completed without work, and preserves permanent publisher failures in the batch result so they are not retried as unknown failures.

Merge readiness

Blocked until real behavior proof from a real setup is added - 4 items remain

Keep this PR open for human review: the patch is a focused repair for two credible exact-review failure modes and I found no discrete correctness defect in the supplied diff, but it lacks contributor-provided after-fix runtime proof for behavior that can suppress queue dispatch or change terminal handling.

Priority: P2
Reviewed head: b427f99b6bbdeae452f5b6264150a7a2dc41aed9

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The implementation and focused coverage look coherent from the supplied diff, but proof is limited to tests and static validation rather than observed after-fix runtime behavior.
Proof confidence 🧂 unranked krab (1/6) Needs real behavior proof before merge: The PR reports focused tests, builds, linting, and static review, but its body explicitly says no live queue dispatch, item comment, workflow dispatch, or state change was performed; add redacted after-fix runtime output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs real behavior proof before merge: The PR reports focused tests, builds, linting, and static review, but its body explicitly says no live queue dispatch, item comment, workflow dispatch, or state change was performed; add redacted after-fix runtime output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 4 items Focused queue change: The supplied PR diff changes queue admission in dashboard/exact-review-queue.ts and adds focused coverage for closed, missing, inaccessible, and failure-classification paths.
Batch outcome preservation: The publisher catch path now writes the structured batch mutation result before writing completion outputs and rethrowing, retaining permanent dispositions such as tuple_protocol_invalid for the coordinator.
Related current-main foundations: The PR builds on merged exact-review retry and dispatch-classification work: #838 and #839. The supplied context describes this PR as complementary rather than a duplicate.
Findings None None.
Security None None.

How this fits together

The exact-review system accepts GitHub events into a Durable Object queue, dispatches bounded review work, and then publishes durable review results. This change sits at queue admission and batch-result publication, affecting whether queued work is dispatched, retried, parked, or completed.

flowchart LR
  A[GitHub review event] --> B[Exact-review durable queue]
  B --> C[Admission target check]
  C -->|Open target| D[Review workflow dispatch]
  C -->|Closed or conclusively gone| E[Complete queue item]
  D --> F[Batch result publisher]
  F --> G[Durable state and GitHub result]
  F --> H[Permanent or retryable outcome]
Loading

Before merge

  • Add real behavior proof - Needs real behavior proof before merge: The PR reports focused tests, builds, linting, and static review, but its body explicitly says no live queue dispatch, item comment, workflow dispatch, or state change was performed; add redacted after-fix runtime output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Resolve merge risk (P1) - Merging changes admission from unconditional dispatch to terminal completion after a live GitHub read; without after-fix runtime evidence, an unexpected visibility or classification edge could suppress review work or park it incorrectly.
  • Resolve merge risk (P1) - The batch-result repair preserves the intended disposition, but the submitted evidence is test and static-check output rather than a redacted runtime transcript showing the coordinator consumes the written permanent result.
  • Complete next step (P2) - The remaining merge blocker is contributor-supplied real behavior proof for operational queue classification, not a narrow code repair ClawSweeper can safely make.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Changed surface 5 files affected; 739 additions, 53 deletions The patch is concentrated in one queue implementation, one publisher implementation, and focused tests, but the queue behavior is operationally sensitive.
Admission concurrency 1 bounded read pool of 8 The new live-target checks add GitHub reads at admission and therefore need runtime proof of the intended classification behavior.

Merge-risk options

Maintainer options:

  1. Add admission and publisher runtime proof (recommended)
    Before merge, provide a redacted after-fix transcript or recording that shows the real queue owner safely distinguishes terminal targets from inaccessible ones and preserves a permanent tuple failure for the batch coordinator.
  2. Pause for targeted operator review
    If equivalent runtime proof cannot be captured safely, keep the PR open for an owner of the queue and publisher paths to review the operational failure classification manually.

Technical review

Best possible solution:

Add redacted after-fix runtime proof that drives the real queue owner through open, closed, visible-404, inaccessible-target, and tuple-failure cases, showing both the resulting queue state and the persisted batch disposition before merge.

Do we have a high-confidence way to reproduce the issue?

No high-confidence independently executed reproduction is available in this review. The PR describes clear current-main paths and adds focused source-level regressions, so the failures are source-reproducible but not independently demonstrated here.

Is this the best way to solve the issue?

Unclear. The patch follows the existing bounded-retry and dispatch-classification design, but runtime proof is needed to show the new terminal admission and batch-output boundaries behave safely with real request handling.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3cb93c6a16bf.

Labels

Label changes:

  • add P2: This is a bounded reliability repair in a production automation queue, with meaningful but not emergency blast radius.
  • add merge-risk: 🚨 availability: Admission may complete or defer queued reviews based on live GitHub responses, so a misclassification can prevent expected review dispatch.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR reports focused tests, builds, linting, and static review, but its body explicitly says no live queue dispatch, item comment, workflow dispatch, or state change was performed; add redacted after-fix runtime output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

  • P2: This is a bounded reliability repair in a production automation queue, with meaningful but not emergency blast radius.
  • merge-risk: 🚨 availability: Admission may complete or defer queued reviews based on live GitHub responses, so a misclassification can prevent expected review dispatch.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR reports focused tests, builds, linting, and static review, but its body explicitly says no live queue dispatch, item comment, workflow dispatch, or state change was performed; add redacted after-fix runtime output before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

Likely related people:

  • yetval: Recent merged work on retry ceilings and recovery behavior directly changed the same durable queue surface this PR extends. (role: recent exact-review queue contributor; confidence: high; commits: c875ba748e2f, cf4888a6e0c0; files: dashboard/exact-review-queue.ts, test/dashboard-worker.test.ts)
  • Takhoffman: The merged dispatch-classification change is the immediate adjacent behavior for this PR's distinction between target-specific and shared failures. (role: dispatch-failure classification contributor; confidence: high; commits: cf01fe706bb7; files: dashboard/exact-review-queue.ts, test/dashboard-worker.test.ts)
  • hxy91819: The merged batch-preparation work is the relevant predecessor for the publisher completion path and coordinator fallback this PR updates. (role: batch-publication contributor; confidence: medium; commits: bcafbf8f9d26; files: src/repair/publish-event-result.ts, test/repair/exact-review-batch-workflow.test.ts)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Capture redacted after-fix queue-owner output for the closed, visible-404, inaccessible-target, and open-target admission outcomes.
  • Capture a redacted publisher/coordinator transcript proving tuple_protocol_invalid remains permanent after the publisher exception path.
  • After adding proof, update the PR body to trigger a fresh ClawSweeper review or ask a maintainer to comment @clawsweeper re-review if needed.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@brokemac79

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(exact-review): handle terminal admissions and tuple failures This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

@brokemac79
brokemac79 merged commit 8006cf0 into main Jul 25, 2026
12 checks passed
@brokemac79
brokemac79 deleted the codex/terminal-admission-tuple-failure branch July 25, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant