Repository navigation
fix(server): settle refused remote delegated launches - #641
Conversation
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 1 found one blocking defect at head 946dfe4661bf42ef38d65ff1d3878a296686abb8.
Why: A confirmed pre-launch invalid_request refusal is marked failed without settling its saved launch, so the running follower can replay it.
So what: Address the refusal settlement gap and add a focused event-driven regression before #542 is considered complete.
- P1: Confirmed invalid_request refusals can still launch later. In RemoteDelegation.ts:694-698, the retained refusal branch calls
completewithstoppedThereundefined. A concrete confirmed refusal is a missing worktree base: project/handlers.ts:233-249 returnsinvalid_requestwith “No thread was created” before launch acceptance. The caller receives failure, but RemoteDelegation.ts:489-490 still starts the follower. Its retry at line 852 re-enters launchPending at line 1242, and the guard at line 542 requires both a result andstoppedThere === true. If the missing base becomes available while the source stays running, the previously refused task can launch. This path predates the patch, but it remains within #542's explicit acceptance criterion that target-reported finalinvalid_requestrefusals must never replay. Confirmed refusals need settlement while uncertain outcomes retain same-key recovery.
Other checks: the new provider/model refusal handling is pre-acceptance; accepted receipt replay precedes admission; unknown/transport recovery and other handlers are unchanged. The new tests await persisted events without sleeps, polling, or per-test timeouts; their delegate assertions would fail on the base's pending state. PR Human summary and Elon record are present. Exact-head Fork Stack Model log ends fork-check: OK. Tests were not rerun under the read-only/no-files restriction; author red/green evidence is summarized rather than a raw transcript.
Model: gpt-6.1-sol (high effort). Harness: Codex in T3 Code.
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
|
What: Fixed round 1's P1 and pushed PR #641 head Review response: #641 (review). The extended existing real-destination table now covers a missing worktree base during both initial delegation and recovery. The Git boundary reports the base missing, while the real destination launch service and handler produce Refusal audit and implementation:
Proof on the pushed code:
Reflection: the earlier reasoning equated an error code with launch outcome and retained an existing uncertainty branch without testing a confirmed invalid-request refusal. The correction lives at the transport provenance boundary and in the existing behavioral tests; no duplicate instruction or preference was added. Requirements and who asked: #542 and the coordinator's round 1 response require all confirmed refusals to settle, with same-key recovery preserved for unknown outcomes. Workspace |
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 2 found two blocking P1 defects at exact head 5f5866c3e89a4fe18ac220d6ac166a26bde514f9.
Why: Definite refusals before forwarding can still launch later, while JSON-RPC internal errors are now mistaken for proof that nothing was accepted.
So what: Correct both classifications and add focused regressions before #542 is complete; this review grants no merge approval.
Reviewed the complete four-file PR diff against base ec50dd0af2af1efc261ab0471966141d1a8062d8, the Issue, round 1 findings, and surrounding transport, forwarding, launch-receipt, orchestration, persistence and test code. Reviewer identity: Codex harness, gpt-6.1-sol, high reasoning effort, this assigned independent round; I did not author or modify the candidate.
Blocking findings
-
P1: Settle definite first-attempt refusals from the forwarding preflight too. The new gate at apps/server/src/peer/RemoteDelegation.ts:700 requires
isToolRefusal, but apps/server/src/peer/PeerForwarding.ts:394 explicitly returns an ordinaryinvalid_requestwhen the destination does not advertise bare launch, saying “nothing was forwarded.” A remote bare delegation to a peer that supports lifecycle v1 but not bare launch has already saved its task and launch marker before this check. On this first attempt there is no acceptance uncertainty, yet it falls through to apps/server/src/peer/RemoteDelegation.ts:717, which records failure withoutstoppedThere. The ensured follower (apps/server/src/peer/RemoteDelegation.ts:489) retries throughfollowOnce(apps/server/src/peer/RemoteDelegation.ts:1265); the settled guard at apps/server/src/peer/RemoteDelegation.ts:542 does not stop it. If the peer is upgraded to advertise bare launch while that follower remains alive, work already reported failed launches. The tool-allowlist and execution-purpose preflights (apps/server/src/peer/PeerForwarding.ts:379, apps/server/src/peer/PeerForwarding.ts:415) have the same mechanism. Explicit session-initialization refusals also remain unmarked (apps/server/src/peer/PeerMcpClient.ts:369). Preserve evidence that dispatch never happened and settle these definite first-attempt failures, while retaining uncertainty for a replay whose earlier attempt may have succeeded. Add a regression that restores the refused capability while the existing follower remains alive and verifies no child launches. This is an in-scope remaining refusal-settlement gap, not an unrelated cleanup. -
P1: Do not classify every JSON-RPC error as a confirmed invalid-request refusal. apps/server/src/peer/PeerMcpClient.ts:881 discards
body.error.codeand constructsPeerToolFailure({ code: "invalid_request" })for every numeric JSON-RPC error. The accepted response shape permits any numeric error code (apps/server/src/peer/PeerMcpClient.ts:267), including-32603(internal error) and implementation-defined server errors, which the JSON-RPC specification distinguishes from invalid request/params. An internal error is not evidence that the launch transaction never committed. For a destination that accepts a launch and then returns an internal error while producing its response, the new classifier at apps/server/src/peer/RemoteDelegation.ts:699 writesstoppedThere: true, both initially and on replay because the code was relabeledinvalid_request. The source then skips same-key receipt recovery and can abandon accepted work while reporting it settled. Previously this error remained unsettled. Preserve internal/unknown RPC outcomes as recoverable and mark only errors that prove refusal; add an accepted-receipt/internal-error regression at the real transport seam. This is a new regression in the candidate. Static protocol trace, not a locally executed reproduction.
Complete contract-code audit
The actual contract has 23 codes at packages/contracts/src/orchestratorMcp.ts:1065. For a provenance-marked explicit destination response, the candidate handles them as follows:
invalid_request,provider_unavailable,model_unavailable: failed and settled on first attempt and replay.capability_denied,runtime_mode_escalation_denied,interaction_mode_escalation_denied,thread_not_found,run_not_found: failed and settled on first attempt; failed but unsettled on replay, with the “work may still be running” warning.parent_not_active,research_host_unavailable,research_budget_exceeded,session_unavailable,session_expired,session_in_use,session_warning,task_not_found,task_not_cancellable,thread_not_sendable,thread_not_interruptible,thread_credential_required,target_required: failed and settled on first attempt; propagated without settlement on replay. Several belong to other tools and have no identified producer in this launch path; their presence in the common failure contract alone is not a defect.outcome_unknown,orchestration_error: propagated and recoverable on both paths, retaining the stored namespace, client key and grant identity. Finding 2 bypasses this intended classification by relabeling numeric internal errors.
Unmarked failures do not gain settlement from the new tool-refusal branch: the five mode/final codes above plus invalid_request use the existing failed/unsettled path; other codes propagate. A separate new first-attempt lifecycle-preflight catch settles its final/mode/invalid_request refusals, but does not cover the forwarding preflights in finding 1.
Cleared paths, proof and limits
- Round 1 P1 resolved: the real missing-worktree-base
invalid_requestnow has local transport provenance and reaches failed/settled completion. The destination checks the base in the admission callback (apps/server/src/mcp/toolkits/project/handlers.ts:233), afterreplayOrLaunchreads the receipt (apps/server/src/orchestration-v2/ThreadLaunchService.ts:317). - Uncertain acceptance: existing lost-response tests retain a single accepted child even after model/headroom loss. Existing lowered-parent, revoked-link and replaced-link cases retain the warning and explicitly assert
stoppedThere !== true(apps/server/src/peer/RemoteDelegation.test.ts:2383). These supported cases are preserved; they do not cover finding 2. - Serialization/migration: the symbol marks only newly constructed in-process errors; the wire failure schema and persisted launch shape are unchanged. Settlement uses the existing
stoppedTherefield and completion command. Existing legacy serialization coverage remains in the suite (apps/server/src/peer/RemoteDelegation.test.ts:3135); no migration execution was performed. - Focused tests: the six new cases use the real destination launch handler/service with catalog, usage and Git boundary fakes.
taskResultawaits storedsubagent.updatedevents (apps/server/src/peer/RemoteDelegation.test.ts:1428); added code has no sleep, polling, per-test timeout or retry. On the PR base, provider/model delegate cases remain pending and the missing-base case fails the settled-marker assertion, matching the author's reported red failures. However, the rows labeled “recovery” callrecordPending, which does not setlaunchStarted(apps/server/src/peer/RemoteDelegation.test.ts:2190), so they still exercisefirstAttempt === true, not a previously dispatched uncertain launch. Neither blocking mechanism is covered. - Existing exact-candidate evidence: inspected the PR/Issue's final 59/59 focused-test report, reported scoped lint/typecheck repair after the aggregate checker failed, and reported test-first failures. These local results are summaries, not raw transcripts available to this review. Exact-head CI Build and Typecheck, Lint, and completed server shards passed; other server shards were still running at inspection. I do not claim the full gate is complete.
- Fork rules/evidence: read
docs/fork.mdandscripts/fork-check.sh; the check enforces a merge-free stack, registered upstream edits, ownership/consolidation and browser-doc rules. Exact-head Fork Stack Model run 42271725310 identifies this SHA and its log endsfork-check: OK.; feature ownership and consolidation checks pass. The map change attaches this PR to the existing lifecycle part. No version bump. - PR Human summary and all four Elon fields are present. Its upstream-first record is linked to #542. This remains an existing-component server bugfix supporting Drafters; no client/provider-adapter changes require browser proof.
- Tests were not rerun because the user forbids file writes. No checkout, reset, worktree, implementation or test-artifact creation was performed. Main checkout remains
38644e601a24b2ae6447b533cce32c1357799f9a; dirty user-ownedpnpm-lock.yamlandscripts/__pycache__/were left untouched. No physical two-machine/provider or live product proof.
Requirements and who asked: User-assigned independent round 2 of #641 for #542: every confirmed refusal settles, unknown outcomes retain same-key recovery, read-only review.
Deleted: No implementation, new components, test execution, workspace changes, PR takeover, watch, merge or deployment.
Bottleneck: Distinguishing definite no-dispatch/refusal evidence from errors that cannot rule out an already accepted launch.
Checked myself: Read exact-candidate diff and contracts, traced initial/replay/preflight/receipt/persistence paths, inspected regression wait/oracle behavior and available candidate-bound proof, and verified live head before publishing.
Verdict: FAIL at 5f5866c3e89a4fe18ac220d6ac166a26bde514f9, two P1 findings. Reviewer: Codex / gpt-6.1-sol / high effort.
|
What: Both round 2 P1s are addressed at head Canonical response, complete red/green evidence and handoff. Final requested suites: 81 passed; corrected scoped lint/typechecks and fork-check passed. The once-only aggregate check found lint/type errors that were corrected and revalidated individually, as detailed in that record. |
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 3 FAIL at exact head 6aef9b574ca070632653bde21eb4e3e462d9656b, with two P1 findings.
Why: Protocol errors during replay can falsely settle accepted work, and one definite no-dispatch preflight still remains replayable.
So what: Both paths need correction and focused regressions before a fresh exact-head independent review; no merge approval is implied.
-
P1: Preserve prior acceptance uncertainty when a replay receives a protocol refusal. PeerMcpClient.ts:898 maps
-32700/-32600/-32601/-32602toinvalid_request; itsPeerRpcFailuremarker at line 304 makes these satisfyisToolRefusal. RemoteDelegation.ts:693 then settles this error even whenfirstAttemptis false. If the original launch committed its receipt but its response was lost, a later protocol/schema refusal before destination receipt lookup disproves only the replay's acceptance. The source nevertheless completes withstoppedThere: true, and the guard at line 542 permanently prevents recovery of the original accepted child. Protocol refusal provenance must stay distinct from a post-receipt admission refusal on replay. Extend the real accepted-receipt test with a protocol-error replay, then restore compatibility and assert same-key attachment to the original child, with exactly one child. -
P1: Mark incompatible durable-request negotiation as definite no-dispatch. PeerMcpClient.ts:1160 still returns ordinary
OrchestratorMcpFailurewhen an HTTP-200 request-protocol response fails decoding, explicitly stating that no mutation was forwarded. PeerForwarding.ts:360 invokes this before the mutation. The launch intent andlaunchStartedmarker already exist, but this error has neither dispatch nor tool-refusal provenance, so RemoteDelegation takes its failed-but-unsettled branch at line 706. Restoring compatible negotiation lets the ensured follower replay and launch work already reported failed. Preserve the dispatch-refusal marker on this decode failure, and cover an incompatible HTTP-200 response followed by restoration while the original follower is alive; assert no child launch and durable settlement.
Prior findings disposition: the named round 2 forwarding capability/session-init refusals and numeric -32603 regression are corrected. The new source regressions restore capability with the follower alive and recover a real accepted receipt through HTTP/MCP; their event/clock synchronization follows project rules. The table now explicitly separates unsent recovery from actually dispatched recovery and asserts launchStarted === true. Round 1's missing-base refusal remains covered. The two paths above are not covered by these regressions.
Review scope/proof: complete six-file diff and surrounding consumers, exact-head source tests, Issue #542 and prior review context, PR body and CI evidence. No files were written, no checkout was changed, and no tests were executed locally under the read-only restriction. The author's 81-pass focused run and repaired scoped checks are reported evidence, not independently rerun proof. The exact-head Fork Stack Model log was inspected and ends fork-check: OK. The PR has its Human summary, Elon record, upstream-search record and map reference; no version bump or unrelated changes were found. This review supports the Objective by preventing unexpected remote Drafter launches and abandoned accepted work.
Requirements and who asked: User and #542 require confirmed launch refusals to settle permanently and uncertain prior acceptance to retain same-key recovery.
Deleted: No implementation changes, new tests, worktree, deployment, or merge are part of this read-only review.
Bottleneck: Evidence must distinguish the current RPC attempt's refusal from proof that the durable launch was never accepted.
Checked myself: Independently traced the protocol-error marker into replay settlement and the incompatible HTTP-200 preflight into the unsettled follower path; inspected the corrected requested regressions and dispatched table rows at the exact SHA.
Model: gpt-6.1-sol, high effort. Harness: Codex in T3 Code.
|
What: Both round 3 P1s and the requested structural redesign are delivered at Canonical response, design boundaries, failing-before evidence and handoff. Final focused proof: 92 tests across the requested suites plus forwarding, scoped lint/typechecks and post-commit fork-check passed. The once-only aggregate-check failures and their scoped revalidation are detailed in that record. |
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 4 FAIL at exact head 764ebe3470f1e1616c6420682bad993f555b62db, with two P1 findings.
Why: HTTP-level refusals still bypass settlement, and an internal session retry can turn uncertainty about an accepted launch into false settlement.
So what: Correct both transport paths and add focused event-driven regressions before a fresh exact-head review; no merge approval is implied.
Reviewed the complete nine-file diff against ec50dd0af2af1efc261ab0471966141d1a8062d8, Issue #542, prior reviews, transport/forwarding/receipt/recovery consumers, and the pinned Effect 4.1.2 source in .repos/effect.
-
P1: Normalize explicit HTTP refusals through the dispatch observer. PeerMcpClient.ts:863-872 yields HTTP 401 and decoded non-200 refusals directly as Effect failures. The
refusedobservation at PeerMcpClient.ts:957-964 only runs for a successfully returnedResult.fail, so neither branch reaches it. After the launch POST, dispatch remainssent; RemoteDelegation.ts:693-707 records a finalcapability_denied/invalid_requestas failed without settlement. Orchestrator.ts:9572-9575 explicitly recordsstoppedThere: false, leaving it eligible for startup recovery and the ensured live follower. This is a real destination path: McpHttpServer.ts:349-369 refuses unusable credentials before handling, and McpHttpServer.ts:419 returns decoded authority refusals as HTTP 403 viapeerAuthorityFailure(lines 290-296). A first-attempt explicit HTTP refusal therefore remains replayable after the refusing condition is restored, violating #542. Cover an actual mutation POST refusal followed by restoration with the same follower/key, asserting failed/settled state and zero children; preserve uncertainty for these responses on replay. -
P1: Treat an internal resend as a replay when the preceding send may have committed. PeerMcpClient.ts:857-861 renews the session and sends the same mutation again on HTTP 404. Both requests share the caller's captured
firstAttempt === trueand mutable dispatch stage. If the renewed request returns a protocol refusal, the client overwritessentwithrefusedat line 963, and RemoteDelegation line 693 settles without negative-receipt evidence. HTTP 404 does not prove that the earlier send was never accepted in this server: pinned McpServer.ts:1975-1985 installs a pre-response hook that replaces the response with 404 if the session disappears after processing. McpHttpServer.ts:2223 enables session termination, which deletes the session before notifying/interruption of outstanding requests (mcpRuntime.ts:486-500); it cannot undo an already committed command receipt. A launch committed before that response edge, followed by session renewal and a pre-lookup protocol refusal, is thus abandoned as settled despite its accepted child. The new separate-launchPendingreplay regression does not cover this internal resend. Preserve earlier-send uncertainty across renewal and cover accepted receipt -> HTTP 404 -> renewed protocol refusal -> restoration, requiring same-key attachment to exactly the original child.
Round 3 disposition: both requested regressions are now present. RemoteDelegation.test.ts:2256-2351 covers all four protocol codes on a later source attempt, recovery-query inclusion, restoration, the original key/thread, and exactly one child. RemoteDelegation.test.ts:2199-2251 covers incompatible HTTP-200 request negotiation with the follower alive and zero launches after restoration. Their synchronization uses stored events and TestClock, without polling/sleeps/retries/per-test timeouts.
The redesign otherwise follows the intended ownership: capability probes omit the observer, preparation starts not_sent, receipt evidence is emitted inside the destination admission callback after lookup with a second receipt check, and only the launch-tool decoder preserves it. Missing receipt services, receipt-read failures, internal errors and ordinary protocol replay do not establish negative-receipt evidence. The old PeerDispatchRefusal, PeerToolFailure, isDispatchRefusal, isToolRefusal and per-preflight wrappers are removed. The Orchestrator change preserves existing confirmed settlement when a late uncertain response arrives.
Remaining dispatch-fact audit:
sentat PeerMcpClient line 825 means attempted HTTP handoff, not bytes flushed. It precedes request/header/body construction, HttpClient URL/tracing work and Fetch invocation. Synchronous construction defects, cancellation before execution, DNS/TCP/TLS failures and aborted/unflushed writes can therefore leave it set without destination receipt. Typed transport errors stay recoverable; no delivery acknowledgment is available here.- A durable
launchStartedis committed before preparation. A crash/interruption between that marker and POST loses the in-memory no-send fact; recovery conservatively treats it as a replay. - The client explicitly resends once after 404 using unchanged input/key, but lacks per-send acceptance history. Finding 2 is the unsafe consequence.
- Production injects FetchHttpClient directly (server.ts:1230,1250). The pinned Fetch adapter performs one fetch per execution, with no retry decorator on this mutation path; its default Fetch redirect behavior can perform additional network requests below this observer. PeerMcpClient's mutation
postdoes not setredirect: "manual". Runtime Fetch/connection internals provide no byte-flush proof, so hidden transport attempts cannot be counted as receipt evidence. - Session initialization, notifications, lifecycle/request negotiation and capability probes correctly do not mark the mutation sent. Errors after successful tool decoding/pair registration retain
sent, which is conservative about an already accepted child. - Explicit HTTP failures bypass
refused, as in finding 1; subsequent normalized refusals overwrite previous-send uncertainty, as in finding 2.
Proof/limits: no files were written or tests rerun under the user's restriction. Exact-head source assertions and the author's reported 92 passing focused cases were inspected; these are not an independently executed runtime reproduction. The exact-head Fork Stack Model log (run 42278899608) names this SHA and ends fork-check: OK.; feature ownership and consolidation passed. Build/typecheck and lint passed; other CI jobs were still running at inspection. PR Human summary, all four Elon fields, upstream-search record and map reference are present; no version bump. No browser/provider or two-machine runtime proof is claimed.
Requirements and who asked: User-assigned round 4 for #542 requires every definite refusal to settle permanently while possible acceptance retains same-key recovery.
Deleted: No implementation, test execution, workspace changes, new components, merge or deployment.
Bottleneck: Transport evidence must cover every explicit response and retain uncertainty across every send, including session renewal.
Checked myself: Read exact-head diff and pinned transport source, traced HTTP refusal bypass and internal resend into settlement/recovery, inspected both round 3 regressions and exact-head fork CI.
Model: gpt-6.1-sol, high effort. Harness: Codex in T3 Code.
|
What: Both round 4 transport P1s and manual redirect handling are fixed at Canonical response and complete red/green proof. Requested files: 92 passed. The once-only changed-scope gate passed all stages, including 36-package typechecking and 148 selected tests; post-commit fork-check passed. |
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 5 FAIL at exact head deeb5f71ea70742232356e0ca66b5137bfa5cfbb, with one blocking P1.
Why: Overlapping same-key launch attempts can each retain first-attempt settlement authority, so one attempt's no-send failure can permanently abandon another attempt's accepted child.
So what: Make first-attempt settlement authority exclusive for the task and add an event-driven overlap regression before a fresh exact-head review; no merge approval is implied.
P1: A concurrent same-key attempt can falsely settle an accepted launch. RemoteDelegation.ts:550 captures firstAttempt from a projection read before dispatching the shared launch marker. Two overlapping delegate_task invocations with the same key can both read launchStarted !== true. Their marker commands have the same command ID (line 573); replaying that command does not change either invocation's captured firstAttempt, and both continue after observing launchStarted === true (lines 590-609).
A reachable interleaving is: both read the unstarted task; A dispatches the marker and sends the launch; B replays the marker and pauses in lifecycle preparation; A's destination commits the child but its response is lost; B's lifecycle preparation fails before its own mutation POST. At RemoteDelegation.ts:693, B has firstAttempt === true and its private dispatch stage is not_sent, so it completes the task with stoppedThere: "stopped". That becomes persisted stoppedThere: true (Orchestrator.ts:9572-9576); subsequent launch recovery returns immediately at RemoteDelegation.ts:542. A's possibly accepted launch therefore loses same-key recovery, despite no destination admission refusal establishing receipt absence.
The production entry path does not serialize these calls: OrchestratorMcpService.ts:3311 invokes remoteDelegation.delegate directly; McpRouting.ts:420-429 binds the request identity and runs the handler, without exclusive execution. McpRequestReceipts.identify binds input but does not claim an execution lease. Individual command serialization cannot protect the read/dispatch/network/settlement sequence. The regression should hold both reads before the marker, commit A's destination receipt and lose its response, fail B before its send, then restore connectivity and require recovery of exactly A's original child with the original key.
Round 4 findings are cleared: HTTP 401 and decoded non-200 refusals now flow through the normalized observer; the observer reports refused only when sends === 1; internal 404 renewal retains earlier-send uncertainty; the mutation POST uses redirect: "manual". The event-driven table in RemoteDelegation.test.ts:2221 covers actual mutation HTTP refusals with restoration and zero children, and the table at line 2282 covers accepted receipt -> 404 -> renewed protocol/HTTP refusal -> recovery of exactly one original child. PeerMcpClient.test.ts:159 covers zero POSTs at a redirect target. Earlier preflight, admission, replay and monotonic-settlement protections remain present. No separate non-blocking hardening finding.
Proof and limits: the concurrency P1 is established by production source tracing, not an executed reproduction. Under the requested read-only restriction, no files were written and no local tests were run. Delegated exact-head CI inspection found 76 RemoteDelegation and 16 PeerMcpClient cases passing; Fork Stack Model passed with fork-check: OK. (run). The PR includes its required Human summary, all four Elon fields, upstream search reference and map update; no version bump. Some full-suite CI jobs were still running at inspection, so complete CI success is not claimed.
Requirements and who asked: User-assigned round 5 against #542 requires definite no-dispatch/refusal settlement while possibly accepted launches retain same-key recovery.
Deleted: No implementation, test execution, checkout changes, new components, merge or deployment.
Bottleneck: First-attempt settlement authority is captured per invocation while acceptance uncertainty belongs to the shared task.
Checked myself: Verified the exact live PR head, inspected the source path from request identity through overlapping launch markers and settlement/recovery, checked round 4 transport fixes and regression assertions, and adjudicated the delegated finding against production callers.
Model: gpt-6.1-sol, high effort. Harness: Codex in T3 Code.
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 6 FAIL at exact head 527b3b864e89362fc279ed8a17f3af742a292620, with one blocking P1.
Why: Marker ownership is exclusive, but launch execution is not, so one invocation's no-send evidence still cannot establish that the shared task was never dispatched.
So what: Correct the remaining overlap and cover both orderings before a fresh exact-head review; no merge approval is implied.
P1: Concurrent same-key launches can still abandon accepted work or launch settled work. RemoteDelegation.ts:592 correctly grants first-attempt authority only to the invocation that newly applied the marker. However, a replay continues into preparation and forwarding too, while the owner retains that authority. The dispatch stage at line 624 is private to each invocation. Two reachable schedules remain:
-
An accepted child loses recovery. A newly applies the marker and waits in the inner lifecycle probe at line 653. B replays the marker, sends the launch, and the destination commits its child/receipt, but its launch response is lost or repeatedly replaced with an internal error. A's independent lifecycle GET then times out (PeerMcpClient.ts:1059, timeout at line 1087). A still has
firstAttempt === trueand its own stagenot_sent, so RemoteDelegation.ts:696 persists failed/stoppedThere: true, without negative-receipt evidence. The settled guard at line 542 then suppresses original-key recovery of B's accepted child. -
A settled no-dispatch task still launches. Hold B after its replayed marker dispatch, as the new test already does. Let A fail before sending and settle the task; restore the refusing condition, then release B. B rereads the task at lines 593-595, but only tests
launchStarted, which remains true. It proceeds to forwarding without reapplying the settled guard. The source authority path also permits a failed task: PeerAuthority.ts:3060 rejects cancelled/interrupted tasks, not this failed/settled task. The destination can therefore launch work after the source reported its definite refusal as permanently settled.
Both arise from the same missing task-wide ordering between launch attempts and settlement. Production does not serialize the whole operation: OrchestratorMcpService.ts:3311 directly invokes delegate; MCP request identification binds input rather than claiming execution; PeerForwarding.ts:1125 sends directly; orchestration locks cover individual commands. The marker owner must not settle from private no-send/refusal evidence after another invocation may have sent, and an invocation already in flight must not send after definite settlement. Cover both orderings with Deferred/event-driven regressions, retaining original-key recovery of exactly the accepted child and zero launches after definite settlement, respectively.
Round 5 disposition: The exact requested owner-accepted/replay-unsent ordering is corrected and covered by the new test at RemoteDelegation.test.ts:2282. It does not exercise the two orderings above. Rounds 1-4 remain cleared in the inspected paths: HTTP refusal normalization, internal 404 resend history, manual Fetch redirects, preflight settlement, post-receipt-lookup admission evidence, conservative protocol/internal replay failures, startup-recovery eligibility and monotonic settlement remain present. No separate non-blocking hardening finding.
Proof and limits: Read the complete nine-file exact-candidate diff against base ec50dd0af2af1efc261ab0471966141d1a8062d8, surrounding production consumers, Issue #542 and prior reviews. Independently challenged the overlap with a delegated reviewer and verified its findings against source. This P1 is established by production source tracing, not an executed reproduction. No local tests ran and no files or checkout were changed under the user's restriction. The new regression uses Deferreds and persisted-event observations without sleeps, polling, retries or per-test timeouts. The PR Human summary, four Elon fields, upstream-search record and map reference are present; no version bump. Exact-head Fork Stack Model run 42285151059 identifies this SHA and ends fork-check: OK.; build/typecheck and lint passed at inspection, while test jobs were still running. Complete CI success is not claimed. Objective contribution: safe remote Drafter launches and recovery.
Requirements and who asked: User-assigned round 6 for #542 requires permanent definite-refusal settlement and same-key recovery whenever acceptance remains possible.
Deleted: No implementation, file writes, local test execution, workspace changes, new components, merge or deployment.
Bottleneck: The shared task's acceptance uncertainty outlives the marker owner's private transport observation.
Checked myself: Verified the exact live head, traced both overlapping schedules through preparation/forwarding/authority/settlement/recovery, inspected the new test's ordering and prior fixes, and checked candidate-bound fork CI.
Model: gpt-6.1-sol, high effort. Harness: Codex in T3 Code.
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review round 7 PASS at exact head fd4ca4bb26c6c2b6776855043f2560e5b28ca81f, with no P1 blockers or nonblocking findings.
Why: Every launch attempt now takes the same per-task lock before reading state, and retains it through preparation, forwarding, settlement and attachment.
So what: Round 6 is cleared; this review does not grant merge approval.
Reviewed against #542 and base ec50dd0af2af1efc261ab0471966141d1a8062d8. The lock prevents both round 6 schedules: private no-send settlement after a competing send, and a queued send after definite settlement. Queued holders re-read settled state. The transport synchronously records task-wide sent evidence before each mutation POST, and restarts retain conservative recovery through the durable marker.
Both Deferred-driven overlap regressions preserve the real lock, source commands/projections, HTTP/MCP and destination receipt path. They check original-key recovery of exactly the accepted child and zero launches after settlement. Static inspection found no nested acquisition: the delegate's ensured follower is forked outside the locked launch effect. The shared KeyedLock uses acquireUseRelease and semaphore permits for failure/interruption cleanup; different task ids have independent semaphores.
Rounds 1-5 remain cleared in the inspected paths: preflight classification, normalized refusals, 404 resend history, manual Fetch redirects, admission evidence only after receipt lookup and recheck, conservative uncertain outcomes, recovery eligibility and monotonic settlement.
Proof limits: no tests or local fork checks rerun under the read-only constraint; interruption cleanup was assessed from source. Exact-head Fork Stack Model checks passed on GitHub. The PR Human summary, four Elon fields, upstream record and map reference are present; no version bump. No files or checkout changed, and user-owned dirty paths remain untouched.
Requirements and who asked: User-assigned round 7 for #542 requires permanent definite-refusal settlement, same-key recovery for possible acceptance, and correction of both round 6 overlap schedules.
Deleted: No implementation, local test execution, workspace changes, merge or deployment.
Bottleneck: Ordering every launch and settlement against the shared remote task.
Checked myself: Verified exact live head/base, traced lock scope, follower/recovery entry points, transport handoffs, receipt evidence and settlement, read both overlap regressions, independently challenged the change through the reviewer role, and checked exact-head fork CI.
Model: gpt-6.1-sol, high effort. Harness: Codex in T3 Code.
What: Remote launch refusals settle permanently while uncertain earlier acceptance remains recoverable across response loss and session renewal.
Why: Record-first recovery replayed rejected work, and incomplete transport classification or overlapping launch attempts could abandon an already accepted child.
So what: Closes #542; the coordinator owns current-head CI and independent review round 7, with human merge approval required.
PeerMcpClient owns dispatch progress. Preparation starts
not_sent; capability probes and negotiation do not mark the mutation sent. Explicit HTTP and MCP refusals use one normalized result path. Each logical RPC retains its count of actual HTTP client handoffs: a refusal may reportrefusedonly after a sole send. A 404-triggered session resend cannot replace earlier-send uncertainty with plain refusal. Fetch redirects are manual, preventing extra POSTs below the observer; other Fetch options and HTTP link-error diagnostics are preserved.RemoteDelegation holds one in-process keyed lock per remote task from its first task read through marker dispatch, preparation, forwarding, settlement and attachment. All startup/follower recovery uses the same lock. One server owns these tasks. Queued attempts re-read settled state before sending. First-attempt settlement authority requires newly applying the durable marker and no earlier task send; a task-level sent set is updated by the transport observer before each mutation POST. Restarts conservatively derive replay behavior from the durable
launchStartedmarker.For the first attempt, no-dispatch failures and explicit final refusals settle. Replay settlement requires destination
launchReceipt: "absent"evidence, emitted only from admission after receipt lookup and a second receipt-store check. Protocol/internal errors, missing receipt services and receipt-read failures provide no such evidence. Only the launch-tool failure decoder accepts it. Unconfirmed failed tasks remain eligible for startup recovery, and late uncertainty cannot undo confirmed settlement.Removed per-site refusal classes/predicates and wrapper maps. The optional receipt-evidence field is the only additive failure-contract change; there is no persisted-schema migration or new retry loop. Older peers without evidence remain conservative on replay.
Two Deferred-driven overlap regressions reproduce both old schedules: an accepted child falsely settles when the marker owner fails before send, and a queued invocation launches after definite settlement. With the lock, the competing holder cannot overtake; lost acceptance followed by a queued no-send failure retains recovery of exactly the original child/key, while pre-send settlement followed by restored capability produces zero launches. The fixture executes real source commands, HTTP/MCP and destination receipts; the lock observer preserves the original lock implementation.
Regressions also cover accepted receipt -> HTTP 404 -> renewed protocol/HTTP refusal -> restored compatibility, with the same key and exactly one child. Actual mutation POST refusals are restored without restarting the source and must launch no child. A real Fetch redirect target receives zero extra POSTs. Existing ordinary replay, preflight, admission, cancellation and monotonic-settlement cases remain.
Upstream: main lacked this pending-launch implementation; related pingdotgg/t3code#30249 concerns recovery without a matching refusal fix. No upstream code was ported. Search record.
Requirements and who asked: User and #542 require permanent confirmed-refusal settlement and same-key recovery for uncertain acceptance; round 6 requires one task-wide in-process lock, task-level sent evidence and both overlap regressions.
Deleted: Separate HTTP refusal handling, hidden redirect resends and the assumption that exclusive marker ownership orders launch execution; no distributed lease, persisted migration or retry layer.
Bottleneck: Every launch attempt must order its preparation, sends and settlement against the shared task, including queued recovery attempts.
Checked myself: Reproduced both round 6 durable failures against the previous exact head; verified queued exclusion, real receipt/key/child identity, zero launches after settlement, and the focused, aggregate and post-commit fork checks.
Proof on head
fd4ca4bb26c6c2b6776855043f2560e5b28ca81f:pnpm exec vp run check:changed --concurrency 1ran once and passed all stages: format, lint, both knip passes, GraphQL, 36-package typecheck and 150 tests in 10 selected files.bash scripts/fork-check.sh --upstream upstreampassed after commit before push. Commit hooks passed format, lint and knip.All local proof was niced and single-job/serialized on rocky. The send count records HTTP handoffs, not a delivery acknowledgment; transport errors stay uncertain. CI owns the full suite/build/release-smoke gate. No merge, version bump, deployment, live-data write, physical two-machine/provider or browser verification. Objective contribution: safe cross-project Drafter launches. Website impact: none, server lifecycle correction only.
Model: gpt-6.1-sol (high effort). Harness: Codex in T3 Code.