Repository navigation
test(closura): add machine-failover drill as an expected-fail test - #109
Merged
Merged
Conversation
Work started on one T3 machine does not survive that machine going down. Nothing outside the machine holds the work record, nothing leases or re-places orphaned units, and a returning machine re-runs its stale copy instead of showing another machine's finished result. The drill runs two isolated T3 servers on loopback with a free mock ACP agent and links them both ways. It starts N units on A, pushes their checkpoints, SIGKILLs A and checks takeover, completion, fencing and reconciliation. Today every C check fails as the gaps file lists. The script exits 0 while the failures match that file and 3 when a listed gap starts passing. Done by Claude Opus 5.5 in Claude Code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the drill found checks that could pass without the bar holding. A failed thread listing read as "no units", so C4.fence passed. C3 accepted any unit's checkpoint in a shared directory. C2 dropped the wrong row. An abort before the verdict could exit 3 and read as an XPASS. Thread listings now snapshot in the main shell and a failure ends the drill. C3 checks each in-flight unit's own branch in its own worktree. C4 requires every unit to be listed. Any exit before the verdict is 1. Processes are signalled only when they carry the run's marker, so orphaned mock agents are stopped too. The ~/.t3 guard resolves symlinks before creating anything. Done by Claude Opus 5.5 in Claude Code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Bot review (one round, Claude Opus 5.5 subagent, independent of the author) and repairs at
Re-verified after the fixes: the drill exits 0 as an expected fail, all five C checks fail as the gaps file lists (18:32–18:36 UTC), and no marked processes are left afterwards. The committed pre-fix script also reproduced the same result with default settings (18:28–18:32 UTC). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The bar Kameron set on 2026-11-19: if a T3 machine goes down mid-work, another machine picks all of it up and finishes it. When the first machine returns, it shows the finished state. Today the fork fails every part of that, and nothing measured it.
Change
scripts/closura/machine-failover-drill.shis a repeatable drill that runs as an expected-fail test, withmachine-failover-drill.gapslisting the open gaps. It starts two isolated T3 servers (A and B) on loopback. Each has a temporary home, a clone of a shared bare origin and the freeacp-mock-agent, and the two are linked both ways. A starts N units: one finishes, the rest hang and push checkpoint branches. Then the drill SIGKILLs A's process group. It never touches~/.t3, scrubs inheritedT3CODE_*variables and points stray CLI calls at an empty guard home.ProviderRuntimeRecoveryServicecontinues interrupted runs without an epoch checkExit codes: 0 means pass, or the failures match the gaps file; 1 means an unexpected failure; 2 means a usage error; 3 means XPASS, so delete the gap's line.
--strictrequires the whole bar to pass. A short ops page is indocs/operations/machine-failover-drill.md.The agreed design (shared with the event-bus lane) is a per-lane Cloudflare Durable Object that holds a lease and epoch for each unit and a resume pointer. Each closed gap removes a line from the gaps file. Known limitation: the units are top-level threads, not a parent with delegated children, because the mock agent can't issue MCP tool calls yet.
Verification
release/native-0.1.10@1a634c53a5, last at head8b1917549eafter the review fixes. Each run exits 0 as an expected fail, with all five C checks failing as listed, in about 4 minutes. No drill processes are left afterwards.bash -n; fmt and lint passed in the pre-commit hook. No product code changed.Deploy: none
Tracking: the machine-failover issue in closura-agent-config (nullStack65/closura-agent-config#873).
🤖 Generated with Claude Code