Skip to content

test(closura): add machine-failover drill as an expected-fail test - #109

Merged
nullStack65 merged 2 commits into
release/native-0.1.10from
drill/machine-failover
Nov 19, 2026
Merged

nullStack65 merged 2 commits into
release/native-0.1.10from
drill/machine-failover

Conversation

@nullStack65

@nullStack65 nullStack65 commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Problem

The bar Kameron set on 2026-11-19: if a T3 machine goes down mid-work, another machine picks all of it up and finishes it. When the first machine returns, it shows the finished state. Today the fork fails every part of that, and nothing measured it.

Change

scripts/closura/machine-failover-drill.sh is a repeatable drill that runs as an expected-fail test, with machine-failover-drill.gaps listing the open gaps. It starts two isolated T3 servers (A and B) on loopback. Each has a temporary home, a clone of a shared bare origin and the free acp-mock-agent, and the two are linked both ways. A starts N units: one finishes, the rest hang and push checkpoint branches. Then the drill SIGKILLs A's process group. It never touches ~/.t3, scrubs inherited T3CODE_* variables and points stray CLI calls at an empty guard home.

Check Today Gap
C1.record: B holds every unit while A is down FAIL (0 of 4) G1: the work record lives only in A's SQLite
C2.lease: B claims A's in-flight units FAIL (0 of 3) G2: there is no heartbeat or lease; peers are probed only when a call is made
C3.takeover: B finishes them from the pushed checkpoint FAIL (0 of 4) G3: placement runs only at launch; checkpoints are local refs
C4.fence: A, back online, runs none of B's units FAIL (A re-ran 3) G4: ProviderRuntimeRecoveryService continues interrupted runs without an epoch check
C4.reconcile: A shows every unit completed FAIL (1 of 4) G4: a returning machine never learns that B finished

Exit codes: 0 means pass, or the failures match the gaps file; 1 means an unexpected failure; 2 means a usage error; 3 means XPASS, so delete the gap's line. --strict requires the whole bar to pass. A short ops page is in docs/operations/machine-failover-drill.md.

The agreed design (shared with the event-bus lane) is a per-lane Cloudflare Durable Object that holds a lease and epoch for each unit and a resume pointer. Each closed gap removes a line from the gaps file. Known limitation: the units are top-level threads, not a parent with delegated children, because the mock agent can't issue MCP tool calls yet.

Verification

  • Ran three times locally on release/native-0.1.10 @ 1a634c53a5, last at head 8b1917549e after the review fixes. Each run exits 0 as an expected fail, with all five C checks failing as listed, in about 4 minutes. No drill processes are left afterwards.
  • bash -n; fmt and lint passed in the pre-commit hook. No product code changed.

Deploy: none

Tracking: the machine-failover issue in closura-agent-config (nullStack65/closura-agent-config#873).

🤖 Generated with Claude Code

nullStack65 and others added 2 commits November 19, 2026 18:27
Work started on one T3 machine does not survive that machine going down. Nothing outside
the machine holds the work record, nothing leases or re-places orphaned units, and a returning
machine re-runs its stale copy instead of showing another machine's finished result.

The drill runs two isolated T3 servers on loopback with a free mock ACP agent and links them
both ways. It starts N units on A, pushes their checkpoints, SIGKILLs A and checks takeover,
completion, fencing and reconciliation. Today every C check fails as the gaps file lists. The
script exits 0 while the failures match that file and 3 when a listed gap starts passing.

Done by Claude Opus 5.5 in Claude Code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the drill found checks that could pass without the bar holding. A failed thread
listing read as "no units", so C4.fence passed. C3 accepted any unit's checkpoint in a shared
directory. C2 dropped the wrong row. An abort before the verdict could exit 3 and read as an
XPASS.

Thread listings now snapshot in the main shell and a failure ends the drill. C3 checks each
in-flight unit's own branch in its own worktree. C4 requires every unit to be listed. Any exit
before the verdict is 1. Processes are signalled only when they carry the run's marker, so
orphaned mock agents are stopped too. The ~/.t3 guard resolves symlinks before creating
anything.

Done by Claude Opus 5.5 in Claude Code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author

Bot review (one round, Claude Opus 5.5 subagent, independent of the author) and repairs at 8b1917549e.

  1. C3 counted any checkpoint in any directory, including unit 0 and the shared clone. Fixed: each in-flight unit must finish in its own worktree, and that worktree's HEAD must contain its own mf-drill/unit-k branch.
  2. C2 dropped the wrong row and counted error/stopped as claimed. Fixed: it drops unit 0 first and counts only running/starting/queued/completed. A comment notes that a lease-owner check belongs here once units record one.
  3. A failed thread listing read as "no units", so C4.fence passed. Fixed: listings snapshot in the main shell and a failure is a sanity failure. C4 requires all units to be listed on A.
  4. project add and pairing failures could exit with a raw child code, including 3, which reads as XPASS. Fixed: they now call || sanity, and any exit before the verdict maps to 1.
  5. Stdin consumed inside the read loop. Fixed: </dev/null.
  6. ss | grep -q could hit SIGPIPE under pipefail. Fixed: the output is captured, then tested.
  7. The guard ran after mkdir and missed symlinks, and orphaned children were left behind when the leader died. Fixed: the path is resolved with realpath -m before creating anything. Processes are signalled only when they carry the run's T3_DRILL_RUN marker.
  8. A free_port TOCTOU race. Not fixed (low): a stranger's listener would make S0 or S1 fail loudly, never pass falsely.

Re-verified after the fixes: the drill exits 0 as an expected fail, all five C checks fail as the gaps file lists (18:32–18:36 UTC), and no marked processes are left afterwards. The committed pre-fix script also reproduced the same result with default settings (18:28–18:32 UTC).

@nullStack65
nullStack65 merged commit 440e116 into release/native-0.1.10 Nov 19, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant