From 9abb0bdc6de454cf90f721a963b70e776ed02b81 Mon Sep 17 00:00:00 2001 From: nullStack65 <112618179+nullStack65@users.noreply.github.com> Date: Thu, 19 Nov 2026 18:27:31 +0000 Subject: [PATCH 1/2] test(closura): add machine-failover drill as an expected-fail test Work started on one T3 machine does not survive that machine going down. Nothing outside the machine holds the work record, nothing leases or re-places orphaned units, and a returning machine re-runs its stale copy instead of showing another machine's finished result. The drill runs two isolated T3 servers on loopback with a free mock ACP agent and links them both ways. It starts N units on A, pushes their checkpoints, SIGKILLs A and checks takeover, completion, fencing and reconciliation. Today every C check fails as the gaps file lists. The script exits 0 while the failures match that file and 3 when a listed gap starts passing. Done by Claude Opus 5.5 in Claude Code. Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 1 + docs/operations/machine-failover-drill.md | 36 +++ scripts/closura/machine-failover-drill.gaps | 7 + scripts/closura/machine-failover-drill.sh | 273 ++++++++++++++++++++ 4 files changed, 317 insertions(+) create mode 100644 docs/operations/machine-failover-drill.md create mode 100644 scripts/closura/machine-failover-drill.gaps create mode 100755 scripts/closura/machine-failover-drill.sh diff --git a/docs/README.md b/docs/README.md index 264d509d97642..85f723a0eaf4b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -66,6 +66,7 @@ source alone does not explain. Most code changes do not need an internal documen - [Development and local builds](./operations/development.md) - [Scenarios: real-app proof for a fix](./operations/scenarios.md) +- [Machine-failover drill](./operations/machine-failover-drill.md) - [T3 Connect setup](./operations/connect-setup.md) - [Release](./operations/release.md) - [Observability](./operations/observability.md) diff --git a/docs/operations/machine-failover-drill.md b/docs/operations/machine-failover-drill.md new file mode 100644 index 0000000000000..28de9f59d1734 --- /dev/null +++ b/docs/operations/machine-failover-drill.md @@ -0,0 +1,36 @@ +# Machine-failover drill + +The bar: work started on one T3 machine survives that machine going down. Another machine +takes it over and finishes it. When the first machine comes back, it shows the finished state +instead of re-running its stale copy. + +```sh +scripts/closura/machine-failover-drill.sh # expected-fail mode (default) +scripts/closura/machine-failover-drill.sh --strict # pass only if the whole bar holds +``` + +Run it from the repository root after `vp i`. It needs Node 24 or newer; set `T3_DRILL_NODE` +if that Node is not on `PATH`. It uses a free mock ACP agent, so it consumes no model usage +and takes about 3–4 minutes. + +The drill starts two servers, A and B, on loopback ports. Each has its own temporary home and +its own clone of a bare origin, and the two are linked both ways. A starts N units: one +finishes and the rest hang as long turns. Each in-flight unit pushes a checkpoint branch. +Then the drill SIGKILLs A's whole process group. It never touches `~/.t3` or a live lane. + +| Check | Asserts | +| ------------ | ---------------------------------------------------------- | +| C1.record | B holds every unit while A is down | +| C2.lease | B claims A's in-flight units within `--takeover-deadline` | +| C3.takeover | B finishes every unit, resuming from the pushed checkpoint | +| C4.fence | A, back online, runs none of the units B owns | +| C4.reconcile | A shows every unit completed, matching B | + +[`machine-failover-drill.gaps`](../../scripts/closura/machine-failover-drill.gaps) lists the +checks that fail today and the gap behind each. Exit codes: 0 means pass, or the failures +match the gaps file; 1 means an unexpected failure; 2 means a usage error; 3 means a listed +gap now passes, so delete its line. Evidence goes to the run directory's `evidence/`, which +defaults to `~/.cache/t3-machine-failover-drill/