Skip to content

test: prove lease revival after sleep and trace drain on continuation release (U6) - #23

Merged
Steel-tech merged 1 commit into
mainfrom
test/ci-repair-gaps
Sep 28, 2026
Merged

Steel-tech merged 1 commit into
mainfrom
test/ci-repair-gaps

Conversation

@Steel-tech

Copy link
Copy Markdown
Contributor

Implements U6 of the factory-quality follow-ups plan (PR #21): closes the two test gaps that CI repair shipped with (R12, R13). This PR changes tests only. No production code changed.

R12: lease revival after a gap (internal/worker/publish_repair_test.go)

Each test runs a real repair round through the worker, control plane and git remote. While the round runs, the lease is expired on the server (expireLease, left unswept) and the lease's injected clock (attemptLease.now) is set. The fence the round meets next is the pre-push authorizeRound.

Test Clock Server state Proves
TestAStaleClockRevivesAnExpiredLeaseBeforeTheRoundIsAuthorized now + HeartbeatInterval (stale) expired, no successor freshen sends a heartbeat, the heartbeat revives the lease, and the round is then authorized, pushed, recorded, and judged green. The attempt is accepted.
TestAFreshClockSkipsTheHeartbeatAndTheExpiredLeaseIsRefused pinned to the last renewal (fresh) expired, no successor freshen skips the heartbeat, so authorization is refused with lease_not_owner and nothing is pushed. This shows it was the heartbeat that revived the lease in the first test.
TestAStaleClockDoesNotReviveASupersededLease stale swept, job retried, successor exists The heartbeat is refused with lease_superseded. Nothing is pushed and no round is recorded.

The publishing runner's Outcome is captured by wrapping w.config.Runner. When the lease stays lost, the attempt's completion is also refused, and the round's own stop code can only be read from that captured outcome. The only other change to the test scaffolding is that scriptedContinuation now keeps the attempt's lease.

R13: releasing the continuation drains the trace (cmd/jig/ci_repair_test.go)

TestReleasingTheRepairContinuationDrainsTheTraceBeforeTheAttemptCompletes runs through the real workerAttemptRunner, which is wrapped in the publishing runner, against an in-process jig serve. The setup it shares with the existing end-to-end test is moved into helpers.

Determinism mechanism. TraceStream.Emit wakes the sender on every event. With a healthy control plane, events therefore reach the store long before release, whether or not the host closes the trace. I confirmed this: the existing end-to-end test still passes with closeTrace removed. An "events == JSONL" comparison on its own proves nothing.

The test puts a gate (a reverse proxy) between the worker and the control plane. The gate refuses POST …/events until CI goes green on the round's fix, and a probe checks that the store holds 0 events at that point. When the continuation is released, every event from both the chain and the round is still buffered, and the background sender is in retry backoff. TraceStream.Close drains synchronously, and the host's release closure runs before the attempt completes. So the assertion runs as soon as ClaimOnce returns, with no sleeping and no polling: the control plane must hold exactly the attempt's JSONL raw record, compared on seq, type, phase, name and payload, and including ci_repair_start.

I chose this over a test flush interval because a longer flush interval would not stop the per-emit wake. It would also have needed a new production seam in OpenTrace.

Mutation checks

Each mutant compiled, and each file was restored with git checkout after the commit.

Mutant Result
(a) remove freshen from authorizeRound The stale-clock revival test fails 3/3 on lease_not_owner. The superseded test also fails, because it gets lease_not_owner instead of lease_superseded. The fresh-clock control still passes, as intended.
(b) remove closeTrace() from the DeferToContinuation cleanup in cmd/jig/worker.go The R13 test fails 10/10: "control plane holds 0 event(s), the JSONL raw record 22". The pre-existing end-to-end test passes 5/5 against this mutant.

Flake checks

  • go test -race -count=20 -run 'StaleClock|FreshClock' ./internal/worker/: ok
  • go test -race -count=20 -run 'CIRepair|RepairContinuation' ./cmd/jig/: ok
  • just check: ok

🤖 Generated with Claude Code

… release (U6)

R12: a repair round whose lease expired server-side while it ran is
revived by freshen's heartbeat when the local clock says the last renewal
is older than HeartbeatInterval, and is then authorized, pushed, and
recorded. The same round with a fresh clock is refused on lease_not_owner,
and a superseded lease (a successor attempt exists) is never revived.

R13: through the real workerAttemptRunner wiring, trace ingestion is
refused until CI goes green on the round's fix, so the whole trace is still
buffered when the continuation is released. Once ClaimOnce returns, the
control plane holds exactly the attempt's JSONL raw record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 28, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 49 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 372df142-6f7f-4657-9bcf-37049e372e76

📥 Commits

Reviewing files that changed from the base of the PR and between 8c542bb and fe39bdd.

📒 Files selected for processing (2)
  • cmd/jig/ci_repair_test.go
  • internal/worker/publish_repair_test.go

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Steel-tech
Steel-tech merged commit 07b753c into main Sep 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant