Repository navigation
Run every heartbeat on the clock its caller passed - #2005
Merged
Merged
Conversation
The loop's TimeProvider was optional and defaulted to TimeProvider.System, so both executor call sites kept the wall clock without saying so: the seam existed and production ignored it. A test could therefore only pin the loop by racing real milliseconds, and A_failing_ping_is_reported_but_does_not_ kill_the_loop asserted that at least two 20ms ticks fitted inside a real 200ms window. On a loaded runner one did, reddening runs whose diffs touched nothing near this loop. Making the parameter required turns "which clock" into a question the compiler asks at every call site, and both executor heartbeats now hand it the DI-registered TimeProvider they already hold for the checkpoint cadence — the system provider's Delay IS the Task.Delay this used before, so the cadence is unchanged. With the fake clock deciding time, the test asserts exactly one ping per elapsed interval instead of "at least two", which was the property meant all along. A fake clock proves the loop honours whatever interval it is handed and says nothing about which one production hands it, so the real cadence is pinned as a number where it is derived.
4 of 5 tasks
ppXD
added a commit
that referenced
this pull request
Sep 23, 2026
The reconciler's spool recovery keeps a run's session-checkpoint columns exactly as its abandon does: both land through the same terminal CAS, which never touches them. Three comments named only the abandon; they now name both, as ArtifactRetention already did. Two retry notes said things that are not always true. The honest-redo line ended "(no pushed branch was found to continue from)", which a dependent supervisor unit reads in the same goal as its producer's "Continue from this branch"; it now says the prior attempt pushed no branch of its own. The lost-host preamble said the machine was lost, but the reconciler abandons the same way when only the process died (ProcessConfirmedDead, or a lapsed lease whose orphan it kills), so it now says "the machine or the process", and the published-branch hint no longer says the unpublished work died with the machine. The grading heartbeat slept on the wall clock, so its tests could only race real milliseconds. It now takes its clock from the caller as a required parameter, as HeartbeatLoop.RunAsync has since #2005, and the tests drive a FakeTimeProvider: exactly one heartbeat per elapsed interval. Converting them showed the cancel test could never fail on the escape it is named for, because Should.NotThrowAsync passes a canceled task; it records the outcome directly now. The listener-death flake was not a slow wait but a lost one. Install handed the accept loop to Task.Run, so a close that landed while a pool thread was still registering the loop's first wait was never delivered by the managed HttpListener, and the loop waited forever with the lease still claimed. Against the real broker, opening and immediately breaking a lease left 23 of 1500 leases claimed; calling the loop directly, which registers its first wait before Install returns, left none. The test now wakes on the drop's warning, its last effect, instead of polling HasLease. The raw NUL byte in that file's ThisHost constant is written as an escape, so grep stops skipping the file as binary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HeartbeatLoop.RunAsync'sTimeProviderwas optional and defaulted toTimeProvider.System, so bothAgentRunExecutorcall sites (launch, line 270; re-attach, line 804) silently stayed on the wall clock. It is now required, and both sites pass the DI-registered_clockthey already hold. The system provider'sDelayIS theTask.Delaythis used before, so the production cadence is byte-identical.A_failing_ping_is_reported_but_does_not_kill_the_loopasserted that at least two real 20 ms ticks fitted inside a real 200 msCancelAfterwindow. On a loaded runner one did — red on Reclaim the settled cleanup receipts nothing cites any more #2001's unit run 35476566751 (pings should be >= 2 but was 1) and on a full-suite run the day before, from diffs touching nothing near this loop. It now advances aFakeTimeProviderand asserts exactly one ping per elapsed interval plus one report per ping, which is the property that was meant.AgentRunLiveness.HeartbeatIntervalis pinned as a literal (100 s = a third of the default 5-minute window). A fake clock proves the loop honours whatever interval it is handed; it says nothing about which one production hands it, and nothing else pinned that number.Test plan
HeartbeatLoopTests(4) +AgentRunLivenessTests(13) green, 0.9 s (was dominated by a 200 ms wall-clock wait)DefaultWindow5 min → 6 min redsHeartbeatInterval_defaults_to_one_hundred_seconds(should be 00:01:40 but was 00:02:00)= null!redsThe_clock_cannot_be_omitted_by_a_call_sitethrow;afteronPingError(ex)reds it (ping 2 never arrived)dotnet build CodeSpace.sln: 0 errors (warnings pre-existing)Follow-up, not in this PR
SupervisorGradingHeartbeatTestsis the one remaining wall-clock heartbeat test in the unit project (SupervisorTurnService.RunGradingHeartbeatLoopAsyncuses a bareTask.Delay, 15 ms interval,ShouldBeGreaterThanOrEqualTo(3)). It is already hardened against the fixed-window shape — it waits for three observed ticks under a 10 s deadline rather than counting ticks in a fixed window — so it is not flaky the same way. Converting it means threading aTimeProviderthroughSupervisorTurnService's 17-parameter constructor at ~11 construction sites across unit, integration and E2E, which does not belong in this diff.AgentProgressLeaseTestswas checked and is already fullyFakeTimeProvider-driven. No other unit test in the heartbeat/lease family counts wall-clock ticks.