Skip to content

Run every heartbeat on the clock its caller passed - #2005

Merged
ppXD merged 1 commit into
mainfrom
fix/drive-heartbeat-loop-on-injected-timeprovider
Sep 21, 2026
Merged

ppXD merged 1 commit into
mainfrom
fix/drive-heartbeat-loop-on-injected-timeprovider

Conversation

@ppXD

@ppXD ppXD commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • HeartbeatLoop.RunAsync's TimeProvider was optional and defaulted to TimeProvider.System, so both AgentRunExecutor call sites (launch, line 270; re-attach, line 804) silently stayed on the wall clock. It is now required, and both sites pass the DI-registered _clock they already hold. The system provider's Delay IS the Task.Delay this used before, so the production cadence is byte-identical.
  • A_failing_ping_is_reported_but_does_not_kill_the_loop asserted that at least two real 20 ms ticks fitted inside a real 200 ms CancelAfter window. On a loaded runner one did — red on Reclaim the settled cleanup receipts nothing cites any more #2001's unit run 35476566751 (pings should be >= 2 but was 1) and on a full-suite run the day before, from diffs touching nothing near this loop. It now advances a FakeTimeProvider and asserts exactly one ping per elapsed interval plus one report per ping, which is the property that was meant.
  • AgentRunLiveness.HeartbeatInterval is pinned as a literal (100 s = a third of the default 5-minute window). A fake clock proves the loop honours whatever interval it is handed; it says nothing about which one production hands it, and nothing else pinned that number.

Test plan

  • Unit: HeartbeatLoopTests (4) + AgentRunLivenessTests (13) green, 0.9 s (was dominated by a 200 ms wall-clock wait)
  • Determinism: converted class run 20x clean and 20x under 8 CPU hogs — 40/40 green
  • Regression: full unit suite 10907/10907 green
  • Mutation — literal cadence pin: DefaultWindow 5 min → 6 min reds HeartbeatInterval_defaults_to_one_hundred_seconds (should be 00:01:40 but was 00:02:00)
  • Mutation — required-clock pin: re-adding = null! reds The_clock_cannot_be_omitted_by_a_call_site
  • Mutation — the converted test still catches its own regression: adding throw; after onPingError(ex) reds it (ping 2 never arrived)
  • dotnet build CodeSpace.sln: 0 errors (warnings pre-existing)

Follow-up, not in this PR

SupervisorGradingHeartbeatTests is the one remaining wall-clock heartbeat test in the unit project (SupervisorTurnService.RunGradingHeartbeatLoopAsync uses a bare Task.Delay, 15 ms interval, ShouldBeGreaterThanOrEqualTo(3)). It is already hardened against the fixed-window shape — it waits for three observed ticks under a 10 s deadline rather than counting ticks in a fixed window — so it is not flaky the same way. Converting it means threading a TimeProvider through SupervisorTurnService's 17-parameter constructor at ~11 construction sites across unit, integration and E2E, which does not belong in this diff.

AgentProgressLeaseTests was checked and is already fully FakeTimeProvider-driven. No other unit test in the heartbeat/lease family counts wall-clock ticks.

The loop's TimeProvider was optional and defaulted to TimeProvider.System,
so both executor call sites kept the wall clock without saying so: the seam
existed and production ignored it. A test could therefore only pin the loop
by racing real milliseconds, and A_failing_ping_is_reported_but_does_not_
kill_the_loop asserted that at least two 20ms ticks fitted inside a real
200ms window. On a loaded runner one did, reddening runs whose diffs touched
nothing near this loop.

Making the parameter required turns "which clock" into a question the
compiler asks at every call site, and both executor heartbeats now hand it
the DI-registered TimeProvider they already hold for the checkpoint cadence
— the system provider's Delay IS the Task.Delay this used before, so the
cadence is unchanged. With the fake clock deciding time, the test asserts
exactly one ping per elapsed interval instead of "at least two", which was
the property meant all along.

A fake clock proves the loop honours whatever interval it is handed and says
nothing about which one production hands it, so the real cadence is pinned
as a number where it is derived.
@ppXD
ppXD merged commit 6ce3768 into main Sep 21, 2026
6 of 7 checks passed
ppXD added a commit that referenced this pull request Sep 23, 2026
The reconciler's spool recovery keeps a run's session-checkpoint columns
exactly as its abandon does: both land through the same terminal CAS,
which never touches them. Three comments named only the abandon; they
now name both, as ArtifactRetention already did.

Two retry notes said things that are not always true. The honest-redo
line ended "(no pushed branch was found to continue from)", which a
dependent supervisor unit reads in the same goal as its producer's
"Continue from this branch"; it now says the prior attempt pushed no
branch of its own. The lost-host preamble said the machine was lost,
but the reconciler abandons the same way when only the process died
(ProcessConfirmedDead, or a lapsed lease whose orphan it kills), so it
now says "the machine or the process", and the published-branch hint
no longer says the unpublished work died with the machine.

The grading heartbeat slept on the wall clock, so its tests could only
race real milliseconds. It now takes its clock from the caller as a
required parameter, as HeartbeatLoop.RunAsync has since #2005, and the
tests drive a FakeTimeProvider: exactly one heartbeat per elapsed
interval. Converting them showed the cancel test could never fail on
the escape it is named for, because Should.NotThrowAsync passes a
canceled task; it records the outcome directly now.

The listener-death flake was not a slow wait but a lost one. Install
handed the accept loop to Task.Run, so a close that landed while a pool
thread was still registering the loop's first wait was never delivered
by the managed HttpListener, and the loop waited forever with the lease
still claimed. Against the real broker, opening and immediately
breaking a lease left 23 of 1500 leases claimed; calling the loop
directly, which registers its first wait before Install returns, left
none. The test now wakes on the drop's warning, its last effect,
instead of polling HasLease. The raw NUL byte in that file's ThisHost
constant is written as an escape, so grep stops skipping the file as
binary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant