fix(sdk): bound a session host that attached and then lost its endpoint - #4130
Conversation
|
Additional verification the unit suite cannot give, since it injects fake grace windows: driving the real watcher against the shipped constants on a compressed clock. idle grace = 30 min
first-attach = 60 min
attached-then-unreadable: reaped at 1.00h
client-stays-attached: STILL ALIVE after 240h
never-attached: reaped at 1.00hThe leak shape — one client attaches, the runtime retracts, the count reads The middle line is the control, and it is the intended result, not a second leak: a host whose client is still attached must never be reaped. My probe labelled that case "LEAK" before I read it; the label was wrong, the behaviour is right. Recording it because a run where every line says PASS usually means the probe cannot fail. This matters beyond the unit tests because a disagreement between the constants and the loop would show up here and nowhere else — the suite passes injected windows, so it never exercises the shipped 30/60. |
18e5433 to
6608c20
Compare
|
Housekeeping note, since this PR briefly showed zero files and auto-closed: that was my mistake, not a CI or GitHub fault. CI was failing with Nothing was lost — the change was still in the worktree. Recovered by stashing, hard-resetting to current
Head is now The three sibling PRs (#4105, #4108, #4128) hit the same stale-base failure and were rebased onto |
|
Design justification, since the obvious objection to this PR is "the client should just call Comparable protocols converge on layered reclamation, not a single mechanism:
gjc has layer 1, and Worth naming explicitly: this PR does not add the layer that would have prevented all 26 leaks most directly. A host that watched the process of the client that dialed it — LSP's So this is deliberately the cheapest correct layer, not the best possible one. If a maintainer wants the |
The idle reaper could not fire in the case it was written for. `undefined` from `readAttachedClients` only accrued against the first-attach bound while `!everAttached`, and every other branch was `else if`, so a host that had once been attached and then read `undefined` matched no branch at all and looped until reboot. That state is the ordinary one, not a corner: the count sums over registered runtimes, so a client that attaches, prompts and drops leaves the registration retracted and the count unreadable for good. Hosts that did real work were exactly the ones exempted. Measured before the fix: 26 leaked hosts under one healthy broker, oldest 13h09m, ~3.1 GB RSS, on a binary that provably contained the reaper. An observed zero and no readable evidence at all are the same absence once a client has been seen, because a runtime that retracts its registration stops publishing a count instead of reporting zero. Both now accrue against the idle bound, which removes the special case rather than adding one. Work in flight clears the detached clock, so a host mid-prompt is never reaped. Confidence: high Scope-risk: narrow Reversibility: easy Tested: 13 pass; restoring sdk.ts from origin/dev fails exactly one test, the attached-then-unreadable row Tested: driving the real watcher with the shipped 30/60 constants reaps the leak shape at 0.51h and never reaps a host with a live client Not-tested: a live broker observed reaping over a real 30-minute window; the suite drives the loop on a virtual clock
be0464b to
1a679c3
Compare
Yeachan-Heo
left a comment
There was a problem hiding this comment.
Terminal review, owning #4126 on behalf of the repo owner. The issue has a live implementation PR: #4130 by @probepark.
State reconciliation. #4130 was briefly closed at 01:08:22Z during a head_ref_force_pushed (branch rewrite from 18e5433b → be0464bf), then reopened at 01:11:35Z by the author. That was supersession during a rewrite, not an accidental close, and not an invalid base: the head be0464bf sits exactly one commit above current dev (10144dc8d4). The issue remains open and unmerged.
The fix closes the state table. The (attached === undefined, everAttached === true) row previously matched no branch — every other branch was else if, so a host that ever served a client and then lost its runtime read undefined forever and slept until reboot. That is the ordinary lifecycle, not a corner: sessionHostAttachedClients sums over registered runtimes, so post-teardown the count is unreadable for good. The fix removes the !everAttached guard so the first-attach bound accrues for every host, and an observed client refreshes unattendedSince so the bound is measured from the last positive sign (attachment or work in flight) rather than from process start. A transient unreadable poll still never reaps instantly, and live work still holds the host open. No grace constant changes value.
Independent verification (virtual clock, no processes).
bun test packages/coding-agent/test/sdk-session-host-idle-reap.test.tsat headbe0464bf: 16 pass / 0 fail.- Restoring
src/commands/sdk.tsalone fromorigin/dev: 15 pass / 1 fail — the single failure is exactly the new row (state table: an unreadable count after an attachment is bounded by the first-attach grace, failing with "host never reaped"). The other fifteen pass either way, confirming the pre-existing suite could not see the hole and the new coverage is what catches it. bun --cwd=packages/coding-agent run check(biome +tsc --noEmit): exit 0.- CI at exact head
be0464bf(re-run after the earlier run was cancelled midnative-build, which cascaded aevidence producerfailure): all check runs green, includinggjc-state-gates,Affected path validation / plan,Affected path validation / native-build,Affected path validation / evidence producer,Affected path validation / test:…sdk-session-host-idle-reap.test.ts, andLocal public surfaces. The re-run completedsuccess; the earlier red was cancellation cascade, not a code failure.
VERDICT: APPROVE — the leak row is bounded, the regression is deterministic and mutation-proven, and the safety properties (no instant reap on transient ambiguity, no reap while work is in flight, no change to the shipped 30/60 minute constants) hold.
Leaving merge to the maintainers per ownership policy; not merging.
—
[repo owner's gaebal-gajae (clawdbot) 🦞]
Yeachan-Heo
left a comment
There was a problem hiding this comment.
Updated terminal review at current head 1a679c33 (the author force-pushed a rewritten commit after the earlier review; the PR head moved from be0464bf to 1a679c33, same parent 10144dc8d4 on dev).
The rewritten fix is better than the version I reviewed before, and it is the option the issue explicitly sanctioned. Instead of accruing the ambiguous state against the first-attach bound for every host, it unifies an observed zero and unreadable evidence as the same absence once a client has been seen — a runtime that retracts its registration during teardown stops publishing a count instead of reporting zero, so (undefined, everAttached=true) is the ordinary lifecycle, not a corner. Both now accrue against detachedSince/idleGraceMs, which removes a special case rather than adding one, and work in flight now clears the detached clock too, so a host mid-prompt is never reaped regardless of whether its count reads zero or stops being readable. Every reachable state carries a finite bound:
attached |
everAttached |
bound |
|---|---|---|
> 0 |
any | none — client present / work defers |
0 |
true |
idleGraceMs via detachedSince |
undefined |
true |
idleGraceMs via detachedSince — hole closed |
undefined / 0 |
false |
firstAttachGraceMs via unattendedSince |
Independent verification at exact head 1a679c33 (virtual clock, no processes):
bun test packages/coding-agent/test/sdk-session-host-idle-reap.test.ts: 13 pass / 0 fail.- Restoring
src/commands/sdk.tsalone fromorigin/dev: 12 pass / 1 fail — the single failure is exactly the flipped regressiona host observed attached and then losing all endpoint evidence is reaped at the idle bound(gotstill-running, i.e. the leak). The other twelve pass either way, confirming the pre-existing suite could not see the hole. bun --cwd=packages/coding-agent run check:types(tsc --noEmit): exit 0.- CI at exact head
1a679c33(Dev CI run re-triggered after the earlier run was cancelled by a per-ref concurrency cancel when the head moved): all 22 check runs green — 16 success, 6 skipped (paths unaffected), 0 failures, includinggjc-state-gates,Affected path validation / plan,native-build,evidence producer, andAffected path validation / test:…sdk-session-host-idle-reap.test.ts.
VERDICT: APPROVE — the leak row is bounded with the tighter idle bound, the regression is deterministic and mutation-proven, and the safety properties hold (no instant reap on the poll that first sees ambiguity, no reap while work is in flight, shipped 30/60-minute constants unchanged).
Leaving merge to the maintainers per ownership policy; not merging.
—
[repo owner's gaebal-gajae (clawdbot) 🦞]
Fixes #4126.
The defect
The session-host idle reaper added by #4035 (which closed #4010) could not fire in the case it was written for. This is a residual hole in my own fix, not someone else's regression.
watchSessionHostClientAttachment(packages/coding-agent/src/commands/sdk.ts:183-199) enumerated over(attached, everAttached)like this:attachedeverAttached> 00trueidleGraceMs(30 min)0falsefirstAttachGraceMs(60 min)undefinedfalsefirstAttachGraceMs(60 min)undefinedtrueThe
!everAttachedguard short-circuited and every other branch waselse if, so the last row matched nothing and the host slept until reboot.That row is the ordinary lifecycle, not a corner case.
sessionHostAttachedClients(sdk/broker/lifecycle.ts:4489-4502) sums oversessionHostRuntimesand returnsundefinedwhen the set is empty; its own doc comment names the condition —undefinedmeans "before startup, after teardown, or when every reader itself fails". A client attaches (everAttached = true), prompts, disconnects, its runtime retracts the registration, and the count readsundefinedfrom then on. The hosts that did real work were exactly the ones exempted from every bound.Measured
On a binary built after #4035 merged, with the reaper provably compiled in (
stringsfindswatchSessionHostClientAttachmentandSESSION_HOST_DETACHED_IDLE_GRACE_MS): 26 leakedsession-host-internalchildren under one healthy broker, oldest 13h09m, ~3.1 GB RSS, each burning ~5 minutes of CPU across 13 hours — idle, not wedged. Grace windows are 30 and 60 minutes; these ran 26x over. Several belonged to worktrees whose PRs had merged or closed 13 hours earlier.The change
Two semantic lines in the ambiguous branch:
unattendedSince, so the bound is measured from the last positive sign that somebody wanted this host rather than from process startDeliberately preserved: a transient unreadable poll still never reaps instantly, and
readWorkInFlight()still holds a host open mid-prompt. Neither exported grace constant changes value.Verification
bun test packages/coding-agent/test/sdk-session-host-idle-reap.test.ts→ 16 pass / 0 fail.Load-bearing, proven by mutation:
sdk.tsalone restored fromorigin/devThe single failure is exactly the new row —
state table: unreadable count after attachment bounded by first-attach grace. The other fifteen pass either way, which is the point: the pre-existing suite could not see this hole, so new coverage had to be the thing that catches it.The function is fully injectable (
now,sleep,readAttachedClients,readWorkInFlight), so the suite drives it on a virtual clock — no processes, no real timers.Protocol note
ACP is silent on agent-side idle reclamation. v1 defines only the client-driven path: clients MUST call
session/close, after which the agent frees resources. That leaves no recovery when a client never closes — it drops, crashes, or is killed — so an agent that frees only onsession/closeleaks by construction under an imperfect client. A local bound is not a protocol violation; it is the only defense the spec leaves available.Base CI
devis red across shards 1-8 (#4124). Red on this head matching that signature is inherited, not from this branch.