Skip to content

fix(sdk): keep an attached chat session alive across a socket drop, and resume it in order - #4033

Merged
probepark merged 10 commits into
Yeachan-Heo:devfrom
probepark:pr/chat-daemon-reconnect
Aug 9, 2026
Merged

fix(sdk): keep an attached chat session alive across a socket drop, and resume it in order#4033
probepark merged 10 commits into
Yeachan-Heo:devfrom
probepark:pr/chat-daemon-reconnect

Conversation

@probepark

Copy link
Copy Markdown
Collaborator

Scoped replacement for the chat-daemon reconnect portion of the closed #4021. Six commits, 9 files, one subsystem: an attached chat session must survive losing its socket, and must resume in order, exactly once.

This is the surface the #4021 reviewer red-teamed twice. Both of their blockers are fixed here, with the integration tests they asked for.

The defects

1. The reconnect budget only covered the initial dial (021399101, 59f5ae2df)
The SDK host drops any session whose client has not ponged within HEARTBEAT_TTL_MS (20 s). SdkClient defaults to 3 attempts at 25 ms base — a 175 ms total budget — so any stall long enough for the host to reap the session was unrecoverable. #4012 fixed this for ACP via ACP_SESSION_RECONNECT; the chat daemon's long-lived attached-session clients still inherited the one-shot default. The constant is hoisted to src/sdk/session-reconnect.ts so the bus layer can reach it without importing the ACP layer, and attach() now dials on it. The per-request broker client keeps the one-shot defaults deliberately — it closes in its own finally.

attach() is a protected chat-daemon lifecycle declaration, so CHAT_DAEMON_GENERATIONS is bumped: an owner still running the old code gives up reconnecting after 175 ms and permanently loses its attachment.

2. An established attachment still died silently (98d194668)
Raising the budget fixed the initial dial and nothing else. ChatDaemonRuntime installed client.onFrame() only inside attach(), and SdkClient never opens a replacement socket on its own — it retires the closed incarnation and re-dials on the next connect/request. A chat attachment is purely passive, so a transient drop silently ended delivery for good, and any later command-triggered reconnect resumed from nowhere. Chat notifications simply stopped.

The attachment now keeps its socket dialed through client.onReconnect — the same mechanism sdk/acp/adapter.ts already relies on — and replays from its own cursor, fenced exactly as attach() already fences: this session id, this endpoint generation. A superseded attachment is disposed, never resurrected.

3. The resume raced live frames, reordering and duplicating them (a7ee32a5f)
Resuming started the replay asynchronously while the live onFrame subscription stayed open, so two producers wrote into the same queue with no barrier. Worse, the cursor guard only advanced on seq > cursor and never rejected seq <= cursor, so the replay republished a frame the live socket had already delivered.

Reproduction: deliver seq 1 → drop the socket → record seq 2 during the outage → accept the replacement → deliver live seq 3 before the replay answers. Required [1, 2, 3]; actual [1, 3, 2, 3].

Ingress is now fenced for the duration of the resume — live frames arriving in the hello→replay window are held and drained in sequence order behind the replayed events — and any frame at or below the cursor is dropped instead of republished. The fence cannot deadlock: an unanswered replay leaves the cursor intact so the next reconnect re-issues it, which is the pre-existing failure path.

4. A unit test burned the real 41.75 s budget (6cccc9441)
sdk-acp-provider-reconnect dialed a dead endpoint with the production budget and timed out at 5 s — failing on clean origin/dev ever since #4012 grew it. Injecting a one-shot client keeps the assertion about the typed rejection and drops the run from 41.9 s to 96 ms.

Verification

Every regression test is load-bearing, proven by restoring only chat-daemon-runtime.ts from origin/dev and re-running:

chat-daemon-session-reconnect    7 pass / 0 fail   →  1 pass / 6 fail without the fix

The four cases the reviewer required, each failing without the fix:

case without with
live seq3 before replay answers [seq1, seq3, seq2, seq3] [seq1, seq2, seq3]
replayed frame at/below cursor duplicate published dropped
stop while replay pending hang (20 s timeout) returns, publishes nothing held
supersession while replay pending held frame replayed onto the new attachment discarded

The full published array is asserted, not a set or a length.

+ control-frames + daemon-control + acp reconnect + provider reconnect   199 pass / 0 fail
bun --cwd=packages/coding-agent run check                                exit 0
telegram-daemon-generation-guard                                         v43 required generation bump verified

packages/bridge-client is untouched: changing reconnect semantics there would hit every consumer, including one-shot request clients.

Relationship to #4021

#4021 bundled twelve unrelated defects into 53 files and was closed with the instruction to open fresh, scoped PRs. This is the second of those, after #4031 (ACP turn lifecycle). Remaining groups — broker/host lifecycle, gc/disk retention, tool diagnostics — follow as separate PRs.

@probepark

Copy link
Copy Markdown
Collaborator Author

Self-review: REQUEST_CHANGES on my own PR. A red-team pass showed the barrier's failure path silently loses events, and the justification I wrote into the code is wrong. Verified in source.

Blocker: a failed replay discards held frames, then the cursor is dragged over the gap

On a replay failure the barrier returns, revokes itself, and drops held[]. My comment justified that two ways; both are false on a live socket:

"the next reconnect re-issues it" — there is no next reconnect. #replayAttachment is driven only by client.onReconnect (chat-daemon-runtime.ts:565-566), which fires from SdkClient.#acceptHello when a NEW connectionId arrives. If the socket never dropped, there is no hello and no retry. #reviveTransport calls connect(), which is a no-op on a live socket, so the reconcile tick does not rescue it either.

"the cursor stands" — it stands for exactly one frame. The next live frame has seq > cursor, so :815 (attached.cursor.seq = seq) drags the cursor straight over the discarded gap, and those events are unrecoverable even if a reconnect happened later.

So the barrier converts a recoverable ordering problem into permanent silent loss — the same class of defect as the reordering it was written to fix, which is worse because nothing reports it.

Related: the hold buffer drops the OLDEST frame

:811-812:

held.push({ seq, frame });
if (held.length > REPLAY_BARRIER_LIMIT) held.shift();   // REPLAY_BARRIER_LIMIT = 1_024

Bounding the buffer is right; shift() is not. Under a long outage with heavy traffic it evicts the oldest held frame — the one nearest the cursor, i.e. the one that must be delivered next for the sequence to stay contiguous. Overflow should either fail the barrier loudly and force a full reattach, or drop from the newest end and reset the cursor to the last contiguous seq. Silently discarding the frame the ordering depends on defeats the barrier.

What the fix has to be

  1. A failed replay must not advance the cursor past unreplayed sequences. Either keep the held frames until a replay succeeds, or reset the cursor to the last contiguous delivered seq so a later reconnect genuinely re-issues them.
  2. Give the barrier a retry path that does not depend on a new hello, since the failure mode being handled occurs on a live socket.
  3. Fix the overflow policy so the buffer cannot evict the next-needed frame, and surface the overflow rather than hiding it.

Not merging until those are in, with regressions covering: replay rejects on a live socket → no cursor advance and no lost events; overflow → contiguity preserved or a loud reattach. The earlier reconnect-budget and resume commits in this PR stand; this is specifically the barrier's failure path.

@probepark

Copy link
Copy Markdown
Collaborator Author

Blocker fixed in 6ad3e4b4b: a failed replay now retries on the live socket with bounded backoff while the hold stays intact, and exhaustion resets the cursor to the last contiguously-delivered seq so a later replay genuinely re-issues the gap. Overflow no longer evicts the next-needed frame — it fails the barrier loudly and reattaches through the endpoint-generation fence. Regressions: no event lost on a rejected live-socket replay (full array asserted), cursor never crosses an un-replayed gap, overflow reattach. 197 pass / 0 fail across the reconnect+control suites; 7 pass/3 fail without src; manifest regenerated (v43 verified).

@probepark
probepark requested a review from Yeachan-Heo August 8, 2026 19:09
… budget

The chat daemon holds its attached-session clients until an explicit detach or
endpoint roll, under the same host liveness reaper that Yeachan-Heo#4012 fixed for ACP, yet
attach() still dialed with the transport's one-shot defaults -- 3 attempts at
25ms base, 175ms total. The host drops any session not ponged within
HEARTBEAT_TTL_MS (20s), so every stall long enough to be reaped left the
attachment permanently lost. ACP_SESSION_RECONNECT is hoisted out of the ACP
adapter into src/sdk/session-reconnect.ts so the bus layer can reach it without
importing the ACP layer, and attach() now connects on that shared budget.

Lore-id: 5b3e1c74
Constraint: exactly one definition of the budget -- ACP call sites keep importing it
Constraint: the bus layer must not import sdk/acp -- the constant moves, not the dependency
Rejected: put it in bus/daemon-paths.ts | that module is deliberately paths-only and would gain an SdkClientOptions dependency
Rejected: raise the bridge-client transport defaults | correct for the one-shot request clients that already rely on them
Confidence: high
Scope-risk: narrow
Reversibility: easy
Directive: leave the per-request broker client on the one-shot defaults -- it closes in its own finally
Tested: the real ChatDaemonRuntime attach path driven to reconnect exhaustion on a fake transport and clock, asserting the concrete 250/500/1000/2000 schedule and a cumulative budget of 41750ms
Not-tested: recovery against a host that comes back mid-schedule
`attach()` is a protected chat-daemon lifecycle declaration, so changing it to
dial on the long-lived session reconnect budget requires a strictly higher
generation: an owner still running the old code gives up reconnecting after
175ms and permanently loses its attachment, and the generation is what stops it
from serving requests captured against the new contract.

Lore-id: 6f21b8ac
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: telegram-daemon-generation-guard against origin/dev
The exhaustion test dialed a dead endpoint with the production budget, which
Yeachan-Heo#4012 deliberately grew to outlive the host heartbeat TTL (41.75s of real
backoff). It has been timing out at 5s on origin/dev ever since, so
check:sdk-closure fails on a clean checkout. Injecting a one-shot client keeps
the assertion about the typed rejection and drops the run from 41.9s to 96ms.

Lore-id: 7a2c8e04
Constraint: the budget itself stays asserted from its constants in acp-session-reconnect.test.ts
Rejected: raise the test timeout past 42s | pays 42s of wall clock on every run to assert an error code
Rejected: shrink the production budget | it is sized against HEARTBEAT_TTL_MS on purpose
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: the file now runs 2 pass / 0 fail in 96ms
Giving the attached-session client the long-lived reconnect budget covered the
initial dial only. `SdkClient` never opens a replacement socket on its own — it
retires the closed incarnation and re-dials on the next `connect`/`request` —
and a chat attachment is purely passive, so a transient drop silently ended
delivery for good and any later command-triggered reconnect resumed from
nowhere. Chat notifications just stopped, which is the opposite of the
session-survival premise the budget change was made for.

The attachment now keeps its socket dialed through `onReconnect` and replays
from its own cursor on the way back, fenced exactly as `attach()` already
fences: this session, this endpoint generation. A superseded attachment is
disposed rather than resurrected.

Lore-id: 3ba7c209
Constraint: replay is fenced by session id and endpoint generation -- a stale incarnation must never replay onto a newer one
Constraint: the cursor advances on every delivered frame, including filtered ones, or a reconnect re-delivers them interleaved
Constraint: revival lives outside #pending -- the reconnect budget outlives the heartbeat TTL and stop() must not wait for it
Rejected: auto-reopen inside SdkClient | changes reconnect semantics for every consumer, including one-shot request clients
Rejected: reattach from scratch on every drop | loses the cursor and re-delivers the whole backlog
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: an established attachment loses its open socket, regains it, and resumes from the last acknowledged event
Tested: a superseded endpoint generation disposes the old attachment instead of resuming it
Tested: the connect-dial budget assertion is intact and unweakened
Not-tested: a live chat daemon across a real network partition
…once

Resuming an attachment started the replay asynchronously while the live
`onFrame` subscription stayed open, so two producers wrote into the same queue
with no barrier. A frame arriving in the hello-to-replay window was published
ahead of the events it followed, and the cursor guard only advanced on
`seq > cursor` without ever rejecting `seq <= cursor`, so the replay published
it a second time. The reviewer reproduced [1, 3, 2, 3] where [1, 2, 3] was
required.

Ingress is now fenced for the duration of the resume: live frames arriving
while a replay is pending are held and drained in sequence order behind the
replayed events, and any frame at or below the cursor is dropped rather than
republished.

Lore-id: 6d81f4b2
Constraint: ordering is by sequence, never by arrival
Constraint: an event at or below the cursor for its generation is never published twice
Constraint: the fence must not deadlock -- an unanswered replay leaves the cursor intact so the next reconnect re-issues it
Constraint: disposal and endpoint supersession discard pending work instead of draining it onto a newer attachment
Rejected: dedupe alone without a barrier | drops the duplicate but still publishes the live frame out of order
Rejected: pausing the frame observer | loses frames outright rather than ordering them
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: live seq3 before the replay answers yields [seq1, seq2, seq3], asserted as the full published array
Tested: a replayed frame at or below the cursor is dropped instead of published twice
Tested: stopping while a replay is pending neither hangs nor publishes what it held
Tested: supersession while a replay is pending discards it instead of replaying onto the new attachment
Not-tested: a live chat daemon across a real network partition
…nnect path

Giving the attached-session client the long-lived reconnect budget and fencing
its resume replay both changed `chat-daemon-runtime.ts:attach`, moving its
declaration digest for the discord and slack surfaces. Regenerated with
--write-manifest, not hand-edited.

Lore-id: 9a3f21c7
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: telegram-daemon-generation-guard against origin/dev
A failed replay dropped its held frames on the theory that the next reconnect
would re-issue them and the cursor stood. Both halves were false on a live
socket: replay is driven only by a new hello, which a live socket never
produces, and the cursor stood for exactly one frame before the next live seq
dragged it over the discarded gap. The barrier converted a recoverable
ordering problem into permanent silent loss. Overflow likewise evicted the
OLDEST held frame — the one needed next for contiguity.

A failed replay now retries on the live socket with bounded backoff while the
hold stays intact, and if the retries exhaust, the cursor is reset to the last
contiguously-delivered seq so a later replay genuinely re-issues the gap.
Overflow no longer evicts the next-needed frame: it fails the barrier loudly
and reattaches through the existing endpoint-generation fence.

Lore-id: 3a8c7e15
Constraint: ordered exactly-once delivery on the happy path is unchanged
Constraint: stop/dispose and supersession still discard pending work
Rejected: unbounded retry | an unanswerable replay must eventually yield to the reattach fence
Rejected: dropping newest on overflow without cursor reset | still silently loses events, just different ones
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: a rejected replay on a live socket loses no events and delivers in order
Tested: the cursor never advances over an un-replayed gap
Tested: overflow triggers the loud reattach instead of a silent skip
Not-tested: sustained overflow under a real network partition
…path

The retry/reset logic changed chat-daemon-runtime.ts declaration digests again.
Regenerated with --write-manifest; the generation bump already on this branch
covers it.

Lore-id: 5d2b8f91
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: telegram-daemon-generation-guard against origin/dev
Conceding a gap the host can no longer replay left the cursor below the
conceded range, so the next reconnect re-requested events that are provably
gone and the attachment could never move past them. A legitimate replay whose
retained slice is empty was also treated as failure.

The cursor now advances to the end of a conceded gap, which is the honest
statement: those events are unrecoverable, and delivery resumes after them
rather than stalling forever.

Lore-id: 7c2e4b19
Constraint: a gap is conceded only when the host proves the events are gone
Constraint: ordered exactly-once delivery after the gap is unchanged
Rejected: retrying the conceded range | the host has already discarded it
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: a conceded gap advances the cursor and delivery resumes after it
Tested: an empty retained slice is not treated as a failed replay
Tested: the earlier reconnect, barrier and overflow behaviours are unchanged
Not-tested: a live host under sustained retention pressure
@probepark

Copy link
Copy Markdown
Collaborator Author

Local codex-pro review gate: APPROVE, no blockers, after 677451a29 (rebased onto current dev).

Conceding a retention gap left the cursor below the conceded range, so the next reconnect re-requested events the host had provably discarded and the attachment could never move past them; a legitimate replay whose retained slice was empty was also treated as a failure. The cursor now advances to the end of a conceded gap — the honest statement that those events are unrecoverable — and delivery resumes after them.

204 pass / 0 fail across reconnect + control-frames + daemon-control; 9 pass / 8 fail with packages/coding-agent/src stashed; check exit 0; generation guard v43 required generation bump verified.

@probepark
probepark force-pushed the pr/chat-daemon-reconnect branch from 677451a to 0906d45 Compare August 9, 2026 01:42
CI failed an ordering assertion this suite passes locally, which is the tell for
a race rather than a flake. The replay answer rides the same socket as the live
frames before it, but ingress is a queue: a frame the socket already delivered
can still be in flight when the answer resolves, and the hold buffer does not
carry it until it lands. Conceding a range then stepped the cursor over a
sequence still on its way in, and that frame was dropped as already delivered
past a cursor no producer had published it past.

The round now joins the ingress tail before reading its own answer, so what live
delivery already carried is readable before the concession is computed.

Lore-id: 8b5f2a71
Constraint: ordered exactly-once delivery is unchanged; only the read point moves
Constraint: the join is bounded and cannot deadlock -- a rejected tail resolves it
Rejected: loosening the assertion to match CI | the pinned order was the property under test
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: the conceded-gap ordering holds across repeated local runs
Tested: barrier, overflow and supersession behaviours are unchanged
Not-tested: the same interleaving under a real network partition
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant