fix(sdk): keep an attached chat session alive across a socket drop, and resume it in order - #4033
Conversation
|
Self-review: REQUEST_CHANGES on my own PR. A red-team pass showed the barrier's failure path silently loses events, and the justification I wrote into the code is wrong. Verified in source. Blocker: a failed replay discards held frames, then the cursor is dragged over the gapOn a replay failure the barrier returns, revokes itself, and drops "the next reconnect re-issues it" — there is no next reconnect. "the cursor stands" — it stands for exactly one frame. The next live frame has So the barrier converts a recoverable ordering problem into permanent silent loss — the same class of defect as the reordering it was written to fix, which is worse because nothing reports it. Related: the hold buffer drops the OLDEST frame
held.push({ seq, frame });
if (held.length > REPLAY_BARRIER_LIMIT) held.shift(); // REPLAY_BARRIER_LIMIT = 1_024Bounding the buffer is right; What the fix has to be
Not merging until those are in, with regressions covering: replay rejects on a live socket → no cursor advance and no lost events; overflow → contiguity preserved or a loud reattach. The earlier reconnect-budget and resume commits in this PR stand; this is specifically the barrier's failure path. |
|
Blocker fixed in |
… budget The chat daemon holds its attached-session clients until an explicit detach or endpoint roll, under the same host liveness reaper that Yeachan-Heo#4012 fixed for ACP, yet attach() still dialed with the transport's one-shot defaults -- 3 attempts at 25ms base, 175ms total. The host drops any session not ponged within HEARTBEAT_TTL_MS (20s), so every stall long enough to be reaped left the attachment permanently lost. ACP_SESSION_RECONNECT is hoisted out of the ACP adapter into src/sdk/session-reconnect.ts so the bus layer can reach it without importing the ACP layer, and attach() now connects on that shared budget. Lore-id: 5b3e1c74 Constraint: exactly one definition of the budget -- ACP call sites keep importing it Constraint: the bus layer must not import sdk/acp -- the constant moves, not the dependency Rejected: put it in bus/daemon-paths.ts | that module is deliberately paths-only and would gain an SdkClientOptions dependency Rejected: raise the bridge-client transport defaults | correct for the one-shot request clients that already rely on them Confidence: high Scope-risk: narrow Reversibility: easy Directive: leave the per-request broker client on the one-shot defaults -- it closes in its own finally Tested: the real ChatDaemonRuntime attach path driven to reconnect exhaustion on a fake transport and clock, asserting the concrete 250/500/1000/2000 schedule and a cumulative budget of 41750ms Not-tested: recovery against a host that comes back mid-schedule
`attach()` is a protected chat-daemon lifecycle declaration, so changing it to dial on the long-lived session reconnect budget requires a strictly higher generation: an owner still running the old code gives up reconnecting after 175ms and permanently loses its attachment, and the generation is what stops it from serving requests captured against the new contract. Lore-id: 6f21b8ac Confidence: high Scope-risk: narrow Reversibility: easy Tested: telegram-daemon-generation-guard against origin/dev
The exhaustion test dialed a dead endpoint with the production budget, which Yeachan-Heo#4012 deliberately grew to outlive the host heartbeat TTL (41.75s of real backoff). It has been timing out at 5s on origin/dev ever since, so check:sdk-closure fails on a clean checkout. Injecting a one-shot client keeps the assertion about the typed rejection and drops the run from 41.9s to 96ms. Lore-id: 7a2c8e04 Constraint: the budget itself stays asserted from its constants in acp-session-reconnect.test.ts Rejected: raise the test timeout past 42s | pays 42s of wall clock on every run to assert an error code Rejected: shrink the production budget | it is sized against HEARTBEAT_TTL_MS on purpose Confidence: high Scope-risk: narrow Reversibility: easy Tested: the file now runs 2 pass / 0 fail in 96ms
Giving the attached-session client the long-lived reconnect budget covered the initial dial only. `SdkClient` never opens a replacement socket on its own — it retires the closed incarnation and re-dials on the next `connect`/`request` — and a chat attachment is purely passive, so a transient drop silently ended delivery for good and any later command-triggered reconnect resumed from nowhere. Chat notifications just stopped, which is the opposite of the session-survival premise the budget change was made for. The attachment now keeps its socket dialed through `onReconnect` and replays from its own cursor on the way back, fenced exactly as `attach()` already fences: this session, this endpoint generation. A superseded attachment is disposed rather than resurrected. Lore-id: 3ba7c209 Constraint: replay is fenced by session id and endpoint generation -- a stale incarnation must never replay onto a newer one Constraint: the cursor advances on every delivered frame, including filtered ones, or a reconnect re-delivers them interleaved Constraint: revival lives outside #pending -- the reconnect budget outlives the heartbeat TTL and stop() must not wait for it Rejected: auto-reopen inside SdkClient | changes reconnect semantics for every consumer, including one-shot request clients Rejected: reattach from scratch on every drop | loses the cursor and re-delivers the whole backlog Confidence: high Scope-risk: narrow Reversibility: easy Tested: an established attachment loses its open socket, regains it, and resumes from the last acknowledged event Tested: a superseded endpoint generation disposes the old attachment instead of resuming it Tested: the connect-dial budget assertion is intact and unweakened Not-tested: a live chat daemon across a real network partition
…once Resuming an attachment started the replay asynchronously while the live `onFrame` subscription stayed open, so two producers wrote into the same queue with no barrier. A frame arriving in the hello-to-replay window was published ahead of the events it followed, and the cursor guard only advanced on `seq > cursor` without ever rejecting `seq <= cursor`, so the replay published it a second time. The reviewer reproduced [1, 3, 2, 3] where [1, 2, 3] was required. Ingress is now fenced for the duration of the resume: live frames arriving while a replay is pending are held and drained in sequence order behind the replayed events, and any frame at or below the cursor is dropped rather than republished. Lore-id: 6d81f4b2 Constraint: ordering is by sequence, never by arrival Constraint: an event at or below the cursor for its generation is never published twice Constraint: the fence must not deadlock -- an unanswered replay leaves the cursor intact so the next reconnect re-issues it Constraint: disposal and endpoint supersession discard pending work instead of draining it onto a newer attachment Rejected: dedupe alone without a barrier | drops the duplicate but still publishes the live frame out of order Rejected: pausing the frame observer | loses frames outright rather than ordering them Confidence: high Scope-risk: narrow Reversibility: easy Tested: live seq3 before the replay answers yields [seq1, seq2, seq3], asserted as the full published array Tested: a replayed frame at or below the cursor is dropped instead of published twice Tested: stopping while a replay is pending neither hangs nor publishes what it held Tested: supersession while a replay is pending discards it instead of replaying onto the new attachment Not-tested: a live chat daemon across a real network partition
…nnect path Giving the attached-session client the long-lived reconnect budget and fencing its resume replay both changed `chat-daemon-runtime.ts:attach`, moving its declaration digest for the discord and slack surfaces. Regenerated with --write-manifest, not hand-edited. Lore-id: 9a3f21c7 Confidence: high Scope-risk: narrow Reversibility: easy Tested: telegram-daemon-generation-guard against origin/dev
A failed replay dropped its held frames on the theory that the next reconnect would re-issue them and the cursor stood. Both halves were false on a live socket: replay is driven only by a new hello, which a live socket never produces, and the cursor stood for exactly one frame before the next live seq dragged it over the discarded gap. The barrier converted a recoverable ordering problem into permanent silent loss. Overflow likewise evicted the OLDEST held frame — the one needed next for contiguity. A failed replay now retries on the live socket with bounded backoff while the hold stays intact, and if the retries exhaust, the cursor is reset to the last contiguously-delivered seq so a later replay genuinely re-issues the gap. Overflow no longer evicts the next-needed frame: it fails the barrier loudly and reattaches through the existing endpoint-generation fence. Lore-id: 3a8c7e15 Constraint: ordered exactly-once delivery on the happy path is unchanged Constraint: stop/dispose and supersession still discard pending work Rejected: unbounded retry | an unanswerable replay must eventually yield to the reattach fence Rejected: dropping newest on overflow without cursor reset | still silently loses events, just different ones Confidence: high Scope-risk: narrow Reversibility: easy Tested: a rejected replay on a live socket loses no events and delivers in order Tested: the cursor never advances over an un-replayed gap Tested: overflow triggers the loud reattach instead of a silent skip Not-tested: sustained overflow under a real network partition
…path The retry/reset logic changed chat-daemon-runtime.ts declaration digests again. Regenerated with --write-manifest; the generation bump already on this branch covers it. Lore-id: 5d2b8f91 Confidence: high Scope-risk: narrow Reversibility: easy Tested: telegram-daemon-generation-guard against origin/dev
Conceding a gap the host can no longer replay left the cursor below the conceded range, so the next reconnect re-requested events that are provably gone and the attachment could never move past them. A legitimate replay whose retained slice is empty was also treated as failure. The cursor now advances to the end of a conceded gap, which is the honest statement: those events are unrecoverable, and delivery resumes after them rather than stalling forever. Lore-id: 7c2e4b19 Constraint: a gap is conceded only when the host proves the events are gone Constraint: ordered exactly-once delivery after the gap is unchanged Rejected: retrying the conceded range | the host has already discarded it Confidence: high Scope-risk: narrow Reversibility: easy Tested: a conceded gap advances the cursor and delivery resumes after it Tested: an empty retained slice is not treated as a failed replay Tested: the earlier reconnect, barrier and overflow behaviours are unchanged Not-tested: a live host under sustained retention pressure
|
Local codex-pro review gate: APPROVE, no blockers, after Conceding a retention gap left the cursor below the conceded range, so the next reconnect re-requested events the host had provably discarded and the attachment could never move past them; a legitimate replay whose retained slice was empty was also treated as a failure. The cursor now advances to the end of a conceded gap — the honest statement that those events are unrecoverable — and delivery resumes after them. 204 pass / 0 fail across reconnect + control-frames + daemon-control; 9 pass / 8 fail with |
677451a to
0906d45
Compare
CI failed an ordering assertion this suite passes locally, which is the tell for a race rather than a flake. The replay answer rides the same socket as the live frames before it, but ingress is a queue: a frame the socket already delivered can still be in flight when the answer resolves, and the hold buffer does not carry it until it lands. Conceding a range then stepped the cursor over a sequence still on its way in, and that frame was dropped as already delivered past a cursor no producer had published it past. The round now joins the ingress tail before reading its own answer, so what live delivery already carried is readable before the concession is computed. Lore-id: 8b5f2a71 Constraint: ordered exactly-once delivery is unchanged; only the read point moves Constraint: the join is bounded and cannot deadlock -- a rejected tail resolves it Rejected: loosening the assertion to match CI | the pinned order was the property under test Confidence: high Scope-risk: narrow Reversibility: easy Tested: the conceded-gap ordering holds across repeated local runs Tested: barrier, overflow and supersession behaviours are unchanged Not-tested: the same interleaving under a real network partition
Scoped replacement for the chat-daemon reconnect portion of the closed #4021. Six commits, 9 files, one subsystem: an attached chat session must survive losing its socket, and must resume in order, exactly once.
This is the surface the #4021 reviewer red-teamed twice. Both of their blockers are fixed here, with the integration tests they asked for.
The defects
1. The reconnect budget only covered the initial dial (
021399101,59f5ae2df)The SDK host drops any session whose client has not ponged within
HEARTBEAT_TTL_MS(20 s).SdkClientdefaults to 3 attempts at 25 ms base — a 175 ms total budget — so any stall long enough for the host to reap the session was unrecoverable. #4012 fixed this for ACP viaACP_SESSION_RECONNECT; the chat daemon's long-lived attached-session clients still inherited the one-shot default. The constant is hoisted tosrc/sdk/session-reconnect.tsso the bus layer can reach it without importing the ACP layer, andattach()now dials on it. The per-request broker client keeps the one-shot defaults deliberately — it closes in its ownfinally.attach()is a protected chat-daemon lifecycle declaration, soCHAT_DAEMON_GENERATIONSis bumped: an owner still running the old code gives up reconnecting after 175 ms and permanently loses its attachment.2. An established attachment still died silently (
98d194668)Raising the budget fixed the initial dial and nothing else.
ChatDaemonRuntimeinstalledclient.onFrame()only insideattach(), andSdkClientnever opens a replacement socket on its own — it retires the closed incarnation and re-dials on the nextconnect/request. A chat attachment is purely passive, so a transient drop silently ended delivery for good, and any later command-triggered reconnect resumed from nowhere. Chat notifications simply stopped.The attachment now keeps its socket dialed through
client.onReconnect— the same mechanismsdk/acp/adapter.tsalready relies on — and replays from its own cursor, fenced exactly asattach()already fences: this session id, this endpoint generation. A superseded attachment is disposed, never resurrected.3. The resume raced live frames, reordering and duplicating them (
a7ee32a5f)Resuming started the replay asynchronously while the live
onFramesubscription stayed open, so two producers wrote into the same queue with no barrier. Worse, the cursor guard only advanced onseq > cursorand never rejectedseq <= cursor, so the replay republished a frame the live socket had already delivered.Reproduction: deliver seq 1 → drop the socket → record seq 2 during the outage → accept the replacement → deliver live seq 3 before the replay answers. Required
[1, 2, 3]; actual[1, 3, 2, 3].Ingress is now fenced for the duration of the resume — live frames arriving in the hello→replay window are held and drained in sequence order behind the replayed events — and any frame at or below the cursor is dropped instead of republished. The fence cannot deadlock: an unanswered replay leaves the cursor intact so the next reconnect re-issues it, which is the pre-existing failure path.
4. A unit test burned the real 41.75 s budget (
6cccc9441)sdk-acp-provider-reconnectdialed a dead endpoint with the production budget and timed out at 5 s — failing on cleanorigin/devever since #4012 grew it. Injecting a one-shot client keeps the assertion about the typed rejection and drops the run from 41.9 s to 96 ms.Verification
Every regression test is load-bearing, proven by restoring only
chat-daemon-runtime.tsfromorigin/devand re-running:The four cases the reviewer required, each failing without the fix:
[seq1, seq3, seq2, seq3][seq1, seq2, seq3]The full published array is asserted, not a set or a length.
packages/bridge-clientis untouched: changing reconnect semantics there would hit every consumer, including one-shot request clients.Relationship to #4021
#4021 bundled twelve unrelated defects into 53 files and was closed with the instruction to open fresh, scoped PRs. This is the second of those, after #4031 (ACP turn lifecycle). Remaining groups — broker/host lifecycle, gc/disk retention, tool diagnostics — follow as separate PRs.