fix(notifier): prevent hung-reconnect wedge (arm timeout/abort before connect)#78
Merged
Merged
Conversation
…onnect wedge waitWebSocket awaited notificationClient.connect() before arming the re-poll timeout and abort listener. A reconnect whose connect() never resolves (half-open / black-holed socket) therefore blocked the fetch loop forever: the timer was never armed, abort was never wired, and the finally never ran. The pulse emitter (separate timer) kept reporting "alive", so the pump appeared healthy while silently processing nothing — recoverable only by a full process restart. The existing main-loop self-heal (#74) only catches thrown errors, not a hang. Arm the timeout + abort listener BEFORE connect(), and race connect() against the resolution promise. A hung connect now resolves the wait at timeoutMs; the loop re-polls (catching up backlog) and attempts a fresh connect each cycle, auto-recovering to push/WS mode the moment a connect succeeds. A connect rejection still surfaces, preserving main-loop self-heal. No sticky poll-mode state. Adds notifier test cases 9 (hung connect must not wedge) and 10 (retries while degraded, returns to WS mode on reconnect). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 tasks
jbiskur
added a commit
to flowcore-io/flowcore-pathways
that referenced
this pull request
Jun 10, 2026
…#90) * fix(pump): vendor @flowcore/data-pump 0.21.4 (hung-connect reconnect self-heal) Bumps the vendored data-pump (dnt inlines the jsr import into the published package, pinned by deno.lock) from 0.21.3 to 0.21.4, which adds the hung-connect fix: notifier.waitWebSocket now arms the re-poll timeout + abort BEFORE connect() and races connect() against the resolution promise, so a hung websocket reconnect degrades to a bounded re-poll and auto-recovers to WS mode instead of wedging the pump until a process restart. This is the delivery vehicle that gets the fix into embedded consumers (control-plane, worker, klokkan-backend, usable-chat, ...). See flowcore-io/data-pump#78 (data-pump 0.21.4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(postgres): accept EAI_AGAIN as well as ENOTFOUND for unresolvable host The "missing connection information" test dials a nonexistent host and asserted the error message contains "ENOTFOUND", but some runners' DNS resolvers report an unresolvable host as "EAI_AGAIN" instead (both mean the host did not resolve). Assert on either to de-flake CI. Unrelated to the data-pump bump; pre-existing fragility surfaced on the current runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The control-plane's internal pathways pump (and the data-pathways worker fleet) can silently wedge: the fetch loop stops consuming events while the pulse emitter keeps reporting "alive", recoverable only by a full process restart. Most recently this took the prod control-plane down for ~14h (2026-06-08T19:40Z) — every
PUT /pathways/:id500'd becausewaitForPathwayToBeProcessedtimed out on the self-loop write.Root cause
FlowcoreNotifier.waitWebSocket()awaitednotificationClient.connect()before arming the re-poll timeout and abort listener:When a reconnect hangs (half-open / black-holed socket),
connect()never resolves: the 20s re-poll timer is never armed, abort is never wired, and thefinallynever runs → the main loop blocks forever. ThestartMainLoopself-heal (#74) only catches thrown errors, so a hang is invisible to it.Fix
Arm the timeout + abort listener before
connect(), and raceconnect()against the resolution promise so a hung connect can't block. A hung connect now resolves the wait attimeoutMs; the loop re-polls (catching up any backlog) and attempts a fresh connect each cycle, auto-recovering to push/WS mode the moment a connect succeeds. A connect rejection still surfaces, preserving the main-loop self-heal. No sticky poll-mode state.Tests
test/tests/notifier.test.ts:connect()must not wedge:wait()resolves via the pre-armed timeout (fails on pre-fix code).Full suite: 12 passed / 57 steps.
deno fmt/lint/checkclean.🤖 Generated with Claude Code