Skip to content

fix(notifier): prevent hung-reconnect wedge (arm timeout/abort before connect)#78

Merged
jbiskur merged 1 commit into
mainfrom
fix/notifier-hung-connect-wedge
Jun 9, 2026
Merged

fix(notifier): prevent hung-reconnect wedge (arm timeout/abort before connect)#78
jbiskur merged 1 commit into
mainfrom
fix/notifier-hung-connect-wedge

Conversation

@jbiskur

@jbiskur jbiskur commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Problem

The control-plane's internal pathways pump (and the data-pathways worker fleet) can silently wedge: the fetch loop stops consuming events while the pulse emitter keeps reporting "alive", recoverable only by a full process restart. Most recently this took the prod control-plane down for ~14h (2026-06-08T19:40Z) — every PUT /pathways/:id 500'd because waitForPathwayToBeProcessed timed out on the self-loop write.

Root cause

FlowcoreNotifier.waitWebSocket() awaited notificationClient.connect() before arming the re-poll timeout and abort listener:

await this.notificationClient.connect()              // unbounded — no timeout/abort
this.timer = setTimeout(() => this.eventResolver?.(), 20000)  // armed only AFTER connect resolves
signal?.addEventListener("abort", () => this.eventResolver?.())
await promise

When a reconnect hangs (half-open / black-holed socket), connect() never resolves: the 20s re-poll timer is never armed, abort is never wired, and the finally never runs → the main loop blocks forever. The startMainLoop self-heal (#74) only catches thrown errors, so a hang is invisible to it.

Fix

Arm the timeout + abort listener before connect(), and race connect() against the resolution promise so a hung connect can't block. A hung connect now resolves the wait at timeoutMs; the loop re-polls (catching up any backlog) and attempts a fresh connect each cycle, auto-recovering to push/WS mode the moment a connect succeeds. A connect rejection still surfaces, preserving the main-loop self-heal. No sticky poll-mode state.

Tests

test/tests/notifier.test.ts:

  • case 9 — hung connect() must not wedge: wait() resolves via the pre-armed timeout (fails on pre-fix code).
  • case 10 — retries while degraded, then returns to WS mode on reconnect.
  • Cases 1–8 unchanged and green.

Full suite: 12 passed / 57 steps. deno fmt/lint/check clean.

🤖 Generated with Claude Code

…onnect wedge

waitWebSocket awaited notificationClient.connect() before arming the
re-poll timeout and abort listener. A reconnect whose connect() never
resolves (half-open / black-holed socket) therefore blocked the fetch
loop forever: the timer was never armed, abort was never wired, and the
finally never ran. The pulse emitter (separate timer) kept reporting
"alive", so the pump appeared healthy while silently processing nothing
— recoverable only by a full process restart. The existing main-loop
self-heal (#74) only catches thrown errors, not a hang.

Arm the timeout + abort listener BEFORE connect(), and race connect()
against the resolution promise. A hung connect now resolves the wait at
timeoutMs; the loop re-polls (catching up backlog) and attempts a fresh
connect each cycle, auto-recovering to push/WS mode the moment a connect
succeeds. A connect rejection still surfaces, preserving main-loop
self-heal. No sticky poll-mode state.

Adds notifier test cases 9 (hung connect must not wedge) and 10 (retries
while degraded, returns to WS mode on reconnect).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jbiskur jbiskur merged commit 0fe4bbb into main Jun 9, 2026
2 checks passed
@jbiskur jbiskur deleted the fix/notifier-hung-connect-wedge branch June 9, 2026 15:22
jbiskur added a commit to flowcore-io/flowcore-pathways that referenced this pull request Jun 10, 2026
…#90)

* fix(pump): vendor @flowcore/data-pump 0.21.4 (hung-connect reconnect self-heal)

Bumps the vendored data-pump (dnt inlines the jsr import into the
published package, pinned by deno.lock) from 0.21.3 to 0.21.4, which adds
the hung-connect fix: notifier.waitWebSocket now arms the re-poll timeout
+ abort BEFORE connect() and races connect() against the resolution
promise, so a hung websocket reconnect degrades to a bounded re-poll and
auto-recovers to WS mode instead of wedging the pump until a process
restart. This is the delivery vehicle that gets the fix into embedded
consumers (control-plane, worker, klokkan-backend, usable-chat, ...).

See flowcore-io/data-pump#78 (data-pump 0.21.4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(postgres): accept EAI_AGAIN as well as ENOTFOUND for unresolvable host

The "missing connection information" test dials a nonexistent host and
asserted the error message contains "ENOTFOUND", but some runners' DNS
resolvers report an unresolvable host as "EAI_AGAIN" instead (both mean
the host did not resolve). Assert on either to de-flake CI. Unrelated to
the data-pump bump; pre-existing fragility surfaced on the current runner.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant