Skip to content

fix(dwn): port the dwn-clients WebSocket liveness discipline - #274

Open
LiranCohen wants to merge 3 commits into
fix/feed-driven-refreshfrom
fix/ws-liveness
Open

fix(dwn): port the dwn-clients WebSocket liveness discipline#274
LiranCohen wants to merge 3 commits into
fix/feed-driven-refreshfrom
fix/ws-liveness

Conversation

@LiranCohen

Copy link
Copy Markdown
Contributor

Fixes #273

Stacked on #271 (fix/feed-driven-refresh), which stacks on #270 — will retarget as the stack merges. Review only the top 3 commits.

What changed

Ports the enbox reference WS client's liveness discipline (dwn-clients json-rpc-socket.ts / web-socket-clients.ts, enbox #1364) to meshd's Go subscription client, so half-dead sockets (system sleep, NAT timeout, link flap) are detected in seconds instead of waiting out TCP or the 5-minute fingerprint settle check:

  • Per-connection heartbeat (internal/dwn/liveness.go): JSON-RPC rpc.ping every 30s; no pong within 10s → warn + force-close, and the existing run-loop backoff + cursor-resumed resubscribe takes over. The ping id is the heartbeat generation — only the pong for the active ping clears the deadline, so a late pong from a superseded ping can never defuse a newer deadline. The deadline is armed before the write (like the probe path), so even a write stalled on a dead socket gets force-closed and unblocked.
  • On-demand health probes (CheckHealth): connected → out-of-band ping with a 5s deadline, coalesced so concurrent probes share one verdict; miss → close + reconnect. In reconnect backoff → the pending wait is fast-forwarded so the next attempt starts immediately. A probe-verified connection also clears any pending heartbeat deadline (one armed before a system sleep must not kill a connection the probe just proved alive).
  • Wake signals (internal/engine/wakeprobe.go): the engine's existing netmon.Monitor gets a change callback — on RebindLikelyRequired || TimeJumped (TimeJumped checked separately because resume-from-sleep can suppress the rebind flag), all active subscription streams (feed or topology+delivery) are probed, debounced at 1s. This is the daemon analog of the browser online/tab-visible wake listeners. Callback is deregistered before watcher stop; late callbacks hit a no-op path.
  • TS-parity subtleties preserved (verified against the TS source in review): a heartbeat pong clears the deadline even when it's an error response (a server without rpc.ping degrades to an inert heartbeat, not a reconnect loop); a probe error response resolves unhealthy without force-closing, mirroring probeConnection. Constants (30s/10s/5s) match the reference and are configurable for tests.

Not ported, deliberately: the pooled-WS control plane (enbox #1387) — meshd runs 1–2 sockets per endpoint; there is nothing to multiplex.

Verification

  • go build ./... — clean
  • go vet ./... — clean
  • go test ./... -count=1 -race — all packages pass, no data races; liveness tests additionally stress-run at -count=3/-count=5 -race with zero flakes
  • New tests: fake-server contract tests (ping emitted after quiet interval, pong clears deadline, missed pong → close + cursor-resumed resubscribe, ack/ping interleaving, per-generation ping-id uniqueness), unit tests for generation discipline / probe coalescing / shutdown resolution, wake-prober debounce + fan-out, watcher-level CheckHealth across streams.
  • Independent adversarial review verified TS parity claims against the actual reference source, the coder/websocket concurrent-write-safety claim against the vendored library, lock ordering, and goroutine lifecycle across reconnect cycles.

🤖 Generated with Claude Code

LiranCohen and others added 3 commits July 27, 2026 04:01
Port the reference client's WebSocket liveness discipline (enbox #1364)
to the Go subscription manager. NAT timeouts, network switches, and
system sleep kill connections silently — TCP never learns, and the
subscription read loop blocks forever on a dead socket.

Per connected socket, a heartbeat goroutine sends a JSON-RPC rpc.ping
("hb-<uuid>") every 30s; a pong missing its 10s deadline logs a warning
and force-closes the connection so the existing run-loop error path
does backoff and resubscribes with the last cursor. The ping id is the
heartbeat generation: only the pong for the ACTIVE ping clears the
deadline, and a late pong for a superseded ping mutates nothing.

Subscription.CheckHealth is the on-demand verdict for wake signals:
ended subscriptions no-op false, reconnect backoff waits are
fast-forwarded through a wake channel, and connected sockets get an
out-of-band probe with a 5s deadline — coalesced across concurrent
callers — whose success also defuses any pending heartbeat deadline
armed before a sleep. SubscriptionManager.CheckHealth probes every
active subscription. Read-loop demux is untouched except that response
frames matching the outstanding ping/probe id are consumed by liveness;
everything else keeps today's behavior. Pings share the acks' write
discipline: whole-message conn.Write calls, serialized by the
websocket library.

Intervals mirror the TS defaults (30s/10s/5s) and are configurable via
WithHeartbeat / WithHealthProbeTimeout for tests.

Refs #273

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the daemon analog of the browser online/visibility wake listeners:
a netmon change callback registered while the engine runs treats
rebind-worthy link changes and wall-clock jumps (resume from sleep) as
wake signals and calls SubscriptionWatcher.CheckHealth, which probes
every active stream (feed, or topology+delivery) through the manager.
Dead sockets are torn down immediately and resubscribe with their
cursor; streams parked in reconnect backoff skip the remaining wait.

Callbacks within one second of the last probe are debounced so a
flapping link cannot stampede the DWN with rpc.ping traffic. The
callback is deregistered on engine stop before the watcher shuts down,
and probing is a safe no-op while the watcher is not running.

Refs #273

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A write stalled on a dead-but-unclosed socket would wedge the heartbeat
goroutine with no deadline armed — the silent-death class the heartbeat
exists to catch. Mirror the probe path: arm first, so deadline expiry
force-closes the connection and unblocks the stalled write too.

Refs #273

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant