fix(dwn): port the dwn-clients WebSocket liveness discipline - #274
Open
LiranCohen wants to merge 3 commits into
Open
fix(dwn): port the dwn-clients WebSocket liveness discipline#274LiranCohen wants to merge 3 commits into
LiranCohen wants to merge 3 commits into
Conversation
Port the reference client's WebSocket liveness discipline (enbox #1364)
to the Go subscription manager. NAT timeouts, network switches, and
system sleep kill connections silently — TCP never learns, and the
subscription read loop blocks forever on a dead socket.
Per connected socket, a heartbeat goroutine sends a JSON-RPC rpc.ping
("hb-<uuid>") every 30s; a pong missing its 10s deadline logs a warning
and force-closes the connection so the existing run-loop error path
does backoff and resubscribes with the last cursor. The ping id is the
heartbeat generation: only the pong for the ACTIVE ping clears the
deadline, and a late pong for a superseded ping mutates nothing.
Subscription.CheckHealth is the on-demand verdict for wake signals:
ended subscriptions no-op false, reconnect backoff waits are
fast-forwarded through a wake channel, and connected sockets get an
out-of-band probe with a 5s deadline — coalesced across concurrent
callers — whose success also defuses any pending heartbeat deadline
armed before a sleep. SubscriptionManager.CheckHealth probes every
active subscription. Read-loop demux is untouched except that response
frames matching the outstanding ping/probe id are consumed by liveness;
everything else keeps today's behavior. Pings share the acks' write
discipline: whole-message conn.Write calls, serialized by the
websocket library.
Intervals mirror the TS defaults (30s/10s/5s) and are configurable via
WithHeartbeat / WithHealthProbeTimeout for tests.
Refs #273
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the daemon analog of the browser online/visibility wake listeners: a netmon change callback registered while the engine runs treats rebind-worthy link changes and wall-clock jumps (resume from sleep) as wake signals and calls SubscriptionWatcher.CheckHealth, which probes every active stream (feed, or topology+delivery) through the manager. Dead sockets are torn down immediately and resubscribe with their cursor; streams parked in reconnect backoff skip the remaining wait. Callbacks within one second of the last probe are debounced so a flapping link cannot stampede the DWN with rpc.ping traffic. The callback is deregistered on engine stop before the watcher shuts down, and probing is a safe no-op while the watcher is not running. Refs #273 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A write stalled on a dead-but-unclosed socket would wedge the heartbeat goroutine with no deadline armed — the silent-death class the heartbeat exists to catch. Mirror the probe path: arm first, so deadline expiry force-closes the connection and unblocks the stalled write too. Refs #273 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #273
What changed
Ports the enbox reference WS client's liveness discipline (dwn-clients
json-rpc-socket.ts/web-socket-clients.ts, enbox #1364) to meshd's Go subscription client, so half-dead sockets (system sleep, NAT timeout, link flap) are detected in seconds instead of waiting out TCP or the 5-minute fingerprint settle check:internal/dwn/liveness.go): JSON-RPCrpc.pingevery 30s; no pong within 10s → warn + force-close, and the existing run-loop backoff + cursor-resumed resubscribe takes over. The ping id is the heartbeat generation — only the pong for the active ping clears the deadline, so a late pong from a superseded ping can never defuse a newer deadline. The deadline is armed before the write (like the probe path), so even a write stalled on a dead socket gets force-closed and unblocked.CheckHealth): connected → out-of-band ping with a 5s deadline, coalesced so concurrent probes share one verdict; miss → close + reconnect. In reconnect backoff → the pending wait is fast-forwarded so the next attempt starts immediately. A probe-verified connection also clears any pending heartbeat deadline (one armed before a system sleep must not kill a connection the probe just proved alive).internal/engine/wakeprobe.go): the engine's existingnetmon.Monitorgets a change callback — onRebindLikelyRequired || TimeJumped(TimeJumped checked separately because resume-from-sleep can suppress the rebind flag), all active subscription streams (feed or topology+delivery) are probed, debounced at 1s. This is the daemon analog of the browseronline/tab-visible wake listeners. Callback is deregistered before watcher stop; late callbacks hit a no-op path.rpc.pingdegrades to an inert heartbeat, not a reconnect loop); a probe error response resolves unhealthy without force-closing, mirroringprobeConnection. Constants (30s/10s/5s) match the reference and are configurable for tests.Not ported, deliberately: the pooled-WS control plane (enbox #1387) — meshd runs 1–2 sockets per endpoint; there is nothing to multiplex.
Verification
go build ./...— cleango vet ./...— cleango test ./... -count=1 -race— all packages pass, no data races; liveness tests additionally stress-run at-count=3/-count=5 -racewith zero flakes🤖 Generated with Claude Code