You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Feature request from the shepherd watch-tower harvest (tracked shepherd-side by the [Upstream nexum-runtime] issue that cites this one). Derives from watch-tower's src/services/chain.ts watchdog and syncStatus metric and its src/services/api.ts/health; upstream watch-tower #85 and PR #209 (160 restarts in 34 hours from health that could not tell behind from stuck).
Host-side chain-delivery observability: a per-chain gauge pair chain_head_height and last_delivered_height, a per-chain no-block deadline knob, and health that distinguishes behind-but-progressing from stuck.
Why
An in-process watchdog that kills the process when a chain goes quiet, and health that conflates syncing with wedged, produce restart storms. The host-native shape inverts it: the runtime exports gauges and lets alerting decide; nothing in-process kills anything. Per-call RPC deadlines already exist in host/impls/chain.rs (per-chain request timeout with a dedicated timeout fault), so the scope here is gauges and health only, with a verification rider.
Scope
Per-chain gauges chain_head_height and last_delivered_height; lag is derived by the consumer, never stored.
Health distinguishes progressing (delivered height advancing), behind (lag above threshold but still advancing), and stuck (no advance within the deadline).
Rider: verify the per-call deadlines in host/impls/chain.rs cover every chain call path; extend only if a path lacks one.
Done when
The gauge pair is exported per chain.
Stuck-vs-progressing is test-covered at the event-loop level.
The no-block deadline is configurable per chain with a documented default.
The per-call deadline verification rider is recorded (covered, or extended where a gap was found).
Host-side chain-delivery observability: a per-chain gauge pair
chain_head_heightandlast_delivered_height, a per-chain no-block deadline knob, and health that distinguishes behind-but-progressing from stuck.Why
An in-process watchdog that kills the process when a chain goes quiet, and health that conflates syncing with wedged, produce restart storms. The host-native shape inverts it: the runtime exports gauges and lets alerting decide; nothing in-process kills anything. Per-call RPC deadlines already exist in
host/impls/chain.rs(per-chain request timeout with a dedicated timeout fault), so the scope here is gauges and health only, with a verification rider.Scope
chain_head_heightandlast_delivered_height; lag is derived by the consumer, never stored.host/impls/chain.rscover every chain call path; extend only if a path lacks one.Done when