Skip to content

Sync-lag gauge pair and progressing-vs-stuck chain health #66

Description

@mfw78

Feature request from the shepherd watch-tower harvest (tracked shepherd-side by the [Upstream nexum-runtime] issue that cites this one). Derives from watch-tower's src/services/chain.ts watchdog and syncStatus metric and its src/services/api.ts /health; upstream watch-tower #85 and PR #209 (160 restarts in 34 hours from health that could not tell behind from stuck).

Host-side chain-delivery observability: a per-chain gauge pair chain_head_height and last_delivered_height, a per-chain no-block deadline knob, and health that distinguishes behind-but-progressing from stuck.

Why

An in-process watchdog that kills the process when a chain goes quiet, and health that conflates syncing with wedged, produce restart storms. The host-native shape inverts it: the runtime exports gauges and lets alerting decide; nothing in-process kills anything. Per-call RPC deadlines already exist in host/impls/chain.rs (per-chain request timeout with a dedicated timeout fault), so the scope here is gauges and health only, with a verification rider.

Scope

  • Per-chain gauges chain_head_height and last_delivered_height; lag is derived by the consumer, never stored.
  • Per-chain no-block deadline knob: quiet chains legitimately skip slots (watch-tower refactor(test): add BootScenario and Refusal boot helpers #85), so the deadline must be configurable per chain.
  • Health distinguishes progressing (delivered height advancing), behind (lag above threshold but still advancing), and stuck (no advance within the deadline).
  • Rider: verify the per-call deadlines in host/impls/chain.rs cover every chain call path; extend only if a path lacks one.

Done when

  • The gauge pair is exported per chain.
  • Stuck-vs-progressing is test-covered at the event-loop level.
  • The no-block deadline is configurable per chain with a documented default.
  • The per-call deadline verification rider is recorded (covered, or extended where a gap was found).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:chainProvider pool, chain-log/block subscriptions, backfill, RPC transportarea:observabilityMetrics, gauges, health endpoints, structured loggingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions