fix(api): make /readyz detect a stalled indexer - #128
Merged
Conversation
The /metrics scrape and its gauge parser were private to the sync-progress handler. The readiness probe needs the same two pieces, so move them into src/api/prometheus.ts unchanged.
Ponder's /ready latches at 200 once historical sync finishes and never goes back, so /readyz inherited a blind spot: an indexer whose realtime subscription dies keeps reporting itself ready while its data ages. In production a dRPC WebSocket stopped delivering newHeads without closing the socket; Ponder logged one warning and idled, and every probe stayed green for a week. /readyz now also reads ponder_sync_block_timestamp per active chain and fails when the newest synced block is older than READINESS_MAX_LAG_SECONDS (default 300, per-chain override with a numeric chain-id suffix). The 503 body names the chain, its block number, and the measured lag. Compares against wall-clock rather than calling eth_blockNumber on purpose: a readiness probe that depends on the RPC turns an RPC outage into a restart loop, and the block timestamp already carries enough to spot a stalled sync. The trade-off is that a genuine chain halt reads as staleness. A chain missing from the metrics counts as stale — absent data is not evidence of freshness.
Two separate gaps kept the week-long stall invisible. The health check probed /ready, which stays 200 forever after the initial sync, so it could never fail no matter how stale the data got. It now probes /readyz, which carries the freshness gate. Also pins an explicit interval and timeout rather than leaning on Docker's defaults. Docker does not restart a container for failing its health check — restart: unless-stopped only reacts to the process exiting — so even a failing check would have left the container sitting there unhealthy. The autoheal label wires it to the host's willfarrell/autoheal daemon, which restarts unhealthy labelled containers. Inert if that daemon is not running on the host.
…ainer Records the /ready latching behaviour that hid the stall, the staleness budget and its env overrides, why the probe avoids an RPC call, and the two easy misreadings of the container health check: Docker never restarts on an unhealthy check, and the 24-hour start period is what covers a cold start.
…aveat Nothing in docs/ mentioned that Ponder can run the HTTP server separately from the indexer, so a team taking over operations had no way to know the option exists. Documents ponder serve, the four constraints read off 0.16.6, and the fact that it has not been run here. Calls out the interaction with the freshness gate: it reads an in-memory gauge that only the indexing process sets, so an API-only container would fail /readyz forever and autoheal would restart it in a loop. Records _ponder_checkpoint as the DB-backed source to switch to, with the internal-table caveat.
The freshness gate parsed ponder_sync_block_timestamp out of the Prometheus text format. Those gauges live in the indexing process's memory, which pinned the probe to a process that also has to be the one serving HTTP. Ponder's /status exposes the same information from the database — it decodes the _ponder_checkpoint table into a block number and timestamp per chain — so the check now reads committed state instead. Concretely this means /readyz stays correct if the API is ever split from the indexer with ponder serve, drops the Prometheus text parsing, and reads a documented JSON endpoint rather than internal gauge names. Also restores the Prometheus parser to sync-progress.ts. It was extracted for the freshness gate to share; with that gone, sync-progress is its only consumer again and the extra module earned nothing.
Trims the endpoint tables and drops the restated rationale, keeping the constraints themselves. Notes that splitting the API from the indexer also enables horizontal scaling on the API side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On the test deploy, the indexer stopped indexing on 29 July. A dRPC WebSocket stopped delivering
newHeadswithout closing the connection, so Ponder never saw an error or close event, logged one "No new block received within expected time" warning per chain, and idled. The process stayed alive and kept serving GraphQL, so every signal we had stayed green: Ponder's/readylatches at 200 once historical sync finishes and never goes back, and that is exactly what the container health check was probing. After a restart it happened again 18 hours later, same shape./readyznow also compares each active chain's newest synced block against wall-clock time, readingponder_sync_block_timestampfrom/metrics, and returns 503 naming the chain, its block number, and the lag when it exceedsREADINESS_MAX_LAG_SECONDS(default 300, per-chain override with a numeric chain-id suffix). It deliberately does not calleth_blockNumberfor the real head: a readiness probe that depends on the RPC turns an RPC outage into a restart loop, and the block timestamp already carries enough to spot a stall — the trade-off being that a genuine chain halt reads as staleness. Two deployment-side changes matter as much as the endpoint: theDockerfilehealth check now probes/readyzinstead of/ready, and the ponder service carries anautoheal: "true"label, because Docker does not restart a container for failing its health check (restart: unless-stoppedonly reacts to the process exiting) and the host'swillfarrell/autohealdaemon is what turns unhealthy into a restart. Worth a reviewer's eye: the freshness gate is inside/readyz, which also gates on the owner backfill, so it needs the backfill count never to climb back above zero after startup — it doesn't, because generators created during live sync are written withhistoryBackfilled = true(composableCow.ts:232), and the 24-hour start period covers the initial drain. The last commit is documentation only, added because we are handing operations to another team: it records that Ponder can run the API separately from the indexer viaponder serve(undocumented until now, and untested here), and that doing so would break this freshness gate, since the metric it reads only exists in the indexing process —_ponder_checkpointis the DB-backed source to move to if anyone splits them.