Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions docs/deployment/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,18 @@ flags as the router.
## Sharing state across routers

When several router instances share one cache, add `--shared-state` to the routers so
they also share consumer-consistency state through it — keeping their "seen block" views
aligned. See the [CLI reference](../reference/cli.md#cache-shared-state).
they also share two things through it:

- **consumer-consistency state** — their "seen block" views stay aligned, so a client
that just read block N from one replica is not served N-1 by another;
- **chain-tracker poll observations** — each replica publishes the upstream polls it
makes and borrows fresh ones from the others, so every upstream is polled about
once per interval fleet-wide instead of once per replica. This is the lever that
stops the router's own polling load from growing with the replica count.

See the [CLI reference](../reference/cli.md#cache-shared-state) for the safety floors and
[`rpc_endpoint_tracker_gate_skips_total`](../reference/metrics.md#endpoint-scoped-rpc_endpoint_)
for how to confirm it is working.

## Observability

Expand Down
64 changes: 59 additions & 5 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,59 @@ See [Failover & retry](../configuration/failover/index.md).
## Polling relief

Each configured upstream gets its own chain tracker, which polls it for the latest
block independently of the relays you send. These two flags lower that load; both
are process-wide. Watch the effect on
block independently of the relays you send. These flags lower that load; all are
process-wide. Watch the effect on
[`rpc_endpoint_tracker_requests_total`](metrics.md#endpoint-scoped-rpc_endpoint_),
and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.

| Flag | Default | Description |
| --- | --- | --- |
| `--enable-fork-detection` | `false` | Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the two savings. |
| `--chain-tracker-poll-divisor` | `2` | The tracker polls every `avgBlockTime ÷ divisor`. `1` halves the polling rate. Allowed `[1,8]`; out-of-range reverts to the default. |
| `--enable-fork-detection` | `false` | Turn on block-hash polling (reorg detection on upstreams). Off by default — the larger of the per-pod savings. |
| `--chain-tracker-poll-divisor` | `2` | The tracker polls every `avgBlockTime ÷ divisor`. `1` halves the polling rate; `0.25` — one poll per **four** block times — cuts it eightfold. Allowed `[0.25,8]`; out-of-range reverts to the default. See [Polling slower than the chain](#polling-slower-than-the-chain). |
| `--shared-state` (with `--cache-be`) | `false` | Share poll observations across router replicas through the cache, so an upstream is polled about once per interval **fleet-wide** rather than once per pod. See [Cache & shared state](#cache-shared-state). |

The tracker also skips a poll whenever something else has already kept the upstream's
tip fresh — the relays it served in the last block time, or (with `--shared-state`) a
poll another replica made. Skipped ticks show up on
[`rpc_endpoint_tracker_gate_skips_total`](metrics.md#endpoint-scoped-rpc_endpoint_) by
`source`, so `requests_total` falling while `gate_skips_total` rises is the relief
working, not the tracker stalling.

### Polling slower than the chain

A divisor below `1` polls less often than the chain produces blocks, which is where
the relief is on a fast chain. Measured requests/min per endpoint:

| Chain | `2` (default) | `1` | `0.5` | `0.25` |
| --- | --- | --- | --- | --- |
| Aptos | 600 | 300 | 150 | 75 |
| Solana | 300 | 150 | 75 | 37.5 |
| Base | 60 | 30 | 15 | 7.5 |
| Ethereum | 9.2 | 4.6 | 2.3 | 1.2 |

What bounds the low end is the staleness window — `max(10 × avgBlockTime, 2s)`, past
which an observation stops counting for consensus, the tip reads unknown, and the probe
scores a healthy upstream not-alive. The window does **not** move with this flag; the
flag moves how long a tip can go unrefreshed, the other side of that comparison. Both
common cases stay well inside it:

- **Idle upstream** — nothing but its own poll refreshes the tip, so the gap *is* the
interval: 4 × `avgBlockTime` at `0.25`, against a 10× window.
- **Served upstream** — relays refresh the same tip, so traffic bounds the gap.

The seam is between them: an upstream that trips the traffic gate and *then* goes quiet
is refreshed by neither, and the worst-case gap becomes `(maxRelaySkips + 1) × interval`
— 20 × `avgBlockTime` at `0.25`. That is a property of the **product** of this flag and
the gate's skip budget, not of either alone, so the router warns once per chain at
startup rather than refusing to start:

```
WRN poll cadence can outrun the staleness window when the traffic gate skips
pollInterval=60s worstCaseGapBetweenPolls=5m0s stalenessWindow=2m30s maxRelaySkips=4
```

It is a line to read, not a failure — the configuration is safe for both common cases.
If you see it on a chain with bursty traffic, step back toward `0.5` or `1`.

## Consistency tuning

Expand All @@ -89,7 +133,17 @@ and confirm what is live via `HashPolling` in `GET /debug/endpoint-state`.
| Flag | Default | Description |
| --- | --- | --- |
| `--cache-be` | — | Address of the cache server (e.g. `127.0.0.1:20100`). In Compose the cache address usually comes from `cache-be:` in the config instead. |
| `--shared-state` | `false` | Share consistency state across router instances via the cache (use with `--cache-be`). |
| `--shared-state` | `false` | Share state across router replicas through the cache (use with `--cache-be`): the consumer-consistency "seen block", **and** per-endpoint chain-tracker poll observations, so the fleet polls each upstream about once per interval instead of once per replica. |

With `--shared-state`, every replica publishes each successful poll of an upstream to
the cache and, before polling, checks whether **another** replica polled that upstream
within the last block time. If one did, the tick is skipped and the peer's block is
adopted (it appears as `Source: peer` in `GET /debug/endpoint-state`). Three floors
keep it safe: a replica never borrows its own observation, so a single replica polls
exactly as without the flag; every replica still polls each upstream itself every few
ticks, so a broken path from one pod stays detectable; and an upstream that was
disabled is only re-enabled by that pod's own successful poll. Latency is never
shared — a peer's round-trip says nothing about this pod's path.

## WebSocket

Expand Down
9 changes: 8 additions & 1 deletion docs/reference/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ If you only graph a handful of things, graph these:
| Are requests failing? | `smartrouter_total_errored`, `smartrouter_errors_total` (by `error_category` / `retryable`) |
| How fast is it? | `smartrouter_end_to_end_latency_milliseconds` (histogram → p50/p95/p99) |
| Is a node degrading? | `rpc_endpoint_overall_health`, `rpc_endpoint_end_to_end_latency_milliseconds`, `rpc_endpoint_latest_block` |
| How much load does the router itself put on my node? | `rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it |
| How much load does the router itself put on my node? | `rpc_endpoint_tracker_requests_total` (by `kind`) — the router's own polling, separate from the relays you sent it; `rpc_endpoint_tracker_gate_skips_total` (by `source`) — the polls it chose not to send |
| Is failover working hard? | `smartrouter_retries_total`, `smartrouter_hedge_total` |
| Is an upstream rate-limiting us? | `smartrouter_rate_limit_holdoffs_total` (by `provider` / `event`), `smartrouter_rate_limit_holdoff_seconds` |
| Is the cache earning its keep? | `smartrouter_cache_success_total` / `smartrouter_cache_requests_total` |
Expand Down Expand Up @@ -202,6 +202,13 @@ They split into **endpoint-scoped** (`rpc_endpoint_*`) and **router-scoped**
| `rpc_endpoint_fetch_latest_success` | Counter | `spec`, `apiInterface`, `endpoint_id` | New-block **detections** by the chain tracker, not successful requests. |
| `rpc_endpoint_fetch_block_success` | Counter | `spec`, `apiInterface`, `endpoint_id` | Successful specific-block fetches. |
| `rpc_endpoint_tracker_requests_total` | Counter | `spec`, `apiInterface`, `endpoint_id`, `kind` | Requests the chain tracker actually sent upstream, by `kind` — `latest_block`, or `block_hash` (zero unless [`--enable-fork-detection`](cli.md#polling-relief)). The only metric that measures tracker **request volume**. |
| `rpc_endpoint_tracker_gate_skips_total` | Counter | `spec`, `apiInterface`, `endpoint_id`, `source` | Poll ticks the tracker skipped because the upstream's tip was already fresh, by `source` — `relay` (served traffic in the last block time) or `peer` (another replica's poll, shared via [`--shared-state`](cli.md#cache-shared-state)). The complement of `requests_total`: together they account for every tick. |

On a multi-replica deployment with `--shared-state`, the fleet-wide
`sum(rate(rpc_endpoint_tracker_requests_total{kind="latest_block"}[5m]))` for an upstream
should settle near a single replica's worth, and `source="peer"` skips should be non-zero on
every replica. `peer` stuck at zero with the flag on means the replicas are not reaching the
same cache.

Two caveats when reading the chain-tracker counters:

Expand Down