fix(web3): querying different RPCs returns different head blocks

[sanbotto]

Fixes a bug in the "Query Contract Events" web3 step where a resolved (non-user-specified) toBlock was computed once via a standalone eth_blockNumber call and then reused for a later eth_getLogs call. Because the RPC provider's endpoint is a load-balanced pool of backend nodes that do not sync in unison, the two calls could land on different endpoints, and if the log query landed on a slightly slower endpoint than the one that answered the head check, it would reject the range with -32602 block range extends beyond current head.

This was discovered when an unrelated issue caused it to surface during the debugging session (see this PM doc).

The step's reported toBlock output is derived only from that same query's results (the highest block among the returned events, or no forward progress if none) rather than a separate getBlockNumber() call, since a second call could itself land on a different, more-advanced replica and overstate what was actually scanned. An explicit user-provided toBlock is left untouched so a genuine misconfiguration still surfaces as an error. Unit tests cover the retry/failover wiring and the tip-batch query against a mocked contract, verifying the "latest" tag, the max-of-events derivation, and the no-events fallback.

Opening as draft since I wanna test it in a PR env.

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-1702.keeperhub.com
• Namespace: pr-1702
• Cluster: techops-staging
• Image Tag: c6a512f
• Commit: c6a512fbade14b169229fc2d6b21a4a2a31b977b

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[sanbotto]

Just finished running a smoke test. The new code path is functionally correct against our RPC endpoints.

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-1702.keeperhub.com
• Namespace: pr-1702
• Cluster: techops-staging
• Image Tag: c0091f9
• Commit: c0091f90eb6eba90ff1f4c6f70fff4824aa5b6ec

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[github-actions]

PR Environment Deployed

Your PR environment has been deployed!

Environment Details:

• URL: https://app-pr-1702.keeperhub.com
• Namespace: pr-1702
• Cluster: techops-staging
• Image Tag: 1f6081f
• Commit: 1f6081fa50b371c5924a79bda87248aed263b9e0

Components:

• Keeperhub Application
• PostgreSQL Database (isolated instance)
• LocalStack (SQS emulation)
• Redis (isolated instance)
• Schedule Dispatcher (staging image)
• Block Dispatcher (staging image)
• Event Tracker (staging image)

The environment will be automatically cleaned up when this PR is closed or merged.

[joelorzet]

Hi @sanbotto, nice fix. One question on scope: isTipBatch keys off end === range.toBlock, so only the final batch queries "latest". When the range length leaves a small remainder over the 2000 batch size (for example a fromBlock/blockCount that makes the tip batch tiny), the batch right before it ends within a few blocks of the head, and a lagging replica can still reject it with the same -32602 beyond current head. Retries don't reliably help there for the same reason they don't in the tip case, since the mismatch can persist across replicas.

Would it be worth scoping the "latest" batch to the near-head region rather than the exact last batch, so no fixed batch ever ends within replica-lag distance of the head? Low frequency, but it's the same failure one batch over.