Enhancement: concurrent chain polling

[faisal-chainlink]

Describe your changes

..

Issue ticket number and link

..

Describe highly relevant files or code snippets that are critical in the review

..

Are there other PRs that should be merged first?

..

[Copilot]

Pull request overview

This PR aims to enhance the chain polling/indexing pipeline to better support concurrent/continuous checkpoint processing and to expand testnet coverage for additional event sources (e.g., OnRamp events).

Changes:

• Updates ChainPoller to alter polling/catch-up behavior and adjusts rescan behavior/logging.
• Modifies checkpoint fetching to stop hydrating transaction events during GetCheckpointData.
• Extends the testnet chainreader integration test to start the indexer and query new OnRamp events from a configured starting checkpoint.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
relayer/client/grpc_client.go	Changes rate-limit label and removes per-transaction event hydration from GetCheckpointData.
relayer/chainreader/indexer/chain_poller.go	Alters polling/catch-up control flow and rescan behavior; adds/adjusts debug logging.
relayer/chainreader/reader/chainreader_testnet_test.go	Starts the indexer in the testnet test, binds OnRamp, and adds an OnRamp events assertion (with several subtests skipped).

Comments suppressed due to low confidence (3)

relayer/chainreader/indexer/chain_poller.go:260

• RescanRecent now starts a background catchUp with context.Background() and isn't tracked by the poller's WaitGroup. This can keep running after Close() and panic when sending on channels that run() closes. It also computes latestCheckpoint-defaultRescanRewindCheckpoints without an underflow guard. A safer approach is to rewind lastProcessed (so the main polling loop re-processes recent checkpoints) rather than launching an untracked goroutine.

	if err != nil {
		cp.logger.Errorw("Failed to get latest checkpoint", "error", err)
		return
	}
	go cp.catchUp(context.Background(), latestCheckpoint-defaultRescanRewindCheckpoints, latestCheckpoint)

relayer/chainreader/indexer/chain_poller.go:271

• The log message says "catchup goroutine", but catchUp can be called synchronously (and even if called via go, it's not guaranteed to be a goroutine long-lived concept worth logging). This wording is misleading and makes logs harder to interpret.

	cp.logger.Infow("Starting checkpoint catchup goroutine", "startSequence", startSeq, "endSequence", endSeq)

relayer/chainreader/indexer/chain_poller.go:321

• This completion log also refers to a "goroutine"; consider using neutral wording so the message remains correct regardless of how catchUp is invoked.

	cp.logger.Debugw("Checkpoint catchup goroutine completed",
		"start", startSeq,
		"end", endSeq,
	)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[FelixFan1992]

Can we make it so that: if we are 100K checkpoints behind when starting, we will spin up Go routines so that:

1. routine 1 will try to get 1-20 blocks, routine 2 will get 21-40 blocks, etc
2. but routine N+1 should not send/merge its batch of blocks into the database before routine N. it should hold on to them until N is done.
3. if a routine is having trouble fetching some checkpoints, it can retry in some ways. it should merge whatever it has into database after certain number of retries (or a timeout). but is this OK?? we should not allow 1 failure to block tho.
4. there will be at most X routines. X can be 8 or other sensible numbers.

[faisal-chainlink]

2. but routine N+1 should not send/merge its batch of blocks into the database before routine N. it should hold on to them until N is done.

What's the advantage of doing this?

there will be at most X routines. X can be 8 or other sensible numbers.

This means the concurrency wouldn't be doing much since there will be a significant amount of waiting around. There's no real advantage to using concurrency if point 2 above is required. It is in fact more risky to do this because more values will be held in memory and a simple relayer start would possibly entirely lose track of what was processed and what wasn't.

Another thing is that this design proposes a shared piece of data across Goroutines, this is another point of failure.

---

A simpler approach would be to simply cap concurrency and use batches but without one Goroutine waiting for another.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

relayer/chainreader/reader/chainreader_testnet_test.go:261

• Start returns an error; ignoring it can hide initialization failures (and can lead to later test flakiness when the indexer never actually started).

	indexerInstance.Start(ctx)