fix(discovery): jitter the refresh retry backoff

[EricAndrechek]

Closes #141. Part of #613.

What

SchemaRegistry.RetryRefresh used to sleep exactly 2s * 2^n, capped at 60s. That means N instances retrying against one recovering ClickHouse fire in lockstep once they all reach the cap. With this change, each sleep is a uniform draw from [0, backoff) (full jitter). The bound still doubles from 2s to 60s.

Why full jitter instead of the ±10% the issue proposed: at the 60s cap, ±10% spreads the retries over only 12s. Full jitter spreads them over the whole 60s window. That makes it the strongest way to break lockstep without state or coordination (AWS's "Exponential Backoff and Jitter" analysis). One side effect: the mean wait halves. So during an outage, a failing tenant's retries, its log lines and wavehouse_schema_refresh_failures_total come about twice as often. Nothing in this repo consumes that counter yet (it was new in #610). The CHANGELOG entry mentions this.

The random source is an injectable retryDelay func(time.Duration) time.Duration field (rand.N by default), set up the same way as the existing firstTick. It adds no dependency.

Tests

• TestRetryRefresh_BackoffIsBounded now records the backoff each sleep is drawn within (1, 2, 4, 4, 4 ms) instead of timing the wall clock, so it can't flake.
• TestRetryRefresh_SleepsTheJitteredDelay: the loop sleeps the drawn delay, not the backoff.
• TestRetryRefresh_DelayIsSpreadOverTheBackoff: 200 draws from the production retryDelay all fall in [0, backoff), reach both the bottom and top quarters, and are nearly all distinct.

Left for later

• The discovery: add jitter to RetryRefresh exponential backoff for clustered mode #141 criterion "confirm the jitter range once clustered mode has a topology config" is conditional and stays with that work.
• There is no shared backoff helper yet. internal/auth (the JWKS retry) has its own unjittered loop, and feat/ch-error-classes is adding a ClickHouse backoff in internal/ingest. Consolidating them is a follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: e0581e82-7488-4548-a0f8-f35aba9b5fbc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📚 Docs preview is live → https://00bdb03b-wavehouse-docs.wave-rf.workers.dev

• Commit — f5d5e04: Merge remote-tracking branch 'origin/main' into fix/discovery-retry-jitter
• Author — @EricAndrechek
• Committed — 2026-09-25 18:44 (UTC-04:00)
• Deployed — 2026-09-25 19:00 EDT

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit f5d5e04 in the fix/discovery-retry-... branch remains at 94%, unchanged from commit cb78f78 in the main branch.

Show a line coverage summary of the most impacted files.

File	main cb78f78	fix/discovery-retry-... f5d5e04	+/-
internal/mq/embedded.go	89%	88%	-1%
internal/ingest/worker.go	98%	97%	-1%
internal/discov...ry/discovery.go	96%	96%	0%

---

_{Updated September 25, 2026 23:01 UTC}