[nick-boss-tech]

🤖 AI text below 🤖 (reviewed by Nick Shanin)

Ready for review, but held as a draft due to the project's open-PR cap; I'll mark it open on request.

https://issues.apache.org/jira/browse/SOLR-6213

What happens today

A failed election watch is retried until the stack overflows.

When a replica is not the leader, LeaderElector.checkIfIamLeader sets a watch on the election node just ahead of it. If setting that watch fails with a KeeperException other than NoNode, the method calls itself again at once (line 161 on the base). Nothing limits the repeats and nothing waits between them. A failure that does not go away, such as the NoAuth error in the 2025 report on the ticket, ends in a StackOverflowError.

What this change does

The retry stops after three attempts, and the third failure is thrown to the caller.

A join then sees the real KeeperException and the log shows it. The NoNode case, where the node ahead disappeared, still checks again at once, because the next check finds a new node to watch. The limit is LeaderElector.MAX_WATCH_FAILURES.

Proof

The new test fails on the base code with a StackOverflowError and passes at this head.

LeaderElectionTest.testRepeatedWatchFailureIsRaisedInsteadOfRetriedForever makes every getData call on the second elector's ZooKeeper client fail with NoAuth. With the new test run against the base code, it fails with StackOverflowError. At head a55a45e8bab it passes, and the test verifies the NoAuth exception reaches the caller after exactly three getData calls. LeaderElectionTest as a whole passes at this head: 6 tests, 0 failures (seed 6213, run 2026-10-23). Tidy, the Error Prone compile, and :solr:core:check -x test also pass at this head.

A choice to check

Two routes were available: stop after three attempts and throw, or keep retrying with a growing pause.

Context: the retry only runs inside a watcher callback or a join, so there is no thread of its own to wait on. A persistent failure such as a missing permission does not clear in the milliseconds a short retry spans.

The deciding questions, in order: Can a retry fix the error? Only a short ZooKeeper hiccup, and three attempts cover that. Is a thread available to wait on? No. Which is worse, an exception that reaches the log or an endless loop? The loop.

Recommendation: stop after three attempts and throw (the route implemented here). Is a limit of three the right number?

Limits

The cap covers this watch retry only.

After the third failure the replica has no watch until it joins the election again, for example after a session expiry. Before this change that replica also ended without a watch, through the overflow. The nested rejoin path in ShardLeaderElectionContext is a different loop and is in a separate change. retryElection also re-joins without a limit; it creates a new election node on each pass and was left alone.

Changelog: changelog/unreleased/SOLR-6213-watch-retry.yml

AI-generated description. Nick Shanin directed the work and takes responsibility for this contribution.