test(network): give the reconnect test a budget above the backoff - #4569
Open
stevenvegt wants to merge 1 commit into
Open
stevenvegt wants to merge 1 commit into
stevenvegt wants to merge 1 commit into
Conversation
TestNetworkIntegration_OutboundConnection11Reconnects waits defaultTimeout, 5 seconds, for node1 to reconnect to node2 after node2 was restarted. That reconnect races the backoff: connect() resets it to a random value between 1 and 5 seconds after a connection that was up is closed cleanly, and the connect loop only picks up expired backoffs once per second. The worst case is therefore already over the budget before dialling starts, and the test fails intermittently under load. The reconnect wait now has its own budget of 15 seconds. The two other waits in the test are an initial connect and a disconnect, neither of which races a backoff, so they keep defaultTimeout. Test-only. The tight budget dates back to #1874, which introduced the random reset. Assisted-by: AI
stevenvegt
requested review from
Dirklectisch,
JorisHeadease,
gerardsn and
reinkrul
as code owners
September 18, 2026 15:49
reinkrul
approved these changes
Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TestNetworkIntegration_OutboundConnection11Reconnectsfails intermittently under load:The test restarts node2 and gives node1
defaultTimeout, 5 seconds, to reconnect by itself. That reconnect races the backoff. When a connection that was up is closed cleanly,connect()ends with:and
connectLooponly picks up expired backoffs on atime.NewTicker(time.Second). So the worst case is 5 seconds of backoff plus up to a second before the loop notices, before dialling and the TLS handshake even start. The budget is exceeded before the node does anything wrong.This is not new and is not caused by any recent change: the random reset came in with #1874 (2023-02-24) and
defaultTimeoutpredates it. I hit it twice while preparing the backports of #4467 and #4562 and checked the counterfactual: 6 clean full-package runs with #4562's changes, 4 clean runs without them, and 10 clean runs of the test in isolation. It only shows up when the whole package runs and the machine is busy, which is also why it is rare in CI rather than absent.Change
The reconnect wait gets its own budget of 15 seconds, comfortably above the 5 second maximum backoff plus the 1 second tick, with the arithmetic written down next to the constant. The two other waits in the test are an initial connect (no prior backoff) and a disconnect, neither of which races a backoff, so they keep
defaultTimeout.Test-only, no production code touched.
Verification
This is a timing budget, so there is no deterministic RED to show: the test passes both before and after when the machine is quiet.
-count=10passes on this branch. What the change does is remove a worst case that was already outside the budget by construction, which is visible in the two constants rather than in a test run.An alternative would be to have the test force an immediate reconnect, but that would defeat the point of the test, which is that the node reconnects on its own.
Related: #4321 reports the same shape (a fixed wait that propagation can exceed under load) on the gossip e2e test.