Skip to content

test(network): give the reconnect test a budget above the backoff - #4569

Open
stevenvegt wants to merge 1 commit into
masterfrom
test/reconnect-timeout-budget
Open

stevenvegt wants to merge 1 commit into
masterfrom
test/reconnect-timeout-budget

Conversation

@stevenvegt

Copy link
Copy Markdown
Member

Problem

TestNetworkIntegration_OutboundConnection11Reconnects fails intermittently under load:

--- FAIL: TestNetworkIntegration_OutboundConnection11Reconnects
    time-out while waiting for node 1 to reconnect to node 2

The test restarts node2 and gives node1 defaultTimeout, 5 seconds, to reconnect by itself. That reconnect races the backoff. When a connection that was up is closed cleanly, connect() ends with:

// Connection was OK, but now disconnected. Add a random wait to prevent simultaneous reconnecting.
contact.backoff.Reset(RandomBackoff(time.Second, 5*time.Second))

and connectLoop only picks up expired backoffs on a time.NewTicker(time.Second). So the worst case is 5 seconds of backoff plus up to a second before the loop notices, before dialling and the TLS handshake even start. The budget is exceeded before the node does anything wrong.

This is not new and is not caused by any recent change: the random reset came in with #1874 (2023-02-24) and defaultTimeout predates it. I hit it twice while preparing the backports of #4467 and #4562 and checked the counterfactual: 6 clean full-package runs with #4562's changes, 4 clean runs without them, and 10 clean runs of the test in isolation. It only shows up when the whole package runs and the machine is busy, which is also why it is rare in CI rather than absent.

Change

The reconnect wait gets its own budget of 15 seconds, comfortably above the 5 second maximum backoff plus the 1 second tick, with the arithmetic written down next to the constant. The two other waits in the test are an initial connect (no prior backoff) and a disconnect, neither of which races a backoff, so they keep defaultTimeout.

Test-only, no production code touched.

Verification

This is a timing budget, so there is no deterministic RED to show: the test passes both before and after when the machine is quiet. -count=10 passes on this branch. What the change does is remove a worst case that was already outside the budget by construction, which is visible in the two constants rather than in a test run.

An alternative would be to have the test force an immediate reconnect, but that would defeat the point of the test, which is that the node reconnects on its own.

Related: #4321 reports the same shape (a fixed wait that propagation can exceed under load) on the gossip e2e test.

TestNetworkIntegration_OutboundConnection11Reconnects waits
defaultTimeout, 5 seconds, for node1 to reconnect to node2 after node2
was restarted. That reconnect races the backoff: connect() resets it to
a random value between 1 and 5 seconds after a connection that was up
is closed cleanly, and the connect loop only picks up expired backoffs
once per second. The worst case is therefore already over the budget
before dialling starts, and the test fails intermittently under load.

The reconnect wait now has its own budget of 15 seconds. The two other
waits in the test are an initial connect and a disconnect, neither of
which races a backoff, so they keep defaultTimeout.

Test-only. The tight budget dates back to #1874, which introduced the
random reset.

Assisted-by: AI
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants