Skip to content

test(network): give the reconnect test a budget above the backoff (backport of #4569) - #4570

Open
stevenvegt wants to merge 1 commit into
V6.2from
V6_2/reconnect-timeout-budget
Open

stevenvegt wants to merge 1 commit into
V6.2from
V6_2/reconnect-timeout-budget

Conversation

@stevenvegt

Copy link
Copy Markdown
Member

Backport of #4569.

Problem

TestNetworkIntegration_OutboundConnection11Reconnects fails intermittently under load with "time-out while waiting for node 1 to reconnect to node 2".

The test restarts node2 and gives node1 defaultTimeout, 5 seconds, to reconnect by itself. That reconnect races the backoff: when a connection that was up is closed cleanly, connect() resets the backoff to RandomBackoff(time.Second, 5*time.Second), and connectLoop only picks up expired backoffs on a one second ticker. The worst case is therefore already over the budget before dialling and the TLS handshake start.

This is not new and is not caused by any recent change. The random reset came in with #1874 (2023-02-24) and defaultTimeout predates it.

Change

The reconnect wait gets its own budget of 15 seconds, with the arithmetic written down next to the constant. The two other waits in the test are an initial connect and a disconnect, neither of which races a backoff, so they keep defaultTimeout.

Test-only, no production code touched. Cherry-picked without conflicts.

Verification

This is a timing budget, so there is no deterministic RED: the test passes before and after when the machine is quiet. -count=5 passes on this branch. What the change removes is a worst case that was outside the budget by construction, visible in the two constants rather than in a test run.

TestNetworkIntegration_OutboundConnection11Reconnects waits
defaultTimeout, 5 seconds, for node1 to reconnect to node2 after node2
was restarted. That reconnect races the backoff: connect() resets it to
a random value between 1 and 5 seconds after a connection that was up
is closed cleanly, and the connect loop only picks up expired backoffs
once per second. The worst case is therefore already over the budget
before dialling starts, and the test fails intermittently under load.

The reconnect wait now has its own budget of 15 seconds. The two other
waits in the test are an initial connect and a disconnect, neither of
which races a backoff, so they keep defaultTimeout.

Test-only. The tight budget dates back to #1874, which introduced the
random reset.

Assisted-by: AI
(cherry picked from commit 71c1629)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants