Repository navigation
Server hangs endlessly (0.00 Mbits/sec) on sudden client disconnect – Works in 3.14, broken in 3.16+ #1986
Description
iperf3 Version(s)
Working: 3.14
Intermittent: 3.15
Broken: 3.16 and newer
Operating System / Architecture
Server: Alma Linux 10
Client: Android
Defect Description
I am encountering a regression in how iperf3 handles sudden client disconnections (specifically with mobile devices losing signal).
When a client device (mobile) abruptly loses connection during a test (e.g., 4G/5G drops or is toggled off), the server instance should detect the loss, report "Connection reset by peer," and return to the listening state.
In versions 3.16 and newer, the server fails to detect the disconnect. Instead of resetting, it hangs indefinitely, printing 0.00 Mbits/sec lines until manually killed.
Regression / Bisect Info
I have tested multiple versions to isolate the behavior:
v3.14: Works reliably. The server detects the drop immediately, prints iperf3: error - unable to read from stream socket: Connection reset by peer, and waits for a new client.
v3.15: Unreliable. It works sometimes, but occasionally exhibits the hanging behavior.
v3.16+ (up to current): consistently broken. The server never resets and hangs on 0.00 Mbits/sec until the process is killed.
Steps to Reproduce
Start an iperf3 server: iperf3 -s
Connect a client (a mobile device over 4G or 5G): iperf3 -c <server_ip> -t 100
Midway through the test, simulate a signal loss on the client (e.g., disable 4G/5G data or toggle airplane mode).
Observe Server Output:
Expected (v3.14): Server errors out with "Connection reset" and restarts the listener.
Actual (v3.16+): Server continues running, printing 0.00 Mbits/sec indefinitely.
Activity
davidBar-On commentedon Dec 18, 2025
@Vili-B, can you try running the server with
--rcv-timeoutoption to see if it helps. E.g. setting--rcv-timeout 10should stop the server about 10 seconds after the "signal loss". Note that for backward compatibility the default is no receive timeout.(Even if
--rcv-timeoutsolve the problem, it is probably only a workaround, as I guess the server's receiving thread is terminated. However, currently the main thread of iperf does not detect termination of sending/receiving threads. E.g. see PR #1654.)WatcherOfTheSkies commentedon Dec 19, 2025
Do you mean
--rcv-timeout?It's default is 120 sec, and range 100sec to 1 day!
davidBar-On commentedon Dec 19, 2025
You are right. Sorry for the confusion. In this case, did you wait the 120 sec after the signal loss and the server still didn't stop the test? Can you try setting the
--rcv-timeoutmanually to shorter time (e.g. 10 sec) and see if it also doesn't cause the test to stop?I have tried that by code changes (I don't have proper test environment for mobile) - stopping sending data from the client in the middle of the test and then also terminating the sending thread at that point. In both cases, the server terminated the test after the
--rcv-timeouttimed out (I set it to 5 seconds and set client test time to-t 20, but that was the same with the default 120 sec receive timeout with-t200).Also, I evaluated the iperf3 code and didn't find a case were no data is received and the
--rcv-timeouttimeout does not cause the server to terminate the test.Vili-B commentedon Dec 19, 2025
Yes I have waited for the 2 minute default mark and tried to change the --rcv-timeout to 5 sec or 10 sec but neither works, it just keeps showing 0 until manually stopped.
davidBar-On commentedon Dec 19, 2025
Can you run the server using
-V --debug=3options? Main purpose is to see the "State change: State set to ..." messages , which may indicate whether the server did identify the test termination condition, and is stuck because it sent a control message to the client and is waiting for a response or something like that.Alternatively (or in addition), if you are able to build iperf3 can you try building and testing using the code of PR #1764? Although it is the same test scenario, the issue seems to be similar, so it may be that the PR change solves also this problem.
Vili-B commentedon Dec 19, 2025
I ran the server with the
-V --debug=3plus--rcv-timeoutwith 5 secodns below there are the logs, client was put to airplane mode at the 5 second mark.davidBar-On commentedon Dec 19, 2025
Thanks for running the test. I don't see anything that can help in the logs.
Can you re-run the test and send the server logs for 15 seconds (instead of 7)? The reason I am asking if that is the client was put into airplane mode after 5 seconds and the receive timeout is 5 seconds, then the server receive timeout should be triggered only at about second 10 of the test (the 15 is just to make usre, in case the trigger is not exactly at the 10's second).
Vili-B commentedon Dec 19, 2025
Good point! The test was run exactly the same as previously.
bmah888 commentedon Dec 19, 2025
Thanks for the issue report! I fixed up some of the pasted program output in a couple comments to make them easier to read.
bmah888 commentedon Dec 19, 2025
@Vili-B, any chance you could try running with some newer code (latest released version is 3.20)? I'm pretty sure we did some changes to some timeouts, etc. in the code after 3.16.
We just started looking at this. I did just a couple of iperf3 tests with VMs (server FreeBSD 14, client Ubuntu 22), where I killed the VM running the client side in the middle of the test (to simulate an ungraceful loss of connectivity). The server continued to run the test for approximately another 50 seconds until it emitted
iperf3: error - select failed: Bad file descriptor, terminated the test, and set up for another client connection. This is (roughly) what we'd expect to see.Vili-B commentedon Dec 19, 2025
Those tests are run with the new 3.20 which I compiled to a debian 12 server. I also let the server idle after dropping the connection and it still continued to show 0mbps at 2700 seconds.
davidBar-On commentedon Dec 20, 2025
Can you send the iperf3 commands you are using for both the client and server? E.g. I see that the test runs with 4 streams (
-P 4) which is not mentioned in the description. Also, can you send the server version information (iperf3 -v)?In any case, it is not clear why the server's receive timeout has no effect or why the server does not terminate the test (or even crash as in my simulation case) 40 seconds after the planned time for the test completion (10+40=50 seconds by default, 100+40 in your test?).
Vili-B commentedon Dec 20, 2025
Server:
iperf3 --server -p 22020 --rcv-timeout 5000 -V --debug=3Client:
iperf3 -c <address> -p 22020 -t 0 -P 4 -Riperf 3.20+ (cJSON 1.7.15) Linux docker 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1 (2025-09-20) x86_64 Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment, POSIX threadsdavidBar-On commentedon Dec 21, 2025
As this is a reverse (
-R) test, the server is the sender, the--rcv-timeouthas not effect. Also, because the test time is unlimited (-t 0), the server will not terminate because the test time expired. In this case, can you try each of the following tests separately to see which mechanism helps:-t 60. In this case the server should terminate the test after 100 seconds of testing (60+40, the equivalent to the 50 seconds @bmah888 mentioned above).--cntl-ka=5/4/3. In this case the server is expected to terminate the test 5+(4*3)=17 seconds after the disconnection from the client.--snd-timeout 5000. If that helps, the server will disconnect the test 5 seconds after the disconnection from the client.Vili-B commentedon Dec 22, 2025
Limiting the test time works, the server starts waiting for a new client after a few seconds. but it is not the intended way for my case. The send timeout does not work. The control channel keep alive seems to work, but I wonder if this is the intended way but for now it will patch the issue for my case.
davidBar-On commentedon Dec 22, 2025
@Vili-B thanks for testing all the options.. You are right that this was not the intended use case for the keep alive. Its main use case is to make sure that the control socket is not terminated during the text by Proxy / High-availability server / etc., as during the test there are no control messages over this socket. A better approach is probably that the main thread will be aware that sockets are disconnected or that data threads fail (as PR #1654 suggests).
bmah888 commentedon May 15, 2026
I've been banging away on this problem trying to understand exactly what's going on. I think I've been able to reproduce it with two VMs running under UTM on my macOS 26 laptop. The reason for using VMs is to have control over the simulated state of the physical machine.
--serverplus other options as below.iperf3 --client 192.168.64.8 -t 0 -R(192.168.64.8 is the IPv4 address of the server)Results vary depending on the server arguments:
--server(no other arguments), the server side never terminates, although it will if the client VM is resumed.--server --cntl-ka=15/4/3the server will terminate about 15-20 seconds after the client VM is paused. This works roughly as I would have expected.--server --snd-timeout=5000the server never terminates, it appears this option has no effect on this case. I didn't expect this.@davidBar-On suggested PR #1654 as a possible solution (I think). It touches quite a few files and it also can't apply cleanly to
masterdue to code changes on that codeline since the PR was submitted. I haven't studied this in-depth.davidBar-On commentedon May 16, 2026
I believe that the
--snd-timeoutdoes help and it causes the sending thread to terminate. However, the server's main thread is not aware of this termination and also it is not waiting for any response from the client (unless--cntl-kais set), so the main thread is not terminated in this case.Merging PR #1861 (rebased now), that adds error message in case a thread is terminated, can help verifying whether this is the case (adding such error message is helpful in any case).
I have rebase now PR #1654 to the latest master. The PR changes should cause the master thread to terminate when one of the worker threads is terminated. That should resolve this issue if the the sending thread does terminate.
I am aware that PR #1654 usage of a counter to determine whether a worker thread was terminated requires careful testing. However, I was not able to find an easier mechanism to allow the master thread to know whether a worker thread was terminated. In general, I believe that such mechanism, to terminate the master thread when a worker thread terminats, is important.