io_uring close paths close the fd while a linked RECV can still resolve its number (the #657 fd-lifetime rule, not yet applied to close/hijack/shutdown)

[FumingPower3925]

Summary

The #657 fix (PR #681) stops the io_uring hand-off from releasing a connection's fd while an operation on it can still resolve the fd. The close paths keep the same shape. They are listed here as the follow-up the #657 design deferred (DECISION step 5). This is found by reading and not observed in any run so far.

• finishClose / finishCloseDetached submit the connection's cancels (cancelConnOps) and then unix.Close the fd at once. A linked RECV that has not been issued yet (still in the SQ ring, or chained behind a SEND) resolves the fd number when it is issued. By then a new accept may hold that number.
• hijackConn hands the fd to net.FileConn with only a pending cancel for the armed recv. A recv that completes before the cancel lands takes the hijacker's first bytes.
• shutdown closes connection fds before tearing down the ring.

On 5.10-5.18 kernels the cancels themselves fail (#682), which makes the close-path case far worse. That half is tracked there. This issue is about the ordering on kernels where the cancels work.

What a fix needs

• Apply the same rule the hand-off now follows: do not close() a connection's fd while any op on it can still resolve the fd. Either hold the close until the kernel owes nothing (kernelInflight == 0 and no recv armed), or make sure the RECV is issued before the fd can be reused.
• Measure it first with a join-key test in the style of adaptive: forced switches leave keep-alive connections behind — async conns miss the 1.2 s flap window in every run, and a one-worker io_uring strands sync conns until their requests fail #657's diagnosis: an unissued linked RECV at close time, a new connection that reuses the fd number, and the old RECV's data landing on the new connection.

Related: #657, #682, PR #681.

[FumingPower3925]

Measured: the close-path case reproduces deterministically (draft PR #781, from #715)

This issue's "measure it first" step is done for finishClose / finishCloseDetached, in the case where the RECV is still in the SQ ring at close.

Draft PR #781 is at head 37db222. Its tests build only under -tags=validation, need an opt-in variable, and are not in CI.

• The trial:
  1. The closing worker queues its cancels behind A's prepared, unsubmitted RECV.
  2. It calls unix.Close and parks.
  3. The sibling io_uring worker accepts a new connection B on the freed number.
  4. The closer's next submit issues A's RECV against B's socket.
• Arm A (the tree as it is): in 80 of 80 trials, B's whole request was stolen and B was never answered.
  • The stale CQE came from the closer, under A's (fd, generation), with res = len(B's request) and B's bytes.
  • stale_recv_data_closed went up by 1 in every trial.
  • Shapes: x86 and arm64, kernels 6.17 and 7.0, with and without -race.
• Control, one Submit before unix.Close: 0 of 80 stolen. The witness close_with_unsubmitted_recv went up by 1 in every trial of both arms.
• An independent skeptic upheld this and went further:
  • Under a synthetic load with no holds, 486 of 1,536,431 requests were lost this way. All 486 joined, by request id, to a stale exemplar.
  • With submit-before-close, 0 of 770,413 were lost.
  • A build without the validation tag lost requests too. Its stale_recv_data_closed equalled the lost count in each of 3 runs.

This is the leading candidate for #715: a fresh connection's request is read off its socket and never answered, then closed at the 10 s header deadline. The skeptic reproduced #715's socket row exactly. It is not yet shown that #715's field events are this mechanism; the #715 comment has the details.

For the fix (engine lane D-2)

• PR test(iouring): deterministic check of the #715 recv-theft window #781's arm A is the acceptance test.
  • The fix must make TestRecvTheft715ArmA pass (0 thefts, B answered) in every shape, and keep the control passing.
  • The fix PR should enable arm A in CI. Arm A needs two io_uring workers, which means memlock of at least 24 MiB. The unit job's 8 MiB gives one worker, so there is no sibling.
• The control arm's submit-before-close is one fix direction. The requirement is this issue's rule: no close() while any op can still resolve the number.
• Still in scope here, and not covered by PR test(iouring): deterministic check of the #715 recv-theft window #781:
  • A linked RECV chained behind a SEND. It is submitted, so the witness does not count it, but it is not yet issued. This comes from code reading only, not a run.
  • hijackConn and shutdown.
• The ENOBUFS re-arm that adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715's reading names needs the provided-buffer ring. That ring is off unless CELERIS_IOURING_MULTISHOT_RECV=1 (worker.go:995 and 2720), so the ENOBUFS re-arm is not reachable in the default config.
• Close sites at main 0cf0c52 (unchanged by the merges since 698bed6):
  • finishClose: cancelConnOps at worker.go:4022, then unix.Close at 4069, 4074 and 4083;
  • finishCloseDetached: cancelConnOps at 4138, then unix.Close at 4162, 4183 and 4192.

Evidence: evidence/celeris-715/hypothesis-a/ and evidence/celeris-715/hypothesis-a-skeptic/.