test(iouring): deterministic check of the #715 recv-theft window

[FumingPower3925]

Refs #715 #685

Summary

This PR adds a deterministic test of hypothesis (a) of #715 (the #685 class). The theft reproduces in every trial, and the control does not. A recv SQE sits unsubmitted in the closing worker's SQ ring when that worker closes the descriptor. A sibling worker accepts a fresh connection B under the freed number. The closing worker's next submit then issues the old recv against B's socket. That recv reads B's whole request: stale_recv_data_closed goes up by 1, and the recorded bytes are exactly B's request. B is never answered.

arm	what differs	prediction under (a)	result, all shapes
A (TestRecvTheft715ArmA)	nothing (the tree as it is)	B's request stolen, B unanswered	stolen and unanswered in 80 of 80 trials
control (TestRecvTheft715Control)	the close paths submit the ring before unix.Close	0 thefts	0 thefts, B answered in 80 of 80
C (TestRecvTheft715ArmC)	hypothesis (c): the unlock-to-Signal window in promoteConnToAsync and the async feed, widened to 2 ms on every hand-off	0 lost if that window loses no wakeup	0 lost of 19,200 requests, 12,800 widened hand-offs

Arm A fails on main on purpose. It asserts the property a #685 fix has to restore. It is not in CI; the fix PR (engine lane D-2) should enable it.

What this does and does not show

It shows:

• the mechanism is real on both arches, on kernel 6.17 (GitHub ubuntu-24.04 / ubuntu-24.04-arm) and on 7.0 (a linuxkit container), with and without -race;
• the stale recv reads the new connection's request byte for byte. Its CQE is res = 41 = len(B's request), it comes from the closing worker under the closed connection's (fd, generation), and its first bytes are GET /b HTTP/1.1\r\nHost: recv-theft-715\r\n\r\n;
• B then waits on an empty socket. The test stops reading after 2 s. Left alone, B waits until its header deadline (celeris's default is 10 s) and is then closed, which is the h2c hang-eof signature of adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715;
• the only difference between arm A and the control is one io_uring_enter before the close, which is the fix direction io_uring close paths close the fd while a linked RECV can still resolve its number (the #657 fd-lifetime rule, not yet applied to close/hijack/shutdown) #685 names.

It does not show:

• that adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715's events are this mechanism. Two holds force the one ordering the theft needs. First, the sibling's accept of the number comes after the close. Second, the closing worker's submit comes before the sibling submits B's own recv. Nothing here measures how often that ordering happens on its own.
• which production paths reach the precondition, and how often. The precondition is a close with the recv prepared and not yet submitted. The test reaches it through the engine's own async path: promoteConnToAsync re-arms, and the dispatch goroutine's asyncClosed is drained in the same iteration. It uses a handler that writes nothing on a Connection: close request. When a response is pending, closeConn defers the close to the SEND completion, and by then the recv has been submitted. That is the common case in the real server. The new witness counter close_with_unsubmitted_recv exists to count the precondition in a container leg; that is step 2 of the adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715 plan.

Arm C is a weak test of (c). The worker is the only feeder, so holding it makes the window longer but adds no second actor to race it. It rules out a lost wakeup in that window under keep-alive and pipelined load. It does not rule out (c) as a whole: the #364 revert, the hand-off claim, the h2c upgrade on the dispatch goroutine, and a response-side loss are all untouched.

How the trial works

engine/iouring/recv_theft_715_linux_test.go starts an async-handler engine with two io_uring workers, then:

1. It opens /dev/null until every descriptor hole is filled, and opens B's candidate client sockets. As a result, the number that A's close frees is the lowest free one.
2. A sends GET /close with Connection: close (an async route, handler writes nothing). The worker W1 that owns A:
   • bails from the inline parse, promotes A and re-arms A's recv (promoteConnToAsync);
   • waits at the gate until the dispatch goroutine has queued A (recvtheft.Options.PromoteGate, the one test-only step on this path);
   • drains the queue and closes A (finishCloseDetached). The witness finds A's recv SQE not yet consumed by the kernel, so W1 counts it and parks right after unix.Close (recvtheft.HoldAfterClose).
3. The test dials candidates for B and sends B's request on each. A candidate that hashes to W1's listener waits in W1's accept queue. The first one the sibling W2 accepts gets the freed number, and W2 parks right after it prepares B's recv, before it submits (recvtheft.AfterAccept).
4. The test releases W1, which submits, then W2, and reads B's response with a 2 s deadline.

The witness compares the SQ sequence number at which the recv SQE was placed with the kernel's SQ head. Every hold is bounded (10 s), and a disarmed trial releases both workers.

Tallies

Every arm is judged from the --- PASS/FAIL/SKIP lines and one RECVTHEFT715 ... result line per trial. Tally script: evidence/celeris-715/hypothesis-a/scripts/tally.py. The tallies ran at ad3ffb7. The two commits after it change only the production layout test. Its first version compared two offsets that kernelInflight's own alignment keeps 2 bytes apart, and failed in a laptop container.

shape	arch, kernel	processes x -count	arm A: FAIL, and stolen + unanswered	control: PASS, and stolen	arm C: PASS, and lost
GitHub-hosted, celeris CI shape (4 vCPU, -race, memlock unlimited)	x86, 6.17.0-1022-azure	5 x 4	20 of 20; 20 of 20	20 of 20; 0	20 of 20; 0 of 4,800
same run	arm64, 6.17.0-1022-azure	5 x 4	20 of 20; 20 of 20	20 of 20; 0	20 of 20; 0 of 4,800
laptop container, CI shape (--cpus 4, -race, memlock unlimited)	arm64, 7.0.12-linuxkit	1 x 20	20 of 20; 20 of 20	20 of 20; 0	20 of 20; 0 of 4,800
laptop container, unconstrained (8 CPUs, no -race, memlock unlimited)	arm64, 7.0.12-linuxkit	1 x 20	20 of 20; 20 of 20	20 of 20; 0	20 of 20; 0 of 4,800
total			80 of 80; 80 of 80	80 of 80; 0	80 of 80; 0 of 19,200

• In every trial of both arms (160 of 160), the gate fired and close_with_unsubmitted_recv went up by exactly 1: both arms reached the same precondition.
• stale_recv_data_closed went up by 1 in every arm A trial, and by 0 in every control trial.
• In every arm A trial the stale completion came from the closing worker, on the freed number, with res = 41 and B's request as its bytes. For example (x86 shard 1): {worker=1 fd=153 gen=3 res=41 head="GET /b HTTP/1.1\r\nHost: recv-theft-715\r\n\r\n"}, and B on worker 0 had no response head within 2 s.
• Not a result: at memlock 8 MiB (the CI unit job's shape) the engine starts one io_uring worker, so there is no sibling. With CELERIS_REQUIRE_IOURING_WORKERS=1 the tests fail that way (workers=1) instead of skipping.
• A smoke run before the tallies (unconstrained, -count=2) gave the same outcome and is not in the totals.

fd-number reuse. Every time the sibling accepted a connection during a hold, it got exactly the number the held close had freed: 160 of 160 sibling accepts. An attempt reached a sibling accept in 160 of 162 attempts. In the other 2 attempts, all six candidates hashed to the held worker's SO_REUSEPORT listener (p = 1/64 per attempt), and the test retried with a new A. 162 candidates queued on the held worker in total.

Commands

Laptop containers, one at a time, through the lane slot lock (evidence/celeris-715/hypothesis-a/scripts/run-shape.sh):

# unconstrained: 8 CPUs, memlock unlimited, no -race
docker run --rm --ulimit memlock=-1:-1 --security-opt seccomp=unconfined -v "$TREE":/src:ro -w /src \
  -e CELERIS_RECV_THEFT_715=1 -e CELERIS_REQUIRE_IOURING_WORKERS=1 golang:1.27 bash -c '
    go test -tags=validation -c -o /tmp/iu.test ./engine/iouring/ && cd engine/iouring &&
    for arm in ArmA Control ArmC; do /tmp/iu.test -test.v -test.count=20 -test.run "^TestRecvTheft715${arm}\$"; done'
# CI shape: the same with --cpus 4 and go test -race

GitHub-hosted runners, both arches, the celeris CI shape (-race, memlock raised as the iouring job raises it): probatorium celeris-stress.yml run 36351992181:

gh workflow run celeris-stress.yml --repo goceleris/probatorium --ref stress/runs -f target=github -f mode=stress \
  -f celeris_ref=ad3ffb707be6c8bbcb9d8c4e76e30245d07353de -f packages=./engine/iouring \
  -f run='^TestRecvTheft715(ArmA|Control|ArmC)$' -f count=4 -f shards=5 -f arches=both -f memlock=unlimited \
  -f race=true -f timeout=20m -f extra='-tags=validation CELERIS_RECV_THEFT_715=1 CELERIS_REQUIRE_IOURING_WORKERS=1'

That run's summary is red because arm A fails, as designed.

Changes

All of the instrumentation is validation-only, following the internal/zcwindow pattern from #693. recvtheft.Enabled is a false constant without -tags=validation, so every call site compiles away. A production engine/iouring test binary contains no recvtheft symbol, and neither recvUnsubmitted nor the two ring accessors.

• internal/recvtheft: the witness counter, the trial (close hold, sibling accept hold, promote gate, control switch, stale-recv exemplars) and the hypothesis (c) wake hold. The production build gets no-ops and a zero-size ArmSeq.
• engine/iouring:
  • noteRecvPlaced records each recv SQE's SQ sequence number;
  • finishClose / finishCloseDetached count a close with an unsubmitted recv, park when a trial is armed, and in the control arm submit before closing;
  • onAcceptedFD, promoteConnToAsync and the async feed get their hooks;
  • staleConnCQE hands a stale data completion's first bytes to the trial;
  • Ring gains sqPlaced / sqConsumed.
• connState.recvArmSeq is zero-size in production and not the last field. TestRecvTheftWitnessCompilesAway pins that. It is a production-build test, and it passed in this PR's Unit job. Sizeof(connState) is 600 bytes on linux/amd64 and linux/arm64, the same as main. Every field after recvArmSeq keeps its offset. Under -tags=validation the 4-byte field fits in existing padding, so the size is still 600.
• The three TestRecvTheft715* tests build only with -tags=validation, run only with CELERIS_RECV_THEFT_715=1, and need two io_uring workers (memlock of at least 24 MiB). Without the opt-in they skip. No CI step builds ./engine/iouring with the validation tag.

Checks run: go vet and golangci-lint v2.13 on GOOS=linux, amd64 and arm64, with and without -tags=validation.

For the #685 fix

• Enabling arm A turns this into its regression test. The fix should make arm A pass in every shape above, with the control unchanged.
• The close paths are not the only ones: io_uring close paths close the fd while a linked RECV can still resolve its number (the #657 fd-lifetime rule, not yet applied to close/hijack/shutdown) #685 also lists hijackConn and shutdown. The witness covers only finishClose / finishCloseDetached.
• A linked recv that is submitted but not yet issued (chained behind a SEND) also resolves the number late. This witness does not count it, because the ring head says it was submitted.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 34.21053% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/recvtheft/recvtheft_off.go	0.00%	10 Missing ⚠️
engine/iouring/handoff_loss.go	0.00%	6 Missing ⚠️
engine/iouring/worker.go	72.22%	5 Missing ⚠️
engine/iouring/ring.go	0.00%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[FumingPower3925]

Superseded by #793, the #685 fix

#793 carries this PR's three test commits (ad3ffb7, 64ba91a, 37db222) unchanged, as 7b661f1, abddf38 and 12b700c. On top of them it adds one test change: arm A's verdict now passes on a fixed tree.

• Why the verdict had to change. The fix keeps a closing connection's descriptor number allocated until its owed recv has ended. So the sibling worker can no longer be given that number, and every attempt would have read INCONCLUSIVE.
• What counts as a hit now. The sibling accepting B while the closer is parked, on whatever number. The trial also records:
  • reused: whether the sibling got A's number;
  • held_open: whether A's number was still open at the hold;
  • released: whether it was closed afterwards.
• The verdict is unchanged. B must be answered, and no stale recv may carry B's bytes.

Arm A on the tree without the fix:

• CI run 36365755720, at fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793's failing-first head 943d54b: FAIL 5 of 5 on x86 and on arm64.
• Laptop: FAIL 10 of 10 in the cp-revert control. In every trial, reused and stolen were true.

On #793's head, arm A passes:

• CI run 36366730324: 5 of 5 on each arch.
• Laptop: 40 of 40, with held_open true, reused false and released true in every trial.

#793 also enables the trials in CI (a new recv-theft job), and it adds the linked-recv and hijack forms.

I am leaving this PR open for whoever closes it.

[FumingPower3925]

Superseded by #793, merged as 3e7abba. #793 carries this PR's three test commits unchanged and changes how arm A is judged (the round-2 hit rule, theftHit), so that the arm passes on the fixed tree and still fails every trial on a tree without the fix. The trials run in the recv-theft CI job on both arches. #715 stays open until a no-fault container leg on the fix shows 0 events.