adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed)

[FumingPower3925]

Summary

On the adaptive engine, after its epoll→io_uring promote, a fresh connection's request is sometimes read off the socket by the engine and never answered:

• an h2c-churn GET / preamble gets no response, and the server closes the connection at the 10 s header deadline (hang-eof, read 10,001 ms);
• a WS-torture upgrade request gets no 101 within the walker's 2 s (handshake-fail-timeout).

Seven events so far, all adaptive, all 20-73 s after the promote. In the five that an in-stall socket snapshot caught, the server socket had received every byte of the request and had nothing left in its receive queue, and had sent no data (no bytes_sent, no data segment out; its one segment out is at most the ACK of the request). Nothing in the process was blocked. The mechanism is not attributed.

This is the bug record for the event class that #588 exists to capture. #588 stays the measurement issue; its comments hold the per-run detail.

The events

#	where	engine / event	write instant	after the switch	server socket in the in-stall ss
1	laptop container, no-fault leg of probatorium#412 run c7c3de4	adaptive, WS handshake-fail-timeout	2026-09-26 12:54:56.43Z	+22.4 s	not captured (before the in-stall dossier existed)
2	container, run 3a62131	adaptive, WS handshake-fail-timeout	17:09:43.13Z	+24.1 s	ESTAB, Recv-Q 0, bytes_received 156 = client bytes_sent 156, segs_out 1, at +1.0 s
3	container, run 3a62131	adaptive, WS handshake-fail-timeout	17:10:19.73Z	+60.7 s	ESTAB, Recv-Q 0, 156 = 156, segs_out 1, at +1.0 s
4	container, run ba0a3e0	adaptive, h2c hang-eof, read 10,001 ms	2026-09-27 05:04:14.27Z	+20.3 s	ESTAB, Recv-Q 0, bytes_received 126 = client bytes_sent 126, segs_out 1, at +3.0 s
5	container, run ba0a3e0	adaptive, WS handshake-fail-timeout	05:04:33.92Z	+39.9 s	ESTAB, Recv-Q 0, 156 = 156, segs_out 1, at +1.0 s
6	container, run ba0a3e0	adaptive, h2c hang-eof, read 10,001 ms	05:05:07.07Z	+73.1 s	ESTAB, Recv-Q 0, 126 = 126, segs_out 1, at +3.0 s
7	cluster, checkptr tier 36283577085 (msr1, cell-07)	adaptive/arm64, h2c hang-eof, read 10,004 ms	2026-09-27 01:55:36.60Z	+48.6 s	not captured (the tier ran main 6f5521c, without probatorium#412)

• Events 1-6 come from the no-fault legs of the lane-M container runs (auth_session_ratelimit, linux/arm64, nothing injected; the switch time is the refapp's engine switch completed line, 1 s resolution). Event 7 is detailed in Measure: capture enough at the next h2c_hang / ws_handshake_fail to root-cause it from the artifact, and try to force recurrence #588's 2026-09-27 04:27Z comment.
• Events 1-6 are each on the first promote of their cell: adaptive_switches is 0 before it and 1 through the event (HANDOFF-*.txt).
• Denominators over the five no-fault legs: adaptive 49,247 h2c and 14,927 WS attempts gave 2 and 4 failures. The plain io_uring, epoll and std cells of the same legs, about 13,500 h2c and 3,000 WS attempts each, gave 0. At adaptive's WS rate the io_uring cell expects about 0.8 WS events, so "adaptive only" is not established.

What the capture shows

The in-stall dossier (probatorium#412) is taken while the read still waits: at +3 s for h2c, at +1 s for WS. It takes ss -ltn / ss -tanpi first, then a goroutine dump from a debug side listener outside the engine.

• The request reached the process. The server socket is ESTAB and owned by the refapp. Its bytes_received equals the client's bytes_sent: 126 for the h2c preamble, 156 for the WS upgrade. Its Recv-Q is 0. It shows no bytes_sent and no data_segs_out, and segs_out:1: it sent no data, only (at most) the ACK of the request, and the client's bytes_acked is bytes_sent + 1 (the request and the SYN were acknowledged). lastrcv is 1.0 or 3.0 s, i.e. the request arrived when it was written.
• Every listener's accept queue is 0 (ss -ltn Recv-Q, every listener in every in-stall snapshot).
• Nothing is blocked. In every in-stall side-listener dump (events 2-6):
  • the two io_uring worker threads are runnable, inside the engine or in inline middleware;
  • the two epoll standby loops are parked in select;
  • no goroutine is in sync.(*Mutex).Lock;
  • 13-22 idle runAsyncHandler goroutines wait in sync.Cond.Wait (the refapp runs async handlers).
  • In the three ba0a3e0 dossiers the dump was fetched 45-50 ms after the trigger. The 3a62131 dossiers predate the fetch stamp.
• The hand-off was quiet (HANDOFF-ba0a3e0.txt, HANDOFF-3a62131.txt, 1 Hz series from 5 s before the switch to the last event):
  • In the ba0a3e0 cell, transplant_detached = transplant_adopted = 69, and from 05:04:05 on (9 s before the first event) sweep_passes stays at 310, residual_busy and standby_active_conns at 0, and handoff_in_flight is 0 throughout.
  • Every zero witness reads 0 over the window in both cells: recv_double_armed, recv_cqe_unaccounted, recv_sq_full, recv_stall_episodes, close_missing_conn_state, adopt_slot_occupied, stale_recv_data_transplanted, stale_recv_data_unattributed.
  • The only engine errors are send_peer_gone (the walkers abandoning connections, adaptive: is the post-promotion engine error rate proportionate to its load, or excessive? (ErrorCount was one undifferentiated counter) #645's class).
  • engine_stale_recv_data_closed = 105 for the ba0a3e0 cell. That says nothing specific: Measure: capture enough at the next h2c_hang / ws_handshake_fail to root-cause it from the artifact, and try to force recurrence #588's checkptr comment shows it is nonzero in cells with no loss.
• The h2c close is the header deadline. The refapp sets no ReadHeaderTimeout, so celeris's 10 s default applies. The read ended at 10,001 ms with an orderly EOF.

What that excludes and what is left

Excluded by the snapshot, for events 2-6:

• (b) an unarmed recv: the bytes would still be in the Recv-Q.
• A connection never accepted: it would be ESTAB with no owning process, and an accept queue would be non-zero.
• A handler or loop stall on a lock: the dump shows nothing blocked.

Also ruled out:

• io_uring: a parked worker keeps a stale cachedNow; if the park outlasted ReadTimeout, conns it adopts or accepts on waking get a past timestamp and the first checkTimeouts closes them (adaptive: 6/6 promotes 31 s after a demote, 0/8 at 18 s; A/B not run) #713 (stale cachedNow after a park). Its mechanism needs a demote before the promote. It predicts closes within ~100 ms of the promote and none at a first promote. These events come 20-73 s after a first promote, and the connection was held for 10 s, not closed early.

Left, and not separable from this artifact:

• (a) A stale completion on a reused descriptor number consumed the request: a RECV left by a closed connection, io_uring close paths close the fd while a linked RECV can still resolve its number (the #657 fd-lifetime rule, not yet applied to close/hijack/shutdown) #685's class. Here the reads come tens of seconds after the promote, on fresh connections.
• (c) The connection's own recv read the bytes, and they never reached the parser or the async handler. That is a lost hand-off or wakeup between handleRecv and the connection's runAsyncHandler.

What would decide it

A per-connection record in the engine, keyed by descriptor and peer port and joined to the walker's local_addr, would separate (a) from (c): bytes received by a recv CQE, bytes handed to the parser or handler, and whether an async handler goroutine was started for the connection. #588's checkptr comment proposes the same kind of bounded per-read exemplar. The in-stall capture already takes a dump inside the stall. It cannot join a goroutine to a descriptor.

Evidence

evidence/measurements-585-587-588/lane-20260926/round3/588/:

• NATURAL-EVENTS.txt, from natural-events-588.py over the five no-fault legs: rows 1-6, the denominators (DENOMINATORS), and the check that all five in-read captures show the same socket state (SIGNATURE, 5/5).
• natural/NATURAL-EVENT.txt, from natural/natural-event-588.py over the ba0a3e0 cell: events 4-6 with both socket rows, the dump and the listeners, and the same check with the client's bytes_acked and the dump's lock-wait frames added (3/3).
• HANDOFF-{c7c3de4,3a62131,ba0a3e0}.txt, from handoff-window-588.py: the series window (with the instant each counter goes flat) and the dump states.
• Row 7 is from Measure: capture enough at the next h2c_hang / ws_handshake_fail to root-cause it from the artifact, and try to force recurrence #588's checkptr comment. The dossiers are under lane-20260926/588/20260926T*/nofault/ and lane-20260926/round2/588/20260927T044910Z-ba0a3e0/nofault/.

Refs #588, #685, #713, goceleris/probatorium#412.

---

Edited 2026-09-27 (lane M, probatorium#412 review round 3): the server's one outgoing segment is described as at most the ACK of the request (the client's bytes_acked = bytes_sent + 1), not as the SYN-ACK; the hand-off counters are flat from 05:04:05, not 05:04:10; the evidence list names the SIGNATURE checks the scripts now print. No conclusion changed.

[FumingPower3925]

Cluster stall-capture nightly 36338747093: zero #715-class events

No #715-class event in this run, so the capture did not fire. The mechanism is still not attributed. The run is still a result: its denominators, and a correction to the cluster history below. The cluster history adds two earlier cluster events that this issue does not list.

The run

• goceleris/probatorium matrix-nightly-tier.yml run 36338747093:
  • head f1acc119, stall_capture=true, conclusion success, 17:55:48Z to 19:14:46Z;
  • it builds the refapps' pinned celeris 9f4d89b171db, not celeris main.
• The capture is run-wide. All 64 cells ran with it: 2 arches x 8 refapps x 4 engines.
• It is a valid denominator:
  • ValidateGate PASS over 64 cells with VALIDATE_GATE_EXPECT_ADAPTIVE_SWITCH: 1 (job log lines 1275 and 1290).
  • ValidateDiff strict OK (line 1303).
  • adaptive_switches is 1 in all 16 adaptive cells. Each promote came about 30 s after ready, and each cell then ran about 81 s (1,295 s in total). That covers the +20 to +73 s offsets of events 1-7.
  • Hosts had been up 18 days, with oom_kill 0.
  • Bench run 36340530922 bootstrapped at 19:14:48Z, after this run's teardown ended at 19:14:46Z.

The events: 0 in 64 cells

arch (each)	engine	cells	h2c fires	h2c read legs	h2c hangs	WS handshakes	WS handshake failures
amd64, arm64	adaptive	8	72,512	48,640	0	2,745	0
amd64, arm64	epoll / iouring / std	8 each	19,776 each	13,616 each	0	549 each	0
total		64	263,680	178,976	0	8,784	0

• Adaptive's h2c fires include 23,872 intentional RSTs per arch, which have no read leg.
• Only the auth_session_ratelimit cells run WS.
• The slow-read rings (h2c_slow_reads, ws_slow_reads) are empty in every cell.
• The other walkers' failure counters are 0 too: requests_error, adv_hang_until_timeout, sse_handshake_fail, ws_echo_handshake_fail/timeout, rfc_read_fail.

The capture's liveness

• It did not fire.
  • The artifact has 0 incidents/ directories.
  • All 32 cell directories on each host have no subdirectory.
  • Both rescued tarballs hold 0 incident entries.
• It could not have fired, armed or not.
  • Every read leg the capture watches is also recorded in the always-on read-leg histogram (validation/h2c.go:329-359 and ws.go:311-355 at f1acc119).
  • All 178,976 h2c and 8,784 WS read legs finished in under 100 ms.
  • A trigger needs a read in flight for 3 s (h2c) or 1 s (WS) (walker_capture.go:57-58).
  • So the zero does not depend on arming.
• Arming in this run is shown only up to the job environment.
  • Job log line 794 shows PROBATORIUM_STALL_CAPTURE: 1 in the mage Validate step.
  • The next hop, stallCaptureExtraVars (mage_validate.go:249) to ansible/validate.yml:258, is not logged.
  • No artifact file records the side listener's debug addr= banner: liveness.go:479 reads it and does not keep it.
• The capture has worked on the cluster. Fault-control run 36325737189 (same day, head 6c3d7f1):
  • It holds 77 dossiers, each with ss.txt and goroutine-stacks.txt. Among them are 22 I-H2C-STALL and 28 I-WS-STALL in-stall dossiers.
  • Measure: capture enough at the next h2c_hang / ws_handshake_fail to root-cause it from the artifact, and try to force recurrence #588 closed on its ValidateFaultControl PASS.
  • That run armed the capture through PROBATORIUM_REFAPP_FAULT. stallCaptureEnabled (validation/liveness.go:126-127) accepts either variable.
  • git diff 6c3d7f1 f1acc119 is empty for validation/, mage_validate.go, ansible/validate.yml and the workflow.
  • Only the stall_capture knob's own hop is unproven.

Correction: two earlier cluster events of this class

Two earlier cluster nightlies each had one event of this class, going by the client-side record. That is the same standard this issue applies to event 7. Neither has a socket snapshot: their dossiers were taken after the failure and hold no ss.txt.

run	celeris	cell	event	write instant (ring ts minus read_ms)	after engine switch completed
34876253223 (2026-09-14)	3b6184a	msr1 (arm64) cell-07 auth_session_ratelimit-adaptive, adaptive_switches 1	h2c hang-eof, read 10,006 ms, n_read 0, local 127.0.0.1:9720	~18:04:03.8Z	~+59 s (switch 18:03:05Z)
34918161309 (2026-09-15)	7a651c3	msa2-server (amd64) cell-07 auth_session_ratelimit-adaptive, adaptive_switches 1	WS handshake-fail-timeout, read 2,000 ms, n_read 0, local 127.0.0.1:41172	~02:00:01.0Z	~+8 s (switch 01:59:53Z)

Caveats:

• 34876253223: the same cell recorded an I-CONN-2 drift of 2 at 18:03:59Z (accepted 9333 - closed 9234 - active 97). That is celeris#624's lost-transplant class, since fixed, so this event may be confounded.
• 34918161309: it is at +8 s, outside the +20 to +73 s range of events 1-7. It is also the only amd64 event so far.

Rates, auth_session_ratelimit/adaptive only. The cluster figures span 14 runs (nightly, checkptr and race tiers) at celeris pins 3b6184a to 9f4d89b.

	cells	events	h2c	WS
cluster	28	3	2 in 228,986 fires	1 in 69,692 handshakes
container, lane M's 7 no-fault legs	7	6	2 in 69,025	4 in 20,917

• Per cell: the container rate is 0.86 events per cell, so the cluster's 28 cells should have had 24.0 events. It had 3, P(3 or fewer) = 9.9e-8. The cluster is about 8x rarer per cell.
• WS alone: 13.3 expected, 1 seen, P = 2.3e-5.
• h2c alone: 6.6 expected, 2 seen, P = 0.039.
• This nightly by itself (2 such cells) expected 1.71 events, so P(0) = 0.18. On its own it is uninformative.

Mechanism: not attributed

Why it is not attributed:

• This run has 0 events and 0 dossiers, so it holds no timeline, socket row, dump or join.
• The two older cluster events have no ss.txt.
• The only joins of a server socket to the walker's local_addr remain container events 2-6.

Code reading, not shown to be reached in any run: a sharper form of (a). Celeris 9f4d89b, engine/iouring:

1. prepareRecv (worker.go:4516-4531) only fills a RECV SQE with the fd number (fixed files are off), sets recvArmed and does kernelInflight++. It does not submit.
2. The non-SQPOLL loop submits only at worker.go:1173-1201 (the refapp logs sqpoll=false). CQE dispatch (1214-1294), drainDetachQueue (1365) and the dirty flush (1381-1430) all run before that next submit.
3. finishClose queues its ASYNC_CANCEL behind the unsubmitted RECV (cancelConnOps 3655-3661, called at 3874-3877). getCancelSQE submits only when the SQ is full (3694-3703). finishClose then calls unix.Close at 3922, 3927 and 3936. finishCloseDetached does the same: cancel at 3991-3993, close at 4015, 4036 and 4045.
4. Another thread's accept may be given the freed number before this worker's next submit. In that case the RECV resolves the number to the new connection's socket, can read its already-written request, and completes under the old (fd, generation).
   • staleConnCQE counts it as stale_recv_data_closed (worker.go:1602-1604, then handoff_loss.go:163-176).
   • The new connection's own recv then waits on an empty socket: Recv-Q 0, bytes_received equal to the client's bytes_sent, nothing sent. That is the signature of events 2-6.
5. The thief has to be another thread. The high tier sets DEFER_TASKRUN|SINGLE_ISSUER (tier.go:113 and 194). So this worker's own multishot accept cannot allocate the number before its next io_uring_enter, which is also when the RECV is submitted. The sibling io_uring worker is the likeliest candidate.
6. handoff_loss.go:35-37 already names this window. The KNOWN RESIDUAL note at worker.go:1584-1586 considers only the same worker's re-accept.
7. Two paths can queue a RECV and close the connection in the same iteration:
   • promoteConnToAsync's re-arm (4079-4083), followed by drainDetachQueue's closeConn when asyncClosed is set (4597-4598);
   • the ENOBUFS re-arm (2580-2588), followed by a close later in the same batch.
8. The counter does not discriminate. stale_recv_data_closed is 61 (arm64) and 69 (amd64) in this run's two auth_session_ratelimit/adaptive cells, and neither cell had an event.

Not excluded:

• (c): handleRecv 2515-3065, the promoteConnToAsync unlock-then-Signal at 4068-4073, and runAsyncHandler 4102-4361. This run says nothing about them.
• A response-side loss: a response built in cs.writeBuf but never flushed, through a lost detachQueue enqueue or a stranded dirty-list entry. It fits the WS socket signature equally well (Recv-Q 0, request fully received, nothing sent). For h2c, the close at exactly the 10 s header deadline argues against it if a parsed request clears the header timer. That is not verified.

Today's merges (celeris main 698bed6). These function bodies are byte-identical to 9f4d89b; only their line numbers moved:

• finishClose, cancelConnOps, closeConn, finishCloseDetached
• staleConnCQE, noteClosedInflight, prepareRecv
• handleRecv, handleAccept, runAsyncHandler, promoteConnToAsync
• drainDetachQueue, getCancelSQE, flushSendLink, armHeaderTimer
• noteStaleRecvData, attachAdoptedFD

run(), which holds the submit, changed in only three places: the listener handling from #674 (the TCP_DEFER_ACCEPT flag and the pause/linger branch), the accept re-arm call, and a validation-only zcwindow.Hold() from #693 that compiles away. Its order is unchanged: CQE dispatch, then drainDetachQueue, then dirty flush, then the next iteration's submit. Nothing merged today touches the candidate window.

Not proven

• The mechanism. (a) comes from code reading only; no run shows the path taken. (c) and a response-side loss are not excluded.
• That this run's stall_capture knob reached the validator. The hop is not logged. The zero event count rests on the histograms, not on arming.
• That the two older cluster events share the socket signature. They have no ss.txt, and the 34876253223 event may be confounded by the I-CONN-2 drift.
• "Adaptive only".
• Whether adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715 persists on 698bed6. That is untested.

Next step

1. Stop using cluster nightlies as the event source for adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715.
   • The cluster gives about 0.1 events per auth_session_ratelimit/adaptive cell (3 in 28), with 2 such cells per nightly.
   • A lane-M no-fault container leg gives about 0.86 events per leg (6 in 7).
2. Separate (a) from (c) with one container leg. First add:

   • In celeris:

     • a witness counter close_with_unsubmitted_recv: finishClose or finishCloseDetached reached while cs.recvArmed and the RECV was prepared after the worker's last submit. This needs a per-worker submit sequence number and a per-connection recvArmSeq.
     • a bounded ring of stale recv CQEs that carry data: ns time, fd, generation, res, first 16 bytes.

   • In probatorium: engine_stale_recv_data_closed in properties_series.csv (only the transplanted and unattributed columns are there now), and the exemplars in the cell document.

   • Reading the leg: under (a), each hang pairs with a stale exemplar within milliseconds of its write instant, and the witness counter is above 0. That exemplar:

     • is on the fd that ss shows for the hung socket;
     • has res equal to that connection's client bytes_sent;
     • starts with the walker's request line. For h2c that is GET / HTTP/1.1 (the Upgrade: h2c preamble, validation/h2c.go:399-413), not the PRI * preface. For WS it is the upgrade GET.

     Match on the per-connection bytes_sent, not on fixed lengths: the 126 and 156 bytes seen in the container depend on the Host header.

     Under (c), there is no such exemplar.

3. A deterministic celeris test in a validation build, using the test(iouring): judge the SEND_ZC first-completion window under -race, with its detachMu mutant (celeris#587) #693 internal/zcwindow hold pattern:
   • Arm A: hold the worker right after unix.Close in finishClose whenever a RECV was prepared since its last submit. Meanwhile, dial the sibling worker and write a request. Prediction under (a): the request is stolen every time, stale_recv_data_closed grows by its length, and the new connection hangs.
   • Control arm: submit before the close. Predicted: 0 thefts.
   • Arm C: widen 4068-4073 to test (c).
   • A promote-boundary injection would test the wrong window: events 1-7 came 20-73 s after the promote.
4. Fix direction, only if arm A reproduces and its control does not: the close paths must not release an fd number while a RECV naming it is prepared but unsubmitted, for example by submitting before unix.Close as the control arm does.
5. Then settle "adaptive only" with a container leg on engine iouring, pinned to 2 workers at the adaptive walker load. The in-stall dumps of events 2-6 show two io_uring worker threads.
6. Probatorium apparatus gaps to file:
   • The stall capture leaves no marker when nothing triggers. Proposal: stall_capture {enabled, debug_addr, thresholds, triggers, dossiers} in tier1_tally.json.
   • properties_series.csv lacks engine_stale_recv_data_closed.

Evidence

evidence/celeris-715/:

• nightly-36338747093/:
  • RATES-cluster-runs.txt, CAPTURE-LIVENESS.txt, ADAPTIVE-EXPOSURE.txt, CODE-EXCERPTS.txt, CODE-PATH-DIFF.txt;
  • RATE-COMPARE-28.txt, the corrected 28-cell comparison. The 18-cell figures in RATE-COMPARE.txt are superseded.
  • PRIOR-CLUSTER-EVENTS.txt: the two older events, their switch lines and dossier contents, and the 36325737189 dossier count.
• skeptic-prior-nightlies/<run>/: the downloaded artifacts of 34876253223, 34918161309, 34983261615, 34893230678, 34864823296, 34818888908 (no adaptive traffic) and 36325737189.
• scripts/: rate_compare_28.py, prior_cluster_events.{sh,py}, and the scripts that produced the files above.

[FumingPower3925]

Hypothesis (a) reproduces, and it produces this issue's signature (draft PR #781)

(a) is a real mechanism. A closing worker's recv, prepared but not yet submitted, reads the request of a new connection that a sibling worker accepted on the freed fd number. That new connection is then never answered. It is not yet shown that events 1-7 are this mechanism.

The deterministic test: draft PR #781

PR #781 is at head 37db222. Its tests build only under -tags=validation, need an opt-in variable, and are not in CI.

• Arm A (the tree as it is). The closing worker parks after unix.Close with A's recv prepared and unsubmitted. The sibling accepts B on the freed number. The closer's next submit then issues A's recv against B's socket.
  • 80 of 80 trials: B's whole request was stolen, and B was never answered.
  • The stale completion came from the closer, under A's (fd, generation), with res = 41 = len(B's request) and B's bytes.
  • stale_recv_data_closed went up by 1 in every trial.
• Control: one Submit before unix.Close, with both holds unchanged. B was answered in 80 of 80 trials, with 0 thefts. The new witness close_with_unsubmitted_recv went up by 1 in every trial of both arms, so both arms reached the same precondition.
• Arm C: the unlock-to-Signal window in promoteConnToAsync and the async feed, widened to 2 ms. 0 of 19,200 requests were lost. This is a weak test of (c), because the widening adds no competing actor.
• Shapes, 20 trials per arm in each:
  • GitHub-hosted x86 and arm64 (kernel 6.17, 4 vCPU, -race; probatorium celeris-stress run 36351992181);
  • a laptop container on arm64 (kernel 7.0), with and without -race.

Independent skeptic: upheld, and stronger than the PR claims

The skeptic re-ran the arms on arm64 in both container shapes (arm A FAIL 20 of 20, control PASS 20 of 20, arm C PASS 20 of 20) and re-tallied the GitHub artifacts. Additional results:

• This issue's socket row, reproduced (5 of 5):
  • B's server socket is ESTAB, with Recv-Q 0.
  • bytes_received 41 equals the client's bytes_sent.
  • No data was sent, segs_out is 1, and the client's bytes_acked is sent + 1.
  • The server ends B with an orderly EOF at 10.001 s, which is the header deadline. That matches events 4 and 6 (hang-eof, 10,001 ms).
• No holds and no gate (synthetic load: an async Connection: close route, 40 client goroutines, 2 workers):
  • 486 of 1,536,431 requests went unanswered. All 486 joined, by a request id, to a stale exemplar that carried that id.
  • With submit-before-close, 0 of 770,413 went unanswered.
• A build without the validation tag loses requests the same way. There, stale_recv_data_closed equals the lost-request count exactly in each of 3 runs: 120, 186 and 180.
• Order matters: swapping only the release order, so that the sibling submits B's recv first, got B answered 5 of 5 times.
• The precondition occurs on its own: 2 to 9% of async Connection: close closes had an unsubmitted recv.

Not proven

• That events 1-7 are (a). (c) and a response-side loss leave the same socket row. The discriminator is still step 2 of the plan above: in a no-fault container leg, look for close_with_unsubmitted_recv > 0 and a stale exemplar joined to the hung socket's fd and bytes_sent.
• The field rate. The load above is synthetic.
• x86, independently. The skeptic's own runs are arm64 only. The x86 result rests on the GitHub run.

What follows

• io_uring close paths close the fd while a linked RECV can still resolve its number (the #657 fd-lifetime rule, not yet applied to close/hijack/shutdown) #685's fix (engine lane D-2) is expected to fix adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715. The control arm shows the fix direction: never release an fd number while a recv naming it is prepared and unsubmitted.
• PR test(iouring): deterministic check of the #715 recv-theft window #781's arm A is that fix's acceptance test. The fix must make TestRecvTheft715ArmA pass (0 thefts, B answered) in every shape, and enable it in CI.
• If adaptive: after the epoll→io_uring promote, a fresh connection's request is read off its socket and never answered (h2c closed at the 10 s header deadline, WS handshake timeout; 7 events, mechanism not attributed) #715-class events still occur after the fix covers every close path, (a) is excluded for them. "Every close path" includes a linked recv chained behind a SEND, which the witness does not count. What would remain is (c) and a response-side loss.
• A lead: in the synthetic runs, every stale completion that carried data was a lost request. So the nonzero stale_recv_data_closed of event-free cells (61 and 69 in nightly 36338747093) may be thefts that no walker counts. properties_series.csv has no engine_stale_recv_data_closed column, so a celeris#715-class request theft cannot be placed in time probatorium#450 adds the series column that can place them in time. The stall capture leaves no marker when nothing triggers: record stall_capture {enabled, debug_addr, thresholds, triggers, dossiers} in tier1_tally.json probatorium#449 covers the capture's missing liveness marker.

The window is unchanged on main 0cf0c52:

• finishClose queues its cancels (worker.go:4022) and closes at 4069, 4074 and 4083.
• finishCloseDetached queues its cancels (4138) and closes at 4162, 4183 and 4192.
• promoteConnToAsync re-arms at 4231.

The #722 and #758 merges since 698bed6 touch neither these paths nor the submit order in run().

Evidence:

• evidence/celeris-715/hypothesis-a/: TALLY.txt, logs, the GitHub run's artifacts, scripts.
• evidence/celeris-715/hypothesis-a-skeptic/: TALLY.txt, 9 logs, and the extras patch on 37db222.