v1.6.0 polish: io_uring engine

[FumingPower3925]

Consolidated on 2026-10-02 from the per-PR 'Follow-ups from #N' bundles, by maintainer decision. From now on, review nits are fixed inside the PR before it merges, so no new bundles are filed. Each line names the bundle and item it came from; the bundle's thread keeps the full context. Everything here stays in v1.6.0.

• Cover the arming half of fix(iouring): read an async conn's header deadline before its dispatch goroutine runs, under detachMu (celeris#722) #743 (mutant Z12 survives engine/iouring): a unit test that parks the goroutine with HeaderDeadlineNs > 0 and asserts the next feed sets headerTimerArmed, and a -race test of handleHeaderTimer's early-fire branch (no test calls handleHeaderTimer). (from Follow-ups from #743: epoll checkTimeouts h1State race, arm-path test coverage, a contended bench for the TryLock feed #762 2)
• TestAsyncH2CUpgradeOnFeedLeavesH1StateToTheGoroutine (async_h2c_h1state_race_linux_test.go:151) separates its two writes with a 100 ms sleep; send the second write only after an observable signal that the first recv was promoted and the goroutine is parked, or assert the second write arrived through the feed, so a coalesced run cannot pass vacuously. (from Follow-ups from #743: epoll checkTimeouts h1State race, arm-path test coverage, a contended bench for the TryLock feed #762 6)
• TestDriverShutdownWaitsForAHandedOffClose (driver_linger_close_linux_test.go, time.Sleep(200ms) now at line 392) syncs on a sleep; add a test-only hook fired when waitDriverCloses starts, wait for it with a deadline before drain(), and rerun the M2 control. (from Follow-ups from #744: parallel driver closes at shutdown, onClose no longer tied to the fd's map removal #763 4)
• Add a deterministic test for runAsyncHandler's re-check exit after taking detachMu (engine/iouring/worker.go): hold detachMu, start the handler with a non-empty asyncInBuf, call closeConn so it sets closeOwed, release, and assert the conn is enqueued and closed exactly once, so mutant Y1 (dropping the owed enqueue) fails. (from Follow-ups from #745: a test for the re-check exit's owed hand-back, closeErr on the shutdown path #764 1)
• TestIouringCloseStillWaitsForABoundedHolder (engine/iouring/async_handler_stall_linux_test.go) treats 'not done after 200 ms' as proof the close waits on the holder; replace the timer with a sync point tied to closeConn's failed TryLock (test hook or polling the mutex until a waiter is queued) so mutants M7/M8 cannot survive on a loaded runner. (from Follow-ups from #745: a test for the re-check exit's owed hand-back, closeErr on the shutdown path #764 4)
• The end-to-end io_uring: four worker-thread sites block on cs.detachMu while an async handler holds it, parking the whole ring (io_uring twin of #669) #704 stall tests synchronise with time.Sleep(50ms) before trigger(slow) and judge a 300 ms wall-clock budget (stallBudget704); have /slow signal a channel on entry, and replace or back the budget with a structural assertion or an opt-in env var. (from Follow-ups from #745: a test for the re-check exit's owed hand-back, closeErr on the shutdown path #764 5)
• queuePendingReleaseDetached's comment (engine/iouring/worker.go, before :4427 on main) lists only detached close paths; add hijack and its reason (the request's views must outlive the handler). (from Follow-ups from #773: an AsyncHandlers arm after #774, io_uring multishot strings read before Hijack, the copy on std, the engine-side cost, doc and witness nits #785 5)
• clientSeesClose in park_close_fin_test.go treats any non-timeout read error (an RST) as the close; accept only io.EOF (or log which arrived and assert EOF), make the single-connection arms wait for BytesRead like the many-connection arms, and re-run the pre-fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 arm to confirm it still fails (CodeRabbit on test(iouring): regression arms for #712 (fixed by #793) #767). (from Follow-ups from #767: a named CI interlock for the #712 tests, and the bare-metal failing-first #795 4)
• Lengthen the setup deadlines in park_close_fin_test.go (single header-timer arm 400 ms to ~1 s, 64-connection arms 1 s to ~3 s) and the dependent close-observation bounds and parkWait712 waits, then re-confirm fail-before/pass-after (CodeRabbit on test(iouring): regression arms for #712 (fixed by #793) #767). (from Follow-ups from #767: a named CI interlock for the #712 tests, and the bare-metal failing-first #795 5)
• On 5.19+, requireAsyncCancelFlags returns nil for asyncCancelUnexpected (fails open). Decide and either refuse on an unexpected answer at any version or document why failing open is preferred, and align the gate-test rows {asyncCancelUnexpected,5,19,false}/{..,7,0,false}. No kernel is known to give such an answer. (from Follow-ups from #792: io_uring 5.19 floor — backport probe covers 1 of 4 cancel forms, floor test keyed on version, no CI seam for adaptive/Server, stale comments, SSE flake, docs pages #796 4)
• classifyAsyncCancelProbe's reason hard-codes 'the kernel predates Linux 5.19' (engine/iouring/probe.go:665 on main), so a 5.19+ rejection error contradicts itself; drop the version clause. (from Follow-ups from #792: io_uring 5.19 floor — backport probe covers 1 of 4 cancel forms, floor test keyed on version, no CI seam for adaptive/Server, stale comments, SSE flake, docs pages #796 5)
• Fix stale or overstated comments/log text from fix(iouring): require Linux 5.19; refuse io_uring on a kernel that rejects the IORING_ASYNC_CANCEL flags (celeris#682) #792: (a) engine.go 'peer gets no FIN' scoped to H1 fast-close; (b) the 'io_uring not available' prefix matters only to test skip helpers, not adaptive's fallback; (c) logAsyncCancelProbe messages (engine.go:301,304 on main) should say New refuses io_uring below 5.19; (d) adaptive/engine.go:136 'cannot disagree' is no longer true; (f) probe/probe.go:121 '5.10-5.18 Base-tier kernel runs epoll' is adaptive-only; (g) engine/iouring/doc.go should mention an unexpected answer below 5.19 also refuses. (from Follow-ups from #792: io_uring 5.19 floor — backport probe covers 1 of 4 cancel forms, floor test keyed on version, no CI seam for adaptive/Server, stale comments, SSE flake, docs pages #796 6)
• probe/diagnostic.go prints io_uring_tier=base and the io_uring feature block on 5.10-5.18 although the engine refuses io_uring there; annotate that the engine needs 5.19+ (log wording only). (from Follow-ups from #792: io_uring 5.19 floor — backport probe covers 1 of 4 cancel forms, floor test keyed on version, no CI seam for adaptive/Server, stale comments, SSE flake, docs pages #796 7)
• runCancelForm682 (engine/iouring/async_cancel_floor_test.go) can close the ring while its recv is still armed if an earlier t.Fatalf runs the ring.Close defer; register a deferred drain (write a byte, reap CQ until recvUD completes, with a deadline, keep buf alive) before the ring.Close defer (test only). (from Follow-ups from #792: io_uring 5.19 floor — backport probe covers 1 of 4 cancel forms, floor test keyed on version, no CI seam for adaptive/Server, stale comments, SSE flake, docs pages #796 11b)
• Now that test(iouring): regression arms for #712 (fixed by #793) #767 (5a085e4) and fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793 are both on main, run test(iouring): regression arms for #712 (fixed by #793) #767's parked-worker FIN tests against mutant M2 (closeFDOwed==0 removed from the park predicate, worker.go) and record that they fail, so the park gate is pinned; the named interlock is Follow-ups from #767: a named CI interlock for the #712 tests, and the bare-metal failing-first #795 item 1. (from Follow-ups from #793: a SEND_ZC notification counted as an owed op, the unpinned park gate and shutdown drain, and two counter/exit nits #798 2)
• Add a shutdown trial that holds the worker between a linked recv's SEND CQE and shutdown() and fails with mutant M1 (endOwedOpsAtShutdown drain disabled), plus one for the multishot ENOBUFS re-arm; if no kernel issues such work after the descriptors close, remove the drain rather than keep its unmeasured cost (up to 250 ms per worker, CQEs dropped). (from Follow-ups from #793: a SEND_ZC notification counted as an owed op, the unpinned park gate and shutdown drain, and two counter/exit nits #798 3)
• engine/iouring/fd_lifetime_close_test.go judges its ZC tests by wall clock (zcShutdownBound 100 ms vs a 250 ms regression; zcReleaseBound 200 ms, both still on main); assert fdOps(cs)==0 before endOwedOpsAtShutdown, count the drain's passes and assert one, widen zcReleaseBound to 2 s, and keep both failing on 0f36096 (CodeRabbit on fix(iouring): never release a descriptor number while an op can still resolve it: close paths, hijack, shutdown (celeris#685) #793). (from Follow-ups from #793: a SEND_ZC notification counted as an owed op, the unpinned park gate and shutdown drain, and two counter/exit nits #798 6)
• hijackConn submits its cancels before handing over the socket only when !w.sqpoll, and endOwedOpsAtShutdown is skipped under SQPOLL; unreachable today (every tier's SQPollIdle is 0; Dormant SQPOLL tier: latent SQ-tail publish race (GetSQE) + architecturally unviable — remove or guard #377 closed). Add a comment/guard at both sites saying enabling SQPOLL must make hijackConn wait for the cancel CQE and give the drain an SQPOLL form. No behaviour change. (from Follow-ups from #793: a SEND_ZC notification counted as an owed op, the unpinned park gate and shutdown drain, and two counter/exit nits #798 8)
• TestIouringSendCompletionDuringASlowAsyncHandlerDoesNotStallItsWorker has no witness that a completion was held; add one (EngineMetrics.RingBytes grew across /slow, or a test hook counting held completions > 0) and make cluster row 52's PASS require it. The interim witness big_ok=false no longer works now that fix(iouring): an async handler's direct write waits for a ring SEND of the conn's earlier bytes (celeris#751) #800 (65d1755) is on main. (from Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 1)
• Add an arm that holds a completion and lets the dispatch goroutine exit through the park claim (with a drain set) and through the h2c exit, asserting drainDetachQueue's entry applies it, so the replay stays at the top of every hand-back entry. (from Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 2)
• Add a test that holds a completion, then releases and reuses the connState (or the fd number), and asserts nothing is applied to the new owner, so releaseConnState clearing heldSends and replayHeldSends' generation check are pinned (mutants Mrelease, Mgen survive today). (from Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 3)
• TestIouringSendCompletionStillWaitsForABoundedHolder does not prove handleSend reached holdOrLockSend's TryLock within its 200 ms window; add a test-only hook fired after a failed TryLock and wait for it before the window (CodeRabbit on fix(iouring): stop a ring SEND completion parking the worker on a running async handler's detachMu (celeris#750) #801). (from Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 6)
• The end-to-end test orders /big and /slow with 150 ms and 50 ms sleeps (send_completion_stall_linux_test.go:490,494 on main); have the /slow handler signal entry on a channel and wait for it before reading /big, together with Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 item 1's witness (CodeRabbit on fix(iouring): stop a ring SEND completion parking the worker on a running async handler's detachMu (celeris#750) #801). (from Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814 7)
• TestAsyncResponseWaitsForAnInFlightRingSend reaches only ringSendOutstanding's cs.sending clause; commit the reviewer's SQ-full probe (zz_review_probe_751_sqfull) as a third arm for the sendBuf clause and add a SEND_ZC arm (first CQE with F_MORE, then the notification) for zcNotifPending. (from Follow-ups from #800: test ringSendOutstanding's sendBuf and zcNotifPending clauses, one copy of the condition, the comment's locking claim, the over-cap drop on the async path #815 1)
• The guarded writeFn keeps its own copy of the condition (worker.go:2761 on main: !cs.fixedFile && !cs.sending && !cs.zcNotifPending && ...); use !ringSendOutstanding(cs) so there is one source. (from Follow-ups from #800: test ringSendOutstanding's sendBuf and zcNotifPending clauses, one copy of the condition, the comment's locking claim, the over-cap drop on the async path #815 2)
• ringSendOutstanding's comment (worker.go:5184 on main) says the worker mutates all four fields under cs.detachMu; two flushSend callers (closeConn's deferred-close flush, the h2Conns drain) do not; reword to the narrower invariant that holds. (from Follow-ups from #800: test ringSendOutstanding's sendBuf and zcNotifPending clauses, one copy of the condition, the comment's locking claim, the over-cap drop on the async path #815 3)
• Over the send cap the async path dropped whole responses without closing (pre-existing). fix(epoll, iouring): send a large response whole, keep pipelined responses in order, never send a body the handler has given back (celeris#761, celeris#802, celeris#817) #805 (711d6f9) addressed it: a refused write sets writeRefused and runAsyncHandler closes on it (worker.go:5113 on main), and HTTP/1 sendCap is now unbounded per request. Remaining: run the reviewer's over-cap probe on main and record that the conn closes and no response is lost silently. (from Follow-ups from #800: test ringSendOutstanding's sendBuf and zcNotifPending clauses, one copy of the condition, the comment's locking claim, the over-cap drop on the async path #815 4)
• The h2c retry and landing arms (handoff_exit_window_test.go) assert only that no hand-off happened; assert f.w.conns[f.fd]==f.cs and slices.Contains(f.w.h2Conns, f.fd) after the drain, and that the landing arms' conn keeps its slot with its recv re-armed (mutant Mkill passes 9/9 today). (from Follow-ups from #799: assert the h2c arms' outcome, write down the exit contract, a rerunHandOff witness, an unattributed merged-tree failure, two flakes on main #816 1)
• Write down the exit contract in endDispatch's doc comment (engine/iouring/conn.go:633 on main does not state it): every runAsyncHandler exit other than a claim must publish asyncClosed or a non-HTTP/1 protocol before endDispatch clears asyncRun; or pin it with a test that drives every exit and checks rerunHandOff and tryTransplant refuse the conn. (from Follow-ups from #799: assert the h2c arms' outcome, write down the exit contract, a rerunHandOff witness, an unattributed merged-tree failure, two flakes on main #816 2)
• Add a log line or test counter in rerunHandOff's exiting branch (fd_lifetime.go:237 on main) and assert it in the retry arms, so they stay non-vacuous if the fixture drifts. (from Follow-ups from #799: assert the h2c arms' outcome, write down the exit contract, a rerunHandOff witness, an unattributed merged-tree failure, two flakes on main #816 3)
• io_uring's WRITEV scatter-gather path (bodyBuf, sendBody, iov, the WRITEV branches of flushSend/completeSend) is unreachable since fix(epoll, iouring): send a large response whole, keep pipelined responses in order, never send a body the handler has given back (celeris#761, celeris#802, celeris#817) #805 stopped installing the zero-copy body writer (celeris#817); remove it, or bring it back only for bodies the engine may keep. (from Follow-ups from #805: the memory bound after the per-request cap, epoll's closing-conn reap by read time, no deterministic H2 refusal test, io_uring's dormant WRITEV path, body nits #818 4)
• engine/iouring/zc_send_buffer.go:59-63 and zcHoldBytesMax's comment (:125-133): the memory bound is stated as 'SEND_ZCs in flight, bounded by the connection limit', which bounds sends not bytes; since fix(epoll, iouring): send a large response whole, keep pipelined responses in order, never send a body the handler has given back (celeris#761, celeris#802, celeris#817) #805 one SEND_ZC/hold can be a whole response (up to 64 MiB on HTTP/2). Reword to 'connections x their response size'. (from Follow-ups from #813: the SEND_ZC hold's memory-bound comment after #805, a WARN that becomes a Debug line under the #801 lag #845 1a)
• Decide whether to count in-flight SEND_ZC bytes against zcHoldBytesMax or cap the size of a single SEND_ZC, so the 16 MiB cap bounds held bytes (today ~62 stalled 256 KiB responses or one >=16 MiB response push a worker into copy-every-send mode while the orphaned socket lives). Review: not a correctness problem; fix(iouring): hold a closed connection's send buffer until its SEND_ZC notification, past the release backstop (celeris#812) #813's Limits section already says the cap does not bound sends in flight. (from Follow-ups from #813: the SEND_ZC hold's memory-bound comment after #805, a WARN that becomes a Debug line under the #801 lag #845 1b)
• Under the fix(iouring): stop a ring SEND completion parking the worker on a running async handler's detachMu (celeris#750) #801 lag, the closing-drain teardown in checkTimeouts (worker.go:6146-6148 at c417fe3) with zc_send_buffer.go:169 does not replay held send completions, so ce.inflight == zcOwed matches wrongly: a Debug line instead of the backstop WARN (worker.go:4485) and CloseZCNotifHeld counts an already-arrived notification. Safe side, needs a kernel anomaly. Fix: replay held completions in the checkTimeouts teardown (intended for Follow-ups from #801: an end-to-end held-completion witness, the replay's place at every hand-back entry, two untested defensive paths, a 15 s wait without #800 #814, not recorded there). Same fix as io_uring: under the #801 lag, fdOps counts a held SEND_ZC first CQE as still naming the descriptor (early release missed; CloseFDForced counts past the backstop) #880. (from Follow-ups from #813: the SEND_ZC hold's memory-bound comment after #805, a WARN that becomes a Debug line under the #801 lag #845 3)

Added 2026-10-03 (cluster row 63, run 37076068695, pre-registered verdict on #813):

• io_uring close-retire bookkeeping added by fix(iouring): hold a closed connection's send buffer until its SEND_ZC notification, past the release backstop (celeris#812) #813 (io_uring: the pendingRelease backstop returns a connection's send buffer to the pool while a SEND_ZC notification is still owed, so a peer that stalls >10 s then reads receives another connection's bytes #812): on arm64, CloseRetireRecv and CloseRetireZCNotif cost +7% (+8.8 ns per closed connection), above the 5% bar; x86 is +3%. Find where the hold bookkeeping adds work on the retire path when nothing is held (zcOwed accounting, pendingRelease checks), and make it free again in that case. Re-measure with the same row 63 design.

Added 2026-10-03 (#907's round-2 reviews, lane L905):

• Worker.run, engine/iouring/worker.go (worker.go:1072-1075 at fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907's head 242d88a): when platform.SaveThreadAffinity fails, the loop still calls PinToCPU and registers no restore, so a main thread the loop ran on is parked still pinned (the epoll, io_uring: a stopped engine hands its loop threads back to Go still pinned to one CPU (PinToCPU is never undone before UnlockOSThread) #905 shape). sched_getaffinity into unix.CPUSet's 1024 bits fails with EINVAL on a kernel with more than 1024 possible CPUs (nr_cpu_ids), while sched_setaffinity still takes the short mask. Either pin only after a successful save (a behaviour change: no pin on such hosts), or read the mask into a buffer as large as the kernel's (the Go runtime uses 8 KiB, runtime/os_linux.go getCPUCount, golang.org/issue/11823). One change in internal/platform serves both engines (epoll: the same item on v1.6.0 polish: epoll engine #885). fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907 states the gap in a comment at both sites. Code reading only, not reproduced. (from fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907)