io_uring default (single-shot) recv stalls a quarter to two-thirds of connections under sustained WS inbound flood

[FumingPower3925]

What

On a sustained inbound WebSocket flood, the io_uring engine in its default configuration drains
connections so slowly that clients hit their write deadlines and give up. epoll, running the same
receive model (single-shot, one read per readiness) on the same workload, never does.

Measured with TestBackpressureInboundSequenceIntegrity (96 connections x 64000 frames of 126
bytes), linux/arm64, docker --cpus 4, over 10+ runs:

engine	framesIn / framesSent	clientCloseFail	closedOK	subtest wall-clock
epoll	6144000 / 6144000, every run	0	96/96	~2.8s
io_uring (default)	~3.0-4.6M, only 17-79 of 96 clients finish sending	81-186	17-64	~20.1-22.1s
io_uring + multishot	often 6144000/6144000	0-52	73-95	~2.5s

clientCloseFail counts a client whose 2-second write deadline expired — i.e. the server stopped
draining that socket for over two seconds. epoll records zero of these in every run; the default
io_uring configuration records 81-186.

The wall-clock is the tell: io_uring's subtest sits at ~20s run after run while epoll and io_uring
multishot both finish in under 3s. That is clients timing out, not throughput scaling.

Reader starvation

Instrumenting chanReader.Read for calls that had to block for a chunk:

engine	Reads that blocked
epoll	829-1027
io_uring + multishot	~1300
io_uring (default)	4023-9418

The handler is starved waiting for inbound data that the engine has not delivered.

Why this is the default path

CELERIS_IOURING_MULTISHOT_RECV is opt-in (worker.go), deliberately, because multishot throttled
an aarch64 6.6.10 kernel to ~90 req/s. So the configuration users get by default is the slow one
here, and the fast one carries its own documented regression.

Why it matters beyond this test

It is the mechanism behind the residual failures on the celeris#484 oracle. Because the server falls
behind, clients abandon connections mid-stream; the resulting RST discards data the server had not
yet read, and the handler reports unexpected EOF or a frame-count mismatch. Those read as
corruption but are downstream of this.

Not investigated here: whether the cost is the pause/resume cancel+re-arm cycle (io_uring must
submit an ASYNC_CANCEL to pause and a fresh recv to resume, where epoll simply stops and starts
reading), or something else. pauses/resumes balance exactly on both epoll and default io_uring,
and no pause exceeded 543ms, so it is not a stuck or lost resume.

[FumingPower3925]

Correction: the comparison in this issue is confounded and the headline claim is not supported by
the data I posted.

The repro pins --ulimit memlock=8388608 (8 MB). That is below the engine's per-worker minimum, so
io_uring was capped to a single worker while epoll ran with four:

WARN io_uring workers capped by RLIMIT_MEMLOCK requested=4 capped_to=1
  memlock_cur_bytes=8388608 min_per_worker_bytes=12582912
INFO io_uring engine listening ... tier=high workers=1 sqpoll=false send_zc=true
WARN SEND_ZC returned ENOMEM (RLIMIT_MEMLOCK), falling back to regular SEND worker=0

Every number in the table above is therefore epoll on 4 workers against io_uring on 1, plus
io_uring silently losing SEND_ZC to the same limit. That is not an engine comparison. The 8 MB cap
was deliberate in that harness — it reproduces CI's one-worker io_uring for the #519/#482 work — and
I carried it into this measurement without accounting for it.

Re-measuring with memlock=134217728 so io_uring gets its four workers. I will post the corrected
numbers and close this if the gap disappears.

The findings that do NOT depend on this comparison still stand, because they were measured
before-and-after within the same one-worker environment: the celeris#484 spill and drain fixes
(default-config oracle 0/10 -> 11/14), and celeris#530's RST-scored-as-clean-close, which is a
property of the test's isCloseErr and not of any engine.

[FumingPower3925]

Corrected measurement — equal workers

Re-ran with --ulimit memlock=134217728 so io_uring gets its four workers (confirmed
workers=4 in the engine's own startup line, and no capped by RLIMIT_MEMLOCK warning). Same
workload: 96 connections x 64000 frames of 126 bytes, --cpus 4, linux/arm64, 6 runs.

engine	framesIn / framesSent	clientCloseFail	closedOK
epoll	6144000 / 6144000, all 6 runs	0	96/96
io_uring + multishot	6144000 / 6144000, all 6 runs	0	96/96
io_uring (default, single-shot)	4.16M - 5.38M	25, 28, 28, 35, 43, 66	65-84

io_uring with multishot recv is now exactly identical to epoll — perfect delivery, not one
client failure, in every run. The dramatic gap in the original report was the one-worker cap, and
that part of the claim is withdrawn.

What survives, and it is still worth fixing

The default configuration — single-shot recv, which is what CELERIS_IOURING_MULTISHOT_RECV
being opt-in leaves users with — stalls 25 to 66 of 96 connections for more than two seconds
under this flood, where both epoll and io_uring-multishot stall none. clientCloseFail counts a
client whose 2-second write deadline expired, i.e. the server stopped draining that socket for over
two seconds. It delivers 68-88% of the offered frames rather than 100%.

All six runs still pass the oracle, because those connections are marked client-failed and
excluded from the per-connection assertions. So this is invisible to the gate today — it shows up
only in clientCloseFail and closedOK, neither of which is asserted.

Narrowed by this data: it is specific to single-shot recv on io_uring. Same engine, same
workload, same worker count, multishot is flawless. Whatever the cost is, it is in the per-recv
re-arm path, not in the engine generally.

Knock-on for celeris#484

At four workers the #484 oracle passes 6 of 6, against 11 of 14 at one worker. So a large part
of that residual was the worker cap as well, not a distinct defect.

[FumingPower3925]

Root cause

epoll applies the WebSocket backpressure pause inline on the loop thread; io_uring defers it
through the detach queue. epoll got that fix, io_uring never did.

epoll/loop.go:1287, inside drainRead's read-until-EAGAIN loop (the function spans 879-1304), with
a comment that names the failure exactly:

Detached WS/SSE backpressure: if the middleware signaled pause during this iteration
(recvPauseDesired toggled by chanReader crossing high-water during Append), apply EPOLL_CTL_MOD
inline and stop reading. Without this, a large single message can over-fill the channel within one
drainRead pass — pause would only land on the NEXT iteration via drainDetachQueue, too late to
prevent overflow → chanReader drop → ErrReadLimit.

io_uring has no equivalent. PauseRecv only sets recvPauseDesired and wakes the worker; the
ASYNC_CANCEL is submitted later from drainDetachQueue and must then round-trip through the
kernel.

There is a second half to it. On resume, epoll re-arms EPOLLIN and the next readiness event drains
the socket completely (for { unix.Read ... } until EAGAIN). io_uring's single-shot path arms
one recv and gets back one buffer, so every drain cycle costs a full async round trip for a
single buffer's worth of data. Multishot escapes this because after one re-arm the kernel keeps
delivering CQEs unattended — which is why multishot matches epoll exactly.

Evidence

Disabling the pause entirely (WS484_BP=100000, so the high-water mark is never crossed) on the
single-shot path, 4 runs each, 4 workers:

config	clientCloseFail	serverRST
BP=256 (pause active)	41, 29, 48, 30	17, 12, 21, 13
BP=100000 (pause never fires)	14, 10, 0, 0	6, 3, 0, 0

Non-overlapping. The stalls live in the pause/resume cycle, not in plain single-shot recv.

Refuted along the way, so nobody repeats them:

• Not a deferred re-arm under SQ pressure. Instrumented: prepRecvSQFull=0, resumeArmFail=0
  across 536 resumes and 99,397 prepareRecv calls. Every resume armed immediately.
• Not pause count. Multishot pauses more often (155-235 episodes vs 28-150 for single-shot)
  and never stalls.
• Not a stuck pause. pauseMaxMs peaked at 543ms with zero pauses over 2s.

Fix direction

The inline pause is the direct analogue of epoll's fix and is available: PauseRecv is invoked from
chanReader.Append, which already runs on the worker thread, so io_uring can cancel the recv
inline instead of deferring it. That addresses the latency half.

The one-buffer-per-resume half is separate and structural: single-shot recv has a pipeline depth of
one per connection. Options are to keep more than one recv in flight per connection, or to make
multishot the default for detached connections specifically — the aarch64 regression that made
multishot opt-in was measured on HTTP churn, not on WebSocket streaming.

Related: this is also the mechanism behind celeris#484's overflow. The comment quoted above names
chanReader drop → ErrReadLimit as precisely what the inline pause prevents.

[FumingPower3925]

Why "drain more per round trip" cannot be done on the single-shot path

Following up on the root cause above, the obvious remedy — let a resume pull more than one buffer —
does not work, and it is worth writing down why so nobody tries it.

epoll drains because drainRead loops unix.Read until EAGAIN: many reads, one readiness event,
strictly ordered because each read completes before the next begins. io_uring cannot copy that shape
with single-shot recv. Arming two recvs on the same connection puts two independent operations in
the ring against one socket, and their CQEs can complete in either order — that is stream
corruption, not a throughput win. Pipeline depth on a single-shot recv is necessarily one.

Which leaves exactly one mechanism that drains more than one buffer per round trip while preserving
order: multishot recv, where the kernel keeps delivering into the provided-buffer ring without
the worker re-arming. That is precisely why multishot measures identical to epoll here —
6144000/6144000, clientCloseFail=0, serverRST=0, every run — while the default single-shot path
stalls 25-66 of 96 connections.

So the fix is a configuration decision, not an algorithm

Multishot is opt-in (CELERIS_IOURING_MULTISHOT_RECV=1) for a documented reason, quoted from
worker.go:

multishot recv on aarch64 kernel 6.6.10 throttles the whole worker to ~30 accepts/s / ~90 req/s —
a 100x collapse vs the epoll engine's ~25k req/s on the identical workload. The profile showed
workers drowning in spurious recv CQEs (25k per worker per second against ~50 useful
completions)... Opt back in for workloads that are known to be dominated by long-lived keep-alive
conns and that benefit from the CQE-batching multishot provides.

Note what that regression was measured on: connection churn, thousands of short-lived
connections. And note what the opt-in advice already says it is for: long-lived keep-alive
connections. A detached WebSocket connection is the extreme case of exactly that — one connection,
no accept churn, sustained inbound streaming.

So the targeted change is to select multishot per connection at detach, rather than globally by
env var: keep single-shot for the HTTP path where the churn regression was measured, use multishot
for detached WS/SSE connections where it is measured identical to epoll.

I am not implementing that blind. It needs dual-arch benchmark evidence — specifically that the
aarch64 churn collapse does not reappear when multishot is scoped this way — and that is a
cluster-sized measurement, not a local one. Recording it as the recommendation so the next step is
measurement rather than rediscovery.