Follow-ups from #671: measure the engine-worker stall under pausedMu, the epoll tail lean, #705-repro vs M5 coverage, prereg predictions

[FumingPower3925]

Follow-ups from PR #671 (#667 + #672: the WebSocket chanReader applies pause/resume under the lock that decides them, and lifts a stale pause). They were deferred under the maintainer's two-round review cap (2026-09-27). #671 merges at 371c76a after #674. Its final two re-reviews APPROVED, and its measured A/B passed: correctness W1/W2 PASS; throughput 4/4 cells PASS on the pre-registered rule; io_uring/m128 replication +0.40% [-0.81, +2.29], p=0.096. The tail secondary is not resolved: epoll at 128 MiB leans against the PR (ops >=10 ms 17 vs 10; per-round max p=0.042, uncorrected), so a small epoll tail cost is possible.

Open items (round-3 re-review, both lenses approve)

1. minor: The PR body says celeris#705's repro, used as middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705's failing-first test, 'is what will kill M5'. It cannot. The repro checks only the channel depth and the spill, and M5 changes neither. So the new re-check's !r.hasSpill() guard has no test, and the plan to test it does not work. M5 (the re-check ignoring the spill) only decides whether resume() fires. The repro therefore fails the same way with or without M5 on today's code. After any middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705 fix that promotes the spill, it passes the same way with or without M5. The guard's own property can be tested today, in the same interleaving, without waiting for middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705: the engine must stay paused while a chunk is spilled. The head reports enginePaused=true there, and M5 would resume. Fix: add that assertion, or rewrite the M5 sentence and put 'pin the re-check's spill guard' into middleware/websocket: a chunk that spills after the handler drained the channel is never promoted, so the connection wedges permanently (stranded spill) #705's scope.
   Evidence: PR body, Mutants section: 'fixed on top of this PR with that repro as its failing-first test, which is what will kill M5'. /Users/fuming/.claude/projects/-Users-fuming-Documents-github-celeris-probatorium/evidence/celeris-667/lane-20260926/05-stranded-spill/stranded_spill_repro_test.go:52 has the repro's only assertion, if len(r.ch) == 0 && r.hasSpill(). run-on-head-d3472e7.log shows depth=0 spill=1 enginePaused=true readerPaused=true pauses=2 resumes=1. Under M5 the re-check at engineread.go:306 would call resume() (len 0 <= lowWater 2), giving enginePaused=false with depth and spill unchanged. round3/04-mutants.txt: M5 SURVIVED.
2. minor: Nothing tracks the engine-worker stall this PR adds. The middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667 fix holds pausedMu across ResumeRecv. On the handler side that means detachQMu plus, on the queue's empty-to-non-empty edge, an eventfd write(2) under the WakeFD read lock. So the event-loop thread (sync mode) or the dispatch goroutine (async), in Append -> requestPause, now waits on a lock a handler goroutine holds, and that wait includes the handler's syscall and its rescheduling. I re-derived the graph and it is deadlock-free: every detachQMu section is a leaf, and WakeFD writers take no other lock. But the wait is bounded only by scheduling, and it freezes every connection on that loop. The body measures the stall and a possible epoll tail cost, then says reducing it 'is not this PR'. No issue was filed. A design exists that keeps middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667's ordering without the hold: drainDetachQueue applies the latest recvPauseDesired, so the chanReader could store the desired state under pausedMu and do the queue append and Signal after unlocking. That needs a small split of the engine's PauseRecv/ResumeRecv API. File it with the measured numbers (search first); it does not block this PR.
   Evidence: engineread.go:275-310 and :317-324 run the callbacks under pausedMu. engine/epoll/loop.go:1770-1797 and engine/iouring/worker.go:2342-2369 show the callbacks: Swap, detachQMu, then Signal. PR body, Worker stall: about once per 440-650 ops, 80-180 us mean wait, largest mean 0.47 ms. Tail: epoll 128 MiB per-round max p=0.042, and post hoc ops >= 2 ms 29 vs 13. PR body: 'Making those waits rarer would mean changing how the callbacks are ordered, and that is not this PR.' gh issue list --search pausedMu finds only middleware/websocket: chanReader applies pause/resume to the engine outside the lock that decides them, so a resume can be lost and the connection stops delivering (found by reading, not reproduced) #667 and middleware/websocket: a pause decided on a stale depth snapshot wedges the connection permanently when the handler drains to empty first (distinct from #667) #672 (plus the unrelated, closed io_uring: a linked recv never starts because its chained SEND blocks on a detached WebSocket peer's closed window #607 and io_uring: a recv-pause whose cancel SQE is unavailable is recorded as paused while the recv stays armed #519).
3. minor: The throughput headline says more than the evidence supports. The section title, the Summary bullet and the sentence "So there is no loss above the floor." all claim no loss, but only the pre-registered rule passes; every other view of io_uring at 128 MiB leans against this PR. In the body, "So there is no loss above the floor" is immediately followed by a pooled estimate that is above its floor at p < 0.05. Suggested wording: "no loss above the floor by the pre-registered rule". The Summary should also give the pooled p and say that the pooled view and the load-sensitivity view both meet the BLOCK rule's letter. The maintainer can then weigh a likely io_uring cost of about 1-2.5% at four workers knowingly.
   Evidence: All four primary cells PASS, and I recomputed them exactly from round4/timing/*.log with my own parser. The replication also PASSes (+0.40%, p=0.0958; round4/35-CONFIRM.txt). But io_uring at 128 MiB leans slower in every view:

• Primary: +2.29%, p=0.0585, floor 0.03%.
• A4 load sensitivity: +3.37%, p=0.019, floor 0.71% (34-SECONDARY.txt, which says "-> BLOCK").
• Pooled, 88 rounds per arm: +1.40%, p=0.0239, floor 0.56% (36-CONFIRM-POOLED.txt; my recompute gives the same).
• My own POST HOC block-paired sign-flip, using the pairing the ABBA design built and the unpaired MWU ignores:
  • timing matrix: head +2.73%, p=0.006; A/A +0.24%, p=0.45;
  • replication: +1.37%, p=0.062;
  • all 43 blocks: +1.72%, p=0.0075 (Wilcoxon 0.013); A/A -0.60%, p=0.81.

The pooled views are biased upward, because round 4 is what flagged the cell. That is why this is minor, not a block.
4. minor: The pre-registration states no predictions. The review's MAJOR 2 asked for "predictions, n, seed, arms, decision rule". The files fix the arms, n, seed procedure and one-sided decision rules, but never state the expected outcome: for example, that this PR equals main while #667 alone exceeds it, or an expected throughput band. This cannot be fixed after the fact. The A/B section should list it as a deviation from the request.
Evidence: A grep for predict|expect|hypothes over round3/ab/10-PREREGISTRATION.md, round4/11-ADDENDUM.md and round4/12-CONFIRM.md returns nothing. The only hit in the lane is 10-wedge/10-PREREGISTRATION.md:72 ("expected null").

Item 2 matters most for the release: measure the engine-worker stall directly (event-loop and dispatch time blocked on pausedMu), in the lane B-2 oracle work or on the cluster stress path, and decide from data. The epoll tail lean belongs to the same question.

Evidence: evidence/celeris-667/lane-20260926/round3/ to round5/ (MANIFESTs, scripts, hashed pre-registration and addendum).

[FumingPower3925]

Item 2 measured: no v1.6.0 fix by the pre-registered rule; the exposure is real but rare

I measured it, with the decision rule written and hashed before the first run (evidence/celeris-b2/b2b/05-716-stall/10-PREREGISTRATION.md).

How. A throwaway branch (tmp/b2b-716-stall, main + instrumentation) records these in requestPause and resumeIfDrained:

• every contended acquisition of pausedMu, timed exactly (the TryLock fast path is left alone otherwise);
• every ResumeRecv call made under the lock.

It ran on GitHub runners (kernel 6.17, 8 MiB memlock), without -race. Workloads: both backpressure tests, the inbound one at its full default size, on both engines. Sync and async mode, 5 processes per arch per mode. Runs 36339356834 (sync) and 36339367373 (async).

The engine side. This is the event-loop thread, or in async mode the goroutine that runs the WebSocket data callback.

	result
requestPause calls	19,806
found pausedMu held	3 (0.015%), all arm64 io_uring PauseCancel
those 3 waits	1.02 ms (sync), 3.84 ms and 0.41 ms (async)
p99.9 over all calls	0 in every cell
blocked time	at most 2.1e-4 of worker time

Rule (pre-registered): a fix is needed if any cell has p99.9 ≥ 1 ms, a single wait ≥ 10 ms, or ≥ 1% of worker time. None met it, so no v1.6.0 fix.

One pre-registered prediction failed: "max < 1 ms". The measured max is 3.84 ms.

The exposure is real, and #716 argued it right. This was reported, not judged. The handler holds pausedMu across ResumeRecv for as long as scheduling takes, and that hold has a heavy tail in this in-process load (96 handlers and 96 clients on 4 vCPUs):

• 262 ms max (arm64 epoll, inbound, sync);
• 193 ms (x86 epoll, inbound, async);
• 68 ms (x86 epoll, PauseCancel, sync).

Its p99 is at most 16 µs everywhere, except arm64 io_uring PauseCancel (at most 262 µs sync, 524 µs async).

A requestPause that lands inside such a hold stalls the loop for the rest of it. It rarely lands there, because the engine takes a connection's pausedMu only when that same connection crosses high water. #716's design (store the desired state under the lock, then queue and Signal after unlocking) removes the exposure if a workload ever shows it.

Numbers: evidence/celeris-b2/b2b/05-716-stall/20-analysis.txt and 21-predictions-vs-outcome.txt.

Items 1, 3 and 4 are not addressed here. The measurement is also in #749's body.