io_uring retains live heap on the only refapp with WebSocket and SSE routes: the sole failing cell in the 24h soak

[FumingPower3925]

The one cell that fails the soak leaks live memory, and it is the only one with streaming routes

The 24-hour soak (probatorium run 34368601903, pin c9f7dad) put 48 cells through an hour each. Forty-six report zero property violations. The two that do not are the same cell on both architectures — auth_session_ratelimit on io_uring — and they fail only I-MEM-1, the heap-growth oracle. The same reference app on epoll and on std is clean on both hosts, and every other io_uring cell is clean.

host	cell	heap_inuse over the hour	violations
msa2 (amd64)	auth_session_ratelimit / io_uring	11.2 → 40.9 MB	2550
msr1 (arm64)	auth_session_ratelimit / io_uring	9.0 → 29.1 MB	2476
msa2	auth_session_ratelimit / epoll	3.1 → 11.8 MB	0
msa2	auth_session_ratelimit / std	1.7 → 6.8 MB	0

Magnitude is not what separates them. observability on io_uring grew further in its own hour, 17.9 → 53.5 MB, and passed. The oracle fits post-warm-up per-150-second minima, so a workload that ramps and then plateaus scores flat however high the plateau sits. What fails is a series whose post-collection floor keeps rising.

It is live memory, not span fragmentation

I reproduced the cell locally under Docker against the same pin and sampled exact runtime.MemStats from the reference app once a second, then re-derived the oracle's own statistic from the series. At eighteen minutes:

series	post-collection trough slope
HeapInuse	1,987 B/s — over the 1,024 B/s budget, same verdict as the cluster
HeapAlloc	1,334 B/s
HeapObjects	5.7 /s

Live object count rising is the part that matters. Fragmentation moves HeapInuse without moving HeapAlloc or HeapObjects; all three move here. Roughly 234 bytes per retained object, against about 71 accepted connections per second.

What is distinctive about this cell

auth_session_ratelimit is the only one of the nine reference apps that routes /ws and /events. The other eight have no WebSocket and no server-sent events at all, and the validator probes for those routes and skips the corresponding walker slices when they are absent. All three engines take the identical streaming load in this cell — 47,598 WebSocket upgrades and 9,185 event streams killed mid-stream per hour, matched to within five requests across engines — and only io_uring retains.

So the intersection is: io_uring, plus connections that detach.

What this is not

• Not goroutines. The failing cell has the smallest goroutine growth of any io_uring cell in the run (+15 over the hour), and I-MEM-3 is zero everywhere. The incident's goroutine profile holds 101 goroutines, two of them in the event-stream handler and none in WebSocket.
• Not connection accumulation. Active connections sit at 44 at the incident, with accepted minus closed accounted for.
• Not the session store. Sessions expired exceeds sessions created over the window, and the same middleware runs on the engines that stay clean.
• Not the reference app's own bookkeeping. Its WebSocket handler is a bare echo loop and its event handler a ticker; neither registers the connection anywhere.

Still open

Which allocation site. The default heap profiler samples one allocation per 512 KB, which cannot resolve this rate over any window shorter than the cell itself — the soak's own incident dossier ships two profiles whose difference is smaller than the profiler's sampling error, which is why it names nothing (probatorium#319). Two controls and a finer-grained profiling run are in flight:

1. epoll on the identical local cell, to confirm the asymmetry off-cluster
2. the same reference app with /ws and /events renamed away, so the streaming walkers are skipped and everything else is byte-identical
3. a repeat io_uring cell with the profiler sampling every 4 KiB, which resolves the growth well enough for a profile diff to name the site

[FumingPower3925]

Correcting the rate in the title: it decays, and over a full cell it is marginal

I opened this from an eighteen-minute reading. The cell has now run its full fifty minutes, and the rate is not constant — it decays as the window lengthens.

window	heap_inuse trough slope	live objects
18 min	1,987 B/s	5.7 /s
45 min	1,144 B/s	3.9 /s
50 min (full cell)	1,085 B/s	1.7 /s

Budget is 1,024 B/s, so the full-cell verdict is still a violation, but only just. The title's "5.7 objects/s sustained" was the steepest early window and I should not have put it there. Retitling.

Whole cell, io_uring, fifty minutes, 216,093 connections accepted and 216,044 closed with 49 active at the end:

heap_inuse   5.94 -> 22.47 MB      heap_alloc  4.57 -> 19.96 MB
goroutines   47 -> 67              span gap (inuse - alloc)  1.36 -> 2.51 MB

Live bytes and live objects still both rise, so the original point stands: this is retained memory rather than span fragmentation. What does not stand is the implication of a constant-rate leak. A decaying slope is what a filling structure looks like — a cache reaching its working size, or a high-water mark being ratcheted up by successively larger bursts. It is not what an unbounded leak looks like.

Two things follow and both are now measuring rather than arguing.

First, the fitted slope carries far more measurement noise than a single number suggests. Two fits eleven seconds apart, on the same run, gave 1,685 and 1,144 B/s. With eighteen trough points the final bucket has enormous leverage, so one late post-collection sample moves the answer by half. Both computations agree to the byte on identical input, so this is the estimator's sensitivity and not a tooling bug — but it means no slope quoted here should be read to three digits.

Second, the default heap profiler cannot see this. The widest post-collection pair from this cell, thirty-two minutes apart, sums to minus 1.5 MB with a largest single entry of two samples. At one sample per 512 KB there is nothing to read. A repeat cell at 4 KiB sampling is running, along with an A/B pair built from two snapshots of the same tree that differ only by #574's one-line queue clear, which is the candidate this decay shape points at.

Holding further conclusions until those land. Flagging the correction now rather than leaving the original rate standing.

[FumingPower3925]

The local cell reproduces the soak's failure closely enough to test fixes against

Worth recording separately from the diagnosis, because it changes what a fix costs to verify.

The local io_uring cell wrote the same single-predicate failure the cluster did, and the streaming workload it generated matches the cluster's to within about one percent once scaled to the same duration. Nothing about that was arranged; the walker simply drives the same reference app at the same concurrency.

	cluster cell (1 hour)	local cell (50 min)
violations	2,550, I-MEM-1 only	1,950, I-MEM-1 only
share of evaluations	5.8%	5.4%
WebSocket upgrades	47,598	39,598
event streams killed mid-stream	9,183	7,629
slope at first violation	3.2 KB/s	2.2 KB/s

The incident dossier has the same structure down to the five-trough window and the floor calculation:

I-MEM-1 violated: heap_inuse trough slope 2.2 KB/s exceeds 1.0 KB/s budget; rise 1.21 MB
(fitted 1.30 MB, trough-to-trough 1.21 MB) over 750 samples over 12m29s (5 troughs) clears the
floor 600.0 KB = max(budget x 10m0s = 600.0 KB, 3% of level 14.14 MB = 434.5 KB, 8x sampling
noise 43.5 KB = 347.9 KB); 16.20 MB -> 17.84 MB

So testing a candidate no longer needs a cluster soak. It needs a fifty-minute container.

One hypothesis the tally already rules out

Every WebSocket connection closed cleanly — ws_closed_correctly equals ws_upgraded, 39,598 of 39,598 — and all but two event streams were killed mid-stream as the walker intends. Whatever is retained is not connections that failed to close.

Setup is --security-opt seccomp=unconfined --ulimit memlock=-1 --cpus 8 against a golang:1.27 image, with the reference app's exact runtime.MemStats polled once a second from /debug/vars, and post-collection heap profiles pulled from /debug/pprof/heap?gc=1 every five minutes. The ?gc=1 matters: without it each profile is a random point on the collection sawtooth and successive ones are not comparable — two taken three minutes apart showed a major allocation site dropping by 2,898 objects.

[FumingPower3925]

The engine asymmetry reproduces locally, on an identical workload

Both cells ran fifty minutes against the same reference app at the same concurrency, and both accepted exactly 216,093 connections. I did not arrange that; the walker is deterministic enough that the workload matched to the connection. So the comparison has no workload term in it.

	io_uring	epoll	ratio
I-MEM-1 violations	1,950	0	—
heap_inuse trough slope	1,085 B/s (24.3%/h)	131 B/s (7.2%/h)	8.3x abs, 3.4x rel
heap_alloc trough slope	686 B/s (23.5%/h)	72 B/s (7.7%/h)	9.5x abs, 3.1x rel
live-object trough slope	1.7 /s (38.8%/h)	0.4 /s (19.5%/h)	4.3x abs, 2.0x rel
heap_inuse over the cell	5.94 → 22.47 MB	2.43 → 6.46 MB
goroutines	47 → 67	47 → 63

Both fitted over eighteen trough points, which is where the estimator settles — below about nine the two engines' relative rates actually flipped between adjacent windows, so anything quoted from a short window here would have been meaningless.

This mirrors the cluster exactly: 0 violations on epoll, thousands on io_uring, same reference app, same load.

Two things this settles

The absolute-budget explanation is finished. io_uring does not merely carry a larger working set; it grows three times faster as a fraction of that working set. A budget scaled to the baseline would still fail it. I argued the opposite in probatorium#318 and have retracted it there.

What is retained is large. io_uring holds roughly four times the objects but nine times the bytes — about 404 bytes per retained object against epoll's 180. Whatever accumulates is a big structure, not many small ones, which fits connection state carrying its read and write buffers.

What is still open

Which reference holds those connection states. The allocation site is confirmed — pprof -traces puts the growth at the connStatePool constructor, reached through sync.Pool.Get from acquireConnState in the accept path — and because the runtime drains a sync.Pool on collection, and this cell ran roughly 14,500 collections, pooled-but-idle states cannot be what accumulates. They are reachable from something.

A repeat io_uring cell at 4 KiB profiler sampling (128x the default density) is running now to name that something, followed by an A/B pair cut from the same commit with and without #574's one-line queue clear — the candidate this shape points at, already merged as hardening but not yet shown to be the cause.

[FumingPower3925]

Profiling at 128x density: half the growth is connection state plus the WebSocket upgrade graph

The default profiler samples one allocation per 512 KB, which is why neither the soak's incident dossier nor my first reproduction could name anything — the widest diff I could take at that rate summed to negative 1.5 MB. Re-running the cell with GODEBUG=memprofilerate=4096 samples 128 times as densely.

Verified live rather than assumed, since the linker zeroes the rate when runtime.MemProfile is unreachable and applies that after GODEBUG is parsed:

heap profile: 595: 6425880 [470919: 116052552] @ heap/8192

The header carries twice the rate, so 8192 confirms 4 KiB sampling. It also did not perturb the workload: this cell and the default-rate one both ran 65 connections/s at t=300s.

Post-collection diffs from the warm-up boundary, cumulative share of total growth:

window	acquireConnState	websocket.tryEngineUpgrade	combined
12 min	15%	25%	40%
22 min	21%	29%	50%
27 min	25%	23%	48%

Those are disjoint subtrees — one on the accept path, one on the request path — so about half of all post-warm-up growth is connection state and the WebSocket upgrade object graph, in roughly equal parts. (*Worker).run accounts for 64% overall. The individual shares trade places between the last two windows and the total moved 2.1 → 1.9 MB, which is the collection sawtooth showing through even in forced-collection snapshots; the combined figure is what holds.

The cell itself behaved like every other io_uring run: 526 I-MEM-1 violations, trough slope 1,196 B/s against the 1,024 budget.

Why both ends of the chain appear

Verified by reading, not inferred: SetWSDetachClose stores its callback on H1State.OnDetachClose, and the callback tryEngineUpgrade installs is a closure capturing the WebSocket Conn. So one retained *connState keeps h1State → the closure → the Conn → its chanReader and buffered reader/writer. A single held pointer drags the whole graph, which is why the accept-path allocation and the upgrade-path allocation both grow together.

tryEngineUpgrade.func1 contributes 8% on its own. That is the per-message copy the upgrade makes because the engine reuses its read buffer, so retained channel readers are holding buffered inbound data rather than being empty shells. It also explains part of the size asymmetry against epoll: 4x the retained objects but 9x the bytes.

What this does not establish

Which reference holds them. A heap profile records allocation stacks, not retention paths, and it cannot distinguish "allocated here and leaked" from "allocated here and held by something three hops away".

One caution for anyone reading the same profiles: do not build a retention count from inuse_objects. Across five forced-collection snapshots with active connections pinned at 44-46, the channel-reader estimate swung between 202 and 361 in adjacent snapshots, and connection state was not monotone (356, 379, 401, 394, 411). These are extrapolations from sampled allocations and the error is comparable to the trend. I started converting them into a per-connection leak rate before noticing.

Next: two cells cut from the same commit, differing only by #574's one-line queue clear. Primary readout is the violation count (1,950 vs 0 separates io_uring from epoll cleanly), with the slope secondary. That pair can confirm a complete fix; it cannot reliably resolve a partial one, so a null result will mean "not the whole cause" rather than "no effect".

[FumingPower3925]

#574 is the fix. I predicted it would not be, and it is.

Two 50-minute cells cut from the same commit (a73d306), differing only by the one-line clear() #574 added to the hand-off queues. Same reference app, same concurrency, and the two accepted 206,211 and 206,355 connections — within 0.07%, so there is no workload term in this comparison.

	base	fix	epoll, for reference
I-MEM-1 violations	1,651	0	0
heap_inuse trough slope	1,071 B/s (23.8%/h)	52 B/s (1.9%/h)	131 B/s (7.2%/h)
heap_alloc trough slope	728 B/s	29 B/s	72 B/s
live-object trough slope	2.8 /s	0.2 /s	0.4 /s
heap_inuse over the cell	5.94 → 18.76 MB	6.05 → 9.66 MB	2.43 → 6.46 MB

Zero violations across 34,350 evaluations, a 20x reduction in slope, and the fixed io_uring engine now retains less in absolute terms than epoll does. The fix half is under the budget at every window from seven trough points onward; the base half is over it at every window.

The base half is a verified control, not an assumption: it reproduced the independently-built pinned cell to within 1.5% (1,071 vs 1,085 B/s).

The prediction I got wrong

I wrote, before running this, that I expected no difference — reasoning that the hand-off queue's depth should be single digits across eight workers, capping what a clear() could recover at roughly 1 MB against ~15 MB of growth. The measured effect is about 9 MB over fifty minutes. The arithmetic was fine; the assumed queue depth was wrong, by more than an order of magnitude.

That matters beyond bookkeeping. If the queue routinely runs hundreds deep, the same reasoning that made me dismiss this would make me dismiss the next one like it. The instrumented tree I built while these cells ran records the queue's deepest drain and its backing array's peak capacity — capacity being what the clear() actually reclaims, since a slice truncated to [:0] pins every slot up to cap. Running it next to get the number.

Why the profile pointed here after all

The retention chain reads the same way it did before, only now with a cause at the root: a stale slot in the queue's backing array holds a *connState, which holds h1State, which holds the OnDetachClose closure, which captures the WebSocket Conn and its chanReader and buffers. One stale pointer per slot, the entire graph behind each. That is why the fine-grained profile found growth split between the accept path and the upgrade path: two ends of one retention.

It also explains the size asymmetry against epoll — 4x the objects but 9x the bytes. epoll has the same un-cleared queue, but it does not drive WebSocket pause/resume through it, so its stale slots hold far less each.

Not closing this yet

One pair, one machine, one architecture, under Docker. The release bar asks for both architectures and a clean 24-hour soak. #574 is already merged, so the verification path is: repin probatorium to current main, nightly, soak, benchmark. This stays open until the soak reports 48 clean cells on both hosts.

[FumingPower3925]

Verified on the cluster, both architectures. Closing.

The 24h soak (probatorium run 34616620237) ran the same 48 cells as the run that opened this issue, pinned to a celeris main carrying #574.

	before #574 (run 34368601903)	after (run 34616620237)
property evaluations	2,116,360	2,116,384
I-MEM-1 violations	5,026 (2,550 amd64 + 2,476 arm64)	0
any property violation, any cell	2 cells	0 cells

Every memory oracle was silent in all 48 cells on both hosts for the full 24 hours. That is the measurement the release bar asks for, and it agrees with the local A/B that predicted it (1,651 → 0 violations, trough slope 1,071 → 52 B/s).

Root cause, restated for the record: four hand-off queues truncated a slice of pointers with [:0] and reused the backing array without clearing it, so every stale slot kept a whole connection reachable — its buffers, its protocol state, and the entire WebSocket object graph hanging off the detach-close closure. Measured peak queue depth was 1,755 entries with a backing array of 2,303. I had predicted single digits and dismissed the candidate on that basis; the arithmetic was fine and the input was wrong by two orders of magnitude.

The soak did fail its gate, on an unrelated single-occurrence SSE counter that six independent reviews could not attribute to celeris and that the harness now records the error value for (probatorium#327). That is tracked separately and does not bear on this issue.