Skip to content

fix(engine): clear the hand-off queue slots so a drained conn can be collected - #574

Merged
FumingPower3925 merged 1 commit into
mainfrom
fix/queue-slot-retention
Sep 11, 2026
Merged

fix(engine): clear the hand-off queue slots so a drained conn can be collected#574
FumingPower3925 merged 1 commit into
mainfrom
fix/queue-slot-retention

Conversation

@FumingPower3925

Copy link
Copy Markdown
Contributor

Four reuse queues across the two loop engines truncate a slice of pointers with [:0] and reuse the backing array. Truncation does not clear the array, so every pointer in it stays reachable until some later drain happens to overwrite that slot. Each queue pins its own high-water mark worth of objects for as long as the worker lives.

file queue element
engine/iouring/worker.go detachQueue / detachQSpare []*connState
engine/epoll/loop.go detachQueue / detachQSpare []*connState
engine/epoll/adopt.go adoptQueue / adoptQSpare []adoptItem
engine/iouring/driver.go driverActionQueue / driverActionSpare []driverAction

The two detach queues are the ones that matter. A detached connState carries its read and write buffers, its H1State, and on the WebSocket path whatever the middleware hung off that state, so one deep burst leaves megabytes reachable behind a queue that is nominally empty.

drainPendingRelease, in the same file as the io_uring one, already guards exactly this:

// Nil out the tail slots so the backing array drops its strong refs
// to released connStates.
for i := len(kept); i < len(w.pendingRelease); i++ {
	w.pendingRelease[i].cs = nil
}

These four never got it. Same shape as #571: one guarded copy of an idea and several unguarded ones.

What this is not

Retention here is bounded by peak queue depth, not unbounded, so this is hardening and not a fix for #573. I found it while reading the detach paths for that leak and am landing it on its own merits rather than attaching it to a cause I have not yet established.

Tests

Two guards per engine. The first reads the spare slice out to its capacity and requires every slot past the new length to be nil. The second states the same guarantee as an outcome — a cleanup registered on an enqueued conn must run once the caller drops its reference — which is what the fix is actually for. detachClosed short-circuits the drain body, so both exercise the real drainDetachQueue without needing a ring or an epoll fd.

Negative control, reverting only the clear() and keeping everything else:

--- FAIL: TestDrainDetachQueueDropsSlotRefs
    drainDetachQueue left 64 of 64 connStates reachable in the reused backing array
--- FAIL: TestDrainDetachQueueLetsDrainedConnsBeCollected
    a connState the detach queue already drained is still reachable after 5 GC cycles

Both fail on both engines, and both pass on the fixed tree.

One note on the second test. runtime.KeepAlive in it is load-bearing, not defensive: without it the compiler may treat the worker as dead after its last use, the queues become collectable along with it, and the cleanup runs no matter what the queue did. It passed against the unfixed code until the KeepAlive went in.

…collected

Four reuse queues across the two loop engines truncate a slice of pointers
with [:0] and reuse the backing array. Truncation does not clear the array,
so every pointer in it stays reachable until some later drain happens to
overwrite that slot -- each queue pins its own high-water mark worth of
objects for the life of the worker.

  engine/iouring/worker.go   detachQueue / detachQSpare   []*connState
  engine/epoll/loop.go       detachQueue / detachQSpare   []*connState
  engine/epoll/adopt.go      adoptQueue / adoptQSpare     []adoptItem
  engine/iouring/driver.go   driverActionQueue / ...Spare []driverAction

The detach queues are the ones that matter. A detached connState carries its
read and write buffers, its H1State, and on the WebSocket path whatever the
middleware hung off that state, so one deep burst leaves megabytes reachable
behind a queue that is nominally empty. drainPendingRelease in the same file
already guards exactly this, with a comment saying why; these four did not.

Retention is bounded by peak queue depth rather than unbounded, so this is
hardening, not a fix for the sustained growth in #573. It is a sibling of
#571 in shape: one guarded copy and several unguarded ones of the same idea.

Each engine gets two guards. The first reads the spare slice out to its
capacity and requires every slot past the new length to be nil. The second
states the same thing as an outcome -- a cleanup on an enqueued conn must
run once the caller drops its reference -- which is what the fix is actually
for. Reverting the clear() alone fails both on both engines:

  drainDetachQueue left 64 of 64 connStates reachable in the reused backing array
  a connState the detach queue already drained is still reachable after 5 GC cycles

runtime.KeepAlive in the second test is load-bearing. Without it the compiler
may treat the worker as dead after its last use, the queues become
collectable with it, and the test passes against the unfixed code -- which it
did, before the KeepAlive went in.
@FumingPower3925
FumingPower3925 merged commit e264f44 into main Sep 11, 2026
10 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/queue-slot-retention branch September 11, 2026 01:35
@FumingPower3925

Copy link
Copy Markdown
Contributor Author

Correcting this PR's own description: it said retention here is bounded by peak queue depth and therefore "hardening, not a fix for the sustained growth in #573". That is wrong, and it was measured wrong by me rather than merely guessed.

Two 50-minute cells cut from a73d306 with and without this one line, same reference app, 206,211 and 206,355 connections accepted (within 0.07%):

without the clear with it
I-MEM-1 violations 1,651 0
heap_inuse trough slope 1,071 B/s 52 B/s
heap_alloc trough slope 728 B/s 29 B/s

Zero violations across 34,350 evaluations, a 20x slope reduction, and io_uring with this line now retains less in absolute terms than epoll does (131 B/s). Full comparison on #573.

The reasoning that led me to undersell it was an assumed hand-off-queue depth of single digits, which caps recoverable memory near 1 MB. The measured effect is about 9 MB over fifty minutes, so that assumption is off by more than an order of magnitude. Measuring the actual depth next, since the same reasoning would have made me dismiss the next bug of this shape.

The change itself needs nothing; it is already merged and correct. Only its stated significance was understated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant