fix(engine): clear the hand-off queue slots so a drained conn can be collected - #574
Conversation
…collected Four reuse queues across the two loop engines truncate a slice of pointers with [:0] and reuse the backing array. Truncation does not clear the array, so every pointer in it stays reachable until some later drain happens to overwrite that slot -- each queue pins its own high-water mark worth of objects for the life of the worker. engine/iouring/worker.go detachQueue / detachQSpare []*connState engine/epoll/loop.go detachQueue / detachQSpare []*connState engine/epoll/adopt.go adoptQueue / adoptQSpare []adoptItem engine/iouring/driver.go driverActionQueue / ...Spare []driverAction The detach queues are the ones that matter. A detached connState carries its read and write buffers, its H1State, and on the WebSocket path whatever the middleware hung off that state, so one deep burst leaves megabytes reachable behind a queue that is nominally empty. drainPendingRelease in the same file already guards exactly this, with a comment saying why; these four did not. Retention is bounded by peak queue depth rather than unbounded, so this is hardening, not a fix for the sustained growth in #573. It is a sibling of #571 in shape: one guarded copy and several unguarded ones of the same idea. Each engine gets two guards. The first reads the spare slice out to its capacity and requires every slot past the new length to be nil. The second states the same thing as an outcome -- a cleanup on an enqueued conn must run once the caller drops its reference -- which is what the fix is actually for. Reverting the clear() alone fails both on both engines: drainDetachQueue left 64 of 64 connStates reachable in the reused backing array a connState the detach queue already drained is still reachable after 5 GC cycles runtime.KeepAlive in the second test is load-bearing. Without it the compiler may treat the worker as dead after its last use, the queues become collectable with it, and the test passes against the unfixed code -- which it did, before the KeepAlive went in.
|
Correcting this PR's own description: it said retention here is bounded by peak queue depth and therefore "hardening, not a fix for the sustained growth in #573". That is wrong, and it was measured wrong by me rather than merely guessed. Two 50-minute cells cut from
Zero violations across 34,350 evaluations, a 20x slope reduction, and io_uring with this line now retains less in absolute terms than epoll does (131 B/s). Full comparison on #573. The reasoning that led me to undersell it was an assumed hand-off-queue depth of single digits, which caps recoverable memory near 1 MB. The measured effect is about 9 MB over fifty minutes, so that assumption is off by more than an order of magnitude. Measuring the actual depth next, since the same reasoning would have made me dismiss the next bug of this shape. The change itself needs nothing; it is already merged and correct. Only its stated significance was understated. |
Four reuse queues across the two loop engines truncate a slice of pointers with
[:0]and reuse the backing array. Truncation does not clear the array, so every pointer in it stays reachable until some later drain happens to overwrite that slot. Each queue pins its own high-water mark worth of objects for as long as the worker lives.engine/iouring/worker.godetachQueue/detachQSpare[]*connStateengine/epoll/loop.godetachQueue/detachQSpare[]*connStateengine/epoll/adopt.goadoptQueue/adoptQSpare[]adoptItemengine/iouring/driver.godriverActionQueue/driverActionSpare[]driverActionThe two detach queues are the ones that matter. A detached
connStatecarries its read and write buffers, itsH1State, and on the WebSocket path whatever the middleware hung off that state, so one deep burst leaves megabytes reachable behind a queue that is nominally empty.drainPendingRelease, in the same file as the io_uring one, already guards exactly this:These four never got it. Same shape as #571: one guarded copy of an idea and several unguarded ones.
What this is not
Retention here is bounded by peak queue depth, not unbounded, so this is hardening and not a fix for #573. I found it while reading the detach paths for that leak and am landing it on its own merits rather than attaching it to a cause I have not yet established.
Tests
Two guards per engine. The first reads the spare slice out to its capacity and requires every slot past the new length to be nil. The second states the same guarantee as an outcome — a cleanup registered on an enqueued conn must run once the caller drops its reference — which is what the fix is actually for.
detachClosedshort-circuits the drain body, so both exercise the realdrainDetachQueuewithout needing a ring or an epoll fd.Negative control, reverting only the
clear()and keeping everything else:Both fail on both engines, and both pass on the fixed tree.
One note on the second test.
runtime.KeepAlivein it is load-bearing, not defensive: without it the compiler may treat the worker as dead after its last use, the queues become collectable along with it, and the cleanup runs no matter what the queue did. It passed against the unfixed code until theKeepAlivewent in.