You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
HTTP/2 streams on the shared H2 worker pool are not drained at shutdown: an async-route stream loses its response on epoll, io_uring and adaptive, and std waits for no h2c stream #759
Found in the review of #746 (celeris#703). Pre-existing on main: measured on 698bed6.
Defect
A graceful shutdown does not wait for HTTP/2 stream handlers that run on the shared H2 worker pool, and on epoll, io_uring and adaptive it cuts their responses off.
epoll, io_uring and adaptive, a stream on an async route (Route.Async() / RouteGroup.Async()). canRunInline (protocol/h2/stream/processor.go) returns false for such a stream, so runHandler submits it to globalH2Pool. At shutdown, Loop.shutdown (engine/epoll/loop.go) and Worker.shutdown (engine/iouring/worker.go) only cancel the conn's streams (conn.CloseH2, then Manager.Close). Their asyncWG does not track pool handlers, and the loop then closes the fd. The handler keeps running, its response goes nowhere, and the client gets unexpected EOF. The OnShutdown hooks run before the handler has finished.
std, every h2c stream. std serves h2c through h2c.NewHandler (engine/std/engine.go), which hijacks the connection. http.Server.Shutdown stops tracking a hijacked connection, so it does not wait for the stream. The response is delivered, but the hooks run before the handler has finished.
What the drain does cover: HTTP/1.1, and on epoll, io_uring and adaptive an H2 stream whose handler runs inline on the worker (a sync route, END_STREAM). #746 scopes its godoc to that.
Measurement
The request is held in its handler for a fixed 400 ms after the shutdown starts. The release does not depend on the hook, so the result does not depend on when the hook runs. h2c prior knowledge, one request, ShutdownTimeout 30 s, linux/arm64 Docker, unlimited memlock. REPRO-H2 lines, main 698bed6:
engine
route
Shutdown: hook before handler / response
cancel: hook before handler / response
std
sync
yes / 200
yes / 200
std
async
yes / 200
yes / 200
epoll
async
yes / unexpected EOF
yes / unexpected EOF
io_uring
async
yes / unexpected EOF
yes / unexpected EOF
adaptive
async
yes / unexpected EOF
yes / unexpected EOF
adaptive
sync
no / 200
no / 200
On main, epoll and io_uring also run the hooks first on the sync route: that part is #703 itself, which #746 fixes. The lost async-route response is independent of #746. The same review measured it at #746's head 76c205c, with a hook-released handler, in all 8 epoll/io_uring async-route cases.
Native engines: count the pool handlers per connection, for example a counter on the Processor incremented in runHandler before Submit and decremented in executeHandler's defer. At shutdown, send GOAWAY and keep the connection's write path running until they have finished, instead of cancelling them. A plain join would deadlock on a handler blocked on flow control, because only the loop reads WINDOW_UPDATE. That makes this the H2 half of graceful shutdown (net/http: GOAWAY, then wait for the streams). The counter is on the async-stream path, so it needs a benchmark.
std: give the h2c server graceful shutdown (http2.ConfigureServer registers GOAWAY on RegisterOnShutdown), and have Engine.Shutdown wait, bounded by ctx, for the h2c connections it hijacked. It must not wait for WebSocket hijacks.
Found in the review of #746 (celeris#703). Pre-existing on main: measured on 698bed6.
Defect
A graceful shutdown does not wait for HTTP/2 stream handlers that run on the shared H2 worker pool, and on epoll, io_uring and adaptive it cuts their responses off.
Route.Async()/RouteGroup.Async()).canRunInline(protocol/h2/stream/processor.go) returns false for such a stream, sorunHandlersubmits it toglobalH2Pool. At shutdown,Loop.shutdown(engine/epoll/loop.go) andWorker.shutdown(engine/iouring/worker.go) only cancel the conn's streams (conn.CloseH2, thenManager.Close). TheirasyncWGdoes not track pool handlers, and the loop then closes the fd. The handler keeps running, its response goes nowhere, and the client getsunexpected EOF. The OnShutdown hooks run before the handler has finished.h2c.NewHandler(engine/std/engine.go), which hijacks the connection.http.Server.Shutdownstops tracking a hijacked connection, so it does not wait for the stream. The response is delivered, but the hooks run before the handler has finished.What the drain does cover: HTTP/1.1, and on epoll, io_uring and adaptive an H2 stream whose handler runs inline on the worker (a sync route, END_STREAM). #746 scopes its godoc to that.
Measurement
The request is held in its handler for a fixed 400 ms after the shutdown starts. The release does not depend on the hook, so the result does not depend on when the hook runs. h2c prior knowledge, one request,
ShutdownTimeout30 s, linux/arm64 Docker, unlimited memlock.REPRO-H2lines, main 698bed6:On main, epoll and io_uring also run the hooks first on the sync route: that part is #703 itself, which #746 fixes. The lost async-route response is independent of #746. The same review measured it at #746's head 76c205c, with a hook-released handler, in all 8 epoll/io_uring async-route cases.
Script: lane evidence
lanes-20260927/LIFECYCLE/round2/repro/run-repro.sh <ref>(test filedrain_gaps_repro_linux_test.go,TestReproH2PoolDrain). Log:round2/logs/repro-698bed6-unlimited.log.Fix direction (not measured)
runHandlerbeforeSubmitand decremented inexecuteHandler's defer. At shutdown, send GOAWAY and keep the connection's write path running until they have finished, instead of cancelling them. A plain join would deadlock on a handler blocked on flow control, because only the loop reads WINDOW_UPDATE. That makes this the H2 half of graceful shutdown (net/http: GOAWAY, then wait for the streams). The counter is on the async-stream path, so it needs a benchmark.http2.ConfigureServerregisters GOAWAY onRegisterOnShutdown), and haveEngine.Shutdownwait, bounded by ctx, for the h2c connections it hijacked. It must not wait for WebSocket hijacks.TestShutdownHooksRunAfterTheDrain(fix(server): Shutdown runs the OnShutdown hooks, and returns, only after the drain on every engine (#703) #746 adds the sync-route H2 cases on the native engines).