Skip to content

HTTP/2 streams on the shared H2 worker pool are not drained at shutdown: an async-route stream loses its response on epoll, io_uring and adaptive, and std waits for no h2c stream #759

Description

@FumingPower3925

Found in the review of #746 (celeris#703). Pre-existing on main: measured on 698bed6.

Defect

A graceful shutdown does not wait for HTTP/2 stream handlers that run on the shared H2 worker pool, and on epoll, io_uring and adaptive it cuts their responses off.

  • epoll, io_uring and adaptive, a stream on an async route (Route.Async() / RouteGroup.Async()). canRunInline (protocol/h2/stream/processor.go) returns false for such a stream, so runHandler submits it to globalH2Pool. At shutdown, Loop.shutdown (engine/epoll/loop.go) and Worker.shutdown (engine/iouring/worker.go) only cancel the conn's streams (conn.CloseH2, then Manager.Close). Their asyncWG does not track pool handlers, and the loop then closes the fd. The handler keeps running, its response goes nowhere, and the client gets unexpected EOF. The OnShutdown hooks run before the handler has finished.
  • std, every h2c stream. std serves h2c through h2c.NewHandler (engine/std/engine.go), which hijacks the connection. http.Server.Shutdown stops tracking a hijacked connection, so it does not wait for the stream. The response is delivered, but the hooks run before the handler has finished.

What the drain does cover: HTTP/1.1, and on epoll, io_uring and adaptive an H2 stream whose handler runs inline on the worker (a sync route, END_STREAM). #746 scopes its godoc to that.

Measurement

The request is held in its handler for a fixed 400 ms after the shutdown starts. The release does not depend on the hook, so the result does not depend on when the hook runs. h2c prior knowledge, one request, ShutdownTimeout 30 s, linux/arm64 Docker, unlimited memlock. REPRO-H2 lines, main 698bed6:

engine route Shutdown: hook before handler / response cancel: hook before handler / response
std sync yes / 200 yes / 200
std async yes / 200 yes / 200
epoll async yes / unexpected EOF yes / unexpected EOF
io_uring async yes / unexpected EOF yes / unexpected EOF
adaptive async yes / unexpected EOF yes / unexpected EOF
adaptive sync no / 200 no / 200

On main, epoll and io_uring also run the hooks first on the sync route: that part is #703 itself, which #746 fixes. The lost async-route response is independent of #746. The same review measured it at #746's head 76c205c, with a hook-released handler, in all 8 epoll/io_uring async-route cases.

Script: lane evidence lanes-20260927/LIFECYCLE/round2/repro/run-repro.sh <ref> (test file drain_gaps_repro_linux_test.go, TestReproH2PoolDrain). Log: round2/logs/repro-698bed6-unlimited.log.

Fix direction (not measured)

  • Native engines: count the pool handlers per connection, for example a counter on the Processor incremented in runHandler before Submit and decremented in executeHandler's defer. At shutdown, send GOAWAY and keep the connection's write path running until they have finished, instead of cancelling them. A plain join would deadlock on a handler blocked on flow control, because only the loop reads WINDOW_UPDATE. That makes this the H2 half of graceful shutdown (net/http: GOAWAY, then wait for the streams). The counter is on the async-stream path, so it needs a benchmark.
  • std: give the h2c server graceful shutdown (http2.ConfigureServer registers GOAWAY on RegisterOnShutdown), and have Engine.Shutdown wait, bounded by ctx, for the h2c connections it hijacked. It must not wait for WebSocket hijacks.
  • Test: add std-h2c and async-route cases to TestShutdownHooksRunAfterTheDrain (fix(server): Shutdown runs the OnShutdown hooks, and returns, only after the drain on every engine (#703) #746 adds the sync-route H2 cases on the native engines).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/engineEngine interface or implementationbugSomething isn't workingengine/epollEpoll engine specificsengine/iouringio_uring engine specificsprotocol/h2HTTP/2 protocol

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions