From 9111ae9a7b9a2e183f58dbccce8820e10180e788 Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Mon, 28 Sep 2026 08:46:44 +0200 Subject: [PATCH 1/3] docs: epoll's shutdown now sends what the sockets have not taken yet, until the deadline (celeris#760) --- src/content/docs/graceful-shutdown.md | 37 +++++++++++++++++---------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/src/content/docs/graceful-shutdown.md b/src/content/docs/graceful-shutdown.md index 52394e2..99486e6 100644 --- a/src/content/docs/graceful-shutdown.md +++ b/src/content/docs/graceful-shutdown.md @@ -216,8 +216,9 @@ engine's `Shutdown`. cancelled (the next step). 3. **Cancel the listen context and wait for the drain.** Cancelling it is what stops a running `epoll` or `io_uring` engine: its workers stop accepting, let the handlers - they are running return, close their connections and wait for their async - handlers. `Shutdown` then waits, bounded by `ctx`, until the engine's `Listen` has + they are running return and wait for their async handlers, send what the sockets + have not taken yet (see [What the drain waits for](#what-the-drain-waits-for)), and + close their connections. `Shutdown` then waits, bounded by `ctx`, until the engine's `Listen` has returned, which is when the drain is over (on `std` and `adaptive`, at once: step 2 drained). What that drain covers, and what it does not, is in [What the drain waits for](#what-the-drain-waits-for). @@ -263,21 +264,28 @@ It does not wait for two kinds of HTTP/2 stream server and stops tracking it, so the drain does not wait for it. The hooks can run while the handler is still running; the response is still delivered. -There is one more gap on `epoll`, and on `adaptive` while it runs `epoll`. The drain -is over when the handlers have returned, and each connection is then closed without -flushing what the socket has not taken yet. A response larger than the socket buffers, -to a client that reads slowly, loses its tail -([celeris#760](https://github.com/goceleris/celeris/issues/760)). `io_uring` keeps -sending for up to 250 ms before it closes, and `std` drains through net/http. +Once the handlers have returned, the native engines keep sending what the sockets have +not taken yet before they close the connections, so a response larger than the socket +buffers still reaches a client that reads slowly. `epoll` (and `adaptive` while it runs +`epoll`) keeps sending until the shutdown's deadline (`Config.ShutdownTimeout` after a +cancel, or the `ctx` of a direct `Shutdown`), and never for less than 250 ms; a client +that never reads holds the shutdown that long and no longer. Before celeris v1.6.0 +`epoll` closed each connection as soon as the handlers had returned, and such a +response lost its tail ([celeris#760](https://github.com/goceleris/celeris/issues/760)). +`io_uring` keeps sending for 250 ms whatever the deadline, so a client slower than that +can still lose the tail ([celeris#806](https://github.com/goceleris/celeris/issues/806)), +and `std` drains through net/http. (Measured on each engine with one request in flight at the shutdown. An h2c request on an `.Async()` route got `unexpected EOF` on `epoll`, `io_uring` and `adaptive`, with the hooks run first. On a route that is not async it got its response, with the hooks run after the handler. On `std` the hooks ran before the h2c handler returned, and the response still arrived. A 3 MiB response was written 200 ms into the shutdown, to a -client with a 64 KiB receive buffer that started reading 1 s later. The client got -2,634,240 of its 3,145,849 bytes on `epoll` and `adaptive`, and all of them on `std` -and `io_uring`.) +client with a 64 KiB receive buffer that started reading 1 s later. The client got all +of it on `std`, `epoll` and `adaptive` (before v1.6.0, 2,634,240 of its 3,145,849 bytes +on `epoll` and `adaptive`). On `io_uring` it depends on what the kernel had taken when +the 250 ms ran out: all of it on one machine, 2,634,119 of 3,145,728 body bytes on a CI +runner.) > **The shutdown context is shared across the engine drain *and* every hook.** Within a > single `Shutdown(ctx)` call, the same `ctx` bounds the drain and then flows into each @@ -656,9 +664,10 @@ Celeris stops waiting for them; it does not interrupt them. When the deadline `Shutdown`) passes before the drain is over, the `OnShutdown` hooks run then, with the expired context, and a direct `Shutdown` returns `context.DeadlineExceeded` once they have run. A handler still running keeps running and keeps its connection, and its -response still reaches the client when it finishes. On `epoll` a response larger than -the socket buffers can lose its tail (see -[What the drain waits for](#what-the-drain-waits-for)). `c.Context()` is not cancelled +response still reaches the client when it finishes. The deadline has passed by then, +so on `epoll` and `io_uring` that response gets 250 ms to go out before the connection +closes: one larger than the socket buffers, to a client that reads slowly, can lose its +tail (see [What the drain waits for](#what-the-drain-waits-for)). `c.Context()` is not cancelled at the deadline, so a handler cannot see it there. When the `Start*` call returns differs by engine: From 01ada449350a208d45a55cb2654b6989ef89605e Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Mon, 28 Sep 2026 11:49:28 +0200 Subject: [PATCH 2/3] docs: epoll's shutdown send drain with a ctx that has no deadline: until it is done, no longer than WriteTimeout (celeris#760) --- src/content/docs/graceful-shutdown.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/content/docs/graceful-shutdown.md b/src/content/docs/graceful-shutdown.md index 99486e6..f80f07b 100644 --- a/src/content/docs/graceful-shutdown.md +++ b/src/content/docs/graceful-shutdown.md @@ -267,9 +267,12 @@ It does not wait for two kinds of HTTP/2 stream Once the handlers have returned, the native engines keep sending what the sockets have not taken yet before they close the connections, so a response larger than the socket buffers still reaches a client that reads slowly. `epoll` (and `adaptive` while it runs -`epoll`) keeps sending until the shutdown's deadline (`Config.ShutdownTimeout` after a -cancel, or the `ctx` of a direct `Shutdown`), and never for less than 250 ms; a client -that never reads holds the shutdown that long and no longer. Before celeris v1.6.0 +`epoll`) keeps sending while the shutdown's context is live: until its deadline +(`Config.ShutdownTimeout` after a cancel, or the `ctx` of a direct `Shutdown`), or, for a +`ctx` with no deadline such as `context.Background()`, until that `ctx` is done. It never +sends for longer than `Config.WriteTimeout` (60 s by default) nor for less than 250 ms, +so a client that never reads holds the shutdown that long and no longer, even a +`Shutdown(context.Background())`. Before celeris v1.6.0 `epoll` closed each connection as soon as the handlers had returned, and such a response lost its tail ([celeris#760](https://github.com/goceleris/celeris/issues/760)). `io_uring` keeps sending for 250 ms whatever the deadline, so a client slower than that From 709202871b172be4293a770e3f3f9908b342ac38 Mon Sep 17 00:00:00 2001 From: Albert Bausili Date: Tue, 29 Sep 2026 14:05:22 +0200 Subject: [PATCH 3/3] docs: the send drain's WriteTimeout bound holds only while WriteTimeout is set; io_uring's 250 ms is not enforced on a stalled send (celeris#806) --- src/content/docs/graceful-shutdown.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/src/content/docs/graceful-shutdown.md b/src/content/docs/graceful-shutdown.md index 9b9b324..6bbf91c 100644 --- a/src/content/docs/graceful-shutdown.md +++ b/src/content/docs/graceful-shutdown.md @@ -272,14 +272,19 @@ buffers still reaches a client that reads slowly. `epoll` (and `adaptive` while `epoll`) keeps sending while the shutdown's context is live: until its deadline (`Config.ShutdownTimeout` after a cancel, or the `ctx` of a direct `Shutdown`), or, for a `ctx` with no deadline such as `context.Background()`, until that `ctx` is done. It never -sends for longer than `Config.WriteTimeout` (60 s by default) nor for less than 250 ms, -so a client that never reads holds the shutdown that long and no longer, even a -`Shutdown(context.Background())`. Before celeris v1.6.0 +sends for less than 250 ms, nor, while `Config.WriteTimeout` is set (60 s by default), +for longer than that, so a client that never reads holds the shutdown that long and no +longer, even a `Shutdown(context.Background())`. With `WriteTimeout: -1` (no timeout) +only the context bounds it, and a `Shutdown(context.Background())` waits for as long as +such a client does not read. Before celeris v1.6.0 `epoll` closed each connection as soon as the handlers had returned, and such a response lost its tail ([celeris#760](https://github.com/goceleris/celeris/issues/760)). `io_uring` keeps sending for 250 ms whatever the deadline, so a client slower than that -can still lose the tail ([celeris#806](https://github.com/goceleris/celeris/issues/806)), -and `std` drains through net/http. +can still lose the tail. A send stalled on a client that does not read at all is not cut +at 250 ms, though: the worker waits in the kernel, and `io_uring`'s `Listen`, and with it +the `Start*` call, returns about 10 s later whatever the budget +([celeris#806](https://github.com/goceleris/celeris/issues/806)). `std` drains through +net/http. (Measured on each engine with one request in flight at the shutdown. An h2c request on an `.Async()` route got `unexpected EOF` on `epoll`, `io_uring` and `adaptive`, with the @@ -671,7 +676,8 @@ expired context, and a direct `Shutdown` returns `context.DeadlineExceeded` once have run. A handler still running keeps running and keeps its connection, and its response still reaches the client when it finishes. The deadline has passed by then, so on `epoll` and `io_uring` that response gets 250 ms to go out before the connection -closes: one larger than the socket buffers, to a client that reads slowly, can lose its +closes (on `io_uring` longer when a send is stalled on a client that does not read at +all, [celeris#806](https://github.com/goceleris/celeris/issues/806)): one larger than the socket buffers, to a client that reads slowly, can lose its tail (see [What the drain waits for](#what-the-drain-waits-for)). `c.Context()` is not cancelled at the deadline, so a handler cannot see it there.