A response can carry another connection's data. On epoll and io_uring (and adaptive, which runs them), an HTTP/1.1 response whose body is 8 KiB or more can go out with bytes that another handler wrote: the next pipelined request's body on the same connection, or another connection's body. The common case is c.JSON (both its reflection-free path and encoding/json), and a buffered response (BufferResponse + FlushResponse); std is not affected. Found in the review of #805 (thanks to the round-1 review); filed publicly, as the maintainer decided for security defects while celeris has no users (no embargo), and fixed in #805.
Mechanism
The H1 response adapter hands a body of 8 KiB or more to the engine's zero-copy body writer (writeBody, internal/conn/response.go) and returns; the writer keeps a reference to the caller's slice and sends it later:
- epoll stages it in
cs.bodyBuf and sends it with writev at the flush after the handler, or much later on EPOLLOUT when the socket is full (the partial-header branch of flushWritesV and an EAGAIN both leave it staged). On a pipelined batch the next handler runs before the flush.
- io_uring puts the slice in a WRITEV SQE's iovec (
flushSend, cs.sendBody = cs.bodyBuf); the kernel reads it only when the ring is next entered, after the handlers of the other connections in the same completion batch have run, and again later if the socket is full.
But the body belongs to the handler, which may reuse it as soon as its write returns, as net/http allows: c.JSON puts its encode buffer back in a pool (jsonFastBufPool, jsonEncPool) right after c.Blob returns, and the next handler to encode, on this connection or another on the same P, takes it and overwrites it; FlushResponse's body is the Context's capturedBody, reused by the next request. internal/conn's SetWriteBodyFn doc even states the opposite contract ("Callers must not mutate the body slice ... until the SEND CQE fires"), which c.JSON never honoured, and context_response.go's own comment says the buffer "is always safe to recycle once Blob returns".
Evidence
Tests of #805 (response_body_ownership_linux_test.go), run by evidence/lanes-20260927/WRITE/scripts/controls.sh with the variant files built by 761/r2-variants.sh (linux/arm64 Docker, 4 CPUs, unlimited memlock; log 761/controls/b518200-variants-r2-unl.log, two runs each). The variant r1head is #805's round-1 head (which already fixes the response order of #802, which otherwise hides this) on current main 3e7abba:
TestPipelinedResponsesOwnTheirBodies (three 16 KiB c.JSON requests pipelined in one packet, 10 rounds): epoll, io_uring and adaptive, fast, encjson and buffered: 10 of 10 rounds bad; response 1 carries request 2's body. std: 0.TestConcurrentResponsesOwnTheirBodies/behind-a-large-body (16 connections at once, each pipelining a 2 MiB static body and its own 16 KiB c.JSON body, reading nothing for 100 ms): io_uring 13 of 32 JSON bodies carried another connection's id, in both runs (e.g. "id 1300000 ... carrying id 400000"); epoll 1 of 32 in both runs; adaptive 0 and 1. std: 0.
On main itself the order bug (#802) and the dropped bodies (#761) break these connections first. The round-1 reviewer's probe without any pipelining (64 connections x 200 requests of 16 KiB c.JSON, on base bd17725) got io_uring 9063 of 12800 bodies wrong, the first carrying another connection's body; epoll 2, adaptive 5, std 0. A probe of the same shape on current main follows in a comment.
Fix (in #805)
The engine must not keep a reference to the body once the write returns. epoll's writer now makes the writev in the call and copies what the kernel did not take into writeBuf (the syscall is the one the flush after the handler would have made); io_uring no longer installs a zero-copy writer (its kernel cannot finish the read before the call returns), so the adapter copies the body into the write buffer. The SetWriteBodyFn contract is rewritten to say so.
A response can carry another connection's data. On epoll and io_uring (and adaptive, which runs them), an HTTP/1.1 response whose body is 8 KiB or more can go out with bytes that another handler wrote: the next pipelined request's body on the same connection, or another connection's body. The common case is
c.JSON(both its reflection-free path and encoding/json), and a buffered response (BufferResponse+FlushResponse); std is not affected. Found in the review of #805 (thanks to the round-1 review); filed publicly, as the maintainer decided for security defects while celeris has no users (no embargo), and fixed in #805.Mechanism
The H1 response adapter hands a body of 8 KiB or more to the engine's zero-copy body writer (
writeBody, internal/conn/response.go) and returns; the writer keeps a reference to the caller's slice and sends it later:cs.bodyBufand sends it withwritevat the flush after the handler, or much later onEPOLLOUTwhen the socket is full (the partial-header branch offlushWritesVand anEAGAINboth leave it staged). On a pipelined batch the next handler runs before the flush.flushSend,cs.sendBody = cs.bodyBuf); the kernel reads it only when the ring is next entered, after the handlers of the other connections in the same completion batch have run, and again later if the socket is full.But the body belongs to the handler, which may reuse it as soon as its write returns, as net/http allows:
c.JSONputs its encode buffer back in a pool (jsonFastBufPool,jsonEncPool) right afterc.Blobreturns, and the next handler to encode, on this connection or another on the same P, takes it and overwrites it;FlushResponse's body is the Context'scapturedBody, reused by the next request.internal/conn'sSetWriteBodyFndoc even states the opposite contract ("Callers must not mutate the body slice ... until the SEND CQE fires"), whichc.JSONnever honoured, andcontext_response.go's own comment says the buffer "is always safe to recycle once Blob returns".Evidence
Tests of #805 (
response_body_ownership_linux_test.go), run byevidence/lanes-20260927/WRITE/scripts/controls.shwith the variant files built by761/r2-variants.sh(linux/arm64 Docker, 4 CPUs, unlimited memlock; log761/controls/b518200-variants-r2-unl.log, two runs each). The variantr1headis #805's round-1 head (which already fixes the response order of #802, which otherwise hides this) on current main 3e7abba:TestPipelinedResponsesOwnTheirBodies(three 16 KiBc.JSONrequests pipelined in one packet, 10 rounds): epoll, io_uring and adaptive,fast,encjsonandbuffered: 10 of 10 rounds bad; response 1 carries request 2's body. std: 0.TestConcurrentResponsesOwnTheirBodies/behind-a-large-body(16 connections at once, each pipelining a 2 MiB static body and its own 16 KiBc.JSONbody, reading nothing for 100 ms): io_uring 13 of 32 JSON bodies carried another connection's id, in both runs (e.g. "id 1300000 ... carrying id 400000"); epoll 1 of 32 in both runs; adaptive 0 and 1. std: 0.On main itself the order bug (#802) and the dropped bodies (#761) break these connections first. The round-1 reviewer's probe without any pipelining (64 connections x 200 requests of 16 KiB
c.JSON, on base bd17725) got io_uring 9063 of 12800 bodies wrong, the first carrying another connection's body; epoll 2, adaptive 5, std 0. A probe of the same shape on current main follows in a comment.Fix (in #805)
The engine must not keep a reference to the body once the write returns. epoll's writer now makes the
writevin the call and copies what the kernel did not take intowriteBuf(the syscall is the one the flush after the handler would have made); io_uring no longer installs a zero-copy writer (its kernel cannot finish the read before the call returns), so the adapter copies the body into the write buffer. TheSetWriteBodyFncontract is rewritten to say so.