Skip to content

io_uring: a shutdown that finds a SEND stalled on a client that does not read returns ~10 s late: the 250 ms send-drain bound is not enforced while the worker waits in SubmitAndWait #806

Description

@FumingPower3925

Found while working on #760. Pre-existing on main (the #760 branch does not touch io_uring).

Defect

When a shutdown finds a response still being sent to a client that does not read, io_uring's Listen (and so the Start* call) returns about 10 s later, whatever the shutdown's budget. The send drain Worker.run runs after its context is cancelled (#595) is meant to give up after shutdownSendDrainNanos = 250 ms, but the bound is only checked at the top of a loop iteration. When the worker submits the rest of a partly sent response, the iteration waits in SubmitAndWait with no timeout ("Mode 3a: SEND SQEs pending — guaranteed CQE on completion", engine/iouring/worker.go), and a SEND to a peer that does not read does not complete, so nothing wakes the worker until some other completion. (Read from the code, not traced. The ~10 s observed matches two 5 s constants, closingDrainTimeoutNanos and pendingReleaseHoldNanos; which completion ends the wait was not traced either.)

The hooks are not delayed: Server.Shutdown waits for Listen only until its ctx is done. The Start* call is, and a main that exits when it returns exits ~10 s late.

Measurement

A 3 MiB response queued to a raw client with a 64 KiB receive buffer that never reads; the shutdown begins 100 ms after the handler returned; time until the StartWithContext call returns (TestProbe760StalledReaderShutdown, linux/arm64 Docker, unlimited memlock, tree = the #760 branch at 96cbfcd, whose io_uring code is main's):

engine direct Shutdown, budget 500 ms / 2 s cancel, ShutdownTimeout 500 ms / 2 s
io_uring 10.164 s / 10.153 s 10.152 s / 10.154 s
epoll (with #760) 503 ms / 2.001 s 505 ms / 2.002 s
adaptive (with #760) 501 ms / 2.001 s 503 ms / 2.001 s

(std's cancel rows in the same run, 59.95 s and more than 60 s, are #753, fixed by #803: the #760 branch does not have it.)

Script: lane evidence lanes-20260927/WRITE/760/run-probe.sh 96cbfcd unlimited (760/probe760_linux_test.go). Log: 760/logs/probe-96cbfcd-unlimited.log.

Fix direction

Bound the wait of a drain iteration by the time left to shutdownDrainDeadline (use the timed wait while ctx.Err() != nil), so the bound is enforced when nothing completes. #760 gives epoll's send drain the budget of the last Engine.Shutdown (never less than 250 ms); io_uring could take the same bound, so the two engines agree.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/engineEngine interface or implementationbugSomething isn't workingengine/iouringio_uring engine specifics

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions