Skip to content

Stop busy-looping and orphaning workers during graceful shutdown - #235

Merged
rafaelfranca merged 6 commits into
Shopify:masterfrom
bscanlan:fix-graceful-shutdown-loop
Nov 19, 2026
Merged

rafaelfranca merged 6 commits into
Shopify:masterfrom
bscanlan:fix-graceful-shutdown-loop

Conversation

@bscanlan

Copy link
Copy Markdown
Contributor

Why?

Since 0cd51fc, a graceful shutdown keeps the monitor on a full CPU core until the workers exit, which can take 22 seconds with the default timeout. A second QUIT or TERM during that drain makes the monitor exit while workers are still serving requests, so nothing kills them when they run past the timeout. A TTIN during the drain also starts new workers that can outlive the monitor.

How?

The shutdown loop now waits up to 0.1 seconds per pass for a child exit, signal or message, and keeps waiting for workers after a repeated QUIT or TERM. TTIN no longer switches respawning back on during a shutdown.

Notes for reviewers

The TTIN change has to land with the loop change. Without it, the faster loop killed and respawned the TTIN worker 371 times in one 3-second drain on macOS.

bscanlan and others added 5 commits November 19, 2026 11:34
The `TTIN` handler set `@respawn = true`. In Unicorn, this line undid
`WINCH`. Pitchfork has no `WINCH`, so `@respawn` is false only during boot
and during shutdown. A `TTIN` during a graceful shutdown made the monitor
spawn new workers, and these workers could outlive the monitor.
Commit 0cd51fc replaced `sleep(0.1)` in the shutdown loop of
`HttpServer#stop` with `monitor_loop(false)`, which does not sleep. While
workers drain, the monitor spins on one CPU core for up to `timeout`
seconds. Because the loop never calls `monitor_sleep`, the monitor also
never reads messages such as `WorkerSpawned` from the control socket.

Wait on the control socket for at most 0.1 seconds per iteration. A child
exit, a signal, or a message wakes the monitor immediately.
`HttpServer#stop` returned early when `monitor_loop` handled a signal that
returns `StopIteration`. For `INT`, the nested `stop(false)` has already
killed the workers. For `QUIT` and `TERM`, nothing has been done yet. A
second `QUIT` or `TERM` during a graceful shutdown made the monitor exit
while workers still handled requests, and the monitor no longer enforced
the hard timeout on them.

Ignore the return value. After a nested `stop(false)`, no children
remain, so the loop ends immediately. A new test makes sure that `INT`
still ends a graceful shutdown immediately.
@rafaelfranca
rafaelfranca force-pushed the fix-graceful-shutdown-loop branch from 0986464 to 74fae88 Compare November 19, 2026 16:35
@rafaelfranca
rafaelfranca merged commit 37d40f0 into Shopify:master Nov 19, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants