Skip to content

Count a timed out spawn once in max_consecutive_spawn_errors - #236

Merged
hmcguire-shopify merged 2 commits into
Shopify:masterfrom
bscanlan:fix-spawn-timeout-error-count
Nov 19, 2026
Merged

hmcguire-shopify merged 2 commits into
Shopify:masterfrom
bscanlan:fix-spawn-timeout-error-count

Conversation

@bscanlan

Copy link
Copy Markdown
Contributor

Why?

The configuration docs say a worker that fails to become ready adds one to the max_consecutive_spawn_errors count. A timed out spawn adds two, once when it times out and again when the killed worker is reaped. With a limit of 4 and one worker, the monitor gives up after 3 timed out spawns instead of 5.

How?

A timeout now adds to the count only for a worker that never registered, because the monitor never reaps those.

Notes for reviewers

The existing timeout test only passed because of the double count, so its limit drops from 4 to 2. That keeps the abort in the first round of spawns, inside the test's 4 second wait.

@bscanlan
bscanlan force-pushed the fix-spawn-timeout-error-count branch from 5a43ee4 to 35e1d30 Compare November 19, 2026 18:21
`hard_timeout` incremented `@consecutive_spawn_errors` for a child that was
not ready. When the monitor reaps the killed child, `reap_worker`
increments the counter again for the same failure. As a result, the
monitor aborted after about half of `max_consecutive_spawn_errors`
timeouts.

Increment the counter in `hard_timeout` only for a child that never
registered. The monitor never reaps such a child.

`test_max_consecutive_spawn_timeout` passed only because of the double
count: 3 timed out spawns counted as 6, which is more than 4. It now sets
the limit to 2, so it still aborts after the first round of spawns.
Comment thread lib/pitchfork/http_server.rb Outdated
@consecutive_spawn_errors += 1 unless child.ready?

if child.pid.nil? # Not yet registered, likely never spawned
@consecutive_spawn_errors += 1 unless child.ready?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if the child.pid.nil? then child can never be ready?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep.

@bscanlan
bscanlan force-pushed the fix-spawn-timeout-error-count branch from 35e1d30 to 4aeb56a Compare November 19, 2026 22:41
A child without a pid has not sent its spawn message, so it cannot be ready yet.
@hmcguire-shopify
hmcguire-shopify merged commit ed95612 into Shopify:master Nov 19, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants