Skip to content

Ignore MoldReady from a mold that was already reaped - #237

Open
bscanlan wants to merge 1 commit into
Shopify:masterfrom
bscanlan:fix-unknown-mold-ready
Open

bscanlan wants to merge 1 commit into
Shopify:masterfrom
bscanlan:fix-unknown-mold-ready

Conversation

@bscanlan

Copy link
Copy Markdown
Contributor

Why?

If a mold exits right after it reports ready, the monitor can reap it before reading that message. Handling the message then raises undefined method 'nr' for an instance of Pitchfork::Message::MoldReady (NoMethodError). Normally the monitor just logs this, but during shutdown the error escapes and the monitor exits without killing the children that are still running.

How?

The child lookup now falls back to the worker number only for messages that have one, and the monitor logs and ignores a ready message from a mold it no longer tracks.

Notes for reviewers

I found this while working on #235. That PR has the shutdown drain read control messages, which makes this easier to hit, but master has the same bug.

The integration test pauses the monitor with SIGSTOP so the mold always exits before its ready message is read. On master it fails with the error above.

If a mold exits right after sending `MoldReady`, `reap_all_workers` can
remove it before the monitor reads the message. `Children#update` then
falls back to a lookup by `nr`, which `MoldReady` doesn't have, and
raises `NoMethodError`. Had it returned nil, the monitor would have
failed on `nil.to_log` instead.

In the main loop the error is rescued and logged. During shutdown it
escapes `stop`, so the monitor exits without killing the children that
are still running.

Only fall back to `nr` for messages that carry one, and have the monitor
log and ignore `MoldReady` for a mold it no longer tracks. The
integration test pauses the monitor with SIGSTOP so the mold always
exits before its `MoldReady` is read.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant