fix(runner,docker,podman): guard containerDied read and release container-wait goroutines on stop#288
Open
damilolaedwards wants to merge 1 commit into
Conversation
…iner-wait goroutines on stop runContainerLifecycle read containerDied without the mutex the death-monitor goroutine writes it under, right before deciding whether to return an error to the caller. Torn visibility there could let a multi-genesis run continue past a group whose container actually died. Now reads it under the same lock as every other access in the function. Separately, both the Docker and Podman container managers already had a wg/done pair meant for exactly this, but WaitForContainerExit never used it: the goroutine it spawns only watched the caller's context, so calling Stop() before that context was cancelled left it (and its underlying wait call) running past Stop() returning. Both managers now derive the wait context through a helper that also cancels on manager shutdown, and track the watcher goroutine on the manager's WaitGroup so Stop() actually waits for everything it spawned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related concurrency issues in the container lifecycle path:
runContainerLifecycle read containerDied without the mutex the death-monitor goroutine writes it under, right before deciding whether to return an error to the caller. The multi-genesis loop uses that return value to decide whether to stop or continue to the next group, so a torn read there could let it continue past a group whose container actually died.
Both the Docker and Podman container managers already had a wg/done pair meant for tracking internally-spawned goroutines, but WaitForContainerExit never used it. The goroutine it spawns only watched the caller's context, so calling Stop() before that context was cancelled left it, and its underlying wait call, running past Stop() returning.
How
Test plan
go test -race ./pkg/docker/... ./pkg/podman/... ./pkg/runner/...passes,go vetandgofmtclean.