Skip to content

fix(runner,docker,podman): guard containerDied read and release container-wait goroutines on stop#288

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/runner-container-wait-lifecycle
Open

fix(runner,docker,podman): guard containerDied read and release container-wait goroutines on stop#288
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/runner-container-wait-lifecycle

Conversation

@damilolaedwards

Copy link
Copy Markdown
Contributor

What

Two related concurrency issues in the container lifecycle path:

  1. runContainerLifecycle read containerDied without the mutex the death-monitor goroutine writes it under, right before deciding whether to return an error to the caller. The multi-genesis loop uses that return value to decide whether to stop or continue to the next group, so a torn read there could let it continue past a group whose container actually died.

  2. Both the Docker and Podman container managers already had a wg/done pair meant for tracking internally-spawned goroutines, but WaitForContainerExit never used it. The goroutine it spawns only watched the caller's context, so calling Stop() before that context was cancelled left it, and its underlying wait call, running past Stop() returning.

How

  • The final containerDied check now reads under the same mutex as every other access to that variable in the function.
  • Both managers now derive their wait context through a helper (ctxWithDone in docker, connWithCtx in podman) that also cancels on manager shutdown, and track the watcher goroutine on the manager's WaitGroup so Stop() actually waits for it. In podman's case this is the same helper nearly every call already goes through, so the fix covers WaitForContainerExit without a special case and hardens the other call sites too.

Test plan

  • New tests directly exercise the fixed mechanism in both packages: the derived context cancels on manager-done and on parent-context-done, the watcher goroutine does not leak after an explicit cancel, and wg.Wait() returns promptly after done closes.
  • containerDied does not get a new dedicated test. It is a local variable inside a very large function with Docker/executor dependencies this suite does not mock, and a synthetic reproduction of the pattern would not exercise the real code, so it would not catch a regression. The fix is a mechanical one-line change reusing a mutex already used four other times in the same function.
  • go test -race ./pkg/docker/... ./pkg/podman/... ./pkg/runner/... passes, go vet and gofmt clean.
  • No Docker daemon was available in this environment, so this could not be verified against a live container the way the API-based fixes were. Verification here is direct unit testing of the actual synchronization mechanism rather than an end-to-end run.

…iner-wait goroutines on stop

runContainerLifecycle read containerDied without the mutex the
death-monitor goroutine writes it under, right before deciding whether to
return an error to the caller. Torn visibility there could let a
multi-genesis run continue past a group whose container actually died.
Now reads it under the same lock as every other access in the function.

Separately, both the Docker and Podman container managers already had a
wg/done pair meant for exactly this, but WaitForContainerExit never used
it: the goroutine it spawns only watched the caller's context, so calling
Stop() before that context was cancelled left it (and its underlying
wait call) running past Stop() returning. Both managers now derive the
wait context through a helper that also cancels on manager shutdown, and
track the watcher goroutine on the manager's WaitGroup so Stop() actually
waits for everything it spawned.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant