fix(dockerutil): drop AutoRemove on one-off helper containers

[whalelephant]

Problem

Three short-lived busybox helper containers in dockerutil run with HostConfig.AutoRemove: true:

• volumeowner.go — chowns a chain's data volume
• filewriter.go — writes files (config/genesis) into the volume
• fileretriever.go — reads files back out

Two of them (volumeowner, filewriter) start the container and then call ContainerWait(..., WaitConditionNotRunning). Their command (a chown) exits in microseconds. With AutoRemove on, the daemon removes the container the moment its process exits — and on some daemons that removal wins the race against the wait being registered, so ContainerWait returns No such container. That fails SetVolumeOwner / the file write, and with it any chain boot.

Concretely this reproduces on Docker Desktop with Engine 28.3.0 (macOS) — boot panics first at set volume owner, then (after fixing that helper) at overwriting config/config.toml — while the same code passes on the older Engine used in CI. It's a timing difference, not a code difference.

Root cause

This is a wait-ordering race, not a Docker version regression. The moby Go client documents that ContainerWait returns its channels only after the wait request has been acknowledged by the daemon, specifically so callers can establish the wait before starting the container:

ContainerWait blocks until the request has been acknowledged by the server (with a response header), then returns two channels on which the caller can wait for the exit status of the container or an error… This allows the caller to synchronize ContainerWait with other calls, such as specifying a "next-exit" condition before issuing a ContainerStart request.
— moby client/container_wait.go

These helpers do the opposite (start, then wait), so for a near-instant command the exit + AutoRemove can complete before the wait attaches. AutoRemove (daemon-side removal on process exit) has existed unchanged since API v1.25 / Docker 1.13, so this is a latent bug that newer daemons' removal timing simply exposes — same race, provider-independent (cf. podman #25479).

Fix

Drop AutoRemove from all three helpers and let the existing deferred ContainerRemove(..., Force: true) clean up the stopped container. This closes the race (the container persists until the wait observes its exit) while still guaranteeing removal. fileretriever never starts its container (it only CopyFromContainers the created filesystem), so it was already race-free — changed only for uniformity.

Verification

TestDchainBoots (dchain interchaintest) boots the 3-validator network to height 5 and PASSes on Docker Engine 28.3.0 / Docker Desktop (macOS), where it previously panicked at set volume owner.

References

• moby client/container_wait.go — documents the wait-before-start pattern the helpers violate
• Docker Engine API version history — AutoRemove added in API v1.25
• Docker Engine 25.0 release notes — the only AutoRemove change in 25.0 is live-restore-on-restart (unrelated to this race)
• podman #25479 — the same auto-remove + wait race, on a different runtime

[Copilot]

Pull request overview

This PR fixes a Docker wait-ordering race in dockerutil one-off helper containers by removing daemon-side AutoRemove so containers persist long enough for ContainerWait(..., NotRunning) to reliably observe exit, and then relying on explicit ContainerRemove(..., Force: true) cleanup.

Changes:

• Removed HostConfig.AutoRemove: true from the volume owner helper container to avoid ContainerWait racing container removal.
• Removed HostConfig.AutoRemove: true from the file writer helper container for the same ContainerWait/removal race.
• Removed HostConfig.AutoRemove: true from the file retriever helper container for uniformity (even though it is never started).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
dockerutil/volumeowner.go	Drops AutoRemove to prevent helper container removal racing ContainerWait.
dockerutil/filewriter.go	Drops AutoRemove to prevent helper container removal racing ContainerWait.
dockerutil/fileretriever.go	Drops AutoRemove for uniformity across helper containers.

Suppressed comments (2)

dockerutil/filewriter.go:82

• The deferred ContainerRemove uses the same ctx that can be canceled (the select below explicitly returns ctx.Err()). If ctx is canceled, this cleanup call will likely fail with "context canceled" and leave the helper container behind. Consider using a non-cancelable cleanup context (with a short timeout) for the remove step so the container is reliably reaped even on test timeouts/cancellation.

	defer func() {
		if err := w.cli.ContainerRemove(ctx, cc.ID, types.ContainerRemoveOptions{
			Force: true,
		}); err != nil {
			w.log.Warn("Failed to remove file content container", zap.String("container_id", cc.ID), zap.Error(err))
		}

dockerutil/volumeowner.go:81

• The deferred ContainerRemove uses the same ctx that can be canceled (the select below explicitly returns ctx.Err()). If ctx is canceled, this cleanup call will likely fail with "context canceled" and leave the helper container behind. Consider using a non-cancelable cleanup context (with a short timeout) for the remove step so the container is reliably reaped even on test timeouts/cancellation.

	defer func() {
		if err := opts.Client.ContainerRemove(ctx, cc.ID, types.ContainerRemoveOptions{
			Force: true,
		}); err != nil {
			opts.Log.Warn("Failed to remove volume-owner container", zap.String("container_id", cc.ID), zap.Error(err))
		}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.