Repository navigation
Recreate a dead Kubernetes workspace pod on devpod up - #27
Merged
Merged
Conversation
The kubernetes driver reused an existing workspace pod whenever the provider options were unchanged, even when the pod was dead. Workspace pods have restart policy Never, so after e.g. a node restart the pod stays in phase Failed with its container terminated, and devpod up skipped the update and then timed out injecting the agent. runContainer now deletes and recreates a pod whose phase is Failed, Succeeded or Unknown, whose dev container has terminated, or that is being deleted. The PVC is kept, as when the options change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
With the
kubernetesprovider,devpod upreused an existing workspace pod whenever the provider options were unchanged, even when the pod was dead. Workspace pods have restart policyNever, so after a node restart (seen when restarting Docker Desktop) the pod stays in phaseFailedwith its containerterminated(exit code 255, reasonUnknown).devpod upthen:FindDevContainerreports the pod asexited, soStartDevContainerrunsrunContainer, which printsPod '...' already exists and nothing changed, skipping update;RunDevContainer(nil)(the reprovisioning path) does the same, so the message appears twice;inject agent: timeout waiting for instance connection: EOF.Fix
runContainer(pkg/driver/kubernetes/run.go) now checks the existing pod withdeadPodReasonfirst. When the pod's phase isFailed,SucceededorUnknown, itsdevpodcontainer has terminated, or it is being deleted, the pod is deleted and recreated, keeping the workspace's PVC (as when the provider options change). A healthy running or pending pod is reused as before. Terminated sidecar containers from a pod template don't count.Testing
TestDeadPodReasonunit test covering running, pending, sidecar, Failed, Succeeded, Unknown, terminated dev container and terminating pods.go test ./pkg/driver/kubernetes/...,go build ./..., golangci-lint on the package.Failedwith the container terminated, then randevpod upwith this build. It loggedPod '...' is in phase Failed, recreating it, created the pod again and finished in about 2 s; the workspace files on the PVC were still there anddevpod sshworked. Anotherdevpod upon the healthy pod reused it. The workspace was deleted afterwards.🤖 Generated with Claude Code