Skip to content

Recreate a dead Kubernetes workspace pod on devpod up - #27

Merged
renatoathaydes merged 1 commit into
mainfrom
claude/recreate-failed-pod-734ed7
Oct 20, 2026
Merged

renatoathaydes merged 1 commit into
mainfrom
claude/recreate-failed-pod-734ed7

Conversation

@renatoathaydes

Copy link
Copy Markdown
Collaborator

Bug

With the kubernetes provider, devpod up reused an existing workspace pod whenever the provider options were unchanged, even when the pod was dead. Workspace pods have restart policy Never, so after a node restart (seen when restarting Docker Desktop) the pod stays in phase Failed with its container terminated (exit code 255, reason Unknown). devpod up then:

  1. FindDevContainer reports the pod as exited, so StartDevContainer runs runContainer, which prints Pod '...' already exists and nothing changed, skipping update;
  2. RunDevContainer(nil) (the reprovisioning path) does the same, so the message appears twice;
  3. agent injection waits on the dead pod for 5 minutes and fails with inject agent: timeout waiting for instance connection: EOF.

Fix

runContainer (pkg/driver/kubernetes/run.go) now checks the existing pod with deadPodReason first. When the pod's phase is Failed, Succeeded or Unknown, its devpod container has terminated, or it is being deleted, the pod is deleted and recreated, keeping the workspace's PVC (as when the provider options change). A healthy running or pending pod is reused as before. Terminated sidecar containers from a pod template don't count.

Testing

  • TestDeadPodReason unit test covering running, pending, sidecar, Failed, Succeeded, Unknown, terminated dev container and terminating pods.
  • go test ./pkg/driver/kubernetes/..., go build ./..., golangci-lint on the package.
  • End to end on Docker Desktop's Kubernetes: created a throwaway workspace, killed the pod's PID 1 so the pod went to Failed with the container terminated, then ran devpod up with this build. It logged Pod '...' is in phase Failed, recreating it, created the pod again and finished in about 2 s; the workspace files on the PVC were still there and devpod ssh worked. Another devpod up on the healthy pod reused it. The workspace was deleted afterwards.

🤖 Generated with Claude Code

The kubernetes driver reused an existing workspace pod whenever the
provider options were unchanged, even when the pod was dead. Workspace
pods have restart policy Never, so after e.g. a node restart the pod
stays in phase Failed with its container terminated, and devpod up
skipped the update and then timed out injecting the agent.

runContainer now deletes and recreates a pod whose phase is Failed,
Succeeded or Unknown, whose dev container has terminated, or that is
being deleted. The PVC is kept, as when the options change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@renatoathaydes
renatoathaydes merged commit cb27d52 into main Oct 20, 2026
17 checks passed
@renatoathaydes
renatoathaydes deleted the claude/recreate-failed-pod-734ed7 branch October 20, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant