Skip to content

[Bug]: workspace up can't restart a Kubernetes workspace whose pod has terminated #1699

Closed

Description

@wsmoak

Before submitting

  • I have searched existing issues to confirm this is not a duplicate
  • I am running the latest version of Devsy

What happened?

When a Kubernetes workspace's pod terminates (PID 1 exits, for example an OOM kill or a
crash), the pod goes Completed and the PVC stays bound. devsy workspace status
correctly reports Stopped, but devsy workspace up fails after about 2 minutes instead
of bringing the workspace back. Deleting the pod by hand makes up work again, and the
PVC contents survive.

Likely cause: FindDevContainer (pkg/driver/kubernetes/find.go) treats any pod not in
phase Running as exited, so status is right. But reconcileExistingPod
(pkg/driver/kubernetes/run.go) finds the existing pod, sees the provider options are
unchanged, logs "Pod already exists and nothing changed, skipping update" and returns
without recreating it. It never checks the pod's phase, and only deletes the pod when
options changed. Agent injection then tries to exec into the terminated container.

What did you expect to happen instead?

workspace up recreates the pod (reusing the existing PVC) and the workspace starts.

Steps to reproduce

  1. devsy provider add kubernetes
  2. devsy provider set kubernetes -o KUBERNETES_CONTEXT=docker-desktop -o KUBERNETES_NAMESPACE=devsy-p3test -o DISK_SIZE=1Gi
  3. devsy workspace up ./p3test --id p3test --provider kubernetes --ide none
  4. kubectl -n devsy-p3test exec <pod> -- kill 1 (simulates PID 1 exiting)
  5. kubectl -n devsy-p3test get pods,pvc shows the pod Completed and the PVC Bound.
  6. devsy workspace status p3test reports "state":"Stopped".
  7. Run the workspace up command from step 3 again. It fails.

devcontainer.json

{ "image": "alpine:3.20" }

Error output / logs

platform-native agent delivery failed: deliver agent (post-start): in-container download failed:
  download in container: Internal error occurred: unable to upgrade connection: container not found ("devsy")
exec-stream delivery failed: Internal error occurred: unable to upgrade connection: container not found ("devsy")
legacy agent injection failed: inject agent: [inject] read ping: EOF

# afterwards
NAME                     READY   STATUS      RESTARTS   AGE
devsy-default-p3-30abe   0/1     Completed   0          2m39s

How often does this happen?

Every time

Operating system

macOS

Linux distribution (if applicable)

N/A (the pod image is alpine:3.20)

Architecture

ARM64

Desktop app or CLI?

CLI only

Devsy version

v1.24.0

Devsy provider

Kubernetes

Provider version

v1.0.0

Screenshots

No response

Anything else?

Suggested fix: in reconcileExistingPod, treat a pod in phase Succeeded or Failed
as needing recreation. Delete it as the options-changed path does (waitPodDeleted) and
fall through to creating a new one.

Any way PID 1 can end (OOM kill, crash, a node drain with restartPolicy: Never) leaves the workspace unrecoverable through Devsy until the pod is deleted.

This also blocks one possible fix for issue #1698 (inactivity timeout), which would otherwise end the pod's main process.

Workaround:

kubectl -n <namespace> delete pod <pod>
devsy workspace up ...

Activity

  1. skevetter commented on Oct 31, 2026

    @skevetter
    Contributor

    Triage result: fixed in v1.25.0-beta.6 and later prereleases

    Reviewed on 30 October 2026 against main at 9bc12ee041ca854c6f81de95fd8c6e49d7cc028c.

    The original terminal-pod restart defect is implemented and merged. PR #1701 merged as b86442407a9ad9854213f08931236d470d783b77. v1.25.0-beta.6 contains that commit; the commit comparison confirms ancestry. The fix is also in the currently published v1.25.0-beta.8. The latest stable release is still v1.24.0, the version in this report, so updating only within the stable channel does not deliver this fix.

    What changed

    The original diagnosis was correct for v1.24.0. The terminal-state decision now runs before the unchanged-provider-options reuse shortcut:

    • classifyPod and reconcileObservedPod classify Succeeded and Failed pods as terminal. A terminated devsy container is also recognized even if another container keeps the pod phase Running.
    • A terminal pod is deleted and awaited before a replacement is created. Ownership validation and UID-preconditioned deletion prevent replacing another workspace's pod or deleting a newly created pod that reused the same name.
    • A healthy running pod with unchanged options is reused. Pending/terminating pods have bounded waits; API lookup errors are not treated as absence. Unknown phase is surfaced instead of deleting a possibly live workspace.
    • The PVC run-information path preserves the existing workspace volume and its identity. Ordinary workspace up is the resume operation; --recreate and manual kubectl delete pod are not needed to solve this report.

    Evidence and closure check

    The existing pod-lifecycle unit tests cover both terminal phases, unchanged-option replacement, running-pod reuse, UID conflicts, ownership rejection, forbidden lookups/deletes, cancellation and PVC preservation. The Kubernetes inactivity/resume E2E suite covers two stop/resume cycles with a preserved file and a separate explicit terminal-pod restart. The Kubernetes inactivity CI job on the final #1701 head completed successfully. PR #1737 subsequently fixed synchronization in a unit-test readiness flag; it does not add another production restart fix and is newer than beta.8.

    For customer confirmation, repeat the original reproduction with beta.6 or later and record:

    1. CLI/agent version, namespace, workspace UID, original Pod UID and PVC UID.
    2. A sentinel written inside the workspace's mounted volume.
    3. The original pod reaching Succeeded or Failed.
    4. A plain devsy workspace up <workspace> succeeding with a different Pod UID, the same PVC/workspace UID, and the sentinel still present.
    5. A second ordinary up preserving that healthy Pod UID.

    There is no outstanding implementation change for the behavior reported here. This issue can be resolved as fixed with the prerelease version identified, or kept open for reporter confirmation under the project's release policy. Do not create another terminal-pod recreation ticket. Files outside mounted volumes are not part of the PVC-preservation contract.

    This review inspected source, release ancestry and recorded CI results; it did not run a new Kubernetes reproduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions