Skip to content

Archive gating cannot see remote background bash jobs surviving an unclean restart (Coder stop policy) #3944

Description

@ThomasK33

Context

PR #3940's crash-orphan guard makes model-driven archives fail closed when durable spawn records under /tmp/mux-bashes/<workspaceId> show a background process that survived an unclean Xum restart. That layout is intentionally local-only: for SSH/Coder runtimes, spawnProcess writes meta.json/exit_code through the remote runtime (runtime.tempDir()), so the records live on the remote host.

Problem (Codex review round 18, P2)

Condition: a dedicated Coder target uses the default stop archive policy, a remote background bash job survives an unclean Xum restart, and an orchestrator archives the workspace via task_workspace_lifecycle.

After the restart the in-memory BackgroundProcessManager map is empty and the local crash-orphan probe cannot see the remote records, so both archive gates report no background activity. The before-archive hook then stops the Coder workspace underneath the still-running remote job.

Non-crash flows are already covered (in-memory tracking + fresh status refresh), as are untrackable native terminals/editors under Coder stop (durable local markers).

Suggested directions (design decision needed)

  1. Durable local admission registry for remote spawns: write a session-dir record (mirroring bashMonitorRegistryStore) when spawning on a non-local runtime; remove it when the process is observed exited/terminated/cleaned. Leftover records after a restart fail the archive gate closed. Needs a reconciliation story so a job that exited during the outage does not permanently refuse model-driven archives (e.g. verify via remote kill -0 when the workspace is reachable).
  2. Remote record probing at gate time: exec a liveness check against the remote spawn layout during archive gating. Must not wake stopped Coder workspaces (coder ssh --wait=yes starts them) — probe only when the workspace is already running.

Either way, the refusal should route to user-mediated archive like the existing gates.

References


Generated with mux • Model: anthropic:claude-fable-5 • Thinking: xhigh

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions