Skip to content

[Bug]: ssh provider workspace doesn't start when zsh is the default user shell #1621

Closed

Description

@souryavarenya

Before submitting

  • I have searched existing issues to confirm this is not a duplicate
  • I am running the latest version of Devsy

What happened?

I'm trying to run devsy via ssh provider on one of my servers whose user has default shell zsh

Running into:

ERROR   main:7: command not found: sh -c

Once I have chsh -s /bin/bash on the remote server, it works again.

What did you expect to happen instead?

Launch my editor upon successful copy

Steps to reproduce

  1. setup remote server with zsh as default shell: chsh -s /bin/zsh
  2. Add ssh provider: devsy provider add ssh --name server -o HOST=user@server
  3. Try to launch a workspace: devsy workspace --provider server .

devcontainer.json

Error output / logs

2026-10-20T16:18:03.693+0200    DEBUG   adding SSH keys to agent
2026-10-20T16:18:03.694+0200    DEBUG   running ssh-add: privateKeyPath=/home/sourya/.ssh/id_ed25519
2026-10-20T16:18:03.695+0200    DEBUG   injecting and running SSH server command: "'/tmp/sourya/devsy/agent' internal ssh-server --stdio --debug"
2026-10-20T16:18:03.695+0200    DEBUG   starting agent injection
2026-10-20T16:18:03.695+0200    DEBUG   start inject
2026-10-20T16:18:03.695+0200    DEBUG   prefer downloading agent from URL: url=https://github.com/devsy-org/devsy/releases/download/v1.24.0
2026-10-20T16:18:03.695+0200    DEBUG   execute inject script
2026-10-20T16:18:03.707+0200    DEBUG   tunnel: setup start
2026-10-20T16:18:03.707+0200    DEBUG   creating SSH client
2026-10-20T16:18:03.956+0200    DEBUG   handshake complete elapsed=260.454959ms
2026-10-20T16:18:03.971+0200    DEBUG   done exec
2026-10-20T16:18:03.971+0200    DEBUG   done inject
2026-10-20T16:18:03.972+0200    DEBUG   payload delivered elapsed=276.364386ms
2026-10-20T16:18:03.972+0200    DEBUG   done injecting
2026-10-20T16:18:03.972+0200    DEBUG   complete elapsed=276.640138ms
2026-10-20T16:18:03.972+0200    DEBUG   retrying injection: [inject] 2026-10-20T16:18:03.965+0200       ERROR   main:7: command not found: sh -c


read inject signal: EOF
2026-10-20T16:18:14.663+0200    DEBUG   start inject
2026-10-20T16:18:14.663+0200    DEBUG   prefer downloading agent from URL: url=https://github.com/devsy-org/devsy/releases/download/v1.24.0
2026-10-20T16:18:14.663+0200    DEBUG   execute inject script
2026-10-20T16:18:14.952+0200    DEBUG   handshake complete elapsed=288.553288ms
2026-10-20T16:18:14.966+0200    DEBUG   done exec
2026-10-20T16:18:14.966+0200    DEBUG   done inject
2026-10-20T16:18:14.967+0200    DEBUG   payload delivered elapsed=303.859002ms
2026-10-20T16:18:14.967+0200    DEBUG   done injecting
2026-10-20T16:18:14.967+0200    DEBUG   complete elapsed=304.105295ms
2026-10-20T16:18:14.967+0200    DEBUG   retrying injection: [inject] 2026-10-20T16:18:14.961+0200       ERROR   main:7: command not found: sh -c

How often does this happen?

Every time

Operating system

Linux

Linux distribution (if applicable)

No response

Architecture

AMD64

Desktop app or CLI?

Both

Devsy version

v1.24.0

Devsy provider

SSH

Provider version

v1.1.44

Screenshots

No response

Anything else?

No response

Activity

  1. skevetter commented on Oct 31, 2026

    @skevetter
    Contributor

    Cleanup audit: source fix delivered to prereleases; finish the existing shell qualification

    Status: the original default-Zsh bootstrap defect has been fixed. Core and provider changes are merged and available in release artifacts. The outstanding work is qualification of the corrected pair through existing provider PR #164, plus the stable/customer upgrade checks.

    Reviewed 30 October 2026. Core source baseline: 9bc12ee041ca854c6f81de95fd8c6e49d7cc028c. SSH-provider release source: c53ba2742db162f587aa57fbc87572d19ec11cd7, v1.1.47. This audit read source, release ancestry and workflow records; it did not execute a customer host or change a login shell.

    1. Verified implementation and release state

    Change Verified state and responsibility
    Core #1627, merge e22be00149ace87d512de0ec953541e3f7642c69 Merged 21 October. Quotes bootstrap literals, uses a fixed POSIX installer, and stages replacement agents before atomic rename.
    SSH-provider #152, merge d31fed1b2695861b76e922c530ad8a2436f5acb9 Merged 21 October. Selects /bin/sh before the remote login shell interprets the POSIX bootstrap. Applies to built-in and external SSH transports.
    Core #1725, merge 9c80b29c7672f4cc3bc54553f57c9a915cad7d19 Merged 26 October. Completes literal host-agent-path quoting at workspace startup, tunnel execution, refresh and related callers. This fixes the later hostile-path blocker found during shell qualification.
    SSH-provider v1.1.45 First provider release listing #152. v1.1.47 includes that merge.
    Core v1.25.0-beta.5 Includes the later #1725 fix. Current beta.8 includes both core fixes.
    Core stable v1.24.0 Predates these core fixes. An updated provider alone does not establish the complete fixed core/provider combination.
    Provider PR #164, head 7f5f401e874e7b835119548ab1f21ec3e5c4dfd9 Still open and Draft. The complete qualification harness is implemented. Ordinary PR workflows passed, but no workflow-dispatch release qualification was found for this head.

    The current provider launcher encodes the script as one /bin/sh -c argument, including Fish-specific handling of literal backslashes. It preserves protocol stdin/stdout separately from diagnostics. Core GenerateScript and QuotePOSIXLiteral preserve literal installer inputs. Current workspace startup and host tunnel quote the agent path with the shared shell-escaping convention.

    This addresses the bootstrap interpreter boundary. It preserves the user's chosen interactive shell. Changing chsh, replacing SHELL, editing shell startup files, or wrapping every subsequent interactive command in Bash is not part of the fix.

    2. What remains, and why the old hold is stale

    Provider #164's last coordination comment waits for a corrected published core candidate after #1725. That candidate now exists: beta.5 and later, including beta.8. The old beta.2 failure and the earlier passing plain-path matrix remain historical evidence. Neither proves the current literal-path/real-sudo gate.

    Use the existing PR and this issue. Do not open a second bootstrap implementation or copy the same test suite into core.

    3. Exact engineering tasks

    T1. Complete the candidate gate in provider #164

    Files on the existing PR: .github/workflows/tests.yaml and e2e/README.md.

    • Retain the existing CORE_FIX_SHA=e22be00149ace87d512de0ec953541e3f7642c69 and PROVIDER_FIX_SHA=d31fed1b2695861b76e922c530ad8a2436f5acb9 ancestry checks.
    • Add a second required core ancestry check for 9c80b29c7672f4cc3bc54553f57c9a915cad7d19, the merged host-path correction. Use the same compare-result allowlist, ahead or identical, and fail before fixture provisioning for any other result. Include both core fix identities in pair-verification.txt. This prevents an artifact containing only fix(inject): quote bootstrap inputs and preserve installed agents #1627 from being selected for the broader literal-path acceptance.
    • Update the workflow input help and README's stale beta.3 example to a concrete corrected candidate. For this audit the explicit pair is core v1.25.0-beta.8 at f4d6039acf68bffcba9d1f9ce927cce144b7dc66, provider v1.1.47 at c53ba2742db162f587aa57fbc87572d19ec11cd7. Resolve and record the live tag identities again when dispatching; never silently substitute a different candidate.
    • Preserve independent cli and desktop artifact rows. Beta.8 publishes both devsy-linux-amd64 and Devsy_linux_amd64.deb, matching this workflow's selectors. Keep verification of published asset digest, printed version and the actual extracted opt/Devsy/resources/bin/devsy.
    • Keep the fail-closed missing-candidate check. Keep DEVSY_AGENT_URL, DEVSY_AGENT_BINARY and DEVSY_AGENT_PREFER_DOWNLOAD unset for qualification. A development agent override can bypass the delivery behavior under test.
    • Make the narrow gate/docs update in signed commits on the existing Draft PR and run its applicable required checks. Preserve protections and the existing acceptance predicate.

    T2. Run the existing corrected native qualification

    Owned implementation files:

    • e2e/tests/integration/login_shells.go

    • login_shells_workspace.go

    • login_shells_sudo.go

    • qualification.go

    • Dispatch the existing Tests workflow at the reviewed PR head with explicit core_tag=v1.25.0-beta.8 and provider_tag=v1.1.47. Run in the existing disposable GitHub Actions environment.

    • Require all six shell/transport cells: Bash, Zsh and Fish, each over external and built-in SSH.

    • In every cell retain the five implemented scenarios: cold remote-download install, current-agent reuse, stale replacement, cold stdin-transfer install/reopen, and live workspace SSH/reopen behavior. Require unchanged login-shell identity and owned cleanup.

    • Keep agent paths containing space, apostrophe, backslash, dollar sign and Unicode. Keep exact delivered-binary hashes and the staged atomic stale-fixture replacement. Do not remove the troublesome characters or truncate a running executable.

    • Keep the six command-boundary stream/exit-status cases and the actual non-root sudo scenario. The latter must prove denied ordinary writes to the root-owned directory, successful authorized install, expected root ownership/version/hash, a live workspace command, and cleanup of the temporary policy.

    • Require all 37 selected cases per artifact source and the complete report predicate. Zero failures with skipped required cases is not a pass. Record the harness commit separately from core and provider artifact commits.

    • Link both uploaded reports, their exact workflow run and any failure disposition here and on docs: add CLAUDE.md repo skill for engineering agents #164. If a runtime defect is exposed, attach the failing cell and first failing boundary before proposing another source change.

    T3. Confirm supported client and Desktop behavior

    The Linux qualification harness tests the Desktop-bundled CLI, not the Electron UI.

    • On the reporter-equivalent Linux client, create a disposable workspace using a remote account whose login shell is Zsh. Repeat normal reopen without rebuild, SSH command execution and cleanup. Keep the account's login shell unchanged.
    • Run the same startup/reopen flow from the actual fixed Desktop package. Record selected CLI/provider versions and verify the remote agent version. Capture only the relevant bootstrap/tunnel evidence.
    • Run targeted native macOS and Windows client smokes for the released pair, including redirected stdin/binary transfer where used. Preserve the existing native Windows stdin/cancellation coverage in the provider; a Linux-only job is not evidence for Windows console behavior.
    • Separate a provider bootstrap failure from in-container agent delivery and IDE launch. The latter are different boundaries and must not be relabeled as this original Zsh parsing defect.
    • After successful candidate qualification, record the stable core version that includes both fixes, the compatible provider version, and the upgrade/reopen instructions. The current stable core remains v1.24.0 as of this audit.

    4. Scope and closure

    No further production bootstrap rewrite is justified by the current evidence. Remaining source work is the qualification gate/docs adjustment above, unless the corrected release matrix finds a concrete defect. Preserve the fixed POSIX launcher, argument quoting, byte-stream ownership, cancellation, atomic replacement and normal interactive-shell behavior.

    The original failure can be called fixed in source and available in prereleases. Keep the issue open for the existing release qualification until the corrected pair passes and the supported stable delivery is recorded. Then close with the exact core/provider versions and evidence. Provider #164 should continue through its normal Draft PR review and checks; this audit does not mark it ready or merge it.

  2. skevetter commented on Oct 31, 2026

    @skevetter
    Contributor

    Completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions