Skip to content

feat(sandbox-vms): wire Coder workspaces to docker-vm and talos-vm sandboxes - #712

Open
ppat wants to merge 1 commit into
mainfrom
feat/sandbox-vm-wiring
Open

feat(sandbox-vms): wire Coder workspaces to docker-vm and talos-vm sandboxes#712
ppat wants to merge 1 commit into
mainfrom
feat/sandbox-vm-wiring

Conversation

@ppat

@ppat ppat commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

Automates the manual setup that made the KubeVirt sandbox VMs usable from a Coder workspace (docker-vm for DOCKER_HOST over SSH, talos-vm for a disposable Talos cluster), so it survives a workspace rebuild instead of being redone from memory each time.

  • SSH: ~/.ssh/config + a dedicated known_hosts.sandbox pin docker-vm's now-stable host key (SHA256:uCqIXbsfvNMxdq8RPjy/CAiUFNFezWbNt+I8JtcGxt4, verified live) and point IdentityFile at a dedicated key with IdentitiesOnly yes — not ssh-agent, which is per-shell and doesn't survive a new terminal. UserKnownHostsFile points at a chezmoi-owned dedicated file rather than the main known_hosts, so managing it declaratively doesn't clobber entries ssh itself accumulates for other hosts.
  • DOCKER_HOST is set in private_dot_env.tmpl once the sandbox key exists on disk (guards against pointing at a dead host before the secret's been fetched).
  • talosctl: context/endpoint/node are fixed up on every apply (run_after_63_sandbox.sh.tmpl), since a talosconfig restored from Bitwarden can come back with an empty top-level context: field.
  • kubeconfig merge: the sandbox kubeconfig is merged into ~/.kube/config via a temp-file flatten (KUBECONFIG=a:b kubectl config view --flatten > a truncates a before it's read — has to go through a temp file and mv), with the existing config listed first so current-context is never silently switched onto the sandbox.
  • act: ~/.config/act/actrc maps ubuntu-24.04 (these repos' runs-on pin, which act's default map doesn't cover until ppat/github-workflows#577) and ships act's own "Medium" tier, so nobody hits the first-run image-size prompt or picks the insufficient Micro image by accident.
  • Machine config decision: controlplane.yaml embeds the sandbox cluster's CA private key, and the workspace home PVC is shared RWX across workspaces. It's kept off disk by defaultexecutable_fetch-sandbox-talos-machine-config fetches it via bws on demand instead of through chezmoi apply. The talosconfig alone (chezmoi-managed) covers all day-to-day use; the machine config is only needed for disaster recovery (talosctl gen secrets --from-controlplane-config).

Degrades gracefully

The only live network call this makes against the VMs is the talosctl kubeconfig fetch (talosctl's config context/endpoint/node are local-only file edits). That call is timeout-bounded and fails soft — logs a warning and leaves the existing kubeconfig untouched — so a down or nonexistent sandbox never blocks a workspace from starting. Verified by pointing the endpoint at a dead hostname and confirming graceful skip with a byte-identical kubeconfig before/after.

Known gap

sandbox_docker_vm_ssh_private_key does not exist yet in Bitwarden Secrets Manager — only its public half (sandbox_docker_vm_ssh_public_key) was ever uploaded. private_sandbox_docker_vm.tmpl carries a placeholder secret ID and a TODO with the exact bws secret create command to close the gap (this doesn't affect CI: these private_dot_ssh/*.tmpl/private_dot_talos/*.tmpl files aren't in any lint job's file set, same as the pre-existing private_id_ed25519.tmpl). Everything else in this change was exercised live against the real VMs from a workspace.

Test plan

  • pre-commit run --all-files clean
  • Rendered every new/changed template through the same sed+chezmoi execute-template pipeline CI uses, then shellchecked the output
  • SSH: ssh docker-vm docker version and docker system dial-stdio both verified live over the managed config
  • act -l and act -j <job> -n verified live against the remote daemon, actrc fix confirmed (ran through an entire job's steps before an artificial timeout cut it off)
  • talosctl: config context/endpoint/node + talosctl get members verified live
  • kubeconfig merge: verified live end-to-end (kubectl --context admin@sandbox-talos get nodes), current-context preservation verified with a simulated existing config
  • Degradation path: endpoint pointed at a dead hostname, confirmed graceful skip + untouched kubeconfig, then restored
  • CI

Closes #710

…ndboxes

Automates the manual setup that made the KubeVirt sandbox VMs (docker-vm for
DOCKER_HOST over SSH, talos-vm for a disposable Talos cluster) usable from a
Coder workspace, so it survives a workspace rebuild instead of being redone
from memory:

- ~/.ssh/config + a dedicated known_hosts.sandbox pin docker-vm's now-stable
  host key and point IdentityFile at a dedicated key (IdentitiesOnly, no
  ssh-agent -- that's per-shell and doesn't survive a new terminal).
- DOCKER_HOST is set once the sandbox key exists (private_dot_env.tmpl).
- talosctl gets its context/endpoint/node fixed up every apply
  (run_after_63_sandbox.sh.tmpl), since a talosconfig restored from Bitwarden
  can come back with an empty context field.
- The sandbox kubeconfig is merged into ~/.kube/config using a temp-file
  flatten (KUBECONFIG=a:b kubectl config view --flatten > a truncates a
  before it's read) with the existing config listed first so current-context
  is never silently switched onto the sandbox.
- act gets an actrc mapping ubuntu-24.04 (these repos' runs-on pin) and
  act's own "Medium" tier, so nobody hits the first-run image-size prompt.
- controlplane.yaml (which embeds the sandbox cluster's CA private key)
  deliberately stays off the shared, RWX workspace home PVC by default --
  fetch_sandbox-talos-machine-config fetches it via bws on demand instead.
  The talosconfig alone covers day-to-day use; the machine config is only
  needed for disaster recovery.

Every network call against the VMs (only the kubeconfig fetch makes one) is
timeout-bounded and fails soft, so a down sandbox never blocks a workspace
from coming up -- verified by pointing the endpoint at a dead host and
confirming the script logs a warning and exits 0 with the kubeconfig
untouched.

Known gap: sandbox_docker_vm_ssh_private_key does not exist yet in Bitwarden
Secrets Manager (only its public half was ever uploaded), so
private_sandbox_docker_vm.tmpl carries a placeholder secret ID and a TODO
with the exact command to create it. Everything else in this change was
exercised live against the real VMs.

Closes #710

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wire Coder workspaces to the KubeVirt sandbox VMs (Docker host + Talos cluster)

1 participant