feat(sandbox-vms): wire Coder workspaces to docker-vm and talos-vm sandboxes - #712
Open
ppat wants to merge 1 commit into
Open
feat(sandbox-vms): wire Coder workspaces to docker-vm and talos-vm sandboxes#712ppat wants to merge 1 commit into
ppat wants to merge 1 commit into
Conversation
…ndboxes Automates the manual setup that made the KubeVirt sandbox VMs (docker-vm for DOCKER_HOST over SSH, talos-vm for a disposable Talos cluster) usable from a Coder workspace, so it survives a workspace rebuild instead of being redone from memory: - ~/.ssh/config + a dedicated known_hosts.sandbox pin docker-vm's now-stable host key and point IdentityFile at a dedicated key (IdentitiesOnly, no ssh-agent -- that's per-shell and doesn't survive a new terminal). - DOCKER_HOST is set once the sandbox key exists (private_dot_env.tmpl). - talosctl gets its context/endpoint/node fixed up every apply (run_after_63_sandbox.sh.tmpl), since a talosconfig restored from Bitwarden can come back with an empty context field. - The sandbox kubeconfig is merged into ~/.kube/config using a temp-file flatten (KUBECONFIG=a:b kubectl config view --flatten > a truncates a before it's read) with the existing config listed first so current-context is never silently switched onto the sandbox. - act gets an actrc mapping ubuntu-24.04 (these repos' runs-on pin) and act's own "Medium" tier, so nobody hits the first-run image-size prompt. - controlplane.yaml (which embeds the sandbox cluster's CA private key) deliberately stays off the shared, RWX workspace home PVC by default -- fetch_sandbox-talos-machine-config fetches it via bws on demand instead. The talosconfig alone covers day-to-day use; the machine config is only needed for disaster recovery. Every network call against the VMs (only the kubeconfig fetch makes one) is timeout-bounded and fails soft, so a down sandbox never blocks a workspace from coming up -- verified by pointing the endpoint at a dead host and confirming the script logs a warning and exits 0 with the kubeconfig untouched. Known gap: sandbox_docker_vm_ssh_private_key does not exist yet in Bitwarden Secrets Manager (only its public half was ever uploaded), so private_sandbox_docker_vm.tmpl carries a placeholder secret ID and a TODO with the exact command to create it. Everything else in this change was exercised live against the real VMs. Closes #710 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automates the manual setup that made the KubeVirt sandbox VMs usable from a Coder workspace (
docker-vmforDOCKER_HOSTover SSH,talos-vmfor a disposable Talos cluster), so it survives a workspace rebuild instead of being redone from memory each time.~/.ssh/config+ a dedicatedknown_hosts.sandboxpindocker-vm's now-stable host key (SHA256:uCqIXbsfvNMxdq8RPjy/CAiUFNFezWbNt+I8JtcGxt4, verified live) and pointIdentityFileat a dedicated key withIdentitiesOnly yes— notssh-agent, which is per-shell and doesn't survive a new terminal.UserKnownHostsFilepoints at a chezmoi-owned dedicated file rather than the mainknown_hosts, so managing it declaratively doesn't clobber entries ssh itself accumulates for other hosts.DOCKER_HOSTis set inprivate_dot_env.tmplonce the sandbox key exists on disk (guards against pointing at a dead host before the secret's been fetched).run_after_63_sandbox.sh.tmpl), since a talosconfig restored from Bitwarden can come back with an empty top-levelcontext:field.~/.kube/configvia a temp-file flatten (KUBECONFIG=a:b kubectl config view --flatten > atruncatesabefore it's read — has to go through a temp file andmv), with the existing config listed first socurrent-contextis never silently switched onto the sandbox.~/.config/act/actrcmapsubuntu-24.04(these repos'runs-onpin, which act's default map doesn't cover untilppat/github-workflows#577) and ships act's own "Medium" tier, so nobody hits the first-run image-size prompt or picks the insufficient Micro image by accident.controlplane.yamlembeds the sandbox cluster's CA private key, and the workspace home PVC is shared RWX across workspaces. It's kept off disk by default —executable_fetch-sandbox-talos-machine-configfetches it viabwson demand instead of throughchezmoi apply. The talosconfig alone (chezmoi-managed) covers all day-to-day use; the machine config is only needed for disaster recovery (talosctl gen secrets --from-controlplane-config).Degrades gracefully
The only live network call this makes against the VMs is the
talosctl kubeconfigfetch (talosctl'sconfig context/endpoint/nodeare local-only file edits). That call istimeout-bounded and fails soft — logs a warning and leaves the existing kubeconfig untouched — so a down or nonexistent sandbox never blocks a workspace from starting. Verified by pointing the endpoint at a dead hostname and confirming graceful skip with a byte-identical kubeconfig before/after.Known gap
sandbox_docker_vm_ssh_private_keydoes not exist yet in Bitwarden Secrets Manager — only its public half (sandbox_docker_vm_ssh_public_key) was ever uploaded.private_sandbox_docker_vm.tmplcarries a placeholder secret ID and aTODOwith the exactbws secret createcommand to close the gap (this doesn't affect CI: theseprivate_dot_ssh/*.tmpl/private_dot_talos/*.tmplfiles aren't in any lint job's file set, same as the pre-existingprivate_id_ed25519.tmpl). Everything else in this change was exercised live against the real VMs from a workspace.Test plan
pre-commit run --all-filescleanchezmoi execute-templatepipeline CI uses, then shellchecked the outputssh docker-vm docker versionanddocker system dial-stdioboth verified live over the managed configact -landact -j <job> -nverified live against the remote daemon, actrc fix confirmed (ran through an entire job's steps before an artificial timeout cut it off)config context/endpoint/node+talosctl get membersverified livekubectl --context admin@sandbox-talos get nodes), current-context preservation verified with a simulated existing configCloses #710