Wire Coder workspaces to the KubeVirt sandbox VMs (Docker host + Talos cluster)

[ppat]

Context

Two KubeVirt-hosted VMs now exist on the homelab cluster and both are working end to end:

• docker-vm (sandbox-docker) — Ubuntu 26.04 running dockerd, reached over SSH. This is what makes docker build, kind and act possible from a Coder workspace, which is otherwise an unprivileged pod with no container runtime.
• talos-vm (sandbox-talos) — a single-node Talos v1.13.7 / Kubernetes v1.36.0 cluster. Disposable, resettable, survives restarts, and doubles as the rehearsal ground for the k3s→Talos migration.

Everything below is currently manual, done from memory each time a workspace is rebuilt. It belongs here rather than in the Coder template: it's per-user configuration, and chezmoi can pull the secrets from Bitwarden while the template can't.

What to set up

1. Fetch credentials from Bitwarden via chezmoi

• sandbox_talos_vm_talosconfig → the talosctl client config.
• sandbox_talos_vm_machine_config → controlplane.yaml. Useful locally because talosctl gen secrets --from-controlplane-config reconstructs the secrets bundle from it, which is the recovery path if the talosconfig is ever lost — regenerating a client cert against the running cluster, since the CA is unchanged.
• sandbox_docker_vm_ssh_private_key (or wherever the Docker VM's user private key lives) → for the SSH transport.

Security note worth deciding deliberately rather than by default: controlplane.yaml contains the cluster CA private key, and the workspace home PVC is shared RWX across workspaces. The talosconfig alone is sufficient for all day-to-day use; the machine config is only needed for disaster recovery. Consider whether it should land on disk by default, or be a chezmoi template that's fetched on demand. Either is defensible — the sandbox CA grants nothing outside a disposable, network-isolated cluster — but it should be a choice.

2. talosctl setup

talosctl is already provided via aqua (siderolabs/talos). What's missing is the config being usable without flags:

• A talosconfig restored from Bitwarden comes back with an empty top-level context: field, producing:
  error constructing client: failed to resolve configuration context: default context "" not found in config
  Fix: talosctl config context sandbox-talos.
• Persist the endpoint and node so -n/-e aren't needed on every invocation:
  talosctl config endpoint talos-vm.sandbox-talos.svc.cluster.local
talosctl config node talos-vm.sandbox-talos.svc.cluster.local

Both are one-time fixups that should just be part of the managed config.

3. Sandbox kubeconfig

From a workspace this works directly — no port-forward. A Coder workspace has no host-cluster credentials at all, and doesn't need them: allow-coder-and-mcp-ingress admits the coder namespace to the VM on 6443 and 50000, and the Service DNS name is in both cert SAN sets.

talosctl kubeconfig ./sandbox-kubeconfig

Then merge into the existing kubeconfig. The merge has a trap: KUBECONFIG=a:b kubectl config view --flatten > a truncates a before reading it. Back up, flatten to a temp file, move into place, chmod 600.

Context lands as admin@sandbox-talos.

Note the original design called for a separate KUBECONFIG_SANDBOX so a bare kubectl hits nothing by default. Merging was chosen instead for convenience — low risk here, since the default context is the OIDC prod one that never authenticates inside a workspace, so a stray command fails rather than doing damage. Worth keeping current-context deliberate regardless, with agents running in the same environment.

4. SSH agent and DOCKER_HOST

DOCKER_HOST=ssh://docker@docker-vm.sandbox-docker.svc.cluster.local

The private key is not a default filename, so ssh:// won't find it unaided. Either load it into an agent at shell init, or add a ~/.ssh/config entry with IdentityFile and IdentitiesOnly yes — the latter is probably better for a managed dotfile, since it needs no running agent.

The VM's host key is now stable across restarts (supplied via cloud-init ssh_keys: from Bitwarden), so a known_hosts entry can finally be managed here rather than re-accepted after every VM restart. Its fingerprint is SHA256:uCqIXbsfvNMxdq8RPjy/CAiUFNFezWbNt+I8JtcGxt4.

No local privilege is needed and none should be added. There is no local daemon and no local socket — the CLI talks to a remote daemon over SSH, and authorisation happens on the VM. A local docker group or sudo rule would be solving a problem that doesn't exist here.

5. Redirecting the tools

• act — honours DOCKER_HOST; likely nothing further needed. Verify.
• chainsaw — needs no Docker at all. It runs against any kubeconfig-reachable cluster, so pointing it at the sandbox is purely a KUBECONFIG/context matter. This is the highest-value one to get right.
• kind — needs more than DOCKER_HOST: networking.apiServerAddress set to something the workspace can route to, a unique apiServerPort per consumer (one daemon, one port space), and matching certSANs. Correction, 2026-08-02: this section previously said sandbox-docker's ingress admits the coder namespace on SSH only, blocking a kind API-server port, and that the fix needed a NetworkPolicy change in the clusters repo. That was wrong by the time it was checked against the live cluster — sandbox-docker's allow-coder-ingress rule has admitted a dedicated kind API-server port range (TCP 6550-6560, alongside SSH) since ppat/homelab-ops-kubernetes-clusters#815 (Phase 3, shipped before either sandbox VM existed). No clusters-repo change is needed; only the dotfiles-side config (address, a tracked per-consumer port from that range, certSANs) remains. Treat kind as a separate follow-up rather than blocking this issue on it — now tracked at Wire kind against the remote Docker daemon (config only — the NetworkPolicy half already shipped) #711.

Invariants

• No real DNS domains committed — parameterise as this repo already does elsewhere.
• No secrets in git; everything sensitive comes from Bitwarden via chezmoi at apply time.
• Should degrade gracefully when the VMs don't exist or aren't reachable — a workspace that fails to start because a sandbox is down would be a bad trade.

Verified live 2026-08-02 — what actually works, and the exact papercuts to automate

All three targets were exercised end to end from a workspace against the real VMs, so the
items below are observed rather than anticipated.

SSH config is the fix, not ssh-agent

DOCKER_HOST=ssh:// failed with:

command [ssh -l docker ... docker system dial-stdio] has exited with exit status 255,
make sure the URL is valid, and Docker 18.09 or later is installed on the remote host:
stderr=docker@docker-vm...: Permission denied (publickey).

ssh-agent is per-shell, so loading the key once doesn't survive a new terminal — and the
error names the URL and the remote Docker version while burying the real cause in stderr=.
A ~/.ssh/config entry fixes it permanently and needs no running agent:

Host docker-vm.sandbox-docker.svc.cluster.local
  User docker
  IdentityFile ~/.ssh/sandbox_docker_vm
  IdentitiesOnly yes

Also ensure the key is 0600 — chezmoi should place it that way.

Manage the known_hosts entry too. The VM's host key is now stable across restarts
(supplied via cloud-init ssh_keys: from Bitwarden), fingerprint
SHA256:uCqIXbsfvNMxdq8RPjy/CAiUFNFezWbNt+I8JtcGxt4. Pinning it here means
StrictHostKeyChecking stays meaningful instead of being cleared reflexively.

Verified working afterwards: docker build with BuildKit/buildx against the remote daemon,
including layer commit (docker run --rm buildtest cat /x → hi).

act needs a platform map and a bigger runner image

act -l works out of the box and correctly reports the remote daemon. act -j <job> -n then
stops at:

🚧  Skipping unsupported platform -- Try running with `-P ubuntu-24.04=...`

because these repos pin runs-on: ubuntu-24.04 and act's default map only covers
ubuntu-latest. Until ppat/github-workflows#577 lands (standardising on ubuntu-latest),
~/.config/act/actrc needs:

-P ubuntu-24.04=catthehacker/ubuntu:act-24.04

Also note act prompts interactively on first run for a default image size and writes the
choice to that same file. The Micro image (<200MB, NodeJS only) is insufficient for real runs —
ship a sensible default so nobody gets the prompt or picks Micro by accident.

Correction worth recording: act does resolve remote reusable workflows — it cloned
ppat/github-workflows at its pinned SHA. The runner label was the only blocker.

chainsaw

Nothing here beyond pointing KUBECONFIG at the sandbox — chainsaw needs no Docker and runs
against any reachable cluster. The harness-side work belongs in
ppat/homelab-ops-kubernetes-apps#3513, not here: that repo is the only place chainsaw tests
exist, and the two things that block a local run (Flux must be pre-installed in the target
cluster, and SkipDelete: true is right for CI but wrong against a standing sandbox) are
properties of the suite and its shared config, not of a workspace.

Once that lands, dotfiles may want a thin wrapper or alias so the right kubeconfig and cleanup
behaviour aren't something anyone has to remember — but wait and see what shape #3513 takes
before building one.

talosctl

Confirmed: a talosconfig restored from Bitwarden comes back with an empty top-level context:,
producing default context "" not found in config. talosctl config context sandbox-talos plus
persisted endpoint/node is the fixup — as described above, and it should just be part of the
managed config rather than a step anyone runs.

Done when

A fresh workspace has working docker against the VM, talosctl and kubectl against the sandbox cluster, and chainsaw able to run a suite — with no manual setup steps and no secrets in the repo.

[ppat]

Origin: this is Phase 6 of ppat/homelab-ops-kubernetes-apps#3503 — the KubeVirt/sandbox effort this workspace wiring completes. That issue's status comment tracks this one as the one piece of Phase 6 currently in progress.