Skip to content

runbooks: one way onto a runner, and replacing one is drain then remove - #13

Merged
aledbf merged 1 commit into
mainfrom
runner-access-coherent
Nov 18, 2026
Merged

aledbf merged 1 commit into
mainfrom
runner-access-coherent

Conversation

@aledbf

@aledbf aledbf commented Nov 18, 2026

Copy link
Copy Markdown
Member

The runner runbooks disagreed about getting onto a runner.

  • host-not-reporting sent the operator onto the machine through Session Manager (systemctl restart, journalctl). Runners have no Session Manager: in spin-terraform-aws only the control plane and the proxy get that policy. And it contradicts host-shell, where a runner is reached only by a recorded shell that empties it first and retires it after. A host that is not connected cannot open a shell anyway, since the shell rides its connection. Now nothing is done on the machine: reboot the instance from the cloud's console, and a runner that keeps failing is emptied and replaced.
  • Replacing a runner is spin admin runner drain <id> --suspend, then spin admin runner remove <id> (its disks are delete_on_termination), said the same way in host-not-reporting, runners-behind, volume-not-served and object-store-requests-runaway. A host-shell is for seeing why, and replaces the runner all the same.
  • runners-behind and host-shell said the group "starts a new one" / "replaces it". remove and a shell's retirement ask the group for one machine less (Scaler.Take); the autoscaler starts another when the fleet needs the room.

task check passes.

🤖 Generated with Claude Code

host-not-reporting sent the operator onto the machine through Session
Manager to run systemctl and journalctl. Runners have no Session Manager
(only the control plane and the proxy do), and host-shell says a runner
is reached only by a recorded shell, which empties it first and retires
it after. A host that is not connected cannot open one either: the shell
rides its connection. So: nothing is done on the machine, a stuck runner
is rebooted from the cloud's console, and one that keeps failing is
emptied and replaced.

Replacing is drain --suspend, then remove, in every runbook that needs
it; a shell is for seeing why, and replaces the runner all the same.
remove asks the group for one machine less (Scaler.Take), not for a new
one: the autoscaler starts another when the fleet needs the room.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit c65f802 into main Nov 18, 2026
1 check passed
@aledbf
aledbf deleted the runner-access-coherent branch November 18, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant