Repository navigation
runbooks: one way onto a runner, and replacing one is drain then remove - #13
Merged
Merged
Conversation
host-not-reporting sent the operator onto the machine through Session Manager to run systemctl and journalctl. Runners have no Session Manager (only the control plane and the proxy do), and host-shell says a runner is reached only by a recorded shell, which empties it first and retires it after. A host that is not connected cannot open one either: the shell rides its connection. So: nothing is done on the machine, a stuck runner is rebooted from the cloud's console, and one that keeps failing is emptied and replaced. Replacing is drain --suspend, then remove, in every runbook that needs it; a shell is for seeing why, and replaces the runner all the same. remove asks the group for one machine less (Scaler.Take), not for a new one: the autoscaler starts another when the fleet needs the room. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The runner runbooks disagreed about getting onto a runner.
systemctl restart,journalctl). Runners have no Session Manager: in spin-terraform-aws only the control plane and the proxy get that policy. And it contradicts host-shell, where a runner is reached only by a recorded shell that empties it first and retires it after. A host that is not connected cannot open a shell anyway, since the shell rides its connection. Now nothing is done on the machine: reboot the instance from the cloud's console, and a runner that keeps failing is emptied and replaced.spin admin runner drain <id> --suspend, thenspin admin runner remove <id>(its disks aredelete_on_termination), said the same way in host-not-reporting, runners-behind, volume-not-served and object-store-requests-runaway. A host-shell is for seeing why, and replaces the runner all the same.removeand a shell's retirement ask the group for one machine less (Scaler.Take); the autoscaler starts another when the fleet needs the room.task checkpasses.🤖 Generated with Claude Code