From da5231042ffb498a6f90df474b6e0666bf0151a5 Mon Sep 17 00:00:00 2001 From: Manuel de Brito Fontes Date: Tue, 17 Nov 2026 23:52:02 -0300 Subject: [PATCH] runbooks: one way onto a runner, and replacing one is drain then remove host-not-reporting sent the operator onto the machine through Session Manager to run systemctl and journalctl. Runners have no Session Manager (only the control plane and the proxy do), and host-shell says a runner is reached only by a recorded shell, which empties it first and retires it after. A host that is not connected cannot open one either: the shell rides its connection. So: nothing is done on the machine, a stuck runner is rebooted from the cloud's console, and one that keeps failing is emptied and replaced. Replacing is drain --suspend, then remove, in every runbook that needs it; a shell is for seeing why, and replaces the runner all the same. remove asks the group for one machine less (Scaler.Take), not for a new one: the autoscaler starts another when the fleet needs the room. Co-Authored-By: Claude Opus 5.5 --- runbooks/host-not-reporting.md | 27 ++++++++++++++--------- runbooks/host-shell.md | 10 ++++++--- runbooks/object-store-requests-runaway.md | 9 ++++---- runbooks/runners-behind.md | 13 ++++++----- runbooks/volume-not-served.md | 6 +++-- 5 files changed, 41 insertions(+), 24 deletions(-) diff --git a/runbooks/host-not-reporting.md b/runbooks/host-not-reporting.md index faf5883..242831c 100644 --- a/runbooks/host-not-reporting.md +++ b/runbooks/host-not-reporting.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet] severity: critical owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-18 --- ## Symptoms @@ -50,19 +50,16 @@ and its volumes' ages are what it last said. 4. Look at the machine itself in the cloud's console: running, stopped, terminated, failing its status checks. On AWS, an autoscaling group may have replaced it. -5. If the machine runs, on it: - ``` - systemctl status spin-runner.service - journalctl -u spin-runner.service --since "1 hour ago" - ``` - - Runners have no SSH; on AWS they are reached through Session Manager. +Nothing here is done on the machine. A runner has no SSH and no Session Manager, and the one way +onto it, a [host-shell](host-shell.md), rides the host's own connection to the control plane: a +host that is not connected cannot open one. ## Fix -- **The runner process stopped**: `systemctl restart spin-runner.service` on the machine. A runner - that restarts comes back as the same host and its workspaces are waited for, nothing lost. +- **The machine runs and the runner does not answer**: reboot the instance from the cloud's + console (on AWS, *Reboot instance*), not from inside it. The runner starts with the machine and + comes back as the same host; its workspaces are waited for, nothing lost. - **The network between them** (the machine runs, the runner logs it cannot reach the control plane): fix the route, the security group or the DNS name. The host reconnects on its own. - **The runner was revoked** (its log says its token is invalid): it will never connect again. @@ -71,6 +68,16 @@ and its volumes' ages are what it last said. another host either by their owners (`spin workspace recover `) or by the control plane on its own, past the installation's rehome threshold ([workspace-rehomed](workspace-rehomed.md)). +- **It comes back but keeps failing**: empty it and replace it. Its workspaces resume on other + runners, and its machine, with everything on its disks, is terminated: + + ``` + spin admin runner drain --suspend + spin admin runner remove + ``` + + To see why first, open a [host-shell](host-shell.md) instead: it empties the runner the same + way and retires it when the shell ends. - **Cannot tell yet**: stop placing on it until you know: `spin admin runner drain `. ## Verify diff --git a/runbooks/host-shell.md b/runbooks/host-shell.md index 83b45db..25d4761 100644 --- a/runbooks/host-shell.md +++ b/runbooks/host-shell.md @@ -8,7 +8,7 @@ applies_to: dashboards: [] severity: info owner: security -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-18 --- ## Symptoms @@ -27,7 +27,11 @@ A root shell is for a while, with a reason, and costs the runner: under Object Lock. - Once the shell ends (its minutes are up, or it is closed) the runner is **retired**: revoked, every disk it held the key of marked to be re-keyed, and its machine given back to its group, - which replaces it. A runner someone had root on runs no workspace again. + which terminates it and is asked for one less; the autoscaler starts another when the fleet + needs the room. A runner someone had root on runs no workspace again. + +A shell is for seeing why. To replace a runner without looking at it, no shell is needed: +`spin admin runner drain --suspend`, then `spin admin runner remove `. ## Diagnose @@ -89,7 +93,7 @@ Settings, the CLI or the configuration file; only the admin API takes them ([spi ## Verify `spin admin shell list` shows the shell `ended`, with RETIRED `true`. The runner is gone from -[Admin → Runners](/admin/runners) or revoked, and its group has started a replacement. +[Admin → Runners](/admin/runners) or revoked, and its group no longer lists its machine. `spin admin volume rekeys` lists the disks it held while they are re-keyed ([key-rotation](key-rotation.md)). [Admin → Audit](/admin/audit) has `host_shell_requested`, `host_shell_approved`, `host_shell_session_opened` and `host_shell_session_ended`. diff --git a/runbooks/object-store-requests-runaway.md b/runbooks/object-store-requests-runaway.md index d4d2e41..65a2db4 100644 --- a/runbooks/object-store-requests-runaway.md +++ b/runbooks/object-store-requests-runaway.md @@ -8,7 +8,7 @@ applies_to: dashboards: [storage] severity: warning owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-18 --- ## Symptoms @@ -49,9 +49,10 @@ hour, and is usually a symptom of a volume the host cannot serve. - **The store refusing** (access denied, throttling): the host's credentials or the bucket's policy changed. Check the host's role in the cloud and *Credentials hosts asked for, and what the store's STS minted* on the storage dashboard. -- **Nothing explains it**: stop the host taking new work (`spin admin runner drain `) - and look at it on a [host-shell](host-shell.md). Draining with `--suspend` moves its running - workspaces elsewhere. +- **Nothing explains it**: stop the host taking new work (`spin admin runner drain `). + To see why, open a [host-shell](host-shell.md): it empties the runner first and retires it + when the shell ends. To replace it without looking, `drain --suspend` and then + `spin admin runner remove`. ## Verify diff --git a/runbooks/runners-behind.md b/runbooks/runners-behind.md index 0e3d744..472804e 100644 --- a/runbooks/runners-behind.md +++ b/runbooks/runners-behind.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet] severity: warning owner: operator -last_reviewed: 2026-11-09 +last_reviewed: 2026-11-18 --- ## Symptoms @@ -46,15 +46,18 @@ or stopped through it. The fleet has that much less room. ## Fix - **An autoscaling group's runner**: make sure the group now starts machines of the control - plane's release (its launch template names the new image), then give the old one back to its - group, which drains it, terminates it and starts a new one: + plane's release (its launch template names the new image), then empty the old one and give it + back. Its group terminates it and is asked for one machine less; the autoscaler asks for a new + one, of the new image, when the fleet needs the room: ``` + spin admin runner drain --suspend spin admin runner remove ``` - A runner that is the only home of a workspace's disk is refused, naming them; their owners stop - or move them first. + `remove` refuses a runner that is still the only home of a workspace's disk, naming them: run + the drain again ([workspace-suspends-failing](workspace-suspends-failing.md)), or their owners + stop them. - **A runner registered with a token** (self-hosted): drain it with `--suspend` so its workspaces move, install the control plane's release on the machine, and join it again with a new token (`spin admin token create`). See [lost-runner](lost-runner.md) for taking the old host out. diff --git a/runbooks/volume-not-served.md b/runbooks/volume-not-served.md index e8602c2..818fcf6 100644 --- a/runbooks/volume-not-served.md +++ b/runbooks/volume-not-served.md @@ -8,7 +8,7 @@ applies_to: dashboards: [fleet, storage] severity: critical owner: operator -last_reviewed: 2026-11-12 +last_reviewed: 2026-11-18 --- ## Symptoms @@ -81,7 +81,9 @@ it writes from now on is lost with its host. Each failed restore reads the objec A refused volume counts as stopped, so recover does not wait for the host. The workspace restarts from the last commit the store has; see [workspace-rehomed](workspace-rehomed.md). - **The host is the cause for many volumes**: stop it taking new work with - `spin admin runner drain `, and look at it on a [host-shell](host-shell.md). + `spin admin runner drain `. To see why, open a [host-shell](host-shell.md): it + empties the runner first and retires it when the shell ends, so the host is replaced either + way. To replace it without looking, `drain --suspend` and then `spin admin runner remove`. - **Which workspaces it is**: list every refused volume, with the refusal, the host, the workspace and its owner (the listing is audited):