Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions runbooks/host-not-reporting.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ applies_to:
dashboards: [fleet]
severity: critical
owner: operator
last_reviewed: 2026-11-09
last_reviewed: 2026-11-18
---

## Symptoms
Expand Down Expand Up @@ -50,19 +50,16 @@ and its volumes' ages are what it last said.

4. Look at the machine itself in the cloud's console: running, stopped, terminated, failing its
status checks. On AWS, an autoscaling group may have replaced it.
5. If the machine runs, on it:

```
systemctl status spin-runner.service
journalctl -u spin-runner.service --since "1 hour ago"
```

Runners have no SSH; on AWS they are reached through Session Manager.
Nothing here is done on the machine. A runner has no SSH and no Session Manager, and the one way
onto it, a [host-shell](host-shell.md), rides the host's own connection to the control plane: a
host that is not connected cannot open one.

## Fix

- **The runner process stopped**: `systemctl restart spin-runner.service` on the machine. A runner
that restarts comes back as the same host and its workspaces are waited for, nothing lost.
- **The machine runs and the runner does not answer**: reboot the instance from the cloud's
console (on AWS, *Reboot instance*), not from inside it. The runner starts with the machine and
comes back as the same host; its workspaces are waited for, nothing lost.
- **The network between them** (the machine runs, the runner logs it cannot reach the control
plane): fix the route, the security group or the DNS name. The host reconnects on its own.
- **The runner was revoked** (its log says its token is invalid): it will never connect again.
Expand All @@ -71,6 +68,16 @@ and its volumes' ages are what it last said.
another host either by their owners (`spin workspace recover <workspace>`) or by the control
plane on its own, past the installation's rehome threshold
([workspace-rehomed](workspace-rehomed.md)).
- **It comes back but keeps failing**: empty it and replace it. Its workspaces resume on other
runners, and its machine, with everything on its disks, is terminated:

```
spin admin runner drain <runner-id> --suspend
spin admin runner remove <runner-id>
```

To see why first, open a [host-shell](host-shell.md) instead: it empties the runner the same
way and retires it when the shell ends.
- **Cannot tell yet**: stop placing on it until you know: `spin admin runner drain <runner-id>`.

## Verify
Expand Down
10 changes: 7 additions & 3 deletions runbooks/host-shell.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ applies_to:
dashboards: []
severity: info
owner: security
last_reviewed: 2026-11-09
last_reviewed: 2026-11-18
---

## Symptoms
Expand All @@ -27,7 +27,11 @@ A root shell is for a while, with a reason, and costs the runner:
under Object Lock.
- Once the shell ends (its minutes are up, or it is closed) the runner is **retired**: revoked,
every disk it held the key of marked to be re-keyed, and its machine given back to its group,
which replaces it. A runner someone had root on runs no workspace again.
which terminates it and is asked for one less; the autoscaler starts another when the fleet
needs the room. A runner someone had root on runs no workspace again.

A shell is for seeing why. To replace a runner without looking at it, no shell is needed:
`spin admin runner drain <runner-id> --suspend`, then `spin admin runner remove <runner-id>`.

## Diagnose

Expand Down Expand Up @@ -89,7 +93,7 @@ Settings, the CLI or the configuration file; only the admin API takes them ([spi
## Verify

`spin admin shell list` shows the shell `ended`, with RETIRED `true`. The runner is gone from
[Admin → Runners](/admin/runners) or revoked, and its group has started a replacement.
[Admin → Runners](/admin/runners) or revoked, and its group no longer lists its machine.
`spin admin volume rekeys` lists the disks it held while they are re-keyed
([key-rotation](key-rotation.md)). [Admin → Audit](/admin/audit) has `host_shell_requested`,
`host_shell_approved`, `host_shell_session_opened` and `host_shell_session_ended`.
Expand Down
9 changes: 5 additions & 4 deletions runbooks/object-store-requests-runaway.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ applies_to:
dashboards: [storage]
severity: warning
owner: operator
last_reviewed: 2026-11-09
last_reviewed: 2026-11-18
---

## Symptoms
Expand Down Expand Up @@ -49,9 +49,10 @@ hour, and is usually a symptom of a volume the host cannot serve.
- **The store refusing** (access denied, throttling): the host's credentials or the bucket's
policy changed. Check the host's role in the cloud and *Credentials hosts asked for, and what the
store's STS minted* on the storage dashboard.
- **Nothing explains it**: stop the host taking new work (`spin admin runner drain <runner-id>`)
and look at it on a [host-shell](host-shell.md). Draining with `--suspend` moves its running
workspaces elsewhere.
- **Nothing explains it**: stop the host taking new work (`spin admin runner drain <runner-id>`).
To see why, open a [host-shell](host-shell.md): it empties the runner first and retires it
when the shell ends. To replace it without looking, `drain --suspend` and then
`spin admin runner remove`.

## Verify

Expand Down
13 changes: 8 additions & 5 deletions runbooks/runners-behind.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ applies_to:
dashboards: [fleet]
severity: warning
owner: operator
last_reviewed: 2026-11-09
last_reviewed: 2026-11-18
---

## Symptoms
Expand Down Expand Up @@ -46,15 +46,18 @@ or stopped through it. The fleet has that much less room.
## Fix

- **An autoscaling group's runner**: make sure the group now starts machines of the control
plane's release (its launch template names the new image), then give the old one back to its
group, which drains it, terminates it and starts a new one:
plane's release (its launch template names the new image), then empty the old one and give it
back. Its group terminates it and is asked for one machine less; the autoscaler asks for a new
one, of the new image, when the fleet needs the room:

```
spin admin runner drain <runner-id> --suspend
spin admin runner remove <runner-id>
```

A runner that is the only home of a workspace's disk is refused, naming them; their owners stop
or move them first.
`remove` refuses a runner that is still the only home of a workspace's disk, naming them: run
the drain again ([workspace-suspends-failing](workspace-suspends-failing.md)), or their owners
stop them.
- **A runner registered with a token** (self-hosted): drain it with `--suspend` so its workspaces
move, install the control plane's release on the machine, and join it again with a new token
(`spin admin token create`). See [lost-runner](lost-runner.md) for taking the old host out.
Expand Down
6 changes: 4 additions & 2 deletions runbooks/volume-not-served.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ applies_to:
dashboards: [fleet, storage]
severity: critical
owner: operator
last_reviewed: 2026-11-12
last_reviewed: 2026-11-18
---

## Symptoms
Expand Down Expand Up @@ -81,7 +81,9 @@ it writes from now on is lost with its host. Each failed restore reads the objec
A refused volume counts as stopped, so recover does not wait for the host. The workspace
restarts from the last commit the store has; see [workspace-rehomed](workspace-rehomed.md).
- **The host is the cause for many volumes**: stop it taking new work with
`spin admin runner drain <runner-id>`, and look at it on a [host-shell](host-shell.md).
`spin admin runner drain <runner-id>`. To see why, open a [host-shell](host-shell.md): it
empties the runner first and retires it when the shell ends, so the host is replaced either
way. To replace it without looking, `drain --suspend` and then `spin admin runner remove`.

- **Which workspaces it is**: list every refused volume, with the refusal, the host, the
workspace and its owner (the listing is audited):
Expand Down
Loading