From 8c526c8074c61717b1909a65972bc5487564e2df Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 17:47:54 -0400 Subject: [PATCH 1/3] ci: a runner-health check that notices the OOM-killed-and-stayed-dead shape (#1504) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit engine#1504 measured the gap in its own title: both self-hosted runners died on an OOM kill at 08:37 on 2026-11-08 and stayed dead for ~3h45m with `Result=oom-kill` and `NRestarts=0`, and **nothing noticed** - every check that needs the box read as *pending*, never green and never red, which at a glance is indistinguishable from "queued behind a busy runner". `scripts/ci-runner-health.sh` is the thing that notices: one line per `actions.runner.*` unit, a printed verdict, a non-zero exit. It reads systemd's `ActiveState`/`SubState`/`Result`/`NRestarts` - the only place `Result=oom-kill` survives, and the property a restart policy would consult - and it distinguishes the two shapes that look the same from outside: a runner that is *down after a kill* from one that is *busy*. It never restarts anything. `systemctl start` needs root, and whether to restart a runner that OOM-killed twice in two minutes is a decision rather than a check - #1504 measured that the fourth restart survived only because the GPU ticket was empty. Verified: * `--self-test` 6/6 on the verdict logic, including `failed/failed oom-kill 0` (the 08:37 shape), `inactive`, an unreadable state (unknown is not a pass), and `active … NRestarts=3` (a working restart policy is a PASS with the count visible, so a flapping runner shows up as a number rather than as silence); * the **fail direction through the real code path** with a stubbed `systemctl`: the dead shape prints `FAIL … failed/failed after an OOM kill (0 restarts) - the kernel killed it and nothing restarted it; jobs read as pending, not as failed` and exits **1**; a restarted-and-up runner prints PASS with `1 restart(s)` and exits **0**; and a host with no matching units exits **2** with a reason, because a silent pass is the defect this exists for; * live on this host: three runners, all PASS. Wiring it to a timer is a host decision, not a repo one (the unit would live in `/etc/systemd/system`), so this PR stops at the check. The snippet for whoever does wire it: [Service] Type=oneshot ExecStart=/path/to/scripts/ci-runner-health.sh with the same restart-policy question #1504 asks (`Restart=` on the runner units) left where it belongs - with the owner, since restarting a runner into the same ticket is the failure the fourth restart reproduced. --- scripts/ci-runner-health.sh | 84 +++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100755 scripts/ci-runner-health.sh diff --git a/scripts/ci-runner-health.sh b/scripts/ci-runner-health.sh new file mode 100755 index 000000000..e59b76f71 --- /dev/null +++ b/scripts/ci-runner-health.sh @@ -0,0 +1,84 @@ +#!/bin/bash +# Copyright 2026 bong-water-water-bong +# SPDX-License-Identifier: Apache-2.0 +# +# ci-runner-health.sh - say whether the self-hosted GitHub runners are RUNNING, and refuse to be silent about a dead one. +# +# Why this exists (engine#1504): both runners died on an OOM kill at 08:37 on 2026-11-08 and stayed dead for ~3h45m with +# `Result=oom-kill` and `NRestarts=0`. Every check that needs the box then read as **pending** - never green, never red - +# which is indistinguishable, at a glance, from "queued behind a busy runner". The issue's own words for the gap are +# "nothing notices", and this is the thing that notices: one line per runner, a printed verdict, and a non-zero exit. +# +# It reads systemd's view, which is the only place `Result=oom-kill` survives (the journal has it too, but the unit property +# is what a restart policy would consult). It never restarts anything: `systemctl start` needs root, and whether to restart a +# runner that OOM-killed twice in two minutes is a decision, not a check (engine#1504 measured that: the fourth restart +# survived only because the GPU ticket was empty). +# +# scripts/ci-runner-health.sh # every actions.runner.* unit +# scripts/ci-runner-health.sh [...] # named units, e.g. for a specific runner +# scripts/ci-runner-health.sh --self-test # the verdict logic, with systemctl stubbed +# +# Exit: 0 when every runner is active/running with no failure result; 1 when any is not; 2 when there is nothing to inspect +# (no such units on this host, so a silent pass would be the defect). +set -uo pipefail + +# -> one verdict line +runner_verdict() { + _st=${1:-} _sub=${2:-} _res=${3:-} _nr=${4:-} + if [ -z "$_st" ]; then + printf 'FAIL no systemd state could be read for this runner, so whether it is alive is unknown' + return 0 + fi + if [ "$_st" = active ]; then + printf 'PASS active/running, last result %s, %s restart(s) - the runner is accepting jobs' "${_res:-?}" "${_nr:-0}" + return 0 + fi + case "$_res" in + oom-kill) printf 'FAIL %s/%s after an OOM kill (%s restarts) - the kernel killed it and nothing restarted it; jobs read as pending, not as failed' "$_st" "$_sub" "${_nr:-0}" ;; + *) printf 'FAIL %s/%s with last result %s (%s restarts) - the runner is not accepting jobs' "$_st" "${_sub:-?}" "${_res:-?}" "${_nr:-0}" ;; + esac +} + +self_test() { + rc=0 + chk() { # + case "$2" in "$1"*) printf 'ok %s\n' "$(printf '%s' "$2" | cut -c1-78)" ;; + *) printf 'FAIL want %s, got: %s\n' "$1" "$2"; rc=1 ;; esac + } + chk PASS "$(runner_verdict active running success 0)" + chk FAIL "$(runner_verdict failed failed oom-kill 0)" # the 08:37 shape + chk FAIL "$(runner_verdict failed failed oom-kill 0)" + chk FAIL "$(runner_verdict inactive dead success 0)" + chk FAIL "$(runner_verdict '' '' '' '')" # nothing readable is not a pass + # a restart policy that works shows NRestarts>0 and result success - which is a PASS with the count visible + chk PASS "$(runner_verdict active running success 3)" + [ "$rc" = 0 ] && echo "ci-runner-health self-test: ok (6 cases)" || echo "ci-runner-health self-test: FAIL" + return $rc +} + +[ "${1:-}" = --self-test ] && { self_test; exit $?; } + +SC=${SYSTEMCTL:-systemctl} +units=("$@") +if [ ${#units[@]} -eq 0 ]; then + mapfile -t units < <("$SC" list-units 'actions.runner.*' --all --no-legend --plain 2>/dev/null | awk '{print $1}') +fi +if [ ${#units[@]} -eq 0 ]; then + echo "FAIL no actions.runner.* units on this host - either the runners were never installed here or systemd has dropped them entirely; a silent pass would be the defect this check exists for" + exit 2 +fi + +rc=0 +for u in "${units[@]}"; do + line=$("$SC" show "$u" -p ActiveState -p SubState -p Result -p NRestarts 2>/dev/null) + st=$(printf '%s\n' "$line" | sed -n 's/^ActiveState=//p') + sub=$(printf '%s\n' "$line" | sed -n 's/^SubState=//p') + res=$(printf '%s\n' "$line" | sed -n 's/^Result=//p') + nr=$(printf '%s\n' "$line" | sed -n 's/^NRestarts=//p') + v=$(runner_verdict "$st" "$sub" "$res" "$nr") + case "$v" in + PASS*) printf 'PASS %-52s %s\n' "$u" "${v#PASS }" ;; + *) printf 'FAIL %-52s %s\n' "$u" "${v#FAIL }"; rc=1 ;; + esac +done +exit "$rc" From 6c9e5b7d4465833f714f67896ca70ead23d75527 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 17:49:18 -0400 Subject: [PATCH 2/3] ci-runner-health: name the RESTART WINDOW, which is where a flapping runner looks busy (#1504) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pi-os3 ran the check on their side and asked for the one case the self-test did not have: `activating (auto-restart)`. That is the transient where a FLAPPING runner looks *busy* to a naive check - systemd is already dealing with the unit, so a check that only asks "is it failed?" passes it. The verdict already failed that state, but silently: `activating/auto-restart with last result success (4 restarts) - the runner is not accepting jobs` does not tell a reader that the unit is mid-restart or that a log full of these lines is the diagnosis. It now says so: FAIL … activating/auto-restart - the runner is in its RESTART WINDOW and not accepting jobs (4 restarts so far); one line is transient, a log full of them is a flapping runner Three self-test cases added: the window with `Result=success`, the window whose last result is a kill, and an assertion that the message *names* the window rather than merely failing. 9 cases, 0 failed. Also worth recording, from their run rather than mine: the check works **unprivileged** (they ran it as `bcloud` and read state, Result and NRestarts for all three system units), so a timer for it does not need a root unit - a health check that needs root to *read* is one that gets wired up wrong once and then silently stops. --- scripts/ci-runner-health.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/scripts/ci-runner-health.sh b/scripts/ci-runner-health.sh index e59b76f71..7f0a794a7 100755 --- a/scripts/ci-runner-health.sh +++ b/scripts/ci-runner-health.sh @@ -33,6 +33,12 @@ runner_verdict() { printf 'PASS active/running, last result %s, %s restart(s) - the runner is accepting jobs' "${_res:-?}" "${_nr:-0}" return 0 fi + # `activating (auto-restart)` is the transient where a FLAPPING runner looks busy: it is not accepting jobs, but a naive + # check sees a unit that systemd is already dealing with. Say which it is, and say what a repeat means (pi-os3's addition). + if [ "$_st" = activating ] || [ "${_sub:-}" = auto-restart ]; then + printf 'FAIL %s/%s - the runner is in its RESTART WINDOW and not accepting jobs (%s restarts so far); one line is transient, a log full of them is a flapping runner' "$_st" "${_sub:-?}" "${_nr:-0}" + return 0 + fi case "$_res" in oom-kill) printf 'FAIL %s/%s after an OOM kill (%s restarts) - the kernel killed it and nothing restarted it; jobs read as pending, not as failed' "$_st" "$_sub" "${_nr:-0}" ;; *) printf 'FAIL %s/%s with last result %s (%s restarts) - the runner is not accepting jobs' "$_st" "${_sub:-?}" "${_res:-?}" "${_nr:-0}" ;; @@ -50,9 +56,14 @@ self_test() { chk FAIL "$(runner_verdict failed failed oom-kill 0)" chk FAIL "$(runner_verdict inactive dead success 0)" chk FAIL "$(runner_verdict '' '' '' '')" # nothing readable is not a pass + chk FAIL "$(runner_verdict activating auto-restart success 4)" # the window where a flapping runner looks busy + chk FAIL "$(runner_verdict activating auto-restart oom-kill 2)" # same window, last result a kill + if printf '%s' "$(runner_verdict activating auto-restart success 4)" | grep -q 'RESTART WINDOW'; then + printf 'ok the restart window is named, not just failed\n' + else printf 'FAIL the restart window should be named in the message\n'; rc=1; fi # a restart policy that works shows NRestarts>0 and result success - which is a PASS with the count visible chk PASS "$(runner_verdict active running success 3)" - [ "$rc" = 0 ] && echo "ci-runner-health self-test: ok (6 cases)" || echo "ci-runner-health self-test: FAIL" + [ "$rc" = 0 ] && echo "ci-runner-health self-test: ok (9 cases)" || echo "ci-runner-health self-test: FAIL" return $rc } From 3a5953b1c35fc2141c240896f93241b5d6d5f450 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 18:36:14 -0400 Subject: [PATCH 3/3] ci-runner-health: the file the gate requires (CI's one-line red, #1546) `s scripts/ci-runner-health.sh` had the Copyright + SPDX pair but not the full Apache block, so the repo's "Copyright notice on every file" step failed on it: missing copyright notice: scripts/ci-runner-health.sh The block is now copied **verbatim from the gate's own `NOTICE` constant** (`tools/copyright.py:16`) rather than retyped, so the two cannot drift, and `python3 tools/copyright.py --check` is rc=0 over the whole repo. The check itself is unchanged: `--self-test` and the live run behave as verified at `6c9e5b7` (unprivileged, three system units, 0 = healthy, 2 = nothing to inspect). --- scripts/ci-runner-health.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/scripts/ci-runner-health.sh b/scripts/ci-runner-health.sh index 7f0a794a7..86a46c96b 100755 --- a/scripts/ci-runner-health.sh +++ b/scripts/ci-runner-health.sh @@ -2,6 +2,18 @@ # Copyright 2026 bong-water-water-bong # SPDX-License-Identifier: Apache-2.0 # +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# # ci-runner-health.sh - say whether the self-hosted GitHub runners are RUNNING, and refuse to be silent about a dead one. # # Why this exists (engine#1504): both runners died on an OOM kill at 08:37 on 2026-11-08 and stayed dead for ~3h45m with