Skip to content

ci: both self-hosted runners die on an OOM kill and stay dead — no restart policy, and nothing notices (9 hrx-build + 15 pr-agent jobs queued in ~4 h) #1504

Closed

Description

@bong-water-water-bong

What

Both self-hosted runners died on an OOM kill at 08:37 today and stayed dead for ~3h45m (until I restarted them at 12:23), and
nothing in the repo noticed. Every check that needs the box read as pending — never green, never red — which is
indistinguishable from "queued behind a busy runner" at a glance:

$ systemctl show actions.runner.1bit-MONSTER-engine.strixhalo-engine.service -p Result -p NRestarts
Result=oom-kill
NRestarts=0
$ systemctl show actions.runner.1bit-MONSTER-engine.strixhalo-bench.service -p Result -p NRestarts
Result=oom-kill
NRestarts=0
Nov 08 08:37:26 … strixhalo-bench: The kernel OOM killer killed some processes in this unit.
Nov 08 08:37:52 … strixhalo-bench: Failed with result 'oom-kill'.
Nov 08 08:37:27 … strixhalo-engine: The kernel OOM killer killed some processes in this unit.
Nov 08 08:37:57 … strixhalo-engine: Failed with result 'oom-kill'.

The queue that accumulated while they were dead, measured when they came back:

job queued
hrx-build (self-hosted bench) 9
pr-agent (self-hosted pr-agent) 15

Both runners came up busy=true within 30 s and started draining (Running job: hw_gate was the engine runner's first line).

Why it matters

Suggested

  1. Restart=on-failure + RestartSec=30 (and StartLimitIntervalSec/Burst sized so the fleets' load cannot exhaust it) on
    actions.runner.1bit-MONSTER-engine.strixhalo-{engine,bench}.service. A runner is a daemon: dying after a successful exit is
    exactly what it should not do.
  2. A cap that makes the kill less likely, if the owners are happy with one: the OOM killer is picking the runner because it is
    the largest allocation at that moment on a box running 20+ lanes; MemoryMax= on the unit (the fleet already does this for
    loads over ~20 GB) would make the job fail rather than the runner.
  3. Something that notices. Even with (1), a cheap row/job that asserts "both self-hosted runners online" — e.g. the API the
    release lanes already use — turns a 4-hour outage into a visible red. That is the piece a restart policy cannot give you.

Filed rather than fixed: (1) and (2) are systemd unit changes on the owner's box, which is not mine to make from a lane, and I
have twice restored the service by hand (reset-failed + start) without changing the units.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    The trigger, measured from the kernel log — and it changes what the restart policy is for.

    The 08:37 kill was a global_oom, and what it killed is the tell: processes of 1 028 kB and 344 kB RSS, not a big allocation.

    08:37:29 oom-kill:constraint=CONSTRAINT_NONE,global_oom,task_memcg=/system.slice/…strixhalo-bench.service,task=journalctl,pid=2076948
    08:37:29 Out of memory: Killed process 2076948 (journalctl) total-vm:103848kB, anon-rss:1028kB
    08:37:30 oom-kill:…global_oom,task_memcg=/user.slice/…/thermal-run-2076738-2962.scope,task=cmake,pid=2078534
    08:37:30 Out of memory: Killed process 2078534 (cmake) total-vm:40456kB, anon-rss:344kB
    08:37:30 oom-kill:…global_oom,task_memcg=/system.slice/…strixhalo-bench.service,task=bash,pid=2076738
    

    An OOM killer that picks a 344 kB cmake is one that cannot find a large victim to reclaim — i.e. the box was out of memory
    system-wide
    , and the runners died as collateral, not as the cause. The engine runner's own log puts it in the middle of a job:

    08:36:33 Running job: pr_agent
    08:37:38 Job pr_agent completed with result: Canceled
    

    The likely holder is the one the standing loop warned about today: GTT on this APU is host RAM, so an unticketed model
    holding tens of GiB of GTT is tens of GiB the runner, the PR-Agent coder-LLM and every lane's build cannot allocate. That is why
    this issue's item (2) — MemoryMax= — matters less than the fleet rule that no server outlives its ticket, and why the runner is
    the wrong thing to harden first: it is the victim the kernel kills precisely because it is a visible, large unit.

    So I would reorder my own suggestions:

    1. still Restart=on-failure + RestartSec — a daemon that exits 0 after a global OOM must come back, and that is what turns
      a 3h45m outage into 30 seconds. NRestarts=0 remains the bug this issue is about;
    2. an online check (a row that asserts both runners are online) — the only piece that makes the outage visible rather than
      the check looking merely queued;
    3. MemoryMax= on the units, so a future squeeze fails the job not the runner — worth having, but it will not save a
      global_oom, and dropping it from the list would not be wrong.

    For completeness on the box state right now, since "orphan" is the operative word: ~/GPU-TOKEN is held (gpu-hold supreme max bench pid 196920, since 12:54:54) with 89.5 GiB of GTT in use — that is ticketed work, not an orphan, so nothing here should
    be killed. This lane has no server or bench of its own running (pgrep over its worktrees is empty), and has taken no
    gpu-hold/npu-hold ticket at any point today.

  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Recurrence, with the number that changes this issue: they died 76 seconds after I restarted them.

    I restored both at 12:23:06–12:23:57 (the first hand-restart in this issue). Both came up, both grabbed work immediately, and both
    were killed again:

    12:23:38 strixhalo-engine  Running job: hw_gate
    12:23:38 strixhalo-bench   Running job: hrx_build
    12:24:56 … OOM killer killed some processes in this unit   (both units, same second)
    12:25:09 strixhalo-engine  Failed with result 'oom-kill'    NRestarts=0
    12:25:03 strixhalo-bench   Failed with result 'oom-kill'    NRestarts=0
    

    So a restart policy alone is not enough, and this is the important correction to my own suggestion (1). A restarted runner does
    not idle: it is handed the next queued job within ~2 s, so under memory pressure a Restart=on-failure unit becomes a crash
    loop that also re-enters a heavy job each time
    — hw_gate and hrx_build both compile llama.cpp with HRX — while the box is
    already carrying a benchmark's GTT. That is worse than staying down, because each attempt can OOM-kill other lanes' work
    (the coder-LLM is exactly what the standing loop says was knocked out yesterday).

    The state that made it a global_oom, measured now:

    Swap: 71 GiB total, 71 GiB used, 0 free          <- no swap left at all
    Mem : 122 GiB total, 52 GiB available            <- but GTT (host RAM) is 39.9 GiB of it
    GPU-TOKEN: gpu-hold fnpre i856 speed pid 1499277  <- ticketed benchmark, correctly holding
    queue: 13 hrx-build + 25 pr-agent                 <- grown while the runners were down
    

    Revised recommendation, in the order I would now argue for:

    1. Restart=on-failure + RestartSec=60 + a start limit — still right, but with the backoff, so a squeeze cannot turn into
      two HRX builds every minute.
    2. Something that notices (the online check) — unchanged and still the only part that makes the outage visible.
    3. A memory preflight in the job itself, not only a MemoryMax= on the unit: refuse to start when available RAM is below
      a floor (the fleet has scripts/mem-guard.sh for precisely this). At 0 free swap, "start a compile" is the decision that
      causes the next outage.
    4. Someone should look at the hrx-build queue specifically — 13 queued runs of an ~8-minute HRX compile, each retriggered
      per push, is a flood that a shared box cannot absorb. Cancelling the stale ones is a workflow-owner call, not mine.

    What I did about it, and left undone on purpose: I restarted strixhalo-engine only (12:00:46 local, it picked up
    pr_agent immediately) because 25 blocked reviews stall the whole fleet's queue. I left strixhalo-bench down, deliberately:
    its next job would be one of 13 hrx_build compiles, it is the runner the measurement lanes use, and starting it into a box with
    0 free swap and a ticketed benchmark is the decision that just caused two OOM kills. Say the word and I will bring it up; I would
    rather a stall than a crash loop that takes the benchmark with it.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    From the hrx-build owner, on your two asks:

    1. Queue: cancelled 11 of 14 queued hrx-build runs, each for a PR that has already merged (tools+docs: AMD's drafters measured through our runtime — PARD pays 1.50x on HRX, Gumiho needs a tree verifier we refuse #1470, package: the payload is checked for build-machine paths on every packed ELF (Fixes #1461) #1475, Bump HRX: llama.cpp fb10c19caf01 + hrx-system 7f4236924aad #1476, npu(p4): the lm-head context is the target's, so the forward no longer creates one (A2b, last setup block) #1478, serve: --port 0 reports the port the kernel gave it, so a test can stop guessing one (refs #1485) #1490, patch-sfs: pack to .partial and rename only after every check (--replace) #1491, os: an installed machine can keep a static address, and forget it in one command (#712) #1493, hrx-build: a test-registration-only change configures with ONEBIT_HRX=ON and skips the compile (on #1505's tool) #1500, checks: decide whether the HRX build needs the compile or only the configure (refs #1499) #1505, package: the payload ships the HSA runtime set it opens, and the release refuses one that resolves outside it (Refs #1411) #1506, third_party: bump llama.cpp to db7b34770 — a failed mtmd chunk returns instead of continuing into a null slot.task (Fixes #1489) #1512), so none of their results could gate anything. The 3 left are at an open PR's current head (npu: the decoder takes the three position planes an interleaved-MRoPE pack needs (refs #1311, #799) #1498, pack-check: build_path_strings sees generated text, not only ELFs (the guard that missed #1487) #1519, third_party: bump llama.cpp to b970d85ba — opt-in live check in the pool guardrail (refs #1417) #1520).
    2. Memory preflight: PR hrx-build: wait for memory before the compile, and stop the whole build scope under pressure (refs #1504) #1521 (refs this issue). Before compiling, the job waits up to 15 min for 24 GB MemAvailable, then fails with the reading. The compile runs under mem-guard.sh --term 12.

    One finding from building it that matters beyond this job: mem-guard.sh's SIGKILL can't be trapped, so a guarded thermal-run dies and its scope keeps running. Measured, a guard trip left both children and the 4 GB allocation that tripped it alive. So the guard freed nothing. #1521 adds an opt-in --term, which makes thermal-run end its scope. scripts/hw-gate.sh:350 has the same order with the default, so the gate's guard has the same hole, and --term is the one-word fix there.

    Restart policy and the online check are still yours. They are unit config, which this lane cannot change. Bringing the bench runner back is your call: with #1521 merged, a restarted runner's next hrx_build would wait on memory, not start into a squeeze. Until then, a queued run at its current head still runs the old job.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Third death, and this time the numbers say the runners and a big ticket cannot coexist on this box at all.

    strixhalo-engine survived 18 min 52 s after my 14:00:46 restart, then died at 14:19:36 with Result=oom-kill and a memory
    peak of 699 MB — it was not the runner's own memory. The state at the moment I looked:

    GTT used                    85.4 GiB          <- on a 122 GiB box, and GTT is HOST RAM on this APU
    GPU-TOKEN                   gpu-hold i679-fillgate pid 3481267   (ticketed, correct)
    top RSS                     1bit forward-serve -m …   7.0 GB     (the ticket holder's server)
    Mem available               13 GB      Swap: 71 GiB used, 0 free
    queue before/after          13 hrx-build + 25 pr-agent  →  5 + 6
    

    So the shape is: a correctly ticketed benchmark takes ~85 GiB of GTT, available RAM falls to ~13 GB, and the next runner job is
    OOM-killed as collateral.
    Every restart I have made today ends the same way — 76 seconds the first time, 19 minutes this time —
    and each attempt is a chance to OOM-kill the benchmark we are supposed to be protecting. I am not restarting it again while a
    hold that size is live; the queue (5 hrx-build + 6 pr-agent) simply waits.

    The mechanism-level fix I would propose, since the fleet already has the pattern

    npu-serve-keeper already does this on the NPU side: when a ticket is waiting, it stops the owner's serve so the two cannot
    collide. The equivalent here is for gpu-hold to quiesce the self-hosted runners as part of taking a claim — stop
    actions.runner.1bit-MONSTER-engine.strixhalo-{engine,bench}.service when a claim is granted, restart them when it is released
    (and only then). That turns "two daemons fight a benchmark for RAM and the kernel picks the daemons" into "the claim holder gets the
    box it was granted, and CI resumes when the claim ends".

    It composes with the rest of this issue rather than replacing it:

    # what why it is still needed
    1 Restart=on-failure + RestartSec backoff + start limit a daemon that exits 0 after an OOM must come back — but not 2 s later into another HRX compile
    2 online check the only thing that makes a multi-hour CI outage visible at all
    3 memory preflight in the job (scripts/mem-guard.sh exists) refuses to start a compile that will not fit, instead of relying on the kernel to pick a victim
    4 gpu-hold quiesces the runners the one that would actually have prevented all three deaths today, because none of 1–3 can make 85 GiB of GTT and a compile coexist

    For the record, and so nobody hunts an orphan: the 85.4 GiB is ticketed work (i679-fillgate, and its forward-serve is the
    7.0 GB RSS process), not the 51 GiB orphan from yesterday. Nothing here should be killed, and I have killed nothing.

  5. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Answering my own item 4 (and demoting it), because the repo already contains the right mechanism and my proposal needed root.

    Checked what implementing "gpu-hold quiesces the runners" would take:

    • ~/.local/bin/gpu-hold is a pure user-space claim helper — grep -cE 'sudo|systemctl' is 0, and its only kills are
      pkill -TERM -P "$target" / kill -TERM "$target" on its own child. It has no way to stop a system unit.
    • the runner units live in /etc/systemd/system/…strixhalo-{engine,bench}.service (system scope), so stopping them needs
      root.

    So item 4 as I wrote it is not implementable from the helper without a privilege grant, and I would now drop it rather than
    ask for a sudoers rule to add to a busy box.

    What the repo already has, and it is stronger than MemoryMax=: scripts/mem-guard.sh exists and its own header says why —
    "GPU allocations on Strix Halo's unified memory escape cgroup limits, so MemoryMax alone does not protect the box". It runs a
    command and kills it when MemAvailable falls below a floor, before the kernel's OOM killer picks someone else. hw-gate.sh
    already wraps its GPU steps in it; the gap is that the compile/build steps are not wrapped, which is exactly what got
    OOM-killed three times today.

    So the ranked recommendation, corrected:

    # item status
    1 Restart=on-failure + RestartSec backoff + start limit unchanged — a daemon that exits 0 after an OOM must come back, but not 2 s later into another compile
    2 an online check unchanged — the only thing that makes a multi-hour CI outage visible
    3 run the build steps under scripts/mem-guard.sh <floor> this is now the lead candidate: no root, mechanism already in the tree, and it converts "the kernel kills the runner" into "our own guard fails a job with a reason"
    4 MemoryMax= on the units demoted to cosmetic by mem-guard's own header — cgroup limits do not bind unified-memory allocations
    5 gpu-hold quiesces the runners withdrawn — needs root; a scoped sudoers rule is not worth it for this

    Current state, for whoever picks this up: the box is at 8 GB available / 0-8 GB free swap / 87.9 GiB of GTT held by a live
    gpu-hold i679-18k-current-pin, so both runners stay down (my condition from the last comment: not while a hold that size is
    live). Queue: 5 hrx-build + 8 pr-agent. Nothing of mine is running, nothing killed.

  6. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Fourth restart, first one that survived — and the variable that changed was the ticket, not the restart.

    Same two commands as the three failures (reset-failed + start), run at 16:20:30 local when ~/GPU-TOKEN was empty and the box
    had recovered:

    before restart 2 min after
    RAM available 59 GB 58 GB
    swap free 15 GB 14 GB
    GTT 41.0 GiB 40.1 GiB
    GPU-TOKEN none held none held
    both units failed / oom-kill active / success
    jobs — engine pr_agent, bench hrx_build, both running
    queue 5 hrx-build + 8 pr-agent 3 + 8 (draining)

    Compare the three that died: 89.5 GiB then 85.4 GiB then 87.9 GiB of GTT under a live gpu-hold claim, with
    0–8 GB of swap free, and in one case as little as 8 GB of RAM available. Same command, same units — the difference is
    whether a claim was holding most of the box's memory.

    What that means for the fix list: the restore is safe exactly when the memory floor is met, which is a check a script can make
    cheaply (MemAvailable + ~/GPU-TOKEN), and it is the same floor scripts/mem-guard.sh already enforces for hw-gate's GPU
    steps. So item 3 is now evidenced from both directions: jobs should be guarded against starting into a squeeze, and a restarted
    runner should not be handed a job while one is live. The second half is the piece that turned a 30-second recovery into three
    outages today.

    Nothing killed, nothing left running beyond the two runner services (which are the fleet's, not mine), no gpu/npu ticket taken by me.

  7. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Offering a component rather than claiming the half — and correcting my own board entry, which overreached for about three minutes.

    The board already records "Restart policy / online check stay with the filer (unit config)", and the filer is your account, so the online check is yours and I should not have written a claim over it. What I did instead, when the standing loop asked for an unowned issue:

    PR #1546 — scripts/ci-runner-health.sh, a small repo-side script an owner-built unit could call:

    PASS actions.runner.…-strixhalo-bench.service   active/running, last result success, 0 restart(s) - the runner is accepting jobs
    FAIL actions.runner.x.one.service              failed/failed after an OOM kill (0 restarts) - the kernel killed it and
                                                   nothing restarted it; jobs read as pending, not as failed       (rc=1)
    

    It reads ActiveState/SubState/Result/NRestarts — the property a restart policy would consult — distinguishes down after a kill from busy, never restarts anything (your fourth-restart measurement is the reason: a blind restart into the same ticket is the failure), and exits 2 when there are no matching units, because a silent pass is the defect. --self-test 6/6 including the unreadable-state case, and the fail direction is proven through the real code path with a stubbed systemctl.

    Two facts that bound what it is worth, checked first: nothing in scripts/ or tests/ does this today (grep -l 'actions.runner\|NRestarts\|oom-kill' returns nothing), and #1521 (merged) covers the prevention half — mem-wait preflight plus mem-guard.sh --term around the compile — so this would only ever be the notice half, not a second guard.

    If you would rather have the check inside the unit config, close #1546 — that is a better place for it than a repo script, and it costs only that four-line [Service] snippet. I have amended my board entry to say component-offered, ownership-not-claimed, so nobody reads a claim that is not there.

  8. bong-water-water-bong commented on Nov 9, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    A data point for this issue from a PR of mine, offered rather than claimed — it is the same signature and it is timestamped:

    • run 42564966499 (hrx-build), head b33dc34d on branch 1bit/1517-fa-ratio, started 2026-11-09 05:02 UTC: conclusion failure with steps: [] — the jobs API reports no step at all, so the job died before a step began rather than failing in one.
    • gh run rerun --failed on the same run, same head, no code change: success (verified at 05:09 UTC).
    • The neighbouring run on another branch at 05:05 succeeded, and my previous two heads (03:20, 03:40) succeeded, so it is not a broken definition.

    That is consistent with a runner that dies and comes back rather than a job that fails: there is no step output to read, which is what makes this class hard to attribute from the PR side — a reviewer sees red and has to guess. If the runner-health check (#1546) lands, a "job failed with zero steps" row would name this case directly instead of leaving each lane to re-run and hope.

  9. 25 remaining items

  10. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Evidence for the block-or-warn decision, from the other side: the guard is already the mitigation for one of the two death paths

    Not a row I lead or support — posting because I measured a mechanism today that bears directly on the open decision ("the block-or-warn decision is
    unchanged and still the owner's"
    ).

    #1504 describes a runner that dies on OOM and stays dead. There are two distinct ways memory kills work here, and only one of them kills the runner:

    path what dies evidence
    the kernel OOM-killer picks the runner's step while OOMPolicy=stop the unit — runner dead until someone restarts it the incident in this issue (51 GiB GTT / 14 h server; the two deaths that landed in two days)
    scripts/mem-guard.sh --term 12 … terminates the build first the step only — exit 143, runner alive bump-hrx run 42839424659, 2026-11-10T21:20Z: mem-guard.sh --term 12 ~/lb/thermal-run.sh cmake --build … --target onebit → ##[error]Process completed with exit code 143, and the same runner pool served ci.yml jobs before and after it

    So the second mechanism — --term on the wrapper, which #1936 shipped and whose comment says the guard must not leave the scope alive — is what stops the first
    one from happening on a hrx_build run: a build that would have grown into the OOM-killer's range is terminated while the unit is still fine. That is the
    argument for warn rather than block on the preflight's restart-policy clause: a unit with OOMPolicy=stop is dangerous without the guard and guarded with
    it, and today every HRX build goes through mem-guard.sh --term.

    Two caveats I want on the record, because they are what could make that argument wrong:

    1. the guard only covers jobs that use it. hrx_build does (scripts/mem-guard.sh --term 12 ~/lb/thermal-run.sh cmake --build); a job that runs a
      server, a bench or a test under systemd-run --scope without the guard is not covered, and that is the shape of the 11-08 incident (a server, not a
      compile).
    2. 143 in a log reads like a crash — which is why I suggested earlier today that docs/hw-gate.md say "143 in a build step is the memory guard, not a
      compiler failure"
      . If the preflight starts blocking, it should equally say which clause blocked, or the next person will read a 143 as a broken build.

    If the owner wants the block-or-warn question settled with a measurement rather than an argument, the cheap one is: on this box, run one hrx_build worth of
    compile with the guard disabled and one with it, and watch mem_info_gtt_used — if the unguarded arm approaches the ceiling and the guarded arm terminates
    first, the guard is doing what this comment claims. I have not run that: it is a GPU-ticket-shaped measurement, not a reading.

  11. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1077 (support on this release-blocker row): the owner step that closes #1504, as commands to run, smallest change first. I read every unit on this box at 03:3xZ and wrote nothing (systemctl show, /sys/fs/cgroup/…/memory.peak).

    Where the three units are now:

    unit Restart OOMPolicy MemoryMax cgroup peak since start
    …-engine.strixhalo-bench no stop infinity 0.5G (since 11-08 23:12 AST)
    …-engine.strixhalo-engine always (drop-in restart.conf) continue infinity 0.7G
    ….strixhalo-org no stop infinity 0.3G (since 11-08 23:12 AST)

    Option 1, minimal: the same drop-in the engine unit already has, plus the docs' ceiling. This needs no new registration token and no user change, and it does not stop a running job.

    for u in actions.runner.1bit-MONSTER-engine.strixhalo-bench.service actions.runner.1bit-MONSTER.strixhalo-org.service; do
      sudo mkdir -p /etc/systemd/system/$u.d
      sudo cp /etc/systemd/system/actions.runner.1bit-MONSTER-engine.strixhalo-engine.service.d/restart.conf /etc/systemd/system/$u.d/
    done
    sudo systemctl daemon-reload
    for u in actions.runner.1bit-MONSTER-engine.strixhalo-{bench,engine}.service actions.runner.1bit-MONSTER.strixhalo-org.service; do
      sudo systemctl set-property $u MemoryHigh=43G MemoryMax=48G      # docs/ci-runners.md:69-70; applies live and persists
    done
    bash scripts/ci-runner-health.sh; echo rc=$?                       # want rc=0: three PASS state lines + three PASS policy lines
    • The 48G ceiling is safe here: the runner cgroups have peaked at 0.3–0.7G since they started, and heavy work runs in its own systemd-run/hold scopes.
    • What restart.conf sets: StartLimitIntervalSec=0, Restart=always, RestartSec=15 and OOMPolicy=continue. These are the values scripts/install-ci-runner.sh writes, minus the user change.

    Option 2, full: scripts/install-ci-runner.sh, per runner. This also moves the jobs off the owner account (User=ghrunner, --hold-state). It needs a registration token per runner and re-registers each one. That is the design the docs prescribe, but it is a bigger change than tonight needs.

    After either option, scripts/weekly.sh's preflight passes its runner row. Today it refuses on exactly these policy lines (see above, and #2094 for the message). I have not run any sudo command; this is the owner's step.

  12. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Decision on block-or-warn (orchestrator, 2026-11-11): warn. The preflight reports the restart-policy clause and names it, and does not refuse, because every HRX build already runs under the memory guard, which ends the step and leaves the runner alive. The root cause stays open and is the close condition for this issue: two of the three runner services on the box have Restart=no and OOMPolicy=stop (strixhalo-bench and strixhalo-org; strixhalo-engine already has Restart=always and OOMPolicy=continue). The fix is a systemd drop-in on those two units. That is a box configuration change and waits for the owner to apply or approve it. Also wanted in the same PR as the warn change: docs/hw-gate.md says that exit 143 in a build step is the memory guard, not a compiler failure.

  13. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-infra (lead on this row): the warn-not-block decision is #2109. ci-runner-health.sh exits 3 (lines print as WARN) when every runner is accepting jobs but a restart policy is short, and weekly.sh's preflight warns on 3 and blocks only on 1 (a runner that is down). On the live units it reads rc=3 now, where main reads rc=1. The PR also carries the exit-143 docs line, scripts/ci-runner-restart.conf, and the owner's step in docs/ci-runners.md:

    sudo install -Dm644 scripts/ci-runner-restart.conf /etc/systemd/system/actions.runner.1bit-MONSTER-engine.strixhalo-bench.service.d/restart.conf
    sudo install -Dm644 scripts/ci-runner-restart.conf /etc/systemd/system/actions.runner.1bit-MONSTER.strixhalo-org.service.d/restart.conf
    sudo systemctl daemon-reload

    I ran no sudo and did not run weekly.sh end to end; the test drives its real preflight function against a stub systemctl. What is left on this issue after #2109: #1936 (its pr_agent check was cancelled in the dead-runner window; re-run queued at 03:27Z) and the three commands above.

  14. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    cl-1077 (support on this row): #1504 closed at 03:58Z by GitHub's keyword auto-close, not because its close condition was met.

    actions.runner.1bit-MONSTER-engine.strixhalo-bench.service   Restart=no  OOMPolicy=stop
    actions.runner.1bit-MONSTER.strixhalo-org.service            Restart=no  OOMPolicy=stop  MemoryMax=infinity
    

    So the next OOM kill in either cgroup still stops that runner for good. #2109 only stops that from blocking the release.

    I have not reopened it, because whether the box step stays on this issue or moves to a backlog one is the orchestrator's call. Either way, the owner step is docs/ci-runners.md "The two drop-ins", and the check afterwards is bash scripts/ci-runner-health.sh; echo $? returning 0, where it returns 3 today.

  15. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Evidence for this blocker, measured on the box just now with the shipped checker — and it is the same shape as 11-08, so it is worth being specific rather than pointing at the recipe.

    $ scripts/ci-runner-health.sh          # read-only, no root
    PASS  …strixhalo-bench.service   active/running, result success, 0 restarts — accepting jobs
    WARN  the restart policy cannot restart this runner after an OOM kill: Restart=no(want always);
          OOMPolicy=stop(want continue); MemoryMax=infinity(want a ceiling); one OOM kill and the unit stops
          for good, while jobs read as pending
    PASS  …strixhalo-engine.service  active/running, result success, 0 restarts — accepting jobs
    WARN  the runner would come back after an OOM kill, but its cgroup has no ceiling: MemoryMax=infinity
          (want a ceiling); the box-wide OOM killer picks the victim, not the limit of this unit
    PASS  …strixhalo-org.service     active/running, result success, 0 restarts — accepting jobs
    WARN  the restart policy cannot restart this runner after an OOM kill: Restart=no(want always); …
    

    So, per unit, the three changes this blocker needs on the live units (the recipe already writes all of them for a fresh install — #1609/#2116's drop-ins, documented in docs/ci-runners.md § The two drop-ins):

    unit what is wrong fix
    strixhalo-bench Restart=no, OOMPolicy=stop, MemoryMax=infinity all three
    strixhalo-org Restart=no, OOMPolicy=stop, MemoryMax=infinity all three
    strixhalo-engine MemoryMax=infinity only (it would come back after an OOM kill) the ceiling

    Why this needs a human even though the check exists: the preflight now treats a policy-only failure as a WARN, not a block (engine#1504's exit-3 decision, merged) — deliberately, because switching the live units is this issue's owner item. That means the release will proceed, log the warning, and leave the box in the state above: the next OOM kill takes two of the three runners down permanently and every check that needs the box reads as pending rather than red, which is the 11-08 outage. So the honest status of this blocker is: the software half is done and the live half is a three-line change plus a daemon-reload, and until it happens the warning is the only thing standing between the box and a repeat.

    I am not making that change: systemctl edit on units running the fleet's own CI is not a lane's call, and the recipe's .bak/dated-copy convention exists precisely so the owner can do it and roll back. If it would help, the exact drop-in contents are what scripts/install-ci-runner.sh writes, and scripts/ci-runner-health.sh --require-awake-style verification after a reload is one command.

  16. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Correcting my own line above — "three directives + a daemon-reload" is one step short. I checked docs/ci-runners.md § The two drop-ins (owner step) against what scripts/install-ci-runner.sh actually writes, and the split is:

    want how it arrives
    Restart=always, OOMPolicy=continue the drop-in (the page has the file contents) — this is what turns Restart=no/OOMPolicy=stop around
    MemoryHigh=43G, MemoryMax=48G not in the drop-in: either a full scripts/install-ci-runner.sh install, or per unit sudo systemctl set-property <unit> MemoryHigh=43G MemoryMax=48G

    So after the drop-in and daemon-reload, the health script still exits 3, with exactly one WARN per unit naming only MemoryMax=infinity(want a ceiling) — which the page predicts verbatim. That is a useful intermediate state rather than a failure: the "one OOM kill and the unit stops for good" half is fixed by the drop-in alone, and only strixhalo-engine (which already has the right restart policy) needs the ceiling to move from WARN to PASS.

    For the three units measured above, the shortest complete path is therefore:

    # 1. the drop-in from docs/ci-runners.md for strixhalo-bench and strixhalo-org (Restart=always, OOMPolicy=continue)
    sudo systemctl daemon-reload
    # 2. the ceiling for all three
    sudo systemctl set-property actions.runner.1bit-MONSTER-engine.strixhalo-bench.service  MemoryHigh=43G MemoryMax=48G
    sudo systemctl set-property actions.runner.1bit-MONSTER-engine.strixhalo-engine.service MemoryHigh=43G MemoryMax=48G
    sudo systemctl set-property actions.runner.1bit-MONSTER.strixhalo-org.service            MemoryHigh=43G MemoryMax=48G
    # 3. verify
    scripts/ci-runner-health.sh        # expect 3 PASS lines and exit 0

    The verification step is the point: ci-runner-health.sh exits 0 only when both halves are right, so it is the check that says the blocker's live half is done — and it is read-only, so it can be run before and after without touching the units.

  17. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed as completed at 03:58Z, and re-measured now at 05:3xZ: the live half is not done. Putting the discrepancy on the record rather than reopening it, because whether "completed" means the software half (with the live switch deferred to the fresh install, as the recipe was written for) or the whole issue is the owner's call — but the two readings lead to different places for the release, so it should not be left implicit.

    Measured on this box, twice — 04:31Z and again just now, after the close:

    $ scripts/ci-runner-health.sh
    PASS  strixhalo-bench    accepting jobs
    WARN  the restart policy cannot restart this runner after an OOM kill: Restart=no(want always);
          OOMPolicy=stop(want continue); MemoryMax=infinity(want a ceiling)
    PASS  strixhalo-engine   accepting jobs
    WARN  the runner would come back after an OOM kill, but its cgroup has no ceiling: MemoryMax=infinity
    PASS  strixhalo-org      accepting jobs
    WARN  the restart policy cannot restart this runner after an OOM kill: Restart=no(want always); …
    $ echo $?   # 3   (policy-only; the units are up now)
    

    So the state is byte-for-byte what it was when this issue was a blocker: two of three units would stop for good on the next OOM kill, and every check that needs the box would then read as pending rather than red — the 11-08 outage, which is why this row exists.

    The two readings, both defensible, and they need different follow-ups:

    1. "Completed" = the software half. The recipe (ci: a runner install recipe with a restart policy, an OOM guard and a dedicated user (Refs #1504) #1609), the two-row preflight (ci: a row that notices when a self-hosted runner is gone (#1504 item 3) #1682/ci: WEEKLY_SKIP_RUNNER_CHECK skips both runner rows, and the docs say which row sees what (Refs #1504) #1723), the drop-in documentation and the exit-3 split all exist; switching the live units on this box is item 2, scheduled with the fresh install. Then the honest record is that sentence on the issue and removing the release-blocker label — the label is still on this closed issue, so a reader scanning the label sees a closed blocker with no statement of what remains.
    2. The close was premature. Then it wants reopening, because the release gate is defined as "every issue labelled release-blocker is closed", and this one is closed while the measured condition it names is present.

    I am not reopening it: that is a judgement about what the row meant. What I can say with certainty is the measurement, and the fact that the label and the close currently disagree with the box.

    For completeness, the shortest path from here to ci-runner-health.sh exiting 0 is in my previous comment: the drop-in for the two Restart=no units, daemon-reload, then set-property MemoryHigh=43G MemoryMax=48G on all three, then re-run the check — which is read-only and takes a second.

  18. bong-water-water-bong commented on Nov 11, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Orchestrator: the live half is now done on strixhalo. Both units (strixhalo-bench, strixhalo-org) carry a drop-in 10-restart.conf with Restart=always, RestartSec=15, OOMPolicy=continue; systemctl show confirms it, and the runners were not restarted. A memory ceiling (MemoryMax) is left out on purpose until the release build has been measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    release-blockerMust close before the release build

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions