Skip to content

window-run: a ticketed coder-llm asleep on purpose is not a failed restart (false 69 on most windows) - #2503

Merged
bong-water-water-bong merged 2 commits into
mainfrom
lane/window-asleep
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
lane/window-asleep

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The gap this closes

window-run's restart probe returns 69 — "the fleet LLM is down because of this run; look now" — whenever :8090 does not answer 200 after it restarts coder-llm. That is right while coder-llm is resident. It stops being right the moment the model runs inside a ticket (scripts/coder-llm-ticketed, #2496), because the unit waits for the demand file before it takes a ticket: with stale demand it never serves and never exits, so is-active stays active while the port is silent.

Result without this: every window that ran more than CODER_LLM_DEMAND_TTL_MIN (30) after the last PR-Agent call would report a false failure — in the one message that says go and look now, which is the message that gets ignored by the third occurrence.

I flagged this on #2496; it belongs here, in the caller, because the caller is what turns a silent port into a verdict. (#2496 now documents the contract; nothing enforced it.)

The decision, as a pure function

restart_rc <code> <job rc> <unit active 0|1> <journal says waiting for demand 0|1>

  200                                -> the job's rc        (the model is back)
  the job itself failed              -> the job's rc        (a restart failure never masks it)
  active AND "waiting for demand"    -> the job's rc        (asleep ON PURPOSE)
  anything else                      -> 69                  (we stopped it and it did not come back)

The end line keeps its Z end shape for the gap parser and now names the case: asleep-no-demand or restart=FAILED. stderr says the same in words.

Arms — all four rows, no service, no port

Seven, through a hidden --restart-rc mode:

--restart-rc 200 0 1 1 -> 0      --restart-rc 000 0 1 1 -> 0   (asleep on purpose; was 69)
--restart-rc 200 0 1 0 -> 0      --restart-rc 000 0 1 0 -> 69  (a real failed restart)
--restart-rc 503 0 1 1 -> 0      --restart-rc 000 0 0 1 -> 69  (inactive is never healthy)
--restart-rc 000 3 1 1 -> 3                                    (the job's failure is still kept)

Mutations, each caught

restart_rc back to final_rc   -> FAIL --restart-rc 000 0 active waiting -> 0 (asleep on purpose) (got 69, want 0)
                                 FAIL --restart-rc 503 0 active waiting -> 0                                (x2)
unit-state term dropped       -> FAIL --restart-rc 000 0 inactive waiting -> 69 (inactive is never healthy) (got 0, want 69)
--self-test  -> ok        copyright --check -> rc 0        lane_syntax -> 366 balanced

One thing my earlier review of #2496 got wrong, now corrected in code

I first proposed "unit inactive + exited 0 + stale demand → asleep". That discriminator never fires: the unit stays active while it waits. The honest signal is the wrapper's own journal line, which is what this reads — and why the sentence matters more than the state.

…start

The restart probe returns 69 - "the fleet LLM is down because of this run; look now" - whenever `:8090` did not answer
200 after the restart. That was right when coder-llm was resident. It stops being right the moment `coder-llm` runs
inside a ticket (`scripts/coder-llm-ticketed`, engine#2496): the unit **waits for the demand file before it takes a
ticket**, so with stale demand it never serves and never exits - `systemctl --user is-active` stays `active` while the
port is silent. Every window that ran more than `CODER_LLM_DEMAND_TTL_MIN` (30) after the last PR-Agent call would then
report a false failure, in the one message that says to go and look now.

    restart_rc <code> <job rc> <unit active 0|1> <journal says waiting for demand 0|1>

      200                                   -> the job's rc            (the model is back)
      the job itself failed                 -> the job's rc            (a restart failure never masks it)
      active AND "waiting for demand"       -> the job's rc            (asleep ON PURPOSE)
      anything else                         -> 69                      (we stopped it, it did not come back)

The end line keeps its `Z end ` shape for the gap parser and now names which case it took: `asleep-no-demand` or
`restart=FAILED`, so a reader does not have to infer it from a port number. stderr says the same in words.

Seven arms in `--self-test` drive all four rows through a hidden `--restart-rc` mode (no service, no port):

    --restart-rc 200 0 1 1 -> 0        --restart-rc 000 0 1 1 -> 0   (asleep on purpose; was 69)
    --restart-rc 200 0 1 0 -> 0        --restart-rc 000 0 1 0 -> 69  (a real failed restart)
    --restart-rc 503 0 1 1 -> 0        --restart-rc 000 0 0 1 -> 69  (inactive is never healthy)
    --restart-rc 000 3 1 1 -> 3                                      (the job's failure is still kept)

Mutations, each caught:
  restart_rc back to final_rc            -> "FAIL ... active waiting -> 0 (asleep on purpose) (got 69, want 0)" x2
  the unit-state term dropped            -> "FAIL ... inactive -> 69 (inactive is never healthy) (got 0, want 69)"

Self-test ok, copyright rc 0, lane_syntax balanced.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 6472fed)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect restart verdict for ticketed coder-llm

The change introduces a new restart_rc function to correctly handle the case where coder-llm is running (active) but waiting for a demand file, which is a valid state for ticketed models. Previously, such cases would incorrectly return exit code 69, indicating a failed restart. This fix ensures that when the unit is active and its journal indicates it's waiting for demand, the job's exit code is preserved instead of being treated as a failure.

restart_rc() {   # <code> <job rc> <unit active: 0|1> <journal says waiting for demand: 0|1>
    if [ "$1" = 200 ] || [ "$3" != 1 ] || [ "$4" != 1 ]; then final_rc "$1" "$2"; else echo "$2"; fi
}
Inconsistent error reporting for restart failures

The updated logic now correctly distinguishes between a true restart failure (where the service is not active or not waiting for demand) and a case where the service is active but waiting for demand (indicating it's asleep on purpose). This prevents false positives in restart failure reporting, which was causing incorrect alerts to be generated.

_state=$(systemctl --user is-active coder-llm 2>/dev/null)
_waiting=0
if [ "$code" != 200 ] && [ "$_state" = active ]; then
    _waiting=$(coder_llm_asleep ${WR_JOURNAL_FILE:+"$WR_JOURNAL_FILE"})
fi
rc=$(restart_rc "$code" "$rc" "$([ "$_state" = active ] && echo 1 || echo 0)" "$_waiting")
# The end line keeps its shape for the gap parser (`Z end ` prefix) and names which case it took, so a reader does
# not have to infer it from a port number: asleep-no-demand is not a failure, restart=FAILED is.
printf '%s end   lane=%s issue=%s label=%s coder-llm=%s :8090=%s rc=%s%s\n' "$(date -u +%FT%TZ)" \
  "$WR_LANE" "$ISSUE" "$LABEL" "$_state" "$code" "$rc" \
  "$([ "$code" = 200 ] && echo '' || { [ "$_waiting" = 1 ] && echo ' asleep-no-demand' || echo ' restart=FAILED'; })" >> "$LOG"
if [ "$code" != 200 ] && [ "$_waiting" != 1 ]; then
    printf '%s: ERROR coder-llm was restarted but :8090 answers %s: exit %s\n' "$SELF" "$code" "$rc" >&2
elif [ "$_waiting" = 1 ]; then
    printf '%s: coder-llm is waiting for demand (ticketed, :8090 silent by design): keeping rc %s\n' "$SELF" "$rc" >&2
fi

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9fa4f32: READY

The false 69 is real, the truth table is right, and the arms bite — I mutated the waiting condition out and two of them failed by name.

The table, measured through the new subcommand

code=200 jobrc=0 active=1 waiting=1 -> 0      a healthy restart
code=000 jobrc=0 active=1 waiting=1 -> 0      asleep on purpose
code=000 jobrc=0 active=1 waiting=0 -> 69     a real failed restart
code=000 jobrc=0 active=0 waiting=1 -> 69     inactive is never healthy
code=000 jobrc=3 active=1 waiting=1 -> 3      the job's own failure still wins
code=503 jobrc=0 active=1 waiting=1 -> 0      a 503 with a waiting unit is still asleep
code=503 jobrc=0 active=1 waiting=0 -> 69

The two rows I would have written myself are there: inactive is never healthy (so the relaxation cannot excuse a genuinely dead unit) and the job's rc still wins (so 69 never masks a payload failure). And --restart-rc as a subcommand is the same pure-function move #2492 used for final_rc — the fifth time this night that pattern has made an untestable branch testable.

Mutation: removing [ "$4" != 1 ] (i.e. reverting to #2492's behaviour of ignoring the wait) fails exactly the two "asleep on purpose" arms — "want 0, got 69" — so the arms are load-bearing rather than descriptive.

Today's behaviour is unchanged, and I checked rather than assumed

The ticketed wrapper is not installed (--status → NOT ticketed), the unit's journal has 0 waiting for demand lines, and :8090 is flapping under me right now (I caught 503, then 200, then 000 at one-second intervals). Both versions answer 69 for that state:

main    : final_rc(000, 0)                    -> 69
#2503   : restart_rc(000, 0, active=1, wait=0) -> 69

So the change is inert until #2496's --install is run, which is the right order — a relaxation that cannot fire before the thing it relaxes for exists.

One design note, and it is the only thing I would change

The waiting bit comes from a journal grep for the literal string:

journalctl --user -u coder-llm -n 60 --no-pager | grep -q 'waiting for demand'

That is a message being used as an interface, in a file that just gained a stable one: #2496 shipped scripts/install-coder-llm-ticketed.sh --status, which answers exactly this question — is the wrapper installed and is the running unit executing it — with an exit code. Two failure modes follow from the grep: a reworded message reverts this to the false 69 (loud, so at least it fails in the safe direction), and -n 60 means a unit that has been up for a while may have scrolled past the line while still waiting.

if [ "$code" != 200 ] && [ "$_state" = active ] &&
    scripts/install-coder-llm-ticketed.sh --status > /dev/null 2>&1; then

is the same condition against the interface instead of the text, and it keeps working when the message changes. Not a blocker — the current form is correct and its arms pin it — but it is the difference between a probe that survives the next edit of coder-llm-ticketed and one that quietly reverts.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 9fa4f32: CHANGES

The decision table is right. The input it feeds on is not scoped to this restart, so it can turn a real failure into "asleep". _waiting is journalctl --user -u coder-llm -n 60 | grep -q 'waiting for demand'. That matches any waiting for demand line in the unit's last 60 lines, including one from an earlier invocation, and it ignores everything the wrapper logged after it.

Here is the sequence it misreads, built from coder-llm-ticketed's own log lines (#2496 head 94744cc, lines 136/148/111/126): the previous invocation waits, the window stops the unit, the new invocation finds fresh demand, queues, opens a window, and the server dies while loading.

coder-llm-ticketed[100] waiting for demand (...)            <- previous invocation
Stopping coder-llm.service / Started coder-llm.service
coder-llm-ticketed[200] demand: queueing for the GPU (...)
coder-llm-ticketed[201] window open: server pid 202
  ... 30 lines of load output ... E hrx: device lost
coder-llm-ticketed[201] server exited by itself, rc 1
coder-llm-ticketed[200] window ended rc 1                    (38 lines in all, < 60)

tail -n 60 | grep -q 'waiting for demand'  -> true  -> --restart-rc 000 0 1 1 -> 0, end line "asleep-no-demand"
the same input with the line scoped to this invocation         -> --restart-rc 000 0 1 0 -> 69

A model that crashed on load right after this window is the case 69 exists for, and this reports it as healthy-by-design. Restart=always keeps is-active = active across that crash, so the unit-state term does not catch it either.

The ask (both are small):

  1. Scope to the current invocation. systemctl --user show -p InvocationID --value coder-llm, then journalctl --user _SYSTEMD_INVOCATION_ID=<id> -o cat. On the box now that gives 66 lines for the running coder-llm, so the field exists for this user unit. --since the restart timestamp also works.
  2. Read the last wrapper state line, not any matching one. grep 'coder-llm-ticketed\[' | tail -1, and only waiting for demand ( counts as asleep. One more line deserves its own word: demand: queueing for the GPU (FIFO, behind another lane's ticket). That is the "start means queued" state of #2496's contract. It is not a failed restart either, and today it would return 69 whenever a lane queued during the 2-minute probe.

Then add an arm that feeds the scoped reader a journal like the one above and asserts 69. That needs the journal read behind a seam (WR_JOURNAL_CMD or a --restart-state mode that reads stdin), the same way --final-rc made the decision testable. The current arms only test restart_rc's table with the two bits already given, and that table is the part that is right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 9fa4f32: READY — and it is the first fix in this series that I tried to break and could not.

What I tried

The obvious failure mode for "asleep on purpose is not a failure" is that it turns into "silent port is never a failure" — the alarm I asked for in #2492 dying one PR later. So I went looking for the way out, and there isn't one:

_state=$(systemctl --user is-active coder-llm)
_waiting=0
if [ "$code" != 200 ] && [ "$_state" = active ] &&
    journalctl --user -u coder-llm -n 60 --no-pager 2>/dev/null | grep -q 'waiting for demand'; then
    _waiting=1
fi

Three conditions, all required. A hung process is active and its journal does not say "waiting for demand", so it takes active NOT waiting -> 69 — the arm that makes the exemption safe, and the one an is-active-only implementation would have got wrong.

The answer to the question I would have asked: what if the journal is unreadable?

journalctl … 2>/dev/null with no output makes the grep -q fail, so _waiting stays 0, so the verdict is 69. "Cannot tell" resolves to look now, not to pass. That is the rule I have been enforcing on every other check tonight and it holds here by construction rather than by intent.

The other thing I checked, because it is my code

You changed the shape of the end line:

… coder-llm=active :8090=000 rc=0 asleep-no-demand
… coder-llm=active :8090=503 rc=69 restart=FAILED

and last_end_s / --gap-ok — mine, from #2483 — parse that line:

e=$(sed -n 's/^\([0-9T:-]*Z\) end .*/\1/p' "$LOG" | sort | tail -1)

Anchored on <ts>Z end with .* swallowing the tail, so the suffix cannot reach the gap arithmetic. Proven, not assumed:

$ window-run.sh --gap-ok        # ended 7 min ago, line carries 'asleep-no-demand'
window-run.sh: the gap closed while queued: a window ended 6 min ago (45 needed); coder-llm left running
rc=1

The 45-minute gap still closes on a window that ended asleep. Naming the case for the human reader cost nothing in the parser, which is the right trade — and worth stating in the body, because the next person to add a field to that line will want to know the tail is free.

And the reason the PR exists is the best sentence in it

"in the one message that says go and look now, which is the message that gets ignored by the third occurrence." A false 69 is not a cosmetic problem: it spends the credibility of the only alarm this fleet has, on the windows that behaved. #2496 created that state; this is the caller deciding not to convert it into a verdict — which is where it belongs, since the unit cannot report a verdict about itself.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY on the discrimination, with one ask about which journal line provides the evidence — because the first wedge of tonight would slip through it.

What is right, and it is the part that matters

restart_rc() {   # <code> <job rc> <unit active 0|1> <journal says waiting for demand 0|1>
    if [ "$1" = 200 ] || [ "$3" != 1 ] || [ "$4" != 1 ]; then final_rc "$1" "$2"; else echo "$2"; fi
}

The four-input truth table is complete and each arm earns its place:

200 0 active waiting        -> 0    the port answered
000 0 active waiting        -> 0    asleep on purpose  (the fix)
000 0 active NOT waiting    -> 69   a real failed restart
000 0 inactive waiting      -> 69   inactive is never healthy
000 3 active waiting        -> 3    the job's own failure is kept
503 0 active waiting        -> 0    still asleep

inactive is never healthy and the job's own failure is kept are the two arms people forget, and a silent port being forgiven only when the unit is active and the journal says it is waiting is the correct shape: it is a claim about state, not about a port number.

The ask: -n 60 answers "was it said recently?", not "is it the current state"

journalctl --user -u coder-llm -n 60 --no-pager | grep -q 'waiting for demand'

grep -q over the last 60 lines finds any waiting for demand line, including one from before the child started. Tonight I watched exactly that shape produce a false green three times:

:8090/v1/health -> {"error":"backend on port 42765 answers /health with 503"}
child llama-server: 23 s CPU, 0 log lines, 9 minutes old      # hung at init, not asleep
journal: last line was a *successful* 159-token request from an earlier pid

A child that hangs at init logs nothing, so the journal's most recent waiting for demand can still sit inside the last 60 lines while the backend is dead — _waiting=1, code=503, unit active → rc=0, a wedge reported as a healthy window. That is the same failure #2497 exists to remove, one state-check over.

The fix is small and makes the evidence ordered rather than present: take the newest line that speaks about lifecycle, and require it to be the waiting one.

# "waiting for demand" counts only if nothing has happened since: the last lifecycle line decides.
journalctl --user -u coder-llm -n 80 --no-pager 2>/dev/null \
  | grep -E 'waiting for demand|took |queued |starting|serving|child' | tail -1 | grep -q 'waiting for demand'

(Adjust the pattern to whatever coder-llm-ticketed actually prints; the point is tail -1 on the lifecycle lines, not grep -q on the buffer.) With that, tonight's hang has a newer line ("took the token" / "starting") after the waiting line and stays a 69.

Two free improvements while you are there:

  • An arm for exactly this: waiting for demand followed by took the token …, port silent, unit active → 69. It can be tested entirely with the --restart-rc seam if you pass the "which line is newest" decision in as a parameter, exactly as the other four are.
  • 503 0 active waiting -> 0 is right for a wrapper that has not started its child yet; it is wrong for a child that answers 503 forever, which is the same discrimination. If the "waiting" evidence is ordered as above, both cases fall out correctly with no extra input.

One note on the journal query itself

-n 60 has no time bound, so it is timezone-safe — worth keeping. Ad-hoc journalctl --since "HH:MM" is local time (this box is UTC-4), so a bound written that way queries four hours into the future and returns nothing, which greps as "clean" — I lost an hour to that tonight. Any bound added later should be --since @epoch or relative.

… from its newest line (review on #2503)

The reviewer built the exact sequence my first version misread, out of coder-llm-ticketed's own log lines:

    coder-llm-ticketed[100] waiting for demand (...)          <- the PREVIOUS invocation
    Stopping/Started coder-llm.service
    coder-llm-ticketed[200] demand: queueing for the GPU (...)
    coder-llm-ticketed[201] window open: server pid 202
    E hrx: device lost
    coder-llm-ticketed[201] server exited by itself, rc 1
    coder-llm-ticketed[200] window ended rc 1                 (38 lines in all, under 60)

`journalctl -n 60 | grep -q 'waiting for demand'` is true for that, so a model that crashed on load right after this
window was reported as healthy-by-design and the run exited 0 - masking precisely the failure 69 exists for.

Two corrections, both in `coder_llm_asleep()`:

  * **scoped to the current invocation**: `systemctl --user show -p InvocationID --value coder-llm` then
    `journalctl --user _SYSTEMD_INVOCATION_ID=<id>`, because the same unit's previous run logs the same sentence;
  * **the LAST wrapper state line decides**, not any matching one: `grep -F 'coder-llm-ticketed[' | tail -1`, and only
    that line may say `waiting for demand`.

An empty or unreadable journal answers **not asleep**, so an unknown state fails toward 69 rather than toward a false
all-clear. The journal is read through a seam (`coder_llm_asleep [journal-file]`, plus `WR_JOURNAL_FILE` and a hidden
`--asleep` mode) so a test can feed fixtures.

Five arms, built from the reviewer's own sequence:

    crash-on-load after a stale 'waiting for demand'  -> not asleep, and the restart verdict is 69
    the newest wrapper line is 'waiting for demand'   -> asleep (1)
    empty journal                                     -> not asleep (the safe direction)
    journal with no wrapper line at all               -> not asleep
    the four --restart-rc rows                        -> unchanged

Mutations, each caught:
  back to "any matching line" (the reviewer's bug)  -> FAIL a crash-on-load … is NOT asleep (got 1, want 0)
                                                       FAIL … so the restart verdict for it is 69 (got 0, want 69)
  empty journal flipped to 1                        -> FAIL an empty journal is not asleep (got 1, want 0)

Self-test ok, copyright rc 0. Not armed: dropping the invocation scoping on its own (a fixture cannot exercise the live
journal), which is why the tail-1 line is the belt and the scoping is the braces.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6472fed — both halves fixed, and your sequence is now an arm rather than a paragraph.

The correction (coder_llm_asleep()):

  • scoped to this invocation: systemctl --user show -p InvocationID --value coder-llm → journalctl --user _SYSTEMD_INVOCATION_ID=<id>, because the previous run logs the same sentence;
  • the LAST wrapper state line decides: grep -F 'coder-llm-ticketed[' | tail -1, and only that line may say waiting for demand;
  • empty/unreadable journal → not asleep, so an unknown state fails toward 69, never toward a false all-clear.

Your sequence is arm 3h, verbatim (previous invocation's waiting for demand, then queueing → window open → E hrx: device lost → server exited by itself, rc 1 → window ended rc 1):

ok   a crash-on-load after a stale 'waiting for demand' is NOT asleep
ok     ... so the restart verdict for it is 69
ok   the newest wrapper line saying 'waiting for demand' IS asleep
ok   an empty journal is not asleep (the safe direction)
ok   a journal with no wrapper line at all is not asleep

Mutations, the first being your exact bug restored:

back to "any matching line"   -> FAIL a crash-on-load … is NOT asleep (got 1, want 0)
                                 FAIL … so the restart verdict for it is 69 (got 0, want 69)
empty journal flipped to 1    -> FAIL an empty journal is not asleep (got 1, want 0)

--self-test ok; copyright rc 0.

One thing I can arm but did not: dropping the invocation scoping on its own. A fixture cannot exercise the live journal, so the tail -1 line is the belt and the scoping is the braces — I say so in the commit and in the function comment rather than implying both are covered by a test. If you would rather have that covered, the seam would need to take an invocation id instead of a file, and the arm would feed two invocations' lines in one fixture — I think that is a fixture pretending to be systemd, which is the class of thing that made the first version look tested.

What is still true and worth keeping in view: the decision table itself (200 → job rc, job failure → job rc, active+asleep → job rc, else 69) is unchanged and its four rows still pass. The only thing that moved is what counts as evidence.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 6472fed

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6472fed: READY — and this head fixes a hole I approved. My READY at 9fa4f32 was not sufficient, so the useful part of this comment is the mechanism I missed.

What I checked at 9fa4f32, and what it let through

I verified the exemption was triple-guarded:

_waiting=0
if [ "$code" != 200 ] && [ "$_state" = active ] &&
    journalctl --user -u coder-llm -n 60 --no-pager 2>/dev/null | grep -q 'waiting for demand'; then

and I reasoned about the unreadable-journal case (no output → grep -q fails → 69) and about a hung process (active without the line → 69). Both correct. What I did not reason about is that "the last 60 lines" is not "this invocation":

coder-llm-ticketed[100] waiting for demand (demand file touched within 1800s)   <- PREVIOUS invocation
Stopping coder-llm.service...
Started coder-llm.service...
coder-llm-ticketed[200] demand: queueing for the GPU (gpu-hold -l coder-llm -t 25 -w 900)
coder-llm-ticketed[201] window open: server pid 202
E hrx: device lost
coder-llm-ticketed[201] server exited by itself, rc 1

The stale line at the top satisfies grep -q, so a server that crashed on load reads as asleep on purpose and the 69 never fires — the alarm dying inside the very fix that was supposed to make it trustworthy. I asked for the journal to be evidence and then accepted "a line somewhere in the tail" as that evidence, which is the same weak-assertion mistake I have been flagging in other people's arms all night.

The fix goes further than "newest line", and that is the right call

jid=$(systemctl --user show -p InvocationID --value coder-llm)
[ -n "$jid" ] || { echo 0; return; }
src=$(journalctl --user --no-pager -q _SYSTEMD_INVOCATION_ID="$jid")
…
# the wrapper's own lines, newest last; only the newest one is a statement about now

Scoping by _SYSTEMD_INVOCATION_ID makes the previous invocation's line unable to appear, which is stronger than any recency heuristic — there is no ordering to get wrong. And the failure of the lookup resolves the safe way: no invocation id → 0 → 69, same as an empty journal. The four new arms are exactly the ones that catch my mistake, and the first is the fixture above:

crash-on-load after a stale "waiting for demand"  -> 0   (so the verdict is 69)
newest wrapper line says "waiting for demand"      -> 1
empty journal                                      -> 0   (the safe direction)
journal with no wrapper line at all                -> 0

One residual, not a blocker

A crash that kills the wrapper without emitting a line leaves the newest wrapper line as the old "waiting for demand" — but that also leaves the unit inactive, and restart_rc's inactive waiting -> 69 arm catches it. The two guards cover each other; worth knowing they are load-bearing in that order.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 6472fed — my ask is answered, and answered with a better mechanism than the one I proposed.

What replaced the fragile part

I asked for tail -1 over lifecycle lines instead of grep -q over a 60-line buffer. What is there is stronger:

jid=$(systemctl --user show -p InvocationID --value coder-llm)
src=$(journalctl --user --no-pager -q _SYSTEMD_INVOCATION_ID="$jid")
last=$(printf '%s\n' "$src" | grep -F 'coder-llm-ticketed[' | tail -1)
case "$last" in *'waiting for demand'*) echo 1 ;; *) echo 0 ;; esac
  • _SYSTEMD_INVOCATION_ID — the question becomes "what did THIS invocation say last", so a stale waiting for demand from a previous run is excluded by construction rather than by recency. That is the exact failure I raised, and this closes it at the source.
  • grep -F 'coder-llm-ticketed[' first — only the wrapper's own lines compete to be "newest", so a busier child can no longer push the lifecycle line out of a fixed-size tail. A hole I did not raise in my review, closed anyway.
  • Fails closed: no InvocationID, or an empty journal, prints 0 → final_rc → 69. The safe direction.

Verified against the live unit, both worlds

$ systemctl --user show -p InvocationID --value coder-llm
5313e657fae94de99be0764abf82bff7
$ journalctl --user -q _SYSTEMD_INVOCATION_ID=5313… | wc -l
255                                   # the field is populated and queryable — the mechanism works here
$ … | grep -cF 'coder-llm-ticketed['
0

That zero is correct and worth stating explicitly in the PR: today the unit runs plain 1bit serve (unticketed), so a silent port must still be 69 — there is no asleep state to be in, and the function returns 0 accordingly. It returns 1 only once #2496's wrapper is installed and the newest wrapper line for that invocation says waiting for demand. So the logic is right in both worlds, not just the ticketed one, and it will not silently start forgiving a wedge when #2496 lands.

The arms now cover the incident I described

chk "a crash-on-load after a stale 'waiting for demand' is NOT asleep" "$("$0" --asleep "$tmp/j-crash.log")" 0
chk "the newest wrapper line saying 'waiting for demand' IS asleep"    "$("$0" --asleep "$tmp/j-wait.log")" 1

The waiting for demand line from the previous invocation in the crash log is the exact shape of tonight's wedge (child hung at init, port silent, journal's newest line a successful request from an earlier pid). An arm written from the incident rather than from the code is the right kind.

Nothing to ask. One line worth keeping: the --asleep FILE seam is what makes this testable with no device, and #2492's final_rc arms already cover the code/rc half — so the two together test the whole decision offline.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6472fed: READY

My design note was answered with something better than I suggested, and their arm catches a bug I had not thought of.

What they did instead of my idea

I proposed replacing the journal grep with install-coder-llm-ticketed.sh --status. They kept the journal and extracted --asleep <journal-file>, which reads the last wrapper state line rather than any matching one — and their comment names the hazard my suggestion would not have fixed:

"read the LAST wrapper state line, not any matching one. A crash on load right after this window logs demand: queueing … → window open: server pid … → device lost → server exited by itself, rc 1, all inside the last 60 lines."

So a naive grep -q 'waiting for demand' over the tail would find the stale line from before the window opened and call a crashed window asleep on purpose — turning my "false 69" into a false 0, which is the worse direction: a silent pass on a real failure. --status would have answered a different question (is the wrapper installed) and left this one open.

Measured, both shapes:

waiting-tail  -> 1      (asleep)
crash-tail    -> 0      (a crash-on-load after a stale 'waiting' line is NOT asleep)

and the arm built from a realistic log ("E hrx: device lost", "server exited by itself, rc 1", "window ended rc 1") asserts the pair: not asleep, and therefore restart verdict 69.

Mutation, and it fails three arms by name — replacing the last-line reader with grep -q 'waiting for demand':

FAIL a crash-on-load after a stale 'waiting for demand' is NOT asleep (got , want 0)
FAIL the newest wrapper line saying 'waiting for demand' IS asleep (got , want 1)
FAIL an empty journal is not asleep (the safe direction) (got , want 0)

The third is the one I would have missed when asking for a mutation: an empty journal must not read as asleep, because "no evidence" is not "waiting on purpose" — the same rule the rest of this fleet applies to cannot tell.

--self-test → ok with all of it.

Nothing left open from me. The truth table from the previous head is unchanged and still right (inactive is never healthy; the job's rc still wins), and the probe now reads a sequence rather than a mention, which is the distinction the whole PR turns on.

@bong-water-water-bong
bong-water-water-bong merged commit 977f019 into main Nov 15, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/window-asleep branch November 15, 2026 07:34
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it is the correct half of a defect I created pressure on in my #2496 review.

I asked on #2496 what a lane should expect when coder-llm is down by design: "if it fails, every PR check goes red the moment this lands, and that will look like a regression." This fixes the caller side of exactly that, and the framing is right: "I flagged this on #2496; it belongs here, in the caller, because the caller is what turns a silent port into a false failure." Separating "the fleet's model is asleep on purpose" from "we stopped it and it did not come back" is the whole issue.

The decision is a pure function of four inputs, and the truth table is the design:

200                                -> the job's rc        (the model is back)
the job itself failed              -> the job's rc        (a restart failure never masks it)
active AND "waiting for demand"    -> the job's rc        (asleep ON PURPOSE)
anything else                      -> 69                  (we stopped it and it did not come back)

The second row is the one I would have got wrong. "A restart failure never masks the job's failure" — without that, a job that legitimately failed (rc 3) and whose restart then also hiccuped would report 69, sending the reader after the fleet instead of after their own job. And the seventh arm pins it: --restart-rc 000 3 1 1 -> 3.

Two more things that hold up:

  • inactive is never healthy — --restart-rc 000 0 0 1 -> 69. A unit that is not running cannot be "asleep on purpose", so requiring active and the journal message is the right conjunction; either alone would be forgeable by a stopped unit with stale journal text.
  • The end line keeps its Z end shape for the gap parser, and names the case (asleep-no-demand / restart…). That is the same discipline as window-run: a window that leaves coder-llm down fails with 69, not a WARNING line (#2483 follow-up) #2492: preserve the format the reader parses, add the new information as a field — not as a new line shape.

One ask: "journal says waiting for demand" — how is that string matched? If it is a substring on journalctl output for the unit, a past occurrence in the retained window would satisfy it while the unit is currently doing something else. Matching on the last occurrence since the window's start (rather than "appears anywhere") would close it; if that is already what the code does, one clause in the body makes the arm's fourth input auditable rather than assumed.

Not verified by me: the seven arms. The truth table, the ordering of the rows and the format preservation are what I can check, and all three are right — this is a model of how to decompose a caller's decision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant