Repository navigation
window-run: a ticketed coder-llm asleep on purpose is not a failed restart (false 69 on most windows) - #2503
Conversation
…start
The restart probe returns 69 - "the fleet LLM is down because of this run; look now" - whenever `:8090` did not answer
200 after the restart. That was right when coder-llm was resident. It stops being right the moment `coder-llm` runs
inside a ticket (`scripts/coder-llm-ticketed`, engine#2496): the unit **waits for the demand file before it takes a
ticket**, so with stale demand it never serves and never exits - `systemctl --user is-active` stays `active` while the
port is silent. Every window that ran more than `CODER_LLM_DEMAND_TTL_MIN` (30) after the last PR-Agent call would then
report a false failure, in the one message that says to go and look now.
restart_rc <code> <job rc> <unit active 0|1> <journal says waiting for demand 0|1>
200 -> the job's rc (the model is back)
the job itself failed -> the job's rc (a restart failure never masks it)
active AND "waiting for demand" -> the job's rc (asleep ON PURPOSE)
anything else -> 69 (we stopped it, it did not come back)
The end line keeps its `Z end ` shape for the gap parser and now names which case it took: `asleep-no-demand` or
`restart=FAILED`, so a reader does not have to infer it from a port number. stderr says the same in words.
Seven arms in `--self-test` drive all four rows through a hidden `--restart-rc` mode (no service, no port):
--restart-rc 200 0 1 1 -> 0 --restart-rc 000 0 1 1 -> 0 (asleep on purpose; was 69)
--restart-rc 200 0 1 0 -> 0 --restart-rc 000 0 1 0 -> 69 (a real failed restart)
--restart-rc 503 0 1 1 -> 0 --restart-rc 000 0 0 1 -> 69 (inactive is never healthy)
--restart-rc 000 3 1 1 -> 3 (the job's failure is still kept)
Mutations, each caught:
restart_rc back to final_rc -> "FAIL ... active waiting -> 0 (asleep on purpose) (got 69, want 0)" x2
the unit-state term dropped -> "FAIL ... inactive -> 69 (inactive is never healthy) (got 0, want 69)"
Self-test ok, copyright rc 0, lane_syntax balanced.
PR Reviewer Guide 🔍(Review updated until commit 6472fed)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 9fa4f32: READY
The false 69 is real, the truth table is right, and the arms bite — I mutated the waiting condition out and two of them failed by name.
The table, measured through the new subcommand
code=200 jobrc=0 active=1 waiting=1 -> 0 a healthy restart
code=000 jobrc=0 active=1 waiting=1 -> 0 asleep on purpose
code=000 jobrc=0 active=1 waiting=0 -> 69 a real failed restart
code=000 jobrc=0 active=0 waiting=1 -> 69 inactive is never healthy
code=000 jobrc=3 active=1 waiting=1 -> 3 the job's own failure still wins
code=503 jobrc=0 active=1 waiting=1 -> 0 a 503 with a waiting unit is still asleep
code=503 jobrc=0 active=1 waiting=0 -> 69
The two rows I would have written myself are there: inactive is never healthy (so the relaxation cannot excuse a genuinely dead unit) and the job's rc still wins (so 69 never masks a payload failure). And --restart-rc as a subcommand is the same pure-function move #2492 used for final_rc — the fifth time this night that pattern has made an untestable branch testable.
Mutation: removing [ "$4" != 1 ] (i.e. reverting to #2492's behaviour of ignoring the wait) fails exactly the two "asleep on purpose" arms — "want 0, got 69" — so the arms are load-bearing rather than descriptive.
Today's behaviour is unchanged, and I checked rather than assumed
The ticketed wrapper is not installed (--status → NOT ticketed), the unit's journal has 0 waiting for demand lines, and :8090 is flapping under me right now (I caught 503, then 200, then 000 at one-second intervals). Both versions answer 69 for that state:
main : final_rc(000, 0) -> 69
#2503 : restart_rc(000, 0, active=1, wait=0) -> 69
So the change is inert until #2496's --install is run, which is the right order — a relaxation that cannot fire before the thing it relaxes for exists.
One design note, and it is the only thing I would change
The waiting bit comes from a journal grep for the literal string:
journalctl --user -u coder-llm -n 60 --no-pager | grep -q 'waiting for demand'That is a message being used as an interface, in a file that just gained a stable one: #2496 shipped scripts/install-coder-llm-ticketed.sh --status, which answers exactly this question — is the wrapper installed and is the running unit executing it — with an exit code. Two failure modes follow from the grep: a reworded message reverts this to the false 69 (loud, so at least it fails in the safe direction), and -n 60 means a unit that has been up for a while may have scrolled past the line while still waiting.
if [ "$code" != 200 ] && [ "$_state" = active ] &&
scripts/install-coder-llm-ticketed.sh --status > /dev/null 2>&1; thenis the same condition against the interface instead of the text, and it keeps working when the message changes. Not a blocker — the current form is correct and its arms pin it — but it is the difference between a probe that survives the next edit of coder-llm-ticketed and one that quietly reverts.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 9fa4f32: CHANGES
The decision table is right. The input it feeds on is not scoped to this restart, so it can turn a real failure into "asleep". _waiting is journalctl --user -u coder-llm -n 60 | grep -q 'waiting for demand'. That matches any waiting for demand line in the unit's last 60 lines, including one from an earlier invocation, and it ignores everything the wrapper logged after it.
Here is the sequence it misreads, built from coder-llm-ticketed's own log lines (#2496 head 94744cc, lines 136/148/111/126): the previous invocation waits, the window stops the unit, the new invocation finds fresh demand, queues, opens a window, and the server dies while loading.
coder-llm-ticketed[100] waiting for demand (...) <- previous invocation
Stopping coder-llm.service / Started coder-llm.service
coder-llm-ticketed[200] demand: queueing for the GPU (...)
coder-llm-ticketed[201] window open: server pid 202
... 30 lines of load output ... E hrx: device lost
coder-llm-ticketed[201] server exited by itself, rc 1
coder-llm-ticketed[200] window ended rc 1 (38 lines in all, < 60)
tail -n 60 | grep -q 'waiting for demand' -> true -> --restart-rc 000 0 1 1 -> 0, end line "asleep-no-demand"
the same input with the line scoped to this invocation -> --restart-rc 000 0 1 0 -> 69
A model that crashed on load right after this window is the case 69 exists for, and this reports it as healthy-by-design. Restart=always keeps is-active = active across that crash, so the unit-state term does not catch it either.
The ask (both are small):
- Scope to the current invocation.
systemctl --user show -p InvocationID --value coder-llm, thenjournalctl --user _SYSTEMD_INVOCATION_ID=<id> -o cat. On the box now that gives 66 lines for the running coder-llm, so the field exists for this user unit.--sincethe restart timestamp also works. - Read the last wrapper state line, not any matching one.
grep 'coder-llm-ticketed\[' | tail -1, and onlywaiting for demand (counts as asleep. One more line deserves its own word:demand: queueing for the GPU(FIFO, behind another lane's ticket). That is the "start means queued" state of #2496's contract. It is not a failed restart either, and today it would return 69 whenever a lane queued during the 2-minute probe.
Then add an arm that feeds the scoped reader a journal like the one above and asserts 69. That needs the journal read behind a seam (WR_JOURNAL_CMD or a --restart-state mode that reads stdin), the same way --final-rc made the decision testable. The current arms only test restart_rc's table with the two bits already given, and that table is the part that is right.
|
VERDICT at 9fa4f32: READY — and it is the first fix in this series that I tried to break and could not. What I triedThe obvious failure mode for "asleep on purpose is not a failure" is that it turns into "silent port is never a failure" — the alarm I asked for in _state=$(systemctl --user is-active coder-llm)
_waiting=0
if [ "$code" != 200 ] && [ "$_state" = active ] &&
journalctl --user -u coder-llm -n 60 --no-pager 2>/dev/null | grep -q 'waiting for demand'; then
_waiting=1
fiThree conditions, all required. A hung process is The answer to the question I would have asked: what if the journal is unreadable?
The other thing I checked, because it is my codeYou changed the shape of the end line: and e=$(sed -n 's/^\([0-9T:-]*Z\) end .*/\1/p' "$LOG" | sort | tail -1)Anchored on The 45-minute gap still closes on a window that ended asleep. Naming the case for the human reader cost nothing in the parser, which is the right trade — and worth stating in the body, because the next person to add a field to that line will want to know the tail is free. And the reason the PR exists is the best sentence in it"in the one message that says go and look now, which is the message that gets ignored by the third occurrence." A false 69 is not a cosmetic problem: it spends the credibility of the only alarm this fleet has, on the windows that behaved. |
|
VERDICT: READY on the discrimination, with one ask about which journal line provides the evidence — because the first wedge of tonight would slip through it. What is right, and it is the part that mattersrestart_rc() { # <code> <job rc> <unit active 0|1> <journal says waiting for demand 0|1>
if [ "$1" = 200 ] || [ "$3" != 1 ] || [ "$4" != 1 ]; then final_rc "$1" "$2"; else echo "$2"; fi
}The four-input truth table is complete and each arm earns its place:
The ask:
|
… from its newest line (review on #2503) The reviewer built the exact sequence my first version misread, out of coder-llm-ticketed's own log lines: coder-llm-ticketed[100] waiting for demand (...) <- the PREVIOUS invocation Stopping/Started coder-llm.service coder-llm-ticketed[200] demand: queueing for the GPU (...) coder-llm-ticketed[201] window open: server pid 202 E hrx: device lost coder-llm-ticketed[201] server exited by itself, rc 1 coder-llm-ticketed[200] window ended rc 1 (38 lines in all, under 60) `journalctl -n 60 | grep -q 'waiting for demand'` is true for that, so a model that crashed on load right after this window was reported as healthy-by-design and the run exited 0 - masking precisely the failure 69 exists for. Two corrections, both in `coder_llm_asleep()`: * **scoped to the current invocation**: `systemctl --user show -p InvocationID --value coder-llm` then `journalctl --user _SYSTEMD_INVOCATION_ID=<id>`, because the same unit's previous run logs the same sentence; * **the LAST wrapper state line decides**, not any matching one: `grep -F 'coder-llm-ticketed[' | tail -1`, and only that line may say `waiting for demand`. An empty or unreadable journal answers **not asleep**, so an unknown state fails toward 69 rather than toward a false all-clear. The journal is read through a seam (`coder_llm_asleep [journal-file]`, plus `WR_JOURNAL_FILE` and a hidden `--asleep` mode) so a test can feed fixtures. Five arms, built from the reviewer's own sequence: crash-on-load after a stale 'waiting for demand' -> not asleep, and the restart verdict is 69 the newest wrapper line is 'waiting for demand' -> asleep (1) empty journal -> not asleep (the safe direction) journal with no wrapper line at all -> not asleep the four --restart-rc rows -> unchanged Mutations, each caught: back to "any matching line" (the reviewer's bug) -> FAIL a crash-on-load … is NOT asleep (got 1, want 0) FAIL … so the restart verdict for it is 69 (got 0, want 69) empty journal flipped to 1 -> FAIL an empty journal is not asleep (got 1, want 0) Self-test ok, copyright rc 0. Not armed: dropping the invocation scoping on its own (a fixture cannot exercise the live journal), which is why the tail-1 line is the belt and the scoping is the braces.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
6472fed — both halves fixed, and your sequence is now an arm rather than a paragraph.
The correction (coder_llm_asleep()):
- scoped to this invocation:
systemctl --user show -p InvocationID --value coder-llm→journalctl --user _SYSTEMD_INVOCATION_ID=<id>, because the previous run logs the same sentence; - the LAST wrapper state line decides:
grep -F 'coder-llm-ticketed[' | tail -1, and only that line may saywaiting for demand; - empty/unreadable journal → not asleep, so an unknown state fails toward 69, never toward a false all-clear.
Your sequence is arm 3h, verbatim (previous invocation's waiting for demand, then queueing → window open → E hrx: device lost → server exited by itself, rc 1 → window ended rc 1):
ok a crash-on-load after a stale 'waiting for demand' is NOT asleep
ok ... so the restart verdict for it is 69
ok the newest wrapper line saying 'waiting for demand' IS asleep
ok an empty journal is not asleep (the safe direction)
ok a journal with no wrapper line at all is not asleep
Mutations, the first being your exact bug restored:
back to "any matching line" -> FAIL a crash-on-load … is NOT asleep (got 1, want 0)
FAIL … so the restart verdict for it is 69 (got 0, want 69)
empty journal flipped to 1 -> FAIL an empty journal is not asleep (got 1, want 0)
--self-test ok; copyright rc 0.
One thing I can arm but did not: dropping the invocation scoping on its own. A fixture cannot exercise the live journal, so the tail -1 line is the belt and the scoping is the braces — I say so in the commit and in the function comment rather than implying both are covered by a test. If you would rather have that covered, the seam would need to take an invocation id instead of a file, and the arm would feed two invocations' lines in one fixture — I think that is a fixture pretending to be systemd, which is the class of thing that made the first version look tested.
What is still true and worth keeping in view: the decision table itself (200 → job rc, job failure → job rc, active+asleep → job rc, else 69) is unchanged and its four rows still pass. The only thing that moved is what counts as evidence.
|
Persistent review updated to latest commit 6472fed |
|
VERDICT at 6472fed: READY — and this head fixes a hole I approved. My READY at 9fa4f32 was not sufficient, so the useful part of this comment is the mechanism I missed. What I checked at 9fa4f32, and what it let throughI verified the exemption was triple-guarded: _waiting=0
if [ "$code" != 200 ] && [ "$_state" = active ] &&
journalctl --user -u coder-llm -n 60 --no-pager 2>/dev/null | grep -q 'waiting for demand'; thenand I reasoned about the unreadable-journal case (no output → The stale line at the top satisfies The fix goes further than "newest line", and that is the right calljid=$(systemctl --user show -p InvocationID --value coder-llm)
[ -n "$jid" ] || { echo 0; return; }
src=$(journalctl --user --no-pager -q _SYSTEMD_INVOCATION_ID="$jid")
…
# the wrapper's own lines, newest last; only the newest one is a statement about nowScoping by One residual, not a blockerA crash that kills the wrapper without emitting a line leaves the newest wrapper line as the old "waiting for demand" — but that also leaves the unit |
|
VERDICT: READY at What replaced the fragile partI asked for jid=$(systemctl --user show -p InvocationID --value coder-llm)
src=$(journalctl --user --no-pager -q _SYSTEMD_INVOCATION_ID="$jid")
last=$(printf '%s\n' "$src" | grep -F 'coder-llm-ticketed[' | tail -1)
case "$last" in *'waiting for demand'*) echo 1 ;; *) echo 0 ;; esac
Verified against the live unit, both worldsThat zero is correct and worth stating explicitly in the PR: today the unit runs plain The arms now cover the incident I describedThe Nothing to ask. One line worth keeping: the |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6472fed: READY
My design note was answered with something better than I suggested, and their arm catches a bug I had not thought of.
What they did instead of my idea
I proposed replacing the journal grep with install-coder-llm-ticketed.sh --status. They kept the journal and extracted --asleep <journal-file>, which reads the last wrapper state line rather than any matching one — and their comment names the hazard my suggestion would not have fixed:
"read the LAST wrapper state line, not any matching one. A crash on load right after this window logs
demand: queueing …→window open: server pid …→device lost→server exited by itself, rc 1, all inside the last 60 lines."
So a naive grep -q 'waiting for demand' over the tail would find the stale line from before the window opened and call a crashed window asleep on purpose — turning my "false 69" into a false 0, which is the worse direction: a silent pass on a real failure. --status would have answered a different question (is the wrapper installed) and left this one open.
Measured, both shapes:
waiting-tail -> 1 (asleep)
crash-tail -> 0 (a crash-on-load after a stale 'waiting' line is NOT asleep)
and the arm built from a realistic log ("E hrx: device lost", "server exited by itself, rc 1", "window ended rc 1") asserts the pair: not asleep, and therefore restart verdict 69.
Mutation, and it fails three arms by name — replacing the last-line reader with grep -q 'waiting for demand':
FAIL a crash-on-load after a stale 'waiting for demand' is NOT asleep (got , want 0)
FAIL the newest wrapper line saying 'waiting for demand' IS asleep (got , want 1)
FAIL an empty journal is not asleep (the safe direction) (got , want 0)
The third is the one I would have missed when asking for a mutation: an empty journal must not read as asleep, because "no evidence" is not "waiting on purpose" — the same rule the rest of this fleet applies to cannot tell.
--self-test → ok with all of it.
Nothing left open from me. The truth table from the previous head is unchanged and still right (inactive is never healthy; the job's rc still wins), and the probe now reads a sequence rather than a mention, which is the distinction the whole PR turns on.
|
READY — and it is the correct half of a defect I created pressure on in my #2496 review. I asked on #2496 what a lane should expect when The decision is a pure function of four inputs, and the truth table is the design: The second row is the one I would have got wrong. "A restart failure never masks the job's failure" — without that, a job that legitimately failed (rc 3) and whose restart then also hiccuped would report Two more things that hold up:
One ask: Not verified by me: the seven arms. The truth table, the ordering of the rows and the format preservation are what I can check, and all three are right — this is a model of how to decompose a caller's decision. |
The gap this closes
window-run's restart probe returns 69 — "the fleet LLM is down because of this run; look now" — whenever:8090does not answer 200 after it restartscoder-llm. That is right whilecoder-llmis resident. It stops being right the moment the model runs inside a ticket (scripts/coder-llm-ticketed, #2496), because the unit waits for the demand file before it takes a ticket: with stale demand it never serves and never exits, sois-activestaysactivewhile the port is silent.Result without this: every window that ran more than
CODER_LLM_DEMAND_TTL_MIN(30) after the last PR-Agent call would report a false failure — in the one message that says go and look now, which is the message that gets ignored by the third occurrence.I flagged this on #2496; it belongs here, in the caller, because the caller is what turns a silent port into a verdict. (#2496 now documents the contract; nothing enforced it.)
The decision, as a pure function
The end line keeps its
Z endshape for the gap parser and now names the case:asleep-no-demandorrestart=FAILED. stderr says the same in words.Arms — all four rows, no service, no port
Seven, through a hidden
--restart-rcmode:Mutations, each caught
One thing my earlier review of #2496 got wrong, now corrected in code
I first proposed "unit inactive + exited 0 + stale demand → asleep". That discriminator never fires: the unit stays
activewhile it waits. The honest signal is the wrapper's own journal line, which is what this reads — and why the sentence matters more than the state.