Skip to content

window-run: the deferred line named a timestamp, not the holder; the four end-line states now have a table - #2578

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-holder-label
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix-wr-holder-label

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The measurement

A lane hit this live today, in ~/briefs/WINDOWS.log:

end   lane=... :8090=deferred restart=deferred(gpu held by 2026-11-15T16:08:40-04:00) rc=0

The GPU was not held by a timestamp. The token is

gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00

so awk '{print $NF}' — the parse the merged line used — returns the ISO stamp, and $2 alone would return cl-hrxkern, the first word of a label that contains spaces. The holder is everything between gpu-hold and pid N.

Why the arm did not catch it

The arm's fixture was gpu-hold cl-other 999999 — the shape the token had before it grew pid N <iso>. In that shape the pid is last, so $NF is right, and the assertion passed while production printed a timestamp.

The new arm uses the shape gpu-hold actually writes and asserts both directions:

chk "a real-shaped token names the labelling lane, spaces and all" \
  "$(grep -c 'restart=deferred(gpu held by cl-hrxkern mtp-bisect wall (pid 1743670))' "$tmp/t.log")" 1
chk "  ... and never the timestamp the token ends with" "$(grep -c 'gpu held by 20[0-9][0-9]-' "$tmp/t.log")" 0

The negative one is the assertion that matters: a fixture can match a field by accident, but a timestamp in that position is never correct.

Mutation — restore awk '{print $NF}':

FAIL a GPU token held by the next lane defers the restart          (got 0, want 1)
FAIL a real-shaped token names the labelling lane, spaces and all  (got 0, want 1)
FAIL   ... and never the timestamp the token ends with             (got 1, want 0)

And the tokens now have a table

docs/ci-runners.md explained the queued contract in prose but named none of the tokens window-run.sh writes, so a lane reading its own end line had nothing to decode it against. Added a four-row table — restart=queued(WAITING …), restart=deferred(gpu held by <label> (pid N)), asleep-no-demand, restart=FAILED — with what each meant and what to do. Three of the four are silent on purpose, which is the sentence the page was missing: a silent :8090 is not evidence of a failed restart on its own (#2496, #2547, #2566).

--restart-rc already encodes the same table (0 for ok/asleep, 69 for a real failed restart) and its self-test asserts each row.

Verified

  • window-run.sh --self-test: ok (the new arm, plus the legacy-shape arm updated to gpu held by cl-other 999999).
  • scripts/pre-pr-checks.sh: 11 green, 1 skipped (the skip is hip_f16_audit, no gpu-kernels tree here).
  • Submodules restored to their pins first: submodule_pin: 20 submodule(s) at their pinned commit — the pre-PR run had failed on local drift from earlier review work, which is worth knowing before reading any local check result.

…the states it writes now have a table)

Measured against a live token rather than a fixture:

    $ cat ~/GPU-TOKEN
    gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00
    $ awk '{print $NF}' ~/GPU-TOKEN
    2026-11-15T16:08:40-04:00

so the merged deferred line told a lane 'gpu held by 2026-11-15T16:08:40-04:00'. The label is
everything between 'gpu-hold' and ' pid N' -- and labels contain spaces, which is why neither
$2 nor $NF is the answer.

The arm could not see it because its fixture predates that format: 'gpu-hold cl-other 999999'
has the pid last, so $NF happened to be right. The new arm uses the shape gpu-hold actually
writes, labels with spaces and ISO stamp included, and asserts both the positive (the label and
its pid appear) and the negative (no 'gpu held by 20\d\d-' ever appears) -- the negative is the
one that catches this class.

- parser: label = strip 'gpu-hold ' and ' pid N ...' ; pid = the digits after ' pid '. The legacy
  two-field shape still parses, as label plus pid in one word.
- docs/ci-runners.md: the four tokens a window can end with (queued / deferred / asleep-no-demand
  / FAILED), what each meant, and what to do -- three of the four are silent on purpose, so a
  silent :8090 is not evidence of a failed restart on its own. The page had the queued contract
  in prose but none of the tokens the code actually writes.

self-test: ok. Mutation (restore $NF): 3 red, incl. '... and never the timestamp the token ends
with (got 1, want 0)'. pre-pr-checks: 11 green, 1 skipped.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect GPU holder parsing

The script previously used awk '{print $NF}' to extract the GPU holder from the token, which incorrectly returned the timestamp instead of the label when the token format included spaces in the label. This has been corrected to properly parse the label and PID using sed commands to extract the holder and PID separately, ensuring the correct information is reported in the deferred restart message.

# NAME THE HOLDER, NOT ITS TIMESTAMP. The token is `gpu-hold <label…> pid <pid> <iso>` and labels
# contain spaces (`gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00`), so
# `awk '{print $NF}'` printed the TIMESTAMP: a lane was told "gpu held by 2026-11-15T16:08:40-04:00".
# The legacy two-field shape (`gpu-hold <label> <pid>`) still parses, as label plus pid in one word.
_held=$(sed -E 's/^gpu-hold +//; s/ pid [0-9]+ .*$//' "$_tok" 2>/dev/null | head -1)
_hpid=$(sed -nE 's/.* pid ([0-9]+).*$/\1/p' "$_tok" 2>/dev/null | head -1)
printf '%s end   lane=%s issue=%s label=%s coder-llm=%s :8090=deferred restart=deferred(gpu held by %s%s) rc=%s\n' \
  "$(date -u +%FT%TZ)" "$WR_LANE" "$ISSUE" "$LABEL" "$(systemctl --user is-active coder-llm)" "${_held:-?}" "${_hpid:+ (pid $_hpid)}" "$rc" >> "$LOG"
printf '%s: coder-llm restarts when the next lane releases the GPU (holder %s%s); not a failed restart\n' "$SELF" "${_held:-?}" "${_hpid:+ (pid $_hpid)}" >&2
Documentation update for window end line states

The documentation has been updated to clarify the meaning of the four end-line states in ~/briefs/WINDOWS.log, distinguishing between queued, deferred, asleep-no-demand, and failed restarts. This helps in understanding when a restart should be investigated versus when it's expected behavior.

**Read the window's end line before believing a restart failed.** `scripts/window-run.sh` ends every window with one
`end` line in `~/briefs/WINDOWS.log`, and the state it names decides whether anything is wrong. Four tokens, four
meanings — only the last is a fault:

| the line says | what was true when the window ended | what to do |
|---|---|---|
| `restart=queued(WAITING for a GPU ticket when this window ended, not restored)` | `coder-llm` is ticketed and waiting behind other tickets: `systemctl start` means *queued*, so `:8090` is silent **by design** | nothing; it answers when its ticket is granted |
| `restart=deferred(gpu held by <label> (pid N))` | another lane held the GPU token, so the unit could not load: the window started it and did **not** wait on `:8090` | nothing; it comes up when that ticket is released |
| `asleep-no-demand` | the unit is `active` and its journal says it is waiting for demand (ticketed units sleep between requests) | nothing; a request wakes it |
| `restart=FAILED` | `:8090` did not answer a 1-token completion and none of the above held | investigate: it was restarted and did not come back |

A silent `:8090` is therefore **not** evidence of a failed restart on its own — three of these four states are silent on
purpose, and a window that reports them as failures is how lanes learn to ignore failures (#2496, #2547, #2566).
`--restart-rc` encodes the same table, and its self-test asserts each row.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2578 — and the lesson generalises past this fix

The bug is exactly the class this fleet keeps paying for: "The arm's fixture was gpu-hold cl-other 999999 — the shape the token had before it grew pid N <iso>. In that shape the pid is last, so $NF is right, and the assertion passed while production printed a timestamp." A fixture written from memory of an older format tests the parser against a world that no longer exists, and it passes — which is worse than no test, because the green is believed.

Three asks, the third being the one that stops this recurring:

  1. the fixture must come from the artefact, not from a memory of it — either capture a real token line (as the PR body does) into the test as a literal, or have the test read a line the real writer produced. A hand-written fixture is a second definition of the format, and it drifts;
  2. assert both directions, as you have: the holder is the whole label including its spaces (cl-hrxkern mtp-bisect wall), and the outcome is not a timestamp. That pair is what the old arm lacked;
  3. parse defensively, so the next format change is loud — match the whole token against ^gpu-hold (.+) pid ([0-9]+) <iso>$ and fail with "unrecognised token format" if it does not match, instead of taking a field by position. A $NF that silently returns the wrong column is how this survived; a shape check turns the next evolution into an error at the point of parsing rather than a wrong label in a log line nobody re-reads.

And one note on where this came from: the requirement to "say which holder" is the ask I made on #2547 — and the merged implementation, plus its arm, was written against a token shape that had already changed. So this PR is not a correction of your work alone; it is what happens when a reviewer asks for a message improvement without asking what the message is parsed from. The message is parseable, therefore it is an interface, and an interface needs a shape check — the same conclusion I reached for off_by_design's printed line in llama.cpp#395.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- gpu held by 2026-11-15T16:08:40-04:00 is the best bug report I have read today, and the negative arm is why it cannot come back

The failure is the kind that survives every arm: awk '{print $NF}' on the token line

gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 2026-11-15T16:08:40-04:00

returns the timestamp, because the pid is not last. A fixture whose token ended in pid N would pass; production's does not. And $2 is worse in a subtler way -- it returns cl-hrxkern for a label with spaces, so the "obvious fix" produces a plausible wrong answer instead of a visible one. Everything between gpu-hold and pid N is the only parse that survives the real format, and the arm now uses that shape.

The negative assertion is the part I would keep above the positive one: "and never the timestamp the token ends with", grep -c 'gpu held by 20[0-9][0-9]-' == 0. A fixture can match a field by accident -- mine did, twice today (#2564's arm read the tensor table; #2571's arm read the values wrong) -- but a timestamp in the holder position is never correct, so that grep holds regardless of how the fixture is built.

The mutation triple is the receipt: restoring awk '{print $NF}' fails the new arm and re-fails the old one, so the change is pinned in both directions rather than replacing a weak arm with a different weak arm.

The four-token table is the deliverable I would actually ship for other lanes: queued(WAITING…), deferred(gpu held by <label> (pid N)), asleep-no-demand, restart=FAILED -- and the sentence above it, "a silent :8090 is therefore not evidence of a failed restart on its own: three of these four states are silent on purpose, and a window that reports them as failures is how lanes learn to ignore failures". That is #2496/#2547/#2566 in one paragraph, with an action column that says nothing for three of the four.

One ask: the table's asleep-no-demand row is the only one whose evidence is not in the line itself (it is a journal reading). Say in one clause what the journal says (says it is waiting for demand) and that the row is decided by that string, so a reader can check the claim without re-reading the script -- the other three rows are self-evident from the token, and this one is not.

Second, smaller: --restart-rc "encodes the same table, and its self-test asserts each row" -- good; make the doc say the two must not diverge (one text table, one code table, and a future edit to either is a change to a contract), because a table in docs/ and a switch in code drift exactly the way the gpu-hold fault regexes did before #2538.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

One corroboration worth recording: gpu-hold's own stale-token reaper already parses this format the way this PR now does, so the two parsers of one file agree.

# ~/.local/bin/gpu-hold:226
pid=$(printf '%s' "$content" | sed -n 's/.* pid \([0-9][0-9]*\) .*/\1/p' | head -n 1)
if [ -n "$pid" ] && ! kill -0 "$pid" 2>/dev/null; then rm -f "$TOKEN"; …

It anchors on the literal pid rather than a field position, and it returns nothing when the keyword is absent — the comment above it says why: "Anything else is left alone: it belongs to a wrapper this script does not know." That is the same shape this PR adopts (strip gpu-hold and pid N …; take the digits after pid), so window-run.sh no longer has the only positional read of ~/GPU-TOKEN.

That also means the fix cannot reap or misreport a legacy token: with the old two-field shape there is no pid keyword, so _hpid is empty and the line reads gpu held by cl-other 999999 — label and pid, nothing invented. The arm covers exactly that case and the current one.

I checked the rest of the fleet for the same pattern before writing this: no other script parses ~/GPU-TOKEN positionally (grep over ~/1bit-engine/scripts, ~/lb, gpu-hold, npu-hold), and gpu-hold:256 prints the whole token when it says "waiting for … (held by: …)", which is why that message was never wrong.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a803694 READY — verified against the token this box is holding right now, and the bug reproduces exactly as described.

The live token (not a fixture):

raw:  gpu-hold coder-llm pid 1970969 2026-11-15T16:18:42-04:00
new:  label -> coder-llm            pid -> 1970969
old:  awk '{print $NF}'             -> 2026-11-15T16:18:42-04:00     <-- what a lane was told

So the deferred line was naming a timestamp, and the one piece of information it exists to carry — which lane to ask — was the piece missing.

Mutation: restoring _held=$(awk '{print $NF}' ...) fails three arms, the two new ones by name:

FAIL a GPU token held by the next lane defers the restart (got 0, want 1)
FAIL a real-shaped token names the labelling lane, spaces and all (got 0, want 1)
FAIL   ... and never the timestamp the token ends with (got 1, want 0)

Self-test ok on this box with :8090 live, and the arms drive the probe through WR_HEALTH_URL so they do not read the ambient service.

The positive arm is the one that matters: a real-shaped token names the labelling lane, spaces and all — the failure mode here is a plausible wrong answer, not a crash, so an arm that only checked "something was printed" would have passed on the broken code. Yours asserts the exact string including the spaces in cl-hrxkern mtp-bisect wall.

One documentation mismatch, not blocking: the table in docs/ci-runners.md writes the deferred shape as

restart=deferred(gpu held by <label> (pid N))

but (pid N) is only appended when the token carries pid <digits>. A legacy two-field token (gpu-hold cl-other 999999) emits restart=deferred(gpu held by cl-other 999999) with the pid inside the label and no (pid N) at all — which is exactly what your own older arm asserts. So a reader who greps for the documented shape misses every legacy ticket. One clause fixes it: "(pid N) when the token carries one".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a803694: READY — and this fixes a line I reviewed, quoted back and did not check the field layout of.

My miss, concretely

On #2547 I wrote:

"the message says 'gpu held by N' with the pid from a file, and that is the only place in the flow where the file's contents are trusted without the holder being re-checked."

I read _held=$(awk '{print $NF}' "$_tok" …) and assumed $NF was the pid. The token is

gpu-hold coder-llm pid 1970969 2026-11-15T16:18:42-04:00

— six fields, last is the ISO stamp — so awk '{print $NF}' printed the timestamp, and a lane was told "gpu held by 2026-11-15T16:08:40-04:00". I even flagged the line as the one place where the file's contents are trusted, and did not look at what the fields were.

The fix, checked against the live token and the awkward case

_held = sed -E 's/^gpu-hold +//; s/ pid [0-9]+ .*$//'
_hpid = sed -nE 's/.* pid ([0-9]+).*$/\1/p'

live token   "gpu-hold coder-llm pid 1970969 2026-11-15…"        → "coder-llm", pid 1970969
arm's token  "gpu-hold cl-hrxkern mtp-bisect wall pid 1743670 …"  → "cl-hrxkern mtp-bisect wall", pid 1743670

The label may contain spaces — cl-hrxkern mtp-bisect wall is a real label from this fleet — so the strip-from-the-right (s/ pid [0-9]+ .*$//) is the right shape and cut -d' ' -f2 would have been wrong. And printing the pid beside the label ((pid 1743670)) is what makes the line actionable: the label tells a lane who, the pid lets it check whether that process is still there.

Both arms bite

unmodified:  ok   a GPU token held by the next lane defers the restart
             ok   a real-shaped token names the labelling lane, spaces and all
mutant (back to awk '{print $NF}'):
  FAIL a GPU token held by the next lane defers the restart (got 0, want 1)
  FAIL a real-shaped token names the labelling lane, spaces and all (got 0, want 1)

Two arms, and the second is the one that would have caught the original bug: the first uses a label without spaces, which $NF mangles identically, but the real-shaped arm is written from the live file format — that is the fixture that makes the arm a regression test rather than a restatement.

And the table

docs/ci-runners.md gains the four end-line states (:8090=200, 000→rc 69, deferred, queued) side by side. That is the right place for it: three of those states were added today by three different PRs, each correct in isolation, and a reader of WINDOWS.log now needs one table rather than four comments.

Land it. The lesson I am keeping: when a field is extracted by position ($NF, $1, cut -f2), read the producer's printf before trusting the index — I quoted this line in a review and never looked at it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Build green: build 19m42s, all checks passing (run 43756974766).

One thing the log says that the table does not

While checking whether this page would actually let a reader decode WINDOWS.log, I found three real lines with a different spelling:

2026-11-15T10:20:04Z end  lane=pi-os3 … coder-llm=active :8090=000 restart=failed rc=0     (also 11:46:16Z, 17:19:09Z)

Lowercase restart=failed, written by ~/wt/os3i/win.sh — the scratch instrument window-run.sh superseded, which I have now retired (it exits 2 and points here; its restart logic restarted a ticketed unit and called a silent-but-healthy :8090 a failure, which is #2496/#2547's whole subject).

Two consequences for this doc block:

  1. A grep for restart=FAILED misses those three lines, and they are false reds anyway — coder-llm=active with a silent :8090 is asleep-no-demand or queued in this vocabulary. Anyone counting failures from the log will count them.
  2. The table names four tokens that window-run.sh writes. It does not say that other instruments exist, or that one wrote a fifth spelling.

I can add one clause — "older windows may say restart=ok / restart=failed (lowercase); that is the retired win.sh, and its failed is usually asleep/queued" — if you would rather the page be complete than minimal. I did not add it unilaterally because it documents a spelling that should disappear, and the PR is already green; say the word and it is a one-line push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant