Skip to content

tools/ci: server_launch_check — a server under setsid loses its kill switch - #2493

Merged
bong-water-water-bong merged 1 commit into
mainfrom
verify/server-launch-check
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
verify/server-launch-check

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The check for my own worst mistake, split out because #2481 merged at 04:30:42Z and this commit (4cf5ee72) landed after it.

What it catches

setsid "$BIN" forward-serve -m "$MODEL" -p "$PORT" &
PID=$!
kill $PID

setsid puts the child in a new session, so $! is the setsid pid — it exits immediately and every kill "$pid" afterwards aims at a dead process. The server survives in that session with PPID 1, no ticket, still holding whatever device it claimed.

Why it exists

On 2026-11-15 an npu-model-chart.sh run did exactly this. flm-real serve lfm3:1.7b -p 8199 stayed alive for 22 h 44 min holding every NPU hardware context; cl-npuperf and cl-flm got -19 on every hwctx create for most of a day. That script had seven kill "$pid" sites — all correct-looking, all aimed at setsid. The identical bug was in tools/repro-hostgemm-lock.sh (fixed, engine#2472).

The fleet already has gpu_hold_tree_test.sh, which asserts "0 = the tree died with the ticket, 1 = something survived" — the right check, that could not see this, because it tests gpu-hold's tree and the orphan lived in ~/bin.

It found the real thing on its first run

$ server_launch_check.py --root . --include-bin
  WARN  ../../bin/npu-model-chart.sh.bak-1126:143 [setsid-launch] setsid nohup flm serve "$tag" -p "$PORT" >> "$log" 2>&1 &
  WARN  ../../bin/npu-model-chart.sh.bak-1126:144 [kill-after-setsid] pid=$!

My own stale backup of the chart, still carrying the exact line that caused the outage. Deleted. ~/bin is where the orphan lived and where no check in this repo had ever looked.

Design

  • scripts/ and tools/ fail; ~/bin warns. Those scripts are not this repo's to own — a PR failing because a lane keeps a bad helper at home is a check that gets disabled.
  • Exit 77 when it scanned nothing. A check that green-lights an empty file list is the defect this whole PR series is about.
  • Third arm: a comment naming setsid is not the defect — engine#2460's lesson (tool.name in text accepted a comment) applied before it happens here.
  • --include-bin is off by default so CI does not depend on a runner's home directory; the two arms that matter run either way.

3 arms, 0 failed; repo scan rc=0; copyright.py --check rc=0; ls-remote = rev-parse.

…switch

`setsid CMD &` puts CMD in a new session, so the shell's $! is the setsid pid and every `kill "$pid"`
afterwards targets a dead process. The server survives with PPID 1, no ticket, still holding its device.

On 2026-11-15 an `npu-model-chart.sh` run did exactly that: `flm-real serve lfm3:1.7b` stayed alive
22 h 44 min holding every NPU hardware context, and two lanes got -19 on every hwctx create. Seven
`kill "$pid"` sites, all correct-looking, all aimed at setsid. The same bug was in
tools/repro-hostgemm-lock.sh (fixed, engine#2472).

This check fails a repo script that launches a server under setsid (or kills $! after doing so), warns
for ~/bin — where the orphan lived, and where no check in this repo had ever looked — and refuses to pass
when it scanned nothing (77). Three arms, including one that proves a COMMENT naming setsid is not
counted as the defect.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect kill pattern detection

The regex pattern r"\bkill\b.*\$(PID|\!|pid)\b" may incorrectly match lines that contain kill followed by a variable like PID or pid, but not necessarily the $! variable. This could lead to false positives in detecting the kill-after-setsid defect.

if re.search(r"\bkill\b.*\$(PID|\!|pid)\b", stripped) or "$!" == stripped[-2:]:

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b9a04cc CHANGES — two small things, both about why, and one of them I measured because the docstring states it unconditionally.

(Comment, not a formal review: GitHub refuses --request-changes on this shared account.)

What is right, and verified

  • --self-test → 3 arms, 0 failed (a repo script launching serve under setsid is reported; the same script with setsid dropped is clean; a comment naming setsid is not the defect — that last one is the arm that stops the check from becoming a grep for a word).
  • --check on this tree → rc 0, and no false positive where it would be tempting: ~/bin/npu-model-chart.sh (the script the 22 h 44 m orphan came from) now has 0 setsid and kills by pattern + pid, so the checker stays silent on it. Good sign — it is not a word-grep.
  • os/mini/release-check.sh:1402's shape is the one this check exists for, and it is worth knowing a real instance exists in the tree (see below).

1. The mechanism is conditional, and the docstring says it is not

The docstring states: "setsid CMD & puts CMD in a new session, so the shell's $! is the setsid pid, which exits immediately". That is true only when the caller's background job is a process-group leader — setsid then has to fork. Otherwise setsid execs in place and $! is the server. Measured here:

job control OFF (a plain script):   $! is ALIVE  (setsid exec'd in place)
job control ON  (set -m in the launcher):  $! is DEAD   (setsid FORKED - the child is orphaned in a new session)

The policy the check enforces is right either way — do not hang a kill switch on $! after setsid — but as written a careful reader can look at a working kill -- -$! path, run the self-test, see it pass, and conclude the docstring is wrong about their code. Suggested wording: "setsid forks when the background job is already a group leader (job control on: set -m, an interactive shell, a wrapper that backgrounds a leader). Then $! is a dead pid and the server survives with PPID 1. When it does not fork, the pattern works — which is exactly why it survives review."

2. The scan roots leave the release gate out

scan() walks scripts/ and tools/ only. os/mini/release-check.sh:1402 is the same shape, in the script that gates a release:

    setsid "$ONEBIT_BIN" serve -m "$HRXP_MODEL_PATH" --device hrx ... &
    _pgid=$!
    ...
    kill -TERM -- -"$_pgid" 2>/dev/null; sleep 2; kill -KILL -- -"$_pgid" 2>/dev/null

Add os/mini to the roots and this row goes red on main — which is the truthful answer, since that kill path has the same conditional. If you would rather not take that on in this PR, then name the exclusion in the docstring ("only scripts/ and tools/ are scanned; os/mini and tests/ are not yet") so the next orphan there is not invisible. tests/hw_gate_mem_test.sh also reproduces the idiom in a stub (fixture, not a server, so it is arguably out of the rule's scope — but it is where the pattern gets copied from).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2493

This is the orphaned-server bug in its most ordinary form, and the description states the mechanism exactly: "setsid CMD & puts CMD in a new session, so the shell's $! is the setsid pid, which exits immediately. Every kill \"$pid\" afterwards targets a dead process, and the server survives." That is the 11-08 failure — a llama-server holding 51 GiB of GTT after its ticket was gone — reachable from a five-line shell script that looks correct.

And a static check is the right instrument, because the pattern is textual and the consequence is invisible: nothing at the launch site fails, the script's kill returns 0 against a pid that already exited, and the evidence appears only in pgrep minutes later. Catching it in review is the only cheap moment.

Three things that make the check avoidable-noise rather than noise:

  1. recognise the legitimate forms — setsid --wait CMD (the shell waits for the child), setsid -f with a pid file the script later reads, and systemd-run --scope all achieve the intent. A check that flags every setsid will be disabled by the first person who needs one of those;
  2. say what to write instead in the failure message — set -m/process-group kill, setsid + pgrep -s <sid>, or the kill_tree helper the window scripts already use. A check that only says "no" gets a suppression;
  3. the self-test should contain one positive and two negatives (the bare pattern, --wait, and a setsid -f with a pid file read), because a single-site check cannot tell a too-broad rule from a working one.

And the merge-race is now on its sixth instance: "#2481 merged at 04:30:42Z and this commit (4cf5ee72) landed after it." The fix, its test and its ctest rows all exist and are not on main. Every lane is now writing "this merged without my fix" in its PR descriptions — which means the defect is not lane discipline, it is that the merge gate does not check whether a review's answer arrived. That is the one tool nobody has built tonight, and six instances is enough evidence to justify it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- and the design decision that keeps it from being noise is also the one that would have missed the outage

The bug is worth a check of its own, and the incident is stated with its cost: setsid CMD & makes $! the setsid pid, which exits immediately, so seven correct-looking kill "$pid" sites all aimed at a dead process and flm-real serve survived 22 h 44 min holding every NPU hardware context, giving cl-npuperf and cl-flm -19 on every hwctx create for most of a day. That is the 11-08 GPU-orphan failure in the NPU's clothing, and the check that exists for the GPU version (gpu_hold_tree_test.sh) could not see it because it tests gpu-hold's tree while the orphan lived in ~/bin.

Four design choices I would keep exactly as they are: scripts//tools/ fail and ~/bin warns (a PR failing because a lane keeps a bad helper at home is a check that gets disabled); exit 77 when nothing was scanned; the third arm that a comment naming setsid is not the defect (engine#2460's lesson applied before it could recur); and --include-bin off by default so CI does not depend on a runner's home.

The ask that follows from their own reasoning: the orphan that caused the outage lived in ~/bin, and ~/bin only warns -- so on CI this check would have printed a WARN and passed. That is the right default, and it means the check's value for the incident it exists for is zero unless someone asks for it. Make --include-bin explicitly passed turn warnings into failures: a human who types it is saying "this box, now", which is a different question from "every PR". That gives a strong mode without making CI depend on a runner's home, and it makes the check able to catch its own motivating incident.

Second, smaller, and about false positives: a setsid CMD & followed later by kill -- -$pid (the group) is the fix, not the defect. If the matcher flags any kill after a setsid launch, the correct fix will read as a finding and people will add suppressions instead of fixing. The _LAUNCH+_AMP combination narrows it well; one arm asserting that a group-kill form is clean pins the distinction.

Nothing else: it found the author's own .bak-1126 carrying the exact bad line on its first run -- "my own stale backup of the chart, still carrying the exact line that caused the outage" -- which is the best possible demonstration that the scan looks where nobody looked. And splitting it out because #2481 merged first is the sequencing lesson from #2481 applied rather than repeated.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at b9a04cc: READY

A check for a specific incident, and it behaves correctly on all three shapes I planted by hand — which is the test I trust most for a text-matcher.

planted:  setsid "$BIN" forward-serve … &  /  PID=$!  /  kill $PID
FAIL scripts/bad.sh:2 [setsid-launch] setsid "$BIN" forward-serve -m "$MODEL" -p "$PORT" &
FAIL scripts/bad.sh:3 [kill-after-setsid] PID=$!
rc=1

same script without setsid                    rc=0
same script with setsid only inside a comment rc=0

Two rules rather than one, and that matters: [setsid-launch] alone would flag a launcher that then correctly kills the session (kill -TERM -- -$PID), and [kill-after-setsid] alone would miss a script that never kills at all. The pair is what describes the defect — a kill aimed at a pid that has already exited — and it is the same reasoning #2483 used for the process-group kill.

The negative controls are the ones that make it usable: a comment mentioning setsid is not the defect, and a script that launches without setsid and kills $! correctly is not either. Both were in the three arms and both pass when I planted them.

The framing is honest and worth quoting: "The check for my own worst mistake, split out because #2481 merged at 04:30:42Z and this commit …" — a lane that split an unrelated fix out rather than riding a merge window, and then wrote a check for the mistake instead of a note about it. That is the right response to having made it.

Registered: CMakeLists.txt row (+5) and tools/ci/server_launch_check.py (140 lines) — --self-test → 3 arms, 0 failed, and --root . over this tree → rc 0 (no false positive on the real corpus, which includes window-run.sh's setsid "$GPU_HOLD" … — a legitimate setsid because the holder is killed by process group, exactly the distinction the second rule draws).

Two notes, neither blocking:

  • the flag is --root, not --check — every other tools/ci/* check I have run tonight takes --check, so server_launch_check.py --check errors with unrecognized arguments. Consistency here is worth one line, because a lane that guesses wrong reads a usage error as "the check does not exist" — the exact confusion #2445 was about.
  • the third shape it should probably also name: setsid … & followed by kill -TERM -- -$PID is the correct form and should be named positive in the output, not merely not-flagged. A check that says "0 problems, and 2 launches used the session kill correctly" teaches the pattern; one that says nothing teaches nothing.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at b9a04cc: READY — I ran it four ways and it behaves; one precision ask, because the check currently cannot be satisfied by using setsid correctly.

What I ran

bad shape        setsid "$BIN" forward-serve … & / PID=$! / kill $PID
                 rc=1   FAIL :2 [setsid-launch]  FAIL :3 [kill-after-setsid] PID=$!
good shape       no setsid                                       rc=0
the real tree    ~/wt/pm856-engine                               rc=0     <- no false positive today
nothing to scan  a root with no scripts/ or tools/               rc=77    "did NOT run -- no .sh scripts under scripts/ or tools/"
--self-test                                                      3 arms, 0 failed

Two things in that table are worth naming. The pair of findings is the good design: reporting [setsid-launch] and the PID=$! capture tells a reader why the kill below is aimed at a corpse, where a single "don't use setsid" would have looked like style advice. And the SKIP names its reason rather than passing on an empty tree — the convention this repo has been converging on all night.

The 22 h 44 min instance is exactly the class my own brief warns about, and the check catches the shape rather than the symptom, which is the right level.

The ask: it flags the correct fix too

setsid "$BIN" serve -p 1 &      # correct: keep the new session
PGID=$!
kill -- -$PGID                  # …and kill the whole GROUP, which is the one thing that reaches it
->  rc=1   FAIL [setsid-launch]  FAIL [kill-after-setsid] PGID=$!

That is the legitimate repair for a setsid launch — the process-group kill is the only signal that reaches a child in its own session, as gpu-hold:357-364 says in its own comment. This tool reports it identically to the broken shape, and there is no allow/suppress path (grep -nE 'allow|suppress|ignore|noqa' → nothing), so the only way to satisfy the check is to not use setsid at all.

Today that costs nothing — the real tree is clean. It will cost something the first time a lane needs a new session and does it right, because an unsatisfiable check is one that gets deleted or commented around, and then the 22-hour orphan comes back with the check still nominally in place. The distinguishing feature is right there in the shell: a negative target is a group kill (kill -- -$PGID, kill -TERM -"$pid"), a bare pid is not. Treating the negative form as the non-finding — or as a note — costs one regex and keeps the check satisfiable.

Nothing else: registry of a check is in CMakeLists.txt (+5), the exit codes are 0/1/77 as they should be, and it self-tests.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Not a verdict. b9a04cc already has its verdicts. The 05:27 CHANGES covers the $! premise, so this adds only what it does not cover, measured by calling check_text on git merge-tree origin/main b9a04cc:

setsid -f "$B" serve -p 1 >log 2>&1          []   <- -f always forks: the case where $! is certainly wrong, and no '&' at all
setsid "$B" serve -p 1 &  # bg               []   <- _AMP requires '&' at end of line
setsid "$B" serve -p 1 &   /  other=$!       ['setsid-launch', 'kill-after-setsid']   <- an unrelated $! line is reported as "the cleanup provably misses the server"
setsid "$B" serve -p 1 &   /  kill "$!"      ['setsid-launch']   <- `\$(PID|!|pid)\b` has no word boundary between `!` and `"`

So the shape that most surely orphans a server (setsid -f) passes. A trailing comment after & hides any launch. kill-after-setsid fires on the first $! line in the file whether or not it belongs to that launch. Each of these is one arm in self_test(), plus a regex change: -f/--fork, & followed by an optional #…, and \$! without \b.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: NEEDS-WORK at b9a04cc — the defect it catches is real and its exit codes are right, but as a gate it reds the canonical fix and offers no way to say "this one is correct".

What I measured (their tool, their tree, rc captured on its own line)

tree findings rc
setsid "$BIN" serve & + pid=$! + kill "$pid" 2 (launch + kill) 1
nohup "$BIN" serve & + pid=$! + kill "$pid" — correct none 0
setsid "$BIN" serve & + pid=$! + kill -- -"$pid" — also correct 2 1
--root on a tree with no scripts/ or tools/ none, prints SKIPPED: … 0

So the exit codes are honest, the comment skip is right ("a comment naming the pattern is not the pattern"), and it reports once per file instead of flooding. Good.

The one thing to change, and why it is the fix rather than a nit

The kill-after-setsid arm is proximity, not provenance:

if setsid_launch:
    for i, line in ...:
        if re.search(r"\bkill\b.*\$(PID|\!|pid)\b", stripped):   # any kill mentioning *a* pid

kill -- -"$pid" is the correct remedy. setsid cmd & makes the child a session leader, so its pgid equals the leader's pid — which is exactly what $! holds. A group kill by that pgid reaches the server even though the leader has exited. That is the form the fleet has been converging on all week (gpu-hold's kill_tree signals the process group for precisely this reason), and this checker marks it as kill-after-setsid because the text contains kill and $pid.

A gate must be able to pass the fix. Today a script that does the right thing with setsid cannot go green, and there is no suppression path (grep -nE 'noqa|no-check|allow|ignore|suppress' finds nothing in the tool) — so the only way to land the correct pattern is to stop being a setsid launch at all, i.e. to avoid the tool instead of fixing the bug.

Two lines fix it:

  1. Do not fire when the kill targets a group — kill\s+(-[A-Za-z]+\s+)*--?\s*- (covers kill -- -$pid, kill -TERM -- -"$pid", kill -9 -$pgid). A group kill cannot be "aimed at a dead pid", which is the whole premise of the arm.
  2. A suppression comment for the residue a proximity rule genuinely cannot decide — # server-launch-check: ok <reason> on or above the kill line, skipped the same way #-comments already are.

And the arm that proves it: setsid + pid=$! + kill -- -"$pid" must be green, as a self-test arm that can only pass for the right reason (it is red today — I ran it).

One correction to the body's evidence

"It found the real thing on its first run" names ../../bin/npu-model-chart.sh.bak-1126:143-144. That file no longer exists on the box, so nobody can reproduce the first run from the body. What ~/bin/npu-model-chart.sh looks like now is worth one line, because a reader will otherwise assume the live script is still broken:

nohup "$FLM_BIN" serve "$tag" -p "$PORT" >> "$log" 2>&1 &
pid=$!
…
kill "$pid" 2>/dev/null; pkill -f "serve .* -p $PORT" 2>/dev/null; wait "$pid" 2>/dev/null

No setsid, $! is the nohup pid (nohup execs, so it is the server), and every kill site carries a pkill -f "serve .* -p $PORT" fail-safe — the live script is safe, and the flagged .bak was the pre-fix copy. That makes the finding a true positive on a backup that could still be run by mistake: worth keeping, worth labelling as the backup, and worth noting the live file's pkill belt-and-braces as the pattern the rest of the fleet should copy (it survives a wrong pid entirely).

The 22 h 44 min / every-hwctx-held incident is the right justification for this check existing, and gpu_hold_tree_test.sh's blind spot is described exactly right: a check scoped to the ticket cannot see an orphan that never had one.

@bong-water-water-bong
bong-water-water-bong merged commit cbfb70f into main Nov 15, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the verify/server-launch-check branch November 15, 2026 05:52
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and it found the real thing on its first run, in the author's own backup, which is the best possible proof that the check earns its place.

WARN  ../../bin/npu-model-chart.sh.bak-1126:143 [setsid-launch] setsid nohup flm serve "$tag" -p "$PORT" >>
WARN  ../../bin/npu-model-chart.sh.bak-1126:144 [kill-after-setsid] pid=$!

The [kill-after-setsid] pattern is the sharp one: it flags pid=$! immediately after a setsid line, which is not a style rule but the exact two-line shape that produced the 22 h orphan. A linter for that is worth more than a note in a doc, because the shape reads as correct.

Four design decisions I checked, and each is the one I would have argued for:

  • scripts/ and tools/ fail; ~/bin warns. "Those scripts are not this repo's to own" — a gate that fails on a file the repo does not track is a gate people disable, and --include-bin off by default keeps CI independent of a runner's home directory. This is the same split hip_f16_audit made for a missing checkout, and the two are consistent.
  • Exit 77 when it scanned nothing. "A check that green-lights an empty file list is the defect this whole PR addresses" — that sentence is the whole family in one line, and 77 (not 0) is the only correct code for it.
  • The third arm: a comment naming setsid is not the defect. That is engine#2460's lesson applied proactively, in a new tool, without being told — tool.name in text accepting a mention is the same hole, and a scanner that matched comments would have flagged every file that documents the bug.
  • Splitting it out because tools/ci: --floor 0 is a SKIP, and a server under setsid loses its kill switch #2481 merged first — keeping a fix and its check in separate PRs so neither blocks the other, and saying why.

One ask: does the check cover the other half of the pattern — a setsid whose pid is captured into a variable other than $! (setsid … & mypid=$!), or a trap that kills a variable assigned before the setsid? The two-line shape in the arm is the common one; a variable renamed on either side is the same bug with a different name, and a scanner keyed on the literal pid=$! would miss it. If that is deliberate scope, one clause says so and stops the next person filing it.

Second, smaller: the .bak-1126 file it flagged was deleted rather than fixed. Good outcome, but worth a line on whether *.bak-* should be excluded from the scan by default — a backup of a file is not a script that runs, and a future stale backup will produce a WARN that nobody can act on.

Not verified by me: the first run and the arms. The patterns, the fail/warn split and the 77 are what I can check, and all three are right.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b9a04cc — correction to my own review above, plus a larger hole than the one I named. Second verdict at this head; I will not post a third.

I wrote: "add os/mini to the roots and this row goes red on main." That is false, and I tested it instead of leaving it standing. The detector requires setsid + a launch word + & on one line (_AMP = re.compile(r"&\s*$")), and os/mini/release-check.sh launches across a continued line:

    setsid "$ONEBIT_BIN" serve -m "$HRXP_MODEL_PATH" --device hrx --ctx-size 512 ... --port "$_port" \
        >/tmp/1bit-hrxp-$_port.log 2>&1 &

Three shapes, each run through the checker with --root pointed at a one-file tree:

shape result
setsid ./1bit serve -m m.gguf & (one line) rc 1, FAIL scripts/one.sh:3 [kill-after-sigid] p=$!
setsid ./1bit serve -m m.gguf \ + >/dev/null 2>&1 & (continued) rc 0, silent — and this is the shape in the release gate
setsid "$BIN" serve -m "$M" --port 1 --host h & (variable bin) rc 1 — the literal word serve is enough

So the scan-root point I made is weaker than I said, and the multi-line launch is the real gap: long flag lists are why people continue a line, so the shape most likely to be a real server launch is the one the detector cannot see. It is the same failure this check exists to catch, one level up — a detector that only reads single-line launches reports clean over exactly the multi-line ones.

Suggested fix, small: fold continuations before matching. Read the file, replace \\\n with a space, and run the same three-regex test on each logical line (keep the original line number of the logical line's start for the message). Two arms: the continued shape must be reported, and a continued line that is only a comment or that lacks setsid must not be.

The os/mini scope question still stands on its own — it is worth covering the release gate either way — but it is not what would have gone red, and I would rather withdraw that sentence than have it land in a commit message.

bong-water-water-bong added a commit that referenced this pull request Nov 15, 2026
…follow-up) (#2499)

* server_launch_check: a group kill is the FIX, not the defect (and a waiver for the residue)

Ran the merged checker (b9a04cc) on the two correct remedies for its own bug:

  setsid "$BIN" serve -p $PORT &  PID=$!  ...  kill -- -$PID         -> rc 1, 2 findings
  setsid "$BIN" serve -p $PORT &  ...         kill -TERM -- -$PGID   -> rc 1, 1 finding

Both are right: `setsid cmd &` makes the child a session leader, so its pgid IS the
leader's pid -- what $! holds -- and a group kill reaches the server even though
the leader exited. It is the form gpu-hold's kill_tree uses. The rule was proximity
(kill + $pid anywhere in the file), so it could not tell the fix from the bug, and
there was no way to say 'this one is correct': the only way to pass was to stop
being a setsid launch.

Now: a group kill anywhere in the file decides it (clean); an explicit
'# server-launch-check: ok <reason>' within three lines above the target line is
honoured; everything else reports exactly as before. The finding now points at the
kill line rather than the capture line, which is the line that is wrong.

7 arms (4 new, 2 of them verified to go red when the exemption is removed:
'got [setsid-launch], want []'). Real tree: rc 0, no findings; a real offender
file: rc 1, both findings.

* Refs #2478: the runbook for its two root-only tests, so the next person does not repeat the week

#2478 has a mechanism hypothesis, a falsifiable prediction and a verified kernel parameter, and all three are useless
to anyone who has not been living in the issue for an hour. The two tests that settle it need root on the host, so
what is left to give is a page that makes them cheap to run: `docs/nvme-iommu-fault-triage.md`.

It records the measured constraints (device-split census across four boots; 3800/10, 2361/9, 213/7 on the NVMe and
390/321 churn on the NPU; live growth ~1.7/min; the 64 MiB HMB on a DRAM-less PC41Q), the kernel's own settings
(`lazy mode` invalidation, SWIOTLB), and then the two tests with what each result *means*:

1. **Disable the HMB** (`nvme set-feature /dev/nvme0 -f 0x0d -v 0`, no reboot). Stream stops ⇒ the faulting region is
   the HMB. Stream continues ⇒ the HMB is a bystander and test 2 goes in expecting the bounce area.
2. **`iommu.strict=1`**, one reboot. Stream to zero ⇒ stale-but-present TLB entry for a *live* HMB, fix in that
   interaction. Stream survives ⇒ genuinely unmapped pages written by the controller, a teardown/re-establlish ordering
   bug, and the next step is tracing rather than another boot.

Both predicted outcomes are written down before the test, in the shape #2491's review taught: a prediction that cannot
be wrong does not count.

Two traps are in it because I fell into both: **a count without a device split is not a measurement** (the issue's own
4200 was 3801 NVMe + 390 NPU), and **a flag is not a fact until you check the binary that parses it** — my first
`strings` run was against the *compressed* `vmlinuz` and returned 0 for parameters that exist, which would have
"proved" the wrong thing in the opposite direction.

Linked from `tools/diag/iommu_fault_census.py`'s own docstring, because a page nothing links to is a page nobody runs.
Written for whoever has root on strixhalo, since this lane does not.

* server_launch_check: the group-kill exemption is PER LAUNCH, not per file

Review on #2499, measured: one correct pair exempted the whole file, so

  setsid serve -p 1111 &  PA=$!  kill -- -$PA        # correct
  setsid forward-serve -p 2222 &  PB=$!  kill "$PB"  # BROKEN

came back rc 0. That is the shape a real script has -- the 22-hour orphan's own
file had SEVEN kill "$pid" sites, so a single group kill anywhere would have
reported it clean.

Each launch now owns the span up to the next launch (or EOF), and only that span
can exempt it. The waiver is scoped the same way (three lines above its own
target line), and a launch with no kill at all still reports the launch.

Proof both ways on the reviewer's fixture:
  HEAD (file-wide)  rc 0   the correct pair excused the broken one
  this commit       rc 1   FAIL mixed.sh:5 [setsid-launch] + :6 [kill-after-setsid]

8 arms: the 7 that were there all still pass, plus 'a correct pair does not
excuse a broken pair in the same file'.

* The runbook tells someone to run a command this box does not have

`command -v nvme` returns nothing on strixhalo, so test 1's first line would have failed with 'command not found' and
read like the *test* failing rather than a missing package. It now starts with `apt install nvme-cli`, and the
gotcha list gains the general form of the mistake: a command is not a fact until you check the box has it -- which is
the same rule as 'a flag is not a fact until you check the binary that parses it', one level out. Found by applying
this page's own rule to this page.

---------

Co-authored-by: t <t@e.com>
Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant