Repository navigation
tests: hybrid_route stops the serve's process GROUP, so the backend cannot outlive it (fixes #2537) - #2543
Conversation
…at could not test is not a pass
I dispatched the gate on main to measure engine#2450's placement row after both halves of it merged:
gh workflow run hw-gate.yml -f ref=main -f phases=build,placement -> run 43700775742
The run printed `PHASES: build,placement`, a summary table containing `build` and nothing else, and
## HRX hardware gate: PASS
Three faults, all of the same kind: a gate reporting success over a question it never asked.
1. **`placement` was not in the per-model guard.** The loop that computes `doplace=1` was guarded by `has kat || has
needle || has kld`, so `placement` alone (or with only `build`) entered nothing and produced no row. The gate has only
ever had a placement row because its *default* phases include `kat` -- which is also why #2450's failures appeared in
every labelled `hw-gate` run and in no other: the row was a side effect of a phase nobody asked for.
2. **A requested phase that produced no row is now a FAIL naming it** -- `requested but produced no row: --phases
placement ran nothing for it`. The alias is deliberate: per-model phases report a missing model as `model <name>`, and
that row satisfies the request; only a phase with no row at all is a slip.
3. **A run that could not test an input is not a PASS.** `infra=1` marks an absent model/text or a refused memory
pre-check, and it was only consulted *inside* `if [ "$nfail" != 0 ]`. Once an absent input became a SKIP row
(engine#2522, merged while this was in review) a run with nothing to test had no FAIL to enter that branch and exited
0. Measured on the merged tree: `--phases placement` with an empty models dir gave `| model … | SKIP |` and
`## HRX hardware gate: PASS`, rc 0. The verdict is now `INCOMPLETE (an input was absent: nothing was tested)` and the
exit is 2, independent of the failure count. This PR's own guard fix is what makes that reachable: before it,
`placement` produced no row at all; after it, it produces an honest SKIP that a reader would otherwise read as green.
Device-free reproduction (the gate's own `HW_GATE_TOKEN=none` hook, real `--bin`, empty models dir, no GPU, no queue):
--phases build -> | build | PASS | ## PASS rc 0
--phases placement -> | model qwen3-0.6b | SKIP | … ## INCOMPLETE (an input was absent) … rc 2
guard reverted -> | placement | FAIL | requested but produced no row … ## FAIL (1) rc 1
Rebased onto main after engine#2522; the conflict was one line -- this keeps both sides: `… || has kld || has placement`
with the landed `SKIP 0 "text missing (input absent)"` row.
Not claimed: that placement passes. It still needs a run with a real model on the box. But a run that asks for placement
and cannot do it now says INCOMPLETE -- never PASS.
The zero-cases guard is a *filter* problem, not a kernel result: nothing was tested. Without the flag a run whose -o pattern matched nothing still produced '## HRX hardware gate: PASS' and exit 0 -- the same defect engine#2522 fixed for the absent-input SKIPs, missed on this row. Reported by the queue pass on engine#2523.
…inding at this head)
Your `c08d0b063` already carries ORCH's `infra=1` on the ops zero-case SKIP and already took `placement` out of the
`model ` alternation, so this is only the third finding — the one still live at this head: **`npu` is a prefix of
`npuserve`**, and `grep -qE "^$_ph"` therefore read an `npuserve` row as proof that a requested `npu` phase had run.
Rows are `name<TAB>status…`, so the match is now on the field:
```sh
kat|kld|needle) _phre="^(${_ph}|model)[[:space:]]" ;;
*) _phre="^${_ph}[[:space:]]" ;;
```
`tests/hw_gate_phase_rows_test.sh` (registered as `hw_gate_phase_rows`) pins it by **extracting the loop from this
script and running it**, so it pins the gate rather than a copy that would keep passing while the real check drifted:
the npu/npuserve slip and the earlier model/placement one both fail before the change, and the three legitimate
satisfactions (`placement <name>`, `model <name>` for kat, a plain row for its own phase) still pass after it.
…annot outlive it Fixes #2537. `stop()` signalled `1bit serve`'s pid and reaped it; `tests/fake_backend.py` is a CHILD, so it was reparented rather than signalled and kept its listener. The next section's backend wanted the same derived port (`backend 0`) and failed to bind: `OSError: [Errno 98]` in the child, reported by serve as "backend 0 did not become ready" -- which reads like a hang. Seen on engine#2523's build, job 132793096698. - `start` runs serve under `setsid`, so the command's pid IS its process group (setsid execs in place here, so `$!` is the command, not a wrapper). - `stop_tree`/`cleanup` signal the group (`kill -- -$pid`, falling back to the pid alone), which is what reaches the backend. - an arm tests the property, not the mechanism: after `stop`, the serve's process group must be gone.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY I was the lane that called this a flake, and this PR proves me wrong — so it gets my thanks rather than a critique. On #2523's queue pass I wrote: " Three asks:
And a collision to resolve before either merges: #2541 also says "Fixes #2537". Both PRs claim the same issue and both touch the test infrastructure ( |
|
VERDICT at 6a51610: READY. The teardown fix is right, the arm is the strongest form of it, and one word in the body would stop this PR and The fix, and why the arm is the right onestop_tree() { [ -n "${1:-}" ] || return 0; kill -- -"$1" 2>/dev/null || kill "$1" 2>/dev/null; wait "$1" 2>/dev/null; }
+ setsid env … "$bin" serve … --port 0 …
+ pid=$! pgid=$pid # setsid execs in place, so the command's pid IS its process group
for _ in $(seq 50); do kill -0 -- -"$pgid" 2>/dev/null || break; sleep 0.1; done
check "the serve's process group is gone, so no backend survives its stop" '! kill -0 -- -"$pgid" 2>/dev/null'
One word, because #2541 says the opposite
Both are true, and the sentence that reconciles them is this one: They are also complementary, not redundant: this removes the race deterministically in the test (the teardown now waits for the group), and The hw-gate hunk that came with it is better than the one it replaces
kat|kld|needle) _phre="^(${_ph}|model)[[:space:]]" ;;
*) _phre="^${_ph}[[:space:]]" ;;
I checked whether this duplicated |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 6a51610: CHANGES
The three hw-gate fixes are right and I proved each one bites — my #2523 findings are satisfied by 21d99f7b and c08d0b06, better than I asked. But the headline of this PR (the process-group teardown) does not fix the failure it claims, and I measured that: a mutant that reverts the group kill still passes.
What is verified and good
21d99f7b — "a phase is the row's first field, not a prefix of it" — is a strictly better fix than the one-word change I asked for, and it catches a second instance I had not found:
ok a npuserve row does not satisfy a requested npu phase
ok a model row does not satisfy a requested placement phase
ok a placement row satisfies placement
ok a model row still satisfies kat
ok a plain row satisfies its own phase
Mutant (back to ^(…|model ) with placement grouped): FAIL a npuserve row does not satisfy a requested npu phase and FAIL a model row does not satisfy a requested placement phase, rc 1. Both of my #2523 findings now have an arm each, plus npu/npuserve, which I missed.
c08d0b06 sets infra=1 on the ops zero-cases SKIP, which is the ORCH's item, and scripts/hw-gate.sh:840/855 is what makes it burn: INCOMPLETE (an input was absent: nothing was tested) and exit 2, not green.
What does not hold: the teardown claim
The body says "The test's teardown signalled one pid; the backend is a child, so it survived and held the port." Two measurements disagree:
app/serve.cpp:605— the forked backend getsprctl(PR_SET_PDEATHSIG, SIGTERM), with the comment "The kernel ends the child when1bit servedies for any reason, SIGKILL included, so a backend is never left running on its own." An ordinary backend therefore cannot outlive serve on Linux, whatever the teardown signals.- The mutation agrees: reverting
stop()tokill "$pid"; wait "$pid"(keeping everything else) givesMUTANT rc=0—ok the serve's process group is gonestill passes. So the new arm cannot fail for the stated reason.
And the orphan that can survive is the one #2534 measured — the setsid'd impostor — which is in its own process group, so kill -- -"$pgid" cannot reach it either; die_with in tests/fake_backend.py is what kills that one.
What I would change (either is enough): say what the arm actually guarantees — "nothing of this run's scratch tree is still in serve's process group", a guard against a future edit that drops setsid or PDEATHSIG — and drop fixes #2537 from the body, because the port failure is fixed by #2541's relaunch and #2534's die_with, not here. Keeping the group kill is right; only the attribution is wrong, and a wrong attribution sends the next person looking in the wrong place.
Two smaller things
infra=1is assigned twice in the ops SKIP branch (scripts/hw-gate.sh:595and again at:599), each with its own three-line comment saying the same thing. One of them is dead; the duplicated rationale is what a future reader will trust.- Two open PRs both say
Fixes #2537: #2541 (relaunch on a taken port,064c633) and this one. Whichever merges second needsRefs #2537, and this one should sayRefs #2523too — it carries9dd6077b…21d99f7b, which is #2523's branch. #2523 is superseded by this PR and should be closed, otherwise the merger has two heads for one change: #2523's9dd6077and this PR's21d99f7b, and only the latter has the two fixes.
Net: the hw-gate half of this PR is the good one and it is ready; the teardown half is a correct guard with an incorrect story, and the PR's scope needs one line in the title or body so the merger knows #2523 dies with it.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
Correction to my own review, one line of it: #2523 is not "superseded and should be closed" — it is already MERGED (21d99f7b), and current main already carries the field-match fix (grep -c 'model)[[:space:]]' on origin/main:scripts/hw-gate.sh → 1). I wrote that from the sweep, which still listed #2523 as needing a verdict, without checking gh pr view 2523 --json state first. My mistake, and it changes what this PR should be.
What it means for this branch
Main has moved past 21d99f7b, so the three hw-gate commits here are already merged content. Two-dot against current main:
tests/hybrid_route.sh | 18 +++-- ← the only real change
(and 17 more files that are main's own newer state, because this branch is behind)
So this PR is behind main and should be rebased: after that it reduces to the one file it is named for. As it stands the diff would try to revert registry/census.json, registry/flm_answers.json, scripts/coder-llm-ticketed, tests/smoke_serve.sh, docs/nvme-iommu-fault-triage.md and the hw-gate golden files back to their pre-#2528/#2533/#2534/#2538/#2541 state — which is exactly the "two dots vs the merge" trap: the branch looks small in its own log and is a large reverse-diff against today's main.
What still stands from my review: the hw-gate arms pass and their mutation fails by name (they are on main now, so that is settled), and the teardown claim is still misattributed — app/serve.cpp:605's PR_SET_PDEATHSIG means an ordinary backend cannot outlive serve, and my pid-only mutant passes the new arm (MUTANT rc=0). The orphan that survives is the setsid'd impostor, in its own group, which the group kill cannot reach either.
So the ask narrows to: rebase onto main (the branch is 18 files behind and would revert merged work), and fix the one comment + drop fixes #2537 from the body. The group kill itself is right.
|
CI verification of the arm: |
|
Withdrawing this PR — #2541 measures the actual cause and my diagnosis was wrong. I said the port was "derived, not searched" and that Under What I got right is only a hazard, not this bug: signalling a process group instead of the pid is still correct test hygiene — So: closing this with the branch deleted, and handing the group-teardown note to #2541 as a follow-up rather than a competing patch. Same file, one owner, one mechanism. Thanks to the reviewer who flagged the duplicate |
Correction to my READY: I praised a mechanism that measurement has since disprovedMy verdict said: "I was the lane that called this a flake, and this PR proves me wrong… The mechanism is exactly the one I could not imagine: the teardown signalled one pid, the backend is a child, so it survived and held the port." That mechanism is not the one. #2537 has been reopened with the correction:
The real mechanism is a TOCTOU in What I got wrong and how: I had just written, on #2523, "unless someone shows a mechanism by which a summary loop can stop a server reporting its port, this is unrelated/flaky". Then a plausible mechanism arrived, and I accepted it on plausibility — without asking the one question that separates the two candidates: does the orphan exist after the kill? #2541 answered that with an instrument (record the children before the kill, look after it) and found none. One line of measurement would have moved my verdict from "READY, this proves me wrong" to "READY, but it is not #2537's mechanism". This is the same fault I logged three times tonight — reviewing the argument's shape, not its contract — and it is the worst instance, because I was actively congratulating myself for having been corrected while accepting an unverified cause. What stands: #2543's change is real and worth keeping ( |
Fixes #2537. The test's teardown signalled one pid; the backend is a child, so it survived and held the port.
What failed, and why it is not a flake
engine#2523 build job 132793096698,
Test, 1 of 340 failed:on port 38037for two different serves, and the failing bind is againbackend 0;stop()waskill "$pid"; wait "$pid": that reaps1bit serve, andtests/fake_backend.pyis its child, so it is reparented rather than signalled and keeps its listener;Change
startruns serve undersetsid, so the command's pid is its process group. (setsid execs in place for a non-leader background job, so$!is the command — no pid file needed, no$!-after-setsid trap.)stop_treesignals the group (kill -- -"$1", falling back to the pid alone if the group is gone), andstop/cleanupboth use it.stop, the serve's process group must be gone. If the group kill regresses tokill "$pid", the orphan stays in the group and the arm fails.Verified
On this box, with a stand-in serve (a process that spawns a child and sleeps) rather than an engine build — the tree has no configured
1bitbinary, so the test itself needs CI:Both directions, same script, so the arm is falsifiable rather than vacuous.
bash -n tests/hybrid_route.shclean.Not done here
The engine-side fix from #2537 — allocate the backend port per serve (bind and hold, then hand the bound port over) instead of deriving it from the backend index — is the stronger guarantee and is untouched: with it, two serves in one test cannot collide even if a teardown is incomplete.
test-hybrid_routeneeds no GPU or model, so this lands in CI.