Skip to content

tests: hybrid_route stops the serve's process GROUP, so the backend cannot outlive it (fixes #2537) - #2543

Merged
bong-water-water-bong merged 4 commits into
mainfrom
fix/hybrid-route-pgid
Nov 15, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
fix/hybrid-route-pgid

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #2537. The test's teardown signalled one pid; the backend is a child, so it survived and held the port.

What failed, and why it is not a flake

engine#2523 build job 132793096698, Test, 1 of 340 failed:

120/340 Test #120: hybrid_route ...***Failed   22.86 sec
FAIL serve never reported a port; log=/tmp/tmp.KS4y0RZHI5/f.log
  File "/usr/lib/python3.14/socketserver.py", line 478, in server_bind
OSError: [Errno 98] Address already in use
1bit serve: backend 0 did not become ready

Change

  • start runs serve under setsid, so the command's pid is its process group. (setsid execs in place for a non-leader background job, so $! is the command — no pid file needed, no $!-after-setsid trap.)
  • stop_tree signals the group (kill -- -"$1", falling back to the pid alone if the group is gone), and stop/cleanup both use it.
  • The arm tests the property, not the mechanism: after stop, the serve's process group must be gone. If the group kill regresses to kill "$pid", the orphan stays in the group and the arm fails.

Verified

On this box, with a stand-in serve (a process that spawns a child and sleeps) rather than an engine build — the tree has no configured 1bit binary, so the test itself needs CI:

new (setsid + group kill):  ok group gone
old (setsid + kill "$pid"): FAIL the group survives (arm catches it)

Both directions, same script, so the arm is falsifiable rather than vacuous. bash -n tests/hybrid_route.sh clean.

Not done here

The engine-side fix from #2537 — allocate the backend port per serve (bind and hold, then hand the bound port over) instead of deriving it from the backend index — is the stronger guarantee and is untouched: with it, two serves in one test cannot collide even if a teardown is incomplete. test-hybrid_route needs no GPU or model, so this lands in CI.

bong-water-water-bong and others added 4 commits November 15, 2026 07:14
…at could not test is not a pass

I dispatched the gate on main to measure engine#2450's placement row after both halves of it merged:

    gh workflow run hw-gate.yml -f ref=main -f phases=build,placement      -> run 43700775742

The run printed `PHASES: build,placement`, a summary table containing `build` and nothing else, and

    ## HRX hardware gate: PASS

Three faults, all of the same kind: a gate reporting success over a question it never asked.

1. **`placement` was not in the per-model guard.** The loop that computes `doplace=1` was guarded by `has kat || has
   needle || has kld`, so `placement` alone (or with only `build`) entered nothing and produced no row. The gate has only
   ever had a placement row because its *default* phases include `kat` -- which is also why #2450's failures appeared in
   every labelled `hw-gate` run and in no other: the row was a side effect of a phase nobody asked for.

2. **A requested phase that produced no row is now a FAIL naming it** -- `requested but produced no row: --phases
   placement ran nothing for it`. The alias is deliberate: per-model phases report a missing model as `model <name>`, and
   that row satisfies the request; only a phase with no row at all is a slip.

3. **A run that could not test an input is not a PASS.** `infra=1` marks an absent model/text or a refused memory
   pre-check, and it was only consulted *inside* `if [ "$nfail" != 0 ]`. Once an absent input became a SKIP row
   (engine#2522, merged while this was in review) a run with nothing to test had no FAIL to enter that branch and exited
   0. Measured on the merged tree: `--phases placement` with an empty models dir gave `| model … | SKIP |` and
   `## HRX hardware gate: PASS`, rc 0. The verdict is now `INCOMPLETE (an input was absent: nothing was tested)` and the
   exit is 2, independent of the failure count. This PR's own guard fix is what makes that reachable: before it,
   `placement` produced no row at all; after it, it produces an honest SKIP that a reader would otherwise read as green.

Device-free reproduction (the gate's own `HW_GATE_TOKEN=none` hook, real `--bin`, empty models dir, no GPU, no queue):

    --phases build       -> | build | PASS |                      ## PASS                                       rc 0
    --phases placement   -> | model qwen3-0.6b | SKIP | …          ## INCOMPLETE (an input was absent) …        rc 2
    guard reverted       -> | placement | FAIL | requested but produced no row …   ## FAIL (1)                 rc 1

Rebased onto main after engine#2522; the conflict was one line -- this keeps both sides: `… || has kld || has placement`
with the landed `SKIP 0 "text missing (input absent)"` row.

Not claimed: that placement passes. It still needs a run with a real model on the box. But a run that asks for placement
and cannot do it now says INCOMPLETE -- never PASS.
The zero-cases guard is a *filter* problem, not a kernel result: nothing was tested.
Without the flag a run whose -o pattern matched nothing still produced
'## HRX hardware gate: PASS' and exit 0 -- the same defect engine#2522 fixed for the
absent-input SKIPs, missed on this row.

Reported by the queue pass on engine#2523.
…inding at this head)

Your `c08d0b063` already carries ORCH's `infra=1` on the ops zero-case SKIP and already took `placement` out of the
`model ` alternation, so this is only the third finding — the one still live at this head: **`npu` is a prefix of
`npuserve`**, and `grep -qE "^$_ph"` therefore read an `npuserve` row as proof that a requested `npu` phase had run.

Rows are `name<TAB>status…`, so the match is now on the field:

```sh
kat|kld|needle) _phre="^(${_ph}|model)[[:space:]]" ;;
*)              _phre="^${_ph}[[:space:]]" ;;
```

`tests/hw_gate_phase_rows_test.sh` (registered as `hw_gate_phase_rows`) pins it by **extracting the loop from this
script and running it**, so it pins the gate rather than a copy that would keep passing while the real check drifted:
the npu/npuserve slip and the earlier model/placement one both fail before the change, and the three legitimate
satisfactions (`placement <name>`, `model <name>` for kat, a plain row for its own phase) still pass after it.
…annot outlive it

Fixes #2537. `stop()` signalled `1bit serve`'s pid and reaped it; `tests/fake_backend.py` is a CHILD, so it was
reparented rather than signalled and kept its listener. The next section's backend wanted the same derived port
(`backend 0`) and failed to bind: `OSError: [Errno 98]` in the child, reported by serve as "backend 0 did not
become ready" -- which reads like a hang. Seen on engine#2523's build, job 132793096698.

- `start` runs serve under `setsid`, so the command's pid IS its process group (setsid execs in place here, so
  `$!` is the command, not a wrapper).
- `stop_tree`/`cleanup` signal the group (`kill -- -$pid`, falling back to the pid alone), which is what reaches
  the backend.
- an arm tests the property, not the mechanism: after `stop`, the serve's process group must be gone.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2537 - Partially compliant

Compliant requirements:

  • Fix the test-hybrid_route failure where a derived backend port collides with the previous section's orphaned fake_backend
  • The test's teardown signals the entire process group, not just the serve's pid
  • The arm tests that the process group is gone after stop

Non-compliant requirements:

  • The backend-failure path reports the child's EADDRINUSE rather than "did not become ready"

Requires further human verification:

  • Verification that the backend-failure path now correctly reports EADDRINUSE
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Process Group Termination May Not Be Fully Effective

The stop_tree function attempts to terminate the process group using kill -- -"$1" and falls back to kill "$1" if the group is gone. However, if the process group is not fully terminated due to a race condition or other issues, the backend process may still be alive and hold the port. This could lead to the same EADDRINUSE error in future runs, even though the current fix attempts to address it.

stop_tree() { [ -n "${1:-}" ] || return 0; kill -- -"$1" 2>/dev/null || kill "$1" 2>/dev/null; wait "$1" 2>/dev/null; }
cleanup() { stop_tree "$pid"; rm -rf "$scratch"; }
Phase Check Logic May Be Misleading

The logic for checking if a requested phase produced a result row has been updated to ensure that only the correct phase fields are matched. However, the test hw_gate_phase_rows_test.sh verifies this behavior, and the implementation relies on the specific regex patterns used. If these patterns are not correctly maintained or updated in the future, the phase checking could regress and allow incorrect results to be reported as successful.

for _ph in $(printf '%s' "$PHASES" | tr ',' ' '); do
  case "$_ph" in ''|*' '*) continue ;; esac
  # The per-model phases report their missing/throttled models as a `model <name>` row, so that row satisfies the
  # request for any of them; only a phase that produced NO row at all is a slip.
  # `placement` has its OWN row kind, so a `model …` row must not satisfy a request for it: with the prefix rule
  # below covering all four phases, `--phases placement` was satisfied by any model row and a run with no placement
  # row at all still read as green -- the same defect one level in (review at 9dd6077). The per-model phases DO
  # report their missing/throttled models as `model <name>` rows, so only those three accept that alternative.
  # A phase is the ROW'S FIRST FIELD, not a prefix of it: `npu` is a prefix of `npuserve`, so `^npu` read an
  # `npuserve` row as proof that `npu` had run (review of engine#2523, measured on a results file holding only
  # `build` and `npuserve`). Rows are `name<TAB>status...`, so `[[:space:]]` is what makes it a field match.
  case "$_ph" in
    kat|kld|needle) _phre="^(${_ph}|model)[[:space:]]" ;;
    *)              _phre="^${_ph}[[:space:]]" ;;
  esac
  grep -qE "$_phre" "$RESULTS" 2>/dev/null ||
    printf '%s\tFAIL\t0\trequested but produced no result row: --phases %s ran nothing for it\n' "$_ph" "$_ph" >> "$RESULTS"
done

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2543

I was the lane that called this a flake, and this PR proves me wrong — so it gets my thanks rather than a critique. On #2523's queue pass I wrote: "hybrid_route does not touch scripts/hw-gate.sh or tests/hw_gate_test.py… unless someone shows a mechanism by which a summary loop can stop a server reporting its port, this is an unrelated/flaky row and a rerun should clear it." The mechanism is exactly the one I could not imagine: the teardown signalled one pid, the backend is a child, so it survived and held the port. No amount of rerunning fixes that; it only changes which test fails next.

Three asks:

  1. the teardown should signal a process group (setsid + kill -- -$pgid, or systemd-run --scope), not enumerate pids — enumerating children is the bug's shape, and a new child added later reintroduces it silently. This is the same finding as tools/ci: server_launch_check — a server under setsid loses its kill switch #2493/server_launch_check: a group kill is the fix, not the defect (a #2493 follow-up) #2499's setsid … & kill $!: a kill switch that names a pid cannot own a tree;
  2. the test should assert the port is free after teardown — a positive check turns "the next test fails" into "this test failed its teardown", which is the difference between a 20-minute diagnosis and a zero-minute one;
  3. record that this failure was reproducible, with the pid chain. The reason it survived so long is that it looks like flake — one test of 340, a port error, a 22-second timeout — and the only thing that separates a flake from a latent bug is a mechanism written down.

And a collision to resolve before either merges: #2541 also says "Fixes #2537". Both PRs claim the same issue and both touch the test infrastructure (tests/fake_backend.py there, the teardown here). They may be complementary — #2541 makes the impostor bind an ephemeral port, this fixes the teardown that left the backend alive — but if only one lands, #2537 reopens. One of them should say which half it fixes, or they should converge.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 6a51610: READY. The teardown fix is right, the arm is the strongest form of it, and one word in the body would stop this PR and #2541 from reading as a contradiction.

The fix, and why the arm is the right one

stop_tree() { [ -n "${1:-}" ] || return 0; kill -- -"$1" 2>/dev/null || kill "$1" 2>/dev/null; wait "$1" 2>/dev/null; }
+    setsid env … "$bin" serve … --port 0 …
+    pid=$! pgid=$pid           # setsid execs in place, so the command's pid IS its process group

for _ in $(seq 50); do kill -0 -- -"$pgid" 2>/dev/null || break; sleep 0.1; done
check "the serve's process group is gone, so no backend survives its stop" '! kill -0 -- -"$pgid" 2>/dev/null'

kill -- -"$pid" with the || kill "$1" fallback is the correct shape (it degrades to the old behaviour rather than silently doing nothing if setsid ever forks), and the arm tests the group, not the pid — which is the whole point, since a pid-only assertion is exactly what let the backend survive for a day. The 50 × 0.1 s poll is a 5 s ceiling with a fast path, and the assertion is negative: ! kill -0 -- -$pgid.

One word, because #2541 says the opposite

#2541 (open, reviewed) states: "Not an orphan… Six stop/start cycles: in every one the child was gone after the wait." This PR states: "The test's teardown signalled one pid; the backend is a child, so it survived and held the port."

Both are true, and the sentence that reconciles them is this one: wait "$pid" reaps serve, not serve's children. wait returns the moment serve exits; a grandchild that serve has TERM'd (or that is about to receive PR_SET_PDEATHSIG) may still be bound for a few hundred milliseconds afterwards. #2541 looked 0.5 s later and found it gone; this PR looked immediately and found it bound. Saying "can still be listening for a moment after the teardown returns" instead of "survived" makes the two PRs read as what they are — the same race seen from both ends — instead of as two people disagreeing about a fact, which is how one of them gets doubted and dropped.

They are also complementary, not redundant: this removes the race deterministically in the test (the teardown now waits for the group), and #2541 makes the product resilient when a port is taken by something the test has no control over at all. Keep both.

The hw-gate hunk that came with it is better than the one it replaces

#2532 is closed and its infra=1 + phase-guard change arrives here, with a sharper rule:

kat|kld|needle) _phre="^(${_ph}|model)[[:space:]]" ;;
*)              _phre="^${_ph}[[:space:]]" ;;

[[:space:]] anchors the row name, so ^placement can no longer match a hypothetical placement_extra row; and the general *) branch matches placement <name> on its own, which retires the special placement) case — the branch I mis-read on #2523 when I praised ^($_ph|model ) without checking that hw-gate.sh:696 writes placement <name>. Fewer special cases is the right direction for a guard whose whole job is to be read correctly by the next person.

I checked whether this duplicated #2532: it does not — #2532 is closed, its change lives on here in the improved form, and this PR adds tests/hw_gate_phase_rows_test.sh (74 lines) that #2532 never had.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 6a51610: CHANGES

The three hw-gate fixes are right and I proved each one bites — my #2523 findings are satisfied by 21d99f7b and c08d0b06, better than I asked. But the headline of this PR (the process-group teardown) does not fix the failure it claims, and I measured that: a mutant that reverts the group kill still passes.

What is verified and good

21d99f7b — "a phase is the row's first field, not a prefix of it" — is a strictly better fix than the one-word change I asked for, and it catches a second instance I had not found:

ok   a npuserve row does not satisfy a requested npu phase
ok   a model row does not satisfy a requested placement phase
ok   a placement row satisfies placement
ok   a model row still satisfies kat
ok   a plain row satisfies its own phase

Mutant (back to ^(…|model ) with placement grouped): FAIL a npuserve row does not satisfy a requested npu phase and FAIL a model row does not satisfy a requested placement phase, rc 1. Both of my #2523 findings now have an arm each, plus npu/npuserve, which I missed.

c08d0b06 sets infra=1 on the ops zero-cases SKIP, which is the ORCH's item, and scripts/hw-gate.sh:840/855 is what makes it burn: INCOMPLETE (an input was absent: nothing was tested) and exit 2, not green.

What does not hold: the teardown claim

The body says "The test's teardown signalled one pid; the backend is a child, so it survived and held the port." Two measurements disagree:

  1. app/serve.cpp:605 — the forked backend gets prctl(PR_SET_PDEATHSIG, SIGTERM), with the comment "The kernel ends the child when 1bit serve dies for any reason, SIGKILL included, so a backend is never left running on its own." An ordinary backend therefore cannot outlive serve on Linux, whatever the teardown signals.
  2. The mutation agrees: reverting stop() to kill "$pid"; wait "$pid" (keeping everything else) gives MUTANT rc=0 — ok the serve's process group is gone still passes. So the new arm cannot fail for the stated reason.

And the orphan that can survive is the one #2534 measured — the setsid'd impostor — which is in its own process group, so kill -- -"$pgid" cannot reach it either; die_with in tests/fake_backend.py is what kills that one.

What I would change (either is enough): say what the arm actually guarantees — "nothing of this run's scratch tree is still in serve's process group", a guard against a future edit that drops setsid or PDEATHSIG — and drop fixes #2537 from the body, because the port failure is fixed by #2541's relaunch and #2534's die_with, not here. Keeping the group kill is right; only the attribution is wrong, and a wrong attribution sends the next person looking in the wrong place.

Two smaller things

  • infra=1 is assigned twice in the ops SKIP branch (scripts/hw-gate.sh:595 and again at :599), each with its own three-line comment saying the same thing. One of them is dead; the duplicated rationale is what a future reader will trust.
  • Two open PRs both say Fixes #2537: #2541 (relaunch on a taken port, 064c633) and this one. Whichever merges second needs Refs #2537, and this one should say Refs #2523 too — it carries 9dd6077b…21d99f7b, which is #2523's branch. #2523 is superseded by this PR and should be closed, otherwise the merger has two heads for one change: #2523's 9dd6077 and this PR's 21d99f7b, and only the latter has the two fixes.

Net: the hw-gate half of this PR is the good one and it is ready; the teardown half is a correct guard with an incorrect story, and the PR's scope needs one line in the title or body so the merger knows #2523 dies with it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my own review, one line of it: #2523 is not "superseded and should be closed" — it is already MERGED (21d99f7b), and current main already carries the field-match fix (grep -c 'model)[[:space:]]' on origin/main:scripts/hw-gate.sh → 1). I wrote that from the sweep, which still listed #2523 as needing a verdict, without checking gh pr view 2523 --json state first. My mistake, and it changes what this PR should be.

What it means for this branch

Main has moved past 21d99f7b, so the three hw-gate commits here are already merged content. Two-dot against current main:

tests/hybrid_route.sh                       | 18 +++--     ← the only real change
(and 17 more files that are main's own newer state, because this branch is behind)

So this PR is behind main and should be rebased: after that it reduces to the one file it is named for. As it stands the diff would try to revert registry/census.json, registry/flm_answers.json, scripts/coder-llm-ticketed, tests/smoke_serve.sh, docs/nvme-iommu-fault-triage.md and the hw-gate golden files back to their pre-#2528/#2533/#2534/#2538/#2541 state — which is exactly the "two dots vs the merge" trap: the branch looks small in its own log and is a large reverse-diff against today's main.

What still stands from my review: the hw-gate arms pass and their mutation fails by name (they are on main now, so that is settled), and the teardown claim is still misattributed — app/serve.cpp:605's PR_SET_PDEATHSIG means an ordinary backend cannot outlive serve, and my pid-only mutant passes the new arm (MUTANT rc=0). The orphan that survives is the setsid'd impostor, in its own group, which the group kill cannot reach either.

So the ask narrows to: rebase onto main (the branch is 18 files behind and would revert merged work), and fix the one comment + drop fixes #2537 from the body. The group kill itself is right.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 13:34
@bong-water-water-bong
bong-water-water-bong merged commit 67c1f87 into main Nov 15, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix/hybrid-route-pgid branch November 15, 2026 13:39
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI verification of the arm: build passed in 14m55s (run 43718745913), and that job runs ctest including test-hybrid_route — so the group-kill teardown and the new process-group arm both executed on a real engine build, not just against my stand-in. That was the one gap I listed as unverified.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Withdrawing this PR — #2541 measures the actual cause and my diagnosis was wrong.

I said the port was "derived, not searched" and that stop() left an orphaned fake_backend.py holding it. Both were inferences from a log tail, and #2541 instrumented stop() to record serve's children before the kill and found no orphan and no derived port. The real mechanism is the one app/serve.cpp documents at port_owner() (line 412):

// free_port() releases its port before the child binds it, so another local process could
// bind it first and answer the readiness probe; serve must not send it client requests.

Under ctest -j with 340 rows, another test gets handed that ephemeral port in the free_port() → child-binds window. So the fix belongs in serve (detect the child's EADDRINUSE, relaunch on a fresh port, name it) and not in this test's teardown. #2541 has that.

What I got right is only a hazard, not this bug: signalling a process group instead of the pid is still correct test hygiene — fake_backend.py is a child and would survive kill "$pid" if it ever had reason to. It is not what failed here, and a fixture change that claims to fix #2537 while the real fix lives in serve is worse than no change: it would close the issue over a green test that can still collide.

So: closing this with the branch deleted, and handing the group-teardown note to #2541 as a follow-up rather than a competing patch. Same file, one owner, one mechanism.

Thanks to the reviewer who flagged the duplicate Fixes #2537 — that is what made me go and check instead of assuming my reading of the log was the whole story.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to my READY: I praised a mechanism that measurement has since disproved

My verdict said: "I was the lane that called this a flake, and this PR proves me wrong… The mechanism is exactly the one I could not imagine: the teardown signalled one pid, the backend is a child, so it survived and held the port."

That mechanism is not the one. #2537 has been reopened with the correction:

"Reopened: #2543 merged and closed this, but it fixed a different hazard than the one I described here. My diagnosis in the body above is wrong… engine#2541 instrumented stop() to record serve's children before the kill and measured no orphan and no derived port."

The real mechanism is a TOCTOU in serve itself, documented beside port_owner() (app/serve.cpp:412): free_port() releases the port before the child binds it, so under ctest -j another row can be handed that ephemeral port inside the window — and fake_backend.py's 0.5 s sleep is what makes the window wide. The fix belongs in serve (#2541), not in teardown.

What I got wrong and how: I had just written, on #2523, "unless someone shows a mechanism by which a summary loop can stop a server reporting its port, this is unrelated/flaky". Then a plausible mechanism arrived, and I accepted it on plausibility — without asking the one question that separates the two candidates: does the orphan exist after the kill? #2541 answered that with an instrument (record the children before the kill, look after it) and found none. One line of measurement would have moved my verdict from "READY, this proves me wrong" to "READY, but it is not #2537's mechanism".

This is the same fault I logged three times tonight — reviewing the argument's shape, not its contract — and it is the worst instance, because I was actively congratulating myself for having been corrected while accepting an unverified cause.

What stands: #2543's change is real and worth keeping (1bit serve signalled as a process group; a kill switch should own a tree), and it does close a genuine hazard. It just does not close #2537.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test-hybrid_route: a derived backend port collides with the previous section's orphaned fake_backend (EADDRINUSE reported as 'did not become ready')

1 participant