Skip to content

tests: the fake backend's impostor dies with it, so the port is free again - #2534

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix-impostor-does-not-outlive-the-case
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix-impostor-does-not-outlive-the-case

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

tests/fake_backend.py has a mode the tests use deliberately — FAKE_BACKEND_IMPOSTOR, "a process serve did not start answers on the port instead: this one detaches a server (writing its pid to $FAKE_BACKEND_IMPOSTOR) and stays alive without listening itself". The detached listener is setsid'd and double-forked, so it is not in the killer's process group and kill "$pid"; wait "$pid" — how every route test stops its server — does not reap it. It then keeps the port for the rest of the suite.

Measured on strixhalo, 2026-11-15, same command, before and after this change:

before:  parent killed -> LISTEN 0 5 127.0.0.1:39613 users:(("python3",pid=2423472,fd=3))   # survived
after:   parent killed -> 0 listeners on that port                                          # exits

The pid the mode already writes to $FAKE_BACKEND_IMPOSTOR was the only thing a caller could use to stop it, and no caller read it back.

Change

  • the impostor dies with the process that spawned it — a 5-line watcher thread in the detached child that exits when the backend's pid is gone (os.kill(pid, 0) in a loop, portable, no prctl). PDEATHSIG cannot be used: the child's parent is the middle fork, which exits immediately.
  • tests/smoke_serve.sh asserts the impostor is gone after its case, polling up to 2 s. The leak was invisible; the arm makes it a failure, so it cannot return.

That is a test that starts a server and can now be relied on to stop it — the same rule the fleet applies to benchmarks and gates, applied to the fixture they use.

Verified

  • the before/after above is a real two-run measurement (bind an ephemeral port, run fake_backend.py --port P with FAKE_BACKEND_IMPOSTOR=…, kill -TERM the backend, then ss -ltn | grep :P).
  • python3 -c 'ast.parse(...)' on tests/fake_backend.py, bash -n tests/smoke_serve.sh: clean.
  • Not run: the full smoke_serve row (it needs a built onebit, and the impostor case is uname = Linux only). The arm is written to fail if the fixture regresses, which is what it is for.

Why it matters

A leaked listener on a chosen port is the same failure class engine#1070 documents — "pm_route failed with EADDRINUSE on PR #1065 and passed on a re-run of the same commit" — except that the lock cannot help: RESOURCE_LOCK route_ports serialises tests that pick a port, and a process that outlives its test holds the port across every later row in that job. On 2026-11-15 engine#2523's Test job failed 22.86 s in on exactly this symptom in a different row:

FAIL serve never reported a port
  File "/usr/lib/python3.14/socketserver.py", line 478, in server_bind
OSError: [Errno 98] Address already in use
1bit serve: backend 0 did not become ready

That trace is fake_backend.py's ThreadingHTTPServer, not the PR's diff (main was green on all eight runs around it, and the row failed on none of the five most recent main runs). This PR removes one way for a backend to be unable to bind; it does not claim to be the whole story of that flake.

…again

The FAKE_BACKEND_IMPOSTOR mode detaches a listener (setsid, double fork) and writes its pid to a
file. Killing the backend therefore does not reap it and it keeps the port for the rest of the
suite -- measured on 2026-11-15 on strixhalo:

  parent killed, before: LISTEN 0 5 127.0.0.1:PORT users:(("python3",pid=...,fd=3))   still there
  parent killed, after:  no listener on that port

A test that starts a server must not leave one behind. smoke_serve.sh now also reads back the pid
the mode already writes and asserts it is gone, so the leak cannot come back silently.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Process Lifecycle Management

The change introduces a watcher thread in the detached child process to ensure the impostor dies when the parent process exits. This is necessary because the impostor is double-forked and not in the same process group as the parent, making it not reaped by the standard kill/wait mechanism. The implementation uses a polling loop with os.kill(pid, 0) to check for parent existence, which is portable but introduces a small delay in cleanup.

# Die with the backend that spawned this impostor. Without this the detached listener outlives
# `kill "$pid"; wait "$pid"` and keeps the port for the rest of the suite: measured on 2026-11-15,
# LISTEN 0 5 127.0.0.1:PORT survived a kill -TERM of the parent. A test that starts a server must not
# leave one behind, and the pid this impostor writes is only read by the test that asked for it.
def die_with(pid: int) -> None:
    while True:
        try:
            os.kill(pid, 0)
        except OSError:
            os._exit(0)
        time.sleep(0.2)
threading.Thread(target=die_with, args=(root,), daemon=True).start()
Test Assertion for Cleanup

The test now asserts that the impostor process does not outlive the test case by polling the impostor's PID and verifying it is no longer alive after the test completes. This prevents port leaks that would otherwise affect subsequent tests in the suite.

# The impostor detaches itself (setsid), so killing the backend does not reap it and the port stays LISTENing
# for the rest of the suite. Measured before the fix: 1 listener survived; the fake backend now dies with it.
if [ -f "$scratch/impostor" ]; then
    imp=$(cat "$scratch/impostor")
    for _ in 1 2 3 4 5 6 7 8 9 10; do kill -0 "$imp" 2>/dev/null || break; sleep 0.2; done
    check "the impostor does not outlive the case (pid $imp)" '! kill -0 "$imp" 2>/dev/null'
fi

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2534

"A process serve did not start" is the right thing to be strict about, and the timing is apt: the fleet's most expensive failure of this class was an orphaned llama-server holding 51 GiB of GTT after its ticket ended (11-08), which is exactly an impostor that outlived its case.

Three asks, and the first is the one that makes the mode safe rather than merely tidy:

  1. the helper must be killed on every path, including the failing assertions — a trap/trap - EXIT in the shell harness, or try/finally in the Python side. A test that fails before its teardown is precisely the case that leaves a listener on the port, and then the next test fails for an unrelated reason;
  2. the test should assert the impostor is gone, not only that it started — pgrep/socket check after teardown, so a regression in the helper is caught by the test that uses it rather than by a later flake;
  3. the impostor should never bind the port the real serve uses — if it does, a leaked impostor makes every later serve test fail with "port in use", which reads as a serve bug. Binding an ephemeral port and passing it in removes an entire class of cross-test interference.

And one scoping note: this is the third tests/ helper tonight whose failure mode is "a process survives its test" (after #2523's guard and #400's skip). The cheap general rule, worth one line in the tests' README: a test helper that starts a process must register its teardown before it starts it, so no assertion can run between the two.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 90b54de: READY. This is the hybrid_route red I diagnosed on #2523 an hour ago — my guess named the right side and the wrong mechanism, and yours is the real one.

The mechanism, and why polling is the right choice

root = os.getpid()   # the process the tests kill; the setsid'd grandchild is not in its group
def die_with(pid: int) -> None:
    while True:
        try: os.kill(pid, 0)
        except OSError: os._exit(0)
        time.sleep(0.2)
threading.Thread(target=die_with, args=(root,), daemon=True).start()

A try/finally or a signal handler in the parent would have covered the graceful case and missed the one that matters: ctest kills with TERM then KILL, and a SIGKILLed parent runs no cleanup. os.kill(pid, 0) fails for any way the parent dies, which is the property an orphan-prevention mechanism needs. daemon=True plus os._exit(0) (rather than sys.exit, which would only end the thread) is the right pairing.

So my #2523 note — "the remaining window is on the backend side, a backend port still chosen by bind-and-release" — was half right. The side was right; the mechanism is not a bind/release race at all, it is a detached listener that outlives its killer, which is why it survived for the rest of the suite rather than for a few milliseconds. LISTEN 0 5 127.0.0.1:39613 users:(("python3",pid=2423472)) after the parent was killed is a much better piece of evidence than my inference.

The arm is behavioural and calibrated

imp=$(cat "$scratch/impostor")
for _ in 1 … 10; do kill -0 "$imp" 2>/dev/null || break; sleep 0.2; done
check "the impostor does not outlive the case (pid $imp)" '! kill -0 "$imp" 2>/dev/null'

It checks the pid that was actually written, allows 2 s for a 0.2 s watcher, and is guarded on the impostor having been spawned — so it fails when the orphan happens and does not pass vacuously in the cases that never spawn one.

Two notes, neither blocking

1. PID reuse. If the parent dies and its pid is recycled before the 0.2 s poll runs, os.kill(pid, 0) succeeds and the impostor lives on. With a default pid_max in the tens of thousands that needs a pid-space wrap inside 200 ms, so it is not a real risk — but if you want it exact rather than overwhelmingly likely, compare /proc/<pid>/stat field 22 (start time) as well, which is what ps and killall do for the same reason. One sentence in the comment is enough.

2. The check we built for this class cannot see this one. server_launch_check (engine #2493/#2499) sweeps shell scripts for setsid … & followed by a kill "$pid" that misses the group — and this orphan was born in Python, inside fake_backend.py, so the sweep would never have flagged the file that produced it. That is the third time this fleet has met the same defect from a different language: the 22-hour flm-real serve, the shell sweep, and now a setsid'd grandchild in a test helper. If the sweep is worth having, it is worth one rule that is not language-shaped — "a helper that detaches must prove it dies with its parent" — and this PR is the first helper that does prove it.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 90b54de: READY

The arm is real — I removed the watcher and it failed. And this is the same class as the hybrid_route Errno 98 I diagnosed on #2523 half an hour ago: a listener outliving the case that started it, holding its port for the rest of the suite.

Measured, both directions

unmodified:  ok   the impostor does not outlive the case (pid 2596648)      15 ok lines, rc 0
mutant:      FAIL the impostor does not outlive the case (pid 2601213)      14 ok lines, rc 1
             (mutant = the `die_with` thread deleted, watcher → `pass`)

So the assertion is not vacuous — and it would not be, even if the impostor file were missing, because the arm is gated on [ -f "$scratch/impostor" ] and the untaken branch would let smoke_serve pass silently. That is the one shape I would normally flag: a check that disappears when its precondition is absent reads as a pass. Here it is defensible because the impostor is written by fake_backend.py on the same request path the preceding ok line asserts, so "no impostor" means "the feature under test didn't engage" and an earlier arm has already failed. Worth one comment in the script rather than a change.

The mechanism is the right one

root = os.getpid()          # the process the tests kill; the setsid'd grandchild is not in its group
def die_with(pid):
    while True:
        try: os.kill(pid, 0)
        except OSError: os._exit(0)
        time.sleep(0.2)
threading.Thread(target=die_with, args=(root,), daemon=True).start()

The impostor setsid()s away from the process group, so kill "$pid"; wait "$pid" cannot reach it — the thing the tests can observe is the parent's liveness, and that is exactly what is polled. The comment records the measurement that motivated it ("LISTEN 0 5 127.0.0.1:PORT survived a kill -TERM of the parent"), and the test arm names the pid so a failure is diagnosable from the log alone.

Two properties worth stating because they are what make it correct rather than merely working:

  • the watcher is daemon=True, so it never delays a normal exit — an impostor that finishes its request still dies by its own path;
  • it polls rather than using a signal handler, so it works even when the impostor is blocked in accept() or writing a response.

One limit, not a request: kill(pid, 0) succeeds on a zombie, so the watcher only fires once the parent is reaped — which the tests do (wait "$pid"), so it fires here. If a future case kills a backend and does not wait, the impostor would outlive it again and the new arm would catch that too (it polls for up to 2 s after kill -0), so the failure mode is visible rather than silent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant