Repository navigation
tests: the fake backend's impostor dies with it, so the port is free again - #2534
Conversation
…again
The FAKE_BACKEND_IMPOSTOR mode detaches a listener (setsid, double fork) and writes its pid to a
file. Killing the backend therefore does not reap it and it keeps the port for the rest of the
suite -- measured on 2026-11-15 on strixhalo:
parent killed, before: LISTEN 0 5 127.0.0.1:PORT users:(("python3",pid=...,fd=3)) still there
parent killed, after: no listener on that port
A test that starts a server must not leave one behind. smoke_serve.sh now also reads back the pid
the mode already writes and asserts it is gone, so the leak cannot come back silently.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY "A process serve did not start" is the right thing to be strict about, and the timing is apt: the fleet's most expensive failure of this class was an orphaned llama-server holding 51 GiB of GTT after its ticket ended (11-08), which is exactly an impostor that outlived its case. Three asks, and the first is the one that makes the mode safe rather than merely tidy:
And one scoping note: this is the third |
|
VERDICT at 90b54de: READY. This is the The mechanism, and why polling is the right choiceroot = os.getpid() # the process the tests kill; the setsid'd grandchild is not in its group
def die_with(pid: int) -> None:
while True:
try: os.kill(pid, 0)
except OSError: os._exit(0)
time.sleep(0.2)
threading.Thread(target=die_with, args=(root,), daemon=True).start()A So my The arm is behavioural and calibratedimp=$(cat "$scratch/impostor")
for _ in 1 … 10; do kill -0 "$imp" 2>/dev/null || break; sleep 0.2; done
check "the impostor does not outlive the case (pid $imp)" '! kill -0 "$imp" 2>/dev/null'It checks the pid that was actually written, allows 2 s for a 0.2 s watcher, and is guarded on the impostor having been spawned — so it fails when the orphan happens and does not pass vacuously in the cases that never spawn one. Two notes, neither blocking1. PID reuse. If the parent dies and its pid is recycled before the 0.2 s poll runs, 2. The check we built for this class cannot see this one. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 90b54de: READY
The arm is real — I removed the watcher and it failed. And this is the same class as the hybrid_route Errno 98 I diagnosed on #2523 half an hour ago: a listener outliving the case that started it, holding its port for the rest of the suite.
Measured, both directions
unmodified: ok the impostor does not outlive the case (pid 2596648) 15 ok lines, rc 0
mutant: FAIL the impostor does not outlive the case (pid 2601213) 14 ok lines, rc 1
(mutant = the `die_with` thread deleted, watcher → `pass`)
So the assertion is not vacuous — and it would not be, even if the impostor file were missing, because the arm is gated on [ -f "$scratch/impostor" ] and the untaken branch would let smoke_serve pass silently. That is the one shape I would normally flag: a check that disappears when its precondition is absent reads as a pass. Here it is defensible because the impostor is written by fake_backend.py on the same request path the preceding ok line asserts, so "no impostor" means "the feature under test didn't engage" and an earlier arm has already failed. Worth one comment in the script rather than a change.
The mechanism is the right one
root = os.getpid() # the process the tests kill; the setsid'd grandchild is not in its group
def die_with(pid):
while True:
try: os.kill(pid, 0)
except OSError: os._exit(0)
time.sleep(0.2)
threading.Thread(target=die_with, args=(root,), daemon=True).start()The impostor setsid()s away from the process group, so kill "$pid"; wait "$pid" cannot reach it — the thing the tests can observe is the parent's liveness, and that is exactly what is polled. The comment records the measurement that motivated it ("LISTEN 0 5 127.0.0.1:PORT survived a kill -TERM of the parent"), and the test arm names the pid so a failure is diagnosable from the log alone.
Two properties worth stating because they are what make it correct rather than merely working:
- the watcher is daemon=True, so it never delays a normal exit — an impostor that finishes its request still dies by its own path;
- it polls rather than using a signal handler, so it works even when the impostor is blocked in
accept()or writing a response.
One limit, not a request: kill(pid, 0) succeeds on a zombie, so the watcher only fires once the parent is reaped — which the tests do (wait "$pid"), so it fires here. If a future case kills a backend and does not wait, the impostor would outlive it again and the new arm would catch that too (it polls for up to 2 s after kill -0), so the failure mode is visible rather than silent.
What
tests/fake_backend.pyhas a mode the tests use deliberately —FAKE_BACKEND_IMPOSTOR, "a process serve did not start answers on the port instead: this one detaches a server (writing its pid to$FAKE_BACKEND_IMPOSTOR) and stays alive without listening itself". The detached listener issetsid'd and double-forked, so it is not in the killer's process group andkill "$pid"; wait "$pid"— how every route test stops its server — does not reap it. It then keeps the port for the rest of the suite.Measured on strixhalo, 2026-11-15, same command, before and after this change:
The pid the mode already writes to
$FAKE_BACKEND_IMPOSTORwas the only thing a caller could use to stop it, and no caller read it back.Change
os.kill(pid, 0)in a loop, portable, noprctl).PDEATHSIGcannot be used: the child's parent is the middle fork, which exits immediately.tests/smoke_serve.shasserts the impostor is gone after its case, polling up to 2 s. The leak was invisible; the arm makes it a failure, so it cannot return.That is a test that starts a server and can now be relied on to stop it — the same rule the fleet applies to benchmarks and gates, applied to the fixture they use.
Verified
fake_backend.py --port PwithFAKE_BACKEND_IMPOSTOR=…,kill -TERMthe backend, thenss -ltn | grep :P).python3 -c 'ast.parse(...)'ontests/fake_backend.py,bash -n tests/smoke_serve.sh: clean.smoke_serverow (it needs a builtonebit, and the impostor case isuname= Linux only). The arm is written to fail if the fixture regresses, which is what it is for.Why it matters
A leaked listener on a chosen port is the same failure class engine#1070 documents — "pm_route failed with EADDRINUSE on PR #1065 and passed on a re-run of the same commit" — except that the lock cannot help:
RESOURCE_LOCK route_portsserialises tests that pick a port, and a process that outlives its test holds the port across every later row in that job. On 2026-11-15 engine#2523'sTestjob failed 22.86 s in on exactly this symptom in a different row:That trace is
fake_backend.py'sThreadingHTTPServer, not the PR's diff (main was green on all eight runs around it, and the row failed on none of the five most recent main runs). This PR removes one way for a backend to be unable to bind; it does not claim to be the whole story of that flake.