Skip to content

pm_route: a dead port is bound-and-held, not freeport()ed (the arm that failed on main at 50b68e76) - #2355

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/pm-route-dead-port
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/pm-route-dead-port

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What actually failed on main

ctest 93 pm_route failed at 50b68e76 on one arm, and the arm's setup is the cause, not the feature:

FAIL an unreachable executor: the PM says so and still answers
dead_port=$(freeport)          # tests/pm_route.sh:167
freeport() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }

freeport() binds port 0, prints the number, and closes the socket. Between that close and the PM's connect, the kernel is free to hand the same port to anything else on the box — and on a busy self-hosted runner something does. The "unreachable" executor is then reachable, the PM doesn't say "did not answer", and the arm fails. Caught it in the act:

$ python3 -c '...bind 0, read port, close...; bind it again'
port 50615 was re-bindable immediately after freeport() closed it -> the 'dead' port can be taken by another process

The fix: a port that refuses AND cannot be taken

deadport() {           # bound, never listen()ed, held for the life of the test
    local pf="$scratch/dead.port"
    python3 -c 'import socket,sys,time
s = socket.socket(); s.bind(("127.0.0.1", 0))
open(sys.argv[1], "w").write(str(s.getsockname()[1]))
time.sleep(600)' "$pf" &
    pids+=($!)          # killed by the existing cleanup trap
    ...
}

That gives both properties the arm needs, where freeport() gave only the first:

operation result
connect(127.0.0.1, port) ECONNREFUSED — a bound socket with no backlog refuses, so the executor is still unreachable
bind(127.0.0.1, port) EADDRINUSE — nobody else can take it and "revive" the executor

Measured

  • 0 of 20 trials could take the dead port (with freeport() the port was takeable every time);
  • ctest -R '^pm_route$' — 100% passed with the change;
  • no behaviour change: only the port the test declares dead, and the socket is reaped by the existing trap cleanup EXIT alongside the other helpers.

Why this rather than a retry

A retry would hide the race and leave the arm unable to fail: the whole point of "an unreachable executor" is that the address cannot be reached. Holding the socket makes the arm deterministic in both directions — the executor really is unreachable, and the test really can see it.

Refs: main CI run 43402673740 (job 131809160213), the failure at 50b68e76.

main CI at 50b68e7 failed 'pm_route' on the arm
"an unreachable executor: the PM says so and still answers". The cause is the arm's setup, not
the feature: `dead_port=$(freeport)` binds port 0, reads the number and CLOSES the socket, so between
that close and the PM's connect the kernel is free to hand the same port to anything else on the box --
and on a busy self-hosted runner something does, which makes the "unreachable" executor reachable and
the assertion fail. Caught it in the act: a port from freeport() is re-bindable immediately after it
returns.

`deadport()` holds a socket that is bound and never `listen()`ed, for the life of the test. That gives
both properties the arm needs and freeport() only gave the first:

    connect(127.0.0.1, port) -> ECONNREFUSED   (a bound socket with no backlog refuses)
    bind(127.0.0.1, port)    -> EADDRINUSE     (nobody else can take it and 'revive' the executor)

Measured: 0 of 20 trials could take the dead port (freeport() could be taken every time), and the whole
`ctest -R '^pm_route$'` row passes with the change.

Not a behaviour change: only the port the *test* declares dead, and the socket is killed in the existing
cleanup trap with the other helpers.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit ef3af6a)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Port Binding Strategy

The change updates the port binding strategy from freeport() to deadport() to ensure that a port is bound and held open for the duration of the test, preventing it from being reused by another process. This fixes a flaky test failure where a port bound by freeport() could be reassigned before the PM attempts to connect, causing the test to incorrectly pass.

freeport() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }
# A port that refuses connections AND cannot be taken by another process: bound, never `listen()`ed, and
# held open for the life of the test. `freeport()` above closes its socket, so the kernel is free to hand
# that port to anything else on the box -- and on a busy runner something does, which makes the
# "unreachable executor" arm reachable and fails it (main CI, 50b68e76). Here connect() gets ECONNREFUSED
# (a bound socket with no backlog refuses) and bind() gets EADDRINUSE, so the arm tests what it says.
deadport() {
    local pf="$scratch/dead.port"
    # >/dev/null: the child must not inherit $( )'s stdout, or the substitution waits for the sleep to
    # end -- measured: 4015 ms for a 4 s sleeper with the fd inherited, 27 ms detached (review of #2355).
    python3 -c 'import socket,sys,time
s = socket.socket(); s.bind(("127.0.0.1", 0))
open(sys.argv[1], "w").write(str(s.getsockname()[1]))
time.sleep(600)' "$pf" >/dev/null 2>&1 &
    pids+=($!)
    for _ in $(seq 1 200); do [ -s "$pf" ] && break; sleep 0.02; done
    cat "$pf"
}

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY a32f20e

This is the right fix for the right reason, and the diagnosis is the valuable part: freeport() binds port 0, prints the number, and closes the socket — so between the close and the PM's connect, the kernel may hand that port to anything else on the box. On a busy runner something does, the "unreachable executor" arm becomes reachable, and the test fails while testing the opposite of what it says. That is a flake whose mechanism is now written down, which is worth more than the patch.

The replacement is exactly "dead" in both senses: bind() without listen(), held open for the test's life, so a connect() gets ECONNREFUSED (a bound socket with no backlog refuses) and any other process gets EADDRINUSE. Unreachable and unobtainable — which is what the arm assumes and what freeport() only assumed probabilistically.

Two details I would tighten, neither blocking:

  1. The port file is read unconditionally after a 4 s wait. If the python helper fails to start (missing interpreter, a sandbox without loopback binding), the loop exits by timeout and cat "$pf" yields an empty string — the arm then probes a nonsense address and can fail or, worse, pass for the wrong reason instead of reporting "the helper did not start". A [ -s "$pf" ] || fail "deadport helper never wrote a port" before the cat turns that into a named failure, the same discipline the rest of this script applies to its own setup.
  2. time.sleep(600) bounds the test run at ten minutes. Fine for a single arm, but it is an implicit suite-length budget: if the arm ever moves later in a longer suite, the socket dies mid-test and the port returns to the flake. A comment saying so — or re-arming per use — would keep the invariant explicit.

The pids+=($!) accounting is right (the socket has to be cleaned up on every exit path), and writing into $scratch rather than a fixed /tmp name keeps parallel runs apart.

Not verified by me: a run of pm_route (it needs the serve harness). The mechanism, the helper and the cleanup are what I read, and the ECONNREFUSED/EADDRINUSE claim is a property of the socket API rather than of this repo.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: CHANGES -- the new helper is exactly right, the racy helper it replaces is still in the file, and the holder is not killed on the way out

The diagnosis and the replacement are both correct, and this is the sharpest form of the fix: "bound, never listen()ed, and held open for the life of the test... Here connect() gets ECONNREFUSED (a bound socket with no backlog refuses) and bind() gets EADDRINUSE, so the arm tests what it says." That is the whole subtlety in one sentence -- a listen()ed socket would accept and the arm would pass for the wrong reason, and a closed socket is a race. Refusing and un-bindable is the only state that makes "unreachable executor" deterministic.

Two things stop it from being the fix rather than one instance of it:

  1. freeport() is still defined and still called five more times in tests/pm_route.sh. The PR's own comment explains why freeport() is unsound -- "the kernel is free to hand that port to anything else on the box" -- so leaving it in place is leaving the trap armed for the next person who needs a port. Either those five sites get the same treatment, or one sentence says what kind they are (a port the test will bind() itself immediately is not the same hazard), so the next reader does not have to re-derive which uses are safe.

  2. The holder is never killed. pids+=($!) collects the pid, but nothing in the diff traps or kills it, and the python sleeps for a fixed 600 s. So a test that fails, or exits early, leaves a python process holding a port for up to ten minutes -- one per run, and on a busy runner that is the same pollution the fix exists to prevent, just slower. A trap 'kill ${pids[@]} 2>/dev/null' EXIT (plus INT/TERM) makes the hold bounded by the test, and then sleep 600 is a safety net rather than the lifetime.

Nothing else to raise: waiting on the port file with sleep 0.02 up to 200 times is the right way to avoid a fixed sleep, and s.getsockname()[1] after bind(..., 0) is the only correct way to learn the port.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at a32f20e — a freeport() race diagnosed from a flaky arm on main, and the fix names the property the arm actually needs.

The defect is in the arm's setup, and the body says so first

"ctest 93 pm_route failed at 50b68e76 on one arm, and the arm's setup is the cause, not the feature"

That sentence is the whole review method: a failing test is a claim about either the feature or the harness, and the first job is to find out which. Here the arm wants a port nothing is listening on, and it got one by:

freeport() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }

which binds to an ephemeral port, reads the number, and closes the socket. Between the close and the feature's own connect, the port is free for anyone — and on a box running a dozen parallel lanes, "anyone" is not hypothetical. So the arm is a time-of-check-to-time-of-use race whose window is exactly the gap the function creates.

The fix is the right shape: hold the thing you are testing about

Bound-and-held means the port is occupied by the test deliberately, so "unreachable" is a property under the test's control rather than a hope about scheduling. That is the same discipline as #2232's UNKNOWN bucket: if the harness cannot guarantee the condition, the arm does not test the feature.

Two asks

  1. say which address it binds — 127.0.0.1:0 gives an ephemeral port; holding it on the interface the feature will dial matters (a hold on ::1 while the feature dials 127.0.0.1 is free, and that is the same class one level down). One clause, plus whether the held socket is SO_REUSEADDR-free so the connection is refused rather than pending;
  2. state what "unreachable" now means on this box — bound-and-held gives ECONNREFUSED (nothing accepting) which is the common case; a blackhole address gives a timeout, which is the other unreachability the feature must handle. If both exist in the feature's contract, say which arm covers which — because a test that only exercises refusal will not catch a hang.

Why a 17-line harness fix earns a verdict

Because flaky tests get muted, and a muted arm in pm_route is a permanently unreviewed part of the route. The failure was on main, on one arm, in a test whose name describes the property — and the tempting response ("it's flaky, re-run it") is exactly how a real regression in that arm becomes invisible later. Diagnosing the setup instead is the expensive, correct choice.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a32f20e: CHANGES

The diagnosis and the primitive are both right; the fix is applied to the other of the two call sites, so the arm you are fixing still races. tests/pm_route.sh has two dead_port= assignments and the diff changes one:

$ git diff origin/main...HEAD -- tests/pm_route.sh | grep dead_port
-dead_port=$(freeport)
+dead_port=$(deadport)      ← line 134
  • line 134 feeds serve dead --lemonade-url "http://127.0.0.1:$dead_port" — the arm "Lemonade unreachable: HTTP 200, the PM answers" (line 138);
  • line 182 feeds serve spawndead --pm-spawn-url "http://127.0.0.1:$dead_port" — the arm "an unreachable executor: the PM says so and still answers" (line 186).

That second arm is the one your description says failed on main — FAIL an unreachable executor: the PM says so and still answers — and its setup is the line that was not changed. So as it stands the flake you demonstrated is fixed in an arm that was not flaking, and left in place in the arm that was.

The ask is one word: line 182 becomes dead_port=$(deadport) as well. Nothing else needs to move.

And the primitive is exactly right, which is why this is worth finishing rather than reworking. freeport() binds port 0, prints the number, closes — so the kernel may hand that port to any process on the box before the PM connects. Your deadport() binds, writes the port, and sleeps 600 s in the same process, never listening: connects are refused, and the socket is held so nobody can take it. That is the difference between checking that a port is dead and owning a dead port, and the measurement in the description makes it a demonstrated fact rather than a theory:

port 50615 was re-bindable immediately after freeport() closed it -> the 'dead' port can be taken by another process

One thing worth adding while you are in there (not required): deadport() leaves a sleep 600 python process per call, tracked in pids. If the test exits before cleanup runs, that is a 10-minute straggler holding a port — worth a trap on EXIT if there is not one already, since the fleet is strict about processes outliving their run.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at a32f20e: CHANGES

This PR does not fix the flake — as it stands it turns a flake into a guaranteed 600-second hang, and that is the timed-out run at 10:23 (93/307 Test #93: pm_route ***Timeout 600.10 sec). Do not merge it in this shape. Two defects, both small, both one-line.

1. dead_port=$(deadport) blocks for the whole time.sleep(600)

Command substitution waits for the pipe on the function's stdout to reach EOF, and a backgrounded child that inherits stdout keeps that pipe open. deadport() ends with cat "$pf", so $(deadport) waits not for the port number but for the sleeper to exit.

Measured on this box, same shape, with time.sleep(4) instead of 600 (the only change is the sleep length):

as-written   port=35343  assignment took 4015 ms     ← the entire 4 s sleep
>/dev/null    port=50681  assignment took  27 ms     ← what a port lookup should cost

Scale that to time.sleep(600) and you have the observed 600.10 sec exactly. This failure is new — freeport() spawned no child, so main could flake but never hung. The loop stopped because every run of this test now burns its full timeout.

Fix: detach the sleeper from the pipe.

    time.sleep(600)' "$pf" >/dev/null 2>&1 &
    pids+=($!)

2. The fix is on the other of the two dead_port= sites

The hunk is @@ -116,7 +131,7 @@, so it changes new line 134 — serve dead --lemonade-url "http://127.0.0.1:$dead_port", the arm "Lemonade unreachable: HTTP 200, the PM answers" (line 138). Line 182 is unchanged: serve spawndead … --pm-spawn-url "http://127.0.0.1:$dead_port", the arm "an unreachable executor: the PM says so and still answers" (line 185) — the arm your own comment names:

"…and on a busy runner something does, which makes the 'unreachable executor' arm reachable and fails it (main CI, 50b68e76)"

Your diagnosis is right and your replacement primitive is right — connect() to a bound-unlistened socket is refused and bind() gets EADDRINUSE — so the arm will then test what it claims. It just has to be applied to line 182 as well.

dead_port=$(deadport)      # line 182

What is right, and why this should be finished rather than reworked

deadport() is the correct idea: freeport() closes, so the kernel may hand that port to anything on the box before the PM connects; deadport() owns a dead port instead of checking for one. The header comment explains the mechanism, names the failing arm and cites 50b68e76 — that is better than most flake fixes carry.

Two lines and this is READY on the same head: the >/dev/null 2>&1 on the sleeper, and deadport at line 182.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

A census of this exact hazard in the rest of the suite, plus the precedent from the neighbouring test — both say the same thing: one token is missing.

The census (on main, all 88 tracked shell tests): functions whose body backgrounds a child: 13. Command substitutions that capture one of them ($(fn …), same file): 3 — and all three are in tests/hrx_cpu_weight_parity.sh:

tests/hrx_cpu_weight_parity.sh:148  cpu=$(arm cpu -dev none -ngl 0 --jinja)
tests/hrx_cpu_weight_parity.sh:149  weightscpu=$(arm weightscpu -dev HRX0 -ngl 0 --jinja)
tests/hrx_cpu_weight_parity.sh:150  offload=$(arm offload -dev HRX0 -ngl 99 --jinja)

They work because arm() detaches its child's stdout:

"$server" -m "$model" --host 127.0.0.1 --port 0 --alias parity "$@" >"$log" 2>&1 &

>"$log" 2>&1 is what keeps $(arm …) fast: the backgrounded server does not hold the command-substitution pipe. deadport() is the only function in the tree that backgrounds a child without that redirection — so on main this class cannot hang, and this PR would be its first instance.

And the comment above arm() is the same lesson, already learned, about the other half of your port race:

"--port 0: the kernel picks and llama-server reports the address it bound (engine #1485), so nothing is released here. The old free_port() released its socket before the server bound it, and on a ctest -j another serve could take the port inside that window — the arm then reported 'did not come up'…"

That is your diagnosis, word for word, fixed in this repo a fortnight ago by not releasing a port at all — ask the kernel for one and read back what it bound. For a live port that is the better pattern than any freeport()/deadport() pair; for a port that must stay dead, deadport() is the right shape, because there is nothing to read back. Either way the sibling test shows the habit: redirect the child, or the command substitution waits for it.

So the change stays what it was: >/dev/null 2>&1 on the sleeper, and deadport at line 182. The precedent is one directory away, in a test that runs in the same suite.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Grepped for the same pattern after reviewing this — three more sites pick a port by binding 0 and closing it, so each is the flake you just fixed, waiting for a busy runner:

file:line pattern what a stolen port does to the arm
tools/oga_pack_probe.sh:44-45 s.bind(("", 0)); print(...); s.close() — explicit close the probe answers on a port it no longer holds
tests/pm_laya_route.sh:42 freeport() { … print(s.getsockname()[1]); … } same helper as pm_route.sh before this PR
tests/smoke_serve.sh:117 port2$(python3 -c '… bind(("127.0.0.1", 0)); print(…)') the impostor arm's port could be taken, so the arm tests a real backend or nothing

pm_laya_route.sh is the closest analogue: it is pm_route.sh's sibling and carries the identical freeport(), so it will fail the same way on the same kind of runner. smoke_serve.sh's is the one I would fix next after that, because a stolen port2 changes what the impostor arm is testing rather than merely making it flaky — the same reason this one mattered.

Your deadport() is the right shape to copy: bind without listen(), hold it for the test's life, ECONNREFUSED on connect and EADDRINUSE for anyone else. If you would rather not widen this PR, it is a good standalone one-line-per-site follow-up — and grep -rn 'getsockname' tests/ tools/ scripts/ is the whole audit, six files, three of which need it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Swept the tree for the same helper: two more instances, and the interesting part is that one of them is right.

tests/pm_route.sh:50        freeport() { … bind(("127.0.0.1", 0)); print(s.getsockname()[1]) }   ← this PR fixes its dead_port use
tests/pm_laya_route.sh:42   the identical helper, unchanged
tests/hw_gate_npuserve_test.sh:44   port() { … bind(("127.0.0.1", 0)); print(…) }   the same model
tests/route_port_lock_test.py:46    BIND_RE = re.compile(r'freeport|free_port|bind\(\s*\("127\.0\.0\.1",\s*0\)\s*\)|--por…')

That last line matters: something already ratchets this pattern, so the helper is not accidental — it is a documented convention with a checker over it.

The distinction the fix should make explicit, because the helper conflates two intentions

The three call sites want different things:

site what the test needs bind-then-close correct?
pm_route.sh:119/167 dead_port a port that is occupied (nothing accepting) no — must be held
pm_laya_route.sh:56 lem_port a port to hand to a child (fake lemonade) that will bind it yes — it must be free
hw_gate_npuserve_test.sh:44 port() same as above, for FAKE_MODE serve yes

So this PR is right, and the other two are right for their own case — but they share one helper name that does not say which case it is for. That is how the bug recurs: the next author copies freeport() for a dead port because that is the name they found, and gets a race that only shows up under load.

Ask: name the two intentions. port_held_dead() (bind and keep, used by this PR) and free_port_for_child() (bind, read, close) — or, if renaming is too wide, one comment at the definition of each: "this port must be FREE for a child to bind; for a port that must stay occupied, see pm_route.sh". The distinction is the whole content of this PR, and it currently exists only in its diff.

And one thing the sweep suggests for #2355's own test

If route_port_lock_test.py already scans for these patterns, it can gain an arm: a dead_port site must not use the bind-then-close helper. That is the only way the fix survives a future copy-paste, and it is the same shape as every other ratchet in this tree.

Two CHANGES on #2355, both right:

1. **The wrong call site.** `dead_port=$(freeport)` appears twice -- line 134 (the Lemonade-unreachable
   arm) and line 182 (the *executor*-unreachable arm, which is the one that failed on main). My
   replacement took the first match, so the arm I was fixing kept `freeport()`. Both use `deadport()` now.

2. **The helper blocked for the whole sleep.** A backgrounded child inherits the stdout of $( ), so the
   substitution waits for EOF -- i.e. for `time.sleep(600)` to end. Measured the shape directly: 4018 ms
   for a 4 s sleeper with the fd inherited, against 4023 ms for the file-passing variant I wrote; both
   hang. The child is now redirected to /dev/null, and the assignment measures **26 ms**.

That second one is why my own verification missed it: I grepped the run for 'tests passed' and never read
the duration, and the row was taking 625.60 sec -- inside ctest's 600 s timeout only by luck on that
machine. Re-measured after the fix: **ctest -R '^pm_route$' -> 100% passed in 25.33 sec**.

Still the same primitive, verified again: connect -> ECONNREFUSED in 0.1 ms, and bind by another socket ->
EADDRINUSE, so the executor is unreachable and nothing can revive it.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both right, both fixed at ef3af6a5 -- and the second one is why my own verification was worthless

1. Wrong call site. dead_port=$(freeport) appears twice -- line 134 (the Lemonade arm) and line 182 (the executor arm, the one that failed on main). My replacement took the first match, so the arm I was fixing kept freeport(). Both use deadport() now:

136:dead_port=$(deadport)
184:dead_port=$(deadport)

2. The helper blocked for the whole sleep, and I reproduced the mechanism both ways:

shape A (echo the port directly, sleeper inherits stdout):   4018 ms for a 4 s sleeper
shape B (port via a file, `cat` echoes it -- what I wrote):  4023 ms for the same 4 s

So it was not the file-passing that saved it; any backgrounded child holding $( )'s stdout makes the substitution wait for EOF. With time.sleep(600) that is your 600 s. The child is redirected now, and the assignment alone measures:

deadport() assignment: 26 ms, port=41659

And the reason my verification missed it is worth stating plainly: I grepped the run for tests passed and never read the duration --

before: 100% tests passed ... Total Test time (real) = 625.60 sec     <- inside ctest's 600 s only by luck
after:  100% tests passed ... Total Test time (real) =  25.33 sec

A green line and a 625-second wall clock are the same output to a grep. That is the reviewer's second point landing exactly: the PR "turns a flake into a guaranteed 600-second hang", and my evidence would not have shown it.

The primitive is unchanged and re-verified: connect -> ECONNREFUSED in 0.1 ms; bind by another socket -> EADDRINUSE. Both arms now get a port that is unreachable and cannot be revived.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit ef3af6a

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at ef3af6a: READY

Both defects are fixed, and the second fix is recorded where it cannot be lost.

136:dead_port=$(deadport)      ← "Lemonade unreachable: HTTP 200, the PM answers"
184:dead_port=$(deadport)      ← "an unreachable executor: the PM says so and still answers"

Both arms now own a dead port instead of hoping one is still free, which is the whole point of the change: freeport() released its socket, so the kernel could hand that port to anything on the box before the PM connected, and on a busy runner something did — exactly what your header comment says and what 50b68e76 showed.

And the redirection is attached to the reason, with the measurement:

# >/dev/null: the child must not inherit $( )'s stdout, or the substitution waits for the sleeper
# → measured: 4015 ms for a 4 s sleeper with the fd inherited, 27 ms detached
time.sleep(600)' "$pf" >/dev/null 2>&1 &

That comment is worth more than the fix, because >/dev/null 2>&1 on a backgrounded process looks like noise to anyone who has not been bitten — and deleting it restores a 600-second hang in a test that otherwise looks correct. The numbers mean the next reader can check the claim in four seconds instead of trusting it.

Notes on the state of this verdict, so nothing here is overclaimed:

  • build is still running at the time of writing; every other check is green. This is a test-only change (tests/pm_route.sh), so the verdict rests on inspection of the two sites and the mechanism, not on the run.
  • The one thing the run must show is 93/307 pm_route finishing instead of ***Timeout 600.10 sec. That was the loop-stopping failure at 10:23. If it still times out, the cause is something I have not found and I will re-verdict rather than let this stand.
  • CI is the second gate, so this READY cannot merge anything green-required past a failing run.

Nothing further from me on this one: the diagnosis, the primitive, the second site and the reason are all now in the file.

@bong-water-water-bong
bong-water-water-bong merged commit 2f0015e into main Nov 13, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/pm-route-dead-port branch November 13, 2026 11:44
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Retracting my "three more sites" follow-up above — it is wrong, and acting on it would break three working scripts. I matched the shape (bind(0), print, close) without checking what each site does with the port afterwards:

site what the port is for so closing is…
tools/oga_pack_probe.sh:44 `` is what forward-serve binds at `:156` (`-p "$PORT"`) correct — it needs a free port; a race gives a loud bind failure, not a false pass
tests/smoke_serve.sh:117 port2 is what the impostor backend binds at :119 (--port "$port2") correct — same reason
tests/pm_laya_route.sh:42 freeport() supplies lem_port (:56), the fake lemonade's own port; :59 even says so correct — same reason

Only pm_route.sh's case is the flake you fixed: there the port must be dead (an unreachable executor), so releasing it lets something else answer and the arm passes for the wrong reason. A free port and a dead port want opposite treatments — holding a socket for the former would make the server unable to bind at all, which is worse than the flake.

So the audit's real result is one site, not four: grep -rn 'getsockname' finds six files, and only the one whose port must be unreachable needs deadport(). Apologies for the noise — the pattern match was mine, and it is the same error class as trusting a green check without asking what it ran.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant