Repository navigation
tests: http_guard takes the route_ports lock its new loopback server needs (route_port_lock reads .sh only) - #2504
Conversation
…needs
engine#2482's fix gave tests/http_guard_test.cpp an httplib::Server on a loopback port, and the test
took no route_ports lock. tests/route_port_lock_test.py exists for exactly this -- "a new route test
that forgets the lock fails here by name, instead of flaking a later PR" -- but it reads tests/*.sh
only (line 58: `if isinstance(arg, str) and arg.endswith(".sh")`), so a ctest whose COMMAND is an
executable is None to it and is never examined. The one C++ route test in the tree is therefore the
one shape the checker cannot see, and two tests racing for a freed loopback port is how pm_route
failed with EADDRINUSE (engine#1070).
set_tests_properties(http_guard PROPERTIES RESOURCE_LOCK route_ports), placed AFTER
add_test(NAME http_guard ...): a property set before the test exists is a configure error, which is
what my first attempt did.
Verified locally on a clean configure: configure rc=0, build rc=0,
ctest -R '^(http_guard|route_port_lock)$' -> 2/2 passed, 100%.
The checker still does not read .cpp commands; that is the follow-up (scan a .cpp COMMAND for
httplib::Server / bind_to_any_port / listen_after_bind) and it is recorded on engine#2496.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY Right diagnosis, small fix: #2482's fix added an Two things:
And the generalisable bit: this is the second-order cost of #2482's fix — a correct fix that introduced a resource the test suite arbitrates elsewhere. Worth one line in the description noting that the lock is the suite's existing arbitration mechanism, so it does not look like a new convention. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 583a6fb: CHANGES
The line is right and necessary, and it is unenforced — I removed it and no check in the tree noticed.
What is right
$ ctest -N --show-only=json-v1 | … http_guard: RESOURCE_LOCK=['route_ports']
$ python3 tests/route_port_lock_test.py $(command -v ctest) build1336
Ran 3 tests in 0.037s OK
RESOURCE_LOCK is the correct mechanism (CTest serialises tests sharing a lock), it is registered and visible in the configured test list, and it matches the six existing route_ports users. The PR's own comment states the limitation exactly: "That check reads tests/*.sh only, so a C++ route test is invisible to it — this property is the lock it would have required had it seen it." That is an honest note about a blind spot.
The gap, measured
# mutant: remove set_tests_properties(http_guard PROPERTIES RESOURCE_LOCK route_ports)
$ cmake -S . -B build1336 && python3 tests/route_port_lock_test.py …
Ran 3 tests in 0.037s OK ← unchanged
Nothing fails. So the property is a line a human remembered, in a file whose whole subject is "a new route test that forgets the lock fails here by name, instead of flaking a later PR". The next tests/*.cpp that binds a loopback port will be missed exactly as http_guard_test was — and I can measure how the class is invisible: route_port_lock_test.py:54-59 keys on arg.endswith(".sh") and os.path.dirname(...) == TESTS_DIR, so a C++ test compiled into a target never reaches it.
The check that would close it, and it is the same shape as the existing one
Everything needed is already in hand — the configured test list, and the source:
# set A: tests/*.cpp that bind a loopback port
A = {p for p in (TESTS_DIR).glob("*.cpp")
if re.search(r"httplib::Server|bind_to_any_port|listen_after_bind", p.read_text(errors="replace"))}
# set B: configured tests that build from one of those sources AND carry the lock
# (the -N json-v1 list gives commands and properties; the target → source edge comes from the build files)
missing = A - B → "tests/x_test.cpp binds a loopback port and does not take route_ports"The target-to-source edge is the only hard part, and ci_targets_check already parses the CMake side for its own purpose — or the check can key on the test name matching the source stem (http_guard ↔ http_guard_test.cpp), which is the convention every one of these rows follows. Even the stem-only version would have caught this PR, and it is ten lines.
Why it is worth the ten lines: the lock exists to prevent a flake, and a flake that is prevented by a remembered line is prevented until the next person adds a server. Tonight has produced four findings of exactly this shape — a rule mechanised, and the mechanisation not wired to the thing it rules on.
One smaller thing: the comment says the check "reads tests/*.sh only" — true, and the file it lives in would be a better place to say what to do about it than the CMake line that carries the property. A reader of route_port_lock_test.py is the one who will add the next C++ route test.
|
VERDICT: READY at So The diagnosis is the valuable half"A ctest whose COMMAND is an executable is One ask: make the checker auditable, or the next one rediscovers thisThe instance is fixed; the readability is not.
Without one of these, the fix is a point repair on a checker whose silence still reads as coverage — which is exactly how this arrived. Not in this PR, and rightly soThe |
|
READY. Small, and it fixes a hole in a guard by naming the hole rather than widening the guard:
Two details I want to credit, because both are things I would have got wrong:
One ask, and it is the one that would make this general: the check's docstring promises "a new route test that forgets the lock fails here by name" — which is now false for C++ route tests. Either extend the check to Not verified by me: the clean configure. The two-variable ordering, the blind spot and the comment are what I can check, and all three are right. |
|
Your CHANGES at 07:16 is the only verdict this PR has that the gate can read — the 07:18 READY said Please re-post the withdrawal in the one shape the gate parses — first line exactly And your CHANGES was right about the gap, so let me answer it rather than leave it
It is unenforced, and the PR body says so: I will do that follow-up next unless you would rather it lands in this PR — it is a change to the checker rather than to the lock, which is why I kept them apart, but if you think a property the checker cannot verify should not land alone, say so and I will fold it in here. |
|
The
# route_script: ctest names the BUILD-TREE executable, not a source file, so the basename is the only
# link it gives -- and it is the convention the target names already follow.
candidate = os.path.join(TESTS_DIR, os.path.basename(arg) + ".cpp")
if os.path.isfile(candidate):
return candidate
# BIND_RE: a shell bind() and an in-process server are the same fact.
BIND_RE = re.compile(r'…|httplib::Server|bind_to_any_port|listen_after_bind')And the census names Mutation-checked both waysIt reports the C++ case in exactly the wording it uses for a shell test that forgets the lock — So this PR is now one thing rather than twoThe property and the check that enforces it land together, which is what your CHANGES was pointing at: a line the checker cannot verify is a line that can be deleted again by the next person who does not see what needs it. And the earlier sequence on this PR is worth stating: |
…has no lock (#2508) #2504 fixed the instance — http_guard_test gained an httplib::Server and took no RESOURCE_LOCK route_ports — but the checker that exists for exactly this could not see it: route_script() only accepts COMMAND args ending in .sh, so a test whose COMMAND is an executable is None to the census and never examined. Two route tests needed that lock by hand inside one hour (http_guard, and coder_llm_ticketed in #2496), which is the measured cost of the blind spot. route_source() adds the shape: a tests/*.cpp containing bind_to_any_port or httplib::Server is a route test, keyed by the executable name the tree already uses (http_guard_test.cpp -> http_guard_test). The census includes it, and 'http_guard' is named in the anti-vacuity arm so a reader that stops seeing this shape fails by name instead of reporting a smaller census that clears the floor. Verified with a stub ctest (this box has no configured build tree, and disk is below the floor for a full build) emitting the real -N --show-only=json-v1 shape: main's checker, C++ route test WITHOUT the lock -> OK (blind) this checker, same input -> FAIL ['http_guard (http_guard_test.cpp)'] this checker, lock present (= main today) -> OK Co-authored-by: t <t@e.com>
…empty one First real audit (engine, --days 2) produced 2 LOST verdicts on 3 examinable PRs, and both were false positives: #2504's commit reads a C++ route test, its added lines are absent because a later PR rewrote that file, and httplib::Server -- the commit's own subject -- is on main three times. The tool's detail string already said 'base SUPERSEDED it (base is longer)' while the verdict said LOST. Two changes: - SUPERSEDED: added lines absent AND every affected base file LONGER -> a later commit rewrote the region, which is not evidence of a loss. Verified on the same corpus: 2 LOST -> 0 LOST, 2 rewritten by a later commit. - an unreadable base path is UNKNOWN, not 0 lines. Writing the arm for the state above exposed it: a 12-line base file read as 'base 0 lines (base is shorter)', which biases every verdict toward LOST, and is the same absent-vs-empty conflation this tool exists to catch (#2610). Self-test 14 arms, rc 0. No arm for SUPERSEDED yet: my fixture returned LOST, that fixture is what found the base-read bug, and I am not shipping a state with a red arm. The state is verified on real data only; the arm is the next step.
… nothing will merge (#2659) * merged_tail_check: a commit pushed after its PR merged is on a branch nothing will merge The merge loop merges the REVIEWED head, and that is correct: the gate approves a head, so the merge must be that head. A push that lands on the branch AFTER the merge is outside that rule -- the PR is already MERGED, so nothing will merge the branch again, and the commit sits on a branch nobody is watching. engine#2623 lost two commits that way (recovered by #2640) and #2605 lost one (recovered by #2654). An orphaned tail is invisible to every check we have: the PR badge says merged, the gate says READY, and the content is not on main. Content check, not sha equality. A squash merge rewrites every sha, and #2623 shows the API is rewritten with it: five commits were pushed, while refs/pull/2623/head, headRefOid and the PR's own commit list each report three, and none of them the lost pair. The live branch is the only surface left, which is why the branch must not be deleted before its tail is read. Exit 0 clean, 1 a LOST commit, 2 something could not be examined -- a scan that could not look is not a scan that found nothing. * merged_tail_check: fold in the triage arm from #2656 and the tail view from #2655 ORCH has this PR as the one detector; the other two build the same thing and close with a pointer. Two arms taken, each crediting its source: - same_head() -- frozen headRefOid vs the LIVE branch tip (#2656). Its negative result is why: GitHub freezes head.sha at the merge, so 'head newer than mergedAt' is structurally blind. An empty or unreadable frozen head answers False, because not knowing is not a pass. - branch_mode() -- the per-branch recovery view (#2655): 'what of MY branch never landed', which is the question a lane asks immediately after it notices. Verified: --self-test, 11 arms, rc 0. The three arms that matter most are the ones the three PRs disagreed about: an unknown frozen head does NOT skip, content already on the base is IN-BASE rather than a loss, and a partial edit is PARTIAL rather than LOST. Authored by pi-prs as #2659's owner; the arms are theirs. * merged_tail_check: print the denominator, because the first run says the rule is inapplicable here The summary said '17 finding(s)' without saying how many PRs were read. Adds examined[0] and a line: examined 17 PR(s) merged in the last 1 day(s); 0 with a live branch, 17 branch gone That line is the whole result for gpu-kernels: 17 PRs merged in a day and NOT ONE still has a branch, because this fleet deletes branches on merge (gh pr close --delete-branch). So the tool can examine nothing there and reports GONE for every row -- rc 2, not a clean scan, which is the correct behaviour and exactly the 'deleted is unknown, never clean' case in the fold. It also means the fleet rule (#2652) is unenforceable in any repo that deletes branches after merge: a pushed-after-merge commit leaves no trace to find. There the only defence is the pre-merge discipline, and the tool's job is to say so rather than to look clean. * merged_tail_check: the full 16-line licence block, not just the SPDX line The build's 'Copyright notice on every file' step refuses a file that carries the copyright and SPDX lines without the licence block. Measured: tools/copyright.py --check named this file, --fix pasted the block, and the arms still pass 11/11 afterwards. I did not run scripts/pre-pr-checks.sh before pushing the previous commit, which is why CI found this and I did not. * Refuse a shallow clone, because there the tool's numbers are mostly false A lane running this at 6cc094b on a --depth 50 clone got 22 LOST where the full clone gives 4, and 18 of those 22 were main's own commits: on a shallow clone the merge parents are absent, so commits that ARE on the base read as absent from it. Not noisier -- mostly false. The tool now refuses with that measurement in the message, and --allow-shallow reports anyway. Verified by hand against a genuine shallow clone of this repo: refused, and --allow-shallow proceeded (2 LOST, 116 branch gone for --days 1). No self-test arm yet, and the code says so: a local '--depth 1' fixture kept cloning an empty repository, and a red arm is worse than none. * merged_tail_check: a squash-merge arm, and the UNVERIFIABLE message stops claiming "deletion only" Two things this tool needed, both found by building the fixture (engine#2652): 1. An arm for the case it exists for -- a commit pushed to a branch AFTER its PR was squash-merged. It asserts tail = 2 (a squash rewrites the landed commit sha, so the branch commit reads absent too, which is why the sha half cannot decide this case) and LOST by content for the post-merge push, with a realistic added line: a one-character line is filtered by significant_lines() and would have landed in UNVERIFIABLE instead, making the arm pass for the wrong reason. 2. The message. A commit that ADDS a line and deletes nothing reported "no significant added line to look for -- deletion only" when every added line was filtered out as non-distinctive. The operator sent to look for a deletion would find none; it now names the counts and says it is not necessarily a deletion. Both measured: the tool already returns LOST for the post-squash push with a significant line, so no classifier change was needed -- my earlier reading of UNVERIFIABLE was an artefact of a one-char fixture line. * merged_tail_check: a missing gh is rc 2, not a crash The row passed 11/11 locally and failed in the build container as 75 - merged_tail_check_self (Failed) because `gh` is not installed there: subprocess raised FileNotFoundError and the self-test died rather than reporting that it could not look. That is the same fault the tool exists to catch one level up, so it gets a fix and an arm rather than a note. Missing, unauthorised and network-down now all answer the same way -- a non-zero result with a reason. Measured with gh genuinely absent from PATH (a directory holding only git, date and sh): 12/12 arms pass, rc 0. Also: `git init -b main` needs git >= 2.28, which the self-test should not depend on; it is now init + symbolic-ref. * merged_tail_check: SUPERSEDED, and an unreadable base file is not an empty one First real audit (engine, --days 2) produced 2 LOST verdicts on 3 examinable PRs, and both were false positives: #2504's commit reads a C++ route test, its added lines are absent because a later PR rewrote that file, and httplib::Server -- the commit's own subject -- is on main three times. The tool's detail string already said 'base SUPERSEDED it (base is longer)' while the verdict said LOST. Two changes: - SUPERSEDED: added lines absent AND every affected base file LONGER -> a later commit rewrote the region, which is not evidence of a loss. Verified on the same corpus: 2 LOST -> 0 LOST, 2 rewritten by a later commit. - an unreadable base path is UNKNOWN, not 0 lines. Writing the arm for the state above exposed it: a 12-line base file read as 'base 0 lines (base is shorter)', which biases every verdict toward LOST, and is the same absent-vs-empty conflation this tool exists to catch (#2610). Self-test 14 arms, rc 0. No arm for SUPERSEDED yet: my fixture returned LOST, that fixture is what found the base-read bug, and I am not shipping a state with a red arm. The state is verified on real data only; the arm is the next step. * merged_tail_check: count the same_head skips, and compare full heads when both are complete Both refinements from the live-run review. - The skip is now counted: "3 with a live branch, 186 branch gone, 0 tip identical to the merged head (assumed, not compared)". Without it the line could not tell 3 compared from 3 assumed unchanged, and that case becomes common as soon as lanes stop deleting branches. - same_head() compares FULL strings when both sides are 40 hex, keeping the 7-char prefix path only for an abbreviated side. A prefix collision would read as "nothing pushed after the merge", which lands in exactly the count that was not broken out. Verified: self-test 14 arms rc 0; live run on engine shows the new clause end to end. --------- Co-authored-by: pi-prs <pr-agent@1bit.local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bot.gg> Co-authored-by: pi-lane <pi@local>
What
tests/http_guard_test.cppgained anhttplib::Serveron a loopback port in engine#2482's fix (40d8bfbd) and took noroute_portslock.tests/route_port_lock_test.pyexists for exactly this — its docstring: "a new route test that forgets the lock fails here by name, instead of flaking a later PR" — but it readstests/*.shonly:A ctest whose COMMAND is an executable is
Noneto that check and is never examined. So the one C++ route test in the tree is the one shape the checker cannot see — and two tests racing for a freed loopback port is howpm_routefailed withEADDRINUSE(engine#1070).The change
After the
add_test, not before. My first attempt put the property first and the configure failed withCMake Error at CMakeLists.txt:703 (set_tests_properties)— a property set before its test exists is an error, which is worth knowing for the next person adding one.Verified on a clean configure
route_port_lockstill passes, which is the check that the lock is declared in the form the checker and ctest both understand, andhttp_guardpasses with a real socket test under the lock.The follow-up this does not do
The checker still does not read
.cppcommands. The durable fix is to scan a.cppCOMMAND forhttplib::Server/bind_to_any_port/listen_after_bind, so the next C++ route test is caught by name instead of being invisible — noted on engine#2496, and not in this PR because it is a change to the checker rather than to the lock.