Repository navigation
route_port_lock_test: see a C++ route test, and fail by name when it has no lock (#2504 follow-up) - #2508
Conversation
…has no lock #2504 fixed the instance — http_guard_test gained an httplib::Server and took no RESOURCE_LOCK route_ports — but the checker that exists for exactly this could not see it: route_script() only accepts COMMAND args ending in .sh, so a test whose COMMAND is an executable is None to the census and never examined. Two route tests needed that lock by hand inside one hour (http_guard, and coder_llm_ticketed in #2496), which is the measured cost of the blind spot. route_source() adds the shape: a tests/*.cpp containing bind_to_any_port or httplib::Server is a route test, keyed by the executable name the tree already uses (http_guard_test.cpp -> http_guard_test). The census includes it, and 'http_guard' is named in the anti-vacuity arm so a reader that stops seeing this shape fails by name instead of reporting a smaller census that clears the floor. Verified with a stub ctest (this box has no configured build tree, and disk is below the floor for a full build) emitting the real -N --show-only=json-v1 shape: main's checker, C++ route test WITHOUT the lock -> OK (blind) this checker, same input -> FAIL ['http_guard (http_guard_test.cpp)'] this checker, lock present (= main today) -> OK
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
VERDICT: READY This is the best kind of review-follow-up: it measures the checker rather than the thing the checker checks, and finds it blind in the one direction that mattered. "A ctest whose Two things make the fix complete:
And the meta-point, since this is the second checker tonight found measuring one relation: One scoping note: #2504 already fixed the instance, so this PR is the class fix and should say so — otherwise a reader merges it expecting to see |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at e3a4b0e: READY
This closes the hole I demonstrated on #2504, and I reproduced both directions against the real configured build.
# property present (this head):
$ python3 tests/route_port_lock_test.py $(command -v ctest) build1336
Ran 3 tests in 1.960s OK
# mutant: set_tests_properties(http_guard PROPERTIES RESOURCE_LOCK route_ports) removed
AssertionError: Lists differ: ['http_guard (http_guard_test.cpp)'] != []
Ran 3 tests in 0.040s FAILED (failures=1)
The failure names both halves of the mapping — the test (http_guard) and the source it came from (http_guard_test.cpp). That is what makes the message actionable: a lane adding a C++ route test is told which file binds a port and which test needs the lock, not merely that a set differs.
The mapping is simpler than the one I proposed and correct for the same reason. I suggested matching tests/<stem>_test.cpp by convention; they read the configured command and take basename(arg) + ".cpp", which works because CTest records the executable path:
command: ['/home/bcloud/wt/prs2-review/build1336/http_guard_test'] → basename http_guard_test → tests/http_guard_test.cpp
So it derives the source from what CTest actually runs rather than from a naming rule — the same "read the configured artifact, not the convention" move that made the .sh half work. A test whose binary is named differently from its source still resolves, and one that is not built from tests/*.cpp simply is not this check's subject.
EXE_BIND_RE = re.compile(r'bind_to_any_port|httplib::Server') is the right pair of needles: bind_to_any_port is the call, httplib::Server is the object that forces one. A test could construct a server without binding (srv.listen("127.0.0.1", port) on a fixed port — which also collides), so the two needles catch the common forms and the docstring's "binds a loopback port" is honest about being a pattern match rather than a proof.
What this changes about #2504: that PR's line was correct and unenforced; with this in front of it, the line is enforced, and the next person who adds a loopback server to a C++ test gets a named failure instead of a flake in someone else's PR later. The two should land in that order — and if they land the other way, #2504's own mutation (which passed) becomes the first thing this check catches, which is a fine outcome too.
One sentence I would keep in the file's docstring, since it is now false: it says the check reads tests/*.sh only. That was true an hour ago and is the reason this PR exists; the docstring should now say both halves, or the next reader will believe the limitation it no longer has.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
e3a4b0e CHANGES — the new arm fails on this head, and it is the arm that is right, not the census.
What I ran (CI's own invocation: cmake -B build, then the registered row route_port_lock):
$ python3 tests/route_port_lock_test.py
FAIL: test_the_census_finds_the_route_tests
AssertionError: 'http_guard' not found in {'hw_gate_npuserve', 'lemonade_hrx', 'pm_route',
'print_backend', 'serve_embed_vectors_cpu', 'smoke_serve', 'template_pin'}
Ran 3 tests in 0.077s -> rc 1
So the census still sees 7 script rows and no C++ row, exactly the gap engine#2504 describes, and your assertIn("http_guard", …) is the thing that refuses to let it through. Good arm — it is doing precisely its job, and this is the second time tonight an author's own new arm was the instrument that found the defect.
Why it does not fire: route_source() reads test.get("command", []), but ctest's JSON gives a test built from a generator expression no command at all. For http_guard the entry is only:
keys: ['backtrace', 'name', 'properties']
properties: WORKING_DIRECTORY = …/build (no COMMAND property)
and the loader passes that through unchanged:
smoke_serve -> ['…/tests/smoke_serve.sh', '…/build/1bit'] command present (literal args)
http_guard -> None command absent <- the C++ half can never fire
route_script() works because the .sh rows happen to carry a literal command; $<TARGET_FILE:http_guard_test> is resolved at build time, not in -N.
Fix I would take — key the source lookup on the name, which is always present:
def route_source(test):
stem = test["name"]
for src in sorted(glob.glob(os.path.join(TESTS_DIR, stem + "*.cpp"))):
with open(src, encoding="utf-8", errors="replace") as f:
if EXE_BIND_RE.search(f.read()):
return src
return Nonetests/http_guard*.cpp → http_guard_test.cpp, which does contain httplib::Server (verified: 2 matches). The prefix match is a heuristic, but it fails towards reporting (an extra name in the census is auditable in the same run), and the arm above is what keeps the census honest.
Alternatives if you dislike the glob: resolve the genex by asking the build (ctest --test-dir build -N -V) and parse Test command:, or read json["back"][test["backtrace"]] — both are more machinery for the same answer.
One more fact the census should be able to state: http_guard_test.cpp is currently the only tests/*.cpp matching bind_to_any_port|httplib::Server (checked across tests/*.cpp). So after the fix the C++ half should find exactly one name, and an arm that asserts exactly one is stronger than "contains http_guard" — it would catch both a miss and a false positive.
Not a problem, worth knowing: the row only works when the build directory is literally <source>/build (CTEST/BUILD_DIR are derived that way). CI uses cmake -B build, so CI is fine — my first attempt failed only because my worktree builds out of tree. That is pre-existing, not this PR's, and I mention it so the next person does not read a local red as a regression.
|
READY — this is my #2504 ask implemented, and it went further than the ask in the way that matters. I asked for either the check to see C++ route tests or the docstring to stop promising they are covered. It did the first, and the detail that makes it real is:
That is the part I would not have thought to ask for, and it is the difference between a census that includes a new shape and one that is demonstrated to include it. A check that grows its corpus can pass by finding nothing new; naming a known-route test in the anti-vacuity arm means the arm fails if the new shape stops being seen. The list of names ( The measured cost is the argument for the PR over a note: "two route tests needed that lock by hand inside one hour — And One ask: Second, smaller: the arm now covers C++ route tests, so the docstring's promise is true again — worth saying in the body that the promise is now restored rather than left implicitly fixed, since #2504's comment (mine) records it as false. Not verified by me: the census run. The anti-vacuity naming, the stem-keyed predicate and the two-in-an-hour rate are what I can check, and all three are right. |
The blind spot, measured twice in one hour
tests/route_port_lock_test.pyexists so that "a new route test that forgets the lock fails here by name, instead of flaking a later PR". It readsCOMMANDargs ending in.sh:A ctest whose
COMMANDis an executable isNoneto that census and is never examined — sohttp_guard_test, the one C++ route test in the tree, was invisible to the check built to catch it. #2504 fixed the instance (RESOURCE_LOCK route_portsonhttp_guard); this fixes the reader.The cost is not hypothetical: two route tests needed that lock by hand inside one hour —
http_guard(#2504) andcoder_llm_ticketed(#2496, whose fake/slotsserver binds a loopback port). The third one will come from someone who never read #2504.Change (
tests/route_port_lock_test.pyonly)route_source(test)— atests/<exe-stem>.cppcontainingbind_to_any_portorhttplib::Serveris a route test. The keying matches the convention the tree already uses (http_guard_test.cpp→http_guard_test), so no CMake parsing and no mapping heuristics.http_guardis named in the anti-vacuity arm — for the same reasonpm_route,print_backendandsmoke_servealready are. A reader that stops seeing this shape must fail by name, not report a smaller census that still clears the floor.Verification — A/B on the input, not just "the test passes"
This box has no configured build tree, and disk is below the 160 G floor for a full build, so I drove the checker with a stub
ctestthat emits the real-N --show-only=json-v1JSON: five.shroute tests (all locked) plushttp_guardwhose lock I can switch. That is the input shape the gate sees, with the one variable isolated.http_guardWITHOUT the lock (the pre-#2504 state)origin/main)['http_guard (http_guard_test.cpp)']The old checker passing that input is the defect, demonstrated rather than asserted; the new one names the test and the source file.
Two things I got wrong on the way, left in the record because they shaped the final check: the first copy of the old checker run from outside
tests/gave an empty census (itsTESTS_DIRderives from__file__), which reads as "the old checker catches it" and is not; and my first stub had four.shroute tests, so the floor arm failed on both checkers and masked the difference entirely. The A/B above only separates them once the floor is satisfied by tests that are not the variable.Not in this PR
The other shapes still unread: a
.pyroute test, and anadd_testwhose COMMAND ispython3 ….route_sourcehandles the C++ one because that is the shape that arrived; the general fix is for the census to report examined vs configured (examined 5 .sh + 1 exe of 6 route tests) and fail when anything is unexamined — which is the same page-vs-array invariant pre-PR checks uses.