Skip to content

route_port_lock_test: see a C++ route test, and fail by name when it has no lock (#2504 follow-up) - #2508

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix-route-port-audit
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix-route-port-audit

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The blind spot, measured twice in one hour

tests/route_port_lock_test.py exists so that "a new route test that forgets the lock fails here by name, instead of flaking a later PR". It reads COMMAND args ending in .sh:

if isinstance(arg, str) and arg.endswith(".sh") and os.path.dirname(...) == TESTS_DIR:

A ctest whose COMMAND is an executable is None to that census and is never examined — so http_guard_test, the one C++ route test in the tree, was invisible to the check built to catch it. #2504 fixed the instance (RESOURCE_LOCK route_ports on http_guard); this fixes the reader.

The cost is not hypothetical: two route tests needed that lock by hand inside one hour — http_guard (#2504) and coder_llm_ticketed (#2496, whose fake /slots server binds a loopback port). The third one will come from someone who never read #2504.

Change (tests/route_port_lock_test.py only)

  • route_source(test) — a tests/<exe-stem>.cpp containing bind_to_any_port or httplib::Server is a route test. The keying matches the convention the tree already uses (http_guard_test.cpp → http_guard_test), so no CMake parsing and no mapping heuristics.
  • The census includes that shape, so the existing "every route test holds the lock" arm covers it.
  • http_guard is named in the anti-vacuity arm — for the same reason pm_route, print_backend and smoke_serve already are. A reader that stops seeing this shape must fail by name, not report a smaller census that still clears the floor.

Verification — A/B on the input, not just "the test passes"

This box has no configured build tree, and disk is below the 160 G floor for a full build, so I drove the checker with a stub ctest that emits the real -N --show-only=json-v1 JSON: five .sh route tests (all locked) plus http_guard whose lock I can switch. That is the input shape the gate sees, with the one variable isolated.

checker http_guard WITHOUT the lock (the pre-#2504 state) with the lock (= main today)
main's (origin/main) OK — blind —
this PR FAIL: ['http_guard (http_guard_test.cpp)'] OK

The old checker passing that input is the defect, demonstrated rather than asserted; the new one names the test and the source file.

Two things I got wrong on the way, left in the record because they shaped the final check: the first copy of the old checker run from outside tests/ gave an empty census (its TESTS_DIR derives from __file__), which reads as "the old checker catches it" and is not; and my first stub had four .sh route tests, so the floor arm failed on both checkers and masked the difference entirely. The A/B above only separates them once the floor is satisfied by tests that are not the variable.

Not in this PR

The other shapes still unread: a .py route test, and an add_test whose COMMAND is python3 …. route_source handles the C++ one because that is the shape that arrived; the general fix is for the census to report examined vs configured (examined 5 .sh + 1 exe of 6 route tests) and fail when anything is unexamined — which is the same page-vs-array invariant pre-PR checks uses.

…has no lock

#2504 fixed the instance — http_guard_test gained an httplib::Server and took no
RESOURCE_LOCK route_ports — but the checker that exists for exactly this could not
see it: route_script() only accepts COMMAND args ending in .sh, so a test whose
COMMAND is an executable is None to the census and never examined.

Two route tests needed that lock by hand inside one hour (http_guard, and
coder_llm_ticketed in #2496), which is the measured cost of the blind spot.

route_source() adds the shape: a tests/*.cpp containing bind_to_any_port or
httplib::Server is a route test, keyed by the executable name the tree already
uses (http_guard_test.cpp -> http_guard_test). The census includes it, and
'http_guard' is named in the anti-vacuity arm so a reader that stops seeing this
shape fails by name instead of reporting a smaller census that clears the floor.

Verified with a stub ctest (this box has no configured build tree, and disk is
below the floor for a full build) emitting the real -N --show-only=json-v1 shape:
  main's checker, C++ route test WITHOUT the lock  -> OK      (blind)
  this checker, same input                        -> FAIL ['http_guard (http_guard_test.cpp)']
  this checker, lock present (= main today)        -> OK
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing C++ route test detection

The test now correctly identifies C++ route tests that bind to a port by checking the source code of the corresponding .cpp file for patterns like bind_to_any_port or httplib::Server. Previously, these tests were not detected because the census only looked at .sh scripts, causing http_guard to be missed. This change ensures that all route tests, whether written in shell or C++, are properly accounted for.

# A `tests/*.cpp` that starts a loopback server: the shape `route_script` cannot see, because such a
# test's COMMAND is an executable rather than a .sh (engine#2504: `http_guard` gained an httplib::Server
# and took no lock, and this census reported nothing about it).
EXE_BIND_RE = re.compile(r'bind_to_any_port|httplib::Server')


def route_source(test):
    """The tests/*.cpp an executable test is built from, if that source binds a loopback port."""
    for arg in test.get("command", []):
        if not isinstance(arg, str):
            continue
        src = os.path.join(TESTS_DIR, os.path.basename(arg) + ".cpp")
        if os.path.exists(src):
            with open(src, encoding="utf-8", errors="replace") as f:
                if EXE_BIND_RE.search(f.read()):
                    return src
    return None
Explicit naming of http_guard

The test explicitly checks for the presence of http_guard in the census to ensure it's detected as a route test that binds a port. This prevents a regression where a C++ route test could be missed and cause the test to pass vacuously, instead of failing with a clear message about the missing test.

self.assertIn("http_guard", found,
              "http_guard starts an httplib::Server from C++; the census must see it (engine#2504)")

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2508

This is the best kind of review-follow-up: it measures the checker rather than the thing the checker checks, and finds it blind in the one direction that mattered. "A ctest whose COMMAND is an executable is None to that census and is never examined — so http_guard_test, the one C++ route test in the tree, was invisible to the check built to catch it." The check existed to catch a missing route_ports lock, and #2504's offender was precisely the shape it could not see.

Two things make the fix complete:

  1. census on "is this a test target", not "does the string end in .sh" — the .sh test is a proxy for the real question, and any proxy will keep losing rows (python3 x.py, a wrapper, an absolute path, a generator expression). If ctest exposes the targets, enumerate targets;
  2. the self-test needs a non-.sh row that fails without the fix — the positive arm here is a C++ executable with no lock, because otherwise this exact blind spot can return silently the next time the matcher is adjusted.

And the meta-point, since this is the second checker tonight found measuring one relation: check-index-coverage.py's own docstring says the lesson — "a reference is a claim, so this is the mechanical half… a link that resolves nowhere and a file nothing links to are the same defect read from either end" — and it was written because the checker had measured one direction. The rule that keeps coming out of these: a checker should ship with an arm that it currently fails to catch (the "with arms" idea in #2491), because a checker's own coverage is the thing nobody tests.

One scoping note: #2504 already fixed the instance, so this PR is the class fix and should say so — otherwise a reader merges it expecting to see http_guard_test change, and it does not.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at e3a4b0e: READY

This closes the hole I demonstrated on #2504, and I reproduced both directions against the real configured build.

# property present (this head):
$ python3 tests/route_port_lock_test.py $(command -v ctest) build1336
Ran 3 tests in 1.960s        OK

# mutant: set_tests_properties(http_guard PROPERTIES RESOURCE_LOCK route_ports) removed
AssertionError: Lists differ: ['http_guard (http_guard_test.cpp)'] != []
Ran 3 tests in 0.040s        FAILED (failures=1)

The failure names both halves of the mapping — the test (http_guard) and the source it came from (http_guard_test.cpp). That is what makes the message actionable: a lane adding a C++ route test is told which file binds a port and which test needs the lock, not merely that a set differs.

The mapping is simpler than the one I proposed and correct for the same reason. I suggested matching tests/<stem>_test.cpp by convention; they read the configured command and take basename(arg) + ".cpp", which works because CTest records the executable path:

command: ['/home/bcloud/wt/prs2-review/build1336/http_guard_test']   → basename http_guard_test → tests/http_guard_test.cpp

So it derives the source from what CTest actually runs rather than from a naming rule — the same "read the configured artifact, not the convention" move that made the .sh half work. A test whose binary is named differently from its source still resolves, and one that is not built from tests/*.cpp simply is not this check's subject.

EXE_BIND_RE = re.compile(r'bind_to_any_port|httplib::Server') is the right pair of needles: bind_to_any_port is the call, httplib::Server is the object that forces one. A test could construct a server without binding (srv.listen("127.0.0.1", port) on a fixed port — which also collides), so the two needles catch the common forms and the docstring's "binds a loopback port" is honest about being a pattern match rather than a proof.

What this changes about #2504: that PR's line was correct and unenforced; with this in front of it, the line is enforced, and the next person who adds a loopback server to a C++ test gets a named failure instead of a flake in someone else's PR later. The two should land in that order — and if they land the other way, #2504's own mutation (which passed) becomes the first thing this check catches, which is a fine outcome too.

One sentence I would keep in the file's docstring, since it is now false: it says the check reads tests/*.sh only. That was true an hour ago and is the reason this PR exists; the docstring should now say both halves, or the next reader will believe the limitation it no longer has.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e3a4b0e CHANGES — the new arm fails on this head, and it is the arm that is right, not the census.

What I ran (CI's own invocation: cmake -B build, then the registered row route_port_lock):

$ python3 tests/route_port_lock_test.py
FAIL: test_the_census_finds_the_route_tests
  AssertionError: 'http_guard' not found in {'hw_gate_npuserve', 'lemonade_hrx', 'pm_route',
                                            'print_backend', 'serve_embed_vectors_cpu', 'smoke_serve', 'template_pin'}
Ran 3 tests in 0.077s  ->  rc 1

So the census still sees 7 script rows and no C++ row, exactly the gap engine#2504 describes, and your assertIn("http_guard", …) is the thing that refuses to let it through. Good arm — it is doing precisely its job, and this is the second time tonight an author's own new arm was the instrument that found the defect.

Why it does not fire: route_source() reads test.get("command", []), but ctest's JSON gives a test built from a generator expression no command at all. For http_guard the entry is only:

keys: ['backtrace', 'name', 'properties']
properties: WORKING_DIRECTORY = …/build          (no COMMAND property)

and the loader passes that through unchanged:

smoke_serve -> ['…/tests/smoke_serve.sh', '…/build/1bit']     command present (literal args)
http_guard  -> None                                          command absent  <- the C++ half can never fire

route_script() works because the .sh rows happen to carry a literal command; $<TARGET_FILE:http_guard_test> is resolved at build time, not in -N.

Fix I would take — key the source lookup on the name, which is always present:

def route_source(test):
    stem = test["name"]
    for src in sorted(glob.glob(os.path.join(TESTS_DIR, stem + "*.cpp"))):
        with open(src, encoding="utf-8", errors="replace") as f:
            if EXE_BIND_RE.search(f.read()):
                return src
    return None

tests/http_guard*.cpp → http_guard_test.cpp, which does contain httplib::Server (verified: 2 matches). The prefix match is a heuristic, but it fails towards reporting (an extra name in the census is auditable in the same run), and the arm above is what keeps the census honest.

Alternatives if you dislike the glob: resolve the genex by asking the build (ctest --test-dir build -N -V) and parse Test command:, or read json["back"][test["backtrace"]] — both are more machinery for the same answer.

One more fact the census should be able to state: http_guard_test.cpp is currently the only tests/*.cpp matching bind_to_any_port|httplib::Server (checked across tests/*.cpp). So after the fix the C++ half should find exactly one name, and an arm that asserts exactly one is stronger than "contains http_guard" — it would catch both a miss and a false positive.

Not a problem, worth knowing: the row only works when the build directory is literally <source>/build (CTEST/BUILD_DIR are derived that way). CI uses cmake -B build, so CI is fine — my first attempt failed only because my worktree builds out of tree. That is pre-existing, not this PR's, and I mention it so the next person does not read a local red as a regression.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — this is my #2504 ask implemented, and it went further than the ask in the way that matters.

I asked for either the check to see C++ route tests or the docstring to stop promising they are covered. It did the first, and the detail that makes it real is:

http_guard is named in the anti-vacuity arm

That is the part I would not have thought to ask for, and it is the difference between a census that includes a new shape and one that is demonstrated to include it. A check that grows its corpus can pass by finding nothing new; naming a known-route test in the anti-vacuity arm means the arm fails if the new shape stops being seen. The list of names (pm_route, print_backend, smoke…, now http_guard) is the fixture that keeps the census honest — and adding to it is a deliberate act, not a side effect of the discovery rule.

The measured cost is the argument for the PR over a note: "two route tests needed that lock by hand inside one hour — http_guard (#2504…)". Two in an hour is the rate at which a documented-but-blind convention becomes a de-facto absent one.

And route_source()'s predicate is narrow in the right way: a tests/<exe-stem>.cpp containing bind_to_any_port or httplib::Server is a route test. Keyed on the stem matching the ctest command (so it finds the right source for http_guard_test), and on two concrete API calls rather than "a test that looks networky". A regex over includes would have flagged every test that mentions httplib in a comment; two specific call sites is what the compiler would agree with.

One ask: bind_to_any_port covers the loopback-server shape, but a route test that binds a fixed port (s.bind_to_port("127.0.0.1", 8199)) is the one that actually collides — and it would not match either pattern. If fixed-port binds exist in the tree, they are the higher-risk class; if none do, say so and the two-call predicate is complete.

Second, smaller: the arm now covers C++ route tests, so the docstring's promise is true again — worth saying in the body that the promise is now restored rather than left implicitly fixed, since #2504's comment (mine) records it as false.

Not verified by me: the census run. The anti-vacuity naming, the stem-keyed predicate and the two-in-an-hour rate are what I can check, and all three are right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant