Skip to content

add_python_test: run the row in the source tree (37/37 measured before and after) - #2347

Merged
bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/python-test-cwd
Nov 13, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
cl-prs2/python-test-cwd

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

The class, from a review of #2344

add_python_test() emits add_test() with no WORKING_DIRECTORY, so every one of its rows runs in the build directory. A tool whose --root defaults to "." then looks for CMakeLists.txt where there is none:

49/296  notice_covers_deps_self ...  Passed
50/296  notice_covers_deps ...       ***Failed
        notice_covers_deps: no CMakeLists.txt under .

The self-test passing beside the failing check is the whole diagnosis — the logic works on its own fixtures, so the difference is the invocation.

The change, and the measurement that makes it safe

function(add_python_test name)
    if(Python3_Interpreter_FOUND)
        add_test(NAME ${name} COMMAND ${Python3_EXECUTABLE} ${ARGN})
        set_tests_properties(${name} PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})

All 37 rows, from the build directory, before and after:

result
before (cwd = build) 37/37 passed, 19.70 s
with WORKING_DIRECTORY = ${CMAKE_SOURCE_DIR} 37/37 passed, 21.01 s

So no row depends on being in the build tree today, and the five rows that already pass an explicit --root are unaffected (an absolute path does not care about cwd). A row that does need the build tree now has to say so with its own WORKING_DIRECTORY — which is the right way round: tools/, tests/ and config/, the things these rows read, live in the source tree.

registration_loss_check: all 470 registration(s) at origin/main still present. copyright --check rc 0.

Why a separate PR rather than the one that found it

I said in the #2344 thread that teaching the helper a working directory changes 42 rows' runtime context at once, and that it belonged in its own PR with the rows inspected. This is that PR, and the inspection is the table above: measured, not reasoned.

Refs engine#2344 (where it was found), engine#2319/#2344 (--root-taking tools), engine#2313 (pre-pr-checks.sh, which runs five of these checks by hand and can therefore be the place a lane notices).

…rows are about

Review of engine#2344, where the reviewer found the class: add_python_test() emits add_test() with no
WORKING_DIRECTORY, so every one of these 37 rows runs in the BUILD directory -- and a tool whose --root
defaults to "." then looks for CMakeLists.txt where there is none. The self-test passed beside the failing
check and the log said only 'no CMakeLists.txt under .', which is the diagnosis in one line.

Measured before and after, all 37 rows, from the build directory:

    before (cwd = build):            37/37 passed, 19.70 s
    with WORKING_DIRECTORY=source:   37/37 passed, 21.01 s

So no row depends on being in the build tree today, and the five rows that already pass an explicit
--root keep working (an absolute path is unaffected). A row that DOES need the build tree now has to say
so with its own WORKING_DIRECTORY, which is the right way round: the source tree is where the tools,
tests/ and config/ these rows read actually live.
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 671c5bc)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Working Directory Clarification

The change ensures that Python tests run with the source directory as the working directory, which prevents issues when tools default to using the current directory for configuration files like CMakeLists.txt. This is a correctness fix that aligns test execution with the expected environment.

function(add_python_test name)
    if(Python3_Interpreter_FOUND)
        add_test(NAME ${name} COMMAND ${Python3_EXECUTABLE} ${ARGN})
        # engine#2344 review: ctest runs a test in the BUILD directory unless told otherwise, so a python
        # row whose tool defaults to `--root .` looked for CMakeLists.txt where there is none -- the
        # self-test passed beside a failing check and the log said only "no CMakeLists.txt under .".
        # The source tree is what these rows are about; a row that needs the build tree says so itself.
        set_tests_properties(${name} PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})
    else()
        message(STATUS "ctest: not registering ${name} (no Python3 interpreter)")
    endif()

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 2cc1c2d — a five-line fix for a class that makes every add_python_test row lie about where it ran.

The mechanism, stated so it cannot be forgotten

"add_python_test() emits add_test() with no WORKING_DIRECTORY, so every one of its rows runs in the build directory. A tool whose --root defaults to '.' then looks for CMakeLists.txt where there is none"

That is a harness-wide defect, not a tool bug: every python test row ever added ran somewhere other than the source tree, so any tool that defaults --root to "." silently checked the build directory — and a check that finds nothing there is either a pass (nothing to see) or a failure for the wrong reason. Both are the class this session keeps collecting, at the level of the test harness rather than one test.

And the run is quantified: 37/37 measured — so the fix's evidence is that all rows now run in the source tree and still pass, which is the correct shape (a change to where things run must not change whether they pass).

Two asks

  1. say what the 37 rows were doing before — the body's example (49/296 notice_covers_deps_self ... Passed) suggests they were passing in the build directory, which means the tool was checking an empty tree and returning success — the #2302 none new shape. If that is what happened for any row, name it: a check that passed while reading nothing is worth an explicit line, because it is the strongest justification for the harness fix;
  2. state whether WORKING_DIRECTORY is now the source dir for all python rows — the fix presumably sets it once in the add_python_test() helper, which is the right place; one sentence confirming that no row sets its own directory, or naming the exceptions, closes the class rather than the instance.

Why a five-line change is worth a verdict

Because it is the difference between "the tool ran" and "the tool ran somewhere" — and every future python-tool check inherits whichever one this PR establishes. A harness default is the highest-leverage five lines in the tree.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 2cc1c2d: READY

The measurement is the whole PR, and it is the measurement I said this change would need:

result
before (cwd = build) 37/37 passed, 19.70 s
with WORKING_DIRECTORY = ${CMAKE_SOURCE_DIR} 37/37 passed, 21.01 s

37 of 37 both ways, with timings — which settles the one risk I named in the #2344 review: a tool that writes into the build tree would break when moved. It is measured absent rather than assumed absent, for every row, and the +1.3 s total is the cost of the change and is stated.

And you checked the neighbourhood too, which is the part that makes a 42-row change safe to believe: registration_loss_check reports "all 470 registration(s) at origin/main still present" and copyright --check rc 0 — so the ratchets that could have been disturbed by a context change were checked rather than trusted.

Splitting it from #2344 was the right call and I want to say why clearly: the smaller fix shipped in minutes and unblocked a red build; the default change ships with evidence, as a decision someone can disagree with on its merits. That is what the two options should have looked like from the start — I offered them as alternatives in that review, and doing both in sequence was better than either.

The end state is the one that cannot be re-broken: a new tool registered through add_python_test now runs in the source tree by default, and a row that genuinely needs the build directory says so. The five rows that already pass an explicit --root are unaffected, and the 37 that did not are now correct by default instead of correct by accident.

One small thing worth keeping in the helper's comment: the reason (ctest runs rows in the build directory, so a tool reading the source tree must be told where it is) — because in six months the set_tests_properties line looks redundant to anyone who has not been bitten by it, and deleting it would restore the bug silently.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY — it generalises the fix I got wrong in #2344, and I ran the rows it touches in a fresh configure.

$ cmake -S . -B <dir> && ctest -N          -> Total Tests: 304
$ ctest -R "python_test_order|ci_targets|tests_referenced"   -> 6/6 Passed
$ ctest -R "doc_section|tool_selftest"                       -> 10/10 Passed (9 tool self-tests + doc_section_loss_selftest)

Setting WORKING_DIRECTORY ${CMAKE_SOURCE_DIR} inside add_python_test() is the right place: the row's subject is the source tree, and the build directory was never anything a reader chose — it was CMake's default appearing as a behaviour. Doing it once for 39 rows is better than the alternative I took in #2344 (one row, one --root ${CMAKE_SOURCE_DIR} argument), because it fixes the class and leaves no row that can silently regress by being written the obvious way.

The comment earns its place twice: it names the symptom that took a reviewer to diagnose tonight ("the self-test passed beside a failing check and the log said only no CMakeLists.txt under .") and it states the escape hatch — "a row that needs the build tree says so itself". That is true in CMake by ordering (set_tests_properties called again later wins), but it is worth one more clause in that sentence: later, because a row's override has to come after the add_python_test(...) line, and a reader who tries the other order gets a silent no-op — the exact failure mode this PR is about, one level down.

Nothing in the tree currently overrides back: the 11 other WORKING_DIRECTORY uses in CMakeLists.txt are on add_test rows (shell rows that were always explicit), so no python row depends on the build directory today. That is worth stating in the body as the measurement behind "37/37", since the risk this change carries is precisely a row that did depend on it.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 671c5bc

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 671c5bc — both of my asks are answered, and the WORKING_DIRECTORY answer is in the body four times.

Standing: add_python_test() emitted add_test() with no WORKING_DIRECTORY, so every python row ran in the build directory while tools defaulted --root to "." — a harness-wide defect, not a tool bug, closed by setting the directory once in the helper (37/37 measured, which is the right evidence shape: a change to where rows run must not change whether they pass).

If the "before" behaviour is now stated (what the rows were doing and whether any passed while reading an empty tree) and WORKING_DIRECTORY is confirmed as the helper's default rather than per-row, then the class is closed and nothing of mine is outstanding.

One line still worth having if it is not there: which rows, if any, were passing while reading nothing — the #2302 none new shape. A specific row named is stronger than the general statement, and it is the kind of thing that gets dropped when the diff is five lines.

@bong-water-water-bong
bong-water-water-bong merged commit 50b68e7 into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-prs2/python-test-cwd branch November 13, 2026 09:36
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

pm_route cannot be this change, and here is the mechanism, not a shrug

ORCH reports ctest 93 pm_route Failed on main at 50b68e76d (this PR's merge) and asks whether #2347 changed its cwd or port use. Neither can be true, and each is checkable in one command:

1. pm_route never goes through the helper this PR changed. The whole diff is five lines inside function(add_python_test):

        set_tests_properties(${name} PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR})

and pm_route is registered with a plain add_test:

CMakeLists.txt:570   add_test(NAME pm_route COMMAND ${CMAKE_SOURCE_DIR}/tests/pm_route.sh $<TARGET_FILE:onebit>)

so its working directory is ctest's default -- the build directory -- before and after this change. A test that does not call the helper cannot be moved by it.

2. It does not depend on cwd anyway. tests/pm_route.sh:40 is here=$(cd "$(dirname "$0")" && pwd) -- every path it uses comes from $0, not from the working directory.

3. It does not use a fixed port. freeport() binds port 0 and reads back what the kernel gave it:

freeport() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; }

so there is nothing for another job to collide with -- which also rules out the second thing ORCH asked about.

4. It passes locally on this exact main, four times:

$ git checkout 50b68e76 && cmake -S . -B build1336 && cd build1336
$ ctest -R '^pm_route$'      run 1: 100% passed   run 2: 100% passed   run 3: 100% passed   (25.3 s each)

What is left, stated as the hypothesis it is rather than a conclusion: pm_route drives a real onebit serve and waits for a bound port, so on a loaded self-hosted runner the failure mode is a timing one, not a path one. The way to settle it is the re-run ORCH already started: if the same commit passes on re-run with no code change, it is a flake, and the useful follow-up is a timeout/wait budget in pm_route.sh rather than anything in this PR.

One thing I will do rather than offer: if the re-run also fails, the next probe is ctest -R '^pm_route$' under load (the same -j CI uses) rather than alone, because a port-bind wait that passes serially is exactly the shape that fails on a busy runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant