Skip to content

fix(test-harness): unique --basetemp per parallel pytest subprocess#1175

Merged
Lexus2016 merged 2 commits into
mainfrom
fix/pytest-parallel-basetemp-race
Jul 19, 2026
Merged

fix(test-harness): unique --basetemp per parallel pytest subprocess#1175
Lexus2016 merged 2 commits into
mainfrom
fix/pytest-parallel-basetemp-race

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Problem

scripts/run_tests_parallel.py runs one python -m pytest <file> subprocess
per file with bounded parallelism (up to cpu*2). None passed --basetemp,
so every subprocess used the shared default /tmp/pytest-of-<user>/ root.

pytest's numbered-basetemp retention GC ("keep the last few pytest-N dirs")
runs at session start and rmtrees sibling subprocesses' live basetemp
out from under them. Result: flaky
FileNotFoundError: /tmp/pytest-of-<user>/pytest-N at fixture setup
(notably the pytest_asyncio tmp_path fixture), with zero relation to the
code under test.

This is the harness flake that reddens unrelated PRs. Concretely, #1170's
slice 4/8 failed with ~126 tmp_path setup errors in
tests/gateway/test_telegram_noise_filter.py ("all tests passed but pytest
exited non-zero") that vanished on rerun.

Fix

Allocate a unique --basetemp (tempfile.mkdtemp) per subprocess, so the
shared parent — and therefore the cross-process rmtree race — no longer
exists. Details:

  • An explicit user-supplied --basetemp in passthrough args is respected.
  • The temp tree is reclaimed on every exit path (normal, timeout, interrupt),
    so local runs don't accumulate one tree per file per invocation. CI runners
    are ephemeral anyway.

Testing

Ran the real harness over the tmp_path-heavy files (including the previously
flaky one) in parallel:

scripts/run_tests.sh tests/gateway/test_telegram_noise_filter.py \
  tests/tools/test_delegate.py tests/tools/test_tool_search.py
=> 3 files, 426 tests passed, 0 failed
  • ruff check clean; py_compile clean.
  • Verified no hermes-pytest-* temp trees leak after the run (cleanup works).
  • test_durations.json is gitignored (harness side-effect, not committed).

@Lexus2016
Lexus2016 force-pushed the fix/pytest-parallel-basetemp-race branch from 5952fc2 to a02b790 Compare July 19, 2026 09:11
…etemp

run_tests_parallel.py runs one `python -m pytest <file>` subprocess per
file with bounded parallelism. None passed --basetemp, so every subprocess
fell back to the shared default /tmp/pytest-of-<user>/ root. pytest's
numbered-basetemp retention GC ("keep the last few numbered dirs") runs on
session start and rmtree's sibling subprocesses' live basetemp out from
under them, surfacing as flaky
`FileNotFoundError: .../pytest-of-<user>/pytest-N` at fixture setup
(notably the pytest_asyncio tmp_path fixture) with no relation to the code
under test. This is the harness flake that reddened unrelated PRs — e.g.
#1170 slice 4/8 hit ~126 tmp_path setup errors that vanished on rerun.

Allocate a unique --basetemp (tempfile.mkdtemp) per subprocess so the
shared parent, and therefore the cross-process rmtree race, is gone. An
explicit user-supplied --basetemp is respected. The temp tree is reclaimed
on every exit path (normal, timeout, interrupt).
@Lexus2016

Copy link
Copy Markdown
Owner Author

Amended: the initial hermes-pytest- mkdtemp prefix put the literal "hermes" into the injected --basetemp path, which is passed as a subprocess arg inside some tests. conftest.py's live-system guard blocks any subprocess whose command contains "hermes"/"gateway" alongside a process-killer token — e.g. test_search_hidden_dirs.py greps for the literal "skill" (a token in _PROCESS_KILLERS), so the run reddened slice 4/8. Switched the prefix to a neutral pytest-rtp- so the injected path is behaviourally identical to pytest's default pytest-of-<user> location. Verified locally: the previously-failing test now passes through the harness, and the tmp_path-heavy files stay green.

@Lexus2016
Lexus2016 force-pushed the fix/pytest-parallel-basetemp-race branch from a02b790 to 8ff75f5 Compare July 19, 2026 09:23
@Lexus2016
Lexus2016 merged commit bde90a8 into main Jul 19, 2026
35 checks passed
@Lexus2016
Lexus2016 deleted the fix/pytest-parallel-basetemp-race branch July 19, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant