CI test runner reports pytest-style files as passing without running them — 11 files uncollected, 2 hiding real failures

[ChuckBuilds]

Summary

scripts/run_plugin_tests.py runs every discovered test_*.py as a subprocess and reads its exit code (0 pass / 2 skip / 1 fail). That is right for the standalone scripts most plugin tests are — but 11 files at plugin root are pytest modules, and running one of those as a script merely imports it, defines its test functions, runs none of them, and exits 0.

CI reports those as passing. Two of the eleven are failing right now.

Reproduction

$ cd ~/LEDMatrix
$ PYTHONPATH=$PWD LEDMATRIX_CORE=$PWD python3 plugin-repos/ledmatrix-flights/test_vegas_map_parity.py
$ echo $?
0                       # <- what CI sees

$ PYTHONPATH=$PWD LEDMATRIX_CORE=$PWD python3 -m pytest plugin-repos/ledmatrix-flights/test_vegas_map_parity.py -q
4 failed, 17 passed in 0.68s

The file has 22 def test_ / class Test definitions and no __main__ guard, so there is nothing for a script run to execute.

The eleven files, and what they are really doing

Checked each one both ways on a rig with all 44 first-party plugins installed:

file	as a script	under pytest
ledmatrix-flights/test_vegas_map_parity.py	exit 0	4 failed, 17 passed
news/test_news_ticker.py	exit 0	1 failed, 15 passed
f1-scoreboard/test_fetch_json_log_levels.py	exit 0	9 passed
jellyfin-now-playing/test_content_width.py	exit 0	16 passed
ledmatrix-flights/test_public_api_throttle.py	exit 0	19 passed
ledmatrix-music/test_album_art_prefetch.py	exit 0	7 passed
ledmatrix-music/test_progress_bar_width.py	exit 0	17 passed
odds-ticker/test_display_defers_network.py	exit 0	3 passed
of-the-day/test_text_fitting.py	exit 0	12 passed
football-scoreboard/test_missing_team_logos.py	exit 1	(import error, see note)
soccer-scoreboard/test_favorite_team_diagnostics.py	exit 0	(import error, see note)

The two sports entries are inconclusive on my rig only: its core is on a branch predating src/common/sports_shared, which both import. Not evidence of a defect in them.

So the exposure is 11 files silently uncollected, of which 2 are hiding real failures — 5 failing tests in total. Both are filed separately: the flights one is a stale test double (fake_fetch missing allow_network), the news one belongs to PR #462.

Suggested fix

Classify before running, and use the right runner for each kind:

def _is_script_style(path):
    src = Path(path).read_text(encoding="utf-8", errors="replace")
    has_items = re.search(r"^\s*(def test_|class Test|async def test_)", src, re.M)
    return "__main__" in src and "__name__" in src and not has_items

Scripts keep the 0/2/1 subprocess treatment; the rest go to pytest. A file that is neither — no __main__ guard and no test items — should be reported rather than counted as a pass, since that is the shape this bug hides behind.

I have this implemented for the core's copy of run_plugin_tests.py (the mirror-image bug, ChuckBuilds/LEDMatrix#532: that one hands everything to pytest and collects 0 items from the scripts). Happy to port the same classifier here.

Also worth knowing

This runner only globs test_*.py at each plugin's root. The 97 test files under test/ subdirectories are not discovered by it at all — separate from this bug, but it means the suite is smaller than the file count suggests.

Environment

44 first-party plugins installed, pytest 9.1.1. Found while fixing ChuckBuilds/LEDMatrix#532, which is the same defect pointing the other way.