Skip to content

fix: run the suite against the installed wheel, not the source tree - #159

Merged
JarbasAl merged 1 commit into
devfrom
ci/build-tests-runs-the-wheel
Sep 26, 2026
Merged

JarbasAl merged 1 commit into
devfrom
ci/build-tests-runs-the-wheel

Conversation

@openvoiceos-bot

@openvoiceos-bot openvoiceos-bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

The defect

The Run Tests step ran python -m pytest. The -m form puts the working
directory first on sys.path, and the working directory is the repository root.
For a flat-layout package the root holds the package directory, so import mypkg
resolved to the source tree and never to site-packages. The job built a wheel,
installed it, and then tested the checkout instead. A packaging defect passed in
every caller: a module left out of the wheel, a dependency not declared, package
data not shipped.

The proof

A control package whose wheel contains no code at all:

$ python -c "import zipfile; print(zipfile.ZipFile('dist/ctlpkg-0.0.1-py3-none-any.whl').namelist())"
['ctlpkg-0.0.1.dist-info/METADATA', 'ctlpkg-0.0.1.dist-info/WHEEL',
 'ctlpkg-0.0.1.dist-info/top_level.txt', 'ctlpkg-0.0.1.dist-info/RECORD']
$ ls .venv/lib/python3.12/site-packages/ | grep ctl
ctlpkg-0.0.1.dist-info          # the dist-info only, no module

The old command and the new one, same tree, same venv:

$ python -m pytest tests -v      # the command this workflow ran
tests/test_ctl.py::test_answer PASSED      1 passed        <- the defect
$ pytest tests -q                # the command it runs now
FAILED tests/test_ctl.py::test_answer - ModuleNotFoundError    1 failed

The reverse control, a correctly packaged repository with a test that asserts
where the import came from:

$ pytest tests -v
tests/test_ok.py::test_answer PASSED
tests/test_ok.py::test_imported_from_site_packages PASSED
tests/test_ok.py::test_relative_cwd_fixture_still_readable PASSED    3 passed
$ python -m pytest tests -q      # the old command, same repository
FAILED tests/test_ok.py::test_imported_from_site_packages     1 failed, 2 passed

The fix

One token: pytest, not python -m pytest. The console script adds no working
directory entry.

The working directory is deliberately not changed. A suite that reads a
repo-relative fixture path keeps working, and prepend import mode still puts the
test directory on sys.path, so a helper module beside the tests still resolves.
Both are controls in the new test file.

--import-mode=importlib was measured and is not needed: dropping -m is
sufficient, and importlib mode would stop a tests-local helper import from
resolving.

Blast radius, counted

callers of build-tests.yml in the checked-out clones:      192
of those, with a test_path set:                            187
of those, flat layout (a package directory at the root):   171   <- were affected

A screen for callers whose tests import a root-level name that the packaging
config never names, which is where a new red is possible:

screened: 187 in, 187 out
candidates: 4
  ml/voiceclonnx           tests   imports: conversion
  ml/wt-spike              test    imports: benchmarks
  ovos/ovos-skill-moon-game test   imports: __init__
  utils/powerguess         test    imports: train

The screen is a screen: a name that appears anywhere in the packaging config
counts as declared, so it can miss a case and it can raise a false one. It is
not a substitute for the sweep.

A limit worth naming

48 of the 182 caller test directories call sys.path themselves. Where a
conftest re-adds the repository root, this fix is defeated and the suite still
tests the source tree. Those repositories need the sys.path line removed, one
at a time, and that is separate work.

Canary runs

Three callers were pointed at this branch. One produced the evidence; the other
two are red for reasons that predate this change, and both are named below as
fleet work, not as a result of this one.

OpenVoiceOS/nebulento#57, run 36218046118: green on all five versions.
The executed step, read from the log of job 108337853495:

# ... the comment this change adds ...
pytest test/ -v  | tee pytest.log
...
rootdir: /home/runner/work/nebulento/nebulento
======================= 178 passed, 5 warnings in 23.05s =======================

The new command form ran, the rootdir is still the repository root, and a real
178-test suite stayed green. nebulento is flat layout and carries a second root
package (benchmark), so it is the shape most at risk.

TigreGotico/linux2mqtt#15: no evidence, the repository has no working CI.
7 of its 9 workflows that call a gh-automations reusable end in
startup_failure, and the 2 that call none (docker.yml,
conventional-label.yml) run. The same is true on its other branches, so it
predates this change. The cause is not this workflow: the repository default is
read and those 7 callers carry no permissions: block, so the forge refuses
the run before any job starts. Diagnosed and fixed in linux2mqtt#16.

VoiceGamez/ovos-skill-moon-game#21: no evidence, red before the tests.
The Install Built Package step fails to build fann2==1.0.7 (ld: cannot find -ldoublefann) on all five versions, so no job reaches Run Tests. Its
build-tests has failed this way since 2026-06. Filed separately.

That leaves the __init__ prediction in the screen above unverified by a
run
: moon-game was the one repository that could have shown it, and it never
gets far enough to import anything.

Local test result

$ pytest test/test_build_tests_runs_the_wheel.py -v      6 passed
$ pytest test -q          779 passed, 6 failed
$ git stash && pytest test/test_lint_actionlint.py test/test_scripts.py -q
                          146 passed, 6 failed        # the same 6, on dev

The 6 are pre-existing and local: no actionlint binary and no
ovos-plugin-manager in this environment.

Mutation check: restoring python -m makes
test_the_workflow_does_not_invoke_pytest_through_dash_m fail, so the guard is
not a silent pass.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 5b36a47a-9b6b-483d-938d-1cffcb763155


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

The Run Tests step called `python -m pytest`. The -m form puts the working
directory first on sys.path, and the working directory is the repository root.
For a flat-layout package the root holds the package directory, so the suite
imported the source tree and never the wheel the job had just built and
installed. A packaging defect passed: a module left out of the wheel, a
dependency not declared, package data not shipped.

Use the console script, which adds no such sys.path entry. The working
directory stays the repository root, so a suite that reads a repo-relative
fixture path is unaffected, and prepend import mode still puts the test
directory on sys.path, so a helper module beside the tests still resolves.

The console script is necessary and not sufficient, and the comment above the
command now says so. Prepend import mode inserts the first directory above the
test file that is not a package, so a test/__init__.py or a conftest.py at the
repository root makes that directory the repository root again and the
shadowing returns. Of 413 flat-layout callers across the five orgs, 228 get the
fix and 185 keep testing their checkout; the remedy there is per repository,
either removing the test package's __init__.py or `--import-mode=importlib` in
pytest_args. T-5492 carries that set, and T-5337 the root-conftest part of it.
The comment also notes that the console script resolves through PATH where the
-m form bound the interpreter, which matters only if a pre_test_command ever
activates a venv.

test_build_tests_runs_the_wheel.py reads the workflow and adds eight controls
that run pytest over a fixture whose installed copy and source copy disagree,
one per property: the old command imported the source, the new command imports
the installed copy, a tests-local helper import still resolves, a
repo-relative fixture path still resolves, a test package puts the rootdir
back, a root conftest.py does the same, and --import-mode=importlib holds in
both of those shapes. The two shape controls assert the failure names the
source copy, so a pytest that fails to start cannot pass them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JarbasAl
JarbasAl force-pushed the ci/build-tests-runs-the-wheel branch from 32e3751 to a429cee Compare September 26, 2026 08:40
@JarbasAl
JarbasAl marked this pull request as ready for review September 26, 2026 10:20
@JarbasAl
JarbasAl merged commit dae3c9c into dev Sep 26, 2026
11 checks passed
@JarbasAl
JarbasAl deleted the ci/build-tests-runs-the-wheel branch September 26, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants