Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ depend on pytest-xdist or implement xdist's flags or fixtures.
```bash
pip install throngtest
pytest
pytest --throngtest-distribution=files
pytest --isolates=2 --throngtest-backend=local
pytest --distribution=files
pytest --isolates=2 --backend=local
```

Installing the plugin enables distribution into up to four isolates by default.
Expand All @@ -38,7 +38,7 @@ Python 3.8+ and pytest 8.3.5–9.x are supported.
All throngtest settings are loaded, converted and validated through
[skelet](https://github.com/mutating/skelet). Sources have this precedence:

1. Explicit `--throngtest-*` arguments, including arguments supplied by pytest's
1. Explicit CLI options listed below, including arguments supplied by pytest's
`addopts` or `PYTEST_ADDOPTS`.
2. Environment variables with the `THRONGTEST_` prefix.
3. `[tool.throngtest]` in `pyproject.toml` at pytest's `rootdir`.
Expand All @@ -55,28 +55,28 @@ preparation = ["python scripts/prepare.py", "python scripts/seed_test_data.py"]

```bash
THRONGTEST_WORKERS=4 pytest
pytest --isolates=2 --throngtest-exclude='[".git/", ".venv/", "large-data/"]'
pytest --isolates=2 --exclude='[".git/", ".venv/", "large-data/"]'
```

| Setting | CLI option | Default | Meaning |
| --- | --- | --- | --- |
| `workers` | `--isolates` | `4` | Maximum number of nonempty test subsets; a nonnegative integer. `0` disables distribution. |
| `check_fingerprints` | `--throngtest-check-fingerprints` | `false` | Require identical ordered collections in the controller and isolates. |
| `backend` | `--throngtest-backend` | `temporary_directory` | Name of an installed throng plugin. |
| `distribution` | `--throngtest-distribution` | `tests` | Split individual tests or keep each file together (`files`). |
| `python` | `--throngtest-python` | Controller's `sys.executable` | Python executable available inside each isolate. |
| `exclude` | `--throngtest-exclude` | See below | Throng snapshot exclusion patterns; a JSON array for CLI/environment sources and an array in TOML. |
| `preparation` | `--throngtest-preparation` | `[]` | Ordered list of nonempty commands run once in each isolate before pytest; JSON for CLI/environment sources and an array in TOML. |
| `check_fingerprints` | `--check-fingerprints` | `false` | Require identical ordered collections in the controller and isolates. |
| `backend` | `--backend` | `temporary_directory` | Name of an installed throng plugin. |
| `distribution` | `--distribution` | `tests` | Split individual tests or keep each file together (`files`). |
| `python` | `--python` | Controller's `sys.executable` | Python executable available inside each isolate. |
| `exclude` | `--exclude` | See below | Throng snapshot exclusion patterns; a JSON array for CLI/environment sources and an array in TOML. |
| `preparation` | `--preparation` | `[]` | Ordered list of nonempty commands run once in each isolate before pytest; JSON for CLI/environment sources and an array in TOML. |

The default exclusions are `.git/`, `.venv/`, `venv/`, `__pycache__/`,
`.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`, `build/`, `dist/`, and `mutants/`.
An explicit exclusion list replaces the defaults. Patterns are interpreted by
throng. Configurations are read afresh for each pytest session.

Fingerprint checks are disabled by default. Enable them with
`--throngtest-check-fingerprints`, `THRONGTEST_CHECK_FINGERPRINTS=true`, or
`--check-fingerprints`, `THRONGTEST_CHECK_FINGERPRINTS=true`, or
`check_fingerprints = true` in `[tool.throngtest]`. Use
`--throngtest-no-check-fingerprints` to override an enabled setting from the
`--no-check-fingerprints` to override an enabled setting from the
environment or TOML. Both CLI flags take no value; if both are supplied, the
last flag wins. The environment accepts `true`/`false`; TOML uses booleans.

Expand All @@ -86,7 +86,7 @@ Use `preparation` to generate files, install dependencies, or otherwise prepare
each isolate before its pytest process starts:

```bash
pytest --throngtest-preparation='["python scripts/prepare.py"]'
pytest --preparation='["python scripts/prepare.py"]'
THRONGTEST_PREPARATION='["python scripts/prepare.py"]' pytest
```

Expand Down Expand Up @@ -223,7 +223,7 @@ budget. Xdist's `worker_id` values, such as `gw0`, are local to each isolate.

The supported schedulers are `load`, `loadfile`, `loadscope`, `loadgroup`, and
`worksteal`. Their grouping guarantees apply within each assigned subset.
For example, use `--throngtest-distribution=files --dist=loadfile` to keep a
For example, use `--distribution=files --dist=loadfile` to keep a
file together at both levels. Xdist groups do not combine tests from different
isolates. `--dist=each` is rejected because it intentionally repeats tests.
Explicit `--tx`/`--px` execution environments and `--looponfail` are unsupported;
Expand All @@ -245,7 +245,7 @@ throng backends must provide stdout and a process return code. The selected
interpreter must have throngtest, pytest, the project's dependencies, and
required pytest plugins available by the end of preparation. Throngtest only
installs packages when explicitly instructed through preparation commands.
For a remote backend, set `--throngtest-python` to an interpreter available in
For a remote backend, set `--python` to an interpreter available in
the isolate, such as `python`; its default is the controller's absolute
`sys.executable` path, which is usually absent on a remote machine.

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "throngtest"
version = "0.0.1"
version = "0.0.2"
authors = [
{ name="Evgeniy Blinov", email="zheni-b@yandex.ru" },
]
Expand Down
4 changes: 2 additions & 2 deletions tests/test_coverage_agents.py
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ def test_other(): assert classify(0) == 'other'
environment.pop('COVERAGE_PROCESS_START', None)
environment.pop('COVERAGE_FILE', None)
arguments = [sys.executable, '-m', 'coverage', 'run', '-m', 'pytest'] if provider == 'coverage' else [sys.executable, '-m', 'pytest', '--cov=app', '--cov-branch', '--cov-fail-under=100']
result = subprocess.run([*arguments, '-q', '--isolates=2', f'--throngtest-backend={backend}'], cwd=pytester.path, env=environment, text=True, capture_output=True, check=False)
result = subprocess.run([*arguments, '-q', '--isolates=2', f'--backend={backend}'], cwd=pytester.path, env=environment, text=True, capture_output=True, check=False)
assert result.returncode == 0, result.stdout + result.stderr
if provider == 'coverage':
combined = subprocess.run([sys.executable, '-m', 'coverage', 'combine', '-q'], cwd=pytester.path, env=environment, text=True, capture_output=True, check=False)
Expand Down Expand Up @@ -568,7 +568,7 @@ def test_other(): assert classify(0) == 'other'
environment.pop('COVERAGE_PROCESS_START', None)
environment.pop('COVERAGE_FILE', None)
result = subprocess.run([
sys.executable, '-m', 'pytest', '-q', '--isolates=2', '--throngtest-backend=temporary_directory',
sys.executable, '-m', 'pytest', '-q', '--isolates=2', '--backend=temporary_directory',
'-n', '2', '--cov=app', '--cov-branch', '--cov-fail-under=100',
], cwd=pytester.path, env=environment, text=True, capture_output=True, check=False)
assert result.returncode == 0, result.stdout + result.stderr
Expand Down
12 changes: 6 additions & 6 deletions tests/test_fingerprints.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ def test_item(index):
assert not marker.exists()
marker.touch()
''')
arguments = ['--isolates=2', f'--throngtest-backend={backend}', '--junitxml=results.xml']
arguments = ['--isolates=2', f'--backend={backend}', '--junitxml=results.xml']
if source == 'cli':
arguments.append('--throngtest-check-fingerprints' if checked else '--throngtest-no-check-fingerprints')
arguments.append('--check-fingerprints' if checked else '--no-check-fingerprints')
elif source == 'environment':
monkeypatch.setenv('THRONGTEST_CHECK_FINGERPRINTS', str(checked).lower())
elif source == 'toml':
Expand Down Expand Up @@ -70,7 +70,7 @@ def test_fingerprint_flag_before_an_absolute_test_path(pytester: pytest.Pytester
child.mkdir()
(child / 'keep.txt').write_text('Keep the invocation directory in the isolate snapshot.')
monkeypatch.chdir(child)
flag = '--throngtest-check-fingerprints' if checked else '--throngtest-no-check-fingerprints'
flag = '--check-fingerprints' if checked else '--no-check-fingerprints'
pytester.runpytest_subprocess(flag, str(test), timeout=30).assert_outcomes(passed=1)


Expand Down Expand Up @@ -114,7 +114,7 @@ def pytest_collection_modifyitems(config, items):
items[:] = items[:count]
''')
pytester.makepyfile(**{f'test_{index}': 'def test_ok(): pass' for index in range(4)})
result = pytester.runpytest_subprocess(f'--throngtest-backend={backend}', f'--throngtest-distribution={distribution}', timeout=30)
result = pytester.runpytest_subprocess(f'--backend={backend}', f'--distribution={distribution}', timeout=30)
result.assert_outcomes(passed=worker_count)
assert result.ret == (pytest.ExitCode.OK if worker_count else pytest.ExitCode.NO_TESTS_COLLECTED)

Expand All @@ -123,5 +123,5 @@ def pytest_collection_modifyitems(config, items):
def test_duplicate_identifiers_remain_valid_in_both_modes(pytester: pytest.Pytester, backend: str, checked: bool) -> None:
"""Preserve duplicate test occurrences with fingerprint checking enabled or disabled."""
test = pytester.makepyfile('def test_ok(): pass')
flag = '--throngtest-check-fingerprints' if checked else '--throngtest-no-check-fingerprints'
pytester.runpytest_subprocess(f'--throngtest-backend={backend}', flag, '--keep-duplicates', str(test), str(test), timeout=30).assert_outcomes(passed=2)
flag = '--check-fingerprints' if checked else '--no-check-fingerprints'
pytester.runpytest_subprocess(f'--backend={backend}', flag, '--keep-duplicates', str(test), str(test), timeout=30).assert_outcomes(passed=2)
16 changes: 8 additions & 8 deletions tests/test_integration.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@


def run(pytester: pytest.Pytester, backend: str, *arguments: str) -> pytest.RunResult:
return pytester.runpytest_subprocess('--isolates=2', f'--throngtest-backend={backend}', *arguments, timeout=30)
return pytester.runpytest_subprocess('--isolates=2', f'--backend={backend}', *arguments, timeout=30)


@pytest.mark.parametrize(('arguments', 'workers'), [((), 4), (('--isolates=2',), 2), (('--isolates', '2'), 2), (('--throngtest-check-fingerprints',), 4)])
@pytest.mark.parametrize(('arguments', 'workers'), [((), 4), (('--isolates=2',), 2), (('--isolates', '2'), 2), (('--check-fingerprints',), 4)])
def test_real_isolates_execute_every_test_once(pytester: pytest.Pytester, backend: str, tmp_path: Path, arguments: tuple, workers: int) -> None:
"""Execute every test once through the selected throng backend and isolate count.

Expand Down Expand Up @@ -46,7 +46,7 @@ def test_item(index):
target.write_text(json.dumps({{'pid': os.getpid(), 'cwd': str(Path.cwd())}}))
Path('isolate-marker').write_text('written')
''')
result = pytester.runpytest_subprocess(f'--throngtest-backend={backend}', *arguments, timeout=30)
result = pytester.runpytest_subprocess(f'--backend={backend}', *arguments, timeout=30)
result.assert_outcomes(passed=8)
records = [json.loads((tmp_path / str(index)).read_text()) for index in range(8)]
dispatched = [path.read_text() for path in tmp_path.glob('dispatch-*')]
Expand Down Expand Up @@ -109,7 +109,7 @@ def test_item(resource, index):
assert resource == 42
'''
pytester.makepyfile(test_a=source, test_b=source)
run(pytester, backend, '--throngtest-distribution=files').assert_outcomes(passed=8)
run(pytester, backend, '--distribution=files').assert_outcomes(passed=8)
assert (tmp_path / 'test_a.fixture').read_text() != (tmp_path / 'test_b.fixture').read_text()
assert len(list(tmp_path.glob('*.done'))) == 2

Expand Down Expand Up @@ -246,7 +246,7 @@ def test_collection_mismatch(pytester: pytest.Pytester, backend: str, tmp_path:
@pytest.mark.parametrize('value', values)
def test_item(value): raise AssertionError('test must not execute')
''')
result = run(pytester, backend, '--throngtest-check-fingerprints')
result = run(pytester, backend, '--check-fingerprints')
assert result.ret == pytest.ExitCode.INTERNAL_ERROR
output = result.stdout.str() + result.stderr.str()
assert 'collection differs' in output
Expand Down Expand Up @@ -277,7 +277,7 @@ def test_collection_diagnostic_for_absolute_parameter_path(pytester: pytest.Pyte
@pytest.mark.parametrize('value', [{path_expression}])
def test_item(value): raise AssertionError('test must not execute')
''')
result = run(pytester, 'temporary_directory', '--throngtest-check-fingerprints')
result = run(pytester, 'temporary_directory', '--check-fingerprints')
assert result.ret == pytest.ExitCode.INTERNAL_ERROR
output = result.stdout.str() + result.stderr.str()
assert 'controller: 1 selected tests\nisolate: 1 selected tests' in output
Expand Down Expand Up @@ -353,7 +353,7 @@ def test_invocation_directory_and_absolute_nodeid(pytester: pytest.Pytester, bac
def test_missing_interpreter(pytester: pytest.Pytester, backend: str) -> None:
"""Report a missing configured Python executable as a worker startup failure."""
pytester.makepyfile('def test_ok(): pass')
result = run(pytester, backend, '--throngtest-python=missing-throngtest-python')
result = run(pytester, backend, '--python=missing-throngtest-python')
assert result.ret == pytest.ExitCode.INTERNAL_ERROR
assert 'worker terminated without a result' in result.stdout.str() + result.stderr.str()

Expand Down Expand Up @@ -384,7 +384,7 @@ def test_excluded_file_is_not_copied(pytester: pytest.Pytester) -> None:
"""Omit explicitly excluded project files from temporary isolate copies."""
(pytester.path / 'private.txt').write_text('not for the snapshot')
pytester.makepyfile('from pathlib import Path\ndef test_excluded(): assert not Path("private.txt").exists()')
run(pytester, 'temporary_directory', '--throngtest-exclude=["private.txt"]').assert_outcomes(passed=1)
run(pytester, 'temporary_directory', '--exclude=["private.txt"]').assert_outcomes(passed=1)


def test_strict_xpass_is_failure(pytester: pytest.Pytester, backend: str) -> None:
Expand Down
12 changes: 6 additions & 6 deletions tests/test_preparation.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,11 @@ def pytest_sessionstart(session):
def test_prepared(index):
assert (Path.cwd().parent / 'second marker').read_text() == 'prepared'
''')
arguments = ['--isolates=2', f'--throngtest-backend={backend}']
arguments = ['--isolates=2', f'--backend={backend}']
if source == 'cli':
arguments.append('--throngtest-preparation=' + json.dumps(commands))
arguments.append('--preparation=' + json.dumps(commands))
elif source == 'cli_separate':
arguments.extend(['--throngtest-preparation', json.dumps(commands)])
arguments.extend(['--preparation', json.dumps(commands)])
elif source == 'environment':
monkeypatch.setenv('THRONGTEST_PREPARATION', json.dumps(commands))
else:
Expand Down Expand Up @@ -106,7 +106,7 @@ def test_preparation_failure_stops_commands_and_tests(pytester: pytest.Pytester,
python_command(f'from pathlib import Path; Path({str(forbidden)!r}).touch()'),
]
pytester.makepyfile(f'from pathlib import Path\ndef test_never(): Path({str(forbidden)!r}).touch()')
result = pytester.runpytest_subprocess('--isolates=1', f'--throngtest-backend={backend}', '--throngtest-preparation=' + json.dumps(commands), timeout=30)
result = pytester.runpytest_subprocess('--isolates=1', f'--backend={backend}', '--preparation=' + json.dumps(commands), timeout=30)
assert result.ret == pytest.ExitCode.INTERNAL_ERROR
output = result.stdout.str() + result.stderr.str()
assert 'preparation command 2 failed with exit code 7' in output
Expand All @@ -124,7 +124,7 @@ def test_preparation_is_not_run_without_isolates(pytester: pytest.Pytester, tmp_
marker = tmp_path / 'must-not-run'
commands = [python_command(f'from pathlib import Path; Path({str(marker)!r}).touch()')]
pytester.makepyfile('def test_ok(): pass')
result = pytester.runpytest_subprocess(*arguments, '--throngtest-preparation=' + json.dumps(commands), timeout=30)
result = pytester.runpytest_subprocess(*arguments, '--preparation=' + json.dumps(commands), timeout=30)
assert result.ret == exitcode
assert not marker.exists()

Expand Down Expand Up @@ -157,7 +157,7 @@ def test_preparation_failure_cancels_other_preparation(pytester: pytest.Pytester
sys.exit(9)
''')
commands = [shlex.join([sys.executable, script.name])]
result = pytester.runpytest_subprocess('--isolates=2', '--throngtest-preparation=' + json.dumps(commands), timeout=30)
result = pytester.runpytest_subprocess('--isolates=2', '--preparation=' + json.dumps(commands), timeout=30)
assert result.ret == pytest.ExitCode.INTERNAL_ERROR
assert 'preparation command 1 failed with exit code 9' in result.stdout.str() + result.stderr.str()
assert not (tmp_path / 'not-cancelled').exists()
Expand Down
21 changes: 19 additions & 2 deletions tests/test_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from throng import AbstractManager

from throngtest.protocol import Request, WorkerError, encode
from throngtest.runner import Runner, execute, relocate, replay
from throngtest.runner import Runner, execute, relocate, replay, worker_arguments
from throngtest.settings import ARGUMENTS, Settings


Expand All @@ -33,10 +33,27 @@ def test_relocate_paths(tmp_path: Path) -> None:
child.mkdir()
(child / '2').touch()
assert relocate('--isolates=2', root, child, root) == '--isolates=2'
preparation = '--throngtest-preparation=["echo ' + 'long command ' * 100 + '"]'
preparation = '--preparation=["echo ' + 'long command ' * 100 + '"]'
assert relocate(preparation, root, tmp_path, root) == preparation


@pytest.mark.parametrize('option', ['--isolates', '--backend', '--distribution', '--python', '--exclude', '--preparation'])
def test_worker_arguments_preserve_unprefixed_option_values(pytester: pytest.Pytester, tmp_path: Path, option: str) -> None:
"""Keep isolate option values intact with both CLI argument spellings.

A same-named file in the invocation directory would trigger path rewriting
if a value were mistaken for a test path during worker dispatch.
"""
root = tmp_path / 'project'
invocation = root / 'subdirectory'
invocation.mkdir(parents=True)
(invocation / 'value').touch()
config = pytester.parseconfig('--isolates=0')
for arguments in ([option, 'value'], [f'{option}=value']):
config.stash[ARGUMENTS] = arguments
assert worker_arguments(config, root, invocation, Path())[:len(arguments)] == arguments


@pytest.mark.parametrize('stage', ['enter', 'run', 'exit'])
def test_isolate_errors_are_reported_and_cleanup_attempted(stage: str) -> None:
"""Name the failing backend phase and attempt cleanup after scope entry.
Expand Down
Loading
Loading