Skip to content

Wrap the test helper of every configured test path - #2

Merged
foxbenjaminfox merged 1 commit into
foxbenjaminfox:masterfrom
timeline-holdings:upstream-test-paths
Oct 17, 2026
Merged

foxbenjaminfox merged 1 commit into
foxbenjaminfox:masterfrom
timeline-holdings:upstream-test-paths

Conversation

@miguelcoba

@miguelcoba miguelcoba commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1.

Problem

For a single (non-umbrella) app, Mutare.Sandbox wraps only test/test_helper.exs. A project that sets test_paths: in its mix.exs, for example ["test/unit", "test/integration"], loads those directories' helpers instead, so the coverage probe never runs: it ends with coverage dump unreadable (:enoent), and every covered mutant runs the whole suite. Kills stay correct; only test selection is lost, which on a large suite makes a run far slower.

Solution

  • New option test_paths: in .mutare.exs, mirroring test_paths: in the target's mix.exs. It defaults to ["test"], so nothing changes unless it's set. Mutare reads none of the target's Mix config, so the option mirrors the setting rather than evaluating mix.exs.
  • Sandbox.helper_rels/3 wraps <path>/test_helper.exs for every entry. The umbrella clause is unchanged.
  • Several helpers in one BEAM reuse the safeguards umbrellas already rely on: the ETS tables are created only if missing, the bootstrap keeps its first evaluation, and each after_suite writes the same dump.
  • Docs: the .mutare.exs reference in mix help mutare, a troubleshooting entry for coverage dump unreadable (:enoent), and the CHANGELOG.

Verification

  • New sandbox test: with two configured test paths, both helpers are wrapped and no test/test_helper.exs is created.
  • New end-to-end test in coverage_runner_test: a project with two test_paths: entries gets selection == :tests with the option. Without it (checked once, not committed), the same project gives selection == :suite.
  • Full suite: passes apart from 6 tests that fail the same way on an untouched master on the same machine (CompilerOptionsIntegrationTest ×3, NamespaceRunnerTest, UmbrellaTest, SwapRedundancyTest). Format and Credo are clean; Dialyzer reports the same harvest.ex warning and Dialyxir crash as master.
  • On a project with 10 test_paths: entries, using this branch as a git dependency, every covered mutant gets per-test selection instead of a whole-suite run.

A single app that sets test_paths: in its mix.exs never loads the
test/test_helper.exs Mutare wrapped, so the probe recorded nothing and every
mutant ran the whole suite. The new test_paths: option names those
directories; each one's helper gets the bootstrap. It defaults to ["test"].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@miguelcoba
miguelcoba marked this pull request as ready for review October 14, 2026 15:33
@foxbenjaminfox
foxbenjaminfox merged commit f3d3e2c into foxbenjaminfox:master Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage selection falls back to the whole suite when a project sets custom test_paths:

2 participants