Skip to content

ci: run cargo-mutants on each pull request diff - #156

Merged
justin13888 merged 2 commits into
masterfrom
ci/130-run-mutants-on-pr-diff
Oct 3, 2026
Merged

justin13888 merged 2 commits into
masterfrom
ci/130-run-mutants-on-pr-diff

Conversation

@justin13888

Copy link
Copy Markdown
Contributor

Summary

  • tests/acceptance.rs: the acceptance suite now copies bench/stubs into a tempdir and sets each copy to 0755 (executable_stubs), and PATH points at that copy instead of the checkout's bench/stubs. Why: cargo-mutants 27.1.0 copies the tree with reflink::reflink where the filesystem supports it (src/copy_tree.rs), and a reflinked file is created with the default mode, so the copied stubs lost their execute bit. On a reflink filesystem (reproduced on XFS) the unmutated baseline failed with 9 of 10 acceptance tests failing, and cargo test failed in an unmutated tree, so no mutants were tested. The suite no longer depends on the checkout's file modes at all.
  • .github/workflows/ci.yml: a new pull-request-only mutants job. It diffs the merge commit against its first parent (the base as it stands now), runs mise run mutants --in-diff pr.diff under timeout-minutes: 60, and uploads mutants.out as the mutants-out artifact even when the job fails. It provisions zsh, the user-namespace setup, bun and the pinned uv the same way check does, because the baseline runs the whole suite. MISE_AUTO_INSTALL=false with install_args: cargo:cargo-mutants keeps it from building the rest of the tool table. A missed or timed-out mutant fails the job. A diff with no Rust source changes passes (Diff changes no Rust source files).
  • mise.toml and .cargo/mutants.toml are unchanged. The mutants task already passes extra arguments through (raw_args = true), and cargo-mutants has no config key or flag that turns reflink off or keeps file modes (checked against the 27.1.0 Config struct), so a config file had nothing to hold.

Validation

  • cargo mutants --file src/version.rs --re 'replace format_version -> String with String::new' on master@8d9fcdf (reflink XFS host): ERROR cargo test failed in an unmutated tree, 9 acceptance tests failed. This reproduces the issue.
  • mise run mutants --file src/version.rs --re 'replace format_version -> String with String::new' at this branch: ok Unmutated baseline in 62s build + 24s test, 1 mutant tested in 2m: 1 caught
  • cargo test --test acceptance: 10 passed; 0 failed
  • git diff HEAD^1 HEAD > pr.diff && mise run mutants --in-diff pr.diff (a workflow-only diff): Diff changes no Rust source files, exits 0
  • mise run actionlint: clean. mise run line-check origin/master: no file newly over 3743 lines
  • mise run format, mise run lint: clean. The pre-push hook (format-check, clippy, tests, coverage) passed on push.

Coverage gaps

  • This pull request's own diff changes only a test file and a workflow, so its mutants job finds no mutants and exits before the baseline. The job's baseline and mutant run on GitHub's runner are first exercised by a later pull request that touches src/. The baseline was shown passing locally through the same mise task.
  • executable_stubs is test scaffolding. The acceptance tests that run the stubs reach it, and nothing tests it separately.

Risks and rollout

  • The new job adds up to an hour of runner time to a pull request that changes Rust source. A diff too large to mutate in that time fails the job. That cap is the decision recorded below.
  • The test change only affects where the acceptance suite finds the stubs. bx's behaviour is unchanged.

Issue

Closes #130

Decisions taken

  1. Mutation scope in CI
    Taken: diff-scoped on pull requests with a time cap (accepted unrebutted)
    Rejected: the whole crate (4036 mutants) per run - far beyond a CI budget
    Reverses: drop the CI job

  2. Where the execute-bit fix lands
    Taken: the acceptance suite runs the stubs from a tempdir copy it marks executable (tests/acceptance.rs), so it does not depend on the checkout's file modes
    Rejected: --in-place in the mise task. It mutates the developer's own source tree, a crash can leave a mutation behind, and it rules out --jobs. Also rejected: a .cargo/mutants.toml setting or a task flag, because cargo-mutants 27.1.0 has none that disables reflink or keeps modes. Also rejected: pointing TMPDIR at another filesystem to force a plain copy, because no such filesystem is guaranteed to exist.
    Reverses: revert commit 6edec17
    Note: this path is outside the planned manifest (.cargo/mutants.toml, .github/workflows/ci.yml, mise.toml). It is the test that depends on the modes, so the fix lands there.

  3. Which base the diff is taken against
    Taken: git diff HEAD^1 HEAD on the pull request's merge commit (checkout fetch-depth: 2), which is the pull request's own change against the base as it stands now
    Rejected: github.event.pull_request.base.sha. That is the base when the event fired, so once master moves the diff would also carry other people's changes.
    Reverses: replace the Diff against the base step's command

  4. Wall-clock cap
    Taken: timeout-minutes: 60 on the job
    Rejected: the six-hour default, which is no cap. Also rejected: a tighter cap. The baseline alone costs about 1.5 minutes of build and test locally, plus building cargo-mutants on a cold cache, before any mutant runs.
    Reverses: change timeout-minutes

This plan was not approved by a human. The run was declared unattended, and each fork above was decided under that run's authority.

A pull-request-only job runs the mise mutants task with --in-diff against the base, under a one-hour cap, and uploads mutants.out.
cargo mutants copies the tree with reflinks where it can, and a reflinked file gets the default mode, so on such a host the copied bench/stubs lost their execute bit and the unmutated baseline failed. The acceptance suite now copies the stubs into a tempdir and marks them executable.
@justin13888 justin13888 added the de-slop Repository cleanup: documentation, structure, and test adequacy label Oct 3, 2026
@justin13888
justin13888 merged commit 69252f5 into master Oct 3, 2026
6 checks passed
@justin13888
justin13888 deleted the ci/130-run-mutants-on-pr-diff branch October 3, 2026 14:39
@github-actions github-actions Bot mentioned this pull request Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

de-slop Repository cleanup: documentation, structure, and test adequacy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make mutation testing runnable here and run it on every pull request's diff

1 participant