ci: run cargo-mutants on each pull request diff - #156
Merged
Merged
Conversation
A pull-request-only job runs the mise mutants task with --in-diff against the base, under a one-hour cap, and uploads mutants.out.
cargo mutants copies the tree with reflinks where it can, and a reflinked file gets the default mode, so on such a host the copied bench/stubs lost their execute bit and the unmutated baseline failed. The acceptance suite now copies the stubs into a tempdir and marks them executable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/acceptance.rs: the acceptance suite now copiesbench/stubsinto a tempdir and sets each copy to0755(executable_stubs), andPATHpoints at that copy instead of the checkout'sbench/stubs. Why: cargo-mutants 27.1.0 copies the tree withreflink::reflinkwhere the filesystem supports it (src/copy_tree.rs), and a reflinked file is created with the default mode, so the copied stubs lost their execute bit. On a reflink filesystem (reproduced on XFS) the unmutated baseline failed with 9 of 10 acceptance tests failing, andcargo test failed in an unmutated tree, so no mutants were tested. The suite no longer depends on the checkout's file modes at all..github/workflows/ci.yml: a new pull-request-onlymutantsjob. It diffs the merge commit against its first parent (the base as it stands now), runsmise run mutants --in-diff pr.diffundertimeout-minutes: 60, and uploadsmutants.outas themutants-outartifact even when the job fails. It provisions zsh, the user-namespace setup, bun and the pinned uv the same waycheckdoes, because the baseline runs the whole suite.MISE_AUTO_INSTALL=falsewithinstall_args: cargo:cargo-mutantskeeps it from building the rest of the tool table. A missed or timed-out mutant fails the job. A diff with no Rust source changes passes (Diff changes no Rust source files).mise.tomland.cargo/mutants.tomlare unchanged. Themutantstask already passes extra arguments through (raw_args = true), and cargo-mutants has no config key or flag that turns reflink off or keeps file modes (checked against the 27.1.0Configstruct), so a config file had nothing to hold.Validation
cargo mutants --file src/version.rs --re 'replace format_version -> String with String::new'on master@8d9fcdf (reflink XFS host):ERROR cargo test failed in an unmutated tree, 9 acceptance tests failed. This reproduces the issue.mise run mutants --file src/version.rs --re 'replace format_version -> String with String::new'at this branch:ok Unmutated baseline in 62s build + 24s test,1 mutant tested in 2m: 1 caughtcargo test --test acceptance:10 passed; 0 failedgit diff HEAD^1 HEAD > pr.diff && mise run mutants --in-diff pr.diff(a workflow-only diff):Diff changes no Rust source files, exits 0mise run actionlint: clean.mise run line-check origin/master:no file newly over 3743 linesmise run format,mise run lint: clean. The pre-push hook (format-check, clippy, tests, coverage) passed on push.Coverage gaps
mutantsjob finds no mutants and exits before the baseline. The job's baseline and mutant run on GitHub's runner are first exercised by a later pull request that touchessrc/. The baseline was shown passing locally through the same mise task.executable_stubsis test scaffolding. The acceptance tests that run the stubs reach it, and nothing tests it separately.Risks and rollout
Issue
Closes #130
Decisions taken
Mutation scope in CI
Taken: diff-scoped on pull requests with a time cap (accepted unrebutted)
Rejected: the whole crate (4036 mutants) per run - far beyond a CI budget
Reverses: drop the CI job
Where the execute-bit fix lands
Taken: the acceptance suite runs the stubs from a tempdir copy it marks executable (
tests/acceptance.rs), so it does not depend on the checkout's file modesRejected:
--in-placein the mise task. It mutates the developer's own source tree, a crash can leave a mutation behind, and it rules out--jobs. Also rejected: a.cargo/mutants.tomlsetting or a task flag, because cargo-mutants 27.1.0 has none that disables reflink or keeps modes. Also rejected: pointingTMPDIRat another filesystem to force a plain copy, because no such filesystem is guaranteed to exist.Reverses: revert commit 6edec17
Note: this path is outside the planned manifest (
.cargo/mutants.toml,.github/workflows/ci.yml,mise.toml). It is the test that depends on the modes, so the fix lands there.Which base the diff is taken against
Taken:
git diff HEAD^1 HEADon the pull request's merge commit (checkoutfetch-depth: 2), which is the pull request's own change against the base as it stands nowRejected:
github.event.pull_request.base.sha. That is the base when the event fired, so once master moves the diff would also carry other people's changes.Reverses: replace the
Diff against the basestep's commandWall-clock cap
Taken:
timeout-minutes: 60on the jobRejected: the six-hour default, which is no cap. Also rejected: a tighter cap. The baseline alone costs about 1.5 minutes of build and test locally, plus building cargo-mutants on a cold cache, before any mutant runs.
Reverses: change
timeout-minutesThis plan was not approved by a human. The run was declared unattended, and each fork above was decided under that run's authority.