Add scheduled mutation testing to generated repos#24
Conversation
Generated projects now ship a thin mutation-testing caller workflow pinned to the shared reusable workflows: mutmut for the Python package and, when the Rust extension is enabled, cargo-mutants for the rust_extension crate. The templated pyproject gains the matching [tool.mutmut] configuration, with a note on also_copy for tests that read repository-root paths.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR raises the copier template's minimum supported Python version to 3.12, adds a computed ChangesMutation testing setup
Sequence Diagram(s)sequenceDiagram
participant Scheduler as Cron/Dispatch
participant Workflow as mutation-testing.yml
participant MutmutJob as mutation job
participant CargoJob as mutation-rust job
participant Shared as Shared reusable workflows
Scheduler->>Workflow: trigger (daily cron or manual dispatch)
Workflow->>MutmutJob: run if mutmut_supported
MutmutJob->>Shared: call mutation-mutmut.yml with paths, python_version
Workflow->>CargoJob: run if use_rust
CargoJob->>Shared: call mutation-cargo.yml with sharding, crate dirs
Possibly related PRs
Suggested labels: Poem
Caution Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
❌ Failed checks (1 error, 3 warnings)
✅ Passed checks (16 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 862e80152e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # repository root, so no module prefix is stripped. | ||
| paths: "{{ package_name }}/" | ||
| module-prefix-strip: "" | ||
| python-version: "{{ python_version }}" |
There was a problem hiding this comment.
When the template uses its default python_version (3.10 in copier.yml), this passes 3.10 into the pinned reusable workflow's setup-uv step, which sets UV_PYTHON; the shared workflow then runs uv run --script for mutation_detect_changes.py, whose inline script metadata requires Python >=3.13. With UV_PYTHON=3.10, uv selects/installs 3.10 and the detection step fails before any mutation testing runs, so default-generated repositories get a broken scheduled mutation workflow. Keep the reusable workflow's script runner on 3.13+ or avoid overriding it below the script requirement.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@coderabbitai Has this now been resolved in the latest commit?
Use codegraph analysis to determine your answer.
If this comment is now resolved, please mark it as such using the API. Otherwise, please provide an AI agent prompt for the remaining work to be done to address this comment.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find copier.yml
fd -a copier.yml
echo "---- copier.yml python_version / mutmut_supported ----"
rg -n -A5 -B5 'mutmut_supported|python_version' copier.yml
echo "---- workflow template filename and content ----"
fd -a 'mutation-testing.yml.jinja'
echo "---- full workflow template content ----"
fd -a 'mutation-testing.yml.jinja' --exec cat -n {}Length of output: 1167
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- find workflow files with fd literal ----"
fd -a -H 'mutation-testing'
echo "---- find via find ----"
find . -iname '*mutation-testing*'
echo "---- list workflows dir ----"
find . -path '*workflows*' -iname '*.jinja'Length of output: 969
🏁 Script executed:
#!/bin/bash
set -euo pipefail
cat -n './template/.github/workflows/{% if mutmut_supported or use_rust %}mutation-testing.yml{% endif %}.jinja'Length of output: 3102
@leynos, confirmed via codegraph analysis — this is resolved.
The gating now prevents the broken combination entirely:
copier.ymlcomputesmutmut_supportedaspython_version.split('.')[1] | int >= 13(Line 26-29), awhen: falsederived flag, not a prompt.- The
mutationjob in the workflow template is wrapped in{%- if mutmut_supported %} … {%- endif %}(Line 28-43), and thepython-version: "{{ python_version }}"input (Line 42) only renders inside that block. - The default
python_versionis"3.12"(raised from 3.10, but still< 3.13), somutmut_supportedevaluates tofalseby default and themutationjob — indeed the whole file, per the{% if mutmut_supported or use_rust %}filename guard — does not render for default-baseline repositories. - Whenever a repository does opt into a
python_version >= 3.13,mutmut_supportedflips true and the renderedpython-versionvalue is guaranteed to satisfy the shared workflow'smutation_detect_changes.py>=3.13script requirement.
So the failure mode codex flagged — UV_PYTHON=3.10 breaking the detection step — can no longer occur; the job simply isn't emitted until the baseline interpreter is high enough.
One note: I can't mark this thread resolved via the API myself, since the original comment was posted by chatgpt-codex-connector[bot], not by me — I can only resolve threads I've authored. Go ahead and resolve it manually on your end.
(=^･ω･^=) ﾉ resolved
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
Review found that the caller passed the template's default python-version (3.10) into the shared mutmut workflow, whose helper scripts require 3.13 or greater: setup-uv exported UV_PYTHON job-wide and change detection failed before any mutation testing ran. Address this per maintainer direction: - bump copier.yml's python_version default from 3.10 to 3.12 (the estate baselines libraries for external consumption at 3.12), and update everything derived from the default (test helpers and the requires-python contract assertion); - add a hidden computed mutmut_supported answer (baseline minor >= 13, numeric comparison) and render the mutmut job and the [tool.mutmut] pyproject section only when it holds; - keep the cargo-mutants job gated on use_rust independently, and use a conditional filename so the workflow file is not generated at all when both gates are off; - bump the shared workflow pin to 859416a90eb3987b46a57682c5d6b8964ad3f0a6, which fails fast on sub-3.13 python-version values; - add contract tests covering the four gating combinations. The mutmut job keeps python-version set to the project's baseline, which is now 3.13 or greater by construction whenever the job exists.
leynos
force-pushed
the
add-mutation-testing-template
branch
from
57d3ac7 to
3a34cba
Compare
|
Reworked per review and maintainer direction. Credit to the review for catching that the caller passed the template's default Changes in the latest push:
Validated by rendering all five variants (YAML + actionlint + TOML parses clean) and running the full contract suite: 37 passed, 2 skipped. PR body updated with the new gating semantics. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@copier.yml`:
- Around line 24-29: The mutmut_supported default in copier.yml assumes
python_version always contains a dotted X.Y value, so rendering can fail on
malformed input. Update the computed default to validate or parse python_version
defensively before calling split, and add a copier validator for the
python_version field if needed so the mutmut_supported expression in this
template cannot break on unexpected version formats.
In `@tests/test_template.py`:
- Around line 513-520: Add a narrow Ruff suppression for the boolean positional
parameters in test_generated_mutation_testing_gating by annotating the test
signature with an inline FBT001 waiver and a brief justification, since pytest
parametrization already binds use_rust and expect_mutmut by name. Keep the fix
localized to this test function only and avoid broader lint disables or
signature restructuring.
- Around line 547-553: The test is duplicating the copier invocation instead of
reusing render_project, which risks the two call sites drifting apart. Update
render_project to accept an optional destination/subdirectory argument and have
it call copier.copy with that path internally, then change this test case to
call render_project(tmp_path / target_dir, copier, project_name=...,
package_name=..., use_rust=..., python_version=...) so the shared logic stays in
one place.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f247b32d-cc5a-474d-90ee-c09062f76d79
📒 Files selected for processing (6)
copier.ymltemplate/.github/workflows/{% if mutmut_supported or use_rust %}mutation-testing.yml{% endif %}.jinjatemplate/pyproject.toml.jinjatests/helpers/pyproject_contracts.pytests/helpers/rendering.pytests/test_template.py
| # Computed: mutmut mutation testing needs a base interpreter of 3.13 or | ||
| # greater because the shared workflow's helper scripts require >= 3.13. | ||
| mutmut_supported: | ||
| type: bool | ||
| default: "{{ python_version.split('.')[1] | int >= 13 }}" | ||
| when: false |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: check for validator/choices guarding python_version format.
sed -n '1,30p' copier.ymlRepository: leynos/agent-template-python
Length of output: 822
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the full copier config and any repo-level constraints around python_version.
sed -n '1,120p' copier.yml
printf '\n---\n'
# Search for other mentions of python_version, validators, choices, or related constraints.
rg -n --hidden --glob '!**/.git/**' -e 'python_version' -e 'choices' -e 'validator' -e 'validate' -e 'mutmut_supported' .Repository: leynos/agent-template-python
Length of output: 8588
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the tests that feed python_version into the template.
sed -n '490,610p' tests/test_template.py
printf '\n---\n'
sed -n '40,100p' tests/helpers/rendering.pyRepository: leynos/agent-template-python
Length of output: 5761
Validate python_version before splitting it. python_version.split('.')[1] assumes a dotted X.Y value, so a malformed answer can break rendering. Add a validator or parse the version more defensively before setting mutmut_supported.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@copier.yml` around lines 24 - 29, The mutmut_supported default in copier.yml
assumes python_version always contains a dotted X.Y value, so rendering can fail
on malformed input. Update the computed default to validate or parse
python_version defensively before calling split, and add a copier validator for
the python_version field if needed so the mutmut_supported expression in this
template cannot break on unexpected version formats.
| def test_generated_mutation_testing_gating( | ||
| copier: CopierFixture, | ||
| tmp_path: Path, | ||
| target_dir: str, | ||
| use_rust: bool, | ||
| python_version: str, | ||
| expect_mutmut: bool, | ||
| ) -> None: |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Ruff flags boolean-typed positional parameters (FBT001).
use_rust: bool and expect_mutmut: bool trip FBT001 since the signature permits positional bool args, even though pytest always injects parametrize values by name. A blanket restructure isn't warranted here; add a narrow, justified suppression instead of leaving the lint failing.
🔧 Proposed fix
def test_generated_mutation_testing_gating(
copier: CopierFixture,
tmp_path: Path,
target_dir: str,
- use_rust: bool,
+ use_rust: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword
python_version: str,
- expect_mutmut: bool,
+ expect_mutmut: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword
) -> None:As per coding guidelines, "Only narrow in-line disables (# noqa: XYZ) are permitted, must be accompanied by justification and used only as a last resort."
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def test_generated_mutation_testing_gating( | |
| copier: CopierFixture, | |
| tmp_path: Path, | |
| target_dir: str, | |
| use_rust: bool, | |
| python_version: str, | |
| expect_mutmut: bool, | |
| ) -> None: | |
| def test_generated_mutation_testing_gating( | |
| copier: CopierFixture, | |
| tmp_path: Path, | |
| target_dir: str, | |
| use_rust: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword | |
| python_version: str, | |
| expect_mutmut: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword | |
| ) -> None: |
🧰 Tools
🪛 Ruff (0.15.20)
[warning] 517-517: Boolean-typed positional argument in function definition
(FBT001)
[warning] 519-519: Boolean-typed positional argument in function definition
(FBT001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_template.py` around lines 513 - 520, Add a narrow Ruff suppression
for the boolean positional parameters in test_generated_mutation_testing_gating
by annotating the test signature with an inline FBT001 waiver and a brief
justification, since pytest parametrization already binds use_rust and
expect_mutmut by name. Keep the fix localized to this test function only and
avoid broader lint disables or signature restructuring.
Sources: Path instructions, Linters/SAST tools
| project = copier.copy( | ||
| tmp_path / target_dir, | ||
| project_name="MutationProj", | ||
| package_name="mutation_pkg", | ||
| use_rust=use_rust, | ||
| python_version=python_version, | ||
| ) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Duplicates render_project's copier invocation instead of reusing it.
This manually re-implements render_project's body (project_name/package_name/use_rust/python_version → copier.copy) purely to pass a per-case subdirectory. Extend render_project with an optional destination/subdirectory argument and reuse it here to avoid drift between the two call sites.
♻️ Proposed direction
def render_project(
tmp_path: Path,
copier: CopierFixture,
*,
project_name: str,
package_name: str,
use_rust: bool = False,
python_version: str = "3.12",
+ destination: Path | None = None,
) -> CopierProject:
...
return copier.copy(
- tmp_path,
+ destination or tmp_path,
...
)Then in the new test: render_project(tmp_path / target_dir, copier, project_name=..., ...).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_template.py` around lines 547 - 553, The test is duplicating the
copier invocation instead of reusing render_project, which risks the two call
sites drifting apart. Update render_project to accept an optional
destination/subdirectory argument and have it call copier.copy with that path
internally, then change this test case to call render_project(tmp_path /
target_dir, copier, project_name=..., package_name=..., use_rust=...,
python_version=...) so the shared logic stays in one place.
1 participant
Summary
Rolls the estate-wide scheduled mutation testing pattern into the copier template so newly generated repositories get it by default.
template/.github/workflows/{% if mutmut_supported or use_rust %}mutation-testing.yml{% endif %}.jinja— thin caller of the shared reusable workflows, pinned toleynos/shared-actions@859416a90eb3987b46a57682c5d6b8964ad3f0a6:mutationjob callsmutation-mutmut.ymlagainst the flat package layout (paths: "{{ package_name }}/",module-prefix-strip: "",python-version: "{{ python_version }}").use_rustis enabled, amutation-rustjob callsmutation-cargo.ymlfor the extension crate.template/pyproject.toml.jinja— adds[tool.mutmut]withsource_paths = ["{{ package_name }}/"]andpytest_add_cli_args_test_selection = ["tests/"](norunnerkey; mutmut 3.6 ignores it), plus a comment pointing tests that read repository-root paths atalso_copy/skipif and the caller guide.Caller guides: mutation-mutmut-workflow.md and mutation-cargo-workflow.md.
Gating semantics (rework)
Review caught that the original caller passed the template's default
python-version(3.10) into the shared mutmut workflow: setup-uv exportsUV_PYTHONjob-wide and the workflow's helper scripts require >= 3.13, so change detection failed before any mutation testing ran. Per maintainer direction:python_versiondefault bumped 3.10 → 3.12 — the estate baselines libraries for external consumption at 3.12, applications and internal work at 3.14.requires-pythonand the rufftarget-versionderive from the answer; test helpers and therequires-pythoncontract assertion were updated to match. Generic doc examples mentioning 3.10 were left alone.mutmut_supported(python_versionminor >= 13, numeric comparison) gates both themutationjob and the[tool.mutmut]pyproject section. A 3.12 baseline therefore gets no mutmut mutation testing.use_rustindependently.{% if mutmut_supported or use_rust %}…{% endif %}, copier skips files whose rendered name is empty), so no empty or invalid workflow can ship.859416a90eb3987b46a57682c5d6b8964ad3f0a6, which includes the sub-3.13python-versionfail-fast guard and thesetup-commandsinput.python-version: "{{ python_version }}"is retained on the mutmut job — now always >= 3.13 by construction whenever the job renders.Scheduling
The template default is
30 9 * * *(09:30 UTC). Existing estate repositories occupy the 03:05–09:05 UTC band; the workflow carries a comment advising per-repository staggering, and downstream adopters should adjust the cron when generating a repository.Judgement calls
crate-type = ["cdylib", "rlib"]with in-crate#[cfg(test)]tests, and the generated Makefile already runscargo testfor it. Verified empirically: a rendereduse_rust=trueproject passescargo test --manifest-path rust_extension/Cargo.toml(pyo3 0.29 links cleanly on Linux). The prosidy-darn/stilyagi opt-out precedent therefore does not apply here.extra-crate-dirsrather than rootpaths. Generated repositories have no rootCargo.toml— the sole crate lives inrust_extension/— so the crate is declared viaextra-crate-dirs: "rust_extension"withpaths: "". Scheduled scoped runs bucketrust_extension/src/**changes to that target correctly. (The cargo job's with-block is unchanged by the rework.)mutation_detect_changes.pyunconditionally emits root-target shards forworkflow_dispatchfull runs, and this repository shape has no root crate, so that leg fails (therust_extensionleg still runs;fail-fast: false). The caller setsshard-count: 1to confine this to a single leg and documents it in-line. A shared-actions follow-up allowing an empty root target would remove the wart..yml.jinjawith${{ "{{" }} … {{ "}}" }}escaping matches the existingci.yml.jinja/act-validation.yml.jinjaconvention (copier's default envops; norawblocks in this template).Validation
python_version3.12 (with and withoutuse_rust), 3.13 (both), and 3.14: the workflow is absent for 3.12 pure, cargo-only for 3.12 + Rust, and includes the mutmut job with the correctpython-versioninput for 3.13/3.14;[tool.mutmut]renders only at >= 3.13.actionlint; every renderedpyproject.tomlTOML-parses withrequires-python/ruff target derived correctly.cargo testin the rendered Rust variant: 1 passed.🤖 Generated with Claude Code