-
Notifications
You must be signed in to change notification settings - Fork 0
Add scheduled mutation testing to generated repos #24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| name: Mutation testing | ||
|
|
||
| # Thin caller of the shared mutation-testing reusable workflows; caller | ||
| # guides: leynos/shared-actions docs/mutation-mutmut-workflow.md and | ||
| # docs/mutation-cargo-workflow.md. | ||
| # Scheduled runs mutate only files changed within the detection window; | ||
| # manual dispatch runs mutate everything. To test a branch, select it in | ||
| # the Actions "Run workflow" control. | ||
|
|
||
| on: | ||
| schedule: | ||
| # Stagger this cron per repository so estate runs do not bunch up; | ||
| # existing repositories occupy the 03:05-09:05 UTC band. | ||
| - cron: "30 9 * * *" # Daily, 09:30 UTC | ||
| workflow_dispatch: | ||
|
|
||
| # Default the token to no scopes; jobs opt in explicitly. | ||
| permissions: {} | ||
|
|
||
| # Serialize runs per ref: a dispatch during a scheduled run (or vice | ||
| # versa) queues rather than racing. Runs are informational, so queued | ||
| # runs wait instead of cancelling. | ||
| concurrency: | ||
| group: mutation-testing-${{ "{{" }} github.ref {{ "}}" }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| {%- if mutmut_supported %} | ||
| mutation: | ||
| permissions: | ||
| contents: read | ||
| id-token: write # OIDC workflow-source resolution | ||
| uses: leynos/shared-actions/.github/workflows/mutation-mutmut.yml@859416a90eb3987b46a57682c5d6b8964ad3f0a6 | ||
| with: | ||
| # Flat package layout: the package directory sits at the | ||
| # repository root, so no module prefix is stripped. | ||
| paths: "{{ package_name }}/" | ||
| module-prefix-strip: "" | ||
| # Always >= 3.13 here: this job renders only when the project's | ||
| # baseline interpreter satisfies the shared workflow's helper | ||
| # scripts. | ||
| python-version: "{{ python_version }}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the template uses its default Useful? React with 👍 / 👎.
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @coderabbitai Has this now been resolved in the latest commit? Use codegraph analysis to determine your answer. If this comment is now resolved, please mark it as such using the API. Otherwise, please provide an AI agent prompt for the remaining work to be done to address this comment. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Find copier.yml
fd -a copier.yml
echo "---- copier.yml python_version / mutmut_supported ----"
rg -n -A5 -B5 'mutmut_supported|python_version' copier.yml
echo "---- workflow template filename and content ----"
fd -a 'mutation-testing.yml.jinja'
echo "---- full workflow template content ----"
fd -a 'mutation-testing.yml.jinja' --exec cat -n {}Length of output: 1167 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "---- find workflow files with fd literal ----"
fd -a -H 'mutation-testing'
echo "---- find via find ----"
find . -iname '*mutation-testing*'
echo "---- list workflows dir ----"
find . -path '*workflows*' -iname '*.jinja'Length of output: 969 🏁 Script executed: #!/bin/bash
set -euo pipefail
cat -n './template/.github/workflows/{% if mutmut_supported or use_rust %}mutation-testing.yml{% endif %}.jinja'Length of output: 3102
The gating now prevents the broken combination entirely:
So the failure mode codex flagged — One note: I can't mark this thread resolved via the API myself, since the original comment was posted by (=^・ω・^=) ノ resolved |
||
| {%- endif %} | ||
| {%- if use_rust %} | ||
|
|
||
| mutation-rust: | ||
| permissions: | ||
| contents: read | ||
| id-token: write # OIDC workflow-source resolution | ||
| uses: leynos/shared-actions/.github/workflows/mutation-cargo.yml@859416a90eb3987b46a57682c5d6b8964ad3f0a6 | ||
| with: | ||
| # The extension crate is the only Cargo target and lives outside | ||
| # the repository root, so it is declared as an extra crate and the | ||
| # root path list is emptied. | ||
| paths: "" | ||
| extra-crate-dirs: "rust_extension" | ||
| # The shared workflow always fans the root target out on manual | ||
| # dispatch, and this repository has no root crate, so that leg | ||
| # fails; keep it to a single shard. Scheduled scoped runs and the | ||
| # rust_extension target are unaffected. | ||
| shard-count: 1 | ||
| {%- endif %} | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -24,7 +24,9 @@ | |||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| from tests.helpers.generated_files import ( | ||||||||||||||||||||||||||||||||||
| parse_toml_file, | ||||||||||||||||||||||||||||||||||
| parse_yaml_mapping, | ||||||||||||||||||||||||||||||||||
| read_generated_text, | ||||||||||||||||||||||||||||||||||
| require_mapping, | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| from tests.helpers.rendering import ( | ||||||||||||||||||||||||||||||||||
| check_generated_import, | ||||||||||||||||||||||||||||||||||
|
|
@@ -494,3 +496,97 @@ def test_generated_github_workflows_match_act_validation_contract( | |||||||||||||||||||||||||||||||||
| package_name=package_name, | ||||||||||||||||||||||||||||||||||
| use_rust=use_rust, | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| MUTATION_WORKFLOW_PIN = "859416a90eb3987b46a57682c5d6b8964ad3f0a6" | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| @pytest.mark.parametrize( | ||||||||||||||||||||||||||||||||||
| ("target_dir", "use_rust", "python_version", "expect_mutmut"), | ||||||||||||||||||||||||||||||||||
| [ | ||||||||||||||||||||||||||||||||||
| ("mutation-312-pure", False, "3.12", False), | ||||||||||||||||||||||||||||||||||
| ("mutation-312-rust", True, "3.12", False), | ||||||||||||||||||||||||||||||||||
| ("mutation-313-pure", False, "3.13", True), | ||||||||||||||||||||||||||||||||||
| ("mutation-314-rust", True, "3.14", True), | ||||||||||||||||||||||||||||||||||
| ], | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| def test_generated_mutation_testing_gating( | ||||||||||||||||||||||||||||||||||
| copier: CopierFixture, | ||||||||||||||||||||||||||||||||||
| tmp_path: Path, | ||||||||||||||||||||||||||||||||||
| target_dir: str, | ||||||||||||||||||||||||||||||||||
| use_rust: bool, | ||||||||||||||||||||||||||||||||||
| python_version: str, | ||||||||||||||||||||||||||||||||||
| expect_mutmut: bool, | ||||||||||||||||||||||||||||||||||
| ) -> None: | ||||||||||||||||||||||||||||||||||
|
Comment on lines
+513
to
+520
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win Ruff flags boolean-typed positional parameters (FBT001).
🔧 Proposed fix def test_generated_mutation_testing_gating(
copier: CopierFixture,
tmp_path: Path,
target_dir: str,
- use_rust: bool,
+ use_rust: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword
python_version: str,
- expect_mutmut: bool,
+ expect_mutmut: bool, # noqa: FBT001 -- pytest.mark.parametrize always passes by keyword
) -> None:As per coding guidelines, "Only narrow in-line disables ( 📝 Committable suggestion
Suggested change
🧰 Tools🪛 Ruff (0.15.20)[warning] 517-517: Boolean-typed positional argument in function definition (FBT001) [warning] 519-519: Boolean-typed positional argument in function definition (FBT001) 🤖 Prompt for AI AgentsSources: Path instructions, Linters/SAST tools |
||||||||||||||||||||||||||||||||||
| """Rendered mutation testing follows the interpreter and Rust gates. | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| Parameters | ||||||||||||||||||||||||||||||||||
| ---------- | ||||||||||||||||||||||||||||||||||
| copier | ||||||||||||||||||||||||||||||||||
| ``pytest-copier`` fixture used to render the template. | ||||||||||||||||||||||||||||||||||
| tmp_path | ||||||||||||||||||||||||||||||||||
| Temporary directory where the rendered project is created. | ||||||||||||||||||||||||||||||||||
| target_dir | ||||||||||||||||||||||||||||||||||
| Temporary project directory name for the rendered variant. | ||||||||||||||||||||||||||||||||||
| use_rust | ||||||||||||||||||||||||||||||||||
| Whether the rendered variant includes the optional Rust extension. | ||||||||||||||||||||||||||||||||||
| python_version | ||||||||||||||||||||||||||||||||||
| Minimum supported Python version answer passed to Copier. | ||||||||||||||||||||||||||||||||||
| expect_mutmut | ||||||||||||||||||||||||||||||||||
| Whether the baseline interpreter supports the mutmut workflow | ||||||||||||||||||||||||||||||||||
| (3.13 or greater). | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| Returns | ||||||||||||||||||||||||||||||||||
| ------- | ||||||||||||||||||||||||||||||||||
| None | ||||||||||||||||||||||||||||||||||
| The test passes when the mutmut job and ``[tool.mutmut]`` section | ||||||||||||||||||||||||||||||||||
| render only for baselines of 3.13 or greater, the cargo-mutants job | ||||||||||||||||||||||||||||||||||
| renders only with the Rust extension, and the workflow file is | ||||||||||||||||||||||||||||||||||
| absent when both gates are off. | ||||||||||||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||||||||||||
| project = copier.copy( | ||||||||||||||||||||||||||||||||||
| tmp_path / target_dir, | ||||||||||||||||||||||||||||||||||
| project_name="MutationProj", | ||||||||||||||||||||||||||||||||||
| package_name="mutation_pkg", | ||||||||||||||||||||||||||||||||||
| use_rust=use_rust, | ||||||||||||||||||||||||||||||||||
| python_version=python_version, | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
|
Comment on lines
+547
to
+553
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win Duplicates This manually re-implements ♻️ Proposed direction def render_project(
tmp_path: Path,
copier: CopierFixture,
*,
project_name: str,
package_name: str,
use_rust: bool = False,
python_version: str = "3.12",
+ destination: Path | None = None,
) -> CopierProject:
...
return copier.copy(
- tmp_path,
+ destination or tmp_path,
...
)Then in the new test: 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||
| pyproject = parse_toml_file(project / "pyproject.toml") | ||||||||||||||||||||||||||||||||||
| mutmut_config = pyproject.get("tool", {}).get("mutmut") | ||||||||||||||||||||||||||||||||||
| if expect_mutmut: | ||||||||||||||||||||||||||||||||||
| assert mutmut_config == { | ||||||||||||||||||||||||||||||||||
| "source_paths": ["mutation_pkg/"], | ||||||||||||||||||||||||||||||||||
| "pytest_add_cli_args_test_selection": ["tests/"], | ||||||||||||||||||||||||||||||||||
| }, "expected mutmut configuration for baselines of 3.13 or greater" | ||||||||||||||||||||||||||||||||||
| else: | ||||||||||||||||||||||||||||||||||
| assert mutmut_config is None, ( | ||||||||||||||||||||||||||||||||||
| "expected no mutmut configuration below a 3.13 baseline" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
| workflow_path = project / ".github" / "workflows" / "mutation-testing.yml" | ||||||||||||||||||||||||||||||||||
| if not expect_mutmut and not use_rust: | ||||||||||||||||||||||||||||||||||
| assert not workflow_path.exists(), ( | ||||||||||||||||||||||||||||||||||
| "expected no mutation workflow when both gates are off" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| return | ||||||||||||||||||||||||||||||||||
| workflow = parse_yaml_mapping( | ||||||||||||||||||||||||||||||||||
| read_generated_text(workflow_path), "mutation workflow" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| jobs = require_mapping(workflow, "jobs", "mutation workflow") | ||||||||||||||||||||||||||||||||||
| assert ("mutation" in jobs) == expect_mutmut, ( | ||||||||||||||||||||||||||||||||||
| "expected the mutmut job only for baselines of 3.13 or greater" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| assert ("mutation-rust" in jobs) == use_rust, ( | ||||||||||||||||||||||||||||||||||
| "expected the cargo-mutants job only for Rust variants" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| if expect_mutmut: | ||||||||||||||||||||||||||||||||||
| mutation_job = require_mapping(jobs, "mutation", "mutation workflow jobs") | ||||||||||||||||||||||||||||||||||
| mutation_inputs = require_mapping(mutation_job, "with", "mutation job") | ||||||||||||||||||||||||||||||||||
| assert mutation_inputs.get("python-version") == python_version, ( | ||||||||||||||||||||||||||||||||||
| "expected the mutmut job to run on the project's baseline Python" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
| for job_name, job in jobs.items(): | ||||||||||||||||||||||||||||||||||
| uses = str(job.get("uses", "")) if isinstance(job, dict) else "" | ||||||||||||||||||||||||||||||||||
| assert uses.endswith(f"@{MUTATION_WORKFLOW_PIN}"), ( | ||||||||||||||||||||||||||||||||||
| f"expected {job_name} to pin the shared mutation workflow" | ||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: leynos/agent-template-python
Length of output: 822
🏁 Script executed:
Repository: leynos/agent-template-python
Length of output: 8588
🏁 Script executed:
Repository: leynos/agent-template-python
Length of output: 5761
Validate
python_versionbefore splitting it.python_version.split('.')[1]assumes a dottedX.Yvalue, so a malformed answer can break rendering. Add a validator or parse the version more defensively before settingmutmut_supported.🤖 Prompt for AI Agents