Conversation
- Added Copilot CLI backend for generating Python unit tests. - Added Gemini CLI backend for generating Python unit tests. - Implemented CLI commands for running tests, reporting, and backend management. - Created context builder for assembling context from source and test files. - Defined models for run requests, patch candidates, and validation results. - Developed orchestrator to manage the test generation workflow. - Implemented patch application with rollback capabilities. - Added reporting functionality with terminal and JSON renderers. - Established validation framework with syntax and pytest validators. - Introduced targeting strategies for selecting files to test. - Summarized validation feedback for improved retry logic.
- Created test modules for various components in the v2 package, including backends, context, patching, reporting, targeting, and validation. - Implemented tests for core models such as RunRequest, TargetSpec, ContextBundle, PatchCandidate, PatchApplication, and ValidationResult. - Developed tests for the V2Orchestrator to ensure proper functionality and error handling. - Added CLI command tests to verify the behavior of the v2 CLI interface.
… to handle artifacts directory
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an initial AIUnitTest v2 “tool-first” execution workflow, including a dry-run mode, run artifact persistence, and a new v2 CLI namespace, alongside documentation and CI/release workflow updates.
Changes:
- Adds AIUnitTest v2 core components (models, orchestrator, targeting, context building, patching, validation, reporting).
- Exposes v2 via a new
ai-unit-test v2 ...CLI surface (run/report/backends/doctor) and wires it into the main CLI. - Adds unit tests and v2 documentation; updates workflows and project metadata.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/v2/init.py | Introduces v2 test package. |
| tests/unit/v2/test_cli.py | Tests for v2 CLI commands (backends/report/doctor). |
| tests/unit/v2/test_models.py | Tests for v2 dataclass models. |
| tests/unit/v2/test_orchestrator.py | Tests for v2 orchestrator behavior including dry-run and retries. |
| tests/unit/v2/backends/init.py | Introduces v2 backends test package. |
| tests/unit/v2/backends/test_backends.py | Tests backend registry and CLI backend adapters. |
| tests/unit/v2/context/init.py | Introduces v2 context test package. |
| tests/unit/v2/context/test_builder.py | Tests file-based context assembly (source/tests/config). |
| tests/unit/v2/patching/init.py | Introduces v2 patching test package. |
| tests/unit/v2/patching/test_workspace.py | Tests patch application guardrails, diffing, rollback, dry-run. |
| tests/unit/v2/reporting/init.py | Introduces v2 reporting test package. |
| tests/unit/v2/reporting/test_reporting.py | Tests artifact persistence and renderers (terminal/JSON). |
| tests/unit/v2/targeting/init.py | Introduces v2 targeting test package. |
| tests/unit/v2/targeting/test_selectors.py | Tests explicit-file target selection behavior. |
| tests/unit/v2/validation/init.py | Introduces v2 validation test package. |
| tests/unit/v2/validation/test_runners.py | Tests syntax/pytest validators and feedback summarization. |
| src/ai_unit_test/v2/init.py | Exposes v2 public API exports. |
| src/ai_unit_test/v2/cli.py | Implements ai-unit-test v2 CLI commands and wiring. |
| src/ai_unit_test/v2/models.py | Defines v2 core dataclasses (request/target/context/report/etc.). |
| src/ai_unit_test/v2/orchestrator.py | Implements retry loop, dry-run flow, validation, reporting integration. |
| src/ai_unit_test/v2/backends/init.py | Exposes backend registry/contracts. |
| src/ai_unit_test/v2/backends/base.py | Defines reasoning backend protocol + registry. |
| src/ai_unit_test/v2/backends/copilot_cli.py | Copilot CLI subprocess adapter and output parsing. |
| src/ai_unit_test/v2/backends/gemini_cli.py | Gemini CLI subprocess adapter and output parsing. |
| src/ai_unit_test/v2/context/init.py | Introduces v2 context subpackage. |
| src/ai_unit_test/v2/context/builder.py | File-based context builder for sources/tests/config. |
| src/ai_unit_test/v2/patching/init.py | Introduces v2 patching subpackage. |
| src/ai_unit_test/v2/patching/workspace.py | Patch apply/rollback logic with test-first guardrails and diffs. |
| src/ai_unit_test/v2/reporting/init.py | Introduces v2 reporting subpackage. |
| src/ai_unit_test/v2/reporting/renderer.py | Terminal + JSON renderers for run reports. |
| src/ai_unit_test/v2/reporting/store.py | Persists run artifacts (report.json, summary.md, optional patch.diff). |
| src/ai_unit_test/v2/targeting/init.py | Introduces v2 targeting subpackage. |
| src/ai_unit_test/v2/targeting/selectors.py | Explicit-file selector for initial v2 targeting. |
| src/ai_unit_test/v2/validation/init.py | Introduces v2 validation subpackage. |
| src/ai_unit_test/v2/validation/feedback.py | Summarizes validator failures for retries. |
| src/ai_unit_test/v2/validation/runners.py | Implements syntax + targeted pytest validators (with log capture). |
| src/ai_unit_test/core/cli_manager.py | Registers the v2 Typer app under the main CLI. |
| README.md | Adds project status note pointing to v2 docs. |
| pyproject.toml | Updates classifiers to include Python 3.14. |
| docs/initial.md | Clarifies this doc is v1 thesis; points to v2 docs. |
| docs/v2/README.md | Adds v2 product positioning and current status. |
| docs/v2/architecture.md | Adds detailed v2 architecture and surfaces description. |
| docs/v2/implementation-plan.md | Adds v2 phased implementation plan and MVP scope. |
| docs/v2/github-issue-v2.md | Adds a draft issue describing v2 redesign goals. |
| .github/workflows/ci.yml | Updates GitHub Actions versions used in CI pipeline. |
| .github/workflows/release.yml | Updates GitHub Actions versions used in release workflow. |
- PatchApplier: fail explicitly on empty/unparseable patches; validate parsed paths (not just touched_files) against guardrails; extract _check_guardrails to reduce complexity - PytestValidator: use sys.executable instead of hardcoded 'python'; treat 'no test files' as failure instead of success - Backends: check returncode != 0 and empty output as runtime errors; copilot doctor uses 'gh copilot --version' for real availability - Orchestrator: catch backend exceptions with structured retry; import TargetSpec directly (fix F821); remove unused variables - CLI: extract report() typer.Option defaults to module-level (fix B008) - Tests: add 9 new tests covering error paths (empty patch, backend errors, sys.executable, orchestrator retry on backend crash) - Lint: add missing docstrings, fix imports, formatting via pre-commit - Docs: wrap long line in README.md; blacken-docs in architecture.md All 259 tests pass. Coverage: 83.44% (threshold: 70%). pre-commit run --all-files passes cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add nosec B404 to subprocess import in runners.py (controlled usage) - Add nosec B603 to subprocess.run call in PytestValidator (safe args) - Replace hardcoded /tmp path in test_models.py with /var/log/app/ - Move subprocess import to module level in test_runners.py with nosec Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
- PatchApplier: use None sentinel for non-existent files in snapshots so rollback correctly preserves pre-existing empty files - Orchestrator: scope feedback to current attempt only (not full history) to avoid stale/duplicated context in retry prompts - CLI: remove unused --last-run parameter from report command - Docs: fix 'backends list' → 'backends' and remove --last-run from architecture.md and implementation-plan.md - Tests: add test_rollback_preserves_empty_files All 260 tests pass. Coverage: 83.43%. pre-commit clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- write pytest validator logs under project-scoped .ai-unit-test/logs instead of leaking temp files in the system temp directory - record timeout output in the validator log file - align release workflow Python version with supported runtime (3.13) - add test coverage for project-scoped validator logs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Description
Implements the complete AIUnitTest v2 MVP — a tool-first test execution layer designed for coding agents. This is a full rewrite of the v2 module, which was previously just a scaffold (models + stub orchestrator), into a working end-to-end pipeline.
What changed
New modules (15 source files):
targeting/selectors.py) —ExplicitFileSelectorresolves file paths toTargetSpeccontext/builder.py) —FileContextBuilderreads source + test files intoContextBundlewith retry feedbackpatching/workspace.py) —PatchApplierwith rollback, test-file guardrails (fnmatch-based), and diff trackingvalidation/runners.py,validation/feedback.py) —SyntaxValidator(py_compile),PytestValidator(subprocess with addopts override),FeedbackSummarizerreporting/store.py,reporting/renderer.py) —RunStorepersists JSON + summary.md + patch.diff;TerminalRendererandJsonRendererbackends/copilot_cli.py,backends/gemini_cli.py) — async subprocess adapters for Copilot CLI and Gemini CLIorchestrator.py) — full loop: target → context → backend → patch → validate → retry → reportcli.py) — typer sub-app withrun,report,backends,doctorcommandsModified files:
cli_manager.py— registers v2 sub-app viaadd_typer()pyproject.toml— new optional dependencies for v2 backendsdocs/v2/README.md— updated current status sectionKey design decisions
fnmatchwith configured patterns, not substring matchingaddoptsto avoid coverage overhead during validationasyncio.create_subprocess_execFixes #51
Type of change
How Has This Been Tested
v2 run --dry-run,v2 backends,v2 doctor,v2 report, error handlinghealth-check, etc.) still work correctlyTest Configuration
--override-ini="addopts="for isolated runspip install -e ".[dev,all]"Checklist