Make 1.0 a runner-only framework for coding-agent users

[sbroenne]

Description

Coding-agent users already have an assistant that can investigate failures alongside their source. The framework should execute real Copilot tasks and preserve what happened, rather than make additional model calls to judge answers or generate recommendations.

• Keep CopilotEval, copilot_eval, execution controls, request audits, usage, repetitions, and ordinary pytest outcomes. Consumers own concrete output checks; the current coding agent interprets the evidence.
• Remove AI judging, analysis, refinement, HTML/Markdown renderers, dashboards, rankings, winner selection, the report command, and their dependencies, demos, fixtures, snapshots, and screenshots.
• Preserve lossless schema-4.0 JSON evidence with atomic saves and visible persistence errors. Unsupported property values fail instead of silently becoming strings; missing and empty image data remain distinct.
• Add the companion skill distributed through npx skills, real-Copilot coverage of test authoring and source-backed investigation, and updated setup, documentation, workflows, and contributor instructions.

Breaking migration: prepares version 1.0.0. Removed APIs and flags are not retained as aliases, and older JSON must be regenerated by its producer. load_suite_report returns SuiteReport directly. Session completion is not task correctness; A/B records share the enclosing pytest outcome and do not establish independent pass rates or a winner. Non-AI case loading and explicit grading export remain available.

Rebased onto current main, preserving its WebSocket lifecycle and exact-budget completion fixes. Historical changelog entries and migration examples remain intentionally. No release was tagged or published.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• I ran the deterministic checks required for this change
• I ran the relevant real-Copilot integration file when behavior changed
• I added contract or integration coverage that proves the outcome
• I updated user and contributor documentation where needed
• I did not include credentials or unsanitized private prompts

Testing

Latest checks after rebasing and legacy cleanup:

uv run python -m pytest "tests\contracts" -q -o addopts=''
Passed: 98 offline framework/evidence contracts. These do not establish model performance.

uv run python -m pytest "tests\integration\copilot\test_01_basic.py" -q
Passed: 2 real-Copilot integration cases.

uv run ruff check src tests examples
Passed.

uv run ruff format --check src tests examples
Passed: 131 files already formatted.

uv run pyright
Passed: 0 errors, 0 warnings, 0 information messages.

uv run python -X utf8 -m mkdocs build --strict --quiet
Passed.

uv build --quiet
Passed: wheel and source distribution built for 1.0.0.

uv lock --check
Passed.

git --no-pager diff --check
Passed after normalizing the edited ignore file.

Additional acceptance checks verified isolated installed-package evidence replay without Copilot credentials, local npx skills installation with all three files matching source, and the rebuilt wheel/source archives containing no retired modules, renderer assets, dependencies, or report entry point. The full paid integration suite was not rerun.

Related Issues

N/A

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details

Package	Version	Score	Details
pip/pytest-skill-engineering	>= 1.0.0,< 2	Unknown	Unknown
pip/pytest-skill-engineering	>= 1.0.0,< 2	Unknown	Unknown
pip/annotated-types	0.8.0	Unknown	Unknown
pip/anyio	4.15.1	Unknown	Unknown
pip/attrs	26.1.0	Unknown	Unknown
pip/certifi	2026.7.22	🟢 5.9	Details Check Score Reason Maintained 🟢 6 7 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 6 Security-Policy 🟢 10 security policy file detected Code-Review 🟢 5 Found 2/4 approved changesets -- score normalized to 5 Binary-Artifacts 🟢 10 no binaries found in the repo Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Pinned-Dependencies 🟢 5 dependency not pinned by hash detected -- score normalized to 5 Token-Permissions 🟢 10 GitHub workflow tokens follow principle of least privilege CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing ⚠️ 0 project is not fuzzed License 🟢 9 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches Packaging ⚠️ -1 packaging workflow not detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/cffi	2.1.1	Unknown	Unknown
pip/click	8.5.0	Unknown	Unknown
pip/colorama	0.4.6	Unknown	Unknown
pip/cryptography	50.0.2	Unknown	Unknown
pip/github-copilot-sdk	1.0.16	Unknown	Unknown
pip/h11	0.16.0	🟢 4.4	Details Check Score Reason Code-Review 🟢 5 Found 9/18 approved changesets -- score normalized to 5 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Binary-Artifacts 🟢 10 no binaries found in the repo Packaging ⚠️ -1 packaging workflow not detected Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Security-Policy ⚠️ 0 security policy file not detected Fuzzing 🟢 10 project is fuzzed Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/httpcore	1.0.9	Unknown	Unknown
pip/httpcore2	2.13.1	Unknown	Unknown
pip/httpx	0.28.1	Unknown	Unknown
pip/httpx2	2.13.1	Unknown	Unknown
pip/httpx2-jsfetch	1.0	Unknown	Unknown
pip/idna	3.20	Unknown	Unknown
pip/iniconfig	2.3.0	Unknown	Unknown
pip/jsonschema	4.26.0	Unknown	Unknown
pip/jsonschema-specifications	2025.9.1	Unknown	Unknown
pip/mcp	2.2.0	Unknown	Unknown
pip/mcp-types	2.2.0	Unknown	Unknown
pip/opentelemetry-api	1.45.0	Unknown	Unknown
pip/outcome	1.3.0.post0	🟢 3.9	Details Check Score Reason Binary-Artifacts 🟢 10 no binaries found in the repo Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Code-Review 🟢 7 Found 7/9 approved changesets -- score normalized to 7 Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Packaging ⚠️ -1 packaging workflow not detected Pinned-Dependencies ⚠️ 1 dependency not pinned by hash detected -- score normalized to 1 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing ⚠️ 0 project is not fuzzed License 🟢 9 license file detected Signed-Releases ⚠️ -1 no releases found Security-Policy ⚠️ 0 security policy file not detected Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/packaging	26.3	Unknown	Unknown
pip/pluggy	1.6.0	Unknown	Unknown
pip/pycparser	3.0	Unknown	Unknown
pip/pydantic	2.13.5	Unknown	Unknown
pip/pydantic-core	2.46.5	🟢 6.5	Details Check Score Reason Code-Review ⚠️ 2 Found 7/29 approved changesets -- score normalized to 2 Maintained 🟢 10 30 commit(s) and 9 issue activity found in the last 90 days -- score normalized to 10 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies 🟢 8 dependency not pinned by hash detected -- score normalized to 8 Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Security-Policy 🟢 10 security policy file detected Branch-Protection 🟢 3 branch protection is not maximal on development and all release branches SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0 Packaging 🟢 10 packaging workflow detected
pip/pygments	2.21.0	Unknown	Unknown
pip/pyjwt	2.15.1	Unknown	Unknown
pip/pytest	9.1.1	Unknown	Unknown
pip/pytest-asyncio	1.4.0	Unknown	Unknown
pip/python-dateutil	2.9.0.post0	🟢 4.9	Details Check Score Reason Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Packaging ⚠️ -1 packaging workflow not detected Maintained 🟢 6 7 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 6 Binary-Artifacts 🟢 10 no binaries found in the repo Code-Review ⚠️ 0 Found 0/15 approved changesets -- score normalized to 0 Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Security-Policy ⚠️ 0 security policy file not detected Fuzzing 🟢 10 project is fuzzed License 🟢 9 license file detected Signed-Releases 🟢 8 2 out of the last 2 releases have a total of 2 signed artifacts. Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/python-frontmatter	1.3.0	Unknown	Unknown
pip/python-multipart	0.0.32	Unknown	Unknown
pip/pywin32	312	Unknown	Unknown
pip/pyyaml	6.0.3	Unknown	Unknown
pip/referencing	0.37.0	Unknown	Unknown
pip/rpds-py	2026.6.3	Unknown	Unknown
pip/six	1.17.0	🟢 3.7	Details Check Score Reason Maintained ⚠️ 0 0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0 Code-Review ⚠️ 2 Found 7/30 approved changesets -- score normalized to 2 Binary-Artifacts 🟢 10 no binaries found in the repo Packaging ⚠️ -1 packaging workflow not detected Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Fuzzing 🟢 10 project is fuzzed Security-Policy ⚠️ 0 security policy file not detected License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ 0 branch protection not enabled on development/release branches SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
pip/sniffio	1.3.1	Unknown	Unknown
pip/sortedcontainers	2.4.0	Unknown	Unknown
pip/sse-starlette	3.5.0	Unknown	Unknown
pip/starlette	1.7.0	Unknown	Unknown
pip/tomlkit	0.15.1	Unknown	Unknown
pip/trio	0.34.0	Unknown	Unknown
pip/truststore	0.10.4	Unknown	Unknown
pip/typing-extensions	4.16.0	Unknown	Unknown
pip/typing-inspection	0.4.4	Unknown	Unknown
pip/uvicorn	0.54.0	Unknown	Unknown
pip/websockets	17.1	Unknown	Unknown
pip/cryptography	50.0.2	Unknown	Unknown
pip/github-copilot-sdk	1.0.16	Unknown	Unknown

Scanned Files

• .github/workflows/hero-tests.yml
• examples/quickstart/pyproject.toml
• examples/skill-dogfood/pyproject.toml
• examples/skill-dogfood/uv.lock
• uv.lock