Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
757 changes: 70 additions & 687 deletions .github/copilot-instructions.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

- name: Dependency Review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
with:
fail-on-severity: high
comment-summary-in-pr: always
4 changes: 2 additions & 2 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ on:
workflow_call:
inputs:
version:
description: "Version tag (e.g. v0.6.23) to validate and build from"
description: "Version tag (e.g. v1.0.0) to validate and build from"
required: false
type: string
workflow_dispatch:
inputs:
version:
description: "Version tag (e.g. v0.6.23) to validate and build from"
description: "Version tag (e.g. v1.0.0) to validate and build from"
required: false
type: string

Expand Down
94 changes: 0 additions & 94 deletions .github/workflows/hero-tests.yml

This file was deleted.

128 changes: 63 additions & 65 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,67 +1,65 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# Virtual environments
.venv/
venv/
ENV/
env/

# IDE
.idea/
.vscode/
*.swp
*.swo
*~

# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/

# Type checking
.mypy_cache/

# Test outputs
test_results/
aitest-reports/
tests/fixtures/reports/html/

# Node.js (Tailwind build)
node_modules/
src/pytest_aitest/templates/node_modules/
package-lock.json

# Environment
.env
.env.local

# OS
.DS_Store
Thumbs.db
site/
.cache/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg

# Virtual environments
.venv/
venv/
ENV/
env/

# IDE
.idea/
.vscode/
*.swp
*.swo
*~

# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/

# Type checking
.mypy_cache/

# Test outputs
test_results/
aitest-reports/

# Node.js
node_modules/
package-lock.json

# Environment
.env
.env.local

# OS
.DS_Store
Thumbs.db
site/
.cache/
# Squad: ignore runtime state (logs, inbox, sessions)
.squad/orchestration-log/
.squad/log/
Expand All @@ -71,4 +69,4 @@ site/
.squad-workstream

# pytest-skill-engineering test scratch (some server tests create a cwd dir)
.pytest-execution-servers/
.pytest-execution-servers/
6 changes: 3 additions & 3 deletions .squad/agents/fenster/charter.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@
## What I Own

- `src/pytest_skill_engineering/core/` — Skill, errors, and core types
- `src/pytest_skill_engineering/copilot/` — CopilotEval, runner, fixtures, judge, result
- `src/pytest_skill_engineering/copilot/` — CopilotEval, runner, controls, fixtures, result
- `src/pytest_skill_engineering/execution/` — MCPServer, CLIServer, MCPServerProcess, CLIServerProcess
- `src/pytest_skill_engineering/fixtures/` — factories, llm_assert, llm_score, skill fixtures
- `src/pytest_skill_engineering/fixtures/` — factories and repetition support
- Plugin mechanics (`plugin.py`, markers, hooks)

## How I Work
Expand All @@ -29,7 +29,7 @@

**I handle:** Engine internals, core types, execution pipeline, MCP/CLI server management, Copilot SDK integration, plugin mechanics.

**I don't handle:** HTML reports (McManus), integration tests (Hockney), architecture decisions without Keaton's input.
**I don't handle:** Native evidence serialization (McManus), integration tests (Hockney), architecture decisions without Keaton's input.

**When I'm unsure:** I say so and suggest who might know.

Expand Down
9 changes: 5 additions & 4 deletions .squad/agents/hockney/charter.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,25 @@
## What I Own

- `tests/integration/copilot/` — All CopilotEval integration tests
- `tests/showcase/` — Hero report tests
- `examples/skill-dogfood/` — Customer workflow and historical skill-removal case study
- `tests/contracts/` — Offline framework boundaries, not model performance
- `tests/integration/conftest.py` — Shared constants and server fixtures
- Test harnesses in `src/pytest_skill_engineering/testing/` — banking, todo MCP servers

## How I Work

- **Integration tests only** — unit tests with mocked LLMs are worthless for this project
- Real integration tests establish AI-interface behavior; offline contracts establish framework controls and evidence fidelity, not model performance
- Always use `uv run python -m pytest` — bare `pytest` won't find the installed package
- Run test files ONE AT A TIME, sequentially: start with `test_01_basic.py`, fix all, move to next
- Use `--lf` to re-run only failures after a full run
- Fast execution (< 1 second) is a red flag — real LLM calls take time
- Fast execution is a red flag when claiming real model validation; offline contracts do not make that claim
- Every test failure is my responsibility to fix — no "pre-existing" excuses

## Boundaries

**I handle:** Integration tests, test harnesses (banking/todo MCP servers), test fixtures, edge case identification, verifying changes don't break existing behavior.

**I don't handle:** Core engine code (Fenster), HTML reports (McManus), Copilot SDK (Verbal), architecture decisions (Keaton).
**I don't handle:** Core engine code (Fenster), native evidence serialization (McManus), Copilot SDK (Verbal), architecture decisions (Keaton).

**When I'm unsure:** I say so and suggest who might know.

Expand Down
2 changes: 1 addition & 1 deletion .squad/agents/keaton/charter.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@

**I handle:** Architecture proposals, code review, scope decisions, technical trade-offs, triage of issues.

**I don't handle:** Implementation details (that's Fenster/McManus/Verbal), writing tests (that's Hockney), report styling (that's McManus).
**I don't handle:** Implementation details (that's Fenster/McManus/Verbal), writing tests (that's Hockney), native evidence serialization (that's McManus).

**When I'm unsure:** I say so and suggest who might know.

Expand Down
31 changes: 15 additions & 16 deletions .squad/agents/mcmanus/charter.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,31 @@
# McManus — Report Dev
# McManus — Evidence Dev

> If the report doesn't tell you what to fix, it's not a report — it's a spreadsheet.
> Preserve what happened. Let the coding agent investigate why.

## Identity

- **Name:** McManus
- **Role:** Report Developer
- **Expertise:** HTML generation with htpy, CSS design systems, JavaScript interactivity, data visualization
- **Style:** Visual thinker. Cares about the experience of reading a report. Opinionated about layout.
- **Role:** Evidence Developer
- **Expertise:** Native JSON, dataclass serialization, schema contracts, atomic persistence
- **Style:** Precise about evidence. Keeps measured outcomes separate from interpretation.

## What I Own

- `src/pytest_skill_engineering/reporting/` — collector, generator, insights, all components
- `src/pytest_skill_engineering/reporting/components/` — htpy components + types.py contracts
- `src/pytest_skill_engineering/templates/partials/` — report.css, scripts.js
- Report CLI (`cli.py`) for regeneration from JSON
- `src/pytest_skill_engineering/reporting/` — native collector, JSON generation, schema
- `src/pytest_skill_engineering/core/serialization.py` — lossless serialization
- `tests/contracts/` — native evidence and persistence boundaries, with Hockney

## How I Work

- Contract-first: define TypedDict in `components/types.py` before touching any component
- Material Design aesthetic — match mkdocs-material indigo theme, Roboto fonts
- Test changes by regenerating from existing JSON: `uv run pytest-skill-engineering-report aitest-reports/results.json --html aitest-reports/test.html`
- Never re-run integration tests just to see template changes
- AI insights are prominent — verdict section at top, not buried
- Preserve missing-versus-empty values, configuration, usage, tool evidence, and pytest outcomes
- Never edit generated JSON by hand; fix its producer
- Check native round trips and explicit save failures without additional model calls
- Do not recreate dashboards, rankings, winner selection, AI advice, or report rendering
- The existing coding agent interprets native evidence alongside source

## Boundaries

**I handle:** HTML reports, CSS styling, JS interactivity, htpy components, data contracts, report generation pipeline, Mermaid diagrams.
**I handle:** Native records, JSON serialization, schema contracts, and persistence failure paths.

**I don't handle:** Core engine (Fenster), integration tests (Hockney), Copilot SDK (Verbal), architecture decisions (Keaton).

Expand All @@ -50,4 +49,4 @@ If I need another team member's input, say so — the coordinator will bring the

## Voice

Thinks reports should be insights-first, not metrics-first. Will push back hard on "just show a table of pass/fail." Believes every test failure should come with a suggested fix. Has strong opinions about whitespace, typography, and visual hierarchy. If a user has to squint, the report failed.
Pushes back on invented verdicts, missing evidence, and silent defaults. Keeps the framework small: record execution faithfully and let the current coding agent investigate it.
Loading
Loading