From fffc34fd0d19717b9aeadab7bec4fc92374f9fae Mon Sep 17 00:00:00 2001 From: Evan Lezar Date: Wed, 30 Sep 2026 10:02:35 +0200 Subject: [PATCH] docs(conformance): route agents to portable test guide Signed-off-by: Evan Lezar --- .agents/skills/build-from-issue/SKILL.md | 44 +++++-- .agents/skills/create-spike/SKILL.md | 5 +- .agents/skills/review-github-pr/SKILL.md | 7 + .agents/skills/sync-agent-infra/SKILL.md | 7 + AGENTS.md | 1 + tests/suites/conformance/README.md | 158 +++++++++++++++++++++++ 6 files changed, 207 insertions(+), 15 deletions(-) create mode 100644 tests/suites/conformance/README.md diff --git a/.agents/skills/build-from-issue/SKILL.md b/.agents/skills/build-from-issue/SKILL.md index 4ec54e61a4..fc11985bc2 100644 --- a/.agents/skills/build-from-issue/SKILL.md +++ b/.agents/skills/build-from-issue/SKILL.md @@ -171,7 +171,7 @@ In the prompt, instruct the reviewer to: 3. Determine the **issue type** — one of: `feat` (new feature), `fix` (bug fix), `refactor`, `chore`, `perf`, `docs`. 4. Propose the minimal set of changes that satisfies the requirements. 5. Sequence the work so each step is independently testable. -6. Identify what tests are needed (unit, integration, e2e) and where they should live. +6. Identify what tests are needed (unit, integration, conformance, e2e) and where they should live. For public CLI or gateway behavior that may need portable installed-artifact coverage, apply the routing criteria in `tests/suites/conformance/README.md` instead of assuming an existing driver or E2E test is sufficient. 7. Assess **complexity** on a scale: - **Low**: Isolated change, < 3 files, clear path forward - **Medium**: Multiple files/components, some design decisions, but well-scoped @@ -212,6 +212,7 @@ gh issue comment --body "$(cat <<'EOF' ### Test Plan - **Unit tests:** - **Integration tests:** +- **Conformance tests:** - **E2E tests:** ### Risks & Open Questions @@ -429,17 +430,31 @@ mise run pre-commit Do not proceed to Phase 2 or PR creation if Phase 1 is not green. -#### Phase 2: E2E Tests (Conditional) +#### Phase 2: Conformance and E2E Tests (Conditional) -**Trigger**: Run this phase if any files under `e2e/` were added or modified in this build. Check with: +**Trigger**: Run this phase when the plan calls for conformance or E2E coverage, +when the implementation changes behavior covered by those suites, or when +executable test code or configuration changes under any of these paths: ```bash -git diff --name-only main -- e2e/ +git diff --name-only main -- \ + e2e/ \ + crates/openshell-conformance/ \ + crates/openshell-conformance-cli/ \ + tests/suites/conformance/ ``` -If there are no changes under `e2e/`, skip this phase entirely. +Treat this path check as a minimum signal, not the sole test-selection rule. A +public CLI or gateway behavior change can require portable conformance coverage +even when none of these paths changed. Use `tests/suites/conformance/README.md` +to decide whether conformance applies and which focused build and live checks to +run. -If E2E files were modified, run the relevant E2E lane for the driver touched by the change: +For changes to the conformance library, runner, or installed-artifact workspace, +run the focused checks documented in that README. When scenario behavior or its +E2E integration changes, also run the narrowest live lane that exercises it. + +For driver- or environment-specific E2E changes, run the relevant lane: ```bash # Docker-backed gateway smoke E2E @@ -448,23 +463,23 @@ mise run e2e:docker Use `mise run e2e:podman`, `mise run e2e:vm`, or a Helm-backed Kubernetes E2E lane when the change targets those drivers. -**E2E retry loop** (up to 3 attempts): +**Conformance/E2E retry loop** (up to 3 attempts): -1. Run the selected E2E lane. +1. Run the selected focused conformance check or E2E lane. 2. If tests fail: - - Read the pytest output carefully — identify which tests failed and why. + - Read the test output carefully — identify which tests failed and why. - Distinguish between **test bugs** (the test itself is wrong) and **implementation bugs** (the code under test is wrong). - Fix the failing code or tests. - Decrement the retry counter and try again. 3. If tests pass, Phase 2 is green. -**If all 3 E2E attempts fail**, stop and report to the user: -- Which E2E tests are failing -- The pytest output from the last attempt +**If all 3 attempts fail**, stop and report to the user: +- Which conformance or E2E tests are failing +- The test output from the last attempt - Whether the failures appear to be test issues or implementation issues - That manual intervention is needed -Do not proceed to PR creation if E2E verification is not green. +Do not proceed to PR creation if required conformance or E2E verification is not green. ### Step 11: Update Documentation @@ -529,6 +544,7 @@ Closes # ## Testing - [x] `mise run pre-commit` passes - [x] Unit tests added/updated +- [x] Conformance tests added/updated (if applicable) - [x] E2E tests added/updated (if applicable) **Tests added:** @@ -727,7 +743,7 @@ User says: "Build issue #42" 6. Implement pagination for both endpoints per the plan 7. Add unit tests for pagination logic, integration tests for both endpoints 8. `mise run pre-commit` passes on first attempt -9. E2E tests skipped (no changes under `e2e/`) +9. Conformance and E2E tests skipped because the plan and changed behavior do not require them 10. Commit, push, create PR with `Closes #42` 11. Post summary comment on issue with PR link 12. No agent-workflow label transition is needed diff --git a/.agents/skills/create-spike/SKILL.md b/.agents/skills/create-spike/SKILL.md index 5c6d16c2ec..0cd4b4b904 100644 --- a/.agents/skills/create-spike/SKILL.md +++ b/.agents/skills/create-spike/SKILL.md @@ -207,8 +207,11 @@ gh issue create \ ## Test Considerations +- For public CLI or gateway behavior that may need portable installed-artifact + coverage, use `tests/suites/conformance/README.md` to determine whether the + contract belongs in conformance, a driver suite, a feature suite, or E2E. - -- +- - - diff --git a/.agents/skills/review-github-pr/SKILL.md b/.agents/skills/review-github-pr/SKILL.md index de56cb4770..1b21f1828a 100644 --- a/.agents/skills/review-github-pr/SKILL.md +++ b/.agents/skills/review-github-pr/SKILL.md @@ -137,6 +137,13 @@ Read through the full diff (and the PR description if available). Produce a summ - Do not fabricate concerns or claim a behavioral regression without evidence for both the prior and proposed behavior. - **Agent infrastructure**: When the PR changes behavior, commands, or development workflows, use the `sync-agent-infra` maintenance map to check that related skills were updated. When it adds, removes, or renames skills or crates; changes workflow relationships or skill coverage; modifies issue or PR templates; or changes agent cross-references, apply the full consistency checklist. Report missing companion updates or drift under **Potential Concerns**. +- **Conformance coverage**: When the PR changes public CLI or gateway behavior, + or touches `crates/openshell-conformance/` or + `tests/suites/conformance/`, apply the placement and verification criteria in + `tests/suites/conformance/README.md`. Report a missing portable scenario, + misplaced driver-specific assertion, duplicated scenario implementation, or + omitted focused verification under **Potential Concerns** when it creates a + concrete coverage risk. ## Step 5: Output diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index 68a623c392..6b273af192 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -24,6 +24,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re | `.agents/skills/create-github-pr/SKILL.md` | Pre-PR agent infrastructure check | | `.agents/skills/review-github-pr/SKILL.md` | Review-time agent infrastructure check | | `.agents/skills/build-from-issue/SKILL.md` | Label awareness and pre-commit agent infrastructure check | +| `tests/suites/conformance/README.md` | Canonical conformance test placement and verification guidance | | `.claude/agents/principal-engineer-reviewer.md` | Shared review-time agent infrastructure check | ## When to Run @@ -32,6 +33,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re - After adding, removing, or renaming a crate in `crates/` - After changing workflow chain relationships between skills - After changing which product or development areas a skill covers +- After changing conformance test placement or verification guidance - After modifying issue or PR templates - Before opening a PR that touches any of the above @@ -56,6 +58,7 @@ Use this map when product behavior, commands, or development workflows change. I | Security review or remediation workflow | `review-security-issue`, `fix-security-issue` | | RFC template, numbering, or lifecycle | `create-rfc` | | Documentation structure, navigation, or doc-update workflow | `update-docs-from-commits` | +| Portable public CLI behavior, installed-artifact validation, or conformance test layout | `build-from-issue`, `create-spike`, `review-github-pr` | | Skills, crates, workflow chains, issue/PR templates, or agent cross-references | `sync-agent-infra` | ## Prerequisites @@ -142,6 +145,10 @@ For each file in the table above, check for the following inconsistencies: 4. **`create-spike`** — Reference to `build-from-issue` as next step must be accurate. 5. **`review-security-issue`** / **`fix-security-issue`** — Cross-references between the two must be accurate. 6. **PR creation and review checks** — The `create-github-pr`, `review-github-pr`, `build-from-issue`, and `principal-engineer-reviewer` references to `sync-agent-infra` must exist and use trigger conditions aligned with this skill. +7. **Conformance test routing** — `AGENTS.md`, `build-from-issue`, + `create-spike`, and `review-github-pr` must route portable public CLI and + gateway behavior to `tests/suites/conformance/README.md` without copying its + detailed placement rules. ### Skill Layout, Metadata, and Portability diff --git a/AGENTS.md b/AGENTS.md index a590b117ce..93b1e83f09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -227,6 +227,7 @@ ocsf_emit!(event); - `mise run test` — Unit test suite. Run after code changes. - `mise run e2e` — End-to-end tests against a running gateway. Run for infrastructure, sandbox, or policy changes. - `mise run ci` — Full local CI (lint + compile/type checks + tests). Run before opening a PR. +- For public CLI or gateway behavior that should be portable across drivers and installation methods, use the [CLI conformance suite guide](tests/suites/conformance/README.md) to decide whether to add conformance coverage and which test layer owns it. ## Go SDK (`sdk/go/`) diff --git a/tests/suites/conformance/README.md b/tests/suites/conformance/README.md new file mode 100644 index 0000000000..a5403bbce2 --- /dev/null +++ b/tests/suites/conformance/README.md @@ -0,0 +1,158 @@ + + +# CLI conformance suite + +This workspace verifies that an installed `openshell` CLI and a selected gateway +implement portable, public behavior. Tests treat the CLI and gateway as black +boxes. They do not inspect driver internals or replace driver-specific +qualification. + +Use this document as the source of truth for deciding whether a change needs +OpenShell conformance coverage and where that coverage belongs. + +## Choose the test location + +Start with the externally observable contract, not the directory changed by the +implementation. + +| Behavior under test | Primary location | Why | +|---|---|---| +| Public CLI or gateway behavior that should work the same for every supporting compute driver and installation method | `crates/openshell-conformance/` and `tests/suites/conformance/` | Exercises the installed product through a portable black-box contract | +| Scenario selection, command execution, polling, parsing, diagnostics, or cleanup mechanics | Unit tests in `crates/openshell-conformance/` or `crates/openshell-conformance-cli/` | Verifies the reusable conformance machinery without requiring a live gateway | +| A compute driver's configuration, host integration, isolation mechanism, or driver-specific contract | `tests/suites/drivers//` | The expected behavior is intentionally driver-specific | +| A feature that requires a dedicated external service or fixture, such as an identity provider | `tests/suites/features//` | The environment is part of the feature contract rather than the portable CLI baseline | +| Deployment topology, wrapper behavior, upgrade behavior, or a workflow that depends on repository-managed gateway setup | The relevant `e2e/` suite or deployment test | The assertion depends on orchestration that an installed-artifact conformance test does not own | +| Pure internal logic or a crate-local API | Unit or integration tests next to the implementation | A black-box installed-product test would be slower and less precise | + +A change can need more than one layer. For example, a new public sandbox +operation can require unit tests for parsing, portable conformance coverage for +the public contract, and a driver-specific test for a backend-only edge case. +Do not leave a portable assertion only in a driver lane or an `e2e/` wrapper +when the same contract should hold for other supported environments. + +## Decide whether behavior is portable conformance + +Add or update conformance coverage when all of these are true: + +- A user or automation can observe the behavior through the public `openshell` + CLI and gateway API surface. +- The contract should remain consistent across every compute driver that claims + to support the capability. +- The test can run against an installed candidate CLI and an already reachable, + selected gateway without importing implementation internals. +- The scenario can own, uniquely name, and clean up its resources after success + or failure. +- Environment-specific prerequisites can be expressed by selecting an + independent capability test rather than adding driver branches to the + assertion. + +Conformance does not mean that every environment supports every capability. A +portable capability may be selected only in environments that support it. Keep +capabilities with different runtime requirements independently selectable so a +test environment can run every applicable contract without broad skips or +short-circuiting unrelated assertions. + +Do not add a portable conformance scenario merely because code under +`crates/openshell-conformance/` is convenient to reuse. If the expected result +depends on Docker, Podman, Kubernetes, VM, GPU, host policy, a particular +installer, or a dedicated external service, place the environment-specific +assertion in the matching driver, feature, or E2E suite. Shared runner helpers +may still live in the conformance crate when they are genuinely reusable. + +## Understand the components + +- `crates/openshell-conformance/` owns reusable scenarios and the runner that + invokes a candidate CLI, parses observations, polls state, reports failures, + and cleans up scenario-owned resources. +- `crates/openshell-conformance-cli/` provides the standalone + `openshell-conformance` binary used by local driver E2E lanes. Its scenario + registry and selection behavior must stay aligned with the reusable library. +- `tests/suites/conformance/` is the separately built installed-artifact test + workspace. Its tests are archived and run inside supported test guests against + the installed `openshell` binary. +- `e2e/` wrappers provision local gateways and run the standalone conformance + binary before their lane-specific tests. They own environment setup, not the + portable assertions. + +The scenario implementation should have one source of truth in +`crates/openshell-conformance/`. The standalone runner and installed-artifact +tests should call that implementation instead of copying commands or +assertions. + +## Add or change a scenario + +1. Define the public contract, capability prerequisites, and supported + environments. If the result is driver- or topology-specific, choose another + suite using the table above. +2. Add or update the scenario in `crates/openshell-conformance/src/scenarios/`. + Keep observations on public output and behavior. Add crate-local tests for + parsing, selection, polling, failure diagnostics, and other mechanics. +3. Export and register the scenario in the conformance library. Use a stable, + capability-oriented name. Give behavior with distinct prerequisites or + cleanup an independently selectable scenario rather than hiding it in a + broad aggregate. +4. Add or update the thin installed-artifact test in + `tests/suites/conformance/cli/tests/`. It should construct the runner from + `OPENSHELL_BIN`, check gateway reachability, invoke the shared scenario, and + always finish through the runner so cleanup and diagnostics are preserved. +5. Keep the standalone `openshell-conformance` runner's listing and selection + behavior aligned with the registered scenarios. +6. Update integration test selection or packaging only when the new capability + changes which tests an environment can run. Do not duplicate scenario logic + in Ansible, workflow YAML, or shell wrappers. + +Each scenario must use its generated run ID for owned resource names, avoid +depending on unrelated gateway state, and delete only resources it created. +Failure output should identify the scenario and step without exposing secrets. + +## Verify changes + +Run focused checks for every component changed: + +```shell +cargo test -p openshell-conformance +cargo test -p openshell-conformance-cli +cargo test \ + --manifest-path tests/suites/conformance/Cargo.toml \ + --no-run +``` + +The first two commands exercise the reusable library and standalone runner. The +third proves that the separate installed-artifact workspace still compiles; it +is not covered merely because the root workspace tests pass. + +When behavior or scenario assertions change, also run the narrowest live lane +that exercises the capability. To run the installed-artifact workspace against +an already reachable and selected gateway, build the candidate CLI if needed, +then run: + +```shell +cargo build -p openshell-cli +OPENSHELL_BIN="$PWD/target/debug/openshell" \ + cargo test \ + --manifest-path tests/suites/conformance/Cargo.toml \ + -- --nocapture --test-threads=1 +``` + +Use `mise run e2e:docker`, `mise run e2e:podman`, `mise run e2e:vm`, or the +relevant Kubernetes lane when the change depends on that environment or changes +the E2E integration. Installed-artifact or cross-distribution behavior should +also be verified through the matching integration test matrix when practical. + +Finally, run the repository-required checks from `AGENTS.md`. Document which +targeted and live checks ran; do not claim conformance verification based only +on an unchanged `e2e/` directory. + +## Review checklist + +- Is the asserted behavior a public, portable contract? +- Does the scenario avoid driver and deployment assumptions? +- Are distinct capabilities and prerequisites independently selectable? +- Do the standalone runner and installed-artifact suite share one scenario + implementation? +- Does every test own uniquely named resources and clean them up on failure? +- Were the separate conformance workspace and an appropriate live lane + considered during verification?