NVIDIA · elezar · Sep 30, 2026
@@ -171,7 +171,7 @@ In the prompt, instruct the reviewer to:
 3. Determine the **issue type** — one of: `feat` (new feature), `fix` (bug fix), `refactor`, `chore`, `perf`, `docs`.
 4. Propose the minimal set of changes that satisfies the requirements.
 5. Sequence the work so each step is independently testable.
-6. Identify what tests are needed (unit, integration, e2e) and where they should live.
+6. Identify what tests are needed (unit, integration, conformance, e2e) and where they should live. For public CLI or gateway behavior that may need portable installed-artifact coverage, apply the routing criteria in `tests/suites/conformance/README.md` instead of assuming an existing driver or E2E test is sufficient.
 7. Assess **complexity** on a scale:
    - **Low**: Isolated change, < 3 files, clear path forward
    - **Medium**: Multiple files/components, some design decisions, but well-scoped
@@ -212,6 +212,7 @@ gh issue comment <id> --body "$(cat <<'EOF'
 ### Test Plan
 - **Unit tests:** <what will be tested and where the tests live>
 - **Integration tests:** <what will be tested, or "N/A" with rationale>
+- **Conformance tests:** <portable installed-artifact behavior to cover, or "N/A" with rationale>
 - **E2E tests:** <what will be tested, or "N/A" with rationale>
 
 ### Risks & Open Questions
@@ -429,17 +430,31 @@ mise run pre-commit
 
 Do not proceed to Phase 2 or PR creation if Phase 1 is not green.
 
-#### Phase 2: E2E Tests (Conditional)
+#### Phase 2: Conformance and E2E Tests (Conditional)
 
-**Trigger**: Run this phase if any files under `e2e/` were added or modified in this build. Check with:
+**Trigger**: Run this phase when the plan calls for conformance or E2E coverage,
+when the implementation changes behavior covered by those suites, or when
+executable test code or configuration changes under any of these paths:
 
 ```bash
-git diff --name-only main -- e2e/
+git diff --name-only main -- \
+  e2e/ \
+  crates/openshell-conformance/ \
+  crates/openshell-conformance-cli/ \
+  tests/suites/conformance/
 ```
 
-If there are no changes under `e2e/`, skip this phase entirely.
+Treat this path check as a minimum signal, not the sole test-selection rule. A
+public CLI or gateway behavior change can require portable conformance coverage
+even when none of these paths changed. Use `tests/suites/conformance/README.md`
+to decide whether conformance applies and which focused build and live checks to
+run.
 
-If E2E files were modified, run the relevant E2E lane for the driver touched by the change:
+For changes to the conformance library, runner, or installed-artifact workspace,
+run the focused checks documented in that README. When scenario behavior or its
+E2E integration changes, also run the narrowest live lane that exercises it.
+
+For driver- or environment-specific E2E changes, run the relevant lane:
 
 ```bash
 # Docker-backed gateway smoke E2E
@@ -448,23 +463,23 @@ mise run e2e:docker
 
 Use `mise run e2e:podman`, `mise run e2e:vm`, or a Helm-backed Kubernetes E2E lane when the change targets those drivers.
 
-**E2E retry loop** (up to 3 attempts):
+**Conformance/E2E retry loop** (up to 3 attempts):
 
-1. Run the selected E2E lane.
+1. Run the selected focused conformance check or E2E lane.
 2. If tests fail:
-   - Read the pytest output carefully — identify which tests failed and why.
+   - Read the test output carefully — identify which tests failed and why.
    - Distinguish between **test bugs** (the test itself is wrong) and **implementation bugs** (the code under test is wrong).
    - Fix the failing code or tests.
    - Decrement the retry counter and try again.
 3. If tests pass, Phase 2 is green.
 
-**If all 3 E2E attempts fail**, stop and report to the user:
-- Which E2E tests are failing
-- The pytest output from the last attempt
+**If all 3 attempts fail**, stop and report to the user:
+- Which conformance or E2E tests are failing
+- The test output from the last attempt
 - Whether the failures appear to be test issues or implementation issues
 - That manual intervention is needed
 
-Do not proceed to PR creation if E2E verification is not green.
+Do not proceed to PR creation if required conformance or E2E verification is not green.
 
 ### Step 11: Update Documentation
 
@@ -529,6 +544,7 @@ Closes #<issue-id>
 ## Testing
 - [x] `mise run pre-commit` passes
 - [x] Unit tests added/updated
+- [x] Conformance tests added/updated (if applicable)
 - [x] E2E tests added/updated (if applicable)
 
 **Tests added:**
@@ -727,7 +743,7 @@ User says: "Build issue #42"
 6. Implement pagination for both endpoints per the plan
 7. Add unit tests for pagination logic, integration tests for both endpoints
 8. `mise run pre-commit` passes on first attempt
-9. E2E tests skipped (no changes under `e2e/`)
+9. Conformance and E2E tests skipped because the plan and changed behavior do not require them
 10. Commit, push, create PR with `Closes #42`
 11. Post summary comment on issue with PR link
 12. No agent-workflow label transition is needed

@@ -207,8 +207,11 @@ gh issue create \
 
 ## Test Considerations
 
+- For public CLI or gateway behavior that may need portable installed-artifact
+  coverage, use `tests/suites/conformance/README.md` to determine whether the
+  contract belongs in conformance, a driver suite, a feature suite, or E2E.
 - <what testing strategy makes sense for this change>
-- <which test levels are needed: unit, integration, e2e>
+- <which test levels are needed: unit, integration, conformance, e2e>
 - <any test infrastructure that may need to be added>
 - <what tests exist for the affected area today, what patterns should be followed, any test infrastructure gaps>
 

@@ -137,6 +137,13 @@ Read through the full diff (and the PR description if available). Produce a summ
   - Do not fabricate concerns or claim a behavioral regression without evidence
     for both the prior and proposed behavior.
 - **Agent infrastructure**: When the PR changes behavior, commands, or development workflows, use the `sync-agent-infra` maintenance map to check that related skills were updated. When it adds, removes, or renames skills or crates; changes workflow relationships or skill coverage; modifies issue or PR templates; or changes agent cross-references, apply the full consistency checklist. Report missing companion updates or drift under **Potential Concerns**.
+- **Conformance coverage**: When the PR changes public CLI or gateway behavior,
+  or touches `crates/openshell-conformance/` or
+  `tests/suites/conformance/`, apply the placement and verification criteria in
+  `tests/suites/conformance/README.md`. Report a missing portable scenario,
+  misplaced driver-specific assertion, duplicated scenario implementation, or
+  omitted focused verification under **Potential Concerns** when it creates a
+  concrete coverage risk.
 
 ## Step 5: Output
 

@@ -24,6 +24,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re
 | `.agents/skills/create-github-pr/SKILL.md` | Pre-PR agent infrastructure check |
 | `.agents/skills/review-github-pr/SKILL.md` | Review-time agent infrastructure check |
 | `.agents/skills/build-from-issue/SKILL.md` | Label awareness and pre-commit agent infrastructure check |
+| `tests/suites/conformance/README.md` | Canonical conformance test placement and verification guidance |
 | `.claude/agents/principal-engineer-reviewer.md` | Shared review-time agent infrastructure check |
 
 ## When to Run
@@ -32,6 +33,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re
 - After adding, removing, or renaming a crate in `crates/`
 - After changing workflow chain relationships between skills
 - After changing which product or development areas a skill covers
+- After changing conformance test placement or verification guidance
 - After modifying issue or PR templates
 - Before opening a PR that touches any of the above
 
@@ -56,6 +58,7 @@ Use this map when product behavior, commands, or development workflows change. I
 | Security review or remediation workflow | `review-security-issue`, `fix-security-issue` |
 | RFC template, numbering, or lifecycle | `create-rfc` |
 | Documentation structure, navigation, or doc-update workflow | `update-docs-from-commits` |
+| Portable public CLI behavior, installed-artifact validation, or conformance test layout | `build-from-issue`, `create-spike`, `review-github-pr` |
 | Skills, crates, workflow chains, issue/PR templates, or agent cross-references | `sync-agent-infra` |
 
 ## Prerequisites
@@ -142,6 +145,10 @@ For each file in the table above, check for the following inconsistencies:
 4. **`create-spike`** — Reference to `build-from-issue` as next step must be accurate.
 5. **`review-security-issue`** / **`fix-security-issue`** — Cross-references between the two must be accurate.
 6. **PR creation and review checks** — The `create-github-pr`, `review-github-pr`, `build-from-issue`, and `principal-engineer-reviewer` references to `sync-agent-infra` must exist and use trigger conditions aligned with this skill.
+7. **Conformance test routing** — `AGENTS.md`, `build-from-issue`,
+   `create-spike`, and `review-github-pr` must route portable public CLI and
+   gateway behavior to `tests/suites/conformance/README.md` without copying its
+   detailed placement rules.
 
 ### Skill Layout, Metadata, and Portability
 

@@ -227,6 +227,7 @@ ocsf_emit!(event);
 - `mise run test` — Unit test suite. Run after code changes.
 - `mise run e2e` — End-to-end tests against a running gateway. Run for infrastructure, sandbox, or policy changes.
 - `mise run ci` — Full local CI (lint + compile/type checks + tests). Run before opening a PR.
+- For public CLI or gateway behavior that should be portable across drivers and installation methods, use the [CLI conformance suite guide](tests/suites/conformance/README.md) to decide whether to add conformance coverage and which test layer owns it.
 
 ## Go SDK (`sdk/go/`)
 

@@ -0,0 +1,158 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# CLI conformance suite
+
+This workspace verifies that an installed `openshell` CLI and a selected gateway
+implement portable, public behavior. Tests treat the CLI and gateway as black
+boxes. They do not inspect driver internals or replace driver-specific
+qualification.
+
+Use this document as the source of truth for deciding whether a change needs
+OpenShell conformance coverage and where that coverage belongs.
+
+## Choose the test location
+
+Start with the externally observable contract, not the directory changed by the
+implementation.
+
+| Behavior under test | Primary location | Why |
+|---|---|---|
+| Public CLI or gateway behavior that should work the same for every supporting compute driver and installation method | `crates/openshell-conformance/` and `tests/suites/conformance/` | Exercises the installed product through a portable black-box contract |
+| Scenario selection, command execution, polling, parsing, diagnostics, or cleanup mechanics | Unit tests in `crates/openshell-conformance/` or `crates/openshell-conformance-cli/` | Verifies the reusable conformance machinery without requiring a live gateway |
+| A compute driver's configuration, host integration, isolation mechanism, or driver-specific contract | `tests/suites/drivers/<driver>/` | The expected behavior is intentionally driver-specific |
+| A feature that requires a dedicated external service or fixture, such as an identity provider | `tests/suites/features/<feature>/` | The environment is part of the feature contract rather than the portable CLI baseline |
+| Deployment topology, wrapper behavior, upgrade behavior, or a workflow that depends on repository-managed gateway setup | The relevant `e2e/` suite or deployment test | The assertion depends on orchestration that an installed-artifact conformance test does not own |
+| Pure internal logic or a crate-local API | Unit or integration tests next to the implementation | A black-box installed-product test would be slower and less precise |
+
+A change can need more than one layer. For example, a new public sandbox
+operation can require unit tests for parsing, portable conformance coverage for
+the public contract, and a driver-specific test for a backend-only edge case.
+Do not leave a portable assertion only in a driver lane or an `e2e/` wrapper
+when the same contract should hold for other supported environments.
+
+## Decide whether behavior is portable conformance
+
+Add or update conformance coverage when all of these are true:
+
+- A user or automation can observe the behavior through the public `openshell`
+  CLI and gateway API surface.
+- The contract should remain consistent across every compute driver that claims
+  to support the capability.
+- The test can run against an installed candidate CLI and an already reachable,
+  selected gateway without importing implementation internals.
+- The scenario can own, uniquely name, and clean up its resources after success
+  or failure.
+- Environment-specific prerequisites can be expressed by selecting an
+  independent capability test rather than adding driver branches to the
+  assertion.
+
+Conformance does not mean that every environment supports every capability. A
+portable capability may be selected only in environments that support it. Keep
+capabilities with different runtime requirements independently selectable so a
+test environment can run every applicable contract without broad skips or
+short-circuiting unrelated assertions.
+
+Do not add a portable conformance scenario merely because code under
+`crates/openshell-conformance/` is convenient to reuse. If the expected result
+depends on Docker, Podman, Kubernetes, VM, GPU, host policy, a particular
+installer, or a dedicated external service, place the environment-specific
+assertion in the matching driver, feature, or E2E suite. Shared runner helpers
+may still live in the conformance crate when they are genuinely reusable.
+
+## Understand the components
+
+- `crates/openshell-conformance/` owns reusable scenarios and the runner that
+  invokes a candidate CLI, parses observations, polls state, reports failures,
+  and cleans up scenario-owned resources.
+- `crates/openshell-conformance-cli/` provides the standalone
+  `openshell-conformance` binary used by local driver E2E lanes. Its scenario
+  registry and selection behavior must stay aligned with the reusable library.
+- `tests/suites/conformance/` is the separately built installed-artifact test
+  workspace. Its tests are archived and run inside supported test guests against
+  the installed `openshell` binary.
+- `e2e/` wrappers provision local gateways and run the standalone conformance
+  binary before their lane-specific tests. They own environment setup, not the
+  portable assertions.
+
+The scenario implementation should have one source of truth in
+`crates/openshell-conformance/`. The standalone runner and installed-artifact
+tests should call that implementation instead of copying commands or
+assertions.
+
+## Add or change a scenario
+
+1. Define the public contract, capability prerequisites, and supported
+   environments. If the result is driver- or topology-specific, choose another
+   suite using the table above.
+2. Add or update the scenario in `crates/openshell-conformance/src/scenarios/`.
+   Keep observations on public output and behavior. Add crate-local tests for
+   parsing, selection, polling, failure diagnostics, and other mechanics.
+3. Export and register the scenario in the conformance library. Use a stable,
+   capability-oriented name. Give behavior with distinct prerequisites or
+   cleanup an independently selectable scenario rather than hiding it in a
+   broad aggregate.
+4. Add or update the thin installed-artifact test in
+   `tests/suites/conformance/cli/tests/`. It should construct the runner from
+   `OPENSHELL_BIN`, check gateway reachability, invoke the shared scenario, and
+   always finish through the runner so cleanup and diagnostics are preserved.
+5. Keep the standalone `openshell-conformance` runner's listing and selection
+   behavior aligned with the registered scenarios.
+6. Update integration test selection or packaging only when the new capability
+   changes which tests an environment can run. Do not duplicate scenario logic
+   in Ansible, workflow YAML, or shell wrappers.
+
+Each scenario must use its generated run ID for owned resource names, avoid
+depending on unrelated gateway state, and delete only resources it created.
+Failure output should identify the scenario and step without exposing secrets.
+
+## Verify changes
+
+Run focused checks for every component changed:
+
+```shell
+cargo test -p openshell-conformance
+cargo test -p openshell-conformance-cli
+cargo test \
+  --manifest-path tests/suites/conformance/Cargo.toml \
+  --no-run
+```
+
+The first two commands exercise the reusable library and standalone runner. The
+third proves that the separate installed-artifact workspace still compiles; it
+is not covered merely because the root workspace tests pass.
+
+When behavior or scenario assertions change, also run the narrowest live lane
+that exercises the capability. To run the installed-artifact workspace against
+an already reachable and selected gateway, build the candidate CLI if needed,
+then run:
+
+```shell
+cargo build -p openshell-cli
+OPENSHELL_BIN="$PWD/target/debug/openshell" \
+  cargo test \
+    --manifest-path tests/suites/conformance/Cargo.toml \
+    -- --nocapture --test-threads=1
+```
+
+Use `mise run e2e:docker`, `mise run e2e:podman`, `mise run e2e:vm`, or the
+relevant Kubernetes lane when the change depends on that environment or changes
+the E2E integration. Installed-artifact or cross-distribution behavior should
+also be verified through the matching integration test matrix when practical.
+
+Finally, run the repository-required checks from `AGENTS.md`. Document which
+targeted and live checks ran; do not claim conformance verification based only
+on an unchanged `e2e/` directory.
+
+## Review checklist
+
+- Is the asserted behavior a public, portable contract?
+- Does the scenario avoid driver and deployment assumptions?
+- Are distinct capabilities and prerequisites independently selectable?
+- Do the standalone runner and installed-artifact suite share one scenario
+  implementation?
+- Does every test own uniquely named resources and clean them up on failure?
+- Were the separate conformance workspace and an appropriate live lane
+  considered during verification?