Skip to content

test(conformance): run driver suites with cargo - #3866

Open
elezar wants to merge 1 commit into
codex/remove-parityfrom
codex/conformance-after-parity
Open

elezar wants to merge 1 commit into
codex/remove-parityfrom
codex/conformance-after-parity

Conversation

@elezar

@elezar elezar commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Summary

Run driver conformance through the portable Cargo test suites instead of building and distributing the standalone conformance CLI. Remove the obsolete CLI crate now that all supported paths use the library-backed test suites. This PR is stacked on #3864.

Related Issue

Follow-up cleanup for #2792.

Stack

Changes

  • Run driver conformance from tests/suites/conformance with Cargo.
  • Remove standalone conformance-CLI build and setup steps from branch, driver E2E, binary-build, and release workflows.
  • Delete crates/openshell-conformance-cli and its workspace lockfile entry.
  • Update test, build, policy-advisor, and test-guest documentation for Cargo tests and installed nextest archives.
  • Rename the Nix archive binding to reflect that it packages conformance tests rather than a CLI.

Testing

  • mise run pre-commit passes
  • cargo test --locked --package openshell-conformance passes (18 tests)
  • cargo test --locked --manifest-path tests/suites/conformance/Cargo.toml --package openshell-test-conformance-cli --no-run passes
  • nix-instantiate --parse tests/artifacts.nix passes
  • Agent infrastructure consistency and public skill discovery checks pass
  • Live driver E2E jobs were not run locally; the affected GitHub workflows provide that coverage

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@github-actions

Copy link
Copy Markdown

@elezar
elezar force-pushed the codex/conformance-after-parity branch from 9fee853 to 9994506 Compare September 30, 2026 06:32
@elezar
elezar added this pull request to stack #3939 September 30, 2026 06:32
@elezar
elezar force-pushed the codex/conformance-after-parity branch from 9994506 to f4da56b Compare September 30, 2026 10:11
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@krishicks

Copy link
Copy Markdown
Collaborator

Analysis by agent found some cleanup that's worth addressing:

Once the CLI crate is removed, these openshell-conformance items have no remaining callers:

  • SCENARIOS, scenarios() and scenario(name)
  • FILE_TRANSFER_SCENARIO and run_file_transfer
  • the Scenario::description field
  • OpenShellRunner::new
  • OpenShellRunner::scenario(), which only its own unit test calls

Without the registry's consumer, a scenario added to SCENARIOS does not run unless a matching test file is also added.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants