Skip to content

refactor(conformance): consolidate the test suite workspace - #4120

Draft
elezar wants to merge 1 commit into
codex/conformance-after-parityfrom
codex/4119-conformance-suite
Draft

elezar wants to merge 1 commit into
codex/conformance-after-parityfrom
codex/4119-conformance-suite

Conversation

@elezar

@elezar elezar commented Oct 2, 2026

Copy link
Copy Markdown
Member

Summary

Consolidate the shared conformance runner and scenarios under tests/suites/conformance, alongside their behavior-grouped Cargo entry points. This implements the first consolidation step in the final RFC 0016 proposal (#3460) after standalone runner removal, while preserving existing assertions and test selection names.

Stacked on #3866; the PR base is codex/conformance-after-parity.

Related Issue

Closes #4119. Part of #3954.

Changes

  • Move crates/openshell-conformance to tests/suites/conformance/support and make it a member of the suite workspace. Retain the package name and lint policy; declare the process runner's required Tokio features explicitly.
  • Update CLI, Podman, and Keycloak consumers, remove the product-workspace lockfile entry, and prune dependencies no longer enabled in the suite lockfiles.
  • Add explicit local and branch-CI formatting, Clippy, and harness unit-test checks so the move preserves source coverage. Keep gateway-backed archive selection limited to the CLI test package.
  • Add suite layout, authoring, and prepared-target execution guidance; update TESTING.md, CI.md, and the architecture inventory.

Effective capability discovery, distinct applicability/infrastructure outcomes, attributed completeness reporting, and further behavioral migrations remain follow-ups. The RFC remains a draft; this PR does not claim new conformance qualification or driver admission evidence.

Testing

  • Checks appropriate to the affected code and behavior pass.
  • Existing 18 harness unit tests pass with the exact branch-CI nextest command after rebasing onto test(conformance): run driver suites with cargo #3866.
  • Conformance workspace Clippy passes with all targets and warnings denied.
  • Gateway-backed conformance and Podman test binaries compile; Keycloak feature targets pass cargo check.
  • All eight tracked Cargo lockfiles validate with --locked.
  • Nextest archive construction passes; listing the archive retains all eight CLI tests across four binaries.
  • tests/artifacts.nix parses; architecture/skill inventory, reference, and public discovery checks pass.
  • Relocated Rust implementation files are byte-identical to the PR base. One existing entry-point formatting issue is normalized by the added format check.
  • mise run pre-commit passed before rebasing. Focused tests, Clippy, consumer compilation, and all lockfiles were verified again on the final base.
  • Full local CI did not complete: the initial invocation lacked the host Z3 library; the retry with nix develop -c mise run ci ran out of disk space during unrelated root-workspace Rust compilation. The task's disposable root build output was cleaned afterward. Full CI is not reported as passing.
  • Live driver/feature E2E runs were not performed locally. Behavioral assertions and target provisioning are unchanged.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off (DCO).
  • Suite guidance and architecture inventory updated.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 2, 2026 14:44
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

@elezar
elezar marked this pull request as draft October 2, 2026 15:05
@copy-pr-bot

copy-pr-bot Bot commented Oct 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant