Skip to content

refactor(conformance): consolidate shared support in the suite workspace #4119

Description

@elezar

User Story

As an OpenShell contributor maintaining the conformance PR stack ending at #3866, I need the conformance tests organized with their shared implementation so I can align the next PR with the final testing-strategy proposal in #3460.

Problem Statement

After #3866 removes the standalone conformance executable, the Cargo entry points live under tests/suites/conformance, but their shared scenarios and runner still live in the product workspace under crates/openshell-conformance. The test implementation and execution guidance remain split across those locations.

Impact / Why This Matters

Contributors must navigate the product workspace to maintain the test suite. Moving the library without updating source checks would silently remove its unit-test and lint coverage from the root workspace. Keeping it in place leaves the first consolidation step of RFC 0016 unfinished.

Proposed Design

Contributors should find the reusable runner, scenario implementations, behavior-grouped CLI entry points, and execution guidance in the conformance suite tree. Existing driver and feature consumers should continue to reuse the runner, and existing Cargo/nextest selection names and artifact execution should continue to work. Keep source checks and gateway-backed execution independently selectable.

Example contributor workflow:

mise run test:conformance-support
cargo test --locked --manifest-path tests/suites/conformance/Cargo.toml \
  --package openshell-test-conformance-cli --no-run

Acceptance Criteria

  • The shared conformance implementation lives under tests/suites/conformance and is absent from the product workspace.
  • Existing gateway-backed test names and behavioral assertions are preserved.
  • Podman and Keycloak suites resolve and compile against the relocated runner.
  • Local and branch source checks retain harness unit tests, formatting, and lint coverage without requiring a gateway.
  • Conformance archive construction preserves the eight current CLI tests.
  • Suite guidance explains the implementation layout, prepared-target execution, and current qualification/reporting limitations.

Alternatives Considered

  • Retain the library in crates/: keeps current source checks but leaves suite ownership split after runner removal.
  • Duplicate the runner into each suite: avoids cross-workspace dependencies but creates divergent execution and cleanup behavior.
  • Add capability discovery and result reporting in the same PR: useful follow-up work, but crosses public API and reporting boundaries beyond this consolidation. Middleware, interceptors, and providers do not address repository test ownership.

Context

Implementation follow-up to #3866 and the consolidation step of RFC PR #3460. Part of #3954; this issue does not cover the full strategy, capability API, reporting, or coverage migrations.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions