docs(rfc): propose OpenShell testing strategy

[elezar]

Summary

Propose RFC 0016 for OpenShell's testing strategy. Define separate ownership for behavioral contracts, test families, provisioning, installation, execution, and CI policy so contributors have one place to discuss the strategy and clear destinations for its living documentation.

Related Issue

Part of #3954, broadened at the maintainer's request to track the overall testing strategy and assign RFC 0016. Accepting the RFC does not close the implementation tracker.

Changes

• Define general conformance, feature-specific, driver-specific, disruption, load/scale, and lower-level testing boundaries; retain SDK compatibility in companion RFC PR docs(rfc): propose SDK conformance testing #3238.
• Define contracts, test cases, assertions, and suites; treat CLI and SDK access as client interfaces rather than separate test families.
• Specify mandatory versus optional support and distinct result semantics; defer classification of capability-dependent portable tests until a concrete migration example.
• Diagram target preparation separately from behavioral testing, reflecting tmachine environments, installers, and test suites.
• Document existing conformance, Podman driver, and Keycloak provider-refresh suite locations and proposed additions.
• Propose moving the shared conformance library under tests/suites/conformance after test(conformance): run driver suites with cargo #3866 removes the standalone conformance executable.
• Assign clear responsibilities to TESTING.md, proposed tests/CONFORMANCE.md, suite READMEs, CI.md, and contributor skills.
• Define incremental migration by behavioral intent and preserve unanswered versioning, admission, portability, reporting, and CI policy questions for review.

The final PR diff contains only the RFC. Current testing and CI reference documents will be updated through focused follow-ups as the strategy is agreed and implemented.

Testing

• mise run pre-commit passes, including Markdown, Rust workspace/E2E/example lint, formatting, license, Helm, Python, and SDK checks.
• Repository Markdown lint passes.
• git diff --cached --check passes; the final diff against the PR merge base contains only the RFC.
• RFC template sections and issue/PR references checked.
• Unit and E2E tests are not applicable to this proposal-only change.

Checklist

• Follows Conventional Commits.
• Commit is signed off for DCO.
• Living architecture and published documentation changes are deferred to implementation, as required by the RFC workflow.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[johntmyers]

If something is only implemented on one driver, I'd assume that should be part of "driver specific"

[elezar]

I think that's generally true, but only if it doesn't rely on some external service. For example, something that only works on podman, but requires some addtional service to work could also be a feature-specific test.

[johntmyers]

is this only unit and integration? is everything else an e2e test then?

[elezar]

Yes, the conformance, feature-specific, and driver-specific tests are end-to-end tests. I'll update the tables as per @drew's suggestion to clarify things. We'll use "integration" for component-integration tests and end-to-end for tests that run CLI commands against a real OpenShell installation.

[matthewgrossman]

is general conformance different than feature-specific where all drivers support it? I'm imagining that there's a set of features w/ venn diagrams including specific drivers, and in a world where all drivers are included, it just becomes a circle and is now deemed a "conformance test"

[matthewgrossman]

effectively wondering if conformance is just a special-case version of feature-specific tests where all drivers are included, or if you see it as something different

[elezar]

The way I have tried to reason about it, "conformance" is table stakes. These are the things that every OpenShell installation should be able to do. Think: "I can start a sandbox and stop it again", "A running sandbox is restarted when the gateway restarts and keeps its data", "The sandbox can't communicate with the outside world by default".

For feature-specific tests, the driver-dependence is secondary. The factor here is that the functionality requires some external service or property of the system. The example that's currently included in the repo is using keycloak to implement provider refresh. This requires a functional keycloak deployment to work. These tests MAY work on all drivers, by may not be applicable to all OpenShell installations becase they require opt-in behaviour from the user.

Does that help / clarify things?

[matthewgrossman]

at a practical level I don't see any issues with global-setting changes if we're isolating tests, but I find this statement a little confusing. is it implying that multiple tests are sharing the same environment, so a good test-citizen should restore the env how they found it?

[elezar]

I think the point is that if a test is a conformance test it should be able to run against any OpenShell installation. This means that it MUST not updated gateway config to get a test to pass. The discussion around global state comes from my agent looking at some of the tests and determined that running them modifies some globabl state that isn't the CAS store (I think it had something to do with OCSF). I need to improve the wording to focus on not modifying a running Gateway in ways that a user with CLI / API access would not be able to do.

[matthewgrossman]

making sure I understand this section:

Rather than encoding in our test-suite which drivers support features a/b/c, that should be encoded as an API contract that consumers can see. The the test suite just becomes one consumer of that API, and decides which tests are worth running against this given openshell deploy?

[elezar]

No, I don't think that's correct. (Which shows that I need to rework this).

The goal of the conformance test suite is that it can run against any OpenShell installation. In cases where this is not possible (or is not yet possible), my thinking was that one should have a programatic way of indicating that a failure is expected and indicates a conformance gap for a particular configuration. For example, the MXC driver does not support openshell sandbox exec which means that a conformance test that checks: "I can run a sandbox and exec something in it" will not work there. A conformance test that fails (as expected) against this driver will then allow us to explicitly capture the gap and also give a strong signal of the change when we finally add support.

The alternative is to have the tests discover what a driver supports and only run a subset of tests against a driver (or positive and negative tests depending on the reported capability). As @drew mentions later this is probably not scalable.

I think with a couple of concrete examples, we can tighten the definitions a bit.

[drew]

its a little confusing to put unit and integration in the same sentence. these are usually distinct types of tests and integration has become very overloaded. we call current conformance tests, "conformance integration", and current feature specific tests "feature specific integration"

[elezar]

They were grouped together in the doc because they are both product internal. That is to say that they interact with Rust primatives at a software level and do not require "real" testing infrastructure.

You're correct that "integration" is overloaded here. In the cases you mention these should be end-to-end tests of which conformance and feature-specific are two classes. As part of this work, we can update the CI jobs too.

[drew]

I would suggest the following taxonomy

Test family	Description
Lint	Check formatting, style, and static rules.
Unit	Verify one component in isolation. Place tests inline in its Rust module or in an adjacent test.rs file under src/.
Integration	Verify interactions between components. Place Rust integration tests in the crate’s top-level tests/ directory.
End-to-end	Verify a configured OpenShell target through a client interface. Place suites in the repository-level e2e/ directory; conformance is a class of end-to-end test for portable public contracts.
Benchmark	Measure performance or scale. Place crate benchmarks in benchmarks/ and full-system benchmarks in a root benchmarks/.

Types of e2e tests

End-to-end type	What it verifies
Conformance	Portable public contracts across applicable drivers, including advertised optional capabilities. Includes recovery and continuity after an induced failure. It should be possible to run these tests out of tree.
Feature	Behavior requiring a named external service or special gateway configuration.
Driver	Behavior specific to a driver, its host integration, or its configuration.
Installation	Installing, upgrading, and uninstalling candidate artifacts.

[elezar]

I like the suggestion.

I also think I've convinced myself that the tests that we're writing are user-facing e2e tests and not system-level integration tests. I'll update naming to address that.

[drew]

It would be good if we could start to define what conformance tests we're going to build. I would propose the following suites. Here's a list to get us started

• Sandbox lifecycle: Create, inspect, execute, stop, restart, and delete.
• Policy: Validate, apply, update, and report effective policy.
• Sandbox enforcement: Enforce filesystem, process, and network rules.
• Providers: Manage providers and verify credential delivery, isolation, rotation, and protection from exposure.
• Identity and authorization: Authenticate and enforce access boundaries.
• Middleware: Verify selection, ordering, transformation, and failure handling.
• Interceptors: Verify request transformation, rejection, and preservation of authorization.

[elezar]

Yes, adding categories / focus areas for the conformance tests makes sense (and already exists to some extend). I'll update the doc to formalize it a bit.

[drew]

why is conformance at the top instead of inside conformance/?

[elezar]

I think this was pulled from k8s, but should be updated. I'll assess the layout again a bit more critically.

[drew]

this whole section is pretty dense. i'm not sure i understand it. conformance tests should

• use public apis
• be parametrized by gateway endpoint
• have the ability to run out of tree so third parties can test for conformance
• run as part of nightly qualification
• run on branch checks when manually triggered.
• it would be great if we can granularly trigger conformance checks by suite on ci

[elezar]

I think I agree with your points, but want to be careful to not mix WHAT the tests are with HOW and WHEN they are run.

Q: What are conformance tests?
A: Conformance tests are end-to-end tests that run against any OpenShell installation (i.e. parameterized endpoint). They use public APIs (e.g. the openshell CLI or SDKs) to test required behaviour against the configured installation.

Q: How are conformance tests run?
A: (Still in progress) They are run as a nextest archive against the configured installation. The archive can be downloaded and run against any OpenShell installation and do not required the OpenShell CI machinery.

Q: When are they run?
A: Whenever feasible.

This last answer could also be "it depends". We should definitely run them as part of automated qualification. It should also be possible to run them locally for development and trigger them in CI. Providing more granular selection should be possible and could also be considered, but then we would have to determine what the selection criteria are. Why are we NOT running all the tests?

[elezar]

(I will work on rewording this).

[drew]

can we please detail what tests run where. for example, unit tests always run on ci, e2e tests run when manually triggered or on release qualification, etc. we also need to cleanup the current labelling approach.

i would also like to see us be more efficient in what tests run on branch checks. @SDAChess mentioned work to dynamically figure out what tests to run per pr. i think we should consider this. i've seen interesting approaches that use llms or jev to figure out what tests should be run based on changes. might be interesting to experiment with.

we don't have to build this all at once, but since this rfc is broadly titled "propose OpenShell testing strategy" and talks about execution stragies, i think we should detail where we're headed with things. current branch checks on prs are too slow.

[drew]

i'm assuming this is just for conformance tests. how do we specify a test to be unsupported or skipped?

[drew]

I'm not sure how scalable this is. For example, what if down the road we support loading more than one compute driver. Some compute drivers might enable different capabilities. This is also going to cause options on the compute driver to explode with every product capability a driver might or might not support. We've already started to see this and it creates quite a change amplification that I'd like to avoid.

As part of RFC-0012 we proposed using validation to assert capabilities. For example, if you create a sandbox with a file system policy, and no filesystem policy exists, that sandbox should fail to create will a validation error. I think this is a more scalable approach.

[elezar]

Yes, gateway-reported capabilities might not be the right approach here. As I mentioned to MG above, we may be able to get a better idea of what things look like once we add some concrete examples.

[drew]

We should handle some level of flaky-ness and include infra for retries. If a test is identified as flaky there should be a report that we monitor and fix out of band. If we hard fail on flakes, we're going to be fighting builds and wind up just manually retrying tests anyways (we already see this today).

Down the road, if there's a report we can have some agent iterate on the report to reduce flakes.

[elezar]

The point here was to simplify the process for calling a test a conformance test. As I understand it, in the k8s case, the tests have to be considered stable for a period of time before being allowed to be labeled as such.

I do agree that we should include some flakiness analysis of our tests in general.