test(conformance): select leaf scenarios by capability

CI.md

@@ -49,10 +49,11 @@ Manually dispatch `Integration Tests` on the candidate branch with an
 [{"environment":"ubuntu-docker-rootful","installer":"binaries","testsuite":"policy-advisor"}]
 ```
 
-This runs the `mechanistic-proposal`, `new-hostname-proposal`, and
-`policy-local` conformance tests in the installed-artifact suite. The artifact run must contain the candidate CLI and
-gateway binaries and runtime images. This manual run does not replace the
-required PR E2E gate.
+This runs the `policy-advisor/mechanistic-proposal`,
+`policy-advisor/new-hostname-proposal`, and `policy-advisor/sandbox-local`
+conformance tests in the installed-artifact suite. The artifact run must contain
+the candidate CLI and gateway binaries and runtime images. This manual run does
+not replace the required PR E2E gate.
 
 ## Informational security reports
 

TESTING.md

@@ -181,11 +181,14 @@ lifecycle management, output parsing, and cleanup.
 Suites:
 
 - Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests.
-- CLI conformance (`openshell-conformance`) - named scenarios for lifecycle,
-  mechanistic drafts, and the sandbox-local API, including agent-authored
-  permission requests. Driver E2E wrappers run every scenario. The
-  installed-artifact conformance suite runs all scenarios and offers a focused
-  `policy-advisor` testsuite for manual integration runs.
+- CLI conformance (`tests/suites/conformance`) - installed-artifact tests of
+  portable public CLI behavior. Coverage is split into independent capability
+  tests so each driver runs only the contracts it implements and failures remain
+  isolated. Scenario implementations live in `openshell-conformance`; the
+  standalone `openshell-conformance` binary supports exact leaf and family-
+  prefix selection for manual runs and driver E2E wrappers. See the
+  [suite README](tests/suites/conformance/README.md) for scope and selection
+  guidance.
 - Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or
   `e2e-vm`) - CLI conformance plus the common and driver-specific coverage for
   the selected deployment.
@@ -222,20 +225,27 @@ starts. The task does not provision a gateway or select a compute driver. Set
 `OPENSHELL_BIN` to test a prebuilt CLI; otherwise, the task builds the CLI from
 the current checkout.
 
-The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without
-depending on how the gateway was installed or which driver is configured. It
-requires machine-readable gRPC status, creates a uniquely named detached
-sandbox with the configured default image, verifies the sandbox is `Ready` by finding its
-unique name in paginated JSON list output, executes `echo` with a run-specific
-marker, deletes the sandbox, and verifies that its name no longer appears.
-Driver suites enable the same profile
-instead of maintaining a separate smoke implementation. Sandbox lifecycle,
-label matrices, VM overlay, and TLS-key permission assertions remain regular
-E2E coverage.
+The smoke contract verifies the CLI-to-gateway-to-driver path without depending
+on how the gateway was installed. Its control-plane test requires
+machine-readable status, creates a uniquely named detached sandbox, verifies it
+is `Ready` through get and paginated list output, deletes it, and verifies its
+name no longer appears. The exec test creates its own sandbox and checks
+`sandbox exec` with a run-specific marker. Drivers without exec support run the
+control-plane test alone. Smoke sandboxes use the runtime's default workload and
+create configuration. Lifecycle coverage follows the same split: its
+control-plane test covers stop and stopped deletion without requiring exec,
+while its restart-persistence test uses exec to verify workspace state across
+stop and start. Future environment or canonical-main coverage should use
+separate leaves when it has distinct runtime requirements. The standalone runner
+expands a family selector into independent leaf runs, while installed-artifact
+CI uses the leaf tests directly as its selection and failure-isolation boundary.
+Label matrices, VM overlay, and TLS-key permission assertions remain regular E2E
+coverage.
 
 Each invocation prints a ten-character run ID before creating resources.
-Conformance sandboxes use names such as `ct-<run-id>-01`. The runner tracks the
-exact name and uses it for cleanup; phase 1 does not add ownership labels.
+Conformance sandboxes use names such as `ct-<run-id>-cp` and
+`ct-<run-id>-ex`. The runner tracks the exact name and uses it for cleanup;
+smoke conformance does not add ownership labels.
 
 The runner deletes owned resources after both success and failure. If the test
 process is interrupted before cleanup, locate leftovers without touching

crates/openshell-conformance-cli/src/main.rs

@@ -3,6 +3,7 @@
 
 //! Standalone runner for `OpenShell` CLI conformance scenarios.
 
+use std::collections::BTreeSet;
 use std::path::PathBuf;
 use std::process::ExitCode;
 
@@ -30,7 +31,7 @@ enum Command {
     },
     /// Run all registered scenarios, or named scenarios.
     Run {
-        /// Scenario names. Omit to run every registered scenario.
+        /// Leaf scenario names or family prefixes. Omit to run every leaf.
         scenarios: Vec<String>,
         /// Explicit path to the `OpenShell` CLI. Defaults to `openshell` on PATH.
         #[arg(long)]
@@ -195,14 +196,35 @@ fn select_scenarios(requested: &[String]) -> Result<Vec<&'static Scenario>, Stri
     if requested.is_empty() {
         return Ok(scenarios().iter().collect());
     }
-    requested
-        .iter()
-        .map(|name| {
-            scenario(name).ok_or_else(|| {
-                format!("unknown scenario '{name}'; run `openshell-conformance list`")
-            })
-        })
-        .collect()
+
+    let mut selected = Vec::new();
+    let mut selected_names = BTreeSet::new();
+    for selector in requested {
+        if let Some(candidate) = scenario(selector) {
+            if selected_names.insert(candidate.name) {
+                selected.push(candidate);
+            }
+            continue;
+        }
+
+        let prefix = format!("{selector}/");
+        let mut matched = false;
+        for candidate in scenarios()
+            .iter()
+            .filter(|candidate| candidate.name.starts_with(&prefix))
+        {
+            matched = true;
+            if selected_names.insert(candidate.name) {
+                selected.push(candidate);
+            }
+        }
+        if !matched {
+            return Err(format!(
+                "unknown scenario or group '{selector}'; run `openshell-conformance list`"
+            ));
+        }
+    }
+    Ok(selected)
 }
 
 #[cfg(test)]
@@ -213,46 +235,82 @@ mod tests {
 
     #[test]
     fn selects_all_scenarios_by_default() {
-        assert_eq!(
-            select_scenarios(&[]).expect("select all").len(),
-            scenarios().len()
-        );
+        let selected = select_scenarios(&[]).expect("select all");
+        assert_eq!(selected.len(), 10);
+        assert_eq!(selected.len(), scenarios().len());
     }
 
     #[test]
     fn selects_named_scenario() {
-        let selected = select_scenarios(&["smoke".to_string()]).expect("select smoke");
-        assert_eq!(selected[0].name, "smoke");
+        let selected = select_scenarios(&["smoke/exec".to_string()]).expect("select smoke exec");
+        assert_eq!(selected.len(), 1);
+        assert_eq!(selected[0].name, "smoke/exec");
     }
 
     #[test]
-    fn unknown_scenario_has_actionable_diagnostic() {
-        let error = select_scenarios(&["missing".to_string()]).expect_err("unknown scenario");
-        assert!(error.contains("openshell-conformance list"));
+    fn expands_group_to_all_leaf_scenarios() {
+        let selected = select_scenarios(&["smoke".to_string()]).expect("select smoke group");
+        let names = selected
+            .iter()
+            .map(|candidate| candidate.name)
+            .collect::<Vec<_>>();
+        assert_eq!(names, ["smoke/control-plane", "smoke/exec"]);
     }
 
     #[test]
-    fn selects_named_policy_scenarios() {
+    fn expands_lifecycle_group_to_independent_capabilities() {
+        let selected =
+            select_scenarios(&["sandbox-lifecycle".to_string()]).expect("select lifecycle group");
+        let names = selected
+            .iter()
+            .map(|candidate| candidate.name)
+            .collect::<Vec<_>>();
+        assert_eq!(
+            names,
+            [
+                "sandbox-lifecycle/control-plane",
+                "sandbox-lifecycle/restart-persistence",
+            ]
+        );
+    }
+
+    #[test]
+    fn overlapping_selectors_do_not_run_a_leaf_twice() {
         let selected = select_scenarios(&[
-            "mechanistic-proposal".to_string(),
-            "policy-local".to_string(),
+            "smoke".to_string(),
+            "smoke/exec".to_string(),
+            "policy-advisor".to_string(),
         ])
-        .unwrap();
+        .expect("select overlapping scenarios");
+        let names = selected
+            .iter()
+            .map(|candidate| candidate.name)
+            .collect::<Vec<_>>();
         assert_eq!(
-            selected
-                .iter()
-                .map(|scenario| scenario.name)
-                .collect::<Vec<_>>(),
-            ["mechanistic-proposal", "policy-local"]
+            names,
+            [
+                "smoke/control-plane",
+                "smoke/exec",
+                "policy-advisor/mechanistic-proposal",
+                "policy-advisor/new-hostname-proposal",
+                "policy-advisor/sandbox-local"
+            ]
         );
     }
 
+    #[test]
+    fn unknown_scenario_has_actionable_diagnostic() {
+        let error = select_scenarios(&["missing".to_string()]).expect_err("unknown scenario");
+        assert!(error.contains("unknown scenario or group 'missing'"));
+        assert!(error.contains("openshell-conformance list"));
+    }
+
     #[test]
     fn parses_binary_override_and_json_output() {
         let cli = Cli::try_parse_from([
             "openshell-conformance",
             "run",
-            "smoke",
+            "smoke/exec",
             "--openshell-bin",
             "/opt/openshell",
             "--output",

crates/openshell-conformance/README.md

@@ -0,0 +1,41 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# OpenShell conformance scenarios
+
+This crate defines reusable black-box conformance scenarios for the public
+`openshell` CLI. It owns command execution, assertions, diagnostics, generated
+resource names, and best-effort cleanup. It does not provision a gateway or
+inspect compute-driver internals.
+
+The crate is a scenario library, not the primary CI entrypoint:
+
+| Component | Responsibility |
+|---|---|
+| [`tests/suites/conformance/cli`](../../tests/suites/conformance/README.md) | Wrap exported scenarios as Cargo tests and select them according to runtime capabilities. New installed-artifact CI should use this workspace. |
+| `openshell-conformance` | Define portable scenarios and the shared `OpenShellRunner`. |
+| [`openshell-conformance-cli`](../openshell-conformance-cli) | Run registered leaf scenarios manually or from existing E2E tooling. |
+
+Each leaf Cargo test is both a selection boundary and a failure-isolation
+boundary. It creates a fresh `OpenShellRunner`, invokes one exported `Scenario`,
+and finishes cleanup independently. The standalone CLI registers the same
+leaves. An exact selector such as `smoke/exec` runs one leaf, while a family
+selector such as `smoke` runs every registered `smoke/...` leaf. Each selected
+leaf receives its own runner and cleanup lifecycle, so one failure does not hide
+results for later capabilities.
+
+Scenarios must exercise public CLI behavior, own only resources created for
+their run ID, and remain independent of unrelated gateway state. Split coverage
+when a behavior requires an optional runtime capability so environments can run
+the largest supported subset without weakening assertions. Driver internals,
+platform enforcement, and hardware qualification belong in driver-specific
+tests rather than this crate.
+
+When adding or splitting a scenario, create one leaf per independently
+selectable runtime capability, export each leaf from the library, and add a
+separate Cargo test wrapper under `tests/suites/conformance/cli`. Register every
+leaf with the standalone CLI and name it `<family>/<capability>`. Family-prefix
+selection provides the grouped manual and E2E entrypoint without introducing an
+aggregate scenario that can stop at its first failed child.

crates/openshell-conformance/src/lib.rs

@@ -25,9 +25,10 @@ use self::executor::{CliExecutionError, CliExecutor, ProcessCli};
 
 pub use scenarios::{
     FILE_TRANSFER_GIT_FILTERING_SCENARIO, FILE_TRANSFER_PATH_SAFETY_SCENARIO,
-    FILE_TRANSFER_ROUND_TRIP_SCENARIO, FILE_TRANSFER_SCENARIO, MECHANISTIC_PROPOSAL_SCENARIO,
-    NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO, SANDBOX_LIFECYCLE_SCENARIO,
-    SMOKE_SCENARIO,
+    FILE_TRANSFER_ROUND_TRIP_SCENARIO, MECHANISTIC_PROPOSAL_SCENARIO,
+    NEW_HOSTNAME_PROPOSAL_SCENARIO, SANDBOX_LIFECYCLE_CONTROL_PLANE_SCENARIO,
+    SANDBOX_LIFECYCLE_RESTART_PERSISTENCE_SCENARIO, SANDBOX_LOCAL_SCENARIO,
+    SMOKE_CONTROL_PLANE_SCENARIO, SMOKE_EXEC_SCENARIO,
 };
 
 /// An installed conformance scenario.
@@ -47,20 +48,24 @@ impl Scenario {
 }
 
 const SCENARIOS: &[Scenario] = &[
-    SMOKE_SCENARIO,
-    SANDBOX_LIFECYCLE_SCENARIO,
-    FILE_TRANSFER_SCENARIO,
+    SMOKE_CONTROL_PLANE_SCENARIO,
+    SMOKE_EXEC_SCENARIO,
+    SANDBOX_LIFECYCLE_CONTROL_PLANE_SCENARIO,
+    SANDBOX_LIFECYCLE_RESTART_PERSISTENCE_SCENARIO,
+    FILE_TRANSFER_ROUND_TRIP_SCENARIO,
+    FILE_TRANSFER_GIT_FILTERING_SCENARIO,
+    FILE_TRANSFER_PATH_SAFETY_SCENARIO,
     MECHANISTIC_PROPOSAL_SCENARIO,
     NEW_HOSTNAME_PROPOSAL_SCENARIO,
-    POLICY_LOCAL_SCENARIO,
+    SANDBOX_LOCAL_SCENARIO,
 ];
 
 /// Returns every scenario compiled into this distribution.
 pub fn scenarios() -> &'static [Scenario] {
     SCENARIOS
 }
 
-/// Finds a scenario by its stable command-line name.
+/// Finds a leaf scenario by its registered command-line name.
 pub fn scenario(name: &str) -> Option<&'static Scenario> {
     scenarios().iter().find(|candidate| candidate.name == name)
 }

crates/openshell-conformance/src/scenarios/file_transfer.rs

@@ -17,13 +17,6 @@ const COMMAND_TIMEOUT: Duration = Duration::from_mins(2);
 const TRANSFER_TIMEOUT: Duration = Duration::from_mins(5);
 const LARGE_FILE_SIZE: usize = 512 * 1024;
 
-/// Certify portable upload and download behavior through the public CLI.
-pub const FILE_TRANSFER_SCENARIO: Scenario = Scenario {
-    name: "file-transfer",
-    description: "Verify sandbox uploads, downloads, Git filtering, and path safety.",
-    run: run_file_transfer,
-};
-
 /// Certify basic file and directory upload and download behavior.
 pub const FILE_TRANSFER_ROUND_TRIP_SCENARIO: Scenario = Scenario {
     name: "file-transfer/round-trip",
@@ -45,14 +38,6 @@ pub const FILE_TRANSFER_PATH_SAFETY_SCENARIO: Scenario = Scenario {
     run: run_path_safety,
 };
 
-fn run_file_transfer(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
-    Box::pin(async move {
-        FILE_TRANSFER_ROUND_TRIP_SCENARIO.run(runner).await?;
-        FILE_TRANSFER_GIT_FILTERING_SCENARIO.run(runner).await?;
-        FILE_TRANSFER_PATH_SAFETY_SCENARIO.run(runner).await
-    })
-}
-
 fn run_round_trip(runner: &mut OpenShellRunner) -> ScenarioFuture<'_> {
     Box::pin(async move {
         let (sandbox_name, remote_root, local) = prepare_sandbox(runner, "round-trip").await?;

crates/openshell-conformance/src/scenarios/mod.rs

@@ -10,10 +10,12 @@ mod smoke;
 
 pub use file_transfer::{
     FILE_TRANSFER_GIT_FILTERING_SCENARIO, FILE_TRANSFER_PATH_SAFETY_SCENARIO,
-    FILE_TRANSFER_ROUND_TRIP_SCENARIO, FILE_TRANSFER_SCENARIO,
+    FILE_TRANSFER_ROUND_TRIP_SCENARIO,
 };
 pub use policy_behavior::{
-    MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, POLICY_LOCAL_SCENARIO,
+    MECHANISTIC_PROPOSAL_SCENARIO, NEW_HOSTNAME_PROPOSAL_SCENARIO, SANDBOX_LOCAL_SCENARIO,
 };
-pub use sandbox_lifecycle::SANDBOX_LIFECYCLE_SCENARIO;
-pub use smoke::SMOKE_SCENARIO;
+pub use sandbox_lifecycle::{
+    SANDBOX_LIFECYCLE_CONTROL_PLANE_SCENARIO, SANDBOX_LIFECYCLE_RESTART_PERSISTENCE_SCENARIO,
+};
+pub use smoke::{SMOKE_CONTROL_PLANE_SCENARIO, SMOKE_EXEC_SCENARIO};