test(odh): add gateway failover coverage

e2e/rust/tests/odh/README.md

@@ -30,7 +30,8 @@ e2e/rust/tests/odh/
 ├── tier2/                     # Tier 2: medium/low priority positive tests
 │   └── mod.rs                    # empty — no scenarios yet
 ├── tier3/                      # Tier 3: negative and destructive tests
-│   └── mod.rs                    # empty — no scenarios yet
+│   ├── mod.rs
+│   └── gateway_failover.rs       # external-PostgreSQL HA gateway failover
 ├── tiers.toml                  # tier → upstream test binaries + ODH module filter
 └── run-odh-test-tier.sh        # runs one tier against a deployed gateway
 ```
@@ -55,8 +56,9 @@ plain module is enough — no new crate and no workspace change.
   `tokio::process::Command` for `oc`, injecting `--context` from
   `OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE` (exported by `e2e/with-kube-gateway.sh`)
   when it is set, so every ODH test targets the same cluster the upstream
-  harness does. `oc_json()` runs a query and parses `-o json` output, panicking
-  with a descriptive message on any failure.
+  harness does. It also centralizes namespace/release resolution and Pod Ready
+  checks. `oc_json()` runs a query and parses `-o json` output, panicking with
+  a descriptive message on any failure.
 - `odh_harness::selinux::SelinuxAudit` — an opt-in scenario guard that detects
   OpenShift, requires every Ready worker to report `Enforcing`, records
   node-local audit cutoffs, and rejects OpenShell AVCs on completion.
@@ -98,8 +100,9 @@ criticality, not just copied as-is.
 
 Smoke covers gateway reachability, sandbox lifecycle, and image provenance.
 Tier 1 checks the process-supervisor SELinux label and its mapped upstream
-tests. Tier 2 and Tier 3 have no ODH scenarios yet; they run their mapped
-upstream tests and the image provenance check. Add scenarios by creating a
+tests. Tier 2 has no ODH scenarios yet; it runs its mapped upstream tests
+and the image provenance check. Tier 3 includes an opt-in  destructive
+external-PostgreSQL HA gateway failover scenario. Add scenarios by creating a
 `.rs` file under the tier's directory and declaring it with a `mod` line in
 that tier's `mod.rs`.
 
@@ -123,6 +126,28 @@ skips in the JUnit report.
   active gateway pointed at the deployed OpenShell instance (`openshell
   gateway add ...` / `openshell gateway select ...`) — the harness shells out
   to this binary and relies on its persisted config, not on any env var.
+- For `tier3::gateway_failover`, a deployment installed with
+  `deploy/helm/openshell/ci/values-high-availability.yaml`, an external
+  PostgreSQL Secret, and a configured CLI gateway name. The test discovers two
+  ready gateway pods and opens its own loopback `oc port-forward` to direct the
+  initial create and session to one pod. It confirms the session disconnects
+  when that pod is deleted, then
+  reconnects through a second port-forward to the pre-existing surviving
+  replica. This pins the check to that replica and does not exercise the
+  configured Service or Route endpoint. The workload heartbeat includes a
+  UUID generated at process start; reconnect must report the same UUID, so a
+  restarted workload does not satisfy the session check. Set
+  `OPENSHELL_GATEWAY` to the configured gateway name; it also selects the
+  gateway used by the image-provenance check that runs with every tier.
+  `OPENSHELL_ODH_HA_GATEWAY_NAME` remains a compatibility fallback when
+  `OPENSHELL_GATEWAY` is not set. The test identity needs permission to get
+  the PostgreSQL Secret referenced by `OPENSHELL_DB_URL`, create pod
+  port-forwards, and delete gateway pods. The test
+  deletes a gateway pod only when `OPENSHELL_ODH_HA_FAILOVER=1` is set. Its
+  default pod selector is the Helm
+  `app.kubernetes.io/name=openshell,app.kubernetes.io/instance=<RELEASE>`
+  selector; override it with `OPENSHELL_ODH_HA_GATEWAY_SELECTOR` for a renamed
+  chart deployment.
 
 ### The `KUBECONFIG` gotcha
 
@@ -162,6 +187,51 @@ context you happen to have active elsewhere. This means:
 | `mise run e2e:odh:tier3` | Tier 3: mapped upstream tests + ODH `tier3::` + image provenance |
 | `cargo nextest run --manifest-path e2e/rust/Cargo.toml --features e2e-odh --test odh -E 'test(=module::test_name)'` | A single ODH test function |
 
+The destructive HA failover scenario is disabled by default. Run it only
+against a disposable or explicitly approved HA deployment:
+
+```bash
+OPENSHELL_ODH_HA_FAILOVER=1 mise run e2e:odh:tier3
+```
+
+The existing `e2e/with-kube-gateway.sh` wrapper can provision the HA fixture
+in an ephemeral namespace: it installs the chart with the HA values overlay,
+deploys the PostgreSQL fixture, configures the OpenShift Route and CLI, then
+tears down its resources after the test. The Rust test itself assumes an
+already-deployed gateway, as do the other ODH tier tests. To run only this
+scenario through the wrapper, use a disposable OpenShift context and a built
+`target/debug/openshell` CLI. The wrapper's existing-context mode uses one
+registry and image tag for gateway, sandbox runtime, and supervisor; deployments
+with different image sources for those components still need explicit Helm
+overrides or a wrapper extension.
+
+```bash
+OPENSHELL_E2E_KUBE_CONTEXT=<disposable-context> \
+OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET=openshell-ha-pg \
+OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-high-availability.yaml \
+OPENSHELL_GATEWAY=<configured-gateway-name> \
+OPENSHELL_ODH_HA_FAILOVER=1 \
+e2e/with-kube-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml \
+  --features e2e-odh --test odh -- \
+  tier3::gateway_failover::gateway_pod_failover_preserves_sandbox_session_and_workspace \
+  --exact --nocapture
+```
+
+Before running failover on an existing deployment, confirm that a normal
+`smoke::sandbox::test_create_delete` passes. The failover test bounds sandbox
+creation to five minutes and requires the initial session to remain attached
+for more than two seconds before deleting any gateway pod. It retries that
+preflight with a replacement pod port-forward when `oc port-forward` stalls
+the SSH-based `sandbox connect` path. A failure after the retries means
+failover has not been exercised.
+After sandbox creation, the test awaits a sandbox deletion attempt even when a
+later assertion fails.
+
+The failover scenario explicitly uses the shared E2E workload image, like the
+smoke tests. It does not use the chart's default sandbox image: that image can
+carry a discovered policy incompatible with the gateway under test, causing
+the supervisor to exit before readiness and masking the failover behavior.
+
 Example, running the Smoke tier against a real cluster:
 
 ```bash
@@ -178,7 +248,9 @@ see below. For RHOAI images, use
 too if your deployment doesn't use the defaults (`openshell`/`openshell`).
 The Quay deployment script sets
 `sandbox.image.pullPolicy=IfNotPresent`; other deployments must
-configure it themselves.
+configure it themselves. Set `SANDBOX_NAMESPACE` only when Sandbox
+custom resources and workload Pods run in a namespace separate from the
+gateway.
 
 ### SELinux-enforcing OCP validation
 
@@ -247,6 +319,9 @@ ALLOWED_IMAGE_REGISTRY_PREFIXES="quay.io/opendatahub/,nvcr.io/nvidia/base/" \
   otherwise match a lookalike host (e.g. `registry.redhat.io` would also
   match `registry.redhat.io.attacker.example/image`).
 - `NAMESPACE`/`RELEASE` env vars default to `openshell`/`openshell`.
+  `SANDBOX_NAMESPACE` defaults to the resolved gateway namespace and selects
+  Sandbox custom resources and workload Pods; `NAMESPACE` selects gateway
+  resources.
 - The check is a registry-prefix allowlist, not an exact image/digest match.
   It checks each observed image against the configured allowed prefixes.
 - The `imagePullPolicy: IfNotPresent` check applies to every container,

e2e/rust/tests/odh/odh_harness/mod.rs

@@ -10,4 +10,5 @@
 //! collide with the upstream `harness` module.
 
 pub mod oc;
+pub mod sandbox;
 pub mod selinux;

e2e/rust/tests/odh/odh_harness/oc.rs

@@ -8,7 +8,63 @@
 //! command construction here keeps every test targeting the same cluster and
 //! reporting failures the same way.
 
+use std::process::Stdio;
+
 use serde_json::Value;
+use tokio::io::AsyncWriteExt as _;
+
+const DEFAULT_DEPLOYMENT_NAME: &str = "openshell";
+
+/// Resolves the namespace containing gateway resources for the ODH deployment.
+///
+/// The names match the standard ODH and Kubernetes e2e configuration
+/// variables.
+pub fn gateway_namespace() -> String {
+    std::env::var("NAMESPACE")
+        .or_else(|_| std::env::var("OPENSHELL_E2E_KUBE_NAMESPACE"))
+        .unwrap_or_else(|_| DEFAULT_DEPLOYMENT_NAME.to_string())
+}
+
+/// Resolves the namespace containing Sandbox custom resources and workload Pods.
+///
+/// Sandbox resources normally share the gateway namespace. Set
+/// `SANDBOX_NAMESPACE` when the compute driver uses a separate namespace.
+pub fn sandbox_namespace() -> String {
+    std::env::var("SANDBOX_NAMESPACE").unwrap_or_else(|_| gateway_namespace())
+}
+
+/// Resolves the Helm release name used by the ODH deployment under test.
+pub fn release() -> String {
+    std::env::var("RELEASE")
+        .or_else(|_| std::env::var("OPENSHELL_E2E_KUBE_RELEASE"))
+        .unwrap_or_else(|_| DEFAULT_DEPLOYMENT_NAME.to_string())
+}
+
+/// Returns whether a Kubernetes Pod JSON object is Running and Ready.
+pub fn pod_is_ready(pod: &Value) -> bool {
+    pod["status"]["phase"].as_str() == Some("Running")
+        && pod["status"]["conditions"]
+            .as_array()
+            .is_some_and(|conditions| {
+                conditions.iter().any(|condition| {
+                    condition["type"].as_str() == Some("Ready")
+                        && condition["status"].as_str() == Some("True")
+                })
+            })
+}
+
+/// Output from an `oc` invocation.
+pub struct OcOutput {
+    pub success: bool,
+    pub stdout: String,
+    pub stderr: String,
+}
+
+impl OcOutput {
+    pub fn diagnostics(&self) -> String {
+        format!("stdout:\n{}\nstderr:\n{}", self.stdout, self.stderr)
+    }
+}
 
 /// Builds an `oc` command targeting the active e2e cluster.
 ///
@@ -27,22 +83,75 @@ pub fn oc_command() -> tokio::process::Command {
     cmd
 }
 
+/// Runs `oc <args>`, optionally writing `input` to standard input.
+///
+/// Returns stdout and stderr even when the command fails, allowing tests to
+/// make assertions with the command's diagnostic output.
+pub async fn oc(args: &[&str], input: Option<&str>) -> OcOutput {
+    let mut cmd = oc_command();
+    cmd.args(args).stdout(Stdio::piped()).stderr(Stdio::piped());
+    if input.is_some() {
+        cmd.stdin(Stdio::piped());
+    }
+    let mut child = cmd.spawn().expect(
+        "failed to run `oc` — required for ODH cluster-state checks; ensure it is in PATH \\
+         and KUBECONFIG targets the cluster",
+    );
+    if let Some(input) = input {
+        child
+            .stdin
+            .take()
+            .expect("piped stdin")
+            .write_all(input.as_bytes())
+            .await
+            .expect("write manifest to oc");
+    }
+    let output = child.wait_with_output().await.expect("wait for oc");
+    OcOutput {
+        success: output.status.success(),
+        stdout: String::from_utf8_lossy(&output.stdout).into_owned(),
+        stderr: String::from_utf8_lossy(&output.stderr).into_owned(),
+    }
+}
+
+/// Returns whether the active cluster exposes the OpenShift Route API.
+///
+/// ODH-only tests use this to skip cleanly on non-OpenShift clusters while
+/// preserving the standard tier entry points.
+pub async fn is_openshift() -> bool {
+    let output = oc_command()
+        .args([
+            "api-resources",
+            "--api-group=route.openshift.io",
+            "--no-headers",
+        ])
+        .output()
+        .await
+        .expect(
+            "failed to run `oc api-resources` — cannot decide whether the cluster is OpenShift; \\
+             ensure `oc` is in PATH and KUBECONFIG targets the cluster",
+        );
+    assert!(
+        output.status.success(),
+        "oc api-resources failed:\n{}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+    !output.stdout.is_empty()
+}
+
 /// Runs `oc <args>` and parses stdout as JSON.
 ///
 /// Panics with a descriptive message if `oc` cannot be launched, exits
 /// non-zero, or does not return valid JSON — use it for `-o json` queries
 /// whose failure should fail the test.
 pub async fn oc_json(args: &[&str]) -> Value {
-    let output = oc_command().args(args).output().await.expect(
-        "failed to run `oc` — required for ODH cluster-state checks; ensure it is in PATH \
-         and KUBECONFIG targets the cluster",
-    );
+    let output = oc(args, None).await;
     assert!(
-        output.status.success(),
-        "oc {args:?} failed: {}",
-        String::from_utf8_lossy(&output.stderr)
+        output.success,
+        "oc {args:?} failed:\n{}",
+        output.diagnostics()
     );
-    serde_json::from_slice(&output.stdout)
+    serde_json::from_str(&output.stdout)
         .unwrap_or_else(|e| panic!("oc {args:?} did not return valid JSON: {e}"))
 }
 
@@ -203,9 +312,32 @@ mod tests {
     use serde_json::json;
 
     use super::{
-        pod_node_and_uid, pod_uid_cgroup_form, sandbox_id_from_json, supervisor_pod_from_json,
+        pod_is_ready, pod_node_and_uid, pod_uid_cgroup_form, sandbox_id_from_json,
+        supervisor_pod_from_json,
     };
 
+    #[test]
+    fn recognizes_only_running_ready_pods() {
+        assert!(pod_is_ready(&json!({
+            "status": {
+                "phase": "Running",
+                "conditions": [{"type": "Ready", "status": "True"}]
+            }
+        })));
+        assert!(!pod_is_ready(&json!({
+            "status": {
+                "phase": "Pending",
+                "conditions": [{"type": "Ready", "status": "True"}]
+            }
+        })));
+        assert!(!pod_is_ready(&json!({
+            "status": {
+                "phase": "Running",
+                "conditions": [{"type": "Ready", "status": "False"}]
+            }
+        })));
+    }
+
     #[test]
     fn resolves_sandbox_id_from_named_sandbox() {
         let sandbox = json!({

e2e/rust/tests/odh/odh_harness/sandbox.rs

@@ -0,0 +1,34 @@
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Helpers for resolving OpenShell sandbox resources on an ODH cluster.
+
+use serde_json::Value;
+
+use super::oc::oc_json;
+
+/// Returns the pod selector reported by the Sandbox custom resource.
+///
+/// The sandbox-agent controller owns pod creation and does not propagate the
+/// OpenShell sandbox-name label to its pod. The custom resource's status is
+/// therefore the stable way for downstream tests to discover that pod.
+pub async fn sandbox_pod_selector(namespace: &str, sandbox_name: &str) -> Option<String> {
+    let selector = format!("openshell.ai/sandbox-name={sandbox_name}");
+    let sandboxes: Value = oc_json(&[
+        "get",
+        "sandboxes.agents.x-k8s.io",
+        "-n",
+        namespace,
+        "-l",
+        &selector,
+        "-o",
+        "json",
+    ])
+    .await;
+    sandboxes
+        .get("items")
+        .and_then(Value::as_array)
+        .and_then(|items| items.first())
+        .and_then(|sandbox| sandbox["status"]["selector"].as_str())
+        .map(str::to_string)
+}