diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 6cb277fbbb..bcbe45f123 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -27,6 +27,16 @@ workspace namespace modes via `workspace_mode`: gateway state; it never deletes or otherwise accesses the operator-managed Kubernetes namespace. +In managed and operator mode, a chart-installed `ValidatingAdmissionPolicy` +matching only the gateway ServiceAccount limits the ClusterRole's workspace +grants: Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes +are admitted only in namespaces labeled as owned by this gateway and namespaces +matching the operator label selector, with Secret writes also admitted in the +credential namespace, and Namespace writes only for namespaces this gateway +owns. The +gateway cannot patch namespaces, so it cannot add ownership labels to an +existing namespace. The policy is on by default and can be disabled. + When the gateway configures `[openshell.gateway.otlp]`, Kubernetes compute-driver spans export to the same OTLP/gRPC collector with the service name `openshell-driver-kubernetes`. The driver preserves the gateway trace diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index df3ea173ad..4fb0fa3e83 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces. ## Cluster-scoped vs namespaced objects Most objects in this chart are namespaced and land in the release namespace. -Only two are cluster-scoped: +These are cluster-scoped: | Object | Default name | | --- | --- | | `ClusterRole` | `-node-reader-` | | `ClusterRoleBinding` | `-node-reader-` | +| `ValidatingAdmissionPolicy` | `-workspace-scope-` | +| `ValidatingAdmissionPolicyBinding` | `-workspace-scope-` | -By default the release creates both, so an install by a cluster-admin is +The admission policy and its binding render only in `managed` and `operator` +workspace modes. By default the release creates all of them, so an install by a cluster-admin is unchanged. On clusters where cluster-scoped RBAC is owned by a different team, split the install in two. @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version " \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of @@ -309,6 +327,7 @@ discovery endpoint or its TLS CA. | Key | Type | Default | Description | |-----|------|---------|-------------| +| admissionPolicy.enabled | bool | `true` | Install a ValidatingAdmissionPolicy that rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, Services, ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned managed namespaces and operator-selected namespaces. Secret writes are also admitted in the credential namespace. | | affinity | object | `{}` | Affinity rules for the gateway pod. | | agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. | | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 1e86e0cbcf..e04e4c7d79 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces. ## Cluster-scoped vs namespaced objects Most objects in this chart are namespaced and land in the release namespace. -Only two are cluster-scoped: +These are cluster-scoped: | Object | Default name | | --- | --- | | `ClusterRole` | `-node-reader-` | | `ClusterRoleBinding` | `-node-reader-` | +| `ValidatingAdmissionPolicy` | `-workspace-scope-` | +| `ValidatingAdmissionPolicyBinding` | `-workspace-scope-` | -By default the release creates both, so an install by a cluster-admin is +The admission policy and its binding render only in `managed` and `operator` +workspace modes. By default the release creates all of them, so an install by a cluster-admin is unchanged. On clusters where cluster-scoped RBAC is owned by a different team, split the install in two. @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version " \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of diff --git a/deploy/helm/openshell/ci/values-namespace-admin.yaml b/deploy/helm/openshell/ci/values-namespace-admin.yaml index 7326be7328..08b59912c4 100644 --- a/deploy/helm/openshell/ci/values-namespace-admin.yaml +++ b/deploy/helm/openshell/ci/values-namespace-admin.yaml @@ -3,8 +3,9 @@ # Namespace-admin overlay — renders only namespaced objects. # -# Use this when a cluster-admin applies the gateway ClusterRole and -# ClusterRoleBinding once, out of band, and the OpenShell release is installed +# Use this when a cluster-admin applies the gateway ClusterRole, +# ClusterRoleBinding, and admission policy once, out of band, and the OpenShell +# release is installed # and upgraded by an installer that holds no cluster-scoped permissions. # # Generate the cluster-scoped objects for the cluster-admin step from the same @@ -15,7 +16,11 @@ # --set rbac.clusterScoped.create=true \ # --set agentSandbox.preflight.enabled=false \ # --show-only templates/clusterrole.yaml \ -# --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +# --show-only templates/clusterrolebinding.yaml \ +# --show-only templates/workspace-admission-policy.yaml | kubectl apply -f - +# +# Omit the admission policy template in shared workspace mode, which renders +# no policy. # # The gateway ServiceAccount name and release namespace are unchanged, so the # pre-created ClusterRoleBinding still matches this release. @@ -23,3 +28,6 @@ rbac: clusterScoped: create: false + +admissionPolicy: + enabled: false diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index ab42458759..6ec88a4c0a 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -261,6 +261,17 @@ shared workspace mode. {{- uniq $names | toJson -}} {{- end }} +{{/* +Whether to render the gateway workspace admission policy. Shared mode grants +the gateway nothing cluster-wide that the policy covers. +*/}} +{{- define "openshell.admissionPolicyEnabled" -}} +{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} +{{- if and .Values.admissionPolicy.enabled (ne $workspaceMode "shared") -}} +true +{{- end -}} +{{- end }} + {{/* Namespace where Kubernetes Secret-backed provider credentials live. */}} diff --git a/deploy/helm/openshell/templates/workspace-admission-policy.yaml b/deploy/helm/openshell/templates/workspace-admission-policy.yaml new file mode 100644 index 0000000000..d1a7df7278 --- /dev/null +++ b/deploy/helm/openshell/templates/workspace-admission-policy.yaml @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if include "openshell.admissionPolicyEnabled" . }} +{{- $kubernetes := .Values.server.drivers.kubernetes }} +{{- $workspaceMode := $kubernetes.workspaceMode }} +{{- $gatewayId := .Values.server.sandboxJwt.gatewayId | default (include "openshell.fullname" .) }} +{{- $credentialSecret := "false" }} +{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }} +{{- $credentialSecret = printf "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == '%s'" (include "openshell.credentialKubernetesSecretsNamespace" .) }} +{{- end }} +{{- $managed := "false" }} +{{- if eq $workspaceMode "managed" }} +{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $gatewayId) }} +{{- fail "admissionPolicy requires the gateway ID (server.sandboxJwt.gatewayId or the release fullname) to be a DNS-1123 label" }} +{{- end }} +{{- $managed = printf "('openshell.ai/managed-by' in variables.labels && variables.labels['openshell.ai/managed-by'] == 'openshell' && 'openshell.ai/gateway-id' in variables.labels && variables.labels['openshell.ai/gateway-id'] == '%s')" $gatewayId }} +{{- end }} +{{- $operator := "false" }} +{{- if eq $workspaceMode "operator" }} +{{- if $kubernetes.operatorNamespaceFile }} +{{- fail "admissionPolicy cannot enforce server.drivers.kubernetes.operatorNamespaceFile; use operatorNamespaceLabel or set admissionPolicy.enabled=false" }} +{{- end }} +{{- $terms := list }} +{{- range splitList "," $kubernetes.operatorNamespaceLabel }} +{{- $term := trim . }} +{{- if not (regexMatch "^([a-z0-9]([-a-z0-9.]*[a-z0-9])?/)?[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?=([A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?)?$" $term) }} +{{- fail (printf "admissionPolicy supports only key=value terms in server.drivers.kubernetes.operatorNamespaceLabel, got %q; set admissionPolicy.enabled=false to use other selectors" $term) }} +{{- end }} +{{- $pair := splitn "=" 2 $term }} +{{- $terms = append $terms (printf "('%s' in variables.labels && variables.labels['%s'] == '%s')" $pair._0 $pair._0 $pair._1) }} +{{- end }} +{{- $operator = printf "(%s)" (join " && " $terms) }} +{{- end }} +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["secrets", "pods", "services", "serviceaccounts", "namespaces"] + - apiGroups: ["networking.k8s.io"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["networkpolicies"] + - apiGroups: ["agents.x-k8s.io"] + apiVersions: ["*"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["sandboxes", "sandboxes/status"] + matchConditions: + - name: gateway-service-account + expression: >- + request.userInfo.username == + 'system:serviceaccount:{{ .Release.Namespace }}:{{ include "openshell.serviceAccountName" . }}' + variables: + - name: isNamespace + expression: "request.resource.group == '' && request.resource.resource == 'namespaces'" + - name: targetNamespace + expression: >- + variables.isNamespace + ? (request.operation == 'CREATE' ? object : oldObject) + : namespaceObject + - name: labels + expression: >- + has(variables.targetNamespace.metadata.labels) + ? variables.targetNamespace.metadata.labels + : {} + - name: managed + expression: {{ $managed | quote }} + - name: operator + expression: {{ $operator | quote }} + - name: credentialSecret + expression: {{ $credentialSecret | quote }} + validations: + - expression: >- + variables.isNamespace + ? variables.managed + : (variables.credentialSecret || variables.managed || variables.operator) + messageExpression: >- + 'the OpenShell gateway may not ' + request.operation + ' ' + + request.resource.resource + ' in namespace ' + + variables.targetNamespace.metadata.name + reason: Forbidden +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + policyName: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + validationActions: ["Deny"] +{{- end }} diff --git a/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml b/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml new file mode 100644 index 0000000000..e70cf1ff72 --- /dev/null +++ b/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: workspace admission policy +templates: + - templates/workspace-admission-policy.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: is not rendered in shared mode + asserts: + - hasDocuments: + count: 0 + + - it: is not rendered when disabled + set: + server.drivers.kubernetes.workspaceMode: managed + admissionPolicy.enabled: false + asserts: + - hasDocuments: + count: 0 + + - it: renders a denying policy and binding in managed mode + set: + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 2 + - isKind: + of: ValidatingAdmissionPolicy + documentIndex: 0 + - equal: + path: spec.failurePolicy + value: Fail + documentIndex: 0 + - isKind: + of: ValidatingAdmissionPolicyBinding + documentIndex: 1 + - equal: + path: spec.validationActions + value: ["Deny"] + documentIndex: 1 + - equal: + path: spec.policyName + value: openshell-workspace-scope-my-namespace + documentIndex: 1 + + - it: matches only the gateway ServiceAccount + set: + server.drivers.kubernetes.workspaceMode: managed + documentIndex: 0 + asserts: + - equal: + path: spec.matchConditions[0].expression + value: request.userInfo.username == 'system:serviceaccount:my-namespace:openshell' + + - it: allows gateway-owned managed namespaces by gateway ID + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxJwt.gatewayId: gw-1 + documentIndex: 0 + asserts: + - matchRegex: + path: spec.variables[3].expression + pattern: "variables.labels\\['openshell.ai/gateway-id'\\] == 'gw-1'" + - equal: + path: spec.variables[4].expression + value: "false" + + - it: allows only Secret writes in the credential namespace + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxNamespace: sandboxes + server.credentialDrivers.kubernetesSecrets.enabled: true + server.credentialDrivers.kubernetesSecrets.namespace: provider-secrets + documentIndex: 0 + asserts: + - equal: + path: spec.variables[5].expression + value: "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == 'provider-secrets'" + - notMatchRegex: + path: spec.validations[0].expression + pattern: "sandboxes" + + - it: allows no namespace exceptions without the credential driver + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxNamespace: sandboxes + documentIndex: 0 + asserts: + - equal: + path: spec.variables[5].expression + value: "false" + - equal: + path: spec.validations[0].expression + value: "variables.isNamespace ? variables.managed : (variables.credentialSecret || variables.managed || variables.operator)" + + - it: renders operator equality selectors + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceLabel: "team.example.com/openshell=enabled, tier=ai" + documentIndex: 0 + asserts: + - equal: + path: spec.variables[3].expression + value: "false" + - equal: + path: spec.variables[4].expression + value: "(('team.example.com/openshell' in variables.labels && variables.labels['team.example.com/openshell'] == 'enabled') && ('tier' in variables.labels && variables.labels['tier'] == 'ai'))" + + - it: rejects set-based operator selectors + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceLabel: "tier in (ai,ml)" + asserts: + - failedTemplate: + errorPattern: admissionPolicy supports only key=value terms + + - it: rejects operator namespace files + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceFile: /etc/openshell/namespaces.json + asserts: + - failedTemplate: + errorPattern: admissionPolicy cannot enforce + + - it: rejects gateway IDs that are not DNS labels + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxJwt.gatewayId: "gw'1" + asserts: + - failedTemplate: + errorPattern: DNS-1123 label diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 4bf21b88a8..d7f3a4752b 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -169,6 +169,17 @@ workspaceResources: # from this chart. Disable for a gateway-only release. enabled: true +# Admission policy limiting gateway workspace requests to namespaces it owns +# or is granted. Rendered in managed and operator +# workspace modes. Requires Kubernetes 1.30 or later. +admissionPolicy: + # -- Install a ValidatingAdmissionPolicy that rejects gateway requests to + # create, update, or delete Secrets, Pods, Sandboxes, Services, + # ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned + # managed namespaces and operator-selected namespaces. Secret writes are also + # admitted in the credential namespace. + enabled: true + # -- Extra annotations to add to the gateway pod. podAnnotations: {} # -- Extra labels to add to the gateway pod. diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index 9cace1a708..08cb16bed3 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -125,7 +125,7 @@ Install the software for the compute driver you use: |---|---|---| | Docker Desktop or Docker Engine | 28.0 | Required for Docker-backed gateways, local image builds, and Docker development workflows. | | Podman | 5.x | Required for Podman-backed gateways. | -| Kubernetes | 1.29 | Required for Helm deployments and Kubernetes sandbox scheduling. | +| Kubernetes | 1.30 | Required for Helm deployments and Kubernetes sandbox scheduling. | | Helm | 3.x | Required to install `deploy/helm/openshell`. | | kubectl | Compatible with your cluster | Required for Kubernetes operational inspection and secret creation. | | Host virtualization | Host dependent | Required for MicroVM-backed gateways. MicroVM uses Hypervisor.framework on macOS and KVM on Linux. | diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index 2b39ddf223..32f050b381 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -38,7 +38,7 @@ Make sure the following are in place before you install. | Prerequisite | Required | Notes | |---|---|---| -| Kubernetes 1.29+ with RBAC enabled | Yes | No additional notes. | +| Kubernetes 1.30+ with RBAC enabled | Yes | Managed and operator workspace modes install a `ValidatingAdmissionPolicy`, which is GA in 1.30. | | CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces | Yes | Verify enforcement on your cluster. | | Helm 3.x | Yes | No additional notes. | | Agent Sandbox controller and CRDs | Yes | Install before the OpenShell chart. Refer to [Install Agent Sandbox](#install-agent-sandbox). | @@ -322,8 +322,9 @@ The chart creates the following RBAC resources in the release namespace: | Role + RoleBinding | Namespace | `openshell-sandbox` | | ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-` | -Every other object the chart creates is namespaced. The `ClusterRole` and -`ClusterRoleBinding` in the last row are the only cluster-scoped objects. +Every other object the chart creates is namespaced, except the +[admission policy](#admission-policy) and its binding in `managed` and +`operator` workspace modes. When the Kubernetes Secrets credential driver is enabled, the chart also creates an `openshell-credential-secrets` Role and RoleBinding in the credential @@ -391,6 +392,10 @@ helm template openshell \ --show-only templates/clusterrolebinding.yaml | kubectl apply -f - ``` +In `managed` and `operator` workspace modes, also add +`--show-only templates/workspace-admission-policy.yaml` so the cluster-admin +applies the [admission policy](#admission-policy). + A namespace-admin then installs and upgrades the chart with cluster-scoped objects omitted. The release renders only namespaced objects and needs no permission on `clusterroles` or `clusterrolebindings`: @@ -404,6 +409,10 @@ helm upgrade --install openshell \ --set rbac.clusterScoped.create=false ``` +In `managed` and `operator` workspace modes, also pass +`--set admissionPolicy.enabled=false`; the namespace-admin cannot create the +cluster-scoped admission policy. + The gateway ServiceAccount name and namespace are unchanged by this flag, so the pre-created `ClusterRoleBinding` still binds the ServiceAccount the namespaced release creates. The same holds with `serviceAccount.create=false`: @@ -425,7 +434,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-" \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of the workspace mode. `managed` and @@ -451,7 +471,7 @@ currently held`. |---|---|---| | `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | | `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | -| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` | `managed` and `operator` render no namespaced sandbox `Role`, so no extra grant is needed there. @@ -478,6 +498,26 @@ helm template openshell \ To grant the installer the sandbox permissions instead, bind it to a Role carrying the same rules as the chart's `openshell-sandbox` Role. +### Admission policy + +In managed and operator workspace modes, the chart installs a +`ValidatingAdmissionPolicy` that applies only to the gateway ServiceAccount. It +rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, +Services, ServiceAccounts, and NetworkPolicies unless the namespace is one of: + +- A managed namespace labeled with `openshell.ai/managed-by=openshell` and this + gateway's `openshell.ai/gateway-id`. +- A namespace matching `server.drivers.kubernetes.operatorNamespaceLabel`. + +Secret writes are also admitted in the credential namespace. The gateway may +create or delete only namespaces labeled as its own. Rejected requests report +`the OpenShell gateway may not in namespace `. + +The policy supports only `key=value` terms in `operatorNamespaceLabel` and +cannot enforce `operatorNamespaceFile`; the chart fails to render with either +unsupported form. Set `admissionPolicy.enabled=false` to run without the policy, +for example on a cluster that does not serve `ValidatingAdmissionPolicy`. + ## Probes The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes: diff --git a/e2e/rust/tests/workspace_namespace_managed.rs b/e2e/rust/tests/workspace_namespace_managed.rs index a8f97cf5ef..4323157694 100644 --- a/e2e/rust/tests/workspace_namespace_managed.rs +++ b/e2e/rust/tests/workspace_namespace_managed.rs @@ -608,6 +608,88 @@ async fn managed_gateway_cannot_read_or_modify_workspace_secrets() { ); } +async fn gateway_dry_run(args: &[&str]) -> (bool, String) { + let mut full = vec!["--as", GATEWAY_SERVICE_ACCOUNT]; + full.extend_from_slice(args); + full.push("--dry-run=server"); + kubectl(&full).await +} + +struct NamespaceCleanup(String); + +impl Drop for NamespaceCleanup { + fn drop(&mut self) { + let _ = std::process::Command::new("kubectl") + .args([ + "--context", + &kube_context(), + "delete", + "namespace", + &self.0, + "--ignore-not-found", + "--wait=false", + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status(); + } +} + +#[tokio::test] +async fn managed_admission_policy_confines_gateway_to_owned_namespaces() { + // Kubernetes refuses to delete built-in namespaces before admission + // policies run, so probe an ordinary unowned namespace. + let ns = unique_workspace("e2e-unowned"); + let (ok, out) = kubectl(&["create", "namespace", &ns]).await; + assert!(ok, "failed to create namespace {ns}: {out}"); + let _cleanup = NamespaceCleanup(ns.clone()); + + for (description, args) in [ + ( + "create a Secret", + &[ + "-n", + &ns, + "create", + "secret", + "generic", + "e2e-probe", + "--from-literal=k=v", + ][..], + ), + ( + "create a Pod", + &["-n", &ns, "run", "e2e-probe", "--image=busybox"][..], + ), + ("delete a Namespace", &["delete", "namespace", &ns][..]), + ( + "create a Pod in the sandbox and credential namespace", + &["-n", "openshell", "run", "e2e-probe", "--image=busybox"][..], + ), + ] { + let (ok, out) = gateway_dry_run(args).await; + assert!( + !ok && out.contains("the OpenShell gateway may not"), + "admission policy must stop the gateway from being able to {description} outside its namespaces: {out}" + ); + } + + let (ok, out) = gateway_dry_run(&[ + "-n", + "openshell", + "create", + "secret", + "generic", + "e2e-probe", + "--from-literal=k=v", + ]) + .await; + assert!( + ok, + "admission policy must admit provider credential Secret writes: {out}" + ); +} + #[tokio::test] async fn managed_rejects_namespace_owned_by_different_gateway() { let ws = unique_workspace("mgdown"); diff --git a/e2e/rust/tests/workspace_namespace_operator.rs b/e2e/rust/tests/workspace_namespace_operator.rs index 926afa800c..6d94f984e6 100644 --- a/e2e/rust/tests/workspace_namespace_operator.rs +++ b/e2e/rust/tests/workspace_namespace_operator.rs @@ -209,6 +209,41 @@ async fn operator_gateway_has_no_secret_access_without_workspace_chart() { ); } +#[tokio::test] +async fn operator_admission_policy_confines_gateway_to_selected_namespaces() { + for (description, args) in [ + ( + "create a Pod", + &["-n", "default", "run", "e2e-probe", "--image=busybox"][..], + ), + ( + "create a Service", + &[ + "-n", + "default", + "create", + "service", + "clusterip", + "e2e-probe", + "--tcp=80", + ][..], + ), + ( + "create a Pod in the sandbox namespace", + &["-n", "openshell", "run", "e2e-probe", "--image=busybox"][..], + ), + ] { + let mut full = vec!["--as", GATEWAY_SERVICE_ACCOUNT]; + full.extend_from_slice(args); + full.push("--dry-run=server"); + let (ok, out) = kubectl(&full).await; + assert!( + !ok && out.contains("the OpenShell gateway may not"), + "admission policy must stop the gateway from being able to {description} outside selected namespaces: {out}" + ); + } +} + #[tokio::test] async fn operator_sandbox_in_labeled_namespace() { let ns = unique_namespace("op"); diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index c8b763e399..fa4f7621c1 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -153,6 +153,17 @@ kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\ kubectl auth can-i get secrets -n --as system:serviceaccount:openshell:openshell ``` +In managed and operator workspace modes, a gateway error containing `the +OpenShell gateway may not` comes from the chart's `ValidatingAdmissionPolicy`. +The target namespace is not labeled as owned by this gateway, does not match +`operatorNamespaceLabel`, and is not the credential namespace for a Secret. Check +the namespace labels and the rendered policy: + +```bash +kubectl get namespace --show-labels +kubectl get validatingadmissionpolicy -l app.kubernetes.io/instance=openshell -o yaml +``` + For a configured Vault credential driver, inspect its endpoint and trust bundle before debugging provider resolution. Non-loopback addresses must use HTTPS, and the driver never follows redirects. A private CA bundle augments platform