From 7d843b32c25644b209d3fd2851fdc625fc0dbb64 Mon Sep 17 00:00:00 2001 From: Kris Hicks Date: Tue, 22 Sep 2026 11:43:57 -0700 Subject: [PATCH] feat(helm)!: confine gateway workspace permissions with an admission policy Install a ValidatingAdmissionPolicy, on by default in managed and operator workspace modes, that matches only the gateway ServiceAccount. It admits Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes only in namespaces labeled as owned by this gateway and namespaces matching the operator selector. Secret writes are also admitted in the credential namespace when the Kubernetes Secrets credential driver is enabled. Namespace writes are admitted only for namespaces owned by this gateway, judged by their existing labels. - Skip rendering the policy when rbac.create or rbac.clusterScoped.create is false; a cluster-admin applies it with the other cluster-scoped objects. - Require Kubernetes 1.30. - Fail rendering with operatorNamespaceFile or set-based operator selectors, which the policy cannot evaluate. admissionPolicy.enabled set to false opts out. - Add e2e checks in managed and operator modes that send server-side dry-run requests as the gateway ServiceAccount outside its namespaces, including Pod creation in the sandbox and credential namespace, and assert the policy rejects them. - Document the policy and opt-out, raise the minimum Kubernetes version in the docs and support matrix, and add policy denials to the cluster debugging skill. Signed-off-by: Kris Hicks --- crates/openshell-driver-kubernetes/README.md | 10 ++ deploy/helm/openshell/README.md | 27 +++- deploy/helm/openshell/README.md.gotmpl | 26 +++- .../openshell/ci/values-namespace-admin.yaml | 14 +- deploy/helm/openshell/templates/_helpers.tpl | 11 ++ .../templates/workspace-admission-policy.yaml | 101 +++++++++++++ .../workspace_admission_policy_test.yaml | 135 ++++++++++++++++++ deploy/helm/openshell/values.yaml | 11 ++ docs/about/support-matrix.mdx | 2 +- docs/kubernetes/setup.mdx | 50 ++++++- e2e/rust/tests/workspace_namespace_managed.rs | 82 +++++++++++ .../tests/workspace_namespace_operator.rs | 35 +++++ skills/debug-openshell-cluster/SKILL.md | 11 ++ 13 files changed, 498 insertions(+), 17 deletions(-) create mode 100644 deploy/helm/openshell/templates/workspace-admission-policy.yaml create mode 100644 deploy/helm/openshell/tests/workspace_admission_policy_test.yaml diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 6cb277fbbb..bcbe45f123 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -27,6 +27,16 @@ workspace namespace modes via `workspace_mode`: gateway state; it never deletes or otherwise accesses the operator-managed Kubernetes namespace. +In managed and operator mode, a chart-installed `ValidatingAdmissionPolicy` +matching only the gateway ServiceAccount limits the ClusterRole's workspace +grants: Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes +are admitted only in namespaces labeled as owned by this gateway and namespaces +matching the operator label selector, with Secret writes also admitted in the +credential namespace, and Namespace writes only for namespaces this gateway +owns. The +gateway cannot patch namespaces, so it cannot add ownership labels to an +existing namespace. The policy is on by default and can be disabled. + When the gateway configures `[openshell.gateway.otlp]`, Kubernetes compute-driver spans export to the same OTLP/gRPC collector with the service name `openshell-driver-kubernetes`. The driver preserves the gateway trace diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index df3ea173ad..4fb0fa3e83 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces. ## Cluster-scoped vs namespaced objects Most objects in this chart are namespaced and land in the release namespace. -Only two are cluster-scoped: +These are cluster-scoped: | Object | Default name | | --- | --- | | `ClusterRole` | `-node-reader-` | | `ClusterRoleBinding` | `-node-reader-` | +| `ValidatingAdmissionPolicy` | `-workspace-scope-` | +| `ValidatingAdmissionPolicyBinding` | `-workspace-scope-` | -By default the release creates both, so an install by a cluster-admin is +The admission policy and its binding render only in `managed` and `operator` +workspace modes. By default the release creates all of them, so an install by a cluster-admin is unchanged. On clusters where cluster-scoped RBAC is owned by a different team, split the install in two. @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version " \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of @@ -309,6 +327,7 @@ discovery endpoint or its TLS CA. | Key | Type | Default | Description | |-----|------|---------|-------------| +| admissionPolicy.enabled | bool | `true` | Install a ValidatingAdmissionPolicy that rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, Services, ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned managed namespaces and operator-selected namespaces. Secret writes are also admitted in the credential namespace. | | affinity | object | `{}` | Affinity rules for the gateway pod. | | agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. | | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 1e86e0cbcf..e04e4c7d79 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces. ## Cluster-scoped vs namespaced objects Most objects in this chart are namespaced and land in the release namespace. -Only two are cluster-scoped: +These are cluster-scoped: | Object | Default name | | --- | --- | | `ClusterRole` | `-node-reader-` | | `ClusterRoleBinding` | `-node-reader-` | +| `ValidatingAdmissionPolicy` | `-workspace-scope-` | +| `ValidatingAdmissionPolicyBinding` | `-workspace-scope-` | -By default the release creates both, so an install by a cluster-admin is +The admission policy and its binding render only in `managed` and `operator` +workspace modes. By default the release creates all of them, so an install by a cluster-admin is unchanged. On clusters where cluster-scoped RBAC is owned by a different team, split the install in two. @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version " \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of diff --git a/deploy/helm/openshell/ci/values-namespace-admin.yaml b/deploy/helm/openshell/ci/values-namespace-admin.yaml index 7326be7328..08b59912c4 100644 --- a/deploy/helm/openshell/ci/values-namespace-admin.yaml +++ b/deploy/helm/openshell/ci/values-namespace-admin.yaml @@ -3,8 +3,9 @@ # Namespace-admin overlay — renders only namespaced objects. # -# Use this when a cluster-admin applies the gateway ClusterRole and -# ClusterRoleBinding once, out of band, and the OpenShell release is installed +# Use this when a cluster-admin applies the gateway ClusterRole, +# ClusterRoleBinding, and admission policy once, out of band, and the OpenShell +# release is installed # and upgraded by an installer that holds no cluster-scoped permissions. # # Generate the cluster-scoped objects for the cluster-admin step from the same @@ -15,7 +16,11 @@ # --set rbac.clusterScoped.create=true \ # --set agentSandbox.preflight.enabled=false \ # --show-only templates/clusterrole.yaml \ -# --show-only templates/clusterrolebinding.yaml | kubectl apply -f - +# --show-only templates/clusterrolebinding.yaml \ +# --show-only templates/workspace-admission-policy.yaml | kubectl apply -f - +# +# Omit the admission policy template in shared workspace mode, which renders +# no policy. # # The gateway ServiceAccount name and release namespace are unchanged, so the # pre-created ClusterRoleBinding still matches this release. @@ -23,3 +28,6 @@ rbac: clusterScoped: create: false + +admissionPolicy: + enabled: false diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index ab42458759..6ec88a4c0a 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -261,6 +261,17 @@ shared workspace mode. {{- uniq $names | toJson -}} {{- end }} +{{/* +Whether to render the gateway workspace admission policy. Shared mode grants +the gateway nothing cluster-wide that the policy covers. +*/}} +{{- define "openshell.admissionPolicyEnabled" -}} +{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} +{{- if and .Values.admissionPolicy.enabled (ne $workspaceMode "shared") -}} +true +{{- end -}} +{{- end }} + {{/* Namespace where Kubernetes Secret-backed provider credentials live. */}} diff --git a/deploy/helm/openshell/templates/workspace-admission-policy.yaml b/deploy/helm/openshell/templates/workspace-admission-policy.yaml new file mode 100644 index 0000000000..d1a7df7278 --- /dev/null +++ b/deploy/helm/openshell/templates/workspace-admission-policy.yaml @@ -0,0 +1,101 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- if include "openshell.admissionPolicyEnabled" . }} +{{- $kubernetes := .Values.server.drivers.kubernetes }} +{{- $workspaceMode := $kubernetes.workspaceMode }} +{{- $gatewayId := .Values.server.sandboxJwt.gatewayId | default (include "openshell.fullname" .) }} +{{- $credentialSecret := "false" }} +{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }} +{{- $credentialSecret = printf "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == '%s'" (include "openshell.credentialKubernetesSecretsNamespace" .) }} +{{- end }} +{{- $managed := "false" }} +{{- if eq $workspaceMode "managed" }} +{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $gatewayId) }} +{{- fail "admissionPolicy requires the gateway ID (server.sandboxJwt.gatewayId or the release fullname) to be a DNS-1123 label" }} +{{- end }} +{{- $managed = printf "('openshell.ai/managed-by' in variables.labels && variables.labels['openshell.ai/managed-by'] == 'openshell' && 'openshell.ai/gateway-id' in variables.labels && variables.labels['openshell.ai/gateway-id'] == '%s')" $gatewayId }} +{{- end }} +{{- $operator := "false" }} +{{- if eq $workspaceMode "operator" }} +{{- if $kubernetes.operatorNamespaceFile }} +{{- fail "admissionPolicy cannot enforce server.drivers.kubernetes.operatorNamespaceFile; use operatorNamespaceLabel or set admissionPolicy.enabled=false" }} +{{- end }} +{{- $terms := list }} +{{- range splitList "," $kubernetes.operatorNamespaceLabel }} +{{- $term := trim . }} +{{- if not (regexMatch "^([a-z0-9]([-a-z0-9.]*[a-z0-9])?/)?[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?=([A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?)?$" $term) }} +{{- fail (printf "admissionPolicy supports only key=value terms in server.drivers.kubernetes.operatorNamespaceLabel, got %q; set admissionPolicy.enabled=false to use other selectors" $term) }} +{{- end }} +{{- $pair := splitn "=" 2 $term }} +{{- $terms = append $terms (printf "('%s' in variables.labels && variables.labels['%s'] == '%s')" $pair._0 $pair._0 $pair._1) }} +{{- end }} +{{- $operator = printf "(%s)" (join " && " $terms) }} +{{- end }} +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["secrets", "pods", "services", "serviceaccounts", "namespaces"] + - apiGroups: ["networking.k8s.io"] + apiVersions: ["v1"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["networkpolicies"] + - apiGroups: ["agents.x-k8s.io"] + apiVersions: ["*"] + operations: ["CREATE", "UPDATE", "DELETE"] + resources: ["sandboxes", "sandboxes/status"] + matchConditions: + - name: gateway-service-account + expression: >- + request.userInfo.username == + 'system:serviceaccount:{{ .Release.Namespace }}:{{ include "openshell.serviceAccountName" . }}' + variables: + - name: isNamespace + expression: "request.resource.group == '' && request.resource.resource == 'namespaces'" + - name: targetNamespace + expression: >- + variables.isNamespace + ? (request.operation == 'CREATE' ? object : oldObject) + : namespaceObject + - name: labels + expression: >- + has(variables.targetNamespace.metadata.labels) + ? variables.targetNamespace.metadata.labels + : {} + - name: managed + expression: {{ $managed | quote }} + - name: operator + expression: {{ $operator | quote }} + - name: credentialSecret + expression: {{ $credentialSecret | quote }} + validations: + - expression: >- + variables.isNamespace + ? variables.managed + : (variables.credentialSecret || variables.managed || variables.operator) + messageExpression: >- + 'the OpenShell gateway may not ' + request.operation + ' ' + + request.resource.resource + ' in namespace ' + + variables.targetNamespace.metadata.name + reason: Forbidden +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + policyName: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }} + validationActions: ["Deny"] +{{- end }} diff --git a/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml b/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml new file mode 100644 index 0000000000..e70cf1ff72 --- /dev/null +++ b/deploy/helm/openshell/tests/workspace_admission_policy_test.yaml @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: workspace admission policy +templates: + - templates/workspace-admission-policy.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: is not rendered in shared mode + asserts: + - hasDocuments: + count: 0 + + - it: is not rendered when disabled + set: + server.drivers.kubernetes.workspaceMode: managed + admissionPolicy.enabled: false + asserts: + - hasDocuments: + count: 0 + + - it: renders a denying policy and binding in managed mode + set: + server.drivers.kubernetes.workspaceMode: managed + asserts: + - hasDocuments: + count: 2 + - isKind: + of: ValidatingAdmissionPolicy + documentIndex: 0 + - equal: + path: spec.failurePolicy + value: Fail + documentIndex: 0 + - isKind: + of: ValidatingAdmissionPolicyBinding + documentIndex: 1 + - equal: + path: spec.validationActions + value: ["Deny"] + documentIndex: 1 + - equal: + path: spec.policyName + value: openshell-workspace-scope-my-namespace + documentIndex: 1 + + - it: matches only the gateway ServiceAccount + set: + server.drivers.kubernetes.workspaceMode: managed + documentIndex: 0 + asserts: + - equal: + path: spec.matchConditions[0].expression + value: request.userInfo.username == 'system:serviceaccount:my-namespace:openshell' + + - it: allows gateway-owned managed namespaces by gateway ID + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxJwt.gatewayId: gw-1 + documentIndex: 0 + asserts: + - matchRegex: + path: spec.variables[3].expression + pattern: "variables.labels\\['openshell.ai/gateway-id'\\] == 'gw-1'" + - equal: + path: spec.variables[4].expression + value: "false" + + - it: allows only Secret writes in the credential namespace + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxNamespace: sandboxes + server.credentialDrivers.kubernetesSecrets.enabled: true + server.credentialDrivers.kubernetesSecrets.namespace: provider-secrets + documentIndex: 0 + asserts: + - equal: + path: spec.variables[5].expression + value: "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == 'provider-secrets'" + - notMatchRegex: + path: spec.validations[0].expression + pattern: "sandboxes" + + - it: allows no namespace exceptions without the credential driver + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxNamespace: sandboxes + documentIndex: 0 + asserts: + - equal: + path: spec.variables[5].expression + value: "false" + - equal: + path: spec.validations[0].expression + value: "variables.isNamespace ? variables.managed : (variables.credentialSecret || variables.managed || variables.operator)" + + - it: renders operator equality selectors + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceLabel: "team.example.com/openshell=enabled, tier=ai" + documentIndex: 0 + asserts: + - equal: + path: spec.variables[3].expression + value: "false" + - equal: + path: spec.variables[4].expression + value: "(('team.example.com/openshell' in variables.labels && variables.labels['team.example.com/openshell'] == 'enabled') && ('tier' in variables.labels && variables.labels['tier'] == 'ai'))" + + - it: rejects set-based operator selectors + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceLabel: "tier in (ai,ml)" + asserts: + - failedTemplate: + errorPattern: admissionPolicy supports only key=value terms + + - it: rejects operator namespace files + set: + server.drivers.kubernetes.workspaceMode: operator + server.drivers.kubernetes.operatorNamespaceFile: /etc/openshell/namespaces.json + asserts: + - failedTemplate: + errorPattern: admissionPolicy cannot enforce + + - it: rejects gateway IDs that are not DNS labels + set: + server.drivers.kubernetes.workspaceMode: managed + server.sandboxJwt.gatewayId: "gw'1" + asserts: + - failedTemplate: + errorPattern: DNS-1123 label diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 4bf21b88a8..d7f3a4752b 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -169,6 +169,17 @@ workspaceResources: # from this chart. Disable for a gateway-only release. enabled: true +# Admission policy limiting gateway workspace requests to namespaces it owns +# or is granted. Rendered in managed and operator +# workspace modes. Requires Kubernetes 1.30 or later. +admissionPolicy: + # -- Install a ValidatingAdmissionPolicy that rejects gateway requests to + # create, update, or delete Secrets, Pods, Sandboxes, Services, + # ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned + # managed namespaces and operator-selected namespaces. Secret writes are also + # admitted in the credential namespace. + enabled: true + # -- Extra annotations to add to the gateway pod. podAnnotations: {} # -- Extra labels to add to the gateway pod. diff --git a/docs/about/support-matrix.mdx b/docs/about/support-matrix.mdx index 9cace1a708..08cb16bed3 100644 --- a/docs/about/support-matrix.mdx +++ b/docs/about/support-matrix.mdx @@ -125,7 +125,7 @@ Install the software for the compute driver you use: |---|---|---| | Docker Desktop or Docker Engine | 28.0 | Required for Docker-backed gateways, local image builds, and Docker development workflows. | | Podman | 5.x | Required for Podman-backed gateways. | -| Kubernetes | 1.29 | Required for Helm deployments and Kubernetes sandbox scheduling. | +| Kubernetes | 1.30 | Required for Helm deployments and Kubernetes sandbox scheduling. | | Helm | 3.x | Required to install `deploy/helm/openshell`. | | kubectl | Compatible with your cluster | Required for Kubernetes operational inspection and secret creation. | | Host virtualization | Host dependent | Required for MicroVM-backed gateways. MicroVM uses Hypervisor.framework on macOS and KVM on Linux. | diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index 2b39ddf223..32f050b381 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -38,7 +38,7 @@ Make sure the following are in place before you install. | Prerequisite | Required | Notes | |---|---|---| -| Kubernetes 1.29+ with RBAC enabled | Yes | No additional notes. | +| Kubernetes 1.30+ with RBAC enabled | Yes | Managed and operator workspace modes install a `ValidatingAdmissionPolicy`, which is GA in 1.30. | | CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces | Yes | Verify enforcement on your cluster. | | Helm 3.x | Yes | No additional notes. | | Agent Sandbox controller and CRDs | Yes | Install before the OpenShell chart. Refer to [Install Agent Sandbox](#install-agent-sandbox). | @@ -322,8 +322,9 @@ The chart creates the following RBAC resources in the release namespace: | Role + RoleBinding | Namespace | `openshell-sandbox` | | ClusterRole + ClusterRoleBinding | Cluster | `openshell-node-reader-` | -Every other object the chart creates is namespaced. The `ClusterRole` and -`ClusterRoleBinding` in the last row are the only cluster-scoped objects. +Every other object the chart creates is namespaced, except the +[admission policy](#admission-policy) and its binding in `managed` and +`operator` workspace modes. When the Kubernetes Secrets credential driver is enabled, the chart also creates an `openshell-credential-secrets` Role and RoleBinding in the credential @@ -391,6 +392,10 @@ helm template openshell \ --show-only templates/clusterrolebinding.yaml | kubectl apply -f - ``` +In `managed` and `operator` workspace modes, also add +`--show-only templates/workspace-admission-policy.yaml` so the cluster-admin +applies the [admission policy](#admission-policy). + A namespace-admin then installs and upgrades the chart with cluster-scoped objects omitted. The release renders only namespaced objects and needs no permission on `clusterroles` or `clusterrolebindings`: @@ -404,6 +409,10 @@ helm upgrade --install openshell \ --set rbac.clusterScoped.create=false ``` +In `managed` and `operator` workspace modes, also pass +`--set admissionPolicy.enabled=false`; the namespace-admin cannot create the +cluster-scoped admission policy. + The gateway ServiceAccount name and namespace are unchanged by this flag, so the pre-created `ClusterRoleBinding` still binds the ServiceAccount the namespaced release creates. The same holds with `serviceAccount.create=false`: @@ -425,7 +434,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-" \ helm.sh/resource-policy=keep --overwrite ``` -The objects then survive the upgrade that sets the flag, and the cluster-admin +In `managed` and `operator` workspace modes, the namespace-admin release also +sets `admissionPolicy.enabled=false`, which deletes the admission policy and its +binding while the gateway keeps its cluster-wide permissions. Annotate them too: + +```shell +kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-" \ + helm.sh/resource-policy=keep --overwrite +``` + +The objects then survive the upgrade that sets the flags, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of the workspace mode. `managed` and @@ -451,7 +471,7 @@ currently held`. |---|---|---| | `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | | `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | -| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | +| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` | `managed` and `operator` render no namespaced sandbox `Role`, so no extra grant is needed there. @@ -478,6 +498,26 @@ helm template openshell \ To grant the installer the sandbox permissions instead, bind it to a Role carrying the same rules as the chart's `openshell-sandbox` Role. +### Admission policy + +In managed and operator workspace modes, the chart installs a +`ValidatingAdmissionPolicy` that applies only to the gateway ServiceAccount. It +rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, +Services, ServiceAccounts, and NetworkPolicies unless the namespace is one of: + +- A managed namespace labeled with `openshell.ai/managed-by=openshell` and this + gateway's `openshell.ai/gateway-id`. +- A namespace matching `server.drivers.kubernetes.operatorNamespaceLabel`. + +Secret writes are also admitted in the credential namespace. The gateway may +create or delete only namespaces labeled as its own. Rejected requests report +`the OpenShell gateway may not in namespace `. + +The policy supports only `key=value` terms in `operatorNamespaceLabel` and +cannot enforce `operatorNamespaceFile`; the chart fails to render with either +unsupported form. Set `admissionPolicy.enabled=false` to run without the policy, +for example on a cluster that does not serve `ValidatingAdmissionPolicy`. + ## Probes The gateway exposes `/healthz` for process liveness and `/readyz` for dependency-aware readiness on the health port. The Helm chart wires both into Kubernetes probes: diff --git a/e2e/rust/tests/workspace_namespace_managed.rs b/e2e/rust/tests/workspace_namespace_managed.rs index a8f97cf5ef..4323157694 100644 --- a/e2e/rust/tests/workspace_namespace_managed.rs +++ b/e2e/rust/tests/workspace_namespace_managed.rs @@ -608,6 +608,88 @@ async fn managed_gateway_cannot_read_or_modify_workspace_secrets() { ); } +async fn gateway_dry_run(args: &[&str]) -> (bool, String) { + let mut full = vec!["--as", GATEWAY_SERVICE_ACCOUNT]; + full.extend_from_slice(args); + full.push("--dry-run=server"); + kubectl(&full).await +} + +struct NamespaceCleanup(String); + +impl Drop for NamespaceCleanup { + fn drop(&mut self) { + let _ = std::process::Command::new("kubectl") + .args([ + "--context", + &kube_context(), + "delete", + "namespace", + &self.0, + "--ignore-not-found", + "--wait=false", + ]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status(); + } +} + +#[tokio::test] +async fn managed_admission_policy_confines_gateway_to_owned_namespaces() { + // Kubernetes refuses to delete built-in namespaces before admission + // policies run, so probe an ordinary unowned namespace. + let ns = unique_workspace("e2e-unowned"); + let (ok, out) = kubectl(&["create", "namespace", &ns]).await; + assert!(ok, "failed to create namespace {ns}: {out}"); + let _cleanup = NamespaceCleanup(ns.clone()); + + for (description, args) in [ + ( + "create a Secret", + &[ + "-n", + &ns, + "create", + "secret", + "generic", + "e2e-probe", + "--from-literal=k=v", + ][..], + ), + ( + "create a Pod", + &["-n", &ns, "run", "e2e-probe", "--image=busybox"][..], + ), + ("delete a Namespace", &["delete", "namespace", &ns][..]), + ( + "create a Pod in the sandbox and credential namespace", + &["-n", "openshell", "run", "e2e-probe", "--image=busybox"][..], + ), + ] { + let (ok, out) = gateway_dry_run(args).await; + assert!( + !ok && out.contains("the OpenShell gateway may not"), + "admission policy must stop the gateway from being able to {description} outside its namespaces: {out}" + ); + } + + let (ok, out) = gateway_dry_run(&[ + "-n", + "openshell", + "create", + "secret", + "generic", + "e2e-probe", + "--from-literal=k=v", + ]) + .await; + assert!( + ok, + "admission policy must admit provider credential Secret writes: {out}" + ); +} + #[tokio::test] async fn managed_rejects_namespace_owned_by_different_gateway() { let ws = unique_workspace("mgdown"); diff --git a/e2e/rust/tests/workspace_namespace_operator.rs b/e2e/rust/tests/workspace_namespace_operator.rs index 926afa800c..6d94f984e6 100644 --- a/e2e/rust/tests/workspace_namespace_operator.rs +++ b/e2e/rust/tests/workspace_namespace_operator.rs @@ -209,6 +209,41 @@ async fn operator_gateway_has_no_secret_access_without_workspace_chart() { ); } +#[tokio::test] +async fn operator_admission_policy_confines_gateway_to_selected_namespaces() { + for (description, args) in [ + ( + "create a Pod", + &["-n", "default", "run", "e2e-probe", "--image=busybox"][..], + ), + ( + "create a Service", + &[ + "-n", + "default", + "create", + "service", + "clusterip", + "e2e-probe", + "--tcp=80", + ][..], + ), + ( + "create a Pod in the sandbox namespace", + &["-n", "openshell", "run", "e2e-probe", "--image=busybox"][..], + ), + ] { + let mut full = vec!["--as", GATEWAY_SERVICE_ACCOUNT]; + full.extend_from_slice(args); + full.push("--dry-run=server"); + let (ok, out) = kubectl(&full).await; + assert!( + !ok && out.contains("the OpenShell gateway may not"), + "admission policy must stop the gateway from being able to {description} outside selected namespaces: {out}" + ); + } +} + #[tokio::test] async fn operator_sandbox_in_labeled_namespace() { let ns = unique_namespace("op"); diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index c8b763e399..fa4f7621c1 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -153,6 +153,17 @@ kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\ kubectl auth can-i get secrets -n --as system:serviceaccount:openshell:openshell ``` +In managed and operator workspace modes, a gateway error containing `the +OpenShell gateway may not` comes from the chart's `ValidatingAdmissionPolicy`. +The target namespace is not labeled as owned by this gateway, does not match +`operatorNamespaceLabel`, and is not the credential namespace for a Secret. Check +the namespace labels and the rendered policy: + +```bash +kubectl get namespace --show-labels +kubectl get validatingadmissionpolicy -l app.kubernetes.io/instance=openshell -o yaml +``` + For a configured Vault credential driver, inspect its endpoint and trust bundle before debugging provider resolution. Non-loopback addresses must use HTTPS, and the driver never follows redirects. A private CA bundle augments platform