Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ workspace namespace modes via `workspace_mode`:
gateway state; it never deletes or otherwise accesses the operator-managed
Kubernetes namespace.

In managed and operator mode, a chart-installed `ValidatingAdmissionPolicy`
matching only the gateway ServiceAccount limits the ClusterRole's workspace
grants: Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes
are admitted only in namespaces labeled as owned by this gateway and namespaces
matching the operator label selector, with Secret writes also admitted in the
credential namespace, and Namespace writes only for namespaces this gateway
owns. The
gateway cannot patch namespaces, so it cannot add ownership labels to an
existing namespace. The policy is on by default and can be disabled.

When the gateway configures `[openshell.gateway.otlp]`, Kubernetes
compute-driver spans export to the same OTLP/gRPC collector with the service
name `openshell-driver-kubernetes`. The driver preserves the gateway trace
Expand Down
27 changes: 23 additions & 4 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces.
## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:
These are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
| `ValidatingAdmissionPolicy` | `<fullname>-workspace-scope-<release namespace>` |
| `ValidatingAdmissionPolicyBinding` | `<fullname>-workspace-scope-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
The admission policy and its binding render only in `managed` and `operator`
workspace modes. By default the release creates all of them, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

Expand All @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <ver
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

In `managed` and `operator` workspace modes, also add
`--show-only templates/workspace-admission-policy.yaml`, and pass
`--set admissionPolicy.enabled=false` to the namespace-admin install below.

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:
Expand Down Expand Up @@ -73,7 +80,7 @@ currently held`.
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.
Expand Down Expand Up @@ -116,7 +123,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
In `managed` and `operator` workspace modes, the namespace-admin release also
sets `admissionPolicy.enabled=false`, which deletes the admission policy and its
binding while the gateway keeps its cluster-wide permissions. Annotate them too:

```shell
kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flags, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
Expand Down Expand Up @@ -309,6 +327,7 @@ discovery endpoint or its TLS CA.

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| admissionPolicy.enabled | bool | `true` | Install a ValidatingAdmissionPolicy that rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, Services, ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned managed namespaces and operator-selected namespaces. Secret writes are also admitted in the credential namespace. |
| affinity | object | `{}` | Affinity rules for the gateway pod. |
| agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. |
| certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). |
Expand Down
26 changes: 22 additions & 4 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces.
## Cluster-scoped vs namespaced objects

Most objects in this chart are namespaced and land in the release namespace.
Only two are cluster-scoped:
These are cluster-scoped:

| Object | Default name |
| --- | --- |
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
| `ValidatingAdmissionPolicy` | `<fullname>-workspace-scope-<release namespace>` |
| `ValidatingAdmissionPolicyBinding` | `<fullname>-workspace-scope-<release namespace>` |

By default the release creates both, so an install by a cluster-admin is
The admission policy and its binding render only in `managed` and `operator`
workspace modes. By default the release creates all of them, so an install by a cluster-admin is
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
split the install in two.

Expand All @@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <ver
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
```

In `managed` and `operator` workspace modes, also add
`--show-only templates/workspace-admission-policy.yaml`, and pass
`--set admissionPolicy.enabled=false` to the namespace-admin install below.

A namespace-admin then installs and upgrades the release with cluster-scoped
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
or the equivalent `--set`:
Expand Down Expand Up @@ -73,7 +80,7 @@ currently held`.
| --- | --- | --- |
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` |

`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
is needed there.
Expand Down Expand Up @@ -116,7 +123,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flag, and the cluster-admin
In `managed` and `operator` workspace modes, the namespace-admin release also
sets `admissionPolicy.enabled=false`, which deletes the admission policy and its
binding while the gateway keeps its cluster-wide permissions. Annotate them too:

```shell
kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-<namespace>" \
helm.sh/resource-policy=keep --overwrite
kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-<namespace>" \
helm.sh/resource-policy=keep --overwrite
```

The objects then survive the upgrade that sets the flags, and the cluster-admin
owns them from that point on. Fresh installs need no such step.

`rbac.clusterScoped.create` is independent of
Expand Down
14 changes: 11 additions & 3 deletions deploy/helm/openshell/ci/values-namespace-admin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@

# Namespace-admin overlay — renders only namespaced objects.
#
# Use this when a cluster-admin applies the gateway ClusterRole and
# ClusterRoleBinding once, out of band, and the OpenShell release is installed
# Use this when a cluster-admin applies the gateway ClusterRole,
# ClusterRoleBinding, and admission policy once, out of band, and the OpenShell
# release is installed
# and upgraded by an installer that holds no cluster-scoped permissions.
#
# Generate the cluster-scoped objects for the cluster-admin step from the same
Expand All @@ -15,11 +16,18 @@
# --set rbac.clusterScoped.create=true \
# --set agentSandbox.preflight.enabled=false \
# --show-only templates/clusterrole.yaml \
# --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
# --show-only templates/clusterrolebinding.yaml \
# --show-only templates/workspace-admission-policy.yaml | kubectl apply -f -
#
# Omit the admission policy template in shared workspace mode, which renders
# no policy.
#
# The gateway ServiceAccount name and release namespace are unchanged, so the
# pre-created ClusterRoleBinding still matches this release.

rbac:
clusterScoped:
create: false

admissionPolicy:
enabled: false
11 changes: 11 additions & 0 deletions deploy/helm/openshell/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,17 @@ shared workspace mode.
{{- uniq $names | toJson -}}
{{- end }}

{{/*
Whether to render the gateway workspace admission policy. Shared mode grants
the gateway nothing cluster-wide that the policy covers.
*/}}
{{- define "openshell.admissionPolicyEnabled" -}}
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
{{- if and .Values.admissionPolicy.enabled (ne $workspaceMode "shared") -}}
true
{{- end -}}
{{- end }}

{{/*
Namespace where Kubernetes Secret-backed provider credentials live.
*/}}
Expand Down
101 changes: 101 additions & 0 deletions deploy/helm/openshell/templates/workspace-admission-policy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

{{- if include "openshell.admissionPolicyEnabled" . }}
{{- $kubernetes := .Values.server.drivers.kubernetes }}
{{- $workspaceMode := $kubernetes.workspaceMode }}
{{- $gatewayId := .Values.server.sandboxJwt.gatewayId | default (include "openshell.fullname" .) }}
{{- $credentialSecret := "false" }}
{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
{{- $credentialSecret = printf "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == '%s'" (include "openshell.credentialKubernetesSecretsNamespace" .) }}
{{- end }}
{{- $managed := "false" }}
{{- if eq $workspaceMode "managed" }}
{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $gatewayId) }}
{{- fail "admissionPolicy requires the gateway ID (server.sandboxJwt.gatewayId or the release fullname) to be a DNS-1123 label" }}
{{- end }}
{{- $managed = printf "('openshell.ai/managed-by' in variables.labels && variables.labels['openshell.ai/managed-by'] == 'openshell' && 'openshell.ai/gateway-id' in variables.labels && variables.labels['openshell.ai/gateway-id'] == '%s')" $gatewayId }}
{{- end }}
{{- $operator := "false" }}
{{- if eq $workspaceMode "operator" }}
{{- if $kubernetes.operatorNamespaceFile }}
{{- fail "admissionPolicy cannot enforce server.drivers.kubernetes.operatorNamespaceFile; use operatorNamespaceLabel or set admissionPolicy.enabled=false" }}
{{- end }}
{{- $terms := list }}
{{- range splitList "," $kubernetes.operatorNamespaceLabel }}
{{- $term := trim . }}
{{- if not (regexMatch "^([a-z0-9]([-a-z0-9.]*[a-z0-9])?/)?[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?=([A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?)?$" $term) }}
{{- fail (printf "admissionPolicy supports only key=value terms in server.drivers.kubernetes.operatorNamespaceLabel, got %q; set admissionPolicy.enabled=false to use other selectors" $term) }}
{{- end }}
{{- $pair := splitn "=" 2 $term }}
{{- $terms = append $terms (printf "('%s' in variables.labels && variables.labels['%s'] == '%s')" $pair._0 $pair._0 $pair._1) }}
{{- end }}
{{- $operator = printf "(%s)" (join " && " $terms) }}
{{- end }}
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE", "DELETE"]
resources: ["secrets", "pods", "services", "serviceaccounts", "namespaces"]
- apiGroups: ["networking.k8s.io"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE", "DELETE"]
resources: ["networkpolicies"]
- apiGroups: ["agents.x-k8s.io"]
apiVersions: ["*"]
operations: ["CREATE", "UPDATE", "DELETE"]
resources: ["sandboxes", "sandboxes/status"]
matchConditions:
- name: gateway-service-account
expression: >-
request.userInfo.username ==
'system:serviceaccount:{{ .Release.Namespace }}:{{ include "openshell.serviceAccountName" . }}'
variables:
- name: isNamespace
expression: "request.resource.group == '' && request.resource.resource == 'namespaces'"
- name: targetNamespace
expression: >-
variables.isNamespace
? (request.operation == 'CREATE' ? object : oldObject)
: namespaceObject
- name: labels
expression: >-
has(variables.targetNamespace.metadata.labels)
? variables.targetNamespace.metadata.labels
: {}
- name: managed
expression: {{ $managed | quote }}
- name: operator
expression: {{ $operator | quote }}
- name: credentialSecret
expression: {{ $credentialSecret | quote }}
validations:
- expression: >-
variables.isNamespace
? variables.managed
: (variables.credentialSecret || variables.managed || variables.operator)
messageExpression: >-
'the OpenShell gateway may not ' + request.operation + ' ' +
request.resource.resource + ' in namespace ' +
variables.targetNamespace.metadata.name
reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
labels:
{{- include "openshell.labels" . | nindent 4 }}
spec:
policyName: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
validationActions: ["Deny"]
{{- end }}
Loading
Loading