feat: Multi Namespace support by camel-k operator, Namespace watcher …

[timmy-mathew-ah]

Summary

This PR is intended primarily as a proof of concept and a source of inspiration rather than something that is expected to be merged in its current form even though the implementation has been fully tested with the code before the release of this fix: #6616 which solves the issue but with a static list of namespaces rather than dynamic.

This PR introduces support for running a namespaced Camel K operator across multiple namespaces without requiring cluster-wide resource access. In addition to supporting a statically configured list of namespaces, the operator can dynamically discover namespaces based on Kubernetes labels and automatically adjust the set of watched namespaces as labels are added or removed.

Because controller-runtime does not allow the watched namespace set to be modified after the manager has been created, the implementation uses a graceful self-restart strategy whenever the effective namespace set changes. This allows the operator to rebuild its caches with the updated namespace configuration while preserving the security benefits of namespaced RBAC.

The implementation is fully compatible with the existing operator sharding mechanism and supports multiple operator shards watching the same dynamically discovered namespaces.

Detailed Design

This PR introduces two new, fully backward-compatible capabilities for the namespaced (non-global) Camel K operator:

1. Static multi-namespace support – WATCH_NAMESPACE now accepts a comma-separated list of namespaces.
2. Dynamic namespace discovery – a new WATCH_NAMESPACE_SELECTOR environment variable enables the operator to automatically discover namespaces based on Kubernetes labels and update the watched namespace set as namespaces are added or removed.

Motivation

On large multi-tenant clusters, running the operator in global mode requires cluster-wide access to workloads, secrets, and other resources, significantly increasing the blast radius if the operator is compromised. Conversely, the existing single-namespace deployment model does not scale well when many teams require their own namespaces.

This proposal provides a middle ground by allowing the operator to watch only explicitly authorized namespaces while keeping RBAC scoped to those namespaces instead of cluster level. Static namespace lists and dynamic label-based discovery are both supported.

Controller-runtime constraint

The design is primarily driven by a limitation of controller-runtime. The namespace scope configured through cache.Options.DefaultNamespaces is fixed when the manager is created and cannot be modified while the process is running. Although multiple namespaces are already supported, there is no supported mechanism for dynamically adding or removing namespaces from an existing cache.

Several approaches were evaluated:

1. Cluster-scoped cache with filtering
   Rejected because it requires cluster-wide resource permissions, defeating the primary security objective.

2. One cache per namespace managed dynamically
   Rejected because it would require substantial changes to controller wiring and significantly increase implementation complexity.

3. Graceful restart when the watched namespace set changes
   Chosen because it preserves scoped RBAC while allowing dynamic namespace discovery. When the effective watched namespace set changes, the operator gracefully shuts down, rebuilds its caches, and resumes reconciliation using the updated namespace configuration. Since reconciliation is level-based, no work is lost during the restart.

Implementation Overview

Static multi-namespace support

When WATCH_NAMESPACE contains multiple comma-separated namespaces (for example team-a,team-b,team-c), the operator watches those namespaces together with its own installation namespace.

• Existing single-namespace behavior remains unchanged.
• Leaving WATCH_NAMESPACE empty or unset continues to enable global mode.
• No additional cluster-scoped RBAC is required beyond the existing namespaced Role and RoleBinding resources in each watched namespace.

Dynamic namespace discovery

When WATCH_NAMESPACE_SELECTOR is configured (for example camel-k-enabled=true), the operator computes its watched namespace set as the union of:

• the operator namespace,
• namespaces specified in WATCH_NAMESPACE, and
• all namespaces matching the configured label selector.

A dedicated cluster-scoped namespace watcher monitors namespace events. When a namespace begins matching or stops matching the selector, the operator performs a graceful restart so that controller-runtime rebuilds its caches using the updated namespace set.

Before adding a namespace to the watched set, the operator performs a SelfSubjectAccessReview to verify that it has permission to watch Camel K resources in that namespace. Namespaces lacking the required RBAC are skipped with a warning rather than preventing the operator from starting or synchronizing its caches.

To avoid excessive restarts, namespace changes are debounced and periodically re-evaluated. This also allows deployments where RBAC is installed after labels have already been applied to recover automatically without requiring any manual intervention.

The namespace watcher itself is leader-election aware, ensuring that only the active leader monitors namespace changes and initiates restarts.

Operator sharding compatibility

The implementation is fully compatible with the existing operator sharding model. Multiple operator shards can independently discover and watch the same label-selected namespaces while reconciling only resources assigned to their respective operator.id.

The accompanying RBAC helper also supports granting permissions to multiple operator ServiceAccounts within a single namespace, simplifying deployments that use multiple shards.

What this PR includes

• Support for static multi-namespace watching through WATCH_NAMESPACE.
• Support for dynamic namespace discovery through WATCH_NAMESPACE_SELECTOR.
• A cluster-scoped namespace watcher that gracefully restarts the operator when the effective watched namespace set changes.
• Pre-flight RBAC validation using SelfSubjectAccessReview.
• Debouncing and periodic reconciliation of namespace membership changes.
• Helm chart support for both static and dynamic configuration.
• Minimal additional cluster-scoped RBAC limited to namespace discovery.
• Documentation covering configuration and installation.
• Unit tests covering namespace computation and watcher behavior.

The remaining sections describe the implementation details, installation examples, sharded deployment configuration, and related Helm changes.

Installing an operator that watches multiple namespaces by label

Install into the operator namespace (e.g. camel-k) in dynamic mode:

helm install camel-k ./helm/camel-k -n camel-k --create-namespace \
  --set-string operator.global=false \
  --set operator.watchNamespaceSelector="camel-k-enabled=true"

This renders the operator + namespaced RBAC in camel-k, plus a minimal cluster-scoped
namespaces-read ClusterRole bound to the operator SA.

--set-string matters for operator.global: the chart compares it as a string, so a bare
--set operator.global=false (parsed as a boolean) fails template rendering.

Onboard each team namespace — install the RBAC first, then label it (the label is the "ready"
signal):

# 1. Grant the operator its namespaced rights in the target namespace,
#    bound to the operator's ServiceAccount in the operator namespace.
./script/install_namespace_rbac.sh team-a camel-k camel-k-operator

# 2. Label the namespace so the operator starts watching it.
kubectl label namespace team-a camel-k-enabled=true

The operator detects the change and self-restarts to watch team-a. If you label before installing
RBAC, the namespace is skipped with a warning and self-heals on the next periodic re-evaluation —
no re-labelling needed. Removing the label (or deleting the namespace) makes the operator stop
watching it; Integrations already running there keep running.

Static alternative (no cluster RBAC, fixed set, changes require helm upgrade):

helm install camel-k ./helm/camel-k -n camel-k --create-namespace \
  --set-string operator.global=false \
  --set "operator.watchNamespaces={team-a,team-b}"
./script/install_namespace_rbac.sh team-a camel-k camel-k-operator
./script/install_namespace_rbac.sh team-b camel-k camel-k-operator

Combining with multiple operator shards (optional)

Multiple operator releases can run side by side (each with a distinct operator.operatorId, hence a
distinct leader lease) and all watch the same label-selected namespaces, reconciling only the
resources whose camel.apache.org/operator.id annotation matches their own id. For this model,
grant every shard's ServiceAccount access in each watched namespace with a single multi-subject
binding:

./script/install_namespace_rbac.sh team-a camel-k camel-k-shard-1-operator,camel-k-shard-2-operator

[timmy-mathew-ah]

@squakez As discussed here's the PR I've created as inspiration for dynamic watching of namespace using labels. It was created before your fix was recently released with static namespaces so I haven't modified or merged that code here.

[squakez]

Thanks for the work. It is interesting to see a different approach to solve the problem identified.

The goal of the recent works we're doing is to make the project lighter for an easier long term maintenance and I think that adding a new set of informers to watch namespaces is something that we should avoid for that reason. Also, the very same behavior can be easily performed with maintaining the static WATCH_NAMESPACE variable and updating the operator deployment when a new namespace has to be watched. If any, you can do a simple external sidecar container doing the same but separate from the main operator duties.

The goal of the new multi tenancy logic is to make sure the operator is picking the namespaces to watch by an admin, instead of letting the user picking them among any available. This is also a security measure we can discuss privately.