Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitleaksignore
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,7 @@ a1b4bf1043a7f3681fdd294579be16aa76a03549:packs/kubewatch-1.0.7/values.yaml:gener
6dcc3f04f80fe8429a878a572bfc59ea94d09f93:packs/sriov-network-operator-1.6.0/values.yaml:private-key:109
c9f6f8d1c46e6bcbde37938b891053075174f1bc:packs/nvidia-nvsentinel-v1.19.0/charts/nvsentinel/charts/mongodb-store/charts/psmdb-db/values.yaml:generic-api-key:675
c9f6f8d1c46e6bcbde37938b891053075174f1bc:packs/nvidia-nvsentinel-v1.19.0/charts/nvsentinel/charts/mongodb-store/charts/psmdb-db/values.yaml:generic-api-key:678
8f96753560fe5c430a59971bb1161af69ec9bfcb:packs/nvidia-network-operator-26.7.0/charts/network-operator/charts/maintenance-operator-chart/values.yaml:private-key:75
8f96753560fe5c430a59971bb1161af69ec9bfcb:packs/nvidia-network-operator-26.7.0/charts/network-operator/charts/sriov-network-operator/values.yaml:private-key:94
8f96753560fe5c430a59971bb1161af69ec9bfcb:packs/nvidia-network-operator-26.7.0/charts/network-operator/values.yaml:private-key:140
8f96753560fe5c430a59971bb1161af69ec9bfcb:packs/nvidia-network-operator-26.7.0/values.yaml:private-key:179
234 changes: 234 additions & 0 deletions packs/nvidia-network-operator-26.7.0/README.md

Large diffs are not rendered by default.

Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
apiVersion: v2
name: network-operator
version: 26.7.0
kubeVersion: '>= 1.32.0-0'
appVersion: v26.7.0
description: Nvidia network operator
type: application
keywords:
- gpu-direct
- rdma
home: https://mellanox.github.io/network-operator
sources:
- https://github.com/Mellanox/network-operator
dependencies:
- condition: nfd.enabled
name: node-feature-discovery
repository: ''
version: 0.2.1
- condition: sriovNetworkOperator.enabled
name: sriov-network-operator
repository: ''
version: 0.1.0
- condition: maintenanceOperator.enabled
name: maintenance-operator-chart
repository: ''
version: 0.0.1
274 changes: 274 additions & 0 deletions packs/nvidia-network-operator-26.7.0/charts/network-operator/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,274 @@
# Nvidia Network Operator Helm Chart

Nvidia Network Operator Helm Chart provides an easy way to install and manage the lifecycle of Nvidia network operator.

## Nvidia Network Operator

Nvidia Network Operator
leverages [Kubernetes CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/)
and [Operator SDK](https://github.com/operator-framework/operator-sdk) to manage Networking related Components in order
to enable Fast networking, RDMA and GPUDirect for workloads in a Kubernetes cluster. Network Operator works in
conjunction with [GPU-Operator](https://github.com/NVIDIA/gpu-operator) to enable GPU-Direct RDMA on compatible systems.

The Goal of Network Operator is to manage _all_ networking related components to enable execution of RDMA and GPUDirect
RDMA workloads in a kubernetes cluster including:

* Mellanox Networking drivers to enable advanced features
* Kubernetes device plugins to provide hardware resources for fast network
* Kubernetes secondary network for Network intensive workloads

### Documentation

For more information please visit the official [documentation](https://docs.nvidia.com/networking/software/cloud-orchestration/index.html).

## Additional components

### Node Feature Discovery

Nvidia Network Operator relies on the existance of specific node labels to operate properly. e.g label a node as having
Nvidia networking hardware available. This can be achieved by either manually labeling Kubernetes nodes or using
[Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) to perform the labeling.

To allow zero touch deployment of the Operator we provide a helm chart to be used to optionally deploy Node Feature
Discovery in the cluster. This is enabled via `nfd.enabled` chart parameter.

### SR-IOV Network Operator

Nvidia Network Operator can operate in unison with SR-IOV Network Operator to enable SR-IOV workloads in a Kubernetes
cluster. We provide a helm chart to be used to optionally
deploy [SR-IOV Network Operator](https://github.com/k8snetworkplumbingwg/sriov-network-operator) in the cluster. This is
enabled via `sriovNetworkOperator.enabled` chart parameter.

SR-IOV Network Operator can work in conjuction with [IB Kubernetes](#ib-kubernetes) to use InfiniBand PKEY Membership
Types

For more information on how to configure SR-IOV in your Kubernetes cluster using SR-IOV Network Operator refer to the
project's github.

## QuickStart

### System Requirements

* RDMA capable hardware: Mellanox ConnectX-5 NIC or newer.
* NVIDIA GPU and driver supporting GPUDirect e.g Quadro RTX 6000/8000 or Tesla T4 or Tesla V100 or Tesla V100.
(GPU-Direct only)
* Operating Systems: Ubuntu 20.04 LTS.

> __NOTE__: ConnectX-6 Lx is not supported.

### Tested Network Adapters

The following Network Adapters have been tested with network-operator:

* ConnectX-5
* ConnectX-6 Dx

### Prerequisites

- Kubernetes v1.17+
- Helm v3.5.3+
- Ubuntu 20.04 LTS

### Install Helm

Helm provides an install script to copy helm binary to your system:

```
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 500 get_helm.sh
$ ./get_helm.sh
```

For additional information and methods for installing Helm, refer to the official [helm website](https://helm.sh/)

### Deploy Network Operator

```
# Add Repo
$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
$ helm repo update

# Install Operator
$ helm install -n network-operator --create-namespace --wait network-operator nvidia/network-operator

# View deployed resources
$ kubectl -n network-operator get pods
```

#### Deploy Network Operator without Node Feature Discovery

By default the network operator
deploys [Node Feature Discovery (NFD)](https://github.com/kubernetes-sigs/node-feature-discovery)
in order to perform node labeling in the cluster to allow proper scheduling of Network Operator resources. If the nodes
where already labeled by other means (either deployed from upstream or deployed within another deployment), it is possible to disable the deployment of NFD by setting
`nfd.enabled=false` chart parameter and make sure that the installed version is `v0.13.2` or newer and has NodeFeatureApi enabled.

##### Deploy NFD from upstream with NodeFeatureApi enabled
```
$ export NFD_NS=node-feature-discovery
$ helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts
$ helm repo update
$ helm install nfd/node-feature-discovery --namespace $NFD_NS --create-namespace --generate-name --set enableNodeFeatureApi='true'
```
For additional information , refer to the official [NVD deployment with Helm](https://kubernetes-sigs.github.io/node-feature-discovery/v0.13/deployment/helm.html)

##### Deploy Network Operator without Node Feature Discovery
```
$ helm install --set nfd.enabled=false -n network-operator --create-namespace --wait network-operator nvidia/network-operator
```

##### Currently the following NFD labels are used:

| Label | Where |
|-----------------------------------------------|---------------------------------------------------|
| `feature.node.kubernetes.io/pci-15b3.present` | Nodes bearing Nvidia Mellanox Networking hardware |
| `nvidia.com/gpu.present` | Nodes bearing Nvidia GPU hardware |

> __Note:__ The labels which Network Operator depends on may change between releases.

> __Note:__ By default the operator is deployed without an instance of `NicClusterPolicy` and `MacvlanNetwork`
> custom resources. The user is required to create it later with configuration matching the cluster or use chart parameters to deploy it together with the operator.

#### Deploy Network Operator with GitOps

The chart runs a `keep-ncp` hook by default to preserve a Helm-managed `NicClusterPolicy` when upgrading from an older
chart version that deployed the custom resource. GitOps controllers such as Argo CD cannot distinguish a Helm install
from an upgrade and run this hook during every sync.

For a first-time GitOps installation where there is no existing Helm-managed `NicClusterPolicy`, disable the hook and
its RBAC resources by setting:

```yaml
keepNCP: false
```

Keep the default value, `keepNCP: true`, when upgrading a deployment that might contain a Helm-managed
`NicClusterPolicy` from an older chart version.

#### Deploy development version of Network Operator

To install development version of Network Operator you need to clone repository first and install helm chart from the
local directory:

```
# Clone Network Operator Repository
$ git clone https://github.com/Mellanox/network-operator.git

# Update chart dependencies
$ cd network-operator/deployment/network-operator && helm dependency update

# Install Operator
$ helm install -n network-operator --create-namespace --wait network-operator ./

# View deployed resources
$ kubectl -n network-operator get pods
```

#### Deploy Network Operator with Admission Controller

The Admission Controller can be optionally included as part of the Network Operator installation process.
It has the capability to validate supported Custom Resource Definitions (CRDs), which currently include NicClusterPolicy and HostDeviceNetwork.
By default, the deployment of the admission controller is disabled. To enable it, you must set `operator.admissionController.enabled` to `true`.

Enabling the admission controller provides you with two options for managing certificates.
You can either utilize [cert-manager](https://cert-manager.io/docs/installation/) for generating a self-signed certificate automatically, or you can provide your own self-signed certificate.

To use `cert-manager`, ensure that `operator.admissionController.useCertManager` is set to `true`. Additionally, make sure that you deploy cert-manager before initiating the Network Operator deployment.

If you prefer not to use `cert-manager`, set `operator.admissionController.useCertManager` to `false`, and then provide your custom certificate and key using `operator.admissionController.certificate.tlsCrt` and `operator.admissionController.certificate.tlsKey`.

> __NOTE__: When using your own certificate, the certificate must be valid for <Release_Name>-webhook-service.<
> Release_Namespace>.svc, e.g. network-operator-webhook-service.network-operator.svc

> __NOTE__: When deploying network operator with admission controller using helm, you need to append `--wait` to helm install and helm upgrade commands
>

##### Generating self-signed certificate using OpenSSL

To generate a self-signed SSL certificate valid for a specific hostname, you can use the `openssl` command-line tool.
First, navigate to the directory where you want to store your certificate and key files. Then, run the following
command:

```bash
SVCNAME="network-operator-webhook-service.network-operator.svc"
openssl req -x509 -nodes -batch -newkey rsa:2048 -keyout server.key -out server.crt -days 365 -addext "subjectAltName=DNS:$SVCNAME"
```

Replace `SVCNAME` with the SVC name follows this convention <Release_Name>-webhook-service.<Release_Namespace>.svc.
This command will generate a new RSA key pair with 2048 bits and create a self-signed certificate (`server.crt`) and
private key (`server.key`) that are valid for 365 days.

#### Deploy Network Operator with Go integration coverage

For integration test coverage collection (e.g. Jinx regression), enable coverage and deploy
with the immutable instrumented image tag published by CI (`<git-sha>-coverage`):

```bash
helm install network-operator ./deployment/network-operator \
--set operator.coverage.enabled=true \
--set operator.tag=<git-sha>-coverage
```

When enabled, the controller sets `GOCOVERDIR=/coverage`, mounts an `emptyDir` at `/coverage`, and
sets `COVERAGE_CLEAR_AFTER_FLUSH=1`. Send **SIGUSR1** to the manager process to flush runtime
coverage counters without restarting the pod. On distroless images, signal the container host PID
rather than relying on `kubectl exec kill`.

## Upgrade

> __NOTE__: Upgrade capabilities are limited now. Additional manual actions required when containerized OFED driver is used

Before starting the upgrade to a specific release version, please, check release notes for this version to ensure that
no additional actions are required.


### Check available releases

```
helm search repo nvidia/network-operator -l
```

> __NOTE__: add `--devel` option if you want to list beta releases as well

### Upgrade CRDs to compatible version

The network-operator helm chart contains a hook(pre-install, pre-upgrade) that will automatically upgrade required CRDs in the cluster.
The hook is enabled by default. If you don't want to upgrade CRDs with helm automatically,
you can disable auto upgrade by setting `upgradeCRDs: false` in the helm chart values.
Then you can follow the guide below to download and apply CRDs for the concrete version of the network-operator.

It is possible to retrieve updated CRDs from the Helm chart or from the release branch on GitHub. Example bellow show
how to download and unpack Helm chart for specified release and then apply CRDs update from it.

```
helm pull nvidia/network-operator --version <VERSION> --untar --untardir network-operator-chart
```

> __NOTE__: `--devel` option required if you want to use the beta release

```
kubectl apply -f network-operator-chart/network-operator/crds \
-f network-operator-chart/network-operator/charts/sriov-network-operator/crds
```

### Prepare Helm values for the new release

Download Helm values for the specific release

```
helm show values nvidia/network-operator --version=<VERSION> > values-<VERSION>.yaml
```

Edit `values-<VERSION>.yaml` file as required for your cluster.

### Apply Helm chart update

```
helm upgrade -n network-operator network-operator nvidia/network-operator --version=<VERSION> -f values-<VERSION>.yaml --force
```

> __NOTE__: `--devel` option required if you want to use the beta release

## Chart parameters

In order to tailor the deployment of the network operator to your cluster needs, Chart parameters are available. See official [documentation](https://docs.nvidia.com/networking/software/cloud-orchestration/index.html).
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: v2
name: maintenance-operator-chart
description: Maintenance Operator Helm Chart
type: application
version: 0.0.1
appVersion: "latest"
Loading
Loading