mbp-1128: Add Network Observability with NooBaa-backed Loki #165
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,3 +22,4 @@ __pycache__/ | |
| .cursorrules | ||
| .cursor/ | ||
| .vscode/ | ||
| .venv/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| apiVersion: v2 | ||
| name: netobserv | ||
| description: OpenShift Network Observability (NooBaa-backed LokiStack + FlowCollector) for ZTVP | ||
| type: application | ||
| version: 0.1.0 | ||
| appVersion: "1.12" | ||
| keywords: | ||
| - network-observability | ||
| - zero-trust | ||
| maintainers: | ||
| - name: Validated Patterns Team | ||
| home: https://validatedpatterns.io/patterns/layered-zero-trust/ | ||
| sources: | ||
| - https://github.com/validatedpatterns/layered-zero-trust |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| {{- define "netobserv.name" -}} | ||
| netobserv | ||
| {{- end }} | ||
|
|
||
| {{- define "netobserv.labels" -}} | ||
| app.kubernetes.io/name: {{ include "netobserv.name" . }} | ||
| app.kubernetes.io/instance: {{ .Release.Name }} | ||
| app.kubernetes.io/managed-by: {{ .Release.Service }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Resolve LokiStack PVC storageClassName: | ||
| explicit value, else the cluster default StorageClass, else fail. | ||
| */}} | ||
| {{- define "netobserv.lokiStorageClassName" -}} | ||
| {{- $sc := .Values.loki.storageClassName | default "" -}} | ||
| {{- if $sc -}} | ||
| {{- $sc -}} | ||
| {{- else -}} | ||
| {{- $default := "" -}} | ||
| {{- range (lookup "storage.k8s.io/v1" "StorageClass" "" "").items | default list -}} | ||
| {{- $annotations := .metadata.annotations | default dict -}} | ||
| {{- if eq (index $annotations "storageclass.kubernetes.io/is-default-class" | default "") "true" -}} | ||
| {{- $default = .metadata.name -}} | ||
| {{- end -}} | ||
| {{- end -}} | ||
| {{- if not $default -}} | ||
| {{- fail "loki.storageClassName is empty and no default StorageClass was found; set loki.storageClassName in charts/netobserv values" -}} | ||
| {{- end -}} | ||
| {{- $default -}} | ||
| {{- end -}} | ||
| {{- end }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| {{- if .Values.enabled }} | ||
| apiVersion: flows.netobserv.io/v1beta2 | ||
| kind: FlowCollector | ||
| metadata: | ||
| name: {{ .Values.flowcollector.name }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "42" | ||
| argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true | ||
| spec: | ||
| namespace: {{ .Values.flowcollector.namespace }} | ||
| deploymentModel: {{ .Values.flowcollector.deploymentModel }} | ||
| agent: | ||
| type: {{ .Values.flowcollector.agent.type }} | ||
| ebpf: | ||
| sampling: {{ .Values.flowcollector.agent.sampling }} | ||
| privileged: {{ .Values.flowcollector.agent.privileged }} | ||
| loki: | ||
| enable: {{ .Values.flowcollector.loki.enable }} | ||
| mode: {{ .Values.flowcollector.loki.mode }} | ||
| lokiStack: | ||
| name: {{ .Values.loki.name }} | ||
| namespace: {{ .Values.loki.namespace }} | ||
| consolePlugin: | ||
| enable: {{ .Values.flowcollector.consolePlugin.enable }} | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| {{- if and .Values.enabled .Values.loki.enabled }} | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: netobserv-loki-s3-setup | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "37" | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: Role | ||
| metadata: | ||
| name: netobserv-loki-s3-setup | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "37" | ||
| rules: | ||
| - apiGroups: [""] | ||
| resources: ["secrets"] | ||
| verbs: ["get", "list", "create", "update", "patch"] | ||
| - apiGroups: [""] | ||
| resources: ["configmaps"] | ||
| verbs: ["get", "list"] | ||
| - apiGroups: ["objectbucket.io"] | ||
| resources: ["objectbucketclaims"] | ||
| verbs: ["get", "list", "watch"] | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: RoleBinding | ||
| metadata: | ||
| name: netobserv-loki-s3-setup | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "37" | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: Role | ||
| name: netobserv-loki-s3-setup | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: netobserv-loki-s3-setup | ||
| namespace: {{ .Values.loki.namespace }} | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: netobserv-loki-s3-credentials | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "39" | ||
| argocd.argoproj.io/hook: Sync | ||
| argocd.argoproj.io/hook-delete-policy: BeforeHookCreation | ||
| spec: | ||
| ttlSecondsAfterFinished: 600 | ||
| backoffLimit: {{ .Values.credentialsJob.backoffLimit }} | ||
| template: | ||
| metadata: | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 8 }} | ||
| spec: | ||
| serviceAccountName: netobserv-loki-s3-setup | ||
| automountServiceAccountToken: true | ||
| restartPolicy: OnFailure | ||
| securityContext: | ||
| runAsNonRoot: true | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| containers: | ||
| - name: remap-obc-secret | ||
| image: {{ .Values.credentialsJob.image }} | ||
| imagePullPolicy: IfNotPresent | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| runAsNonRoot: true | ||
| readOnlyRootFilesystem: true | ||
| capabilities: | ||
| drop: | ||
| - ALL | ||
| seccompProfile: | ||
| type: RuntimeDefault | ||
| resources: | ||
| {{- toYaml .Values.credentialsJob.resources | nindent 12 }} | ||
| volumeMounts: | ||
| - name: tmp | ||
| mountPath: /tmp | ||
| env: | ||
| - name: NS | ||
| value: {{ .Values.loki.namespace | quote }} | ||
| - name: OBC_NAME | ||
| value: {{ .Values.loki.objectBucketClaim.name | quote }} | ||
| - name: LOKI_SECRET | ||
| value: {{ .Values.loki.storageSecretName | quote }} | ||
| - name: REGION | ||
| value: {{ .Values.loki.region | quote }} | ||
| command: | ||
| - /bin/bash | ||
| - -ec | ||
| - | | ||
| # NooBaa OBC creates a Secret (keys) + ConfigMap (bucket/host/port) | ||
| # with the same name as the claim — not all keys live in the Secret. | ||
| echo "Checking ObjectBucketClaim ${OBC_NAME} in ${NS}..." | ||
| oc get objectbucketclaim "${OBC_NAME}" -n "${NS}" | ||
|
|
||
| echo "Waiting for ObjectBucketClaim ${OBC_NAME} to be Bound (timeout: 10 minutes)..." | ||
| oc wait --for=jsonpath='{.status.phase}'=Bound \ | ||
| "objectbucketclaim/${OBC_NAME}" -n "${NS}" --timeout=600s || { | ||
| echo "ERROR: ObjectBucketClaim failed to reach Bound state within timeout" >&2 | ||
| oc describe objectbucketclaim "${OBC_NAME}" -n "${NS}" >&2 || true | ||
| exit 1 | ||
| } | ||
|
|
||
| echo "Waiting for OBC Secret and ConfigMap ${OBC_NAME}..." | ||
| found=0 | ||
| for i in $(seq 1 60); do | ||
|
Collaborator
There was a problem hiding this comment. I'm missing error handling if resources are unavailable after 5 minutes (60 attempts). With the current logic, the script would continue executing, but the |
||
| if oc get secret "${OBC_NAME}" -n "${NS}" >/dev/null 2>&1 \ | ||
| && oc get configmap "${OBC_NAME}" -n "${NS}" >/dev/null 2>&1; then | ||
| found=1 | ||
| break | ||
| fi | ||
| sleep 5 | ||
| done | ||
| if [[ "${found}" -ne 1 ]]; then | ||
| echo "ERROR: OBC Secret and/or ConfigMap ${OBC_NAME} not available after 5 minutes" >&2 | ||
| oc describe objectbucketclaim "${OBC_NAME}" -n "${NS}" >&2 || true | ||
| exit 1 | ||
| fi | ||
|
|
||
| ACCESS=$(oc get secret "${OBC_NAME}" -n "${NS}" -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d) | ||
| SECRET=$(oc get secret "${OBC_NAME}" -n "${NS}" -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d) | ||
| BUCKET=$(oc get configmap "${OBC_NAME}" -n "${NS}" -o jsonpath='{.data.BUCKET_NAME}') | ||
| HOST=$(oc get configmap "${OBC_NAME}" -n "${NS}" -o jsonpath='{.data.BUCKET_HOST}') | ||
| PORT=$(oc get configmap "${OBC_NAME}" -n "${NS}" -o jsonpath='{.data.BUCKET_PORT}') | ||
|
|
||
| if [[ -z "${ACCESS}" || -z "${SECRET}" || -z "${BUCKET}" || -z "${HOST}" ]]; then | ||
| echo "OBC Secret/ConfigMap is missing required keys" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ -n "${PORT}" && "${PORT}" != "443" ]]; then | ||
| ENDPOINT="https://${HOST}:${PORT}" | ||
| else | ||
| ENDPOINT="https://${HOST}" | ||
| fi | ||
|
|
||
| oc create secret generic "${LOKI_SECRET}" -n "${NS}" \ | ||
| --from-literal=access_key_id="${ACCESS}" \ | ||
| --from-literal=access_key_secret="${SECRET}" \ | ||
| --from-literal=bucketnames="${BUCKET}" \ | ||
| --from-literal=endpoint="${ENDPOINT}" \ | ||
| --from-literal=region="${REGION}" \ | ||
| --dry-run=client -o yaml | oc apply -f - | ||
| echo "Created/updated ${LOKI_SECRET} for LokiStack (endpoint=${ENDPOINT})" | ||
| volumes: | ||
| - name: tmp | ||
| emptyDir: {} | ||
| {{- end }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| {{- if and .Values.enabled .Values.loki.enabled }} | ||
| apiVersion: loki.grafana.com/v1 | ||
| kind: LokiStack | ||
| metadata: | ||
| name: {{ .Values.loki.name }} | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "41" | ||
| argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true | ||
| spec: | ||
| size: {{ .Values.loki.size }} | ||
| storage: | ||
| schemas: | ||
| - version: v13 | ||
| effectiveDate: "2022-06-01" | ||
| secret: | ||
| name: {{ .Values.loki.storageSecretName }} | ||
| type: s3 | ||
| storageClassName: {{ include "netobserv.lokiStorageClassName" . }} | ||
|
Collaborator
There was a problem hiding this comment. You can make it conditional to address my earlier comment: |
||
| tenants: | ||
| mode: openshift-network | ||
| {{- end }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| {{- if and .Values.enabled .Values.loki.enabled }} | ||
| apiVersion: objectbucket.io/v1alpha1 | ||
| kind: ObjectBucketClaim | ||
| metadata: | ||
| name: {{ .Values.loki.objectBucketClaim.name }} | ||
| namespace: {{ .Values.loki.namespace }} | ||
| labels: | ||
| {{- include "netobserv.labels" . | nindent 4 }} | ||
| annotations: | ||
| argocd.argoproj.io/sync-wave: "36" | ||
| argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true | ||
| spec: | ||
| generateBucketName: {{ .Values.loki.objectBucketClaim.bucketName }} | ||
| storageClassName: {{ .Values.loki.objectBucketClaim.storageClass }} | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| enabled: true | ||
|
|
||
| loki: | ||
| enabled: true | ||
| namespace: netobserv-loki | ||
| name: loki | ||
| size: 1x.extra-small | ||
| # Empty: use the cluster default StorageClass (or set explicitly) | ||
| storageClassName: "" | ||
| storageSecretName: loki-s3 | ||
| objectBucketClaim: | ||
| name: netobserv-loki-obc | ||
| bucketName: netobserv-loki | ||
| storageClass: openshift-storage.noobaa.io | ||
| # NooBaa MCG default region used in remapped secret | ||
| region: us-east-1 | ||
|
|
||
| flowcollector: | ||
| name: cluster | ||
| namespace: netobserv | ||
| deploymentModel: Direct | ||
| agent: | ||
| type: eBPF | ||
| sampling: 50 | ||
| privileged: false | ||
| loki: | ||
| enable: true | ||
| mode: LokiStack | ||
| consolePlugin: | ||
| enable: true | ||
|
|
||
| credentialsJob: | ||
| image: registry.redhat.io/openshift4/ose-cli-rhel9:latest | ||
| backoffLimit: 5 | ||
| resources: | ||
| requests: | ||
| cpu: 50m | ||
| memory: 128Mi | ||
| limits: | ||
| cpu: 500m | ||
| memory: 256Mi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've tested this in OCP 4.22 and using Helm lookup to find the default StorageClass works with helm install but fails in ArgoCD because ArgoCD's Helm renderer doesn't support lookup. I had to set loki.storageClassName explicitly via an ArgoCD helm parameter override.
If I'm not wrong, when storageClassName is omitted from the LokiStack CR, the Loki Operator uses the cluster's default StorageClass automatically (https://docs.redhat.com/en/documentation/red_hat_openshift_logging/6.5/html/installing_logging/verifying-cluster-prerequisites).
You can update the lokistack.yaml to make storageClassName conditional (see my next comment).