Skip to content

[PERFSCALE-5187] Enable Control plane baseline tests on existing BM CPT Lab for Telco Core RDS - OCP 5.0 Nightly builds #83169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sraviteja-maker wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[PERFSCALE-5187] Enable Control plane baseline tests on existing BM CPT Lab for Telco Core RDS - OCP 5.0 Nightly builds #83169

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a2ef598
[PERFSCALE-5187] Enable telco-core daily job for OCP 5.0 Nightly builds
sraviteja-maker Aug 10, 2026
150ab4d
[PERFSCALE-5187] Add IDMS fallback mechanism and broader OCP version …
sraviteja-maker Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .../openshift-eng/ocp-perfscale/openshift-eng-ocp-perfscale-main__metal-5.0-nightly-x86.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,47 @@ tests:
- ref: openshift-qe-installer-bm-day2-network-workloads-post
- ref: openshift-qe-egressip
- ref: openshift-qe-evpn
- as: daily-telco-core-6nodes
capabilities:
- intranet
cron: 0 19 * * *
reporter_config:
channel: '#ocp-qe-scale-ci-results'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :white_check_mark: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> :white_check_mark:
{{else}} :warning: Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View
logs> :warning: {{end}}'
restrict_network_access: false
steps:
allow_skip_on_success: true
cluster_profile: metal-perfscale-cpt
env:
CHURN_CYCLES: "2"
CHURN_PERCENT: "50"
EXTRA_FLAGS: --perf-profile=cpt-pao --gc-metrics=false --gc=true --phased=true
--idle-duration=15m
ITERATIONS: "15"
LABEL: node-role.kubernetes.io/worker-dpdk,node-role.kubernetes.io/worker-metallb
OCP_BUILD: dev
OPERATOR_SOURCE_INDEX: prega-operator-index
RUN_ORION: "true"
TELCO: "true"
post:
- ref: openshift-qe-installer-bm-gather-extra
- ref: openshift-qe-installer-bm-must-gather
test:
- ref: openshift-qe-installer-bm-day2-label
- ref: openshift-qe-installer-bm-day2-prega-operators
- ref: openshift-qe-installer-bm-day2-sriov
- ref: openshift-qe-installer-bm-day2-metallb
- ref: openshift-qe-installer-bm-day2-perf-profile-rds-core
- ref: openshift-qe-rds-core
- ref: openshift-qe-orion-rds-core
workflow: openshift-qe-installer-bm-deploy
Comment on lines +110 to +150

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

make update
git status --short
git diff --check

Repository: openshift/release

Length of output: 197

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- git status ---\n'
git status --short

printf '\n--- Makefile metadata generation references ---\n'
if [ -f Makefile ]; then
  rg -n 'update|zz_generated_metadata|ci-operator|Prow|prow|metadata' Makefile || true
  printf '\n--- Makefile update target context ---\n'
  sed -n '/^update:/,/^$/p' Makefile | head -80
else
  printf 'No Makefile found\n'
fi

printf '\n--- Relevant diff summary ---\n'
git diff --stat
printf '\n--- Diff for modified CI config ---\n'
git diff -- ci-operator/config/openshift-eng/ocp-perfscale/openshift-eng-ocp-perfscale-main__metal-5.0-nightly-x86.yaml | sed -n '1,220p'

Repository: openshift/release

Length of output: 246

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- git status ---'
git status --short

printf '%s\n' ''
printf '%s\n' '--- Makefile generation references ---'
if [ -f Makefile ]; then
  rg -n 'update|zz_generated_metadata|ci-operator|Prow|prow|metadata' Makefile || true
  printf '%s\n' ''
  printf '%s\n' '--- Makefile update target context ---'
  sed -n '/^update:/,/^$/p' Makefile | head -80
else
  printf '%s\n' 'No Makefile found'
fi

printf '%s\n' ''
printf '%s\n' '--- Relevant diff summary ---'
git diff --stat
printf '%s\n' ''
printf '%s\n' '--- Diff for modified CI config ---'
git diff -- ci-operator/config/openshift-eng/ocp-perfscale/openshift-eng-ocp-perfscale-main__metal-5.0-nightly-x86.yaml | sed -n '1,220p'

Repository: openshift/release

Length of output: 11712

Regenerate CI configuration artifacts before merge.

This change adds a scheduled CI job under ci-operator/config/openshift-eng/ocp-perfscale. Run make update and include any generated Prow job config, determinized CI config, and registry metadata changes.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift-eng/ocp-perfscale/openshift-eng-ocp-perfscale-main__metal-5.0-nightly-x86.yaml`
around lines 110 - 150, Run make update to regenerate artifacts for the new
daily-telco-core-6nodes scheduled job, then include all resulting generated Prow
job configuration, determinized CI configuration, and registry metadata changes
in the commit.

Sources: Coding guidelines, Learnings

zz_generated_metadata:
branch: main
org: openshift-eng
Expand Down
95 changes: 95 additions & 0 deletions ci-operator/jobs/openshift-eng/ocp-perfscale/openshift-eng-ocp-perfscale-main-periodics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5235,6 +5235,101 @@ periodics:
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build12
cron: 0 19 * * *
decorate: true
decoration_config:
skip_cloning: true
extra_refs:
- base_ref: main
org: openshift-eng
repo: ocp-perfscale
labels:
capability/intranet: intranet
ci-operator.openshift.io/cloud: metal-perfscale-cpt
ci-operator.openshift.io/cloud-cluster-profile: metal-perfscale-cpt
ci-operator.openshift.io/variant: metal-5.0-nightly-x86
ci.openshift.io/generator: prowgen
job-release: "5.0"
pj-rehearse.openshift.io/can-be-rehearsed: "true"
name: periodic-ci-openshift-eng-ocp-perfscale-main-metal-5.0-nightly-x86-daily-telco-core-6nodes
reporter_config:
slack:
channel: '#ocp-qe-scale-ci-results'
job_states_to_report:
- success
- failure
- error
report_template: '{{if eq .Status.State "success"}} :white_check_mark: Job *{{.Spec.Job}}*
ended with *{{.Status.State}}*. <{{.Status.URL}}|View logs> :white_check_mark:
{{else}} :warning: Job *{{.Spec.Job}}* ended with *{{.Status.State}}*. <{{.Status.URL}}|View
logs> :warning: {{end}}'
spec:
containers:
- args:
- --gcs-upload-secret=/secrets/gcs/service-account.json
- --image-import-pull-secret=/etc/pull-secret/.dockerconfigjson
- --lease-server-credentials-file=/etc/boskos/credentials
- --report-credentials-file=/etc/report/credentials
- --secret-dir=/secrets/ci-pull-credentials
- --target=daily-telco-core-6nodes
- --variant=metal-5.0-nightly-x86
command:
- ci-operator
env:
- name: HTTP_SERVER_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
image: quay-proxy.ci.openshift.org/openshift/ci:ci_ci-operator_latest
imagePullPolicy: Always
name: ""
ports:
- containerPort: 8080
name: http
resources:
requests:
cpu: 10m
volumeMounts:
- mountPath: /etc/boskos
name: boskos
readOnly: true
- mountPath: /secrets/ci-pull-credentials
name: ci-pull-credentials
readOnly: true
- mountPath: /secrets/gcs
name: gcs-credentials
readOnly: true
- mountPath: /secrets/manifest-tool
name: manifest-tool-local-pusher
readOnly: true
- mountPath: /etc/pull-secret
name: pull-secret
readOnly: true
- mountPath: /etc/report
name: result-aggregator
readOnly: true
serviceAccountName: ci-operator
volumes:
- name: boskos
secret:
items:
- key: credentials
path: credentials
secretName: boskos-credentials
- name: ci-pull-credentials
secret:
secretName: ci-pull-credentials
- name: manifest-tool-local-pusher
secret:
secretName: manifest-tool-local-pusher
- name: pull-secret
secret:
secretName: registry-pull-credentials
- name: result-aggregator
secret:
secretName: result-aggregator
- agent: kubernetes
cluster: build12
cron: 30 0 * * *
Expand Down
38 changes: 29 additions & 9 deletions ...taller/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,39 @@ bastion=$(cat ${CLUSTER_PROFILE_DIR}/address)
get_idms_manifest() {
echo "Getting the ImageDigestMirrorSet manifest from the PREGA build server"
QUAY_URL="https://quay.io/api/v1/repository/prega/prega-operator-index/tag/?limit=100&page=1"
OCP_VERSION=$(oc get clusterversion --no-headers | grep -o '[4].[0-9][0-9]' | head -1 | awk '{print "v"$0}')
OCP_VERSION=$(oc get clusterversion --no-headers | grep -oE '[0-9]+\.[0-9]+' | head -1 | awk '{print "v"$0}')
DIGEST=$(curl -s -H "Authorization: Bearer ${QUAY_ACCESS_TOKEN}" ${QUAY_URL} | jq -r --arg tag "$OCP_VERSION" '.tags[] | select(.name == $tag) | .manifest_digest' | head -1)
OPERATOR_PREGA_VERSION=$(curl -s -H "Authorization: Bearer ${QUAY_ACCESS_TOKEN}" ${QUAY_URL} | jq -r --arg digest "$DIGEST" --arg tag "$OCP_VERSION" '.tags[] | select(.manifest_digest == $digest and .name != $tag) | .name' | sort -u)
Comment on lines 17 to 18

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='ci-operator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh'
rg -n -C 3 'set .*x|set -o xtrace|set \+x|QUAY_ACCESS_TOKEN' "$file"

Repository: openshift/release

Length of output: 1496

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='cioperator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh'

printf 'Tracing/token occurrences in %s:\n' "$file"
rg -n -C 2 'set .*x|set -o xtrace|set \+x|QUAY_ACCESS_TOKEN|curl -s -H "Authorization: Bearer \${QUAY_ACCESS_TOKEN}"|^ *set -x$' "$file" || true

printf '\nFirst 90 lines:\n'
sed -n '1,90p' "$file"

Repository: openshift/release

Length of output: 801

🏁 Script executed:

#!/bin/bash
set -euo pipefail

file='ci-operator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh'

printf 'Tracing/token occurrences in %s:\n' "$file"
rg -n -C 2 'set .*x|set -o xtrace|set \+x|QUAY_ACCESS_TOKEN|Bearer \${QUAY_ACCESS_TOKEN}|^ *set -x$' "$file" || true

printf '\nFirst 90 lines:\n'
sed -n '1,90p' "$file"

python3 - <<'PY'
from pathlib import Path
import re

p = Path('ci-operator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh')
lines = p.read_text().splitlines()
sensitive_pat = re.compile(r'Authorization:\s*Bearer\s+\${QUAY_ACCESS_TOKEN}')
for i, line in enumerate(lines, 1):
    if sensitive_pat.search(line):
        # Walk back to find nearest preceding set -x / set -o xtrace
        enabled_before = False
        disabled_since = None
        for j in range(i, 0, -1):
            stripped = lines[j-1].strip()
            if stripped in ('set -x', 'set -o xtrace') or stripped.startswith('set -o xtrace '):
                enabled_before = True
                break
            if stripped in ('set +x', 'set +o xtrace') or stripped.startswith('set +o xtrace '):
                disabled_since = j
                break
        print(f"line {i}: enabled_before={enabled_before} since_disabled_line={disabled_since}")
        print(f"  {line}")
PY

Repository: openshift/release

Length of output: 5445

Disable xtrace around the Quay token requests.

The script enables set -x in its prologue, but lines 17-18 expand QUAY_ACCESS_TOKEN in curl commands. Surround these authenticated requests with set +x / set -x to keep the bearer token out of CI logs.

🧰 Tools
🪛 Shellcheck (0.11.0)

[info] 17-17: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 18-18: Double quote to prevent globbing and word splitting.

(SC2086)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh`
around lines 17 - 18, Disable shell tracing before the authenticated curl
requests that expand QUAY_ACCESS_TOKEN in the DIGEST and OPERATOR_PREGA_VERSION
assignments, then re-enable tracing immediately afterward with set +x and set
-x. Preserve the existing command behavior and ordering.

Source: Coding guidelines

if [[ -z "${OPERATOR_PREGA_VERSION}" ]]; then
echo "OPERATOR_PREGA_VERSION could not be resolved from Quay; falling back to OCP_VERSION: ${OCP_VERSION}"
OPERATOR_PREGA_VERSION="${OCP_VERSION}"
fi
echo "PREGA Operator Version: ${OPERATOR_PREGA_VERSION} for OCP Version: ${OCP_VERSION}"
ssh ${SSH_ARGS} root@${bastion} "
set -e
set -o pipefail
curl -k -o /tmp/idms.yaml https://${PREGA_BUILD_SERVER_IP}/${OPERATOR_PREGA_VERSION}/imageDigestMirrorSet.yaml
curl -k -o /tmp/idms_${OCP_VERSION}.yaml https://${PREGA_BUILD_SERVER_IP}/${OPERATOR_PREGA_VERSION}/imageDigestMirrorSet.yaml
"
scp -q ${SSH_ARGS} root@${bastion}:/tmp/idms.yaml /tmp/idms.yaml
echo "ImageDigestMirrorSet manifest saved to /tmp/idms.yaml"
scp -q ${SSH_ARGS} root@${bastion}:/tmp/idms_${OCP_VERSION}.yaml /tmp/idms_${OCP_VERSION}.yaml

validate_yaml() {
local f="$1"
[[ -s "$f" ]] || return 1
if command -v yq &>/dev/null; then
yq eval 'has("kind") and has("apiVersion")' "$f" 2>/dev/null | grep -q '^true$'
else
grep -qE '^(apiVersion|kind):' "$f"
fi
}

if ! validate_yaml /tmp/idms_${OCP_VERSION}.yaml; then
echo "Downloaded /tmp/idms_${OCP_VERSION}.yaml is not valid YAML; falling back to bastion artifact for ${OCP_VERSION}"
scp -q ${SSH_ARGS} root@${bastion}:/root/prega_artifacts/idms_${OCP_VERSION}.yaml /tmp/idms_${OCP_VERSION}.yaml
validate_yaml /tmp/idms_${OCP_VERSION}.yaml \
|| { echo "Fallback IDMS /root/prega_artifacts/idms_${OCP_VERSION}.yaml is also invalid or missing"; exit 1; }
Comment on lines +31 to +45

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Validate the expected ImageDigestMirrorSet schema.

validate_yaml accepts any nonempty YAML object with apiVersion and kind. A Kubernetes Status response or an unrelated resource then bypasses the fallback on Lines 41-46. The later oc apply can fail or apply an unintended resource.

Require apiVersion: config.openshift.io/v1 and kind: ImageDigestMirrorSet for both sources.

Proposed fix 
     [[ -s "$f" ]] || return 1
     if command -v yq &>/dev/null; then
-      yq eval 'has("kind") and has("apiVersion")' "$f" 2>/dev/null | grep -q '^true$'
+      yq eval '.apiVersion == "config.openshift.io/v1" and .kind == "ImageDigestMirrorSet"' "$f" 2>/dev/null | grep -q '^true$'
     else
-      grep -qE '^(apiVersion|kind):' "$f"
+      grep -qx 'apiVersion: config.openshift.io/v1' "$f" &&
+        grep -qx 'kind: ImageDigestMirrorSet' "$f"
     fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
validate_yaml() {
local f="$1"
[[ -s "$f" ]] || return 1
if command -v yq &>/dev/null; then
yq eval 'has("kind") and has("apiVersion")' "$f" 2>/dev/null | grep -q '^true$'
else
grep -qE '^(apiVersion|kind):' "$f"
fi
}
if ! validate_yaml /tmp/idms_${OCP_VERSION}.yaml; then
echo "Downloaded /tmp/idms_${OCP_VERSION}.yaml is not valid YAML; falling back to bastion artifact for ${OCP_VERSION}"
scp -q ${SSH_ARGS} root@${bastion}:/root/prega_artifacts/idms_${OCP_VERSION}.yaml /tmp/idms_${OCP_VERSION}.yaml
validate_yaml /tmp/idms_${OCP_VERSION}.yaml \
|| { echo "Fallback IDMS /root/prega_artifacts/idms_${OCP_VERSION}.yaml is also invalid or missing"; exit 1; }
validate_yaml() {
local f="$1"
[[ -s "$f" ]] || return 1
if command -v yq &>/dev/null; then
yq eval '.apiVersion == "config.openshift.io/v1" and .kind == "ImageDigestMirrorSet"' "$f" 2>/dev/null | grep -q '^true$'
else
grep -qx 'apiVersion: config.openshift.io/v1' "$f" &&
grep -qx 'kind: ImageDigestMirrorSet' "$f"
fi
}
if ! validate_yaml /tmp/idms_${OCP_VERSION}.yaml; then
echo "Downloaded /tmp/idms_${OCP_VERSION}.yaml is not valid YAML; falling back to bastion artifact for ${OCP_VERSION}"
scp -q ${SSH_ARGS} root@${bastion}:/root/prega_artifacts/idms_${OCP_VERSION}.yaml /tmp/idms_${OCP_VERSION}.yaml
validate_yaml /tmp/idms_${OCP_VERSION}.yaml \
|| { echo "Fallback IDMS /root/prega_artifacts/idms_${OCP_VERSION}.yaml is also invalid or missing"; exit 1; }
🧰 Tools
🪛 Shellcheck (0.11.0)

[info] 41-41: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 43-43: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 43-43: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 43-43: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 43-43: Double quote to prevent globbing and word splitting.

(SC2086)

[info] 44-44: Double quote to prevent globbing and word splitting.

(SC2086)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift-qe/installer/bm/day2/prega-operators/openshift-qe-installer-bm-day2-prega-operators-commands.sh`
around lines 31 - 45, Update validate_yaml to require the exact
ImageDigestMirrorSet schema: apiVersion must be config.openshift.io/v1 and kind
must be ImageDigestMirrorSet, using yq when available and equivalent field
checks in the grep fallback. Ensure this validation is applied to both the
downloaded and bastion artifact files before proceeding to oc apply.

fi

echo "ImageDigestMirrorSet manifest saved to /tmp/idms_${OCP_VERSION}.yaml"
}

oc config view
Expand All @@ -39,15 +61,13 @@ if [ ${OCP_BUILD} == "dev" ]; then
jq -s '.[0] * .[1]' /tmp/existing_pull_secret.json /tmp/prega_pull_secret.json > /tmp/merged_pull_secret.json
oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=/tmp/merged_pull_secret.json
sleep 300
kubectl wait --for jsonpath='{.status.updatedMachineCount}'="$(oc get node --no-headers -l node-role.kubernetes.io/worker= | wc -l)" --timeout=60m mcp worker
oc adm wait-for-stable-cluster --minimum-stable-period=2m --timeout=20m
oc adm wait-for-stable-cluster --minimum-stable-period=2m --timeout=40m

echo "Applying the ImageDigestMirrorSet manifest"
get_idms_manifest
oc apply -f /tmp/idms.yaml
oc apply -f /tmp/idms_${OCP_VERSION}.yaml
sleep 300
kubectl wait --for jsonpath='{.status.updatedMachineCount}'="$(oc get node --no-headers -l node-role.kubernetes.io/worker= | wc -l)" --timeout=60m mcp worker
oc adm wait-for-stable-cluster --minimum-stable-period=2m --timeout=20m
oc adm wait-for-stable-cluster --minimum-stable-period=2m --timeout=40m

echo "Creating CatalogSource for PREGA Operator Index"
cat << EOF| oc apply -f -
Expand Down