Add CI job for checking E2E coverage on ZTWIM operator - #78969
Conversation
WalkthroughAdds a coverage-specific operator image and two AWS E2E coverage test flows to the zero-trust-workload-identity-manager CI config: an optional presubmit (e2e-coverage) and a postsubmit (publish-e2e-coverage). Both install the operator, run coverage setup using the coverage image, execute e2e tests, and attempt best-effort Codecov collection with a mounted token. ChangesCoverage Test Flows
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
the
approved
|
@rausingh-rh, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml (1)
96-108:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winSame presubmit Codecov context concern applies here.
The
collect-coveragepoststep in thee2e-coveragepresubmit onmainfaces the identical issue raised fore2e-coverage-ci.yaml: Prow does not set GitHub Actions–compatible CI environment variables, sohack/e2e-coverage.sh collectmust explicitly supply--pr $PULL_NUMBER/--branchcontext to Codecov CLI to avoid mis-attributing the PR upload as a direct main-branch commit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml` around lines 96 - 108, The presubmit `collect-coverage` post step runs `hack/e2e-coverage.sh collect` without supplying explicit Codecov context, so change the `commands` invocation to pass the PR/branch context to the Codecov CLI (e.g., add flags like `--pr $PULL_NUMBER` and `--branch $PULL_BASE_REF` or appropriate env vars) from the `collect-coverage` block so uploads are not attributed as direct main commits; update the `commands` line invoking `hack/e2e-coverage.sh collect` to forward those arguments (using the Prow env vars available) so Codecov receives the correct PR/branch info.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-e2e-coverage-ci.yaml`:
- Around line 141-196: The postsubmit job "publish-e2e-coverage" (the block with
postsubmit: true and the post step that runs commands: hack/e2e-coverage.sh
collect / as: collect-coverage) should be removed from this e2e-coverage-ci
config so coverage uploads from the e2e-coverage-ci branch do not pollute
Codecov's main baseline; locate the job by the identifiers postsubmit: true, as:
publish-e2e-coverage, and the collect-coverage step (commands:
hack/e2e-coverage.sh collect) and delete that entire job block (or set
postsubmit to false) leaving the remaining test/setup/test steps intact.
---
Duplicate comments:
In
`@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml`:
- Around line 96-108: The presubmit `collect-coverage` post step runs
`hack/e2e-coverage.sh collect` without supplying explicit Codecov context, so
change the `commands` invocation to pass the PR/branch context to the Codecov
CLI (e.g., add flags like `--pr $PULL_NUMBER` and `--branch $PULL_BASE_REF` or
appropriate env vars) from the `collect-coverage` block so uploads are not
attributed as direct main commits; update the `commands` line invoking
`hack/e2e-coverage.sh collect` to forward those arguments (using the Prow env
vars available) so Codecov receives the correct PR/branch info.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: d3152586-64e3-42a3-a0fd-69113ea99754
📒 Files selected for processing (2)
ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-e2e-coverage-ci.yamlci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml
|
/pj-rehearse e2e-coverage |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@rausingh-rh: job(s): e2e-coverage either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
f475087 to
93ef1d3
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml`:
- Around line 91-100: The presubmit currently makes the coverage setup ("as:
setup-coverage", "commands: hack/e2e-coverage.sh setup", "name:
zero-trust-workload-identity-manager-coverage", "cli: latest", "from: src") a
required step in the e2e-operator signal; remove or move this job out of the
required e2e-operator presubmit path so coverage setup is optional — either
delete the setup-coverage entry from the e2e-operator signal or place it in a
separate optional presubmit job (keeping the same commands and dependencies) so
failures in hack/e2e-coverage.sh or the coverage image don’t block operator E2E
tests.
- Around line 120-132: The postsubmit uses a step labeled "as: collect-coverage"
inside the "publish-e2e-coverage" job but it is marked best_effort: true so
coverage baseline publication can silently fail; change the step definition for
collect-coverage (the block with as: collect-coverage, commands:
hack/e2e-coverage.sh collect, credentials: ztwim-codecov-token) to remove or set
best_effort to false so the job fails when baseline publication fails, ensuring
the postsubmit fails on publication errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 6a9a4f07-d9d9-4bbc-bf76-4fe847fcd8a1
⛔ Files ignored due to path filters (2)
ci-operator/jobs/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main-postsubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml
| - as: setup-coverage | ||
| cli: latest | ||
| commands: hack/e2e-coverage.sh setup | ||
| dependencies: | ||
| - env: COVERAGE_IMAGE | ||
| name: zero-trust-workload-identity-manager-coverage | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m |
There was a problem hiding this comment.
Keep coverage setup out of the required e2e-operator signal.
setup-coverage is now a mandatory step inside the existing presubmit. If hack/e2e-coverage.sh setup or the coverage image regresses, the functional E2E lane starts failing for coverage/tooling reasons rather than operator behavior. Given the stated goal here was an optional coverage presubmit, this should stay decoupled from the required e2e-operator path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml`
around lines 91 - 100, The presubmit currently makes the coverage setup ("as:
setup-coverage", "commands: hack/e2e-coverage.sh setup", "name:
zero-trust-workload-identity-manager-coverage", "cli: latest", "from: src") a
required step in the e2e-operator signal; remove or move this job out of the
required e2e-operator presubmit path so coverage setup is optional — either
delete the setup-coverage entry from the e2e-operator signal or place it in a
separate optional presubmit job (keeping the same commands and dependencies) so
failures in hack/e2e-coverage.sh or the coverage image don’t block operator E2E
tests.
| - as: collect-coverage | ||
| best_effort: true | ||
| cli: latest | ||
| commands: hack/e2e-coverage.sh collect | ||
| credentials: | ||
| - mount_path: /var/run/secrets/codecov | ||
| name: ztwim-codecov-token | ||
| namespace: test-credentials | ||
| from: src | ||
| resources: | ||
| requests: | ||
| cpu: 100m | ||
| timeout: 15m0s |
There was a problem hiding this comment.
Fail the postsubmit when baseline publication breaks.
In publish-e2e-coverage, collect-coverage is the step that actually publishes the baseline. Leaving it best_effort: true means the job can stay green while coverage publication is silently broken, which defeats the purpose of this postsubmit.
Suggested change
post:
- as: collect-coverage
- best_effort: true
cli: latest
commands: hack/e2e-coverage.sh collect🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/zero-trust-workload-identity-manager/openshift-zero-trust-workload-identity-manager-main.yaml`
around lines 120 - 132, The postsubmit uses a step labeled "as:
collect-coverage" inside the "publish-e2e-coverage" job but it is marked
best_effort: true so coverage baseline publication can silently fail; change the
step definition for collect-coverage (the block with as: collect-coverage,
commands: hack/e2e-coverage.sh collect, credentials: ztwim-codecov-token) to
remove or set best_effort to false so the job fails when baseline publication
fails, ensuring the postsubmit fails on publication errors.
534ed24 to
29f10cb
Compare
openshift-ci
Bot
removed
the
approved
|
/pj-rehearse max |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/retest operator-e2e-coverage |
|
@rausingh-rh: The The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/pj-rehearse operator-e2e-coverage |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@rausingh-rh: job(s): operator-e2e-coverage either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse pull-ci-openshift-zero-trust-workload-identity-manager-main-operator-e2e-coverage |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-zero-trust-workload-identity-manager-main-ci-bundle-zero-trust-workload-identity-manager-bundle |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-ci
Bot
added
the
needs-rebase
- Build coverage-instrumented image from Dockerfile.coverage - Add e2e-coverage presubmit (optional, always_run) - Add publish-e2e-coverage postsubmit for baseline on main - Mount ztwim-codecov-token secret for Codecov upload
1ff1432 to
00196ae
Compare
|
/pj-rehearse max |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse max |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-zero-trust-workload-identity-manager-main-operator-e2e-coverage |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse max |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse max |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse ack |
|
@rausingh-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bharath-b-rh, danilo-gemoli, rausingh-rh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@rausingh-rh: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
ac9b2ff
into
openshift:main
* Add e2e-coverage optional job for zero-trust-workload-identity-manager - Build coverage-instrumented image from Dockerfile.coverage - Add e2e-coverage presubmit (optional, always_run) - Add publish-e2e-coverage postsubmit for baseline on main - Mount ztwim-codecov-token secret for Codecov upload * Regenerate Prow job config * Merge coverage into e2e-operator job, remove separate e2e-coverage job * Regenerate Prow job config for ztwim * + run e2e coverage as separate job
3 participants
Add E2E coverage reporting for zero-trust-workload-identity-manager
Adds Codecov E2E coverage integration for the ZTWIM operator
Changes
zero-trust-workload-identity-manager-coveragefromDockerfile.coverageoperator-e2e-coveragepresubmit (always_run: false): Standalone coverage job that claims an AWS cluster, installs the operator, runshack/e2e-coverage.sh setup, executesmake test-e2e, and collects coverage viahack/e2e-coverage.sh collect(best-effort, 15m timeout)publish-e2e-coveragepostsubmit: Mirrors the presubmit to publish baseline coverage onmainzero-trust-workload-identity-manager(main branch) in the pipeline controller sooperator-e2e-coveragetriggers automatically afterlgtmztwim-codecov-tokenmounted at/var/run/secrets/codecov