Skip to content

MON-4471: test/extended/prometheus: move alert tests to "Test Framework" - #30703

Open
simonpasquier wants to merge 1 commit into
openshift:mainfrom
simonpasquier:rename-alert-tests
Open

MON-4471: test/extended/prometheus: move alert tests to "Test Framework"#30703
simonpasquier wants to merge 1 commit into
openshift:mainfrom
simonpasquier:rename-alert-tests

Conversation

@simonpasquier

@simonpasquier simonpasquier commented Jan 20, 2026

Copy link
Copy Markdown
Contributor

This commit renames 2 tests related to alerting rules:

  • 1 test checking that no rule fails to evaluate.
  • 1 test checking that there's no firing alert other than approved exceptions.

When these tests fail, the associated Jira used to be assigned to the Monitoring component because of the "sig-instrumentation" prefix. But in practice, it can be any component and monitoring is (often) only the messenger. The tests are renamed with the [Jira:"Test Framework"] prefix instead which redirects the tickets to TRT.

Summary by CodeRabbit

  • Tests
    • Expanded Prometheus monitoring coverage for rule-evaluation failures and unexpected firing alerts.
    • Improved validation of standard and platform-specific alert exceptions, including managed-service, no-upgrade, and OperatorHub scenarios.
    • Removed duplicate alert validation coverage to make test results clearer and more reliable.

@openshift-ci-robot

Copy link
Copy Markdown

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci
openshift-ci Bot requested review from jan--f and machine424 January 20, 2026 16:42
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 20, 2026
@openshift-ci-robot

Copy link
Copy Markdown

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-trt

openshift-trt Bot commented Jan 20, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 616e9bd

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade Medium - "[Jira:"Test Framework"] Prometheus [apigroup:coreos.monitoring.com] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" is a new test, and was only seen in one job.

New tests seen in this PR at sha: 616e9bd

  • "[Jira:"Test Framework"] Prometheus [apigroup:coreos.monitoring.com] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" [Total: 1, Pass: 1, Fail: 0, Flake: 0]

@simonpasquier simonpasquier changed the title test/extended/prometheus: move alert tests to "Test Framework" MON-4471: test/extended/prometheus: move alert tests to "Test Framework" Jan 21, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 21, 2026
@openshift-ci-robot

openshift-ci-robot commented Jan 21, 2026

Copy link
Copy Markdown

@simonpasquier: This pull request references MON-4471 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This commit renames 2 tests related to alerting rules:

  • 1 test checking that no rule fails to evaluate.
  • 1 test checking that there's no firing alert other than approved exceptions.

When these tests fail, the associated Jira used to be assigned to the Monitoring component because of the "sig-instrumentation" prefix. But in practice, it can be any component and monitoring is (often) only the messenger. The tests are renamed with the [Jira:"Test Framework"] prefix instead which redirects the tickets to TRT.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@dgoodwin

Copy link
Copy Markdown
Contributor

Sorry had this open days ago and lost track of it.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jan 23, 2026
@simonpasquier

Copy link
Copy Markdown
Contributor Author

@dgoodwin I'm confused because the renamed test doesn't appear in the e2e output logs. Is it because we need to merge openshift-eng/ci-test-mapping#579?

@dgoodwin

dgoodwin commented Jan 26, 2026

Copy link
Copy Markdown
Contributor

It looks like only one job ran it #30703 (comment)

Probably a good reason why that is but may need investigation, #forum-ocp-release-oversight may be able to help

@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 26, 2026
@simonpasquier

Copy link
Copy Markdown
Contributor Author

/test images

@openshift-ci-robot

Copy link
Copy Markdown

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-trt

openshift-trt Bot commented Jan 27, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 814e9f7

  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel]" [Total: 5, Pass: 5, Fail: 0, Flake: 0]
  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" [Total: 8, Pass: 8, Fail: 0, Flake: 0]

@openshift-ci-robot

Copy link
Copy Markdown

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-trt

openshift-trt Bot commented Feb 17, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 854b5da

  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel]" [Total: 5, Pass: 5, Fail: 0, Flake: 0]
  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" [Total: 8, Pass: 8, Fail: 0, Flake: 0]

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 17, 2026
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 820b2d69-7909-4aa6-bb0d-9a8429354669

📥 Commits

Reviewing files that changed from the base of the PR and between 99099a9 and d26784b.

📒 Files selected for processing (1)
  • test/extended/prometheus/prometheus.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/extended/prometheus/prometheus.go

Walkthrough

The Prometheus test suite adds config API group checks for rule-evaluation failures and unexpected firing alerts. It applies standard and platform-specific alert exclusions and removes duplicate checks.

Changes

Prometheus alert validation

Layer / File(s) Summary
Alert validation tests
test/extended/prometheus/prometheus.go
Adds config API group checks for rule-evaluation failures and unexpected non-info firing alerts. The checks use standard, no-upgrade, managed-service, and OperatorHub-specific exclusions. Duplicate checks are removed from the existing suite.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jan--f, machine424

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Test Structure And Quality ❓ Inconclusive Investigation is still in progress. Need inspect helper wait behavior and changed assertions before deciding.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes moving alert tests to the Test Framework, which matches the pull request objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All changed Ginkgo titles are literal, static strings; the renamed alert title removes specific alert names and adds only stable tags. No dynamic values appear in titles.
Microshift Test Compatibility ✅ Passed Both new tests are wrapped in Describe tagged [apigroup:config.openshift.io], so MicroShift CI skips them automatically.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The two tests only query Prometheus rule and alert metrics, use standard APIs, and contain no node-count, scheduling, failover, drain, or multi-endpoint assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed The only changed path is a Prometheus test file; the diff moves alert tests and changes alert-name collection, with no deployment, controller, manifest, or scheduling constraint changes.
Ote Binary Stdout Contract ✅ Passed The added suite contains only BeforeEach and It callbacks and no stdout writes; the existing fmt.Printf is unchanged and called only from an It callback.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The two moved Ginkgo tests use Kubernetes APIs and the cluster's Prometheus/Thanos route only; no IPv4 literals, IP parsing, IPv4 CIDRs, or public-network calls were added.
No-Weak-Crypto ✅ Passed The PR only moves Prometheus alert tests; added code has no MD5/SHA1/DES/RC4/Blowfish/ECB, crypto APIs, custom crypto, or secret/token comparisons.
Container-Privileges ✅ Passed The PR changes only one Go test file. The added test uses Pod Security admission LevelBaseline and adds no privileged, host namespace, SYS_ADMIN, root, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed The added code has no logging or credential output; it reuses unchanged RunQueries, and its PromQL contains only fixed alert names, namespaces, and test duration.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci-robot

openshift-ci-robot commented Aug 6, 2026

Copy link
Copy Markdown

@simonpasquier: This pull request references MON-4471 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

This commit renames 2 tests related to alerting rules:

  • 1 test checking that no rule fails to evaluate.
  • 1 test checking that there's no firing alert other than approved exceptions.

When these tests fail, the associated Jira used to be assigned to the Monitoring component because of the "sig-instrumentation" prefix. But in practice, it can be any component and monitoring is (often) only the messenger. The tests are renamed with the [Jira:"Test Framework"] prefix instead which redirects the tickets to TRT.

Summary by CodeRabbit

  • Tests
  • Added Prometheus coverage for rule-evaluation failures and unexpected firing alerts.
  • Added handling for managed-service, no-upgrade, and OperatorHub-specific alert exceptions.
  • Removed duplicate alert validation tests from the existing Prometheus suite.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/prometheus/prometheus.go`:
- Line 624: Replace the context.TODO() instances in both Ginkgo g.It specs with
the provided g.SpecContext, updating each spec signature to accept it. Pass that
context through NewPrometheusClient, RunQueries, and IsManagedServiceCluster so
Prometheus operations honor spec cancellation.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 518bf721-8f75-46ff-8916-0648d53b7183

📥 Commits

Reviewing files that changed from the base of the PR and between 6b7ae27 and 28976f9.

📒 Files selected for processing (1)
  • test/extended/prometheus/prometheus.go

Comment thread test/extended/prometheus/prometheus.go Outdated
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-trt

openshift-trt Bot commented Aug 6, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 99099a9

  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel]" [Total: 3, Pass: 3, Fail: 0, Flake: 0]
  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" [Total: 5, Pass: 5, Fail: 0, Flake: 0]

@simonpasquier

Copy link
Copy Markdown
Contributor Author

/hold

needs openshift-eng/ci-test-mapping#624

@simonpasquier

Copy link
Copy Markdown
Contributor Author

/verified by CI

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 7, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@simonpasquier: This PR has been marked as verified by CI.

Details

In response to this:

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Aug 7, 2026
This commit renames 2 tests related to alerting rules:
* 1 test checking that no rule fails to evaluate.
* 1 test checking that there's no firing alert other than approved
  exceptions.

When these tests fail, the associated Jira used to be assigned to the
Monitoring component because of the "sig-instrumentation" prefix. But in
practice, it can be any component and monitoring is (often) only the
messenger. The tests are renamed with the [Jira:"Test Framework"] prefix
instead which redirects the tickets to TRT.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Aug 10, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 10, 2026

Copy link
Copy Markdown

@simonpasquier: This pull request references MON-4471 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

This commit renames 2 tests related to alerting rules:

  • 1 test checking that no rule fails to evaluate.
  • 1 test checking that there's no firing alert other than approved exceptions.

When these tests fail, the associated Jira used to be assigned to the Monitoring component because of the "sig-instrumentation" prefix. But in practice, it can be any component and monitoring is (often) only the messenger. The tests are renamed with the [Jira:"Test Framework"] prefix instead which redirects the tickets to TRT.

Summary by CodeRabbit

  • Tests
  • Expanded Prometheus monitoring coverage for rule-evaluation failures and unexpected firing alerts.
  • Improved validation of standard and platform-specific alert exceptions, including managed-service, no-upgrade, and OperatorHub scenarios.
  • Removed duplicate alert validation coverage to make test results clearer and more reliable.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Aug 10, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@machine424

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2026
@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, machine424, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@simonpasquier: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-microshift-serial d26784b link true /test e2e-aws-ovn-microshift-serial

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt

openshift-trt Bot commented Aug 10, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: d26784b

  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster should have no unexpected alerts in firing state [Early][apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]" [Total: 6, Pass: 6, Fail: 0, Flake: 0]
  • "[Jira:"Test Framework"] Prometheus [apigroup:config.openshift.io] when installed on the cluster shouldn't have failing rules evaluation [Suite:openshift/conformance/parallel]" [Total: 3, Pass: 3, Fail: 0, Flake: 0]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants