OCPBUGS-23746: Add availableInertia support to operator status controller

[tchap]

The OperatorAvailable condition now respects an inertia window, similar to the existing degradedInertia behavior. This allows operators to suppress transient availability fluctuations within a configurable time window. A new WithAvailableInertia method enables operators to set custom inertia thresholds, and comprehensive tests validate the inertia behavior for various condition scenarios.

This basically replaces #2426 . I am pushing this now since the other author is on vacation and I would like to proceed.

#2057 can also be closed once this is merged IMO.

Summary by CodeRabbit

• New Features
  • Added configurable inertia for operator availability status updates.
  • Availability can now remain active briefly after a temporary unavailable condition, reducing status fluctuation.
  • Multiple unavailable conditions are aggregated when determining the final availability status.

[openshift-ci-robot]

@tchap: This pull request references Jira Issue OCPBUGS-23746, which is invalid:

• expected the bug to target either version "5.1.0." or "openshift-5.1.0.", but it targets "4.22.0" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

The OperatorAvailable condition now respects an inertia window, similar to the existing degradedInertia behavior. This allows operators to suppress transient availability fluctuations within a configurable time window. A new WithAvailableInertia method enables operators to set custom inertia thresholds, and comprehensive tests validate the inertia behavior for various condition scenarios.

This basically replaces #2426 . I am pushing this now since the other author is on vacation and I would like to proceed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: abbcd700-429d-4b0d-a28d-b24bccaf139c

📥 Commits

Reviewing files that changed from the base of the PR and between a0584a6 and 9760562.

📒 Files selected for processing (2)

• pkg/operator/status/status_controller.go
• pkg/operator/status/status_controller_test.go

Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review.

---

Walkthrough

StatusSyncer now supports configurable inertia for OperatorAvailable conditions. Tests cover propagation, suppression, expiry, and multiple unavailable conditions.

Changes

Available condition inertia

Layer / File(s)	Summary
Configure and apply available inertia pkg/operator/status/status_controller.go	StatusSyncer stores an available-condition inertia value. WithAvailableInertia configures it on a copied controller. OperatorAvailable updates use the configured inertia.
Validate inertia behavior pkg/operator/status/status_controller_test.go	Table-driven tests verify immediate propagation, suppression during the inertia window, propagation after expiry, and aggregation of multiple unavailable conditions.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 97605

The PR adds configurable inertia handling for operator availability conditions; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR adds standard Go subtests, not Ginkgo tests; all five t.Run names are static scenario strings and contain no runtime values.
Test Structure And Quality	✅ Passed	The changed test uses standard Go subtests with fake clients, not Ginkgo; it has no Eventually/Consistently waits, creates no real cluster resources, and follows existing package patterns.
Microshift Test Compatibility	✅ Passed	The pull request adds a standard Go TestAvailableInertia unit test, not a Ginkgo e2e test; the MicroShift API compatibility check is therefore inapplicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR adds TestAvailableInertia as a standard testing.T unit test, not a Ginkgo e2e test, and it contains no multi-node or HA assumptions.
Topology-Aware Scheduling Compatibility	✅ Passed	The commit changes only status-condition inertia and tests; inspection found no deployments, replicas, affinity, topology spread, node selectors, tolerations, or PDB scheduling constraints.
Ote Binary Stdout Contract	✅ Passed	The PR diff adds inertia logic and tests only; it introduces no main/init/suite setup or stdout writes. Existing klog calls are unchanged.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The added test is a local Go table test using fake clients and an in-memory clock; it adds no Ginkgo e2e test, IPv4 assumption, or external network access.
No-Weak-Crypto	✅ Passed	The diff adds available-condition inertia and tests only; added-line scanning found no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparisons.
Container-Privileges	✅ Passed	The commit changes only Go source and tests. No container or Kubernetes manifest changes add privileged, host namespace, SYS_ADMIN, or privilege-escalation settings.
No-Sensitive-Data-In-Logs	✅ Passed	The PR adds inertia configuration and tests only; it introduces no logging calls or log message changes, and existing status-diff logging remains unchanged.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the issue and the main change: adding availableInertia support to the operator status controller.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[tchap]

/jira refresh

[openshift-ci-robot]

@tchap: This pull request references Jira Issue OCPBUGS-23746, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.1.0) matches configured target version for branch (5.1.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @gangwgr

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@tchap: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bertinatto]

/assign @dusk125

[tmshort]

/lgtm

[tjungblu]

/approve

[fgiudici]

/lgtm

[dusk125]

/lgtm

[tmshort]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dusk125, fgiudici, tchap, tjungblu, tmshort
Once this PR has been reviewed and has the lgtm label, please assign jsafrane for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/operator/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bertinatto]

Since this was considered and nacked before, I think we should talk about what are the implications of this and then make an informed decision. See this comment in the PR where the inertia framework was introduced: #632 (comment). The statement, If someone thinks up a case, they'll need to argue it out in a library-go PR from @wking is what need to happen now :)

I understand this is intended to fix the flipping of the Available condition, however, I believe the lack of support for inertia here was intentional. From my understanding, the current rationale is that it's acceptable to delay reporting Degraded because nothing breaks if it's slow to flip. However, delaying moving out from a Available=false state can trigger more strong reactions, like blocking upgrades, alerts, etc. The other way around is also problematic: delaying moving out from a Available=true is lying about availability (which seems to be a bigger sin than lying about degradation :) ).

I'd like to hear @wking thoughts on this.

[bertinatto]

Also, if we decide to allow this, would it make sense define a default inertia value for Available, so that operators can't overdo it?

[wking]

There's a lot of diversity of view around how this kind of thing should be handled, and I've personally given up on trying to convince folks to consolidate around some kind of consistent policy here ;). Trying to describe the main approaches I've heard:

• Some folks feel that if an component is not available for even a short time (e.g. while a leader lease is being handed off between outgoing and incoming instances of a controller), then the controller is not available during that lease-handoff period, and the ClusterOperator should show Available=False. For these folks, Available inertia doesn't seem to be very useful.
• Some folks feel like there are expected, blippy things that happen in the life of a healthy cluster, and even though ClusterOperatorDown cooks for: 10m before sounding its critical alert, someone watching the ClusterOperator directly (e.g. in oc get clusteroperators ... or the web-console /settings/cluster/clusteroperators page) might see an Available=False claim and panic, without realizing that the issue was regular business and expected to resolve itself shortly. These folks might find Available inertia useful, but can also apply their own inertia internally before feeding into library-go helpers to provide as much buffering as they think "regular business" deserves. For example, retrying a failed Kube API request a few times before setting an initial Available* to False or Unknown. So while this pull would help folks with this approach to denoise their ClusterOperator claims vs. expected disruption, I don't see this pull being a requirement for that kind of denoising.
• Some folks feel like a highly-available cluster should have no disruption, in which case there's no need to have any inertia or set Available=False. I feel like we do a pretty solid job at zero-disruption in general, but we're certainly not perfect.

The current OpenShift origin test-policy says that the "someone watching the ClusterOperator directly might see an Available=False claim and panic" concerns have weight. If the folks setting policy decide to soften that approach, or grant exceptions that last forever and which we don't plan on ever fixing, that's up to them (e.g. openshift/origin#30775 adjusted the policy to stop caring what happened on single-node clusters). If the policies require denoising, both the inertial/retry option (whether via this pull or other tooling) and the "reduce the actual disruption" options are on the table, and I'm fine leaving it up to the impacted component's maintainers to pick the approach that seems most appropriate to them for that particular disruption. If folks overdo it on the "decide not to bubble up disruption because it might be expected/acceptable" direction, anyone who notices (e.g. a bitten cluster admin) can file a bug reporting that. And if folks underdo it and pester admins about issues that shortly afterward resolve on their own, anyone who notices can file bugs reporting that too (which is what this pull's OCPBUGS-23746 is). So there are forcing functions in both directions to keep the component maintainers honest.