OCPEDGE-2710: add TNF alert runbooks

[lucaconsalvi]

Summary

• Add 12 operational runbooks for Two Nodes with Fencing (TNF) alerts in alerts/cluster-etcd-operator/
• Each runbook follows the existing CEO format: Meaning > Impact > Diagnosis > Mitigation
• Covers cluster-level (2), node-level (6), and resource-level (4) Pacemaker alerts

Alerts covered

Alert	Severity	Category
TNFNodeCountMismatch	critical	cluster
TNFClusterInMaintenance	warning	cluster
TNFNodeOffline	critical	node
TNFNodeFencingUnavailable	critical	node
TNFNodeFencingDegraded	warning	node
TNFNodeUnclean	critical	node
TNFNodeInMaintenance	warning	node
TNFNodeStandby	warning	node
TNFResourceStopped	critical	resource
TNFResourceFailed	critical	resource
TNFResourceUnmanaged	warning	resource
TNFResourceDisabled	warning	resource

Context

These runbooks correspond to the TNF PrometheusRule alerts added in cluster-etcd-operator#1650. The runbook_url annotations in that PR already point to these file paths.

Validation

All 12 runbooks validated on live TNF clusters (OCP 5.0 nightly, Pacemaker 3.0.1):

• Diagnosis and mitigation commands tested end-to-end
• API blind spot caveats documented for etcd-related alerts
• Resource disable cascade behavior documented

JIRA

OCPEDGE-2710

[openshift-ci-robot]

@lucaconsalvi: This pull request references OCPEDGE-2710 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• Add 12 operational runbooks for Two Nodes with Fencing (TNF) alerts in alerts/cluster-etcd-operator/
• Each runbook follows the existing CEO format: Meaning > Impact > Diagnosis > Mitigation
• Covers cluster-level (2), node-level (6), and resource-level (4) Pacemaker alerts

Alerts covered

Alert	Severity	Category
TNFNodeCountMismatch	critical	cluster
TNFClusterInMaintenance	warning	cluster
TNFNodeOffline	critical	node
TNFNodeFencingUnavailable	critical	node
TNFNodeFencingDegraded	warning	node
TNFNodeUnclean	critical	node
TNFNodeInMaintenance	warning	node
TNFNodeStandby	warning	node
TNFResourceStopped	critical	resource
TNFResourceFailed	critical	resource
TNFResourceUnmanaged	warning	resource
TNFResourceDisabled	warning	resource

Context

These runbooks correspond to the TNF PrometheusRule alerts added in cluster-etcd-operator#1650. The runbook_url annotations in that PR already point to these file paths.

Validation

All 12 runbooks validated on live TNF clusters (OCP 5.0 nightly, Pacemaker 3.0.1):

• Diagnosis and mitigation commands tested end-to-end
• API blind spot caveats documented for etcd-related alerts
• Resource disable cascade behavior documented

JIRA

OCPEDGE-2710

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lucaconsalvi
Once this PR has been reviewed and has the lgtm label, please assign hasbro17 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• alerts/cluster-etcd-operator/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@lucaconsalvi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[lucaconsalvi]

Validation

All 12 runbooks were validated on live TNF clusters (OCP 4.22 and 5.0 nightlies, RHCOS 9.x and 10.2, Pacemaker 2.1.9 and 3.0.1).

Scenarios tested

Scenario	Alert(s) triggered	Result
pcs node maintenance master-0	TNFNodeInMaintenance, TNFNodeStandby	PASS
pcs resource disable kubelet-clone	TNFResourceDisabled, TNFResourceStopped	PASS (cascade: disabling kubelet-clone also stops etcd-clone)
pcs resource unmanage kubelet-clone	TNFResourceUnmanaged	PASS
pcs cluster maintenance	TNFClusterInMaintenance	PASS
virsh destroy master-0 (node power-off)	TNFNodeOffline, TNFNodeFencingUnavailable	CONFIRMED (API blind spot — alerts can't fire through normal pipeline when quorum is lost)
pcs node standby master-0	TNFNodeStandby	PASS
pcs resource fail kubelet-clone master-0	TNFResourceFailed	PASS
TNFNodeInMaintenance PromQL fix	TNFNodeInMaintenance	PASS (bug found and fixed — OCPBUGS-100294 / CEO PR #1664, merged and verified on 5.0 nightly)

Key findings incorporated into runbooks

1. Resource ordering cascade — disabling any Pacemaker clone resource (etcd-clone or kubelet-clone) is destructive in 2-node. Resource ordering constraints cause cascading stops. Added warning to TNFResourceDisabled and TNFResourceStopped runbooks.
2. API blind spot — losing etcd quorum in a 2-node cluster means the API goes down, so node-offline alerts can't fire through the normal Prometheus pipeline. Documented in TNFNodeOffline runbook.
3. PromQL bug — TNFNodeInMaintenance used and instead of and on(), preventing it from ever firing due to label set mismatch. Fix merged upstream.