Skip to content

[GPTEINFRA-18460] Retry HCO readiness through transient webhook outages - #200

Merged
prakhar1985 merged 1 commit into
mainfrom
GPTEINFRA-18460-hco-readiness
Nov 6, 2026
Merged

prakhar1985 merged 1 commit into
mainfrom
GPTEINFRA-18460-hco-readiness

Conversation

@prakhar1985

Copy link
Copy Markdown
Contributor

Summary

  • Replace the Kubernetes module internal wait with bounded Ansible retries of the HCO API request.
  • Require Available=True, Progressing=False, and Degraded=False; missing resources/conditions and failed queries cannot count as ready.
  • Add configurable retry/delay defaults (120 retries, 10 seconds), documentation, and offline regression coverage.

Why

During a MachineConfig rollout, a node reboot can temporarily remove the HCO conversion webhook. The module internal wait exits on the resulting HTTP 500 even though installation subsequently recovers. Retrying the API query handles this transient outage without suppressing persistent failures.

Validation

  • Offline production-task regression tests passed on ansible-core 2.18.20 and 2.22.0.
  • Coverage includes transient API failure recovery, absent resources/conditions, unhealthy conditions, failed responses with stale resources, and retry-budget exhaustion.
  • Targeted YAML lint, git diff whitespace checks, and secret-pattern scan passed.
  • Not yet validated on a fresh live-cluster installation; full collection lint was not run.

Scope

No changes to MachineConfigs, catalog versions, or the live cluster. This role does not manage MachineConfigs; callers remain responsible for sequencing external rollouts.

Jira: https://redhat.atlassian.net/browse/GPTEINFRA-18460

@prakhar1985
prakhar1985 merged commit 8966fa8 into main Nov 6, 2026
1 check passed
@prakhar1985
prakhar1985 deleted the GPTEINFRA-18460-hco-readiness branch November 6, 2026 03:32
andrew-jones pushed a commit that referenced this pull request Nov 11, 2026
The 'Create the HyperConverged instance' task applies the HyperConverged CR
immediately after the kubevirt-hyperconverged CSV is subscribed. While the
operator is still rolling out, the HCO conversion webhook
(hco-webhook-service:4343) is not yet serving, so the apply fails with:

  conversion webhook for hco.kubevirt.io/v1, Kind=HyperConverged failed:
  Post "https://hco-webhook-service.openshift-cnv.svc:4343/convert":
  connect: connection refused (HTTP 500)

#200 added retry/until to the subsequent read ('Wait until HyperConverged is
installed') but left the apply itself unprotected, so a webhook outage during
the apply still fails the deploy outright. Add the same retry/until to the
apply, reusing the existing hco_wait_retries/hco_wait_delay vars.

Observed intermittently on the RHACS CNV demo (rhacs-demo-cnv); the operator
took ~11 min to reach CSV Succeeded, and the apply landed in that window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants