Skip to content

fix: retry HyperConverged apply through HCO webhook outages - #201

Merged
andrew-jones merged 1 commit into
mainfrom
fix/hco-create-webhook-retry
Nov 11, 2026
Merged

andrew-jones merged 1 commit into
mainfrom
fix/hco-create-webhook-retry

Conversation

@rhjcd

@rhjcd rhjcd commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Problem

The Create the HyperConverged instance task in ocp4_workload_openshift_virtualization applies the HyperConverged CR right after the kubevirt-hyperconverged CSV is subscribed. While the operator is still rolling out, the HCO conversion webhook (hco-webhook-service:4343) is not yet serving, so the apply fails with:

conversion webhook for hco.kubevirt.io/v1, Kind=HyperConverged failed:
Post "https://hco-webhook-service.openshift-cnv.svc:4343/convert?timeout=30s":
connect: connection refused   (HTTP 500)

#200 (GPTEINFRA-18460) added retry/until to the subsequent read (Wait until HyperConverged is installed) but left the apply unprotected. A webhook outage during the apply still fails the deploy outright.

Fix

Add the same register + until: is succeeded + retries/delay to the apply task, reusing the existing ocp4_workload_openshift_virtualization_hco_wait_retries / _hco_wait_delay defaults (120 × 10s). No new variables.

Evidence

Reproduced on the RHACS CNV demo (rhacs-demo-cnv) on OCP 5.0.1: the cluster and operators installed cleanly (CSV kubevirt-hyperconverged-operator.v5.0.0 Succeeded, rhbk-operator Succeeded), but the HyperConverged apply landed during the ~11-min operator rollout window (hco-webhook pod up 23:40:15, CSV Succeeded 23:51:51, apply failed 23:50:14) and 500'd. On the live cluster afterward the webhook endpoint (10.235.0.16:4343) was serving normally and the HyperConverged CR was healthy — confirming a pure readiness race, not an OCP-version or catalog issue.

Refs RHDPOPS-24698, RHDPSPPT-1388. Complements #200.

The 'Create the HyperConverged instance' task applies the HyperConverged CR
immediately after the kubevirt-hyperconverged CSV is subscribed. While the
operator is still rolling out, the HCO conversion webhook
(hco-webhook-service:4343) is not yet serving, so the apply fails with:

  conversion webhook for hco.kubevirt.io/v1, Kind=HyperConverged failed:
  Post "https://hco-webhook-service.openshift-cnv.svc:4343/convert":
  connect: connection refused (HTTP 500)

#200 added retry/until to the subsequent read ('Wait until HyperConverged is
installed') but left the apply itself unprotected, so a webhook outage during
the apply still fails the deploy outright. Add the same retry/until to the
apply, reusing the existing hco_wait_retries/hco_wait_delay vars.

Observed intermittently on the RHACS CNV demo (rhacs-demo-cnv); the operator
took ~11 min to reach CSV Succeeded, and the apply landed in that window.
@andrew-jones
andrew-jones merged commit 80c04f8 into main Nov 11, 2026
1 check passed
@andrew-jones
andrew-jones deleted the fix/hco-create-webhook-retry branch November 11, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants