Investigate making plugin repo changes immediately available for OADP-operator e2e tests

[redhat-chai-bot]

Problem

Currently, when code changes land in OADP plugin repositories (e.g. openshift/openshift-velero-plugin), the updated images are not immediately available for end-to-end tests in the openshift/oadp-operator repository. Instead, the flow requires:

1. Plugin PR merges (e.g. to oadp-1.6 branch)
2. CI builds and promotes the image
3. The image mirroring job (configured via core-services/image-mirroring/konveyor/mapping_konveyor_latest) mirrors the promoted image to quay.io
4. Only then can OADP-operator e2e tests that depend on the updated plugin image pass

This delay blocks PRs in the OADP-operator repo that add or modify e2e tests exercising new plugin functionality. For example, oadp-operator#2328 is currently blocked waiting on openshift-velero-plugin#451 to merge and the resulting image to be mirrored.

Investigation Goal

Explore options for making plugin repo changes immediately available for OADP-operator e2e tests without waiting for the full mirror pipeline. Possible approaches to investigate:

• Using CI image overrides or tag_specification to pull the plugin image directly from the CI registry
• Configuring images stanzas in ci-operator config to build the plugin from source as part of the OADP-operator PR test
• Using image_stream_tag references that resolve to the latest promoted (but not yet mirrored) image
• Any other mechanism to short-circuit the mirror dependency for pre-merge testing

Acceptance Criteria

• Document the viable options with tradeoffs
• Implement the preferred approach (or explain why the current flow is necessary)
• Ensure OADP-operator e2e tests can validate plugin changes without waiting for quay.io mirroring

[redhat-chai-bot]

Investigation Findings: Plugin Image Availability for OADP-Operator E2E Tests

Current Pipeline Analysis

After examining the CI configuration across openshift/release and openshift/oadp-operator, here is the root cause and the full image flow:

1. Plugin CI build & promotion: Each plugin repo (e.g. openshift/openshift-velero-plugin on oadp-1.6) has a ci-operator config that builds the image (e.g. openshift-velero-plugin-oadp-1.6) and promotes it to the konveyor namespace ImageStream in the CI registry (registry.ci.openshift.org/konveyor/openshift-velero-plugin:openshift-velero-plugin-oadp-1.6).

2. Mirror job delay: The image-mirroring job (configured in core-services/image-mirroring/konveyor/mapping_konveyor_latest) copies from the CI public mirror (quay.io/openshift/ci:konveyor_*) to quay.io/konveyor/*. This runs on a periodic cadence, introducing a delay.

3. OADP operator hardcoded defaults: The operator deployment (config/manager/manager.yaml) has all RELATED_IMAGE_* env vars hardcoded to quay.io/konveyor/*:latest — the mirrored destinations.

4. Missing CI dependency wiring: The oadp-operator's ci-operator config (openshift-oadp-operator-oadp-1.6.yaml) has no dependencies stanza and no image overrides for the plugin repos. This means e2e tests always consume whatever is already on quay.io, not what was just promoted.

Key insight: Plugin images are available in the CI registry immediately after the plugin's postsubmit promotion completes — the bottleneck is entirely that the oadp-operator CI jobs don't know to pull from the CI ImageStream instead of quay.io.

---

Potential Fix Approaches (Ranked by Implementation Difficulty)

Approach 1: dependencies stanza in ci-operator config ⭐ (Easiest)

Add a dependencies stanza to the oadp-operator ci-operator config. ci-operator resolves ImageStreamTag references at test time and injects them as environment variables into test pods, automatically overriding the RELATED_IMAGE_* defaults.

# In openshift-oadp-operator-oadp-1.6.yaml
dependencies:
  RELATED_IMAGE_OPENSHIFT_VELERO_PLUGIN:
    name: openshift-velero-plugin
    namespace: konveyor
    tag: openshift-velero-plugin-oadp-1.6
  RELATED_IMAGE_VELERO_PLUGIN_FOR_AWS:
    name: velero-plugin-for-aws
    namespace: konveyor
    tag: velero-plugin-for-aws-oadp-1.6
  RELATED_IMAGE_VELERO_PLUGIN_FOR_GCP:
    name: velero-plugin-for-gcp
    namespace: konveyor
    tag: velero-plugin-for-gcp-oadp-1.6
  RELATED_IMAGE_VELERO_PLUGIN_FOR_MICROSOFT_AZURE:
    name: velero-plugin-for-microsoft-azure
    namespace: konveyor
    tag: velero-plugin-for-microsoft-azure-oadp-1.6
  RELATED_IMAGE_VELERO_PLUGIN_FOR_LEGACY_AWS:
    name: velero-plugin-for-legacy-aws
    namespace: konveyor
    tag: velero-plugin-for-legacy-aws-oadp-1.6
  RELATED_IMAGE_VELERO:
    name: velero
    namespace: konveyor
    tag: velero-oadp-1.6
  # ... repeat for each RELATED_IMAGE_* that has a promoted CI image

Pros:

• Config-only change in openshift/release — no code changes needed
• ci-operator automatically resolves the latest promoted image from the CI registry
• If the dependency image hasn't been promoted yet, CI will warn/fail clearly
• Standard OpenShift CI pattern used by many operators

Cons:

• Must maintain the dependencies list when adding/removing plugins
• Tags must match the to: field in each plugin's ci-operator config exactly

Difficulty: ⭐ Easiest — single PR to openshift/release, purely declarative

---

Approach 2: Env overrides in multi-stage test step config ⭐⭐ (Low)

Set RELATED_IMAGE_* overrides directly in the e2e test step definitions, pointing at the CI registry pullspec:

# In the e2e test step definition
env:
  RELATED_IMAGE_OPENSHIFT_VELERO_PLUGIN: "registry.ci.openshift.org/konveyor/openshift-velero-plugin:openshift-velero-plugin-oadp-1.6"
  RELATED_IMAGE_VELERO_PLUGIN_FOR_AWS: "registry.ci.openshift.org/konveyor/velero-plugin-for-aws:velero-plugin-for-aws-oadp-1.6"
  # ...

Pros:

• Quick, explicit, easy to understand

Cons:

• Hardcoded registry paths that could break if the CI registry topology changes
• Doesn't benefit from ci-operator's dependency resolution (no automatic pull secret handling, no automatic mirroring for disconnected clusters)
• More fragile than Approach 1

Difficulty: ⭐⭐ Low — but less robust

---

Approach 3: Build plugins from source via images stanza ⭐⭐⭐ (Medium)

Add plugin repos as build inputs in the oadp-operator ci-operator config using the images stanza with source context from other repos:

base_images:
  openshift-velero-plugin-src:
    name: src
    namespace: konveyor
    tag: openshift-velero-plugin-oadp-1.6
images:
  - context_dir: .
    dockerfile_path: Dockerfile
    from: builder
    inputs:
      openshift-velero-plugin-src:
        as:
          - src
    to: openshift-velero-plugin-ci

Pros:

• Builds exactly the code from the plugin repo — tests against source of truth
• Could enable testing against unmerged plugin PRs (via multi-repo PR coordination)

Cons:

• Significantly increases build time (building N plugin images per PR test run)
• Complex config — must replicate each plugin's build setup
• Higher CI resource consumption

Difficulty: ⭐⭐⭐ Medium — complex config, longer CI feedback loops

---

Approach 4: Operator code change to prefer CI registry ⭐⭐⭐⭐ (Hard)

Modify pkg/common/common.go to detect OPENSHIFT_CI=true and swap the default image constants to CI registry paths, or add a mechanism for the test framework to inject image overrides.

Pros:

• Flexible, could support other CI environments

Cons:

• Requires code review and testing
• Mixes CI-specific logic into production code
• The RELATED_IMAGE_* env var override mechanism already exists — the problem is just that CI doesn't set them

Difficulty: ⭐⭐⭐⭐ Hard — unnecessary complexity when Approach 1 solves it declaratively

---

Recommendation

Approach 1 (dependencies stanza) is the clear winner. It is the standard OpenShift CI pattern, requires zero code changes, and leverages ci-operator's built-in dependency resolution. The operator already reads RELATED_IMAGE_* from env vars — ci-operator will inject the CI registry pullspecs automatically, overriding the quay.io defaults.

This analysis was generated from examination of:

• openshift-oadp-operator-oadp-1.6.yaml — oadp-operator ci-operator config
• openshift-openshift-velero-plugin-oadp-1.6.yaml — plugin ci-operator config
• mapping_konveyor_latest — image mirror mapping
• config/manager/manager.yaml — RELATED_IMAGE env var definitions
• pkg/common/common.go — default image constants

[redhat-chai-bot]

Update: Fix in progress

This issue is being actively addressed by the following PRs:

1. openshift/release#82762 — CI image substitutions (functional fix)

Adds base_images + operator.substitutions to the oadp-operator ci-operator configs so that all plugin images are resolved directly from CI ImageStreams at e2e test time, bypassing the quay.io/konveyor/* mirror delay entirely.

How it works:

• base_images imports each plugin image from the konveyor namespace CI ImageStream (the same IS that each plugin repo promotes to on merge)
• operator.substitutions rewrites the quay.io/konveyor/* pullspecs in the CSV with CI @sha256 digests before the OLM bundle is built
• Every e2e test that uses OO_INDEX: ci-index automatically gets the latest CI-built plugin images — no test code changes needed

Coverage: oadp-1.3 through oadp-dev, all variant configs.

2. openshift/oadp-operator#2346 — Developer documentation

Adds CI sync documentation to ensure future plugin additions maintain the substitution list:

• Go comment block in pkg/common/common.go above the // Images const

Both PRs are currently in draft status.

[kaovilai]

fixed via openshift/release#82762