fix(pipelines): add retry wrappers to network calls in image mirror script (AROSLSRE-1490) - #269
Open
trevorwilliams2025 wants to merge 2 commits into
Open
Conversation
…ror script (AROSLSRE-1490) The on-demand sync script (on-demand.sh) had several network-touching az/oras/oc calls without retry logic. A transient IMDS/ARM connection reset (BadStatusLine HTTP/1.1 000) on az keyvault secret download caused the mirror-oc-mirror-image Ev2 step to fail in uksouth (stg), even though the retry helper and pattern already existed in the script. Wrap all bare network calls with the existing retry helper: - az keyvault secret download (direct cause of the outage) - az cloud show (both registry and OCI layout paths, deduplicated) - oc registry login - oras login (both paths) - oras cp (both paths) Also fixes unquoted variables in the OCI-layout oras login call.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves the resiliency of the pipelines/types/on-demand.sh image mirroring script by applying retry behavior to previously bare network-touching az/oc/oras calls, reducing transient IMDS/ARM-related rollout failures.
Changes:
- Adds a shared
get_acr_domain_suffix()helper and uses it from both registry and OCI-layout flows. - Wraps additional network-dependent commands (
az keyvault secret download,oc registry login,oras login,oras cp) with the existingretry 5helper. - Fixes quoting in the OCI-layout
oras logininvocation and cleans up minor formatting.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
trevorwilliams2025
force-pushed
the
fix/image-mirror-retry-wrappers
branch
from
July 15, 2026 02:51
eaef36f to
4e58828
Compare
trevorwilliams2025
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
az/oras/occalls inon-demand.shwith the existingretry 5helper (exponential backoff)az cloud showinto a shared top-levelget_acr_domain_suffix()function used by both code pathsoras logincallContext
Ev2 step
mirror-oc-mirror-imagefailed inuksouth(public/stg) during rolloutd3208ea2-7650-4209-8e14-db081fba362d. The root cause was a transient IMDS/ARM connection reset (BadStatusLine HTTP/1.1 000) hittingaz keyvault secret download— the only network call in that block without a retry wrapper. Theaz login --identitycall hit the same error moments earlier but recovered becausecommon.shwraps it with
az_retry 15.This is the same transient IMDS failure class addressed by:
common.shaddedaz_retryaroundaz login --identityforBadStatusLine/IMDS resetsprow-job-executoradded retry with exponential backoff around Key Vault prow-token lookup for IMDS connection resets/EOFs on Ev2 runnerson-demand.shaddedretry 5aroundaz acr loginfor the same error classThis PR closes the gap by applying the same pattern to the remaining bare network calls in
on-demand.sh.Calls now wrapped with
retry 5:az keyvault secret downloadaz cloud showoc registry loginoras loginoras cpJira: AROSLSRE-1490
Test plan
bash -n on-demand.sh— syntax validatesgo test ./pipelines/types/...— existing tests passDRY_RUN=true(exercises all auth/login paths)