Skip to content

CI: retry digest-pinned image pulls in the codec deadline cells - #935

Merged
rmcdaniel merged 1 commit into
mainfrom
rust-325-cell-pull-retry
Nov 2, 2026
Merged

rmcdaniel merged 1 commit into
mainfrom
rust-325-cell-pull-retry

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

mirror.gcr.io has twice failed to serve one blob for a few minutes, with docker: failed to copy: httpReadSeeker: failed open: could not fetch content descriptor …:

Each time, a deadline-health cell's plain docker run failed, and the merge queue dropped #926. The image pulled fine a few minutes later.

Change

codec-operations-cell.sh pulls its runtime, observer and database images before it starts anything, with up to four attempts at 15, 30 and 45 s apart, and a warning annotation for each failed attempt. All three images are pinned by digest, so a repeat pull fetches the same content. A pull that fails four times still fails the cell, as before.

Checks

  • node --test tests/Unit/ServerParityCodecDeadlineCellsTest.mjs: 9 pass. The test lifts the cell's cleanup() only, which is unchanged.
  • bash -n on the script.
  • The function, extracted and run against a fake docker, succeeds on the third attempt after two failures and refuses after four.

🤖 Generated with Claude Code

mirror.gcr.io twice failed to serve one blob for a few minutes ("could not
fetch content descriptor"): a MariaDB 10.11 layer at 00:42-00:47 UTC and the
MySQL 8.4 attestation at 03:01 UTC. Each time a deadline-health cell's docker
run failed and knocked #926 out of the merge queue. Each cell now pulls its
runtime, observer and database images up to four times (15, 30 and 45 s
apart) before it starts. All three are pinned by digest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security bounded review of exact head e3be66eabd6dc1c4ff9168f163a53f4fbd2a9d76: clear; no Security hold. The helper retries exactly four times, delays only between attempts (15/30/45 s), and returns failure after the fourth so set -e still fails the cell closed. All three possible inputs are digest-pinned at this head (runtime, observer, and every non-SQLite database image); the repeated pull cannot float tags. Pulls happen before the network or containers are created, so an exhausted retry needs no earlier cleanup trap. The annotation contains only the public image reference. I independently checked script syntax and the committed pin formats; the owner-provided fake-Docker success-on-third/fail-on-fourth controls are proportionate. This CI-only hardening does not alter workload retries, deadlines, product behavior, or rc.3 authority.

@rmcdaniel
rmcdaniel enabled auto-merge November 2, 2026 03:18
@rmcdaniel

Copy link
Copy Markdown
Member Author

General bounded source review of exact e3be66eabd6dc1c4ff9168f163a53f4fbd2a9d76: clear. The sole 19-line script delta pre-pulls the unchanged digest-pinned runtime, observer and selected database images before creating any network/container. Exact manifest references are pinned at this head. Each failure is visible, waits are 15/30/45 seconds, success returns immediately, and the fourth failure returns nonzero under existing set -e; there is no tag/registry fallback or swallowed terminal failure. SQLite skips the database pull. Existing cleanup, cell resources, conformance assertions and job-level limits are unchanged.

This is source review only. I did not execute native fixtures, pull images, run a benchmark or resume stopped observation qualification. The reported fake-Docker success/exhaustion controls and 9 Node tests are owner-reported. No rc.3 gate/order change requested.

@sonarqubecloud

sonarqubecloud Bot commented Nov 2, 2026

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit 1db78c2 Nov 2, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant