Skip to content

Add ECR Public mirroring for slim images via dispatch - #286

Open
avallete wants to merge 6 commits into
mainfrom
claude/mirror-slim-images-github-nvvrhy
Open

avallete wants to merge 6 commits into
mainfrom
claude/mirror-slim-images-github-nvvrhy

Conversation

@avallete

@avallete avallete commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

Mirror slim images (and their native OCI artifacts) from GHCR to AWS ECR Public by reusing the mirror workflow and AWS credentials that already live in supabase/cli. This repository sends a repository_dispatch event after each release and verifies, with anonymous pulls, that the ECR destination serves the same digest as GHCR. No AWS access is added here.

This PR also carries #308 (merged into this branch), which moved the tooling to TypeScript and added native artifact publication. Design and the dispatch contract: docs/design/ecr-mirror-dispatch.md. CLI-side consumption is recorded in supabase/cli ADR 0026.

What changes

  • scripts/ecr-mirror.ts (bun) with payload, published-digest, destination-repo, request, verify, and sync subcommands. request dispatches mirror-slim-image to supabase/cli and polls public.ecr.aws/supabase/cli/<service>:<version> anonymously until its index digest matches. sync audits every published release and, with --request, re-dispatches out-of-sync images (this is also the backfill path).
  • scripts/publish-native-oci.ts pushes each release's native triplet (tar.zst, manifest.json, checksum) to GHCR as OCI artifacts under <version>-native-<target> tags; the dispatch payload carries them as natives: [{tag, digest}] for the cli handler to copy best-effort.
  • service-release.yml: new publish-natives (best-effort, continue-on-error) and mirror-ecr jobs. publish-release depends on mirror-ecr. Until the CLI_MIRROR_DISPATCH_TOKEN secret exists, mirror-ecr skips with a notice and releases behave exactly as today; once set, an unverified image mirror fails the release and verified releases list the ECR references in their notes.
  • ecr-mirror-check.yml: daily compare-only audit; manual request: true backfills.
  • Cross-review fixes (latest commit): a published release with no GHCR image (older postgres releases) is skipped and counted instead of aborting the whole audit, and native tag drift is reported without failing the audit or blocking --request on a handler that copies images only. ECR_MIRROR_REQUIRE_NATIVES=1 restores strict native checks once supabase/cli copies natives on develop.
  • Tests: scripts/ecr-mirror.test.ts and scripts/publish-native-oci.test.ts (bun), wired into repository-checks.yml; scripts/test-external-workflows.sh guards the workflow wiring.

Naming

Destination is public.ecr.aws/supabase/cli/<service> (same path as GHCR, only the registry host changes). Slim tags reuse upstream version strings, so they cannot share the upstream mirror repos at public.ecr.aws/supabase/<service>.

Test plan

  • bun test ./scripts/ecr-mirror.test.ts ./scripts/publish-native-oci.test.ts (21 tests) and bun build --no-bundle run in repository checks.
  • scripts/test-external-workflows.sh executes the release-notes block and asserts the job wiring.
  • Stubbed end-to-end runs of dispatch → poll → verify, the skip path, the non-blocking native path, and the required-natives path.

Rollout (outside this PR)

  1. Land the mirror-slim-image handler in supabase/cli (image copy is on develop; native copy is in ci(repo): mirror slim images and native OCI artifacts to ECR Public cli#6661).
  2. Create the cli/<service> ECR Public repositories or grant the handler's role ecr-public:CreateRepository.
  3. Set CLI_MIRROR_DISPATCH_TOKEN here. Leave it unset until the cli handler is ready; the daily audit is compare-only without it.
  4. Backfill by running ECR mirror check with request: true, in service-sized slices rather than one shot: each image verify can take up to 15 minutes and there are ~232 published releases.

This PR is safe to merge before any of the above: every new path is inert until the secret exists.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YaJgsq9xTCpMnEhqesr87S

Slim images are published to ghcr.io/supabase/cli/<service> only, so the
CLI cannot pull them through public.ecr.aws like its other images, and
mirror drift keeps surfacing as manifest-not-found failures in dependent
CI. Reuse the mirror machinery and AWS credentials that already live in
supabase/cli instead of provisioning AWS access for this repository.

- scripts/ecr-mirror.sh sends a mirror-slim-image repository_dispatch to
  supabase/cli and polls public.ecr.aws/supabase/cli/<service>:<version>
  anonymously until its index digest matches the published GHCR digest.
  Its sync mode audits every published release and can re-request
  out-of-sync tags, which doubles as the backfill path.
- service-release.yml gains a mirror-ecr job between publish-image and
  publish-release. It skips with a notice until the
  CLI_MIRROR_DISPATCH_TOKEN secret exists; once configured, an
  unverified mirror fails the release, and verified releases list the
  ECR references in their notes.
- ecr-mirror-check.yml runs the sync audit daily.
- docs/design/ecr-mirror-dispatch.md records the dispatch contract the
  supabase/cli handler must implement (digest-preserving copy, payload
  validation, repository creation) and the setup checklist.
- scripts/test-ecr-mirror.sh covers the payload contract, input
  validation, and the token guard, and is wired into repository checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YaJgsq9xTCpMnEhqesr87S
avallete and others added 2 commits September 1, 2026 12:05
list_releases dropped tags that fail today's tag_pattern (live
postgres-15 releases), dest digest inherited ambient registry
creds, and the daily check skipped entirely without the write
token. Audit by service prefix, isolate dest lookups, and always
run compare-only sync.

Co-authored-by: Cursor <cursoragent@cursor.com>
Overwrite GHCR tags and GitHub assets on republish, always dispatch ECR, and publish natives as :version-native-<target>.
avallete and others added 3 commits September 17, 2026 19:03
Replace the inline Python/shell mirror and publish helpers with bun
scripts so the same paths can be tested and replayed outside CI.
Publish native OCI artifacts and force same-version ECR mirrors
Two cross-review findings on the sync path:

- syncReleases failed on the first published GitHub Release with no
  GHCR image (postgres 15.8.1.085 predates image publication), so the
  audit and backfill never reached the remaining ~230 tags. Such
  releases are now logged, counted, and skipped.
- --request waited up to ECR_MIRROR_TIMEOUT per release for the native
  tags to land, but the live supabase/cli handler copies images only.
  232 serial waits cannot finish inside the job timeout. Native drift
  is now reported without failing the audit or blocking the backfill;
  ECR_MIRROR_REQUIRE_NATIVES=1 restores the strict behaviour once the
  handler copies natives.

Image drift still fails the audit, and the release-time mirror-ecr job
is unchanged. Tests cover the skip, the non-blocking default, and the
required-natives mode.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YaJgsq9xTCpMnEhqesr87S
@avallete
avallete requested a review from jgoux September 21, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants