Skip to content

ci: publish the container for linux/arm64 as well as linux/amd64 - #233

Merged
dkijania merged 5 commits into
mainfrom
ci/multiarch-arm64
Oct 7, 2026
Merged

dkijania merged 5 commits into
mainfrom
ci/multiarch-arm64

Conversation

@dkijania

@dkijania dkijania commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Why

ghcr.io/o1-labs/archive-node-api ships one architecture now. Its manifest index lists only:

linux amd64
unknown unknown   <- build provenance attestation, not an architecture

docker/build-push-action had no platforms:, so it built only for the ubuntu-latest runner. On Apple Silicon, this causes emulation and a platform-mismatch warning, or no matching manifest for linux/arm64 when emulation is off.

What changed

Workflow (.github/workflows/build.yaml)

  • platforms: linux/amd64,linux/arm64 on the build step.
  • Set up QEMU registers the binfmt handlers, so that the amd64 runner can execute aarch64 build steps.
    • The step runs a --privileged container. Thus it comes before "Authenticate to Google Cloud", as the ORDERING RULE at the top of the job requires.
    • The binfmt image is pinned by digest: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873…. The action's default is the mutable :latest tag. Dependabot does not bump a with: image: input, so a QEMU update is a manual change.
    • cache-image: false. The action's cache entry is keyed by tag, not digest, and adds ~32 MB per PR (The buildx cache is written on every PR and read by nothing #241).
  • Move cache uses if [ -d … ]; then mv …; fi. A build that wrote no new cache passes, and a real mv failure fails the step.

Dockerfile: three stages, node:24-alpine

A single stage ran npm ci under QEMU for the arm64 leg (752 s against 112 s native) and shipped the dev tree in the runtime image. Now:

  1. deps (target platform): npm ci --omit=dev --ignore-scripts. Production dependencies only. None of them has an install script.
  2. build (--platform=$BUILDPLATFORM): npm ci --ignore-scripts and tsc. The compiled output does not depend on the architecture, so this stage runs natively one time for all targets.
  3. runtime: node_modules from deps (never from build, which has amd64 modules) and build/ from build. Entrypoint (tini), CMD, USER nodeuser, WORKDIR, EXPOSE 8080 and HEALTHCHECK do not change.

All stages use node:24-alpine@sha256:ebfe2f90…, a multi-arch index (amd64, arm64/v8, s390x). Node 24 is an LTS line, supported to 2028-04-30. It replaces node:25-alpine from #239, which is end-of-life.

Measured results:

before after
build-and-deploy, CI 18m41s (amd64 + arm64, single stage) ~4m37s (amd64 + arm64)
image size, amd64 1.30 GB 222 MB
runtime node_modules 780.7 MB ~54 MB

The runtime image has no typescript, artillery, eslint, o1js or Playwright, and no native .node files.

Versioning policy (docs/versioning.md)

A move of the container image between LTS lines is minor, if engines and the image's HTTP contract (port, endpoints, environment variables, entrypoint, user) do not change. A move to a non-LTS line (Current or end-of-life) is still breaking. A raise of engines, or of the Node version that CI uses to publish, is still breaking. #235 makes the same edit, so the two PRs merge cleanly.

Release notes

How to verify after the first tag build

docker buildx imagetools inspect ghcr.io/o1-labs/archive-node-api:latest   # linux/amd64 + linux/arm64
docker run --rm --platform linux/arm64 --entrypoint node ghcr.io/o1-labs/archive-node-api:latest -e 'console.log(process.version, process.arch)'
docker run --rm --platform linux/amd64 --entrypoint node ghcr.io/o1-labs/archive-node-api:latest -e 'console.log(process.version, process.arch)'

🤖 Generated with Claude Code

@dkijania

Copy link
Copy Markdown
Contributor Author

CI is green, and the emulation cost is now measured rather than guessed.

build-and-deploy on this branch took 18m41s. The same job on #232, which is also based on current main but builds amd64 only, took 6m04s. So adding emulated arm64 costs about +12m30s, roughly tripling that job. Every other check is unchanged.

The arm64 layers do build and push, so the approach works — this is purely a wall-clock question.

If 18 minutes is too slow for the main branch, the alternative I mentioned in the description is a matrix over ubuntu-latest and ubuntu-24.04-arm that builds each architecture natively and merges the manifests. It is a larger diff, but it would bring the job back to roughly single-architecture time. Happy to switch this PR to that shape if you prefer it.

Note that PR builds push dev tags to GCP Artifact Registry only, so the multi-arch manifest cannot be inspected on ghcr.io until this lands and a release tag is built.

@SanabriaRusso

Copy link
Copy Markdown
Collaborator

Review

The PR does what it says, and every factual claim in the description checks out. I verified rather than read:

  • The pin is correct. gh api repos/docker/setup-qemu-action/git/ref/tags/v4.3.0 → 1f40c72289eff860ee54a304f1438e3cff362e0a. Exact match. (I also re-checked the neighbouring setup-buildx-action pin 37fe631… against its v4.3.0 tag — also exact. The file's pin comments are truthful.)
  • platforms: arm64 is the documented form, not a typo for linux/arm64. action.yml at that SHA documents the input as "Platforms to install (e.g. arm64,riscv64,arm)". The run log shows binfmt registering linux/arm64 correctly.
  • The base-image claim is true. Queried the Docker Hub registry API directly: node:22-alpine@sha256:16e22a55… is an OCI image index carrying linux/arm64/v8 (sha256:d51cff3f…) alongside amd64, arm/v6, arm/v7 and s390x. Both Dockerfile stages use that identical digest, so there is no second base image to catch us out. The digest pin and its reproducibility guarantee are unaffected.
  • Nothing downstream breaks. build.yaml has exactly one job and nothing after the build reads steps.build.outputs.digest or imageid. The SBOM job in security.yaml scans the source tree (path: .), never the image. smoke-load lives in a separate workflow, builds from source, and never pulls the image — it was absent from this PR's checks because of path filters, not because of anything here.

So: no blocking defects. What I would like to resolve before merge is the cost, because measurement shows it is being paid on the wrong events and for the wrong work.


1. Every PR pays the 3.3x, but only tag builds reach GHCR

The trigger is pull_request: branches: [main] plus push: tags: ['v*'], with no if: gating the new platform list — so both architectures are built on every same-repo PR.

Meanwhile Determine tags (.github/workflows/build.yaml:209-243) emits GHCR tags only on a tag push. PR builds push dev-<sha> to GAR only:

# PR build - use dev tags with SHA (GCP only)

The 13 extra minutes on every PR therefore produce an arm64 layer that never reaches ghcr.io/o1-labs/archive-node-api — the registry this PR exists to fix.

There is a genuine counter-argument, and I want to state it rather than skip it: building arm64 on PRs is what proves the arm64 build still works before you cut a tag. Gating it to tags means a broken arm64 build is discovered at release time, which is the worst moment. I do not think the answer is to give up that signal — I think it is to make the signal cheap, which is finding 2.

2. The real cost is npm ci under emulation — and most of it should not be running at all

Per-step timings from run 34880710868:

Dockerfile step amd64 arm64 (QEMU) ratio
RUN npm ci 111.8s 752.2s 6.7x
RUN npm run build (tsc) 4.6s 40.6s 8.8x
RUN apk add --no-cache tini 0.9s 3.6s 4.0x

The emulated npm ci alone is ~89% of the 847s build graph and sits on the critical path.

Dockerfile:7 is a bare npm ci, so the image build installs artillery, o1js, typescript, eslint and @playwright/browser-chromium — and runs their install scripts — then Dockerfile:18 COPY --from=build /app/node_modules ships every one of them into the runtime image. Under emulation that means a browser download and a unix-dgram node-gyp compile executed through qemu-aarch64. They didn't break (exit 0), but they are a latent breakage source and nearly all of those 752 seconds.

Two facts make this cheap to fix:

  • tsc output is architecture-independent, so the compile stage can be pinned to the build platform and run natively once for all targets.
  • No production code imports a devDependency. I checked: grep -rn "from '(o1js|graphql-request|ts-node|cross-env|artillery|c8)" src/ returns nothing, and the entrypoint is node build/src/index.js. --omit=dev is safe for the runtime image.

Suggested Dockerfile:

# Stage 1: runtime dependencies — TARGET platform, production only.
FROM node:22-alpine@sha256:16e22a550f3863206a3f701448c45f7912c6896a62de43add43bb9c86130c3e2 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev --ignore-scripts

# Stage 2: compile. tsc output is architecture-independent, so pin this stage
# to the BUILD platform — it runs natively once and serves every target.
FROM --platform=$BUILDPLATFORM node:22-alpine@sha256:16e22a550f3863206a3f701448c45f7912c6896a62de43add43bb9c86130c3e2 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --ignore-scripts
COPY src ./src
COPY tsconfig.json ./
RUN npm run build

# Stage 3: runtime — unchanged except these two COPYs.
COPY --from=deps  /app/node_modules ./node_modules
COPY --from=build /app/build ./build

The trap, flagged explicitly: with --platform=$BUILDPLATFORM on the build stage you must not keep COPY --from=build /app/node_modules — that would put amd64-resolved node_modules into the arm64 image. The separate deps stage is what makes this safe. Please keep the two COPY --from= lines exactly as written above.

This also shrinks the runtime image and drops the playwright browser and the node-gyp compile from it entirely — worth having on its own merits, independent of arm64.

Acceptance criteria

  • build-and-deploy on a PR returns to under ~9 minutes with both platforms still built.
  • docker buildx imagetools inspect <image> lists linux/amd64 and linux/arm64.
  • docker run --platform linux/arm64 <image> serves GET /healthcheck with 200.
  • The runtime image no longer contains node_modules/typescript or node_modules/@playwright.

How to test

docker buildx build --platform linux/amd64,linux/arm64 -t ana:multiarch .
docker run --rm --platform linux/arm64 -e PG_CONN=... -p 8080:8080 ana:multiarch &
curl -fsS localhost:8080/healthcheck
docker run --rm --platform linux/arm64 ana:multiarch ls node_modules/typescript   # expect: not found

If you would rather keep this PR strictly to the workflow, the fallback is to gate the platform list to tag builds — but I'd take the Dockerfile fix, since it keeps the PR-time arm64 signal that finding 1 would otherwise cost us.

3. On the matrix alternative you offered

I'd skip it. Native arm runners are free on a public repo, but splitting this job means duplicating WIF auth and GAR login across both legs — which widens credential exposure against the grain of the ORDERING RULE comment at :35-36 and issue 03 — or restructuring so npm publish stays in one place. Realistically 100+ lines with regression risk on exactly the credential-hygiene invariants this job was hardened for. Your instinct that the matrix is the escape hatch is right in spirit, but it aims at the wrong bottleneck: the fix is to stop doing the expensive work, not to find a faster machine to do it on.


Separately: the buildx cache has never hit once (pre-existing, not yours)

Worth an issue rather than a change here. From this PR's log:

Cache not found for input keys: Linux-buildx-deb9e9b3…, Linux-buildx-

Every buildx cache entry in the repo is scoped to refs/pull/N/merge — there is not one on refs/heads/main. build.yaml has no push: branches: [main] trigger, so the base-branch scope is never written, and GitHub scopes a PR-created cache to that PR's own ref, invisible to every other PR. The repo has been writing ~691 MB per PR into a cache only a second run of the same PR could read.

This PR doubles that to 1.39 GB (measured via gh api .../actions/caches: 691,198,082 B on #234 vs 1,390,076,079 B here — exactly 2.01x), plus a new 32 MB binfmt entry. The repo sits at 4.23 GB of the 10 GB limit; at 1.39 GB per PR, ~7 concurrent PRs start LRU-evicting — and the victims include the node-cache-…-npm entry on refs/heads/main that lint, unit-tests and smoke-load depend on.

Two things, both follow-ups:

  • Make the key platform-aware so single- and multi-arch entries can't cross-contaminate.
  • Either add push: branches: [main] so the base-branch scope gets populated and the cache starts earning its keep, or delete the cache steps — right now they are pure cost.

Also a nit while you're in there: Move cache lacks if: always(), so on a failed build the rotation is skipped and the post step saves the stale cache under the new key.


Disposition: approving the approach — the change is correct and the pins are clean. I'd like finding 2 in before merge (it's ~10 lines of Dockerfile and it's what makes arm64 affordable). Finding 3's cache work belongs in its own issue.

@dkijania

Copy link
Copy Markdown
Contributor Author

Finding 2 is in, as 17885ca, and the branch is rebased onto current main. Finding 3 is filed separately (link below). Everything here is measured, not estimated.

The Dockerfile now has three stages

Exactly the shape you specified, including the trap you flagged — node_modules comes from deps, never from the $BUILDPLATFORM-pinned build stage.

One thing I checked before taking --ignore-scripts, since it is the part that could bite silently: no production dependency has an install script. The node-gyp compile you saw is unix-dgram, reached only through artillery → artillery-plugin-publish-metrics → hot-shots, and the lockfile marks it optional: true, dev: true. --omit=dev removes it outright, so there is nothing for the ignored scripts to have built.

Measured, amd64, cold cache

before after
image size 1,297,918,614 B (1.30 GB) 221,929,240 B (222 MB)
runtime node_modules 780.7 MB 54.7 MB

And the number your acceptance criterion asks about — a cold build of both architectures took 2m30s, against 5m28s for the old single-architecture build on the same machine. Dual-arch is now faster than single-arch was. CI will have its own numbers, but the direction is not in doubt.

Acceptance criteria, run

  • ✅ Both platforms in one manifest: a --platform linux/amd64,linux/arm64 build emits an OCI index listing linux/amd64 and linux/arm64.
  • ✅ arm64 image serves: started under emulation, /healthcheck 200, /readiness 200, POST / {__typename} → {"data":{"__typename":"Query"}}, tini still PID 1, process.arch = arm64, zero native *.node files.
  • ✅ Dev tree gone from the runtime image: typescript, artillery, eslint and o1js all absent.

One correction to that last criterion. node_modules/@playwright still exists — as an empty scope directory, 4.0K, nothing inside. npm leaves the scope behind when it omits the package. The package is gone; the directory is not. Saying the criterion passed without that caveat would have been inaccurate.

Also

Added if: always() to Move cache, your nit, plus a guard so the mv cannot fail the step when the build produced no new cache directory.

Finding 3

Filed as its own issue rather than changed here, as you suggested — see the link in the next comment.

🤖 Generated with Claude Code

@dkijania

Copy link
Copy Markdown
Contributor Author

Finding 3 is now #241 — the buildx cache is written on every PR and read by nothing, and this PR doubles what it writes. Options laid out there; nothing needed in this PR for it.

🤖 Generated with Claude Code

@dkijania

Copy link
Copy Markdown
Contributor Author

CI numbers are in, and they beat the acceptance criterion by more than expected.

build-and-deploy on this branch: 4m37s.

For comparison, all on this same job:

build time
this branch before the Dockerfile fix (amd64 + arm64) 18m41s
main, amd64 only (measured on #232) 6m04s
this branch now (amd64 + arm64) 4m37s

So building both architectures is now faster than building one was — the emulated npm ci was costing more than the entire rest of the job. Your acceptance criterion was under ~9 minutes with both platforms; it lands at roughly half that.

All seven checks pass: Check Schema 18s, Linting 1m29s, Run-Tests 14m28s, SBOM 9s, build-and-deploy 4m37s, npm audit 1m51s, unit-tests 1m59s.

🤖 Generated with Claude Code

dkijania and others added 2 commits September 24, 2026 11:00
The image carried one architecture, because build-push-action defaults to
the runner's. Apple Silicon users therefore pulled an amd64 image and ran
it under emulation, or hit "no matching manifest for linux/arm64" outright.

The pinned node:22-alpine digest is already a multi-arch index carrying
linux/arm64/v8, so the pin needs no change — only QEMU on the runner and
the platform list on the build step.

Cost: arm64 layers are emulated on an amd64 runner, so this job gets
slower. If that becomes a problem, the alternative is a matrix over
ubuntu-latest and ubuntu-24.04-arm with a manifest merge, which builds
each architecture natively. This repository is public, so the arm runners
are free.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KfPTtp9B879CdXFu2hSU7d
Addresses finding 2 of the review. Building both architectures on an amd64
runner put `npm ci` under QEMU: 752s against 112s native, about 89% of the whole
build graph, and it installed the dev tree — a Playwright browser download and a
node-gyp compile, emulated — only to copy all of it into the runtime image.

Three stages now:

- `deps`, on the TARGET platform, runs `npm ci --omit=dev --ignore-scripts`.
  Safe: no production dependency has an install script, and the node-gyp build
  that used to run belongs to unix-dgram, an optional transitive dependency of
  artillery, which --omit=dev removes.
- `build`, pinned to $BUILDPLATFORM, compiles natively once and serves every
  target, because tsc output is architecture-independent.
- the runtime stage takes node_modules from `deps`, never from `build`. Copying
  it from a build-platform stage would put amd64 binaries in the arm64 image.

Measured locally, amd64, cold cache: the image drops from 1.30 GB to 222 MB and
its node_modules from 780.7 MB to 54.7 MB. A cold build of BOTH architectures
now takes 2m30s, against 5m28s for the old single-architecture build.

Verified on the arm64 image under emulation: /healthcheck 200, /readiness 200,
`{ __typename }` correct, tini still PID 1, node reports arch arm64, zero native
.node files, and typescript, artillery, eslint and o1js all absent.

Also guards the cache rotation with if: always(), as the review noted — a failed
build otherwise skips it and the post step saves the stale cache under the new
key.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dkijania

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #239 merged, and the base moves with it.

#239 bumped the image to node:25-alpine, which is end-of-life — v25 is an odd-numbered line, never LTS, end: 2026-06-01. So this branch now pins all three stages to node:24-alpine@sha256:ebfe2f90…, Active LTS until 2028-04-30, matching #235. That way main stops shipping an unsupported runtime whichever of the two lands first.

Re-verified after the rebase, on Node 24:

  • cold --platform linux/amd64,linux/arm64 build: 3m13s, both platforms in the manifest;
  • arm64 image runs v24.21.0, process.arch = arm64;
  • runtime node_modules 54.5 MB, with typescript, artillery, eslint and o1js all absent.

Overlap with #235, stated plainly: both PRs now set the same FROM digest, so whichever merges second will show a small conflict on those lines. It resolves by keeping this PR's three-stage structure with that same Node 24 digest — no judgement call in it. Everything else is disjoint: #235 owns the CI matrix, the EOL Node 20 workflow fixes and the dependabot rule; this PR owns QEMU, platforms: and the Dockerfile stages.

🤖 Generated with Claude Code

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks for turning this around. I re-checked every point from my 21 Sep review against e584b3a and against the images CI pushed, not just the thread.

Resolved

  • F1/F2, Dockerfile split: in, exactly as specified. node_modules comes from deps (Dockerfile:45), not from the $BUILDPLATFORM stage. The CI log shows build running only as [linux/amd64 build …]. The arm64 npm ci --omit=dev dropped from 752s to 68.2s, and build-and-deploy went from ~18m41s to ~4m37s.
  • The --ignore-scripts premise holds. Every lockfile entry with hasInstallScript (@playwright/browser-chromium, fsevents, protobufjs, unix-dgram) is dev: true. Every bare import in src/ resolves to a production dependency.
  • Multi-arch is real. dev-2227080 in GAR is an OCI index with linux/amd64, linux/arm64 and two provenance manifests. A --platform linux/amd64 pull resolves to the amd64 config (ec99e8e…), so existing amd64 consumers are unaffected. The arm64 image:
    • runs v24.21.0 arm64;
    • has a 52 MB node_modules with no typescript, artillery, eslint, o1js, rimraf or playwright;
    • has no native .node files;
    • keeps entrypoint tini / node build/src/index.js and user nodeuser.
  • Move cache now has if: always(). F3 (buildx cache) is tracked in #241.

Backward compatibility

  • Tag names and forms are unchanged: dev-<merge sha>, the short form, semver, major.minor, major and latest, in both GAR and GHCR.
  • Digest pinning is not newly affected. Today's latest on GHCR and GAR is already an OCI index (amd64 plus an attestation), so this PR only adds entries.
  • Consumers:
    • The gitops deployments pin the GAR image by dev-… tag on amd64 Hetzner nodes, so they are unaffected.
    • mina-explorer and mina-explorer-api use the API over HTTP only and never pull this image.
  • No workflow reads the build step's digest outputs. WIF auth and step ordering are unchanged.
  • actionlint is clean, and the branch merges cleanly with current main (#162).

Non-blocking. Please do the first one before merging.

  1. The PR description is stale. It still says "No Dockerfile change is needed" and refers to node:22-alpine@sha256:16e22a55…. On a squash merge that text becomes the commit message, so please describe the three-stage Dockerfile and the Node 24 pin instead.
  2. build.yaml:260: stale comment (node:22-alpine); see the inline suggestion.
  3. Runtime moves to Node 24 while the test jobs stay on 22. The current GHCR latest runs NODE_VERSION=22.23.1, and the next release will run 24.21.0. Meanwhile every CI test job still uses node-version: '22' (run-tests.yaml:33, unit-tests.yaml:20). #235 closes that gap, so please either merge #235 right after this one or call out the runtime bump in the release notes. docs/versioning.md:166 still says "Node 22".
  4. build.yaml:274-275: [ -d … ] && mv …; exit 0 also hides a real mv failure. Suggested:
    run: |
      rm -rf /tmp/.buildx-cache
      if [ -d /tmp/.buildx-cache-new ]; then mv /tmp/.buildx-cache-new /tmp/.buildx-cache; fi
    Acceptance: actionlint stays clean; a run whose build fails still exits 0 from this step, while a failing mv now fails the step.
  5. Dockerfile:5-8: the "main briefly shipped node:25 … #235 carries the same move" note will read oddly once both PRs land. Consider trimming it to the LTS rationale. When you resolve the expected FROM conflict with #235, keep this PR's three-stage structure.
  6. The branch is behind main (#162). The merge is clean, but please rebase or re-run CI before merging so the last dev-* image includes #162.

Post-merge check (first tag build)

docker buildx imagetools inspect ghcr.io/o1-labs/archive-node-api:latest   # expect linux/amd64 + linux/arm64
docker run --rm --platform linux/arm64 --entrypoint node ghcr.io/o1-labs/archive-node-api:latest -e 'console.log(process.version, process.arch)'
docker run --rm --platform linux/amd64 --entrypoint node ghcr.io/o1-labs/archive-node-api:latest -e 'console.log(process.version, process.arch)'

Comment thread .github/workflows/build.yaml Outdated
with:
context: .
push: true
# The pinned node:22-alpine digest is a multi-arch index that carries

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale now that the pin is node:24-alpine.

Suggested change
# The pinned node:22-alpine digest is a multi-arch index that carries
# The pinned node:24-alpine digest is a multi-arch index that carries

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9edaeb3. The comment now says node:24-alpine.

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Withdrawing my approval from earlier today, sorry for the churn. A second pass found one supply-chain issue that should be fixed before merge. Everything in my previous review still stands: backward compatibility is fine, F1/F2 are resolved, and the tag forms and amd64 resolution are unchanged. This is the only blocker.

Blocking: Set up QEMU runs a privileged, unpinned container after credentials are on disk

The job header states the ORDERING RULE: every step that runs dependency code comes before Authenticate to Google Cloud, and after it only the registry logins, npm publish and the docker build may run.

The new step sits after Authenticate to Google Cloud, both docker/login-action steps and Publish to npm registry. With no image: input, setup-qemu-action runs the action's default, docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install arm64 (visible in run 35979103452). At that point ~/.docker/config.json holds the GAR access token and, on tag builds, the packages: write GITHUB_TOKEN for GHCR. A privileged container can read the host filesystem, and :latest is a mutable Docker Hub tag, so what runs here can change without any diff in this repo. The action also caches that image in the GHA cache under a tag-keyed entry (~32 MB per PR, which adds to #241).

QEMU needs no credentials, so the fix costs nothing: move the step above the auth step, pin the image by digest, and disable the image cache.

Proposed fix (actionlint clean; the digest is the one CI already resolved for :latest on this PR):

@@ (just above "Authenticate to Google Cloud") @@
+      # arm64 layers are built on an amd64 runner, so register the binfmt
+      # handlers that let the runner execute aarch64 binaries. Must come before
+      # buildx sets up the builder. It runs a --privileged container, so it sits
+      # BEFORE "Authenticate to Google Cloud" (ORDERING RULE) and the image is
+      # pinned by digest rather than the action's default mutable :latest.
+      # cache-image is off: the action's GHA cache entry is keyed by tag, not
+      # digest, and adds ~32 MB per PR to the cache budget (#241).
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4.3.0
+        with:
+          image: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0
+          platforms: arm64
+          cache-image: false
+
       # Keyless: the runner presents its GitHub OIDC token and receives a
@@ (current position, after the second credential-file assertion) @@
-      # arm64 layers are built on an amd64 runner, so register the binfmt
-      # handlers that let the runner execute aarch64 binaries. Must come before
-      # buildx sets up the builder.
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4.3.0
-        with:
-          platforms: arm64
-
       - name: Set up Docker Buildx

How the digest was checked:

  • skopeo inspect shows that tonistiigi/binfmt:qemu-v10.2.3 resolves to the same index, sha256:400a4873…, with revision annotation e29e7d7.
  • podman run --platform linux/amd64 …@sha256:400a4873… --version prints binfmt/e29e7d7 qemu/v10.2.3 go/1.26.4, which matches the Binfmt version group in the CI log.
  • At the pinned SHA, the action passes image straight to docker pull / docker run, so tag@digest works.

Dependabot does not bump a with: image: input, so a newer QEMU has to be bumped by hand. That is the point of the pin.

Acceptance criteria

  • Set up QEMU appears before Authenticate to Google Cloud in build.yaml.
  • The image: input is pinned with @sha256:. The run log shows docker pull …@sha256:400a4873… and no binfmt:latest.
  • There is no Caching docker.io--tonistiigi--binfmt… post step in the log.
  • build-and-deploy still shows both the linux/amd64 and linux/arm64 legs and exporting manifest list, and pushes both dev-* tags.
  • actionlint .github/workflows/build.yaml is clean.

How to test

  1. Run actionlint .github/workflows/build.yaml.
  2. Push, then open the build-and-deploy log.
  3. Check the step order: QEMU, then Authenticate, then the logins, then publish, then Buildx.
  4. Check that the Binfmt version group prints binfmt/e29e7d7 qemu/v10.2.3.
  5. Check that [linux/arm64 deps …] RUN npm ci --omit=dev --ignore-scripts and exporting manifest list both appear.
  6. With GAR access: docker buildx imagetools inspect europe-west3-docker.pkg.dev/o1labs-192920/euro-docker-repo/archive-node-api:dev-<sha> should list linux/amd64 and linux/arm64.

Still open from my previous review (non-blocking)

  • The PR description is stale (it still says "No Dockerfile change is needed" / node:22-alpine), and on a squash merge it becomes the commit message. This PR is also a runtime major change: v1.0.0 shipped Node 22, and the next release ships 24. Please call that out, or land #235 right after this so CI tests the shipped runtime.
  • The build.yaml:260 comment is stale (node:22-alpine).
  • Move cache: actions/cache declares post-if: success(), so on a failed build the post step does not save a stale cache anyway. if: always() is harmless, but its comment's rationale does not hold. [ -d … ] && mv …; exit 0 also hides a real mv failure; if [ -d … ]; then mv …; fi is safer.
  • Rebase onto main (#162) before merging.

Comment thread .github/workflows/build.yaml Outdated
# arm64 layers are built on an amd64 runner, so register the binfmt
# handlers that let the runner execute aarch64 binaries. Must come before
# buildx sets up the builder.
- name: Set up QEMU

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: with no image: input this runs docker run --privileged docker.io/tonistiigi/binfmt:latest (see the run log) after GCP auth, both registry logins and npm publish. At that point ~/.docker/config.json holds the GAR token and, on tags, the GHCR GITHUB_TOKEN, which breaks the ORDERING RULE at the top of the job. Please move this step above Authenticate to Google Cloud and add image: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0 plus cache-image: false. That digest is the one CI resolved for :latest on this run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9edaeb3, as you proposed:

  • Set up QEMU is now before Authenticate to Google Cloud, after the first credential-file assertion and before Set up Docker Buildx.
  • image: docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873…. I checked it myself: the Docker Hub registry returns docker-content-digest: sha256:400a4873b838d1b89194d982c45e5fb3cda4593fbfd7e08a02e76b03b21166f0 for the tag qemu-v10.2.3.
  • cache-image: false. Both inputs are in the action.yml of setup-qemu-action at 1f40c72.
  • actionlint is clean.


# always(): without it a failed build skips the rotation, and the post
# step then saves the stale cache under the new key.
- name: Move cache

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: actions/cache declares post-if: success(), so on a failed build the post step doesn't save at all. if: always() is harmless, but the comment's rationale doesn't hold.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. actions/cache@55cc834 declares post-if: "success()". Fixed in 9edaeb3:

  • I removed if: always().
  • The comment now gives the correct reason.
  • The step uses if [ -d /tmp/.buildx-cache-new ]; then mv …; fi. A build that wrote no new cache passes, and a real mv failure fails the step.

@SanabriaRusso

Copy link
Copy Markdown
Collaborator

Cross-reference to my review on #235. docs/versioning.md:39-40 classes moving the Docker base image to a newer Node runtime as breaking. This PR also moves the image from Node 22 (1.0.x) to node:24-alpine, so the same policy decision applies here: either clarify the policy (LTS→LTS image moves are minor while engines and the HTTP contract are unchanged), or plan the next release as a MAJOR. Proposed wording and acceptance criteria are in the #235 review. Whichever of #233 and #235 lands first should carry the policy change.

dkijania and others added 2 commits October 2, 2026 22:09
- Move "Set up QEMU" above "Authenticate to Google Cloud". It runs a
  --privileged container, so the ORDERING RULE applies to it. Pin the
  binfmt image to qemu-v10.2.3@sha256:400a4873… instead of the action's
  default mutable :latest, and set cache-image: false (#241).
- Move cache: drop always() (actions/cache saves only on success) and
  replace `[ -d ] && mv; exit 0` so that a real mv failure fails the step.
- Fix the stale node:22-alpine comment and trim the Dockerfile Node note
  to the LTS rationale.
- docs/versioning.md: moving the image between LTS lines, with engines
  and the HTTP contract unchanged, is minor. A non-LTS image line stays
  breaking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dkijania

dkijania commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Your review is addressed in 9edaeb3. Before this commit, I merged main (#162 and #237) into the branch. This was a merge, not a rebase, so the history did not change.

Blocking: QEMU step

Acceptance criterion Result (run 37058938747)
Set up QEMU comes before Authenticate to Google Cloud ✅ Step order: QEMU → Authenticate → logins → publish → Buildx.
The image is pinned with @sha256:, with no binfmt:latest ✅ docker pull docker.io/tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873…. Binfmt version shows binfmt/e29e7d7 qemu/v10.2.3 go/1.26.4. There are 0 binfmt:latest lines.
No Caching docker.io--tonistiigi--binfmt… post step ✅ The log has 0 such lines (cache-image: false).
Both platform legs and a manifest list ✅ [linux/arm64 deps 4/4] RUN npm ci --omit=dev --ignore-scripts, [linux/amd64 build …] runs natively, exporting manifest list sha256:de5b361b…. Both dev-* tags are pushed.
actionlint is clean ✅ actionlint 1.7.7.

build-and-deploy took 3m09s.

I checked the pin myself:

  • For the tag qemu-v10.2.3, the Docker Hub registry returns docker-content-digest: sha256:400a4873….
  • The action.yml of setup-qemu-action at 1f40c72 declares the image and cache-image inputs.

Non-blocking items

  • PR description: I wrote it again. It now gives the three-stage Dockerfile, the node:24-alpine pin, the QEMU order and pin, and the policy change. It also says that the runtime moves from Node 22 to Node 24 in the next release notes, and that build(docker): move the image to Node 24 LTS, and test what we ship #235 must merge directly after this PR.
  • build.yaml comment: it now says node:24-alpine.
  • Move cache:
    • I removed if: always(), because actions/cache declares post-if: "success()".
    • The comment gives the correct reason now.
    • The step uses if [ -d … ]; then mv …; fi.
  • Dockerfile header: I made the Node note shorter. It now gives only the LTS reason.
  • Versioning policy: your option (a) from the build(docker): move the image to Node 24 LTS, and test what we ship #235 review is applied to docs/versioning.md, with the same text as in your review. A move of the image between LTS lines is minor if engines and the HTTP contract do not change. A move to a non-LTS line is still breaking. build(docker): move the image to Node 24 LTS, and test what we ship #235 gets the same edit, so the two PRs merge without a conflict. On this branch, the 1.0.0 migration note already says Node 22, so I did not change it.
  • Behind main: fixed by the merge.

When I wrote this comment, Run-Tests was still running. All other checks pass.

🤖 Generated with Claude Code

SanabriaRusso
SanabriaRusso previously approved these changes Oct 7, 2026

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I re-checked 9edaeb3 against my 2 Oct review and against the log of run 37058938747.

  • Blocker resolved.
    • Set up QEMU now runs before Authenticate to Google Cloud and both registry logins.
    • The log shows docker pull tonistiigi/binfmt:qemu-v10.2.3@sha256:400a4873… and binfmt/e29e7d7 qemu/v10.2.3, with no :latest.
    • There is no binfmt cache post step.
  • Build output. Both platforms build, and build runs natively, only as [linux/amd64 build …]. arm64 deps uses --omit=dev --ignore-scripts. exporting manifest list appears, and both dev-* tags are pushed to GAR. All 7 checks are green.
  • Earlier nits are all fixed: the stale node:22 comment, Move cache (no always(), and a real mv failure now fails the step), the Dockerfile note, and the PR description.
  • Branch. It is level with main (#239/#240/#162 included). The QEMU step is the only new action pin, and node:24-alpine correctly replaces the non-LTS node:25 from #239.
  • Backward compatibility. Tag forms and registries are unchanged, and amd64 hosts still resolve to the amd64 manifest. mina-explorer and mina-explorer-api consume the API over HTTP only, so they are unaffected.
  • Follow-ups.
    • Merge #235 right after this PR, so CI tests the Node 24 runtime that ships.
    • In the next release notes, call out the runtime moving to Node 24.
    • Optional: if arm64 ever becomes flaky, gate it behind a repo variable so amd64 publishing can't be blocked.

dkijania added a commit that referenced this pull request Oct 7, 2026
…235)

Supersedes #222. Takes **option B** from @SanabriaRusso's review on that
PR: go to Node **24 (LTS)** rather than 26, and land the runtime move
whole — image, CI, and docs together.

## Why not Node 26

Re-checked against `nodejs/Release/schedule.json` today (2026-09-21),
not quoted from the review:

| line | status today | end |
|---|---|---|
| v20 | **EOL** since 2026-04-30 | 2026-04-30 |
| v22 | Maintenance LTS | 2027-04-30 |
| v24 | **Active LTS** until 2026-10-20, then Maintenance LTS |
2028-04-30 |
| v26 | **Current until 2026-10-28** | 2029-04-30 |

Node 26 is still Current for another five weeks. Current lines take
semver-major changes, so a routine Dependabot digest refresh on
`26-alpine` can move a major straight into the production image.

## What changed

**Image** — both stages to
`node:24-alpine@sha256:ebfe2f90462722a7a4de65e91990e97fe0d401c70e0e762c5b53302f905ec1c1`.

Note this is **not** the digest named in the review. That one
(`e67514e5…`, verified 2026-09-02) is no longer what the tag resolves
to. I re-resolved `24-alpine` from the registry: a genuine OCI index
carrying `linux/amd64`, `linux/arm64/v8` and `linux/s390x`, image config
`NODE_VERSION=24.21.0`, built 2026-09-17. The arm64 manifest matters for
#233.

Correction to an earlier version of this description: the image *config*
has no `org.opencontainers.image.version` label, but the OCI *index
annotations* do carry `org.opencontainers.image.version=24-alpine`.
`NODE_VERSION=24.21.0` in the image config is further evidence.

**CI now runs the runtime we ship.** This was the review's blocking
objection.

| workflow | was | now |
|---|---|---|
| `run-tests.yaml` | `'22'` | matrix `['22', '24']` |
| `unit-tests.yaml` | `'22'` | matrix `['22.12.0', '24']` |
| `live-integration`, `publish-npm`, `build`, `graphql-inspector`,
`lint` | `22` | `24` |
| `smoke-load-test.yaml` | `'22.12.0'` | `'24'` |
| `security.yaml` | **`'20'` — EOL 2026-04-30** | `24` |
| `nightly-devnet-dump.yaml` | **`'20'` — EOL 2026-04-30** | `24` |

22 stays in the matrix because it is the `engines` floor; 24 is what the
image runs. `unit-tests.yaml` pins the exact floor `22.12.0`, because a
bare `'22'` resolves to the newest 22.x. `run-tests.yaml` keeps `'22'`,
because it starts a Mina local network and is expensive. The two Node 20
jobs were running an EOL runtime on `main` and are fixed here, as the
review's criterion 3 requires.

The matrix doubles `run-tests.yaml`, which starts a Mina local network.
I followed the review's instruction literally; say the word and I will
pin that job to `24` alone and leave the floor coverage to
`unit-tests.yaml`.

**Docs** — `docs/getting-started.md` said "Node.js 20+" while `engines`
requires `>=22.12.0` (wrong on `main` already); `docs/versioning.md` now
names both the npm floor and the image's major. `AGENTS.md` now says
Volta pins 22.12.0, not 20.18.0.

**`engines` unchanged at `>=22.12.0`.** Raising it breaks npm consumers
and is a separate decision (criterion 5). `volta.node` stays at
`22.12.0` so local development runs the floor.

**Kept from the #222 branch**, both cleared by the review: `ENV
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1` (build stage only) and `npm ci
--no-audit --no-fund`. Confirmed in the built image: `@playwright` is
124 KB of package metadata with no browser binary anywhere, and the
audit gate is a separate job (`security.yaml`), untouched.

**Beyond option B, easy to drop if you disagree:** `dependabot.yml` now
ignores Node **semver-major** bumps. Digest and patch refreshes stay
automatic, which is the security value; the major moves deliberately,
with CI and docs. Without this, Dependabot re-opens the Node 26 PR next
week.

## Versioning policy

`docs/versioning.md` classed a move of the Docker base image to a newer
Node as **breaking**. This PR takes option (a) from the review: a move
of the image between **LTS lines**, with `engines` and the image's HTTP
contract (port, endpoints, environment variables, entrypoint, user)
unchanged, is **minor**. A move to a non-LTS line (Current or
end-of-life) stays breaking. So the next release can be a MINOR.

The 0.0.6 → 1.0.0 migration note again says what 1.0.x shipped:
`engines` 22.12, and the image on Node 22 (the `v1.0.0` tag's
`Dockerfile` has `FROM node:22-alpine`). **The move to Node 24 goes in
the release notes of the release that contains this PR.** #233 moves the
image to the same digest and gets the identical policy text, so the two
merge cleanly.

## Verification — the #189 contract, executed

Built the image and ran it against a Postgres carrying the 17
`USED_TABLES`:

| check | result |
|---|---|
| `GET /healthcheck` | **200** |
| `GET /readiness` | **200** |
| `POST / {"query":"{ __typename }"}` |
`{"data":{"__typename":"Query"}}` |
| PID 1 | `/sbin/tini -- node build/src/index.js` |
| `wget -qO- "http://127.0.0.1:${PORT:-8080}/healthcheck"` inside the
container | exit **0** |
| Docker `HEALTHCHECK` state | **healthy** |
| `SIGTERM` | `Shutting down (SIGTERM)…`, exit code **0** |

Runtime facts from the built image: node v24.21.0, npm 11.19.0, OpenSSL
**3.5.8**, TLS defaults TLSv1.2/TLSv1.3, alpine 3.24.2, `/usr/bin/wget`
→ busybox, **0** native `*.node` modules. Image 1.31 GB.

Acceptance criteria 1, 3 and 5 re-run and passing; both `FROM` lines
carry the identical `tag@digest`, the LTS check prints `LTS OK` for
major 24, and no workflow pins an EOL major.

## Not fixed here

Three files this PR touches already fail `prettier` on `main` — as does
the rest of the repo (47 files). `lint.yaml` runs `prettier
--debug-check`, which never exits non-zero, so nothing enforces it. That
is #212 and #213, left alone rather than mixed into a runtime change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: SanabriaRusso <luis@o1labs.org>
Resolve the Dockerfile conflict with #235 as the review asked: keep this
branch's three stages and the shared node:24-alpine digest. From main, keep
--no-audit --no-fund on both npm ci lines and the LTS rationale. Drop
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD, because --ignore-scripts already stops the
browser download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dkijania

dkijania commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

main is merged into this branch in e2b9fc4, because #235 landed first. This was a normal push, not a force-push.

The Dockerfile conflict is resolved as your #235 review asked:

  • The three stages from this PR stay. All three use the same node:24-alpine@sha256:ebfe2f90… digest.
  • From main, I kept --no-audit --no-fund on both npm ci lines and the sentence about Current lines in the LTS comment.
  • I removed PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD, because --ignore-scripts already stops the browser download.

build.yaml merged with no conflict. The step order is still QEMU, then Authenticate, then Buildx, and the workflow uses Node 24 from #235.

Local check (amd64): the image builds, runs Node v24.21.0, is 230 MB, and has no typescript, artillery, eslint or o1js.

🤖 Generated with Claude Code

@SanabriaRusso SanabriaRusso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving (re-review). The only push since the previous approval is merge commit e2b9fc4, which brings in #235. git range-diff shows the PR's own commits (17f1edf, e584b3a, 9edaeb3) are unchanged.

What changed since the last approval (the merge with #235)

  • The Dockerfile conflict is resolved correctly.
    • All three stages keep the shared node:24-alpine@sha256:… pin.
    • Both npm ci lines take main's --no-audit --no-fund.
    • Dropping PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 is safe. Both install stages use --ignore-scripts, so the Playwright postinstall never runs. That ENV was only ever set in the build stage, so the runtime image is unaffected.
  • The branch is level with main (0 commits behind). The diff against main is now only Dockerfile and .github/workflows/build.yaml.

Whole PR, re-checked at head e2b9fc4

  • Step order. Set up QEMU still runs before the registry logins and before Set up Docker Buildx.
  • binfmt version. binfmt is pinned (qemu-v10.2.3@sha256:…); nothing runs from :latest.
  • Pinning and permissions.
    • setup-qemu-action is pinned by SHA (v4.3.0).
    • Permissions are unchanged.
    • Authentication is still WIF only, with no stored credential.
  • Backward compatibility for amd64 consumers.
    • Determine tags is untouched. PR images are still tagged with the merge-commit sha. Release tags (X.Y.Z, X.Y, X, latest) are unchanged.
    • ghcr.io/o1-labs/archive-node-api:latest is already an OCI index. This PR only adds a linux/arm64 entry, so amd64 hosts resolve exactly as before, and digests consumers have already pinned are not re-pushed.
  • No native-module risk on arm64. Every lockfile entry with an install script is dev: true, and the runtime install uses --omit=dev --ignore-scripts.
  • CI.
    • All 12 checks pass, including unit and integration tests on Node 22 and 24.
    • build-and-deploy exported the manifest list and pushed both tags.

Non-blocking: the PR description is out of date. With a squash merge, the description becomes the commit message. Since #235 landed first, three parts no longer apply. Please trim them before merging:

  • the docs/versioning.md section;
  • the "Merge #235 directly after this PR" line;
  • the "CI test jobs run on Node 22 only" line.

@dkijania
dkijania merged commit 0728877 into main Oct 7, 2026
12 checks passed
@dkijania
dkijania deleted the ci/multiarch-arm64 branch October 7, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants