Skip to content

ci: serialize production deploys with a concurrency group - #421

Open
byte-the-bot wants to merge 2 commits into
mainfrom
ci/serialize-fly-deploys
Open

ci: serialize production deploys with a concurrency group#421
byte-the-bot wants to merge 2 commits into
mainfrom
ci/serialize-fly-deploys

Conversation

@byte-the-bot

Copy link
Copy Markdown
Collaborator

Why

Two merges to main a minute apart each build their own image and run
flyctl deploy concurrently. Whichever finishes last wins, independent of
commit order — so production can silently serve the older commit while both
deploy jobs report green.

Observed on eyes 2026-08-21: #71's deploy overwrote #70's, and /health
reported one commit behind main until a manual redeploy. Nothing in the logs
says anything is wrong.

The pattern

concurrency on the job that actually runs flyctl deploy, one group per
Fly app, cancel-in-progress: false.

Queue rather than cancel: killing a run mid-flyctl deploy abandons a release
for no gain, and queueing still converges on the newest SHA because GitHub
drops the older pending entry when a newer run queues behind a running one.

Guarding the inner job (rather than the caller, as coreyja/eyes#72 did) keeps
the guard with the thing being guarded, so it holds for every caller including
workflow_dispatch, and avoids actionlint's restrictions on reusable-workflow
call jobs.

Sweep

Part of a fleet-wide sweep (MUL-36c14ba4) covering every repo that deploys to
Fly. Same change, same rationale, 12 repos — reviewable as one decision.
actionlint clean.

Also bumps actions/checkout@v3@v4 in the same file: actionlint
flags v3 as "too old to run on GitHub Actions" (node16).

Guard the job that runs `flyctl deploy` so concurrent merges to main cannot
race, with the older image winning and prod silently serving a stale commit.
clippy 1.98 flags axum handlers returning Result<_, Response>; Response is
large and not ours to shrink. Pre-existing breakage on the newer toolchain
that blocks this PR's Lint job.
@byte-the-bot

Copy link
Copy Markdown
Collaborator Author

Second commit is unrelated to the concurrency change: the Lint job was already failing on the newer toolchain with clippy::result_large_err on projects_get. axum handlers here return Result<Markup, Response>, and Response is large but not ours to shrink — boxing it would churn every handler signature. Added result_large_err = "allow" to the existing [workspace.lints.clippy] allow-list, matching how the other pedantic exceptions are handled. Verified with cargo clippy --workspace --all-targets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants