ci: serialize production deploys with a concurrency group - #421
Open
byte-the-bot wants to merge 2 commits into
Open
ci: serialize production deploys with a concurrency group#421byte-the-bot wants to merge 2 commits into
byte-the-bot wants to merge 2 commits into
Conversation
Guard the job that runs `flyctl deploy` so concurrent merges to main cannot race, with the older image winning and prod silently serving a stale commit.
clippy 1.98 flags axum handlers returning Result<_, Response>; Response is large and not ours to shrink. Pre-existing breakage on the newer toolchain that blocks this PR's Lint job.
Collaborator
Author
|
Second commit is unrelated to the concurrency change: the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two merges to
maina minute apart each build their own image and runflyctl deployconcurrently. Whichever finishes last wins, independent ofcommit order — so production can silently serve the older commit while both
deploy jobs report green.
Observed on
eyes2026-08-21: #71's deploy overwrote #70's, and/healthreported one commit behind
mainuntil a manual redeploy. Nothing in the logssays anything is wrong.
The pattern
concurrencyon the job that actually runsflyctl deploy, one group perFly app,
cancel-in-progress: false.Queue rather than cancel: killing a run mid-
flyctl deployabandons a releasefor no gain, and queueing still converges on the newest SHA because GitHub
drops the older pending entry when a newer run queues behind a running one.
Guarding the inner job (rather than the caller, as coreyja/eyes#72 did) keeps
the guard with the thing being guarded, so it holds for every caller including
workflow_dispatch, and avoidsactionlint's restrictions on reusable-workflowcall jobs.
Sweep
Part of a fleet-wide sweep (MUL-36c14ba4) covering every repo that deploys to
Fly. Same change, same rationale, 12 repos — reviewable as one decision.
actionlintclean.Also bumps
actions/checkout@v3→@v4in the same file:actionlintflags v3 as "too old to run on GitHub Actions" (node16).