Skip to content

fix(deploy): wait for the new build before reporting the rollout - #41

Merged
eersnington merged 1 commit into
mainfrom
fix/deploy-rollout-status
Nov 16, 2026
Merged

eersnington merged 1 commit into
mainfrom
fix/deploy-rollout-status

Conversation

@eersnington

Copy link
Copy Markdown
Member

Right after the pool PUT, Rivet Cloud can still return the pool status from before it, already with the new build_id. orange deploy read that old status as the result.

Pool status read 2 s after a PUT on a real cluster:

18:58:21  build 489a417d  ready  since 18:55:02  1/1 updated   <- status of the previous build
18:58:26  build 489a417d  updating  since 18:58:19
18:59:31  build 489a417d  degraded  since 18:59:30

Same cluster, orange deploy --build <id>:

Deploy main This PR
A good build ✓ Deployed 3 s after the PUT. The pool was updating for 81 s more. ✓ when the pool is ready on it
A build that crashes on start ✓ Deployed, exit 0 degraded (user fault): … App keeps exiting, exit 1, Roll back with: … --build <previous good build>
A rollback from that build to the good one exit 1, Build <good> failed, Roll back with: … --build <crashing build> ✓ Deployed
The build the pool already runs ✓ in 3 s
  • A status counts only after the pool was seen rolling, or when it differs from the status read before the PUT. Only Rivet Cloud values are compared, not the local clock.
  • The rest follows wait_for_rollout in rivet deploy (engine/packages/cli/src/commands/deploy/cluster.rs): the pool must stay on the build, every node updated and ready, and degraded or down fails only when action_needed is set.
  • A redeploy that changes nothing (same build, env, secrets, node count, machine type) takes the current status.
  • The rollback hint names the previous build only if the pool was ready on it.

Right after the pool PUT, Rivet Cloud can still return the pool status
from before it, with the new build_id. Deploy read that old status as the
result: a false success for a new build, and a false failure for a
rollback from a degraded pool.

A status now counts only after the pool was seen rolling, or when it
differs from the status read before the PUT. The rest of the wait follows
rivet deploy: nodes updated and ready, and degraded or down fails only
when action_needed. A redeploy that changes nothing takes the current
status. The rollback hint names the previous build only if it was ready.
@eersnington
eersnington merged commit be4ce54 into main Nov 16, 2026
10 checks passed
@eersnington
eersnington deleted the fix/deploy-rollout-status branch November 16, 2026 19:45
@github-actions

Copy link
Copy Markdown

Preview removed; R2 assets deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant