Repository navigation
Harden autoupdate deploy and recover the RustFS bucket - #339
Conversation
Follow-ups to #338: - file-service health checks the bucket and recreates it when missing (RustFS uses an emptyDir, so a RustFS restart drops the bucket) - readiness probe timeout raised to fit the storage check - autoupdate: generate manifests before pulling images, source the env file only in that subshell, delete workers before applying manifests, and time-box the health check requests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f297229207
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| exists, err := s.s3Client.BucketExists(ctx, BUCKET_NAME) | ||
| if err == nil && !exists { | ||
| err = s.ensureBucket(ctx) |
There was a problem hiding this comment.
If bucket recreation succeeds but SetBucketLifecycle fails—for example, because the shared three-second context expires—the bucket remains without its one-day expiration rule. Subsequent health probes see that the bucket exists, skip ensureBucket, and report success, so lifecycle configuration is never retried and newly uploaded artifacts accumulate indefinitely. Track successful lifecycle setup or reconcile the rule on subsequent probes before considering recovery complete.
Useful? React with 👍 / 👎.
- Revert deleting workers before apply: a failing apply or rustfs wait would leave control with only deleted workers - Retry the bucket lifecycle rule until it is set - Don't exit on startup when storage is not reachable yet; stay up and let the readiness probe report it - Keep both errors when bucket creation and the existence check fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Test Results SummaryThis comment shows the latest test results for this pull request.
|
Follow-ups to #338.
/api/v1/healthnow checks the upload bucket and recreates it (including its expiry rule) when it's missing. RustFS stores data in anemptyDir, so a RustFS restart used to drop the bucket and break uploads untilfilerestarted.upgrade_k3s_containers.sh:generate.shsubshell, so it can't override script variables.--max-timeto the health checkcurls, so a hung request can't hold the deploy lock forever.🤖 Generated with Claude Code