Repository navigation
Apply k8 manifests on autoupdate so config changes reach prod - #338
Merged
Merged
Conversation
upgrade_k3s_containers.sh only pulled images and restarted deployments, so manifest changes never reached the cluster. The RustFS migration (#308) and the /file-uploads ingress route (#334) were never applied in production; file-service kept pointing at MinIO, whose Docker Hub image no longer exists, and crash-looped after the Oct 8 reboot. - Regenerate and apply k8/ on every deploy using /etc/try-playwright/env - Delete resources removed from k8/ via RETIRED_RESOURCES (MinIO for now) - Pull and restart log-aggregator as well - Add a readiness probe to file-service so a pod without storage is not ready - Check object storage after deploy, not only control health - Wait with rollout status instead of `kubectl wait -l`, which fails on pods that are still terminating Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Test Results SummaryThis comment shows the latest test results for this pull request.
|
mxschmitt
added a commit
that referenced
this pull request
Oct 26, 2026
* Harden autoupdate deploy and recover the RustFS bucket Follow-ups to #338: - file-service health checks the bucket and recreates it when missing (RustFS uses an emptyDir, so a RustFS restart drops the bucket) - readiness probe timeout raised to fit the storage check - autoupdate: generate manifests before pulling images, source the env file only in that subshell, delete workers before applying manifests, and time-box the health check requests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: keep worker deletion after apply, retry expiry rule - Revert deleting workers before apply: a failing apply or rustfs wait would leave control with only deleted workers - Retry the bucket lifecycle rule until it is set - Don't exit on startup when storage is not reachable yet; stay up and let the readiness probe report it - Keep both errors when bucket creation and the existence check fail Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
upgrade_k3s_containers.shonly pulled images and ranrollout restart. It never appliedk8/, so production stayed on manifests from Jan 30:filestill usedS3_ENDPOINT=minio:9000./file-uploadsingress route (Fix screenshot images failing with RustFS presign host mismatch #334) andlog-aggregatorwere missing.minio/miniocould not be pulled anymore (the repository is gone from Docker Hub), sominiowas stuck inImagePullBackOffandfilecrash-looped. File artifacts were broken for 17 days.filehad no readiness probe.Changes
k8/generate.sh) andkubectl apply -f k8/on every deploy, using secrets from/etc/try-playwright/env. The deploy fails loudly if that file is missing.RETIRED_RESOURCES: resources removed fromk8/get deleted, sinceapplynever deletes anything. Starts withdeployment/minioandservice/minio.log-aggregatortoo.file(/api/v1/health, served only after the bucket check succeeds).HEADon a missing object must return 404 (a storage outage returns 5xx).rollout statusinstead ofkubectl wait -l …, which failed withNotFoundon pods that were still terminating after the restart.Production
/file-uploads/works.--continue) against the cluster with a temporary state dir. It finisheddeployed test successfully.🤖 Generated with Claude Code