Repository navigation
Site deployment: cleaner wait for initial database replication - #500
Merged
Merged
Conversation
timbeccue
changed the title
timbeccue
marked this pull request as ready for review
October 21, 2026 06:33
timbeccue
changed the title
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The missing-table recovery path and propagation of unrelated database errors need regression coverage.
1 open issue
What changed in this PR
This PR reduces stacking-supervisor crash-loop noise while a new site's database initializes and replicates.
Changes:
- Retry camera discovery every 30 seconds when the table or matching cameras are unavailable.
- Log waiting status immediately, then every 10 minutes.
- Update the supervisor test to verify startup after cameras become available.
| File | Description |
|---|---|
| banzai/tests/test_smart_stacking.py | Tests retrying empty discovery and subsequent worker startup. |
| banzai/stacking.py | Adds startup retries, missing-table handling, and throttled logging. |
Review effort: Balanced
Comment on lines
+188
to
+191
| except ProgrammingError as exc: | ||
| if not isinstance(exc.orig, UndefinedTable): | ||
| raise | ||
| instruments = [] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a fresh site deployment the stacking supervisor starts as soon as Postgres is healthy, before
banzai-cache-inithas created theinstrumentstable or replication has filled it. Its first query then fails, docker restarts it, and the container crash-loops with error logs until the database catches up. This isn't a big problem besides the fact that it pollutes the logs with lots of noise.Fix
In the startup camera discovery lookup, sleep for 30s and retry instead of craching and relying on docker to restart. Log 'info' the first time this happens, and then once every 10 minutes.
Note: this change has been running in the branch deployed to sites for a while with no issues so far.