Conversation
Adding targets already retried when Hetzner answered locked, conflict, robot_unavailable or 5xx, but the other changes a reconcile makes to a balancer failed the whole reconcile on the first such answer. A reconcile that changes the type and then the services is likely to hit the lock the type change just took, and then waits for the next retry of the service. During a reconcile, removing targets, changing the type or the algorithm, attaching or detaching a network, and adding, updating or deleting a service now go through the same short retry. The retry constants lose their add-target names since they now cover all of these calls. Deleting a balancer when its service goes away is left as it was. A target Hetzner reports as already defined now counts as live. That answer comes when an earlier attempt was applied but answered with a 5xx, and without it a balancer with a single target reported that no target could be added although the target was there. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes that a reconcile makes to a busy balancer now retry, not only adding targets. Before this, a
locked,conflict,robot_unavailableor 5xx answer on any other call failed the whole reconcile. A reconcile that changes the type and then the services often hit the lock the type change had just taken.Removing targets, changing the type or the algorithm, attaching or detaching a network, and adding, updating or deleting a service now use the same retry as adding targets. That is two more tries, after 1s and 2s. The constants are renamed to
BUSY_RETRIESandBUSY_RETRY_UNIT, since they are not about targets any more.Adding a target that is already there now counts as success. Hetzner answers
target_already_definedwhen a retry follows a 5xx that it had applied anyway. Before, a balancer with a single target then reported "No target could be added" although the target was there.The other calls have no such special case. If Hetzner applies one of them and still answers 5xx, the retry may get a permanent error, for example a service that already exists. The reconcile then fails the same way it did before this change, and the next run reads the new state. A type change can also hold the lock longer than the three seconds of retries, and then the reconcile still waits for the next run.
Two smaller changes come with it. With the wrappers the services reconcile went over the clippy line limit, so the "service already matches" check moved into
service_is_current. Deleting the balancer incleanup()gets no retry here, because another open PR rewrites that function. #65 tracks it.Tests cover the
target_already_definedclassification only. Removing that arm fromreconcile_targets, or a wrapped call that skips the retry, keeps the suite green. The call sites are async and the repo has no HTTP mock.This adds a one-line conflict with the pipeline port in ci/port-pipeline, in
reconcile_algorithm. Keep the retry wrapper and useSome(self.algorithm.clone()).Stacked on #59.
Closes #60