fix: add exponential recovery backoff#390
Open
nanookclaw wants to merge 1 commit into
Open
Conversation
Comment on lines
+150
to
+168
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn defaults_use_exponential_retry_strategy_with_cap() { | ||
| let options = ArrowStreamConfigurationOptions::default(); | ||
|
|
||
| assert_eq!( | ||
| options.retry_strategy, | ||
| RetryStrategy::ExponentialBackoffWithJitter | ||
| ); | ||
| assert_eq!( | ||
| options.max_recovery_backoff_ms, | ||
| defaults::MAX_RECOVERY_BACKOFF_MS | ||
| ); | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
I'm not sure we need these types of tests, we don't have them for the rest of the configs.
Comment on lines
+195
to
+213
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn defaults_use_exponential_retry_strategy_with_cap() { | ||
| let options = StreamConfigurationOptions::default(); | ||
|
|
||
| assert_eq!( | ||
| options.retry_strategy, | ||
| RetryStrategy::ExponentialBackoffWithJitter | ||
| ); | ||
| assert_eq!( | ||
| options.max_recovery_backoff_ms, | ||
| defaults::MAX_RECOVERY_BACKOFF_MS | ||
| ); | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
Same as in arrow_configuration file.
Comment on lines
+12
to
+18
| pub enum RetryStrategy { | ||
| /// Retry after the configured `recovery_backoff_ms` interval. | ||
| Fixed, | ||
| /// Retry with exponential backoff and jitter, starting from `recovery_backoff_ms`. | ||
| #[default] | ||
| ExponentialBackoffWithJitter, | ||
| } |
Collaborator
There was a problem hiding this comment.
I think fixed should be kept as a default one until we add support for this in all SDKs because if we upgrade Rust version in other SDKs before exposing retry_strategy/max_recovery_backoff_ms in them, there would be no way for users to opt back to Fixed and it would be a silent behavioral change.
Otherwise we would have to commit to add support for this when we upgrade other SDKs. What do you think @teodordelibasic-db?
Comment on lines
+76
to
+123
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn retry_strategy_defaults_to_exponential_with_jitter() { | ||
| assert_eq!( | ||
| RetryStrategy::default(), | ||
| RetryStrategy::ExponentialBackoffWithJitter | ||
| ); | ||
| assert_eq!(defaults::MAX_RECOVERY_BACKOFF_MS, 30_000); | ||
| } | ||
|
|
||
| #[test] | ||
| fn fixed_retry_strategy_uses_configured_interval() { | ||
| let mut strategy = recovery_retry_strategy(RetryStrategy::Fixed, 123, 30_000); | ||
|
|
||
| assert_eq!(strategy.next(), Some(Duration::from_millis(123))); | ||
| assert_eq!(strategy.next(), Some(Duration::from_millis(123))); | ||
| } | ||
|
|
||
| #[test] | ||
| fn exponential_retry_strategy_starts_from_initial_delay_and_caps() { | ||
| assert_eq!( | ||
| capped_exponential_delay(Duration::from_millis(2), 100, 1_000), | ||
| Duration::from_millis(100) | ||
| ); | ||
| assert_eq!( | ||
| capped_exponential_delay(Duration::from_millis(4), 100, 1_000), | ||
| Duration::from_millis(200) | ||
| ); | ||
| assert_eq!( | ||
| capped_exponential_delay(Duration::from_millis(16), 100, 250), | ||
| Duration::from_millis(250) | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn exponential_retry_strategy_applies_jitter_under_cap() { | ||
| let mut strategy = | ||
| recovery_retry_strategy(RetryStrategy::ExponentialBackoffWithJitter, 100, 250); | ||
|
|
||
| for _ in 0..10 { | ||
| assert!(strategy.next().unwrap() <= Duration::from_millis(250)); | ||
| } | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
I would change these tests for tests against the mock server (in rust/test), for consistency.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #192.
This changes Rust stream creation and recovery retries from a fixed interval default to exponential backoff with full jitter.
recovery_backoff_msremains the fixed interval whenRetryStrategy::Fixedis selected, and becomes the base delay forRetryStrategy::ExponentialBackoffWithJitter;max_recovery_backoff_mscaps the computed exponential delay before jitter.The same retry helper is used by gRPC stream creation, Arrow Flight stream creation, and Arrow supervisor recovery sleeps so the behavior stays consistent across both stream implementations. The default strategy is now exponential backoff with jitter, so wrapper SDKs that rely on the Rust core inherit the safer retry behavior without new dependencies.
Verification
cargo fmt --manifest-path rust/sdk/Cargo.tomlcargo check -p databricks-zerobus-ingest-sdkcargo check -p databricks-zerobus-ingest-sdk --features arrow-flightcargo test -p databricks-zerobus-ingest-sdk --libcargo test -p databricks-zerobus-ingest-sdk --features arrow-flight --libgit diff --check