Skip to content
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -4666,7 +4666,7 @@ definitions:
- "([-+]?\\d+)"
max_waiting_time_in_seconds:
title: Max Waiting Time in Seconds
description: Stop the stream instead of waiting, when the value extracted from the header is greater than or equal to this value. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait.
description: 'Stop the stream instead of waiting, when the value extracted from the header is greater than or equal to this value. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.'
anyOf:
- type: number
- type: string
Expand Down Expand Up @@ -4802,7 +4802,7 @@ definitions:
- "([-+]?\\d+)"
max_waiting_time_in_seconds:
title: Max Waiting Time in Seconds
description: Stop the stream instead of waiting, when the wait this strategy computes is longer than this value. The comparison is against the computed wait rather than the raw header, since the header holds an absolute timestamp, and it is applied after `min_wait`, so a cap below the floor still wins -- including for the fallback where the header is absent and `min_wait` supplies the wait on its own. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait.
description: 'Stop the stream instead of waiting, when the wait this strategy computes is greater than or equal to this value. The comparison is against the computed wait rather than the raw header, since the header holds an absolute timestamp, and it is applied after `min_wait`, so a cap below the floor still wins -- including for the fallback where the header is absent and `min_wait` supplies the wait on its own. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.'
anyOf:
- type: number
- type: string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1449,7 +1449,7 @@ class WaitTimeFromHeader(BaseModel):
)
max_waiting_time_in_seconds: Optional[Union[float, str]] = Field(
None,
description="Stop the stream instead of waiting, when the value extracted from the header is greater than or equal to this value. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait.",
description="Stop the stream instead of waiting, when the value extracted from the header is greater than or equal to this value. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.",
examples=[3600, "{{ config['max_waiting_time'] * 60 }}"],
title="Max Waiting Time in Seconds",
)
Expand Down Expand Up @@ -1478,7 +1478,7 @@ class WaitUntilTimeFromHeader(BaseModel):
)
max_waiting_time_in_seconds: Optional[Union[float, str]] = Field(
None,
description="Stop the stream instead of waiting, when the wait this strategy computes is longer than this value. The comparison is against the computed wait rather than the raw header, since the header holds an absolute timestamp, and it is applied after `min_wait`, so a cap below the floor still wins -- including for the fallback where the header is absent and `min_wait` supplies the wait on its own. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait.",
description="Stop the stream instead of waiting, when the wait this strategy computes is greater than or equal to this value. The comparison is against the computed wait rather than the raw header, since the header holds an absolute timestamp, and it is applied after `min_wait`, so a cap below the floor still wins -- including for the fallback where the header is absent and `min_wait` supplies the wait on its own. Can be a hardcoded number, or a string interpolated from the connector config so that the bound can be changed without a connector release. A value of 0 means never wait. Not evaluated when a rate-limited retry can rotate to another credential with quota; any other retryable error still consults this strategy.",
examples=[3600, "{{ config['max_waiting_time'] * 60 }}"],
title="Max Waiting Time in Seconds",
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,13 @@ def evaluate_max_waiting_time(
check would silently disable it. Only an absent field means "no cap"; a field that is present
but resolves to nothing raises, rather than quietly leaving the wait unbounded.

A cap is only read while handling an error the requester is already going to retry, so an
interpolation that cannot be resolved would otherwise surface as an unhandled jinja
UndefinedError or ValueError in the middle of a sync that has been running fine. It is raised
as a system error rather than a config error because the field lives in the manifest: whether
it is a bad expression or a config key the manifest reads but the spec does not expose, the
connector is at fault and there is nothing for the user to correct.
Called once when the strategy is constructed, and again on each cap check. The eager call is
what makes an unresolvable interpolation a startup failure rather than something discovered at
whichever retryable error happens to reach a strategy -- a distinction that matters because
`HttpClient` skips the strategies entirely when a rate-limited retry can rotate credentials.
It is raised as a system error rather than a config error because the field lives in the
manifest: whether it is a bad expression or a config key the manifest reads but the spec does
not expose, the connector is at fault and there is nothing for the user to correct.

:param max_waiting_time_in_seconds: the interpolated field, or None when no cap is configured
:param config: the connector config to interpolate against
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,11 @@ class WaitTimeFromHeaderBackoffStrategy(BackoffStrategy):
header (str): header to read wait time from
regex (Optional[str]): optional regex to apply on the header to extract its value
max_waiting_time_in_seconds (Optional[Union[float, InterpolatedString, str]]): stop the stream
rather than wait longer than this
rather than wait this long or longer -- the bound is inclusive, so a wait exactly
equal to it is refused. Only governs waits that are actually taken: on a
rate-limited response where the authenticator holds another credential with quota,
`HttpClient` rotates onto it instead of asking this strategy for a wait, and the
bound does not apply. Any other retryable error still consults this strategy.
"""

header: Union[InterpolatedString, str]
Expand All @@ -50,6 +54,12 @@ def __post_init__(self, parameters: Mapping[str, Any]) -> None:
self._max_waiting_time_in_seconds = interpolated_max_waiting_time(
self.max_waiting_time_in_seconds, parameters
)
# Resolved here rather than only at the first retryable error. `config` is a field and this
# cap interpolates over `config` alone, so it is fully knowable the moment the component
# exists -- and since `HttpClient` decides token rotation before it asks a strategy for a
# wait, a cap that cannot be evaluated would otherwise stay silent for as long as a spare
# credential keeps the strategies from running. A manifest mistake belongs at startup.
evaluate_max_waiting_time(self._max_waiting_time_in_seconds, self.config)

def backoff_time(
self,
Expand All @@ -68,10 +78,14 @@ def backoff_time(
max_waiting_time = evaluate_max_waiting_time(
self._max_waiting_time_in_seconds, self.config
)
# Not always reached: `HttpClient` decides token rotation before it asks a strategy
# for a wait, so on a rate limit where the authenticator has another credential with
# quota this check does not run. The cap bounds waiting, and that path is not
# waiting.
# `max_waiting_time is not None` rather than a truthiness check, so that 0 means
# "never wait" instead of silently disabling the cap. The comparison stays `>=`,
# which is what this cap has always done; `WaitUntilTimeFromHeader` stops at `>`,
# so a wait exactly equal to the cap is allowed there and refused here.
# which is what this cap has always done, and `WaitUntilTimeFromHeader` matches it --
# a wait exactly equal to the cap is refused by both.
# `header_value` is checked for truthiness rather than `is not None` on purpose: a
# header of `0` asks for no wait at all, which no cap -- not even 0 -- should refuse.
if max_waiting_time is not None and header_value and header_value >= max_waiting_time:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,11 @@ class WaitUntilTimeFromHeaderBackoffStrategy(BackoffStrategy):
min_wait (Optional[Union[float, InterpolatedString, str]]): minimum time to wait for safety
regex (Optional[str]): optional regex to apply on the header to extract its value
max_waiting_time_in_seconds (Optional[Union[float, InterpolatedString, str]]): stop the stream
rather than wait longer than this
rather than wait this long or longer -- the bound is inclusive, so a wait exactly
equal to it is refused. Only governs waits that are actually taken: on a
rate-limited response where the authenticator holds another credential with quota,
`HttpClient` rotates onto it instead of asking this strategy for a wait, and the
bound does not apply. Any other retryable error still consults this strategy.
"""

header: Union[InterpolatedString, str]
Expand All @@ -56,6 +60,12 @@ def __post_init__(self, parameters: Mapping[str, Any]) -> None:
self._max_waiting_time_in_seconds = interpolated_max_waiting_time(
self.max_waiting_time_in_seconds, parameters
)
# Resolved here rather than only at the first retryable error. `config` is a field and this
# cap interpolates over `config` alone, so it is fully knowable the moment the component
# exists -- and since `HttpClient` decides token rotation before it asks a strategy for a
# wait, a cap that cannot be evaluated would otherwise stay silent for as long as a spare
# credential keeps the strategies from running. A manifest mistake belongs at startup.
evaluate_max_waiting_time(self._max_waiting_time_in_seconds, self.config)

def backoff_time(
self,
Expand Down Expand Up @@ -84,13 +94,18 @@ def backoff_time(
return self._capped(wait_time)

def _capped(self, wait_time: float) -> float:
"""Raise rather than wait longer than `max_waiting_time_in_seconds`.
"""Raise rather than wait `max_waiting_time_in_seconds` or longer.

The cap is compared against the wait this strategy is about to return, not against the
raw header: unlike `Retry-After`, the header here is an absolute timestamp, so only the
computed difference is a duration. It is also applied after the `min_wait` floor, so a
cap below the floor wins -- a caller asking never to wait more than N seconds means it,
even when the floor would otherwise round the wait up past N.

Not always reached: `HttpClient` decides token rotation before it asks a strategy for a
wait, so on a rate limit where the authenticator has another credential with quota this
method does not run and the cap does not apply. Waiting is what the cap bounds, and that
path is not waiting.
"""
max_waiting_time = evaluate_max_waiting_time(self._max_waiting_time_in_seconds, self.config)
# `>=` rather than `>` to match WaitTimeFromHeader, so one field name does not mean two
Expand All @@ -99,8 +114,8 @@ def _capped(self, wait_time: float) -> float:
if max_waiting_time is not None and wait_time >= max_waiting_time:
raise AirbyteTracedException(
internal_message=(
f"Rate limit wait time {wait_time}s is greater than the maximum of "
f"{max_waiting_time}s this stream is allowed to wait. Stopping the stream..."
f"Rate limit wait time {wait_time}s is greater than or equal to the maximum "
f"of {max_waiting_time}s this stream is allowed to wait. Stopping the stream..."
),
message="The rate limit wait time is longer than the connector is allowed to wait.",
failure_type=FailureType.transient_error,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@ def backoff_time(
"""
Override this method to dynamically determine backoff time e.g: by reading the X-Retry-After header.

This method is called only if should_backoff() returns True for the input request.
Not called for every retryable response. `HttpClient` skips the strategies entirely when a
rate-limited response can be retried on another credential -- the authenticator says so via
`TokenRotatingAuthenticator.has_alternative_token` -- because the wait computed here is
derived from the credential that was rejected, and the retry will not use it. Implementations
must therefore not rely on being called for side effects such as counting attempts or
emitting metrics.

:param response_or_exception: The response or exception that caused the backoff.
:param attempt_count: The number of attempts already performed for this request.
Expand Down
57 changes: 38 additions & 19 deletions airbyte_cdk/sources/streams/http/http_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -588,29 +588,48 @@ def _handle_error_resolution(
ResponseAction.REFRESH_TOKEN_THEN_RETRY,
):
user_defined_backoff_time = None
for backoff_strategy in self._backoff_strategies:
backoff_time = backoff_strategy.backoff_time(
response_or_exception=response if response is not None else exc,
attempt_count=self._request_attempt_count[request],
)
if backoff_time:
user_defined_backoff_time = backoff_time
break

if (
user_defined_backoff_time
and error_resolution.response_action == ResponseAction.RATE_LIMITED
# Asked before the strategies, not after. The backoff they compute describes the
# credential the server just rejected, so when another credential can serve the
# retry that wait is irrelevant -- and a strategy is allowed to refuse a wait by
# raising (`max_waiting_time_in_seconds`), which would otherwise end the stream
# before rotation was ever considered. Rotating is strictly the better outcome
# there: it is the same retry, seconds from now, on a credential with quota.
#
# Two consequences of not calling the strategies, both deliberate. A rate limit
# that yields no backoff at all now rotates too, rather than falling through to
# the default exponential retry -- on a rotating credential that is the better
# behaviour, and `has_alternative_token` only answers True when the retry will
# rotate -- the CDK's own authenticator narrows that further, to a sending credential
# that is tracked and spent. And a `max_waiting_time_in_seconds` the manifest got
# wrong -- one that cannot be evaluated -- is not reported from here, since that
# error is raised from inside the strategy. Both capped strategies therefore resolve
# the field once in `__post_init__` too, so a manifest mistake fails at startup
# rather than waiting for a rate limit that finds no spare credential.
rotate_instead_of_waiting = (
error_resolution.response_action == ResponseAction.RATE_LIMITED
and self._can_retry_on_another_token(request)
):
# The backoff was derived from this response's headers, which only describe the
# credential that was rejected. Another one has quota, and the retry re-signs the
# request, so waiting out this window would idle for nothing.
)

if rotate_instead_of_waiting:
# Says that a wait was skipped without the number, which is no longer computed,
# and names the cap explicitly: a connector that configured one gets no other
# signal that the retry went ahead without consulting it.
self._logger.info(
f"Rate limited on the current credential; retrying in "
f"{self.TOKEN_ROTATION_BACKOFF}s with another one instead of waiting "
f"{user_defined_backoff_time:.0f}s for the rate limit to reset."
"Rate limited on the current credential; retrying in "
f"{self.TOKEN_ROTATION_BACKOFF}s with another one instead of waiting for the "
"rate limit to reset. Any configured backoff, including a wait cap, is not "
"evaluated for this retry."
)
user_defined_backoff_time = self.TOKEN_ROTATION_BACKOFF
else:
for backoff_strategy in self._backoff_strategies:
backoff_time = backoff_strategy.backoff_time(
response_or_exception=response if response is not None else exc,
attempt_count=self._request_attempt_count[request],
)
if backoff_time:
user_defined_backoff_time = backoff_time
break

error_message = (
error_resolution.error_message
Expand Down
Loading
Loading