Activity

1. patrykks

   changed the title [-]Default retry logic does not retry HTTP 503 errors on `BigQuery.getTable()`[/-] [+]Default retry logic does not retry HTTP 503 errors from BigQuery API on `BigQuery.getTable()`[/+] on Apr 21, 2026
2. lqiu96

   added

   type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

   priority: p2Moderately-important priority. Fix may not be included in next release.Moderately-important priority. Fix may not be included in next release.

   on Apr 28, 2026
3. lqiu96

   self-assigned this

   on Apr 28, 2026
4. lqiu96 commented on Apr 29, 2026

   

   lqiu96

   on Apr 29, 2026

   Member

   More actions

   BigQuery's public documentation describes 5xx as transient and retryable.

   Can you link me to the public documentation about this? I see references to https://cloud.google.com/bigquery/troubleshooting-errors but that leads me to a 404. From a quick search, I think the new guide would be here: https://docs.cloud.google.com/bigquery/docs/error-messages which I can update references.

   5xx errors are service-side issues and the client has no way to fix or control them. From the client side, in order to mitigate the impact of 5xx errors, you need to retry your requests using [truncated exponential backoffs](https://cloud.google.com/monitoring/api/troubleshooting?_gl=1*1i4wlad*_ga*MTMwNzg3MTcwNC4xNzc3MDQzMTcz*_ga_WH2QY8WWF5*czE3Nzc0OTA1ODIkbzUkZzEkdDE3Nzc0OTA2MTEkajMxJGwwJGgw#exponential-retry).
   

   I agree that retrying on the 5xx errors would a better UX. As of right now, the public doc suggests that the behavior is WAI as the expectation is that users will need to configure retries. I'll talk to the internal team about any concerns with client-side retries enabled by default.
5. patrykks commented on May 1, 2026

   

   patrykks

   on May 1, 2026

   Last edited by patrykks

   Author

   More actions

   Thanks for the response and for taking a look at this I really appreciate the attention to the issue.

   I’d like to clarify what I meant regarding the documentation, as I think we’re referring to two slightly different layers:

   1. BigQuery API

   I fully agree here. If someone is using the API directly, then implementing retries is the client’s responsibility. I was referring to the same documentation you mentioned:

   “5xx errors are service-side issues (…) you need to retry your requests using truncated exponential backoffs.”

   This is clear and makes perfect sense at the API level.

   2. Java SDK (client library)

   My concern is specifically about how this is handled in the SDK layer, which already introduces its own retry abstraction.

   In the Java client library:

   • there is a built-in retry mechanism
   • users typically rely on this instead of implementing retries themselves
   • the SDK effectively defines a higher-level contract over the raw API behavior

   Given that, there are a few points that make the current behavior surprising:

   • 5xx errors are explicitly documented as transient and retryable at the API level
   • these errors are not actionable by the client, so retrying is the only mitigation strategy
   • the SDK already includes retry logic for other transient failures

   Additionally, from the library’s internal contract:

   • the exception thrown is BigQueryException
   • this class exposes a set of RETRYABLE_ERRORS
   • and HTTP 503 is included in that set, which strongly suggests it is expected to be retried

   From a user perspective, this creates an inconsistency:

   • the API says “retry this”
   • the SDK has a retry mechanism and classifies 503 as retryable
   • but the retry is not actually performed in this case

   This effectively forces users to:

   either reimplement retry logic on top of the SDK
   or wrap calls defensively, which partially defeats the purpose of having a client library abstraction

   I don’t dispute that this is “working as intended” at the API level. However, at the SDK level, this behavior appears inconsistent with:

   • the existing retry infrastructure
   • and the expectations set by BigQueryException

   For that reason, treating this as a bug still seems justified from the client library perspective.

   Happy to hear more context if there are constraints or design decisions that explain why retries are not applied here
6. patrykks commented on May 2, 2026

   

   patrykks

   on May 2, 2026

   Last edited by patrykks

   Author

   More actions

   My previous explanation might not have been precise enough, so I’d like to add more context on why I expected the SDK to retry these errors.

   Additional context from SDK behavior

   When using the client like this:

   Table table = bigquery.getTable(TableId.of("test_dataset", "test_table"));

   the library internally calls:

   BigQueryRetryHelper.runWithRetries(
       new Callable<com.google.api.services.bigquery.model.Table>() {
         @Override
         public com.google.api.services.bigquery.model.Table call() throws IOException {
           return bigQueryRpc.getTableSkipExceptionTranslation(
               completeTableId.getProject(),
               completeTableId.getDataset(),
               completeTableId.getTable(),
               optionsMap);
         }
       },
       getOptions().getRetrySettings(),
       getOptions().getResultRetryAlgorithm(),
       getOptions().getClock(),
       EMPTY_RETRY_CONFIG,
       getOptions().isOpenTelemetryTracingEnabled(),
       getOptions().getOpenTelemetryTracer());

   This uses:

   getOptions().getResultRetryAlgorithm()

   My understanding is that if the user does not explicitly configure a resultRetryAlgorithm, the default is used.

   From BigQueryOptions:

   if (builder.resultRetryAlgorithm != null) {
     this.resultRetryAlgorithm = builder.resultRetryAlgorithm;
   } else {
     this.resultRetryAlgorithm = BigQueryBaseService.DEFAULT_BIGQUERY_EXCEPTION_HANDLER;
   }

   So effectively, the default is:

   DEFAULT_BIGQUERY_EXCEPTION_HANDLER

   which is defined as:

   public static final ExceptionHandler DEFAULT_BIGQUERY_EXCEPTION_HANDLER =
       ExceptionHandler.newBuilder()
           .abortOn(RuntimeException.class)
           .retryOn(java.net.ConnectException.class)
           .retryOn(java.net.UnknownHostException.class)
           .retryOn(java.net.SocketException.class)
           .addInterceptors(EXCEPTION_HANDLER_INTERCEPTOR)
           .build();

   The key part here is the EXCEPTION_HANDLER_INTERCEPTOR, which behaves as follows:

   @Override
   public RetryResult beforeEval(Exception exception) {
     if (exception instanceof BaseServiceException) {
       boolean retriable = ((BaseServiceException) exception).isRetryable();
       return retriable
           ? Interceptor.RetryResult.RETRY
           : Interceptor.RetryResult.CONTINUE_EVALUATION;
     }
     return Interceptor.RetryResult.CONTINUE_EVALUATION;
   }

   Observed behavior

   From the user perspective, the error returned is:

   BigQueryException: Visibility check was unavailable. Please retry the request
   

   This corresponds to an HTTP 503, which:

   • is marked as retryable at the API level
   • and (via BaseServiceException.isRetryable()) should also be considered retryable by the interceptor

   Where the inconsistency seems to come from

   The reason it is not retried appears to be the use of:

   getTableSkipExceptionTranslation

   The retry wrapper (runWithRetries) operates on methods that do not yet translate exceptions into BigQueryException.

   Only after the retry block completes, the exception is translated and surfaced to the user as BigQueryException.

   As a result:

   • the retry logic operates on lower-level exceptions
   • but the retryability decision (via isRetryable) is effectively tied to BaseServiceException / BigQueryException
   • and in this flow, that signal is not consistently applied

   Why this feels problematic from a user perspective

   From the outside, this creates a mismatch:

   • the API says: retry 5xx
   • the SDK exposes retry infrastructure and marks 503 as retryable
   • but in practice, the retry does not happen

   This makes it difficult to reason about expected behavior and forces users to:

   • implement their own retry logic on top of the SDK, or
   • defensively wrap calls

   which somewhat undermines the value of having a client library abstraction.

   ---

   It’s possible I’m missing some internal design constraint here, but from my perspective this does not fully align with “working as intended” at the SDK level.

   I’d really appreciate any clarification on whether:

   • this behavior is intentional due to specific design decisions, or
   • it’s a gap between the retry mechanism and exception translation that could be improved

   Thanks again for taking the time to look into this. Happy to dig deeper if needed.
7. lqiu96 commented on May 4, 2026

   

   lqiu96

   on May 4, 2026

   Member

   More actions

   I think your confusion is absolutely reasonable. From what I've been gathering, the retries configured in Bigquery are a bit confusing and not in an ideal state.

   I'm still probing around and I think the changes that I'm leaning towards (this is also what I'm seeing in other languages) is that a lot of the query operations should retry on 5xx errors. I'm going to leave the stuff with jobs as-is.

   I think we'll slowly make these changes to try and improve this behavior. I won't be able to promise a timeline on this, but we are absolutely happy to review any contributions and PRs
8. goutamadwant commented on Jun 6, 2026

   

   goutamadwant

   on Jun 6, 2026

   More actions

   @lqiu96 i was able to reproduce this with a focused regression test around BigQuery.getTable()

   The test throws the same BigQuery JSON HTTP 503 shape from getTableSkipExceptionTranslation.. on current main it does not retry because the retry helper sees HttpResponseException before it is translated to BigQueryException so the existing retry check for 503 is not reached

   I opened a small PR for getTable. It translates HttpResponseException inside the retry callable so the existing BigQueryException retry rules can handle transient server errors. Kept it scoped to getTable and did not touch job retry behavior or raw network IOExceptions

   I did not use a live BigQuery 503 for this since that is not something we can trigger deterministically. Pls review to see if this makes sense
9. lqiu96 commented on Jun 26, 2026

   

   lqiu96

   on Jun 26, 2026

   Member

   More actions

   Should be resolved via #13416
10. lqiu96

    closed this as completedon Jun 26, 2026