Skip to content

Fix fragile retry loop - #4

Open
felipecrv wants to merge 1 commit into
mainfrom
retries
Open

Fix fragile retry loop#4
felipecrv wants to merge 1 commit into
mainfrom
retries

Conversation

@felipecrv

Copy link
Copy Markdown
  1. RoundTripper contract violation (internal/client/client.go, errorHandler). On retry exhaustion it returned (resp, werr) with both non-nil. retryablehttp hands that straight to Go's net/http, which at net/http/client.go:262 logs "RoundTripper returned a response & error; ignoring response", discards the response, and leaks the connection — later surfacing as connection is already closed.
  2. Fragile poll loop (connection.go, pollOperation). Long queries are driven by a 1s poll of GetOperationStatus. A single failed poll (idle timeout, reset, 5xx, EOF) made the sentinel abort the whole query — even though the operation handle is still valid server-side and the query keeps running. This is what "loses the statement handle."

Changes made:

  • errorHandler now drains + closes the body and returns (nil, werr) — honoring the contract, matching retryablehttp's own default, and preserving the enriched error (reason-phrase headers).
  • pollOperation now swallows transient GetOperationStatus errors and keeps polling, bounded by a new maxPollFailureWindow = 5 * time.Minute (so a genuinely-dead warehouse still terminates), and aborts immediately on context cancellation/deadline.
  • Added tests: TestErrorHandler (nil response + body closed) and a TestConn_pollOperation subtest (recovers from a transient error instead of aborting).

Test verification

  • TestErrorHandler, TestRetryPolicy, and all TestConn_pollOperation subtests pass (including the new resilience test).

1. RoundTripper contract violation (internal/client/client.go, errorHandler). On
  retry exhaustion it returned (resp, werr) with both non-nil. retryablehttp
  hands that straight to Go's net/http, which at net/http/client.go:262 logs
  "RoundTripper returned a response & error; ignoring response", discards the
  response, and leaks the connection — later surfacing as connection is already
  closed.
2. Fragile poll loop (connection.go, pollOperation). Long queries are driven by
  a 1s poll of GetOperationStatus. A single failed poll (idle timeout, reset, 5xx,
  EOF) made the sentinel abort the whole query — even though the operation handle
  is still valid server-side and the query keeps running. This is what "loses the
  statement handle."

Changes made:

- errorHandler now drains + closes the body and returns (nil, werr) — honoring
  the contract, matching retryablehttp's own default, and preserving the
  enriched error (reason-phrase headers).
- pollOperation now swallows transient GetOperationStatus errors and keeps
  polling, bounded by a new maxPollFailureWindow = 5 * time.Minute (so a
  genuinely-dead warehouse still terminates), and aborts immediately on
  context cancellation/deadline.
- Added tests: TestErrorHandler (nil response + body closed) and a
  TestConn_pollOperation subtest (recovers from a transient error instead of
  aborting).

Test verification

- TestErrorHandler, TestRetryPolicy, and all TestConn_pollOperation subtests
  pass (including the new resilience test).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant