[nijeesh-stream]

TLDR

• Problem: the load balancer front end in front of the Stream API (Google's front-end proxy) sometimes drops connections, for example during scheduled proxy restarts. Google documents that clients must cope with that. This SDK has no retry, so customers see errors like OpenSSL::SSL::SSLSocket#sysread_nonblock: SSL_read: unexpected eof while reading (Shape-Construction on 3.26.2).
• What breaks today: when the proxy closes an idle pooled connection, the next call on it fails, whatever the method, even though the request never reached the server. Net::HTTP's built-in retry can't help because faraday-net_http_persistent turns it off (max_retries = 0).
• What this changes: a small Faraday middleware (StreamChat::RetryMiddleware) on the client connection retries up to 2 times (~100ms, then ~200ms, with jitter). It retries a call only when that is safe: any method if the failure provably happened before the request was sent, and GET/HEAD/OPTIONS also if the connection dropped mid-flight. It has no per-endpoint logic and adds no new gem.
• Risks: a POST/PATCH/PUT/DELETE that drops after it was sent is not retried and still raises the same error as today. Successful calls are unaffected. A failed call can take up to ~450ms longer before raising. See Risks below.

Ticket

• https://linear.app/stream/issue/CHA-5724/ssl-read-unexpected-eof-curl-error-56-on-api-calls-since-gcp-edge

CLA

• I have signed the Stream CLA (required).
• The code changes follow best practices
• Code changes are tested (add some information if not applicable)

Description of the pull request

Adds lib/stream-chat/retry_middleware.rb and registers it in Client#initialize after faraday.request :multipart and before the adapter. Because it is middleware, it covers every request on @conn: make_http_request and the upload path (send_file, which calls @conn.post directly).

Design rules

Failure	Where it happened	Methods retried	Why it is safe
Faraday::SSLError / Faraday::ConnectionFailed	a Net::HTTP#begin_transport frame (stale pooled connection) in the cause chain	all	the server received 0 requests
Faraday::SSLError / Faraday::ConnectionFailed	a Net::HTTP#connect frame (refused, DNS, TCP reset or TLS handshake failure) in the cause chain	all	the connection never opened
Faraday::SSLError / Faraday::ConnectionFailed	later (while sending the request or reading the response)	GET, HEAD, OPTIONS only	the server may have the request; repeating these methods is harmless
Faraday::TimeoutError (read and open timeouts)	any	none	never retried
HTTP error response (4xx/5xx)	n/a	none	never retried
IO body (multipart) that does not respond to rewind	any	none	the body can't be sent twice

• At most 2 retries (3 attempts), with a backoff of 100ms·2^(n-1) plus up to 50% jitter.
• Before a retry, an IO body (multipart CompositeReadIO) is rewound. If it can't be rewound, the original error is raised.
• When the middleware doesn't retry, or runs out of retries, it re-raises the original exception, so callers get the same error class as today.

Evidence

Facts this design relies on (local reproductions with the SDK's exact stack: Faraday 2.12+, faraday-net_http_persistent 2.3, net-http-persistent 4.0, idle_timeout = 59):

• faraday-net_http_persistent sets http.max_retries = 0 if http.respond_to?(:max_retries=) in configure_request (faraday-net_http_persistent 2.3.1, lib/faraday/adapter/net_http_persistent.rb). Net::HTTP's idempotent retry is therefore off.
• The raw OpenSSL::SSL::SSLError / EOFError / Errno::ECONNRESET never reaches the caller. The adapter wraps it as Faraday::SSLError (SSL errors) or Faraday::ConnectionFailed (everything else), with the original in #cause.
• Stale pooled connection: the server closed an idle keep-alive connection without a TLS close_notify. The next request fails for every method with Faraday::SSLError (cause SSL_read: unexpected eof while reading). It is raised from Net::HTTP#begin_transport (the @socket.eof? liveness check), and the server receives 0 requests.
• Mid-flight drop: the exception class and message are the same, but the error is raised while the response is being read (transport_request, no begin_transport), and the server did receive the request.
• How the connect-phase failures surface (probed on Ruby 3.4.2). Each one has a Net::HTTP#connect frame in its cause chain:
  • refused: Faraday::ConnectionFailed ← Net::HTTP::Persistent::Error ← Errno::ECONNREFUSED
  • DNS: Faraday::ConnectionFailed ← Socket::ResolutionError
  • bad TLS handshake: Faraday::SSLError ← OpenSSL::SSL::SSLError (SSL_connect ...)
  • reset during the handshake: Faraday::ConnectionFailed ← Errno::ECONNRESET
  • open timeout: Faraday::TimeoutError ← Net::OpenTimeout. This is a TimeoutError, so it is not retried.
• The frame regex matches both backtrace label styles: `connect' (Ruby ≤ 3.3) and 'Net::HTTP#connect' (Ruby ≥ 3.4). The new specs pass on Ruby 3.2.0 (old style) and 3.4.2 (new style).

How a drop surfaces depends on the Ruby/OpenSSL build

The first CI run showed that the same drop surfaces differently on different builds:

	macOS, Ruby 3.4.2 (OpenSSL 3.6)	Ubuntu CI runners
mid-flight drop	Faraday::SSLError (SSL_read: unexpected eof while reading)	Faraday::ConnectionFailed wrapping EOFError: end of file reached (from read_status_line)
stale pooled connection	Faraday::SSLError from begin_transport; the middleware retries it	no error at all: Net::HTTP's eof? check sees a clean EOF and reconnects by itself

The middleware handles both. It catches both Faraday::SSLError and Faraday::ConnectionFailed, decides by the backtrace frame rather than the error class or message, and re-raises whichever error the adapter produced. The specs were updated to match (commit c507368):

• Where a transport error is expected, the specs accept either class and still check how many attempts reached the server.
• The stale-connection specs assert only the outcome that holds on both builds: the call succeeds and the server receives the request exactly once. On a build where Net::HTTP reconnects by itself, these specs no longer exercise the middleware. The not-sent rule is still covered on every build by the connection-refused specs.
• The rewind rule is checked with a refused connection, which fails before anything is sent on every build. A counting middleware after RetryMiddleware shows 1 attempt (and 0 reads) for a body that can't be rewound, and 3 attempts for a rewindable StringIO.

Real-socket specs (spec/retry_middleware_spec.rb)

The specs use a local HTTPS server (spec/support/tls_test_server.rb) with a self-signed certificate. TLS verification is turned off only in the test client. They go through the real StreamChat::Client request path and need no network access or API keys. Output on the branch (Ruby 3.4.2, macOS):

$ bundle exec rspec spec/retry_middleware_spec.rb --require spec_helper
StreamChat::RetryMiddleware
  when the pooled connection went stale
    succeeds with a POST, and the server receives it once
    succeeds with a GET, and the server receives it once
    succeeds with a multipart upload, and the server receives the full body once
  when the request body is an IO
    does not retry a body that cannot be rewound
    retries a body that can be rewound
  when the connection drops after the request was sent
    retries a GET, and the server receives it twice
    does not retry a POST and raises the transport error
    stops after 3 attempts
  when the connection is refused
    retries a POST once the server is listening
  when the server answers or times out
    does not retry an HTTP 500
    does not retry a read timeout
11 examples, 0 failures

• The same 11 examples pass on Ruby 3.2.0.
• On macOS, removing the middleware from Client#initialize made the "retries…", stale-connection and attempt-cap examples fail. That was measured on the first commit, so the specs exercise the middleware there.
• The mid-flight POST is not retried: the server receives exactly one POST, and the caller gets the same transport error it gets today.

Other checks:

• bundle exec rake rubocop reports 29 files inspected, no offenses.
• bundle exec srb tc reports no errors.
• The specs that need no credentials (errors_spec, webhook_compression_spec, retry_middleware_spec) all pass.

Risks

• A failed call can take longer: at most 2 extra attempts and ~450ms of backoff before the original error is raised. Successful calls get no extra latency.
• Repeated GET/HEAD/OPTIONS: after a mid-flight drop, one of these can reach the server up to 3 times. That is harmless for reads.
• Stack-trace detection: whether a request was sent is decided from Net::HTTP backtrace frame names (connect, begin_transport). If a future Net::HTTP renames or reshapes these, the effect is fail-safe: non-GET requests stop being retried. Nothing gets retried that shouldn't. CI's Ruby 3.0/3.1/4.0 matrix runs the new specs. Locally only 3.2.0 and 3.4.2 were run.
• Open timeouts aren't retried: they surface as Faraday::TimeoutError, and the rule is to never retry timeouts. A request that never connected within the timeout therefore still fails on the first attempt.

🤖 Generated with Claude Code

[nijeesh-stream]

Closing without merging: CHA-5724 has been closed (no further customer reports). Keeping this PR and branch as a reference implementation in case the issue comes back.

Context for whoever picks it up:

• Root cause sits in Google's front-end proxy (scheduled proxy_restarts drop connections); our edge closes connections cleanly. Google documents that clients must tolerate dropped connections.
• No Stream server SDK retries by default. The generated SDKs share the CHA-2959 retry policy (off by default, GET/HEAD, transport errors + 429 with Retry-After, 3 attempts). This PR is stricter on safety (retries any method only when the request provably never left the client) but defaults ON, which differs from CHA-2959 — align with that policy before reviving.
• How the drop surfaces varies by build (curl errno 52 vs 56; Ruby SSLError vs ConnectionFailed/EOFError); the tests were adjusted to assert behaviour, not error spelling.

Findings are documented on https://linear.app/stream/issue/CHA-5724