Skip to content

fix(sdk): wrap all transport errors and stop replaying writes - #295

Merged
loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:fix/sdk-transport-errors-and-write-retries
Oct 10, 2026
Merged

loveRhythm1990 merged 2 commits into
matrixorigin:mainfrom
loveRhythm1990:fix/sdk-transport-errors-and-write-retries

Conversation

@loveRhythm1990

Copy link
Copy Markdown
Collaborator

Summary

Two coupled transport defects in the Python SDK. Grouped because #269's acceptance criteria explicitly depends on the retry decision in #265 ("POST may have committed before disconnection, so wrapping must not introduce unsafe replay") — and both live in the same except clauses.

Leaked exceptions — #269. Both paths caught (ConnectError, TimeoutException, NetworkError). httpx.RemoteProtocolError — "server disconnected without sending a response", a routine production event — descends from ProtocolError, which is a sibling of NetworkError, not a subclass:

TransportError
├── TimeoutException → ConnectTimeout, ReadTimeout, …
├── NetworkError     → ConnectError, ReadError, WriteError, CloseError
├── ProtocolError    → LocalProtocolError, RemoteProtocolError   ← escaped
└── ProxyError, UnsupportedProtocol                              ← escaped

So it escaped the SDK hierarchy and applications handling MemoriaError/MemoriaConnectionError missed the failure. Now the httpx.TransportError base class is caught: every subclass means no response was produced, which is exactly what MemoriaConnectionError describes, and future httpx subclasses can't slip through. The original exception is preserved as __cause__.

Unsafe write replay — #265. POST/PATCH were retried on 502/503/504. No status code proves the upstream skipped the write — a gateway can return 504 while the upstream keeps going and commits, and 502/503 can follow a commit if the upstream dies right after writing — so retrying without an idempotency key risks committing one logical write twice. Non-idempotent retries are now opt-in via retry_unsafe_writes=True (the lighter of the two options the issue proposes; a real idempotency key needs server-side dedup support). ConnectError is still retried for every method, since the request never reached the server, and idempotent methods are unchanged.

Worth noting for severity: memories.store is already protected server-side by near-duplicate detection, which supersedes rather than duplicates identical content — so the practical exposure was on the other write endpoints. That's recorded in the docs.

Verification

Fault injection via httpx.MockTransport, sync and async:

Injected Before After (sync & async)
RemoteProtocolError leaked raw httpx exception MemoriaConnectionError
LocalProtocolError, ProxyError leaked MemoriaConnectionError
ConnectError, ReadTimeout, ReadError wrapped wrapped (unchanged)

Commit counting with a transport that records a commit then returns a gateway error:

Case Before After
POST + 502/503/504 2 commits for one logical write 1 commit, error surfaced
POST + 502/503/504, retry_unsafe_writes=True 2 commits 2 commits (opt-in, documented)
POST + RemoteProtocolError n/a (leaked) 1 request, not replayed
GET + 500/502/503/504 retried, recovered retried, recovered (unchanged)
POST + ConnectError retried retried (never reached the server)

Test plan

  • pytest tests/unit passes — 159 tests (121 baseline + new test_transport_errors.py covering all 8 TransportError branches × GET/POST/async, plus the replay matrix)
  • Two existing tests that asserted POST-retry-on-502 were rewritten to the new policy, with the old behavior still covered under retry_unsafe_writes=True so the opt-in path isn't untested
  • ruff check src tests — no new findings (7 pre-existing, identical on unmodified main, from a newer ruff than the project's >=0.4 floor)

Fixes #265
Fixes #269

🤖 Generated with Claude Code

Two coupled transport defects.

Leaked exceptions (matrixorigin#269): both the sync and async paths caught
(ConnectError, TimeoutException, NetworkError). httpx.RemoteProtocolError
— "server disconnected without sending a response", a routine production
event — descends from ProtocolError, a *sibling* of NetworkError, so it
escaped the SDK exception hierarchy and applications handling
MemoriaError/MemoriaConnectionError missed the failure entirely. The same
held for LocalProtocolError and ProxyError. Catch the httpx.TransportError
base class instead: every subclass means no response was produced, which
is exactly what MemoriaConnectionError describes, and new httpx subclasses
can no longer slip through.

Unsafe write replay (matrixorigin#265): POST/PATCH were retried on 502/503/504. No
status code proves the upstream skipped the write — a gateway can return
504 while the upstream keeps going and commits, and 502/503 can follow a
commit if the upstream dies right after writing — so retrying without an
idempotency key risks committing one logical write twice. Non-idempotent
retries are now opt-in via retry_unsafe_writes=True; the same rule applies
to the newly-caught transport errors, so wrapping them does not introduce
replay. ConnectError is still retried for every method because the request
never reached the server, and idempotent methods are unchanged.

Retry semantics are now documented as a per-method table.

Fixes matrixorigin#265
Fixes matrixorigin#269

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@loveRhythm1990 loveRhythm1990 changed the title fix(sdk/python): wrap all transport errors and stop replaying writes fix(sdk): wrap all transport errors and stop replaying writes Oct 10, 2026
@loveRhythm1990

Copy link
Copy Markdown
Collaborator Author

Reviewed head 9461a808a6e9721437b38a3704c1cf5143a96790.

One medium-severity finding remains:

[P2] Treat memory correction as an unsafe operation even though its HTTP method is PUT. The retry eligibility helpers classify every PUT as idempotent, so retry_unsafe_writes=False does not protect memories.correct(), which sends PUT /v1/memories/{id}/correct. The service creates a new UUID-backed replacement before superseding the old memory; replaying this operation is not safely idempotent. If the first request committed, a retry can instead report that the old memory is missing; if requests overlap before superseding, they can create multiple replacements.

I reproduced two requests with the default options in both sync and async clients, for both a first-response 504 and a first-attempt httpx.RemoteProtocolError, followed by success. In particular, retrying RemoteProtocolError expands the unsafe behavior in this PR: it previously escaped without replay, but the new ProtocolError retry branch now reissues this PUT automatically.

Please make retry eligibility operation-aware (or explicitly mark the correction endpoint unsafe), apply the opt-in policy to ambiguous correction failures, and cover this PUT endpoint with sync/async regression tests for both status and transport errors. The POST/PATCH tests do not exercise it.

Validation: existing Python suite passed, 159/159 tests. Additional httpx.MockTransport checks reproduced the request counts above. The consequences for correction follow from the service's read/create/supersede implementation; database duplication was not experimentally reproduced.

Coverage: 6/6 changed files reviewed, including documentation and tests; 0 skipped (100%).

Follow-up to review on this PR; reproduced before fixing.

Retry eligibility was decided purely by HTTP verb, and PUT is in the
idempotent set, so retry_unsafe_writes=False did not protect
memories.correct() — which sends PUT /v1/memories/{id}/correct. The
server looks the memory up, mints a new UUID-backed replacement and
supersedes the original, so a replay either 404s on the
already-superseded memory or creates a second replacement. Worse, the
ProtocolError retry branch added in this PR *expanded* the exposure:
a disconnect on that PUT previously escaped without replay and would
now have been reissued automatically.

Eligibility is now per operation: _request/_arequest take an optional
`idempotent` override that defaults to the method-based classification,
and the two correct() call sites pass idempotent=False. Verified one
attempt per logical correction for 504 and RemoteProtocolError, sync and
async; retry_unsafe_writes=True still replays; DELETE and other
genuinely idempotent methods keep their behavior.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@loveRhythm1990

Copy link
Copy Markdown
Collaborator Author

lgtm

Re-reviewed 579ad90a8975e383e6fa61a9cb3ad72728f65063. The previous unsafe PUT correction retry finding is resolved: both sync and async correction explicitly pass idempotent=False, and both status-error and transport-error retry decisions honor that override. Opt-in retries and ordinary idempotent operations retain their intended behavior.

Validation: reran the Python unit suite on this head, 167/167 passed, including the added correction gateway-error/disconnect regressions.

Coverage: 7/7 changed files reviewed, including documentation and tests; 0 skipped (100%).

@loveRhythm1990
loveRhythm1990 merged commit a39f364 into matrixorigin:main Oct 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant