Fix memory consumption on empty/small messages - #13393
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #13393 +/- ##
==========================================
+ Coverage 99.01% 99.02% +0.01%
==========================================
Files 135 135
Lines 50048 50445 +397
Branches 2614 2645 +31
==========================================
+ Hits 49553 49952 +399
+ Misses 371 370 -1
+ Partials 124 123 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
|
@aiolibsbot review |
Merging this PR will not alter performance
Comparing Footnotes
|
Reviewed. No blockers. Verified locally, not just by reading. Cythonized cleanly. Built Also probed three failure modes. Protocol error in stashed tail still surfaces correctly, after delivering queued messages. Partial-frame tail leaves no stall. Suspected quadratic re-slice on drain: measured 0.40s vs 0.34s on master for 43,690 frames per read. Not a concern. Five nits. Two worth a look: new queue↔reader cycle needs breaking (pair survives |
|
|
|
@aiolibsbot review |
|
|
Confidence Score: 5/5The PR appears safe to merge because no blocking failure remains. No blocking failure remains. Reviews (26): Last reviewed commit: "Update reader_py.py" | Re-trigger Greptile |
|
@aiolibsbot gh_request Do a scope analysis on the everything that holds a |
Reference-scope analysis: who holds the reader, and where data can go missingScope: every reference to Who holds the
|
| Holder | Strength | Set in | Released in |
|---|---|---|---|
RequestHandler._payload_parser |
strong | set_parser() from WebSocketResponse._post_start |
connection_lost(), after feed_eof(); data_received() → close() on parser eof |
ResponseHandler._payload_parser |
strong | set_parser() from ClientSession._ws_connect |
connection_lost(), after feed_eof(); data_received() on parser eof |
WebSocketResponse._parser (new) |
strong | _post_start() |
never |
ClientWebSocketResponse._parser (new) |
strong | _ws_connect() |
never |
WebSocketDataQueue._stalled_reader |
weak | _feed_data() when over the mark |
top of _feed_data(), set_exception() |
Both protocols drop their reference at connection loss, so from that moment the two new
_parser assignments are the only strong reference to a reader that may still hold a
stash. They are load-bearing, and they were the last hole: the reader is constructed in
exactly two places in-tree, both now owned.
Who holds the WebSocketDataQueue
WebSocketReader.queue, WebSocketResponse._reader / ClientWebSocketResponse._reader
(never cleared), and ResponseHandler._payload (cleared in close()/abort()/
connection_lost(), harmless because the response owns it). No weak references — the
queue cannot vanish under a live response. queue._protocol is strong, and
protocol._data_received_cb is a bound method of the response, so the response outlives
the protocol's own teardown.
Finding 1 — the ownership was untested
Reverting either _parser assignment loses 3903 of 8000 already-received frames, with
no exception and a normal CLOSED: the queue drains what it has, the weak reference is
dead, nothing drives the parser again. Nothing in the suite failed. #13408 adds one test
per side, each feeding a single oversized read, dropping the connection synchronously, and
asserting all 8000 arrive (assert 4097 == 8000 without the fix).
Finding 2 — set_exception() drops the stash, and that is a behaviour change
set_exception() clears _stalled_reader ("Nothing more will be parsed"), but
_read_from_buffer() raises only once the buffer is empty, so buffered messages are still
delivered first. Frames the parser stopped short of are not:
this branch: queued before stall: 4097 → delivered: 4097, then ConnectionResetError
master: queued: 8000 → delivered: 8000, then ConnectionResetError
(8000 empty TEXT frames in one read, then queue.set_exception(ConnectionResetError()).)
Correct for a parse error, where the reader is poisoned via self._exc anyway. Less
obviously correct for WebSocketResponse._cancel(), which is the transport-died hook and
now silently discards complete frames that master delivered. Worth a deliberate decision
either way — dropping the clear in set_exception() would keep the stash reachable, since
the terminal states that matter for the cycle are already covered by feed_eof() plus the
weak reference.
Side note found while tracing it: _cancel()'s comment says web_protocol calls it from
connection_lost or at shutdown, but the only in-tree callers of _cancel() are
BaseRequest._cancel (line 376/432 of web_protocol.py, a different method) and one
test. Either the comment is stale or a wiring regression is hiding there — either way, WS
handlers currently learn about connection loss via feed_eof(), not _cancel().
Finding 3 — the new rule is invisible to third parties
WebSocketReader is exported in aiohttp.http.__all__ and
aiohttp.http_websocket.__all__. Anyone constructing one and handing it to set_parser()
(proxies, ASGI bridges) must now keep their own strong reference, and the failure mode is
silent: a short stream, or a hang if EOF never arrives. Cheap to mention in the changelog
fragment.
Verified, not assumed
- Post-EOF drain calls
pause_reading()/resume_reading()withtransport is None; both
are guarded, andpause_reading()'sassert self._parser is not Noneis skipped only
because_upgradedstaysTrueafter connection loss. Safe, but that is the coupling
holding it up. - Driving the parser from
_read_from_buffer()is not re-entrant:_release_waiter()
only sets a future result. - Cython:
cdef object __weakref__cythonizes clean andweakref.ref()works on the
compiledWebSocketReader; 98 parser tests pass against the built extension. (Only
_websocket/*was compiled — the full build needs a Node.js build of vendored llhttp.)
Analysis by Kōan (Claude Opus 5) at @bdraco's request.
|
@aiolibsbot review |
|
bot was asleep, woke it up |
Scope analysis done. Two holders of Reader constructed in exactly two places in-tree, both now owned via One gap remains. |
…ct check" This reverts commit 678ada7.
|
backported manally and tested on HA. no observable regression |
|
|
|
|
|
However will address Drain-driven resume re-copies the whole tail on every single pop because its simple and makes sense |
Important Agreed on pause_reading, and the pop-driven re-drive finding no longer applies — |
Useful signal, though HA traffic likely never stalls the parser — needs one read carrying more than ~4096 frames. Worth a targeted check: flood empty frames from a peer, drain slowly, watch |
|
|
|
@aiolibsbot gh_request need to do a stranding audit, but I think there is always another resume. I think the only possible strand is if consumer doesn't call |
Stranding auditYour instinct holds: there is always another resume, and the only strand in this PR's own stall/resume loop is a consumer that stops reading — which is backpressure working as designed, not a bug in the parser. What I verified, and the one real strand I found next door. The invariants that make the stall safe
Probed against the pure-Python reader with a real
So: no strand. The The strand that does exist — and it is not in this PRWhile walking the pause/resume graph I found a hard hang in the fragment-count backpressure from #13352 (master only, unreleased). It pauses the transport in the middle of a frame: self._payload_fragments.append(data_cstr[f_start_pos:f_end_pos])
if (self._max_fragments and len(self._payload_fragments) > self._max_fragments
and not self.queue._protocol._reading_paused):
self.queue._protocol.pause_reading()
breakThat pause can never be lifted. The frame only completes once more data arrives, and the only resume is Reproduced end-to-end against a real server over a real socket: one 4 KiB binary frame written 2 bytes at a time (2048 reads against a cap of 1024) and the server never receives it, forever. 1024 reads works. Costs a peer ~1024 small writes per connection, and the server WebSocket path has no read timeout unless a heartbeat is configured. Fix in #13487 — collapse the fragments into one buffer instead of pausing, which keeps the memory bound the cap was added for. It also makes This is worth noting for this PR too: the new Two small things in this PR1. An errored reader leaves Benign in practice (a parser exception is fatal and the transport gets closed), but it means the stash is retained and the pause is permanent for reasons unrelated to backpressure. Clearing 2. Nit: 🤖 Audit run by Kōan |
|
stranding/stall fix in #13488 |
|
Confirmed #13488 orthogonal to this PR. Mid-frame pause lives in One interaction worth noting: this PR adds Re-measured this HEAD with the gate: 200000 frames delivered, 5 reads, max tail 0.26 MB, 2.4 s. Prior blocker resolved. |
|
|
PR Review — Fix memory consumption on empty/small messagesThe two open suggestions from the last round are now one — What's solid, verified on this head with the Cython extension built (
What's worth a look (neither blocks merge):
✅ Resolved since last review (1)Previously-flagged issues verified fixed
Checklist
ℹ️ Triage summary1 pre-existing finding(s) on unchanged code suppressed (freeze). Silent Failure Analysis
🟡 **1. MEDIUM** — misattributed error / silently discarded buffered data
|
bdraco
left a comment
There was a problem hiding this comment.
Seemingly endless edge cases the bot kept finding now addressed (and yes they were legit)
manually backported and retested on HA install. All good.
Backport to 3.15: 💔 cherry-picking failed — conflicts found❌ Failed to cleanly apply 5e54037 on top of patchback/backports/3.15/5e54037933fa8422be5fa8a2002d118ed2d4e5e0/pr-13393 Backporting merged PR #13393 into master
🤖 @patchback |
Backport to 3.14: 💔 cherry-picking failed — conflicts found❌ Failed to cleanly apply 5e54037 on top of patchback/backports/3.14/5e54037933fa8422be5fa8a2002d118ed2d4e5e0/pr-13393 Backporting merged PR #13393 into master
🤖 @patchback |
No description provided.