fix: use TextDecoder in FetchResponse.bodyText to prevent OOM on large responses - #5174
Open
yutonakamura-dev wants to merge 1 commit into
Open
fix: use TextDecoder in FetchResponse.bodyText to prevent OOM on large responses#5174yutonakamura-dev wants to merge 1 commit into
yutonakamura-dev wants to merge 1 commit into
Conversation
…e responses Fixes ethers-io#5168 `FetchResponse.bodyText` decoded every response body through `toUtf8String`, which internally allocates one array element per byte and one string per code point via `getUtf8CodePoints` + `_toUtf8String`. On large JSON-RPC responses (e.g. 18.7MB `trace_block` results), three concurrent calls exhaust a 1GB heap and crash the process. Replaced `toUtf8String(this.#body)` with `new TextDecoder("utf-8", { fatal: true }).decode(this.#body)` in both `bodyText` getter and the error handler. `fatal: true` preserves the existing behavior of throwing on invalid UTF-8 data. The existing `toUtf8String` function and its error callback API remain unchanged — this fix only affects the HTTP response decoding path. Before: 18.7MB × 3 concurrent → OOM crash at 256MB heap (928ms, 794MB per decode) After: 18.7MB × 3 concurrent → 31ms total, 62MB heap, no crash
Member
|
Oh wow! I was unfamiliar with the Thanks! |
Author
|
It is still humble and just an idea. I am glad to hear that it was of some help. Happy to adjust anything if needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5168
FetchResponse.bodyTextdecodes every response body throughtoUtf8String, which allocates one array element per byte and one string per code point. On large JSON-RPC responses (e.g. 18.7MBtrace_blockresults), three concurrent calls exhaust a 1GB heap and crash the process.Changes
Replaced
toUtf8String(this.#body)withnew TextDecoder("utf-8", { fatal: true }).decode(this.#body)in two places insrc.ts/utils/fetch.ts:bodyTextgetter (line 813) — the main decode path for every HTTP responseRemoved the now-unused
toUtf8Stringimport fromfetch.ts. ThetoUtf8Stringfunction itself and its error callback API remain unchanged for other callers.Why TextDecoder
TextDecoderis a native API available in all supported environments (Node.js 12+, all modern browsers)fatal: truethrowsTypeErroron invalid UTF-8, preserving the existing error behaviorReproduction & Results
Tested with 3 concurrent 18.7MB response bodies under a 256MB heap cap:
Testing
test-utils-utf8tests pass (30 tests)test-utils-misc,test-utils-maths,test-utils-unitstests pass (92 tests)TextDecoderoutput matchestoUtf8Stringoutput for all valid UTF-8 inputs