Skip to content

fix(fetch): avoid buffering streamed uploads in memory#5455

Open
cesarvspr wants to merge 1 commit into
nodejs:mainfrom
cesarvspr:fix/fetch-stream-upload-tee-4058
Open

fix(fetch): avoid buffering streamed uploads in memory#5455
cesarvspr wants to merge 1 commit into
nodejs:mainfrom
cesarvspr:fix/fetch-stream-upload-tee-4058

Conversation

@cesarvspr

@cesarvspr cesarvspr commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

This relates to...

Fixes #4058.

Rationale

Streaming a large upload with fetch (e.g. body: fs.createReadStream(...), duplex: 'half') ends up holding the whole file in memory.

fetch clones the request so it can follow a redirect, and cloning tees the body's stream: the original request keeps one branch and the wire sends the other. A stream body has a null source, so it can't be replayed across a redirect anyway. http-redirect-fetch returns a network error for a non-303 redirect with a null source, and on any other redirect the body is re-extracted from its source rather than from the teed branch. So the branch kept on the original request is never read, but the tee still buffers every chunk waiting for it, which is the entire upload.

The clone happens on every fetch with a body, not only when a redirect actually occurs, so this affected all streamed uploads.

Changes

Skip the tee for null-source bodies in the internal redirect clone and reuse the stream directly. cloneBody takes a forFetch flag that only the clone in http-network-or-cache fetch sets. Request.clone() and Response.clone() don't pass it, so they still tee and keep two independently readable branches. Bodies with a non-null source are unchanged.

Features

N/A

Bug Fixes

A streamed request body is no longer buffered in memory.

Breaking Changes and Deprecations

N/A

Status

  • I have read and agreed to the Developer's Certificate of Origin
  • Tested (test/fetch/issue-4058.js: a 128 MiB streamed PUT peaks at a few MiB with the fix and ~128 MiB without it; the clone and redirect suites pass; lint clean)
  • [S] Benchmarked (optional)
  • Documented
  • In review
  • Merge ready

Notes

cloneBody/cloneRequest are also touched by #5415 (lazy Request/Response internals), so a textual conflict is expected and whichever lands second will need a small rebase. This is an independent change: it only affects the redirect-following clone of a null-source body, not the lazy-init refactor.

@codecov-commenter

codecov-commenter commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (0f1f890) to head (44f9885).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5455   +/-   ##
=======================================
  Coverage   93.46%   93.47%           
=======================================
  Files         110      110           
  Lines       37106    37122   +16     
=======================================
+ Hits        34682    34698   +16     
  Misses       2424     2424           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cesarvspr cesarvspr force-pushed the fix/fetch-stream-upload-tee-4058 branch from e6235db to c235156 Compare June 28, 2026 03:32
fetch clones the request so it can follow redirects, and cloning tees the
body's stream: the original keeps one branch and the wire sends the other. A
stream body has a null source and can never be replayed across a redirect
(http-redirect-fetch returns a network error for a non-303 redirect with a null
source, and otherwise re-extracts from the source, not from the teed branch), so
the branch kept on the original request is never read. The tee still buffers
every chunk for it, so a streamed upload ends up fully held in memory.

Skip the tee for null-source bodies in the internal redirect clone and reuse the
stream directly. Request.clone() and Response.clone() still tee, so their two
branches stay independently readable.

Fixes nodejs#4058
@cesarvspr cesarvspr force-pushed the fix/fetch-stream-upload-tee-4058 branch from c235156 to 44f9885 Compare June 28, 2026 03:42

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked this against the Fetch Standard: the HTTP-network-or-cache fetch note explicitly encourages avoiding teeing when the request body source is null, and redirects/auth retries for such bodies either fail or do not need the retained tee branch. Request.clone()/Response.clone() behavior remains unchanged.

@KhafraDev KhafraDev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that is what the spec says, this is not the way to implement it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request stream contents are kept in memory when using fetch

4 participants