fix(repeater): send request bodies to libcurl as raw bytes

[aborovsky]

relates-to: #744

[Copilot]

Pull request overview

This PR fixes a regression in the HTTP repeater where request bodies containing non‑UTF8 bytes could be silently corrupted when passed to libcurl as a JS string. It switches the body write path to a binary-safe libcurl upload callback and adjusts virtual-script body transformation to preserve the original body/encoding when scripts do not modify the body.

Changes:

• Replaced CURLOPT_POSTFIELDS usage with UPLOAD + INFILESIZE_LARGE + READFUNCTION (and added SEEKFUNCTION) to send bodies as raw bytes.
• Updated transformScript so the original body+encoding are restored when a virtual script leaves the body unchanged.
• Added extensive unit tests validating byte-exact body forwarding, header behavior, method preservation, and correct libcurl callback behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/RequestExecutor/HttpRequestExecutor.ts	Switches libcurl body handling to upload callbacks for binary safety; updates virtual-script body restoration logic.
src/RequestExecutor/HttpRequestExecutor.spec.ts	Adds a comprehensive “request body” test suite covering binary integrity, headers, methods, and upload callback replay/seek behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aborovsky]

Regression fix for v13.12.0, based on next @ 13.13.0-next.1.

Problem

applyCurlBody handed the decoded body to CURLOPT_POSTFIELDS as a JS string:

curl.setOpt('POSTFIELDS', bodyBuffer.toString());   // lossy UTF-8 decode
curl.setOpt('POSTFIELDSIZE', bodyBuffer.length);    // truncates the expanded buffer

Buffer.prototype.toString('utf8') replaces every byte that is not part of a valid UTF-8 sequence with U+FFFD, and the native binding re-encodes each of those as the three bytes EF BF BD. POSTFIELDSIZE still declared the original length, so only the first N bytes of the roughly 2x longer buffer went out. The request succeeded, Content-Length matched, and the target received different bytes than the client sent.

Any body carrying bytes that do not form valid UTF-8 was affected: binary uploads, gzip/deflate bodies, protobuf, DER, encrypted blobs. Text that is valid UTF-8 survives the round trip unharmed, so reproducing with emoji proves nothing — use a 0x00..0xFF pattern. Measured end to end through bridges-proxy -> bridges -> bright-cli -> target, a 10 MB 0x00..0xFF body diverged at offset 128 with 7,802,880 of 10,485,760 bytes wrong.

POSTFIELDS cannot be fixed caller-side: the binding types it string | number | boolean | null and registers it in curlOptionString, not curlOptionBlob (the only category that accepts a Buffer). COPYPOSTFIELDS is equally string-only.

Affected versions

range	body write path	binary-safe
<= v13.11.0	req.end(iconv.encode(body, encoding)), a Buffer	yes
v13.11.0-next.5 … v13.13.0-next.1	POSTFIELDS + bodyBuffer.toString()	no

Introduced by 5857011 (refactor(repeater): add custom HTTP client #744). v13.12.0 is the first stable release carrying it.

Fix

src/RequestExecutor/HttpRequestExecutor.ts

1. applyCurlBody now feeds the buffer to libcurl through UPLOAD + INFILESIZE_LARGE + READFUNCTION, which copies from the buffer verbatim. The callback contract is native-verified: Easy.cc:1616 hands the callback a fresh Buffer of size * nmemb and memcpys back exactly the number of bytes returned.
2. A SEEKFUNCTION is registered alongside it. libcurl replays the body when a request has to be sent twice (authentication negotiation); a read callback without a seek callback is reported as non-seekable and the replay fails.
3. transformScript was lossy in the same way, and additionally dropped encoding (Request.toJSON() does not carry it), which re-encoded the already-substituted body a second time. The original body and encoding are now restored when the script leaves the body untouched and sets no encoding of its own. A script that adds encoding is asking for its own body to be decoded, so that result is passed through unchanged. This path was latent: it only triggers when a virtual script is registered for the target host.

Behaviour change to note

libcurl adds Content-Type: application/x-www-form-urlencoded by default for a POSTFIELDS body but not for an upload, so that default header no longer appears on bodies sent without an explicit Content-Type. Measured directly against the binding with the same 8 KiB body:

--- postfields ---   Host, Accept: */*, Content-Length: 8192, Content-Type: application/x-www-form-urlencoded
--- upload ---       Host, Accept: */*, Content-Length: 8192

This restores the pre-v13.12.0 behaviour: the old node http path added no default Content-Type either. A caller-supplied Content-Type is forwarded via HTTPHEADER and overrides the libcurl default either way, so it is unaffected.

Regression checks

Verified against the binding and pinned by tests:

• no Expect: 100-continue on either path, so no added round trip or stall
• no Transfer-Encoding; Content-Length equals the decoded byte length
• the request method survives UPLOAD (which implies PUT) for POST, PUT, PATCH, DELETE and GET, because CUSTOMREQUEST is applied afterwards
• a caller-supplied Content-Length still overrides the body length and is not duplicated, so request smuggling payloads keep working
• requests with no body are untouched: the early return runs before any upload option is set
• a body that decodes to zero bytes sends Content-Length: 0 and does not hang
• the read offset lives in a closure created per Curl handle, so the parallel per-certificate requests in tryRequestWithCertificates cannot share it

Test coverage

request body suite, 24 tests, 259/259 for the whole unit suite on Node 22.22.2. Lint, Prettier and the webpack build are clean.

• 4 KiB and 256 KiB 0x00..0xFF base64 bodies byte-exact (256 KiB spans several read-callback invocations)
• multi-byte UTF-8 forwarded verbatim
• Content-Length equals the decoded byte length, no Transfer-Encoding
• no Expect: 100-continue, no default Content-Type, caller Content-Type forwarded verbatim, caller Content-Length neither duplicated nor overridden
• method preserved for POST / PUT / PATCH / DELETE / GET
• absent body and zero-byte decoded body
• virtual scripts: body byte-exact when the script leaves it alone, rewritten body honoured when the script modifies it, body byte-exact when the script only edits headers, and a script-supplied encoding honoured when the script touches nothing else
• libcurl upload callbacks: seek callback registered alongside the read callback, byte-for-byte replay after seeking back to the start, seek origins SET/CUR/END, out-of-range seeks rejected, and the read callback never writing past the window libcurl asked for

Counterfactual: restoring the two v13.12.0 lines fails 16 of the 24, with firstMismatch === 128 — the same offset the live stack measured.

Follow-ups, deliberately not in this PR

• An e2e test that routes a binary body through a real repeater and asserts byte-for-byte integrity on the target side (requested on the ticket). This repo's tests/e2e/repeater harness drives requests through a scan, and the engine mutates the body by design, so a byte-equality assertion is not deterministic there. It belongs in engine-e2e, which is where the ticket's DoD places it.
• Request.toJSON() also drops timeout, maxContentSize, decompress and keepAlive, so a virtual script rebuild loses them. Separate defect, separate ticket.
• WsRequestExecutor (src/RequestExecutor/WsRequestExecutor.ts:191) calls client.send(request.body) without honouring encoding at all, so a base64 body goes out as literal base64 text. Not exercised here (Protocol.HTTP), noted so it is not rediscovered as the same bug.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[aborovsky]

E2E check.

[shevchenkov-neuralegion]

Just suggestions I think might be valid. Feel free to reject.