Apply part encodings in MultipartWriter.as_bytes()

[2sumtech]

What do these changes do?

MultipartWriter.as_bytes() ignored the per-part Content-Encoding / Content-Transfer-Encoding that write() applies via MultipartPayloadWriter, so its result did not match the wire body. The output contradicted the parts' own headers, and digest authentication with qop=auth-int always failed for such payloads because the middleware hashes as_bytes() while the wire carries the encoded body. This applies the same transformations (ZLibCompressor for gzip/deflate, base64/quoted-printable for CTE) in as_bytes(), making the output byte-identical to write().

Are there changes in behavior for the user?

as_bytes() now returns compressed/encoded part bodies for non-form-data multiparts that declare those headers, matching what is actually sent on the wire.

Is it a substantial burden for the maintainers to support this?

No. The change reuses the exact transformation code path write() already exercises, so the two stay in sync by construction. No new public API, no new dependency.

Related issue number

Fixes #13495

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes (no docs change needed, behavior now matches the documented wire format)
• If you provide code modification, please add yourself to CONTRIBUTORS.txt
• Add a new news fragment into the CHANGES/ folder (13495.bugfix.rst)

Test runs

New parametrized tests assert byte-equality between as_bytes() and the write() wire output for gzip/deflate Content-Encoding and base64/quoted-printable CTE, plus content round-trips. 6 failed without the fix and 6 pass with it. Full test_multipart.py + test_formdata.py + test_client_middleware_digest_auth.py + test_payload.py: 448 passed. flake8 and mypy clean on touched files.

Drafted with Claude Code (Fable 5); reviewed by @2sumtech.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.01%. Comparing base (394959f) to head (50228e3).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #13496    +/-   ##
========================================
  Coverage   99.01%   99.01%            
========================================
  Files         135      135            
  Lines       50026    50162   +136     
  Branches     2611     2626    +15     
========================================
+ Hits        49531    49668   +137     
  Misses        371      371            
+ Partials      124      123     -1     

Flag	Coverage Δ
Autobahn	21.99% <12.31%> (-0.03%)	⬇️
CI-GHA	98.90% <100.00%> (+<0.01%)	⬆️
OS-Linux	98.68% <100.00%> (+<0.01%)	⬆️
OS-Windows	97.01% <100.00%> (+0.01%)	⬆️
OS-macOS	97.93% <100.00%> (+<0.01%)	⬆️
Py-3.10	98.10% <100.00%> (+<0.01%)	⬆️
Py-3.11	98.34% <100.00%> (+<0.01%)	⬆️
Py-3.12	98.42% <100.00%> (+<0.01%)	⬆️
Py-3.13	98.40% <100.00%> (+<0.01%)	⬆️
Py-3.14	98.44% <100.00%> (+<0.01%)	⬆️
Py-3.14t	97.52% <100.00%> (+<0.01%)	⬆️
Py-pypy-3.11	97.38% <100.00%> (+<0.01%)	⬆️
VM-macos	97.93% <100.00%> (+<0.01%)	⬆️
VM-ubuntu	98.68% <100.00%> (+<0.01%)	⬆️
VM-windows	97.01% <100.00%> (+0.01%)	⬆️
cython-coverage	82.76% <8.51%> (-0.15%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[codspeed-hq]

Merging this PR will degrade performance by 16.96%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 91 untouched benchmarks
⏩ 83 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_read_base64_part	14.3 ms	17.2 ms	-16.96%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing 2sumtech:fix/multipart-as-bytes-encodings (50228e3) with master (394959f)}

Footnotes

1. 83 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[aiolibsbot]

PR Review — Apply part encodings in MultipartWriter.as_bytes()

Correct, well-targeted fix — as_bytes() now mirrors write()'s per-part encoding. Merge-ready; four non-blocking notes.

Specific things done well:

• The transformations match write()'s ordering exactly (compress → transfer-encode), including the suppress_deflate_header=True flag and the identity/binary normalisation already done in append_payload(), so no special-casing was needed for identity, binary, or form-data parts.
• The tests assert against bytes(buf) — the real write() output — rather than a hand-written expected literal. That is the right assertion for a "these two paths must agree" bug and is what makes the fix trustworthy.
• I verified the end-to-end qop=auth-int flow on the PR head: as_bytes() is idempotent across calls, and hash-then-write stays consistent, so the middleware fix actually holds.
• Verified locally on the PR branch: 445 passed / 6 skipped across test_multipart.py, test_formdata.py, test_client_middleware_digest_auth.py, test_payload.py; flake8 and black clean on the touched files. (mypy is not installed in this environment — unverified.)
• Docs entry is correctly placed under the MultipartWriter directive, which also resolves the :meth: cross-reference the changelog fragment needs.

Key points:

• quoted-printable still diverges from write() for parts written in multiple chunks (>256 KB) — b2a_qp restarts its 76-column counter per call, so soft line breaks land at different offsets. Reproduced: a 400 KB QP part differs at byte 269338. gzip/deflate/base64 are chunk-invariant and match at any size, so this is the only residual gap.
• decode()'s docstring still points users at as_bytes().decode() as a drop-in, but the two now disagree for encoded parts — and raise UnicodeDecodeError for gzip ones.
• Missing the CHANGES/13496.bugfix.rst -> 13495.bugfix.rst symlink that AGENTS.md asks for (six such symlinks already exist in CHANGES/).
• No test for a part carrying both Content-Encoding and Content-Transfer-Encoding — the composed path works today (verified manually) but is the one most exposed to a future ordering regression.

---

🟢 Suggestions

1. quoted-printable still diverges from write() for multi-chunk payloads
aiohttp/multipart.py:1151-1152

The binascii.b2a_qp(part_bytes) call encodes the whole body at once, but write() calls b2a_qp per chunk in MultipartPayloadWriter.write() (multipart.py:1257). b2a_qp inserts soft line breaks (=\n) every 76 columns and restarts its column counter on each call, so the two paths place the breaks at different offsets once a part is written in more than one chunk.

I verified this empirically against the PR head — a 400 KB file part with Content-Transfer-Encoding: quoted-printable (written in 2 chunks of 262144 + 137856 bytes):

qp no-newline 400k: match=False  wire=410823  as_bytes=410823
  first diff at byte 269338
  wire : b'AAAA...AAA=\nAAAAAAAA'   # break lands at the chunk boundary
  asb  : b'AAAA...AA=\nAAAAAAAAAA'  # break lands on the 76-col grid

Why it matters: this is exactly the case the PR sets out to fix. A qop=auth-int digest hash over a large quoted-printable part will still silently mismatch the wire body, so the auth failure the PR describes persists for that input — just for a narrower set of payloads than before. Note the two encodings are both valid QP; only the bytes differ.

Both in-memory cases and gzip/deflate/base64 at any size do match (zlib and the base64 3-byte buffering in MultipartPayloadWriter are chunk-invariant), so this is the only residual gap. Options: chunk as_bytes() the same way write() does, or note the limitation in the changelog/docs so the remaining case is known rather than assumed fixed.

elif te_encoding == "quoted-printable":
    part_bytes = binascii.b2a_qp(part_bytes)

2. decode() docstring now recommends a non-equivalent replacement
aiohttp/multipart.py:1112

decode() still iterates for part, _e, _te in self._parts and emits raw part bodies, so after this change decode() and as_bytes().decode() no longer agree for encoded parts — but the docstring directs users to swap one for the other:

WARNING: This method may do blocking I/O ... Use as_bytes().decode() instead.

Verified on the PR head:

• Content-Transfer-Encoding: base64 — decode() yields the body 'hello', as_bytes().decode() yields 'aGVsbG8='.
• Content-Encoding: gzip — decode() yields 'hello', as_bytes().decode() raises UnicodeDecodeError on the gzip bytes.

Why it matters: anyone following that guidance to migrate off the blocking method gets either different output or an exception, with nothing in the changelog pointing at it. Worth either updating the docstring to say the replacement now reflects the wire body (and can be non-UTF-8), or applying the same transformations in decode() so the two stay in step.

        WARNING: This method may do blocking I/O if parts contain file payloads.
        It should not be called in the event loop. Use as_bytes().decode() instead.

3. Missing PR-numbered changelog symlink
CHANGES/13495.bugfix.rst:1

AGENTS.md asks for both numbers when a PR fixes an issue:

Both issue and PR number wanted: keep the issue-numbered file and symlink: ln -s 1234.bugfix.rst CHANGES/1240.bugfix.rst

Only the issue-numbered fragment 13495.bugfix.rst is present. This is a live convention in the tree — CHANGES/ currently carries six such symlinks (13330.bugfix.rst -> 13329.bugfix.rst, 13249.bugfix.rst -> 13203.bugfix.rst, 13122.feature.rst -> 13006.feature.rst, …).

Fix: ln -s 13495.bugfix.rst CHANGES/13496.bugfix.rst

4. No test for combined Content-Encoding + Content-Transfer-Encoding
tests/test_multipart.py:1331-1380

The three new tests cover compression and transfer-encoding in isolation, but not both on one part — which is the branch where ordering matters (write() composes them as compress-then-encode via enable_compression() + enable_encoding(), and the new code has to mirror that order).

I verified manually that the composed path does match today (gzip + base64 over a 5000-byte part: match=True), so this is a coverage gap, not a bug. But the ordering is precisely the thing a future refactor of either path could silently invert, and nothing would catch it.

Suggest one more case in the same style:

writer.append("Time to Relax!", {
    CONTENT_ENCODING: "gzip",
    CONTENT_TRANSFER_ENCODING: "base64",
})
await writer.write(stream)
assert await writer.as_bytes() == bytes(buf)

---

Checklist

• Fix matches write()'s encoding semantics — suggestion ＃1
• New behaviour covered by tests — suggestion ＃4
• Related test suites pass locally (445 passed)
• Lint/format clean (flake8, black)
• Changelog fragment follows AGENTS.md conventions — suggestion ＃3
• Docs updated for user-visible behaviour change — suggestion ＃2
• CONTRIBUTORS.txt entry in alphabetical position
• No scope creep beyond the stated fix
• No double-encoding risk (as_bytes does not cache into the write path)
• Form-data multiparts unaffected

---

Automated review by Kōan (Claude) HEAD=ca142f2 6 min 8s

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the earlier quoted-printable chunk-divergence and unbounded partial-line buffering issues are both addressed by the shared fixed-boundary encoder and streaming block flush.

_{Reviews (3): Last reviewed commit: "Bound the quoted-printable partial-line ..." | Re-trigger Greptile}

[Polandia94]

Please follow the PR template: https://github.com/aio-libs/aiohttp/blob/master/.github/PULL_REQUEST_TEMPLATE.md

[2sumtech]

done, restructured the description to the full template