Decode compressed response bodies incrementally

[Kludex]

Summary

The content decoders used to inflate each raw chunk fully in a single decompress() call and re-chunk the result afterwards, so a small compressed chunk could inflate to an arbitrarily large buffer and iter_bytes(chunk_size) did not actually bound peak memory.

This reworks the decoders to yield bounded pieces as they decode:

• gzip/deflate drain a shared ZlibDecompressor with max_length.
• brotli uses output_buffer_limit (now requires brotli>=1.2.0).
• zstd uses max_length on the stdlib compression.zstd backend.
• MultiDecoder pipes its children lazily, so the bound holds across stacked encodings.

iter_bytes(chunk_size) / aiter_bytes(chunk_size) now bound peak memory the way urllib3's read(amt) does, rather than materializing a whole decoded chunk before re-slicing it.

It also closes the underlying stream when decoding raises part-way through, so a decode error releases the connection instead of leaking it (with a regression test that fails under strict async-generator finalization if the cleanup regresses).

AI Disclaimer

This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.

[github-actions]

Docs preview: https://7d6e7736-httpx2-docs.pydantic.workers.dev

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 472041bacf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use the engineer as the sole commit author

The reviewed commit records Codex <codex@openai.com> as both author and committer, which violates the repository requirement that authorship belong only to the engineer. Recreate the commit with the engineer as its sole author before merging.

AGENTS.md reference: AGENTS.md:L3-L3

Useful? React with 👍 / 👎.

[cubic-dev-ai]

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/httpx2/httpx2/_decoders.py">

<violation number="1" location="src/httpx2/httpx2/_decoders.py:132">
P2: Benchmark CI now fails because `test_bench_gzip_decode` adds the iterator results from `GZipDecoder.decode()` and `flush()`. Consume and join both iterators in that benchmark.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Benchmark CI now fails because test_bench_gzip_decode adds the iterator results from GZipDecoder.decode() and flush(). Consume and join both iterators in that benchmark.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/httpx2/httpx2/_decoders.py, line 132:

<comment>Benchmark CI now fails because `test_bench_gzip_decode` adds the iterator results from `GZipDecoder.decode()` and `flush()`. Consume and join both iterators in that benchmark.</comment>

<file context>
@@ -86,22 +123,23 @@ class DeflateDecoder(ContentDecoder):
         self.first_attempt = False
         try:
-            return self.decompressor.decompress(data)
+            yield from self.decompressor(data)
         except zlib.error as exc:
             if was_first_attempt:
</file context>

[codspeed-hq]

Merging this PR will not alter performance

✅ 17 untouched benchmarks
⏩ 7 skipped benchmarks¹

---

_{Comparing incremental-content-decoding (33c8c53) with main (a966320)}

Footnotes

1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩