PatchCompressor: write the decompressed size into the zstd frame header - #26
Merged
Merged
Conversation
ZstdOutputStream never pledges a source size, so patch .zst files carried no content-size field. The Otzaria client decompresses through the zstandard FFI plugin, which sizes its output buffer from ZSTD_getFrameContentSize and, when the field is missing, falls back to compressedSize * 20 in one allocation. A ~560 MB patch therefore requested 11,694,234,840 bytes and failed with "Invalid argument(s): Could not allocate 11694234840 bytes". Compress via ZstdCompressCtx streaming with setPledgedSrcSize(fileSize), keeping memory bounded (1 MiB input chunks) and preserving level/workers. Tests assert Zstd.getFrameContentSize equals the input size (single and multi-threaded), a byte-exact round trip, matching sha256s, and the empty patch case. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Users updating the library via delta patches hit:
PatchCompressorusedZstdOutputStream, which never pledges a source size, so the produced.zstframes carry no content-size field. On the client, thezstandardFFI plugin sizes its output buffer fromZSTD_getFrameContentSize; when the field is missing it falls back tocompressedSize * 20in a single contiguous allocation. 11,694,234,840 = 584,711,742 × 20 exactly, i.e. a ~558 MB patch asked for ~10.9 GiB of RAM.Fix
Compress through
ZstdCompressCtxstreaming withsetPledgedSrcSize(fileSize)andEndDirective.ENDon the last chunk, so the exact decompressed size lands in the frame header. Memory stays bounded (1 MiB input chunks, zstd-recommended output buffer); level, worker count and the returnedResultmetadata are unchanged.Tests
New
PatchCompressorTest:Zstd.getFrameContentSizeequals the input size, single-threaded and withworkers = 2:generator-common:jvmTestpasses (87 tests).Related
The Otzaria client side is also being changed to download and decompress patches as a stream to disk, so large patches no longer sit in RAM regardless of the header. This PR makes existing clients (in-memory decompress) allocate the right size instead of 20× the compressed size.
🤖 Generated with Claude Code