fix: decode all members of concatenated gzip in gunzipSync#287
Open
spokodev wants to merge 1 commit into
Open
Conversation
RFC 1952 2.2 defines a gzip stream as a series of members that must all be decoded and concatenated. gunzipSync (and decompressSync, which routes to it) only decoded the first member: it sized the output from the trailing ISIZE of the whole buffer (the last member's size) and inflt returned at the first member's BFINAL with no member-iteration loop. Concatenated streams were silently truncated to the first member, and a large first member combined with a small last-member ISIZE could overflow the undersized output buffer. The streaming Gunzip class gained concatenated-member support in v0.8.0 (101arrowz#102) but the sync path was missed. Walk the members in gunzipSync the same way: decode a member, advance past its header, DEFLATE data, and 8-byte trailer via the bit position inflt leaves behind, and repeat until the input is exhausted, then concatenate. Each member's output is sized by inflt itself, since the trailing ISIZE only describes the last member.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
gunzipSync(anddecompressSync, which routes to it) silently truncates a concatenated multi-member gzip stream to its first member, and an undersized output buffer can throwRangeError: offset is out of bounds. fflate's own streamingGunzipclass decodes the same input correctly.CLI oracle agrees with node's zlib:
A larger case truncates the same way:
Spec
RFC 1952 section 2.2: a gzip file consists of a series of members, all of which must be decoded and concatenated to recover the original data.
Root cause
Concatenated-member support was added to the streaming classes in v0.8.0 (#102), but the synchronous path was not updated.
gunzipSyncwas:Two issues:
gzl(data), which reads the trailing ISIZE at the end of the whole buffer, i.e. the last member's size, not the total. When the first member is larger than the last, the buffer is too small (RangeError); otherwise it just holds the first member.inflt(...)decodes a single DEFLATE stream and returns at the first member'sBFINAL. There is no member-iteration loop, unlikeGunzip.push("process concatenated GZIP").Fix
Iterate the members in
gunzipSyncthe same way the streaming path does: decode a member, advance past its header + DEFLATE data + 8-byte trailer (using the bit positioninfltleaves in the state object), and repeat until the input is exhausted, then concatenate. Each member's output is sized byinfltitself, since the trailing ISIZE only describes the last member. The single-member fast path is preserved, and truncated/invalid input still throwsinvalid gzip data.Verification
A new test (
test/6-multimember.ts) covers two-member, five-member, anddecompressSyncmulti-member streams against nodezlibas the oracle, plus a single-member regression guard. It fails on master ("AAAAAAAABB"decodes to"AA"; the five-member case to"member-0-payload") and passes with this change. The full suite is green:Also verified manually: empty middle member, a 350 KB two-member stream, and
opts.outwith multi-member input all match the zlib oracle.