Repository navigation
compress: validate repcodes against retained dictionary content - #4915
Merged
Merged
Conversation
Cyan4973
requested changes
Nov 10, 2026
Cyan4973
left a comment
Contributor
There was a problem hiding this comment.
The new check at the end of ZSTD_loadZstdDictionary() also runs when the dictionary is loaded directly into a CCtx.
In this case, the window may have been reduced by overflow correction,
so a CCtx which has already compressed a few GB can reject a valid dictionary that a fresh CCtx would accept.
The CCtx path already discards out-of-range repcodes, so this check is only needed for the CDict:
I would limit it to tfp == ZSTD_tfp_forCDict.
minor: the + 1 in tests/fuzzer.c:1130 aligns src on rep - 1, so the boundary cases never actually encode a match at rep.
minor: when the dictionary is rejected, it shows up as a NULL CDict, which the CLI reports as "Allocation error". That's a bit misleading.
Full-format dictionaries may contain repeat offsets into content discarded by the short-cache CDict limit or the general index limit. Revalidate all three repcodes after loading the retained suffix and enforce the same bound before attaching a CDict, preventing wrapped dictMatchState indices and wild reads. Add regression coverage for 20 MiB dictionaries, every repcode slot, fast/dfast levels, deferred and streaming dictionary loading, the short-cache boundary, and valid non-truncating controls.
terrelln
force-pushed
the
2026-11-10-large-cdict
branch
from
November 10, 2026 23:05
02522f6 to
4f5dce9
Compare
Contributor
Author
|
Cyan4973
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full-format dictionaries may contain repeat offsets into content discarded by the short-cache CDict limit or the general index limit. Revalidate all three repcodes after loading the retained suffix and enforce the same bound before attaching a CDict, preventing wrapped dictMatchState indices and wild reads.
Add regression coverage for 20 MiB dictionaries, every repcode slot, fast/dfast levels, deferred and streaming dictionary loading, the short-cache boundary, and valid non-truncating controls.
NOTE: This will reject dictionaries that have repcodes > 16 MiB when using the short-cache CDict. But I think this is reasonable, and if it becomes a problem in the future, we can fix it then, as it is an encoder-side only limitation.