Skip to content

GH-404: Clarify dictionary fallback encoding #609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
d33bs wants to merge 1 commit into
base: master
Choose a base branch
from
Open

GH-404: Clarify dictionary fallback encoding #609

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions Encodings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,17 @@ endian integer, followed by the bytes.
The dictionary encoding builds a dictionary of values encountered in a given column. The
dictionary will be stored in a dictionary page per column chunk. The values are stored as integers
using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size
or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
written first, before the data pages of the column chunk.
or number of distinct values, the writer may stop using dictionary encoding and fall back to another
valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded
in the data page header's `encoding` field; readers must use that field to determine whether each
data page is dictionary encoded or uses another encoding. The dictionary page is written first,
before the data pages of the column chunk.
Comment on lines -80 to +84

@divjotarora divjotarora Aug 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new wording implies that the first N pages of a column chunk can be dictionary-encoded and all other pages after need to be non-dictionary (specifically the "fall back to another
valid encoding for subsequent data pages"). However, IIUC it is valid to interleave non-dict- and dict-encoded pages within a column chunk as long as only one dictionary is used. If you agree, maybe simpler wording could be something like:

Within a column chunk, the dictionary page is written first, before the data pages of the column chunk. After the dictionary page, data pages can use dictionary encodings or another valid encoding for the column's data type. Dictionary and non-dictionary encoded data pages may be interleaved. Readers must use each page header's encoding field to determine how the page should be decoded.


Dictionary page format: the entries in the dictionary using the [plain](#PLAIN) encoding.

Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
followed by the values encoded using the RLE/Bit-Packing described above (with the given bit width).
Dictionary-encoded data page format: the bit width used to encode the entry ids stored as 1 byte
(max bit width = 32), followed by the values encoded using the RLE/Bit-Packing described above
(with the given bit width).

Using the `PLAIN_DICTIONARY` enum value is deprecated, use `RLE_DICTIONARY`
in a data page and `PLAIN` in a dictionary page for new Parquet files.
Expand Down