From 54f175afa3032dc9332faf8a065d2a7d01b42f29 Mon Sep 17 00:00:00 2001 From: Dave Bunten Date: Sat, 22 Aug 2026 12:53:40 -0600 Subject: [PATCH] GH-404: Clarify dictionary fallback encoding The dictionary encoding docs currently say that writers fall back to PLAIN encoding when a dictionary gets too large. That is not always true. A writer can stop using dictionary encoding and write later data pages with another valid encoding. The actual encoding is already stored in each data page header, so readers should use that field instead of assuming the fallback is PLAIN. Update the wording to describe that behavior, and clarify that the dictionary data page layout only applies to dictionary-encoded pages. --- Encodings.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/Encodings.md b/Encodings.md index 0a07100a..2a9778b2 100644 --- a/Encodings.md +++ b/Encodings.md @@ -77,13 +77,17 @@ endian integer, followed by the bytes. The dictionary encoding builds a dictionary of values encountered in a given column. The dictionary will be stored in a dictionary page per column chunk. The values are stored as integers using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size -or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is -written first, before the data pages of the column chunk. +or number of distinct values, the writer may stop using dictionary encoding and fall back to another +valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded +in the data page header's `encoding` field; readers must use that field to determine whether each +data page is dictionary encoded or uses another encoding. The dictionary page is written first, +before the data pages of the column chunk. Dictionary page format: the entries in the dictionary using the [plain](#PLAIN) encoding. -Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32), -followed by the values encoded using the RLE/Bit-Packing described above (with the given bit width). +Dictionary-encoded data page format: the bit width used to encode the entry ids stored as 1 byte +(max bit width = 32), followed by the values encoded using the RLE/Bit-Packing described above +(with the given bit width). Using the `PLAIN_DICTIONARY` enum value is deprecated, use `RLE_DICTIONARY` in a data page and `PLAIN` in a dictionary page for new Parquet files.