From 53dc52bf86d41869bb0c5ed691af669dd29ec129 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Wed, 29 Jul 2026 20:02:41 +0300 Subject: [PATCH 01/19] Specify compression for FILE self-references --- LogicalTypes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/LogicalTypes.md b/LogicalTypes.md index fae874c0..7fb4f079 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -756,6 +756,10 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. +The bytes referenced by a self-reference are compressed with the `CompressionCodec` +of the inline column chunk's `ColumnMetaData`. This does not apply to external +references. + Parquet files containing self-references must not use Parquet modular encryption. Self-referenced byte ranges are not Parquet encryption modules and therefore cannot be encrypted or authenticated independently. Encryption of external files referenced From 7b729c6a42766790fcf780000d338a710d9aabbd Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 02:30:48 +0300 Subject: [PATCH 02/19] Define FILE self-reference compression semantics --- LogicalTypes.md | 37 +++++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 7fb4f079..006dfd9c 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -688,11 +688,11 @@ external reference (`uri` set). `offset` must not be < 0. ##### size -The byte length of the referenced data. Must be zero or a positive integer if set; a -value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), -in which case the range runs to the end of the referenced file. Because a self-reference -always sets `offset`, it always sets `size` as well. +The byte length of the stored representation. Must be zero or a positive integer if set. +A value of 0 indicates an empty stored representation. `size` must be set whenever +`offset` is set. It may be omitted only for a whole-file external reference (`uri` set, +`offset` not set), in which case the range runs to the end of the referenced file. +Because a self-reference always sets `offset`, it always sets `size` as well. ##### content_type @@ -756,9 +756,30 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -The bytes referenced by a self-reference are compressed with the `CompressionCodec` -of the inline column chunk's `ColumnMetaData`. This does not apply to external -references. +A schema that permits self-references must include the `inline` field and encode it +using Data Page V2. + +Each self-reference corresponds positionally to a value in the `inline` column. Its +compression state is inherited from the Data Page V2 containing that position: + +* If `is_compressed` is true, the referenced byte range is compressed independently + using the `CompressionCodec` of the `inline` column chunk. +* If `is_compressed` is false, the referenced byte range contains the resolved bytes + without compression. + +All self-references corresponding to positions in the same data page share the page's +compression decision. A writer that requires a different decision must begin a new +Data Page V2. + +Each compressed byte range is an independent compression block. Compression state is +not shared with the data page or with other referenced ranges. + +For a self-reference, `offset` and `size` identify the stored representation. When +compressed, `size` is the compressed byte length. The complete range is supplied to +the codec, and its decompressed output is the resolved value. + +`content_type` and `checksum` describe the resolved bytes after decompression. These +compression rules do not apply to external references. Parquet files containing self-references must not use Parquet modular encryption. Self-referenced byte ranges are not Parquet encryption modules and therefore cannot From f5eb8854ca10b9b990a197424e5b4a40d6654db5 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 09:34:48 +0300 Subject: [PATCH 03/19] Define inherited encryption for FILE self-references --- Encryption.md | 74 ++++++++++++++++++++++++++++--------------------- LogicalTypes.md | 44 +++++++++++++++++++---------- 2 files changed, 73 insertions(+), 45 deletions(-) diff --git a/Encryption.md b/Encryption.md index d3c8c9fa..9625772a 100644 --- a/Encryption.md +++ b/Encryption.md @@ -78,9 +78,9 @@ In a plaintext footer mode, the contents of the footer structure is visible and in order to verify its integrity. New footer fields keep an information about the file encryption algorithm and the footer signing key. -For encrypted columns, the following modules are always encrypted, with the same column key: -pages and page headers (both dictionary and data), column indexes, offset indexes, bloom filter -headers and bitsets. If the +For encrypted columns, the following modules are always encrypted, with the same column key: +pages and page headers (both dictionary and data), self-reference payloads, column indexes, +offset indexes, bloom filter headers and bitsets. If the column key is different from the footer encryption key, the column metadata is serialized separately and encrypted with the column key. In this case, the column metadata is also considered to be a module. @@ -135,11 +135,12 @@ related to the "uniqueness requirement of IVs and keys" (section 8 in the NIST s one IV is ever repeated, then the implementation may be vulnerable"*. *"Compliance with this requirement is crucial to the security of GCM"*. -The bulk of modules in a Parquet file are page headers and data pages. Therefore, one encryption -key shall not be used for more than 2^32 total module encryptions, as per the NIST specification. -Since each data page requires two module encryptions (header + data), this means in practice no -more than 2^31 pages per key. In Parquet files encrypted with multiple keys (footer and column -keys), the constraint on the number of invocations is applied to each key separately. +The bulk of modules in a Parquet file are page headers, data pages, and self-references. +Therefore, one encryption key shall not be used for more than 2^32 total module encryptions, +as per the NIST specification. Each data page requires two module encryptions (header and +data), and each self-reference requires one. In Parquet files encrypted with multiple keys +(footer and column keys), the constraint on the number of invocations is applied to each key +separately. When running in the context of a larger system, any particular Parquet writer implementation likely does not have sufficient context to enforce key invocation limits system-wide. Therefore, @@ -163,11 +164,11 @@ tag used to verify the ciphertext and AAD integrity. #### 4.2.2 AES_GCM_CTR_V1 -In this Parquet algorithm, all modules except pages are encrypted with the GCM cipher, as described -above. The pages are encrypted by the CTR cipher without padding. This makes it possible to encrypt/decrypt -the bulk of the data faster, while still verifying the metadata integrity and making -sure the file has not been replaced with a wrong version. However, tampering with the -page data might go unnoticed. The AES CTR cipher +In this Parquet algorithm, all modules except pages and self-references are encrypted with the +GCM cipher, as described above. Pages and self-references are encrypted by the CTR cipher +without padding. This makes it possible to encrypt/decrypt the bulk of the data faster, while +still verifying the metadata integrity and making sure the file has not been replaced with a +wrong version. However, tampering with page or self-reference data might go unnoticed. The AES CTR cipher must be implemented by a cryptographic provider according to the NIST SP 800-38A specification. In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext. IVs are comprised of @@ -256,7 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page and header only] page ordinal (2-byte short, little-endian) +5. [Data page, data page header, and self-reference only] page ordinal (2-byte short, little-endian) +6. [Self-reference only] value ordinal within the data page (4-byte integer, little-endian) The following module types are defined: @@ -270,28 +272,35 @@ The following module types are defined: * OffsetIndex (7) * BloomFilter Header (8) * BloomFilter Bitset (9) + * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal| -|----------------------|------------------|-------------|-------------------|----------------|-------------| -| Footer | yes | yes (0) | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | -| Data Page | yes | yes (2) | yes | yes | yes | -| Dictionary Page | yes | yes (3) | yes | yes | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Value ordinal | +|-----------------------|------------------|-------------|-------------------|----------------|--------------|---------------| +| Footer | yes | yes (0) | no | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | no | +| Data Page | yes | yes (2) | yes | yes | yes | no | +| Dictionary Page | yes | yes (3) | yes | yes | no | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | no | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | +| Self-Reference | yes | yes (10) | yes | yes | yes | yes | + +For a self-reference, the column ordinal and page ordinal are those of the corresponding +position in the `inline` column. The value ordinal is the zero-based position within that +page, counting every position represented by its repetition and definition levels, +including null positions. ## 5. File Format ### 5.1 Encrypted module serialization -All modules, except column pages, are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm, -the column pages are also encrypted with AES GCM. For each module, the GCM encryption +All modules, except column pages and self-references, are encrypted with the GCM cipher. In the +AES_GCM_V1 algorithm, column pages and self-references are also encrypted with AES GCM. For each module, the GCM encryption buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. @@ -299,14 +308,18 @@ the encryption buffer (a 4-byte little endian) is written to the output stream, |-----------------|------------------|------------------------------|----------------| -In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR. -For each page, the CTR encryption buffer is comprised of a nonce and ciphertext, +In the AES_GCM_CTR_V1 algorithm, column pages and self-references are encrypted with AES CTR. +For each page or self-reference, the CTR encryption buffer is comprised of a nonce and ciphertext, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. |length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) | |-----------------|------------------|------------------------------| +For a self-reference, `offset` points to the beginning of the 4-byte length and `size` +is the size of the complete encrypted module, including the length, nonce, ciphertext, +and GCM tag when present. + ### 5.2 Crypto structures Parquet file encryption algorithm is specified in a union of the following Thrift structures: @@ -540,4 +553,3 @@ algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data The throughput overhead of Parquet modular encryption depends on whether AES enciphering is done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers) instead of on much smaller individual data values causes AES to work at its maximal speed. - diff --git a/LogicalTypes.md b/LogicalTypes.md index 006dfd9c..da7a9b67 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -744,7 +744,7 @@ set: | - | set | set | set | external `uri`, `[offset, offset + size)` | | - | - | set | - | invalid | | - | - | - | set | invalid | -| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | +| - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | | - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an @@ -759,8 +759,11 @@ can be renamed or relocated as a single unit. A schema that permits self-references must include the `inline` field and encode it using Data Page V2. -Each self-reference corresponds positionally to a value in the `inline` column. Its -compression state is inherited from the Data Page V2 containing that position: +Each self-reference inherits the compression and encryption settings of the `inline` +column at the same position. The corresponding position is the position representing +the same `FILE` value in the `inline` column's repetition and definition level stream. + +The compression state is inherited from the Data Page V2 containing that position: * If `is_compressed` is true, the referenced byte range is compressed independently using the `CompressionCodec` of the `inline` column chunk. @@ -774,17 +777,30 @@ Data Page V2. Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. -For a self-reference, `offset` and `size` identify the stored representation. When -compressed, `size` is the compressed byte length. The complete range is supplied to -the codec, and its decompressed output is the resolved value. - -`content_type` and `checksum` describe the resolved bytes after decompression. These -compression rules do not apply to external references. - -Parquet files containing self-references must not use Parquet modular encryption. -Self-referenced byte ranges are not Parquet encryption modules and therefore cannot -be encrypted or authenticated independently. Encryption of external files referenced -by `uri` is outside the scope of the Parquet format. +For an unencrypted self-reference, `offset` and `size` identify either the independent +compressed block or the uncompressed bytes. For a compressed block, the complete range +is supplied to the codec, and its decompressed output is the resolved value. + +The encryption state and key are inherited from the `inline` column chunk. If the +column chunk is encrypted, each self-reference is encrypted independently using the +same column key and file encryption algorithm. Compression is applied before +encryption. If the column chunk is not encrypted, its self-references are not +encrypted. See [Parquet Modular Encryption](Encryption.md) for the encryption layout +and AAD construction. + +A self-reference identifies a stored representation, not necessarily the resolved +bytes. Consumers must use a Parquet reader to resolve a self-reference; copying +`[offset, offset + size)` directly may return compressed or encrypted data. To resolve +a self-reference, a reader: + +1. reads the stored representation identified by `offset` and `size`; +2. decrypts it when the corresponding `inline` column chunk is encrypted; +3. decompresses it when `is_compressed` is true for the corresponding Data Page V2; +4. returns the resulting bytes. + +`content_type` and `checksum` describe the resolved bytes after these transformations. +These compression and encryption rules do not apply to external references. Encryption +of external files referenced by `uri` is outside the scope of the Parquet format. #### Validation From 86479df673a307e69eac9cc1f6318747cff4001a Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 09:42:55 +0300 Subject: [PATCH 04/19] Avoid prescribing FILE page boundaries --- LogicalTypes.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index da7a9b67..257e3a95 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -770,10 +770,6 @@ The compression state is inherited from the Data Page V2 containing that positio * If `is_compressed` is false, the referenced byte range contains the resolved bytes without compression. -All self-references corresponding to positions in the same data page share the page's -compression decision. A writer that requires a different decision must begin a new -Data Page V2. - Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. From 3047a52fb64b0f0cde35fb2b4474f632064761eb Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 30 Jul 2026 19:14:37 +0300 Subject: [PATCH 05/19] Make FILE storage inheritance page-independent --- Encryption.md | 40 ++++++++++++++++++++-------------------- LogicalTypes.md | 19 ++++++++----------- 2 files changed, 28 insertions(+), 31 deletions(-) diff --git a/Encryption.md b/Encryption.md index 9625772a..733525c0 100644 --- a/Encryption.md +++ b/Encryption.md @@ -257,8 +257,8 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page, data page header, and self-reference only] page ordinal (2-byte short, little-endian) -6. [Self-reference only] value ordinal within the data page (4-byte integer, little-endian) +5. [Data page and data page header only] page ordinal (2-byte short, little-endian) +6. [Self-reference only] self-reference ordinal within the column chunk (8-byte integer, little-endian) The following module types are defined: @@ -275,24 +275,24 @@ The following module types are defined: * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Value ordinal | -|-----------------------|------------------|-------------|-------------------|----------------|--------------|---------------| -| Footer | yes | yes (0) | no | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | no | -| Data Page | yes | yes (2) | yes | yes | yes | no | -| Dictionary Page | yes | yes (3) | yes | yes | no | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | no | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | -| Self-Reference | yes | yes (10) | yes | yes | yes | yes | - -For a self-reference, the column ordinal and page ordinal are those of the corresponding -position in the `inline` column. The value ordinal is the zero-based position within that -page, counting every position represented by its repetition and definition levels, -including null positions. +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference ordinal | +|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------| +| Footer | yes | yes (0) | no | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | no | +| Data Page | yes | yes (2) | yes | yes | yes | no | +| Dictionary Page | yes | yes (3) | yes | yes | no | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | no | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | +| Self-Reference | yes | yes (10) | yes | yes | no | yes | + +For a self-reference, the column ordinal is that of the `inline` column. The +self-reference ordinal is the zero-based position representing the same `FILE` value +in that column chunk's repetition and definition level stream, including null +positions. It is derived by the reader and is not stored separately. diff --git a/LogicalTypes.md b/LogicalTypes.md index 257e3a95..e90b407e 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -756,19 +756,15 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. -A schema that permits self-references must include the `inline` field and encode it -using Data Page V2. +A schema that permits self-references must include the `inline` field. Each self-reference inherits the compression and encryption settings of the `inline` -column at the same position. The corresponding position is the position representing -the same `FILE` value in the `inline` column's repetition and definition level stream. +column chunk in the same row group. The corresponding position is the position +representing the same `FILE` value in the `inline` column's repetition and definition +level stream. -The compression state is inherited from the Data Page V2 containing that position: - -* If `is_compressed` is true, the referenced byte range is compressed independently - using the `CompressionCodec` of the `inline` column chunk. -* If `is_compressed` is false, the referenced byte range contains the resolved bytes - without compression. +Each referenced byte range is compressed independently using the `CompressionCodec` +of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. Each compressed byte range is an independent compression block. Compression state is not shared with the data page or with other referenced ranges. @@ -791,7 +787,8 @@ a self-reference, a reader: 1. reads the stored representation identified by `offset` and `size`; 2. decrypts it when the corresponding `inline` column chunk is encrypted; -3. decompresses it when `is_compressed` is true for the corresponding Data Page V2; +3. decompresses it using the `CompressionCodec` of the corresponding `inline` column + chunk, unless the codec is `UNCOMPRESSED`; 4. returns the resulting bytes. `content_type` and `checksum` describe the resolved bytes after these transformations. From 81f61ed74074d58794953ea330e64aee6ae879a1 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Fri, 31 Jul 2026 00:35:08 +0300 Subject: [PATCH 06/19] Align FILE resolution table --- LogicalTypes.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index e90b407e..f807da1d 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -735,17 +735,17 @@ only. A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set: -| `inline` | `uri` | `offset` | `size` | Resolves to | -|----------|-------|----------|--------|-------------------------------------------------------| -| set | - | - | - | the inline bytes | -| - | set | - | - | whole external file at `uri` | -| - | set | set | - | invalid | -| - | set | - | set | external `uri`, `[0, size)` | -| - | set | set | set | external `uri`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-----------------------------------------------------------------------| +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | | - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | -| - | - | - | - | nothing - invalid | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also From 5d7bad5df50b967387cebd83d130ecba7c10651d Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Wed, 5 Aug 2026 11:00:23 +0300 Subject: [PATCH 07/19] Address review feedback on FILE self-references Key the self-reference AAD on `offset` rather than a derived ordinal, so a reader can resolve a self-reference without decoding the pages it skips. State that the encryption buffer of a self-reference must fit the 4-byte length field, and that larger values must use an external reference. State that the decompressed size is not stored and how readers obtain it. State that an encrypted stored representation is bound to a single column chunk and must not be shared between column chunks. --- Encryption.md | 22 +++++++++++++++------- LogicalTypes.md | 18 +++++++++++++----- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/Encryption.md b/Encryption.md index 733525c0..e71d4283 100644 --- a/Encryption.md +++ b/Encryption.md @@ -258,7 +258,7 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) 5. [Data page and data page header only] page ordinal (2-byte short, little-endian) -6. [Self-reference only] self-reference ordinal within the column chunk (8-byte integer, little-endian) +6. [Self-reference only] offset of the self-reference within the file (8-byte integer, little-endian) The following module types are defined: @@ -275,7 +275,7 @@ The following module types are defined: * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference ordinal | +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference offset | |-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------| | Footer | yes | yes (0) | no | no | no | no | | ColumnMetaData | yes | yes (1) | yes | yes | no | no | @@ -289,10 +289,15 @@ The following module types are defined: | BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | | Self-Reference | yes | yes (10) | yes | yes | no | yes | -For a self-reference, the column ordinal is that of the `inline` column. The -self-reference ordinal is the zero-based position representing the same `FILE` value -in that column chunk's repetition and definition level stream, including null -positions. It is derived by the reader and is not stored separately. +For a self-reference, the column ordinal is that of the `inline` column and the +self-reference offset is the value of its `offset` field. Because `offset` locates the +stored representation within the file, it is available to a reader without counting +preceding values. A reader may therefore resolve a self-reference without decoding the +pages it skips. + +The AAD of a self-reference binds its stored representation to a single column chunk. A +writer must not share one encrypted byte range between column chunks; each column chunk +that references the same bytes stores its own representation. @@ -318,7 +323,10 @@ described in the Algorithms section. The length of the encryption buffer For a self-reference, `offset` points to the beginning of the 4-byte length and `size` is the size of the complete encrypted module, including the length, nonce, ciphertext, -and GCM tag when present. +and GCM tag when present. The encryption buffer of a self-reference must therefore fit +the 4-byte length field, which limits it to 2 GiB as for any other module. A value whose +encryption buffer would exceed this limit must be stored as an external reference (`uri`) +instead of a self-reference. ### 5.2 Crypto structures diff --git a/LogicalTypes.md b/LogicalTypes.md index f807da1d..00a010b3 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -759,9 +759,9 @@ can be renamed or relocated as a single unit. A schema that permits self-references must include the `inline` field. Each self-reference inherits the compression and encryption settings of the `inline` -column chunk in the same row group. The corresponding position is the position -representing the same `FILE` value in the `inline` column's repetition and definition -level stream. +column chunk of the same row group. These settings are properties of the column chunk, +so all self-references of a column chunk share them regardless of the page a value is +stored in. Each referenced byte range is compressed independently using the `CompressionCodec` of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. @@ -773,12 +773,20 @@ For an unencrypted self-reference, `offset` and `size` identify either the indep compressed block or the uncompressed bytes. For a compressed block, the complete range is supplied to the codec, and its decompressed output is the resolved value. +The decompressed size of a self-reference is not stored. Readers must rely on the +framing of the codec where it provides one, or decompress into a dynamically sized +buffer. A future revision of this specification may add an explicit decompressed size. + The encryption state and key are inherited from the `inline` column chunk. If the column chunk is encrypted, each self-reference is encrypted independently using the same column key and file encryption algorithm. Compression is applied before encryption. If the column chunk is not encrypted, its self-references are not -encrypted. See [Parquet Modular Encryption](Encryption.md) for the encryption layout -and AAD construction. +encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the +encrypted module; a value too large to store this way must use an external reference +(`uri`). An encrypted stored representation is bound to a single column chunk and must +not be shared between column chunks. See +[Parquet Modular Encryption](Encryption.md) for the encryption layout and AAD +construction. A self-reference identifies a stored representation, not necessarily the resolved bytes. Consumers must use a Parquet reader to resolve a self-reference; copying From c9e6381d385bec539ae88780f71e5f0e96595a1e Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 6 Aug 2026 18:27:07 +0300 Subject: [PATCH 08/19] Require size for FILE self-references Align the wording with the validation rules, which state the requirement normatively. Co-authored-by: Isaac --- LogicalTypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 00a010b3..a36f7c20 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -692,7 +692,7 @@ The byte length of the stored representation. Must be zero or a positive integer A value of 0 indicates an empty stored representation. `size` must be set whenever `offset` is set. It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), in which case the range runs to the end of the referenced file. -Because a self-reference always sets `offset`, it always sets `size` as well. +Because a self-reference always sets `offset`, it must always set `size` as well. ##### content_type From aac4d72cc24ce7d97ff54800126fa96aaa1680c7 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 6 Aug 2026 18:28:26 +0300 Subject: [PATCH 09/19] Address review feedback on FILE self-reference wording Describe self-reference storage as being anywhere in the file rather than in a page, since the referenced bytes are outside the page data. Drop the forward-looking note about a possible decompressed size field. State that `offset` and `size` cover the encrypted module for an encrypted self-reference, as already stated for the unencrypted case. Describe rather than mandate how the stored bytes are interpreted. Co-authored-by: Isaac --- LogicalTypes.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index a36f7c20..70cfefc7 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -760,8 +760,8 @@ A schema that permits self-references must include the `inline` field. Each self-reference inherits the compression and encryption settings of the `inline` column chunk of the same row group. These settings are properties of the column chunk, -so all self-references of a column chunk share them regardless of the page a value is -stored in. +so all self-references of a column chunk share them regardless of where in the file the +referenced value is stored. Each referenced byte range is compressed independently using the `CompressionCodec` of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. @@ -775,23 +775,23 @@ is supplied to the codec, and its decompressed output is the resolved value. The decompressed size of a self-reference is not stored. Readers must rely on the framing of the codec where it provides one, or decompress into a dynamically sized -buffer. A future revision of this specification may add an explicit decompressed size. +buffer. The encryption state and key are inherited from the `inline` column chunk. If the column chunk is encrypted, each self-reference is encrypted independently using the same column key and file encryption algorithm. Compression is applied before encryption. If the column chunk is not encrypted, its self-references are not -encrypted. An encrypted self-reference is limited to 2 GiB by the length field of the -encrypted module; a value too large to store this way must use an external reference -(`uri`). An encrypted stored representation is bound to a single column chunk and must -not be shared between column chunks. See -[Parquet Modular Encryption](Encryption.md) for the encryption layout and AAD -construction. +encrypted. For an encrypted self-reference, `offset` and `size` identify the encrypted +module, so `size` is the size after compression and encryption. An encrypted +self-reference is limited to 2 GiB by the length field of the encrypted module; a value +too large to store this way must use an external reference (`uri`). An encrypted stored +representation is bound to a single column chunk and must not be shared between column +chunks. See [Parquet Modular Encryption](Encryption.md) for the encryption layout and +AAD construction. A self-reference identifies a stored representation, not necessarily the resolved -bytes. Consumers must use a Parquet reader to resolve a self-reference; copying -`[offset, offset + size)` directly may return compressed or encrypted data. To resolve -a self-reference, a reader: +bytes. Copying `[offset, offset + size)` directly may return compressed or encrypted +data. To resolve a self-reference, a reader: 1. reads the stored representation identified by `offset` and `size`; 2. decrypts it when the corresponding `inline` column chunk is encrypted; From ad3d1fab58c434d8aa5a227ec9e2896a543d19ef Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Fri, 7 Aug 2026 15:51:52 +0300 Subject: [PATCH 10/19] Point to the encrypted module layout for self-reference size The previous wording described `size` as the size after compression and encryption, which omits the length prefix and nonce that the range also covers. Reference the layout in Encryption.md so the extent is defined in one place. Co-authored-by: Isaac --- LogicalTypes.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 70cfefc7..b9d41110 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -782,12 +782,13 @@ column chunk is encrypted, each self-reference is encrypted independently using same column key and file encryption algorithm. Compression is applied before encryption. If the column chunk is not encrypted, its self-references are not encrypted. For an encrypted self-reference, `offset` and `size` identify the encrypted -module, so `size` is the size after compression and encryption. An encrypted -self-reference is limited to 2 GiB by the length field of the encrypted module; a value -too large to store this way must use an external reference (`uri`). An encrypted stored -representation is bound to a single column chunk and must not be shared between column -chunks. See [Parquet Modular Encryption](Encryption.md) for the encryption layout and -AAD construction. +module, whose exact extent is defined in +[Encrypted module serialization](Encryption.md#51-encrypted-module-serialization). An +encrypted self-reference is limited to 2 GiB by the length field of the encrypted +module; a value too large to store this way must use an external reference (`uri`). An +encrypted stored representation is bound to a single column chunk and must not be shared +between column chunks. See [Parquet Modular Encryption](Encryption.md) for the +encryption layout and AAD construction. A self-reference identifies a stored representation, not necessarily the resolved bytes. Copying `[offset, offset + size)` directly may return compressed or encrypted From d022d27795d84a02421eb2ae1a6de688b9120518 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Tue, 18 Aug 2026 16:33:33 -0700 Subject: [PATCH 11/19] Remove FILE self-references Drop self-references from the FILE specification. `offset` and `size` now apply only to data referenced by `uri`, and a byte range within the current file cannot be referenced. Storing a large value elsewhere in the same file is better served by out-of-band pages, which can hold the value outside the page data without the FILE type having to define its own compression, encryption, and addressing rules. That will be proposed separately. This reverts the compression and encryption semantics added earlier on this branch, so Encryption.md returns to its state before it. Co-authored-by: Isaac --- Encryption.md | 82 +++++++++++++-------------------- LogicalTypes.md | 118 ++++++++++++++---------------------------------- 2 files changed, 64 insertions(+), 136 deletions(-) diff --git a/Encryption.md b/Encryption.md index e71d4283..d3c8c9fa 100644 --- a/Encryption.md +++ b/Encryption.md @@ -78,9 +78,9 @@ In a plaintext footer mode, the contents of the footer structure is visible and in order to verify its integrity. New footer fields keep an information about the file encryption algorithm and the footer signing key. -For encrypted columns, the following modules are always encrypted, with the same column key: -pages and page headers (both dictionary and data), self-reference payloads, column indexes, -offset indexes, bloom filter headers and bitsets. If the +For encrypted columns, the following modules are always encrypted, with the same column key: +pages and page headers (both dictionary and data), column indexes, offset indexes, bloom filter +headers and bitsets. If the column key is different from the footer encryption key, the column metadata is serialized separately and encrypted with the column key. In this case, the column metadata is also considered to be a module. @@ -135,12 +135,11 @@ related to the "uniqueness requirement of IVs and keys" (section 8 in the NIST s one IV is ever repeated, then the implementation may be vulnerable"*. *"Compliance with this requirement is crucial to the security of GCM"*. -The bulk of modules in a Parquet file are page headers, data pages, and self-references. -Therefore, one encryption key shall not be used for more than 2^32 total module encryptions, -as per the NIST specification. Each data page requires two module encryptions (header and -data), and each self-reference requires one. In Parquet files encrypted with multiple keys -(footer and column keys), the constraint on the number of invocations is applied to each key -separately. +The bulk of modules in a Parquet file are page headers and data pages. Therefore, one encryption +key shall not be used for more than 2^32 total module encryptions, as per the NIST specification. +Since each data page requires two module encryptions (header + data), this means in practice no +more than 2^31 pages per key. In Parquet files encrypted with multiple keys (footer and column +keys), the constraint on the number of invocations is applied to each key separately. When running in the context of a larger system, any particular Parquet writer implementation likely does not have sufficient context to enforce key invocation limits system-wide. Therefore, @@ -164,11 +163,11 @@ tag used to verify the ciphertext and AAD integrity. #### 4.2.2 AES_GCM_CTR_V1 -In this Parquet algorithm, all modules except pages and self-references are encrypted with the -GCM cipher, as described above. Pages and self-references are encrypted by the CTR cipher -without padding. This makes it possible to encrypt/decrypt the bulk of the data faster, while -still verifying the metadata integrity and making sure the file has not been replaced with a -wrong version. However, tampering with page or self-reference data might go unnoticed. The AES CTR cipher +In this Parquet algorithm, all modules except pages are encrypted with the GCM cipher, as described +above. The pages are encrypted by the CTR cipher without padding. This makes it possible to encrypt/decrypt +the bulk of the data faster, while still verifying the metadata integrity and making +sure the file has not been replaced with a wrong version. However, tampering with the +page data might go unnoticed. The AES CTR cipher must be implemented by a cryptographic provider according to the NIST SP 800-38A specification. In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext. IVs are comprised of @@ -257,8 +256,7 @@ Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenat 2. [All modules] module type (1 byte) 3. [All modules except footer] row group ordinal (2-byte short, little-endian) 4. [All modules except footer] column ordinal (2-byte short, little-endian) -5. [Data page and data page header only] page ordinal (2-byte short, little-endian) -6. [Self-reference only] offset of the self-reference within the file (8-byte integer, little-endian) +5. [Data page and header only] page ordinal (2-byte short, little-endian) The following module types are defined: @@ -272,40 +270,28 @@ The following module types are defined: * OffsetIndex (7) * BloomFilter Header (8) * BloomFilter Bitset (9) - * Self-Reference (10) -| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal | Self-reference offset | -|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------| -| Footer | yes | yes (0) | no | no | no | no | -| ColumnMetaData | yes | yes (1) | yes | yes | no | no | -| Data Page | yes | yes (2) | yes | yes | yes | no | -| Dictionary Page | yes | yes (3) | yes | yes | no | no | -| Data Page Header | yes | yes (4) | yes | yes | yes | no | -| Dictionary Page Header| yes | yes (5) | yes | yes | no | no | -| ColumnIndex | yes | yes (6) | yes | yes | no | no | -| OffsetIndex | yes | yes (7) | yes | yes | no | no | -| BloomFilter Header | yes | yes (8) | yes | yes | no | no | -| BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | -| Self-Reference | yes | yes (10) | yes | yes | no | yes | - -For a self-reference, the column ordinal is that of the `inline` column and the -self-reference offset is the value of its `offset` field. Because `offset` locates the -stored representation within the file, it is available to a reader without counting -preceding values. A reader may therefore resolve a self-reference without decoding the -pages it skips. - -The AAD of a self-reference binds its stored representation to a single column chunk. A -writer must not share one encrypted byte range between column chunks; each column chunk -that references the same bytes stores its own representation. +| | Internal File ID | Module type | Row group ordinal | Column ordinal | Page ordinal| +|----------------------|------------------|-------------|-------------------|----------------|-------------| +| Footer | yes | yes (0) | no | no | no | +| ColumnMetaData | yes | yes (1) | yes | yes | no | +| Data Page | yes | yes (2) | yes | yes | yes | +| Dictionary Page | yes | yes (3) | yes | yes | no | +| Data Page Header | yes | yes (4) | yes | yes | yes | +| Dictionary Page Header| yes | yes (5) | yes | yes | no | +| ColumnIndex | yes | yes (6) | yes | yes | no | +| OffsetIndex | yes | yes (7) | yes | yes | no | +| BloomFilter Header | yes | yes (8) | yes | yes | no | +| BloomFilter Bitset | yes | yes (9) | yes | yes | no | ## 5. File Format ### 5.1 Encrypted module serialization -All modules, except column pages and self-references, are encrypted with the GCM cipher. In the -AES_GCM_V1 algorithm, column pages and self-references are also encrypted with AES GCM. For each module, the GCM encryption +All modules, except column pages, are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm, +the column pages are also encrypted with AES GCM. For each module, the GCM encryption buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. @@ -313,21 +299,14 @@ the encryption buffer (a 4-byte little endian) is written to the output stream, |-----------------|------------------|------------------------------|----------------| -In the AES_GCM_CTR_V1 algorithm, column pages and self-references are encrypted with AES CTR. -For each page or self-reference, the CTR encryption buffer is comprised of a nonce and ciphertext, +In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR. +For each page, the CTR encryption buffer is comprised of a nonce and ciphertext, described in the Algorithms section. The length of the encryption buffer (a 4-byte little endian) is written to the output stream, followed by the buffer itself. |length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) | |-----------------|------------------|------------------------------| -For a self-reference, `offset` points to the beginning of the 4-byte length and `size` -is the size of the complete encrypted module, including the length, nonce, ciphertext, -and GCM tag when present. The encryption buffer of a self-reference must therefore fit -the 4-byte length field, which limits it to 2 GiB as for any other module. A value whose -encryption buffer would exceed this limit must be stored as an external reference (`uri`) -instead of a self-reference. - ### 5.2 Crypto structures Parquet file encryption algorithm is specified in a union of the following Thrift structures: @@ -561,3 +540,4 @@ algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data The throughput overhead of Parquet modular encryption depends on whether AES enciphering is done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers) instead of on much smaller individual data values causes AES to work at its maximal speed. + diff --git a/LogicalTypes.md b/LogicalTypes.md index b9d41110..aaf61c7e 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -639,10 +639,10 @@ are found during reading, they must be ignored. ### FILE -`FILE` annotates a group that represents a reference to a range of bytes, which may -be stored inline in the value, elsewhere within the current file, or in an external file. It -is intended for use cases such as storing file inventories, manifests, and unstructured -data references (e.g., images or audio files stored in object storage). +`FILE` annotates a group that represents a reference to a range of bytes, which may be +stored inline in the value or in an external file. It is intended for use cases such as +storing file inventories, manifests, and unstructured data references (e.g., images or +audio files stored in object storage). The annotated group may contain the following fields, identified by name case sensitively, not by field order. Field IDs, if they exist, may also be used for projection. Every field @@ -675,24 +675,21 @@ when it is absent from the group, or is present but null or empty. A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`). The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top -of the user-provided data. If `uri` is not set, the value refers to the current file -(a self-reference). +of the user-provided data. ##### offset A byte offset indicating the start of the byte range within the referenced data. If not set, readers must treat the value as 0. If set and non-zero, readers must seek to this offset to retrieve the referenced data. -`offset` must be set for a self-reference (`uri` not set); it is optional for an -external reference (`uri` set). `offset` must not be < 0. +`offset` may only be set together with `uri`. `offset` must not be < 0. ##### size -The byte length of the stored representation. Must be zero or a positive integer if set. -A value of 0 indicates an empty stored representation. `size` must be set whenever -`offset` is set. It may be omitted only for a whole-file external reference (`uri` set, -`offset` not set), in which case the range runs to the end of the referenced file. -Because a self-reference always sets `offset`, it must always set `size` as well. +The byte length of the referenced data. Must be zero or a positive integer if set; a +value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. +It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), +in which case the range runs to the end of the referenced file. ##### content_type @@ -735,84 +732,35 @@ only. A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set: -| `inline` | `uri` | `offset` | `size` | Resolves to | -|----------|-------|----------|--------|-----------------------------------------------------------------------| -| set | - | - | - | the inline bytes | -| - | set | - | - | whole external file at `uri` | -| - | set | set | - | invalid | -| - | set | - | set | external `uri`, `[0, size)` | -| - | set | set | set | external `uri`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | -| - | - | set | set | stored bytes in this file, `[offset, offset + size)` (self-reference) | -| - | - | - | - | nothing - invalid | +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-------------------------------------------| +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | +| - | - | set | set | invalid | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an -explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also -`size`. `size` may be omitted only for a whole-file external reference, where the range -runs to the end of the referenced file. - -A self-reference points within the same Parquet file using `offset` and `size` (both -required). A self-reference is when `uri` is not set. A file containing self-references -can be renamed or relocated as a single unit. - -A schema that permits self-references must include the `inline` field. - -Each self-reference inherits the compression and encryption settings of the `inline` -column chunk of the same row group. These settings are properties of the column chunk, -so all self-references of a column chunk share them regardless of where in the file the -referenced value is stored. - -Each referenced byte range is compressed independently using the `CompressionCodec` -of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes uncompressed. - -Each compressed byte range is an independent compression block. Compression state is -not shared with the data page or with other referenced ranges. - -For an unencrypted self-reference, `offset` and `size` identify either the independent -compressed block or the uncompressed bytes. For a compressed block, the complete range -is supplied to the codec, and its decompressed output is the resolved value. - -The decompressed size of a self-reference is not stored. Readers must rely on the -framing of the codec where it provides one, or decompress into a dynamically sized -buffer. - -The encryption state and key are inherited from the `inline` column chunk. If the -column chunk is encrypted, each self-reference is encrypted independently using the -same column key and file encryption algorithm. Compression is applied before -encryption. If the column chunk is not encrypted, its self-references are not -encrypted. For an encrypted self-reference, `offset` and `size` identify the encrypted -module, whose exact extent is defined in -[Encrypted module serialization](Encryption.md#51-encrypted-module-serialization). An -encrypted self-reference is limited to 2 GiB by the length field of the encrypted -module; a value too large to store this way must use an external reference (`uri`). An -encrypted stored representation is bound to a single column chunk and must not be shared -between column chunks. See [Parquet Modular Encryption](Encryption.md) for the -encryption layout and AAD construction. - -A self-reference identifies a stored representation, not necessarily the resolved -bytes. Copying `[offset, offset + size)` directly may return compressed or encrypted -data. To resolve a self-reference, a reader: - -1. reads the stored representation identified by `offset` and `size`; -2. decrypts it when the corresponding `inline` column chunk is encrypted; -3. decompresses it using the `CompressionCodec` of the corresponding `inline` column - chunk, unless the codec is `UNCOMPRESSED`; -4. returns the resulting bytes. - -`content_type` and `checksum` describe the resolved bytes after these transformations. -These compression and encryption rules do not apply to external references. Encryption -of external files referenced by `uri` is outside the scope of the Parquet format. +explicit `size`. `size` may be omitted only for a whole-file external reference, where +the range runs to the end of the referenced file. A byte range within the current file +cannot be referenced: `offset` and `size` apply only to data referenced by `uri`. + +Encryption of external files referenced by `uri` is outside the scope of the Parquet +format. #### Validation -* A value must resolve to some referenced data. It resolves only if `inline`, `uri`, or - `offset` is set; if none of them are set, the value does not resolve and is invalid, even - if `size` is set. -* A self-reference (`uri` not set) must set `offset`. A value with neither `uri` nor - `offset` set (and not `inline`) does not resolve and is invalid. +* A value must resolve to some referenced data. It resolves only if `inline` or `uri` is + set; if neither is set, the value does not resolve and is invalid, even if `offset` or + `size` is set. +* `offset` may only be set together with `uri`. A value that sets `offset` without `uri` + (and not `inline`) does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` - is invalid. Because a self-reference must set `offset`, it must also set `size`. + is invalid. * If `inline` is set, it supplies the bytes for readers; producers may treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. From f4b79fed8e0fddb54bda95409757551db1b3ab2a Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Wed, 19 Aug 2026 08:32:10 -0700 Subject: [PATCH 12/19] Clarify FILE reference forms after removing self-references Resolve `uri` uniformly as an external reference, including when it names the file that contains it, rather than forbidding a reference to the current file, which a writer or reader cannot check. Allow `inline` together with the locator fields, which must denote the same bytes, and record the locator as provenance. State that the fields of a `FILE` group are ordinary columns and are encoded, compressed, and encrypted like any other column. Co-authored-by: Isaac --- LogicalTypes.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index aaf61c7e..98618c11 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -724,8 +724,10 @@ object-store eTag for the whole file referenced by `uri`. ##### inline The referenced bytes stored inline in the value. If `inline` is set, it supplies the -bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance -only. +bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance only. +Both representations must denote the same bytes: a locator set alongside `inline` records +where those bytes came from, and must not be a partial or otherwise different +representation of the value. #### Resolution @@ -734,7 +736,7 @@ set: | `inline` | `uri` | `offset` | `size` | Resolves to | |----------|-------|----------|--------|-------------------------------------------| -| set | - | - | - | the inline bytes | +| set | any | any | any | the inline bytes | | - | set | - | - | whole external file at `uri` | | - | set | set | - | invalid | | - | set | - | set | external `uri`, `[0, size)` | @@ -746,11 +748,18 @@ set: `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. `size` may be omitted only for a whole-file external reference, where -the range runs to the end of the referenced file. A byte range within the current file -cannot be referenced: `offset` and `size` apply only to data referenced by `uri`. +the range runs to the end of the referenced file. `offset` and `size` apply only to data +referenced by `uri`; there is no form that addresses a byte range in the current file +directly. + +A `uri` is always resolved as an external reference, even when it names the file that +contains it. Parquet applies no compression or encryption of its own to the referenced +bytes, and a reference remains the writer's responsibility if the file is copied or +renamed. Encryption of external files referenced by `uri` is outside the scope of the Parquet -format. +format. The fields of a `FILE`-annotated group are ordinary columns and are encoded, +compressed, and encrypted like any other column, `inline` included. #### Validation @@ -758,11 +767,12 @@ format. set; if neither is set, the value does not resolve and is invalid, even if `offset` or `size` is set. * `offset` may only be set together with `uri`. A value that sets `offset` without `uri` - (and not `inline`) does not resolve and is invalid. + does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. -* If `inline` is set, it supplies the bytes for readers; producers may treat `inline` and the - locator fields as mutually exclusive. +* If `inline` is set, it supplies the bytes for readers, and any locator fields that are + also set must denote the same bytes. Producers may treat `inline` and the locator + fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. From 5facb7f857509928345099135e060a80f64535cc Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Wed, 19 Aug 2026 09:18:32 -0700 Subject: [PATCH 13/19] State that inline and a locator resolve identically A reader that projects the locator fields without `inline` cannot tell whether `inline` was set, so resolving from the locator has to be a valid path. Say so, and say that the value is invalid when the two disagree, without requiring readers to detect it. Co-authored-by: Isaac --- LogicalTypes.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 98618c11..cc3bcad6 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -725,9 +725,10 @@ object-store eTag for the whole file referenced by `uri`. The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance only. -Both representations must denote the same bytes: a locator set alongside `inline` records -where those bytes came from, and must not be a partial or otherwise different -representation of the value. +Both representations must denote the same bytes, so a reader may resolve the value from +either and obtain the same result; reading `inline` requires no external access and is +the cheaper path. A locator set alongside `inline` records where those bytes came from, +and must not be a partial or otherwise different representation of the value. #### Resolution @@ -736,7 +737,7 @@ set: | `inline` | `uri` | `offset` | `size` | Resolves to | |----------|-------|----------|--------|-------------------------------------------| -| set | any | any | any | the inline bytes | +| set | any | any | any | the inline bytes (same as any locator) | | - | set | - | - | whole external file at `uri` | | - | set | set | - | invalid | | - | set | - | set | external `uri`, `[0, size)` | @@ -770,9 +771,10 @@ compressed, and encrypted like any other column, `inline` included. does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. -* If `inline` is set, it supplies the bytes for readers, and any locator fields that are - also set must denote the same bytes. Producers may treat `inline` and the locator - fields as mutually exclusive. +* If `inline` and a locator are both set, they must denote the same bytes, and a reader + may resolve the value from either. If they disagree the value is invalid; a reader is + not required to detect this and may return the bytes of either representation. + Producers may treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. From ed8bb0bcf289389373b0db52f0ee6f302d314e69 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 20 Aug 2026 09:08:03 -0700 Subject: [PATCH 14/19] Keep locator rules when inline is set, and update the Thrift comment The resolution table allowed any locator combination alongside `inline`, which contradicted the validation rules: `offset` still requires `uri` and `size`. Mark those cells as any combination that is valid on its own. Update the FileType comment in parquet.thrift, which still described bytes stored elsewhere in this file. Co-authored-by: Isaac --- LogicalTypes.md | 12 ++++++++---- src/main/thrift/parquet.thrift | 3 +-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index cc3bcad6..65f8b32b 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -640,9 +640,9 @@ are found during reading, they must be ignored. ### FILE `FILE` annotates a group that represents a reference to a range of bytes, which may be -stored inline in the value or in an external file. It is intended for use cases such as -storing file inventories, manifests, and unstructured data references (e.g., images or -audio files stored in object storage). +stored inline or in an external file. It is intended for use cases such as storing file +inventories, manifests, and unstructured data references (e.g., images or audio files +stored in object storage). The annotated group may contain the following fields, identified by name case sensitively, not by field order. Field IDs, if they exist, may also be used for projection. Every field @@ -737,7 +737,7 @@ set: | `inline` | `uri` | `offset` | `size` | Resolves to | |----------|-------|----------|--------|-------------------------------------------| -| set | any | any | any | the inline bytes (same as any locator) | +| set | † | † | † | the inline bytes (same as any locator) | | - | set | - | - | whole external file at `uri` | | - | set | set | - | invalid | | - | set | - | set | external `uri`, `[0, size)` | @@ -747,6 +747,10 @@ set: | - | - | set | set | invalid | | - | - | - | - | nothing - invalid | +† Any combination of the locator fields that is valid on its own. A locator set +alongside `inline` must satisfy the same rules as one used on its own, so `offset` +requires `uri` and `size`. + `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. `size` may be omitted only for a whole-file external reference, where the range runs to the end of the referenced file. `offset` and `size` apply only to data diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index a5922bb3..07597a59 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -472,8 +472,7 @@ struct GeographyType { * File logical type annotation * * Annotates a group that represents a reference to a file, or to a range of - * bytes that may be stored inline, elsewhere in this file, or in an external - * file. + * bytes that may be stored inline or in an external file. * * See LogicalTypes.md for details. */ From 92203686e631a72b6ab46d9d3fc1c12001141aaf Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 20 Aug 2026 09:14:19 -0700 Subject: [PATCH 15/19] Give inline its own column in the FILE resolution table The locator combinations and `inline` are independent, so enumerating them together needed a row per pair and still did not say what `inline` means. Move `inline` to a single cell spanning the table, which states once that it may accompany any combination and holds the same bytes that resolving the rest of the row returns. State that `inline` resolves a value even when its locator is invalid. Co-authored-by: Isaac --- LogicalTypes.md | 77 +++++++++++++++++++++++++++++++++++-------------- 1 file changed, 55 insertions(+), 22 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 65f8b32b..f8ee7655 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -732,24 +732,55 @@ and must not be a partial or otherwise different representation of the value. #### Resolution -A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are -set: - -| `inline` | `uri` | `offset` | `size` | Resolves to | -|----------|-------|----------|--------|-------------------------------------------| -| set | † | † | † | the inline bytes (same as any locator) | -| - | set | - | - | whole external file at `uri` | -| - | set | set | - | invalid | -| - | set | - | set | external `uri`, `[0, size)` | -| - | set | set | set | external `uri`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | -| - | - | set | set | invalid | -| - | - | - | - | nothing - invalid | - -† Any combination of the locator fields that is valid on its own. A locator set -alongside `inline` must satisfy the same rules as one used on its own, so `offset` -requires `uri` and `size`. +A value resolves to bytes based on which of `uri`, `offset`, and `size` are set. The +`inline` field is independent of that combination and is shown in the last column: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
urioffsetsizeResolves toinline
set--whole external file at uriMay be set in any of these combinations. When set, it holds the + same bytes that resolving the rest of the row would return, so a reader may use + either and obtain the same result. Reading inline requires no + external access and is the cheaper path. A value whose locator fields are + invalid still resolves from inline.
setset-invalid
set-setexternal uri, [0, size)
setsetsetexternal uri, [offset, offset + size)
-set-invalid
--setinvalid
-setsetinvalid
---nothing - invalid, unless inline is set
`size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. `size` may be omitted only for a whole-file external reference, where @@ -771,10 +802,12 @@ compressed, and encrypted like any other column, `inline` included. * A value must resolve to some referenced data. It resolves only if `inline` or `uri` is set; if neither is set, the value does not resolve and is invalid, even if `offset` or `size` is set. -* `offset` may only be set together with `uri`. A value that sets `offset` without `uri` - does not resolve and is invalid. -* `size` must be set whenever `offset` is set. A value that sets `offset` without `size` - is invalid. +* `offset` may only be set together with `uri`. A locator that sets `offset` without + `uri` does not resolve and is invalid. +* `size` must be set whenever `offset` is set. A locator that sets `offset` without + `size` is invalid. +* An invalid locator does not invalidate a value whose `inline` is set, because `inline` + resolves the value on its own. * If `inline` and a locator are both set, they must denote the same bytes, and a reader may resolve the value from either. If they disagree the value is invalid; a reader is not required to detect this and may return the bytes of either representation. From d7a713767a0ca0657ba0977698b1161c5b4e1454 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 20 Aug 2026 09:18:30 -0700 Subject: [PATCH 16/19] Revert "Give inline its own column in the FILE resolution table" This reverts commit 92203686e631a72b6ab46d9d3fc1c12001141aaf. --- LogicalTypes.md | 77 ++++++++++++++----------------------------------- 1 file changed, 22 insertions(+), 55 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index f8ee7655..65f8b32b 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -732,55 +732,24 @@ and must not be a partial or otherwise different representation of the value. #### Resolution -A value resolves to bytes based on which of `uri`, `offset`, and `size` are set. The -`inline` field is independent of that combination and is shown in the last column: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
urioffsetsizeResolves toinline
set--whole external file at uriMay be set in any of these combinations. When set, it holds the - same bytes that resolving the rest of the row would return, so a reader may use - either and obtain the same result. Reading inline requires no - external access and is the cheaper path. A value whose locator fields are - invalid still resolves from inline.
setset-invalid
set-setexternal uri, [0, size)
setsetsetexternal uri, [offset, offset + size)
-set-invalid
--setinvalid
-setsetinvalid
---nothing - invalid, unless inline is set
+A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are +set: + +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-------------------------------------------| +| set | † | † | † | the inline bytes (same as any locator) | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | +| - | - | set | set | invalid | +| - | - | - | - | nothing - invalid | + +† Any combination of the locator fields that is valid on its own. A locator set +alongside `inline` must satisfy the same rules as one used on its own, so `offset` +requires `uri` and `size`. `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. `size` may be omitted only for a whole-file external reference, where @@ -802,12 +771,10 @@ compressed, and encrypted like any other column, `inline` included. * A value must resolve to some referenced data. It resolves only if `inline` or `uri` is set; if neither is set, the value does not resolve and is invalid, even if `offset` or `size` is set. -* `offset` may only be set together with `uri`. A locator that sets `offset` without - `uri` does not resolve and is invalid. -* `size` must be set whenever `offset` is set. A locator that sets `offset` without - `size` is invalid. -* An invalid locator does not invalidate a value whose `inline` is set, because `inline` - resolves the value on its own. +* `offset` may only be set together with `uri`. A value that sets `offset` without `uri` + does not resolve and is invalid. +* `size` must be set whenever `offset` is set. A value that sets `offset` without `size` + is invalid. * If `inline` and a locator are both set, they must denote the same bytes, and a reader may resolve the value from either. If they disagree the value is invalid; a reader is not required to detect this and may return the bytes of either representation. From c2a8c6f8c086001076d3cd5aaa080319b0e71288 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Thu, 20 Aug 2026 16:51:11 -0700 Subject: [PATCH 17/19] Allow inline with no locator in the resolution footnote The footnote required any locator alongside `inline` to be valid on its own, which read as making a value with only `inline` set invalid, since no locator fields are set in that case. Co-authored-by: Isaac --- LogicalTypes.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 65f8b32b..0d1a5168 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -747,9 +747,8 @@ set: | - | - | set | set | invalid | | - | - | - | - | nothing - invalid | -† Any combination of the locator fields that is valid on its own. A locator set -alongside `inline` must satisfy the same rules as one used on its own, so `offset` -requires `uri` and `size`. +† The locator fields may all be unset. Otherwise, fields set alongside `inline` must +form a locator valid on its own, so `offset` requires `uri` and `size`. `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. `size` may be omitted only for a whole-file external reference, where From 17aa46ba9b85a9d71d027b297cf5f083bdf5b83a Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Mon, 24 Aug 2026 16:50:54 +0200 Subject: [PATCH 18/19] Do not require inline and a locator to hold the same bytes Nothing can verify that the bytes behind a `uri` match `inline` without fetching and comparing them, so requiring it stated a rule no reader enforces. Keep the part that readers need, which is that either may be read, and leave agreement to producers. Co-authored-by: Isaac --- LogicalTypes.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 0d1a5168..d3dbf2ec 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -724,11 +724,9 @@ object-store eTag for the whole file referenced by `uri`. ##### inline The referenced bytes stored inline in the value. If `inline` is set, it supplies the -bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance only. -Both representations must denote the same bytes, so a reader may resolve the value from -either and obtain the same result; reading `inline` requires no external access and is -the cheaper path. A locator set alongside `inline` records where those bytes came from, -and must not be a partial or otherwise different representation of the value. +bytes and any locator fields (`uri`, `offset`, `size`) that are set record where those +bytes came from. A reader may resolve the value from `inline` or from the locator, +whichever suits it; `inline` requires no external access and is the cheaper path. #### Resolution @@ -737,7 +735,7 @@ set: | `inline` | `uri` | `offset` | `size` | Resolves to | |----------|-------|----------|--------|-------------------------------------------| -| set | † | † | † | the inline bytes (same as any locator) | +| set | † | † | † | the inline bytes, or the locator | | - | set | - | - | whole external file at `uri` | | - | set | set | - | invalid | | - | set | - | set | external `uri`, `[0, size)` | @@ -774,10 +772,10 @@ compressed, and encrypted like any other column, `inline` included. does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. -* If `inline` and a locator are both set, they must denote the same bytes, and a reader - may resolve the value from either. If they disagree the value is invalid; a reader is - not required to detect this and may return the bytes of either representation. - Producers may treat `inline` and the locator fields as mutually exclusive. +* If `inline` and a locator are both set, a reader may resolve the value from either. + Producers are expected to write the same bytes in both, but a reader is not required + to check this and may return the bytes of either. Producers may treat `inline` and the + locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. From 1125912d84ecf333bdc2838b17f55605a9067928 Mon Sep 17 00:00:00 2001 From: Alkis Evlogimenos Date: Tue, 25 Aug 2026 10:21:02 +0200 Subject: [PATCH 19/19] Drop the note on which resolution path is cheaper Which path costs less is an implementation matter and does not need to be stated to make the rule clear. Co-authored-by: Isaac --- LogicalTypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index d3dbf2ec..f7fe984f 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -726,7 +726,7 @@ object-store eTag for the whole file referenced by `uri`. The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`uri`, `offset`, `size`) that are set record where those bytes came from. A reader may resolve the value from `inline` or from the locator, -whichever suits it; `inline` requires no external access and is the cheaper path. +whichever suits it. #### Resolution