From 0327ca349925168d20e9d703ea164754649dd484 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Mon, 22 Jun 2026 06:22:17 +0000 Subject: [PATCH 1/8] wip --- CONTRIBUTING.md | 79 ++++++++++++++++++-------------- ParxMagicNumber.md | 84 ++++++++++++++++++++++++++++++++++ src/main/thrift/parquet.thrift | 9 +++- 3 files changed, 136 insertions(+), 36 deletions(-) create mode 100644 ParxMagicNumber.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d6049a887..75cf033bc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,7 +91,7 @@ demonstrate that the feature is mergeable to its implementation. 1. To the greatest extent possible changes should have an option for forward compatibility (old readers can still read files). The [compatibility and - feature enablement](#compatibility-and-feature-enablement) section below + feature enablement](#compatibility-and-feature-enablement) section below provides more details on expectations for changes that break compatibility. 2. New encodings should be fully specified in this repository and not @@ -110,10 +110,17 @@ demonstrate that the feature is mergeable to its implementation. The Parquet PMC aims to do releases of the format package only as needed when new features are introduced. If multiple new features are being proposed simultaneously some features might be consolidated into the same release. -Guidance is provided below on when implementations should enable features added -to the specification. Due to confusion in the past over Parquet versioning it -is not expected that there will be a 3.x release of the specification in the -foreseeable future. +Guidance is provided below on when implementations should enable features introduced +by new specification versions. + +The Parquet Format versioning follows [SemVer](http://semver.org/) from release +3.0 onwards. Every forward incompatible change added to the specification requires +a major version bump (e.g. new encodings, new compression algorithms, new page layouts, +structural changes to the footer, etc). Forward compatible changes (e.g. new +logical types and sort orders) will be released as minor version changes. + +Versions 2.x.x did not follow SemVer. + ### Compatibility and Feature Enablement @@ -131,45 +138,51 @@ For the purposes of this discussion we classify features into the following buck 3. Forward incompatible. A file written under a newer version of the format with the feature enabled cannot be read under an older version of the format (e.g. - adding and using a new compression algorithm). It is expected any feature in + adding and using a new compression algorithm). Any feature in this category will provide a signal to older readers, so they can - unambiguously determine that they cannot properly read the file (e.g. via - adding a new value to an existing enum). + unambiguously determine that they cannot properly read the file. There are + two mechanisms for providing this functionality: + 1. Adding values to existing enums/unions (e.g. Sort Order, + encodings, compression) are all covered by this mechanism. + 2. Reserving a new bit in the [PARX](ParxMagicNumber.md) footer feature bitmap for other + structural changes that an older reader would not be able to accept. New features are intended to be widely beneficial to users of Parquet, and therefore it is hoped third-party implementations will adopt them quickly after -they are introduced. It is assumed that writing new parts of the format, and -especially forward incompatible features, will be configured with a feature flag -defaulted to "off", and at some future point the feature is turned on by default -(reading of the new feature will typically be enabled without configuration or -defaulted to on). Some amount of lead time is desirable to ensure a critical -mass of Parquet implementations support a feature to avoid compatibility issues +they are introduced. It is expected that implementations will provide a configuration +mechanism to users to enable features. It is recommended that implementations provide +at least a way to enable all relevant features given a specification version +(e.g. major and minor version). In addition, implementations might also choose to +enable features at a finer-grained level, with feature flags initially defaulted to "off". + +Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a given specification version across the ecosystem. Therefore, the Parquet PMC gives the following -recommendations for managing features: +recommendations for managing the default specification version used for writing: 1. Backward compatibility is the concern of implementations but given the ubiquity of Parquet and the length of time it has been used, libraries should support reading older versions of the format to the greatest extent possible. -2. Forward compatible features/changes may be enabled and used by default in +2. Minor format versions may be enabled and used by default in implementations once the parquet-format containing those changes has been - formally released. For features that may pose a significant performance - regression to older format readers, libaries should consider delaying default - enablement until 1 year after the release of the parquet-java implementation - that contains the feature implementation. - -3. Forward incompatible features/changes should not be turned on by default - until 2 years after the parquet-java implementation containing the feature is - released. It is recommended that changing the default value for a forward - incompatible feature flag should be clearly advertised to consumers (e.g. via - a major version release if using Semantic Versioning, or highlighed in + formally released. For releases that may pose a significant performance + regression to older format readers, libraries should consider delaying default + enablement until 1 year after the parquet-java implementation for that format + version is released. + +3. Specifications with major version upgrades should not be turned on by default + until 2 years after the parquet-java implementation for the specification has been + released. It is recommended that changing the default value for a major version bump + be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighted in release notes). For forward compatible changes which have a high chance of performance regression for older readers and forward incompatible changes, implementations should clearly document the compatibility issues. Additionally, while it is up to maintainers of individual open-source implementations to make the best decision to serve -their ecosystem, they are encouraged to start enabling features by default along +their ecosystem, they are encouraged to start enabling specific format versions by default along the same timelines as `parquet-java`. Parquet-java will wait to enable features by default until the most conservative timelines outlined above have been exceeded. This timeline is an attempt to balance ensuring @@ -179,14 +192,13 @@ encourage earlier adoption of new features when an organization using Parquet can guarantee that all readers of the parquet files they produce can read a new feature. -After turning a feature on by default implementations -are encouraged to keep a configuration to turn off the feature. +After changing defaults implementations are encouraged to keep a configuration +mechanism to specify a prior format version or turn off specific features. A recommendation for full deprecation will be made in a future iteration of this document. -For features released prior to October 2024, target dates for each of these -categories will be updated as part of the `parquet-java 2.0` release process -based on a collected feature compatibility matrix. +As of June 2026, the current recommended default specification release +version to use is 2.10.0. For each release of `parquet-java` or `parquet-format` that influences this guidance it is expected exact dates will be added to parquet-format to provide @@ -198,5 +210,4 @@ implementation date/release version information when updating the feature matrix. End users of software are generally encouraged to consult the feature matrix -and vendor documentation before enabling features that are not yet widely -adopted. +and vendor documentation before enabling a specific format version. diff --git a/ParxMagicNumber.md b/ParxMagicNumber.md new file mode 100644 index 000000000..4661bb91b --- /dev/null +++ b/ParxMagicNumber.md @@ -0,0 +1,84 @@ +# PARX Parquet Format Specification + +This specification details a new magic number and associated fixed length footer metadata changes +that accompany the footer. + +## Motivation + +Most parts of the parquet specification lend themselves naturally to comptability checks +when a new feature is added (e.g. encodings and compression values have an enum value added) +and fail appropriately. +However, some semantic changes or footer changes are impossible to communicate appropriately +within existing structures (e.g. changing the serialization of the footer). The motivation +for the new magic number and layout is to accomodate the latter set of changes by introducing +a new extensible mechanism for readers to detect these changes and fail accordingly. + +## Design Motivations + +* Provide a mechanism to only introduce a single new magic number for parquet that can + last at least a decade. +* Provide integrity checks for the for the footer. +* Provide the ability for readers to have a granular understanding of structural and semantic + backward incompatible features that are required to read a particular file. + +## File Layout + +A PARX file has the same overall structure as a standard Parquet file, with two differences: +the leading and trailing magic bytes are `PARX` instead of `PAR1/PARE`, and the trailing footer is +16 bytes instead of 8. + +The file layout is as follows: + +``` ++-----------+-------------------+--------------------+-------------+ +| 'PARX' | File Data | Footer Metadata | Footer tail | +| (4 bytes) | (variable length) | (variable length) | (16 bytes) | ++-----------+----------+--------+--------------------+-------------+ +``` + +All multi-byte integer fields are **little-endian**. + + +### PARX Footer Tail — 16 bytes + +``` ++------------------+-----------+----------+--------+ +| metadata_len | flags | crc32 | 'PARX' | ++------------------+-----------+----------+--------+ + offset 0 offset 4 offset 8 offset 12 +``` + +| Field | Type | Offset | Description | +|----------------|---------|--------|-------------------------------------------------------------------| +| `metadata_len` | i32 LE | 0 | Byte length of the Thrift-encoded `FileMetaData` block | +| `flags` | u32 LE | 4 | Feature flags (see [Feature Flags](#feature-flags)) | +| `crc32` | u32 LE | 8 | CRC32 checksum (see [Integrity Check](#integrity-check)) | +| `magic` | [u8; 4] | 12 | Always the bytes `P A R X` (0x50 0x41 0x52 0x58) | + +## Feature Flags + +The `flags` field is a 32-bit bitfield. A reader **must** reject any file whose `flags` field +contains bits not either not recognized or not supported, because unknown flags may imply structural changes +to the metadata or semantic changes to the file layout that the reader cannot properly interpret. + +| Bit Index| Name | Description | +|----------|-------------------------|-------------------------------------------------------------------------------------------------------------| +| 0 | `ENCRYPTED_FOOTER` | The `FileMetaData` block is encrypted (equivalent to the `PARE` format). | +| 1 | `OMIT_PATH_IN_SCHEMA` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| + + +The zero index is least signficant bit in the field. +All other bits are reserved and must be zero. + + +## Integrity Check + +The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) +computed over the following byte sequence, in order: + +1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 16-byte footer tail) +2. The first 8 bytes of the footer tail (metadata_len and flags bitmap) + +A reader should verify the checksum **after** validating the feature flags (in the rare case that feature flag indicates +a change in the fixed size tail of the file). + diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index e99c46195..64ec78ce7 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -880,8 +880,13 @@ struct ColumnMetaData { * whether we can decode those pages. **/ 2: required list encodings - /** Path in schema **/ - 3: required list path_in_schema + /** Path in schema + * + * Made optional in parquet-format 3.0. If not written + * PARX magic number must be used (Bit 1 in in feature flag bitmap + * must be set). + **/ + 3: optional list path_in_schema /** Compression codec **/ 4: required CompressionCodec codec From 974df154bd111920307cbcc4bb18b2fcb0fd400c Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Mon, 22 Jun 2026 06:35:10 +0000 Subject: [PATCH 2/8] wip --- CONTRIBUTING.md | 14 +++++++------- ParxMagicNumber.md | 17 +++++++++++------ src/main/thrift/parquet.thrift | 4 ++-- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 75cf033bc..4ddd49f68 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -143,16 +143,16 @@ For the purposes of this discussion we classify features into the following buck unambiguously determine that they cannot properly read the file. There are two mechanisms for providing this functionality: 1. Adding values to existing enums/unions (e.g. Sort Order, - encodings, compression) are all covered by this mechanism. - 2. Reserving a new bit in the [PARX](ParxMagicNumber.md) footer feature bitmap for other + encodings, compression) is covered by this mechanism. + 2. Reserving a new bit in the [PARX](ParxMagicNumber.md) footer feature bitmap covers other structural changes that an older reader would not be able to accept. New features are intended to be widely beneficial to users of Parquet, and therefore it is hoped third-party implementations will adopt them quickly after they are introduced. It is expected that implementations will provide a configuration -mechanism to users to enable features. It is recommended that implementations provide +mechanism for users to enable features. It is recommended that implementations provide at least a way to enable all relevant features given a specification version -(e.g. major and minor version). In addition, implementations might also choose to +(e.g. major and minor version). In addition, implementations might choose to enable features at a finer-grained level, with feature flags initially defaulted to "off". Some amount of lead time is desirable to ensure a critical @@ -171,9 +171,9 @@ recommendations for managing the default specification version used for writing: enablement until 1 year after the parquet-java implementation for that format version is released. -3. Specifications with major version upgrades should not be turned on by default +3. Major version upgrades should not be enabled by default until 2 years after the parquet-java implementation for the specification has been - released. It is recommended that changing the default value for a major version bump + released. It is recommended that changing the default format version for a major version bump be clearly advertised to consumers (e.g. via a major version release if using Semantic Versioning, or highlighted in release notes). @@ -192,7 +192,7 @@ encourage earlier adoption of new features when an organization using Parquet can guarantee that all readers of the parquet files they produce can read a new feature. -After changing defaults implementations are encouraged to keep a configuration +After changing defaults, implementations are encouraged to keep a configuration mechanism to specify a prior format version or turn off specific features. A recommendation for full deprecation will be made in a future iteration of this document. diff --git a/ParxMagicNumber.md b/ParxMagicNumber.md index 4661bb91b..2fc9af842 100644 --- a/ParxMagicNumber.md +++ b/ParxMagicNumber.md @@ -17,7 +17,7 @@ a new extensible mechanism for readers to detect these changes and fail accordin * Provide a mechanism to only introduce a single new magic number for parquet that can last at least a decade. -* Provide integrity checks for the for the footer. +* Provide integrity checks for the footer. * Provide the ability for readers to have a granular understanding of structural and semantic backward incompatible features that are required to read a particular file. @@ -58,13 +58,16 @@ All multi-byte integer fields are **little-endian**. ## Feature Flags The `flags` field is a 32-bit bitfield. A reader **must** reject any file whose `flags` field -contains bits not either not recognized or not supported, because unknown flags may imply structural changes +contains bits that are not recognized or not supported, because unknown flags may imply structural changes to the metadata or semantic changes to the file layout that the reader cannot properly interpret. +The PARX format is independent of the `version` field in `FileMetaData`; a file may use the PARX +magic number regardless of which specification version its metadata declares. + | Bit Index| Name | Description | |----------|-------------------------|-------------------------------------------------------------------------------------------------------------| | 0 | `ENCRYPTED_FOOTER` | The `FileMetaData` block is encrypted (equivalent to the `PARE` format). | -| 1 | `OMIT_PATH_IN_SCHEMA` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| +| 1 | `PATH_IN_SCHEMA_OMITTED` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| The zero index is least signficant bit in the field. @@ -73,12 +76,14 @@ All other bits are reserved and must be zero. ## Integrity Check -The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) +The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) computed over the following byte sequence, in order: 1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 16-byte footer tail) 2. The first 8 bytes of the footer tail (metadata_len and flags bitmap) -A reader should verify the checksum **after** validating the feature flags (in the rare case that feature flag indicates -a change in the fixed size tail of the file). +When `ENCRYPTED_FOOTER` (bit 0) is set, the CRC is computed over the footer bytes **as they appear in the +file** (i.e. the encrypted bytes). The CRC itself is always stored unencrypted in the footer tail. +A reader should verify the checksum **before** decrypting the footer, and **after** validating the feature +flags (in the rare case that a feature flag indicates a change in the fixed size tail of the file). diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index 64ec78ce7..c05ad5110 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -883,8 +883,8 @@ struct ColumnMetaData { /** Path in schema * * Made optional in parquet-format 3.0. If not written - * PARX magic number must be used (Bit 1 in in feature flag bitmap - * must be set). + * PARX magic number must be used (PATH_IN_SCHEMA_OMITTED, bit 1 in the + * feature flag bitmap, must be set). **/ 3: optional list path_in_schema From bd585cbc26d655a7862b2e2805368d4fd3ae1348 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Mon, 22 Jun 2026 23:21:43 +0000 Subject: [PATCH 3/8] mention backports --- CONTRIBUTING.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4ddd49f68..295dc0a50 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -111,7 +111,9 @@ The Parquet PMC aims to do releases of the format package only as needed when new features are introduced. If multiple new features are being proposed simultaneously some features might be consolidated into the same release. Guidance is provided below on when implementations should enable features introduced -by new specification versions. +by new specification versions. If there is contributor bandwidth forwards-compatible +features may be backported to previous major versions of the specification and released +as a new minor version. The Parquet Format versioning follows [SemVer](http://semver.org/) from release 3.0 onwards. Every forward incompatible change added to the specification requires From 07d868f43475d7d739f0b800bfa74d62e3a1b3b9 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Mon, 22 Jun 2026 23:31:43 +0000 Subject: [PATCH 4/8] fix typ --- ParxMagicNumber.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ParxMagicNumber.md b/ParxMagicNumber.md index 2fc9af842..243092929 100644 --- a/ParxMagicNumber.md +++ b/ParxMagicNumber.md @@ -5,7 +5,7 @@ that accompany the footer. ## Motivation -Most parts of the parquet specification lend themselves naturally to comptability checks +Most parts of the parquet specification lend themselves naturally to compatibility checks when a new feature is added (e.g. encodings and compression values have an enum value added) and fail appropriately. However, some semantic changes or footer changes are impossible to communicate appropriately From 389ba54d837ce7842eed1664e86f95919c6874e7 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Wed, 8 Jul 2026 16:15:10 +0000 Subject: [PATCH 5/8] remove semver reference --- CONTRIBUTING.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 295dc0a50..dafd4fb12 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -115,13 +115,13 @@ by new specification versions. If there is contributor bandwidth forwards-compat features may be backported to previous major versions of the specification and released as a new minor version. -The Parquet Format versioning follows [SemVer](http://semver.org/) from release -3.0 onwards. Every forward incompatible change added to the specification requires +Every forward incompatible change added to the specification requires a major version bump (e.g. new encodings, new compression algorithms, new page layouts, structural changes to the footer, etc). Forward compatible changes (e.g. new -logical types and sort orders) will be released as minor version changes. +logical types and sort orders) are as released as minor version changes. -Versions 2.x.x did not follow SemVer. +Versions 2.x.x did not follow this versioning, both forward incompatible changes and +forward compatible changes where released with minor version bumps. ### Compatibility and Feature Enablement From 0af8e7a7732ad128c527dfa603dabc8096c20795 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Fri, 14 Aug 2026 19:02:01 +0000 Subject: [PATCH 6/8] Revert CONTRIBUTING.md to master --- CONTRIBUTING.md | 81 +++++++++++++++++++++---------------------------- 1 file changed, 34 insertions(+), 47 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dafd4fb12..d6049a887 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,7 +91,7 @@ demonstrate that the feature is mergeable to its implementation. 1. To the greatest extent possible changes should have an option for forward compatibility (old readers can still read files). The [compatibility and - feature enablement](#compatibility-and-feature-enablement) section below + feature enablement](#compatibility-and-feature-enablement) section below provides more details on expectations for changes that break compatibility. 2. New encodings should be fully specified in this repository and not @@ -110,19 +110,10 @@ demonstrate that the feature is mergeable to its implementation. The Parquet PMC aims to do releases of the format package only as needed when new features are introduced. If multiple new features are being proposed simultaneously some features might be consolidated into the same release. -Guidance is provided below on when implementations should enable features introduced -by new specification versions. If there is contributor bandwidth forwards-compatible -features may be backported to previous major versions of the specification and released -as a new minor version. - -Every forward incompatible change added to the specification requires -a major version bump (e.g. new encodings, new compression algorithms, new page layouts, -structural changes to the footer, etc). Forward compatible changes (e.g. new -logical types and sort orders) are as released as minor version changes. - -Versions 2.x.x did not follow this versioning, both forward incompatible changes and -forward compatible changes where released with minor version bumps. - +Guidance is provided below on when implementations should enable features added +to the specification. Due to confusion in the past over Parquet versioning it +is not expected that there will be a 3.x release of the specification in the +foreseeable future. ### Compatibility and Feature Enablement @@ -140,51 +131,45 @@ For the purposes of this discussion we classify features into the following buck 3. Forward incompatible. A file written under a newer version of the format with the feature enabled cannot be read under an older version of the format (e.g. - adding and using a new compression algorithm). Any feature in + adding and using a new compression algorithm). It is expected any feature in this category will provide a signal to older readers, so they can - unambiguously determine that they cannot properly read the file. There are - two mechanisms for providing this functionality: - 1. Adding values to existing enums/unions (e.g. Sort Order, - encodings, compression) is covered by this mechanism. - 2. Reserving a new bit in the [PARX](ParxMagicNumber.md) footer feature bitmap covers other - structural changes that an older reader would not be able to accept. + unambiguously determine that they cannot properly read the file (e.g. via + adding a new value to an existing enum). New features are intended to be widely beneficial to users of Parquet, and therefore it is hoped third-party implementations will adopt them quickly after -they are introduced. It is expected that implementations will provide a configuration -mechanism for users to enable features. It is recommended that implementations provide -at least a way to enable all relevant features given a specification version -(e.g. major and minor version). In addition, implementations might choose to -enable features at a finer-grained level, with feature flags initially defaulted to "off". - -Some amount of lead time is desirable to ensure a critical -mass of Parquet implementations support a given specification version +they are introduced. It is assumed that writing new parts of the format, and +especially forward incompatible features, will be configured with a feature flag +defaulted to "off", and at some future point the feature is turned on by default +(reading of the new feature will typically be enabled without configuration or +defaulted to on). Some amount of lead time is desirable to ensure a critical +mass of Parquet implementations support a feature to avoid compatibility issues across the ecosystem. Therefore, the Parquet PMC gives the following -recommendations for managing the default specification version used for writing: +recommendations for managing features: 1. Backward compatibility is the concern of implementations but given the ubiquity of Parquet and the length of time it has been used, libraries should support reading older versions of the format to the greatest extent possible. -2. Minor format versions may be enabled and used by default in +2. Forward compatible features/changes may be enabled and used by default in implementations once the parquet-format containing those changes has been - formally released. For releases that may pose a significant performance - regression to older format readers, libraries should consider delaying default - enablement until 1 year after the parquet-java implementation for that format - version is released. - -3. Major version upgrades should not be enabled by default - until 2 years after the parquet-java implementation for the specification has been - released. It is recommended that changing the default format version for a major version bump - be clearly advertised to consumers (e.g. via - a major version release if using Semantic Versioning, or highlighted in + formally released. For features that may pose a significant performance + regression to older format readers, libaries should consider delaying default + enablement until 1 year after the release of the parquet-java implementation + that contains the feature implementation. + +3. Forward incompatible features/changes should not be turned on by default + until 2 years after the parquet-java implementation containing the feature is + released. It is recommended that changing the default value for a forward + incompatible feature flag should be clearly advertised to consumers (e.g. via + a major version release if using Semantic Versioning, or highlighed in release notes). For forward compatible changes which have a high chance of performance regression for older readers and forward incompatible changes, implementations should clearly document the compatibility issues. Additionally, while it is up to maintainers of individual open-source implementations to make the best decision to serve -their ecosystem, they are encouraged to start enabling specific format versions by default along +their ecosystem, they are encouraged to start enabling features by default along the same timelines as `parquet-java`. Parquet-java will wait to enable features by default until the most conservative timelines outlined above have been exceeded. This timeline is an attempt to balance ensuring @@ -194,13 +179,14 @@ encourage earlier adoption of new features when an organization using Parquet can guarantee that all readers of the parquet files they produce can read a new feature. -After changing defaults, implementations are encouraged to keep a configuration -mechanism to specify a prior format version or turn off specific features. +After turning a feature on by default implementations +are encouraged to keep a configuration to turn off the feature. A recommendation for full deprecation will be made in a future iteration of this document. -As of June 2026, the current recommended default specification release -version to use is 2.10.0. +For features released prior to October 2024, target dates for each of these +categories will be updated as part of the `parquet-java 2.0` release process +based on a collected feature compatibility matrix. For each release of `parquet-java` or `parquet-format` that influences this guidance it is expected exact dates will be added to parquet-format to provide @@ -212,4 +198,5 @@ implementation date/release version information when updating the feature matrix. End users of software are generally encouraged to consult the feature matrix -and vendor documentation before enabling a specific format version. +and vendor documentation before enabling features that are not yet widely +adopted. From 22d705263d2e1976ba63e3b64f3b2243721f3250 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Fri, 14 Aug 2026 19:05:06 +0000 Subject: [PATCH 7/8] Revert parquet.thrift to master --- src/main/thrift/parquet.thrift | 167 +++++++++++++++++++++++++++------ 1 file changed, 138 insertions(+), 29 deletions(-) diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index c05ad5110..225f85f96 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -280,7 +280,7 @@ struct Statistics { */ 1: optional binary max; 2: optional binary min; - /** + /** * Count of null values in the column. * * Writers SHOULD always write this field even if it is zero (i.e. no null value) @@ -309,6 +309,13 @@ struct Statistics { 7: optional bool is_max_value_exact; /** If true, min_value is the actual minimum value for a column */ 8: optional bool is_min_value_exact; + /** + * Count of NaN values in the column; only present if physical type is FLOAT + * or DOUBLE, or logical type is FLOAT16. + * If this field is not present, readers MUST assume NaNs may be present + * (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0). + */ + 9: optional i64 nan_count; } /** Empty structs to use as logical type annotations */ @@ -712,9 +719,14 @@ struct DictionaryPageHeader { } /** - * New page format allowing reading levels without decompressing the data + * Alternate page format allowing reading levels without decompressing the data * Repetition and definition levels are uncompressed * The remaining section containing the data is compressed if is_compressed is true + * + * Implementation note - this header is not necessarily a strict improvement over + * `DataPageHeader` (in particular the original header might provide better compression + * in some scenarios). Page indexes require pages to start and end at row boundaries, + * regardless of which page header is used. **/ struct DataPageHeaderV2 { /** Number of values, including NULLs, in this data page. **/ @@ -880,13 +892,8 @@ struct ColumnMetaData { * whether we can decode those pages. **/ 2: required list encodings - /** Path in schema - * - * Made optional in parquet-format 3.0. If not written - * PARX magic number must be used (PATH_IN_SCHEMA_OMITTED, bit 1 in the - * feature flag bitmap, must be set). - **/ - 3: optional list path_in_schema + /** Path in schema **/ + 3: required list path_in_schema /** Compression codec **/ 4: required CompressionCodec codec @@ -897,7 +904,7 @@ struct ColumnMetaData { /** total byte size of all uncompressed pages in this column chunk (including the headers) **/ 6: required i64 total_uncompressed_size - /** total byte size of all compressed, and potentially encrypted, pages + /** total byte size of all compressed, and potentially encrypted, pages * in this column chunk (including the headers) **/ 7: required i64 total_compressed_size @@ -963,6 +970,21 @@ union ColumnCryptoMetaData { struct ColumnChunk { /** File where column data is stored. If not set, assumed to be same file as * metadata. This path is relative to the current file. + * + * As of December 2025, the only known use-case for this field is writing summary + * parquet files (i.e. "_metadata" files). These files consolidate footers from + * multiple parquet files to allow for efficient reading of footers to avoid file + * listing costs and prune out files that do not need to be read based on statistics. + * + * These files do not appear to have ever been formally specified in the specification. + * and are potentially problematic from a correctness perspective [1]. + * + * [1] https://lists.apache.org/thread/ootf2kmyg3p01b1bvplpvp4ftd1bt72d + * + * There is no other known usage of this field. Specifically, there are no known + * reference implementations that will read externally stored column data if this field is populated + * within a standard parquet file. Making use of the field for this purpose is + * not considered part of the Parquet specification. **/ 1: optional string file_path @@ -1024,10 +1046,10 @@ struct RowGroup { * in this row group **/ 5: optional i64 file_offset - /** Total byte size of all compressed (and potentially encrypted) column data + /** Total byte size of all compressed (and potentially encrypted) column data * in this row group **/ 6: optional i64 total_compressed_size - + /** Row group ordinal in the file **/ 7: optional i16 ordinal } @@ -1035,6 +1057,9 @@ struct RowGroup { /** Empty struct to signal the order defined by the physical or logical type */ struct TypeDefinedOrder {} +/** Empty struct to signal IEEE 754 total order for floating point types */ +struct IEEE754TotalOrder {} + /** * Union to specify the order used for the min_value and max_value fields for a * column. This union takes the role of an enhanced enum that allows rich @@ -1043,6 +1068,7 @@ struct TypeDefinedOrder {} * Possible values are: * * TypeDefinedOrder - the column uses the order defined by its logical or * physical type (if there is no logical type). + * * IEEE754TotalOrder - the floating point column uses IEEE 754 total order. * * If the reader does not support the value of this union, min and max stats * for this column should be ignored. @@ -1096,23 +1122,78 @@ union ColumnOrder { * 64-bit signed integer (nanos) * See https://github.com/apache/parquet-format/issues/502 for more details * - * (*) Because the sorting order is not specified properly for floating - * point values (relations vs. total ordering) the following + * (*) Because TYPE_ORDER is ambiguous for floating point types due to + * underspecified handling of NaN and -0/+0, it is recommended that writers + * use IEEE_754_TOTAL_ORDER for these types. + * + * If TYPE_ORDER is used for floating point types, then the following * compatibility rules should be applied when reading statistics: * - If the min is a NaN, it should be ignored. * - If the max is a NaN, it should be ignored. + * - If the nan_count field is set, a reader can compute + * nan_count + null_count == num_values to deduce whether all non-null + * values are NaN. * - If the min is +0, the row group may contain -0 values as well. * - If the max is -0, the row group may contain +0 values as well. * - When looking for NaN values, min and max should be ignored. - * - * When writing statistics the following rules should be followed: - * - NaNs should not be written to min or max statistics fields. + * If the nan_count field is set, it can be used to check whether + * NaNs are present. + * + * When writing page or column chunk statistics for columns with + * TYPE_ORDER order, the following rules must be followed: + * - The nan_count field must be set for floating point types, even if + * it is zero. + * - If the nan_count field is set, min and max statistics fields, when + * present, must not contain NaN values and must be computed from + * non-NaN values only. This signals to readers that the min and max + * statistics are reliable for non-NaN values. + * - If all non-null values are NaN, min and max statistics must not be + * written. * - If the computed max value is zero (whether negative or positive), * `+0.0` should be written into the max statistics field. * - If the computed min value is zero (whether negative or positive), * `-0.0` should be written into the min statistics field. + * + * When writing column indexes for columns with TYPE_ORDER order, the + * following rules must be followed: + * - NaNs must not be written to min_values or max_values. + * - If all non-null values of a page are NaN, a column index must not + * be written for this column chunk because min_values and max_values + * are required. + * - If the computed max value is zero (whether negative or positive), + * `+0.0` should be written into the corresponding max_values entry. + * - If the computed min value is zero (whether negative or positive), + * `-0.0` should be written into the corresponding min_values entry. */ 1: TypeDefinedOrder TYPE_ORDER; + + /* + * The floating point type is ordered according to the totalOrder predicate, + * as defined in section 5.10 of IEEE-754 (2008 revision). Only columns of + * physical type FLOAT or DOUBLE, or logical type FLOAT16 may use this ordering. + * + * Intuitively, this orders floats mathematically, but defines -0 to be less + * than +0, -NaN to be less than anything else, and +NaN to be greater than + * anything else. It also defines an order between different bit representations + * of the same value. + * + * When writing statistics for columns with IEEE_754_TOTAL_ORDER order, then + * following rules must be followed: + * - Writing the nan_count field is mandatory when using this ordering. + * - Min and max statistics must contain the smallest and largest non-NaN + * values respectively, or if all non-null values are NaN, the smallest and + * largest NaN values as defined by IEEE 754 total order. + * + * When reading statistics for columns with this order, the following rules + * should be followed: + * - Readers should consult the nan_count field to determine whether NaNs + * are present. + * - A reader can compute nan_count + null_count == num_values to deduce + * whether all non-null values are NaN. In the page index, which does not + * have a num_values field, the presence of a NaN value in min_values + * or max_values indicates that all non-null values are NaN. + */ + 2: IEEE754TotalOrder IEEE_754_TOTAL_ORDER; } struct PageLocation { @@ -1184,6 +1265,18 @@ struct ColumnIndex { * Such more compact values must still be valid values within the column's * logical type. Readers must make sure that list entries are populated before * using them by inspecting null_pages. + * + * For columns of physical type FLOAT or DOUBLE, or logical type FLOAT16, + * NaN values are not to be included in these bounds. If all non-null values + * of a page are NaN, then a writer must do the following: + * - If the order of this column is TYPE_ORDER, then a column index must + * not be written for this column chunk. While this is unfortunate for + * performance, it is necessary to avoid conflict with legacy files that + * still included NaN in min_values and max_values even if the page had + * non-NaN values. To mitigate this, IEEE754_TOTAL_ORDER is recommended. + * - If the order of this column is IEEE754_TOTAL_ORDER, then min_values[i] + * and max_values[i] of that page must be set to the smallest and largest + * NaN values as defined by IEEE 754 total order. */ 2: required list min_values 3: required list max_values @@ -1197,13 +1290,13 @@ struct ColumnIndex { 4: required BoundaryOrder boundary_order /** - * A list containing the number of null values for each page + * A list containing the number of null values for each page * * Writers SHOULD always write this field even if no null values * are present or the column is not nullable. - * Readers MUST distinguish between null_counts not being present + * Readers MUST distinguish between null_counts not being present * and null_count being 0. - * If null_counts are not present, readers MUST NOT assume all + * If null_counts are not present, readers MUST NOT assume all * null counts are 0. */ 5: optional list null_counts @@ -1225,6 +1318,15 @@ struct ColumnIndex { * Same as repetition_level_histograms except for definitions levels. **/ 7: optional list definition_level_histograms; + + /** + * A list containing the number of NaN values for each page. Only present + * for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16. + * If this field is not present, readers MUST assume that there might be + * NaN values in any page. + */ + 8: optional list nan_counts + } struct AesGcmV1 { @@ -1260,7 +1362,14 @@ union EncryptionAlgorithm { * Description for file metadata */ struct FileMetaData { - /** Version of this file **/ + /** Version of this file + * + * As of December 2025, there is no agreed upon consensus of what constitutes + * version 2 of the file. For maximum compatibility with readers, writers should + * always populate "1" for version. For maximum compatibility with writers, + * readers should accept "1" and "2" interchangeably. All other versions are + * reserved for potential future use-cases. + */ 1: required i32 version /** Parquet schema for this file. This schema contains metadata for all the columns. @@ -1304,30 +1413,30 @@ struct FileMetaData { */ 7: optional list column_orders; - /** + /** * Encryption algorithm. This field is set only in encrypted files * with plaintext footer. Files with encrypted footer store algorithm id * in FileCryptoMetaData structure. */ 8: optional EncryptionAlgorithm encryption_algorithm - /** - * Retrieval metadata of key used for signing the footer. - * Used only in encrypted files with plaintext footer. - */ + /** + * Retrieval metadata of key used for signing the footer. + * Used only in encrypted files with plaintext footer. + */ 9: optional binary footer_signing_key_metadata } /** Crypto metadata for files with encrypted footer **/ struct FileCryptoMetaData { - /** + /** * Encryption algorithm. This field is only used for files * with encrypted footer. Files with plaintext footer store algorithm id * inside footer (FileMetaData structure). */ 1: required EncryptionAlgorithm encryption_algorithm - - /** Retrieval metadata of key used for encryption of footer, + + /** Retrieval metadata of key used for encryption of footer, * and (possibly) columns **/ 2: optional binary key_metadata } From 99c14d7d43ebc2e96c645847624b1aaf6cfade25 Mon Sep 17 00:00:00 2001 From: Micah Kornfield Date: Wed, 26 Aug 2026 07:45:50 +0000 Subject: [PATCH 8/8] tighten and add verison --- ParxMagicNumber.md | 67 +++++++++++++++++++++++++++++++++------------- 1 file changed, 48 insertions(+), 19 deletions(-) diff --git a/ParxMagicNumber.md b/ParxMagicNumber.md index 243092929..c4c17661c 100644 --- a/ParxMagicNumber.md +++ b/ParxMagicNumber.md @@ -3,15 +3,15 @@ This specification details a new magic number and associated fixed length footer metadata changes that accompany the footer. -## Motivation +## Motivation Most parts of the parquet specification lend themselves naturally to compatibility checks -when a new feature is added (e.g. encodings and compression values have an enum value added) +when a new feature is added (e.g. encodings and compression values have an enum value added) and fail appropriately. -However, some semantic changes or footer changes are impossible to communicate appropriately +However, some semantic changes or footer changes are impossible to communicate appropriately within existing structures (e.g. changing the serialization of the footer). The motivation for the new magic number and layout is to accomodate the latter set of changes by introducing -a new extensible mechanism for readers to detect these changes and fail accordingly. +a new extensible mechanism for readers to detect these changes and fail accordingly. ## Design Motivations @@ -25,35 +25,36 @@ a new extensible mechanism for readers to detect these changes and fail accordin A PARX file has the same overall structure as a standard Parquet file, with two differences: the leading and trailing magic bytes are `PARX` instead of `PAR1/PARE`, and the trailing footer is -16 bytes instead of 8. +17 bytes instead of 8. The file layout is as follows: ``` +-----------+-------------------+--------------------+-------------+ | 'PARX' | File Data | Footer Metadata | Footer tail | -| (4 bytes) | (variable length) | (variable length) | (16 bytes) | +| (4 bytes) | (variable length) | (variable length) | (17 bytes) | +-----------+----------+--------+--------------------+-------------+ ``` All multi-byte integer fields are **little-endian**. -### PARX Footer Tail — 16 bytes +### PARX Footer Tail — 17 bytes ``` -+------------------+-----------+----------+--------+ -| metadata_len | flags | crc32 | 'PARX' | -+------------------+-----------+----------+--------+ - offset 0 offset 4 offset 8 offset 12 ++------------------+-----------+-----------+----------+--------+ +| metadata_len | flags | version | crc32 | 'PARX' | ++------------------+-----------+-----------+----------+--------+ + offset 0 offset 4 offset 8 offset 9 offset 13 ``` | Field | Type | Offset | Description | |----------------|---------|--------|-------------------------------------------------------------------| -| `metadata_len` | i32 LE | 0 | Byte length of the Thrift-encoded `FileMetaData` block | +| `metadata_len` | u32 LE | 0 | Byte length of the Thrift-encoded `FileMetaData` block | | `flags` | u32 LE | 4 | Feature flags (see [Feature Flags](#feature-flags)) | -| `crc32` | u32 LE | 8 | CRC32 checksum (see [Integrity Check](#integrity-check)) | -| `magic` | [u8; 4] | 12 | Always the bytes `P A R X` (0x50 0x41 0x52 0x58) | +| `version` | u8 | 8 | Parquet format major version (see [Version](#version)) | +| `crc32` | u32 LE | 9 | CRC32 checksum (see [Integrity Check](#integrity-check)) | +| `magic` | [u8; 4] | 13 | Always the bytes `P A R X` (0x50 0x41 0x52 0x58) | ## Feature Flags @@ -68,22 +69,50 @@ magic number regardless of which specification version its metadata declares. |----------|-------------------------|-------------------------------------------------------------------------------------------------------------| | 0 | `ENCRYPTED_FOOTER` | The `FileMetaData` block is encrypted (equivalent to the `PARE` format). | | 1 | `PATH_IN_SCHEMA_OMITTED` | Column `path_in_schema` fields are omitted from ColumnChunk metadata (this was a previously required field).| +| 2 | `PREVIEW_FEATURES` | The file uses preview (experimental) features not yet part of the stable feature set of the major `version` written in the footer tail. Preview features that are not part of the major version are either detectable via additional feature flags in this bitmap or when parsing the encoded metadata. A preview feature (e.g. a new encoding) might cause thrift parsers to fail to parse the metadata, making this flag useful for providing better error messages to the user. | -The zero index is least signficant bit in the field. +The zero index is least significant bit in the field. All other bits are reserved and must be zero. +## Version + +The `version` field holds the Parquet format major version that produced the file. This is stored +outside of the Thrift footer so that a reader can always read it, even when the reader cannot parse +(or decrypt) the footer metadata itself. + +The `version` field is a coarse-grained means for a reader to decide whether it can read a +file. The feature flags together with the encoded metadata provide readers a finer-grained +means to determine whether they support the feature set used to write the file, without explicitly +checking `version`. If parsing of metadata fails (e.g. because a new encoding enum that the reader +can't handle was present) version can also be used to give more informative error messages, e.g.: +`Failed to parse file written with major version 'X'; current reader fully supports up to version 'Y'.` + +This is important because some Thrift parsers fail hard when they encounter a unknown enum +value for a required field. To preserve the ability of readers with custom Thrift parsers to still +read as much of the file as possible, the recommended order of operations is: + +1. Verify the `crc32` checksum (see [Integrity Check](#integrity-check)). +2. Validate that the reader supports all set `flags`. +3. Attempt to parse the footer. +4. If footer parsing fails, consult `version` and the `PREVIEW_FEATURES` flag to produce an + appropriate error message, e.g.: + +``` +Failed to parse footer. New version XX detected. +Failed to parse footer written with preview features enabled. +``` + + ## Integrity Check The `crc32` field holds a CRC-32 (ISO 3309 / ITU-T V.42 polynomial, the same used for page level CRC values) computed over the following byte sequence, in order: -1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 16-byte footer tail) -2. The first 8 bytes of the footer tail (metadata_len and flags bitmap) +1. The raw `FileMetaData`/`Footer` bytes (i.e. the `metadata_len` bytes immediately before the 17-byte footer tail) +2. The first 9 bytes of the footer tail (metadata_len, flags bitmap, and version) When `ENCRYPTED_FOOTER` (bit 0) is set, the CRC is computed over the footer bytes **as they appear in the file** (i.e. the encrypted bytes). The CRC itself is always stored unencrypted in the footer tail. -A reader should verify the checksum **before** decrypting the footer, and **after** validating the feature -flags (in the rare case that a feature flag indicates a change in the fixed size tail of the file).