From 31719200a6b63958e5623dd7f603c386f297327e Mon Sep 17 00:00:00 2001 From: Mandy Alimaa Date: Tue, 8 Sep 2026 15:58:31 -0500 Subject: [PATCH 1/2] GH-51143: [Format] Add Parquet FILE canonical extension type --- docs/source/format/CanonicalExtensions.rst | 249 +++++++++++++++++++++ 1 file changed, 249 insertions(+) diff --git a/docs/source/format/CanonicalExtensions.rst b/docs/source/format/CanonicalExtensions.rst index f1faa4a9e741..4d86a2960d27 100644 --- a/docs/source/format/CanonicalExtensions.rst +++ b/docs/source/format/CanonicalExtensions.rst @@ -546,6 +546,255 @@ Primitive Type Mappings | UUID extension type | UUID | +----------------------+------------------------+ +.. _parquet_file_extension: + +Parquet File +============ + +File represents a reference to a range of bytes stored inline or in an external +file. It is intended for use cases such as file inventories, manifests, and +references to unstructured data such as images or audio files in object +storage. This extension type preserves the semantics of the +`Parquet FILE logical type `__ +in Arrow columns. + +* Extension name: ``arrow.parquet.file``. + +* The storage type of this extension is a ``Struct`` containing at least one of + the following nullable fields: + + .. list-table:: + :header-rows: 1 + + * - Field + - Type + * - ``uri`` + - ``String``, ``LargeString``, or ``StringView`` + * - ``offset`` + - ``Int64`` + * - ``size`` + - ``Int64`` + * - ``content_type`` + - ``String``, ``LargeString``, or ``StringView`` + * - ``checksum`` + - ``String``, ``LargeString``, or ``StringView`` + * - ``inline`` + - ``Binary``, ``LargeBinary``, or ``BinaryView`` + + The fields may be in any order and must be accessed by name, not by position. + Field names are case sensitive and must not be renamed. Each field may occur + at most once and may be omitted from the ``Struct`` when it is not used. The + ``Struct`` must not contain fields other than those listed above; additional + metadata about a file should be stored adjacent to the extension type. + +* Extension type parameters: + + This type does not have any parameters. + +* Description of the serialization: + + Extension metadata is an empty string. + +Field semantics +--------------- + +A field is *set* for a value when it is present in the storage ``Struct`` and +non-null. An empty string is considered not set when interpreting the value, +although implementations are not required to convert empty strings to nulls. + +``uri`` +~~~~~~~ + +A URI-reference as defined by RFC 3986, for example +``s3://bucket/file.jpg``. It may be absolute or relative. No additional +encoding, such as URI encoding, is applied to the user-provided string. A URI +is always an external reference, even when it identifies the file containing +the Arrow data. + +``offset`` +~~~~~~~~~~ + +A byte offset indicating the start of the byte range within the referenced +data. If not set, readers must treat the value as 0. If set and non-zero, +readers must seek to this offset to retrieve the referenced data. ``offset`` +may only be set together with ``uri``. ``offset`` must not be ``< 0``. + +``size`` +~~~~~~~~ + +``size`` is the byte length of the referenced data. It must be non-negative; +zero denotes empty referenced data. It must be set whenever ``offset`` is set. +It may be omitted only for a whole-file external reference, in which case the +range extends to the end of the file. + +``content_type`` +~~~~~~~~~~~~~~~~ + +``content_type`` is the media type (MIME type), as defined by RFC 2046, of the +resolved bytes, for example ``image/png``. When it is not set, the type may be +assumed to be ``application/octet-stream``. + +``checksum`` +~~~~~~~~~~~~ + +``checksum`` is a self-describing integrity token of the form +``:``. Readers should ignore unknown algorithms. The +recognized algorithms and digest encodings are: + +.. list-table:: + :header-rows: 1 + + * - Algorithm + - Encoding + - Description + * - ``ETAG`` + - opaque + - The object-store eTag, used only for equality comparison and not + otherwise interpreted. + * - ``MD5`` + - lowercase hexadecimal + - RFC 1321 digest represented by 32 hexadecimal characters. + * - ``CRC32`` + - lowercase hexadecimal + - RFC 2083 checksum represented by 8 hexadecimal characters. + * - ``CRC32C`` + - lowercase hexadecimal + - RFC 3385 checksum represented by 8 hexadecimal characters. + * - ``SHA-256`` + - lowercase hexadecimal + - RFC 6234 digest represented by 64 hexadecimal characters. + +A lowercase hexadecimal digest contains two characters per byte and no +separators, for example ``MD5:d41d8cd98f00b204e9800998ecf8427e``. An opaque +digest is the token supplied verbatim by the object store. A checksum applies +to the resolved bytes, except that ``ETAG`` applies to the whole file +referenced by ``uri``. + +``inline`` +~~~~~~~~~~ + +``inline`` contains the referenced bytes in the value. If locator fields are +also set, they record where those bytes came from. A reader may resolve the +value from ``inline`` or the locator, whichever is suitable. + +Resolution +---------- + +A value resolves to bytes based on which locator fields are set: + +.. list-table:: + :header-rows: 1 + + * - ``inline`` + - ``uri`` + - ``offset`` + - ``size`` + - Resolves to + * - set + - optional + - optional + - optional + - The inline bytes or a valid locator. + * - not set + - set + - not set + - not set + - The whole external file at ``uri``. + * - not set + - set + - set + - not set + - Invalid. + * - not set + - set + - not set + - set + - The external byte range ``[0, size)``. + * - not set + - set + - set + - set + - The external byte range ``[offset, offset + size)``. + * - not set + - not set + - set + - not set + - Invalid. + * - not set + - not set + - not set + - set + - Invalid. + * - not set + - not set + - set + - set + - Invalid. + * - not set + - not set + - not set + - not set + - Invalid; the value does not reference any data. + +When ``inline`` is set, the locator fields may all be unset. Otherwise, the +fields set alongside ``inline`` must form a valid locator on their own. In +particular, ``offset`` requires both ``uri`` and ``size``, and external locator +fields require ``uri``. + +The ``offset`` and ``size`` fields apply only to data referenced by ``uri``; +there is no form that addresses a byte range in the file containing the Arrow +data. The extension type does not apply compression or encryption to the +referenced bytes. Its storage fields, including ``inline``, are ordinary Arrow +arrays and are serialized like any other arrays. + +Validation +---------- + +* A value must set either ``inline`` or ``uri``. A value that sets neither does + not resolve, even if ``offset`` or ``size`` is set. + +* ``offset`` may only be set together with ``uri``. + +* ``size`` must be set whenever ``offset`` is set. + +* If ``inline`` and a locator are both set, producers are expected to write the + same bytes to both sources. Readers are not required to verify this and may + return the bytes from either source. Producers may treat ``inline`` and the + locator fields as mutually exclusive. + +* An invalid file reference may be returned as a null file reference by a + reader. + +Examples +-------- + +A storage type containing all defined fields is:: + + Struct< + uri: String, + offset: Int64, + size: Int64, + content_type: String, + checksum: String, + inline: Binary + > + +All six child fields in this example are nullable. An inline-only storage type +may contain only the fields it uses:: + + Struct< + inline: Binary, + content_type: String + > + +Similarly, a whole-file external reference may use:: + + Struct< + uri: String, + content_type: String, + checksum: String + > + .. _timestamp_with_offset_extension: Timestamp With Offset From 8010628252e1276f8839a70755be790d12e01154 Mon Sep 17 00:00:00 2001 From: Mandy Alimaa Date: Tue, 8 Sep 2026 16:35:14 -0500 Subject: [PATCH 2/2] revision --- docs/source/format/CanonicalExtensions.rst | 25 ++++++++++++++-------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/source/format/CanonicalExtensions.rst b/docs/source/format/CanonicalExtensions.rst index 4d86a2960d27..6aa80f8dce98 100644 --- a/docs/source/format/CanonicalExtensions.rst +++ b/docs/source/format/CanonicalExtensions.rst @@ -581,11 +581,10 @@ in Arrow columns. * - ``inline`` - ``Binary``, ``LargeBinary``, or ``BinaryView`` - The fields may be in any order and must be accessed by name, not by position. - Field names are case sensitive and must not be renamed. Each field may occur - at most once and may be omitted from the ``Struct`` when it is not used. The - ``Struct`` must not contain fields other than those listed above; additional - metadata about a file should be stored adjacent to the extension type. + The fields may be in any order and must be accessed by case-sensitive name, + not by position. Each field may occur at most once and may be omitted from + the ``Struct`` when it is not used. The ``Struct`` must not contain fields + other than those listed above. * Extension type parameters: @@ -750,18 +749,26 @@ arrays and are serialized like any other arrays. Validation ---------- -* A value must set either ``inline`` or ``uri``. A value that sets neither does - not resolve, even if ``offset`` or ``size`` is set. +* A value must resolve to some referenced data. It resolves only if ``inline`` + or ``uri`` is set; if neither is set, the value does not resolve and is + invalid, even if ``offset`` or ``size`` is set. -* ``offset`` may only be set together with ``uri``. +* ``offset`` may only be set together with ``uri``. A value that sets + ``offset`` without ``uri`` does not resolve and is invalid. -* ``size`` must be set whenever ``offset`` is set. +* ``size`` must be set whenever ``offset`` is set. A value that sets ``offset`` + without ``size`` is invalid. * If ``inline`` and a locator are both set, producers are expected to write the same bytes to both sources. Readers are not required to verify this and may return the bytes from either source. Producers may treat ``inline`` and the locator fields as mutually exclusive. +* Field names within the storage ``Struct`` must not be renamed. + +* Additional metadata about the file, such as a modification timestamp, must + be stored adjacent to the extension type, not inside its storage ``Struct``. + * An invalid file reference may be returned as a null file reference by a reader.