GH-530: Add modular footer array-page specification - #611
Closed
Jiayi-Wang-db wants to merge 7 commits into
Closed
Conversation
A struct-of-arrays, column-major, modular replacement for FileMetaData that lets a reader decode work proportional to the projection instead of the table width. - Small always-read core (footer scalars, inline schema and placement, and a directory); optional modules (row-group stats, offset index, column index, file metadata) located by a directory of absolute file offsets. - The page index lives outside the footer; offset index and column index are separate modules, each located via a per-column PageIndexDirectory (relative cumulative offsets), so a projection fetches only its columns' blobs. - BITPACK integer arrays for O(1) column-selective random access; VarLenColumn for min/max with common-prefix dedup and inexact-truncation flags. - Forward compatibility via Parquet versioning; adoption targets the sole footer; encryption keeps placement plaintext and encrypts per-column statistics. Co-authored-by: Isaac
Make the offset index and column index per column chunk (per (column, row group)), matching standard Parquet's per-chunk OffsetIndex/ColumnIndex, instead of one blob per leaf column spanning all row groups: - OffsetIndexColumn/ColumnIndexColumn -> OffsetIndexChunk/ColumnIndexChunk, each holding a single chunk's per-page arrays. - Drop first_page_index: each blob is one chunk, so its parallel per-page arrays self-delimit (their common length is the page count); no page-count field. - boundary_orders (per-row-group array) -> boundary_order (a single parquet.BoundaryOrder scalar per chunk). - PageIndexDirectory.column_offsets (num_columns+1) -> chunk_offsets (num_chunks+1, column-major), so a reader fetches page index only for surviving (column, row group) chunks after row-group pruning; a whole column's chunks stay contiguous for a single range fetch. Co-authored-by: Isaac
Clarify why the presence bitsets exist: a column-oriented positional array cannot distinguish a genuine 0 (or empty bytes) from an unset value, so an optional field pairs its value array with a presence bitset (has_null_count, has_minmax, ...) that records which entries are actually set -- restoring the optional/absent semantics the array-of-structs FileMetaData gets from Thrift field presence. Also note in the bitset definition that a bitset serves two roles: boolean data (null_pages, is_fully_dict_encoded) vs a presence flag gating a paired value array. Co-authored-by: Isaac
…tatsMatrix -> RowGroupStatsMatrix No module is carried inline anymore. ModularFooter now holds only the footer-level scalars plus a required directory[] that locates every module (SCHEMA and PLACEMENT always present; the rest optional). Also renames the stats module RgStatsMatrix -> RowGroupStatsMatrix and the enum value RG_STATS -> ROW_GROUP_STATS. Co-authored-by: Isaac <no-reply@databricks.com>
…y_dict_encoded docs - Header: per-chunk (column-major) unless noted; numeric arrays BITPACK-encoded for column-selective random access. - Explain first_dict_page + dictionary_page_offsets: a flat offset array plus a per-chunk cumulative (CSR) index, so a chunk can hold zero, one, or several dictionary pages without a format change. - Explain why is_fully_dict_encoded replaces ColumnMetaData.encoding_stats: the fast path only needs whether the whole chunk is dictionary-encoded, so a single per-chunk bit replaces the variable-length list<PageEncodingStats>. Co-authored-by: Isaac <no-reply@databricks.com>
…s evidence Keep the single BITPACK encoding (simplest) but document the encoding goals (random access, align with Parquet encodings, minimal size) and the plan to benchmark candidate encodings against real fleet footers before adopting one. Motivated by a real 2.29M-chunk footer where sparse min/max full-length offsets cost ~10 MB to index ~0.44 MB of values. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Jiayi <wjyi2015@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The traditional array-of-structs footer requires readers to walk metadata for every column even
when a query projects only a few columns. The modular footer transposes metadata into independent
column-major modules, but storing every encoded array as a Thrift
binaryfield obscures theboundary between the Thrift control structures and the actual array encoding.
This draft explores an explicit metadata-page model analogous to Parquet data pages: a
Compact-Thrift header followed by a raw, uncompressed array payload. It keeps placement,
statistics, and page indexes as independent modules and supports projected random access without
coupling placement to optional statistics.
What changes are included in this PR?
evaluation criteria.
BIT_PACKEDfor dense arrays andSPARSEfor sparse optional values.Do these changes have PoC implementations?
Not yet. This is a draft specification intended to establish the wire model before updating the
existing modular-footer PoC. The proposal lists the required size, projection, sparse-statistics,
malformed-input, and cross-implementation evaluations.