Skip to content

Experimental: parquet bounded streaming of large column chunks (companion to arrow-rs#10410) #23795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adriangb wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Experimental: parquet bounded streaming of large column chunks (companion to arrow-rs#10410) #23795

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
069d68d
Add experimental bounded streaming option for reading large parquet c…
adriangb Jul 22, 2026
5e4bc31
TEMPORARY: patch arrow-rs to the apache/arrow-rs#10410 draft branch
adriangb Jul 22, 2026
a2b95df
Update arrow-rs pin: dictionary page cache across decode windows
adriangb Jul 22, 2026
4a9f09d
Add bounded_streaming to remaining ParquetOptions test initializers
adriangb Jul 22, 2026
c69caf7
Fix clippy: box the AdaptiveFetcher variant, collapse nested if
adriangb Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 37 additions & 51 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

21 changes: 21 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,3 +294,24 @@ debug = false
debug-assertions = false
strip = "debuginfo"
incremental = false

# TEMPORARY (remove before merge): build against the arrow-rs bounded streaming
# draft (apache/arrow-rs#10410) until it is released to crates.io.
[patch.crates-io]
arrow = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-arith = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-array = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-avro = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-buffer = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-cast = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-csv = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-data = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-flight = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-ipc = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-json = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-ord = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-row = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-schema = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-select = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
arrow-string = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
parquet = { git = "https://github.com/pydantic/arrow-rs", branch = "bounded-column-streaming" }
11 changes: 11 additions & 0 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1189,6 +1189,17 @@ config_namespace! {
/// parquet reader setting. 0 means no caching.
pub max_predicate_cache_size: Option<usize>, default = None

/// (reading) If true, large parquet column chunks are read with
/// bounded streaming: instead of buffering every projected column
/// chunk of a row group in memory before decoding, chunks over a
/// threshold are fetched with a single ranged request each whose body
/// feeds the decoder incrementally through a bounded buffer. This
/// bounds reader memory by the buffer size rather than the row group
/// size, and overlaps decoding with data transfer. Requires the
/// parquet offset index (see `enable_page_index`); row groups without
/// one, or with pushed-down filters, fall back to the buffered path.
pub bounded_streaming: bool, default = false

// The following options affect writing to parquet files
// and map to parquet::file::properties::WriterProperties

Expand Down
3 changes: 3 additions & 0 deletions datafusion/common/src/file_options/parquet_writer.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ impl ParquetOptions {
pushdown_filters: _,
reorder_filters: _,
force_filter_selections: _, // not used for writer props
bounded_streaming: _, // not used for writer props
allow_single_file_parallelism: _,
maximum_parallel_row_group_writers: _,
maximum_buffered_record_batches_per_stream: _,
Expand Down Expand Up @@ -494,6 +495,7 @@ mod tests {
pushdown_filters: defaults.pushdown_filters,
reorder_filters: defaults.reorder_filters,
force_filter_selections: defaults.force_filter_selections,
bounded_streaming: defaults.bounded_streaming,
allow_single_file_parallelism: defaults.allow_single_file_parallelism,
maximum_parallel_row_group_writers: defaults
.maximum_parallel_row_group_writers,
Expand Down Expand Up @@ -613,6 +615,7 @@ mod tests {
pushdown_filters: global_options_defaults.pushdown_filters,
reorder_filters: global_options_defaults.reorder_filters,
force_filter_selections: global_options_defaults.force_filter_selections,
bounded_streaming: global_options_defaults.bounded_streaming,
allow_single_file_parallelism: global_options_defaults
.allow_single_file_parallelism,
maximum_parallel_row_group_writers: global_options_defaults
Expand Down
Loading
Loading