Enable file pruning on nested columns#820
Open
jvansanten wants to merge 7 commits intoduckdb:mainfrom
Open
Conversation
jvansanten
added a commit
to jvansanten/duckdb-iceberg
that referenced
this pull request
Mar 19, 2026
Revert "Add primitive support for conjunction filters" This reverts commit 9584122. This reverts commit c31e003. Better implementation in duckdb#820
Member
|
Changes look great on the outset, but this needs some tests still |
jvansanten
force-pushed
the
filter-nested-columns
branch
from
894940f to
18df3a4
Compare
Tishj
reviewed
Mar 20, 2026
| ---- | ||
| 1 Alice {'street': 123 Main St, 'city': Metropolis, 'zip': 12345} [123-456-7890, 987-654-3210] {age=30, membership=gold} | ||
|
|
||
| # nested-column filters prune files from scan |
Member
There was a problem hiding this comment.
Thanks for the tests, can we extend this to cover:
- multiple types
- deeper nesting
- lists
- maps
- IS NULL / IS NOT NULL on child columns
- IS NULL / IS NOT NULL on structs/lists/maps (not elements)
- conjunction AND on combinations of the above items
Contributor
Author
There was a problem hiding this comment.
I've added a few new tests in test/sql/local/irc_any_catalog/reads/test_nested_column_pruning.test. Files are not currently pruned when filtering on elements of lists or maps, as PushDownFilterIntoExpr() leaves these as generic ExpressionFilters, nor with IS NULL or IS NOT NULL predicates, as FilterCombiner::TryPushdownExpression() does not treat them.
Note that many of these demonstrate cases where filter pushdown does _not_ work.
IcebergPredicateStats::DeserializeBounds refuses to set null bounds, making it impossible to distinguish between columns where the bounds are unset and columns that contain only null. Use has_not_null to disambiguate.
fixes lakekeeper test
Contributor
Author
|
Ahoy, is there anything else that needs to happen before this can be merged? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After thinking more carefully about what I was trying to achieve in #766, I realized that most of my underlying problem was related to the way
FilterCombinertreats nested columns. Given astruct_extractexpression, it wraps the inner filter in aStructFilterand pushes it into the filter set as a filter on the containing column. This has two unhelpful side-effects:ConjunctionAndFilter.This PR attempts to address both of these by inverting the combination and wrapping that
FilterCombinerdoes to create a decomposedTableFilterSetthat refers to individual columns. This is then used both for partition and range pruning inFileMatchesFilter().I suspect it would have been cleaner to apply this transformation once to
IcebergMultiFileList::table_filtersrather than for every single data file, but there appears to be some tight coupling betweenIcebergMultiFileList::table_filtersandIcebergMultiFileReaderthat I can't quite wrap my head around yet. Suggestions welcome!