feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) by zhf999 · Pull Request #375 · alibaba/paimon-cpp

zhf999 · 2026-06-17T07:28:07Z

Purpose

This PR follows PR #371.

Support pushing selection bitmaps down to Parquet page-level filtering so only matching pages are read when both predicate and bitmap pruning are available.
Refactor target row-group filtering metadata into a dedicated TargetRowGroup type to carry row-group index, partial-match status, row ranges, and read-range exclusion flags consistently across planning and reading phases.

Changes

src/paimon/format/parquet/target_row_group.h
- Introduced TargetRowGroup class and TargetRowGroups alias.
- Replaced ad-hoc struct usage from row_ranges.h.
src/paimon/format/parquet/parquet_file_batch_reader.h
- Updated row-group filtering APIs to operate on TargetRowGroups instead of raw std::vector<int32_t>.
src/paimon/format/parquet/parquet_file_batch_reader.cpp
- In SetReadSchema, switched row-group candidate list to TargetRowGroups and applied bitmap filter before page-index filtering.
- Added bitmap-driven row-group/page pruning:
  - If any nested columns are included in read schema, bitmap filtering fallbacks to row-group-level.
  - Translate bitmap to row_ranges to filter page with bitmap.
  - The translation strategy follows Paimon, which prunes unneeded rows at the start and end of each page.

Tests

Added bitmap + page-level integration tests:
- BitmapAllPagesSomeRowGroups
- BitmapPartialPagesSingleRowGroup
- BitmapAllAndPartialPagesMixed
- BitmapAllPagesWithPredicate
- BitmapPartialPagesWithPredicate
- BitmapMixedWithPredicate
- NestedMapBitmapFallback
- NestedListBitmapFallback
- NestedStructBitmapFallback
Coverage includes bitmap-only pruning, bitmap+predicate pruning, and mixed partial/full page hit cases.

Datasets

Multiple performance testing were conducted on 2 dataset.

Dataset	File size	num_rows	num_columns	num_rowgroups	page length (in rows)
Dataset#1	6.7GB	2M	1,176	39	1024
Dataset#2	3GB	11M	4	23	1024

Test Methodology

To validate bitmap behavior, we organize tests into six groups based on bitmap distribution patterns. Each group contains five distinct datasets, and the final result reported is the average of these five runs.

Test Cases

Matching records are contiguous and only one row is queried.
Matching records are contiguous and 100 rows are queried.
Matching records are contiguous and 10,000 rows are queried.
Matching records are scattered in 5 segments, each segment containing 1 row.
Matching records are scattered in 5 segments, each segment containing 100 rows.
Matching records are scattered in 100 segments, each segment containing 1 row.
Scattered bitmap, skip 1 row, read 1 row, skip 1 row, read 1 row until EOF (i.e. 1010101010...).
Scattered bitmap, skip 99 rows , read 1 row, skip 99 rows, skip 1 row until EOF.
Scattered bitmap, skip 9999 rows , read 1 row, skip 9999 rows, skip 1 row until EOF.

Test Results

On Dataset#1

Branch	Case 1	Case 2	Case3	Case 4	Case 5	Case 6	Case 7	Case 8	Case 9
Main	2.27s	2.28s	2.19s	8.74ss	8.75s	105.58s	126.08s	125.96s	126.00s
This PR	0.33s	0.32s	0.90s	1.38s	1.35s	15.82s	126.81s	116.24s	20.77s

On Dataset#2

Branch	Case 1	Case 2	Case3	Case 4	Case 5	Case 6	Case 7	Case 8	Case 9
Main	1.30s	1.26s	1.26s	2.77s	2.76s	30.48s	31.27s	31.28s	31.28s
This PR	8ms	7ms	39ms	25ms	26ms	0.46s	35.18s	32.31s	1.99s

API and Format

No public API changes.
No storage format/protocol changes.

Documentation

No user-facing docs required for this change.

Generative AI tooling

gpt-5.3-codex

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

lxy-9602 · 2026-06-21T01:57:52Z

+            // bitmap pushdown if there is any nested field in the schema
+            PAIMON_ASSIGN_OR_RAISE(target_row_groups,
+                                   FilterRowGroupsByBitmap(selection_bitmap.value(),
+                                                           target_row_groups, has_nested_field));


PARQUET_READ_ENABLE_PAGE_INDEX_FILTER=false is not respected for bitmap page pruning.

SetReadSchema calls FilterRowGroupsByBitmap before reading PARQUET_READ_ENABLE_PAGE_INDEX_FILTER, and FilterPagesByBitmap may still load offset/page index metadata and mark row groups as partially matched. So even when page-index filtering is explicitly disabled, a bitmap that only hits part of a row group can still trigger page-level pruning.

Could we gate bitmap page-level pruning behind the same option? When disabled, bitmap filtering should only prune at row-group level. A regression test with PARQUET_READ_ENABLE_PAGE_INDEX_FILTER=false and a partial-page bitmap would be useful too.

lxy-9602 · 2026-06-21T02:18:44Z

+        }
+    }
+    return -1;
+}


Why does bitmap page pruning derive RowRanges from the first arbitrary column that has an offset index?

Since the bitmap already gives us row ids, the target row ranges can be derived independently of any specific column layout. Using one column’s page layout makes the pruning granularity depend on whichever column FindColumnWithOffsetIndex happens to return. If that column has larger/coarser pages than other columns, we may mark a much wider range as matched and lose a lot of the intended page-pruning benefit.

Could we compute row ranges directly from the bitmap, or at least choose the most selective/fine-grained offset index among the projected columns instead of returning on the first available column?

…ering level

…ge index

…ousBatchFirstRowId to GetGlobalRowId

lxy-9602 · 2026-07-09T05:51:18Z

+    TargetRowGroup FilterRowGroupPagesByBitmap(const RoaringBitmap32& bitmap,
+                                               const TargetRowGroup& row_group,
+                                               const std::vector<int32_t>& column_indices,
+                                               ::parquet::PageIndexReader& page_index_reader) const;


For page_index_reader, please use a pointer if it is an output parameter, and const & if it is an input parameter.

lxy-9602 · 2026-07-09T05:58:38Z

        } else {
-            // all within the range, data before readers_[reader_idx]->GetNextRowToRead() has been
-            // effectively consumed
            readers_pos_[reader_idx]->store(readers_[reader_idx]->GetNextRowToRead());


Why rm this comments?

It's a accident cause by git merge ... Modified.

lxy-9602 · 2026-07-09T06:30:11Z

+///
+/// Schema: { id: int32, props: map<utf8, int32> }
+/// 100 rows, 10 per page, 1 row group.
+/// Predicate: id >= 30 would be a partial-row-group match at first 50-row group.


1 row group or 2 row groups?

The comment has been corrected.

lxy-9602 · 2026-07-09T06:31:46Z

+/// Schema: { id: int32, tags: list<item: int32> }
+/// Predicate: id >= 30 would be a partial-row-group match at first 50-row group.
+/// Because nested schema disables page-level filtering, the entire first row group (0..49) is read.
+TEST_F(PageFilteredRowGroupReaderTest, NestedListBitmapFallback) {


the entire first row group (0..49) is read first or second?

The comment has been corrected.

lxy-9602 · 2026-07-09T08:38:51Z

+        for (auto rg_index : parquet_fragment->row_groups()) {
+            target_row_groups.emplace_back(src_row_groups[rg_index]);
+        }
    }


[P2] Do not index TargetRowGroups by file row-group id

MakeSerialRowGroups may skip empty row groups, so the position in
src_row_groups is not guaranteed to equal the file row-group id returned by
parquet_fragment->row_groups(). FilterRowGroupsByPredicate currently does
src_row_groups[rg_index], which can select the wrong TargetRowGroup or go out
of bounds when earlier row groups were skipped. Please look up TargetRowGroup by
GetRowGroupIndex() instead of treating the row-group id as a vector index.

lxy-9602 · 2026-07-09T08:50:00Z

+        row_ranges = RowRanges::Intersection(row_ranges, page_ranges);
+    }
+    return TargetRowGroup(row_group_idx, true, std::move(row_ranges));
+}


Why do we assume this is always true here? What if the bitmap covers the entire row group?

zjw1111

Thanks for the follow-up work on page-level bitmap pushdown, and for the solid test coverage. A few follow-up nits on stale comments and naming, on top of the points already raised in the earlier review.

zjw1111 · 2026-07-09T09:02:52Z

        }
        // Apply page-level filtering after bitmap pruning so we don't read page index
        // pages for row groups that the bitmap already excluded.
        // If no predicate is provided, skip page-level filtering, row_group_row_ranges will be


This comment still refers to row_group_row_ranges, but that map was removed in this PR (page-filter results now go straight into target_row_groups). Could you update the wording to describe target_row_groups so it no longer points at a variable that doesn't exist anymore?

zjw1111 · 2026-07-09T09:02:52Z

+///
+/// Schema: { id: int32, props: map<utf8, int32> }
+/// 100 rows, 10 per page, 1 row group.
+/// Predicate: id >= 30 would be a partial-row-group match at first 50-row group.


Thanks for adding these fallback tests. Two things in this doc comment don't match the test body:

It says Predicate: id >= 30, but the test passes nullptr as the predicate — only the bitmap (AddRange(70, 100)) drives the filtering.

It says rows [0, 99] should all be returned, but the assertion expects 50 rows (data->Slice(50, 50), i.e. rows 50-99).

The same stale Predicate: id >= 30 line also appears in NestedListBitmapFallback and NestedStructBitmapFallback. Could you refresh these comments to match the actual bitmap-only behavior?

zjw1111 · 2026-07-09T09:02:52Z

+/// Test: bitmap hits partial pages of a row group, with page-filtered option disabled.
+///
+/// 200 rows, 10 rows per page, 100 rows per row group → 2 row groups.
+/// Bitmap: {0..99} hits all of RG0 + {120..149} hits pages 2-4 of RG1.


This comment says the bitmap is {0..99} hits all of RG0 + {120..149}, but the test only does bitmap.AddRange(120, 150) (no {0..99}), and the expected result (100 rows, 100-199) confirms only RG1 is hit. Could you drop the {0..99} part so the comment reflects the actual input?

zjw1111 · 2026-07-09T09:02:52Z

+        return row_ranges_;
+    }
+
+    static TargetRowGroups MakeSerialRowGroups(


nit: MakeSerialRowGroups doesn't quite convey what it does — it builds the initial fully-matched list for all row groups. Would a name like MakeAllRowGroups / MakeFullRowGroups be clearer? A short comment noting that it skips empty (0-row) row groups would also help, since that skipping is exactly what makes list positions diverge from file row-group ids (related to the indexing issue flagged in FilterRowGroupsByPredicate).

zjw1111 · 2026-07-09T09:02:52Z

+
+    // Compute the set of row ranges within a single column's pages that
+    // overlap with the given bitmap. For each page, the bitmap is queried to
+    // find the precise [first, last] matching rows.


Minor wording: 'find the precise [first, last] matching rows' reads as if per-row filtering happens here, but the implementation only trims the non-matching rows at the page boundaries — gaps inside [first, last] are kept (consistent with the page-level trimming design, as ScatteredBitmapTest demonstrates). Could you reword to something like 'the first/last matching row in each page, used to trim the page head/tail'?

zjw1111 · 2026-07-09T09:02:52Z

+using TargetRowGroups = std::vector<TargetRowGroup>;
+class TargetRowGroup {
+ public:
+    explicit TargetRowGroup(int32_t rg_index) : row_group_index_(rg_index) {}


nit: this single-arg constructor produces a TargetRowGroup with an empty row_ranges_, and I don't see a caller for it in this PR (the others use the 3-arg ctor or copy). If it's unused, could we drop it to avoid an ambiguous empty-ranges object? Feel free to ignore if there's a caller I missed.

zhf999 and others added 8 commits June 16, 2026 18:34

test: add test cases for nested columns

94c434a

Merge branch 'main' into nested-col-fallback

4cdf107

walkaround: fallback nested field to RowGroup reading

30ea1d8

style: add comments

bed9fdf

fix: judge has_nested_field on read-schema

dfe419d

fix: make tests clearer

2ceaf60

feat: support page-level bitmap pushdown

cb1a632

fix: fallback bitmap pushdown to rowgroup-level

9a5f4ee

Copilot AI review requested due to automatic review settings June 17, 2026 07:28

Copilot AI reviewed Jun 17, 2026

zhf999 and others added 4 commits June 17, 2026 15:28

style: update tests names

09d2121

stye: update comments

3ea428d

fix: set 'SupportPreciseBitmapSelection to true'

76e7282

Merge branch 'main' into paged-bitmap2

72be706

zhf999 changed the title ~~feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)~~ [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 18, 2026

fxi: set 'SupportPreciseBitmapSelection to false'

8078ca4

zhf999 changed the title ~~[WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)~~ feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 18, 2026

lxy-9602 reviewed Jun 21, 2026

View reviewed changes

zhf999 added 4 commits June 22, 2026 10:06

fix: add assign operator for TargetRowGroup

48cd610

fix: use PARQUET_READ_ENABLE_PAGE_INDEX_FILTER to control bitmap filt…

246e854

…ering level

fix: use bitmap to get row ranges instead of the first column with pa…

d9c952d

…ge index

fix: test

1c495e4

zhf999 changed the title ~~feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)~~ [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 24, 2026

zhf999 added 7 commits June 26, 2026 10:36

fix: FileBatchReader returns discontinuous batch, and change GetPrevi…

5d534db

…ousBatchFirstRowId to GetGlobalRowId

style: change interface name

ad03498

update header files

cd7bd44

Merge branch 'main' into fix-rowid

f9721a8

fix: return Status::Invalid intead of returning max value

41f932d

fix: lance and blob return NotImplemented

6bd98d8

fix: add inclusive extend for fully matched rowgroup in SetReadSchema

a694567

zhf999 added 7 commits June 29, 2026 14:17

fix: small fixes

f1c02db

fix: blob test

ad66b22

fix: blob and lance tests

25ef3d0

fix: unittest

4409afb

Merge branch 'fix-rowid' into paged-bitmap-tmp

5bdca4a

fix: blob

8482e90

Merge branch 'fix-rowid' into paged-bitmap-tmp

a2d7036

zhf999 changed the title ~~[WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)~~ feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 29, 2026

zhf999 marked this pull request as ready for review June 29, 2026 08:30

zhf999 marked this pull request as draft June 29, 2026 10:18

zhf999 added 9 commits July 6, 2026 11:22

feat: use java-like bitmap pushing down stratagy

9e2ed67

fix: bitmap filtering logic

2e8365c

style: modify function interfaces and comments

a27a87e

test: add test

d780597

refractor

cfe72e9

Merge branch 'main' into paged-bitmap2

8b96b96

resolve conflics

3a132b6

fix: enable page level filtering in parquet_file_batch_reader_test.cpp

37b8989

Merge branch 'main' into paged-bitmap2

acd3912

zhf999 marked this pull request as ready for review July 6, 2026 09:34

zhf999 added 4 commits July 7, 2026 11:14

update comments

becc3a8

code style

1244b5f

pre-commit

15f7158

style comment

7c9dc47

lxy-9602 reviewed Jul 9, 2026

View reviewed changes

zjw1111 reviewed Jul 9, 2026

View reviewed changes

zhf999 added 2 commits July 9, 2026 17:30

modify for pr review 1

3be534b

fix: modify for pr review 2

74c658d

zhf999 requested review from lxy-9602 and zjw1111 July 9, 2026 10:10

Uh oh!

Conversation

zhf999 commented Jun 17, 2026 • edited by lxy-9602 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

Tests

Datasets

Test Methodology

Test Cases

Test Results

API and Format

Documentation

Generative AI tooling

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

zhf999 commented Jun 17, 2026 •

edited by lxy-9602

Loading