Skip to content

perf(parquet): decode RLE booleans into bitmaps - #1246

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-rle-bool-bitmap
Open

perf(parquet): decode RLE booleans into bitmaps#1246
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-rle-bool-bitmap

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this PR do?

  • Writes repeated RLE boolean runs directly into the output bitmap
  • Copies literal bit-packed runs directly without a []uint64 staging buffer
  • Keeps partial reads and truncated-page errors aligned with the existing decoder

Why?

RleBooleanDecoder.DecodeToBitmap currently decodes through a [1024]uint64 buffer and writes each value one at a time. This change uses bulk bitmap fills for repeated runs and packed bitmap copies for literal runs.

Benchmarks

Apple M1 Pro, GOMAXPROCS=1, -benchmem, 1M values:

pattern before after change
all true 2.65 ms 2.45 us -99.9%
alternating 3.45 ms 0.20 ms -94.2%
short runs 3.50 ms 0.20 ms -94.3%

The benchmark also covers 64K values, unaligned output offsets, and nullable spaced decoding.

Tests

  • go test ./parquet/internal/encoding ./parquet/internal/utils ./parquet/pqarrow
  • go test -race ./parquet/internal/encoding ./parquet/internal/utils ./parquet/pqarrow
  • go test -tags noasm ./parquet/internal/encoding ./parquet/internal/utils

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just needs the merge conflicts to get resolved

@fallintoplace
fallintoplace force-pushed the perf/parquet-rle-bool-bitmap branch from 83cc160 to 7503b83 Compare August 28, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants