Skip to content

perf(parquet): reuse dictionary RLE decoder across pages - #1247

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-dict-rle-reuse
Open

perf(parquet): reuse dictionary RLE decoder across pages#1247
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-dict-rle-reuse

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Reuse the dictionary RLE decoder between data pages.
  • Reset the existing TypedRleDecoder and its bytes.Reader in SetData.
  • Keep the current bit-width validation and empty-page behavior.
  • Add a page-transition regression test covering partial runs, literal runs, and empty data.
  • Add small-page and SetData allocation benchmarks.

Benchmark

Local Apple M1 Pro run with GOMAXPROCS=1: repeated SetData improved from about 0.6-1.0 us / 4.9 KiB / 3 allocs to 6.3 ns / 0 B / 0 allocs.

Tests

  • go test ./parquet/internal/encoding ./parquet/internal/utils
  • go test -race ./parquet/internal/encoding ./parquet/internal/utils
  • go vet -composites=false ./parquet/internal/encoding ./parquet/internal/utils
  • GOOS=linux GOARCH=386 go test -c -o /dev/null ./parquet/internal/encoding

User-facing changes

No.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants