Skip to content

perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage - #1252

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decode-arena
Open

perf(parquet): decode DELTA_BYTE_ARRAY into contiguous storage#1252
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-decode-arena

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Decode DELTA_BYTE_ARRAY suffixes in one batch.
  • Store reconstructed values in one contiguous byte arena per Decode call.
  • Keep prefix validation before allocation and reconstruction.
  • Keep separate arenas for separate decode calls so earlier results stay valid.
  • Add coverage for partial reads, spaced reads, empty values, page resets, and invalid state.
  • Add benchmarks for high and low prefix sharing with 1K and 64K values.

Benchmark

Apple M1 Pro. Compared with main at 6b039a76. The benchmark uses 65,536 values, 1 second per sample, 5 samples, and GOMAXPROCS=1.

case main branch change allocs/op
prefix-heavy, batch 128 3.678 ms 2.714 ms 26.2% faster 65,553 -> 529
prefix-heavy, full page 3.625 ms 2.727 ms 24.8% faster 65,551 -> 17
low-prefix, batch 128 3.172 ms 2.424 ms 23.6% faster 65,552 -> 529
low-prefix, full page 3.169 ms 2.382 ms 24.8% faster 65,551 -> 17

Command:

GOMAXPROCS=1 go test ./parquet/internal/encoding -run '^$' -bench '^BenchmarkDeltaByteArrayDecoderDecode/(prefix-heavy|low-prefix)/65536/batch-(128|65536)$' -benchmem -benchtime=1s -count=5 -cpu=1

Checks

  • go test ./parquet/internal/encoding -count=1
  • go test -race ./parquet/internal/encoding -count=1
  • go vet ./parquet/internal/encoding
  • go test ./parquet/... -run '^$'
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant