Skip to content

perf(parquet/pqarrow): write FixedSizeBinary values directly - #1263

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-fixed-size-binary
Open

perf(parquet/pqarrow): write FixedSizeBinary values directly#1263
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-fixed-size-binary

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Writes Arrow FixedSizeBinary values directly from the fixed-width value buffer.
  • Skips null slots with validity bit runs.
  • Keeps stats, bloom filters, byte-stream-split, slices, nested lists, and dictionary paths covered.

Benchmark

Apple M1 Pro. 64K rows. 16-byte values. Median of 3 runs.

case upstream main this PR change
required, stats off 1.77 ms, 5.84 MB 0.50 ms, 4.26 MB -72% time, -27% bytes
required, stats on 2.28 ms, 5.85 MB 1.14 ms, 4.28 MB -50% time, -27% bytes
nullable, stats off 1.92 ms, 5.95 MB 1.13 ms, 4.38 MB -41% time, -26% bytes
nullable, stats on 2.42 ms, 5.95 MB 1.78 ms, 4.39 MB -27% time, -26% bytes

Command:

go test ./parquet/pqarrow -run "^$" -bench "^BenchmarkWriteArrowFixedSizeBinary$" -benchmem -benchtime=2s -count=3

Tests

  • PARQUET_TEST_DATA=/path/to/parquet-testing/data go test ./parquet/...
  • PARQUET_TEST_DATA=/path/to/parquet-testing/data go test -race ./parquet/internal/encoding ./parquet/file ./parquet/metadata ./parquet/pqarrow
  • go vet ./parquet/internal/encoding ./parquet/file ./parquet/metadata ./parquet/pqarrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant