Skip to content

perf(parquet/pqarrow): write binary values from Arrow offsets - #1262

Open
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-binary-offsets
Open

perf(parquet/pqarrow): write binary values from Arrow offsets#1262
fallintoplace wants to merge 2 commits into
apache:mainfrom
fallintoplace:perf/parquet-pqarrow-binary-offsets

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Writes Binary, String, LargeBinary, and LargeString values from Arrow buffers and offsets.
  • Removes the full []parquet.ByteArray allocation from non-dictionary writes.
  • Keeps nullable writes, page statistics, bloom filters, page limits, and dictionary fallback behavior.
  • Adds coverage for 32-bit and 64-bit offsets, nulls, slices, small batches, and all byte-array encodings.

Benchmark

64K String rows, uncompressed, dictionary disabled, Apple M1 Pro. Median of 3 runs with 3 seconds per run.

stats before after heap/op
false 1.798 ms 1.260 ms 10.62 MB -> 9.04 MB
true 2.343 ms 1.861 ms 10.62 MB -> 9.06 MB

Benchmark command: go test ./parquet/pqarrow -run '^$' -bench '^BenchmarkWriteArrowBinaryOffsets$' -benchmem -benchtime=3s -count=3

Tests

  • go test ./parquet/...
  • go vet ./parquet/internal/encoding ./parquet/metadata ./parquet/file ./parquet/pqarrow

@zeroshade

Copy link
Copy Markdown
Member

Just needs to fix the linting issue

@fallintoplace
fallintoplace force-pushed the perf/parquet-pqarrow-binary-offsets branch from 672ea23 to 657d17c Compare August 31, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants