Skip to content

perf(parquet/pqarrow): write record ranges directly - #1248

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-row-group-range-writes
Open

perf(parquet/pqarrow): write record ranges directly#1248
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-row-group-range-writes

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Write row-group ranges directly from a record batch.
  • Reuse one *arrow.Chunked wrapper per column when a batch spans row groups.
  • Keep the existing path when a batch fits in one row group.
  • Preserve buffered row-group boundaries, including full and zero-row cases.
  • Add nested/null round-trip coverage and benchmarks.

Benchmark

Local Apple M1 Pro. -benchtime=500ms -count=3. 8 int64 columns, 8,192 rows, row-group size 256.

Method main change allocs/op
Write ~2.31 ms, 4.95 MB ~2.11 ms, 4.87 MB 21,905 -> 20,761
WriteBuffered ~2.54 ms, 6.69 MB ~2.45 ms, 6.61 MB 25,407 -> 24,263

The wall-time result varies a bit between runs. The allocation reduction is consistent.

Tests

  • go test ./parquet/pqarrow -run '^(TestFileWriter|TestWriteOnClosedFileWriter|TestBufferedRecWrite|TestByteArrayStatisticsStreamingReleaseBetweenBatches)$'
  • go test -race ./parquet/pqarrow -run '^(TestFileWriterRangeWritesPreserveData|TestFileWriterZeroRowRecord)$'
  • go vet -composites=false ./parquet/pqarrow
  • go test ./... -run '^$'
  • GOOS=linux GOARCH=386 go test -c -o /dev/null ./parquet/pqarrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants