Skip to content

perf(parquet/pqarrow): avoid goroutines for serial struct reads - #1250

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-struct-serial
Open

perf(parquet/pqarrow): avoid goroutines for serial struct reads#1250
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-struct-serial

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Use direct child loops in structReader.SeekToRow and LoadBatch when Parallel=false.
  • Keep the existing errgroup path for parallel reads.
  • Continue visiting every child after an error and return the first error, matching the old serial behavior.
  • Add a regression test and a nested struct benchmark.

Benchmark

Apple M1 Pro. upstream/main at 6b039a76. 500ms per sample, 5 samples.

The benchmark reads a 1024-row nested struct across 16 uncompressed row groups and includes SeekToRow(0) plus NextBatch(1024).

children main this PR change allocs/op
1 120.5 us 103.6 us 14.1% faster 723 -> 715
8 869.1 us 784.0 us 9.8% faster 5708 -> 5672
32 2.927 ms 2.614 ms 10.7% faster 23300 -> 23166
128 10.017 ms 9.140 ms 8.8% faster 93066 -> 92549

Command:

go test ./parquet/pqarrow -run '^$' -bench '^BenchmarkReadNestedStructSerial$' -benchmem -benchtime=500ms -count=5

Checks

  • PARQUET_TEST_DATA=<parquet-testing-data> go test ./parquet/...
  • go test -race ./parquet/pqarrow -run '^(TestStructReaderSerialOperationsVisitEveryChild|TestRecordReaderSerial|TestRecordReaderParallel|TestRecordReaderSeekToRow|TestPartialStructColumnRead)$' -count=1
  • go vet ./parquet/pqarrow
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants