Skip to content

perf(parquet/compress): reuse Brotli writers across encodes - #1254

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-brotli-writer-pool
Open

perf(parquet/compress): reuse Brotli writers across encodes#1254
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-brotli-writer-pool

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Reuse Brotli writers for one-shot Encode and EncodeLevel calls.
  • Keep a separate sync.Pool for each supported compression level.
  • Reset pooled writers with a nil destination before putting them back.
  • Keep the streaming writer APIs unchanged.
  • Add serial and parallel benchmarks plus sequential and concurrent correctness tests.

Benchmark

Compared with main at 6b039a76:

Case Main This PR Main allocations This PR allocations
compressible / 256 KiB ~0.78 ms/op ~0.51 ms/op ~22 allocs/op, ~11.9 MB/op 1 alloc/op, ~10 KB/op
semi-random / 256 KiB ~2.04 ms/op ~1.77 ms/op ~21 allocs/op, ~12.0 MB/op 1 alloc/op, ~37 KB/op

The parallel benchmark also stays at one allocation per operation in the pooled path.

Checks

  • go test ./parquet/compress -count=1
  • go test -race ./parquet/compress
  • go vet ./parquet/compress
  • Parquet package tests with PARQUET_TEST_DATA set

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant