Skip to content

perf(parquet): reuse boolean Bloom filter hashes - #1258

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-boolean-bloom-hashes
Open

perf(parquet): reuse boolean Bloom filter hashes#1258
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-boolean-bloom-hashes

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Precompute the xxHash values for false ([]byte{0}) and true ([]byte{1}).
  • Reuse them in dense and spaced boolean bitmap Bloom-filter paths.
  • Keep custom Hasher implementations on the existing path.

This removes a Sum64 call for every boolean while keeping the same hash values.

Benchmark

Apple M1 Pro, Go 1.26.3, 7 samples, one CPU:

go test -vet=off ./parquet/metadata -run '^$' -bench '^BenchmarkBloomFilter(HashingFromBitmap|BooleanBitmap)' -benchmem -benchtime=150ms -count=7 -cpu=1

The benchmark matrix covers 100K and 1M booleans, all false, all true, alternating, random, and spaced values with 10% and 50% nulls.

Median results for alternating values:

Case Before After
Hash, 100K 636.0 us, 802,819 B/op, 2 allocs 181.6 us, 802,816 B/op, 1 alloc
Insert, 100K 1.840 ms, 8,193 B/op, 2 allocs 943.3 us, 8,193 B/op, 2 allocs
Hash, 1M 7.206 ms, 8,003,596 B/op, 2 allocs 1.510 ms, 8,003,584 B/op, 1 alloc
Insert, 1M 18.832 ms, 8,193 B/op, 2 allocs 9.767 ms, 8,193 B/op, 2 allocs

For alternating data, hashing is 3.5x to 4.8x faster and insertion is about 1.9x faster.

Tests

  • go test ./parquet/metadata
  • PARQUET_TEST_DATA="$PWD/parquet-testing/data" go test ./...
  • go vet ./parquet/metadata

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant