Skip to content

perf(parquet): reuse DELTA byte-array scratch - #1261

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-scratch
Open

perf(parquet): reuse DELTA byte-array scratch#1261
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-delta-byte-array-scratch

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Reuse []parquet.ByteArray scratch for spaced compaction in DELTA_LENGTH_BYTE_ARRAY.
  • Reuse the same scratch pattern in DELTA_BYTE_ARRAY.
  • Clear the scratch after encoding so input payload slices are not retained.
  • Add offset, correctness, reuse, and benchmark coverage.

Benchmark

64K values with 16-byte values on an Apple M1 Pro, Go 1.26.3, 1 CPU, 250 ms per sample, 9 samples. Values are medians.

Encoder / validity ns/op before ns/op after B/op before B/op after allocs/op before allocs/op after
DELTA_LENGTH_BYTE_ARRAY / all valid 1,209,684 886,099 1,577,916 5,000 2,055 2,054
DELTA_LENGTH_BYTE_ARRAY / 50% null 1,049,068 692,072 1,575,636 2,696 1,031 1,030
DELTA_BYTE_ARRAY / all valid 2,618,095 1,957,687 1,633,022 57,918 4,111 4,109
DELTA_BYTE_ARRAY / 50% null 1,776,488 1,141,407 1,604,656 31,066 2,063 2,061

Checks

  • go test ./...
  • go test -race ./parquet/internal/encoding -count=1
  • go vet ./parquet/internal/encoding
  • git diff --check

No public API changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant