Skip to content

perf(arrow/array): reuse full chunks in NewChunkedSlice - #1257

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-new-chunked-slice-full-range
Open

perf(arrow/array): reuse full chunks in NewChunkedSlice#1257
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-new-chunked-slice-full-range

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What

  • Reuse the original chunk when NewChunkedSlice selects the full chunk.
  • Keep using NewSlice for partial chunks.
  • Preserve the existing ownership pattern by retaining the reused chunks before passing them to NewChunked.

This avoids creating an ArrayData and typed array wrapper for every full chunk in an aligned slice.

Benchmark

Apple M1 Pro, Go 1.26.3, 7 samples, one CPU:

go test -vet=off ./arrow/array -run '^$' -bench '^BenchmarkNewChunkedSlice$' -benchmem -benchtime=150ms -count=7 -cpu=1

Aligned, non-null input with 64 rows per chunk:

Chunks Before After
64 10.122 us, 13.312 KiB, 131 allocs 2.264 us, 2.312 KiB, 3 allocs
1,024 173.15 us, 212.06 KiB, 2,051 allocs 37.14 us, 36.06 KiB, 3 allocs
8,192 1.383 ms, 1.664 MiB, 16,387 allocs 272.3 us, 256.1 KiB, 3 allocs

Across all 12 benchmark cases, the geometric mean changed from 136.8 us to 29.24 us (-78.63%), 167.5 KiB to 28.44 KiB (-83.02%), and 1,639 to 4.583 allocations (-99.72%).

Tests

  • PARQUET_TEST_DATA=parquet-testing/data ARROW_TEST_DATA=arrow-testing/data go test ./... -count=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant