Skip to content

perf(compute): pre-size SetLookupState memo tables - #1253

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/compute-set-lookup-memo-sizing
Open

perf(compute): pre-size SetLookupState memo tables#1253
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/compute-set-lookup-memo-sizing

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Pre-size the memo table used by SetLookupState from the value-set length.
  • Use the memo table's 50% growth threshold when choosing the initial table size.
  • Keep the other hash-state paths at their current zero-size hint.
  • Add a benchmark for numeric and string value sets with unique and repeated values.

Benchmark

Apple M1 Pro. Compared with main at 6b039a76.

For a 64K unique value set:

Case Main B/op This change B/op Main allocs/op This change allocs/op
int64 8.66 MB 3.41 MB 58 52
string 10.13 MB 6.56 MB 64,099 64,062

The main benefit is avoiding repeated table growth and rehashing while building a large lookup set.

Checks

  • go test ./arrow/compute/... -count=1
  • go test -race ./arrow/compute/...
  • go vet ./arrow/compute/internal/kernels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant