Skip to content

perf(compute): use typed memo insertion for numeric hashes - #1251

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-dict-numeric-memo
Open

perf(compute): use typed memo insertion for numeric hashes#1251
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/compute-dict-numeric-memo

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Use typed memo insertion in the shared numeric hash path.
  • Keep the existing normalized uint8, uint16, uint32, and uint64 mappings.
  • Avoid the per-value interface call to MemoTable.GetOrInsert.
  • Add coverage for int32, int64, float32, float64, and both null modes.
  • Add a numeric dictionary_encode benchmark.

Benchmark

Apple M1 Pro. Compared with main at 6b039a76. The benchmark uses 65,535 values, 100 unique values, 500ms per sample, and 5 samples. The table shows median results.

type main branch change allocs/op
int32 1.168 ms 0.989 ms 15.3% faster 56 -> 56
int64 1.138 ms 1.016 ms 10.8% faster 56 -> 56
float32 1.662 ms 1.114 ms 33.0% faster 64,935 -> 56
float64 1.707 ms 1.251 ms 26.7% faster 64,935 -> 56

Command:

go test ./arrow/compute -run '^$' -bench '^BenchmarkDictionaryEncodeNumeric$' -benchmem -benchtime=500ms -count=5

Checks

  • go test ./arrow/compute/...
  • go test -race ./arrow/compute/internal/kernels ./arrow/compute -run '^(TestDictionaryEncodeNumericTypes|TestDictionaryEncode|TestDictionaryEncodeArrayWithSmallExecChunkSize|TestDictionaryEncodeResizesMemoTable|TestDictionaryEncodeStateResetAfterFinalize)$' -count=1
  • go vet ./arrow/compute/internal/kernels
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant