Skip to content

perf(arrow/compute): avoid goroutines for serial record filtering - #1245

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-filter-serial
Open

perf(arrow/compute): avoid goroutines for serial record filtering#1245
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-filter-serial

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Run FilterRecordBatch columns synchronously for serial execution.
  • One-column batches use the direct path even when parallelism is configured.
  • NumParallel <= 1 uses the direct path for all columns.
  • Keep the errgroup path for multi-column parallel execution.
  • Add coverage for one-column, zero/one worker, parallel, null selection, and length errors.

Benchmark

Local Apple M1 Pro run with NumParallel=1:

  • 8 columns / 16 rows: 53.3 us -> 26.2 us, 263 -> 240 allocs/op
  • 128 columns / 16 rows: 709.8 us -> 420.9 us, 3,983 -> 3,720 allocs/op
  • 128 columns / 4,096 rows: 1.07 ms -> 0.77 ms, 4,046 -> 3,783 allocs/op

Tests

  • go test ./...
  • GOOS=linux GOARCH=386 go build ./arrow/compute/...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants