Skip to content

perf: Compute array_length of first dimension from offsets - #25121

Open
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/perf-array-length-from-offsets
Open

perf: Compute array_length of first dimension from offsets#25121
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/perf-array-length-from-offsets

Conversation

@neilconway

@neilconway neilconway commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

array_length worked by forming a slice of values for each logical row in the given dimension, and then appending the length of that slice to the result. That is inefficient; when computing the length of the first array dimension, we can get that from the offset buffer instead, which is ~200x faster for typical inputs. For other array dimensions, we fallback to the previous code path.

Benchmarks: (M4 Max)

  • list/8192, 175.379 µs -> 1.137 µs, -99.35%
  • fixed_size_list/8192, 174.770 µs -> 0.963 µs, -99.45%
  • list/1, 0.171 µs -> 0.088 µs, -48.48%

What changes are included in this PR?

  • Compute array_length of dimension 1 for List and LargeList from offsets
  • Compute array_length of dimension 1 for FixedSizeList from the FSL's size
  • Treat array_length(x, 1) as equivalent to array_length(x)
  • Simplify the code to remove a no-longer-necessary macro
  • Add unit tests
  • Add benchmark for array_length

What is the testing strategy for this PR?

Existing tests pass; new unit tests added.

Are there any user-facing changes?

No.

@AdamGS AdamGS left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome improvement, LGTM!

@neilconway

Copy link
Copy Markdown
Contributor Author

@AdamGS @Rich-T-kid Thanks for the reviews!

@github-actions github-actions Bot added the functions Changes to functions implementation label Sep 9, 2026
@neilconway
neilconway enabled auto-merge September 9, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compute array_length from offsets

3 participants