Skip to content

fix: Handle nulls correctly when extracting nested arrays from nullable struct/map - #25122

Open
AdamGS wants to merge 3 commits into
apache:mainfrom
AdamGS:adamg/fix-25120
Open

fix: Handle nulls correctly when extracting nested arrays from nullable struct/map#25122
AdamGS wants to merge 3 commits into
apache:mainfrom
AdamGS:adamg/fix-25120

Conversation

@AdamGS

@AdamGS AdamGS commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Fixes null handling when accessing nested map/struct arrays by AND-ing the null buffers as we go down into the array tree.

Running the get_field benchmarks on my M3 macbook, one micro benchmark has a small repeatable regression:

get_field_map_1024_entries_4_first
                        time:   [18.811 µs 18.881 µs 18.954 µs]
                        change: [+6.4039% +6.8632% +7.3094%] (p = 0.00 < 0.05)
                        Performance has regressed.

What changes are included in this PR?

Taking parent nulls into account when extracting nested arrays from map/struct arrays.

What is the testing strategy for this PR?

Additional unit tests and SLT tests.

Are there any user-facing changes?

Correct behavior.

Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
@AdamGS AdamGS changed the title Handle nulls correctly when extracting nested arrays from nullable struct/map fix: Handle nulls correctly when extracting nested arrays from nullable struct/map Sep 9, 2026
Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 9, 2026
@AdamGS
AdamGS marked this pull request as ready for review September 9, 2026 16:05
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.80%. Comparing base (4048898) to head (8a137f8).

Files with missing lines Patch % Lines
datafusion/functions/src/core/getfield.rs 84.94% 3 Missing and 11 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25122      +/-   ##
==========================================
- Coverage   81.80%   81.80%   -0.01%     
==========================================
  Files        1130     1130              
  Lines      417754   417843      +89     
  Branches   417754   417843      +89     
==========================================
+ Hits       341754   341820      +66     
- Misses      55875    55881       +6     
- Partials    20125    20142      +17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested field access on nullable structs discard parent nulls

2 participants