Problem
When GeoSeries.fillna(limit=...) receives an independently constructed distributed GeoSeries replacement, it runs positional_join.agg(...).first() before returning in order to choose between positional and label alignment. The eventual result action then recomputes the uncached alignment subtree, and the missing/non-missing union references that subtree twice.
A Spark 3.5 probe confirmed that constructing this result starts Spark jobs while scalar and same-frame replacements remain lazy. The final plan repeats both inputs and does not reuse the alignment exchange. Exact job counts depend on the Spark configuration, but the eager pass and repeated scans are structural.
Desired behavior
Build a lazy distributed plan for independent GeoSeries replacements while preserving the current GeoPandas-compatible rules:
- exact duplicate axes pair positionally;
- a unique replacement index broadcasts to duplicate left labels;
- a non-identical duplicate replacement index raises the pandas-compatible error;
- mixed Index and MultiIndex shapes do not match;
- the exact left axis and natural order are preserved.
A simple unconditional left join would break the duplicate-axis cases. Persisting inside fillna would also require a safe cache lifecycle. This likely needs a plan-level alignment strategy rather than either shortcut.
Follow-up to #2068 and #3302. PR #3302 includes a lazy same-frame fast path; this issue covers arbitrary independent GeoSeries replacements.
Problem
When
GeoSeries.fillna(limit=...)receives an independently constructed distributed GeoSeries replacement, it runspositional_join.agg(...).first()before returning in order to choose between positional and label alignment. The eventual result action then recomputes the uncached alignment subtree, and the missing/non-missing union references that subtree twice.A Spark 3.5 probe confirmed that constructing this result starts Spark jobs while scalar and same-frame replacements remain lazy. The final plan repeats both inputs and does not reuse the alignment exchange. Exact job counts depend on the Spark configuration, but the eager pass and repeated scans are structural.
Desired behavior
Build a lazy distributed plan for independent GeoSeries replacements while preserving the current GeoPandas-compatible rules:
A simple unconditional left join would break the duplicate-axis cases. Persisting inside
fillnawould also require a safe cache lifecycle. This likely needs a plan-level alignment strategy rather than either shortcut.Follow-up to #2068 and #3302. PR #3302 includes a lazy same-frame fast path; this issue covers arbitrary independent GeoSeries replacements.