GeoPandas: GeoSeries constructor fails for leading-null local input on Spark 3.5

[jiayuasu]

Expected behavior

GeoSeries should accept local geometry data with leading nulls, preserving the nulls, geometries, and row order. Construction should not depend on whether the first row contains a geometry.

Actual behavior

On Spark 3.5.4, both of these fail during construction, before any spatial operation runs:

GeoSeries([None, Point(1, 0)])
GeoSeries(gpd.GeoSeries([None, Point(1, 0)]))

The error is:

pyarrow.lib.ArrowInvalid: Could not convert <POINT (1 0)> with type Point: did not recognize Python value type when inferring an Arrow data type

The equivalent non-null-first inputs and [None, None] succeed. All five cases also succeed with Spark 4.1.1 and the same Python dependencies. Other Spark versions were not tested in this reproduction.

Steps to reproduce

In a Sedona-enabled Spark 3.5.4 environment:

import geopandas as gpd
from shapely.geometry import Point
from sedona.spark import SedonaContext

spark = SedonaContext.create(
    SedonaContext.builder()
    .master("local[2]")
    .config("spark.sql.ansi.enabled", "false")
    .getOrCreate()
)

from sedona.spark.geopandas import GeoSeries

# Fails during construction.
GeoSeries([None, Point(1, 0)])

# Run separately: also fails.
GeoSeries(gpd.GeoSeries([None, Point(1, 0)]))

# Controls: both succeed.
GeoSeries([Point(1, 0), None]).to_geopandas()
GeoSeries([None, None]).to_geopandas()

Cause and scope

The local-input path in GeoSeries.__init__ converts the input to an object-typed pandas Series and delegates to pandas-on-Spark. In Spark 3.5.4, infer_pd_series_spark_type checks only pser.iloc[0] for the geometry's __UDT__ attribute. A leading null bypasses that check, so it tries Arrow inference on the Shapely values and fails.

Spark 4.1.1 checks the first non-null value instead, which explains the passing control run.

This was found while reviewing #3302, but is a separate constructor bug. It reproduces without calling fillna; the constructor is unchanged by that PR. A fix should preserve index, name, CRS, and input order, with regression coverage for leading-null local inputs on Spark 3.5 as well as the existing non-null-first and all-null cases.

Sedona version

Development Python source at b3bd16f87eb32de9fda40ca43ab1e4c8a18e48d3. The constructor is identical to master at f000c4346cdcaadcee17e2ea74e2ef57949a3e52. The Spark 3.5 reproduction used the Sedona 1.9.1 shaded JAR for Spark 3.5 / Scala 2.12.

Apache Spark version

Reproduced on PySpark 3.5.4; passing control on PySpark 4.1.1.

API type

Python

JRE version

17.0.13

Python version

3.11.13

Environment

macOS arm64, local Spark. pandas 2.2.3, GeoPandas 1.1.4, Shapely 2.1.2, PyArrow 16.1.0.

Existing issues

• I searched the existing issues and did not find a duplicate.