Storing GeoParquet with zstd compression

[JakobMiksch]

I am exploring to store GeoParquet files with Sedona Spark (Python). According to this best practice guide (https://github.com/opengeospatial/geoparquet/blob/main/format-specs/distributing-geoparquet.md) the compression type should be zstd. In the docs about string a GeoParquet this option is not mentionend: https://sedona.apache.org/latest/tutorial/files/geoparquet-sedona-spark/

does Sedona support storing GeoParquet with zstd compression? If yes, how does it work?

[jiayuasu]

Yes, Sedona supports zstd (and any Parquet codec) for GeoParquet writes. The GeoParquet writer reuses Spark's standard Parquet compression handling, so you set it the same way you would for a plain Parquet write.

Per-write, via the compression option:

df.write.format("geoparquet") \
    .option("compression", "zstd") \
    .mode("overwrite") \
    .save("path/to/output")

Or globally on the session:

config = SedonaContext.builder() \
    .config("spark.sql.parquet.compression.codec", "zstd") \
    .getOrCreate()

Supported codec values are none/uncompressed, snappy (the default), gzip, lzo, brotli, lz4, and zstd. zstd is bundled in the Parquet library that ships with Spark 3.3+, so no extra/native libraries are needed.

I checked this locally on a small point dataset (5,000 rows). Writing with the default vs. zstd:

Write	Output file	Codec in footer	Size
default	…-c000.snappy.parquet	SNAPPY	146,931 B
compression=zstd	…-c000.zstd.parquet	ZSTD	42,486 B

The output filename gets a .zstd.parquet suffix, the codec shows as ZSTD in the Parquet footer, the file keeps its GeoParquet geo metadata, and it reads back through sedona.read.format("geoparquet") fine.

The option just isn't called out in the GeoParquet docs — worth adding a note there.

[JakobMiksch]

I added a PR to the best practice document in the GeoParquet repo where your solutions is used in the example: opengeospatial/geoparquet#293

[JakobMiksch]

here also a minor update to the Sedona Spark docs #3117