[GH-1024] Add Sedona R instructions to the AWS EMR setup page

[minhpham1810]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject. Closes Install instructions Sedona/R for AWS EMR? #1024

What changes were proposed in this PR?

Adds a ## Use Sedona in R section to docs/setup/emr.md and its Chinese mirror docs/setup/emr.zh.md.

There is currently no R deployment guidance anywhere in the mkdocs site — both "Install Sedona R" nav entries point at the generated pkgdown site, which documents the R API but not how to get Sedona onto a cluster. The EMR page covers Scala and Python and stops there.

The new section covers three things:

• extending the page's existing bootstrap script to install R, sparklyr and apache.sedona;
• connecting with spark_connect(master = "yarn", spark_home = "/usr/lib/spark"), pointing SEDONA_JAR_FILES at the two jars the bootstrap script already downloads into /jars;
• a one-liner to verify the R side.

The Sedona-specific claims are grounded in R/R/dependencies.R: apache.sedona registers itself as a sparklyr extension whose initializer calls SedonaContext.create(spark_session(sc), "r"), so there is no manual registration step in R; SEDONA_JAR_FILES is split on : and, when set, replaces both Maven coordinates, which is why the geotools-wrapper jar has to be listed alongside the Sedona jar; and the R interface rejects Spark 4.x outright, so this is Spark 3.x only.

Two open questions for reviewers

I don't have an EMR cluster to validate against, and these two would change the text:

1. Which EMR release should this target? The page still documents EMR 6.9.0 / Spark 3.3.0 with sedona-spark-shaded-3.3_2.12. EMR 7.x ships Spark 3.5. I wrote the R section against the page's existing baseline rather than silently changing it, but the whole page looks due a refresh — happy to do that here or in a follow-up, whichever you prefer.

2. Is SEDONA_JAR_FILES actually necessary here? The page's spark-defaults classification already sets spark.yarn.dist.jars to the same two /jars paths. If spark_connect() inherits that, the Sys.setenv() block is redundant and should be presented as a no-internet fallback rather than as the default path.

Three further assumptions on the AWS side are flagged in #1024 (comment) — whether the EMR AMI already ships R, whether core/task nodes need the R packages in YARN client mode, and how R is expected to be driven on EMR now that Studio has no R kernel and the RStudio integration is deprecated.

How was this patch tested?

• prek run --files docs/setup/emr.md docs/setup/emr.zh.md — all hooks pass (markdownlint, codespell, editorconfig-checker, license, whitespace).
• mkdocs build — succeeds; the {{ sedona.current_version }} / {{ sedona.current_geotools }} macros resolve to 1.9.1 and 1.9.1-33.5 in both locales, the three new headings anchor, and both new !!!note admonitions render.
• External links checked for lychee: all resolve 200 after redirects.
• The R snippets themselves were not executed against a live EMR cluster — see the open questions above.

Did this PR include necessary documentation updates?

• Yes, I have updated the documentation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LGHuMH6Jk2ForsfJPEvev6

[jiayuasu]

I ran this on EMR 7.9.0 (Spark 3.5.5) before reviewing: the page's bootstrap from master plus the R lines from this PR, verbatim, on a 1 primary + 1 core m5.xlarge cluster, then a second cluster with the libcurl fix. Two things need to change before this can go in, details in the inline comments:

1. yum install -y R alone doesn't get you sparklyr on EMR 7. The R curl package fails to build because the libcurl headers are missing, and R -e still exits 0 so the bootstrap looks fine. sudo yum install -y R libcurl-devel fixes it.
2. The SEDONA_JAR_FILES snippet still points at the 3.3_2.12 jar. Master moved this page to 3.5_2.12 in #3300, so this needs a rebase.

The note about SEDONA_JAR_FILES replacing both Maven coordinates is also wrong: it only replaces the Sedona one, sparklyr still passes --packages org.datasyslab:geotools-wrapper:... and pulls it from Maven Central.

With those fixed everything in the section works. spark_connect(master = "yarn", spark_home = "/usr/lib/spark") comes up in about 26 s, the verify one-liner prints <POINT (0 0)>, ST_Transform works, and no extra library() is needed because sparklyr exports %>% and collect. The Chinese mirror is a faithful translation and just needs the same edits.

On your open questions:

• EMR release: the page is already on 7.9.0 / Spark 3.5 after #3300, so rebasing is all that's needed. Any 7.x works.
• SEDONA_JAR_FILES is not redundant. spark.yarn.dist.jars only ships the jars to executors; in yarn-client mode the driver never sees them. I checked with plain sparklyr and no apache.sedona loaded: ClassNotFoundException: org.apache.sedona.viz.sql.SedonaVizExtensions on the driver and ST_AsText unresolved. The notebook path only works because EMR runs Livy in cluster deploy mode. Worth a sentence in the doc.
• The AMI doesn't ship R. yum install R gives R 4.5.3 from the AL2023 repos in about 85 s.
• Core/task nodes don't need the R packages for sdf_sql(). An executor-side aggregation ran fine with sparklyr missing on the core node. Only spark_apply() needs R on workers.
• I didn't try RStudio, only Rscript in an EMR step.

Unrelated to this PR but noticed on the way: the existing pip lines break /usr/bin/aws on EMR 7.9 (No module named 'dateutil') and the last pip line aborts on Cannot uninstall packaging 21.3, leaving matplotlib without its deps. That should be a separate issue.

[jiayuasu]

This doesn't work on EMR 7.x as written. The R curl package fails to configure because the libcurl headers aren't on the image, and that takes httr, sparklyr and apache.sedona down with it:

Configuration failed because libcurl was not found. Try installing:
 * rpm: libcurl-devel (Fedora, CentOS, RHEL)
ERROR: dependency 'curl' is not available for package 'httr'
ERROR: dependency 'httr' is not available for package 'sparklyr'
ERROR: dependency 'sparklyr' is not available for package 'apache.sedona'

R -e still exits 0, so the bootstrap reports success and you only find out at library(sparklyr).

sudo yum install -y R libcurl-devel

is enough. I re-ran with that and all 38 packages built on both nodes (about 1.5 min for yum and 6.5 min for install.packages per node). openssl and xml2 are fine already because R-core-devel brings their headers.

Since this runs on every node, it may also be worth making the install fail loudly and use all cores:

install.packages(c("sparklyr", "apache.sedona"), repos = "https://cloud.r-project.org", Ncpus = parallel::detectCores())
stopifnot(all(c("sparklyr", "apache.sedona") %in% rownames(installed.packages())))

[jiayuasu]

This is the Spark 3.3 jar, but since #3300 the bootstrap on master downloads sedona-spark-shaded-3.5_2.12. The hunk merges cleanly so nothing flags it, and spark_connect() dies with FileNotFoundException: File file:/jars/sedona-spark-shaded-3.3_2.12-1.9.1.jar does not exist. Rebase and change this to 3.5_2.12.

[jiayuasu]

This isn't what the code does. In dependencies.R the geotools-wrapper coordinate goes into packages unconditionally; SEDONA_JAR_FILES only drops the Sedona coordinate. With the env var set exactly like the snippet above, spark-submit was launched with

--jars /jars/sedona-spark-shaded-3.5_2.12-1.9.1.jar,/jars/geotools-wrapper-1.9.1-33.5.jar
--packages org.datasyslab:geotools-wrapper:1.9.1-33.5

and ~/.ivy2/jars went from empty to containing the geotools-wrapper jar. So the driver still needs Maven Central (or a warm Ivy cache), and listing the geotools jar is harmless but not for the reason given here. Please reword this, and the "instead of resolving ... from Maven Central" sentence above the code block.

"every connection downloads" is also too strong: Ivy caches, so it's the first connection per user. On EMR that first connect took 26.7 s with the default 60 s timeout (both jars, about 109 MB), the second 24.3 s. The timeout warning is still fair for slow egress, just say "first connection".

[jiayuasu]

Worth saying here that SEDONA_JAR_FILES (or letting sparklyr pull --packages) is required, not an optimisation. spark.yarn.dist.jars only distributes the jars to executors; a yarn-client driver never gets them on its classpath. With plain sparklyr and nothing adding --jars, the driver logged ClassNotFoundException: org.apache.sedona.viz.sql.SedonaVizExtensions and ST_AsText was unresolved. The Jupyter section works only because EMR runs Livy in cluster deploy mode. Without a sentence on this, readers will drop the env var assuming spark-defaults covers it, which is what your open question 2 asks.

[jiayuasu]

A bootstrap action runs on every node, not just the one you run R from. That's fine in practice, and workers only need the R packages for spark_apply() (an executor-side aggregation ran without sparklyr on the core node), but the wording suggests otherwise.

[jiayuasu]

Maybe "apache.sedona 1.9.1 supports Spark 3.x only". Spark 4 support went in with #3340 for the next R release, so this will be stale soon without the version.

[jiayuasu]

Confirmed, no change needed. One side effect worth knowing: attaching the package sets spark.kryo.registrator to SedonaVizKryoRegistrator through an R option, which sparklyr passes as --conf, overriding the SedonaKryoRegistrator from spark-defaults. Harmless, it's a superset.

[jiayuasu]

Same edits needed here: libcurl-devel, 3.5_2.12, and the two reworded notes. The translation itself reads fine.