[#2771] feat(spark): Support Spark 4.2#2772
Draft
LuciferYang wants to merge 2 commits into
Draft
Conversation
Add a mutually-exclusive `spark4.2` Maven profile that builds the existing
client-spark/spark4 module against Apache Spark 4.2.0, mirroring spark4.1.
The three cross-major APIs called from the spark4 module (MapStatus.apply,
the ExternalSorter ctor, WebUIPage.render) are byte-identical between 4.1.1
and 4.2.0 (verified with javap), so the src/main/java-spark4_1 Spark4Compat
shim and the scala-jakarta servlet variant are reused as-is — no new source.
Dependency pins for the profile:
- spark.version 4.2.0, scala.version 2.13.18
- netty.version 4.2.13.Final (what Spark 4.2.0 ships; still Netty 4.2)
- jackson.version 2.21.2 (jackson-module-scala 2.21.2 requires databind in
[2.21.0, 2.22.0), else RDDOperationScope's initializer throws)
- lz4 at.yawk.lz4:1.11.0 (Spark 4.2 calls LZ4BlockInputStream.newBuilder,
added in the 1.11 line; lz4-java relocated org.lz4 -> at.yawk.lz4 after
1.8.1). Introduce a ${lz4.groupId} property, flipped to at.yawk.lz4 in the
spark4.2 profile; both coordinates are version-managed in root DM.
CI: add a spark4.2 / java 17 matrix entry (parallel.yml) and a -Pspark4.2
step (sequential.yml).
LuciferYang
marked this pull request as draft
July 22, 2026 08:13
…rofile The prior commit added at.yawk.lz4:lz4-java:1.11.0 to the root dependencyManagement with a hardcoded version. Because org.lz4:lz4-java:1.8.1 is a relocation stub that Maven rewrites to at.yawk.lz4:1.8.1, that unconditional 1.11.0 entry version-managed every transitive lz4 consumer (server, client, coordinator, cli, ...) up to 1.11.0 in the default build and all non-spark4.2 profiles -- an out-of-scope, undocumented change (the comments claimed the opposite), overriding the 1.8.1 pin from apache#2693. Route the at.yawk.lz4 version through a new ${lz4.version} property: 1.8.1 by default (restores the pre-PR state for the default build and spark2/3/4/4.1), 1.11.0 only in the spark4.2 profile where Spark needs LZ4BlockInputStream.newBuilder(). Also fix the misleading relocation comments and document that the spark4-shaded org.lz4 exclusion must not be parameterized (Maven matches exclusions on the as-declared coordinate; flipping it would leak at.yawk.lz4 into the shaded jar under -Pspark4.2). Verified: default build client/coordinator/server-common resolve at.yawk.lz4:1.8.1 again; -Pspark4.2 spark4 client + integration tests stay on 1.11.0 and pass; -Pspark4.1 unaffected; spark4.2 shaded jar still has no net.jpountz classes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a mutually-exclusive
spark4.2Maven profile that builds the existingclient-spark/spark4module against Apache Spark 4.2.0, mirroring thespark4.1profile added in #2751.pom.xml: newspark4.2profile — a structural clone ofspark4.1, differing only inspark.version=4.2.0,scala.version=2.13.18,netty.version=4.2.13.Final,jackson.version=2.21.2.client-spark/spark4/pom.xml:spark4.2profile reuses thesrc/main/java-spark4_1compat source root (the 4.1Spark4Compatshim).client-spark/extension/pom.xml:spark4.2profile reuses thescala-jakartaservlet source root.${lz4.groupId}property (org.lz4by default,at.yawk.lz4underspark4.2); the two integration-test modules declare${lz4.groupId}:lz4-java, and both coordinates are version-managed in rootdependencyManagement(org.lz4:1.8.1,at.yawk.lz4:1.11.0).parallel.ymlgains aspark4.2 / java 17matrix entry;sequential.ymlgains a-Pspark4.2step gated onjava-version == 17.No new Java source and no new Maven module.
client-spark/spark4-shadedneeds no change (its bouncycastle/jctools relocations are already global).Why are the changes needed?
Fix: #2771
Users should be able to build and run Uniffle's Spark client against Spark 4.2.0 with full parity to the existing 4.1 support.
The three cross-major APIs called from
client-spark/spark4are byte-identical between 4.1.1 and 4.2.0 (verified withjavap), so no shim change is required:MapStatus.apply(BlockManagerId, long[], long, long)ExternalSorter(…, RowBasedChecksum[])ctorWebUIPage.render(jakarta…HttpServletRequest)Dependency notes:
jackson-module-scala_2.132.21.2, which validates the classpathjackson-databindon registration and throws inRDDOperationScope's static initializer if it is outside[2.21.0, 2.22.0). Same class of failure the 4.1 work hit.NioIoHandler/refCnt handling still applies — no code change).org.lz4toat.yawk.lz4after 1.8.1 (org.lz4only publishes a 1.8.1 relocation stub). Spark 4.2's shuffle path callsLZ4BlockInputStream.newBuilder()(added in the 1.11 line); without the bump the integration tests fail at runtime withNoSuchMethodError. A managedat.yawk.lz4:1.11.0alone does not win because the modules declareorg.lz4and Maven's relocation carries 1.8.1's version — hence the${lz4.groupId}flip.Does this PR introduce any user-facing change?
No behavioral change. Adds a new opt-in
-Pspark4.2build profile. Existing-Pspark4(4.0.2) and-Pspark4.1(4.1.1) are unchanged.How was this patch tested?
Verified locally on JDK 17 (Zulu 17.0.18):
mvn clean install -Pspark4.2 -DskipTests— compiles + shades against 4.2.0 (spark-core/sql/catalyst 4.2.0, scala 2.13.18, netty 4.2.13.Final, jackson 2.21.2, lz4 1.11.0 resolved).mvn clean install -Pspark4.1 -DskipTests+ integration tests — 4.1.1 build andAQERepartitionTest/MapSideCombineTeststill green (lz4 staysorg.lz4:1.8.1); confirms no regression.AQERepartitionTest,MapSideCombineTestpass under-Pspark4.2.TransportFrameDecoderTest(2 tests) passes on Netty 4.2.13 under-Pspark4.2.RepartitionWithLocalFileRssTest(NOOP/ZSTD/LZ4 codecs) passes under-Pspark4.2— exercises the lz4 runtime path.dev/scripts/checkshade.shon the spark4.2 shaded jar — "check success", 0 unshaded classes; bouncycastle/jctools relocated.CI runs the same via the new
spark4.2matrix entry /-Pspark4.2step.