[improve][build] Require Java 21 for servers while preserving Java 17 client compatibility

.github/workflows/pulsar-ci.yaml

@@ -181,7 +181,9 @@ jobs:
         uses: actions/setup-java@v5
         with:
           distribution: ${{ env.JDK_DISTRIBUTION }}
-          java-version: ${{ env.CI_JDK_MAJOR_VERSION }}
+          java-version: |
+            17
+            ${{ env.CI_JDK_MAJOR_VERSION }}
 
       - name: Setup Gradle
         uses: ./.github/actions/setup-gradle
@@ -194,6 +196,9 @@ jobs:
           ./gradlew assemble rat spotlessCheck checkstyleMain checkstyleTest
           --no-configuration-cache
 
+      - name: Verify client and Functions API Java compatibility
+        run: ./gradlew :tests:pulsar-client-java-compatibility:test -PtestRetryCount=0
+
       - name: Check binary licenses
         run: ./gradlew checkBinaryLicense --no-configuration-cache
 

ARCHITECTURE.md

@@ -94,8 +94,9 @@ concurrency model.
 ## Build infrastructure
 
 Apache Pulsar uses a **Gradle** build (migrated from Maven via PIP-463; some older tooling and docs
-elsewhere still reference Maven). The wrapper `./gradlew` requires **JDK 21, 25 or 26** (bytecode targets
-Java 17). See [`CONTRIBUTING.md` → Building](CONTRIBUTING.md#building) for the build and lint commands.
+elsewhere still reference Maven). The wrapper `./gradlew` requires **JDK 21, 25 or 26** (server bytecode
+targets Java 21; client/API bytecode targets Java 17). See
+[`CONTRIBUTING.md` → Building](CONTRIBUTING.md#building) for the build and lint commands.
 
 - `settings.gradle.kts` — all modules, organized in dependency tiers (Tier 0 has no internal deps,
   higher tiers build on lower ones).
@@ -122,6 +123,28 @@ preserved. Most importantly:
 Always use the Gradle project path (left of any `--tests`), e.g. `./gradlew :pulsar-client-original:test`.
 Check `settings.gradle.kts` when a path is ambiguous.
 
+### Java compatibility boundaries
+
+`pulsar.java-conventions` defaults main sources to Java 21 and explicitly lists the Java 17 client
+and public API dependency closure. This includes both client generations, admin/auth/crypto clients,
+TLS/HTTP SPIs, shared common/package APIs, and the Functions/IO interfaces. Functions implementations,
+brokers, and other server components target Java 21. Client CLI tools also remain Java 17 compatible.
+`pulsarJavaVersion` and `pulsarClientJavaVersion` control these targets (defaults 21 and 17).
+Test sources default to `pulsarJavaVersion` so client tests can use server fixtures; the dedicated
+consumer tests use `pulsarClientJavaVersion` for compilation and their runtime toolchain.
+The bytecode check and published JVM metadata follow the configured client target.
+
+When adding a client dependency, keep its full compile/runtime closure Java 17 compatible. JVM
+variant attributes reject Java 21 project dependencies; `verifyClientJavaCompatibility` also checks
+class-file versions of dependencies without Gradle metadata and the final shaded client jars.
+Multi-release jars are checked using the entries selected by Java 17. This verifies bytecode, not
+all possible reflective or JDK API usage in third-party libraries; the Java 17 consumer tests provide
+runtime coverage. Do not mark a server implementation as Java 17 just to bypass a dependency error.
+
+The client fastutil minimizer reads CLI classes as build-only reachability roots. Its published jar has no
+transitive project dependencies and is checked as Java 17.
+`buildtools` and `testmocks` also stay Java 17 so the consumer compatibility tests can load them.
+
 ### Changing the build
 
 When editing `build-logic/`, `settings.gradle.kts`, a module `build.gradle.kts`, `gradle.properties`,

CONTRIBUTING.md

@@ -29,8 +29,39 @@ workflow (build, test, PR, CI). For the big-picture module map and the Gradle bu
 
 ## Building
 
-**JDK 21, 25 or 26** is required to build `master` (bytecode targets Java 17; `-PskipJavaVersionCheck`
-bypasses the check); `zip` is also needed. Use the bundled wrapper `./gradlew` (Linux/macOS) or
+Standard Pulsar 5 server components and Functions implementations require Java 21 or later.
+Client CLI tools remain Java 17 compatible.
+Client libraries (including V5), their shared dependencies, and Functions/IO public interfaces
+remain Java 17 compatible. Functions compiled on Java 17 can run in a Java 21+ Functions instance.
+The build uses `--release` and publishes the corresponding JVM requirement in Gradle metadata.
+
+`assemble` checks Java 17 client/API bytecode and its compile/runtime dependencies, including shaded
+jars. Run `./gradlew :tests:pulsar-client-java-compatibility:test -PtestRetryCount=0` to compile and run
+consumer examples on an installed JDK 17. Ordinary tests target Java 21 because even client tests
+can depend on broker/Functions fixtures. `-PtestJavaVersion=17` is only suitable for test modules
+whose entire test dependency graph supports Java 17; it does not lower the server baseline.
+
+For custom builds, `-PpulsarJavaVersion=17` targets Java 17 for server code and ordinary test
+sources. `-PpulsarClientJavaVersion=17` controls the client/public API target and consumer test
+toolchain (17 is already the default). For example:
+
+```shell
+./gradlew assemble -PpulsarJavaVersion=17
+PULSAR_MIN_JAVA_VERSION=17 bin/pulsar standalone
+```
+
+Gradle Docker builds bake `pulsarJavaVersion` into the image as `PULSAR_MIN_JAVA_VERSION`.
+Direct Docker builds can set `--build-arg PULSAR_MIN_JAVA_VERSION=17`; both Alpine and Wolfi
+Dockerfiles default to 21. The environment variable can also be overridden when running a container.
+
+`PULSAR_MIN_JAVA_VERSION` overrides the launcher check, which defaults to 21; it does not change
+compiled bytecode or dependency requirements. A custom Java 17 build is only possible while the
+sources and dependencies remain compatible; use of Java 21 features such as virtual threads will
+prevent targeting Java 17. Java 17 server builds are not part of CI. These properties do not change
+Gradle's build-JDK requirement. Use `-PtestJavaVersion=17` as well to run ordinary tests on an installed Java 17 JDK.
+
+**JDK 21, 25 or 26** is required to build `master` (`-PskipJavaVersionCheck` bypasses the check);
+`zip` is also needed. Use the bundled wrapper `./gradlew` (Linux/macOS) or
 `gradlew.bat` (Windows) — no separate Gradle install. See the
 [build-tooling setup guide](https://pulsar.apache.org/contribute/setup-buildtools/) and the
 [IDE setup guide](https://pulsar.apache.org/contribute/setup-ide/).
@@ -106,7 +137,7 @@ entire group. CI splits `pulsar-broker` tests into groups (see
 group are treated as `other` at runtime. `./gradlew verifyTestGroups` reports group assignments and
 flags tests not covered by any CI group.
 
-Other test-related properties: `-PtestJavaVersion=17` (run tests on a different JDK toolchain),
+Other test-related properties: `-PtestJavaVersion=21` (compile and run tests for a different JDK toolchain),
 `-PtestRetryCount=N`, `-PtestFailFast=true|false`, `-PprotobufVersion=4.31.1` (protobuf v4
 compatibility tests).
 

bin/pulsar

@@ -85,6 +85,7 @@ Environment variables:
    PULSAR_PROXY_CONF             Configuration file for Pulsar proxy (default: $DEFAULT_PROXY_CONF)
    PULSAR_WORKER_CONF            Configuration file for functions worker (default: $DEFAULT_WORKER_CONF)
    PULSAR_STANDALONE_CONF        Configuration file for standalone (default: $DEFAULT_STANDALONE_CONF)
+   PULSAR_MIN_JAVA_VERSION       Minimum Java version accepted by the launcher (default: 21)
    PULSAR_EXTRA_OPTS             Extra options to be passed to the jvm
    PULSAR_EXTRA_CLASSPATH        Add extra paths to the pulsar classpath
    PULSAR_PID_DIR                Folder where the pulsar server PID file should be stored
@@ -147,8 +148,9 @@ if [[ -z $JAVA_MAJOR_VERSION ]]; then
   done
 fi
 
-if [[ $JAVA_MAJOR_VERSION -lt 17 ]]; then
-    echo "Error: Pulsar requires Java 17 or later." 1>&2
+PULSAR_MIN_JAVA_VERSION=${PULSAR_MIN_JAVA_VERSION:-21}
+if [[ $JAVA_MAJOR_VERSION -lt $PULSAR_MIN_JAVA_VERSION ]]; then
+    echo "Error: Pulsar server commands require Java $PULSAR_MIN_JAVA_VERSION or later." 1>&2
     exit 1
 fi
 

build-logic/conventions/src/main/kotlin/VerifyJavaCompatibility.kt

@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import org.gradle.api.DefaultTask
+import org.gradle.api.GradleException
+import org.gradle.api.file.ConfigurableFileCollection
+import org.gradle.api.provider.Property
+import org.gradle.api.tasks.Classpath
+import org.gradle.api.tasks.Input
+import org.gradle.api.tasks.TaskAction
+import org.gradle.work.DisableCachingByDefault
+import java.io.DataInputStream
+import java.io.InputStream
+import java.util.jar.JarFile
+
+/** Checks bytecode as well as Gradle's JVM attributes (many Maven jars publish no JVM metadata). */
+@DisableCachingByDefault(because = "Verification has no outputs")
+abstract class VerifyJavaCompatibility : DefaultTask() {
+    @get:Classpath
+    abstract val classpath: ConfigurableFileCollection
+
+    @get:Input
+    abstract val javaVersion: Property<Int>
+
+    @TaskAction
+    fun verify() {
+        val target = javaVersion.get()
+        val failures = mutableListOf<String>()
+        fun inspect(name: String, stream: InputStream) {
+            DataInputStream(stream).use { input ->
+                if (input.readInt() != 0xCAFEBABE.toInt()) {
+                    throw GradleException("Invalid class file: $name")
+                }
+                val minor = input.readUnsignedShort()
+                val major = input.readUnsignedShort()
+                if (major > target + 44 || minor == 65535) {
+                    failures.add("$name requires Java ${major - 44}" + if (minor == 65535) " preview" else "")
+                }
+            }
+        }
+        for (file in classpath.files) {
+            if (file.isDirectory) {
+                file.walkTopDown().filter { it.isFile && it.extension == "class" }.forEach {
+                    inspect(it.path, it.inputStream())
+                }
+            } else if (file.extension == "jar") {
+                JarFile(file).use { jar ->
+                    val multiRelease = jar.manifest?.mainAttributes?.getValue("Multi-Release") == "true"
+                    // Select the same class entries as the target JVM, not classes for newer JVMs.
+                    val selected = mutableMapOf<String, Pair<Int, java.util.jar.JarEntry>>()
+                    for (entry in jar.entries()) {
+                        if (!entry.name.endsWith(".class")) continue
+                        val match = Regex("META-INF/versions/([0-9]+)/(.*)").matchEntire(entry.name)
+                        val version = match?.groupValues?.get(1)?.toInt() ?: 0
+                        if (match != null && (!multiRelease || version > target)) continue
+                        val name = match?.groupValues?.get(2) ?: entry.name
+                        if (version >= (selected[name]?.first ?: -1)) selected[name] = version to entry
+                    }
+                    selected.values.forEach { (_, entry) ->
+                        inspect("${file.name}!/${entry.name}", jar.getInputStream(entry))
+                    }
+                }
+            }
+        }
+        if (failures.isNotEmpty()) {
+            throw GradleException("Java $target compatibility violated:\n" + failures.take(30).joinToString("\n") +
+                if (failures.size > 30) "\n... ${failures.size} incompatible classes in total" else "")
+        }
+    }
+}

build-logic/conventions/src/main/kotlin/pulsar.client-shade-conventions.gradle.kts

@@ -22,6 +22,7 @@ import org.gradle.api.attributes.Bundling
 import org.gradle.api.attributes.Category
 import org.gradle.api.attributes.LibraryElements
 import org.gradle.api.attributes.Usage
+import org.gradle.api.attributes.java.TargetJvmVersion
 import org.gradle.api.component.AdhocComponentWithVariants
 import org.gradle.api.tasks.PathSensitivity
 import java.util.zip.ZipFile
@@ -37,6 +38,7 @@ plugins {
 
 val shadePrefix = "org.apache.pulsar.shade"
 extra["shadePrefix"] = shadePrefix
+val targetJavaVersion = extensions.getByType<JavaPluginExtension>().targetCompatibility.majorVersion.toInt()
 
 // ---- Published dependency scopes for non-bundled dependencies ----
 // The Shadow plugin publishes the `shadow` configuration's dependencies as the dependency-reduced
@@ -66,6 +68,7 @@ val shadowApiElements = configurations.consumable("shadowApiElements") {
         attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.LIBRARY))
         attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements.JAR))
         attribute(Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling.SHADOWED))
+        attribute(TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, targetJavaVersion)
     }
     // Carry the shaded jar so this variant is a complete API variant (like apiElements does for the
     // standard java-library component).

build-logic/conventions/src/main/kotlin/pulsar.java-conventions.gradle.kts

@@ -18,6 +18,7 @@
  */
 
 import java.io.File
+import org.gradle.api.attributes.java.TargetJvmVersion
 
 plugins {
     `java-library`
@@ -65,12 +66,72 @@ configurations.matching { it.name in platformAlignedClasspaths }.configureEach {
     extendsFrom(internalPlatform)
 }
 
+// Java 17 is a compatibility promise for client libraries and user-written Functions/IO APIs.
+// Keep this list explicit: a new server module must not silently lower its baseline, and a new
+// client dependency must be reviewed before joining the Java 17 dependency closure. Gradle's JVM
+// attributes reject project dependencies from this group onto Java 21 modules.
+val clientProjects = setOf(
+    ":pulsar-client-api", ":pulsar-client-api-v5", ":pulsar-client-admin-api",
+    ":pulsar-tls-factory-api", ":pulsar-http-client-api", ":pulsar-common",
+    ":pulsar-client-original", ":pulsar-client-v5", ":pulsar-client-admin-original",
+    ":pulsar-client-auth-athenz", ":pulsar-client-auth-sasl", ":pulsar-client-messagecrypto-bc",
+    ":pulsar-client-shaded", ":pulsar-client-all", ":pulsar-client-admin-shaded",
+    ":pulsar-client-v5-shaded", ":pulsar-client-v5-all", ":pulsar-client-fastutil-minimized",
+    ":pulsar-client-tools-api", ":pulsar-client-tools", ":pulsar-client-tools-test",
+    ":pulsar-client-tools-customcommand-example", ":pulsar-cli-utils",
+    ":pulsar-package-management:pulsar-package-core",
+    ":pulsar-functions:pulsar-functions-api", ":pulsar-io:pulsar-io-core",
+    // Test support must also load in the Java 17 consumer compatibility test JVM.
+    ":buildtools", ":testmocks", ":tests:pulsar-client-java-compatibility",
+)
+val pulsarJavaVersion = providers.gradleProperty("pulsarJavaVersion").map { it.toInt() }.orElse(21)
+val pulsarClientJavaVersion = providers.gradleProperty("pulsarClientJavaVersion").map { it.toInt() }.orElse(17)
+val mainJavaVersion = if (path in clientProjects) pulsarClientJavaVersion.get() else pulsarJavaVersion.get()
+// Client tests can embed the broker and Functions implementation. Test bytecode and dependency
+// resolution therefore have their own baseline, independent of the published main artifact.
+val testJavaVersion = if (path == ":tests:pulsar-client-java-compatibility") {
+    pulsarClientJavaVersion
+} else {
+    providers.gradleProperty("testJavaVersion").map { it.toInt() }
+}
+val testRelease = testJavaVersion.getOrElse(pulsarJavaVersion.get())
+if (path == ":tests:pulsar-client-java-compatibility") {
+    tasks.withType<Test>().configureEach {
+        systemProperty("pulsarClientJavaVersion", pulsarClientJavaVersion.get())
+    }
+}
+java {
+    sourceCompatibility = JavaVersion.toVersion(mainJavaVersion)
+    targetCompatibility = JavaVersion.toVersion(mainJavaVersion)
+}
+configurations.matching { it.name in setOf("testCompileClasspath", "testRuntimeClasspath") }.configureEach {
+    // Follow explicit module overrides too (for example the Java 21 performance tools).
+    attributes.attributeProvider(TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE,
+        tasks.named<JavaCompile>("compileTestJava").flatMap { it.options.release })
+}
+
 tasks.withType<JavaCompile>().configureEach {
     options.encoding = "UTF-8"
-    options.release.set(17)
+    options.release.set(mainJavaVersion)
     options.compilerArgs.addAll(listOf("-parameters", "-Xlint:deprecation", "-Xlint:unchecked"))
 }
 
+tasks.named<JavaCompile>("compileTestJava") {
+    options.release.set(testRelease)
+}
+
+if (path in clientProjects) {
+    val verifyClientJavaCompatibility = tasks.register<VerifyJavaCompatibility>("verifyClientJavaCompatibility") {
+        group = "verification"
+        description = "Check client/API classes and dependencies against pulsarClientJavaVersion."
+        javaVersion.set(pulsarClientJavaVersion)
+        classpath.from(sourceSets.main.get().output.classesDirs,
+            configurations.named("compileClasspath"), configurations.named("runtimeClasspath"))
+    }
+    tasks.named("check") { dependsOn(verifyClientJavaCompatibility) }
+    tasks.named("assemble") { dependsOn(verifyClientJavaCompatibility) }
+}
+
 configurations.all {
     // Force Jackson version to match the version catalog. Transitive dependencies
     // (e.g. from jackson-bom) can pull in newer versions that break API compatibility
@@ -199,8 +260,7 @@ dependencies {
     "testRuntimeOnly"(catalog.findLibrary("log4j-jul").get())
 }
 
-// Allow overriding the JDK used for running tests via -PtestJavaVersion=17
-val testJavaVersion = providers.gradleProperty("testJavaVersion").map { it.toInt() }
+// Allow overriding the JDK used for running tests via -PtestJavaVersion=17.
 val javaToolchains = extensions.getByType<JavaToolchainService>()
 // Effective Java major version used to run tests: the -PtestJavaVersion override when set,
 // otherwise the JVM running Gradle.

build-logic/conventions/src/main/kotlin/pulsar.minimized-dependencies-conventions.gradle.kts

@@ -91,3 +91,8 @@ val verifyMinimizedJar = tasks.register("verifyMinimizedJar") {
 tasks.named("check") {
     dependsOn(verifyMinimizedJar)
 }
+
+// Verify only the published classes: reachability roots are not bundled in this artifact.
+tasks.withType<VerifyJavaCompatibility>().configureEach {
+    classpath.setFrom(tasks.named<ShadowJar>("shadowJar").flatMap { it.archiveFile })
+}

build-logic/conventions/src/main/kotlin/pulsar.shadow-conventions.gradle.kts

@@ -72,3 +72,9 @@ configurations {
         }
     }
 }
+
+// Check the actual shaded contents too, including relocated classes and multi-release entries.
+tasks.withType<VerifyJavaCompatibility>().configureEach {
+    classpath.from(tasks.named<com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar>("shadowJar")
+        .flatMap { it.archiveFile })
+}