Skip to content

Commit 20fc417

Browse files
lqiu96lqiu96
authored
ci: trigger downstream module integration tests on upstream component changes (#14222)
## Problem Following #14217, integration tests for split modules only run when their own module directory is modified or when global parent POMs change. However, some modules depend on other sibling modules in the monorepo: - `java-bigquery` depends on `java-bigquerystorage` - `java-bigquery-jdbc` depends on `java-bigquery` and `java-bigquerystorage` - `java-spanner` depends on `grpc-gcp-java` - `java-spanner-jdbc` depends on `java-spanner` and `grpc-gcp-java` - `java-storage-nio` depends on `java-storage` - `java-logging-logback` depends on `java-logging` Additionally, changes in `sdk-platform-java` or `google-auth-library-java` impact all client libraries across the entire repository and should trigger all integration tests. ## Changes - **Global Triggers (`should_test_all_modules`)**: Updated in `.kokoro/common.sh` to trigger monorepo-wide integration tests if either `sdk-platform-java` or `google-auth-library-java` is modified. - **Upstream Mapping (`get_upstream_modules` & `is_upstream_module_modified`)**: Added in `.kokoro/common.sh` to map intra-monorepo upstream dependencies and check if any upstream module was modified in the PR diff. - **Split IT & GraalVM Jobs (`.kokoro/build.sh`)**: Updated `integration-single` and `graalvm-single` to execute when an upstream dependency is modified. - **Batch Jobs (`generate_modified_modules_list`)**: Updated to include downstream modules when their upstream dependencies are modified. - **Unit Tests (`.kokoro/common_test.sh`)**: Added test coverage for `sdk-platform-java`, `google-auth-library-java`, and all upstream-downstream combinations.
1 parent 5e9e3cd commit 20fc417

3 files changed

Lines changed: 210 additions & 21 deletions

File tree

‎.kokoro/build.sh‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -121,12 +121,14 @@ case ${JOB_TYPE} in
121121
integration-single)
122122
if [[ "$(release_please_snapshot_pull_request)" == "true" ]]; then
123123
echo "Not running integration checks -- this is Release Please SNAPSHOT pull request."
124-
# Run tests if either global overrides require testing all modules (e.g. parent POM or
125-
# shared dependencies) OR if this specific module was modified. Otherwise skip.
126-
elif ! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"; then
127-
echo "${BUILD_SUBDIR} not modified, skipping split integration test"
124+
# Run tests if:
125+
# 1. Global overrides require testing all modules (e.g. parent POM, sdk-platform-java, auth)
126+
# 2. This specific module was modified
127+
# 3. An upstream dependency of this module was modified (e.g. java-spanner or grpc-gcp-java for java-spanner-jdbc)
128+
elif ! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}" && ! is_upstream_module_modified "${BUILD_SUBDIR}"; then
129+
echo "${BUILD_SUBDIR} not modified and no upstream dependencies modified, skipping split integration test"
128130
else
129-
echo "${BUILD_SUBDIR} modified, running split integration test"
131+
echo "${BUILD_SUBDIR} (or an upstream dependency) modified, running split integration test"
130132
echo "Compiling and building all modules for ${BUILD_SUBDIR}"
131133
install_modules "${BUILD_SUBDIR}"
132134
echo "Running in subdir: ${BUILD_SUBDIR}"
@@ -191,12 +193,14 @@ case ${JOB_TYPE} in
191193
graalvm-single)
192194
if [[ "$(release_please_snapshot_pull_request)" == "true" ]]; then
193195
echo "Not running GraalVM checks -- this is Release Please SNAPSHOT pull request."
194-
# Run tests if either global overrides require testing all modules (e.g. parent POM or
195-
# shared dependencies) OR if this specific module was modified. Otherwise skip.
196-
elif ! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}"; then
197-
echo "${BUILD_SUBDIR} not modified, skipping split GraalVM test"
196+
# Run tests if:
197+
# 1. Global overrides require testing all modules (e.g. parent POM, sdk-platform-java, auth)
198+
# 2. This specific module was modified
199+
# 3. An upstream dependency of this module was modified (e.g. java-spanner or grpc-gcp-java for java-spanner-jdbc)
200+
elif ! should_test_all_modules && ! is_module_modified "${BUILD_SUBDIR}" && ! is_upstream_module_modified "${BUILD_SUBDIR}"; then
201+
echo "${BUILD_SUBDIR} not modified and no upstream dependencies modified, skipping split GraalVM test"
198202
else
199-
echo "${BUILD_SUBDIR} modified, running split GraalVM test"
203+
echo "${BUILD_SUBDIR} (or an upstream dependency) modified, running split GraalVM test"
200204
echo "Compiling and building all modules for ${BUILD_SUBDIR}"
201205
install_modules "${BUILD_SUBDIR}"
202206
echo "Running in subdir: ${BUILD_SUBDIR}"

‎.kokoro/common.sh‎

Lines changed: 72 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -250,12 +250,14 @@ function get_modified_files() {
250250

251251
# Determines if the entire monorepo must be tested.
252252
#
253-
# Monorepo-wide testing is triggered under three conditions:
253+
# Monorepo-wide testing is triggered under four conditions:
254254
# 1. TEST_ALL_MODULES is set to "true" (used by nightly and scheduled CI builds).
255255
# 2. Root parent POMs (google-cloud-jar-parent or google-cloud-pom-parent) are modified,
256256
# as changes to parent POMs affect shared dependency versions and compiler/build plugins.
257-
# 3. Core shared dependencies (sdk-platform-java/java-shared-dependencies) are modified,
258-
# as gax, auth, and transport changes can break downstream client library integration tests.
257+
# 3. Core SDK platform libraries (sdk-platform-java) are modified, as gax, generators,
258+
# and core transport changes can break downstream client library integration tests.
259+
# 4. Core authentication libraries (google-auth-library-java) are modified, as auth/credential
260+
# changes affect all client libraries.
259261
function should_test_all_modules() {
260262
local files
261263
files=$(get_modified_files)
@@ -264,7 +266,8 @@ function should_test_all_modules() {
264266
# stdin of grep, avoiding an external subshell pipeline (like 'echo "$var" | grep').
265267
if [[ "${TEST_ALL_MODULES}" == "true" ]] || \
266268
grep -q -E '^google-cloud-(pom|jar)-parent/pom.xml$' <<< "${files}" || \
267-
grep -q -E '^sdk-platform-java/java-shared-dependencies/' <<< "${files}"; then
269+
grep -q -E '^sdk-platform-java/' <<< "${files}" || \
270+
grep -q -E '^google-auth-library-java/' <<< "${files}"; then
268271
return 0
269272
fi
270273
return 1
@@ -317,6 +320,16 @@ function generate_modified_modules_list() {
317320
else
318321
echo "Found no changes in the java modules"
319322
fi
323+
324+
# Also include downstream modules if any of their upstream dependencies were modified,
325+
# ensuring batch integration tests cover dependent client libraries.
326+
for module in "${maven_modules[@]}"; do
327+
if is_upstream_module_modified "${module}"; then
328+
if [[ ! " ${modified_module_list[*]} " =~ " ${module} " ]]; then
329+
modified_module_list+=("${module}")
330+
fi
331+
fi
332+
done
320333
fi
321334
}
322335

@@ -337,6 +350,61 @@ function is_module_modified() {
337350
grep -q -E "^${module}/" <<< "${files}"
338351
}
339352

353+
# Maps a module to its intra-monorepo upstream dependencies.
354+
#
355+
# Certain libraries in this repository directly depend on sibling modules (for example,
356+
# java-bigquery depends on java-bigquerystorage, JDBC drivers wrap client SDKs, and
357+
# java-spanner depends on grpc-gcp-java). Without this mapping, changes to an upstream
358+
# dependency would not trigger integration tests for downstream consumers in PR CI.
359+
#
360+
# Returns space-separated module names that the given module depends on, or empty if none.
361+
function get_upstream_modules() {
362+
local module="$1"
363+
case "${module}" in
364+
java-bigquery)
365+
echo "java-bigquerystorage"
366+
;;
367+
java-bigquery-jdbc)
368+
echo "java-bigquery java-bigquerystorage"
369+
;;
370+
java-spanner)
371+
echo "grpc-gcp-java"
372+
;;
373+
java-spanner-jdbc)
374+
echo "java-spanner grpc-gcp-java"
375+
;;
376+
java-storage-nio)
377+
echo "java-storage"
378+
;;
379+
java-logging-logback)
380+
echo "java-logging"
381+
;;
382+
*)
383+
;;
384+
esac
385+
}
386+
387+
# Checks if any upstream dependency of the given module was modified in the PR diff.
388+
#
389+
# Takes a module name (e.g. BUILD_SUBDIR), retrieves its upstream dependencies using
390+
# get_upstream_modules, and checks if any of those upstream directories were touched.
391+
# Returns 0 (true) if an upstream module was modified, triggering downstream tests;
392+
# otherwise returns 1 (false).
393+
function is_upstream_module_modified() {
394+
local module="$1"
395+
if [[ -z "${module}" ]]; then
396+
return 1
397+
fi
398+
399+
local upstream
400+
for upstream in $(get_upstream_modules "${module}"); do
401+
if is_module_modified "${upstream}"; then
402+
return 0
403+
fi
404+
done
405+
return 1
406+
}
407+
340408
# Filters the modified_module_list to only include modules that contain
341409
# integration test files (matching IT*.java or *IT.java in src/test/java).
342410
# Not all modules will have ITs written and there is not need to test

‎.kokoro/common_test.sh‎

Lines changed: 124 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -155,17 +155,31 @@ function test_should_test_all_modules {
155155
exit 1
156156
fi
157157

158-
# Core shared dependencies
159-
TEST_MODIFIED_FILES="sdk-platform-java/java-shared-dependencies/pom.xml"
158+
# Core SDK platform
159+
TEST_MODIFIED_FILES="sdk-platform-java/gapic-generator/src/main/Foo.java"
160160
if ! should_test_all_modules; then
161-
echo "should_test_all_modules should return true for java-shared-dependencies change"
161+
echo "should_test_all_modules should return true for sdk-platform-java change"
162162
exit 1
163163
fi
164164

165-
# Prefix collision check: sibling paths starting with java-shared-dependencies must not match
166-
TEST_MODIFIED_FILES="sdk-platform-java/java-shared-dependencies-bom/pom.xml"
165+
# Prefix collision check: sibling paths starting with sdk-platform-java must not match
166+
TEST_MODIFIED_FILES="sdk-platform-java-extra/pom.xml"
167167
if should_test_all_modules; then
168-
echo "should_test_all_modules should return false for java-shared-dependencies prefix match"
168+
echo "should_test_all_modules should return false for sdk-platform-java prefix match"
169+
exit 1
170+
fi
171+
172+
# Core auth library
173+
TEST_MODIFIED_FILES="google-auth-library-java/oauth2_http/src/main/Foo.java"
174+
if ! should_test_all_modules; then
175+
echo "should_test_all_modules should return true for google-auth-library-java change"
176+
exit 1
177+
fi
178+
179+
# Prefix collision check: sibling paths starting with google-auth-library-java must not match
180+
TEST_MODIFIED_FILES="google-auth-library-java-extra/pom.xml"
181+
if should_test_all_modules; then
182+
echo "should_test_all_modules should return false for google-auth-library-java prefix match"
169183
exit 1
170184
fi
171185

@@ -179,6 +193,109 @@ function test_should_test_all_modules {
179193
unset TEST_MODIFIED_FILES
180194
}
181195

196+
# Test is_upstream_module_modified triggers correctly for downstream components.
197+
function test_is_upstream_module_modified {
198+
# Empty input should return false (1)
199+
if is_upstream_module_modified ""; then
200+
echo "is_upstream_module_modified should return false for empty input"
201+
exit 1
202+
fi
203+
204+
# 1. java-bigquerystorage modified:
205+
# - java-bigquery should trigger
206+
# - java-bigquery-jdbc should trigger
207+
# - java-bigquerystorage itself should NOT have upstream trigger
208+
TEST_MODIFIED_FILES="java-bigquerystorage/src/main/Foo.java"
209+
if ! is_upstream_module_modified "java-bigquery"; then
210+
echo "is_upstream_module_modified should return true for java-bigquery when java-bigquerystorage is modified"
211+
exit 1
212+
fi
213+
if ! is_upstream_module_modified "java-bigquery-jdbc"; then
214+
echo "is_upstream_module_modified should return true for java-bigquery-jdbc when java-bigquerystorage is modified"
215+
exit 1
216+
fi
217+
if is_upstream_module_modified "java-bigquerystorage"; then
218+
echo "is_upstream_module_modified should return false for java-bigquerystorage when java-bigquerystorage is modified"
219+
exit 1
220+
fi
221+
222+
# 2. java-bigquery modified:
223+
# - java-bigquery-jdbc should trigger
224+
# - java-bigquerystorage should NOT trigger
225+
TEST_MODIFIED_FILES="java-bigquery/src/main/Foo.java"
226+
if ! is_upstream_module_modified "java-bigquery-jdbc"; then
227+
echo "is_upstream_module_modified should return true for java-bigquery-jdbc when java-bigquery is modified"
228+
exit 1
229+
fi
230+
if is_upstream_module_modified "java-bigquerystorage"; then
231+
echo "is_upstream_module_modified should return false for java-bigquerystorage when java-bigquery is modified"
232+
exit 1
233+
fi
234+
235+
# 3. grpc-gcp-java modified:
236+
# - java-spanner should trigger
237+
# - java-spanner-jdbc should trigger
238+
TEST_MODIFIED_FILES="grpc-gcp-java/src/main/Foo.java"
239+
if ! is_upstream_module_modified "java-spanner"; then
240+
echo "is_upstream_module_modified should return true for java-spanner when grpc-gcp-java is modified"
241+
exit 1
242+
fi
243+
if ! is_upstream_module_modified "java-spanner-jdbc"; then
244+
echo "is_upstream_module_modified should return true for java-spanner-jdbc when grpc-gcp-java is modified"
245+
exit 1
246+
fi
247+
248+
# 4. java-spanner modified:
249+
# - java-spanner-jdbc should trigger
250+
# - java-spanner itself should NOT have upstream trigger
251+
TEST_MODIFIED_FILES="java-spanner/src/main/Foo.java"
252+
if ! is_upstream_module_modified "java-spanner-jdbc"; then
253+
echo "is_upstream_module_modified should return true for java-spanner-jdbc when java-spanner is modified"
254+
exit 1
255+
fi
256+
if is_upstream_module_modified "java-spanner"; then
257+
echo "is_upstream_module_modified should return false for java-spanner when java-spanner is modified"
258+
exit 1
259+
fi
260+
261+
# 5. java-storage modified:
262+
# - java-storage-nio should trigger
263+
# - java-storage itself should NOT have upstream trigger
264+
TEST_MODIFIED_FILES="java-storage/src/main/Foo.java"
265+
if ! is_upstream_module_modified "java-storage-nio"; then
266+
echo "is_upstream_module_modified should return true for java-storage-nio when java-storage is modified"
267+
exit 1
268+
fi
269+
if is_upstream_module_modified "java-storage"; then
270+
echo "is_upstream_module_modified should return false for java-storage when java-storage is modified"
271+
exit 1
272+
fi
273+
274+
# 6. java-logging modified:
275+
# - java-logging-logback should trigger
276+
# - java-logging itself should NOT have upstream trigger
277+
TEST_MODIFIED_FILES="java-logging/src/main/Foo.java"
278+
if ! is_upstream_module_modified "java-logging-logback"; then
279+
echo "is_upstream_module_modified should return true for java-logging-logback when java-logging is modified"
280+
exit 1
281+
fi
282+
if is_upstream_module_modified "java-logging"; then
283+
echo "is_upstream_module_modified should return false for java-logging when java-logging is modified"
284+
exit 1
285+
fi
286+
287+
# 7. Unrelated module modified:
288+
TEST_MODIFIED_FILES="java-asset/src/main/Foo.java"
289+
if is_upstream_module_modified "java-asset" || \
290+
is_upstream_module_modified "java-spanner" || \
291+
is_upstream_module_modified "java-bigquery"; then
292+
echo "is_upstream_module_modified should return false when only an unrelated module is modified"
293+
exit 1
294+
fi
295+
296+
unset TEST_MODIFIED_FILES
297+
}
298+
182299
# Test mock get_modified_files returns simulated files.
183300
function test_mock_get_modified_files {
184301
TEST_MODIFIED_FILES="dummy/file.txt"
@@ -197,4 +314,4 @@ test_parse_pom_version
197314
test_mock_get_modified_files
198315
test_should_test_all_modules
199316
test_is_module_modified
200-
317+
test_is_upstream_module_modified

0 commit comments

Comments
 (0)