Skip to content

test(spanner): unflake ITBulkConnectionTest.testBulkCreateConnectionsMultiThreaded - #14604

Merged
sakthivelmanii merged 1 commit into
mainfrom
unflake-b571072226-pro
Oct 8, 2026
Merged

sakthivelmanii merged 1 commit into
mainfrom
unflake-b571072226-pro

Conversation

@sakthivelmanii

@sakthivelmanii sakthivelmanii commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Root Cause

ITBulkConnectionTest.testBulkCreateConnectionsMultiThreaded submits 250 connection and query tasks across a 50-thread pool and calls executor.awaitTermination(10L, TimeUnit.SECONDS) without asserting its return value or waiting for the submitted task Futures. When initial Spanner client and metrics setup plus 250 connection handshakes exceed 10 seconds, awaitTermination returns false while worker threads still have open connections in flight, causing closeSpanner() in @AfterClass to fail with FAILED_PRECONDITION: There is/are 1 connection(s) still open. Close all connections before calling closeSpanner().

Fix

  • Collect all submitted task Futures and assert assertNull(future.get()) to propagate any worker exception and ensure all connections are closed before test completion.
  • Increase executor.awaitTermination timeout from 10 seconds to 60 seconds and assert assertTrue(executor.awaitTermination(60L, TimeUnit.SECONDS)).

@sakthivelmanii
sakthivelmanii requested review from a team as code owners October 8, 2026 05:06

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the multi-threaded bulk connection creation test in ITBulkConnectionTest.java to track the submitted tasks using Future objects. It now asserts that the executor terminates successfully within 60 seconds and verifies that each task completed without throwing an exception. There are no review comments, so I have no feedback to provide.

@sakthivelmanii
sakthivelmanii force-pushed the unflake-b571072226-pro branch 3 times, most recently from bd97433 to e901fcf Compare October 8, 2026 14:52
@sakthivelmanii sakthivelmanii added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 8, 2026
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 8, 2026
@sakthivelmanii
sakthivelmanii force-pushed the unflake-b571072226-pro branch from e901fcf to f330820 Compare October 8, 2026 15:49
@sakthivelmanii
sakthivelmanii enabled auto-merge (squash) October 8, 2026 16:21
@Test
public void testBulkCreateConnectionsMultiThreaded() throws InterruptedException {
public void testBulkCreateConnectionsMultiThreaded() throws Exception {
ExecutorService executor = Executors.newFixedThreadPool(50);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we add a try-finally block to ensure that the executor is shut down also when the test fails, like this (and also use JUnit style assertions):

@Test
public void testBulkCreateConnectionsMultiThreaded() throws Exception {
  ExecutorService executor = Executors.newFixedThreadPool(50);
  try {
    List<Future<?>> futures = new ArrayList<>(NUMBER_OF_TEST_CONNECTIONS);
    for (int i = 0; i < NUMBER_OF_TEST_CONNECTIONS; i++) {
      futures.add(
          executor.submit(
              () -> {
                try (ITConnection connection = createConnection()) {
                  try (ResultSet resultSet = connection.executeQuery(Statement.of("select 1"))) {
                    assertTrue(resultSet.next());
                    assertNotNull(connection.getReadTimestamp());
                  }
                }
                return null;
              }));
    }
    executor.shutdown();
    assertTrue(
        "Executor did not terminate within timeout; active threads remain",
        executor.awaitTermination(60L, TimeUnit.SECONDS));
    for (Future<?> future : futures) {
      assertNull(future.get());
    }
  } finally {
    if (!executor.isTerminated()) {
      executor.shutdownNow();
      executor.awaitTermination(5L, TimeUnit.SECONDS);
    }
  }
  closeSpanner();
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in follow-up PR #14610.

@sakthivelmanii
sakthivelmanii merged commit a37cff6 into main Oct 8, 2026
206 checks passed
@sakthivelmanii
sakthivelmanii deleted the unflake-b571072226-pro branch October 8, 2026 16:53
sakthivelmanii added a commit that referenced this pull request Oct 8, 2026
…MultiThreaded (#14610)

### Summary
Follow-up to #14604 to address review feedback on
`ITBulkConnectionTest.testBulkCreateConnectionsMultiThreaded`:
- Wrap the `ExecutorService` execution in a `try-finally` block calling
`executor.shutdownNow()` and `executor.awaitTermination(5L,
TimeUnit.SECONDS)` if the executor has not terminated when the test
exits.
- Include a descriptive failure message in `assertTrue(...,
executor.awaitTermination(60L, TimeUnit.SECONDS))` and use JUnit
`assertTrue` / `assertNotNull` assertions inside the worker task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants