From 192abf56cbc8ed7e68a3ed4b5b30c92c65392d0c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Mon, 2 Nov 2026 14:37:38 +0100 Subject: [PATCH 1/4] fix(mysql): run scheduler transactions at READ COMMITTED so the claim stops blocking publish() Under MySQL's default REPEATABLE READ the claim's FOR UPDATE SKIP LOCKED takes gap locks on (status, created_at); SKIP LOCKED only skips record locks. With a backlog smaller than the batch the gap runs to the end of the index, where every publish() inserts, so business transactions waited for the whole delivery and could fail with "Lock wait timeout exceeded". - Spring: okapiTransactionRunner pins READ COMMITTED when its TransactionTemplate leaves isolation at DEFAULT, using a copy so shared template beans are not mutated; explicit isolation levels are still honoured verbatim. - Exposed: ExposedTransactionRunner gains transactionIsolation, defaulting to READ_COMMITTED (null = database default). - MysqlClaimDoesNotBlockPublishersTest reproduces the lock at REPEATABLE READ (control) and proves both runners no longer block a concurrent publish. Closes #129 Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 17 ++ README.md | 1 + .../okapi/exposed/ExposedTransactionRunner.kt | 13 +- .../exposed/ExposedTransactionRunnerTest.kt | 33 ++++ okapi-integration-tests/build.gradle.kts | 1 + .../MysqlClaimDoesNotBlockPublishersTest.kt | 145 ++++++++++++++++++ .../springboot/OutboxAutoConfiguration.kt | 26 +++- ...xAutoConfigurationTransactionRunnerTest.kt | 62 +++++++- .../TransactionTemplateHijackProofTest.kt | 18 ++- 9 files changed, 301 insertions(+), 15 deletions(-) create mode 100644 okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt create mode 100644 okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fdd6ed0..bea98e29 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,23 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * not send the `OutboxId`; that is no longer the case, and the "Deduplicating on `x-outbox-id`" section documents the recipe and both caveats. (KOJAK-78) +### Fixed + +- **MySQL: the processor no longer blocks `publish()` for the duration of delivery.** Under MySQL's + default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on + `(status, created_at)`; with a backlog smaller than the batch, the gap ran to the end of the index, + exactly where every `publish()` inserts, so business transactions waited for the whole HTTP/Kafka + delivery and could fail with `Lock wait timeout exceeded`. The scheduler transaction now runs at + `READ COMMITTED`, which takes no gap locks: + - Spring Boot: the autoconfigured `TransactionRunner` pins `READ COMMITTED` when its + `TransactionTemplate` leaves isolation at `DEFAULT`. It uses a copy, so a user's or Boot's + template bean is not mutated and its timeout/propagation/read-only settings carry over; a + template with an explicit isolation level is still used as-is. + - `ExposedTransactionRunner` gains a `transactionIsolation` parameter defaulting to + `READ_COMMITTED` (`null` restores the database default). + + On Postgres `READ COMMITTED` is already the default, so behaviour there is unchanged. (#129) + ### Changed - **`CompositeMessageDeliverer.deliverBatch`** now dispatches transport groups concurrently — one diff --git a/README.md b/README.md index 837c41c7..39a15b8a 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,7 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co - **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id). - **Best-effort ordering.** Rows are claimed by `created_at`, oldest first. However, parallel delivery and retries mean messages may reach consumers in a different order. Strict delivery ordering is not guaranteed. - **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not. +- **Scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` and `ExposedTransactionRunner` both use it. On MySQL this matters: at the default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner` or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. - **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff. ### Deduplicating on `x-outbox-id` diff --git a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt index ca800c65..3d637bba 100644 --- a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt +++ b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt @@ -3,6 +3,7 @@ package com.softwaremill.okapi.exposed import com.softwaremill.okapi.core.TransactionRunner import org.jetbrains.exposed.v1.jdbc.Database import org.jetbrains.exposed.v1.jdbc.transactions.transaction +import java.sql.Connection /** * Exposed implementation of [TransactionRunner]. @@ -11,10 +12,18 @@ import org.jetbrains.exposed.v1.jdbc.transactions.transaction * Used by the outbox scheduler/processor when running outside of * an existing transactional context (e.g., background processing thread). * + * Runs at [transactionIsolation], READ COMMITTED by default (issue #129). Under MySQL's default + * REPEATABLE READ the claim's `FOR UPDATE SKIP LOCKED` takes gap locks at the end of the + * `(status, created_at)` index -- where every `publish()` inserts -- so publishers block until the + * claiming transaction, and with it delivery, finishes. On Postgres READ COMMITTED is already the + * default. Pass `null` to use the [database]'s default isolation instead. + * * @param database The [Database] instance where the outbox table resides. + * @param transactionIsolation a `java.sql.Connection.TRANSACTION_*` level, or `null` for the database default. */ -class ExposedTransactionRunner( +class ExposedTransactionRunner @JvmOverloads constructor( private val database: Database, + private val transactionIsolation: Int? = Connection.TRANSACTION_READ_COMMITTED, ) : TransactionRunner { - override fun runInTransaction(block: () -> T): T = transaction(database) { block() } + override fun runInTransaction(block: () -> T): T = transaction(database, transactionIsolation = transactionIsolation) { block() } } diff --git a/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt new file mode 100644 index 00000000..df90dcbc --- /dev/null +++ b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt @@ -0,0 +1,33 @@ +package com.softwaremill.okapi.exposed + +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import org.jetbrains.exposed.v1.jdbc.Database +import org.jetbrains.exposed.v1.jdbc.transactions.TransactionManager +import org.jetbrains.exposed.v1.jdbc.transactions.transactionManager +import java.sql.Connection + +class ExposedTransactionRunnerTest : FunSpec({ + + val db = Database.connect( + "jdbc:h2:mem:runner_test_${System.nanoTime()};DB_CLOSE_DELAY=-1", + driver = "org.h2.Driver", + ) + + fun isolationSeenBy(runner: ExposedTransactionRunner): Int = runner.runInTransaction { + (TransactionManager.current().connection.connection as Connection).transactionIsolation + } + + test("runs at READ COMMITTED by default (no MySQL gap locks on the claim, #129)") { + isolationSeenBy(ExposedTransactionRunner(db)) shouldBe Connection.TRANSACTION_READ_COMMITTED + } + + test("honours an explicit isolation level") { + isolationSeenBy(ExposedTransactionRunner(db, Connection.TRANSACTION_SERIALIZABLE)) shouldBe + Connection.TRANSACTION_SERIALIZABLE + } + + test("null falls back to the database's default isolation") { + isolationSeenBy(ExposedTransactionRunner(db, null)) shouldBe db.transactionManager.defaultIsolationLevel + } +}) diff --git a/okapi-integration-tests/build.gradle.kts b/okapi-integration-tests/build.gradle.kts index 6ea46c65..58d1a63a 100644 --- a/okapi-integration-tests/build.gradle.kts +++ b/okapi-integration-tests/build.gradle.kts @@ -11,6 +11,7 @@ dependencies { testImplementation(project(":okapi-http")) testImplementation(project(":okapi-spring-boot")) testImplementation(project(":okapi-micrometer")) + testImplementation(project(":okapi-exposed")) // Test framework testImplementation(libs.kotestRunnerJunit5) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt new file mode 100644 index 00000000..6b521548 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -0,0 +1,145 @@ +package com.softwaremill.okapi.test.transaction + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.exposed.ExposedConnectionProvider +import com.softwaremill.okapi.exposed.ExposedTransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import com.softwaremill.okapi.springboot.OutboxAutoConfiguration +import com.softwaremill.okapi.springboot.SpringConnectionProvider +import com.softwaremill.okapi.test.support.MysqlTestSupport +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldHaveSize +import io.kotest.matchers.longs.shouldBeLessThan +import io.kotest.matchers.string.shouldContain +import org.jetbrains.exposed.v1.jdbc.Database +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.transaction.PlatformTransactionManager +import java.sql.Connection +import java.sql.SQLException +import java.time.Clock +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import javax.sql.DataSource + +/** + * Regression test for issue #129: under MySQL's default REPEATABLE READ, the processor's claim + * (`FOR UPDATE SKIP LOCKED` on `(status, created_at)`) gap-locks the end of the index when the + * backlog is smaller than the batch. Every `publish()` inserts exactly there, so publishers blocked + * until the claiming transaction -- held open for the whole delivery -- committed. + * + * Each case opens a claim through the runner under test, keeps that transaction open (standing in + * for an in-flight delivery), and publishes from another connection with a short + * `innodb_lock_wait_timeout`. The REPEATABLE READ case is the control: it proves the setup really + * reproduces the lock, so the READ COMMITTED cases passing means the fix works, not that the test + * is blind. + */ +class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ + + val db = MysqlTestSupport() + val clock = Clock.systemUTC() + + beforeSpec { db.start() } + afterSpec { db.stop() } + beforeEach { db.truncate() } + + fun seedPending(count: Int) { + val store = MysqlOutboxStore(db.jdbc) + db.jdbc.withTransaction { + repeat(count) { + store.persist(OutboxEntry.createPending(OutboxMessage("seed", "{}"), ClaimTestDeliveryInfo, clock.instant())) + } + } + } + + /** Publishes on its own connection, failing fast with a lock-wait timeout if the insert is blocked. */ + fun publishWithShortLockWait() { + val store = MysqlOutboxStore(db.jdbc) + db.jdbc.withTransaction { + db.jdbc.withConnection { conn -> + conn.createStatement().use { it.execute("SET SESSION innodb_lock_wait_timeout = $LOCK_WAIT_TIMEOUT_SECONDS") } + } + store.persist(OutboxEntry.createPending(OutboxMessage("published", "{}"), ClaimTestDeliveryInfo, clock.instant())) + } + } + + /** Runs [publishWithShortLockWait] while [runner] holds a claim transaction open on [claimStore]. */ + fun publishWhileClaimIsOpen(runner: TransactionRunner, claimStore: OutboxStore) { + seedPending(2) // fewer than the batch size: the claim scans to the end of the PENDING range + val claimed = CountDownLatch(1) + val release = CountDownLatch(1) + val executor = Executors.newSingleThreadExecutor() + try { + val claim = executor.submit { + runner.runInTransaction { + claimStore.claimPending(10) shouldHaveSize 2 + claimed.countDown() + release.await(30, TimeUnit.SECONDS) + } + } + check(claimed.await(30, TimeUnit.SECONDS)) { "claim transaction did not start" } + try { + publishWithShortLockWait() + } finally { + release.countDown() + claim.get(30, TimeUnit.SECONDS) + } + } finally { + executor.shutdownNow() + } + } + + fun assertNotBlocked(runner: TransactionRunner, claimStore: OutboxStore) { + val start = System.nanoTime() + publishWhileClaimIsOpen(runner, claimStore) + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) shouldBeLessThan LOCK_WAIT_TIMEOUT_SECONDS * 1000L + } + + test("control: a REPEATABLE READ claim blocks a concurrent publish (reproduces #129)") { + val database = Database.connect(db.dataSource) + val runner = ExposedTransactionRunner(database, Connection.TRANSACTION_REPEATABLE_READ) + val error = shouldThrow { + publishWhileClaimIsOpen(runner, MysqlOutboxStore(ExposedConnectionProvider(database))) + } + error.message shouldContain "Lock wait timeout" + } + + test("ExposedTransactionRunner's default isolation does not block a concurrent publish") { + val database = Database.connect(db.dataSource) + assertNotBlocked(ExposedTransactionRunner(database), MysqlOutboxStore(ExposedConnectionProvider(database))) + } + + test("the Spring autoconfigured TransactionRunner does not block a concurrent publish") { + ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withPropertyValues("okapi.processor.enabled=false") + .withBean(DataSource::class.java, { db.dataSource }) + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(db.dataSource) }) + .withBean(OutboxStore::class.java, { MysqlOutboxStore(SpringConnectionProvider(db.dataSource)) }) + .withBean(MessageDeliverer::class.java, { NoopDeliverer }) + .run { ctx -> + assertNotBlocked(ctx.getBean(TransactionRunner::class.java), ctx.getBean(OutboxStore::class.java)) + } + } +}) + +private const val LOCK_WAIT_TIMEOUT_SECONDS = 3 + +private object NoopDeliverer : MessageDeliverer { + override val type: String = ClaimTestDeliveryInfo.type + override fun deliver(entry: OutboxEntry): DeliveryResult = DeliveryResult.Success +} + +private object ClaimTestDeliveryInfo : DeliveryInfo { + override val type: String = "claim-test" + override fun serialize(): String = """{"type":"claim-test"}""" +} diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index bb2a526b..aace98e8 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -30,6 +30,7 @@ import org.springframework.context.annotation.Bean import org.springframework.context.annotation.Configuration import org.springframework.core.annotation.Order import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.TransactionDefinition import org.springframework.transaction.support.ResourceTransactionManager import org.springframework.transaction.support.TransactionTemplate import java.time.Clock @@ -138,7 +139,8 @@ class OutboxAutoConfiguration( validatePtmDataSourceMatch(ptm, resolveDataSource(), okapiProperties, dataSources.size) // Use the supplied TT verbatim ONLY when it actually wraps the PTM we picked (preserves // timeout/propagation/isolation). When qualifier forced a different PTM, build a fresh TT - // around it — otherwise we'd run on a TT bound to the wrong PTM. NOTE: with + // around it — otherwise we'd run on a TT bound to the wrong PTM. Either way the result is + // pinned to READ COMMITTED when it leaves isolation at DEFAULT (withOutboxIsolation). NOTE: with // PROPAGATION_REQUIRED a tick joining an outer readOnly=true TX inherits that flag // silently — keep scheduler invocations outside @Transactional scopes to avoid FOR UPDATE // SKIP LOCKED failures. @@ -163,7 +165,7 @@ class OutboxAutoConfiguration( } TransactionTemplate(ptm).apply { isReadOnly = false } } - return SpringTransactionRunner(ttToUse) + return SpringTransactionRunner(ttToUse.withOutboxIsolation()) } @Bean @@ -289,6 +291,26 @@ class OutboxAutoConfiguration( companion object { private val logger = LoggerFactory.getLogger(OutboxAutoConfiguration::class.java) + /** + * Pins the scheduler transaction to READ COMMITTED unless [this] already sets an explicit + * isolation level (issue #129). Under MySQL's default REPEATABLE READ, the claim's + * `FOR UPDATE SKIP LOCKED` takes gap locks on `(status, created_at)` -- SKIP LOCKED only + * skips record locks -- and with a backlog smaller than the batch the gap runs to the end of + * the index, exactly where every `publish()` inserts. Publishers then block until delivery + * finishes and the claim commits. READ COMMITTED takes no gap locks; on Postgres it is + * already the default, so nothing changes there. + * + * A copy rather than a mutation: the template may be the user's or Boot's shared bean, and + * the copy keeps its timeout/propagation/read-only settings. + */ + internal fun TransactionTemplate.withOutboxIsolation(): TransactionTemplate { + if (isolationLevel != TransactionDefinition.ISOLATION_DEFAULT) return this + val ptm = checkNotNull(transactionManager) { "TransactionTemplate has no PlatformTransactionManager" } + return TransactionTemplate(ptm, this).apply { + isolationLevel = TransactionDefinition.ISOLATION_READ_COMMITTED + } + } + internal fun resolveDataSource( dataSources: Map, primaryDataSource: DataSource, diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt index 351b72b5..75464a00 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt @@ -172,7 +172,39 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ .withUserConfiguration(CustomTransactionTemplateConfiguration::class.java) .run { ctx -> val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - runner.transactionTemplate.shouldBeSameInstanceAs(CustomTransactionTemplateConfiguration.TEMPLATE) + // The template leaves isolation at DEFAULT, so the runner gets a READ COMMITTED copy + // (issue #129) -- which must still carry the user's own settings. + runner.transactionTemplate.timeout shouldBe 42 + runner.transactionTemplate.transactionManager + .shouldBeSameInstanceAs(CustomTransactionTemplateConfiguration.TEMPLATE.transactionManager) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + withClue("the user's shared template bean itself must not be mutated") { + CustomTransactionTemplateConfiguration.TEMPLATE.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + } + } + } + + test("user-provided TransactionTemplate with an explicit isolation level is used verbatim") { + val ds: DataSource = SimpleDriverDataSource() + baseRunner + .withBean(DataSource::class.java, { ds }) + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(SerializableTransactionTemplateConfiguration::class.java) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.shouldBeSameInstanceAs(SerializableTransactionTemplateConfiguration.TEMPLATE) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_SERIALIZABLE + } + } + + test("auto-built TransactionTemplate runs at READ COMMITTED (no MySQL gap locks on the claim, #129)") { + val ds: DataSource = SimpleDriverDataSource() + baseRunner + .withBean(DataSource::class.java, { ds }) + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED } } @@ -302,7 +334,7 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ } } - test("qualifier matching Boot's auto-TT's PTM reuses that TT verbatim (preserves its TX settings)") { + test("qualifier matching Boot's auto-TT's PTM builds on that TT (preserves its TX settings)") { // Edge of the qualifier-precedence rule: when the qualifier happens to name the same PTM // that Boot's auto-TT already wraps, the factory must NOT build a fresh TransactionTemplate // around it — that would silently discard timeout/isolation/propagation Boot configured. @@ -331,9 +363,13 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ ctx.startupFailure.shouldBeNull() val bootTt = ctx.getBean(org.springframework.transaction.support.TransactionTemplate::class.java) val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - // The runner's TT must be the SAME OBJECT as Boot's auto-TT — proves the factory - // didn't rebuild a fresh TT just because qualifier was set. - runner.transactionTemplate shouldBeSameInstanceAs bootTt + // The runner's TT must be built from Boot's auto-TT — proves the factory didn't + // rebuild a fresh TT just because qualifier was set. Boot's TT leaves isolation at + // DEFAULT, so it is a READ COMMITTED copy (issue #129) carrying the same settings. + runner.transactionTemplate.transactionManager shouldBeSameInstanceAs bootTt.transactionManager + runner.transactionTemplate.timeout shouldBe bootTt.timeout + runner.transactionTemplate.propagationBehavior shouldBe bootTt.propagationBehavior + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED } } @@ -655,6 +691,22 @@ private class CustomTransactionTemplateConfiguration { } } +@Configuration(proxyBeanMethods = false) +private class SerializableTransactionTemplateConfiguration { + @Bean + fun serializableTemplate(ptm: PlatformTransactionManager): org.springframework.transaction.support.TransactionTemplate { + TEMPLATE.transactionManager = ptm + return TEMPLATE + } + + companion object { + val TEMPLATE: org.springframework.transaction.support.TransactionTemplate = + org.springframework.transaction.support.TransactionTemplate().apply { + isolationLevel = TransactionDefinition.ISOLATION_SERIALIZABLE + } + } +} + @Configuration(proxyBeanMethods = false) private class TwoPtmsNoPrimaryUserConfig { @Bean diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt index 9916edf8..6b6f9de1 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt @@ -7,6 +7,7 @@ import io.kotest.assertions.withClue import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.nulls.shouldBeNull import io.kotest.matchers.nulls.shouldNotBeNull +import io.kotest.matchers.shouldBe import io.kotest.matchers.string.shouldContain import io.kotest.matchers.types.shouldBeInstanceOf import io.kotest.matchers.types.shouldBeSameInstanceAs @@ -15,6 +16,7 @@ import org.springframework.boot.test.context.runner.ApplicationContextRunner import org.springframework.jdbc.datasource.DataSourceTransactionManager import org.springframework.jdbc.datasource.SimpleDriverDataSource import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.TransactionDefinition import org.springframework.transaction.support.TransactionTemplate import javax.sql.DataSource @@ -23,9 +25,10 @@ import javax.sql.DataSource * `TransactionAutoConfiguration` has registered a `TransactionTemplate` bean for the single PTM * in the context: * - * 1. ADOPTS: the factory uses Boot's `TransactionTemplate` verbatim (reference identity), so the - * user's / Boot's TX semantics — timeout, propagation, isolation — are preserved instead of - * being silently replaced by a fresh default `TransactionTemplate`. + * 1. ADOPTS: the factory builds on Boot's `TransactionTemplate`, so the user's / Boot's TX + * semantics — PTM, timeout, propagation, read-only — are preserved instead of being silently + * replaced by a fresh default `TransactionTemplate`. Boot's template leaves isolation at + * DEFAULT, so the runner gets a copy pinned to READ COMMITTED (issue #129). * * 2. VALIDATES: even though Boot's TT is adopted, `validatePtmDataSourceMatch` still runs and * fails the context refresh when the PTM is bound to a different DataSource than okapi's @@ -50,7 +53,7 @@ class TransactionTemplateHijackProofTest : FunSpec({ "spring-boot-autoconfigure (3.x) is on testRuntimeClasspath.", ) - test("ADOPTS: okapiTransactionRunner reuses Spring Boot's auto-configured TransactionTemplate verbatim") { + test("ADOPTS: okapiTransactionRunner builds on Spring Boot's auto-configured TransactionTemplate") { val ds: DataSource = SimpleDriverDataSource() val ourPtm: PlatformTransactionManager = DataSourceTransactionManager(ds) ApplicationContextRunner() @@ -63,8 +66,11 @@ class TransactionTemplateHijackProofTest : FunSpec({ ctx.startupFailure.shouldBeNull() val bootTt = ctx.getBean(TransactionTemplate::class.java).shouldNotBeNull() val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - withClue("okapiTransactionRunner must adopt Boot's TransactionTemplate by reference, not wrap a fresh default TT") { - runner.transactionTemplate.shouldBeSameInstanceAs(bootTt) + withClue("okapiTransactionRunner must carry over Boot's TransactionTemplate settings, not wrap a fresh default TT") { + runner.transactionTemplate.transactionManager.shouldBeSameInstanceAs(bootTt.transactionManager) + runner.transactionTemplate.timeout shouldBe bootTt.timeout + runner.transactionTemplate.propagationBehavior shouldBe bootTt.propagationBehavior + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED } } } From 38e064e4612003251bc04c46eb2523e8d9fe0652 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Tue, 3 Nov 2026 12:12:34 +0100 Subject: [PATCH 2/4] fix(mysql): pin READ COMMITTED only on MySQL and only where the PTM accepts it (#129) Addresses review on #139: - Spring: pin READ COMMITTED only when the store is MysqlOutboxStore. Postgres already defaults to it, so other stores keep the template verbatim (no extra round trip, no behaviour change). Skip pinning with a startup WARN for PTMs that reject custom isolation levels (JtaTransactionManager, JpaTransactionManager without the Hibernate/EclipseLink JpaDialect); otherwise every tick would throw InvalidIsolationLevelException. - Exposed: ExposedTransactionRunner(database) uses READ COMMITTED only on MySQL/MariaDB dialects; ExposedTransactionRunner(database, level) sets it explicitly. - MysqlClaimDoesNotBlockPublishersTest drives a real OutboxProcessor tick with a blocking deliverer instead of calling claimPending directly, so it follows whatever claim query the processor runs (e.g. after #120). Adds Hibernate-JPA and DefaultJpaDialect cases. - New OutboxAutoConfigurationIsolationTest, including a JtaTransactionManager case. - CHANGELOG: behaviour-change note for Exposed overriding defaultIsolationLevel on MySQL. README: binlog_format=ROW/MIXED requirement and the JTA/JPA fallback. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 31 ++- README.md | 2 +- gradle/libs.versions.toml | 3 + .../okapi/exposed/ExposedTransactionRunner.kt | 31 ++- .../exposed/ExposedTransactionRunnerTest.kt | 18 +- .../MysqlClaimDoesNotBlockPublishersTest.kt | 120 +++++++++--- okapi-spring-boot/build.gradle.kts | 2 + .../springboot/OutboxAutoConfiguration.kt | 89 +++++++-- .../OutboxAutoConfigurationIsolationTest.kt | 177 ++++++++++++++++++ ...xAutoConfigurationTransactionRunnerTest.kt | 62 +----- .../TransactionTemplateHijackProofTest.kt | 18 +- 11 files changed, 421 insertions(+), 132 deletions(-) create mode 100644 okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index bea98e29..5f687de9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,19 +27,30 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on `(status, created_at)`; with a backlog smaller than the batch, the gap ran to the end of the index, exactly where every `publish()` inserts, so business transactions waited for the whole HTTP/Kafka - delivery and could fail with `Lock wait timeout exceeded`. The scheduler transaction now runs at - `READ COMMITTED`, which takes no gap locks: - - Spring Boot: the autoconfigured `TransactionRunner` pins `READ COMMITTED` when its - `TransactionTemplate` leaves isolation at `DEFAULT`. It uses a copy, so a user's or Boot's - template bean is not mutated and its timeout/propagation/read-only settings carry over; a - template with an explicit isolation level is still used as-is. - - `ExposedTransactionRunner` gains a `transactionIsolation` parameter defaulting to - `READ_COMMITTED` (`null` restores the database default). - - On Postgres `READ COMMITTED` is already the default, so behaviour there is unchanged. (#129) + delivery and could fail with `Lock wait timeout exceeded`. On MySQL and MariaDB the scheduler + transaction now runs at `READ COMMITTED`, which takes no gap locks: + - Spring Boot: the autoconfigured `TransactionRunner` pins `READ COMMITTED` when the store is + `MysqlOutboxStore` and its `TransactionTemplate` leaves isolation at `DEFAULT`. It uses a copy, + so a user's or Boot's template bean is not mutated and its timeout/propagation/read-only + settings carry over; a template with an explicit isolation level is still used as-is. Pinning is + skipped, with a startup WARN, for PTMs that reject custom isolation levels — a + `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink + `JpaDialect` — since every tick would otherwise throw `InvalidIsolationLevelException`. + - `ExposedTransactionRunner(database)` uses `READ COMMITTED` when the database's dialect is MySQL or + MariaDB. A new `ExposedTransactionRunner(database, transactionIsolation)` constructor sets the + level explicitly (`null` = the database default). + + Other stores and databases are unchanged: Postgres already defaults to `READ COMMITTED`. MySQL at + `READ COMMITTED` needs `binlog_format=ROW` or `MIXED` (see the README). (#129) ### Changed +- **`ExposedTransactionRunner` on MySQL/MariaDB now overrides `DatabaseConfig.defaultIsolationLevel`.** + Scheduler transactions run at `READ COMMITTED` even when the application configured another default + isolation (e.g. `SERIALIZABLE`) for its `Database`; business transactions keep that default. Pass + the level you want to `ExposedTransactionRunner(database, transactionIsolation)`, or `null` to keep + the database default, but note that `REPEATABLE READ` or stricter brings back the `publish()` + blocking described under *Fixed*. (#129) - **`CompositeMessageDeliverer.deliverBatch`** now dispatches transport groups concurrently — one virtual thread per group, with the last group run on the calling thread — instead of blocking on each group in turn. A batch spanning N transports costs ~`max(Tᵢ)` rather than `sum(Tᵢ)`; a batch diff --git a/README.md b/README.md index 39a15b8a..e0ed6541 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co - **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id). - **Best-effort ordering.** Rows are claimed by `created_at`, oldest first. However, parallel delivery and retries mean messages may reach consumers in a different order. Strict delivery ordering is not guaranteed. - **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not. -- **Scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` and `ExposedTransactionRunner` both use it. On MySQL this matters: at the default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner` or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. +- **On MySQL, scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` (with `MysqlOutboxStore`) and `ExposedTransactionRunner` (on a MySQL/MariaDB dialect) both use it. At MySQL's default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner`, an `OutboxStore` of your own, or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. With a `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink `JpaDialect`, okapi cannot set it (startup logs a WARN): configure `transaction_isolation = 'READ-COMMITTED'` on the server or in the JDBC URL instead. `READ COMMITTED` requires `binlog_format=ROW` (the default since MySQL 5.7.7) or `MIXED`; with `STATEMENT`, InnoDB rejects the scheduler's writes with `ERROR 1665`. - **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff. ### Deduplicating on `x-outbox-id` diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7de781a5..c7a1843f 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -17,6 +17,7 @@ slf4j = "2.0.20" logback = "1.6.5" assertj = "3.27.7" h2 = "2.5.252" +jakartaTransaction = "2.0.1" micrometer = "1.17.2" jmh = "1.37" jmhGradlePlugin = "0.7.3" @@ -66,6 +67,8 @@ slf4jApi = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" } slf4jSimple = { module = "org.slf4j:slf4j-simple", version.ref = "slf4j" } logbackClassic = { module = "ch.qos.logback:logback-classic", version.ref = "logback" } h2 = { module = "com.h2database:h2", version.ref = "h2" } +# Test-only: a fake UserTransaction for JtaTransactionManager in okapi-spring-boot tests. +jakartaTransactionApi = { module = "jakarta.transaction:jakarta.transaction-api", version.ref = "jakartaTransaction" } jmhCore = { module = "org.openjdk.jmh:jmh-core", version.ref = "jmh" } jmhGeneratorAnnprocess = { module = "org.openjdk.jmh:jmh-generator-annprocess", version.ref = "jmh" } diff --git a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt index 3d637bba..e05bad14 100644 --- a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt +++ b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt @@ -1,6 +1,7 @@ package com.softwaremill.okapi.exposed import com.softwaremill.okapi.core.TransactionRunner +import org.jetbrains.exposed.v1.core.vendors.MysqlDialect import org.jetbrains.exposed.v1.jdbc.Database import org.jetbrains.exposed.v1.jdbc.transactions.transaction import java.sql.Connection @@ -12,18 +13,36 @@ import java.sql.Connection * Used by the outbox scheduler/processor when running outside of * an existing transactional context (e.g., background processing thread). * - * Runs at [transactionIsolation], READ COMMITTED by default (issue #129). Under MySQL's default + * On MySQL and MariaDB it runs at READ COMMITTED by default (issue #129). Under MySQL's default * REPEATABLE READ the claim's `FOR UPDATE SKIP LOCKED` takes gap locks at the end of the * `(status, created_at)` index -- where every `publish()` inserts -- so publishers block until the - * claiming transaction, and with it delivery, finishes. On Postgres READ COMMITTED is already the - * default. Pass `null` to use the [database]'s default isolation instead. + * claiming transaction, and with it delivery, finishes. This overrides a + * `DatabaseConfig.defaultIsolationLevel` set for the rest of the application. Other databases keep + * their default (Postgres already defaults to READ COMMITTED). * * @param database The [Database] instance where the outbox table resides. - * @param transactionIsolation a `java.sql.Connection.TRANSACTION_*` level, or `null` for the database default. */ -class ExposedTransactionRunner @JvmOverloads constructor( +class ExposedTransactionRunner private constructor( private val database: Database, - private val transactionIsolation: Int? = Connection.TRANSACTION_READ_COMMITTED, + explicitIsolation: Int?, + isolationIsExplicit: Boolean, ) : TransactionRunner { + constructor(database: Database) : this(database, null, isolationIsExplicit = false) + + /** + * @param transactionIsolation a `java.sql.Connection.TRANSACTION_*` level, or `null` for the + * [database]'s default isolation -- on every database, MySQL included. + */ + constructor(database: Database, transactionIsolation: Int?) : this(database, transactionIsolation, isolationIsExplicit = true) + + // Lazy: resolving the dialect may open a connection, which the constructor shouldn't do. + private val transactionIsolation: Int? by lazy { + when { + isolationIsExplicit -> explicitIsolation + database.dialect is MysqlDialect -> Connection.TRANSACTION_READ_COMMITTED // MariaDBDialect extends it + else -> null + } + } + override fun runInTransaction(block: () -> T): T = transaction(database, transactionIsolation = transactionIsolation) { block() } } diff --git a/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt index df90dcbc..adb07c0e 100644 --- a/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt +++ b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt @@ -2,6 +2,7 @@ package com.softwaremill.okapi.exposed import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.shouldBe +import org.jetbrains.exposed.v1.core.DatabaseConfig import org.jetbrains.exposed.v1.jdbc.Database import org.jetbrains.exposed.v1.jdbc.transactions.TransactionManager import org.jetbrains.exposed.v1.jdbc.transactions.transactionManager @@ -18,8 +19,21 @@ class ExposedTransactionRunnerTest : FunSpec({ (TransactionManager.current().connection.connection as Connection).transactionIsolation } - test("runs at READ COMMITTED by default (no MySQL gap locks on the claim, #129)") { - isolationSeenBy(ExposedTransactionRunner(db)) shouldBe Connection.TRANSACTION_READ_COMMITTED + // The MySQL default (READ COMMITTED, #129) is covered against a real MySQL in + // okapi-integration-tests' MysqlClaimDoesNotBlockPublishersTest. + test("keeps the database's default isolation on a non-MySQL database") { + // H2's own default is already READ COMMITTED, so configure another one to tell them apart. + val serializableDb = Database.connect( + "jdbc:h2:mem:runner_test_serializable_${System.nanoTime()};DB_CLOSE_DELAY=-1", + driver = "org.h2.Driver", + databaseConfig = DatabaseConfig { defaultIsolationLevel = Connection.TRANSACTION_SERIALIZABLE }, + ) + isolationSeenBy(ExposedTransactionRunner(serializableDb)) shouldBe Connection.TRANSACTION_SERIALIZABLE + } + + test("honours an explicit READ COMMITTED") { + isolationSeenBy(ExposedTransactionRunner(db, Connection.TRANSACTION_READ_COMMITTED)) shouldBe + Connection.TRANSACTION_READ_COMMITTED } test("honours an explicit isolation level") { diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt index 6b521548..5b9914ea 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -4,8 +4,11 @@ import com.softwaremill.okapi.core.DeliveryInfo import com.softwaremill.okapi.core.DeliveryResult import com.softwaremill.okapi.core.MessageDeliverer import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.TransactionRunner import com.softwaremill.okapi.exposed.ExposedConnectionProvider import com.softwaremill.okapi.exposed.ExposedTransactionRunner @@ -15,14 +18,20 @@ import com.softwaremill.okapi.springboot.SpringConnectionProvider import com.softwaremill.okapi.test.support.MysqlTestSupport import io.kotest.assertions.throwables.shouldThrow import io.kotest.core.spec.style.FunSpec -import io.kotest.matchers.collections.shouldHaveSize import io.kotest.matchers.longs.shouldBeLessThan +import io.kotest.matchers.nulls.shouldBeNull +import io.kotest.matchers.shouldBe import io.kotest.matchers.string.shouldContain import org.jetbrains.exposed.v1.jdbc.Database import org.springframework.boot.autoconfigure.AutoConfigurations import org.springframework.boot.test.context.runner.ApplicationContextRunner import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.orm.jpa.DefaultJpaDialect +import org.springframework.orm.jpa.JpaTransactionManager +import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean +import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.support.TransactionSynchronizationManager import java.sql.Connection import java.sql.SQLException import java.time.Clock @@ -37,11 +46,13 @@ import javax.sql.DataSource * backlog is smaller than the batch. Every `publish()` inserts exactly there, so publishers blocked * until the claiming transaction -- held open for the whole delivery -- committed. * - * Each case opens a claim through the runner under test, keeps that transaction open (standing in - * for an in-flight delivery), and publishes from another connection with a short - * `innodb_lock_wait_timeout`. The REPEATABLE READ case is the control: it proves the setup really - * reproduces the lock, so the READ COMMITTED cases passing means the fix works, not that the test - * is blind. + * Each case runs a real processor tick through the runner under test with a deliverer that blocks + * mid-delivery, so the claim transaction stays open exactly as it does during a slow HTTP/Kafka call. + * Going through [OutboxProcessor.processNext] (not a store claim method directly) keeps the test on + * whatever claim query the processor actually runs. While delivery is blocked, it publishes from + * another connection with a short `innodb_lock_wait_timeout`. The REPEATABLE READ case is the + * control: it proves the setup really reproduces the lock, so the READ COMMITTED cases passing + * means the fix works, not that the test is blind. */ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ @@ -72,35 +83,29 @@ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ } } - /** Runs [publishWithShortLockWait] while [runner] holds a claim transaction open on [claimStore]. */ - fun publishWhileClaimIsOpen(runner: TransactionRunner, claimStore: OutboxStore) { + /** Runs [publishWithShortLockWait] while a processor tick inside [runner] is blocked mid-delivery on [store]. */ + fun publishWhileClaimIsOpen(runner: TransactionRunner, store: OutboxStore) { seedPending(2) // fewer than the batch size: the claim scans to the end of the PENDING range - val claimed = CountDownLatch(1) - val release = CountDownLatch(1) + val deliverer = BlockingDeliverer() + val processor = OutboxProcessor(store, OutboxEntryProcessor(deliverer, RetryPolicy(maxRetries = 0), clock)) val executor = Executors.newSingleThreadExecutor() try { - val claim = executor.submit { - runner.runInTransaction { - claimStore.claimPending(10) shouldHaveSize 2 - claimed.countDown() - release.await(30, TimeUnit.SECONDS) - } - } - check(claimed.await(30, TimeUnit.SECONDS)) { "claim transaction did not start" } + val tick = executor.submit { runner.runInTransaction { processor.processNext(10) } } + check(deliverer.inDelivery.await(30, TimeUnit.SECONDS)) { "processor tick did not reach delivery" } try { publishWithShortLockWait() } finally { - release.countDown() - claim.get(30, TimeUnit.SECONDS) + deliverer.release.countDown() + tick.get(30, TimeUnit.SECONDS) shouldBe 2 } } finally { executor.shutdownNow() } } - fun assertNotBlocked(runner: TransactionRunner, claimStore: OutboxStore) { + fun assertNotBlocked(runner: TransactionRunner, store: OutboxStore) { val start = System.nanoTime() - publishWhileClaimIsOpen(runner, claimStore) + publishWhileClaimIsOpen(runner, store) TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) shouldBeLessThan LOCK_WAIT_TIMEOUT_SECONDS * 1000L } @@ -118,22 +123,77 @@ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ assertNotBlocked(ExposedTransactionRunner(database), MysqlOutboxStore(ExposedConnectionProvider(database))) } + fun springContext(ptm: PlatformTransactionManager) = ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withPropertyValues("okapi.processor.enabled=false") + .withBean(DataSource::class.java, { db.dataSource }) + .withBean(PlatformTransactionManager::class.java, { ptm }) + .withBean(OutboxStore::class.java, { MysqlOutboxStore(SpringConnectionProvider(db.dataSource)) }) + .withBean(MessageDeliverer::class.java, { NoopDeliverer }) + + fun hibernateEmfBean() = LocalContainerEntityManagerFactoryBean().apply { + dataSource = db.dataSource + jpaVendorAdapter = HibernateJpaVendorAdapter() + setPackagesToScan() + afterPropertiesSet() + } + test("the Spring autoconfigured TransactionRunner does not block a concurrent publish") { - ApplicationContextRunner() - .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) - .withPropertyValues("okapi.processor.enabled=false") - .withBean(DataSource::class.java, { db.dataSource }) - .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(db.dataSource) }) - .withBean(OutboxStore::class.java, { MysqlOutboxStore(SpringConnectionProvider(db.dataSource)) }) - .withBean(MessageDeliverer::class.java, { NoopDeliverer }) - .run { ctx -> + springContext(DataSourceTransactionManager(db.dataSource)).run { ctx -> + assertNotBlocked(ctx.getBean(TransactionRunner::class.java), ctx.getBean(OutboxStore::class.java)) + } + } + + test("the Spring autoconfigured TransactionRunner over a Hibernate JpaTransactionManager does not block a concurrent publish") { + val emfBean = hibernateEmfBean() + try { + springContext(JpaTransactionManager(emfBean.`object`!!)).run { ctx -> + val isolationInTick = ctx.getBean(TransactionRunner::class.java).runInTransaction { + TransactionSynchronizationManager.getCurrentTransactionIsolationLevel() ?: -1 + } + isolationInTick shouldBe Connection.TRANSACTION_READ_COMMITTED // HibernateJpaDialect accepts it assertNotBlocked(ctx.getBean(TransactionRunner::class.java), ctx.getBean(OutboxStore::class.java)) } + } finally { + emfBean.destroy() + } + } + + test("JpaTransactionManager without a vendor JpaDialect: isolation is not pinned, so ticks still run") { + // A raw EntityManagerFactory (not Spring's EntityManagerFactoryInfo proxy) leaves the PTM on + // DefaultJpaDialect, which throws InvalidIsolationLevelException for any non-DEFAULT level. + val emfBean = hibernateEmfBean() + try { + val ptm = JpaTransactionManager(emfBean.nativeEntityManagerFactory) + ptm.jpaDialect.javaClass shouldBe DefaultJpaDialect::class.java + springContext(ptm).run { ctx -> + ctx.startupFailure.shouldBeNull() + // Non-null result: SpringTransactionRunner rejects a null block result. + val isolationPinned = ctx.getBean(TransactionRunner::class.java).runInTransaction { + TransactionSynchronizationManager.getCurrentTransactionIsolationLevel() != null + } + isolationPinned shouldBe false + } + } finally { + emfBean.destroy() + } } }) private const val LOCK_WAIT_TIMEOUT_SECONDS = 3 +/** Stands in for a slow transport: the first delivery parks until [release], holding the claim open. */ +private class BlockingDeliverer : MessageDeliverer { + val inDelivery = CountDownLatch(1) + val release = CountDownLatch(1) + override val type: String = ClaimTestDeliveryInfo.type + override fun deliver(entry: OutboxEntry): DeliveryResult { + inDelivery.countDown() + check(release.await(30, TimeUnit.SECONDS)) { "delivery was never released" } + return DeliveryResult.Success + } +} + private object NoopDeliverer : MessageDeliverer { override val type: String = ClaimTestDeliveryInfo.type override fun deliver(entry: OutboxEntry): DeliveryResult = DeliveryResult.Success diff --git a/okapi-spring-boot/build.gradle.kts b/okapi-spring-boot/build.gradle.kts index 7226f674..fa92f893 100644 --- a/okapi-spring-boot/build.gradle.kts +++ b/okapi-spring-boot/build.gradle.kts @@ -98,6 +98,8 @@ dependencies { // LiquibaseDisabledNotice breadcrumb + our PTM↔DS validation cannot-verify WARN) — slf4j-simple // does not provide an introspectable appender. testImplementation(libs.logbackClassic) + // JtaTransactionManager (spring-tx) needs the JTA API to be built around a fake UserTransaction. + testImplementation(libs.jakartaTransactionApi) testImplementation(kotlin("test")) } diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index aace98e8..98a61be2 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -118,6 +118,7 @@ class OutboxAutoConfiguration( fun okapiTransactionRunner( transactionManager: ObjectProvider, transactionTemplate: ObjectProvider, + outboxStore: ObjectProvider, beanFactory: BeanFactory, ): TransactionRunner { val anyTemplate = transactionTemplate.getIfUnique() @@ -139,8 +140,8 @@ class OutboxAutoConfiguration( validatePtmDataSourceMatch(ptm, resolveDataSource(), okapiProperties, dataSources.size) // Use the supplied TT verbatim ONLY when it actually wraps the PTM we picked (preserves // timeout/propagation/isolation). When qualifier forced a different PTM, build a fresh TT - // around it — otherwise we'd run on a TT bound to the wrong PTM. Either way the result is - // pinned to READ COMMITTED when it leaves isolation at DEFAULT (withOutboxIsolation). NOTE: with + // around it — otherwise we'd run on a TT bound to the wrong PTM. Either way, on MySQL the + // result is pinned to READ COMMITTED when it leaves isolation at DEFAULT (withOutboxIsolation). NOTE: with // PROPAGATION_REQUIRED a tick joining an outer readOnly=true TX inherits that flag // silently — keep scheduler invocations outside @Transactional scopes to avoid FOR UPDATE // SKIP LOCKED failures. @@ -165,7 +166,7 @@ class OutboxAutoConfiguration( } TransactionTemplate(ptm).apply { isReadOnly = false } } - return SpringTransactionRunner(ttToUse.withOutboxIsolation()) + return SpringTransactionRunner(ttToUse.withOutboxIsolation(outboxStore.getIfUnique())) } @Bean @@ -292,25 +293,83 @@ class OutboxAutoConfiguration( private val logger = LoggerFactory.getLogger(OutboxAutoConfiguration::class.java) /** - * Pins the scheduler transaction to READ COMMITTED unless [this] already sets an explicit - * isolation level (issue #129). Under MySQL's default REPEATABLE READ, the claim's + * Pins the scheduler transaction to READ COMMITTED when [store] is a MySQL store and [this] + * leaves isolation at DEFAULT (issue #129). Under MySQL's default REPEATABLE READ, the claim's * `FOR UPDATE SKIP LOCKED` takes gap locks on `(status, created_at)` -- SKIP LOCKED only * skips record locks -- and with a backlog smaller than the batch the gap runs to the end of * the index, exactly where every `publish()` inserts. Publishers then block until delivery - * finishes and the claim commits. READ COMMITTED takes no gap locks; on Postgres it is - * already the default, so nothing changes there. + * finishes and the claim commits. READ COMMITTED takes no gap locks. + * + * Returned unchanged when: + * - the template sets an explicit isolation level (the user's choice wins); + * - the store is not MySQL: Postgres already defaults to READ COMMITTED, so pinning would buy + * nothing and cost an isolation round trip per tick (a custom store wrapping MySQL is not + * recognised -- give it a template with an explicit isolation level); + * - the PTM rejects custom isolation levels ([customIsolationUnsupportedReason]). Pinning + * would make every tick throw `InvalidIsolationLevelException`, so a WARN tells the user + * how to get READ COMMITTED instead. * * A copy rather than a mutation: the template may be the user's or Boot's shared bean, and * the copy keeps its timeout/propagation/read-only settings. */ - internal fun TransactionTemplate.withOutboxIsolation(): TransactionTemplate { + internal fun TransactionTemplate.withOutboxIsolation(store: OutboxStore?): TransactionTemplate { if (isolationLevel != TransactionDefinition.ISOLATION_DEFAULT) return this + if (store == null || !isInstanceOfAny(store, setOf(MYSQL_OUTBOX_STORE_CLASS))) return this val ptm = checkNotNull(transactionManager) { "TransactionTemplate has no PlatformTransactionManager" } + customIsolationUnsupportedReason(ptm)?.let { reason -> + logger.warn( + "okapi could not pin the scheduler transaction to READ COMMITTED: PlatformTransactionManager " + + "'{}' {}. It runs at the database default instead -- on MySQL that is usually REPEATABLE " + + "READ, where the claim's gap locks block concurrent publish() calls until delivery " + + "finishes (issue #129). Fix: define a TransactionTemplate bean with isolationLevel = " + + "ISOLATION_READ_COMMITTED (okapi uses it as-is) once the PTM accepts custom isolation " + + "levels, or set transaction_isolation = 'READ-COMMITTED' on the MySQL server or in the " + + "JDBC URL.", + ptm.javaClass.name, + reason, + ) + return this + } return TransactionTemplate(ptm, this).apply { isolationLevel = TransactionDefinition.ISOLATION_READ_COMMITTED } } + // By name: okapi-mysql is compileOnly, so a class literal would fail to link without it. + private const val MYSQL_OUTBOX_STORE_CLASS = "com.softwaremill.okapi.mysql.MysqlOutboxStore" + + private const val JTA_PTM_CLASS = "org.springframework.transaction.jta.JtaTransactionManager" + private const val JPA_PTM_CLASS = "org.springframework.orm.jpa.JpaTransactionManager" + + // JpaDialects whose beginTransaction applies a custom isolation level; Spring's + // DefaultJpaDialect (and so any JpaTransactionManager without a vendor dialect) rejects it. + private val ISOLATION_CAPABLE_JPA_DIALECTS = setOf( + "org.springframework.orm.jpa.vendor.HibernateJpaDialect", + "org.springframework.orm.jpa.vendor.EclipseLinkJpaDialect", + ) + + /** + * Why [ptm] would throw `InvalidIsolationLevelException` on a non-DEFAULT isolation level, or + * `null` if it accepts one. Checked by class name so spring-orm stays optional. + * + * `JtaTransactionManager` rejects custom levels unless `allowCustomIsolationLevels` is set, + * which has no public getter -- so JTA is always treated as unsupported; a user who enabled + * it can supply a template with an explicit level. + */ + internal fun customIsolationUnsupportedReason(ptm: PlatformTransactionManager): String? = when { + isInstanceOfAny(ptm, setOf(JTA_PTM_CLASS)) -> + "is a JtaTransactionManager, which rejects custom isolation levels unless allowCustomIsolationLevels is set" + isInstanceOfAny(ptm, setOf(JPA_PTM_CLASS)) -> { + val dialect = ptm.javaClass.getMethod("getJpaDialect").invoke(ptm) + if (dialect != null && isInstanceOfAny(dialect, ISOLATION_CAPABLE_JPA_DIALECTS)) { + null + } else { + "is a JpaTransactionManager with JpaDialect '${dialect?.javaClass?.name}', which rejects custom isolation levels" + } + } + else -> null + } + internal fun resolveDataSource( dataSources: Map, primaryDataSource: DataSource, @@ -496,14 +555,16 @@ class OutboxAutoConfiguration( "org.springframework.orm.hibernate5.HibernateTransactionManager", ) - // Walk the superclass chain (not just `ptm.javaClass.name`) so user-defined subclasses - // of `JpaTransactionManager` / `HibernateTransactionManager` are also recognised. Reads + private fun isJpaHibernatePtm(ptm: PlatformTransactionManager): Boolean = isInstanceOfAny(ptm, JPA_HIBERNATE_PTM_CLASSES) + + // Walk the superclass chain (not just `obj.javaClass.name`) so user-defined subclasses + // (e.g. of `JpaTransactionManager` / `HibernateTransactionManager`) are also recognised. Reads // `.name` / `.superclass` on the already-loaded hierarchy only — no `Class.forName`, so - // the "no hard spring-orm compile dependency" constraint is preserved. - private fun isJpaHibernatePtm(ptm: PlatformTransactionManager): Boolean { - var c: Class<*>? = ptm.javaClass + // the "no hard spring-orm / okapi-mysql compile dependency" constraint is preserved. + private fun isInstanceOfAny(obj: Any, classNames: Set): Boolean { + var c: Class<*>? = obj.javaClass while (c != null) { - if (c.name in JPA_HIBERNATE_PTM_CLASSES) return true + if (c.name in classNames) return true c = c.superclass } return false diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt new file mode 100644 index 00000000..c03dbeab --- /dev/null +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt @@ -0,0 +1,177 @@ +package com.softwaremill.okapi.springboot + +import ch.qos.logback.classic.Level +import ch.qos.logback.classic.Logger +import ch.qos.logback.classic.spi.ILoggingEvent +import ch.qos.logback.core.read.ListAppender +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.assertions.withClue +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldBeEmpty +import io.kotest.matchers.collections.shouldHaveSize +import io.kotest.matchers.nulls.shouldBeNull +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldContain +import io.kotest.matchers.types.shouldBeInstanceOf +import io.kotest.matchers.types.shouldBeSameInstanceAs +import jakarta.transaction.Status +import jakarta.transaction.UserTransaction +import org.slf4j.LoggerFactory +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Configuration +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.jdbc.datasource.SimpleDriverDataSource +import org.springframework.transaction.InvalidIsolationLevelException +import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.TransactionDefinition +import org.springframework.transaction.jta.JtaTransactionManager +import org.springframework.transaction.support.TransactionTemplate +import javax.sql.DataSource + +/** + * When `okapiTransactionRunner` pins READ COMMITTED (issue #129): only for a MySQL store, only when + * the template leaves isolation at DEFAULT, and only when the PTM accepts a custom level. + */ +class OutboxAutoConfigurationIsolationTest : FunSpec({ + + val ds: DataSource = SimpleDriverDataSource() + + fun runner(store: () -> OutboxStore) = ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withBean(DataSource::class.java, { ds }) + .withBean(OutboxStore::class.java, store) + .withBean(MessageDeliverer::class.java, { stubDeliverer() }) + + val mysqlRunner = runner { MysqlOutboxStore(SpringConnectionProvider(ds)) } + + fun captureWarnings(block: () -> Unit): List { + val logger = LoggerFactory.getLogger(OutboxAutoConfiguration::class.java) as Logger + val appender = ListAppender().apply { start() } + logger.addAppender(appender) + try { + block() + } finally { + logger.detachAppender(appender) + } + return appender.list.filter { it.level == Level.WARN } + } + + test("MySQL store: the auto-built TransactionTemplate runs at READ COMMITTED") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + } + } + + test("MySQL store: a user TransactionTemplate at DEFAULT isolation is copied, keeping its settings, not mutated") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(DefaultIsolationTemplateConfiguration::class.java) + .run { ctx -> + val template = ctx.getBean(TransactionTemplate::class.java) + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + runner.transactionTemplate.timeout shouldBe 42 + runner.transactionTemplate.transactionManager shouldBeSameInstanceAs template.transactionManager + withClue("the user's shared template bean itself must not be mutated") { + template.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + } + } + } + + test("MySQL store: a user TransactionTemplate with an explicit isolation level is used verbatim") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(SerializableTemplateConfiguration::class.java) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate shouldBeSameInstanceAs ctx.getBean(TransactionTemplate::class.java) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_SERIALIZABLE + } + } + + test("non-MySQL store: isolation is left at DEFAULT and a user template is used verbatim") { + runner { stubStore() } + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(DefaultIsolationTemplateConfiguration::class.java) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate shouldBeSameInstanceAs ctx.getBean(TransactionTemplate::class.java) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + } + } + + test("premise: JtaTransactionManager rejects READ COMMITTED by default, so pinning it would fail every tick") { + val jta = JtaTransactionManager(FakeUserTransaction()).apply { afterPropertiesSet() } + shouldThrow { + TransactionTemplate(jta).apply { isolationLevel = TransactionDefinition.ISOLATION_READ_COMMITTED }.execute { } + } + } + + test("MySQL store + JtaTransactionManager: isolation is not pinned, ticks still run, and a WARN explains the fix") { + val warnings = captureWarnings { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { JtaTransactionManager(FakeUserTransaction()) }) + .run { ctx -> + ctx.startupFailure.shouldBeNull() + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + runner.runInTransaction { "tick" } shouldBe "tick" + } + } + val warning = warnings.filter { it.formattedMessage.contains("READ COMMITTED") }.shouldHaveSize(1).single() + warning.formattedMessage shouldContain JtaTransactionManager::class.java.name + warning.formattedMessage shouldContain "allowCustomIsolationLevels" + warning.formattedMessage shouldContain "ISOLATION_READ_COMMITTED" + } + + test("non-MySQL store + JtaTransactionManager: no isolation WARN (nothing to pin)") { + val warnings = captureWarnings { + runner { stubStore() } + .withBean(PlatformTransactionManager::class.java, { JtaTransactionManager(FakeUserTransaction()) }) + .run { ctx -> ctx.startupFailure.shouldBeNull() } + } + warnings.filter { it.formattedMessage.contains("READ COMMITTED") }.shouldBeEmpty() + } +}) + +/** Just enough of a JTA `UserTransaction` for `JtaTransactionManager` to begin and commit. */ +private class FakeUserTransaction : UserTransaction { + private var status = Status.STATUS_NO_TRANSACTION + override fun begin() { + status = Status.STATUS_ACTIVE + } + override fun commit() { + status = Status.STATUS_NO_TRANSACTION + } + override fun rollback() { + status = Status.STATUS_NO_TRANSACTION + } + override fun setRollbackOnly() { + status = Status.STATUS_MARKED_ROLLBACK + } + override fun getStatus(): Int = status + override fun setTransactionTimeout(seconds: Int) = Unit +} + +@Configuration(proxyBeanMethods = false) +private class DefaultIsolationTemplateConfiguration { + @Bean + fun defaultIsolationTemplate(ptm: PlatformTransactionManager) = TransactionTemplate(ptm).apply { timeout = 42 } +} + +@Configuration(proxyBeanMethods = false) +private class SerializableTemplateConfiguration { + @Bean + fun serializableTemplate(ptm: PlatformTransactionManager) = TransactionTemplate(ptm).apply { + isolationLevel = TransactionDefinition.ISOLATION_SERIALIZABLE + } +} diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt index 75464a00..351b72b5 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationTransactionRunnerTest.kt @@ -172,39 +172,7 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ .withUserConfiguration(CustomTransactionTemplateConfiguration::class.java) .run { ctx -> val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - // The template leaves isolation at DEFAULT, so the runner gets a READ COMMITTED copy - // (issue #129) -- which must still carry the user's own settings. - runner.transactionTemplate.timeout shouldBe 42 - runner.transactionTemplate.transactionManager - .shouldBeSameInstanceAs(CustomTransactionTemplateConfiguration.TEMPLATE.transactionManager) - runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED - withClue("the user's shared template bean itself must not be mutated") { - CustomTransactionTemplateConfiguration.TEMPLATE.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT - } - } - } - - test("user-provided TransactionTemplate with an explicit isolation level is used verbatim") { - val ds: DataSource = SimpleDriverDataSource() - baseRunner - .withBean(DataSource::class.java, { ds }) - .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) - .withUserConfiguration(SerializableTransactionTemplateConfiguration::class.java) - .run { ctx -> - val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - runner.transactionTemplate.shouldBeSameInstanceAs(SerializableTransactionTemplateConfiguration.TEMPLATE) - runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_SERIALIZABLE - } - } - - test("auto-built TransactionTemplate runs at READ COMMITTED (no MySQL gap locks on the claim, #129)") { - val ds: DataSource = SimpleDriverDataSource() - baseRunner - .withBean(DataSource::class.java, { ds }) - .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) - .run { ctx -> - val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + runner.transactionTemplate.shouldBeSameInstanceAs(CustomTransactionTemplateConfiguration.TEMPLATE) } } @@ -334,7 +302,7 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ } } - test("qualifier matching Boot's auto-TT's PTM builds on that TT (preserves its TX settings)") { + test("qualifier matching Boot's auto-TT's PTM reuses that TT verbatim (preserves its TX settings)") { // Edge of the qualifier-precedence rule: when the qualifier happens to name the same PTM // that Boot's auto-TT already wraps, the factory must NOT build a fresh TransactionTemplate // around it — that would silently discard timeout/isolation/propagation Boot configured. @@ -363,13 +331,9 @@ class OutboxAutoConfigurationTransactionRunnerTest : FunSpec({ ctx.startupFailure.shouldBeNull() val bootTt = ctx.getBean(org.springframework.transaction.support.TransactionTemplate::class.java) val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - // The runner's TT must be built from Boot's auto-TT — proves the factory didn't - // rebuild a fresh TT just because qualifier was set. Boot's TT leaves isolation at - // DEFAULT, so it is a READ COMMITTED copy (issue #129) carrying the same settings. - runner.transactionTemplate.transactionManager shouldBeSameInstanceAs bootTt.transactionManager - runner.transactionTemplate.timeout shouldBe bootTt.timeout - runner.transactionTemplate.propagationBehavior shouldBe bootTt.propagationBehavior - runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + // The runner's TT must be the SAME OBJECT as Boot's auto-TT — proves the factory + // didn't rebuild a fresh TT just because qualifier was set. + runner.transactionTemplate shouldBeSameInstanceAs bootTt } } @@ -691,22 +655,6 @@ private class CustomTransactionTemplateConfiguration { } } -@Configuration(proxyBeanMethods = false) -private class SerializableTransactionTemplateConfiguration { - @Bean - fun serializableTemplate(ptm: PlatformTransactionManager): org.springframework.transaction.support.TransactionTemplate { - TEMPLATE.transactionManager = ptm - return TEMPLATE - } - - companion object { - val TEMPLATE: org.springframework.transaction.support.TransactionTemplate = - org.springframework.transaction.support.TransactionTemplate().apply { - isolationLevel = TransactionDefinition.ISOLATION_SERIALIZABLE - } - } -} - @Configuration(proxyBeanMethods = false) private class TwoPtmsNoPrimaryUserConfig { @Bean diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt index 6b6f9de1..9916edf8 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/TransactionTemplateHijackProofTest.kt @@ -7,7 +7,6 @@ import io.kotest.assertions.withClue import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.nulls.shouldBeNull import io.kotest.matchers.nulls.shouldNotBeNull -import io.kotest.matchers.shouldBe import io.kotest.matchers.string.shouldContain import io.kotest.matchers.types.shouldBeInstanceOf import io.kotest.matchers.types.shouldBeSameInstanceAs @@ -16,7 +15,6 @@ import org.springframework.boot.test.context.runner.ApplicationContextRunner import org.springframework.jdbc.datasource.DataSourceTransactionManager import org.springframework.jdbc.datasource.SimpleDriverDataSource import org.springframework.transaction.PlatformTransactionManager -import org.springframework.transaction.TransactionDefinition import org.springframework.transaction.support.TransactionTemplate import javax.sql.DataSource @@ -25,10 +23,9 @@ import javax.sql.DataSource * `TransactionAutoConfiguration` has registered a `TransactionTemplate` bean for the single PTM * in the context: * - * 1. ADOPTS: the factory builds on Boot's `TransactionTemplate`, so the user's / Boot's TX - * semantics — PTM, timeout, propagation, read-only — are preserved instead of being silently - * replaced by a fresh default `TransactionTemplate`. Boot's template leaves isolation at - * DEFAULT, so the runner gets a copy pinned to READ COMMITTED (issue #129). + * 1. ADOPTS: the factory uses Boot's `TransactionTemplate` verbatim (reference identity), so the + * user's / Boot's TX semantics — timeout, propagation, isolation — are preserved instead of + * being silently replaced by a fresh default `TransactionTemplate`. * * 2. VALIDATES: even though Boot's TT is adopted, `validatePtmDataSourceMatch` still runs and * fails the context refresh when the PTM is bound to a different DataSource than okapi's @@ -53,7 +50,7 @@ class TransactionTemplateHijackProofTest : FunSpec({ "spring-boot-autoconfigure (3.x) is on testRuntimeClasspath.", ) - test("ADOPTS: okapiTransactionRunner builds on Spring Boot's auto-configured TransactionTemplate") { + test("ADOPTS: okapiTransactionRunner reuses Spring Boot's auto-configured TransactionTemplate verbatim") { val ds: DataSource = SimpleDriverDataSource() val ourPtm: PlatformTransactionManager = DataSourceTransactionManager(ds) ApplicationContextRunner() @@ -66,11 +63,8 @@ class TransactionTemplateHijackProofTest : FunSpec({ ctx.startupFailure.shouldBeNull() val bootTt = ctx.getBean(TransactionTemplate::class.java).shouldNotBeNull() val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() - withClue("okapiTransactionRunner must carry over Boot's TransactionTemplate settings, not wrap a fresh default TT") { - runner.transactionTemplate.transactionManager.shouldBeSameInstanceAs(bootTt.transactionManager) - runner.transactionTemplate.timeout shouldBe bootTt.timeout - runner.transactionTemplate.propagationBehavior shouldBe bootTt.propagationBehavior - runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + withClue("okapiTransactionRunner must adopt Boot's TransactionTemplate by reference, not wrap a fresh default TT") { + runner.transactionTemplate.shouldBeSameInstanceAs(bootTt) } } } From 97959cb33afe46d8d710f22bbad31e7270bfe284 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Tue, 3 Nov 2026 16:30:31 +0100 Subject: [PATCH 3/4] docs(mysql): describe gap locks on the route-aware claim index After #120 the claim runs on (status, delivery_type, created_at, id), so the gap at the end of each claimed type's range is what blocks publish() for that type. Update KDoc, the regression test's description and the CHANGELOG entry, and fold #120's Fixed entry into the single Unreleased Fixed section. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 27 +++++++++---------- .../okapi/exposed/ExposedTransactionRunner.kt | 4 +-- .../MysqlClaimDoesNotBlockPublishersTest.kt | 5 ++-- .../springboot/OutboxAutoConfiguration.kt | 6 ++--- 4 files changed, 20 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bf2a7e1c..7e6f5b11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,10 +24,10 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * ### Fixed - **MySQL: the processor no longer blocks `publish()` for the duration of delivery.** Under MySQL's - default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on - `(status, created_at)`; with a backlog smaller than the batch, the gap ran to the end of the index, - exactly where every `publish()` inserts, so business transactions waited for the whole HTTP/Kafka - delivery and could fail with `Lock wait timeout exceeded`. On MySQL and MariaDB the scheduler + default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on the claim index; + with a backlog smaller than the batch, the gap ran to the end of the claimed type's range, exactly + where every `publish()` of that type inserts, so business transactions waited for the whole + HTTP/Kafka delivery and could fail with `Lock wait timeout exceeded`. On MySQL and MariaDB the scheduler transaction now runs at `READ COMMITTED`, which takes no gap locks: - Spring Boot: the autoconfigured `TransactionRunner` pins `READ COMMITTED` when the store is `MysqlOutboxStore` and its `TransactionTemplate` leaves isolation at `DEFAULT`. It uses a copy, @@ -42,6 +42,14 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * Other stores and databases are unchanged: Postgres already defaults to `READ COMMITTED`. MySQL at `READ COMMITTED` needs `binlog_format=ROW` or `MIXED` (see the README). (#129) +- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed + all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent + failure. Each worker now claims only its registered delivery types, in at most two queries per + batch. Workers on an older version still claim every type, so publish a newly introduced type only + after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried + automatically. Cross-type claim order is no longer globally oldest first: the processor rotates + the starting type, and any leftover batch capacity goes to the other types alphabetically. + ([#116](https://github.com/softwaremill/okapi/issues/116)) ### Changed @@ -112,17 +120,6 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * without it, and the PostgreSQL script must run outside a transaction (`CREATE INDEX CONCURRENTLY`). ([#120](https://github.com/softwaremill/okapi/pull/120)) -### Fixed - -- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed - all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent - failure. Each worker now claims only its registered delivery types, in at most two queries per - batch. Workers on an older version still claim every type, so publish a newly introduced type only - after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried - automatically. Cross-type claim order is no longer globally oldest first: the processor rotates - the starting type, and any leftover batch capacity goes to the other types alphabetically. - ([#116](https://github.com/softwaremill/okapi/issues/116)) - ## [1.0.0] — 2026-07-28 First stable release. The public API now follows semantic versioning — breaking changes will diff --git a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt index e05bad14..5740daf4 100644 --- a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt +++ b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt @@ -14,8 +14,8 @@ import java.sql.Connection * an existing transactional context (e.g., background processing thread). * * On MySQL and MariaDB it runs at READ COMMITTED by default (issue #129). Under MySQL's default - * REPEATABLE READ the claim's `FOR UPDATE SKIP LOCKED` takes gap locks at the end of the - * `(status, created_at)` index -- where every `publish()` inserts -- so publishers block until the + * REPEATABLE READ the claim's `FOR UPDATE SKIP LOCKED` takes gap locks at the end of each claimed + * delivery type's range in the claim index -- where every `publish()` of that type inserts -- so publishers block until the * claiming transaction, and with it delivery, finishes. This overrides a * `DatabaseConfig.defaultIsolationLevel` set for the rest of the application. Other databases keep * their default (Postgres already defaults to READ COMMITTED). diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt index 5b9914ea..dde858ef 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -42,8 +42,9 @@ import javax.sql.DataSource /** * Regression test for issue #129: under MySQL's default REPEATABLE READ, the processor's claim - * (`FOR UPDATE SKIP LOCKED` on `(status, created_at)`) gap-locks the end of the index when the - * backlog is smaller than the batch. Every `publish()` inserts exactly there, so publishers blocked + * (`FOR UPDATE SKIP LOCKED` on `(status, delivery_type, created_at, id)`) gap-locks the end of the + * claimed type's range when its backlog is smaller than the batch. Every `publish()` of that type + * inserts exactly there, so publishers blocked * until the claiming transaction -- held open for the whole delivery -- committed. * * Each case runs a real processor tick through the runner under test with a deliverer that blocks diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index c5f4a7e7..32cf256d 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -302,9 +302,9 @@ class OutboxAutoConfiguration( /** * Pins the scheduler transaction to READ COMMITTED when [store] is a MySQL store and [this] * leaves isolation at DEFAULT (issue #129). Under MySQL's default REPEATABLE READ, the claim's - * `FOR UPDATE SKIP LOCKED` takes gap locks on `(status, created_at)` -- SKIP LOCKED only - * skips record locks -- and with a backlog smaller than the batch the gap runs to the end of - * the index, exactly where every `publish()` inserts. Publishers then block until delivery + * `FOR UPDATE SKIP LOCKED` takes gap locks on `(status, delivery_type, created_at, id)` -- SKIP + * LOCKED only skips record locks -- and with a backlog smaller than the batch the gap runs to the + * end of the claimed type's range, exactly where every `publish()` of that type inserts. Publishers then block until delivery * finishes and the claim commits. READ COMMITTED takes no gap locks. * * Returned unchanged when: From 5cb694c7ff9384d6ae9de2e1835935a96d76e9e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Tue, 3 Nov 2026 17:26:32 +0100 Subject: [PATCH 4/4] test(mysql): assert the #129 regression test runs the route-aware claim The test already drove OutboxProcessor.processNext(), so after #120 it exercised the route-aware claim, but nothing proved it. A recording store now asserts each tick calls claimPending(deliveryTypes, limit) and never the legacy claimPending(limit), so the test fails if the processor's claim path changes. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../MysqlClaimDoesNotBlockPublishersTest.kt | 35 ++++++++++++++++--- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt index dde858ef..cea34e62 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -9,6 +9,7 @@ import com.softwaremill.okapi.core.OutboxMessage import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxStore import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore import com.softwaremill.okapi.core.TransactionRunner import com.softwaremill.okapi.exposed.ExposedConnectionProvider import com.softwaremill.okapi.exposed.ExposedTransactionRunner @@ -18,6 +19,7 @@ import com.softwaremill.okapi.springboot.SpringConnectionProvider import com.softwaremill.okapi.test.support.MysqlTestSupport import io.kotest.assertions.throwables.shouldThrow import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly import io.kotest.matchers.longs.shouldBeLessThan import io.kotest.matchers.nulls.shouldBeNull import io.kotest.matchers.shouldBe @@ -44,13 +46,15 @@ import javax.sql.DataSource * Regression test for issue #129: under MySQL's default REPEATABLE READ, the processor's claim * (`FOR UPDATE SKIP LOCKED` on `(status, delivery_type, created_at, id)`) gap-locks the end of the * claimed type's range when its backlog is smaller than the batch. Every `publish()` of that type - * inserts exactly there, so publishers blocked - * until the claiming transaction -- held open for the whole delivery -- committed. + * inserts exactly there, so publishers blocked until the claiming transaction -- held open for the + * whole delivery -- committed. * * Each case runs a real processor tick through the runner under test with a deliverer that blocks * mid-delivery, so the claim transaction stays open exactly as it does during a slow HTTP/Kafka call. * Going through [OutboxProcessor.processNext] (not a store claim method directly) keeps the test on - * whatever claim query the processor actually runs. While delivery is blocked, it publishes from + * whatever claim query the processor actually runs; [RecordingStore] asserts that this is the + * route-aware `claimPending(deliveryTypes, limit)` from #120, never the legacy `claimPending(limit)`. + * While delivery is blocked, it publishes from * another connection with a short `innodb_lock_wait_timeout`. The REPEATABLE READ case is the * control: it proves the setup really reproduces the lock, so the READ COMMITTED cases passing * means the fix works, not that the test is blind. @@ -88,7 +92,8 @@ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ fun publishWhileClaimIsOpen(runner: TransactionRunner, store: OutboxStore) { seedPending(2) // fewer than the batch size: the claim scans to the end of the PENDING range val deliverer = BlockingDeliverer() - val processor = OutboxProcessor(store, OutboxEntryProcessor(deliverer, RetryPolicy(maxRetries = 0), clock)) + val recordingStore = RecordingStore(store as RouteAwareOutboxStore) + val processor = OutboxProcessor(recordingStore, OutboxEntryProcessor(deliverer, RetryPolicy(maxRetries = 0), clock)) val executor = Executors.newSingleThreadExecutor() try { val tick = executor.submit { runner.runInTransaction { processor.processNext(10) } } @@ -98,6 +103,7 @@ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ } finally { deliverer.release.countDown() tick.get(30, TimeUnit.SECONDS) shouldBe 2 + recordingStore.claims shouldContainExactly listOf("claimPending([${ClaimTestDeliveryInfo.type}], 10)") } } finally { executor.shutdownNow() @@ -184,6 +190,27 @@ class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ private const val LOCK_WAIT_TIMEOUT_SECONDS = 3 /** Stands in for a slow transport: the first delivery parks until [release], holding the claim open. */ +/** + * Delegates to the real store and records which claim method the processor calls, so the test fails + * if the processor stops using the route-aware claim this regression test is meant to cover. + */ +private class RecordingStore(private val delegate: RouteAwareOutboxStore) : RouteAwareOutboxStore by delegate { + val claims = mutableListOf() + + override fun claimPending(limit: Int): List { + claims += "claimPending($limit)" + return delegate.claimPending(limit) + } + + override fun claimPending(deliveryTypes: Set, limit: Int): List { + claims += "claimPending($deliveryTypes, $limit)" + return delegate.claimPending(deliveryTypes, limit) + } + + // Overridden so the single-type call is recorded too instead of going straight to the delegate. + override fun claimPending(deliveryType: String, limit: Int): List = claimPending(setOf(deliveryType), limit) +} + private class BlockingDeliverer : MessageDeliverer { val inDelivery = CountDownLatch(1) val release = CountDownLatch(1)