diff --git a/CHANGELOG.md b/CHANGELOG.md index 58d274a5..7e6f5b11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,8 +21,44 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * not send the `OutboxId`; that is no longer the case, and the "Deduplicating on `x-outbox-id`" section documents the recipe and both caveats. (KOJAK-78) +### Fixed + +- **MySQL: the processor no longer blocks `publish()` for the duration of delivery.** Under MySQL's + default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on the claim index; + with a backlog smaller than the batch, the gap ran to the end of the claimed type's range, exactly + where every `publish()` of that type inserts, so business transactions waited for the whole + HTTP/Kafka delivery and could fail with `Lock wait timeout exceeded`. On MySQL and MariaDB the scheduler + transaction now runs at `READ COMMITTED`, which takes no gap locks: + - Spring Boot: the autoconfigured `TransactionRunner` pins `READ COMMITTED` when the store is + `MysqlOutboxStore` and its `TransactionTemplate` leaves isolation at `DEFAULT`. It uses a copy, + so a user's or Boot's template bean is not mutated and its timeout/propagation/read-only + settings carry over; a template with an explicit isolation level is still used as-is. Pinning is + skipped, with a startup WARN, for PTMs that reject custom isolation levels — a + `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink + `JpaDialect` — since every tick would otherwise throw `InvalidIsolationLevelException`. + - `ExposedTransactionRunner(database)` uses `READ COMMITTED` when the database's dialect is MySQL or + MariaDB. A new `ExposedTransactionRunner(database, transactionIsolation)` constructor sets the + level explicitly (`null` = the database default). + + Other stores and databases are unchanged: Postgres already defaults to `READ COMMITTED`. MySQL at + `READ COMMITTED` needs `binlog_format=ROW` or `MIXED` (see the README). (#129) +- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed + all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent + failure. Each worker now claims only its registered delivery types, in at most two queries per + batch. Workers on an older version still claim every type, so publish a newly introduced type only + after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried + automatically. Cross-type claim order is no longer globally oldest first: the processor rotates + the starting type, and any leftover batch capacity goes to the other types alphabetically. + ([#116](https://github.com/softwaremill/okapi/issues/116)) + ### Changed +- **`ExposedTransactionRunner` on MySQL/MariaDB now overrides `DatabaseConfig.defaultIsolationLevel`.** + Scheduler transactions run at `READ COMMITTED` even when the application configured another default + isolation (e.g. `SERIALIZABLE`) for its `Database`; business transactions keep that default. Pass + the level you want to `ExposedTransactionRunner(database, transactionIsolation)`, or `null` to keep + the database default, but note that `REPEATABLE READ` or stricter brings back the `publish()` + blocking described under *Fixed*. (#129) - **`CompositeMessageDeliverer.deliverBatch`** now dispatches transport groups concurrently — one virtual thread per group, with the last group run on the calling thread — instead of blocking on each group in turn. A batch spanning N transports costs ~`max(Tᵢ)` rather than `sum(Tᵢ)`; a batch @@ -84,17 +120,6 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * without it, and the PostgreSQL script must run outside a transaction (`CREATE INDEX CONCURRENTLY`). ([#120](https://github.com/softwaremill/okapi/pull/120)) -### Fixed - -- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed - all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent - failure. Each worker now claims only its registered delivery types, in at most two queries per - batch. Workers on an older version still claim every type, so publish a newly introduced type only - after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried - automatically. Cross-type claim order is no longer globally oldest first: the processor rotates - the starting type, and any leftover batch capacity goes to the other types alphabetically. - ([#116](https://github.com/softwaremill/okapi/issues/116)) - ## [1.0.0] — 2026-07-28 First stable release. The public API now follows semantic versioning — breaking changes will diff --git a/README.md b/README.md index ea1d04b3..7be4fa3f 100644 --- a/README.md +++ b/README.md @@ -96,6 +96,7 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co - **Different workers can handle different delivery types.** Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom `OutboxStore` used for processing must implement `RouteAwareOutboxStore`; otherwise processing fails before an unsafe claim. The older `claimPending(limit)` method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself. - **Rolling upgrades need sequencing.** Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked `FAILED` by older workers. - **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not. +- **On MySQL, scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` (with `MysqlOutboxStore`) and `ExposedTransactionRunner` (on a MySQL/MariaDB dialect) both use it. At MySQL's default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner`, an `OutboxStore` of your own, or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. With a `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink `JpaDialect`, okapi cannot set it (startup logs a WARN): configure `transaction_isolation = 'READ-COMMITTED'` on the server or in the JDBC URL instead. `READ COMMITTED` requires `binlog_format=ROW` (the default since MySQL 5.7.7) or `MIXED`; with `STATEMENT`, InnoDB rejects the scheduler's writes with `ERROR 1665`. - **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff. ### Deduplicating on `x-outbox-id` diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7de781a5..c7a1843f 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -17,6 +17,7 @@ slf4j = "2.0.20" logback = "1.6.5" assertj = "3.27.7" h2 = "2.5.252" +jakartaTransaction = "2.0.1" micrometer = "1.17.2" jmh = "1.37" jmhGradlePlugin = "0.7.3" @@ -66,6 +67,8 @@ slf4jApi = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" } slf4jSimple = { module = "org.slf4j:slf4j-simple", version.ref = "slf4j" } logbackClassic = { module = "ch.qos.logback:logback-classic", version.ref = "logback" } h2 = { module = "com.h2database:h2", version.ref = "h2" } +# Test-only: a fake UserTransaction for JtaTransactionManager in okapi-spring-boot tests. +jakartaTransactionApi = { module = "jakarta.transaction:jakarta.transaction-api", version.ref = "jakartaTransaction" } jmhCore = { module = "org.openjdk.jmh:jmh-core", version.ref = "jmh" } jmhGeneratorAnnprocess = { module = "org.openjdk.jmh:jmh-generator-annprocess", version.ref = "jmh" } diff --git a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt index ca800c65..5740daf4 100644 --- a/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt +++ b/okapi-exposed/src/main/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunner.kt @@ -1,8 +1,10 @@ package com.softwaremill.okapi.exposed import com.softwaremill.okapi.core.TransactionRunner +import org.jetbrains.exposed.v1.core.vendors.MysqlDialect import org.jetbrains.exposed.v1.jdbc.Database import org.jetbrains.exposed.v1.jdbc.transactions.transaction +import java.sql.Connection /** * Exposed implementation of [TransactionRunner]. @@ -11,10 +13,36 @@ import org.jetbrains.exposed.v1.jdbc.transactions.transaction * Used by the outbox scheduler/processor when running outside of * an existing transactional context (e.g., background processing thread). * + * On MySQL and MariaDB it runs at READ COMMITTED by default (issue #129). Under MySQL's default + * REPEATABLE READ the claim's `FOR UPDATE SKIP LOCKED` takes gap locks at the end of each claimed + * delivery type's range in the claim index -- where every `publish()` of that type inserts -- so publishers block until the + * claiming transaction, and with it delivery, finishes. This overrides a + * `DatabaseConfig.defaultIsolationLevel` set for the rest of the application. Other databases keep + * their default (Postgres already defaults to READ COMMITTED). + * * @param database The [Database] instance where the outbox table resides. */ -class ExposedTransactionRunner( +class ExposedTransactionRunner private constructor( private val database: Database, + explicitIsolation: Int?, + isolationIsExplicit: Boolean, ) : TransactionRunner { - override fun runInTransaction(block: () -> T): T = transaction(database) { block() } + constructor(database: Database) : this(database, null, isolationIsExplicit = false) + + /** + * @param transactionIsolation a `java.sql.Connection.TRANSACTION_*` level, or `null` for the + * [database]'s default isolation -- on every database, MySQL included. + */ + constructor(database: Database, transactionIsolation: Int?) : this(database, transactionIsolation, isolationIsExplicit = true) + + // Lazy: resolving the dialect may open a connection, which the constructor shouldn't do. + private val transactionIsolation: Int? by lazy { + when { + isolationIsExplicit -> explicitIsolation + database.dialect is MysqlDialect -> Connection.TRANSACTION_READ_COMMITTED // MariaDBDialect extends it + else -> null + } + } + + override fun runInTransaction(block: () -> T): T = transaction(database, transactionIsolation = transactionIsolation) { block() } } diff --git a/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt new file mode 100644 index 00000000..adb07c0e --- /dev/null +++ b/okapi-exposed/src/test/kotlin/com/softwaremill/okapi/exposed/ExposedTransactionRunnerTest.kt @@ -0,0 +1,47 @@ +package com.softwaremill.okapi.exposed + +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import org.jetbrains.exposed.v1.core.DatabaseConfig +import org.jetbrains.exposed.v1.jdbc.Database +import org.jetbrains.exposed.v1.jdbc.transactions.TransactionManager +import org.jetbrains.exposed.v1.jdbc.transactions.transactionManager +import java.sql.Connection + +class ExposedTransactionRunnerTest : FunSpec({ + + val db = Database.connect( + "jdbc:h2:mem:runner_test_${System.nanoTime()};DB_CLOSE_DELAY=-1", + driver = "org.h2.Driver", + ) + + fun isolationSeenBy(runner: ExposedTransactionRunner): Int = runner.runInTransaction { + (TransactionManager.current().connection.connection as Connection).transactionIsolation + } + + // The MySQL default (READ COMMITTED, #129) is covered against a real MySQL in + // okapi-integration-tests' MysqlClaimDoesNotBlockPublishersTest. + test("keeps the database's default isolation on a non-MySQL database") { + // H2's own default is already READ COMMITTED, so configure another one to tell them apart. + val serializableDb = Database.connect( + "jdbc:h2:mem:runner_test_serializable_${System.nanoTime()};DB_CLOSE_DELAY=-1", + driver = "org.h2.Driver", + databaseConfig = DatabaseConfig { defaultIsolationLevel = Connection.TRANSACTION_SERIALIZABLE }, + ) + isolationSeenBy(ExposedTransactionRunner(serializableDb)) shouldBe Connection.TRANSACTION_SERIALIZABLE + } + + test("honours an explicit READ COMMITTED") { + isolationSeenBy(ExposedTransactionRunner(db, Connection.TRANSACTION_READ_COMMITTED)) shouldBe + Connection.TRANSACTION_READ_COMMITTED + } + + test("honours an explicit isolation level") { + isolationSeenBy(ExposedTransactionRunner(db, Connection.TRANSACTION_SERIALIZABLE)) shouldBe + Connection.TRANSACTION_SERIALIZABLE + } + + test("null falls back to the database's default isolation") { + isolationSeenBy(ExposedTransactionRunner(db, null)) shouldBe db.transactionManager.defaultIsolationLevel + } +}) diff --git a/okapi-integration-tests/build.gradle.kts b/okapi-integration-tests/build.gradle.kts index 6ea46c65..58d1a63a 100644 --- a/okapi-integration-tests/build.gradle.kts +++ b/okapi-integration-tests/build.gradle.kts @@ -11,6 +11,7 @@ dependencies { testImplementation(project(":okapi-http")) testImplementation(project(":okapi-spring-boot")) testImplementation(project(":okapi-micrometer")) + testImplementation(project(":okapi-exposed")) // Test framework testImplementation(libs.kotestRunnerJunit5) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt new file mode 100644 index 00000000..cea34e62 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -0,0 +1,233 @@ +package com.softwaremill.okapi.test.transaction + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessor +import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.exposed.ExposedConnectionProvider +import com.softwaremill.okapi.exposed.ExposedTransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import com.softwaremill.okapi.springboot.OutboxAutoConfiguration +import com.softwaremill.okapi.springboot.SpringConnectionProvider +import com.softwaremill.okapi.test.support.MysqlTestSupport +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.longs.shouldBeLessThan +import io.kotest.matchers.nulls.shouldBeNull +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldContain +import org.jetbrains.exposed.v1.jdbc.Database +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.orm.jpa.DefaultJpaDialect +import org.springframework.orm.jpa.JpaTransactionManager +import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean +import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter +import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.support.TransactionSynchronizationManager +import java.sql.Connection +import java.sql.SQLException +import java.time.Clock +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Executors +import java.util.concurrent.TimeUnit +import javax.sql.DataSource + +/** + * Regression test for issue #129: under MySQL's default REPEATABLE READ, the processor's claim + * (`FOR UPDATE SKIP LOCKED` on `(status, delivery_type, created_at, id)`) gap-locks the end of the + * claimed type's range when its backlog is smaller than the batch. Every `publish()` of that type + * inserts exactly there, so publishers blocked until the claiming transaction -- held open for the + * whole delivery -- committed. + * + * Each case runs a real processor tick through the runner under test with a deliverer that blocks + * mid-delivery, so the claim transaction stays open exactly as it does during a slow HTTP/Kafka call. + * Going through [OutboxProcessor.processNext] (not a store claim method directly) keeps the test on + * whatever claim query the processor actually runs; [RecordingStore] asserts that this is the + * route-aware `claimPending(deliveryTypes, limit)` from #120, never the legacy `claimPending(limit)`. + * While delivery is blocked, it publishes from + * another connection with a short `innodb_lock_wait_timeout`. The REPEATABLE READ case is the + * control: it proves the setup really reproduces the lock, so the READ COMMITTED cases passing + * means the fix works, not that the test is blind. + */ +class MysqlClaimDoesNotBlockPublishersTest : FunSpec({ + + val db = MysqlTestSupport() + val clock = Clock.systemUTC() + + beforeSpec { db.start() } + afterSpec { db.stop() } + beforeEach { db.truncate() } + + fun seedPending(count: Int) { + val store = MysqlOutboxStore(db.jdbc) + db.jdbc.withTransaction { + repeat(count) { + store.persist(OutboxEntry.createPending(OutboxMessage("seed", "{}"), ClaimTestDeliveryInfo, clock.instant())) + } + } + } + + /** Publishes on its own connection, failing fast with a lock-wait timeout if the insert is blocked. */ + fun publishWithShortLockWait() { + val store = MysqlOutboxStore(db.jdbc) + db.jdbc.withTransaction { + db.jdbc.withConnection { conn -> + conn.createStatement().use { it.execute("SET SESSION innodb_lock_wait_timeout = $LOCK_WAIT_TIMEOUT_SECONDS") } + } + store.persist(OutboxEntry.createPending(OutboxMessage("published", "{}"), ClaimTestDeliveryInfo, clock.instant())) + } + } + + /** Runs [publishWithShortLockWait] while a processor tick inside [runner] is blocked mid-delivery on [store]. */ + fun publishWhileClaimIsOpen(runner: TransactionRunner, store: OutboxStore) { + seedPending(2) // fewer than the batch size: the claim scans to the end of the PENDING range + val deliverer = BlockingDeliverer() + val recordingStore = RecordingStore(store as RouteAwareOutboxStore) + val processor = OutboxProcessor(recordingStore, OutboxEntryProcessor(deliverer, RetryPolicy(maxRetries = 0), clock)) + val executor = Executors.newSingleThreadExecutor() + try { + val tick = executor.submit { runner.runInTransaction { processor.processNext(10) } } + check(deliverer.inDelivery.await(30, TimeUnit.SECONDS)) { "processor tick did not reach delivery" } + try { + publishWithShortLockWait() + } finally { + deliverer.release.countDown() + tick.get(30, TimeUnit.SECONDS) shouldBe 2 + recordingStore.claims shouldContainExactly listOf("claimPending([${ClaimTestDeliveryInfo.type}], 10)") + } + } finally { + executor.shutdownNow() + } + } + + fun assertNotBlocked(runner: TransactionRunner, store: OutboxStore) { + val start = System.nanoTime() + publishWhileClaimIsOpen(runner, store) + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) shouldBeLessThan LOCK_WAIT_TIMEOUT_SECONDS * 1000L + } + + test("control: a REPEATABLE READ claim blocks a concurrent publish (reproduces #129)") { + val database = Database.connect(db.dataSource) + val runner = ExposedTransactionRunner(database, Connection.TRANSACTION_REPEATABLE_READ) + val error = shouldThrow { + publishWhileClaimIsOpen(runner, MysqlOutboxStore(ExposedConnectionProvider(database))) + } + error.message shouldContain "Lock wait timeout" + } + + test("ExposedTransactionRunner's default isolation does not block a concurrent publish") { + val database = Database.connect(db.dataSource) + assertNotBlocked(ExposedTransactionRunner(database), MysqlOutboxStore(ExposedConnectionProvider(database))) + } + + fun springContext(ptm: PlatformTransactionManager) = ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withPropertyValues("okapi.processor.enabled=false") + .withBean(DataSource::class.java, { db.dataSource }) + .withBean(PlatformTransactionManager::class.java, { ptm }) + .withBean(OutboxStore::class.java, { MysqlOutboxStore(SpringConnectionProvider(db.dataSource)) }) + .withBean(MessageDeliverer::class.java, { NoopDeliverer }) + + fun hibernateEmfBean() = LocalContainerEntityManagerFactoryBean().apply { + dataSource = db.dataSource + jpaVendorAdapter = HibernateJpaVendorAdapter() + setPackagesToScan() + afterPropertiesSet() + } + + test("the Spring autoconfigured TransactionRunner does not block a concurrent publish") { + springContext(DataSourceTransactionManager(db.dataSource)).run { ctx -> + assertNotBlocked(ctx.getBean(TransactionRunner::class.java), ctx.getBean(OutboxStore::class.java)) + } + } + + test("the Spring autoconfigured TransactionRunner over a Hibernate JpaTransactionManager does not block a concurrent publish") { + val emfBean = hibernateEmfBean() + try { + springContext(JpaTransactionManager(emfBean.`object`!!)).run { ctx -> + val isolationInTick = ctx.getBean(TransactionRunner::class.java).runInTransaction { + TransactionSynchronizationManager.getCurrentTransactionIsolationLevel() ?: -1 + } + isolationInTick shouldBe Connection.TRANSACTION_READ_COMMITTED // HibernateJpaDialect accepts it + assertNotBlocked(ctx.getBean(TransactionRunner::class.java), ctx.getBean(OutboxStore::class.java)) + } + } finally { + emfBean.destroy() + } + } + + test("JpaTransactionManager without a vendor JpaDialect: isolation is not pinned, so ticks still run") { + // A raw EntityManagerFactory (not Spring's EntityManagerFactoryInfo proxy) leaves the PTM on + // DefaultJpaDialect, which throws InvalidIsolationLevelException for any non-DEFAULT level. + val emfBean = hibernateEmfBean() + try { + val ptm = JpaTransactionManager(emfBean.nativeEntityManagerFactory) + ptm.jpaDialect.javaClass shouldBe DefaultJpaDialect::class.java + springContext(ptm).run { ctx -> + ctx.startupFailure.shouldBeNull() + // Non-null result: SpringTransactionRunner rejects a null block result. + val isolationPinned = ctx.getBean(TransactionRunner::class.java).runInTransaction { + TransactionSynchronizationManager.getCurrentTransactionIsolationLevel() != null + } + isolationPinned shouldBe false + } + } finally { + emfBean.destroy() + } + } +}) + +private const val LOCK_WAIT_TIMEOUT_SECONDS = 3 + +/** Stands in for a slow transport: the first delivery parks until [release], holding the claim open. */ +/** + * Delegates to the real store and records which claim method the processor calls, so the test fails + * if the processor stops using the route-aware claim this regression test is meant to cover. + */ +private class RecordingStore(private val delegate: RouteAwareOutboxStore) : RouteAwareOutboxStore by delegate { + val claims = mutableListOf() + + override fun claimPending(limit: Int): List { + claims += "claimPending($limit)" + return delegate.claimPending(limit) + } + + override fun claimPending(deliveryTypes: Set, limit: Int): List { + claims += "claimPending($deliveryTypes, $limit)" + return delegate.claimPending(deliveryTypes, limit) + } + + // Overridden so the single-type call is recorded too instead of going straight to the delegate. + override fun claimPending(deliveryType: String, limit: Int): List = claimPending(setOf(deliveryType), limit) +} + +private class BlockingDeliverer : MessageDeliverer { + val inDelivery = CountDownLatch(1) + val release = CountDownLatch(1) + override val type: String = ClaimTestDeliveryInfo.type + override fun deliver(entry: OutboxEntry): DeliveryResult { + inDelivery.countDown() + check(release.await(30, TimeUnit.SECONDS)) { "delivery was never released" } + return DeliveryResult.Success + } +} + +private object NoopDeliverer : MessageDeliverer { + override val type: String = ClaimTestDeliveryInfo.type + override fun deliver(entry: OutboxEntry): DeliveryResult = DeliveryResult.Success +} + +private object ClaimTestDeliveryInfo : DeliveryInfo { + override val type: String = "claim-test" + override fun serialize(): String = """{"type":"claim-test"}""" +} diff --git a/okapi-spring-boot/build.gradle.kts b/okapi-spring-boot/build.gradle.kts index 7226f674..fa92f893 100644 --- a/okapi-spring-boot/build.gradle.kts +++ b/okapi-spring-boot/build.gradle.kts @@ -98,6 +98,8 @@ dependencies { // LiquibaseDisabledNotice breadcrumb + our PTM↔DS validation cannot-verify WARN) — slf4j-simple // does not provide an introspectable appender. testImplementation(libs.logbackClassic) + // JtaTransactionManager (spring-tx) needs the JTA API to be built around a fake UserTransaction. + testImplementation(libs.jakartaTransactionApi) testImplementation(kotlin("test")) } diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index 1254bfd2..32cf256d 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -31,6 +31,7 @@ import org.springframework.context.annotation.Bean import org.springframework.context.annotation.Configuration import org.springframework.core.annotation.Order import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.TransactionDefinition import org.springframework.transaction.support.ResourceTransactionManager import org.springframework.transaction.support.TransactionTemplate import java.time.Clock @@ -118,6 +119,7 @@ class OutboxAutoConfiguration( fun okapiTransactionRunner( transactionManager: ObjectProvider, transactionTemplate: ObjectProvider, + outboxStore: ObjectProvider, beanFactory: BeanFactory, ): TransactionRunner { val anyTemplate = transactionTemplate.getIfUnique() @@ -139,7 +141,8 @@ class OutboxAutoConfiguration( validatePtmDataSourceMatch(ptm, resolveDataSource(), okapiProperties, dataSources.size) // Use the supplied TT verbatim ONLY when it actually wraps the PTM we picked (preserves // timeout/propagation/isolation). When qualifier forced a different PTM, build a fresh TT - // around it — otherwise we'd run on a TT bound to the wrong PTM. NOTE: with + // around it — otherwise we'd run on a TT bound to the wrong PTM. Either way, on MySQL the + // result is pinned to READ COMMITTED when it leaves isolation at DEFAULT (withOutboxIsolation). NOTE: with // PROPAGATION_REQUIRED a tick joining an outer readOnly=true TX inherits that flag // silently — keep scheduler invocations outside @Transactional scopes to avoid FOR UPDATE // SKIP LOCKED failures. @@ -164,7 +167,7 @@ class OutboxAutoConfiguration( } TransactionTemplate(ptm).apply { isReadOnly = false } } - return SpringTransactionRunner(ttToUse) + return SpringTransactionRunner(ttToUse.withOutboxIsolation(outboxStore.getIfUnique())) } @Bean @@ -296,6 +299,84 @@ class OutboxAutoConfiguration( companion object { private val logger = LoggerFactory.getLogger(OutboxAutoConfiguration::class.java) + /** + * Pins the scheduler transaction to READ COMMITTED when [store] is a MySQL store and [this] + * leaves isolation at DEFAULT (issue #129). Under MySQL's default REPEATABLE READ, the claim's + * `FOR UPDATE SKIP LOCKED` takes gap locks on `(status, delivery_type, created_at, id)` -- SKIP + * LOCKED only skips record locks -- and with a backlog smaller than the batch the gap runs to the + * end of the claimed type's range, exactly where every `publish()` of that type inserts. Publishers then block until delivery + * finishes and the claim commits. READ COMMITTED takes no gap locks. + * + * Returned unchanged when: + * - the template sets an explicit isolation level (the user's choice wins); + * - the store is not MySQL: Postgres already defaults to READ COMMITTED, so pinning would buy + * nothing and cost an isolation round trip per tick (a custom store wrapping MySQL is not + * recognised -- give it a template with an explicit isolation level); + * - the PTM rejects custom isolation levels ([customIsolationUnsupportedReason]). Pinning + * would make every tick throw `InvalidIsolationLevelException`, so a WARN tells the user + * how to get READ COMMITTED instead. + * + * A copy rather than a mutation: the template may be the user's or Boot's shared bean, and + * the copy keeps its timeout/propagation/read-only settings. + */ + internal fun TransactionTemplate.withOutboxIsolation(store: OutboxStore?): TransactionTemplate { + if (isolationLevel != TransactionDefinition.ISOLATION_DEFAULT) return this + if (store == null || !isInstanceOfAny(store, setOf(MYSQL_OUTBOX_STORE_CLASS))) return this + val ptm = checkNotNull(transactionManager) { "TransactionTemplate has no PlatformTransactionManager" } + customIsolationUnsupportedReason(ptm)?.let { reason -> + logger.warn( + "okapi could not pin the scheduler transaction to READ COMMITTED: PlatformTransactionManager " + + "'{}' {}. It runs at the database default instead -- on MySQL that is usually REPEATABLE " + + "READ, where the claim's gap locks block concurrent publish() calls until delivery " + + "finishes (issue #129). Fix: define a TransactionTemplate bean with isolationLevel = " + + "ISOLATION_READ_COMMITTED (okapi uses it as-is) once the PTM accepts custom isolation " + + "levels, or set transaction_isolation = 'READ-COMMITTED' on the MySQL server or in the " + + "JDBC URL.", + ptm.javaClass.name, + reason, + ) + return this + } + return TransactionTemplate(ptm, this).apply { + isolationLevel = TransactionDefinition.ISOLATION_READ_COMMITTED + } + } + + // By name: okapi-mysql is compileOnly, so a class literal would fail to link without it. + private const val MYSQL_OUTBOX_STORE_CLASS = "com.softwaremill.okapi.mysql.MysqlOutboxStore" + + private const val JTA_PTM_CLASS = "org.springframework.transaction.jta.JtaTransactionManager" + private const val JPA_PTM_CLASS = "org.springframework.orm.jpa.JpaTransactionManager" + + // JpaDialects whose beginTransaction applies a custom isolation level; Spring's + // DefaultJpaDialect (and so any JpaTransactionManager without a vendor dialect) rejects it. + private val ISOLATION_CAPABLE_JPA_DIALECTS = setOf( + "org.springframework.orm.jpa.vendor.HibernateJpaDialect", + "org.springframework.orm.jpa.vendor.EclipseLinkJpaDialect", + ) + + /** + * Why [ptm] would throw `InvalidIsolationLevelException` on a non-DEFAULT isolation level, or + * `null` if it accepts one. Checked by class name so spring-orm stays optional. + * + * `JtaTransactionManager` rejects custom levels unless `allowCustomIsolationLevels` is set, + * which has no public getter -- so JTA is always treated as unsupported; a user who enabled + * it can supply a template with an explicit level. + */ + internal fun customIsolationUnsupportedReason(ptm: PlatformTransactionManager): String? = when { + isInstanceOfAny(ptm, setOf(JTA_PTM_CLASS)) -> + "is a JtaTransactionManager, which rejects custom isolation levels unless allowCustomIsolationLevels is set" + isInstanceOfAny(ptm, setOf(JPA_PTM_CLASS)) -> { + val dialect = ptm.javaClass.getMethod("getJpaDialect").invoke(ptm) + if (dialect != null && isInstanceOfAny(dialect, ISOLATION_CAPABLE_JPA_DIALECTS)) { + null + } else { + "is a JpaTransactionManager with JpaDialect '${dialect?.javaClass?.name}', which rejects custom isolation levels" + } + } + else -> null + } + internal fun resolveDataSource( dataSources: Map, primaryDataSource: DataSource, @@ -481,14 +562,16 @@ class OutboxAutoConfiguration( "org.springframework.orm.hibernate5.HibernateTransactionManager", ) - // Walk the superclass chain (not just `ptm.javaClass.name`) so user-defined subclasses - // of `JpaTransactionManager` / `HibernateTransactionManager` are also recognised. Reads + private fun isJpaHibernatePtm(ptm: PlatformTransactionManager): Boolean = isInstanceOfAny(ptm, JPA_HIBERNATE_PTM_CLASSES) + + // Walk the superclass chain (not just `obj.javaClass.name`) so user-defined subclasses + // (e.g. of `JpaTransactionManager` / `HibernateTransactionManager`) are also recognised. Reads // `.name` / `.superclass` on the already-loaded hierarchy only — no `Class.forName`, so - // the "no hard spring-orm compile dependency" constraint is preserved. - private fun isJpaHibernatePtm(ptm: PlatformTransactionManager): Boolean { - var c: Class<*>? = ptm.javaClass + // the "no hard spring-orm / okapi-mysql compile dependency" constraint is preserved. + private fun isInstanceOfAny(obj: Any, classNames: Set): Boolean { + var c: Class<*>? = obj.javaClass while (c != null) { - if (c.name in JPA_HIBERNATE_PTM_CLASSES) return true + if (c.name in classNames) return true c = c.superclass } return false diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt new file mode 100644 index 00000000..c03dbeab --- /dev/null +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfigurationIsolationTest.kt @@ -0,0 +1,177 @@ +package com.softwaremill.okapi.springboot + +import ch.qos.logback.classic.Level +import ch.qos.logback.classic.Logger +import ch.qos.logback.classic.spi.ILoggingEvent +import ch.qos.logback.core.read.ListAppender +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.assertions.withClue +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldBeEmpty +import io.kotest.matchers.collections.shouldHaveSize +import io.kotest.matchers.nulls.shouldBeNull +import io.kotest.matchers.shouldBe +import io.kotest.matchers.string.shouldContain +import io.kotest.matchers.types.shouldBeInstanceOf +import io.kotest.matchers.types.shouldBeSameInstanceAs +import jakarta.transaction.Status +import jakarta.transaction.UserTransaction +import org.slf4j.LoggerFactory +import org.springframework.boot.autoconfigure.AutoConfigurations +import org.springframework.boot.test.context.runner.ApplicationContextRunner +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Configuration +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.jdbc.datasource.SimpleDriverDataSource +import org.springframework.transaction.InvalidIsolationLevelException +import org.springframework.transaction.PlatformTransactionManager +import org.springframework.transaction.TransactionDefinition +import org.springframework.transaction.jta.JtaTransactionManager +import org.springframework.transaction.support.TransactionTemplate +import javax.sql.DataSource + +/** + * When `okapiTransactionRunner` pins READ COMMITTED (issue #129): only for a MySQL store, only when + * the template leaves isolation at DEFAULT, and only when the PTM accepts a custom level. + */ +class OutboxAutoConfigurationIsolationTest : FunSpec({ + + val ds: DataSource = SimpleDriverDataSource() + + fun runner(store: () -> OutboxStore) = ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withBean(DataSource::class.java, { ds }) + .withBean(OutboxStore::class.java, store) + .withBean(MessageDeliverer::class.java, { stubDeliverer() }) + + val mysqlRunner = runner { MysqlOutboxStore(SpringConnectionProvider(ds)) } + + fun captureWarnings(block: () -> Unit): List { + val logger = LoggerFactory.getLogger(OutboxAutoConfiguration::class.java) as Logger + val appender = ListAppender().apply { start() } + logger.addAppender(appender) + try { + block() + } finally { + logger.detachAppender(appender) + } + return appender.list.filter { it.level == Level.WARN } + } + + test("MySQL store: the auto-built TransactionTemplate runs at READ COMMITTED") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + } + } + + test("MySQL store: a user TransactionTemplate at DEFAULT isolation is copied, keeping its settings, not mutated") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(DefaultIsolationTemplateConfiguration::class.java) + .run { ctx -> + val template = ctx.getBean(TransactionTemplate::class.java) + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_READ_COMMITTED + runner.transactionTemplate.timeout shouldBe 42 + runner.transactionTemplate.transactionManager shouldBeSameInstanceAs template.transactionManager + withClue("the user's shared template bean itself must not be mutated") { + template.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + } + } + } + + test("MySQL store: a user TransactionTemplate with an explicit isolation level is used verbatim") { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(SerializableTemplateConfiguration::class.java) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate shouldBeSameInstanceAs ctx.getBean(TransactionTemplate::class.java) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_SERIALIZABLE + } + } + + test("non-MySQL store: isolation is left at DEFAULT and a user template is used verbatim") { + runner { stubStore() } + .withBean(PlatformTransactionManager::class.java, { DataSourceTransactionManager(ds) }) + .withUserConfiguration(DefaultIsolationTemplateConfiguration::class.java) + .run { ctx -> + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate shouldBeSameInstanceAs ctx.getBean(TransactionTemplate::class.java) + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + } + } + + test("premise: JtaTransactionManager rejects READ COMMITTED by default, so pinning it would fail every tick") { + val jta = JtaTransactionManager(FakeUserTransaction()).apply { afterPropertiesSet() } + shouldThrow { + TransactionTemplate(jta).apply { isolationLevel = TransactionDefinition.ISOLATION_READ_COMMITTED }.execute { } + } + } + + test("MySQL store + JtaTransactionManager: isolation is not pinned, ticks still run, and a WARN explains the fix") { + val warnings = captureWarnings { + mysqlRunner + .withBean(PlatformTransactionManager::class.java, { JtaTransactionManager(FakeUserTransaction()) }) + .run { ctx -> + ctx.startupFailure.shouldBeNull() + val runner = ctx.getBean(TransactionRunner::class.java).shouldBeInstanceOf() + runner.transactionTemplate.isolationLevel shouldBe TransactionDefinition.ISOLATION_DEFAULT + runner.runInTransaction { "tick" } shouldBe "tick" + } + } + val warning = warnings.filter { it.formattedMessage.contains("READ COMMITTED") }.shouldHaveSize(1).single() + warning.formattedMessage shouldContain JtaTransactionManager::class.java.name + warning.formattedMessage shouldContain "allowCustomIsolationLevels" + warning.formattedMessage shouldContain "ISOLATION_READ_COMMITTED" + } + + test("non-MySQL store + JtaTransactionManager: no isolation WARN (nothing to pin)") { + val warnings = captureWarnings { + runner { stubStore() } + .withBean(PlatformTransactionManager::class.java, { JtaTransactionManager(FakeUserTransaction()) }) + .run { ctx -> ctx.startupFailure.shouldBeNull() } + } + warnings.filter { it.formattedMessage.contains("READ COMMITTED") }.shouldBeEmpty() + } +}) + +/** Just enough of a JTA `UserTransaction` for `JtaTransactionManager` to begin and commit. */ +private class FakeUserTransaction : UserTransaction { + private var status = Status.STATUS_NO_TRANSACTION + override fun begin() { + status = Status.STATUS_ACTIVE + } + override fun commit() { + status = Status.STATUS_NO_TRANSACTION + } + override fun rollback() { + status = Status.STATUS_NO_TRANSACTION + } + override fun setRollbackOnly() { + status = Status.STATUS_MARKED_ROLLBACK + } + override fun getStatus(): Int = status + override fun setTransactionTimeout(seconds: Int) = Unit +} + +@Configuration(proxyBeanMethods = false) +private class DefaultIsolationTemplateConfiguration { + @Bean + fun defaultIsolationTemplate(ptm: PlatformTransactionManager) = TransactionTemplate(ptm).apply { timeout = 42 } +} + +@Configuration(proxyBeanMethods = false) +private class SerializableTemplateConfiguration { + @Bean + fun serializableTemplate(ptm: PlatformTransactionManager) = TransactionTemplate(ptm).apply { + isolationLevel = TransactionDefinition.ISOLATION_SERIALIZABLE + } +}