diff --git a/CHANGELOG.md b/CHANGELOG.md index 60925d8..7a71d60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -122,6 +122,23 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * `OutboxStore.claimPending(limit)` ignores the backoff. Tests or callers that relied on immediate retries need `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. ([#131](https://github.com/softwaremill/okapi/issues/131)) +- **`OutboxProcessorListener` events are emitted after the transaction commits.** `OutboxProcessor` + notified the listener right after writing a batch, still inside the scheduler's transaction, so a + failed commit (connection loss, serialization failure, timeout) left `okapi.entries.delivered` / + `okapi.entries.failed` incremented for rows that rolled back to `PENDING`, which were then counted + again on redelivery, and a custom listener could act on state that was never persisted. + `OutboxProcessor` now has `processBatch(limit)`, which claims, delivers and persists without + notifying and returns a `ProcessedBatch`, and `publishEvents(batch)`. `OutboxScheduler` runs + `processBatch` in the transaction and `publishEvents` after it returns, so under the scheduler the + listener runs after the commit, outside the transaction, and not at all for a batch whose commit + failed. A listener that relied on running inside the processing transaction (e.g. to write to the + database atomically with the status update) no longer can. `onBatchProcessed`'s duration, and with + it `okapi.batch.duration`, now includes the commit (and excludes the listener callbacks); empty + polls are still reported. Counters now count committed status changes rather than delivery + attempts. `TransactionRunner` now documents that `runInTransaction` must have committed when it + returns, and `publishEvents` throws if a batch is published twice. + `processNext(limit)` keeps notifying at once, for callers that drive the processor themselves. + ([#137](https://github.com/softwaremill/okapi/issues/137)) - **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the `CompositeMessageDeliverer` entry above). `@JvmOverloads` preserves the previous four-argument JVM constructor, so Java callers are unaffected, but adding a property to a Kotlin `data class` diff --git a/README.md b/README.md index 373ae70..532e0ed 100644 --- a/README.md +++ b/README.md @@ -279,7 +279,7 @@ Metrics are then available at `/actuator/prometheus`. | `okapi.entries.delivered` | Counter | Successfully delivered entries | | `okapi.entries.retry.scheduled` | Counter | Failed attempts rescheduled for retry | | `okapi.entries.failed` | Counter | Permanently failed entries | -| `okapi.batch.duration` | Timer | Processing time per batch | +| `okapi.batch.duration` | Timer | Processing time per batch; under the scheduler from the claim until the transaction has committed (not including listener callbacks) | | `okapi.entries.count` | Gauge | Current entry count (tag: `status`) | | `okapi.entries.lag.seconds` | Gauge | Age of the oldest entry (tag: `status`) | @@ -297,7 +297,7 @@ max by (job, status) (okapi_entries_count) Without Spring Boot, construct `MicrometerOutboxListener` and `MicrometerOutboxMetrics` with your `MeterRegistry`. Refresh gauges with `OutboxMetricsRefresher` or your own scheduler. -For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. +For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. Counters therefore count committed status changes, not delivery attempts: a send whose transaction then rolled back (a failed commit, or a block Exposed retried) is not counted, though the message may have reached the consumer. If you drive `OutboxProcessor` yourself, call `processBatch` inside your transaction and `publishEvents` after it commits; `processNext` does both at once, so inside a transaction its events precede the commit. ## Modules diff --git a/UPGRADING.md b/UPGRADING.md index ac724b0..ef1315b 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -105,6 +105,25 @@ rows and replaces 002's claim index with one on `(delivery_type, next_attempt_at that drive retries tick by tick need a zero backoff: `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. +### `OutboxProcessorListener` runs after the commit (#137) + +Under `OutboxScheduler` (and so with Spring Boot autoconfiguration), listener callbacks now run after +the processing transaction has committed, outside of it, and are skipped for a batch whose commit +failed. + +- A custom listener that wrote to the database expecting to join the processing transaction now + runs without one: open your own transaction in the callback, or make the write idempotent. This + fails quietly: an Exposed listener gets "No transaction in context", a Spring call with + `MANDATORY` propagation throws, and okapi only logs each such exception as a WARN. +- Counters (`okapi.entries.delivered`, `okapi.entries.failed`, ...) now count committed status + changes. A send whose transaction rolled back is no longer counted. +- A custom `TransactionRunner` must have committed when `runInTransaction` returns (now documented on + the interface); one that joins an outer transaction brings back pre-commit events. +- `okapi.batch.duration` now includes the commit, so it reads slightly higher than before. +- If you call `OutboxProcessor.processNext` inside your own transaction, events still fire before + your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by + `processor.publishEvents(batch)` to get the after-commit behaviour. + ## From 0.2.x to 1.0.0 Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt index b62bc0f..b062bd6 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt @@ -59,7 +59,7 @@ class CompositeMessageDeliverer @JvmOverloads constructor( * satisfy that opt out via [TransportDispatch.SEQUENTIAL]. * * The outbox transaction is open across delivery, and stays that way: the scheduler wraps the - * whole [OutboxProcessor.processNext] cycle — claim, deliver, update — in one transaction, so + * whole [OutboxProcessor.processBatch] cycle — claim, deliver, update — in one transaction, so * that `FOR UPDATE SKIP LOCKED` holds its row locks until the results are written. What this * dispatch changes is which thread the transport runs on, not that scope. The store round-trips * still run on the calling thread inside the transaction, but a group handed to a virtual diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index 8afb39c..4a11b0d 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -13,6 +13,12 @@ import java.util.concurrent.atomic.AtomicInteger * full batch. Exceptions in the listener are caught and logged — they never break * processing. Transaction management is the caller's responsibility. * + * Listener events describe state the caller's transaction has persisted, so they belong after + * it commits (issue #137): call [processBatch] inside the transaction and [publishEvents] after + * it commits, as [OutboxScheduler] does. A commit that fails then emits nothing, and the entries + * are reported again when they are re-processed. [processNext] does both in one call, so inside a + * transaction its events precede the commit. + * * In the batch processing path, the per-entry `duration` reported in * [OutboxProcessingEvent] reflects the **wall-clock duration of the whole batch** * (because transports may overlap their per-entry I/O internally — e.g. Kafka's @@ -29,11 +35,21 @@ class OutboxProcessor @JvmOverloads constructor( private val nextStartingType = AtomicInteger() /** - * Claims up to [limit] pending entries, processes them as a batch, and persists - * each result. Returns the number of entries processed (0 if the store had nothing). + * Claims up to [limit] pending entries, processes them as a batch, persists each result and + * notifies the listener at once. Returns the number of entries processed (0 if the store had + * nothing). Called inside a transaction, the events precede its commit; use [processBatch] and + * [publishEvents] to emit them only after the commit. + */ + @JvmOverloads + fun processNext(limit: Int = 10): Int = processBatch(limit).also { publishEvents(it) }.count + + /** + * Claims up to [limit] pending entries, processes them as a batch and persists each result, + * without notifying the listener. Run it inside the processing transaction and pass the result + * to [publishEvents] once that transaction has committed. */ @JvmOverloads - fun processNext(limit: Int = 10): Int { + fun processBatch(limit: Int = 10): ProcessedBatch { val batchStart = clock.instant() val routeAwareStore = store as? RouteAwareOutboxStore ?: error( @@ -63,19 +79,26 @@ class OutboxProcessor @JvmOverloads constructor( first + other } } - if (claimed.isEmpty()) { - notifyBatch(0, Duration.between(batchStart, clock.instant())) - return 0 - } + if (claimed.isEmpty()) return ProcessedBatch(emptyList(), batchStart, Duration.ZERO) val processed = entryProcessor.processBatch(claimed) - val batchDuration = Duration.between(batchStart, clock.instant()) + val deliveryDuration = Duration.between(batchStart, clock.instant()) store.updateAfterProcessingBatch(processed) - processed.forEach { updated -> notifyEntry(updated, batchDuration) } + return ProcessedBatch(processed, batchStart, deliveryDuration) + } - notifyBatch(processed.size, Duration.between(batchStart, clock.instant())) - return processed.size + /** + * Notifies the listener of [batch]: [OutboxProcessorListener.onEntryProcessed] for each entry, + * then [OutboxProcessorListener.onBatchProcessed], whose duration runs from the start of the batch + * until this call, so it includes the commit when called after one, but not the listener + * callbacks. A batch's events can be published only once; a second call throws. + */ + fun publishEvents(batch: ProcessedBatch) { + check(batch.markPublished()) { "Events of this batch were already published" } + val batchDuration = Duration.between(batch.startedAt, clock.instant()) + batch.entries.forEach { updated -> notifyEntry(updated, batch.deliveryDuration) } + notifyBatch(batch.count, batchDuration) } private fun notifyEntry(updated: OutboxEntry, duration: Duration) { diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt index 65fa38c..364211d 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt @@ -11,6 +11,11 @@ import java.time.Duration * * [OutboxProcessor] accepts a single listener. To combine multiple listeners, * implement a composite that delegates to each. + * + * Under [OutboxScheduler], callbacks run after the processing transaction has committed, outside of + * it, and not at all for a batch whose commit failed (issue #137). A callback therefore cannot take + * part in that transaction, and every event describes persisted state. With [OutboxProcessor.processNext] + * called inside your own transaction, events still precede your commit. */ interface OutboxProcessorListener { /** Called after each entry is processed (delivered, retried, or failed). */ diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt index 0f427e8..3876d60 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt @@ -12,7 +12,8 @@ import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicBoolean /** - * Standalone scheduler that periodically calls [OutboxProcessor.processNext]. + * Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and, + * once it has committed, [OutboxProcessor.publishEvents]. * * Each worker's call runs inside [transactionRunner]. The runner is required: without a * surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the @@ -179,7 +180,9 @@ class OutboxScheduler @JvmOverloads constructor( private fun processBatch() { try { - transactionRunner.runInTransaction { outboxProcessor.processNext(config.batchSize) } + val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } + // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. + outboxProcessor.publishEvents(batch) logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize) } catch (e: Exception) { logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt new file mode 100644 index 0000000..f07eff5 --- /dev/null +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt @@ -0,0 +1,30 @@ +package com.softwaremill.okapi.core + +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.concurrent.atomic.AtomicBoolean + +/** + * The outcome of [OutboxProcessor.processBatch]: [entries] in the state just persisted, whose + * listener events are held back until [OutboxProcessor.publishEvents]. Only [OutboxProcessor] + * creates one; its constructor and timing fields are internal to okapi-core. + */ +class ProcessedBatch internal constructor( + entries: List, + internal val startedAt: Instant, + internal val deliveryDuration: Duration, +) { + /** The processed entries, in the state persisted for them. Read-only. */ + val entries: List = Collections.unmodifiableList(entries.toList()) + + /** Number of entries processed (0 if the store had nothing to claim). */ + val count: Int get() = entries.size + + private val published = AtomicBoolean(false) + + /** Returns `true` the first time only. */ + internal fun markPublished(): Boolean = published.compareAndSet(false, true) + + override fun toString(): String = "ProcessedBatch(count=$count, startedAt=$startedAt, deliveryDuration=$deliveryDuration)" +} diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt index bc17a92..fa85276 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt @@ -7,6 +7,13 @@ package com.softwaremill.okapi.core * - `okapi-spring-boot`: wraps Spring's `TransactionTemplate` via `SpringTransactionRunner` * - `okapi-exposed`: wraps Exposed's `transaction {}` via `ExposedTransactionRunner` * - Standalone: user-provided lambda + * + * Contract: when [runInTransaction] returns normally, the transaction it ran [block] in has + * committed. [OutboxScheduler] relies on this to emit listener events only for committed state + * (issue #137). A runner that joins an outer transaction instead (e.g. Spring's `REQUIRED` called + * inside `@Transactional`, or a nested Exposed `transaction {}` without nested transactions) breaks + * that. A runner may run [block] more than once, as Exposed does when a statement or the commit + * throws an `SQLException`; only the result of the attempt that committed is returned. */ interface TransactionRunner { fun runInTransaction(block: () -> T): T diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt new file mode 100644 index 0000000..aa8cc27 --- /dev/null +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt @@ -0,0 +1,174 @@ +package com.softwaremill.okapi.core + +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldBeEmpty +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +private val info = object : DeliveryInfo { + override val type = "stub" + override fun serialize() = "{}" +} + +/** Hands out one fresh PENDING entry per claim. */ +private class OneEntryPerClaimStore : RouteAwareOutboxStore { + override fun persist(entry: OutboxEntry) = entry + override fun claimPending(limit: Int): List = error("not used") + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = + listOf(OutboxEntry.createPending(OutboxMessage("e", "{}"), info, Instant.EPOCH)) + override fun updateAfterProcessing(entry: OutboxEntry) = entry + override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 + override fun findOldestCreatedAt(statuses: Set) = emptyMap() + override fun countByStatuses() = emptyMap() +} + +private class RecordingListener(private val log: MutableList, private val onBatch: () -> Unit = {}) : OutboxProcessorListener { + override fun onEntryProcessed(event: OutboxProcessingEvent) { + log += "entry:${event::class.simpleName}" + } + + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + log += "batch:$processedCount" + onBatch() + } +} + +private fun processor(listener: OutboxProcessorListener, store: RouteAwareOutboxStore = OneEntryPerClaimStore()) = OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = "stub" + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 1), + Clock.systemUTC(), + ), + listener = listener, +) + +/** Issue #137: listener events describe committed state, so the scheduler emits them after the commit. */ +class ListenerAfterCommitTest : FunSpec({ + test("processBatch persists without notifying; publishEvents notifies") { + val log = mutableListOf() + val processor = processor(RecordingListener(log)) + + val batch = processor.processBatch(10) + batch.count shouldBe 1 + log.shouldBeEmpty() + + processor.publishEvents(batch) + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } + + test("processNext still notifies at once") { + val log = mutableListOf() + processor(RecordingListener(log)).processNext(10) shouldBe 1 + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } + + test("the scheduler emits events only after the transaction has committed") { + val log = Collections.synchronizedList(mutableListOf()) + val batchSeen = CountDownLatch(1) + val committingRunner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = block().also { log += "commit" } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log) { batchSeen.countDown() }), + committingRunner, + OutboxSchedulerConfig(interval = Duration.ofMinutes(1)), + ) + + scheduler.start() + batchSeen.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.take(3) shouldContainExactly listOf("commit", "entry:Delivered", "batch:1") + } + + test("a batch whose commit fails emits no events") { + val log = Collections.synchronizedList(mutableListOf()) + val ticks = CountDownLatch(2) + val failingCommit = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + block() + ticks.countDown() + throw IllegalStateException("commit failed: connection reset") + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log)), + failingCommit, + OutboxSchedulerConfig(interval = Duration.ofMillis(20)), + ) + + scheduler.start() + ticks.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.shouldBeEmpty() + } + + test("a runner that retries the block, as Exposed does on SQLException, publishes one set of events") { + val log = Collections.synchronizedList(mutableListOf()) + val batchSeen = CountDownLatch(1) + val retryingRunner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + block() // first attempt: its commit fails and it is rolled back + log += "rollback" + return block().also { log += "commit" } + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log) { batchSeen.countDown() }), + retryingRunner, + OutboxSchedulerConfig(interval = Duration.ofMinutes(1)), + ) + + scheduler.start() + batchSeen.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.take(4) shouldContainExactly listOf("rollback", "commit", "entry:Delivered", "batch:1") + } + + test("a processBatch that throws publishes nothing") { + val log = Collections.synchronizedList(mutableListOf()) + val ticks = CountDownLatch(2) + val failingStore = object : RouteAwareOutboxStore by OneEntryPerClaimStore() { + override fun updateAfterProcessingBatch(entries: List): List { + ticks.countDown() + throw IllegalStateException("update failed") + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log), failingStore), + object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = block() + }, + OutboxSchedulerConfig(interval = Duration.ofMillis(20)), + ) + + scheduler.start() + ticks.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.shouldBeEmpty() + } + + test("a batch's events can be published only once") { + val log = mutableListOf() + val processor = processor(RecordingListener(log)) + val batch = processor.processBatch(10) + + processor.publishEvents(batch) + shouldThrow { processor.publishEvents(batch) } + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } +}) diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt index b2f88a6..4efc130 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt @@ -24,7 +24,7 @@ import java.util.concurrent.atomic.AtomicReference class OutboxSchedulerTest : FunSpec({ - test("tick calls processNext with configured batchSize") { + test("tick calls processBatch with configured batchSize") { var capturedLimit: Int? = null val latch = CountDownLatch(1) val processor = stubProcessor { limit -> diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt new file mode 100644 index 0000000..d9e3142 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt @@ -0,0 +1,111 @@ +package com.softwaremill.okapi.test.transaction + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessingEvent +import com.softwaremill.okapi.core.OutboxProcessor +import com.softwaremill.okapi.core.OutboxProcessorListener +import com.softwaremill.okapi.core.OutboxScheduler +import com.softwaremill.okapi.core.OutboxSchedulerConfig +import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.exposed.ExposedConnectionProvider +import com.softwaremill.okapi.exposed.ExposedTransactionRunner +import com.softwaremill.okapi.postgres.PostgresOutboxStore +import com.softwaremill.okapi.springboot.SpringConnectionProvider +import com.softwaremill.okapi.springboot.SpringTransactionRunner +import com.softwaremill.okapi.test.support.PostgresTestSupport +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import org.jetbrains.exposed.v1.jdbc.Database +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.transaction.support.TransactionTemplate +import java.time.Clock +import java.time.Duration +import java.util.concurrent.CompletableFuture +import java.util.concurrent.TimeUnit + +private object AfterCommitDeliveryInfo : DeliveryInfo { + override val type: String = "after-commit" + override fun serialize(): String = """{"type":"after-commit"}""" +} + +/** + * Issue #137 against a real database: when the scheduler's listener hears that an entry was + * delivered, another connection must already see it as DELIVERED, i.e. the transaction committed. + */ +class ListenerAfterCommitEndToEndTest : FunSpec({ + val db = PostgresTestSupport() + + beforeSpec { db.start() } + afterSpec { db.stop() } + + /** Runs one scheduler tick with [runner] and [store]; returns the status the listener saw from another connection. */ + fun statusSeenByListener(runner: TransactionRunner, store: OutboxStore): String { + db.truncate() + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), AfterCommitDeliveryInfo, Clock.systemUTC().instant()) + db.jdbc.withTransaction { PostgresOutboxStore(db.jdbc).persist(entry) } + + val seen = CompletableFuture() + val listener = object : OutboxProcessorListener { + override fun onEntryProcessed(event: OutboxProcessingEvent) { + try { + // A connection of its own: it only sees committed data. + db.dataSource.connection.use { conn -> + conn.prepareStatement("SELECT status FROM okapi_outbox WHERE id = ?::uuid").use { stmt -> + stmt.setString(1, entry.outboxId.raw.toString()) + stmt.executeQuery().use { rs -> + check(rs.next()) { "row ${entry.outboxId} not found" } + seen.complete(rs.getString("status")) + } + } + } + } catch (e: Exception) { + seen.completeExceptionally(e) + } + } + } + val processor = OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = AfterCommitDeliveryInfo.type + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 0), + Clock.systemUTC(), + ), + listener = listener, + ) + val scheduler = OutboxScheduler(processor, runner, OutboxSchedulerConfig(interval = Duration.ofMinutes(1))) + scheduler.start() + try { + return seen.get(10, TimeUnit.SECONDS) + } finally { + scheduler.stop() + } + } + + test("plain JDBC runner: the listener sees the delivered status already committed") { + val runner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = db.jdbc.withTransaction(block) + } + statusSeenByListener(runner, PostgresOutboxStore(db.jdbc)) shouldBe "DELIVERED" + } + + test("SpringTransactionRunner: the listener sees the delivered status already committed") { + val runner = SpringTransactionRunner(TransactionTemplate(DataSourceTransactionManager(db.dataSource))) + statusSeenByListener(runner, PostgresOutboxStore(SpringConnectionProvider(db.dataSource))) shouldBe "DELIVERED" + } + + test("ExposedTransactionRunner: the listener sees the delivered status already committed") { + val database = Database.connect(db.dataSource) + statusSeenByListener(ExposedTransactionRunner(database), PostgresOutboxStore(ExposedConnectionProvider(database))) shouldBe + "DELIVERED" + } +})