From a005818373ad20a46a681ea5a2c6959acd123d3e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Fri, 6 Nov 2026 10:30:29 +0100 Subject: [PATCH 1/3] fix(core)!: emit OutboxProcessorListener events after the transaction commits (#137) OutboxProcessor notified the listener right after writing a batch, inside the scheduler's transaction, so a failed commit left delivered/failed counters incremented for rows that rolled back to PENDING and were counted again on redelivery. OutboxProcessor gains processBatch (claim, deliver, persist, no events) and publishEvents; OutboxScheduler runs the first in the transaction and the second after it returns, so events describe committed state and a failed commit emits none. processNext keeps notifying at once for callers that drive the processor themselves. BREAKING CHANGE: under the scheduler, listener callbacks run after the commit, outside the processing transaction, and are skipped when the commit fails; okapi.batch.duration includes the commit. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 14 +++ README.md | 4 +- UPGRADING.md | 13 ++ .../okapi/core/OutboxProcessor.kt | 42 +++++-- .../okapi/core/OutboxProcessorListener.kt | 5 + .../okapi/core/OutboxScheduler.kt | 7 +- .../softwaremill/okapi/core/ProcessedBatch.kt | 17 +++ .../okapi/core/ListenerAfterCommitTest.kt | 116 ++++++++++++++++++ .../ListenerAfterCommitEndToEndTest.kt | 84 +++++++++++++ 9 files changed, 287 insertions(+), 15 deletions(-) create mode 100644 okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt create mode 100644 okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt create mode 100644 okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index 60925d89..4434cde4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -122,6 +122,20 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * `OutboxStore.claimPending(limit)` ignores the backoff. Tests or callers that relied on immediate retries need `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. ([#131](https://github.com/softwaremill/okapi/issues/131)) +- **`OutboxProcessorListener` events are emitted after the transaction commits.** `OutboxProcessor` + notified the listener right after writing a batch, still inside the scheduler's transaction, so a + failed commit (connection loss, serialization failure, timeout) left `okapi.entries.delivered` / + `okapi.entries.failed` incremented for rows that rolled back to `PENDING`, which were then counted + again on redelivery, and a custom listener could act on state that was never persisted. + `OutboxProcessor` now has `processBatch(limit)`, which claims, delivers and persists without + notifying and returns a `ProcessedBatch`, and `publishEvents(batch)`. `OutboxScheduler` runs + `processBatch` in the transaction and `publishEvents` after it returns, so under the scheduler the + listener runs after the commit, outside the transaction, and not at all for a batch whose commit + failed. A listener that relied on running inside the processing transaction (e.g. to write to the + database atomically with the status update) no longer can. `onBatchProcessed`'s duration, and with + it `okapi.batch.duration`, now includes the commit; empty polls are still reported. + `processNext(limit)` keeps notifying at once, for callers that drive the processor themselves. + ([#137](https://github.com/softwaremill/okapi/issues/137)) - **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the `CompositeMessageDeliverer` entry above). `@JvmOverloads` preserves the previous four-argument JVM constructor, so Java callers are unaffected, but adding a property to a Kotlin `data class` diff --git a/README.md b/README.md index 373ae702..e283d9da 100644 --- a/README.md +++ b/README.md @@ -279,7 +279,7 @@ Metrics are then available at `/actuator/prometheus`. | `okapi.entries.delivered` | Counter | Successfully delivered entries | | `okapi.entries.retry.scheduled` | Counter | Failed attempts rescheduled for retry | | `okapi.entries.failed` | Counter | Permanently failed entries | -| `okapi.batch.duration` | Timer | Processing time per batch | +| `okapi.batch.duration` | Timer | Processing time per batch, from the claim until the transaction has committed | | `okapi.entries.count` | Gauge | Current entry count (tag: `status`) | | `okapi.entries.lag.seconds` | Gauge | Age of the oldest entry (tag: `status`) | @@ -297,7 +297,7 @@ max by (job, status) (okapi_entries_count) Without Spring Boot, construct `MicrometerOutboxListener` and `MicrometerOutboxMetrics` with your `MeterRegistry`. Refresh gauges with `OutboxMetricsRefresher` or your own scheduler. -For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. +For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. If you drive `OutboxProcessor` yourself, call `processBatch` inside your transaction and `publishEvents` after it commits; `processNext` does both at once, so inside a transaction its events precede the commit. ## Modules diff --git a/UPGRADING.md b/UPGRADING.md index ac724b01..9bdb800d 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -105,6 +105,19 @@ rows and replaces 002's claim index with one on `(delivery_type, next_attempt_at that drive retries tick by tick need a zero backoff: `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. +### `OutboxProcessorListener` runs after the commit (#137) + +Under `OutboxScheduler` (and so with Spring Boot autoconfiguration), listener callbacks now run after +the processing transaction has committed, outside of it, and are skipped for a batch whose commit +failed. + +- A custom listener that wrote to the database expecting to join the processing transaction now + runs without one: open your own transaction in the callback, or make the write idempotent. +- `okapi.batch.duration` now includes the commit, so it reads slightly higher than before. +- If you call `OutboxProcessor.processNext` inside your own transaction, events still fire before + your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by + `processor.publishEvents(batch)` to get the after-commit behaviour. + ## From 0.2.x to 1.0.0 Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index 8afb39c3..6cea89e3 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -13,6 +13,12 @@ import java.util.concurrent.atomic.AtomicInteger * full batch. Exceptions in the listener are caught and logged — they never break * processing. Transaction management is the caller's responsibility. * + * Listener events describe state the caller's transaction has persisted, so they belong after + * it commits (issue #137): call [processBatch] inside the transaction and [publishEvents] after + * it commits, as [OutboxScheduler] does. A commit that fails then emits nothing, and the entries + * are reported again when they are re-processed. [processNext] does both in one call, so inside a + * transaction its events precede the commit. + * * In the batch processing path, the per-entry `duration` reported in * [OutboxProcessingEvent] reflects the **wall-clock duration of the whole batch** * (because transports may overlap their per-entry I/O internally — e.g. Kafka's @@ -29,11 +35,21 @@ class OutboxProcessor @JvmOverloads constructor( private val nextStartingType = AtomicInteger() /** - * Claims up to [limit] pending entries, processes them as a batch, and persists - * each result. Returns the number of entries processed (0 if the store had nothing). + * Claims up to [limit] pending entries, processes them as a batch, persists each result and + * notifies the listener at once. Returns the number of entries processed (0 if the store had + * nothing). Called inside a transaction, the events precede its commit; use [processBatch] and + * [publishEvents] to emit them only after the commit. + */ + @JvmOverloads + fun processNext(limit: Int = 10): Int = processBatch(limit).also { publishEvents(it) }.count + + /** + * Claims up to [limit] pending entries, processes them as a batch and persists each result, + * without notifying the listener. Run it inside the processing transaction and pass the result + * to [publishEvents] once that transaction has committed. */ @JvmOverloads - fun processNext(limit: Int = 10): Int { + fun processBatch(limit: Int = 10): ProcessedBatch { val batchStart = clock.instant() val routeAwareStore = store as? RouteAwareOutboxStore ?: error( @@ -63,19 +79,23 @@ class OutboxProcessor @JvmOverloads constructor( first + other } } - if (claimed.isEmpty()) { - notifyBatch(0, Duration.between(batchStart, clock.instant())) - return 0 - } + if (claimed.isEmpty()) return ProcessedBatch(emptyList(), batchStart, Duration.ZERO) val processed = entryProcessor.processBatch(claimed) - val batchDuration = Duration.between(batchStart, clock.instant()) + val deliveryDuration = Duration.between(batchStart, clock.instant()) store.updateAfterProcessingBatch(processed) - processed.forEach { updated -> notifyEntry(updated, batchDuration) } + return ProcessedBatch(processed, batchStart, deliveryDuration) + } - notifyBatch(processed.size, Duration.between(batchStart, clock.instant())) - return processed.size + /** + * Notifies the listener of [batch]: [OutboxProcessorListener.onEntryProcessed] for each entry, + * then [OutboxProcessorListener.onBatchProcessed], whose duration runs from the start of the batch + * until now, so it includes the commit when called after one. Call it at most once per batch. + */ + fun publishEvents(batch: ProcessedBatch) { + batch.entries.forEach { updated -> notifyEntry(updated, batch.deliveryDuration) } + notifyBatch(batch.count, Duration.between(batch.startedAt, clock.instant())) } private fun notifyEntry(updated: OutboxEntry, duration: Duration) { diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt index 65fa38cb..364211d7 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessorListener.kt @@ -11,6 +11,11 @@ import java.time.Duration * * [OutboxProcessor] accepts a single listener. To combine multiple listeners, * implement a composite that delegates to each. + * + * Under [OutboxScheduler], callbacks run after the processing transaction has committed, outside of + * it, and not at all for a batch whose commit failed (issue #137). A callback therefore cannot take + * part in that transaction, and every event describes persisted state. With [OutboxProcessor.processNext] + * called inside your own transaction, events still precede your commit. */ interface OutboxProcessorListener { /** Called after each entry is processed (delivered, retried, or failed). */ diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt index 0f427e8f..3876d602 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt @@ -12,7 +12,8 @@ import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicBoolean /** - * Standalone scheduler that periodically calls [OutboxProcessor.processNext]. + * Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and, + * once it has committed, [OutboxProcessor.publishEvents]. * * Each worker's call runs inside [transactionRunner]. The runner is required: without a * surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the @@ -179,7 +180,9 @@ class OutboxScheduler @JvmOverloads constructor( private fun processBatch() { try { - transactionRunner.runInTransaction { outboxProcessor.processNext(config.batchSize) } + val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } + // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. + outboxProcessor.publishEvents(batch) logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize) } catch (e: Exception) { logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt new file mode 100644 index 00000000..9a71f47c --- /dev/null +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt @@ -0,0 +1,17 @@ +package com.softwaremill.okapi.core + +import java.time.Duration +import java.time.Instant + +/** + * The outcome of [OutboxProcessor.processBatch]: [entries] in the state just persisted, whose + * listener events are held back until [OutboxProcessor.publishEvents]. + */ +class ProcessedBatch internal constructor( + val entries: List, + internal val startedAt: Instant, + internal val deliveryDuration: Duration, +) { + /** Number of entries processed (0 if the store had nothing to claim). */ + val count: Int get() = entries.size +} diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt new file mode 100644 index 00000000..b231ffd3 --- /dev/null +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt @@ -0,0 +1,116 @@ +package com.softwaremill.okapi.core + +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldBeEmpty +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +private val info = object : DeliveryInfo { + override val type = "stub" + override fun serialize() = "{}" +} + +/** Hands out one fresh PENDING entry per claim. */ +private class OneEntryPerClaimStore : RouteAwareOutboxStore { + override fun persist(entry: OutboxEntry) = entry + override fun claimPending(limit: Int): List = error("not used") + override fun claimPending(deliveryTypes: Set, limit: Int): List = + listOf(OutboxEntry.createPending(OutboxMessage("e", "{}"), info, Instant.EPOCH)) + override fun updateAfterProcessing(entry: OutboxEntry) = entry + override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 + override fun findOldestCreatedAt(statuses: Set) = emptyMap() + override fun countByStatuses() = emptyMap() +} + +private class RecordingListener(private val log: MutableList, private val onBatch: () -> Unit = {}) : OutboxProcessorListener { + override fun onEntryProcessed(event: OutboxProcessingEvent) { + log += "entry:${event::class.simpleName}" + } + + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + log += "batch:$processedCount" + onBatch() + } +} + +private fun processor(listener: OutboxProcessorListener) = OutboxProcessor( + store = OneEntryPerClaimStore(), + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = "stub" + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 1), + Clock.systemUTC(), + ), + listener = listener, +) + +/** Issue #137: listener events describe committed state, so the scheduler emits them after the commit. */ +class ListenerAfterCommitTest : FunSpec({ + test("processBatch persists without notifying; publishEvents notifies") { + val log = mutableListOf() + val processor = processor(RecordingListener(log)) + + val batch = processor.processBatch(10) + batch.count shouldBe 1 + log.shouldBeEmpty() + + processor.publishEvents(batch) + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } + + test("processNext still notifies at once") { + val log = mutableListOf() + processor(RecordingListener(log)).processNext(10) shouldBe 1 + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } + + test("the scheduler emits events only after the transaction has committed") { + val log = Collections.synchronizedList(mutableListOf()) + val batchSeen = CountDownLatch(1) + val committingRunner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = block().also { log += "commit" } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log) { batchSeen.countDown() }), + committingRunner, + OutboxSchedulerConfig(interval = Duration.ofMinutes(1)), + ) + + scheduler.start() + batchSeen.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.take(3) shouldContainExactly listOf("commit", "entry:Delivered", "batch:1") + } + + test("a batch whose commit fails emits no events") { + val log = Collections.synchronizedList(mutableListOf()) + val ticks = CountDownLatch(2) + val failingCommit = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + block() + ticks.countDown() + throw IllegalStateException("commit failed: connection reset") + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log)), + failingCommit, + OutboxSchedulerConfig(interval = Duration.ofMillis(20)), + ) + + scheduler.start() + ticks.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.shouldBeEmpty() + } +}) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt new file mode 100644 index 00000000..cbf46880 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt @@ -0,0 +1,84 @@ +package com.softwaremill.okapi.test.transaction + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessingEvent +import com.softwaremill.okapi.core.OutboxProcessor +import com.softwaremill.okapi.core.OutboxProcessorListener +import com.softwaremill.okapi.core.OutboxScheduler +import com.softwaremill.okapi.core.OutboxSchedulerConfig +import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.postgres.PostgresOutboxStore +import com.softwaremill.okapi.test.support.PostgresTestSupport +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.util.concurrent.CompletableFuture +import java.util.concurrent.TimeUnit + +private object AfterCommitDeliveryInfo : DeliveryInfo { + override val type: String = "after-commit" + override fun serialize(): String = """{"type":"after-commit"}""" +} + +/** + * Issue #137 against a real database: when the scheduler's listener hears that an entry was + * delivered, another connection must already see it as DELIVERED, i.e. the transaction committed. + */ +class ListenerAfterCommitEndToEndTest : FunSpec({ + val db = PostgresTestSupport() + + beforeSpec { db.start() } + afterSpec { db.stop() } + + test("the scheduler's listener sees the delivered status already committed") { + val store = PostgresOutboxStore(db.jdbc) + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), AfterCommitDeliveryInfo, Clock.systemUTC().instant()) + db.jdbc.withTransaction { store.persist(entry) } + + val statusSeenByListener = CompletableFuture() + val listener = object : OutboxProcessorListener { + override fun onEntryProcessed(event: OutboxProcessingEvent) { + // A connection of its own: it only sees committed data. + db.dataSource.connection.use { conn -> + conn.prepareStatement("SELECT status FROM okapi_outbox WHERE id = ?::uuid").use { stmt -> + stmt.setString(1, entry.outboxId.raw.toString()) + stmt.executeQuery().use { rs -> + rs.next() + statusSeenByListener.complete(rs.getString("status")) + } + } + } + } + } + val processor = OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = AfterCommitDeliveryInfo.type + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 0), + Clock.systemUTC(), + ), + listener = listener, + ) + val runner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = db.jdbc.withTransaction(block) + } + val scheduler = OutboxScheduler(processor, runner, OutboxSchedulerConfig(interval = Duration.ofMinutes(1))) + + scheduler.start() + try { + statusSeenByListener.get(10, TimeUnit.SECONDS) shouldBe "DELIVERED" + } finally { + scheduler.stop() + } + } +}) From d894d9bcb348a63c9172a91603780ae5ea4fcff5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Fri, 6 Nov 2026 11:05:28 +0100 Subject: [PATCH 2/3] fix(core): address review of after-commit listener events (#137) - TransactionRunner documents its contract: runInTransaction has committed when it returns, and may run the block more than once (Exposed retries). - publishEvents reads the batch end time before the callbacks and is one-shot per batch; ProcessedBatch exposes a read-only entry list and a toString. - Fix stale processNext references. - Tests: the end-to-end test runs with SpringTransactionRunner and ExposedTransactionRunner as well as plain JDBC and fails fast if the row is missing; unit tests for a retrying runner, a failing processBatch and double publishing. - Docs: counters count committed status changes; batch timer wording; how a listener that relied on the processing transaction now fails. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 5 +- README.md | 4 +- UPGRADING.md | 8 ++- .../okapi/core/CompositeMessageDeliverer.kt | 2 +- .../okapi/core/OutboxProcessor.kt | 7 ++- .../softwaremill/okapi/core/ProcessedBatch.kt | 17 ++++- .../okapi/core/TransactionRunner.kt | 7 +++ .../okapi/core/ListenerAfterCommitTest.kt | 62 ++++++++++++++++++- .../okapi/core/OutboxSchedulerTest.kt | 2 +- .../ListenerAfterCommitEndToEndTest.kt | 59 +++++++++++++----- 10 files changed, 145 insertions(+), 28 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4434cde4..7a71d607 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -133,7 +133,10 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * listener runs after the commit, outside the transaction, and not at all for a batch whose commit failed. A listener that relied on running inside the processing transaction (e.g. to write to the database atomically with the status update) no longer can. `onBatchProcessed`'s duration, and with - it `okapi.batch.duration`, now includes the commit; empty polls are still reported. + it `okapi.batch.duration`, now includes the commit (and excludes the listener callbacks); empty + polls are still reported. Counters now count committed status changes rather than delivery + attempts. `TransactionRunner` now documents that `runInTransaction` must have committed when it + returns, and `publishEvents` throws if a batch is published twice. `processNext(limit)` keeps notifying at once, for callers that drive the processor themselves. ([#137](https://github.com/softwaremill/okapi/issues/137)) - **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the diff --git a/README.md b/README.md index e283d9da..532e0eda 100644 --- a/README.md +++ b/README.md @@ -279,7 +279,7 @@ Metrics are then available at `/actuator/prometheus`. | `okapi.entries.delivered` | Counter | Successfully delivered entries | | `okapi.entries.retry.scheduled` | Counter | Failed attempts rescheduled for retry | | `okapi.entries.failed` | Counter | Permanently failed entries | -| `okapi.batch.duration` | Timer | Processing time per batch, from the claim until the transaction has committed | +| `okapi.batch.duration` | Timer | Processing time per batch; under the scheduler from the claim until the transaction has committed (not including listener callbacks) | | `okapi.entries.count` | Gauge | Current entry count (tag: `status`) | | `okapi.entries.lag.seconds` | Gauge | Age of the oldest entry (tag: `status`) | @@ -297,7 +297,7 @@ max by (job, status) (okapi_entries_count) Without Spring Boot, construct `MicrometerOutboxListener` and `MicrometerOutboxMetrics` with your `MeterRegistry`. Refresh gauges with `OutboxMetricsRefresher` or your own scheduler. -For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. If you drive `OutboxProcessor` yourself, call `processBatch` inside your transaction and `publishEvents` after it commits; `processNext` does both at once, so inside a transaction its events precede the commit. +For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. Counters therefore count committed status changes, not delivery attempts: a send whose transaction then rolled back (a failed commit, or a block Exposed retried) is not counted, though the message may have reached the consumer. If you drive `OutboxProcessor` yourself, call `processBatch` inside your transaction and `publishEvents` after it commits; `processNext` does both at once, so inside a transaction its events precede the commit. ## Modules diff --git a/UPGRADING.md b/UPGRADING.md index 9bdb800d..ef1315bd 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -112,7 +112,13 @@ the processing transaction has committed, outside of it, and are skipped for a b failed. - A custom listener that wrote to the database expecting to join the processing transaction now - runs without one: open your own transaction in the callback, or make the write idempotent. + runs without one: open your own transaction in the callback, or make the write idempotent. This + fails quietly: an Exposed listener gets "No transaction in context", a Spring call with + `MANDATORY` propagation throws, and okapi only logs each such exception as a WARN. +- Counters (`okapi.entries.delivered`, `okapi.entries.failed`, ...) now count committed status + changes. A send whose transaction rolled back is no longer counted. +- A custom `TransactionRunner` must have committed when `runInTransaction` returns (now documented on + the interface); one that joins an outer transaction brings back pre-commit events. - `okapi.batch.duration` now includes the commit, so it reads slightly higher than before. - If you call `OutboxProcessor.processNext` inside your own transaction, events still fire before your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt index b62bc0f3..b062bd66 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/CompositeMessageDeliverer.kt @@ -59,7 +59,7 @@ class CompositeMessageDeliverer @JvmOverloads constructor( * satisfy that opt out via [TransportDispatch.SEQUENTIAL]. * * The outbox transaction is open across delivery, and stays that way: the scheduler wraps the - * whole [OutboxProcessor.processNext] cycle — claim, deliver, update — in one transaction, so + * whole [OutboxProcessor.processBatch] cycle — claim, deliver, update — in one transaction, so * that `FOR UPDATE SKIP LOCKED` holds its row locks until the results are written. What this * dispatch changes is which thread the transport runs on, not that scope. The store round-trips * still run on the calling thread inside the transaction, but a group handed to a virtual diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index 6cea89e3..4a11b0da 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -91,11 +91,14 @@ class OutboxProcessor @JvmOverloads constructor( /** * Notifies the listener of [batch]: [OutboxProcessorListener.onEntryProcessed] for each entry, * then [OutboxProcessorListener.onBatchProcessed], whose duration runs from the start of the batch - * until now, so it includes the commit when called after one. Call it at most once per batch. + * until this call, so it includes the commit when called after one, but not the listener + * callbacks. A batch's events can be published only once; a second call throws. */ fun publishEvents(batch: ProcessedBatch) { + check(batch.markPublished()) { "Events of this batch were already published" } + val batchDuration = Duration.between(batch.startedAt, clock.instant()) batch.entries.forEach { updated -> notifyEntry(updated, batch.deliveryDuration) } - notifyBatch(batch.count, Duration.between(batch.startedAt, clock.instant())) + notifyBatch(batch.count, batchDuration) } private fun notifyEntry(updated: OutboxEntry, duration: Duration) { diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt index 9a71f47c..f07eff5b 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/ProcessedBatch.kt @@ -2,16 +2,29 @@ package com.softwaremill.okapi.core import java.time.Duration import java.time.Instant +import java.util.Collections +import java.util.concurrent.atomic.AtomicBoolean /** * The outcome of [OutboxProcessor.processBatch]: [entries] in the state just persisted, whose - * listener events are held back until [OutboxProcessor.publishEvents]. + * listener events are held back until [OutboxProcessor.publishEvents]. Only [OutboxProcessor] + * creates one; its constructor and timing fields are internal to okapi-core. */ class ProcessedBatch internal constructor( - val entries: List, + entries: List, internal val startedAt: Instant, internal val deliveryDuration: Duration, ) { + /** The processed entries, in the state persisted for them. Read-only. */ + val entries: List = Collections.unmodifiableList(entries.toList()) + /** Number of entries processed (0 if the store had nothing to claim). */ val count: Int get() = entries.size + + private val published = AtomicBoolean(false) + + /** Returns `true` the first time only. */ + internal fun markPublished(): Boolean = published.compareAndSet(false, true) + + override fun toString(): String = "ProcessedBatch(count=$count, startedAt=$startedAt, deliveryDuration=$deliveryDuration)" } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt index bc17a929..fa852769 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/TransactionRunner.kt @@ -7,6 +7,13 @@ package com.softwaremill.okapi.core * - `okapi-spring-boot`: wraps Spring's `TransactionTemplate` via `SpringTransactionRunner` * - `okapi-exposed`: wraps Exposed's `transaction {}` via `ExposedTransactionRunner` * - Standalone: user-provided lambda + * + * Contract: when [runInTransaction] returns normally, the transaction it ran [block] in has + * committed. [OutboxScheduler] relies on this to emit listener events only for committed state + * (issue #137). A runner that joins an outer transaction instead (e.g. Spring's `REQUIRED` called + * inside `@Transactional`, or a nested Exposed `transaction {}` without nested transactions) breaks + * that. A runner may run [block] more than once, as Exposed does when a statement or the commit + * throws an `SQLException`; only the result of the attempt that committed is returned. */ interface TransactionRunner { fun runInTransaction(block: () -> T): T diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt index b231ffd3..e1d153b6 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt @@ -1,5 +1,6 @@ package com.softwaremill.okapi.core +import io.kotest.assertions.throwables.shouldThrow import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.collections.shouldBeEmpty import io.kotest.matchers.collections.shouldContainExactly @@ -39,8 +40,8 @@ private class RecordingListener(private val log: MutableList, private va } } -private fun processor(listener: OutboxProcessorListener) = OutboxProcessor( - store = OneEntryPerClaimStore(), +private fun processor(listener: OutboxProcessorListener, store: RouteAwareOutboxStore = OneEntryPerClaimStore()) = OutboxProcessor( + store = store, entryProcessor = OutboxEntryProcessor( object : MessageDeliverer { override val type = "stub" @@ -113,4 +114,61 @@ class ListenerAfterCommitTest : FunSpec({ log.shouldBeEmpty() } + + test("a runner that retries the block, as Exposed does on SQLException, publishes one set of events") { + val log = Collections.synchronizedList(mutableListOf()) + val batchSeen = CountDownLatch(1) + val retryingRunner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + block() // first attempt: its commit fails and it is rolled back + log += "rollback" + return block().also { log += "commit" } + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log) { batchSeen.countDown() }), + retryingRunner, + OutboxSchedulerConfig(interval = Duration.ofMinutes(1)), + ) + + scheduler.start() + batchSeen.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.take(4) shouldContainExactly listOf("rollback", "commit", "entry:Delivered", "batch:1") + } + + test("a processBatch that throws publishes nothing") { + val log = Collections.synchronizedList(mutableListOf()) + val ticks = CountDownLatch(2) + val failingStore = object : RouteAwareOutboxStore by OneEntryPerClaimStore() { + override fun updateAfterProcessingBatch(entries: List): List { + ticks.countDown() + throw IllegalStateException("update failed") + } + } + val scheduler = OutboxScheduler( + processor(RecordingListener(log), failingStore), + object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = block() + }, + OutboxSchedulerConfig(interval = Duration.ofMillis(20)), + ) + + scheduler.start() + ticks.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + log.shouldBeEmpty() + } + + test("a batch's events can be published only once") { + val log = mutableListOf() + val processor = processor(RecordingListener(log)) + val batch = processor.processBatch(10) + + processor.publishEvents(batch) + shouldThrow { processor.publishEvents(batch) } + log shouldContainExactly listOf("entry:Delivered", "batch:1") + } }) diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt index b2f88a66..4efc1307 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt @@ -24,7 +24,7 @@ import java.util.concurrent.atomic.AtomicReference class OutboxSchedulerTest : FunSpec({ - test("tick calls processNext with configured batchSize") { + test("tick calls processBatch with configured batchSize") { var capturedLimit: Int? = null val latch = CountDownLatch(1) val processor = stubProcessor { limit -> diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt index cbf46880..d9e3142d 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/ListenerAfterCommitEndToEndTest.kt @@ -11,12 +11,20 @@ import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxProcessorListener import com.softwaremill.okapi.core.OutboxScheduler import com.softwaremill.okapi.core.OutboxSchedulerConfig +import com.softwaremill.okapi.core.OutboxStore import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.exposed.ExposedConnectionProvider +import com.softwaremill.okapi.exposed.ExposedTransactionRunner import com.softwaremill.okapi.postgres.PostgresOutboxStore +import com.softwaremill.okapi.springboot.SpringConnectionProvider +import com.softwaremill.okapi.springboot.SpringTransactionRunner import com.softwaremill.okapi.test.support.PostgresTestSupport import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.shouldBe +import org.jetbrains.exposed.v1.jdbc.Database +import org.springframework.jdbc.datasource.DataSourceTransactionManager +import org.springframework.transaction.support.TransactionTemplate import java.time.Clock import java.time.Duration import java.util.concurrent.CompletableFuture @@ -37,23 +45,28 @@ class ListenerAfterCommitEndToEndTest : FunSpec({ beforeSpec { db.start() } afterSpec { db.stop() } - test("the scheduler's listener sees the delivered status already committed") { - val store = PostgresOutboxStore(db.jdbc) + /** Runs one scheduler tick with [runner] and [store]; returns the status the listener saw from another connection. */ + fun statusSeenByListener(runner: TransactionRunner, store: OutboxStore): String { + db.truncate() val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), AfterCommitDeliveryInfo, Clock.systemUTC().instant()) - db.jdbc.withTransaction { store.persist(entry) } + db.jdbc.withTransaction { PostgresOutboxStore(db.jdbc).persist(entry) } - val statusSeenByListener = CompletableFuture() + val seen = CompletableFuture() val listener = object : OutboxProcessorListener { override fun onEntryProcessed(event: OutboxProcessingEvent) { - // A connection of its own: it only sees committed data. - db.dataSource.connection.use { conn -> - conn.prepareStatement("SELECT status FROM okapi_outbox WHERE id = ?::uuid").use { stmt -> - stmt.setString(1, entry.outboxId.raw.toString()) - stmt.executeQuery().use { rs -> - rs.next() - statusSeenByListener.complete(rs.getString("status")) + try { + // A connection of its own: it only sees committed data. + db.dataSource.connection.use { conn -> + conn.prepareStatement("SELECT status FROM okapi_outbox WHERE id = ?::uuid").use { stmt -> + stmt.setString(1, entry.outboxId.raw.toString()) + stmt.executeQuery().use { rs -> + check(rs.next()) { "row ${entry.outboxId} not found" } + seen.complete(rs.getString("status")) + } } } + } catch (e: Exception) { + seen.completeExceptionally(e) } } } @@ -69,16 +82,30 @@ class ListenerAfterCommitEndToEndTest : FunSpec({ ), listener = listener, ) - val runner = object : TransactionRunner { - override fun runInTransaction(block: () -> T): T = db.jdbc.withTransaction(block) - } val scheduler = OutboxScheduler(processor, runner, OutboxSchedulerConfig(interval = Duration.ofMinutes(1))) - scheduler.start() try { - statusSeenByListener.get(10, TimeUnit.SECONDS) shouldBe "DELIVERED" + return seen.get(10, TimeUnit.SECONDS) } finally { scheduler.stop() } } + + test("plain JDBC runner: the listener sees the delivered status already committed") { + val runner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = db.jdbc.withTransaction(block) + } + statusSeenByListener(runner, PostgresOutboxStore(db.jdbc)) shouldBe "DELIVERED" + } + + test("SpringTransactionRunner: the listener sees the delivered status already committed") { + val runner = SpringTransactionRunner(TransactionTemplate(DataSourceTransactionManager(db.dataSource))) + statusSeenByListener(runner, PostgresOutboxStore(SpringConnectionProvider(db.dataSource))) shouldBe "DELIVERED" + } + + test("ExposedTransactionRunner: the listener sees the delivered status already committed") { + val database = Database.connect(db.dataSource) + statusSeenByListener(ExposedTransactionRunner(database), PostgresOutboxStore(ExposedConnectionProvider(database))) shouldBe + "DELIVERED" + } }) From 235f53f29a75f8f0de4d3388511544b752d9d459 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Mon, 9 Nov 2026 13:11:13 +0100 Subject: [PATCH 3/3] test(core): adapt ListenerAfterCommitTest store to claimPending(..., now) from #131 Co-Authored-By: Claude Opus 5.5 (1M context) --- .../com/softwaremill/okapi/core/ListenerAfterCommitTest.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt index e1d153b6..aa8cc274 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/ListenerAfterCommitTest.kt @@ -21,7 +21,7 @@ private val info = object : DeliveryInfo { private class OneEntryPerClaimStore : RouteAwareOutboxStore { override fun persist(entry: OutboxEntry) = entry override fun claimPending(limit: Int): List = error("not used") - override fun claimPending(deliveryTypes: Set, limit: Int): List = + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = listOf(OutboxEntry.createPending(OutboxMessage("e", "{}"), info, Instant.EPOCH)) override fun updateAfterProcessing(entry: OutboxEntry) = entry override fun removeDeliveredBefore(time: Instant, limit: Int) = 0