From 5ef0f646bc066502fc36f553798f29ae261b8e16 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Tue, 10 Nov 2026 10:36:50 +0100 Subject: [PATCH 1/2] fix(core)!: drain the backlog within a scheduler tick instead of one batch per worker (#132) Each worker processed exactly one batch per tick, capping throughput at about concurrency x batchSize / interval (under 10 msg/s with the defaults), so any higher sustained publish rate grew the backlog without bound. A worker now keeps processing batches, each in its own transaction, while the previous one was full and made progress (at least one entry DELIVERED or FAILED), up to OutboxSchedulerConfig.maxBatchesPerTick (default 1000, Spring property okapi.processor.max-batches-per-tick). A batch whose entries were all rescheduled for retry ends the tick, so a failing downstream is not sent the whole backlog at once; stop() lets the batch in progress finish and starts no further one. maxBatchesPerTick = 1 restores the 1.x behaviour. BREAKING CHANGE: the poll interval no longer limits throughput; downstream systems receive a backlog as fast as okapi can deliver it. OutboxSchedulerConfig and OutboxProcessorProperties gained a constructor parameter. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 14 + README.md | 9 +- UPGRADING.md | 18 ++ .../okapi/core/OutboxScheduler.kt | 53 +++- .../okapi/core/OutboxSchedulerConfig.kt | 11 + .../okapi/core/OutboxSchedulerConfigTest.kt | 7 + .../okapi/core/OutboxSchedulerDrainTest.kt | 260 ++++++++++++++++++ .../test/e2e/DrainBacklogEndToEndTest.kt | 107 +++++++ .../springboot/OutboxAutoConfiguration.kt | 1 + .../springboot/OutboxProcessorProperties.kt | 7 + .../spring-configuration-metadata.json | 10 +- .../OutboxProcessorAutoConfigurationTest.kt | 49 ++++ 12 files changed, 528 insertions(+), 18 deletions(-) create mode 100644 okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt create mode 100644 okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a71d607..50067a5d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,20 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * ### Changed (BREAKING) +- **The scheduler drains the backlog within a tick instead of processing one batch per worker.** + Each worker used to process exactly one batch per tick, capping throughput at about + `concurrency × batchSize / interval`, under 10 messages/s with the defaults, so any sustained + higher publish rate grew the backlog without bound. A worker now keeps processing batches, each in + its own transaction, while the previous one was full and at least one of its entries was delivered + or failed for good, up to the new `OutboxSchedulerConfig.maxBatchesPerTick` (default 1000, + Spring property `okapi.processor.max-batches-per-tick`). A batch whose entries were all rescheduled + for retry ends the tick, so a failing downstream is not sent the whole backlog at once. The poll + interval now only sets the idle polling rate: downstream systems receive a backlog as fast as okapi + can deliver it, and `onBatchProcessed` / `okapi.batch.duration` fire more often per tick. + `maxBatchesPerTick = 1` restores the previous behaviour. `OutboxSchedulerConfig` and + `OutboxProcessorProperties` gained a constructor parameter (`@JvmOverloads` keeps the previous Java + constructors; Kotlin callers of `copy(...)` must recompile). + ([#132](https://github.com/softwaremill/okapi/issues/132)) - **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).** A retriable failure used to be retried on the very next tick, so with the defaults (`interval` 1s, `max-retries` 5) a message was `FAILED` — terminally — after about six seconds of downstream diff --git a/README.md b/README.md index 532e0eda..72ddb176 100644 --- a/README.md +++ b/README.md @@ -132,8 +132,9 @@ In a typical single-DataSource application, all properties are optional. Multi-D | Property | Default | Description | |---|---|---| | `okapi.processor.enabled` | `true` | Set `false` to disable delivery entirely (e.g. on instances that only publish). | -| `okapi.processor.interval` | `1s` | How often the scheduler polls for pending entries. | -| `okapi.processor.batch-size` | `10` | Maximum entries claimed per worker per tick. | +| `okapi.processor.interval` | `1s` | Delay between scheduler ticks, i.e. how often an idle outbox is polled. A tick with a backlog keeps going (see `max-batches-per-tick`), so this does not cap throughput. | +| `okapi.processor.batch-size` | `10` | Maximum entries claimed per batch; each batch runs in its own transaction. | +| `okapi.processor.max-batches-per-tick` | `1000` | Batches each worker processes per tick. A worker continues while the previous batch was full and at least one of its entries was delivered or failed for good; a batch that was entirely rescheduled for retry ends the tick. `1` restores the 1.x one batch per tick, which caps throughput at about `concurrency × batch-size / interval`. | | `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. | | `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. | | `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). | @@ -238,7 +239,7 @@ okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liqui ## Transactions -Each processor worker runs its claim, delivery and state update inside one transaction, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed. +Each processor worker runs its claim, delivery and state update inside one transaction per batch, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. While a backlog remains, a worker runs batch after batch within one tick, each in a transaction of its own. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed. **Multiple data sources or transaction managers?** Set both `okapi.datasource-qualifier` and `okapi.transaction-manager-qualifier` to the beans used for the outbox. okapi fails fast when it detects a mismatch. If the selected transaction manager does not expose its `DataSource`, okapi cannot verify the pairing and logs a warning instead. @@ -342,6 +343,8 @@ The storage modules use plain JDBC. With Spring Boot, they participate in the tr Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See [`benchmarks/`](benchmarks/) for methodology and measured results. +Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and `interval` only sets how quickly a new message is picked up when the outbox was idle. To limit the rate at which okapi sends to a downstream system, set `max-batches-per-tick` (with `batch-size` and `interval`) rather than relying on the poll interval alone. + ## Building ```sh diff --git a/UPGRADING.md b/UPGRADING.md index ef1315bd..ab7c1587 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -124,6 +124,24 @@ failed. your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by `processor.publishEvents(batch)` to get the after-commit behaviour. +### The scheduler drains the backlog in each tick (#132) + +A worker no longer stops after one batch per tick: it keeps processing batches, each in its own +transaction, while they come back full and make progress, up to `max-batches-per-tick` (default +1000). + +- **Downstream load.** With a backlog, okapi now sends as fast as the database and transports allow. + If you relied on `batch-size` / `interval` to throttle a rate-limited downstream system, set + `okapi.processor.max-batches-per-tick=1` (`OutboxSchedulerConfig(maxBatchesPerTick = 1)`) to keep + the 1.x rate, or a small value to allow some draining. +- **Database load.** A backlog is now worked through in back-to-back transactions instead of one + per interval. Size `concurrency` with that in mind. +- **Metrics.** `onBatchProcessed` and `okapi.batch.duration` are still per batch, so a tick can now + report many batches. +- **Code.** `OutboxSchedulerConfig` and `OutboxProcessorProperties` gained a `maxBatchesPerTick` + parameter. Java callers keep the previous constructors (`@JvmOverloads`); Kotlin code calling + `copy(...)` must be recompiled. + ## From 0.2.x to 1.0.0 Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt index 3876d602..5a67298f 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt @@ -15,6 +15,14 @@ import java.util.concurrent.atomic.AtomicBoolean * Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and, * once it has committed, [OutboxProcessor.publishEvents]. * + * Each tick drains (issue #132): a worker keeps processing batches, each in its own transaction, + * while the previous one was full and made progress -- at least one entry ended `DELIVERED` or + * `FAILED` rather than being rescheduled -- up to [OutboxSchedulerConfig.maxBatchesPerTick]. + * [OutboxSchedulerConfig.interval] is the delay after a tick, so it sets the polling rate of an + * idle outbox, not peak throughput. A batch whose entries were all rescheduled for retry ends the + * tick, so a failing downstream is not sent the whole backlog in one go. [stop] lets the batch in + * progress finish and starts no further one. + * * Each worker's call runs inside [transactionRunner]. The runner is required: without a * surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the * claim SELECT and concurrent processor instances deliver the same entry multiple times. @@ -23,7 +31,7 @@ import java.util.concurrent.atomic.AtomicBoolean * [OutboxSchedulerConfig.workerExecutorFactory]'s pool. `FOR UPDATE SKIP LOCKED` guarantees * disjoint claims, but [outboxProcessor] (and its dependencies) must be safe for concurrent use. * The tick waits for every worker before the next scheduled interval, so ticks never overlap. - * `concurrency = 1` (default) calls the batch inline, preserving single-worker semantics. + * `concurrency = 1` (default) runs the worker inline, preserving single-worker semantics. * * At `concurrency > 1`, [outboxProcessor] -- and by extension its [OutboxStore], [MessageDeliverer], * and any [OutboxProcessorListener] -- is invoked concurrently from multiple worker threads, so all @@ -110,7 +118,7 @@ class OutboxScheduler @JvmOverloads constructor( if (config.concurrency == 1) { // Skips workers/Lazy entirely -- not just an inline no-op path -- so this configuration // truly pays no executor/synchronization overhead per tick, matching the class KDoc. - processBatch() + drain() return } val pool = try { @@ -130,13 +138,13 @@ class OutboxScheduler @JvmOverloads constructor( } /** - * Submits one worker's [processBatch] without letting [RejectedExecutionException] escape + * Submits one worker's [drain] without letting [RejectedExecutionException] escape * [tick] -- a custom [OutboxSchedulerConfig.workerExecutorFactory] may return an executor that * can reject submissions (bounded queue, shutdown executor), and an uncaught exception here * would make `scheduleWithFixedDelay` suppress all future ticks. */ private fun submitWorker(pool: ExecutorService): Future<*>? = try { - pool.submit(::processBatch) + pool.submit(::drain) } catch (e: RejectedExecutionException) { logger.error("Outbox worker submission rejected, will retry at next scheduled interval", e) null @@ -145,7 +153,7 @@ class OutboxScheduler @JvmOverloads constructor( /** * Awaits one worker's completion without letting anything escape [tick] -- an uncaught * exception here would make `scheduleWithFixedDelay` suppress all future ticks, silently - * killing the scheduler. [processBatch] already catches [Exception] internally, so this + * killing the scheduler. [drain] already catches [Exception] internally, so this * only guards against an [Error] surfacing via [ExecutionException], a [CancellationException] * if a custom [OutboxSchedulerConfig.workerExecutorFactory] cancels queued/running tasks (e.g. * as a backpressure strategy), or an interrupt while awaiting. @@ -178,17 +186,36 @@ class OutboxScheduler @JvmOverloads constructor( } } - private fun processBatch() { - try { - val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } - // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. - outboxProcessor.publishEvents(batch) - logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize) - } catch (e: Exception) { - logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) + /** One worker's share of a tick: processes batches until [shouldContinue] says to stop. */ + private fun drain() { + var batches = 0 + while (true) { + val batch = processBatch() ?: return + batches++ + if (!shouldContinue(batch, batches)) { + logger.debug("Outbox processor tick completed [batches={}, lastBatchSize={}]", batches, batch.count) + return + } } } + private fun shouldContinue(batch: ProcessedBatch, batches: Int): Boolean = batch.count >= config.batchSize && + batches < config.maxBatchesPerTick && + running.get() && + !Thread.currentThread().isInterrupted && + batch.entries.any { it.status != OutboxStatus.PENDING } + + /** Processes one batch in its own transaction; returns `null` if that failed. */ + private fun processBatch(): ProcessedBatch? = try { + val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } + // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. + outboxProcessor.publishEvents(batch) + batch + } catch (e: Exception) { + logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) + null + } + companion object { private val logger = LoggerFactory.getLogger(OutboxScheduler::class.java) private const val SHUTDOWN_TIMEOUT_SECONDS = 5L diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt index 5a9d8125..7f7c780a 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt @@ -24,14 +24,25 @@ data class OutboxSchedulerConfig @JvmOverloads constructor( * switching away from the default. */ val workerExecutorFactory: (Int) -> ExecutorService = ::defaultPlatformPool, + /** + * Upper bound on the batches each worker processes per tick (issue #132). A worker keeps + * claiming, each batch in its own transaction, while the previous batch was full and made + * progress, so [interval] only sets the idle polling delay and a backlog drains as fast as the + * database and transports allow. `1` restores the 1.x behaviour of one batch per worker per + * tick, which caps throughput at about `concurrency × batchSize / interval`. + */ + val maxBatchesPerTick: Int = DEFAULT_MAX_BATCHES_PER_TICK, ) { init { require(!interval.isNegative && interval.toMillis() > 0) { "interval must be at least 1ms, got: $interval" } require(batchSize > 0) { "batchSize must be positive, got: $batchSize" } require(concurrency in 1..256) { "concurrency must be between 1 and 256, got: $concurrency" } + require(maxBatchesPerTick > 0) { "maxBatchesPerTick must be positive, got: $maxBatchesPerTick" } } companion object { + const val DEFAULT_MAX_BATCHES_PER_TICK = 1000 + /** Fixed pool of [n] daemon platform threads, named `outbox-worker-N`. */ @JvmStatic fun defaultPlatformPool(n: Int): ExecutorService { diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt index 4bb2c998..68218a3a 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt @@ -17,6 +17,13 @@ class OutboxSchedulerConfigTest : FunSpec({ config.interval shouldBe ofSeconds(1) config.batchSize shouldBe 10 config.concurrency shouldBe 1 + config.maxBatchesPerTick shouldBe OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK + } + + test("rejects non-positive maxBatchesPerTick") { + listOf(0, -1).forEach { value -> + shouldThrow { OutboxSchedulerConfig(maxBatchesPerTick = value) } + } } test("accepts custom valid values") { diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt new file mode 100644 index 00000000..54015e68 --- /dev/null +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt @@ -0,0 +1,260 @@ +package com.softwaremill.okapi.core + +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.ints.shouldBeLessThan +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +private val drainInfo = object : DeliveryInfo { + override val type = "stub" + override fun serialize() = "{}" +} + +/** In-memory store that claims due PENDING entries in order, as the real stores do. */ +private class BacklogStore(size: Int) : RouteAwareOutboxStore { + private val rows = LinkedHashMap() + + init { + repeat(size) { + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), drainInfo, Instant.EPOCH) + rows[entry.outboxId] = entry + } + } + + fun count(status: OutboxStatus): Int = synchronized(this) { rows.values.count { it.status == status } } + + override fun persist(entry: OutboxEntry) = synchronized(this) { entry.also { rows[it.outboxId] = it } } + override fun claimPending(limit: Int): List = error("not used") + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = synchronized(this) { + rows.values.filter { it.status == OutboxStatus.PENDING && it.deliveryType in deliveryTypes && !it.nextAttemptAt.isAfter(now) } + .take(limit) + // Stands in for FOR UPDATE SKIP LOCKED: another worker must not claim these until they are updated. + .onEach { rows[it.outboxId] = it.copy(status = OutboxStatus.FAILED, lastError = "claimed") } + } + override fun updateAfterProcessing(entry: OutboxEntry) = persist(entry) + override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 + override fun findOldestCreatedAt(statuses: Set) = emptyMap() + override fun countByStatuses() = emptyMap() +} + +/** Records committed batch sizes; [onBatch] runs after each one. */ +private class BatchRecorder(private val onBatch: (Int) -> Unit = {}) : OutboxProcessorListener { + val batches: MutableList = Collections.synchronizedList(mutableListOf()) + + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + batches += processedCount + onBatch(processedCount) + } +} + +private class CountingRunner : TransactionRunner { + val transactions = AtomicInteger() + override fun runInTransaction(block: () -> T): T = block().also { transactions.incrementAndGet() } +} + +private fun drainProcessor(store: BacklogStore, listener: OutboxProcessorListener, result: DeliveryResult = DeliveryResult.Success) = + OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = "stub" + override fun deliver(entry: OutboxEntry) = result + }, + RetryPolicy(maxRetries = 5), + Clock.systemUTC(), + ), + listener = listener, + ) + +/** Waits [millis] and asserts nothing more was processed, for checks that a tick stopped early. */ +private fun assertQuiet(recorder: BatchRecorder, millis: Long = 300) { + val before = recorder.batches.toList() + Thread.sleep(millis) + recorder.batches.toList() shouldBe before +} + +/** Issue #132: a tick drains the backlog instead of processing one batch per worker. */ +class OutboxSchedulerDrainTest : FunSpec({ + test("one tick drains the whole backlog, each batch in its own transaction") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val runner = CountingRunner() + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + runner, + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + ) + + scheduler.start() + drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 35 + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + runner.transactions.get() shouldBe 4 + } + + test("an exactly full last batch is followed by one empty claim, then the tick ends") { + val store = BacklogStore(20) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it == 0) drained.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + ) + + scheduler.start() + drained.await(5, TimeUnit.SECONDS) shouldBe true + assertQuiet(recorder) + scheduler.stop() + + recorder.batches shouldContainExactly listOf(10, 10, 0) + } + + test("maxBatchesPerTick = 1 keeps the 1.x one batch per tick") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, maxBatchesPerTick = 1), + ) + + scheduler.start() + firstBatch.await(5, TimeUnit.SECONDS) shouldBe true + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 10 + recorder.batches shouldContainExactly listOf(10) + } + + test("the tick stops at maxBatchesPerTick") { + val store = BacklogStore(100) + val third = CountDownLatch(3) + val recorder = BatchRecorder { third.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, maxBatchesPerTick = 3), + ) + + scheduler.start() + third.await(5, TimeUnit.SECONDS) shouldBe true + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 30 + } + + test("a full batch whose entries were all rescheduled for retry ends the tick") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder, DeliveryResult.RetriableFailure("503")), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + ) + + scheduler.start() + firstBatch.await(5, TimeUnit.SECONDS) shouldBe true + assertQuiet(recorder) + scheduler.stop() + + // The downstream is failing: only one batch is sent to it, not the whole backlog. + recorder.batches shouldContainExactly listOf(10) + store.count(OutboxStatus.PENDING) shouldBe 35 + } + + test("permanent failures are progress, so the tick keeps draining") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder, DeliveryResult.PermanentFailure("400")), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + ) + + scheduler.start() + drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + store.count(OutboxStatus.FAILED) shouldBe 35 + } + + test("a failed transaction ends the tick; the next tick resumes") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val attempts = AtomicInteger() + val failSecond = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + if (attempts.incrementAndGet() == 2) throw IllegalStateException("connection reset") + return block() + } + } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + failSecond, + OutboxSchedulerConfig(interval = Duration.ofMillis(200), batchSize = 10), + ) + + scheduler.start() + drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + // Tick 1: one batch, then the failure. Tick 2 drains the rest. + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + attempts.get() shouldBe 5 + } + + test("concurrent workers each drain until the backlog is empty") { + val store = BacklogStore(200) + val recorder = BatchRecorder() + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, concurrency = 4), + ) + + scheduler.start() + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) + while (store.count(OutboxStatus.DELIVERED) < 200 && System.nanoTime() < deadline) Thread.sleep(10) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 200 + recorder.batches.sum() shouldBe 200 + } + + test("stop() during a drain finishes the batch in progress and starts no further one") { + val store = BacklogStore(100_000) + val fewBatches = CountDownLatch(3) + val recorder = BatchRecorder { fewBatches.countDown() } + val scheduler = OutboxScheduler( + drainProcessor(store, recorder), + CountingRunner(), + OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + ) + + scheduler.start() + fewBatches.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + val processed = recorder.batches.sum() + store.count(OutboxStatus.DELIVERED) shouldBe processed + processed shouldBeLessThan 10 * OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK + assertQuiet(recorder) + } +}) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt new file mode 100644 index 00000000..57699191 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt @@ -0,0 +1,107 @@ +package com.softwaremill.okapi.test.e2e + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessor +import com.softwaremill.okapi.core.OutboxProcessorListener +import com.softwaremill.okapi.core.OutboxScheduler +import com.softwaremill.okapi.core.OutboxSchedulerConfig +import com.softwaremill.okapi.core.OutboxStatus +import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import com.softwaremill.okapi.postgres.PostgresOutboxStore +import com.softwaremill.okapi.test.support.JdbcConnectionProvider +import com.softwaremill.okapi.test.support.MysqlTestSupport +import com.softwaremill.okapi.test.support.PostgresTestSupport +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.shouldBe +import java.sql.Connection +import java.time.Clock +import java.time.Duration +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +private object DrainDeliveryInfo : DeliveryInfo { + override val type: String = "drain" + override fun serialize(): String = """{"type":"drain"}""" +} + +private const val BACKLOG = 250 +private const val BATCH_SIZE = 10 + +/** + * Issue #132 against real databases: with an interval far longer than the test, a single tick + * delivers the whole backlog, one transaction per batch. + */ +class DrainBacklogEndToEndTest : FunSpec({ + val postgres = PostgresTestSupport() + val mysql = MysqlTestSupport() + + beforeSpec { + postgres.start() + mysql.start() + } + afterSpec { + postgres.stop() + mysql.stop() + } + + fun drainsInOneTick(jdbc: JdbcConnectionProvider, store: RouteAwareOutboxStore, isolation: Int?) { + val now = Clock.systemUTC().instant() + jdbc.withTransaction { + repeat(BACKLOG) { store.persist(OutboxEntry.createPending(OutboxMessage("e", "{}"), DrainDeliveryInfo, now)) } + } + + val batches = Collections.synchronizedList(mutableListOf()) + val drained = CountDownLatch(1) + val listener = object : OutboxProcessorListener { + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + batches += processedCount + if (processedCount < BATCH_SIZE) drained.countDown() + } + } + val processor = OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = DrainDeliveryInfo.type + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 0), + Clock.systemUTC(), + ), + listener = listener, + ) + val runner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = jdbc.withTransaction(isolation, block) + } + val scheduler = OutboxScheduler(processor, runner, OutboxSchedulerConfig(interval = Duration.ofHours(1), batchSize = BATCH_SIZE)) + + scheduler.start() + drained.await(30, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + jdbc.withTransaction { store.countByStatuses() }[OutboxStatus.DELIVERED] shouldBe BACKLOG.toLong() + // 25 full batches, then an empty claim ends the tick. + batches shouldContainExactly List(BACKLOG / BATCH_SIZE) { BATCH_SIZE } + 0 + } + + test("Postgres: one tick drains a backlog of $BACKLOG entries") { + postgres.truncate() + drainsInOneTick(postgres.jdbc, PostgresOutboxStore(postgres.jdbc), isolation = null) + } + + test("MySQL: one tick drains a backlog of $BACKLOG entries") { + mysql.truncate() + // The scheduler's isolation on MySQL (issue #129). + drainsInOneTick(mysql.jdbc, MysqlOutboxStore(mysql.jdbc), Connection.TRANSACTION_READ_COMMITTED) + } +}) diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index 0ffa57a4..3fa1b492 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -234,6 +234,7 @@ class OutboxAutoConfiguration( interval = props.interval, batchSize = props.batchSize, concurrency = props.concurrency, + maxBatchesPerTick = props.maxBatchesPerTick, ), ) } diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt index 394c00eb..1ef40522 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt @@ -1,5 +1,6 @@ package com.softwaremill.okapi.springboot +import com.softwaremill.okapi.core.OutboxSchedulerConfig import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.TransportDispatch import org.springframework.boot.context.properties.ConfigurationProperties @@ -31,12 +32,18 @@ data class OutboxProcessorProperties @JvmOverloads constructor( val maxBackoff: Duration = RetryPolicy.DEFAULT_MAX_BACKOFF, /** Fraction (0 to 1) by which each delay is shortened at random, to spread retries out. */ val backoffJitter: Double = RetryPolicy.DEFAULT_JITTER, + /** + * Maximum batches each worker processes per tick while a backlog remains; `1` restores the 1.x + * one batch per tick. See [com.softwaremill.okapi.core.OutboxSchedulerConfig.maxBatchesPerTick]. + */ + val maxBatchesPerTick: Int = OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK, ) { init { require(!interval.isNegative && interval.toMillis() > 0) { "interval must be at least 1ms" } require(batchSize > 0) { "batchSize must be positive" } require(maxRetries >= 0) { "maxRetries must be >= 0" } require(concurrency in 1..256) { "concurrency must be between 1 and 256" } + require(maxBatchesPerTick > 0) { "maxBatchesPerTick must be positive" } // Fail at startup, with the property names, rather than when the RetryPolicy is built. require(!initialBackoff.isNegative) { "initialBackoff must be >= 0" } require(backoffMultiplier >= 1.0 && backoffMultiplier.isFinite()) { "backoffMultiplier must be finite and >= 1" } diff --git a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json index ae1d9ae5..c4af5bf6 100644 --- a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json +++ b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json @@ -61,13 +61,19 @@ "name": "okapi.processor.interval", "type": "java.time.Duration", "defaultValue": "1s", - "description": "How often the processor runs." + "description": "Delay between scheduler ticks: how often an idle outbox is polled. A tick with a backlog keeps processing batches (see max-batches-per-tick), so this does not cap throughput." }, { "name": "okapi.processor.batch-size", "type": "java.lang.Integer", "defaultValue": 10, - "description": "Maximum number of pending entries to process per tick." + "description": "Maximum number of pending entries claimed and processed per batch, each batch in its own transaction." + }, + { + "name": "okapi.processor.max-batches-per-tick", + "type": "java.lang.Integer", + "defaultValue": 1000, + "description": "Maximum number of batches each worker processes per tick. A worker continues while the previous batch was full and at least one of its entries was delivered or failed. 1 restores the 1.x behaviour of one batch per worker per tick." }, { "name": "okapi.processor.max-retries", diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt index 5ac616e8..eeec1361 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt @@ -8,8 +8,10 @@ import com.softwaremill.okapi.core.OutboxEntryProcessor import com.softwaremill.okapi.core.OutboxMessage import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxPublisher +import com.softwaremill.okapi.core.OutboxSchedulerConfig import com.softwaremill.okapi.core.OutboxStore import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore import com.softwaremill.okapi.core.TransactionRunner import com.softwaremill.okapi.core.TransportDispatch import com.softwaremill.okapi.micrometer.MicrometerOutboxListener @@ -36,6 +38,7 @@ import java.time.Duration.ofMinutes import java.time.Duration.ofSeconds import java.time.Instant import java.util.concurrent.ConcurrentLinkedQueue +import java.util.concurrent.atomic.AtomicInteger import javax.sql.DataSource class OutboxProcessorAutoConfigurationTest : FunSpec({ @@ -200,6 +203,52 @@ class OutboxProcessorAutoConfigurationTest : FunSpec({ } } + test("max-batches-per-tick is bound and validated, defaulting to OutboxSchedulerConfig's (#132)") { + contextRunner.withPropertyValues("okapi.processor.max-batches-per-tick=7").run { ctx -> + ctx.getBean().maxBatchesPerTick shouldBe 7 + } + contextRunner.run { ctx -> + ctx.getBean().maxBatchesPerTick shouldBe OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK + } + contextRunner.withPropertyValues("okapi.processor.max-batches-per-tick=0").run { ctx -> + ctx.startupFailure.shouldNotBeNull() + } + } + + test("the scheduler drains up to max-batches-per-tick batches in a tick (#132)") { + listOf(1, 3).forEach { maxBatches -> + val claims = AtomicInteger() + // Every claim comes back full, so only max-batches-per-tick ends the tick. + val alwaysFull = object : RouteAwareOutboxStore by stubStore() { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { + claims.incrementAndGet() + return List(limit) { entryOfType("stub") } + } + + // Delegation would otherwise send the processor's single-type claim to the stub. + override fun claimPending(deliveryType: String, limit: Int, now: Instant) = claimPending(setOf(deliveryType), limit, now) + } + ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withBean(OutboxStore::class.java, { alwaysFull }) + .withBean(MessageDeliverer::class.java, { stubDeliverer() }) + .withBean(DataSource::class.java, { SimpleDriverDataSource() }) + .withBean(TransactionRunner::class.java, { noOpTransactionRunner() }) + .withPropertyValues( + "okapi.processor.interval=1h", + "okapi.processor.batch-size=2", + "okapi.processor.max-batches-per-tick=$maxBatches", + ) + .run { ctx -> + ctx.startupFailure shouldBe null + val deadline = System.nanoTime() + ofSeconds(5).toNanos() + while (claims.get() < maxBatches && System.nanoTime() < deadline) Thread.sleep(10) + Thread.sleep(300) + withClue("max-batches-per-tick=$maxBatches") { claims.get() shouldBe maxBatches } + } + } + } + test("default properties when nothing is configured") { contextRunner.run { ctx -> val props = ctx.getBean() From 0e88e63790ff32e08a1471da2a6ebe925793e9cb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rafa=C5=82=20Wokacz?= Date: Tue, 10 Nov 2026 10:55:16 +0100 Subject: [PATCH 2/2] fix(core): address review of the backlog drain (#132) - Skip, for the rest of a worker's tick, every delivery type with nothing delivered in a batch (all retried or failed for good), so one failing or misconfigured transport gets one batch per tick while healthy types keep draining; end the tick once every type is skipped. - Claim every batch of a tick with the tick's start as "now", so an entry retried during the tick waits for the next one even with a zero backoff. - Log maxBatchesPerTick at startup and say how many batches had committed when a later one fails. - Docs: describe the per-type stop rule; the README concurrency row; the binary break also covers compiled Kotlin calls relying on default arguments. - Tests: two delivery types with one failing, a type that only fails for good, mixed results in a batch, zero backoff, stop() with a batch in flight. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 20 +- README.md | 6 +- UPGRADING.md | 11 +- .../okapi/core/OutboxProcessor.kt | 25 +- .../okapi/core/OutboxScheduler.kt | 48 ++-- .../okapi/core/OutboxSchedulerConfig.kt | 9 +- .../okapi/core/OutboxSchedulerDrainTest.kt | 238 +++++++++++------- .../springboot/OutboxProcessorProperties.kt | 2 +- .../spring-configuration-metadata.json | 2 +- 9 files changed, 226 insertions(+), 135 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 50067a5d..f68768c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -106,15 +106,19 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * Each worker used to process exactly one batch per tick, capping throughput at about `concurrency × batchSize / interval`, under 10 messages/s with the defaults, so any sustained higher publish rate grew the backlog without bound. A worker now keeps processing batches, each in - its own transaction, while the previous one was full and at least one of its entries was delivered - or failed for good, up to the new `OutboxSchedulerConfig.maxBatchesPerTick` (default 1000, - Spring property `okapi.processor.max-batches-per-tick`). A batch whose entries were all rescheduled - for retry ends the tick, so a failing downstream is not sent the whole backlog at once. The poll - interval now only sets the idle polling rate: downstream systems receive a backlog as fast as okapi - can deliver it, and `onBatchProcessed` / `okapi.batch.duration` fire more often per tick. + its own transaction, while the previous one was full, up to the new + `OutboxSchedulerConfig.maxBatchesPerTick` (default 1000, Spring property + `okapi.processor.max-batches-per-tick`). A delivery type with nothing delivered in a batch (all + rescheduled for retry or failed for good) is skipped for the rest of the tick, so a failing or + misconfigured downstream still gets one batch per tick while other types keep draining, and the + tick ends when every type is skipped. Batches of a tick claim only entries due when the tick + started, so a retry waits for the next tick even with a zero backoff. The poll interval now only + sets the idle polling rate: a healthy downstream receives a backlog as fast as okapi can deliver + it, and `onBatchProcessed` / `okapi.batch.duration` fire more often per tick. `maxBatchesPerTick = 1` restores the previous behaviour. `OutboxSchedulerConfig` and - `OutboxProcessorProperties` gained a constructor parameter (`@JvmOverloads` keeps the previous Java - constructors; Kotlin callers of `copy(...)` must recompile). + `OutboxProcessorProperties` gained a constructor parameter: `@JvmOverloads` keeps the previous + Java constructors, but compiled Kotlin code that calls `copy(...)` or a constructor relying on + default arguments fails with `NoSuchMethodError` until recompiled. ([#132](https://github.com/softwaremill/okapi/issues/132)) - **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).** A retriable failure used to be retried on the very next tick, so with the defaults (`interval` diff --git a/README.md b/README.md index 72ddb176..a09be970 100644 --- a/README.md +++ b/README.md @@ -134,13 +134,13 @@ In a typical single-DataSource application, all properties are optional. Multi-D | `okapi.processor.enabled` | `true` | Set `false` to disable delivery entirely (e.g. on instances that only publish). | | `okapi.processor.interval` | `1s` | Delay between scheduler ticks, i.e. how often an idle outbox is polled. A tick with a backlog keeps going (see `max-batches-per-tick`), so this does not cap throughput. | | `okapi.processor.batch-size` | `10` | Maximum entries claimed per batch; each batch runs in its own transaction. | -| `okapi.processor.max-batches-per-tick` | `1000` | Batches each worker processes per tick. A worker continues while the previous batch was full and at least one of its entries was delivered or failed for good; a batch that was entirely rescheduled for retry ends the tick. `1` restores the 1.x one batch per tick, which caps throughput at about `concurrency × batch-size / interval`. | +| `okapi.processor.max-batches-per-tick` | `1000` | Batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. Batches of a tick claim only entries due when the tick started, so a retry waits for the next tick. `1` restores the 1.x one batch per tick, which caps throughput at about `concurrency × batch-size / interval`. | | `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. | | `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. | | `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). | | `okapi.processor.max-backoff` | `10m` | Upper bound on the delay between retries. | | `okapi.processor.backoff-jitter` | `0.2` | Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. | -| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see [Performance](#performance). | +| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each draining its own batches. Tune based on database capacity and delivery latency; see [Performance](#performance). | | `okapi.processor.transport-dispatch` | `parallel` | How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. | ### Purger @@ -343,7 +343,7 @@ The storage modules use plain JDBC. With Spring Boot, they participate in the tr Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See [`benchmarks/`](benchmarks/) for methodology and measured results. -Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and `interval` only sets how quickly a new message is picked up when the outbox was idle. To limit the rate at which okapi sends to a downstream system, set `max-batches-per-tick` (with `batch-size` and `interval`) rather than relying on the poll interval alone. +Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and `interval` only sets how quickly a new message is picked up when the outbox was idle. A failing or misconfigured downstream gets one batch per worker per tick, as before, while other delivery types keep draining. To limit the rate at which okapi sends to a healthy downstream system, set `max-batches-per-tick` (with `batch-size` and `interval`) rather than relying on the poll interval alone. ## Building diff --git a/UPGRADING.md b/UPGRADING.md index ab7c1587..2a6cdb40 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -127,8 +127,10 @@ failed. ### The scheduler drains the backlog in each tick (#132) A worker no longer stops after one batch per tick: it keeps processing batches, each in its own -transaction, while they come back full and make progress, up to `max-batches-per-tick` (default -1000). +transaction, while they come back full, up to `max-batches-per-tick` (default 1000). A delivery +type with nothing delivered in a batch (all rescheduled for retry, or failed for good, e.g. after a +401) is skipped for the rest of the tick, so a failing or misconfigured downstream still gets one +batch per worker per tick, as in 1.x. - **Downstream load.** With a backlog, okapi now sends as fast as the database and transports allow. If you relied on `batch-size` / `interval` to throttle a rate-limited downstream system, set @@ -139,8 +141,9 @@ transaction, while they come back full and make progress, up to `max-batches-per - **Metrics.** `onBatchProcessed` and `okapi.batch.duration` are still per batch, so a tick can now report many batches. - **Code.** `OutboxSchedulerConfig` and `OutboxProcessorProperties` gained a `maxBatchesPerTick` - parameter. Java callers keep the previous constructors (`@JvmOverloads`); Kotlin code calling - `copy(...)` must be recompiled. + parameter. Java callers keep the previous constructors (`@JvmOverloads`); compiled Kotlin code that + calls `copy(...)` or a constructor relying on default arguments must be recompiled, or it fails + with `NoSuchMethodError`. ## From 0.2.x to 1.0.0 diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index 4a11b0da..a5f27d0b 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -3,6 +3,7 @@ package com.softwaremill.okapi.core import org.slf4j.LoggerFactory import java.time.Clock import java.time.Duration +import java.time.Instant import java.util.concurrent.atomic.AtomicInteger /** @@ -49,30 +50,42 @@ class OutboxProcessor @JvmOverloads constructor( * to [publishEvents] once that transaction has committed. */ @JvmOverloads - fun processBatch(limit: Int = 10): ProcessedBatch { + fun processBatch(limit: Int = 10): ProcessedBatch = processBatch(limit, claimTime = null, skipTypes = emptySet()) + + /** Delivery types this processor claims. */ + internal val supportedDeliveryTypes: Set get() = deliveryTypes.toSet() + + /** + * [processBatch] for [OutboxScheduler]'s drain loop (issue #132): claims only entries due at + * [claimTime] (the start of the batch when `null`), so a retry rescheduled during a tick waits + * for the next tick, and leaves out [skipTypes]. + */ + internal fun processBatch(limit: Int, claimTime: Instant?, skipTypes: Set): ProcessedBatch { val batchStart = clock.instant() + val claimAt = claimTime ?: batchStart + val types = deliveryTypes.filterNot { it in skipTypes } val routeAwareStore = store as? RouteAwareOutboxStore ?: error( "OutboxProcessor requires a RouteAwareOutboxStore to avoid claiming unsupported delivery types, " + "got ${store.javaClass.name}. A wrapper or proxy around a store must implement RouteAwareOutboxStore itself.", ) check(deliveryTypes.isNotEmpty()) { "OutboxProcessor requires at least one delivery type" } - val claimed = if (limit <= 0) { + val claimed = if (limit <= 0 || types.isEmpty()) { emptyList() } else { - val startingType = deliveryTypes[Math.floorMod(nextStartingType.getAndIncrement(), deliveryTypes.size)] - val first = routeAwareStore.claimPending(startingType, limit, batchStart) + val startingType = types[Math.floorMod(nextStartingType.getAndIncrement(), types.size)] + val first = routeAwareStore.claimPending(startingType, limit, claimAt) check(first.size <= limit && first.all { it.deliveryType == startingType }) { "RouteAwareOutboxStore returned entries outside the requested type or limit" } - val remainingTypes = deliveryTypes.filterTo(LinkedHashSet()) { it != startingType } + val remainingTypes = types.filterTo(LinkedHashSet()) { it != startingType } if (first.size == limit || remainingTypes.isEmpty()) { first } else { val remaining = limit - first.size // Stores order this claim by delivery type, so leftover capacity favours types that sort first. // Each type is still claimed first once every N ticks, so none is starved. - val other = routeAwareStore.claimPending(remainingTypes, remaining, batchStart) + val other = routeAwareStore.claimPending(remainingTypes, remaining, claimAt) check(other.size <= remaining && other.all { it.deliveryType in remainingTypes }) { "RouteAwareOutboxStore returned entries outside the requested types or limit" } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt index 5a67298f..500c8829 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt @@ -1,6 +1,7 @@ package com.softwaremill.okapi.core import org.slf4j.LoggerFactory +import java.time.Instant import java.util.concurrent.CancellationException import java.util.concurrent.ExecutionException import java.util.concurrent.ExecutorService @@ -16,12 +17,14 @@ import java.util.concurrent.atomic.AtomicBoolean * once it has committed, [OutboxProcessor.publishEvents]. * * Each tick drains (issue #132): a worker keeps processing batches, each in its own transaction, - * while the previous one was full and made progress -- at least one entry ended `DELIVERED` or - * `FAILED` rather than being rescheduled -- up to [OutboxSchedulerConfig.maxBatchesPerTick]. + * while the previous one was full, up to [OutboxSchedulerConfig.maxBatchesPerTick]. * [OutboxSchedulerConfig.interval] is the delay after a tick, so it sets the polling rate of an - * idle outbox, not peak throughput. A batch whose entries were all rescheduled for retry ends the - * tick, so a failing downstream is not sent the whole backlog in one go. [stop] lets the batch in - * progress finish and starts no further one. + * idle outbox, not peak throughput. A delivery type none of whose entries in a batch was delivered + * (all rescheduled for retry or failed for good) is skipped for the rest of the worker's tick, so a + * failing or misconfigured downstream gets one batch per tick, as in 1.x, while healthy types keep + * draining; the tick ends once every type is skipped. All batches of a tick claim entries due at + * the tick's start, so an entry retried during the tick waits for the next one even with a zero + * backoff. [stop] lets the batch in progress finish and starts no further one. * * Each worker's call runs inside [transactionRunner]. The runner is required: without a * surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the @@ -67,10 +70,11 @@ class OutboxScheduler @JvmOverloads constructor( check(!scheduler.isShutdown) { "OutboxScheduler cannot be restarted after stop()" } if (!running.compareAndSet(false, true)) return logger.info( - "Outbox processor started [interval={}, batchSize={}, concurrency={}]", + "Outbox processor started [interval={}, batchSize={}, concurrency={}, maxBatchesPerTick={}]", config.interval, config.batchSize, config.concurrency, + config.maxBatchesPerTick, ) scheduler.scheduleWithFixedDelay(::tick, 0L, config.interval.toMillis(), TimeUnit.MILLISECONDS) } @@ -189,30 +193,46 @@ class OutboxScheduler @JvmOverloads constructor( /** One worker's share of a tick: processes batches until [shouldContinue] says to stop. */ private fun drain() { var batches = 0 + var claimTime: Instant? = null + val stalledTypes = mutableSetOf() while (true) { - val batch = processBatch() ?: return + val batch = processBatch(claimTime, stalledTypes, batches) ?: return batches++ - if (!shouldContinue(batch, batches)) { - logger.debug("Outbox processor tick completed [batches={}, lastBatchSize={}]", batches, batch.count) + claimTime = claimTime ?: batch.startedAt + // A type with nothing delivered in this batch is failing (or misconfigured): no more of it this tick. + batch.entries.groupBy { it.deliveryType } + .filterValues { entries -> entries.none { it.status == OutboxStatus.DELIVERED } } + .keys.let(stalledTypes::addAll) + if (!shouldContinue(batch, batches, stalledTypes)) { + logger.debug( + "Outbox processor tick completed [batches={}, lastBatchSize={}, skippedTypes={}]", + batches, + batch.count, + stalledTypes, + ) return } } } - private fun shouldContinue(batch: ProcessedBatch, batches: Int): Boolean = batch.count >= config.batchSize && + private fun shouldContinue(batch: ProcessedBatch, batches: Int, stalledTypes: Set): Boolean = batch.count >= config.batchSize && batches < config.maxBatchesPerTick && running.get() && !Thread.currentThread().isInterrupted && - batch.entries.any { it.status != OutboxStatus.PENDING } + !stalledTypes.containsAll(outboxProcessor.supportedDeliveryTypes) /** Processes one batch in its own transaction; returns `null` if that failed. */ - private fun processBatch(): ProcessedBatch? = try { - val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } + private fun processBatch(claimTime: Instant?, skipTypes: Set, committedBatches: Int): ProcessedBatch? = try { + val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize, claimTime, skipTypes) } // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. outboxProcessor.publishEvents(batch) batch } catch (e: Exception) { - logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) + logger.error( + "Outbox processor batch failed after {} committed batch(es) in this tick, will retry at next scheduled interval", + committedBatches, + e, + ) null } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt index 7f7c780a..71f80a79 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt @@ -26,10 +26,11 @@ data class OutboxSchedulerConfig @JvmOverloads constructor( val workerExecutorFactory: (Int) -> ExecutorService = ::defaultPlatformPool, /** * Upper bound on the batches each worker processes per tick (issue #132). A worker keeps - * claiming, each batch in its own transaction, while the previous batch was full and made - * progress, so [interval] only sets the idle polling delay and a backlog drains as fast as the - * database and transports allow. `1` restores the 1.x behaviour of one batch per worker per - * tick, which caps throughput at about `concurrency × batchSize / interval`. + * claiming, each batch in its own transaction, while the previous batch was full, skipping for + * the rest of the tick any delivery type with nothing delivered in a batch; see [OutboxScheduler]. + * So [interval] only sets the idle polling delay and a backlog drains as fast as the database and + * transports allow. `1` restores the 1.x behaviour of one batch per worker per tick, which caps + * throughput at about `concurrency × batchSize / interval`. */ val maxBatchesPerTick: Int = DEFAULT_MAX_BATCHES_PER_TICK, ) { diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt index 54015e68..538785ab 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt @@ -2,7 +2,6 @@ package com.softwaremill.okapi.core import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.collections.shouldContainExactly -import io.kotest.matchers.ints.shouldBeLessThan import io.kotest.matchers.shouldBe import java.time.Clock import java.time.Duration @@ -12,28 +11,38 @@ import java.util.concurrent.CountDownLatch import java.util.concurrent.TimeUnit import java.util.concurrent.atomic.AtomicInteger -private val drainInfo = object : DeliveryInfo { - override val type = "stub" +private class TypeInfo(override val type: String) : DeliveryInfo { override fun serialize() = "{}" } /** In-memory store that claims due PENDING entries in order, as the real stores do. */ -private class BacklogStore(size: Int) : RouteAwareOutboxStore { +private class BacklogStore(backlog: Map) : RouteAwareOutboxStore { + constructor(size: Int) : this(mapOf("stub" to size)) + private val rows = LinkedHashMap() init { - repeat(size) { - val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), drainInfo, Instant.EPOCH) - rows[entry.outboxId] = entry + backlog.forEach { (type, size) -> + repeat(size) { + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), TypeInfo(type), Instant.EPOCH) + rows[entry.outboxId] = entry + } } } - fun count(status: OutboxStatus): Int = synchronized(this) { rows.values.count { it.status == status } } + fun count(status: OutboxStatus, type: String? = null): Int = synchronized(this) { + rows.values.count { it.status == status && (type == null || it.deliveryType == type) } + } + + fun ids(): List = synchronized(this) { rows.keys.toList() } + + fun retries(type: String): List = synchronized(this) { rows.values.filter { it.deliveryType == type }.map { it.retries } } override fun persist(entry: OutboxEntry) = synchronized(this) { entry.also { rows[it.outboxId] = it } } override fun claimPending(limit: Int): List = error("not used") override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = synchronized(this) { rows.values.filter { it.status == OutboxStatus.PENDING && it.deliveryType in deliveryTypes && !it.nextAttemptAt.isAfter(now) } + .sortedWith(compareBy({ it.deliveryType }, { it.nextAttemptAt })) .take(limit) // Stands in for FOR UPDATE SKIP LOCKED: another worker must not claim these until they are updated. .onEach { rows[it.outboxId] = it.copy(status = OutboxStatus.FAILED, lastError = "claimed") } @@ -59,27 +68,38 @@ private class CountingRunner : TransactionRunner { override fun runInTransaction(block: () -> T): T = block().also { transactions.incrementAndGet() } } -private fun drainProcessor(store: BacklogStore, listener: OutboxProcessorListener, result: DeliveryResult = DeliveryResult.Success) = - OutboxProcessor( - store = store, - entryProcessor = OutboxEntryProcessor( - object : MessageDeliverer { - override val type = "stub" - override fun deliver(entry: OutboxEntry) = result - }, - RetryPolicy(maxRetries = 5), - Clock.systemUTC(), - ), - listener = listener, - ) - -/** Waits [millis] and asserts nothing more was processed, for checks that a tick stopped early. */ +private fun delivererOf(type: String, result: (OutboxEntry) -> DeliveryResult) = object : MessageDeliverer { + override val type = type + override fun deliver(entry: OutboxEntry) = result(entry) +} + +private fun drainProcessor( + store: BacklogStore, + listener: OutboxProcessorListener, + deliverer: MessageDeliverer = delivererOf("stub") { DeliveryResult.Success }, + retryPolicy: RetryPolicy = RetryPolicy(maxRetries = 5), +) = OutboxProcessor(store, OutboxEntryProcessor(deliverer, retryPolicy, Clock.systemUTC()), listener) + +private fun scheduler(processor: OutboxProcessor, runner: TransactionRunner = CountingRunner(), config: OutboxSchedulerConfig) = + OutboxScheduler(processor, runner, config) + +private val oneTickOnly = OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10) + +/** + * Waits [millis] and asserts nothing more was processed, for checks that a tick stopped early. + * It cannot fail spuriously, only miss a late extra batch. + */ private fun assertQuiet(recorder: BatchRecorder, millis: Long = 300) { val before = recorder.batches.toList() Thread.sleep(millis) recorder.batches.toList() shouldBe before } +private fun OutboxScheduler.runFirstTick(done: CountDownLatch) { + start() + done.await(5, TimeUnit.SECONDS) shouldBe true +} + /** Issue #132: a tick drains the backlog instead of processing one batch per worker. */ class OutboxSchedulerDrainTest : FunSpec({ test("one tick drains the whole backlog, each batch in its own transaction") { @@ -87,14 +107,9 @@ class OutboxSchedulerDrainTest : FunSpec({ val drained = CountDownLatch(1) val recorder = BatchRecorder { if (it < 10) drained.countDown() } val runner = CountingRunner() - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - runner, - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), - ) + val scheduler = scheduler(drainProcessor(store, recorder), runner, oneTickOnly) - scheduler.start() - drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(drained) scheduler.stop() store.count(OutboxStatus.DELIVERED) shouldBe 35 @@ -106,14 +121,9 @@ class OutboxSchedulerDrainTest : FunSpec({ val store = BacklogStore(20) val drained = CountDownLatch(1) val recorder = BatchRecorder { if (it == 0) drained.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), - ) + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly) - scheduler.start() - drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(drained) assertQuiet(recorder) scheduler.stop() @@ -124,14 +134,9 @@ class OutboxSchedulerDrainTest : FunSpec({ val store = BacklogStore(35) val firstBatch = CountDownLatch(1) val recorder = BatchRecorder { firstBatch.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, maxBatchesPerTick = 1), - ) + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(maxBatchesPerTick = 1)) - scheduler.start() - firstBatch.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(firstBatch) assertQuiet(recorder) scheduler.stop() @@ -143,14 +148,9 @@ class OutboxSchedulerDrainTest : FunSpec({ val store = BacklogStore(100) val third = CountDownLatch(3) val recorder = BatchRecorder { third.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, maxBatchesPerTick = 3), - ) + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(maxBatchesPerTick = 3)) - scheduler.start() - third.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(third) assertQuiet(recorder) scheduler.stop() @@ -161,14 +161,10 @@ class OutboxSchedulerDrainTest : FunSpec({ val store = BacklogStore(35) val firstBatch = CountDownLatch(1) val recorder = BatchRecorder { firstBatch.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder, DeliveryResult.RetriableFailure("503")), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), - ) + val failing = delivererOf("stub") { DeliveryResult.RetriableFailure("503") } + val scheduler = scheduler(drainProcessor(store, recorder, failing), config = oneTickOnly) - scheduler.start() - firstBatch.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(firstBatch) assertQuiet(recorder) scheduler.stop() @@ -177,21 +173,77 @@ class OutboxSchedulerDrainTest : FunSpec({ store.count(OutboxStatus.PENDING) shouldBe 35 } - test("permanent failures are progress, so the tick keeps draining") { + test("a type that only fails for good gets one batch per tick, so a misconfiguration does not fail the backlog at full speed") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val rejecting = delivererOf("stub") { DeliveryResult.PermanentFailure("401") } + val scheduler = scheduler(drainProcessor(store, recorder, rejecting), config = oneTickOnly) + + scheduler.runFirstTick(firstBatch) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.FAILED) shouldBe 10 + store.count(OutboxStatus.PENDING) shouldBe 25 + } + + test("a batch with some deliveries and some retries keeps draining") { val store = BacklogStore(35) val drained = CountDownLatch(1) val recorder = BatchRecorder { if (it < 10) drained.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder, DeliveryResult.PermanentFailure("400")), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), + val calls = AtomicInteger() + val flaky = delivererOf("stub") { + if (calls.incrementAndGet() % 3 == 0) DeliveryResult.RetriableFailure("503") else DeliveryResult.Success + } + val scheduler = scheduler(drainProcessor(store, recorder, flaky), config = oneTickOnly) + + scheduler.runFirstTick(drained) + scheduler.stop() + + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + store.count(OutboxStatus.PENDING) shouldBe 11 + } + + test("a failing delivery type is skipped for the rest of the tick while a healthy one keeps draining") { + val store = BacklogStore(mapOf("a" to 500, "b" to 500)) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val deliverer = CompositeMessageDeliverer( + listOf( + delivererOf("a") { DeliveryResult.Success }, + delivererOf("b") { DeliveryResult.RetriableFailure("503") }, + ), ) + val scheduler = scheduler(drainProcessor(store, recorder, deliverer), config = oneTickOnly) - scheduler.start() - drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(drained) + assertQuiet(recorder) scheduler.stop() - store.count(OutboxStatus.FAILED) shouldBe 35 + store.count(OutboxStatus.DELIVERED, "a") shouldBe 500 + // The failing downstream is sent one batch in the tick, not b's whole backlog. + store.retries("b").count { it > 0 } shouldBe 10 + } + + test("with a zero backoff, an entry retried during the tick waits for the next tick") { + val store = BacklogStore(200) + val failing = store.ids().take(3).toSet() + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + // Same type as the healthy entries, so only the tick's claim time keeps the retries out. + val deliverer = delivererOf("stub") { + if (it.outboxId in failing) DeliveryResult.RetriableFailure("503") else DeliveryResult.Success + } + val noBackoff = RetryPolicy(maxRetries = 5, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO) + val scheduler = scheduler(drainProcessor(store, recorder, deliverer, noBackoff), config = oneTickOnly) + + scheduler.runFirstTick(drained) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 197 + store.retries("stub").filter { it > 0 } shouldContainExactly listOf(1, 1, 1) } test("a failed transaction ends the tick; the next tick resumes") { @@ -205,14 +257,9 @@ class OutboxSchedulerDrainTest : FunSpec({ return block() } } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - failSecond, - OutboxSchedulerConfig(interval = Duration.ofMillis(200), batchSize = 10), - ) + val scheduler = scheduler(drainProcessor(store, recorder), failSecond, OutboxSchedulerConfig(interval = Duration.ofMillis(200))) - scheduler.start() - drained.await(5, TimeUnit.SECONDS) shouldBe true + scheduler.runFirstTick(drained) scheduler.stop() // Tick 1: one batch, then the failure. Tick 2 drains the rest. @@ -223,11 +270,7 @@ class OutboxSchedulerDrainTest : FunSpec({ test("concurrent workers each drain until the backlog is empty") { val store = BacklogStore(200) val recorder = BatchRecorder() - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10, concurrency = 4), - ) + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(concurrency = 4)) scheduler.start() val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) @@ -238,23 +281,30 @@ class OutboxSchedulerDrainTest : FunSpec({ recorder.batches.sum() shouldBe 200 } - test("stop() during a drain finishes the batch in progress and starts no further one") { - val store = BacklogStore(100_000) - val fewBatches = CountDownLatch(3) - val recorder = BatchRecorder { fewBatches.countDown() } - val scheduler = OutboxScheduler( - drainProcessor(store, recorder), - CountingRunner(), - OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10), - ) + test("stop() during a drain finishes the batch in flight and starts no further one") { + val store = BacklogStore(1_000) + val inThirdBatch = CountDownLatch(1) + val release = CountDownLatch(1) + val delivered = AtomicInteger() + val blocking = delivererOf("stub") { + if (delivered.incrementAndGet() == 21) { + inThirdBatch.countDown() + release.await(5, TimeUnit.SECONDS) + } + DeliveryResult.Success + } + val recorder = BatchRecorder() + val scheduler = scheduler(drainProcessor(store, recorder, blocking), config = oneTickOnly) scheduler.start() - fewBatches.await(5, TimeUnit.SECONDS) shouldBe true - scheduler.stop() - - val processed = recorder.batches.sum() - store.count(OutboxStatus.DELIVERED) shouldBe processed - processed shouldBeLessThan 10 * OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK - assertQuiet(recorder) + inThirdBatch.await(5, TimeUnit.SECONDS) shouldBe true + val stopping = Thread { scheduler.stop() }.apply { start() } + Thread.sleep(100) // let stop() clear the running flag while the third batch is still in flight + release.countDown() + stopping.join(10_000) + + stopping.isAlive shouldBe false + recorder.batches shouldContainExactly listOf(10, 10, 10) + store.count(OutboxStatus.DELIVERED) shouldBe 30 } }) diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt index 1ef40522..3bd4597a 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt @@ -12,7 +12,7 @@ data class OutboxProcessorProperties @JvmOverloads constructor( val batchSize: Int = 10, val maxRetries: Int = 5, /** - * Number of parallel workers fanned out per scheduler tick, each claiming its own batch + * Number of parallel workers fanned out per scheduler tick, each draining its own batches * via `FOR UPDATE SKIP LOCKED`. See [com.softwaremill.okapi.core.OutboxSchedulerConfig]. */ val concurrency: Int = 1, diff --git a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json index c4af5bf6..e6d01b06 100644 --- a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json +++ b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json @@ -73,7 +73,7 @@ "name": "okapi.processor.max-batches-per-tick", "type": "java.lang.Integer", "defaultValue": 1000, - "description": "Maximum number of batches each worker processes per tick. A worker continues while the previous batch was full and at least one of its entries was delivered or failed. 1 restores the 1.x behaviour of one batch per worker per tick." + "description": "Maximum number of batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. 1 restores the 1.x behaviour of one batch per worker per tick." }, { "name": "okapi.processor.max-retries",