diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a71d60..f68768c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,24 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * ### Changed (BREAKING) +- **The scheduler drains the backlog within a tick instead of processing one batch per worker.** + Each worker used to process exactly one batch per tick, capping throughput at about + `concurrency × batchSize / interval`, under 10 messages/s with the defaults, so any sustained + higher publish rate grew the backlog without bound. A worker now keeps processing batches, each in + its own transaction, while the previous one was full, up to the new + `OutboxSchedulerConfig.maxBatchesPerTick` (default 1000, Spring property + `okapi.processor.max-batches-per-tick`). A delivery type with nothing delivered in a batch (all + rescheduled for retry or failed for good) is skipped for the rest of the tick, so a failing or + misconfigured downstream still gets one batch per tick while other types keep draining, and the + tick ends when every type is skipped. Batches of a tick claim only entries due when the tick + started, so a retry waits for the next tick even with a zero backoff. The poll interval now only + sets the idle polling rate: a healthy downstream receives a backlog as fast as okapi can deliver + it, and `onBatchProcessed` / `okapi.batch.duration` fire more often per tick. + `maxBatchesPerTick = 1` restores the previous behaviour. `OutboxSchedulerConfig` and + `OutboxProcessorProperties` gained a constructor parameter: `@JvmOverloads` keeps the previous + Java constructors, but compiled Kotlin code that calls `copy(...)` or a constructor relying on + default arguments fails with `NoSuchMethodError` until recompiled. + ([#132](https://github.com/softwaremill/okapi/issues/132)) - **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).** A retriable failure used to be retried on the very next tick, so with the defaults (`interval` 1s, `max-retries` 5) a message was `FAILED` — terminally — after about six seconds of downstream diff --git a/README.md b/README.md index 532e0ed..a09be97 100644 --- a/README.md +++ b/README.md @@ -132,14 +132,15 @@ In a typical single-DataSource application, all properties are optional. Multi-D | Property | Default | Description | |---|---|---| | `okapi.processor.enabled` | `true` | Set `false` to disable delivery entirely (e.g. on instances that only publish). | -| `okapi.processor.interval` | `1s` | How often the scheduler polls for pending entries. | -| `okapi.processor.batch-size` | `10` | Maximum entries claimed per worker per tick. | +| `okapi.processor.interval` | `1s` | Delay between scheduler ticks, i.e. how often an idle outbox is polled. A tick with a backlog keeps going (see `max-batches-per-tick`), so this does not cap throughput. | +| `okapi.processor.batch-size` | `10` | Maximum entries claimed per batch; each batch runs in its own transaction. | +| `okapi.processor.max-batches-per-tick` | `1000` | Batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. Batches of a tick claim only entries due when the tick started, so a retry waits for the next tick. `1` restores the 1.x one batch per tick, which caps throughput at about `concurrency × batch-size / interval`. | | `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. | | `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. | | `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). | | `okapi.processor.max-backoff` | `10m` | Upper bound on the delay between retries. | | `okapi.processor.backoff-jitter` | `0.2` | Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. | -| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see [Performance](#performance). | +| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each draining its own batches. Tune based on database capacity and delivery latency; see [Performance](#performance). | | `okapi.processor.transport-dispatch` | `parallel` | How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. | ### Purger @@ -238,7 +239,7 @@ okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liqui ## Transactions -Each processor worker runs its claim, delivery and state update inside one transaction, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed. +Each processor worker runs its claim, delivery and state update inside one transaction per batch, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. While a backlog remains, a worker runs batch after batch within one tick, each in a transaction of its own. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed. **Multiple data sources or transaction managers?** Set both `okapi.datasource-qualifier` and `okapi.transaction-manager-qualifier` to the beans used for the outbox. okapi fails fast when it detects a mismatch. If the selected transaction manager does not expose its `DataSource`, okapi cannot verify the pairing and logs a warning instead. @@ -342,6 +343,8 @@ The storage modules use plain JDBC. With Spring Boot, they participate in the tr Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See [`benchmarks/`](benchmarks/) for methodology and measured results. +Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and `interval` only sets how quickly a new message is picked up when the outbox was idle. A failing or misconfigured downstream gets one batch per worker per tick, as before, while other delivery types keep draining. To limit the rate at which okapi sends to a healthy downstream system, set `max-batches-per-tick` (with `batch-size` and `interval`) rather than relying on the poll interval alone. + ## Building ```sh diff --git a/UPGRADING.md b/UPGRADING.md index ef1315b..2a6cdb4 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -124,6 +124,27 @@ failed. your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by `processor.publishEvents(batch)` to get the after-commit behaviour. +### The scheduler drains the backlog in each tick (#132) + +A worker no longer stops after one batch per tick: it keeps processing batches, each in its own +transaction, while they come back full, up to `max-batches-per-tick` (default 1000). A delivery +type with nothing delivered in a batch (all rescheduled for retry, or failed for good, e.g. after a +401) is skipped for the rest of the tick, so a failing or misconfigured downstream still gets one +batch per worker per tick, as in 1.x. + +- **Downstream load.** With a backlog, okapi now sends as fast as the database and transports allow. + If you relied on `batch-size` / `interval` to throttle a rate-limited downstream system, set + `okapi.processor.max-batches-per-tick=1` (`OutboxSchedulerConfig(maxBatchesPerTick = 1)`) to keep + the 1.x rate, or a small value to allow some draining. +- **Database load.** A backlog is now worked through in back-to-back transactions instead of one + per interval. Size `concurrency` with that in mind. +- **Metrics.** `onBatchProcessed` and `okapi.batch.duration` are still per batch, so a tick can now + report many batches. +- **Code.** `OutboxSchedulerConfig` and `OutboxProcessorProperties` gained a `maxBatchesPerTick` + parameter. Java callers keep the previous constructors (`@JvmOverloads`); compiled Kotlin code that + calls `copy(...)` or a constructor relying on default arguments must be recompiled, or it fails + with `NoSuchMethodError`. + ## From 0.2.x to 1.0.0 Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index 4a11b0d..a5f27d0 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -3,6 +3,7 @@ package com.softwaremill.okapi.core import org.slf4j.LoggerFactory import java.time.Clock import java.time.Duration +import java.time.Instant import java.util.concurrent.atomic.AtomicInteger /** @@ -49,30 +50,42 @@ class OutboxProcessor @JvmOverloads constructor( * to [publishEvents] once that transaction has committed. */ @JvmOverloads - fun processBatch(limit: Int = 10): ProcessedBatch { + fun processBatch(limit: Int = 10): ProcessedBatch = processBatch(limit, claimTime = null, skipTypes = emptySet()) + + /** Delivery types this processor claims. */ + internal val supportedDeliveryTypes: Set get() = deliveryTypes.toSet() + + /** + * [processBatch] for [OutboxScheduler]'s drain loop (issue #132): claims only entries due at + * [claimTime] (the start of the batch when `null`), so a retry rescheduled during a tick waits + * for the next tick, and leaves out [skipTypes]. + */ + internal fun processBatch(limit: Int, claimTime: Instant?, skipTypes: Set): ProcessedBatch { val batchStart = clock.instant() + val claimAt = claimTime ?: batchStart + val types = deliveryTypes.filterNot { it in skipTypes } val routeAwareStore = store as? RouteAwareOutboxStore ?: error( "OutboxProcessor requires a RouteAwareOutboxStore to avoid claiming unsupported delivery types, " + "got ${store.javaClass.name}. A wrapper or proxy around a store must implement RouteAwareOutboxStore itself.", ) check(deliveryTypes.isNotEmpty()) { "OutboxProcessor requires at least one delivery type" } - val claimed = if (limit <= 0) { + val claimed = if (limit <= 0 || types.isEmpty()) { emptyList() } else { - val startingType = deliveryTypes[Math.floorMod(nextStartingType.getAndIncrement(), deliveryTypes.size)] - val first = routeAwareStore.claimPending(startingType, limit, batchStart) + val startingType = types[Math.floorMod(nextStartingType.getAndIncrement(), types.size)] + val first = routeAwareStore.claimPending(startingType, limit, claimAt) check(first.size <= limit && first.all { it.deliveryType == startingType }) { "RouteAwareOutboxStore returned entries outside the requested type or limit" } - val remainingTypes = deliveryTypes.filterTo(LinkedHashSet()) { it != startingType } + val remainingTypes = types.filterTo(LinkedHashSet()) { it != startingType } if (first.size == limit || remainingTypes.isEmpty()) { first } else { val remaining = limit - first.size // Stores order this claim by delivery type, so leftover capacity favours types that sort first. // Each type is still claimed first once every N ticks, so none is starved. - val other = routeAwareStore.claimPending(remainingTypes, remaining, batchStart) + val other = routeAwareStore.claimPending(remainingTypes, remaining, claimAt) check(other.size <= remaining && other.all { it.deliveryType in remainingTypes }) { "RouteAwareOutboxStore returned entries outside the requested types or limit" } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt index 3876d60..500c882 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxScheduler.kt @@ -1,6 +1,7 @@ package com.softwaremill.okapi.core import org.slf4j.LoggerFactory +import java.time.Instant import java.util.concurrent.CancellationException import java.util.concurrent.ExecutionException import java.util.concurrent.ExecutorService @@ -15,6 +16,16 @@ import java.util.concurrent.atomic.AtomicBoolean * Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and, * once it has committed, [OutboxProcessor.publishEvents]. * + * Each tick drains (issue #132): a worker keeps processing batches, each in its own transaction, + * while the previous one was full, up to [OutboxSchedulerConfig.maxBatchesPerTick]. + * [OutboxSchedulerConfig.interval] is the delay after a tick, so it sets the polling rate of an + * idle outbox, not peak throughput. A delivery type none of whose entries in a batch was delivered + * (all rescheduled for retry or failed for good) is skipped for the rest of the worker's tick, so a + * failing or misconfigured downstream gets one batch per tick, as in 1.x, while healthy types keep + * draining; the tick ends once every type is skipped. All batches of a tick claim entries due at + * the tick's start, so an entry retried during the tick waits for the next one even with a zero + * backoff. [stop] lets the batch in progress finish and starts no further one. + * * Each worker's call runs inside [transactionRunner]. The runner is required: without a * surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the * claim SELECT and concurrent processor instances deliver the same entry multiple times. @@ -23,7 +34,7 @@ import java.util.concurrent.atomic.AtomicBoolean * [OutboxSchedulerConfig.workerExecutorFactory]'s pool. `FOR UPDATE SKIP LOCKED` guarantees * disjoint claims, but [outboxProcessor] (and its dependencies) must be safe for concurrent use. * The tick waits for every worker before the next scheduled interval, so ticks never overlap. - * `concurrency = 1` (default) calls the batch inline, preserving single-worker semantics. + * `concurrency = 1` (default) runs the worker inline, preserving single-worker semantics. * * At `concurrency > 1`, [outboxProcessor] -- and by extension its [OutboxStore], [MessageDeliverer], * and any [OutboxProcessorListener] -- is invoked concurrently from multiple worker threads, so all @@ -59,10 +70,11 @@ class OutboxScheduler @JvmOverloads constructor( check(!scheduler.isShutdown) { "OutboxScheduler cannot be restarted after stop()" } if (!running.compareAndSet(false, true)) return logger.info( - "Outbox processor started [interval={}, batchSize={}, concurrency={}]", + "Outbox processor started [interval={}, batchSize={}, concurrency={}, maxBatchesPerTick={}]", config.interval, config.batchSize, config.concurrency, + config.maxBatchesPerTick, ) scheduler.scheduleWithFixedDelay(::tick, 0L, config.interval.toMillis(), TimeUnit.MILLISECONDS) } @@ -110,7 +122,7 @@ class OutboxScheduler @JvmOverloads constructor( if (config.concurrency == 1) { // Skips workers/Lazy entirely -- not just an inline no-op path -- so this configuration // truly pays no executor/synchronization overhead per tick, matching the class KDoc. - processBatch() + drain() return } val pool = try { @@ -130,13 +142,13 @@ class OutboxScheduler @JvmOverloads constructor( } /** - * Submits one worker's [processBatch] without letting [RejectedExecutionException] escape + * Submits one worker's [drain] without letting [RejectedExecutionException] escape * [tick] -- a custom [OutboxSchedulerConfig.workerExecutorFactory] may return an executor that * can reject submissions (bounded queue, shutdown executor), and an uncaught exception here * would make `scheduleWithFixedDelay` suppress all future ticks. */ private fun submitWorker(pool: ExecutorService): Future<*>? = try { - pool.submit(::processBatch) + pool.submit(::drain) } catch (e: RejectedExecutionException) { logger.error("Outbox worker submission rejected, will retry at next scheduled interval", e) null @@ -145,7 +157,7 @@ class OutboxScheduler @JvmOverloads constructor( /** * Awaits one worker's completion without letting anything escape [tick] -- an uncaught * exception here would make `scheduleWithFixedDelay` suppress all future ticks, silently - * killing the scheduler. [processBatch] already catches [Exception] internally, so this + * killing the scheduler. [drain] already catches [Exception] internally, so this * only guards against an [Error] surfacing via [ExecutionException], a [CancellationException] * if a custom [OutboxSchedulerConfig.workerExecutorFactory] cancels queued/running tasks (e.g. * as a backpressure strategy), or an interrupt while awaiting. @@ -178,17 +190,52 @@ class OutboxScheduler @JvmOverloads constructor( } } - private fun processBatch() { - try { - val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) } - // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. - outboxProcessor.publishEvents(batch) - logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize) - } catch (e: Exception) { - logger.error("Outbox processor tick failed, will retry at next scheduled interval", e) + /** One worker's share of a tick: processes batches until [shouldContinue] says to stop. */ + private fun drain() { + var batches = 0 + var claimTime: Instant? = null + val stalledTypes = mutableSetOf() + while (true) { + val batch = processBatch(claimTime, stalledTypes, batches) ?: return + batches++ + claimTime = claimTime ?: batch.startedAt + // A type with nothing delivered in this batch is failing (or misconfigured): no more of it this tick. + batch.entries.groupBy { it.deliveryType } + .filterValues { entries -> entries.none { it.status == OutboxStatus.DELIVERED } } + .keys.let(stalledTypes::addAll) + if (!shouldContinue(batch, batches, stalledTypes)) { + logger.debug( + "Outbox processor tick completed [batches={}, lastBatchSize={}, skippedTypes={}]", + batches, + batch.count, + stalledTypes, + ) + return + } } } + private fun shouldContinue(batch: ProcessedBatch, batches: Int, stalledTypes: Set): Boolean = batch.count >= config.batchSize && + batches < config.maxBatchesPerTick && + running.get() && + !Thread.currentThread().isInterrupted && + !stalledTypes.containsAll(outboxProcessor.supportedDeliveryTypes) + + /** Processes one batch in its own transaction; returns `null` if that failed. */ + private fun processBatch(claimTime: Instant?, skipTypes: Set, committedBatches: Int): ProcessedBatch? = try { + val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize, claimTime, skipTypes) } + // After the commit (issue #137): a batch whose commit failed is rolled back and is not reported. + outboxProcessor.publishEvents(batch) + batch + } catch (e: Exception) { + logger.error( + "Outbox processor batch failed after {} committed batch(es) in this tick, will retry at next scheduled interval", + committedBatches, + e, + ) + null + } + companion object { private val logger = LoggerFactory.getLogger(OutboxScheduler::class.java) private const val SHUTDOWN_TIMEOUT_SECONDS = 5L diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt index 5a9d812..71f80a7 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfig.kt @@ -24,14 +24,26 @@ data class OutboxSchedulerConfig @JvmOverloads constructor( * switching away from the default. */ val workerExecutorFactory: (Int) -> ExecutorService = ::defaultPlatformPool, + /** + * Upper bound on the batches each worker processes per tick (issue #132). A worker keeps + * claiming, each batch in its own transaction, while the previous batch was full, skipping for + * the rest of the tick any delivery type with nothing delivered in a batch; see [OutboxScheduler]. + * So [interval] only sets the idle polling delay and a backlog drains as fast as the database and + * transports allow. `1` restores the 1.x behaviour of one batch per worker per tick, which caps + * throughput at about `concurrency × batchSize / interval`. + */ + val maxBatchesPerTick: Int = DEFAULT_MAX_BATCHES_PER_TICK, ) { init { require(!interval.isNegative && interval.toMillis() > 0) { "interval must be at least 1ms, got: $interval" } require(batchSize > 0) { "batchSize must be positive, got: $batchSize" } require(concurrency in 1..256) { "concurrency must be between 1 and 256, got: $concurrency" } + require(maxBatchesPerTick > 0) { "maxBatchesPerTick must be positive, got: $maxBatchesPerTick" } } companion object { + const val DEFAULT_MAX_BATCHES_PER_TICK = 1000 + /** Fixed pool of [n] daemon platform threads, named `outbox-worker-N`. */ @JvmStatic fun defaultPlatformPool(n: Int): ExecutorService { diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt index 4bb2c99..68218a3 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerConfigTest.kt @@ -17,6 +17,13 @@ class OutboxSchedulerConfigTest : FunSpec({ config.interval shouldBe ofSeconds(1) config.batchSize shouldBe 10 config.concurrency shouldBe 1 + config.maxBatchesPerTick shouldBe OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK + } + + test("rejects non-positive maxBatchesPerTick") { + listOf(0, -1).forEach { value -> + shouldThrow { OutboxSchedulerConfig(maxBatchesPerTick = value) } + } } test("accepts custom valid values") { diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt new file mode 100644 index 0000000..538785a --- /dev/null +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerDrainTest.kt @@ -0,0 +1,310 @@ +package com.softwaremill.okapi.core + +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +private class TypeInfo(override val type: String) : DeliveryInfo { + override fun serialize() = "{}" +} + +/** In-memory store that claims due PENDING entries in order, as the real stores do. */ +private class BacklogStore(backlog: Map) : RouteAwareOutboxStore { + constructor(size: Int) : this(mapOf("stub" to size)) + + private val rows = LinkedHashMap() + + init { + backlog.forEach { (type, size) -> + repeat(size) { + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), TypeInfo(type), Instant.EPOCH) + rows[entry.outboxId] = entry + } + } + } + + fun count(status: OutboxStatus, type: String? = null): Int = synchronized(this) { + rows.values.count { it.status == status && (type == null || it.deliveryType == type) } + } + + fun ids(): List = synchronized(this) { rows.keys.toList() } + + fun retries(type: String): List = synchronized(this) { rows.values.filter { it.deliveryType == type }.map { it.retries } } + + override fun persist(entry: OutboxEntry) = synchronized(this) { entry.also { rows[it.outboxId] = it } } + override fun claimPending(limit: Int): List = error("not used") + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = synchronized(this) { + rows.values.filter { it.status == OutboxStatus.PENDING && it.deliveryType in deliveryTypes && !it.nextAttemptAt.isAfter(now) } + .sortedWith(compareBy({ it.deliveryType }, { it.nextAttemptAt })) + .take(limit) + // Stands in for FOR UPDATE SKIP LOCKED: another worker must not claim these until they are updated. + .onEach { rows[it.outboxId] = it.copy(status = OutboxStatus.FAILED, lastError = "claimed") } + } + override fun updateAfterProcessing(entry: OutboxEntry) = persist(entry) + override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 + override fun findOldestCreatedAt(statuses: Set) = emptyMap() + override fun countByStatuses() = emptyMap() +} + +/** Records committed batch sizes; [onBatch] runs after each one. */ +private class BatchRecorder(private val onBatch: (Int) -> Unit = {}) : OutboxProcessorListener { + val batches: MutableList = Collections.synchronizedList(mutableListOf()) + + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + batches += processedCount + onBatch(processedCount) + } +} + +private class CountingRunner : TransactionRunner { + val transactions = AtomicInteger() + override fun runInTransaction(block: () -> T): T = block().also { transactions.incrementAndGet() } +} + +private fun delivererOf(type: String, result: (OutboxEntry) -> DeliveryResult) = object : MessageDeliverer { + override val type = type + override fun deliver(entry: OutboxEntry) = result(entry) +} + +private fun drainProcessor( + store: BacklogStore, + listener: OutboxProcessorListener, + deliverer: MessageDeliverer = delivererOf("stub") { DeliveryResult.Success }, + retryPolicy: RetryPolicy = RetryPolicy(maxRetries = 5), +) = OutboxProcessor(store, OutboxEntryProcessor(deliverer, retryPolicy, Clock.systemUTC()), listener) + +private fun scheduler(processor: OutboxProcessor, runner: TransactionRunner = CountingRunner(), config: OutboxSchedulerConfig) = + OutboxScheduler(processor, runner, config) + +private val oneTickOnly = OutboxSchedulerConfig(interval = Duration.ofMinutes(1), batchSize = 10) + +/** + * Waits [millis] and asserts nothing more was processed, for checks that a tick stopped early. + * It cannot fail spuriously, only miss a late extra batch. + */ +private fun assertQuiet(recorder: BatchRecorder, millis: Long = 300) { + val before = recorder.batches.toList() + Thread.sleep(millis) + recorder.batches.toList() shouldBe before +} + +private fun OutboxScheduler.runFirstTick(done: CountDownLatch) { + start() + done.await(5, TimeUnit.SECONDS) shouldBe true +} + +/** Issue #132: a tick drains the backlog instead of processing one batch per worker. */ +class OutboxSchedulerDrainTest : FunSpec({ + test("one tick drains the whole backlog, each batch in its own transaction") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val runner = CountingRunner() + val scheduler = scheduler(drainProcessor(store, recorder), runner, oneTickOnly) + + scheduler.runFirstTick(drained) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 35 + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + runner.transactions.get() shouldBe 4 + } + + test("an exactly full last batch is followed by one empty claim, then the tick ends") { + val store = BacklogStore(20) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it == 0) drained.countDown() } + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly) + + scheduler.runFirstTick(drained) + assertQuiet(recorder) + scheduler.stop() + + recorder.batches shouldContainExactly listOf(10, 10, 0) + } + + test("maxBatchesPerTick = 1 keeps the 1.x one batch per tick") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(maxBatchesPerTick = 1)) + + scheduler.runFirstTick(firstBatch) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 10 + recorder.batches shouldContainExactly listOf(10) + } + + test("the tick stops at maxBatchesPerTick") { + val store = BacklogStore(100) + val third = CountDownLatch(3) + val recorder = BatchRecorder { third.countDown() } + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(maxBatchesPerTick = 3)) + + scheduler.runFirstTick(third) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 30 + } + + test("a full batch whose entries were all rescheduled for retry ends the tick") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val failing = delivererOf("stub") { DeliveryResult.RetriableFailure("503") } + val scheduler = scheduler(drainProcessor(store, recorder, failing), config = oneTickOnly) + + scheduler.runFirstTick(firstBatch) + assertQuiet(recorder) + scheduler.stop() + + // The downstream is failing: only one batch is sent to it, not the whole backlog. + recorder.batches shouldContainExactly listOf(10) + store.count(OutboxStatus.PENDING) shouldBe 35 + } + + test("a type that only fails for good gets one batch per tick, so a misconfiguration does not fail the backlog at full speed") { + val store = BacklogStore(35) + val firstBatch = CountDownLatch(1) + val recorder = BatchRecorder { firstBatch.countDown() } + val rejecting = delivererOf("stub") { DeliveryResult.PermanentFailure("401") } + val scheduler = scheduler(drainProcessor(store, recorder, rejecting), config = oneTickOnly) + + scheduler.runFirstTick(firstBatch) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.FAILED) shouldBe 10 + store.count(OutboxStatus.PENDING) shouldBe 25 + } + + test("a batch with some deliveries and some retries keeps draining") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val calls = AtomicInteger() + val flaky = delivererOf("stub") { + if (calls.incrementAndGet() % 3 == 0) DeliveryResult.RetriableFailure("503") else DeliveryResult.Success + } + val scheduler = scheduler(drainProcessor(store, recorder, flaky), config = oneTickOnly) + + scheduler.runFirstTick(drained) + scheduler.stop() + + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + store.count(OutboxStatus.PENDING) shouldBe 11 + } + + test("a failing delivery type is skipped for the rest of the tick while a healthy one keeps draining") { + val store = BacklogStore(mapOf("a" to 500, "b" to 500)) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val deliverer = CompositeMessageDeliverer( + listOf( + delivererOf("a") { DeliveryResult.Success }, + delivererOf("b") { DeliveryResult.RetriableFailure("503") }, + ), + ) + val scheduler = scheduler(drainProcessor(store, recorder, deliverer), config = oneTickOnly) + + scheduler.runFirstTick(drained) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED, "a") shouldBe 500 + // The failing downstream is sent one batch in the tick, not b's whole backlog. + store.retries("b").count { it > 0 } shouldBe 10 + } + + test("with a zero backoff, an entry retried during the tick waits for the next tick") { + val store = BacklogStore(200) + val failing = store.ids().take(3).toSet() + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + // Same type as the healthy entries, so only the tick's claim time keeps the retries out. + val deliverer = delivererOf("stub") { + if (it.outboxId in failing) DeliveryResult.RetriableFailure("503") else DeliveryResult.Success + } + val noBackoff = RetryPolicy(maxRetries = 5, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO) + val scheduler = scheduler(drainProcessor(store, recorder, deliverer, noBackoff), config = oneTickOnly) + + scheduler.runFirstTick(drained) + assertQuiet(recorder) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 197 + store.retries("stub").filter { it > 0 } shouldContainExactly listOf(1, 1, 1) + } + + test("a failed transaction ends the tick; the next tick resumes") { + val store = BacklogStore(35) + val drained = CountDownLatch(1) + val recorder = BatchRecorder { if (it < 10) drained.countDown() } + val attempts = AtomicInteger() + val failSecond = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T { + if (attempts.incrementAndGet() == 2) throw IllegalStateException("connection reset") + return block() + } + } + val scheduler = scheduler(drainProcessor(store, recorder), failSecond, OutboxSchedulerConfig(interval = Duration.ofMillis(200))) + + scheduler.runFirstTick(drained) + scheduler.stop() + + // Tick 1: one batch, then the failure. Tick 2 drains the rest. + recorder.batches shouldContainExactly listOf(10, 10, 10, 5) + attempts.get() shouldBe 5 + } + + test("concurrent workers each drain until the backlog is empty") { + val store = BacklogStore(200) + val recorder = BatchRecorder() + val scheduler = scheduler(drainProcessor(store, recorder), config = oneTickOnly.copy(concurrency = 4)) + + scheduler.start() + val deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5) + while (store.count(OutboxStatus.DELIVERED) < 200 && System.nanoTime() < deadline) Thread.sleep(10) + scheduler.stop() + + store.count(OutboxStatus.DELIVERED) shouldBe 200 + recorder.batches.sum() shouldBe 200 + } + + test("stop() during a drain finishes the batch in flight and starts no further one") { + val store = BacklogStore(1_000) + val inThirdBatch = CountDownLatch(1) + val release = CountDownLatch(1) + val delivered = AtomicInteger() + val blocking = delivererOf("stub") { + if (delivered.incrementAndGet() == 21) { + inThirdBatch.countDown() + release.await(5, TimeUnit.SECONDS) + } + DeliveryResult.Success + } + val recorder = BatchRecorder() + val scheduler = scheduler(drainProcessor(store, recorder, blocking), config = oneTickOnly) + + scheduler.start() + inThirdBatch.await(5, TimeUnit.SECONDS) shouldBe true + val stopping = Thread { scheduler.stop() }.apply { start() } + Thread.sleep(100) // let stop() clear the running flag while the third batch is still in flight + release.countDown() + stopping.join(10_000) + + stopping.isAlive shouldBe false + recorder.batches shouldContainExactly listOf(10, 10, 10) + store.count(OutboxStatus.DELIVERED) shouldBe 30 + } +}) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt new file mode 100644 index 0000000..5769919 --- /dev/null +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/DrainBacklogEndToEndTest.kt @@ -0,0 +1,107 @@ +package com.softwaremill.okapi.test.e2e + +import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer +import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor +import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessor +import com.softwaremill.okapi.core.OutboxProcessorListener +import com.softwaremill.okapi.core.OutboxScheduler +import com.softwaremill.okapi.core.OutboxSchedulerConfig +import com.softwaremill.okapi.core.OutboxStatus +import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore +import com.softwaremill.okapi.core.TransactionRunner +import com.softwaremill.okapi.mysql.MysqlOutboxStore +import com.softwaremill.okapi.postgres.PostgresOutboxStore +import com.softwaremill.okapi.test.support.JdbcConnectionProvider +import com.softwaremill.okapi.test.support.MysqlTestSupport +import com.softwaremill.okapi.test.support.PostgresTestSupport +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.collections.shouldContainExactly +import io.kotest.matchers.shouldBe +import java.sql.Connection +import java.time.Clock +import java.time.Duration +import java.util.Collections +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +private object DrainDeliveryInfo : DeliveryInfo { + override val type: String = "drain" + override fun serialize(): String = """{"type":"drain"}""" +} + +private const val BACKLOG = 250 +private const val BATCH_SIZE = 10 + +/** + * Issue #132 against real databases: with an interval far longer than the test, a single tick + * delivers the whole backlog, one transaction per batch. + */ +class DrainBacklogEndToEndTest : FunSpec({ + val postgres = PostgresTestSupport() + val mysql = MysqlTestSupport() + + beforeSpec { + postgres.start() + mysql.start() + } + afterSpec { + postgres.stop() + mysql.stop() + } + + fun drainsInOneTick(jdbc: JdbcConnectionProvider, store: RouteAwareOutboxStore, isolation: Int?) { + val now = Clock.systemUTC().instant() + jdbc.withTransaction { + repeat(BACKLOG) { store.persist(OutboxEntry.createPending(OutboxMessage("e", "{}"), DrainDeliveryInfo, now)) } + } + + val batches = Collections.synchronizedList(mutableListOf()) + val drained = CountDownLatch(1) + val listener = object : OutboxProcessorListener { + override fun onBatchProcessed(processedCount: Int, duration: Duration) { + batches += processedCount + if (processedCount < BATCH_SIZE) drained.countDown() + } + } + val processor = OutboxProcessor( + store = store, + entryProcessor = OutboxEntryProcessor( + object : MessageDeliverer { + override val type = DrainDeliveryInfo.type + override fun deliver(entry: OutboxEntry) = DeliveryResult.Success + }, + RetryPolicy(maxRetries = 0), + Clock.systemUTC(), + ), + listener = listener, + ) + val runner = object : TransactionRunner { + override fun runInTransaction(block: () -> T): T = jdbc.withTransaction(isolation, block) + } + val scheduler = OutboxScheduler(processor, runner, OutboxSchedulerConfig(interval = Duration.ofHours(1), batchSize = BATCH_SIZE)) + + scheduler.start() + drained.await(30, TimeUnit.SECONDS) shouldBe true + scheduler.stop() + + jdbc.withTransaction { store.countByStatuses() }[OutboxStatus.DELIVERED] shouldBe BACKLOG.toLong() + // 25 full batches, then an empty claim ends the tick. + batches shouldContainExactly List(BACKLOG / BATCH_SIZE) { BATCH_SIZE } + 0 + } + + test("Postgres: one tick drains a backlog of $BACKLOG entries") { + postgres.truncate() + drainsInOneTick(postgres.jdbc, PostgresOutboxStore(postgres.jdbc), isolation = null) + } + + test("MySQL: one tick drains a backlog of $BACKLOG entries") { + mysql.truncate() + // The scheduler's isolation on MySQL (issue #129). + drainsInOneTick(mysql.jdbc, MysqlOutboxStore(mysql.jdbc), Connection.TRANSACTION_READ_COMMITTED) + } +}) diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index 0ffa57a..3fa1b49 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -234,6 +234,7 @@ class OutboxAutoConfiguration( interval = props.interval, batchSize = props.batchSize, concurrency = props.concurrency, + maxBatchesPerTick = props.maxBatchesPerTick, ), ) } diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt index 394c00e..3bd4597 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt @@ -1,5 +1,6 @@ package com.softwaremill.okapi.springboot +import com.softwaremill.okapi.core.OutboxSchedulerConfig import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.TransportDispatch import org.springframework.boot.context.properties.ConfigurationProperties @@ -11,7 +12,7 @@ data class OutboxProcessorProperties @JvmOverloads constructor( val batchSize: Int = 10, val maxRetries: Int = 5, /** - * Number of parallel workers fanned out per scheduler tick, each claiming its own batch + * Number of parallel workers fanned out per scheduler tick, each draining its own batches * via `FOR UPDATE SKIP LOCKED`. See [com.softwaremill.okapi.core.OutboxSchedulerConfig]. */ val concurrency: Int = 1, @@ -31,12 +32,18 @@ data class OutboxProcessorProperties @JvmOverloads constructor( val maxBackoff: Duration = RetryPolicy.DEFAULT_MAX_BACKOFF, /** Fraction (0 to 1) by which each delay is shortened at random, to spread retries out. */ val backoffJitter: Double = RetryPolicy.DEFAULT_JITTER, + /** + * Maximum batches each worker processes per tick while a backlog remains; `1` restores the 1.x + * one batch per tick. See [com.softwaremill.okapi.core.OutboxSchedulerConfig.maxBatchesPerTick]. + */ + val maxBatchesPerTick: Int = OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK, ) { init { require(!interval.isNegative && interval.toMillis() > 0) { "interval must be at least 1ms" } require(batchSize > 0) { "batchSize must be positive" } require(maxRetries >= 0) { "maxRetries must be >= 0" } require(concurrency in 1..256) { "concurrency must be between 1 and 256" } + require(maxBatchesPerTick > 0) { "maxBatchesPerTick must be positive" } // Fail at startup, with the property names, rather than when the RetryPolicy is built. require(!initialBackoff.isNegative) { "initialBackoff must be >= 0" } require(backoffMultiplier >= 1.0 && backoffMultiplier.isFinite()) { "backoffMultiplier must be finite and >= 1" } diff --git a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json index ae1d9ae..e6d01b0 100644 --- a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json +++ b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json @@ -61,13 +61,19 @@ "name": "okapi.processor.interval", "type": "java.time.Duration", "defaultValue": "1s", - "description": "How often the processor runs." + "description": "Delay between scheduler ticks: how often an idle outbox is polled. A tick with a backlog keeps processing batches (see max-batches-per-tick), so this does not cap throughput." }, { "name": "okapi.processor.batch-size", "type": "java.lang.Integer", "defaultValue": 10, - "description": "Maximum number of pending entries to process per tick." + "description": "Maximum number of pending entries claimed and processed per batch, each batch in its own transaction." + }, + { + "name": "okapi.processor.max-batches-per-tick", + "type": "java.lang.Integer", + "defaultValue": 1000, + "description": "Maximum number of batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. 1 restores the 1.x behaviour of one batch per worker per tick." }, { "name": "okapi.processor.max-retries", diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt index 5ac616e..eeec136 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt @@ -8,8 +8,10 @@ import com.softwaremill.okapi.core.OutboxEntryProcessor import com.softwaremill.okapi.core.OutboxMessage import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxPublisher +import com.softwaremill.okapi.core.OutboxSchedulerConfig import com.softwaremill.okapi.core.OutboxStore import com.softwaremill.okapi.core.RetryPolicy +import com.softwaremill.okapi.core.RouteAwareOutboxStore import com.softwaremill.okapi.core.TransactionRunner import com.softwaremill.okapi.core.TransportDispatch import com.softwaremill.okapi.micrometer.MicrometerOutboxListener @@ -36,6 +38,7 @@ import java.time.Duration.ofMinutes import java.time.Duration.ofSeconds import java.time.Instant import java.util.concurrent.ConcurrentLinkedQueue +import java.util.concurrent.atomic.AtomicInteger import javax.sql.DataSource class OutboxProcessorAutoConfigurationTest : FunSpec({ @@ -200,6 +203,52 @@ class OutboxProcessorAutoConfigurationTest : FunSpec({ } } + test("max-batches-per-tick is bound and validated, defaulting to OutboxSchedulerConfig's (#132)") { + contextRunner.withPropertyValues("okapi.processor.max-batches-per-tick=7").run { ctx -> + ctx.getBean().maxBatchesPerTick shouldBe 7 + } + contextRunner.run { ctx -> + ctx.getBean().maxBatchesPerTick shouldBe OutboxSchedulerConfig.DEFAULT_MAX_BATCHES_PER_TICK + } + contextRunner.withPropertyValues("okapi.processor.max-batches-per-tick=0").run { ctx -> + ctx.startupFailure.shouldNotBeNull() + } + } + + test("the scheduler drains up to max-batches-per-tick batches in a tick (#132)") { + listOf(1, 3).forEach { maxBatches -> + val claims = AtomicInteger() + // Every claim comes back full, so only max-batches-per-tick ends the tick. + val alwaysFull = object : RouteAwareOutboxStore by stubStore() { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { + claims.incrementAndGet() + return List(limit) { entryOfType("stub") } + } + + // Delegation would otherwise send the processor's single-type claim to the stub. + override fun claimPending(deliveryType: String, limit: Int, now: Instant) = claimPending(setOf(deliveryType), limit, now) + } + ApplicationContextRunner() + .withConfiguration(AutoConfigurations.of(OutboxAutoConfiguration::class.java)) + .withBean(OutboxStore::class.java, { alwaysFull }) + .withBean(MessageDeliverer::class.java, { stubDeliverer() }) + .withBean(DataSource::class.java, { SimpleDriverDataSource() }) + .withBean(TransactionRunner::class.java, { noOpTransactionRunner() }) + .withPropertyValues( + "okapi.processor.interval=1h", + "okapi.processor.batch-size=2", + "okapi.processor.max-batches-per-tick=$maxBatches", + ) + .run { ctx -> + ctx.startupFailure shouldBe null + val deadline = System.nanoTime() + ofSeconds(5).toNanos() + while (claims.get() < maxBatches && System.nanoTime() < deadline) Thread.sleep(10) + Thread.sleep(300) + withClue("max-batches-per-tick=$maxBatches") { claims.get() shouldBe maxBatches } + } + } + } + test("default properties when nothing is configured") { contextRunner.run { ctx -> val props = ctx.getBean()