Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,24 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *

### Changed (BREAKING)

- **The scheduler drains the backlog within a tick instead of processing one batch per worker.**
Each worker used to process exactly one batch per tick, capping throughput at about
`concurrency × batchSize / interval`, under 10 messages/s with the defaults, so any sustained
higher publish rate grew the backlog without bound. A worker now keeps processing batches, each in
its own transaction, while the previous one was full, up to the new
`OutboxSchedulerConfig.maxBatchesPerTick` (default 1000, Spring property
`okapi.processor.max-batches-per-tick`). A delivery type with nothing delivered in a batch (all
rescheduled for retry or failed for good) is skipped for the rest of the tick, so a failing or
misconfigured downstream still gets one batch per tick while other types keep draining, and the
tick ends when every type is skipped. Batches of a tick claim only entries due when the tick
started, so a retry waits for the next tick even with a zero backoff. The poll interval now only
sets the idle polling rate: a healthy downstream receives a backlog as fast as okapi can deliver
it, and `onBatchProcessed` / `okapi.batch.duration` fire more often per tick.
`maxBatchesPerTick = 1` restores the previous behaviour. `OutboxSchedulerConfig` and
`OutboxProcessorProperties` gained a constructor parameter: `@JvmOverloads` keeps the previous
Java constructors, but compiled Kotlin code that calls `copy(...)` or a constructor relying on
default arguments fails with `NoSuchMethodError` until recompiled.
([#132](https://github.com/softwaremill/okapi/issues/132))
- **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).**
A retriable failure used to be retried on the very next tick, so with the defaults (`interval`
1s, `max-retries` 5) a message was `FAILED` — terminally — after about six seconds of downstream
Expand Down
11 changes: 7 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,15 @@ In a typical single-DataSource application, all properties are optional. Multi-D
| Property | Default | Description |
|---|---|---|
| `okapi.processor.enabled` | `true` | Set `false` to disable delivery entirely (e.g. on instances that only publish). |
| `okapi.processor.interval` | `1s` | How often the scheduler polls for pending entries. |
| `okapi.processor.batch-size` | `10` | Maximum entries claimed per worker per tick. |
| `okapi.processor.interval` | `1s` | Delay between scheduler ticks, i.e. how often an idle outbox is polled. A tick with a backlog keeps going (see `max-batches-per-tick`), so this does not cap throughput. |
| `okapi.processor.batch-size` | `10` | Maximum entries claimed per batch; each batch runs in its own transaction. |
| `okapi.processor.max-batches-per-tick` | `1000` | Batches each worker processes per tick. A worker continues while the previous batch was full; a delivery type with nothing delivered in a batch (all rescheduled for retry or failed for good) is skipped for the rest of the tick, and the tick ends when every type is skipped. Batches of a tick claim only entries due when the tick started, so a retry waits for the next tick. `1` restores the 1.x one batch per tick, which caps throughput at about `concurrency × batch-size / interval`. |
| `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. |
| `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. |
| `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). |
| `okapi.processor.max-backoff` | `10m` | Upper bound on the delay between retries. |
| `okapi.processor.backoff-jitter` | `0.2` | Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. |
| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see [Performance](#performance). |
| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each draining its own batches. Tune based on database capacity and delivery latency; see [Performance](#performance). |
| `okapi.processor.transport-dispatch` | `parallel` | How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. |

### Purger
Expand Down Expand Up @@ -238,7 +239,7 @@ okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liqui

## Transactions

Each processor worker runs its claim, delivery and state update inside one transaction, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed.
Each processor worker runs its claim, delivery and state update inside one transaction per batch, which keeps `FOR UPDATE SKIP LOCKED` active until delivery state is saved. While a backlog remains, a worker runs batch after batch within one tick, each in a transaction of its own. Each purge batch also runs in its own transaction. With a single `DataSource` and `PlatformTransactionManager`, no additional transaction configuration is needed.

**Multiple data sources or transaction managers?** Set both `okapi.datasource-qualifier` and `okapi.transaction-manager-qualifier` to the beans used for the outbox. okapi fails fast when it detects a mismatch. If the selected transaction manager does not expose its `DataSource`, okapi cannot verify the pairing and logs a warning instead.

Expand Down Expand Up @@ -342,6 +343,8 @@ The storage modules use plain JDBC. With Spring Boot, they participate in the tr

Performance depends on the selected transport, database, batch size, concurrency and downstream latency. See [`benchmarks/`](benchmarks/) for methodology and measured results.

Each tick drains: a worker keeps claiming batches while they come back full, so a backlog is delivered as fast as the database and transports allow, and `interval` only sets how quickly a new message is picked up when the outbox was idle. A failing or misconfigured downstream gets one batch per worker per tick, as before, while other delivery types keep draining. To limit the rate at which okapi sends to a healthy downstream system, set `max-batches-per-tick` (with `batch-size` and `interval`) rather than relying on the poll interval alone.

## Building

```sh
Expand Down
21 changes: 21 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,27 @@ failed.
your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by
`processor.publishEvents(batch)` to get the after-commit behaviour.

### The scheduler drains the backlog in each tick (#132)

A worker no longer stops after one batch per tick: it keeps processing batches, each in its own
transaction, while they come back full, up to `max-batches-per-tick` (default 1000). A delivery
type with nothing delivered in a batch (all rescheduled for retry, or failed for good, e.g. after a
401) is skipped for the rest of the tick, so a failing or misconfigured downstream still gets one
batch per worker per tick, as in 1.x.

- **Downstream load.** With a backlog, okapi now sends as fast as the database and transports allow.
If you relied on `batch-size` / `interval` to throttle a rate-limited downstream system, set
`okapi.processor.max-batches-per-tick=1` (`OutboxSchedulerConfig(maxBatchesPerTick = 1)`) to keep
the 1.x rate, or a small value to allow some draining.
- **Database load.** A backlog is now worked through in back-to-back transactions instead of one
per interval. Size `concurrency` with that in mind.
- **Metrics.** `onBatchProcessed` and `okapi.batch.duration` are still per batch, so a tick can now
report many batches.
- **Code.** `OutboxSchedulerConfig` and `OutboxProcessorProperties` gained a `maxBatchesPerTick`
parameter. Java callers keep the previous constructors (`@JvmOverloads`); compiled Kotlin code that
calls `copy(...)` or a constructor relying on default arguments must be recompiled, or it fails
with `NoSuchMethodError`.

## From 0.2.x to 1.0.0

Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package com.softwaremill.okapi.core
import org.slf4j.LoggerFactory
import java.time.Clock
import java.time.Duration
import java.time.Instant
import java.util.concurrent.atomic.AtomicInteger

/**
Expand Down Expand Up @@ -49,30 +50,42 @@ class OutboxProcessor @JvmOverloads constructor(
* to [publishEvents] once that transaction has committed.
*/
@JvmOverloads
fun processBatch(limit: Int = 10): ProcessedBatch {
fun processBatch(limit: Int = 10): ProcessedBatch = processBatch(limit, claimTime = null, skipTypes = emptySet())

/** Delivery types this processor claims. */
internal val supportedDeliveryTypes: Set<String> get() = deliveryTypes.toSet()

/**
* [processBatch] for [OutboxScheduler]'s drain loop (issue #132): claims only entries due at
* [claimTime] (the start of the batch when `null`), so a retry rescheduled during a tick waits
* for the next tick, and leaves out [skipTypes].
*/
internal fun processBatch(limit: Int, claimTime: Instant?, skipTypes: Set<String>): ProcessedBatch {
val batchStart = clock.instant()
val claimAt = claimTime ?: batchStart
val types = deliveryTypes.filterNot { it in skipTypes }
val routeAwareStore = store as? RouteAwareOutboxStore
?: error(
"OutboxProcessor requires a RouteAwareOutboxStore to avoid claiming unsupported delivery types, " +
"got ${store.javaClass.name}. A wrapper or proxy around a store must implement RouteAwareOutboxStore itself.",
)
check(deliveryTypes.isNotEmpty()) { "OutboxProcessor requires at least one delivery type" }
val claimed = if (limit <= 0) {
val claimed = if (limit <= 0 || types.isEmpty()) {
emptyList()
} else {
val startingType = deliveryTypes[Math.floorMod(nextStartingType.getAndIncrement(), deliveryTypes.size)]
val first = routeAwareStore.claimPending(startingType, limit, batchStart)
val startingType = types[Math.floorMod(nextStartingType.getAndIncrement(), types.size)]
val first = routeAwareStore.claimPending(startingType, limit, claimAt)
check(first.size <= limit && first.all { it.deliveryType == startingType }) {
"RouteAwareOutboxStore returned entries outside the requested type or limit"
}
val remainingTypes = deliveryTypes.filterTo(LinkedHashSet()) { it != startingType }
val remainingTypes = types.filterTo(LinkedHashSet()) { it != startingType }
if (first.size == limit || remainingTypes.isEmpty()) {
first
} else {
val remaining = limit - first.size
// Stores order this claim by delivery type, so leftover capacity favours types that sort first.
// Each type is still claimed first once every N ticks, so none is starved.
val other = routeAwareStore.claimPending(remainingTypes, remaining, batchStart)
val other = routeAwareStore.claimPending(remainingTypes, remaining, claimAt)
check(other.size <= remaining && other.all { it.deliveryType in remainingTypes }) {
"RouteAwareOutboxStore returned entries outside the requested types or limit"
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package com.softwaremill.okapi.core

import org.slf4j.LoggerFactory
import java.time.Instant
import java.util.concurrent.CancellationException
import java.util.concurrent.ExecutionException
import java.util.concurrent.ExecutorService
Expand All @@ -15,6 +16,16 @@ import java.util.concurrent.atomic.AtomicBoolean
* Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and,
* once it has committed, [OutboxProcessor.publishEvents].
*
* Each tick drains (issue #132): a worker keeps processing batches, each in its own transaction,
* while the previous one was full, up to [OutboxSchedulerConfig.maxBatchesPerTick].
* [OutboxSchedulerConfig.interval] is the delay after a tick, so it sets the polling rate of an
* idle outbox, not peak throughput. A delivery type none of whose entries in a batch was delivered
* (all rescheduled for retry or failed for good) is skipped for the rest of the worker's tick, so a
* failing or misconfigured downstream gets one batch per tick, as in 1.x, while healthy types keep
* draining; the tick ends once every type is skipped. All batches of a tick claim entries due at
* the tick's start, so an entry retried during the tick waits for the next one even with a zero
* backoff. [stop] lets the batch in progress finish and starts no further one.
*
* Each worker's call runs inside [transactionRunner]. The runner is required: without a
* surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the
* claim SELECT and concurrent processor instances deliver the same entry multiple times.
Expand All @@ -23,7 +34,7 @@ import java.util.concurrent.atomic.AtomicBoolean
* [OutboxSchedulerConfig.workerExecutorFactory]'s pool. `FOR UPDATE SKIP LOCKED` guarantees
* disjoint claims, but [outboxProcessor] (and its dependencies) must be safe for concurrent use.
* The tick waits for every worker before the next scheduled interval, so ticks never overlap.
* `concurrency = 1` (default) calls the batch inline, preserving single-worker semantics.
* `concurrency = 1` (default) runs the worker inline, preserving single-worker semantics.
*
* At `concurrency > 1`, [outboxProcessor] -- and by extension its [OutboxStore], [MessageDeliverer],
* and any [OutboxProcessorListener] -- is invoked concurrently from multiple worker threads, so all
Expand Down Expand Up @@ -59,10 +70,11 @@ class OutboxScheduler @JvmOverloads constructor(
check(!scheduler.isShutdown) { "OutboxScheduler cannot be restarted after stop()" }
if (!running.compareAndSet(false, true)) return
logger.info(
"Outbox processor started [interval={}, batchSize={}, concurrency={}]",
"Outbox processor started [interval={}, batchSize={}, concurrency={}, maxBatchesPerTick={}]",
config.interval,
config.batchSize,
config.concurrency,
config.maxBatchesPerTick,
)
scheduler.scheduleWithFixedDelay(::tick, 0L, config.interval.toMillis(), TimeUnit.MILLISECONDS)
}
Expand Down Expand Up @@ -110,7 +122,7 @@ class OutboxScheduler @JvmOverloads constructor(
if (config.concurrency == 1) {
// Skips workers/Lazy entirely -- not just an inline no-op path -- so this configuration
// truly pays no executor/synchronization overhead per tick, matching the class KDoc.
processBatch()
drain()
return
}
val pool = try {
Expand All @@ -130,13 +142,13 @@ class OutboxScheduler @JvmOverloads constructor(
}

/**
* Submits one worker's [processBatch] without letting [RejectedExecutionException] escape
* Submits one worker's [drain] without letting [RejectedExecutionException] escape
* [tick] -- a custom [OutboxSchedulerConfig.workerExecutorFactory] may return an executor that
* can reject submissions (bounded queue, shutdown executor), and an uncaught exception here
* would make `scheduleWithFixedDelay` suppress all future ticks.
*/
private fun submitWorker(pool: ExecutorService): Future<*>? = try {
pool.submit(::processBatch)
pool.submit(::drain)
} catch (e: RejectedExecutionException) {
logger.error("Outbox worker submission rejected, will retry at next scheduled interval", e)
null
Expand All @@ -145,7 +157,7 @@ class OutboxScheduler @JvmOverloads constructor(
/**
* Awaits one worker's completion without letting anything escape [tick] -- an uncaught
* exception here would make `scheduleWithFixedDelay` suppress all future ticks, silently
* killing the scheduler. [processBatch] already catches [Exception] internally, so this
* killing the scheduler. [drain] already catches [Exception] internally, so this
* only guards against an [Error] surfacing via [ExecutionException], a [CancellationException]
* if a custom [OutboxSchedulerConfig.workerExecutorFactory] cancels queued/running tasks (e.g.
* as a backpressure strategy), or an interrupt while awaiting.
Expand Down Expand Up @@ -178,17 +190,52 @@ class OutboxScheduler @JvmOverloads constructor(
}
}

private fun processBatch() {
try {
val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) }
// After the commit (issue #137): a batch whose commit failed is rolled back and is not reported.
outboxProcessor.publishEvents(batch)
logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize)
} catch (e: Exception) {
logger.error("Outbox processor tick failed, will retry at next scheduled interval", e)
/** One worker's share of a tick: processes batches until [shouldContinue] says to stop. */
private fun drain() {
var batches = 0
var claimTime: Instant? = null
val stalledTypes = mutableSetOf<String>()
while (true) {
val batch = processBatch(claimTime, stalledTypes, batches) ?: return
batches++
claimTime = claimTime ?: batch.startedAt
// A type with nothing delivered in this batch is failing (or misconfigured): no more of it this tick.
batch.entries.groupBy { it.deliveryType }
.filterValues { entries -> entries.none { it.status == OutboxStatus.DELIVERED } }
.keys.let(stalledTypes::addAll)
if (!shouldContinue(batch, batches, stalledTypes)) {
logger.debug(
"Outbox processor tick completed [batches={}, lastBatchSize={}, skippedTypes={}]",
batches,
batch.count,
stalledTypes,
)
return
}
}
}

private fun shouldContinue(batch: ProcessedBatch, batches: Int, stalledTypes: Set<String>): Boolean = batch.count >= config.batchSize &&
batches < config.maxBatchesPerTick &&
running.get() &&
!Thread.currentThread().isInterrupted &&
!stalledTypes.containsAll(outboxProcessor.supportedDeliveryTypes)

/** Processes one batch in its own transaction; returns `null` if that failed. */
private fun processBatch(claimTime: Instant?, skipTypes: Set<String>, committedBatches: Int): ProcessedBatch? = try {
val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize, claimTime, skipTypes) }
// After the commit (issue #137): a batch whose commit failed is rolled back and is not reported.
outboxProcessor.publishEvents(batch)
batch
} catch (e: Exception) {
logger.error(
"Outbox processor batch failed after {} committed batch(es) in this tick, will retry at next scheduled interval",
committedBatches,
e,
)
null
}

companion object {
private val logger = LoggerFactory.getLogger(OutboxScheduler::class.java)
private const val SHUTDOWN_TIMEOUT_SECONDS = 5L
Expand Down
Loading