Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,23 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *
`OutboxStore.claimPending(limit)` ignores the backoff. Tests or callers that relied on immediate
retries need `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`.
([#131](https://github.com/softwaremill/okapi/issues/131))
- **`OutboxProcessorListener` events are emitted after the transaction commits.** `OutboxProcessor`
notified the listener right after writing a batch, still inside the scheduler's transaction, so a
failed commit (connection loss, serialization failure, timeout) left `okapi.entries.delivered` /
`okapi.entries.failed` incremented for rows that rolled back to `PENDING`, which were then counted
again on redelivery, and a custom listener could act on state that was never persisted.
`OutboxProcessor` now has `processBatch(limit)`, which claims, delivers and persists without
notifying and returns a `ProcessedBatch`, and `publishEvents(batch)`. `OutboxScheduler` runs
`processBatch` in the transaction and `publishEvents` after it returns, so under the scheduler the
listener runs after the commit, outside the transaction, and not at all for a batch whose commit
failed. A listener that relied on running inside the processing transaction (e.g. to write to the
database atomically with the status update) no longer can. `onBatchProcessed`'s duration, and with
it `okapi.batch.duration`, now includes the commit (and excludes the listener callbacks); empty
polls are still reported. Counters now count committed status changes rather than delivery
attempts. `TransactionRunner` now documents that `runInTransaction` must have committed when it
returns, and `publishEvents` throws if a batch is published twice.
`processNext(limit)` keeps notifying at once, for callers that drive the processor themselves.
([#137](https://github.com/softwaremill/okapi/issues/137))
- **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the
`CompositeMessageDeliverer` entry above). `@JvmOverloads` preserves the previous four-argument
JVM constructor, so Java callers are unaffected, but adding a property to a Kotlin `data class`
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ Metrics are then available at `/actuator/prometheus`.
| `okapi.entries.delivered` | Counter | Successfully delivered entries |
| `okapi.entries.retry.scheduled` | Counter | Failed attempts rescheduled for retry |
| `okapi.entries.failed` | Counter | Permanently failed entries |
| `okapi.batch.duration` | Timer | Processing time per batch |
| `okapi.batch.duration` | Timer | Processing time per batch; under the scheduler from the claim until the transaction has committed (not including listener callbacks) |
| `okapi.entries.count` | Gauge | Current entry count (tag: `status`) |
| `okapi.entries.lag.seconds` | Gauge | Age of the oldest entry (tag: `status`) |

Expand All @@ -297,7 +297,7 @@ max by (job, status) (okapi_entries_count)

Without Spring Boot, construct `MicrometerOutboxListener` and `MicrometerOutboxMetrics` with your `MeterRegistry`. Refresh gauges with `OutboxMetricsRefresher` or your own scheduler.

For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own.
For custom reactions to delivery events, implement `OutboxProcessorListener`. `OutboxProcessor` takes a single listener; combine several with a composite of your own. The scheduler calls the listener after the processing transaction has committed, outside of it, and not for a batch whose commit failed, so every event describes persisted state and counters do not over-count when a commit fails. Counters therefore count committed status changes, not delivery attempts: a send whose transaction then rolled back (a failed commit, or a block Exposed retried) is not counted, though the message may have reached the consumer. If you drive `OutboxProcessor` yourself, call `processBatch` inside your transaction and `publishEvents` after it commits; `processNext` does both at once, so inside a transaction its events precede the commit.

## Modules

Expand Down
19 changes: 19 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,25 @@ rows and replaces 002's claim index with one on `(delivery_type, next_attempt_at
that drive retries tick by tick need a zero backoff:
`RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`.

### `OutboxProcessorListener` runs after the commit (#137)

Under `OutboxScheduler` (and so with Spring Boot autoconfiguration), listener callbacks now run after
the processing transaction has committed, outside of it, and are skipped for a batch whose commit
failed.

- A custom listener that wrote to the database expecting to join the processing transaction now
runs without one: open your own transaction in the callback, or make the write idempotent. This
fails quietly: an Exposed listener gets "No transaction in context", a Spring call with
`MANDATORY` propagation throws, and okapi only logs each such exception as a WARN.
- Counters (`okapi.entries.delivered`, `okapi.entries.failed`, ...) now count committed status
changes. A send whose transaction rolled back is no longer counted.
- A custom `TransactionRunner` must have committed when `runInTransaction` returns (now documented on
the interface); one that joins an outer transaction brings back pre-commit events.
- `okapi.batch.duration` now includes the commit, so it reads slightly higher than before.
- If you call `OutboxProcessor.processNext` inside your own transaction, events still fire before
your commit. Switch to `val batch = runInTransaction { processor.processBatch(limit) }` followed by
`processor.publishEvents(batch)` to get the after-commit behaviour.

## From 0.2.x to 1.0.0

Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ class CompositeMessageDeliverer @JvmOverloads constructor(
* satisfy that opt out via [TransportDispatch.SEQUENTIAL].
*
* The outbox transaction is open across delivery, and stays that way: the scheduler wraps the
* whole [OutboxProcessor.processNext] cycle — claim, deliver, update — in one transaction, so
* whole [OutboxProcessor.processBatch] cycle — claim, deliver, update — in one transaction, so
* that `FOR UPDATE SKIP LOCKED` holds its row locks until the results are written. What this
* dispatch changes is which thread the transport runs on, not that scope. The store round-trips
* still run on the calling thread inside the transaction, but a group handed to a virtual
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ import java.util.concurrent.atomic.AtomicInteger
* full batch. Exceptions in the listener are caught and logged — they never break
* processing. Transaction management is the caller's responsibility.
*
* Listener events describe state the caller's transaction has persisted, so they belong after
* it commits (issue #137): call [processBatch] inside the transaction and [publishEvents] after
* it commits, as [OutboxScheduler] does. A commit that fails then emits nothing, and the entries
* are reported again when they are re-processed. [processNext] does both in one call, so inside a
* transaction its events precede the commit.
*
* In the batch processing path, the per-entry `duration` reported in
* [OutboxProcessingEvent] reflects the **wall-clock duration of the whole batch**
* (because transports may overlap their per-entry I/O internally — e.g. Kafka's
Expand All @@ -29,11 +35,21 @@ class OutboxProcessor @JvmOverloads constructor(
private val nextStartingType = AtomicInteger()

/**
* Claims up to [limit] pending entries, processes them as a batch, and persists
* each result. Returns the number of entries processed (0 if the store had nothing).
* Claims up to [limit] pending entries, processes them as a batch, persists each result and
* notifies the listener at once. Returns the number of entries processed (0 if the store had
* nothing). Called inside a transaction, the events precede its commit; use [processBatch] and
* [publishEvents] to emit them only after the commit.
*/
@JvmOverloads
fun processNext(limit: Int = 10): Int = processBatch(limit).also { publishEvents(it) }.count

/**
* Claims up to [limit] pending entries, processes them as a batch and persists each result,
* without notifying the listener. Run it inside the processing transaction and pass the result
* to [publishEvents] once that transaction has committed.
*/
@JvmOverloads
fun processNext(limit: Int = 10): Int {
fun processBatch(limit: Int = 10): ProcessedBatch {
val batchStart = clock.instant()
val routeAwareStore = store as? RouteAwareOutboxStore
?: error(
Expand Down Expand Up @@ -63,19 +79,26 @@ class OutboxProcessor @JvmOverloads constructor(
first + other
}
}
if (claimed.isEmpty()) {
notifyBatch(0, Duration.between(batchStart, clock.instant()))
return 0
}
if (claimed.isEmpty()) return ProcessedBatch(emptyList(), batchStart, Duration.ZERO)

val processed = entryProcessor.processBatch(claimed)
val batchDuration = Duration.between(batchStart, clock.instant())
val deliveryDuration = Duration.between(batchStart, clock.instant())

store.updateAfterProcessingBatch(processed)
processed.forEach { updated -> notifyEntry(updated, batchDuration) }
return ProcessedBatch(processed, batchStart, deliveryDuration)
}

notifyBatch(processed.size, Duration.between(batchStart, clock.instant()))
return processed.size
/**
* Notifies the listener of [batch]: [OutboxProcessorListener.onEntryProcessed] for each entry,
* then [OutboxProcessorListener.onBatchProcessed], whose duration runs from the start of the batch
* until this call, so it includes the commit when called after one, but not the listener
* callbacks. A batch's events can be published only once; a second call throws.
*/
fun publishEvents(batch: ProcessedBatch) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "Call it at most once per batch" is only documented. Calling it twice double-counts every Micrometer counter, which is exactly the class of bug this PR fixes. A one-shot guard is cheap: an AtomicBoolean in ProcessedBatch, with a second call either logged and ignored or failing via check. It would make the new public API harder to misuse.

check(batch.markPublished()) { "Events of this batch were already published" }
val batchDuration = Duration.between(batch.startedAt, clock.instant())
batch.entries.forEach { updated -> notifyEntry(updated, batch.deliveryDuration) }
notifyBatch(batch.count, batchDuration)
}

private fun notifyEntry(updated: OutboxEntry, duration: Duration) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ import java.time.Duration
*
* [OutboxProcessor] accepts a single listener. To combine multiple listeners,
* implement a composite that delegates to each.
*
* Under [OutboxScheduler], callbacks run after the processing transaction has committed, outside of
* it, and not at all for a batch whose commit failed (issue #137). A callback therefore cannot take
* part in that transaction, and every event describes persisted state. With [OutboxProcessor.processNext]
* called inside your own transaction, events still precede your commit.
*/
interface OutboxProcessorListener {
/** Called after each entry is processed (delivered, retried, or failed). */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicBoolean

/**
* Standalone scheduler that periodically calls [OutboxProcessor.processNext].
* Standalone scheduler that periodically calls [OutboxProcessor.processBatch] in a transaction and,
* once it has committed, [OutboxProcessor.publishEvents].
*
* Each worker's call runs inside [transactionRunner]. The runner is required: without a
* surrounding transaction, `FOR UPDATE SKIP LOCKED` releases its row lock at the end of the
Expand Down Expand Up @@ -179,7 +180,9 @@ class OutboxScheduler @JvmOverloads constructor(

private fun processBatch() {
try {
transactionRunner.runInTransaction { outboxProcessor.processNext(config.batchSize) }
val batch = transactionRunner.runInTransaction { outboxProcessor.processBatch(config.batchSize) }
// After the commit (issue #137): a batch whose commit failed is rolled back and is not reported.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix (contract docs): the after-commit guarantee now rests on an unstated TransactionRunner contract: runInTransaction must have committed by the time it returns (it must not join an ambient transaction or defer the commit). The shipped runners meet it on the scheduler's threads (see the review body). But TransactionRunner's KDoc advertises "Standalone: user-provided lambda", and the Spring autoconfig already warns about ticks "joining an outer TX". A custom runner that participates in an outer transaction (Spring REQUIRED under an outer @Transactional, or Exposed transaction {} nested with useNestedTransactions = false) would silently bring back #137. Please spell out the requirement on TransactionRunner (and maybe in ExposedTransactionRunner/SpringTransactionRunner). It would also help to note that Exposed may retry the block, in which case only the attempt that commits is reported.

outboxProcessor.publishEvents(batch)
logger.debug("Outbox processor tick completed [batchSize={}]", config.batchSize)
} catch (e: Exception) {
logger.error("Outbox processor tick failed, will retry at next scheduled interval", e)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package com.softwaremill.okapi.core

import java.time.Duration
import java.time.Instant
import java.util.Collections
import java.util.concurrent.atomic.AtomicBoolean

/**
* The outcome of [OutboxProcessor.processBatch]: [entries] in the state just persisted, whose
* listener events are held back until [OutboxProcessor.publishEvents]. Only [OutboxProcessor]
* creates one; its constructor and timing fields are internal to okapi-core.
*/
class ProcessedBatch internal constructor(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (Java interop / API surface): internal doesn't hide these from Java. javap on the built class shows a plain public ProcessedBatch(List, Instant, Duration) constructor (Kotlin does not mangle internal constructors) and public getters getStartedAt$okapi_okapi_core() / getDeliveryDuration$okapi_okapi_core(). So Java callers can construct one, and those mangled names become de facto ABI. If the goal is "opaque token", a public class with a private constructor and a factory internal to the module (or an interface with an internal implementation) is tighter. Smaller points: no toString, so it logs as ProcessedBatch@1a2b; and entries exposes whatever list OutboxEntryProcessor returned (Java can cast and mutate it). Neither matters much, but this is new public API in a major release, so it's worth deciding now.

entries: List<OutboxEntry>,
internal val startedAt: Instant,
internal val deliveryDuration: Duration,
) {
/** The processed entries, in the state persisted for them. Read-only. */
val entries: List<OutboxEntry> = Collections.unmodifiableList(entries.toList())

/** Number of entries processed (0 if the store had nothing to claim). */
val count: Int get() = entries.size

private val published = AtomicBoolean(false)

/** Returns `true` the first time only. */
internal fun markPublished(): Boolean = published.compareAndSet(false, true)

override fun toString(): String = "ProcessedBatch(count=$count, startedAt=$startedAt, deliveryDuration=$deliveryDuration)"
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ package com.softwaremill.okapi.core
* - `okapi-spring-boot`: wraps Spring's `TransactionTemplate` via `SpringTransactionRunner`
* - `okapi-exposed`: wraps Exposed's `transaction {}` via `ExposedTransactionRunner`
* - Standalone: user-provided lambda
*
* Contract: when [runInTransaction] returns normally, the transaction it ran [block] in has
* committed. [OutboxScheduler] relies on this to emit listener events only for committed state
* (issue #137). A runner that joins an outer transaction instead (e.g. Spring's `REQUIRED` called
* inside `@Transactional`, or a nested Exposed `transaction {}` without nested transactions) breaks
* that. A runner may run [block] more than once, as Exposed does when a statement or the commit
* throws an `SQLException`; only the result of the attempt that committed is returned.
*/
interface TransactionRunner {
fun <T> runInTransaction(block: () -> T): T
Expand Down
Loading