diff --git a/CHANGELOG.md b/CHANGELOG.md index 24ebbb2b..60925d89 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,26 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with * ### Changed (BREAKING) +- **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).** + A retriable failure used to be retried on the very next tick, so with the defaults (`interval` + 1s, `max-retries` 5) a message was `FAILED` — terminally — after about six seconds of downstream + outage, and failing rows, always the oldest, were re-claimed first on every tick. Each entry now + carries `nextAttemptAt` (column `next_attempt_at`); a retriable failure sets it `RetryPolicy`'s + backoff into the future, and the processor claims only due entries, ordered by + `(delivery_type, next_attempt_at, id)` through a new index that replaces 002's. The default backoff + is 10s, doubling, capped at 10m, with 20% jitter, which covers about five minutes of outage before + `FAILED`. New Spring properties `okapi.processor.initial-backoff`, `backoff-multiplier`, + `max-backoff` and `backoff-jitter`. API changes: `RetryPolicy` gained `initialBackoff`, + `backoffMultiplier`, `maxBackoff` and `jitter` (all defaulted, `@JvmOverloads`) and `backoff(retry)`; + `OutboxEntry` gained `nextAttemptAt` (defaults to `createdAt`) and `retry` an optional + `nextAttemptAt`; `RouteAwareOutboxStore.claimPending` takes the current time (`now`); and + `OutboxProcessorProperties` gained four parameters. `OutboxEntry`'s constructor is `@JvmOverloads`, + so Java callers can still omit `nextAttemptAt`. As with every data-class change, Kotlin code + calling `copy(...)` or default-argument constructors of these classes must be recompiled; custom + `RouteAwareOutboxStore`s must filter on and write `next_attempt_at`. The legacy + `OutboxStore.claimPending(limit)` ignores the backoff. Tests or callers that relied on immediate + retries need `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. + ([#131](https://github.com/softwaremill/okapi/issues/131)) - **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the `CompositeMessageDeliverer` entry above). `@JvmOverloads` preserves the previous four-argument JVM constructor, so Java callers are unaffected, but adding a property to a Kotlin `data class` diff --git a/README.md b/README.md index 2981ead9..373ae702 100644 --- a/README.md +++ b/README.md @@ -92,12 +92,12 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co ### Guarantees and limits - **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id). -- **Best-effort ordering.** Rows are claimed by `created_at, id` within each delivery type. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed. +- **Best-effort ordering.** Rows are claimed by due time (`next_attempt_at`, then `id`) within each delivery type: a new row is due when it is created, a retried one once its backoff has passed, so a failing row does not hold up newer ones. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed. - **Different workers can handle different delivery types.** Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom `OutboxStore` used for processing must implement `RouteAwareOutboxStore`; otherwise processing fails before an unsafe claim. The older `claimPending(limit)` method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself. - **Rolling upgrades need sequencing.** Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked `FAILED` by older workers. - **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not. - **On MySQL, scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` (with `MysqlOutboxStore`) and `ExposedTransactionRunner` (on a MySQL/MariaDB dialect) both use it. At MySQL's default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner`, an `OutboxStore` of your own, or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. With a `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink `JpaDialect`, okapi cannot set it (startup logs a WARN): configure `transaction_isolation = 'READ-COMMITTED'` on the server or in the JDBC URL instead. `READ COMMITTED` requires `binlog_format=ROW` (the default since MySQL 5.7.7) or `MIXED`; with `STATEMENT`, InnoDB rejects the scheduler's writes with `ERROR 1665`. -- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff. +- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. A retriable failure is retried after an exponential backoff with jitter: `okapi.processor.initial-backoff` (default 10s) before the first retry, `backoff-multiplier` (default 2) times longer for each further one, capped at `max-backoff` (default 10m) and shortened at random by up to `backoff-jitter` (default 0.2). With the defaults, a message survives roughly five minutes of downstream outage before it becomes `FAILED`; raise `max-retries` to cover longer ones (e.g. 10 retries span about 50 minutes). The backoff only applies to rows claimed by 2.0 workers. ### Deduplicating on `x-outbox-id` @@ -135,6 +135,10 @@ In a typical single-DataSource application, all properties are optional. Multi-D | `okapi.processor.interval` | `1s` | How often the scheduler polls for pending entries. | | `okapi.processor.batch-size` | `10` | Maximum entries claimed per worker per tick. | | `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. | +| `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. | +| `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). | +| `okapi.processor.max-backoff` | `10m` | Upper bound on the delay between retries. | +| `okapi.processor.backoff-jitter` | `0.2` | Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. | | `okapi.processor.concurrency` | `1` | Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see [Performance](#performance). | | `okapi.processor.transport-dispatch` | `parallel` | How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. | @@ -215,10 +219,10 @@ With `okapi-spring-boot` and Liquibase on the classpath, these run automatically If you use another migration tool, apply every SQL migration for your database, in order: -- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/003__timestamptz_columns.sql) `TIMESTAMPTZ` timestamp columns -- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns +- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/003__timestamptz_columns.sql) `TIMESTAMPTZ` timestamp columns, [004](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql) retry backoff column and claim index (its second changeset must run outside a transaction) +- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns, [004](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql) retry backoff column and claim index -For an existing installation, apply `002` before starting updated workers. MySQL claims explicitly select the new index and fail if it is missing. PostgreSQL's `002` uses `CREATE INDEX CONCURRENTLY`, so run it outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout. +For an existing installation, apply every new migration (up to `004`) before starting updated workers. MySQL claims explicitly select the claim index from `004` and fail if it is missing. PostgreSQL's `002` and the index changeset of `004` use `CREATE INDEX CONCURRENTLY`, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout. PostgreSQL's `003` converts the timestamp columns from `TIMESTAMP` (the okapi JVM's local wall-clock time) to `TIMESTAMPTZ`. It reads existing values in the session `TimeZone`; pgjdbc sets that to the JVM default zone, so the bundled Liquibase changelog converts them correctly when it runs inside the application. When applying it by hand, run it inside a transaction (`psql -1`) after `SET TIME ZONE ''` in the same session. It takes an `ACCESS EXCLUSIVE` lock: it first waits (up to 30 s, then fails and retries on the next start) for open transactions on `okapi_outbox`, queuing every other statement on the table meanwhile, and then, unless the session zone is UTC (PostgreSQL 12+), rewrites the table while reads and writes wait. 1.x instances keep working after it, apart from one failed poll per pooled connection. See UPGRADING.md for when the in-app conversion uses the wrong zone. See [UPGRADING.md](UPGRADING.md#from-1x-to-200). diff --git a/UPGRADING.md b/UPGRADING.md index b9b50c35..ac724b01 100644 --- a/UPGRADING.md +++ b/UPGRADING.md @@ -72,6 +72,39 @@ them, and the conversion reads them in the session `TimeZone`: correct too. A claim query a 1.x connection prepared before the migration fails once with `cached plan must not change result type`; that tick is retried on the next poll. +### Retry backoff and `next_attempt_at` (#131) + +Changeset `004__retry_backoff` adds `next_attempt_at`, sets it to `created_at` on existing `PENDING` +rows and replaces 002's claim index with one on `(delivery_type, next_attempt_at, id)`. + +- **PostgreSQL:** the column is `NOT NULL DEFAULT CURRENT_TIMESTAMP` (added without a table rewrite + on PostgreSQL 11+), so rows that 1.x instances insert during a rolling deploy are claimable at once. + The `ADD COLUMN` holds an `ACCESS EXCLUSIVE` lock only for itself (waiting at most 30 s for it); the + backfill of `PENDING` rows runs in a separate changeset with row locks only. The index is built in a second changeset with `CREATE INDEX CONCURRENTLY`, which must run outside a + transaction when you apply the script by hand, and which waits for every transaction that started + before it. +- **MySQL:** the column is nullable (an instant change on 8.0.12+); `PENDING` and `FAILED` rows are + backfilled, `DELIVERED` rows stay `NULL`. 1.x instances are already stopped for 003, so nothing + writes rows without it. Each step is a changeset of its own with a precondition, so a migration that + stopped half-way (e.g. on its 30 s `lock_wait_timeout`) resumes where it stopped. +- **Behaviour:** retriable failures now wait 10s, 20s, 40s, ... (capped at 10m, 20% jitter) before + the next attempt, so with `max-retries` 5 an entry becomes `FAILED` after about five minutes rather + than six seconds. Set `okapi.processor.initial-backoff`, `backoff-multiplier`, `max-backoff`, + `backoff-jitter` or `max-retries` to change that. 1.x workers still running during a rolling deploy + ignore the backoff for the rows they claim. +- **Clocks:** a new entry is due at its `createdAt`, taken from the publishing instance's clock, and + claimed once the processing instance's clock reaches it, so a publisher whose clock runs ahead of + the processors' delays its messages by the difference. Keep instance clocks NTP-synchronised. +- **Requeuing a `FAILED` row by hand** now needs a due time as well: + `UPDATE okapi_outbox SET status = 'PENDING', retries = 0, next_attempt_at = WHERE ...` + (`CURRENT_TIMESTAMP` on PostgreSQL, `UTC_TIMESTAMP(6)` on MySQL). A row whose `next_attempt_at` + lies in the future is not claimed until then. +- **Code:** a custom `RouteAwareOutboxStore` must implement `claimPending(deliveryTypes, limit, now)`, + return only rows with `next_attempt_at <= now`, and write `nextAttemptAt` in `persist` and the update + methods. `OutboxEntry` keeps a Java constructor without `nextAttemptAt` (`@JvmOverloads`). Tests + that drive retries tick by tick need a zero backoff: + `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`. + ## From 0.2.x to 1.0.0 Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntry.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntry.kt index 176e455f..2e7a6ec2 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntry.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntry.kt @@ -8,7 +8,7 @@ import java.time.Instant * Created via [createPending] and progressed through [retry], [toDelivered], * or [toFailed] — each returning a new immutable copy. */ -data class OutboxEntry( +data class OutboxEntry @JvmOverloads constructor( @get:JvmName("getOutboxId") val outboxId: OutboxId, val messageType: String, @@ -21,14 +21,24 @@ data class OutboxEntry( val lastAttempt: Instant?, val lastError: String?, val deliveryMetadata: String, + /** + * When a `PENDING` entry becomes eligible for its next delivery attempt (issue #131): [createdAt] + * for a new entry, later after a retriable failure. Only meaningful while the entry is `PENDING`. + */ + val nextAttemptAt: Instant = createdAt, ) { - /** Returns a copy scheduled for another delivery attempt. */ - fun retry(now: Instant, lastError: String): OutboxEntry = copy( + /** + * Returns a copy scheduled for another delivery attempt at [nextAttemptAt], which defaults to + * [now] (retry on the next poll). + */ + @JvmOverloads + fun retry(now: Instant, lastError: String, nextAttemptAt: Instant = now): OutboxEntry = copy( status = OutboxStatus.PENDING, updatedAt = now, retries = retries + 1, lastAttempt = now, lastError = sanitizeLastError(lastError), + nextAttemptAt = nextAttemptAt, ) /** Returns a copy marked as permanently failed. */ diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntryProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntryProcessor.kt index fbf00c43..1d98ec4b 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntryProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxEntryProcessor.kt @@ -31,15 +31,18 @@ class OutboxEntryProcessor( check(entries.all { it.deliveryType in supportedDeliveryTypes }) { "Batch contains an entry with no registered deliverer" } + val outcomes = deliverer.deliverBatch(entries) + // Read after delivery, like process(): a retry's backoff counts from when the attempt ended, so + // a slow failure (e.g. a 30 s HTTP timeout) does not eat into it (issue #131). val now = clock.instant() - return deliverer.deliverBatch(entries).map { (entry, result) -> applyResult(entry, result, now) } + return outcomes.map { (entry, result) -> applyResult(entry, result, now) } } private fun applyResult(entry: OutboxEntry, result: DeliveryResult, now: java.time.Instant): OutboxEntry = when (result) { is DeliveryResult.Success -> entry.toDelivered(now) is DeliveryResult.RetriableFailure -> if (retryPolicy.shouldRetry(entry.retries)) { - entry.retry(now, result.error) + entry.retry(now, result.error, nextAttemptAt = now + retryPolicy.backoff(entry.retries + 1)) } else { entry.toFailed(now, result.error) } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt index c4f76f65..8afb39c3 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxProcessor.kt @@ -45,7 +45,7 @@ class OutboxProcessor @JvmOverloads constructor( emptyList() } else { val startingType = deliveryTypes[Math.floorMod(nextStartingType.getAndIncrement(), deliveryTypes.size)] - val first = routeAwareStore.claimPending(startingType, limit) + val first = routeAwareStore.claimPending(startingType, limit, batchStart) check(first.size <= limit && first.all { it.deliveryType == startingType }) { "RouteAwareOutboxStore returned entries outside the requested type or limit" } @@ -56,7 +56,7 @@ class OutboxProcessor @JvmOverloads constructor( val remaining = limit - first.size // Stores order this claim by delivery type, so leftover capacity favours types that sort first. // Each type is still claimed first once every N ticks, so none is starved. - val other = routeAwareStore.claimPending(remainingTypes, remaining) + val other = routeAwareStore.claimPending(remainingTypes, remaining, batchStart) check(other.size <= remaining && other.all { it.deliveryType in remainingTypes }) { "RouteAwareOutboxStore returned entries outside the requested types or limit" } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxStore.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxStore.kt index 87068d3b..a5ada0ed 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxStore.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/OutboxStore.kt @@ -7,12 +7,13 @@ interface OutboxStore { fun persist(entry: OutboxEntry): OutboxEntry /** - * Claims up to [limit] PENDING entries without filtering by delivery type. - * Kept for direct callers; [OutboxProcessor] uses [RouteAwareOutboxStore] instead. + * Claims up to [limit] PENDING entries without filtering by delivery type or by + * [OutboxEntry.nextAttemptAt], so it ignores retry backoff. Kept for direct callers; + * [OutboxProcessor] uses [RouteAwareOutboxStore] instead. */ fun claimPending(limit: Int): List - /** Updates an entry after a delivery attempt (status change, retries, lastError). */ + /** Updates an entry after a delivery attempt (status change, retries, lastAttempt, lastError, nextAttemptAt). */ fun updateAfterProcessing(entry: OutboxEntry): OutboxEntry /** diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RetryPolicy.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RetryPolicy.kt index b0052b0d..00851184 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RetryPolicy.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RetryPolicy.kt @@ -1,11 +1,58 @@ package com.softwaremill.okapi.core -/** Determines how many retries are allowed after the initial delivery attempt before an entry is marked as [OutboxStatus.FAILED]. */ -data class RetryPolicy(val maxRetries: Int) { +import java.time.Duration +import java.util.concurrent.ThreadLocalRandom +import java.util.random.RandomGenerator + +/** + * Decides whether a retriable failure is retried and when (issue #131). + * + * [maxRetries] counts retries after the initial delivery attempt; once they are used up the entry is + * marked [OutboxStatus.FAILED]. Retry `n` (1-based) is due [initialBackoff] × [backoffMultiplier]^(n-1) + * after the failed attempt, capped at [maxBackoff] and shortened by up to [jitter] (a fraction in + * `[0, 1]`) at random, so entries that failed together do not all retry in the same tick. With the + * defaults, five retries span roughly five minutes of downstream outage before an entry fails. + */ +data class RetryPolicy @JvmOverloads constructor( + val maxRetries: Int, + val initialBackoff: Duration = DEFAULT_INITIAL_BACKOFF, + val backoffMultiplier: Double = DEFAULT_BACKOFF_MULTIPLIER, + val maxBackoff: Duration = DEFAULT_MAX_BACKOFF, + val jitter: Double = DEFAULT_JITTER, +) { init { require(maxRetries >= 0) { "maxRetries must be >= 0, got: $maxRetries" } + require(!initialBackoff.isNegative) { "initialBackoff must be >= 0, got: $initialBackoff" } + require( + backoffMultiplier >= 1.0 && backoffMultiplier.isFinite(), + ) { "backoffMultiplier must be finite and >= 1, got: $backoffMultiplier" } + require(maxBackoff >= initialBackoff) { "maxBackoff ($maxBackoff) must be >= initialBackoff ($initialBackoff)" } + require(maxBackoff <= MAX_ALLOWED_BACKOFF) { "maxBackoff must be at most $MAX_ALLOWED_BACKOFF, got: $maxBackoff" } + require(jitter in 0.0..1.0) { "jitter must be between 0 and 1, got: $jitter" } } /** Returns `true` if [currentRetries] has not yet reached [maxRetries]. */ fun shouldRetry(currentRetries: Int): Boolean = currentRetries < maxRetries + + /** How long to wait before retry number [retry] (1 for the first retry), jitter applied. */ + @JvmOverloads + fun backoff(retry: Int, random: RandomGenerator = ThreadLocalRandom.current()): Duration { + require(retry >= 1) { "retry must be >= 1, got: $retry" } + // In nanoseconds as doubles: no sub-millisecond truncation, and pow() overflowing to infinity + // is simply capped. MAX_ALLOWED_BACKOFF keeps the result inside Duration.ofNanos's range. + val exponential = initialBackoff.toNanos() * Math.pow(backoffMultiplier, (retry - 1).toDouble()) + val capped = minOf(exponential, maxBackoff.toNanos().toDouble()) + val jittered = capped * (1.0 - jitter * random.nextDouble()) + return Duration.ofNanos(jittered.toLong()) + } + + companion object { + val DEFAULT_INITIAL_BACKOFF: Duration = Duration.ofSeconds(10) + const val DEFAULT_BACKOFF_MULTIPLIER: Double = 2.0 + val DEFAULT_MAX_BACKOFF: Duration = Duration.ofMinutes(10) + const val DEFAULT_JITTER: Double = 0.2 + + /** Upper bound on [maxBackoff]. */ + val MAX_ALLOWED_BACKOFF: Duration = Duration.ofDays(365) + } } diff --git a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxStore.kt b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxStore.kt index f036c824..8d66cefe 100644 --- a/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxStore.kt +++ b/okapi-core/src/main/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxStore.kt @@ -1,14 +1,16 @@ package com.softwaremill.okapi.core +import java.time.Instant + /** - * Claims only entries for delivery types handled by the current worker. + * Claims only entries for delivery types handled by the current worker, and only those that are due. * The returned rows must be locked until the processing transaction completes. - * Implementations should return at most the requested number of rows ordered by delivery type, - * creation time and ID. + * Implementations should return at most the requested number of `PENDING` rows whose + * [OutboxEntry.nextAttemptAt] is at or before `now`, ordered by delivery type, next attempt time and ID. * The processor rejects results with another [OutboxEntry.deliveryType] or more than requested. */ interface RouteAwareOutboxStore : OutboxStore { - fun claimPending(deliveryTypes: Set, limit: Int): List + fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List - fun claimPending(deliveryType: String, limit: Int): List = claimPending(setOf(deliveryType), limit) + fun claimPending(deliveryType: String, limit: Int, now: Instant): List = claimPending(setOf(deliveryType), limit, now) } diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxProcessorTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxProcessorTest.kt index bc177783..5edc6f3d 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxProcessorTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxProcessorTest.kt @@ -35,7 +35,7 @@ class OutboxProcessorTest : override fun claimPending(limit: Int): List = pendingEntries.take(limit) - override fun claimPending(deliveryTypes: Set, limit: Int): List = + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = pendingEntries.filter { it.deliveryType in deliveryTypes }.take(limit) override fun updateAfterProcessing(entry: OutboxEntry): OutboxEntry = entry.also { processedEntries += it } @@ -340,7 +340,7 @@ class OutboxProcessorTest : val batchAwareStore = object : RouteAwareOutboxStore { override fun persist(entry: OutboxEntry) = entry override fun claimPending(limit: Int): List = pendingEntries.take(limit) - override fun claimPending(deliveryTypes: Set, limit: Int): List = + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List = pendingEntries.filter { it.deliveryType in deliveryTypes }.take(limit) override fun updateAfterProcessing(entry: OutboxEntry): OutboxEntry = entry.also { individualCalls += it } override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt index e250ead7..b2f88a66 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/OutboxSchedulerTest.kt @@ -11,6 +11,7 @@ import io.kotest.matchers.shouldBe import org.slf4j.LoggerFactory import java.time.Duration.ofMillis import java.time.Duration.ofMinutes +import java.time.Instant import java.util.concurrent.CancellationException import java.util.concurrent.CountDownLatch import java.util.concurrent.ExecutorService @@ -692,7 +693,7 @@ private fun stubProcessor(onProcessNext: (Int) -> Unit): OutboxProcessor { onProcessNext(limit) return emptyList() } - override fun claimPending(deliveryTypes: Set, limit: Int): List { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { onProcessNext(limit) return emptyList() } diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RetryPolicyTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RetryPolicyTest.kt new file mode 100644 index 00000000..1d1a1cd2 --- /dev/null +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RetryPolicyTest.kt @@ -0,0 +1,126 @@ +package com.softwaremill.okapi.core + +import io.kotest.assertions.throwables.shouldThrow +import io.kotest.core.spec.style.FunSpec +import io.kotest.matchers.comparables.shouldBeGreaterThanOrEqualTo +import io.kotest.matchers.comparables.shouldBeLessThanOrEqualTo +import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration +import java.time.Instant +import java.time.ZoneOffset +import java.util.random.RandomGenerator + +/** A [RandomGenerator] whose `nextDouble()` always returns [value]. */ +private fun fixedRandom(value: Double) = object : RandomGenerator { + override fun nextLong(): Long = 0 + override fun nextDouble(): Double = value +} + +class RetryPolicyTest : FunSpec({ + val noJitter = RetryPolicy( + maxRetries = 10, + initialBackoff = Duration.ofSeconds(10), + backoffMultiplier = 2.0, + maxBackoff = Duration.ofMinutes(1), + jitter = 0.0, + ) + + test("backoff grows exponentially from initialBackoff and is capped at maxBackoff") { + (1..5).map { noJitter.backoff(it) } shouldBe listOf( + Duration.ofSeconds(10), + Duration.ofSeconds(20), + Duration.ofSeconds(40), + Duration.ofMinutes(1), + Duration.ofMinutes(1), + ) + } + + test("a large retry number does not overflow") { + noJitter.backoff(10_000) shouldBe Duration.ofMinutes(1) + } + + test("jitter shortens the delay by at most the jitter fraction") { + val policy = noJitter.copy(jitter = 0.2) + policy.backoff(1, fixedRandom(0.0)) shouldBe Duration.ofSeconds(10) + policy.backoff(1, fixedRandom(0.5)) shouldBe Duration.ofSeconds(9) + repeat(1_000) { + val delay = policy.backoff(2) + delay shouldBeGreaterThanOrEqualTo Duration.ofSeconds(16) + delay shouldBeLessThanOrEqualTo Duration.ofSeconds(20) + } + } + + test("defaults give a retry window of minutes, not seconds") { + val policy = RetryPolicy(maxRetries = 5) + val total = (1..5).map { policy.backoff(it, fixedRandom(0.0)) }.reduce(Duration::plus) + total shouldBe Duration.ofSeconds(10 + 20 + 40 + 80 + 160) + } + + test("invalid settings are rejected") { + shouldThrow { RetryPolicy(1, initialBackoff = Duration.ofSeconds(-1)) } + shouldThrow { RetryPolicy(1, backoffMultiplier = 0.5) } + shouldThrow { RetryPolicy(1, initialBackoff = Duration.ofMinutes(2), maxBackoff = Duration.ofMinutes(1)) } + shouldThrow { RetryPolicy(1, jitter = 1.5) } + shouldThrow { noJitter.backoff(0) } + shouldThrow { RetryPolicy(1, maxBackoff = Duration.ofDays(366)) } + shouldThrow { RetryPolicy(1, backoffMultiplier = Double.POSITIVE_INFINITY) } + } + + test("sub-millisecond backoffs are kept") { + RetryPolicy(1, initialBackoff = Duration.ofNanos(500_000), maxBackoff = Duration.ofMillis(1), jitter = 0.0).backoff(1) shouldBe + Duration.ofNanos(500_000) + } + + test("a retriable failure schedules the next attempt after the backoff") { + val now = Instant.parse("2024-01-01T00:00:00Z") + val deliverer = object : MessageDeliverer { + override val type = "stub" + override fun deliver(entry: OutboxEntry) = DeliveryResult.RetriableFailure("503") + } + val info = object : DeliveryInfo { + override val type = "stub" + override fun serialize(): String = """{"type":"stub"}""" + } + val processor = OutboxEntryProcessor(deliverer, noJitter, Clock.fixed(now, ZoneOffset.UTC)) + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), info, Instant.EPOCH) + + val first = processor.process(entry) + first.nextAttemptAt shouldBe now.plusSeconds(10) + processor.process(first).nextAttemptAt shouldBe now.plusSeconds(20) + } + + test("in a batch the backoff counts from when the slow attempt ended, not when it started") { + var now = Instant.parse("2024-01-01T00:00:00Z") + val clock = object : Clock() { + override fun getZone() = ZoneOffset.UTC + override fun withZone(zone: java.time.ZoneId) = this + override fun instant() = now + } + val timingOut = object : MessageDeliverer { + override val type = "stub" + override fun deliver(entry: OutboxEntry): DeliveryResult { + now = now.plusSeconds(30) // e.g. an HTTP request timeout + return DeliveryResult.RetriableFailure("timeout") + } + } + val info = object : DeliveryInfo { + override val type = "stub" + override fun serialize(): String = "{}" + } + val entry = OutboxEntry.createPending(OutboxMessage("e", "{}"), info, Instant.EPOCH) + + val retried = OutboxEntryProcessor(timingOut, noJitter, clock).processBatch(listOf(entry)).single() + + retried.nextAttemptAt shouldBe Instant.parse("2024-01-01T00:00:40Z") + } + + test("a new entry is due when it is created") { + val info = object : DeliveryInfo { + override val type = "stub" + override fun serialize(): String = "{}" + } + val createdAt = Instant.parse("2024-05-01T00:00:00Z") + OutboxEntry.createPending(OutboxMessage("e", "{}"), info, createdAt).nextAttemptAt shouldBe createdAt + } +}) diff --git a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxProcessorTest.kt b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxProcessorTest.kt index c418b65e..c0d41f2e 100644 --- a/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxProcessorTest.kt +++ b/okapi-core/src/test/kotlin/com/softwaremill/okapi/core/RouteAwareOutboxProcessorTest.kt @@ -23,12 +23,14 @@ private fun routeDeliverer(type: String) = object : MessageDeliverer { private class RoutingStore(var pending: List) : RouteAwareOutboxStore { val queriedTypes = mutableListOf>() + val claimTimes = mutableListOf() val processed = mutableListOf() override fun persist(entry: OutboxEntry) = entry override fun claimPending(limit: Int): List = error("Unsafe claimPending(limit) must not be used") - override fun claimPending(deliveryTypes: Set, limit: Int): List { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { queriedTypes += deliveryTypes + claimTimes += now val claimed = pending.filter { it.deliveryType in deliveryTypes }.take(limit) pending = pending - claimed.toSet() return claimed @@ -40,6 +42,21 @@ private class RoutingStore(var pending: List) : RouteAwareOutboxSto } class RouteAwareOutboxProcessorTest : FunSpec({ + test("claims only entries due at the processor clock's now (#131)") { + val now = Instant.parse("2024-06-01T00:00:00Z") + val store = RoutingStore(listOf(routeEntry("kafka"))) + val clock = Clock.fixed(now, java.time.ZoneOffset.UTC) + val processor = OutboxProcessor( + store, + OutboxEntryProcessor(routeDeliverer("kafka"), RetryPolicy(1), clock), + clock = clock, + ) + + processor.processNext(10) + + store.claimTimes shouldContainExactly listOf(now) + } + test("a single deliverer refuses an entry of another type before delivery") { val entryProcessor = OutboxEntryProcessor(routeDeliverer("kafka"), RetryPolicy(1), Clock.systemUTC()) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/HttpEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/HttpEndToEndTest.kt index 7505a224..a9eb2a94 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/HttpEndToEndTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/HttpEndToEndTest.kt @@ -22,6 +22,7 @@ import com.softwaremill.okapi.test.support.PostgresTestSupport import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.maps.shouldContain import java.time.Clock +import java.time.Duration class HttpEndToEndTest : FunSpec({ val db = PostgresTestSupport() @@ -49,7 +50,7 @@ class HttpEndToEndTest : FunSpec({ val urlResolver = ServiceUrlResolver { "http://localhost:${wiremock.port()}" } val entryProcessor = OutboxEntryProcessor( HttpMessageDeliverer(urlResolver), - RetryPolicy(maxRetries = maxRetries), + RetryPolicy(maxRetries = maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO), clock, ) return Triple(publisher, OutboxProcessor(store, entryProcessor), store) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/ObservabilityEndToEndTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/ObservabilityEndToEndTest.kt index 8d14bf57..12e2d9fc 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/ObservabilityEndToEndTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/e2e/ObservabilityEndToEndTest.kt @@ -23,6 +23,7 @@ import io.kotest.matchers.doubles.shouldBeGreaterThanOrEqual import io.kotest.matchers.shouldBe import io.micrometer.core.instrument.simple.SimpleMeterRegistry import java.time.Clock +import java.time.Duration import java.util.concurrent.TimeUnit class ObservabilityEndToEndTest : FunSpec({ @@ -62,7 +63,12 @@ class ObservabilityEndToEndTest : FunSpec({ val metrics = MicrometerOutboxMetrics(store, registry, transactionRunner = jdbcTransactionRunner, clock = clock) val urlResolver = ServiceUrlResolver { "http://localhost:${wiremock.port()}" } - val entryProcessor = OutboxEntryProcessor(HttpMessageDeliverer(urlResolver), RetryPolicy(maxRetries = 3), clock) + val entryProcessor = + OutboxEntryProcessor( + HttpMessageDeliverer(urlResolver), + RetryPolicy(maxRetries = 3, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO), + clock, + ) val processor = OutboxProcessor(store, entryProcessor, listener = listener, clock = clock) // Stub: first call → 500 (retriable), second call → 200 (success) @@ -115,7 +121,12 @@ class ObservabilityEndToEndTest : FunSpec({ val metrics = MicrometerOutboxMetrics(store, registry, transactionRunner = jdbcTransactionRunner, clock = clock) val urlResolver = ServiceUrlResolver { "http://localhost:${wiremock.port()}" } - val entryProcessor = OutboxEntryProcessor(HttpMessageDeliverer(urlResolver), RetryPolicy(maxRetries = 3), clock) + val entryProcessor = + OutboxEntryProcessor( + HttpMessageDeliverer(urlResolver), + RetryPolicy(maxRetries = 3, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO), + clock, + ) val processor = OutboxProcessor(store, entryProcessor, listener = listener, clock = clock) wiremock.stubFor( @@ -140,7 +151,12 @@ class ObservabilityEndToEndTest : FunSpec({ val listener = MicrometerOutboxListener(registry) val urlResolver = ServiceUrlResolver { "http://localhost:${wiremock.port()}" } - val entryProcessor = OutboxEntryProcessor(HttpMessageDeliverer(urlResolver), RetryPolicy(maxRetries = 3), clock) + val entryProcessor = + OutboxEntryProcessor( + HttpMessageDeliverer(urlResolver), + RetryPolicy(maxRetries = 3, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO), + clock, + ) val processor = OutboxProcessor(store, entryProcessor, listener = listener, clock = clock) wiremock.stubFor( diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlDatetimeMigrationTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlDatetimeMigrationTest.kt index cb137716..f6543564 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlDatetimeMigrationTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlDatetimeMigrationTest.kt @@ -61,7 +61,9 @@ class MysqlDatetimeMigrationTest : FunSpec({ """ INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, delivery_metadata) VALUES ('00000000-0000-0000-0000-000000000001', 'm', '{}', 'http', 'PENDING', - '2024-06-01 17:00:00.250000', '2024-06-01 17:00:01', 1, '2024-06-01 17:00:02', '{}') + '2024-06-01 17:00:00.250000', '2024-06-01 17:00:01', 1, '2024-06-01 17:00:02', '{}'), + ('00000000-0000-0000-0000-000000000002', 'm', '{}', 'http', 'FAILED', + '2024-06-01 16:00:00', '2024-06-01 16:00:01', 5, '2024-06-01 16:00:02', '{}') """.trimIndent(), ) } @@ -113,10 +115,28 @@ class MysqlDatetimeMigrationTest : FunSpec({ setPassword(container.password) }, ) - val entry = jdbc.withTransaction { MysqlOutboxStore(jdbc).claimPending(10) }.single() + val store = MysqlOutboxStore(jdbc) + // The route-aware claim, which only returns rows whose next_attempt_at (backfilled by 004) is due. + val entry = jdbc.withTransaction { store.claimPending(setOf("http"), 10, Instant.now()) }.single() entry.createdAt shouldBe Instant.parse("2024-06-01T12:00:00.25Z") entry.updatedAt shouldBe Instant.parse("2024-06-01T12:00:01Z") entry.lastAttempt shouldBe Instant.parse("2024-06-01T12:00:02Z") + entry.nextAttemptAt shouldBe entry.createdAt entry.status shouldBe OutboxStatus.PENDING + + // 004 also backfilled the FAILED row, so requeuing it by hand makes it claimable. + jdbc.withTransaction { + jdbc.withConnection { conn -> + conn.createStatement().use { + it.executeUpdate( + "UPDATE okapi_outbox SET status = 'PENDING', retries = 0 WHERE id = '00000000-0000-0000-0000-000000000002'", + ) + } + } + } + jdbc.withTransaction { store.claimPending(setOf("http"), 10, Instant.now()) }.map { it.outboxId.raw.toString() } shouldBe listOf( + "00000000-0000-0000-0000-000000000002", + "00000000-0000-0000-0000-000000000001", + ) } }) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlOutboxStoreTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlOutboxStoreTest.kt index b4154b6a..2e4524ab 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlOutboxStoreTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/MysqlOutboxStoreTest.kt @@ -4,6 +4,7 @@ import com.softwaremill.okapi.mysql.MysqlOutboxStore import com.softwaremill.okapi.test.support.MysqlTestSupport import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.collections.shouldBeEmpty +import java.time.Instant class MysqlOutboxStoreTest : FunSpec({ val db = MysqlTestSupport() @@ -24,7 +25,7 @@ class MysqlOutboxStoreTest : FunSpec({ test("[mysql] route-aware claim raises no server warnings") { val store = MysqlOutboxStore(connectionProvider = db.jdbc) val warnings = db.jdbc.withTransaction { - store.claimPending(setOf("kafka", "http"), 10) + store.claimPending(setOf("kafka", "http"), 10, Instant.now()) db.jdbc.withConnection { conn -> conn.createStatement().use { stmt -> stmt.executeQuery("SHOW WARNINGS").use { rs -> diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/OutboxStoreContractTests.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/OutboxStoreContractTests.kt index fd5a67a4..656c63f5 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/OutboxStoreContractTests.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/OutboxStoreContractTests.kt @@ -1,17 +1,26 @@ package com.softwaremill.okapi.test.store import com.softwaremill.okapi.core.DeliveryInfo +import com.softwaremill.okapi.core.DeliveryResult +import com.softwaremill.okapi.core.MessageDeliverer import com.softwaremill.okapi.core.OutboxEntry +import com.softwaremill.okapi.core.OutboxEntryProcessor import com.softwaremill.okapi.core.OutboxMessage +import com.softwaremill.okapi.core.OutboxProcessor import com.softwaremill.okapi.core.OutboxStatus import com.softwaremill.okapi.core.OutboxStore +import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.RouteAwareOutboxStore import com.softwaremill.okapi.test.support.JdbcConnectionProvider import io.kotest.core.spec.style.FunSpec import io.kotest.matchers.collections.shouldHaveSize import io.kotest.matchers.maps.shouldContain import io.kotest.matchers.shouldBe +import java.time.Clock +import java.time.Duration import java.time.Instant +import java.time.ZoneId +import java.time.ZoneOffset private class StubDeliveryInfo( override val type: String = "test", @@ -89,10 +98,10 @@ fun FunSpec.outboxStoreContractTests( } val routed = store as RouteAwareOutboxStore - val claimed = jdbc.withTransaction { routed.claimPending("kafka", 10) } + val claimed = jdbc.withTransaction { routed.claimPending("kafka", 10, Instant.now()) } claimed.map { it.outboxId } shouldBe listOf(supported.outboxId) - jdbc.withTransaction { routed.claimPending("http", 10) }.map { it.outboxId } shouldBe listOf(unsupported.outboxId) + jdbc.withTransaction { routed.claimPending("http", 10, Instant.now()) }.map { it.outboxId } shouldBe listOf(unsupported.outboxId) } test("[$dbName] route-aware claim matches delivery types exactly") { @@ -104,8 +113,8 @@ fun FunSpec.outboxStoreContractTests( } val routed = store as RouteAwareOutboxStore - jdbc.withTransaction { routed.claimPending("kafka", 10) }.map { it.outboxId } shouldBe listOf(lower.outboxId) - jdbc.withTransaction { routed.claimPending("KAFKA", 10) }.map { it.outboxId } shouldBe listOf(upper.outboxId) + jdbc.withTransaction { routed.claimPending("kafka", 10, Instant.now()) }.map { it.outboxId } shouldBe listOf(lower.outboxId) + jdbc.withTransaction { routed.claimPending("KAFKA", 10, Instant.now()) }.map { it.outboxId } shouldBe listOf(upper.outboxId) } test("[$dbName] route-aware claim selects a limited batch across several types") { @@ -119,11 +128,83 @@ fun FunSpec.outboxStoreContractTests( } val routed = store as RouteAwareOutboxStore - val claimed = jdbc.withTransaction { routed.claimPending(setOf("kafka", "http"), 2) } + val claimed = jdbc.withTransaction { routed.claimPending(setOf("kafka", "http"), 2, Instant.now()) } claimed.map { it.outboxId } shouldBe listOf(http.outboxId, kafka.outboxId) } + test("[$dbName] route-aware claim skips entries whose next attempt is not yet due (#131)") { + val now = Instant.parse("2024-01-10T00:00:00Z") + val backingOff = createTestEntry(now = Instant.parse("2024-01-01T00:00:00Z"), deliveryInfo = StubDeliveryInfo("kafka")) + .retry(Instant.parse("2024-01-09T23:59:00Z"), "503", nextAttemptAt = now.plusSeconds(60)) + val due = createTestEntry(now = Instant.parse("2024-01-05T00:00:00Z"), deliveryInfo = StubDeliveryInfo("kafka")) + jdbc.withTransaction { + store.persist(backingOff) + store.persist(due) + } + + val routed = store as RouteAwareOutboxStore + jdbc.withTransaction { routed.claimPending("kafka", 10, now) }.map { it.outboxId } shouldBe listOf(due.outboxId) + jdbc.withTransaction { routed.claimPending("kafka", 10, now.plusSeconds(60)) }.map { it.outboxId } shouldBe + listOf(due.outboxId, backingOff.outboxId) + } + + test("[$dbName] route-aware claim orders by next attempt, so a retried entry does not block newer ones (#131)") { + val retried = createTestEntry(now = Instant.parse("2024-01-01T00:00:00Z"), deliveryInfo = StubDeliveryInfo("kafka")) + .retry(Instant.parse("2024-01-02T00:00:00Z"), "503", nextAttemptAt = Instant.parse("2024-01-03T00:00:00Z")) + val newer = createTestEntry(now = Instant.parse("2024-01-02T12:00:00Z"), deliveryInfo = StubDeliveryInfo("kafka")) + jdbc.withTransaction { + store.persist(retried) + store.persist(newer) + } + + val routed = store as RouteAwareOutboxStore + jdbc.withTransaction { routed.claimPending("kafka", 1, Instant.parse("2024-01-04T00:00:00Z")) }.map { it.outboxId } shouldBe + listOf(newer.outboxId) + } + + test("[$dbName] updateAfterProcessing persists nextAttemptAt (#131)") { + val entry = createTestEntry(deliveryInfo = StubDeliveryInfo("kafka")) + jdbc.withTransaction { store.persist(entry) } + val nextAttemptAt = Instant.parse("2030-01-01T00:00:00.123456Z") + val retried = entry.retry(Instant.parse("2024-01-01T00:00:01Z"), "503", nextAttemptAt) + + jdbc.withTransaction { store.updateAfterProcessingBatch(listOf(retried)) } + + val routed = store as RouteAwareOutboxStore + jdbc.withTransaction { routed.claimPending("kafka", 10, Instant.parse("2029-12-31T23:59:59Z")) } shouldHaveSize 0 + val claimed = jdbc.withTransaction { routed.claimPending("kafka", 10, nextAttemptAt) }.single() + claimed.nextAttemptAt shouldBe nextAttemptAt + claimed.retries shouldBe 1 + } + + test("[$dbName] a retriable failure is not redelivered until its backoff has passed (#131)") { + var now = Instant.parse("2024-01-01T00:00:00Z") + val clock = object : Clock() { + override fun getZone(): ZoneId = ZoneOffset.UTC + override fun withZone(zone: ZoneId): Clock = this + override fun instant(): Instant = now + } + var attempts = 0 + val deliverer = object : MessageDeliverer { + override val type = "kafka" + override fun deliver(entry: OutboxEntry): DeliveryResult { + attempts++ + return DeliveryResult.RetriableFailure("503") + } + } + val policy = RetryPolicy(maxRetries = 5, initialBackoff = Duration.ofSeconds(10), jitter = 0.0) + val processor = OutboxProcessor(store, OutboxEntryProcessor(deliverer, policy, clock), clock = clock) + jdbc.withTransaction { store.persist(createTestEntry(now = now, deliveryInfo = StubDeliveryInfo("kafka"))) } + + jdbc.withTransaction { processor.processNext(10) } shouldBe 1 + now = now.plusSeconds(9) + jdbc.withTransaction { processor.processNext(10) } shouldBe 0 + now = now.plusSeconds(1) + jdbc.withTransaction { processor.processNext(10) } shouldBe 1 + attempts shouldBe 2 + } + test("[$dbName] claimPending returns entries ordered by created_at ASC") { val t1 = Instant.parse("2024-01-01T00:00:00Z") val t2 = Instant.parse("2024-01-02T00:00:00Z") diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/PostgresTimestamptzMigrationTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/PostgresTimestamptzMigrationTest.kt index e00f7322..9f734b08 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/PostgresTimestamptzMigrationTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/store/PostgresTimestamptzMigrationTest.kt @@ -119,7 +119,9 @@ class PostgresTimestamptzMigrationTest : FunSpec({ } } val (jdbc, store) = store(url) - val entry = jdbc.withTransaction { store.claimPending(10) }.single() + // The route-aware claim, which only returns rows whose next_attempt_at (backfilled by 004) is due. + val entry = jdbc.withTransaction { store.claimPending(setOf("http"), 10, Instant.now()) }.single() + entry.nextAttemptAt shouldBe createdAt entry.createdAt shouldBe createdAt entry.updatedAt shouldBe createdAt entry.lastAttempt shouldBe lastAttempt @@ -158,6 +160,8 @@ class PostgresTimestamptzMigrationTest : FunSpec({ } val (jdbc, store) = store(url) - jdbc.withTransaction { store.claimPending(10) }.map { it.createdAt } shouldBe listOf(before, after) + // `after` was inserted by 1.x after 004, so only the column DEFAULT made it claimable. + jdbc.withTransaction { store.claimPending(setOf("http"), 10, Instant.now().plusSeconds(5)) } + .map { it.createdAt }.sorted() shouldBe listOf(before, after) } }) diff --git a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt index cea34e62..25086ae1 100644 --- a/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt +++ b/okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test/transaction/MysqlClaimDoesNotBlockPublishersTest.kt @@ -37,6 +37,7 @@ import org.springframework.transaction.support.TransactionSynchronizationManager import java.sql.Connection import java.sql.SQLException import java.time.Clock +import java.time.Instant import java.util.concurrent.CountDownLatch import java.util.concurrent.Executors import java.util.concurrent.TimeUnit @@ -202,13 +203,14 @@ private class RecordingStore(private val delegate: RouteAwareOutboxStore) : Rout return delegate.claimPending(limit) } - override fun claimPending(deliveryTypes: Set, limit: Int): List { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { claims += "claimPending($deliveryTypes, $limit)" - return delegate.claimPending(deliveryTypes, limit) + return delegate.claimPending(deliveryTypes, limit, now) } // Overridden so the single-type call is recorded too instead of going straight to the delegate. - override fun claimPending(deliveryType: String, limit: Int): List = claimPending(setOf(deliveryType), limit) + override fun claimPending(deliveryType: String, limit: Int, now: Instant): List = + claimPending(setOf(deliveryType), limit, now) } private class BlockingDeliverer : MessageDeliverer { diff --git a/okapi-mysql/src/main/kotlin/com/softwaremill/okapi/mysql/MysqlOutboxStore.kt b/okapi-mysql/src/main/kotlin/com/softwaremill/okapi/mysql/MysqlOutboxStore.kt index 1f8802ef..24878dd4 100644 --- a/okapi-mysql/src/main/kotlin/com/softwaremill/okapi/mysql/MysqlOutboxStore.kt +++ b/okapi-mysql/src/main/kotlin/com/softwaremill/okapi/mysql/MysqlOutboxStore.kt @@ -33,6 +33,7 @@ class MysqlOutboxStore( val lastError = entry.lastError if (lastError != null) stmt.setString(10, OutboxEntry.sanitizeLastError(lastError)) else stmt.setNull(10, Types.VARCHAR) stmt.setString(11, entry.deliveryMetadata) + stmt.setUtc(12, entry.nextAttemptAt) stmt.executeUpdate() } } @@ -63,20 +64,21 @@ class MysqlOutboxStore( } } - override fun claimPending(deliveryTypes: Set, limit: Int): List { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { if (deliveryTypes.isEmpty() || limit <= 0) return emptyList() val routes = deliveryTypes.sorted() val placeholders = routes.joinToString(",") { "?" } val binaryPlaceholders = routes.joinToString(",") { "CAST(? AS BINARY)" } - // A matching index avoids scanning and locking rows for other delivery types. + // A matching index avoids scanning and locking rows for other delivery types, and its + // next_attempt_at range stops at `now`, so entries still backing off are neither scanned nor locked. // The binary predicate excludes case-insensitive collation matches for another route. // CAST rather than the BINARY operator, which is deprecated since 8.0.27 and warns on every call. val sql = """ SELECT * FROM okapi_outbox - FORCE INDEX (idx_okapi_outbox_status_delivery_created_id) - WHERE status = ? AND delivery_type IN ($placeholders) + FORCE INDEX (idx_okapi_outbox_status_delivery_next_attempt_id) + WHERE status = ? AND delivery_type IN ($placeholders) AND next_attempt_at <= ? AND CAST(delivery_type AS BINARY) IN ($binaryPlaceholders) - ORDER BY delivery_type ASC, created_at ASC, id ASC + ORDER BY delivery_type ASC, next_attempt_at ASC, id ASC LIMIT ? FOR UPDATE SKIP LOCKED """.trimIndent() @@ -85,8 +87,9 @@ class MysqlOutboxStore( conn.prepareStatement(sql).use { stmt -> stmt.setString(1, OutboxStatus.PENDING.name) routes.forEachIndexed { index, route -> stmt.setString(index + 2, route) } - routes.forEachIndexed { index, route -> stmt.setString(index + routes.size + 2, route) } - stmt.setInt(routes.size * 2 + 2, limit) + stmt.setUtc(routes.size + 2, now) + routes.forEachIndexed { index, route -> stmt.setString(index + routes.size + 3, route) } + stmt.setInt(routes.size * 2 + 3, limit) stmt.executeQuery().use { rs -> generateSequence { if (rs.next()) rs.toOutboxEntry() else null }.toList() } @@ -189,6 +192,8 @@ class MysqlOutboxStore( lastAttempt = getUtc("last_attempt"), lastError = getString("last_error"), deliveryMetadata = getString("delivery_metadata"), + // NULL only on rows that were not PENDING when changeset 004 added the column. + nextAttemptAt = getUtc("next_attempt_at") ?: checkNotNull(getUtc("created_at")), ) private fun PreparedStatement.prepareForUpdateFrom(entry: OutboxEntry): PreparedStatement { @@ -198,8 +203,9 @@ class MysqlOutboxStore( setUtc(4, entry.lastAttempt) val lastError = entry.lastError if (lastError != null) setString(5, OutboxEntry.sanitizeLastError(lastError)) else setNull(5, Types.VARCHAR) + setUtc(6, entry.nextAttemptAt) // CHAR(36) column — MySQL has no native UUID type, so the id travels as a string. - setString(6, entry.outboxId.raw.toString()) + setString(7, entry.outboxId.raw.toString()) return this } @@ -214,20 +220,21 @@ class MysqlOutboxStore( companion object { private val upsertSql = """ - INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, last_error, delivery_metadata) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, last_error, delivery_metadata, next_attempt_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE status = VALUES(status), updated_at = VALUES(updated_at), retries = VALUES(retries), last_attempt = VALUES(last_attempt), last_error = VALUES(last_error), - delivery_metadata = VALUES(delivery_metadata) + delivery_metadata = VALUES(delivery_metadata), + next_attempt_at = VALUES(next_attempt_at) """.trimIndent() private val updateSql = """ UPDATE okapi_outbox - SET status = ?, updated_at = ?, retries = ?, last_attempt = ?, last_error = ? + SET status = ?, updated_at = ?, retries = ?, last_attempt = ?, last_error = ?, next_attempt_at = ? WHERE id = ? """.trimIndent() } diff --git a/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql b/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql new file mode 100644 index 00000000..33902edb --- /dev/null +++ b/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql @@ -0,0 +1,46 @@ +--liquibase formatted sql + +-- next_attempt_at: when a PENDING entry is next due, in UTC like the other DATETIME columns (issue +-- #131). A retriable failure pushes it out by the RetryPolicy backoff; the claim skips entries that +-- are not yet due. The new claim index replaces the (status, delivery_type, created_at, id) index +-- from 002. +-- +-- MySQL commits every DDL statement on its own, so a changeset holding several would leave the schema +-- half-changed if a later statement failed (e.g. on lock_wait_timeout) and could not run again. Each +-- step is therefore its own changeset, skipped by its precondition when it already took effect. + +--changeset outbox:004 +--preconditions onFail:MARK_RAN +--precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox' AND COLUMN_NAME = 'next_attempt_at' +-- Adding a nullable column at the end is an INSTANT change on MySQL 8.0.12+ (in place on older +-- versions). It is NULL until the backfill below; 1.x instances must be stopped before 003, so +-- nothing else inserts rows without it. +SET @okapi_previous_lock_wait_timeout = @@session.lock_wait_timeout; +SET SESSION lock_wait_timeout = 30; +ALTER TABLE okapi_outbox ADD COLUMN next_attempt_at DATETIME(6) NULL; +SET SESSION lock_wait_timeout = @okapi_previous_lock_wait_timeout; + +--changeset outbox:004-backfill +-- PENDING rows keep their order; FAILED rows get a value too, so one set back to PENDING by hand is +-- claimable. DELIVERED rows never need one and are read with next_attempt_at = created_at. +UPDATE okapi_outbox SET next_attempt_at = created_at WHERE next_attempt_at IS NULL AND status <> 'DELIVERED'; + +--changeset outbox:004-claim-index +--preconditions onFail:MARK_RAN +--precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM information_schema.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox' AND INDEX_NAME = 'idx_okapi_outbox_status_delivery_next_attempt_id' +SET @okapi_previous_lock_wait_timeout = @@session.lock_wait_timeout; +SET SESSION lock_wait_timeout = 30; +ALTER TABLE okapi_outbox + ADD INDEX idx_okapi_outbox_status_delivery_next_attempt_id (status, delivery_type, next_attempt_at, id), + ALGORITHM=INPLACE, LOCK=NONE; +SET SESSION lock_wait_timeout = @okapi_previous_lock_wait_timeout; + +--changeset outbox:004-drop-created-index +--preconditions onFail:MARK_RAN +--precondition-sql-check expectedResult:1 SELECT COUNT(DISTINCT INDEX_NAME) FROM information_schema.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox' AND INDEX_NAME = 'idx_okapi_outbox_status_delivery_created_id' +SET @okapi_previous_lock_wait_timeout = @@session.lock_wait_timeout; +SET SESSION lock_wait_timeout = 30; +ALTER TABLE okapi_outbox + DROP INDEX idx_okapi_outbox_status_delivery_created_id, + ALGORITHM=INPLACE, LOCK=NONE; +SET SESSION lock_wait_timeout = @okapi_previous_lock_wait_timeout; diff --git a/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/changelog.xml b/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/changelog.xml index 12d53f91..16bb26c5 100644 --- a/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/changelog.xml +++ b/okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/changelog.xml @@ -6,4 +6,5 @@ + diff --git a/okapi-postgres/src/main/kotlin/com/softwaremill/okapi/postgres/PostgresOutboxStore.kt b/okapi-postgres/src/main/kotlin/com/softwaremill/okapi/postgres/PostgresOutboxStore.kt index b9409745..e54f85c1 100644 --- a/okapi-postgres/src/main/kotlin/com/softwaremill/okapi/postgres/PostgresOutboxStore.kt +++ b/okapi-postgres/src/main/kotlin/com/softwaremill/okapi/postgres/PostgresOutboxStore.kt @@ -33,6 +33,7 @@ class PostgresOutboxStore( val lastError = entry.lastError if (lastError != null) stmt.setString(10, OutboxEntry.sanitizeLastError(lastError)) else stmt.setNull(10, Types.VARCHAR) stmt.setString(11, entry.deliveryMetadata) + stmt.setInstant(12, entry.nextAttemptAt) stmt.executeUpdate() } } @@ -59,15 +60,17 @@ class PostgresOutboxStore( } } - override fun claimPending(deliveryTypes: Set, limit: Int): List { + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant): List { if (deliveryTypes.isEmpty() || limit <= 0) return emptyList() val routes = deliveryTypes.sorted() val placeholders = routes.joinToString(",") { "?" } - // Keep PENDING literal so a generic prepared plan can use the partial index. + // Keep PENDING literal so a generic prepared plan can use the partial index. Walking + // (delivery_type, next_attempt_at, id) up to `now` skips entries still backing off without + // scanning them, however many there are. val sql = """ SELECT * FROM okapi_outbox - WHERE status = 'PENDING' AND delivery_type IN ($placeholders) - ORDER BY delivery_type ASC, created_at ASC, id ASC + WHERE status = 'PENDING' AND delivery_type IN ($placeholders) AND next_attempt_at <= ? + ORDER BY delivery_type ASC, next_attempt_at ASC, id ASC LIMIT ? FOR UPDATE SKIP LOCKED """.trimIndent() @@ -75,7 +78,8 @@ class PostgresOutboxStore( return connectionProvider.withConnection { conn -> conn.prepareStatement(sql).use { stmt -> routes.forEachIndexed { index, route -> stmt.setString(index + 1, route) } - stmt.setInt(routes.size + 1, limit) + stmt.setInstant(routes.size + 1, now) + stmt.setInt(routes.size + 2, limit) stmt.executeQuery().use { rs -> generateSequence { if (rs.next()) rs.toOutboxEntry() else null }.toList() } @@ -176,6 +180,7 @@ class PostgresOutboxStore( lastAttempt = getInstant("last_attempt"), lastError = getString("last_error"), deliveryMetadata = getString("delivery_metadata"), + nextAttemptAt = checkNotNull(getInstant("next_attempt_at")), ) private fun PreparedStatement.prepareForUpdateFrom(entry: OutboxEntry): PreparedStatement { @@ -185,7 +190,8 @@ class PostgresOutboxStore( setInstant(4, entry.lastAttempt) val lastError = entry.lastError if (lastError != null) setString(5, OutboxEntry.sanitizeLastError(lastError)) else setNull(5, Types.VARCHAR) - setObject(6, entry.outboxId.raw) + setInstant(6, entry.nextAttemptAt) + setObject(7, entry.outboxId.raw) return this } @@ -204,20 +210,21 @@ class PostgresOutboxStore( companion object { private val insertSql = """ - INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, last_error, delivery_metadata) - VALUES (?::uuid, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb) + INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, last_error, delivery_metadata, next_attempt_at) + VALUES (?::uuid, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?::jsonb, ?) ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, updated_at = EXCLUDED.updated_at, retries = EXCLUDED.retries, last_attempt = EXCLUDED.last_attempt, last_error = EXCLUDED.last_error, - delivery_metadata = EXCLUDED.delivery_metadata + delivery_metadata = EXCLUDED.delivery_metadata, + next_attempt_at = EXCLUDED.next_attempt_at """.trimIndent() private val updateSql = """ UPDATE okapi_outbox - SET status = ?, updated_at = ?, retries = ?, last_attempt = ?, last_error = ? + SET status = ?, updated_at = ?, retries = ?, last_attempt = ?, last_error = ?, next_attempt_at = ? WHERE id = ? """.trimIndent() } diff --git a/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql b/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql new file mode 100644 index 00000000..558dda07 --- /dev/null +++ b/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql @@ -0,0 +1,33 @@ +--liquibase formatted sql +--changeset outbox:004 + +-- next_attempt_at: when a PENDING entry is next due (issue #131). A retriable failure pushes it out +-- by the RetryPolicy backoff; the claim skips entries that are not yet due. +-- +-- The DEFAULT keeps rows inserted by 1.x instances during a rolling deploy claimable (they do not +-- know the column). CURRENT_TIMESTAMP is not volatile, so PostgreSQL 11+ records it in the catalog +-- instead of rewriting the table. The ADD COLUMN needs an ACCESS EXCLUSIVE lock, held until the +-- transaction of this step commits, so the step does nothing else; lock_timeout bounds the wait. +SET LOCAL lock_timeout = '30s'; + +ALTER TABLE okapi_outbox ADD COLUMN IF NOT EXISTS next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP; + +--changeset outbox:004-backfill + +-- In its own transaction, so it takes only row locks: existing PENDING rows get their created_at and +-- keep their order instead of all becoming due at the migration time. +UPDATE okapi_outbox SET next_attempt_at = created_at WHERE status = 'PENDING' AND next_attempt_at <> created_at; + +--changeset outbox:004-claim-index runInTransaction:false + +-- The route-aware claim walks (delivery_type, next_attempt_at, id) up to now, so entries that are +-- backing off are never scanned. It replaces the (delivery_type, created_at, id) index from 002. +-- A failed or cancelled CREATE INDEX CONCURRENTLY leaves an INVALID index behind; drop it first so a +-- retried changeset rebuilds it. +DROP INDEX CONCURRENTLY IF EXISTS idx_okapi_outbox_pending_delivery_next_attempt_id; + +CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_okapi_outbox_pending_delivery_next_attempt_id + ON okapi_outbox (delivery_type, next_attempt_at, id) + WHERE status = 'PENDING'; + +DROP INDEX CONCURRENTLY IF EXISTS idx_okapi_outbox_pending_delivery_created_id; diff --git a/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/changelog.xml b/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/changelog.xml index bbc4f37b..50d3db49 100644 --- a/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/changelog.xml +++ b/okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/changelog.xml @@ -6,4 +6,5 @@ + diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt index 32cf256d..0ffa57a4 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxAutoConfiguration.kt @@ -57,7 +57,7 @@ import javax.sql.DataSource * documented, not enforced, and MySQL could silently win instead depending on undocumented * nested-`@Configuration` processing order). Override by defining your own `@Bean OutboxStore`. * - [Clock] — defaults to [Clock.systemUTC] - * - [RetryPolicy] — defaults to `maxRetries = 5` + * - [RetryPolicy] — built from `okapi.processor.max-retries` and the `okapi.processor.*backoff*` properties * * Multi-datasource / multi-PlatformTransactionManager support: * - Set `okapi.datasource-qualifier` to the bean name of the [DataSource] that holds the outbox table. @@ -180,7 +180,15 @@ class OutboxAutoConfiguration( ): OutboxEntryProcessor { return OutboxEntryProcessor( deliverer = if (deliverers.size == 1) deliverers.single() else CompositeMessageDeliverer(deliverers, props.transportDispatch), - retryPolicy = retryPolicy.getIfAvailable { RetryPolicy(maxRetries = props.maxRetries) }, + retryPolicy = retryPolicy.getIfAvailable { + RetryPolicy( + maxRetries = props.maxRetries, + initialBackoff = props.initialBackoff, + backoffMultiplier = props.backoffMultiplier, + maxBackoff = props.maxBackoff, + jitter = props.backoffJitter, + ) + }, clock = clock.getIfAvailable { Clock.systemUTC() }, ) } diff --git a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt index a0eb16f0..394c00eb 100644 --- a/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt +++ b/okapi-spring-boot/src/main/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorProperties.kt @@ -1,5 +1,6 @@ package com.softwaremill.okapi.springboot +import com.softwaremill.okapi.core.RetryPolicy import com.softwaremill.okapi.core.TransportDispatch import org.springframework.boot.context.properties.ConfigurationProperties import java.time.Duration @@ -22,11 +23,25 @@ data class OutboxProcessorProperties @JvmOverloads constructor( * [com.softwaremill.okapi.core.TransportDispatch]. */ val transportDispatch: TransportDispatch = TransportDispatch.PARALLEL, + /** Delay before the first retry of a retriable failure. See [com.softwaremill.okapi.core.RetryPolicy]. */ + val initialBackoff: Duration = RetryPolicy.DEFAULT_INITIAL_BACKOFF, + /** Factor applied to the delay after each further retry. */ + val backoffMultiplier: Double = RetryPolicy.DEFAULT_BACKOFF_MULTIPLIER, + /** Upper bound on the delay between retries. */ + val maxBackoff: Duration = RetryPolicy.DEFAULT_MAX_BACKOFF, + /** Fraction (0 to 1) by which each delay is shortened at random, to spread retries out. */ + val backoffJitter: Double = RetryPolicy.DEFAULT_JITTER, ) { init { require(!interval.isNegative && interval.toMillis() > 0) { "interval must be at least 1ms" } require(batchSize > 0) { "batchSize must be positive" } require(maxRetries >= 0) { "maxRetries must be >= 0" } require(concurrency in 1..256) { "concurrency must be between 1 and 256" } + // Fail at startup, with the property names, rather than when the RetryPolicy is built. + require(!initialBackoff.isNegative) { "initialBackoff must be >= 0" } + require(backoffMultiplier >= 1.0 && backoffMultiplier.isFinite()) { "backoffMultiplier must be finite and >= 1" } + require(maxBackoff >= initialBackoff) { "maxBackoff must be >= initialBackoff" } + require(maxBackoff <= RetryPolicy.MAX_ALLOWED_BACKOFF) { "maxBackoff must be at most ${RetryPolicy.MAX_ALLOWED_BACKOFF}" } + require(backoffJitter in 0.0..1.0) { "backoffJitter must be between 0 and 1" } } } diff --git a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json index 214d96ea..ae1d9ae5 100644 --- a/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json +++ b/okapi-spring-boot/src/main/resources/META-INF/spring-configuration-metadata.json @@ -75,6 +75,30 @@ "defaultValue": 5, "description": "Maximum number of delivery retries before marking an entry as failed." }, + { + "name": "okapi.processor.initial-backoff", + "type": "java.time.Duration", + "defaultValue": "10s", + "description": "Delay before the first retry of a retriable delivery failure. Each further retry waits backoff-multiplier times longer, up to max-backoff." + }, + { + "name": "okapi.processor.backoff-multiplier", + "type": "java.lang.Double", + "defaultValue": 2.0, + "description": "Factor applied to the retry delay after each retry. Must be >= 1." + }, + { + "name": "okapi.processor.max-backoff", + "type": "java.time.Duration", + "defaultValue": "10m", + "description": "Upper bound on the delay between two retries of an entry." + }, + { + "name": "okapi.processor.backoff-jitter", + "type": "java.lang.Double", + "defaultValue": 0.2, + "description": "Fraction (0 to 1) by which each retry delay is shortened at random, so entries that failed together do not all retry at once." + }, { "name": "okapi.datasource-qualifier", "type": "java.lang.String", diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OkapiSpringTestStubs.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OkapiSpringTestStubs.kt index a8c7bc6f..0a19489f 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OkapiSpringTestStubs.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OkapiSpringTestStubs.kt @@ -19,7 +19,7 @@ import java.time.Instant internal fun stubStore() = object : RouteAwareOutboxStore { override fun persist(entry: OutboxEntry) = entry override fun claimPending(limit: Int) = emptyList() - override fun claimPending(deliveryTypes: Set, limit: Int) = emptyList() + override fun claimPending(deliveryTypes: Set, limit: Int, now: Instant) = emptyList() override fun updateAfterProcessing(entry: OutboxEntry) = entry override fun removeDeliveredBefore(time: Instant, limit: Int) = 0 override fun findOldestCreatedAt(statuses: Set) = emptyMap() diff --git a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt index c75ab342..5ac616e8 100644 --- a/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt +++ b/okapi-spring-boot/src/test/kotlin/com/softwaremill/okapi/springboot/OutboxProcessorAutoConfigurationTest.kt @@ -163,6 +163,43 @@ class OutboxProcessorAutoConfigurationTest : FunSpec({ } } + test("retry backoff properties are bound, with RetryPolicy's defaults when unset (#131)") { + contextRunner + .withPropertyValues( + "okapi.processor.initial-backoff=2s", + "okapi.processor.backoff-multiplier=3.5", + "okapi.processor.max-backoff=1h", + "okapi.processor.backoff-jitter=0", + ) + .run { ctx -> + val props = ctx.getBean() + props.initialBackoff shouldBe ofSeconds(2) + props.backoffMultiplier shouldBe 3.5 + props.maxBackoff shouldBe java.time.Duration.ofHours(1) + props.backoffJitter shouldBe 0.0 + } + contextRunner.run { ctx -> + val props = ctx.getBean() + props.initialBackoff shouldBe RetryPolicy.DEFAULT_INITIAL_BACKOFF + props.backoffMultiplier shouldBe RetryPolicy.DEFAULT_BACKOFF_MULTIPLIER + props.maxBackoff shouldBe RetryPolicy.DEFAULT_MAX_BACKOFF + props.backoffJitter shouldBe RetryPolicy.DEFAULT_JITTER + } + } + + test("invalid retry backoff properties fail startup (#131)") { + listOf( + "okapi.processor.backoff-multiplier=0.5", + "okapi.processor.backoff-jitter=2", + "okapi.processor.initial-backoff=15m", + "okapi.processor.max-backoff=400d", + ).forEach { property -> + contextRunner.withPropertyValues(property).run { ctx -> + withClue(property) { ctx.startupFailure.shouldNotBeNull() } + } + } + } + test("default properties when nothing is configured") { contextRunner.run { ctx -> val props = ctx.getBean()