Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,26 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *

### Changed (BREAKING)

- **Retries back off exponentially instead of firing on every poll (new migration `004__retry_backoff`).**
A retriable failure used to be retried on the very next tick, so with the defaults (`interval`
1s, `max-retries` 5) a message was `FAILED` — terminally — after about six seconds of downstream
outage, and failing rows, always the oldest, were re-claimed first on every tick. Each entry now
carries `nextAttemptAt` (column `next_attempt_at`); a retriable failure sets it `RetryPolicy`'s
backoff into the future, and the processor claims only due entries, ordered by
`(delivery_type, next_attempt_at, id)` through a new index that replaces 002's. The default backoff
is 10s, doubling, capped at 10m, with 20% jitter, which covers about five minutes of outage before
`FAILED`. New Spring properties `okapi.processor.initial-backoff`, `backoff-multiplier`,
`max-backoff` and `backoff-jitter`. API changes: `RetryPolicy` gained `initialBackoff`,
`backoffMultiplier`, `maxBackoff` and `jitter` (all defaulted, `@JvmOverloads`) and `backoff(retry)`;
`OutboxEntry` gained `nextAttemptAt` (defaults to `createdAt`) and `retry` an optional
`nextAttemptAt`; `RouteAwareOutboxStore.claimPending` takes the current time (`now`); and
`OutboxProcessorProperties` gained four parameters. `OutboxEntry`'s constructor is `@JvmOverloads`,
so Java callers can still omit `nextAttemptAt`. As with every data-class change, Kotlin code
calling `copy(...)` or default-argument constructors of these classes must be recompiled; custom
`RouteAwareOutboxStore`s must filter on and write `next_attempt_at`. The legacy
`OutboxStore.claimPending(limit)` ignores the backoff. Tests or callers that relied on immediate
retries need `RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`.
([#131](https://github.com/softwaremill/okapi/issues/131))
- **`OutboxProcessorProperties` gained a `transportDispatch` constructor parameter** (see the
`CompositeMessageDeliverer` entry above). `@JvmOverloads` preserves the previous four-argument
JVM constructor, so Java callers are unaffected, but adding a property to a Kotlin `data class`
Expand Down
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,12 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co
### Guarantees and limits

- **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id).
- **Best-effort ordering.** Rows are claimed by `created_at, id` within each delivery type. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.
- **Best-effort ordering.** Rows are claimed by due time (`next_attempt_at`, then `id`) within each delivery type: a new row is due when it is created, a retried one once its backoff has passed, so a failing row does not hold up newer ones. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.
- **Different workers can handle different delivery types.** Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom `OutboxStore` used for processing must implement `RouteAwareOutboxStore`; otherwise processing fails before an unsafe claim. The older `claimPending(limit)` method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself.
- **Rolling upgrades need sequencing.** Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked `FAILED` by older workers.
- **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not.
- **On MySQL, scheduler transactions run at `READ COMMITTED`.** The autoconfigured Spring `TransactionRunner` (with `MysqlOutboxStore`) and `ExposedTransactionRunner` (on a MySQL/MariaDB dialect) both use it. At MySQL's default `REPEATABLE READ`, the claim's gap locks would block every concurrent `publish()` until delivery finishes. If you supply your own `TransactionRunner`, an `OutboxStore` of your own, or a `TransactionTemplate` with an explicit isolation level, keep it at `READ COMMITTED` on MySQL. With a `JtaTransactionManager`, or a `JpaTransactionManager` without the Hibernate/EclipseLink `JpaDialect`, okapi cannot set it (startup logs a WARN): configure `transaction_isolation = 'READ-COMMITTED'` on the server or in the JDBC URL instead. `READ COMMITTED` requires `binlog_format=ROW` (the default since MySQL 5.7.7) or `MIXED`; with `STATEMENT`, InnoDB rejects the scheduler's writes with `ERROR 1665`.
- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff.
- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. A retriable failure is retried after an exponential backoff with jitter: `okapi.processor.initial-backoff` (default 10s) before the first retry, `backoff-multiplier` (default 2) times longer for each further one, capped at `max-backoff` (default 10m) and shortened at random by up to `backoff-jitter` (default 0.2). With the defaults, a message survives roughly five minutes of downstream outage before it becomes `FAILED`; raise `max-retries` to cover longer ones (e.g. 10 retries span about 50 minutes). The backoff only applies to rows claimed by 2.0 workers.

### Deduplicating on `x-outbox-id`

Expand Down Expand Up @@ -135,6 +135,10 @@ In a typical single-DataSource application, all properties are optional. Multi-D
| `okapi.processor.interval` | `1s` | How often the scheduler polls for pending entries. |
| `okapi.processor.batch-size` | `10` | Maximum entries claimed per worker per tick. |
| `okapi.processor.max-retries` | `5` | Retries after the initial attempt before an entry becomes `FAILED`. |
| `okapi.processor.initial-backoff` | `10s` | Delay before the first retry of a retriable failure. |
| `okapi.processor.backoff-multiplier` | `2.0` | Factor applied to the delay for each further retry (>= 1). |
| `okapi.processor.max-backoff` | `10m` | Upper bound on the delay between retries. |
| `okapi.processor.backoff-jitter` | `0.2` | Fraction (0 to 1) by which each delay is shortened at random, so entries that failed together do not retry together. |
| `okapi.processor.concurrency` | `1` | Parallel workers per tick, each claiming its own batch. Tune based on database capacity and delivery latency; see [Performance](#performance). |
| `okapi.processor.transport-dispatch` | `parallel` | How a batch spanning several `MessageDeliverer` beans is dispatched: `parallel` (all groups but one go to a virtual thread each — N transports use N-1 extra threads — while the last runs inline on the calling thread, so the batch costs ~`max(Tᵢ)`) or `sequential` (one group after another, all on the calling thread). Use `sequential` only if a custom deliverer depends on caller thread context — MDC, security context, or a transaction-bound resource such as a connection obtained via `DataSourceUtils`, since a group on a virtual thread does not inherit the caller's transaction. Batches with a single delivery type always run inline, regardless of this setting. |

Expand Down Expand Up @@ -215,10 +219,10 @@ With `okapi-spring-boot` and Liquibase on the classpath, these run automatically

If you use another migration tool, apply every SQL migration for your database, in order:

- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/003__timestamptz_columns.sql) `TIMESTAMPTZ` timestamp columns
- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns
- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/003__timestamptz_columns.sql) `TIMESTAMPTZ` timestamp columns, [004](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql) retry backoff column and claim index (its second changeset must run outside a transaction)
- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns, [004](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql) retry backoff column and claim index

For an existing installation, apply `002` before starting updated workers. MySQL claims explicitly select the new index and fail if it is missing. PostgreSQL's `002` uses `CREATE INDEX CONCURRENTLY`, so run it outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.
For an existing installation, apply every new migration (up to `004`) before starting updated workers. MySQL claims explicitly select the claim index from `004` and fail if it is missing. PostgreSQL's `002` and the index changeset of `004` use `CREATE INDEX CONCURRENTLY`, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.

PostgreSQL's `003` converts the timestamp columns from `TIMESTAMP` (the okapi JVM's local wall-clock time) to `TIMESTAMPTZ`. It reads existing values in the session `TimeZone`; pgjdbc sets that to the JVM default zone, so the bundled Liquibase changelog converts them correctly when it runs inside the application. When applying it by hand, run it inside a transaction (`psql -1`) after `SET TIME ZONE '<zone your okapi JVMs run in>'` in the same session. It takes an `ACCESS EXCLUSIVE` lock: it first waits (up to 30 s, then fails and retries on the next start) for open transactions on `okapi_outbox`, queuing every other statement on the table meanwhile, and then, unless the session zone is UTC (PostgreSQL 12+), rewrites the table while reads and writes wait. 1.x instances keep working after it, apart from one failed poll per pooled connection. See UPGRADING.md for when the in-app conversion uses the wrong zone. See [UPGRADING.md](UPGRADING.md#from-1x-to-200).

Expand Down
33 changes: 33 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,39 @@ them, and the conversion reads them in the session `TimeZone`:
correct too. A claim query a 1.x connection prepared before the migration fails once with
`cached plan must not change result type`; that tick is retried on the next poll.

### Retry backoff and `next_attempt_at` (#131)

Changeset `004__retry_backoff` adds `next_attempt_at`, sets it to `created_at` on existing `PENDING`
rows and replaces 002's claim index with one on `(delivery_type, next_attempt_at, id)`.

- **PostgreSQL:** the column is `NOT NULL DEFAULT CURRENT_TIMESTAMP` (added without a table rewrite
on PostgreSQL 11+), so rows that 1.x instances insert during a rolling deploy are claimable at once.
The `ADD COLUMN` holds an `ACCESS EXCLUSIVE` lock only for itself (waiting at most 30 s for it); the
backfill of `PENDING` rows runs in a separate changeset with row locks only. The index is built in a second changeset with `CREATE INDEX CONCURRENTLY`, which must run outside a
transaction when you apply the script by hand, and which waits for every transaction that started
before it.
- **MySQL:** the column is nullable (an instant change on 8.0.12+); `PENDING` and `FAILED` rows are
backfilled, `DELIVERED` rows stay `NULL`. 1.x instances are already stopped for 003, so nothing
writes rows without it. Each step is a changeset of its own with a precondition, so a migration that
stopped half-way (e.g. on its 30 s `lock_wait_timeout`) resumes where it stopped.
- **Behaviour:** retriable failures now wait 10s, 20s, 40s, ... (capped at 10m, 20% jitter) before
the next attempt, so with `max-retries` 5 an entry becomes `FAILED` after about five minutes rather
than six seconds. Set `okapi.processor.initial-backoff`, `backoff-multiplier`, `max-backoff`,
`backoff-jitter` or `max-retries` to change that. 1.x workers still running during a rolling deploy
ignore the backoff for the rows they claim.
- **Clocks:** a new entry is due at its `createdAt`, taken from the publishing instance's clock, and
claimed once the processing instance's clock reaches it, so a publisher whose clock runs ahead of
the processors' delays its messages by the difference. Keep instance clocks NTP-synchronised.
- **Requeuing a `FAILED` row by hand** now needs a due time as well:
`UPDATE okapi_outbox SET status = 'PENDING', retries = 0, next_attempt_at = <now in UTC> WHERE ...`
(`CURRENT_TIMESTAMP` on PostgreSQL, `UTC_TIMESTAMP(6)` on MySQL). A row whose `next_attempt_at`
lies in the future is not claimed until then.
- **Code:** a custom `RouteAwareOutboxStore` must implement `claimPending(deliveryTypes, limit, now)`,
return only rows with `next_attempt_at <= now`, and write `nextAttemptAt` in `persist` and the update
methods. `OutboxEntry` keeps a Java constructor without `nextAttemptAt` (`@JvmOverloads`). Tests
that drive retries tick by tick need a zero backoff:
`RetryPolicy(maxRetries, initialBackoff = Duration.ZERO, maxBackoff = Duration.ZERO)`.

## From 0.2.x to 1.0.0

Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import java.time.Instant
* Created via [createPending] and progressed through [retry], [toDelivered],
* or [toFailed] — each returning a new immutable copy.
*/
data class OutboxEntry(
data class OutboxEntry @JvmOverloads constructor(
@get:JvmName("getOutboxId")
val outboxId: OutboxId,
val messageType: String,
Expand All @@ -21,14 +21,24 @@ data class OutboxEntry(
val lastAttempt: Instant?,
val lastError: String?,
val deliveryMetadata: String,
/**
* When a `PENDING` entry becomes eligible for its next delivery attempt (issue #131): [createdAt]
* for a new entry, later after a retriable failure. Only meaningful while the entry is `PENDING`.
*/
val nextAttemptAt: Instant = createdAt,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (API docs): Java callers lose the 11-argument constructor.

OutboxEntry's constructor has no @JvmOverloads, so the 11-argument JVM constructor is gone and Java callers of new OutboxEntry(...) get a compile error. The CHANGELOG only says Kotlin copy/default-argument callers must recompile. Either add @JvmOverloads constructor here, as RetryPolicy and OutboxProcessorProperties have, or mention Java constructor callers in the CHANGELOG.

) {
/** Returns a copy scheduled for another delivery attempt. */
fun retry(now: Instant, lastError: String): OutboxEntry = copy(
/**
* Returns a copy scheduled for another delivery attempt at [nextAttemptAt], which defaults to
* [now] (retry on the next poll).
*/
@JvmOverloads
fun retry(now: Instant, lastError: String, nextAttemptAt: Instant = now): OutboxEntry = copy(
status = OutboxStatus.PENDING,
updatedAt = now,
retries = retries + 1,
lastAttempt = now,
lastError = sanitizeLastError(lastError),
nextAttemptAt = nextAttemptAt,
)

/** Returns a copy marked as permanently failed. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,18 @@ class OutboxEntryProcessor(
check(entries.all { it.deliveryType in supportedDeliveryTypes }) {
"Batch contains an entry with no registered deliverer"
}
val outcomes = deliverer.deliverBatch(entries)
// Read after delivery, like process(): a retry's backoff counts from when the attempt ended, so
// a slow failure (e.g. a 30 s HTTP timeout) does not eat into it (issue #131).
val now = clock.instant()
return deliverer.deliverBatch(entries).map { (entry, result) -> applyResult(entry, result, now) }
return outcomes.map { (entry, result) -> applyResult(entry, result, now) }
}

private fun applyResult(entry: OutboxEntry, result: DeliveryResult, now: java.time.Instant): OutboxEntry = when (result) {
is DeliveryResult.Success -> entry.toDelivered(now)
is DeliveryResult.RetriableFailure ->
if (retryPolicy.shouldRetry(entry.retries)) {
entry.retry(now, result.error)
entry.retry(now, result.error, nextAttemptAt = now + retryPolicy.backoff(entry.retries + 1))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix: the backoff is measured from before delivery, so a slow failure uses up most or all of it.

In processBatch, now is read before deliverer.deliverBatch(entries) (line 34), so nextAttemptAt = batchStart + backoff. A slow failure is the typical shape of a real outage. For example, HttpMessageDeliverer has a 30 s exchange timeout and a 10 s connect timeout.

The single-entry process() path reads the clock after deliver(), so the two paths also disagree.

Suggestion: in processBatch, take a second clock.instant() after deliverBatch returns and use it as the base for nextAttemptAt. lastAttempt/updatedAt can keep the start time if that is intended. A unit test with a clock that advances inside deliverBatch would pin this down.

} else {
entry.toFailed(now, result.error)
}
Expand Down
Loading