Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,35 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *
`OutboxStore`/`OutboxPublisher` operation routes through `withConnection`. Construct one
instance per `Database`, matching `ExposedTransactionRunner` and
`ExposedTransactionContextValidator`. ([#96](https://github.com/softwaremill/okapi/issues/96))
- **`OutboxProcessor` requires a `RouteAwareOutboxStore`.** The processor now claims only the
delivery types its `OutboxEntryProcessor` handles, through the new
`RouteAwareOutboxStore.claimPending(deliveryTypes, limit)`; `PostgresOutboxStore` and
`MysqlOutboxStore` implement it. A custom `OutboxStore`, or a hand-written wrapper around a
built-in store, must implement `RouteAwareOutboxStore` too: otherwise `processNext` throws, and
Spring autoconfiguration fails at startup while the processor is enabled. `claimPending(limit)`
stays on `OutboxStore` for direct callers. ([#120](https://github.com/softwaremill/okapi/pull/120))
- **`OutboxEntryProcessor.process`/`processBatch` throw `IllegalStateException` for an entry whose
delivery type has no registered deliverer.** Previously `CompositeMessageDeliverer` returned a
`PermanentFailure` and the entry was marked `FAILED`. Inside `OutboxProcessor` the exception rolls
the transaction back and the row stays `PENDING`; direct callers must handle the exception.
([#120](https://github.com/softwaremill/okapi/pull/120))
- **New migration `002__route_aware_claim_index`** adds a `(status, delivery_type, created_at, id)`
index on MySQL and a partial `(delivery_type, created_at, id) WHERE status = 'PENDING'` index on
PostgreSQL. The bundled Liquibase changelogs apply it automatically. If you manage the schema
yourself, apply it before starting updated workers: the MySQL claim uses `FORCE INDEX` and fails
without it, and the PostgreSQL script must run outside a transaction (`CREATE INDEX CONCURRENTLY`).
([#120](https://github.com/softwaremill/okapi/pull/120))

### Fixed

- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed
all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent
failure. Each worker now claims only its registered delivery types, in at most two queries per
batch. Workers on an older version still claim every type, so publish a newly introduced type only
after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried
automatically. Cross-type claim order is no longer globally oldest first: the processor rotates
the starting type, and any leftover batch capacity goes to the other types alphabetically.
([#116](https://github.com/softwaremill/okapi/issues/116))

## [1.0.0] — 2026-07-28

Expand Down
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,14 +85,16 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co
## How it works

1. `publish()` writes a `PENDING` row to `okapi_outbox` in your transaction.
2. A background scheduler polls for pending rows (every second by default), claiming them with `FOR UPDATE SKIP LOCKED` so workers do not process the same row concurrently.
2. A background scheduler polls for pending rows (every second by default), claiming only the delivery types configured on that worker with `FOR UPDATE SKIP LOCKED` so workers do not process the same row concurrently.
3. Each row goes to the transport matching its delivery type. Success marks it `DELIVERED`; a retriable failure leaves it `PENDING` while retry attempts remain; an exhausted retry budget or permanent failure marks it `FAILED`.
4. A purger deletes delivered rows after a retention period.

### Guarantees and limits

- **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id).
- **Best-effort ordering.** Rows are claimed by `created_at`, oldest first. However, parallel delivery and retries mean messages may reach consumers in a different order. Strict delivery ordering is not guaranteed.
- **Best-effort ordering.** Rows are claimed by `created_at, id` within each delivery type. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.
- **Different workers can handle different delivery types.** Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom `OutboxStore` used for processing must implement `RouteAwareOutboxStore`; otherwise processing fails before an unsafe claim. The older `claimPending(limit)` method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself.
- **Rolling upgrades need sequencing.** Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked `FAILED` by older workers.
- **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not.
- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff.

Expand Down Expand Up @@ -210,10 +212,14 @@ okapi ships Liquibase changelogs that create its table and indexes:

With `okapi-spring-boot` and Liquibase on the classpath, these run automatically against the configured `DataSource` at startup, tracked in dedicated Liquibase tables by default to avoid conflicts with the application's migration history.

If you use another migration tool, copy the SQL for your database into your application's migrations:
If you use another migration tool, apply both SQL migrations for your database in order:

- [PostgreSQL SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql)
- [MySQL SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql)
| Database | 001: table and initial indexes | 002: route-aware claim index |
|---|---|---|
| PostgreSQL | [001 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) |
| MySQL | [001 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) |

For an existing installation, apply `002` before starting updated workers. MySQL claims explicitly select the new index and fail if it is missing. PostgreSQL's `002` uses `CREATE INDEX CONCURRENTLY`, so run it outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.

okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:

Expand Down
23 changes: 23 additions & 0 deletions benchmarks/route-aware-claim/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Route-aware claiming on a large outbox

Exploratory local measurements with PostgreSQL 16 and MySQL 8.0 in Docker. The setup SQL creates 1,000,000 `PENDING` rows: 900,000 older rows have an unsupported `delivery_type`, while 100,000 newer rows are spread across 100 supported types. The legacy `(status, created_at)` index stays in place for the public unrestricted claim API. The new PostgreSQL index is partial on `(delivery_type, created_at, id) WHERE status = 'PENDING'`; MySQL adds `(status, delivery_type, created_at, id)`.

Run the matching `*-setup.sql` and `*-queries.sql` files with `psql` or `mysql`. They compare one route with `IN` over 100 present or absent routes, all with `FOR UPDATE SKIP LOCKED` and limit 100. The `*-client-100.sql` files preserve an earlier alternative: 100 separate empty claims sent through one local SQL client process. The processor now uses **at most two claims per batch**: one for a rotating starting type, then one `IN` claim for all remaining types when the first claim did not fill the batch.

| Case | PostgreSQL observed execution | MySQL observed execution |
| --- | ---: | ---: |
| One present route, first 100 rows | ~0.15 ms | ~0.26–0.49 ms |
| `IN` with 100 absent routes, ordered by type and creation time | ~0.16 ms | ~0.21 ms |
| 100 separate empty route claims, server-side loop | 1.57 ms | 2.04–2.37 ms |
| 100 separate empty route claims through one local SQL client process | 24 ms | 15 ms |

Both current `IN` plans use the new route index, including PostgreSQL with a generic prepared plan. The PostgreSQL query must contain literal `status = 'PENDING'` so a generic plan can use its partial index. A preliminary full PostgreSQL index `(status, delivery_type, created_at, id)` produced a bad plan for the 100-absent-type query: it scanned 1,000,000 rows in ~165 ms. The partial index avoids that observed plan. The MySQL implementation also checks `BINARY delivery_type` to preserve exact type matching under case-insensitive database collations; the primary `IN` predicate still drives the index range scan.

The extra index costs writes. In five alternating series of 10,000 committed inserts and status updates against million-row tables, median times with / without the additional route index were:

| Operation | PostgreSQL partial index | MySQL full index |
| --- | ---: | ---: |
| Insert 10,000 | 57.46 / 39.03 ms (1.47×) | 94.95 / 80.63 ms (1.18×) |
| Update 10,000 statuses | 88.36 / 82.56 ms (1.07×) | 204.30 / 158.82 ms (1.29×) |

The `*-write.sql` files provide a small, rollback-based write probe; the table above is from repeated committed runs on comparable million-row tables, so its figures are not directly reproduced by those scripts. Results depend on cache, hardware, row distribution and workload. The plans and scanned-row counts matter more than these timings. Removing the legacy index made unrestricted `claimPending(limit)` very slow in this dataset, so both indexes remain. Integration tests separately cover migrations, route isolation, exact type matching, multiple routes and concurrent workers on both databases.
Loading