Skip to content

Commit 4c90bb5

Browse files
endrju19rawoclaude
authored
Claim only delivery types supported by each worker (#120)
## Summary Workers now claim only `PENDING` entries for their registered delivery types. The processor rotates its starting type and uses at most two claim queries per batch, independent of the number of types: one for the starting type and, when needed, one `IN` query for the rest. A mismatched entry is also rejected before delivery. PostgreSQL gets a partial `(delivery_type, created_at, id) WHERE status = 'PENDING'` index; MySQL gets `(status, delivery_type, created_at, id)` plus an exact, binary type check (`CAST(... AS BINARY)`) for case-insensitive collations. The unrestricted `OutboxStore.claimPending(limit)` remains available to direct callers and keeps its existing index. Custom stores used by Okapi's processor must implement `RouteAwareOutboxStore`, and so must hand-written wrappers around a built-in store. Spring autoconfiguration rejects an unsafe store at startup and names its class in the error. ## Behaviour changes (see CHANGELOG) - **BREAKING:** `OutboxProcessor` requires a `RouteAwareOutboxStore`. - **BREAKING:** `OutboxEntryProcessor.process`/`processBatch` throw `IllegalStateException` for an entry with no registered deliverer. Previously such an entry was marked `FAILED`. Inside the processor the transaction rolls back and the row stays `PENDING`. - **BREAKING:** new migration `002__route_aware_claim_index`. - Cross-type claim order is no longer globally oldest first. When the starting type doesn't fill the batch, the leftover capacity goes to the other types alphabetically. Every type is still claimed first at least once every N batches, so none is starved. This keeps each claim a cheap ordered index range. ## Verification - `./gradlew ktlintCheck build test` passes on top of current main. - The HTTP batch concurrency test now uses a server-side request barrier. Every request must arrive before any successful response is sent; intentionally switching delivery to sequential makes the test fail. Both PR #120 and #121 include the same test fix. - Core, Spring Boot and integration tests passed locally on PostgreSQL and MySQL Docker containers, including migrations, concurrent heterogeneous workers, exact type matching, HTTP and Kafka. - `MysqlOutboxStoreTest` asserts that the route-aware claim raises no server warnings. The deprecated `BINARY` operator raised warning 1287 on every claim. - On a 1,000,000-row outbox with 900,000 older unsupported rows, a 100-type absent `IN` claim used the route index and took about 0.16 ms on PostgreSQL and 0.21 ms on MySQL in local measurements. The previous full PostgreSQL index produced a 165 ms scan for that case. Scripts and context are in `benchmarks/route-aware-claim/`. - There is a write cost from the additional index. In exploratory repeated local runs, PostgreSQL insert/update medians were 1.47×/1.07× versus the legacy index alone; MySQL medians were 1.18×/1.29×. These are workload-specific measurements, not throughput guarantees. ## Rollout The branch includes the current main, preserving #119 startup validation and #115 outbox-id headers. For manually managed schemas, the README links both migrations: apply 001 and 002 in order for a new installation, or apply 002 before starting updated workers on an existing schema. Run PostgreSQL migration 002 outside a transaction because it creates the index concurrently. It first drops any `INVALID` index left by an interrupted earlier attempt. `CREATE INDEX CONCURRENTLY` waits for every older transaction in the database, so run it when no long transactions are open, or apply it ahead of the rollout. Older workers can still claim unsupported types, so publish a new type only after all old processors are upgraded or stopped. Previously `FAILED` rows need separate recovery. Closes #116 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Rafał Wokacz <rafal.wokacz@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1 parent e5630a1 commit 4c90bb5

30 files changed

Lines changed: 753 additions & 19 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,35 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *
6565
`OutboxStore`/`OutboxPublisher` operation routes through `withConnection`. Construct one
6666
instance per `Database`, matching `ExposedTransactionRunner` and
6767
`ExposedTransactionContextValidator`. ([#96](https://github.com/softwaremill/okapi/issues/96))
68+
- **`OutboxProcessor` requires a `RouteAwareOutboxStore`.** The processor now claims only the
69+
delivery types its `OutboxEntryProcessor` handles, through the new
70+
`RouteAwareOutboxStore.claimPending(deliveryTypes, limit)`; `PostgresOutboxStore` and
71+
`MysqlOutboxStore` implement it. A custom `OutboxStore`, or a hand-written wrapper around a
72+
built-in store, must implement `RouteAwareOutboxStore` too: otherwise `processNext` throws, and
73+
Spring autoconfiguration fails at startup while the processor is enabled. `claimPending(limit)`
74+
stays on `OutboxStore` for direct callers. ([#120](https://github.com/softwaremill/okapi/pull/120))
75+
- **`OutboxEntryProcessor.process`/`processBatch` throw `IllegalStateException` for an entry whose
76+
delivery type has no registered deliverer.** Previously `CompositeMessageDeliverer` returned a
77+
`PermanentFailure` and the entry was marked `FAILED`. Inside `OutboxProcessor` the exception rolls
78+
the transaction back and the row stays `PENDING`; direct callers must handle the exception.
79+
([#120](https://github.com/softwaremill/okapi/pull/120))
80+
- **New migration `002__route_aware_claim_index`** adds a `(status, delivery_type, created_at, id)`
81+
index on MySQL and a partial `(delivery_type, created_at, id) WHERE status = 'PENDING'` index on
82+
PostgreSQL. The bundled Liquibase changelogs apply it automatically. If you manage the schema
83+
yourself, apply it before starting updated workers: the MySQL claim uses `FORCE INDEX` and fails
84+
without it, and the PostgreSQL script must run outside a transaction (`CREATE INDEX CONCURRENTLY`).
85+
([#120](https://github.com/softwaremill/okapi/pull/120))
86+
87+
### Fixed
88+
89+
- **A worker no longer marks rows it cannot deliver as `FAILED`.** Previously every worker claimed
90+
all `PENDING` rows, so a worker without a deliverer for a row's type turned it into a permanent
91+
failure. Each worker now claims only its registered delivery types, in at most two queries per
92+
batch. Workers on an older version still claim every type, so publish a newly introduced type only
93+
after all old processors are upgraded or stopped. Rows already marked `FAILED` are not retried
94+
automatically. Cross-type claim order is no longer globally oldest first: the processor rotates
95+
the starting type, and any leftover batch capacity goes to the other types alphabetically.
96+
([#116](https://github.com/softwaremill/okapi/issues/116))
6897

6998
## [1.0.0] — 2026-07-28
7099

‎README.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -85,14 +85,16 @@ Runnable, self-contained applications live in [okapi-examples](https://github.co
8585
## How it works
8686

8787
1. `publish()` writes a `PENDING` row to `okapi_outbox` in your transaction.
88-
2. A background scheduler polls for pending rows (every second by default), claiming them with `FOR UPDATE SKIP LOCKED` so workers do not process the same row concurrently.
88+
2. A background scheduler polls for pending rows (every second by default), claiming only the delivery types configured on that worker with `FOR UPDATE SKIP LOCKED` so workers do not process the same row concurrently.
8989
3. Each row goes to the transport matching its delivery type. Success marks it `DELIVERED`; a retriable failure leaves it `PENDING` while retry attempts remain; an exhausted retry budget or permanent failure marks it `FAILED`.
9090
4. A purger deletes delivered rows after a retention period.
9191

9292
### Guarantees and limits
9393

9494
- **Duplicate delivery is possible.** A crash between a successful delivery and the status update means the message may be sent again after restart. If processing a message more than once would cause unwanted effects, make the consumer idempotent. okapi sends your payload, your configured headers, and an `x-outbox-id` header to deduplicate on — see [Deduplicating on `x-outbox-id`](#deduplicating-on-x-outbox-id).
95-
- **Best-effort ordering.** Rows are claimed by `created_at`, oldest first. However, parallel delivery and retries mean messages may reach consumers in a different order. Strict delivery ordering is not guaranteed.
95+
- **Best-effort ordering.** Rows are claimed by `created_at, id` within each delivery type. The processor rotates which type it checks first between batches; when that type has fewer rows than the batch size, the remaining capacity goes to the other types in alphabetical order, so under mixed backlog alphabetically earlier types get more of it. Every type is still claimed first at least once every N batches, where N is the number of delivery types the worker handles. Across types, rows are no longer claimed globally oldest first. Parallel delivery and retries can also change arrival order. Strict delivery ordering across types or within a type is not guaranteed.
96+
- **Different workers can handle different delivery types.** Each worker claims only its registered types, using at most two claim queries per batch regardless of the number of types. A custom `OutboxStore` used for processing must implement `RouteAwareOutboxStore`; otherwise processing fails before an unsafe claim. The older `claimPending(limit)` method remains available for callers that need it, but Okapi's processor does not use it. Apply the new index migration before starting updated workers if you manage the schema yourself.
97+
- **Rolling upgrades need sequencing.** Workers running an older Okapi version can still claim rows of a newly introduced delivery type. Upgrade or stop all old processors before publishing that type. An upgrade does not automatically retry rows already marked `FAILED` by older workers.
9698
- **Failure classification is the transport's job.** Each deliverer decides what is retriable. HTTP: 5xx, 429, 408 and connection errors are retriable; other responses and TLS errors are permanent. Kafka: broker-side retriable exceptions are retried; authorization and configuration errors are not.
9799
- **Retry budget.** `okapi.processor.max-retries` (default 5) counts retries *after* the first attempt — six attempts in total before a row becomes `FAILED`. `FAILED` is terminal. Retriable messages become eligible again on the next processor poll; there is no per-message backoff.
98100

@@ -210,10 +212,14 @@ okapi ships Liquibase changelogs that create its table and indexes:
210212

211213
With `okapi-spring-boot` and Liquibase on the classpath, these run automatically against the configured `DataSource` at startup, tracked in dedicated Liquibase tables by default to avoid conflicts with the application's migration history.
212214

213-
If you use another migration tool, copy the SQL for your database into your application's migrations:
215+
If you use another migration tool, apply both SQL migrations for your database in order:
214216

215-
- [PostgreSQL SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql)
216-
- [MySQL SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql)
217+
| Database | 001: table and initial indexes | 002: route-aware claim index |
218+
|---|---|---|
219+
| PostgreSQL | [001 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) |
220+
| MySQL | [001 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) |
221+
222+
For an existing installation, apply `002` before starting updated workers. MySQL claims explicitly select the new index and fail if it is missing. PostgreSQL's `002` uses `CREATE INDEX CONCURRENTLY`, so run it outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.
217223

218224
okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:
219225

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Route-aware claiming on a large outbox
2+
3+
Exploratory local measurements with PostgreSQL 16 and MySQL 8.0 in Docker. The setup SQL creates 1,000,000 `PENDING` rows: 900,000 older rows have an unsupported `delivery_type`, while 100,000 newer rows are spread across 100 supported types. The legacy `(status, created_at)` index stays in place for the public unrestricted claim API. The new PostgreSQL index is partial on `(delivery_type, created_at, id) WHERE status = 'PENDING'`; MySQL adds `(status, delivery_type, created_at, id)`.
4+
5+
Run the matching `*-setup.sql` and `*-queries.sql` files with `psql` or `mysql`. They compare one route with `IN` over 100 present or absent routes, all with `FOR UPDATE SKIP LOCKED` and limit 100. The `*-client-100.sql` files preserve an earlier alternative: 100 separate empty claims sent through one local SQL client process. The processor now uses **at most two claims per batch**: one for a rotating starting type, then one `IN` claim for all remaining types when the first claim did not fill the batch.
6+
7+
| Case | PostgreSQL observed execution | MySQL observed execution |
8+
| --- | ---: | ---: |
9+
| One present route, first 100 rows | ~0.15 ms | ~0.26–0.49 ms |
10+
| `IN` with 100 absent routes, ordered by type and creation time | ~0.16 ms | ~0.21 ms |
11+
| 100 separate empty route claims, server-side loop | 1.57 ms | 2.04–2.37 ms |
12+
| 100 separate empty route claims through one local SQL client process | 24 ms | 15 ms |
13+
14+
Both current `IN` plans use the new route index, including PostgreSQL with a generic prepared plan. The PostgreSQL query must contain literal `status = 'PENDING'` so a generic plan can use its partial index. A preliminary full PostgreSQL index `(status, delivery_type, created_at, id)` produced a bad plan for the 100-absent-type query: it scanned 1,000,000 rows in ~165 ms. The partial index avoids that observed plan. The MySQL implementation also checks `BINARY delivery_type` to preserve exact type matching under case-insensitive database collations; the primary `IN` predicate still drives the index range scan.
15+
16+
The extra index costs writes. In five alternating series of 10,000 committed inserts and status updates against million-row tables, median times with / without the additional route index were:
17+
18+
| Operation | PostgreSQL partial index | MySQL full index |
19+
| --- | ---: | ---: |
20+
| Insert 10,000 | 57.46 / 39.03 ms (1.47×) | 94.95 / 80.63 ms (1.18×) |
21+
| Update 10,000 statuses | 88.36 / 82.56 ms (1.07×) | 204.30 / 158.82 ms (1.29×) |
22+
23+
The `*-write.sql` files provide a small, rollback-based write probe; the table above is from repeated committed runs on comparable million-row tables, so its figures are not directly reproduced by those scripts. Results depend on cache, hardware, row distribution and workload. The plans and scanned-row counts matter more than these timings. Removing the legacy index made unrestricted `claimPending(limit)` very slow in this dataset, so both indexes remain. Integration tests separately cover migrations, route isolation, exact type matching, multiple routes and concurrent workers on both databases.

0 commit comments

Comments
 (0)