Repository navigation
Commit 4c90bb5
Claim only delivery types supported by each worker (#120)
## Summary
Workers now claim only `PENDING` entries for their registered delivery
types. The processor rotates its starting type and uses at most two
claim queries per batch, independent of the number of types: one for the
starting type and, when needed, one `IN` query for the rest. A
mismatched entry is also rejected before delivery.
PostgreSQL gets a partial `(delivery_type, created_at, id) WHERE status
= 'PENDING'` index; MySQL gets `(status, delivery_type, created_at, id)`
plus an exact, binary type check (`CAST(... AS BINARY)`) for
case-insensitive collations. The unrestricted
`OutboxStore.claimPending(limit)` remains available to direct callers
and keeps its existing index. Custom stores used by Okapi's processor
must implement `RouteAwareOutboxStore`, and so must hand-written
wrappers around a built-in store. Spring autoconfiguration rejects an
unsafe store at startup and names its class in the error.
## Behaviour changes (see CHANGELOG)
- **BREAKING:** `OutboxProcessor` requires a `RouteAwareOutboxStore`.
- **BREAKING:** `OutboxEntryProcessor.process`/`processBatch` throw
`IllegalStateException` for an entry with no registered deliverer.
Previously such an entry was marked `FAILED`. Inside the processor the
transaction rolls back and the row stays `PENDING`.
- **BREAKING:** new migration `002__route_aware_claim_index`.
- Cross-type claim order is no longer globally oldest first. When the
starting type doesn't fill the batch, the leftover capacity goes to the
other types alphabetically. Every type is still claimed first at least
once every N batches, so none is starved. This keeps each claim a cheap
ordered index range.
## Verification
- `./gradlew ktlintCheck build test` passes on top of current main.
- The HTTP batch concurrency test now uses a server-side request
barrier. Every request must arrive before any successful response is
sent; intentionally switching delivery to sequential makes the test
fail. Both PR #120 and #121 include the same test fix.
- Core, Spring Boot and integration tests passed locally on PostgreSQL
and MySQL Docker containers, including migrations, concurrent
heterogeneous workers, exact type matching, HTTP and Kafka.
- `MysqlOutboxStoreTest` asserts that the route-aware claim raises no
server warnings. The deprecated `BINARY` operator raised warning 1287 on
every claim.
- On a 1,000,000-row outbox with 900,000 older unsupported rows, a
100-type absent `IN` claim used the route index and took about 0.16 ms
on PostgreSQL and 0.21 ms on MySQL in local measurements. The previous
full PostgreSQL index produced a 165 ms scan for that case. Scripts and
context are in `benchmarks/route-aware-claim/`.
- There is a write cost from the additional index. In exploratory
repeated local runs, PostgreSQL insert/update medians were 1.47×/1.07×
versus the legacy index alone; MySQL medians were 1.18×/1.29×. These are
workload-specific measurements, not throughput guarantees.
## Rollout
The branch includes the current main, preserving #119 startup validation
and #115 outbox-id headers. For manually managed schemas, the README
links both migrations: apply 001 and 002 in order for a new
installation, or apply 002 before starting updated workers on an
existing schema.
Run PostgreSQL migration 002 outside a transaction because it creates
the index concurrently. It first drops any `INVALID` index left by an
interrupted earlier attempt. `CREATE INDEX CONCURRENTLY` waits for every
older transaction in the database, so run it when no long transactions
are open, or apply it ahead of the rollout.
Older workers can still claim unsupported types, so publish a new type
only after all old processors are upgraded or stopped. Previously
`FAILED` rows need separate recovery.
Closes #116
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Rafał Wokacz <rafal.wokacz@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>1 parent e5630a1 commit 4c90bb5
30 files changed
Lines changed: 753 additions & 19 deletions
File tree
- benchmarks/route-aware-claim
- okapi-core/src
- main/kotlin/com/softwaremill/okapi/core
- test/kotlin/com/softwaremill/okapi/core
- okapi-integration-tests/src/test/kotlin/com/softwaremill/okapi/test
- concurrency
- store
- okapi-mysql/src/main
- kotlin/com/softwaremill/okapi/mysql
- resources/com/softwaremill/okapi/db/mysql
- okapi-postgres/src/main
- kotlin/com/softwaremill/okapi/postgres
- resources/com/softwaremill/okapi/db/postgres
- okapi-spring-boot/src
- main/kotlin/com/softwaremill/okapi/springboot
- test/kotlin/com/softwaremill/okapi/springboot
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
68 | 97 | | |
69 | 98 | | |
70 | 99 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
96 | 98 | | |
97 | 99 | | |
98 | 100 | | |
| |||
210 | 212 | | |
211 | 213 | | |
212 | 214 | | |
213 | | - | |
| 215 | + | |
214 | 216 | | |
215 | | - | |
216 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
217 | 223 | | |
218 | 224 | | |
219 | 225 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
0 commit comments