Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,27 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *
they try again on their next one. A runner call inside an existing transaction, and every other
Exposed transaction in the application, keep their configured retries.
([#134](https://github.com/softwaremill/okapi/issues/134))
- **One rejected `last_error` no longer rolls back a whole batch, whatever the cause.** #130 removed the
two known values the database rejected, but any other value still made the batch update fail, rolling
back the batch's results, delivered entries included, which were then delivered again on every tick.
When the batch update fails, `PostgresOutboxStore` and `MysqlOutboxStore` now roll back to a savepoint
and write the rows one by one, each under its own savepoint. A row the database still rejects is
written with a placeholder `last_error`, keeping its status and `retries`, and the original message is
logged at WARN with the entry id. Only a value error (SQLState class 22, or MySQL error 1366) gets the
placeholder; any other failure, such as a deadlock, a timeout or a lost connection, still fails the
batch as before. A successful batch costs two extra statements, `SAVEPOINT` and `RELEASE SAVEPOINT`.
On PostgreSQL the row-by-row fallback opens a subtransaction per row; above 64 rows that overflows the
per-backend subtransaction cache, so keep `batch-size` at or below 64 if rejected values are frequent.
([#141](https://github.com/softwaremill/okapi/issues/141))
- **MySQL: `last_error` is `utf8mb4` (new migration `005__last_error_utf8mb4`).** 001 created
`okapi_outbox` in the schema's default character set. On a `latin1` or `utf8mb3` schema, common on
servers upgraded from MySQL 5.7, an error message with an emoji or any other character outside that
set was rejected with `Incorrect string value`. 005 converts the column to `utf8mb4`. Its
precondition skips it where the column is already `utf8mb4`, the MySQL 8 default. Otherwise it copies
the table: it first waits up to 30 s for open transactions on `okapi_outbox`, queuing every other
statement on the table, and then blocks writes, `publish()` included, until the copy finishes. On a
large outbox, run it in a quiet period.
([#141](https://github.com/softwaremill/okapi/issues/141))
- **MySQL: the processor no longer blocks `publish()` for the duration of delivery.** Under MySQL's
default `REPEATABLE READ`, the claim's `FOR UPDATE SKIP LOCKED` took gap locks on the claim index;
with a backlog smaller than the batch, the gap ran to the end of the claimed type's range, exactly
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,16 @@ With `okapi-spring-boot` and Liquibase on the classpath, these run automatically
If you use another migration tool, apply every SQL migration for your database, in order:

- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/003__timestamptz_columns.sql) `TIMESTAMPTZ` timestamp columns, [004](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/004__retry_backoff.sql) retry backoff column and claim index (its second changeset must run outside a transaction)
- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns, [004](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql) retry backoff column and claim index
- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns, [004](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/004__retry_backoff.sql) retry backoff column and claim index, [005](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/005__last_error_utf8mb4.sql) `utf8mb4` `last_error` (a no-op where it already is)

For an existing installation, apply every new migration (up to `004`) before starting updated workers. MySQL claims explicitly select the claim index from `004` and fail if it is missing. PostgreSQL's `002` and the index changeset of `004` use `CREATE INDEX CONCURRENTLY`, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.
For an existing installation, apply every new migration (up to `004`, and on MySQL `005`) before starting updated workers. MySQL claims explicitly select the claim index from `004` and fail if it is missing. PostgreSQL's `002` and the index changeset of `004` use `CREATE INDEX CONCURRENTLY`, so run them outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.

PostgreSQL's `003` converts the timestamp columns from `TIMESTAMP` (the okapi JVM's local wall-clock time) to `TIMESTAMPTZ`. It reads existing values in the session `TimeZone`; pgjdbc sets that to the JVM default zone, so the bundled Liquibase changelog converts them correctly when it runs inside the application. When applying it by hand, run it inside a transaction (`psql -1`) after `SET TIME ZONE '<zone your okapi JVMs run in>'` in the same session. It takes an `ACCESS EXCLUSIVE` lock: it first waits (up to 30 s, then fails and retries on the next start) for open transactions on `okapi_outbox`, queuing every other statement on the table meanwhile, and then, unless the session zone is UTC (PostgreSQL 12+), rewrites the table while reads and writes wait. 1.x instances keep working after it, apart from one failed poll per pooled connection. See UPGRADING.md for when the in-app conversion uses the wrong zone. See [UPGRADING.md](UPGRADING.md#from-1x-to-200).

MySQL's `003` converts the timestamp columns from `TIMESTAMP(6)`, which ends in January 2038, to `DATETIME(6)` holding UTC. Stop every 1.x instance, publishers included, before it runs: 1.x writes these columns in the JVM zone. It copies the table; it first waits (up to 30 s, then fails and retries on the next start) for open transactions on `okapi_outbox` to finish, queuing every other statement on the table meanwhile, and then blocks writes, including `publish()`, until the copy finishes. See [UPGRADING.md](UPGRADING.md#from-1x-to-200), including when rows written by 1.x can read back shifted.

MySQL's `005` converts `last_error` to `utf8mb4`, so that error messages with characters outside a `latin1` or `utf8mb3` schema's character set can be stored. Where the column is already `utf8mb4`, the MySQL 8 default, it does nothing. Otherwise it copies the table like `003`: it waits up to 30 s for open transactions on `okapi_outbox`, queuing every other statement on the table meanwhile, and then blocks writes, including `publish()`, until the copy finishes. Without it, okapi still works and stores a placeholder for an error message the column rejects.

okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:

| Table | Purpose |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ data class OutboxEntry @JvmOverloads constructor(
* rest is cut to [MAX_LAST_ERROR_LENGTH], never between the two halves of a surrogate pair,
* which MySQL would reject as an invalid string.
*
* The built-in stores also survive a value rejected for any other reason, by writing a
* placeholder in its place (issue #141). Sanitizing still matters, because the placeholder
* loses the message.
*
* [retry] and [toFailed] apply it, and the built-in stores apply it again when writing, so an
* entry built with `copy(lastError = ...)` is covered too. Custom [OutboxStore]s should call
* it before writing `last_error`. It is idempotent.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ interface OutboxStore {
* Updates a batch of entries after a delivery attempt in a single roundtrip where
* possible. Default loops over [updateAfterProcessing] one at a time; storage modules
* override this with a single JDBC `executeBatch()` call.
*
* It runs in the claim's transaction, so if it throws, the results of the whole batch roll
* back, including entries already delivered, and they are delivered again. The built-in stores
* therefore write the rows one by one when the batch update fails, and store a placeholder
* `last_error` for a row the database still rejects (issue #141).
*/
fun updateAfterProcessingBatch(entries: List<OutboxEntry>): List<OutboxEntry> = entries.map { updateAfterProcessing(it) }
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
package com.softwaremill.okapi.test.store

import com.mysql.cj.jdbc.MysqlDataSource
import com.softwaremill.okapi.core.DeliveryInfo
import com.softwaremill.okapi.core.DeliveryResult
import com.softwaremill.okapi.core.MessageDeliverer
import com.softwaremill.okapi.core.OutboxEntry
import com.softwaremill.okapi.core.OutboxEntryProcessor
import com.softwaremill.okapi.core.OutboxMessage
import com.softwaremill.okapi.core.OutboxProcessor
import com.softwaremill.okapi.core.OutboxStatus
import com.softwaremill.okapi.core.RetryPolicy
import com.softwaremill.okapi.mysql.MysqlOutboxStore
import com.softwaremill.okapi.test.support.JdbcConnectionProvider
import io.kotest.assertions.throwables.shouldThrowAny
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
import io.kotest.matchers.string.shouldStartWith
import liquibase.Liquibase
import liquibase.database.DatabaseFactory
import liquibase.database.jvm.JdbcConnection
import liquibase.resource.ClassLoaderResourceAccessor
import org.testcontainers.containers.MySQLContainer
import java.sql.Connection
import java.sql.DriverManager
import java.time.Clock

private const val CHANGELOG = "com/softwaremill/okapi/db/mysql/changelog.xml"

/** Changesets in 001 to 004, i.e. everything before 005. */
private const val CHANGESETS_BEFORE_005 = 7

private const val EMOJI_ERROR = "HTTP 503: upstream is on fire 🔥"

private object CharsetTestDeliveryInfo : DeliveryInfo {
override val type: String = "charset-test"
override fun serialize(): String = """{"type":"charset-test"}"""
}

private object EmojiErrorDeliverer : MessageDeliverer {
override val type: String = CharsetTestDeliveryInfo.type
override fun deliver(entry: OutboxEntry): DeliveryResult = when (entry.messageType) {
"healthy" -> DeliveryResult.Success
else -> DeliveryResult.RetriableFailure(EMOJI_ERROR)
}
}

/**
* Issue #141 on a MySQL schema whose default character set is latin1, as on servers upgraded from
* 5.7. 001 creates `okapi_outbox` in the schema's character set, where an emoji in `last_error` is
* rejected; changeset 005 converts the column to utf8mb4.
*/
class MysqlLastErrorCharsetTest : FunSpec({
val container = MySQLContainer<Nothing>("mysql:8.0")
val clock = Clock.systemUTC()

beforeSpec {
container.start()
DriverManager.getConnection(container.jdbcUrl, "root", container.password).use { conn ->
conn.createStatement().use {
it.execute("CREATE DATABASE legacy_before_005 CHARACTER SET latin1")
it.execute("CREATE DATABASE legacy_upgraded CHARACTER SET latin1")
it.execute("CREATE DATABASE modern")
}
}
}
afterSpec { container.stop() }

fun url(database: String) = container.jdbcUrl.replaceAfterLast('/', database)

fun connect(database: String): Connection = DriverManager.getConnection(url(database), "root", container.password)

/** Runs the first [changes] changesets (all of them when null). */
fun migrate(database: String, changes: Int? = null) {
connect(database).use { conn ->
Liquibase(
CHANGELOG,
ClassLoaderResourceAccessor(),
DatabaseFactory.getInstance().findCorrectDatabaseImplementation(JdbcConnection(conn)),
).use { if (changes != null) it.update(changes, "") else it.update("") }
}
}

fun jdbc(database: String) = JdbcConnectionProvider(
MysqlDataSource().apply {
setURL(url(database))
user = "root"
setPassword(container.password)
},
)

fun query(database: String, sql: String): String? = connect(database).use { conn ->
conn.createStatement().use { stmt -> stmt.executeQuery(sql).use { rs -> if (rs.next()) rs.getString(1) else null } }
}

fun lastErrorCharset(database: String) = query(
database,
"SELECT CHARACTER_SET_NAME FROM information_schema.COLUMNS " +
"WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox' AND COLUMN_NAME = 'last_error'",
)

fun execType005(database: String) = query(database, "SELECT EXECTYPE FROM DATABASECHANGELOG WHERE ID = '005'")

/** Seeds a `healthy` and a `poisoned` entry, processes them in one batch and returns the stored rows. */
fun processHealthyAndPoisoned(database: String): Map<String, Pair<OutboxStatus, String?>> {
connect(database).use { conn -> conn.createStatement().use { it.execute("DELETE FROM okapi_outbox") } }
val jdbc = jdbc(database)
val store = MysqlOutboxStore(jdbc)
jdbc.withTransaction {
listOf("healthy", "poisoned").forEach {
store.persist(OutboxEntry.createPending(OutboxMessage(it, "{}"), CharsetTestDeliveryInfo, clock.instant()))
}
}
val processor = OutboxProcessor(store, OutboxEntryProcessor(EmojiErrorDeliverer, RetryPolicy(maxRetries = 5), clock))
jdbc.withTransaction { processor.processNext(10) } shouldBe 2

return connect(database).use { conn ->
conn.createStatement().use { stmt ->
stmt.executeQuery("SELECT message_type, status, last_error FROM okapi_outbox").use { rs ->
generateSequence {
if (rs.next()) rs.getString(1) to (OutboxStatus.from(rs.getString(2)) to rs.getString(3)) else null
}.toMap()
}
}
}
}

test("before 005, a latin1 last_error rejects an emoji, and the store keeps the batch with a placeholder error") {
migrate("legacy_before_005", CHANGESETS_BEFORE_005)
lastErrorCharset("legacy_before_005") shouldBe "latin1"
// Control: the raw value really is rejected.
shouldThrowAny {
connect("legacy_before_005").use { conn ->
conn.prepareStatement(
"INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, delivery_metadata, last_error) " +
"VALUES (UUID(), 'm', '{}', 'charset-test', '{}', ?)",
).use {
it.setString(1, EMOJI_ERROR)
it.executeUpdate()
}
}
}

val rows = processHealthyAndPoisoned("legacy_before_005")

rows.getValue("healthy").first shouldBe OutboxStatus.DELIVERED
rows.getValue("poisoned").first shouldBe OutboxStatus.PENDING
checkNotNull(rows.getValue("poisoned").second) shouldStartWith "okapi: the database rejected this entry's error message"
}

test("005 converts last_error to utf8mb4, which stores the emoji") {
migrate("legacy_upgraded", CHANGESETS_BEFORE_005)
lastErrorCharset("legacy_upgraded") shouldBe "latin1"
migrate("legacy_upgraded")
execType005("legacy_upgraded") shouldBe "EXECUTED"
lastErrorCharset("legacy_upgraded") shouldBe "utf8mb4"

val rows = processHealthyAndPoisoned("legacy_upgraded")

rows.getValue("healthy").first shouldBe OutboxStatus.DELIVERED
rows.getValue("poisoned") shouldBe (OutboxStatus.PENDING to EMOJI_ERROR)
}

test("005 does nothing where last_error is already utf8mb4") {
migrate("modern")
lastErrorCharset("modern") shouldBe "utf8mb4"
execType005("modern") shouldBe "MARK_RAN"
}
})
Loading