Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,20 @@ Until `1.0.0`, breaking changes may appear in any release and are flagged with *
yourself, apply it before starting updated workers: the MySQL claim uses `FORCE INDEX` and fails
without it, and the PostgreSQL script must run outside a transaction (`CREATE INDEX CONCURRENTLY`).
([#120](https://github.com/softwaremill/okapi/pull/120))
- **MySQL: timestamp columns are `DATETIME(6)` in UTC (new migration `003__datetime_columns`).**
`created_at`, `updated_at` and `last_attempt` were `TIMESTAMP(6)`, which cannot hold anything
after `2038-01-19 03:14:07 UTC`; from then on every `publish()` would fail. The migration converts
existing values without shifting them (it pins the session to UTC for the conversion) and drops the
`CURRENT_TIMESTAMP` column defaults, which okapi never relied on. `MysqlOutboxStore` now binds and
reads these columns as UTC `LocalDateTime`, so stored values no longer depend on the JVM time zone,
the session `time_zone` or Connector/J's `connectionTimeZone`/`preserveInstants`. Stop every 1.x
instance, publishers included, before the migration runs (1.x writes these columns in the JVM
zone), and drain the outbox first if your JVMs and MySQL session ran in different zones: rows 1.x
wrote then were stored shifted by the difference. The migration copies the table, waits at most
30 s for a metadata lock (then fails and retries on the next start) and blocks writes while it
copies. Anything else reading these columns directly must now treat them as UTC wall-clock values.
See UPGRADING.md.
([#138](https://github.com/softwaremill/okapi/issues/138))

## [1.0.0] — 2026-07-28

Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,15 +213,15 @@ okapi ships Liquibase changelogs that create its table and indexes:

With `okapi-spring-boot` and Liquibase on the classpath, these run automatically against the configured `DataSource` at startup, tracked in dedicated Liquibase tables by default to avoid conflicts with the application's migration history.

If you use another migration tool, apply both SQL migrations for your database in order:
If you use another migration tool, apply every SQL migration for your database, in order:

| Database | 001: table and initial indexes | 002: route-aware claim index |
|---|---|---|
| PostgreSQL | [001 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) |
| MySQL | [001 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) | [002 SQL](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) |
- **PostgreSQL:** [001](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-postgres/src/main/resources/com/softwaremill/okapi/db/postgres/002__route_aware_claim_index.sql) route-aware claim index
- **MySQL:** [001](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/001__create_okapi_outbox_table.sql) table and initial indexes, [002](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/002__route_aware_claim_index.sql) route-aware claim index, [003](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) `DATETIME(6)` timestamp columns

For an existing installation, apply `002` before starting updated workers. MySQL claims explicitly select the new index and fail if it is missing. PostgreSQL's `002` uses `CREATE INDEX CONCURRENTLY`, so run it outside a transaction when using another migration tool. The bundled Liquibase changelog handles this automatically. `CREATE INDEX CONCURRENTLY` waits for every transaction in the database that started before it, not only those on `okapi_outbox`, so long-running transactions (including old workers delivering a claimed batch) delay it, and with Liquibase they delay application startup while the changelog lock is held. Run it when no long transactions are open, or apply it ahead of the rollout.

MySQL's `003` converts the timestamp columns from `TIMESTAMP(6)`, which ends in January 2038, to `DATETIME(6)` holding UTC. Stop every 1.x instance, publishers included, before it runs: 1.x writes these columns in the JVM zone. It copies the table; it first waits (up to 30 s, then fails and retries on the next start) for open transactions on `okapi_outbox` to finish, queuing every other statement on the table meanwhile, and then blocks writes, including `publish()`, until the copy finishes. See [UPGRADING.md](UPGRADING.md#from-1x-to-200), including when rows written by 1.x can read back shifted.

okapi stores messages in the fixed `okapi_outbox` table. When the built-in Liquibase integration is used, it also uses two dedicated migration tracking tables:

| Table | Purpose |
Expand Down
32 changes: 32 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,38 @@
This guide contains actions required when upgrading between incompatible okapi versions.
For the full release history, see [CHANGELOG.md](CHANGELOG.md).

## From 1.x to 2.0.0

### MySQL: `DATETIME(6)` timestamp columns (#138)

Changeset `003__datetime_columns` converts `created_at`, `updated_at` and `last_attempt` from
`TIMESTAMP(6)` to `DATETIME(6)` holding UTC. With the bundled Liquibase changelog it runs
automatically at startup; otherwise apply
[`003__datetime_columns.sql`](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql)
yourself, **including its `SET time_zone` statements**, before starting 2.0.0 workers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix (docs): say that 1.x instances must be stopped once 003 runs, not only that 2.0 must not start before it.

The Spring Boot autoconfig runs 003 when the first 2.0 instance starts. In a rolling deploy, 1.x instances are still publishing and claiming at that point. After the migration, 1.x still binds java.sql.Timestamp. Connector/J renders a Timestamp in connectionTimeZone (by default the JVM zone) and the DATETIME column stores that text unconverted. So with a non-UTC JVM, every row 1.x writes or updates in that window gets a local wall-clock value that 2.0 then reads as UTC. 1.x also misreads the rows 2.0 wrote. The wrong values stay in the table for good.

Suggest stating it here, in the README and in the CHANGELOG: either stop every 1.x instance before 003 runs (no rolling deploy across this version), or note that mixed operation is only safe when the 1.x JVMs run in UTC.


- **Stop every 1.x instance, publishers included, before 003 runs.** 1.x binds these columns as
`java.sql.Timestamp`, which Connector/J shifts into the JVM (or `connectionTimeZone`) zone. Against
`DATETIME` columns a 1.x instance on a non-UTC JVM writes local wall-clock time instead of UTC and
misreads rows written by 2.0. A rolling deploy that mixes the two is only safe if every 1.x JVM
runs in UTC.
- **Rows written by 1.x may read back shifted if the JVM and MySQL zones differed.** Without
`connectionTimeZone`, 1.x sent `Timestamp`s as JVM-local wall-clock time and MySQL interpreted them
in the session `time_zone`; when the two zones differed, the stored instant was off by the
difference, and 1.x reading it back undid the shift. 003 converts what is actually stored, so 2.0
reads such rows off by that difference: old `PENDING` rows sort out of order, the lag gauge is skewed
and old `DELIVERED` rows are purged early or late. If your JVMs and MySQL session ran in different
zones, drain the outbox (let every `PENDING` row be delivered) before upgrading.
- **Locking.** The `ALTER TABLE` copies `okapi_outbox` (`ALGORITHM=COPY, LOCK=SHARED`). Before it
starts it waits for a metadata lock behind every open transaction on the table, such as a 1.x worker
delivering a claimed batch, and while it waits every new statement on the table, reads included,
queues behind it. The changeset bounds that wait with `lock_wait_timeout = 30` seconds; on timeout
the migration fails and runs again on the next start. During the copy, writes, including
`publish()`, wait. Stop the workers first, and purge delivered rows or pick a quiet period on a
large table.
- Queries or dashboards that read these columns directly now get UTC wall-clock values, not values
converted to the session time zone.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix (docs), related to the review body: besides direct readers, existing rows may themselves read back shifted after the upgrade. That happens when 1.x ran with JVM zone ≠ MySQL session zone and connectionTimeZone unset: 1.x stored shifted instants and hid the shift on read. Worth a bullet here, e.g. "drain the outbox first if your JVM and MySQL time zones differed".


## From 0.2.x to 1.0.0

Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
package com.softwaremill.okapi.test.store

import com.mysql.cj.jdbc.MysqlDataSource
import com.softwaremill.okapi.core.OutboxStatus
import com.softwaremill.okapi.mysql.MysqlOutboxStore
import com.softwaremill.okapi.test.support.JdbcConnectionProvider
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe
import liquibase.Liquibase
import liquibase.database.DatabaseFactory
import liquibase.database.jvm.JdbcConnection
import liquibase.resource.ClassLoaderResourceAccessor
import org.testcontainers.containers.MySQLContainer
import java.sql.Connection
import java.sql.DriverManager
import java.time.Instant
import java.util.Properties

private const val CHANGELOG = "com/softwaremill/okapi/db/mysql/changelog.xml"

/**
* Upgrade path for issue #138: rows written while the columns were TIMESTAMP must keep their
* instants when changeset 003 converts them to DATETIME, even though the migrating session runs in
* another time zone (MODIFY converts through the session zone).
*/
class MysqlDatetimeMigrationTest : FunSpec({
val container = MySQLContainer<Nothing>("mysql:8.0")

beforeSpec { container.start() }
afterSpec { container.stop() }

/** A connection whose session time zone is +05:00, as a non-UTC server or pool would give. */
fun offsetSessionConnection(): Connection = DriverManager.getConnection(
container.jdbcUrl,
Properties().apply {
setProperty("user", container.username)
setProperty("password", container.password)
setProperty("sessionVariables", "time_zone='+05:00'")
},
)

test("changeset 003 keeps existing instants and restores the session time zone") {
offsetSessionConnection().use { conn ->
val defaultLockWaitTimeout = conn.createStatement().use { stmt ->
stmt.executeQuery("SELECT @@session.lock_wait_timeout").use { rs ->
rs.next()
rs.getLong(1)
}
}
val liquibase =
Liquibase(
CHANGELOG,
ClassLoaderResourceAccessor(),
DatabaseFactory.getInstance().findCorrectDatabaseImplementation(JdbcConnection(conn)),
)
liquibase.update(2, "")

// A TIMESTAMP literal is read in the session zone: 17:00 at +05:00 is 12:00Z.
conn.createStatement().use {
it.execute(
"""
INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, delivery_metadata)
VALUES ('00000000-0000-0000-0000-000000000001', 'm', '{}', 'http', 'PENDING',
'2024-06-01 17:00:00.250000', '2024-06-01 17:00:01', 1, '2024-06-01 17:00:02', '{}')
""".trimIndent(),
)
}

liquibase.update("")

conn.createStatement().use { stmt ->
stmt.executeQuery("SELECT @@session.time_zone AS tz, @@session.lock_wait_timeout AS lwt").use { rs ->
rs.next()
rs.getString("tz") shouldBe "+05:00"
rs.getLong("lwt") shouldBe defaultLockWaitTimeout
}
val columns = stmt.executeQuery(
"""
SELECT COLUMN_NAME, DATA_TYPE, DATETIME_PRECISION, IS_NULLABLE, COLUMN_DEFAULT, EXTRA
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox'
AND COLUMN_NAME IN ('created_at', 'updated_at', 'last_attempt')
ORDER BY COLUMN_NAME
""".trimIndent(),
).use { rs ->
generateSequence {
if (rs.next()) {
listOf(
rs.getString("COLUMN_NAME"),
rs.getString("DATA_TYPE"),
rs.getInt("DATETIME_PRECISION").toString(),
rs.getString("IS_NULLABLE"),
rs.getString("COLUMN_DEFAULT") ?: "null",
rs.getString("EXTRA"),
).joinToString("|")
} else {
null
}
}.toList()
}
columns shouldBe listOf(
"created_at|datetime|6|NO|null|",
"last_attempt|datetime|6|YES|null|",
"updated_at|datetime|6|NO|null|",
)
}
}

val jdbc = JdbcConnectionProvider(
MysqlDataSource().apply {
setURL(container.jdbcUrl)
user = container.username
setPassword(container.password)
},
)
val entry = jdbc.withTransaction { MysqlOutboxStore(jdbc).claimPending(10) }.single()
entry.createdAt shouldBe Instant.parse("2024-06-01T12:00:00.25Z")
entry.updatedAt shouldBe Instant.parse("2024-06-01T12:00:01Z")
entry.lastAttempt shouldBe Instant.parse("2024-06-01T12:00:02Z")
entry.status shouldBe OutboxStatus.PENDING
}
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
package com.softwaremill.okapi.test.store

import com.softwaremill.okapi.mysql.MysqlOutboxStore
import com.softwaremill.okapi.test.support.MysqlTestSupport
import io.kotest.core.spec.style.FunSpec

class MysqlTimestampColumnsTest : FunSpec({
val db = MysqlTestSupport()

timestampColumnTests(
dbName = "mysql",
jdbcProvider = { db.jdbc },
storeFactory = { MysqlOutboxStore(db.jdbc) },
startDb = { db.start() },
stopDb = { db.stop() },
truncate = { db.truncate() },
utcText = { column -> "DATE_FORMAT($column, '%Y-%m-%d %H:%i:%s.%f')" },
setSessionZone = "SET time_zone = '+05:00'",
)
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
package com.softwaremill.okapi.test.store

import com.softwaremill.okapi.core.DeliveryInfo
import com.softwaremill.okapi.core.OutboxEntry
import com.softwaremill.okapi.core.OutboxMessage
import com.softwaremill.okapi.core.OutboxStatus
import com.softwaremill.okapi.core.OutboxStore
import com.softwaremill.okapi.test.support.JdbcConnectionProvider
import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.collections.shouldContainExactly
import io.kotest.matchers.shouldBe
import java.time.Instant
import java.util.TimeZone

private object TimestampTestDeliveryInfo : DeliveryInfo {
override val type: String = "timestamp-test"
override fun serialize(): String = """{"type":"timestamp-test"}"""
}

/**
* Timestamp columns must hold instants that survive any JVM, session or driver time zone, and dates
* past 2038 (issues #135, #138). [utcText] renders a column as UTC `yyyy-MM-dd HH:mm:ss.SSSSSS` in
* SQL, so the tests check what is stored, not only what the store reads back. [setSessionZone] is a
* statement that moves the session time zone away from UTC.
*/
fun FunSpec.timestampColumnTests(
dbName: String,
jdbcProvider: () -> JdbcConnectionProvider,
storeFactory: () -> OutboxStore,
startDb: () -> Unit,
stopDb: () -> Unit,
truncate: () -> Unit,
utcText: (column: String) -> String,
setSessionZone: String,
) {
beforeSpec { startDb() }
afterSpec { stopDb() }
beforeEach { truncate() }

fun storedUtc(id: String, column: String): String = jdbcProvider().withTransaction {
jdbcProvider().withConnection { conn ->
conn.prepareStatement("SELECT ${utcText(column)} AS v FROM okapi_outbox WHERE CAST(id AS CHAR(36)) = ?").use { stmt ->
stmt.setString(1, id)
stmt.executeQuery().use { rs ->
check(rs.next()) { "no row $id" }
rs.getString("v")
}
}
}
}

fun entry(createdAt: Instant): OutboxEntry =
OutboxEntry.createPending(OutboxMessage("timestamp.test", "{}"), TimestampTestDeliveryInfo, createdAt)

test("[$dbName] instants after 2038 are stored and read back") {
val store = storeFactory()
val createdAt = Instant.parse("2040-02-29T12:34:56.123456Z")
val attempted = entry(createdAt).retry(Instant.parse("2100-01-01T00:00:00.000001Z"), "boom")
jdbcProvider().withTransaction { store.persist(attempted) }

val claimed = jdbcProvider().withTransaction { store.claimPending(10) }
claimed.map { it.createdAt } shouldContainExactly listOf(createdAt)
claimed.single().lastAttempt shouldBe Instant.parse("2100-01-01T00:00:00.000001Z")
storedUtc(attempted.outboxId.raw.toString(), "created_at") shouldBe "2040-02-29 12:34:56.123456"
}

test("[$dbName] the purger cutoff works past 2038") {
val store = storeFactory()
val delivered = entry(Instant.parse("2045-01-01T00:00:00Z")).toDelivered(Instant.parse("2045-01-01T00:00:01Z"))
jdbcProvider().withTransaction { store.persist(delivered) }

jdbcProvider().withTransaction { store.removeDeliveredBefore(Instant.parse("2045-01-01T00:00:00Z"), 10) } shouldBe 0
jdbcProvider().withTransaction { store.removeDeliveredBefore(Instant.parse("2045-01-01T00:00:02Z"), 10) } shouldBe 1
}

test("[$dbName] values are stored as UTC whatever the JVM and session time zones") {
val store = storeFactory()
// Europe/Warsaw falls back at 01:00Z on this day: 00:30Z and 01:30Z are both 02:30 local, so a
// value stored as local wall-clock time reads back an hour early.
val createdAt = Instant.parse("2024-10-27T01:30:00.5Z")
val original = TimeZone.getDefault()
val pending = entry(createdAt)
try {
TimeZone.setDefault(TimeZone.getTimeZone("Europe/Warsaw"))
jdbcProvider().withTransaction {
jdbcProvider().withConnection { conn -> conn.createStatement().use { it.execute(setSessionZone) } }
store.persist(pending)
}
storedUtc(pending.outboxId.raw.toString(), "created_at") shouldBe "2024-10-27 01:30:00.500000"

TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles"))
val readBack = jdbcProvider().withTransaction {
jdbcProvider().withConnection { conn -> conn.createStatement().use { it.execute(setSessionZone) } }
store.findOldestCreatedAt(setOf(OutboxStatus.PENDING))
}
readBack shouldBe mapOf(OutboxStatus.PENDING to createdAt)
jdbcProvider().withTransaction { store.claimPending(10) }.single().createdAt shouldBe createdAt
} finally {
TimeZone.setDefault(original)
}
}
}
Loading