Repository navigation
fix(mysql)!: store timestamps as UTC DATETIME(6) so the outbox survives 2038 (#138) #144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,38 @@ | |
| This guide contains actions required when upgrading between incompatible okapi versions. | ||
| For the full release history, see [CHANGELOG.md](CHANGELOG.md). | ||
|
|
||
| ## From 1.x to 2.0.0 | ||
|
|
||
| ### MySQL: `DATETIME(6)` timestamp columns (#138) | ||
|
|
||
| Changeset `003__datetime_columns` converts `created_at`, `updated_at` and `last_attempt` from | ||
| `TIMESTAMP(6)` to `DATETIME(6)` holding UTC. With the bundled Liquibase changelog it runs | ||
| automatically at startup; otherwise apply | ||
| [`003__datetime_columns.sql`](okapi-mysql/src/main/resources/com/softwaremill/okapi/db/mysql/003__datetime_columns.sql) | ||
| yourself, **including its `SET time_zone` statements**, before starting 2.0.0 workers. | ||
|
|
||
| - **Stop every 1.x instance, publishers included, before 003 runs.** 1.x binds these columns as | ||
| `java.sql.Timestamp`, which Connector/J shifts into the JVM (or `connectionTimeZone`) zone. Against | ||
| `DATETIME` columns a 1.x instance on a non-UTC JVM writes local wall-clock time instead of UTC and | ||
| misreads rows written by 2.0. A rolling deploy that mixes the two is only safe if every 1.x JVM | ||
| runs in UTC. | ||
| - **Rows written by 1.x may read back shifted if the JVM and MySQL zones differed.** Without | ||
| `connectionTimeZone`, 1.x sent `Timestamp`s as JVM-local wall-clock time and MySQL interpreted them | ||
| in the session `time_zone`; when the two zones differed, the stored instant was off by the | ||
| difference, and 1.x reading it back undid the shift. 003 converts what is actually stored, so 2.0 | ||
| reads such rows off by that difference: old `PENDING` rows sort out of order, the lag gauge is skewed | ||
| and old `DELIVERED` rows are purged early or late. If your JVMs and MySQL session ran in different | ||
| zones, drain the outbox (let every `PENDING` row be delivered) before upgrading. | ||
| - **Locking.** The `ALTER TABLE` copies `okapi_outbox` (`ALGORITHM=COPY, LOCK=SHARED`). Before it | ||
| starts it waits for a metadata lock behind every open transaction on the table, such as a 1.x worker | ||
| delivering a claimed batch, and while it waits every new statement on the table, reads included, | ||
| queues behind it. The changeset bounds that wait with `lock_wait_timeout = 30` seconds; on timeout | ||
| the migration fails and runs again on the next start. During the copy, writes, including | ||
| `publish()`, wait. Stop the workers first, and purge delivered rows or pick a quiet period on a | ||
| large table. | ||
| - Queries or dashboards that read these columns directly now get UTC wall-clock values, not values | ||
| converted to the session time zone. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should-fix (docs), related to the review body: besides direct readers, existing rows may themselves read back shifted after the upgrade. That happens when 1.x ran with JVM zone ≠ MySQL session zone and |
||
|
|
||
| ## From 0.2.x to 1.0.0 | ||
|
|
||
| Releases up to 0.2.x used the domain table `outbox` and the shared Liquibase tracking | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| package com.softwaremill.okapi.test.store | ||
|
|
||
| import com.mysql.cj.jdbc.MysqlDataSource | ||
| import com.softwaremill.okapi.core.OutboxStatus | ||
| import com.softwaremill.okapi.mysql.MysqlOutboxStore | ||
| import com.softwaremill.okapi.test.support.JdbcConnectionProvider | ||
| import io.kotest.core.spec.style.FunSpec | ||
| import io.kotest.matchers.shouldBe | ||
| import liquibase.Liquibase | ||
| import liquibase.database.DatabaseFactory | ||
| import liquibase.database.jvm.JdbcConnection | ||
| import liquibase.resource.ClassLoaderResourceAccessor | ||
| import org.testcontainers.containers.MySQLContainer | ||
| import java.sql.Connection | ||
| import java.sql.DriverManager | ||
| import java.time.Instant | ||
| import java.util.Properties | ||
|
|
||
| private const val CHANGELOG = "com/softwaremill/okapi/db/mysql/changelog.xml" | ||
|
|
||
| /** | ||
| * Upgrade path for issue #138: rows written while the columns were TIMESTAMP must keep their | ||
| * instants when changeset 003 converts them to DATETIME, even though the migrating session runs in | ||
| * another time zone (MODIFY converts through the session zone). | ||
| */ | ||
| class MysqlDatetimeMigrationTest : FunSpec({ | ||
| val container = MySQLContainer<Nothing>("mysql:8.0") | ||
|
|
||
| beforeSpec { container.start() } | ||
| afterSpec { container.stop() } | ||
|
|
||
| /** A connection whose session time zone is +05:00, as a non-UTC server or pool would give. */ | ||
| fun offsetSessionConnection(): Connection = DriverManager.getConnection( | ||
| container.jdbcUrl, | ||
| Properties().apply { | ||
| setProperty("user", container.username) | ||
| setProperty("password", container.password) | ||
| setProperty("sessionVariables", "time_zone='+05:00'") | ||
| }, | ||
| ) | ||
|
|
||
| test("changeset 003 keeps existing instants and restores the session time zone") { | ||
| offsetSessionConnection().use { conn -> | ||
| val defaultLockWaitTimeout = conn.createStatement().use { stmt -> | ||
| stmt.executeQuery("SELECT @@session.lock_wait_timeout").use { rs -> | ||
| rs.next() | ||
| rs.getLong(1) | ||
| } | ||
| } | ||
| val liquibase = | ||
| Liquibase( | ||
| CHANGELOG, | ||
| ClassLoaderResourceAccessor(), | ||
| DatabaseFactory.getInstance().findCorrectDatabaseImplementation(JdbcConnection(conn)), | ||
| ) | ||
| liquibase.update(2, "") | ||
|
|
||
| // A TIMESTAMP literal is read in the session zone: 17:00 at +05:00 is 12:00Z. | ||
| conn.createStatement().use { | ||
| it.execute( | ||
| """ | ||
| INSERT INTO okapi_outbox (id, message_type, payload, delivery_type, status, created_at, updated_at, retries, last_attempt, delivery_metadata) | ||
| VALUES ('00000000-0000-0000-0000-000000000001', 'm', '{}', 'http', 'PENDING', | ||
| '2024-06-01 17:00:00.250000', '2024-06-01 17:00:01', 1, '2024-06-01 17:00:02', '{}') | ||
| """.trimIndent(), | ||
| ) | ||
| } | ||
|
|
||
| liquibase.update("") | ||
|
|
||
| conn.createStatement().use { stmt -> | ||
| stmt.executeQuery("SELECT @@session.time_zone AS tz, @@session.lock_wait_timeout AS lwt").use { rs -> | ||
| rs.next() | ||
| rs.getString("tz") shouldBe "+05:00" | ||
| rs.getLong("lwt") shouldBe defaultLockWaitTimeout | ||
| } | ||
| val columns = stmt.executeQuery( | ||
| """ | ||
| SELECT COLUMN_NAME, DATA_TYPE, DATETIME_PRECISION, IS_NULLABLE, COLUMN_DEFAULT, EXTRA | ||
| FROM information_schema.COLUMNS | ||
| WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'okapi_outbox' | ||
| AND COLUMN_NAME IN ('created_at', 'updated_at', 'last_attempt') | ||
| ORDER BY COLUMN_NAME | ||
| """.trimIndent(), | ||
| ).use { rs -> | ||
| generateSequence { | ||
| if (rs.next()) { | ||
| listOf( | ||
| rs.getString("COLUMN_NAME"), | ||
| rs.getString("DATA_TYPE"), | ||
| rs.getInt("DATETIME_PRECISION").toString(), | ||
| rs.getString("IS_NULLABLE"), | ||
| rs.getString("COLUMN_DEFAULT") ?: "null", | ||
| rs.getString("EXTRA"), | ||
| ).joinToString("|") | ||
| } else { | ||
| null | ||
| } | ||
| }.toList() | ||
| } | ||
| columns shouldBe listOf( | ||
| "created_at|datetime|6|NO|null|", | ||
| "last_attempt|datetime|6|YES|null|", | ||
| "updated_at|datetime|6|NO|null|", | ||
| ) | ||
| } | ||
| } | ||
|
|
||
| val jdbc = JdbcConnectionProvider( | ||
| MysqlDataSource().apply { | ||
| setURL(container.jdbcUrl) | ||
| user = container.username | ||
| setPassword(container.password) | ||
| }, | ||
| ) | ||
| val entry = jdbc.withTransaction { MysqlOutboxStore(jdbc).claimPending(10) }.single() | ||
| entry.createdAt shouldBe Instant.parse("2024-06-01T12:00:00.25Z") | ||
| entry.updatedAt shouldBe Instant.parse("2024-06-01T12:00:01Z") | ||
| entry.lastAttempt shouldBe Instant.parse("2024-06-01T12:00:02Z") | ||
| entry.status shouldBe OutboxStatus.PENDING | ||
| } | ||
| }) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| package com.softwaremill.okapi.test.store | ||
|
|
||
| import com.softwaremill.okapi.mysql.MysqlOutboxStore | ||
| import com.softwaremill.okapi.test.support.MysqlTestSupport | ||
| import io.kotest.core.spec.style.FunSpec | ||
|
|
||
| class MysqlTimestampColumnsTest : FunSpec({ | ||
| val db = MysqlTestSupport() | ||
|
|
||
| timestampColumnTests( | ||
| dbName = "mysql", | ||
| jdbcProvider = { db.jdbc }, | ||
| storeFactory = { MysqlOutboxStore(db.jdbc) }, | ||
| startDb = { db.start() }, | ||
| stopDb = { db.stop() }, | ||
| truncate = { db.truncate() }, | ||
| utcText = { column -> "DATE_FORMAT($column, '%Y-%m-%d %H:%i:%s.%f')" }, | ||
| setSessionZone = "SET time_zone = '+05:00'", | ||
| ) | ||
| }) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| package com.softwaremill.okapi.test.store | ||
|
|
||
| import com.softwaremill.okapi.core.DeliveryInfo | ||
| import com.softwaremill.okapi.core.OutboxEntry | ||
| import com.softwaremill.okapi.core.OutboxMessage | ||
| import com.softwaremill.okapi.core.OutboxStatus | ||
| import com.softwaremill.okapi.core.OutboxStore | ||
| import com.softwaremill.okapi.test.support.JdbcConnectionProvider | ||
| import io.kotest.core.spec.style.FunSpec | ||
| import io.kotest.matchers.collections.shouldContainExactly | ||
| import io.kotest.matchers.shouldBe | ||
| import java.time.Instant | ||
| import java.util.TimeZone | ||
|
|
||
| private object TimestampTestDeliveryInfo : DeliveryInfo { | ||
| override val type: String = "timestamp-test" | ||
| override fun serialize(): String = """{"type":"timestamp-test"}""" | ||
| } | ||
|
|
||
| /** | ||
| * Timestamp columns must hold instants that survive any JVM, session or driver time zone, and dates | ||
| * past 2038 (issues #135, #138). [utcText] renders a column as UTC `yyyy-MM-dd HH:mm:ss.SSSSSS` in | ||
| * SQL, so the tests check what is stored, not only what the store reads back. [setSessionZone] is a | ||
| * statement that moves the session time zone away from UTC. | ||
| */ | ||
| fun FunSpec.timestampColumnTests( | ||
| dbName: String, | ||
| jdbcProvider: () -> JdbcConnectionProvider, | ||
| storeFactory: () -> OutboxStore, | ||
| startDb: () -> Unit, | ||
| stopDb: () -> Unit, | ||
| truncate: () -> Unit, | ||
| utcText: (column: String) -> String, | ||
| setSessionZone: String, | ||
| ) { | ||
| beforeSpec { startDb() } | ||
| afterSpec { stopDb() } | ||
| beforeEach { truncate() } | ||
|
|
||
| fun storedUtc(id: String, column: String): String = jdbcProvider().withTransaction { | ||
| jdbcProvider().withConnection { conn -> | ||
| conn.prepareStatement("SELECT ${utcText(column)} AS v FROM okapi_outbox WHERE CAST(id AS CHAR(36)) = ?").use { stmt -> | ||
| stmt.setString(1, id) | ||
| stmt.executeQuery().use { rs -> | ||
| check(rs.next()) { "no row $id" } | ||
| rs.getString("v") | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| fun entry(createdAt: Instant): OutboxEntry = | ||
| OutboxEntry.createPending(OutboxMessage("timestamp.test", "{}"), TimestampTestDeliveryInfo, createdAt) | ||
|
|
||
| test("[$dbName] instants after 2038 are stored and read back") { | ||
| val store = storeFactory() | ||
| val createdAt = Instant.parse("2040-02-29T12:34:56.123456Z") | ||
| val attempted = entry(createdAt).retry(Instant.parse("2100-01-01T00:00:00.000001Z"), "boom") | ||
| jdbcProvider().withTransaction { store.persist(attempted) } | ||
|
|
||
| val claimed = jdbcProvider().withTransaction { store.claimPending(10) } | ||
| claimed.map { it.createdAt } shouldContainExactly listOf(createdAt) | ||
| claimed.single().lastAttempt shouldBe Instant.parse("2100-01-01T00:00:00.000001Z") | ||
| storedUtc(attempted.outboxId.raw.toString(), "created_at") shouldBe "2040-02-29 12:34:56.123456" | ||
| } | ||
|
|
||
| test("[$dbName] the purger cutoff works past 2038") { | ||
| val store = storeFactory() | ||
| val delivered = entry(Instant.parse("2045-01-01T00:00:00Z")).toDelivered(Instant.parse("2045-01-01T00:00:01Z")) | ||
| jdbcProvider().withTransaction { store.persist(delivered) } | ||
|
|
||
| jdbcProvider().withTransaction { store.removeDeliveredBefore(Instant.parse("2045-01-01T00:00:00Z"), 10) } shouldBe 0 | ||
| jdbcProvider().withTransaction { store.removeDeliveredBefore(Instant.parse("2045-01-01T00:00:02Z"), 10) } shouldBe 1 | ||
| } | ||
|
|
||
| test("[$dbName] values are stored as UTC whatever the JVM and session time zones") { | ||
| val store = storeFactory() | ||
| // Europe/Warsaw falls back at 01:00Z on this day: 00:30Z and 01:30Z are both 02:30 local, so a | ||
| // value stored as local wall-clock time reads back an hour early. | ||
| val createdAt = Instant.parse("2024-10-27T01:30:00.5Z") | ||
| val original = TimeZone.getDefault() | ||
| val pending = entry(createdAt) | ||
| try { | ||
| TimeZone.setDefault(TimeZone.getTimeZone("Europe/Warsaw")) | ||
| jdbcProvider().withTransaction { | ||
| jdbcProvider().withConnection { conn -> conn.createStatement().use { it.execute(setSessionZone) } } | ||
| store.persist(pending) | ||
| } | ||
| storedUtc(pending.outboxId.raw.toString(), "created_at") shouldBe "2024-10-27 01:30:00.500000" | ||
|
|
||
| TimeZone.setDefault(TimeZone.getTimeZone("America/Los_Angeles")) | ||
| val readBack = jdbcProvider().withTransaction { | ||
| jdbcProvider().withConnection { conn -> conn.createStatement().use { it.execute(setSessionZone) } } | ||
| store.findOldestCreatedAt(setOf(OutboxStatus.PENDING)) | ||
| } | ||
| readBack shouldBe mapOf(OutboxStatus.PENDING to createdAt) | ||
| jdbcProvider().withTransaction { store.claimPending(10) }.single().createdAt shouldBe createdAt | ||
| } finally { | ||
| TimeZone.setDefault(original) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should-fix (docs): say that 1.x instances must be stopped once 003 runs, not only that 2.0 must not start before it.
The Spring Boot autoconfig runs 003 when the first 2.0 instance starts. In a rolling deploy, 1.x instances are still publishing and claiming at that point. After the migration, 1.x still binds
java.sql.Timestamp. Connector/J renders aTimestampinconnectionTimeZone(by default the JVM zone) and theDATETIMEcolumn stores that text unconverted. So with a non-UTC JVM, every row 1.x writes or updates in that window gets a local wall-clock value that 2.0 then reads as UTC. 1.x also misreads the rows 2.0 wrote. The wrong values stay in the table for good.Suggest stating it here, in the README and in the CHANGELOG: either stop every 1.x instance before 003 runs (no rolling deploy across this version), or note that mixed operation is only safe when the 1.x JVMs run in UTC.