Skip to content

feat: model EXCHANGE TABLES so structural key changes can cut over without data loss #202

Description

@alvarogar4

Problem

Changing a table's primaryKey, orderBy, engine, partitionBy, or uniqueKey is a structural change. requiresTableRecreate() (packages/core/src/planner.ts:145) routes those through diffTables() (packages/core/src/planner.ts:299), which emits exactly two operations:

DROP TABLE IF EXISTS db.t;   -- risk: danger
CREATE TABLE db.t (...);     -- risk: safe

There is no data movement between them. On a populated table this is total data loss. safety.allowDestructive: false blocks it by default — which is the correct guard — but the result is that chkit has no supported path for changing a sort key on a table that has data.

Changing the sort key is one of the most common consequential ClickHouse migrations there is (it's how you fix a table whose query patterns have moved). Today chkit's only answer is "drop and recreate", so users have to step outside the tool for precisely the migration where they'd most want it to help.

The pattern users actually need

The safe procedure is shadow table → backfill → atomic swap:

  1. Define t_v2 with the new key alongside the existing t; chkit generate + chkit migrate --apply (create-only, risk=safe).
  2. Backfill t_v2 from tchkit plugin backfill submit (managed) or run (local).
  3. Swap t and t_v2 atomically. chkit does not model this, so it has to happen outside chkit — currently by running EXCHANGE TABLES db.t AND db.t_v2; directly against the cluster.
  4. Reconcile the schema files and snapshot to the post-swap reality.

Steps 1 and 2 are well supported. Step 3 is the gap, and step 4 is awkward as a consequence: after an out-of-band swap the snapshot describes tables whose names no longer match their contents, so drift/check need manual reconciliation.

Proposal

Model EXCHANGE TABLES as a first-class chkit operation. Rough shape:

  • An exchange_tables migration operation type, so the swap lands in a real migration file with a checksum and journal entry instead of being an out-of-band action.
  • A way to author it — e.g. chkit exchange <db.a> <db.b> generating that migration, rather than requiring a hand-written generate --empty stub.
  • Snapshot handling so the post-swap state reconciles cleanly instead of reading as drift.
  • A sensible risk level: EXCHANGE TABLES is atomic and symmetric (re-running it swaps back), so it is meaningfully less dangerous than drop_table and probably shouldn't sit behind --allow-destructive.

Open questions

  • Primitive or plan? Should this stay a manual primitive the user sequences themselves, or should chkit detect a structural key change on a populated table and generate the whole shadow + backfill + swap plan? The latter is far more valuable and far more machinery; the primitive is a useful step toward it either way.
  • Engine support. EXCHANGE TABLES requires the Atomic database engine. Needs a capability check and a clear error rather than a raw ClickHouse failure on Ordinary.
  • Multi-pair form. ClickHouse supports EXCHANGE TABLES a AND b, c AND d. Worth modelling, or is the single pair enough?
  • Interaction with renamedFrom. chkit already has rename metadata for tables; is exchange a distinct concept or should it reuse any of that plumbing?

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions