Repository navigation
DO NOT MERGE: CI for the 1.0 line (dy/ch-migrate-1.0) - #14
Merged
Merged
Conversation
Repeat the accepted session experiment through actual Alembic revisions
in clean environments. Preserve the observed legacy URL limitation and
Python 3.9 incompatibility without switching production dependencies.
Solves: DRY-1391
Tests: Four real-server dialect modes; clean 3.10 import and 3.9 refusal;
411 unit tests on Python 3.9-3.14
(cherry picked from commit 63c58127e0bfba8cdd181bad8b772c5cfeffa14f)
Use one checked session per run and fail if it expires. Move migration
runtime into the package behind a versioned shim, preserve legacy tables
and migrations through an explicit upgrade, and reject stale environments
before connecting. Require Python 3.10 and remove the legacy dialect.
Solves: DRY-1392
Tests: 409 unit tests on Python 3.10-3.14; 17 integration tests on 3.10/3.14;
expired session, released 0.4.1 upgrade, Python 3.9 install refusal
(cherry picked from commit b38bbda891767611896b8caef9fe1f2eda17b297)
Refuse unsafe pending SQL before Alembic applies anything, with visible
reasoned waivers and a comment-preserving script-head baseline. Keep
non-gate findings as warnings in up. Fixtures now declare intentional
inserts and exchanges explicitly; replace the obsolete hard MV gate
test with real nonblocking-warning coverage.
Solves: DRY-1394
Tests: 111-statement corpus; 553 unit tests on Python 3.10-3.14;
24 integration tests on 3.10/3.14 including refusal and recovery
(cherry picked from commit 6d23b3872253911ad4bebe70818f5ebae930ba80)
Share the static environment marker check between CLI refusal and lint.
Flag bare upgrade SET statements on old environments without confusing
SETTINGS clauses or string literals, and retain the fixed gate severity.
Solves: DRY-1395
Tests: 559 unit tests on Python 3.10-3.14; 25 integration tests;
old-environment refusal followed by upgrade and observed SET 3
(cherry picked from commit 488210fa8c9246ebd050e2ec083bc7fc0c30b911)
Own a two-node cluster and embedded Keeper on a private test network.
Prove replication, per-host distributed DDL completion, a stopped-node
timeout and restart catch-up. Preserve cleanup after failure and handled
interrupts, and limit cluster CI to Python 3.12.
Solves: DRY-1397
Tests: 559 unit tests on Python 3.10-3.14; 25 single-node tests on 3.10/3.14;
2 cluster tests on 3.12; stopped-node interrupt cleanup
(cherry picked from commit 86eee16c11423a362fbf462e1dff13d0ae287c57)
Choose version-table engines from the deployment while retaining the
official INSERT then waited DELETE path and preserving existing tables.
Read replica state after synchronization, surface failed version mutations
across replicas, and document manual conversion instead of performing it.
Solves: DRY-1412
Tests: 565 unit tests on Python 3.10-3.14; 33 integration tests on 3.12;
27 single-node tests on 3.10/3.14; interrupted-delete replay proof
(cherry picked from commit 23dd0aa84e47268af1ac421b5eb3b017a42a17fd)
Preserve accepted-but-unacknowledged mutation evidence and the retention counterexample. Keep the unconditional recovery contract blocked for a decision rather than implementing silent replay. Solves: DRY-1398 evidence and recovery decision record Tests: 16 local probes; 3 crash repeats; 565 tests on Python 3.10-3.14 (cherry picked from commit 0ab51016552e68c2e6e2ee90a3eb2983433a27d0)
Give automation stable status, history, lint and drift documents. Keep diagnostics off stdout and distinguish drift from read failures. Resolve applied ancestors and preserve explicit waiver reasons. Solves: DRY-1244 Tests: 18 JSON contracts; 35 integration; 583 tests on Python 3.10-3.14 (cherry picked from commit 57b3e81ef85f3891c1729717f5cbcef50039aec4)
Distinguish metadata, background mutations and in-place changes requiring rebuilds. Compare live column types and sorting keys rather than labeling every ALTER alike. Keep unknown operations explicit. Solves: DRY-1402 Tests: 58 unit cases; 40 server proofs; 641 tests on Python 3.10-3.14 (cherry picked from commit 3540ccb8c0329e3bb785e7772b003dc44acfef77)
Compare a committed environment snapshot without regenerating the expected schema. Preserve JSON evidence before failing the job and isolate temporary database credentials from artifacts and logs. Solves: DRY-1404 Tests: Workflow shell steps and actionlint; 641 tests on Python 3.10-3.14 (cherry picked from commit 097060fff06e3ef7fc8acd255639e51cfa282afb)
Replay archived experiments without their fixed container or broad process-kill wrapper. Attribute duplicates with server query and flush times, and drain pending async writes before measuring missing rows. Solves: DRY-1406 Tests: A/D/E probes; two 6M-row runs; Python 3.10-3.14; 78 integration (cherry picked from commit 73e4b2cb952269af1803eb25163917bce4b01d30)
Record Dan’s approved recovery exception without treating missing ownership evidence as completion or permission to replay a write. Keep normal reattachment and the local-only execution boundaries. Solves: DRY-1398 Tests: 16 local probes; 3 crash repeats; 641 tests on Python 3.10-3.14 (cherry picked from commit 22b4c1a57c1a9c41e02e8c0b65eb6e0fa263ef27)
Persist intent before submission and resume only proven server work. Poll every required replica in the checked session, preserve completed writes, and fail closed when ownership evidence cannot prove an outcome. Keep the CLI as the owning process and journal version bookkeeping. Solves: DRY-1399 Tests: 655 tests on Python 3.10-3.14; 127 integration; 3 crash repeats (cherry picked from commit 1d52b89677e0d72517a78e255896f7732012e5fb)
Own queue entries with durable markers and require successful results from every target host. Resume the existing entry after interruption, then wait for any mutations it created before completing the revision. Apply the same deadline to cold version-table creation on a cluster. Solves: DRY-1400 Tests: DDL host/crash/TTL cases; 655 tests on Python 3.10-3.14; 137 integration (cherry picked from commit 34d95ebe602ee853024a7a7870247329e38cff68)
Expose rewrite precision and rebuild risks before applying migrations. Use shared preflight and refuse missing evidence rather than assuming writers or replicas are safe. Compact-part column counters are unknown, so report a whole-part ceiling instead of an exact zero-byte rewrite. Solves: DRY-1403 Tests: 661 unit tests on Python 3.10-3.14; 151 owned-container integration tests; installed CLI human/JSON smoke with unchanged source data. (cherry picked from commit bce1e4ffebb971567a6ff76c361cb67119f10928)
Preserve acknowledged writes across key and engine changes without allowing expired owners to regain control. Keep progress on the server and require operator reconciliation before releasing interrupted locks. Solves: DRY-1407 Tests: 687 unit tests each Python 3.10-3.14; 169 owned-container integration tests; installed CLI checksum/rollback smoke; acceptance account valid. (cherry picked from commit 34b232443f74ce8d90cfe56a2aa6b7abef1c6cdb)
Prevent a schema swap from breaking insert-trigger paths or leaving dictionaries on an old cache. Reject invalid projections while only empty owned helpers exist, so repaired SQL can restart safely. Solves: DRY-1408 Tests: 694 unit tests each Python 3.10-3.14; 180 owned-container integration tests; live writer/window and dictionary-reload CLI scenarios. (cherry picked from commit 271c1a4c1161ef7de705352008423b33d6c2cee3)
Let SQL authors edit live replacement DDL without writing Python. Keep legacy exchange available with its data-loss warning and expose rebuild suggestions in both plan formats. Solves: DRY-1409 Tests: 702 unit tests on Python 3.10-3.14; 181 owned integrations (cherry picked from commit 9aa0d9aa31cd57c74337e93ed472d665106daf36)
Exercise continuous writers, checkpoint recovery, both replicas, and both async acknowledgement modes with real snapshot and swap barriers. Resolve queued writers from active settings before query-log flushes without weakening unknown-writer or cross-host safety checks. Solves: DRY-1410 Tests: Five consecutive runs of all five variants; 707 unit tests on Python 3.10-3.14; 186 owned integrations; final recovery scenario (cherry picked from commit f9c3074eca879d986a800ba3140943f5c7184331)
Keep multi-line waiting and unknown-outcome instructions when SQL execution wraps a server error, while retaining concise database errors. Solves: Operator recovery detail lost in compact upgrade output Tests: Docker mutation failure and unknown-outcome integration cases
Use the old import solely in the 0.4.1 fixture, keep status a nonblocking reporter, and assert compact waiver output and the SQL statement line rather than the preceding waiver comment. Solves: Integration assertions diverged across rename and output cutover Tests: Six targeted Docker integration cases
Keep the deprecated alias only for historical examples and explicit 0.4.1 compatibility tests. Solves: Remaining executable old-package imports after rename Tests: Six targeted Docker integration cases
Apply repository isort/black settings to changed files and remove unused imports and local variables without altering the dialect and rebuild registration imports. Solves: Changed-file formatter and Ruff findings on port branch Tests: 715 unit tests; changed-file isort, black, Ruff checks
Replicated and Shared MergeTree drop an INSERT block identical to an earlier one. After `down` re-inserted a previous head, or `up` re-inserted a revision applied before a downgrade, the row was silently deduplicated while the DELETE of the old head still ran, leaving alembic_version empty so the next `up` replays everything. Each version-row INSERT now carries its own deduplication token, and downgrade version writes take the same path. Cloud ignored insert_deduplicate = 0 in probes; only a unique token kept the row. Solves: DRY-1447 Tests: round-trip down/up on Docker replicas (failed before) and single node; irreversible + version-table suite 5x on Cloud; 715 unit
Cloud refuses things the self-hosted suite relied on: SYSTEM STOP MERGES, *.* grants without CURRENT GRANTS, weak passwords, a default dictionary source user and changing log_query_settings. Query logs and Memory rows are per node, engines are Shared*, and metadata ALTERs leave done, block-less system.mutations rows. The harness now knows when it runs on Cloud. Tests use Cloud-valid techniques there, or skip with the Cloud restriction as the reason, and say so where an expectation is engine-specific. Adds the `cloud` marker with checks that only run on Cloud: a SET reaches 50 probes over 98 s from one replica, version-table round trips on SharedMergeTree, and the version row is written after a slow mutation finishes. Adds scripts/cloud_sweep.py, a README section on running against Cloud, and the Q9 Cloud results with the metadata-row decision in the waiting note. Solves: DRY-1448, DRY-1446, DRY-1413 Tests: Cloud integration and not cluster, Docker integration incl. cluster, 715 unit
Cloud refuses explicit table UUIDs, SYSTEM STOP MERGES and SYSTEM FLUSH ASYNC INSERT QUEUE for every user. The guarded rebuild needs all three for crash-safe ownership and the swap, so on Cloud it failed at lock CREATE with a misleading "inspect replicas" error. Preflight now refuses Shared* sources with `cloud_unsupported` before any lock, helper or swap exists, and plan reports the same finding. The Cloud spike re-run shows EXCHANGE, ATTACH/MOVE PARTITION, view name-following and in-flight inserts behave as on self-hosted. Its redacted output is in spikes/2026-10-02-rebuild/cloud/, and section C of the rebuild spike records it. Draft note 2026-10-13-rebuild-on-cloud.md sets out Cloud-capable replacements for Dan's 1.0 scope decision. Solves: DRY-1445 (interim refusal; Cloud support pending decision) Tests: scaffold rebuild on Cloud (refusal, table/rows/catalog unchanged) and Docker (rebuild); preflight unit and live tests
Cloud cannot hold a mutation with SYSTEM STOP MERGES, so the
self-hosted SIGKILL tests skip there. A deliberately slow UPDATE
(sleepEachRow) gives a real window instead. `up` is killed while the
mutation is running, and the rerun must resume from its journal ("not
reissuing SQL"), produce exactly one increment and one owned mutation,
and then advance the version.
Solves: DRY-1401 (W03/W23 on Cloud)
Tests: integration and cloud on the DRY Cloud service, 4 passed
Unreleased listed the new features but not the changes that break or surprise 0.5 users. It now names the Python 3.10 floor, the move to the official dialect and its upgrade-env step, the gate on upgraded projects, blocking waits with --timeout, and the version-table replay fix, as the 0.6 and 0.7 acceptance tickets require. Solves: DRY-1396, DRY-1401 changelog callouts Tests: 716 unit
The Cloud SIGKILL test now proves journal resume on Cloud, so the "not exercised" note on crash persistence was stale. Only indefinite holds still need SYSTEM STOP MERGES and stay self-hosted evidence. Solves: DRY-1401 Q9 record Tests: integration and cloud, SIGKILL test 4 of 4
Dan chose full Cloud support for 1.0, rebuild included. Cloud probes show workable replacements for all three blocked fences: - ownership: owner-token comments with read-back UUIDs; - snapshot: plain, TTL-free snapshot/stage helpers whose merges keep every row, for all four collapsing engine families; - swap: no forced flush, since EXCHANGE drains the old async queue in about 0.2 s. A self-hosted probe shows SYSTEM STOP MERGES lapses on server restart. Writers on other compute services remain a gap that needs a declared opt-in. The note is a proposal awaiting approval; nothing changes in code. Solves: DRY-1445 design for approval Tests: Cloud protocol probes (2 rounds), Docker restart probe
SYSTEM STOP MERGES does not survive a server restart. Afterwards a Replacing, Collapsing, Summing or Aggregating snapshot helper could merge rows away, so the stage never reached the snapshot count and the rebuild could not finish. Snapshot and stage helpers are now plain MergeTree without table or column TTL, in the source's replication family, so their merges keep every row and STOP MERGES is gone. Partition transfer between MergeTree families works on 26.3 (single node and Replicated) and on Cloud, so preflight compares only replication families. ClickHouse Cloud cannot grant STOP MERGES at all, so this also removes one Cloud blocker (DRY-1445; protocol approved by Dan 2026-10-13). Solves: DRY-1445 part 1 Tests: helper-merge test fails before (stage count timeout) and passes after; Docker rebuild suites 41 passed; 717 unit
ClickHouse Cloud routes each TCP connection, not each session id, to a replica, so a session exists only on the replica its connection reached. The migration engine shared clickhouse-connect's default pool with other in-process clients; rebuild heartbeats opened a second connection under the session and Cloud answered SESSION_NOT_FOUND. The session now gets a single-connection pool of its own. A dropped connection still fails closed through session_check. Solves: DRY-1445 SESSION_NOT_FOUND during Cloud rebuilds Tests: 726 unit tests; Cloud scaffold and rebuild command suites
Preflight read the last hour of every INSERT on the service, capped at 10,001 rows, and only then filtered to the table being rebuilt. On a busy service, inserts into other tables filled the cap and preflight refused with "Writer log exceeds bounded 60-minute inspection window". The log query now filters to direct INSERTs into the source in SQL and groups by host, user and the two async settings. Queued inserts the log cannot vouch for are resolved from the queue first and excluded from the logged groups, so their live per-query settings still decide. Solves: DRY-1445 preflight refusal on a busy Cloud service Tests: 717 unit tests at this commit; Cloud preflight live test
Cloud refuses the three fences the rebuild relied on: chosen table
UUIDs, SYSTEM STOP MERGES and SYSTEM FLUSH ASYNC INSERT QUEUE. This
implements the protocol Dan approved on 2026-10-13
(docs/design/2026-10-13-rebuild-on-cloud.md):
- On a Shared database the lock and helpers carry an owner-token
COMMENT (before AS for the view). An object is ours only when every
replica shows our token and one UUID, which is then journaled and
checked as before. A changed token stops for the operator.
- No forced queue flush on Cloud. EXCHANGE drains the old queue; the
async rejection report becomes a lower bound that names the replicas
it read ("complete": false).
- Cloud migrations must declare writers_on_this_service_only=True;
preflight refuses with cloud_writers_undeclared until they do, and
the rebuild waits grace_seconds (default 10) after the copying view
reaches every replica, keeping the session alive meanwhile.
- Inspection, KILL QUERY, SYNC REPLICA and FLUSH LOGS go through
cluster 'default' on Cloud.
The rebuild integration suites now run against both Docker and Cloud.
On Cloud, dictionaries load at CREATE, so the dependents test reloads
one after seeding, and a single-shard route test accepts Cloud's
one-shard, two-replica cluster. A new Docker cluster test restarts the
other replica mid-copy.
Solves: DRY-1445 rebuild_table on ClickHouse Cloud
Tests: 726 unit; Docker rebuild suites 35 passed plus replica restart;
Cloud scaffold, rebuild commands, dependents (11), continuous (4)
The async0 continuous-writer variant failed about one run in five, also at 1d512b7: a duplicated row's flush was logged 6 ms after the duplicate window closed although the snapshot already held its rows. An async insert is in flight from queueing to the end of its flush, so it now counts as inside the window when that interval overlaps it, the same rule synchronous inserts use. The single-node DDL-queue test counted every read of the queue in the server's log. On Cloud, earlier tests' ON CLUSTER commands read it too; the count is now limited to this run, on every replica. Solves: async0 attribution flake; Cloud DDL-queue false failure Tests: async0 variant 25/25 (was 15/20); DDL-queue test on Docker, Cloud
An insert that resolves the table just before EXCHANGE and builds its view chain just after it writes only to the old storage. The copying view never sees it, so an acknowledged row is silently missing from the rebuilt table. A synthetic probe (eight writers, one EXCHANGE per trial) hit async inserts with wait_for_async_insert=1 in 3 of 160 trials on 26.3 and 1 of 40 on Cloud; synchronous inserts were never hit. The same race made the async0 report undercount once in CI: the stranded flush logs Ok, not FlushError. Dan chose detect-and-stop (2026-10-13). After the async report, the engine reads part_log for NewPart in the old UUID since the swap from queries with no NewPart in the new UUID; the probe matched this rule exactly in every trial. Stranded rows stop the run with their count and query ids and keep the lock. The operator copies them and releases the lock; the next run records the stop as acknowledged and finishes. Preflight refuses hosts without system.part_log, and bootstrap grants SELECT on it. The continuous-writer test performs that operator recovery if the race strikes, so CI checks the recovery path instead of failing. Solves: DRY-1445 acknowledged rows lost at the swap Tests: 727 unit; Docker rebuild suites 42 incl. cluster; stranded stop-and-resume on Docker and Cloud
The Cloud rebuild note gains its implementation commits, the stranded-row amendment with the probe's trial rates, and the acceptance table at bb3772e. Rebuild-spike section C records spike E re-run without the queue flush and copies the redacted outputs. The 1.0 definition states the detect-and-stop exception, and the EXCHANGE race probe is kept under spikes/2026-10-13-exchange-race. Solves: DRY-1445 evidence record Tests: none (documentation)
0.9.0 ships everything planned for 1.0 under a 0.x number, so the stability promises (frozen JSON output, the rebuild_table guarantee) can meet real workloads before 1.0 (Dan, 2026-10-13). 0.6-0.8 were never published, so the version skips them. A plain pip upgrade reaches 0.9 from 0.5.x, so the CHANGELOG section opens with the upgrade steps and the README says to pin 0.5.1 until each project has run upgrade-env. References to "0.6" as the next version now say 0.9, and the partition-key refusal no longer names a version. --version now passes the version explicitly. Click could not map the ch_migrate package to the ch-migrate-cli distribution in an editable install and raised RuntimeError. Solves: 0.9.0 release preparation (no tag) Tests: 728 unit; --version regression test fails before, passes after; clean wheel quick start on Python 3.10 and 3.14; 3.9 refuses the wheel
CI at f246ac2 hit the EXCHANGE race in the async0 variant: the rebuild stopped with "36 acknowledged rows ... reached only events__chm_old_bbbb". The test's operator-recovery branch then looked for the exception class name, which the runner does not print, and failed instead of recovering. Solves: async0 CI failure at f246ac2 (detection worked; test check wrong) Tests: async0 variant 30/30 locally (race not hit locally)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge, and do not mark this ready for review.
mainis the released 0.5.x line. Merging the 1.0 line into it is not approved, and tagging and publishing are Dan's decision on DRY-1246.This draft exists only because
ci.ymlruns on pushes tomainand on pull requests, sody/ch-migrate-1.0had never had a remote CI run. The 0.6, 0.7, 0.8 and 1.0 acceptance tickets (DRY-1396, DRY-1401, DRY-1405, DRY-1411) require CI on Python 3.10–3.14. Dan approved this approach on 2026-10-13.New on the branch since its last push on 2026-10-09:
c9d19eb: version rows can no longer be deduplicated away on Replicated/Shared MergeTree (DRY-1447).fad701e: the integration suite runs on ClickHouse Cloud (DRY-1448, DRY-1446, DRY-1413).df23886:rebuild_tablerefuses on ClickHouse Cloud withcloud_unsupported(DRY-1445, interim).c687095: Cloud crash-resume test, SIGKILL during a running mutation (DRY-1401).bea2911: CHANGELOG behaviour-change callouts (DRY-1396, DRY-1401).122b298: Q9 Cloud record in the waiting design note.Close this PR when CI evidence is no longer needed.