feat(kit): effect-orm pull — definitions from an existing ClickHouse database

.changeset/pull-clickhouse.md

@@ -0,0 +1,11 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+`effect-orm pull` writes table definitions from an existing ClickHouse database, so adopting effect-orm on a database that already has tables no longer means writing every `CH.table` by hand.
+
+- It reads the database the config's `driver` connects to and writes one `CH.table` per table: engine and arguments, keys as declared, partition key, TTL, settings, comments, column defaults, codecs and skipping indexes. The definitions yield exactly what the database holds, so `generate --baseline` adopts them and the next `generate` has nothing to do.
+- Column types map to the built-ins that write the same SQL; others become `CH.custom` with a fitting schema (`UUID`, `Date`, `Decimal`, `Enum8`, …) or `CH.untyped`. Views and non-MergeTree tables are written as `external: true`. Materialized views, `SAMPLE BY` and `EPHEMERAL` columns are listed for review in the file's header.
+- `pull` is also exported from `/kit`.
+
+`verify` reads a table's sorting and primary keys as declared. It had compared `system.tables.sorting_key`, which has a `VersionedCollapsingMergeTree`'s version column appended, so such a table always showed drift.

.gitignore

@@ -4,3 +4,4 @@ coverage/
 *.tgz
 .env*
 .DS_Store
+tests/.pull-*

README.md

@@ -226,7 +226,7 @@ regressions live in [`src/docs-examples.test.ts`](./src/docs-examples.test.ts).
 | `@maple-dev/effect-orm/sql`         | The low-level `SqlFragment` AST (`raw`, `ident`, `compile`, …) for hand-rolling fragments.                                                                    |
 | `@maple-dev/effect-orm/tinybird`    | Tinybird datasources and materialized views, SDK-compatible, that are also query tables; `buildProject` writes the datafiles. See [Tinybird](./docs/tinybird.md). |
 | `@maple-dev/effect-orm/schema`      | Migration tooling over `table` values: DDL rendering, snapshots, the schema diff. Pure.                                                                       |
-| `@maple-dev/effect-orm/kit`, `/migrate` | `effect-orm generate` and `check`; applying migrations through a driver you provide. See [Schema and migrations](./docs/migrations.md).                  |
+| `@maple-dev/effect-orm/kit`, `/migrate` | `effect-orm generate`, `check` and `pull` (definitions from an existing ClickHouse database); applying migrations through a driver you provide. See [Schema and migrations](./docs/migrations.md).                  |
 | `@maple-dev/effect-orm/database`    | `Database` over your `SqlClient`: `run`, `execute`, `transaction` with retry.                                                                                  |
 | `@maple-dev/effect-orm/clickhouse-http` | A ClickHouse `SqlClient` over `fetch`, for Cloudflare Workers and any runtime with `fetch`; give it to `Database`. |
 

docs/migrations.md

@@ -151,6 +151,45 @@ tables. Independent branches are fine: the next `generate` records both as paren
 change the same table conflict; delete one migration and generate it again on top of the other.
 Run `check` in CI.
 
+### Starting from an existing database
+
+A ClickHouse database that predates effect-orm does not need its tables written by hand.
+`effect-orm pull` reads the database the config's `driver` connects to and writes a schema
+module, one `CH.table` per table; then `generate --baseline` starts the history from it:
+
+```sh
+effect-orm pull                         # writes the config's schema module; --out <path> for another
+effect-orm generate --baseline          # a migration that runs nothing, with this schema as its snapshot
+effect-orm migrate                      # runs no statement; records the baseline as applied
+```
+
+The module is written so that its definitions yield exactly what the database holds: engine
+and its arguments, keys as declared, partition key, TTL, settings, comments, column defaults
+(`DEFAULT`, `MATERIALIZED`, `ALIAS`), codecs and skipping indexes, with every expression as the
+server's own text. The first `generate` after the baseline finds nothing to do. A `Replicated`
+or `Shared` engine is written as its plain family; replication is a render option.
+
+Column types map to the built-ins that write the same SQL (`LowCardinality(Nullable(String))` is
+`CH.lowCardinality(CH.nullable(CH.string))`). A type without one becomes `CH.custom` with the
+schema its JSON value decodes with (`UUID` and `Date` as strings, `Decimal` and `Int8` as
+numbers, an `Enum8` as its labels), and a type the generator does not know (`Tuple`, `JSON`, a
+`Map` keyed by numbers) becomes `CH.untyped`. The file's header lists what to review:
+
+- **Materialized views** are not generated: a view's body is a query-DSL value, which SQL text
+  cannot be turned back into. The header quotes each body; write it with `CH.materializedView`.
+- **Views, dictionaries, `Distributed`, `Kafka` and other non-MergeTree tables** are written as
+  `external: true`: queryable, never migrated.
+- A table with a name that is not a plain identifier (`my-col`), or a MergeTree declared in the
+  old `MergeTree(date, key, granularity)` syntax, is written as external too.
+- **Not modeled, so left out:** `SAMPLE BY`, projections, constraints and `EPHEMERAL` columns.
+  Column-level TTLs are not read at all.
+- A `Decimal` wider than 15 digits decodes to a number, which rounds; the header flags each one.
+- A user who may not read `system.data_skipping_indices` gets no indexes, and a note saying so.
+
+`--tables a,b` writes only those tables; `--force` overwrites an existing file. `verify` reports
+the views and materialized views as `unexpected` until the schema declares them. Postgres is
+not supported yet.
+
 ## Applying migrations
 
 The library opens no connection. Give it a `MigrationDriver`, usually built from the

docs/reference.md

@@ -515,7 +515,7 @@ placeholders, double-quoted identifiers.
 | `DatabaseError`, `TransactionCommitFailed`, `TransactionRollbackFailed` and the other transaction errors | `/database` |
 | `withQuerySummary`, `QuerySummary` — the `db.query.summary` of the statements an effect runs, for spans | `/database` |
 | `ClickhouseHttpClient`, `layer`, `make`, `ClickhouseHttpClientConfig` — a ClickHouse `SqlClient` over `fetch`, for Workers | `/clickhouse-http` |
-| `defineConfig`, `generate`, `check`, `loadSchema`, `readMigrations`, `analyze`, `KitError`       | `/kit`            |
+| `defineConfig`, `generate`, `check`, `pull`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit`            |
 | `defineSuite`, `query`, `caseFromCompiled`, `runSuite`, `compareRuns`, `compareBudgets`          | `/benchmark`      |
 | `Suite`, `RunOutput`, `BenchmarkError` and benchmark contracts                                   | `/benchmark`      |
 | `makeHttpClient`, `makeHttpTransport`, `httpConfigFromEnv`                                       | `/benchmark/http` |

src/kit.ts

@@ -16,4 +16,5 @@ export {
 	type GenerateResult,
 	type KitConfig,
 } from "./kit/generate"
+export { pull, type PullOptions, type PullResult } from "./kit/pull"
 export { runCli } from "./kit/cli"