Add ch dictionary primitive

.changeset/dictionary-schema-primitive.md

@@ -0,0 +1,16 @@
+---
+"chkit": patch
+"@chkit/core": patch
+"@chkit/clickhouse": patch
+"@chkit/plugin-pull": patch
+"@chkit/plugin-codegen": patch
+---
+
+Add `dictionary()` as a first-class ClickHouse schema primitive, mirroring `materializedView()` across the full lifecycle: DSL authoring, validation, canonicalization, SQL rendering, migration planning/diff, drift, `check`, destructive-op safety, `pull` introspection, and `codegen` typed interfaces.
+
+- `dictionary({ database, name, attributes, primaryKey, source, layout, lifetime, range?, settings?, comment? })` — attributes support `default`/`expression` (mutually exclusive), `hierarchical`, `bidirectional` (requires `hierarchical`), `injective`, and `isObjectId`. `range: { min, max }` renders `RANGE(MIN ... MAX ...)` for `RANGE_HASHED`/`COMPLEX_KEY_RANGE_HASHED` layouts, and `settings` renders `SETTINGS(...)`.
+- ClickHouse has no `ALTER DICTIONARY`, so any structural change plans a single atomic `CREATE OR REPLACE DICTIONARY`. Dropping a dictionary is treated as destructive and blocked without `--allow-destructive`.
+- Set `renamedFrom` on a dictionary (or pass `--rename-dictionary old_db.old=new_db.new` to `chkit generate`) to rename a dictionary via `RENAME DICTIONARY IF EXISTS ... TO ...` instead of a destructive drop + create.
+- `chkit pull` introspects live dictionaries (including `RANGE`/`SETTINGS` and all attribute modifiers) into typed schema files, preserving ClickHouse's `[HIDDEN]` password redaction on `SOURCE(...)` credentials; the planner masks passwords before diffing so a live `[HIDDEN]` value never shows as perpetual drift.
+- `codegen` generates a typed interface (and optional Zod schema) for each dictionary from its `attributes`, always included regardless of `includeViews`.
+- `ON CLUSTER` mode now correctly stamps every dictionary DDL statement, including `CREATE OR REPLACE DICTIONARY` and `RENAME DICTIONARY`.

apps/docs/src/content/docs/cli/generate.md

@@ -21,6 +21,7 @@ chkit generate [flags]
 | `--migration-id <id>` | string | — | Escape hatch: override the default timestamp migration prefix |
 | `--rename-table <mapping>` | string | — | Explicit table rename: `old_db.old_table=new_db.new_table` |
 | `--rename-column <mapping>` | string | — | Explicit column rename: `db.table.old_column=new_column` |
+| `--rename-dictionary <mapping>` | string | — | Explicit dictionary rename: `old_db.old_dict=new_db.new_dict` |
 | `--table <selector>` | string | — | Scope operations to matching tables |
 | `--dryrun` | boolean | `false` | Print the plan without writing any files |
 | `--empty` | boolean | `false` | Scaffold a blank manual migration without diffing the schema |
@@ -63,10 +64,12 @@ An empty match set emits a warning and produces no output.
 chkit detects potential renames through two mechanisms:
 
 1. **Schema metadata** — set `renamedFrom` on your schema definition
-2. **CLI flags** — `--rename-table old_db.old_table=new_db.new_table` and `--rename-column db.table.old_col=new_col`
+2. **CLI flags** — `--rename-table old_db.old_table=new_db.new_table`, `--rename-column db.table.old_col=new_col`, and `--rename-dictionary old_db.old_dict=new_db.new_dict`
 
 CLI flags take priority when both sources specify a mapping for the same object. Rename flags accept comma-separated values for multiple mappings.
 
+A dictionary rename emits a single `RENAME DICTIONARY IF EXISTS ... TO ...` statement instead of a `drop_dictionary` + `create_dictionary` pair — see [Dictionary rename](/schema/dsl-reference/#dictionary-rename).
+
 Validation errors are raised for conflicting, chained, or cyclic rename mappings.
 
 ### Dryrun mode
@@ -79,7 +82,7 @@ Plans for objects in a database lead with a `create_database` operation (`CREATE
 
 With `--empty`, the command skips the schema diff entirely and writes a blank, timestamped migration stub for you to hand-edit. Use it for DDL that chkit does not model — raw `INSERT`/backfill statements, `OPTIMIZE`, manual dictionary reloads, or one-off data fixes.
 
-The stub carries the standard migration header (with `operation-count: 0`) plus a placeholder comment. The snapshot is left untouched, so an empty migration never absorbs pending schema drift. The `--name` and `--migration-id` flags apply; without `--name`, the file defaults to `manual`. Schema-diff flags (`--table`, `--rename-table`, `--rename-column`, `--dryrun`) are not used in empty mode.
+The stub carries the standard migration header (with `operation-count: 0`) plus a placeholder comment. The snapshot is left untouched, so an empty migration never absorbs pending schema drift. The `--name` and `--migration-id` flags apply; without `--name`, the file defaults to `manual`. Schema-diff flags (`--table`, `--rename-table`, `--rename-column`, `--rename-dictionary`, `--dryrun`) are not used in empty mode.
 
 `chkit migrate` picks the file up like any other migration and applies it in filename order. Write your SQL into the stub *before* applying it — editing a migration after it has run triggers a checksum mismatch.
 
@@ -129,6 +132,12 @@ chkit generate --rename-table old_db.users=new_db.accounts
 chkit generate --rename-column analytics.events.old_name=new_name
 ```
 
+**Explicit dictionary rename:**
+
+```sh
+chkit generate --rename-dictionary old_db.old_dict=new_db.new_dict
+```
+
 ## Exit codes
 
 | Code | Meaning |

apps/docs/src/content/docs/guides/clickhouse-compatibility.md

@@ -20,6 +20,7 @@ A few schema features depend on the ClickHouse version of your target:
 | [Refreshable materialized views](/schema/refreshable-views/) | Production-ready on **24.10+** (no flag). Experimental and flag-gated on 23.12–24.9. chkit targets 24.10+. |
 | `set` data-skipping index | **ClickHouse 26+** requires the `set(0)` form rather than a bare `set`; chkit emits `set(maxRows)` accordingly. See the [DSL reference](/schema/dsl-reference/). |
 | `uniqueKey` | Renders `UNIQUE KEY` DDL, which is supported on ObsessionDB / ClickHouse Cloud engines but rejected by vanilla `MergeTree`. |
+| [`dictionary()`](/schema/dsl-reference/#dictionary) | DDL `CREATE DICTIONARY` only — supported broadly on 21.x+. chkit does not model XML-config dictionaries or `RENAME`/`SYSTEM RELOAD DICTIONARY`. |
 
 If you target a single-node open-source ClickHouse, prefer the standard `MergeTree` engine family and avoid the Cloud/Shared-only features above.
 

apps/docs/src/content/docs/plugins/codegen.md

@@ -10,6 +10,7 @@ This document covers practical usage of the optional `codegen` plugin.
 ## What it does
 
 - Generates deterministic TypeScript row types from chkit schema definitions.
+- Generates a typed interface (and optional Zod schema) for each `dictionary()` from its `attributes` — dictionaries are always included, regardless of `includeViews`.
 - Optionally generates Zod schemas from the same definitions.
 - Optionally generates typed ingestion functions for inserting rows into ClickHouse tables. Generated ingest helpers gzip-compress request bodies by default and can opt out per call.
 - Optionally generates a self-contained runtime migration module with all migration SQL inlined, for environments without filesystem access (e.g., Cloudflare Workers).
@@ -186,4 +187,4 @@ When `runOnGenerate` is enabled (the default), the migration module is regenerat
 
 - Query-level type inference is not included.
 - Arbitrary SQL expression typing is not included.
-- Views/materialized views are opt-in and emitted conservatively.
+- Views/materialized views are opt-in (`includeViews`) and emitted conservatively (`{ [key: string]: unknown }`). Dictionaries are always included and typed from `attributes`, since their shape is structured rather than an arbitrary query.

apps/docs/src/content/docs/plugins/pull.md

@@ -1,6 +1,6 @@
 ---
 title: Pull Plugin
-description: Introspect live ClickHouse tables, views, and materialized views and generate chkit schema files.
+description: Introspect live ClickHouse tables, views, materialized views, and dictionaries and generate chkit schema files.
 sidebar:
   order: 3
 ---
@@ -11,6 +11,7 @@ This document covers practical usage of the optional `pull` plugin.
 
 - Connects to a live ClickHouse instance and introspects table metadata (columns, engines, indexes, projections, partitioning, TTL, settings).
 - Introspects views and materialized views (including `TO` clause parsing).
+- Introspects dictionaries (attributes — including `HIERARCHICAL`/`BIDIRECTIONAL`/`INJECTIVE`/`IS_OBJECT_ID` modifiers — primary key, `SOURCE`/`LAYOUT`/`LIFETIME`/`RANGE`/`SETTINGS`), preserving ClickHouse's `[HIDDEN]` password redaction — see [Credential handling](#credential-handling-hidden-passwords).
 - Generates a deterministic TypeScript schema file using `@chkit/core` builders.
 - Supports filtering by database and dry-run previews.
 
@@ -106,7 +107,39 @@ export default schema(app_events, app_events_view, app_events_mv)
 
 Tables may also include `uniqueKey`, `ttl`, `settings`, `indexes`, and `projections` when present in the source metadata.
 
+## Credential handling (`[HIDDEN]` passwords)
+
+ClickHouse redacts inline `SOURCE(...)` passwords to `[HIDDEN]` on introspection (`system.tables.create_table_query`, `SHOW CREATE DICTIONARY`) and does not offer a way to recover the original value via `pull` — chkit does not attempt `format_display_secrets_in_show_and_select`. When a pulled dictionary's `source` contains `[HIDDEN]`, the generated file emits it verbatim with a leading comment:
+
+```ts
+// NOTE: password redacted by ClickHouse — replace '[HIDDEN]' with your credential (e.g. process.env.X).
+const default_users_dict = dictionary({
+  database: "default",
+  name: "users_dict",
+  attributes: [
+    { name: "id", type: "UInt64" },
+    { name: "name", type: "String" },
+  ],
+  primaryKey: ["id"],
+  source: "MYSQL(host 'db' port 3306 user 'reader' password '[HIDDEN]' db 'app' table 'users')",
+  layout: "HASHED()",
+  lifetime: "300",
+})
+```
+
+Replace `[HIDDEN]` with a real credential — typically an environment-variable interpolation, matching how you'd author the dictionary by hand (see [Credentials in `source`](/schema/dsl-reference/#credentials-in-source)):
+
+```ts
+source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`,
+```
+
+For round-trip fidelity without a manual edit, use [named collections](https://clickhouse.com/docs/operations/named-collections) on the ClickHouse side instead of an inline password — chkit does not require this, but it's the ClickHouse-native way to avoid the redaction entirely.
+
+Because chkit masks the `password '...'` token before comparing `source` for drift, a live `[HIDDEN]` value is never reported as perpetual drift against your real secret once you've filled it in.
+
 ## Current limits
 
 - Materialized views without a `TO` clause are skipped.
+- Dictionaries whose `create_table_query` can't be parsed (attributes, primary key, or `SOURCE`/`LAYOUT`/`LIFETIME` missing) are skipped.
+- XML-config dictionaries (not created via DDL) are not introspected.
 - Requires a live ClickHouse connection.

apps/docs/src/content/docs/schema/dsl-reference.md

@@ -8,7 +8,7 @@ sidebar:
 Schema files are TypeScript files that export definitions using functions from `@chkit/core`. All exported definitions are collected when chkit loads schema files matched by the `schema` glob in your [configuration](/configuration/overview/).
 
 ```ts
-import { schema, table, view, materializedView } from '@chkit/core'
+import { schema, table, view, materializedView, dictionary } from '@chkit/core'
 ```
 
 ## `schema()`
@@ -353,6 +353,87 @@ const dailyReport = materializedView({
 
 See [Refreshable materialized views](/schema/refreshable-views/) for the full `refresh` field reference, including APPEND mode, `DEPENDS ON`, and the ClickHouse rules that chkit validates.
 
+## `dictionary()`
+
+Creates a [ClickHouse dictionary](https://clickhouse.com/docs/sql-reference/dictionaries) definition — a key-value lookup structure backed by an external or in-database source, queried with `dictGet()`.
+
+```ts
+dictionary(input: Omit<DictionaryDefinition, 'kind'>): DictionaryDefinition
+```
+
+```ts
+import { dictionary } from '@chkit/core'
+
+const usersDict = dictionary({
+  database: 'default',
+  name: 'users_dict',
+  attributes: [
+    { name: 'id', type: 'UInt64' },
+    { name: 'name', type: 'String' },
+    { name: 'email', type: 'String', default: '' },
+  ],
+  primaryKey: ['id'],
+  source: `MYSQL(host 'db' port 3306 user 'reader' password '${process.env.MYSQL_PASSWORD}' db 'app' table 'users')`,
+  layout: `HASHED()`,
+  lifetime: `300`,
+  comment: 'User lookup dictionary',
+})
+```
+
+### Required fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `database` | `string` | ClickHouse database name |
+| `name` | `string` | Dictionary name |
+| `attributes` | `DictionaryAttribute[]` | Attribute definitions (see [Dictionary attributes](#dictionary-attributes)) |
+| `primaryKey` | `string[]` | Key attribute name(s) — every entry must name a declared attribute |
+| `source` | `string` | Raw `SOURCE(...)` body, e.g. `` `MYSQL(host '...' password '...' ...)` `` |
+| `layout` | `string` | Raw `LAYOUT(...)` body, e.g. `` `HASHED()` `` or `` `COMPLEX_KEY_HASHED()` `` |
+| `lifetime` | `string` | Raw `LIFETIME(...)` body, e.g. `` `300` `` or `` `MIN 300 MAX 360` `` |
+
+### Optional fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `range` | `{ min: string; max: string }` | `RANGE(MIN ... MAX ...)` — required by `RANGE_HASHED` / `COMPLEX_KEY_RANGE_HASHED` layouts. Both `min` and `max` must name declared attributes |
+| `settings` | `Record<string, string \| number>` | Raw `SETTINGS(...)` key/value pairs, e.g. `{ dictionary_use_async_executor: 1 }` |
+| `comment` | `string` | Dictionary comment |
+| `renamedFrom` | `{ database?: string; name: string }` | Previous identity for rename tracking |
+
+:::note
+`source`, `layout`, and `lifetime` are **raw strings**, not a typed sub-DSL — chkit passes them through to ClickHouse verbatim inside `SOURCE(...)`, `LAYOUT(...)`, and `LIFETIME(...)` respectively. There is no key/layout coupling validation or required-parameter checking (e.g. `size_in_cells` for `HASHED`); ClickHouse validates the DDL when it's applied.
+:::
+
+### Dictionary attributes
+
+Each entry in the `attributes` array is a `DictionaryAttribute`.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | `string` | Attribute name |
+| `type` | `string` | ClickHouse type |
+| `default` | `string \| number \| boolean` | `DEFAULT` value for missing keys. Mutually exclusive with `expression` |
+| `expression` | `string` | `EXPRESSION` computed from source columns. Mutually exclusive with `default` |
+| `hierarchical` | `boolean` | Marks the attribute `HIERARCHICAL` |
+| `bidirectional` | `boolean` | Marks the attribute `BIDIRECTIONAL` — enables parent/child lookups in both directions. Only valid alongside `hierarchical` |
+| `injective` | `boolean` | Marks the attribute `INJECTIVE` |
+| `isObjectId` | `boolean` | Marks the attribute `IS_OBJECT_ID` (MongoDB sources) |
+
+### Credentials in `source`
+
+Inline credentials in `source` (e.g. a MySQL/PostgreSQL `password '...'`) should be interpolated from environment variables at schema-authoring time, the same way you'd handle any other secret in a TypeScript config file:
+
+```ts
+source: `MYSQL(host 'db' password '${process.env.MYSQL_PASSWORD}' ...)`,
+```
+
+ClickHouse redacts inline passwords back to `[HIDDEN]` on introspection (`SHOW CREATE DICTIONARY`, `system.dictionaries`). chkit masks the `password '...'` token before comparing `source` for drift/diff purposes, so a live `[HIDDEN]` value is never reported as perpetual drift against your real secret — see [Pull: credential handling](/plugins/pull/#credential-handling-hidden-passwords).
+
+### No `ALTER DICTIONARY`
+
+ClickHouse has no `ALTER DICTIONARY` — every structural change to a dictionary is rendered as a single `CREATE OR REPLACE DICTIONARY` statement (atomic, dependency-safe). See [Structural vs. alterable properties](#structural-vs-alterable-properties). A pure rename (`renamedFrom` with no other change) is the one exception — it renders as `RENAME DICTIONARY`, not a replace; see [Dictionary rename](#dictionary-rename).
+
 ## Type system reference
 
 The [codegen plugin](/plugins/codegen/) maps ClickHouse types to TypeScript types using these rules:
@@ -402,7 +483,22 @@ columns: [
 ]
 ```
 
-Both table and column renames can be overridden by CLI flags `--rename-table` and `--rename-column`.
+### Dictionary rename
+
+Set `renamedFrom` on a dictionary definition to rename a dictionary. This emits a single `RENAME DICTIONARY IF EXISTS ... TO ...` statement instead of a `drop_dictionary` + `create_dictionary` pair:
+
+```ts
+const lookupDict = dictionary({
+  database: 'app',
+  name: 'lookup_dict', // new name
+  renamedFrom: { name: 'users_dict' }, // old name
+  // ...
+})
+```
+
+The `database` field in `renamedFrom` is optional and defaults to the dictionary's current database.
+
+Table, column, and dictionary renames can all be overridden by CLI flags: `--rename-table`, `--rename-column`, and `--rename-dictionary`.
 
 ## Plugin configuration
 
@@ -448,6 +544,12 @@ chkit validates schema definitions and throws a `ChxValidationError` if any issu
 - **Empty codec chain** (`codec_chain_empty`) -- a `codec` array with no steps; provide at least one codec or omit the field
 - **Multiple general codecs** (`codec_chain_multiple_general`) -- more than one general codec in a chain; only one is allowed
 - **Codec chain must end with a general codec** (`codec_chain_must_end_with_general`) -- preprocessors must precede the single general codec (`NONE`, `LZ4`, `LZ4HC`, `ZSTD`, `T64`, `GCD`, `ALP`)
+- **Dictionary missing primary key** (`dictionary_missing_primary_key`) -- a dictionary's `primaryKey` is empty
+- **Dictionary primary key references missing attribute** (`dictionary_primary_key_missing_attribute`) -- a `primaryKey` entry doesn't name a declared attribute
+- **Dictionary missing source/layout/lifetime** (`dictionary_missing_source`, `dictionary_missing_layout`, `dictionary_missing_lifetime`) -- one of these raw-string fields is empty
+- **Dictionary attribute default/expression exclusive** (`dictionary_attribute_default_expression_exclusive`) -- an attribute sets both `default` and `expression`
+- **Dictionary range references missing attribute** (`dictionary_range_missing_attribute`) -- `range.min`/`range.max` doesn't name a declared attribute
+- **Dictionary bidirectional requires hierarchical** (`dictionary_bidirectional_requires_hierarchical`) -- an attribute sets `bidirectional` without `hierarchical`
 
 ## Structural vs. alterable properties
 
@@ -459,6 +561,8 @@ When a property changes, chkit determines whether the table can be altered in pl
 
 Views and materialized views always use drop + recreate.
 
+Dictionaries have no ALTER at all: any change to `attributes`, `primaryKey`, `layout`, `lifetime`, or `comment` renders as a single `CREATE OR REPLACE DICTIONARY` (`risk=caution`). A `source` change is compared with the password masked (see [Credentials in `source`](#credentials-in-source)), so only a real credential/connection change triggers a replace. Removing a dictionary from schema emits `DROP DICTIONARY` (`risk=danger`, requires `--allow-destructive`).
+
 :::danger
 Changing a structural property on an existing table generates a `DROP TABLE` followed by `CREATE TABLE` — **all rows are permanently deleted and the table is recreated empty**. The data is not copied over. The drop is classified `risk=danger` (blocked without `--allow-destructive`) and `chkit migrate` flags it with the distinct `table_recreate_data_loss` warning. To preserve data, migrate by hand instead: create a new table with the desired structure, `INSERT INTO new SELECT ... FROM old`, then swap names and drop the old table.
 :::