feat(database): relational reads with defineRelations and findMany

.changeset/relational-reads.md

@@ -0,0 +1,5 @@
+---
+"@maple-dev/effect-orm": minor
+---
+
+`Db.defineRelations(tables, (r) => ...)` and `Db.findMany(schema.table, { where, orderBy, limit, with })` in `/database` read rows with their related rows, nested to any depth, as Drizzle's relational queries and Prisma's `include` do. `r.one` and `r.many` are checked against the tables. Each level of `with` is one statement, `WHERE <key> IN (...)` over the keys of the level above, decoded through its own row codec, so a nested `timestamptz` is a `DateTime.Utc`. A level whose parent rows carry one tenant is pinned to it, and `requireTenantScope` checks every level. See docs/relations.md and design/relations.md.

README.md

@@ -207,6 +207,7 @@ Full guides live in [`docs/`](./docs/README.md):
 | [Decoding results](./docs/decoding-results.md)             | `rowSchema`, `decodeRows`, decode errors                        |
 | [Running a query](./docs/running-queries.md)               | Executing the SQL with a real client, wire settings, `SETTINGS` |
 | [Tenant scoping](./docs/tenant-scoping.md)                 | `tenantColumn`, what marks a query scoped, `crossTenant()`      |
+| [Relational reads](./docs/relations.md)                    | `defineRelations`, `findMany` with nested `with`                |
 | [Postgres](./docs/postgres.md)                             | The Postgres dialect, its column types and functions            |
 | [Schema and migrations](./docs/migrations.md)              | DDL from `table`, `effect-orm generate`, applying migrations    |
 | [Extending the DSL](./docs/extending.md)                   | `defineFn`, raw escape hatches, handwritten SQL                 |

design/relations.md

@@ -0,0 +1,108 @@
+# Relational reads: `defineRelations` and `findMany`
+
+Status: first slice built (issue #79; `docs/relations.md`). `one` and `many` relations over
+single-column keys, `with` to any depth, `where` and `orderBy` per level, `limit` and `offset` on
+the top level. Section 5 lists what is not built.
+
+## 1. Goal
+
+Read a row with its related rows in one call, as Drizzle's relational queries
+(`db.query.users.findMany({ with: { posts: true } })`) and Prisma's `include` do. Keep what the
+rest of the library gives a query:
+
+- rows decoded through the column codecs, at every level;
+- a tenant scope derived for every statement, which `requireTenantScope` checks;
+- a result type derived from the `with` tree.
+
+## 2. The API
+
+```ts
+const schema = Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	users: { posts: r.many("posts", { from: "id", to: "authorId" }) },
+	posts: { author: r.one("users", { from: "authorId", to: "id" }) },
+}))
+
+Db.findMany(schema.users, { where: ($) => [$.orgId.eq(orgId)], with: { posts: { with: { author: true } } } })
+```
+
+- **One callback over all tables**, as Drizzle v2 `defineRelations`. A relation names its target
+  by key, not by value, so tables that point at each other need no thunks.
+- **Relations live beside the tables, not in them.** `table()` stays as it is, and the DDL, the
+  diff and the importers do not change. A Prisma importer can write the `defineRelations` call.
+- **`findMany` lives in `/database`.** It runs more than one statement, so it needs the
+  `Database` service. `defineRelations` sits next to it, so the dialect entries do not change.
+- **Checked in the types.** The target must be a table key, `to` a column of the target, `from` a
+  column of the source, and both must decode to `string | number | bigint`. A relation name must
+  not be a column name.
+
+## 3. One statement per level, not JSON aggregation
+
+The issue proposed one statement with JSON aggregation (`jsonb_agg` / `json_object`), as Drizzle
+RQB v2 does. This slice runs **one statement per level** instead:
+`SELECT <all columns> FROM <target> WHERE <to> IN (<keys from the level above>) [AND <tenant pin>]
+[AND <where>] [ORDER BY ...]`. The loader then groups the rows by key in memory. Prisma does the
+same by default.
+
+Why:
+
+1. **Decoding stays correct for free.** Each level is an ordinary builder query, decoded by its
+   own derived row codec. With JSON aggregation, every column comes back in its JSON form, not
+   its wire form. An `int8` becomes a JSON number and loses precision past 2^53, a `timestamptz`
+   becomes an ISO string in a different format, `bytea` becomes hex, and a `numeric` can change.
+   Each column type would need a second "from JSON" codec per dialect, or a cast to text in the
+   SQL. That is a large change to the type system for one feature.
+2. **Tenant scope needs no new analysis.** Each level is a query the existing analysis already
+   understands. A nested JSON subquery would need correlated-subquery evidence through
+   `jsonb_agg`, which the analysis does not have yet.
+3. **Every dialect.** The SQL is only `IN (...)`, so the same code runs on Postgres, on
+   ClickHouse, and later on SQLite / D1 and MySQL. JSON aggregation needs a different function
+   set per dialect, and ClickHouse has no `jsonb_agg`.
+4. **The statement count is fixed.** It is the depth of the `with` tree, not the row count, so
+   there is no N+1.
+
+The costs:
+
+- **More round trips.** Three levels are three statements. On a server close to the app this is
+  small. JSON aggregation is better on a slow link.
+- **No single snapshot.** Outside a transaction, a write can land between two levels. The docs
+  say to use `Db.transaction({ isolationLevel: "repeatable read" })`. `findMany` does not open a
+  transaction itself: ClickHouse has none, and the caller may already be in one.
+- **Long `IN` lists.** All keys of a level are in one list. Postgres has a limit of 65,535 bound
+  values. Chunking belongs with #57 (parameter limits).
+
+JSON aggregation can come later as a second strategy behind the same API, per dialect, once the
+column types have JSON codecs. The result type and the tenant rule would not change.
+
+## 4. Tenant scope on nested levels
+
+Each level goes through `Database.run`, so it is checked like any other statement. A nested
+level filters on the parent keys, which says nothing about the tenant. So the loader adds a pin:
+
+> When the parent table and the target table both declare a `tenantColumn`, and every parent row
+> carries the same tenant value `V`, the nested query also gets `<target tenant column> = V`.
+
+This is sound. The pin is an ordinary `eq` on a literal, so the existing analysis proves
+`"single-tenant"`. `V` comes from rows the parent query read, so a level never reads a tenant its
+parent did not read. The pin also hides bad data: a child row whose tenant differs from its
+parent's.
+
+No pin when the parent table is untenanted, or the parent rows carry more than one tenant. The
+level is then `"cross-tenant"`, and `requireTenantScope` refuses it, which is the safe default. A
+level whose `to` column is the target's tenant column, with one key, is scoped by the one-value
+`IN` already. `crossTenant: true` declares every level `.crossTenant()`.
+
+The pin does not pass through an untenanted level: a grandparent's tenant says nothing about a
+grandchild reached through a shared table.
+
+## 5. Not built
+
+| What | Why not now |
+| --- | --- |
+| JSON aggregation strategy | Needs JSON codecs per column type (section 3) |
+| `columns` per level | Small, but changes the row type of every level; next slice |
+| `limit` per parent on a nested level | Needs `row_number()` over a partition, or a lateral join |
+| Composite keys | The loader would match tuples; no issue asks for it yet |
+| `findFirst` | `findMany` with `limit: 1`; cardinality helpers are #69 |
+| `_count`, `some` / `every` / `none` | Issue #80 |
+| Params in nested `where` | `findMany` takes no params; values go in as literals |
+| SQLite, MySQL, D1 | Not on main yet; the SQL here is plain `IN`, so no dialect work is expected |

docs/README.md

@@ -65,6 +65,7 @@ Roughly in reading order.
 | [Postgres](./postgres.md)                             | The Postgres dialect, its column types and functions                                |
 | [Schema and migrations](./migrations.md)              | DDL from `CH.table` / `PG.table`, `materializedView`, `effect-orm generate`, applying migrations, adopting drizzle-kit |
 | [Statements and transactions](./database.md)          | `Database` over your `SqlClient`: `run`, `execute`, `transaction`, retry            |
+| [Relational reads](./relations.md)                    | `defineRelations`, `findMany` with nested `with`, tenant scope per level            |
 
 ## Reference
 

docs/reference.md

@@ -513,6 +513,7 @@ placeholders, double-quoted identifiers.
 | `run`, `status`, `verify`, `baseline`, `MigrationDriver` and the migrate errors                  | `/migrate`        |
 | `Database`, `run`, `sql`, `query`, `execute`, `transaction`, `requireTransaction`, `retryContention`, `Transaction` | `/database` |
 | `DatabaseError`, `TransactionCommitFailed`, `TransactionRollbackFailed` and the other transaction errors | `/database` |
+| `defineRelations`, `findMany`, `Relation`, `RelationBuilder`, `RelationsConfig`, `RelationTables`, `RelationalSchema`, `RelationalTable`, `RelationOptions`, `FindManyOptions`, `FindManyRow`, `WithOf`, `KeyColumn` — relational reads, one statement per level ([Relational reads](./relations.md)) | `/database` |
 | `withQuerySummary`, `QuerySummary` — the `db.query.summary` of the statements an effect runs, for spans | `/database` |
 | `ClickhouseHttpClient`, `layer`, `make`, `ClickhouseHttpClientConfig` — a ClickHouse `SqlClient` over `fetch`, for Workers | `/clickhouse-http` |
 | `defineConfig`, `generate`, `check`, `pull`, `loadSchema`, `readMigrations`, `analyze`, `KitError` | `/kit`            |

docs/relations.md

@@ -0,0 +1,129 @@
+# Relational reads
+
+`Db.defineRelations` declares how tables relate. `Db.findMany` reads rows of one table with the
+related rows under them, nested to any depth, as Drizzle's relational queries and Prisma's
+`include` do. Both are in `@maple-dev/effect-orm/database`.
+
+```ts title="relations-find-many.ts"
+import { PgliteClient } from "@effect/sql-pglite"
+import { Effect, Layer } from "effect"
+import * as Db from "@maple-dev/effect-orm/database"
+import * as PG from "@maple-dev/effect-orm/postgres"
+
+const Users = PG.table("users", { columns: { id: PG.int4, name: PG.text }, primaryKey: ["id"] })
+const Posts = PG.table("posts", {
+	columns: { id: PG.int4, authorId: PG.column(PG.int4, { name: "author_id" }), title: PG.text },
+	primaryKey: ["id"],
+})
+
+export const schema = Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	users: { posts: r.many("posts", { from: "id", to: "authorId" }) },
+	posts: { author: r.one("users", { from: "authorId", to: "id" }) },
+}))
+
+const DatabaseLive = Db.layerSqlClient({ dialect: PG.postgresDialect }).pipe(
+	Layer.provideMerge(PgliteClient.layer({ postgresqlconf: "timezone = 'UTC'" })),
+)
+
+export const users = Effect.gen(function* () {
+	yield* Db.execute(Db.sql`CREATE TABLE users (id int4 PRIMARY KEY, name text NOT NULL)`)
+	yield* Db.execute(Db.sql`CREATE TABLE posts (id int4 PRIMARY KEY, author_id int4 NOT NULL, title text NOT NULL)`)
+	yield* Db.execute(Db.sql`INSERT INTO users VALUES (1, 'ada'), (2, 'bob')`)
+	yield* Db.execute(Db.sql`INSERT INTO posts VALUES (10, 1, 'hello')`)
+	return yield* Db.findMany(schema.users, {
+		orderBy: ($) => [[$.id, "asc"]],
+		with: { posts: { with: { author: true } } },
+	})
+}).pipe(Effect.provide(DatabaseLive))
+// [
+//   { id: 1, name: "ada", posts: [{ id: 10, authorId: 1, title: "hello", author: { id: 1, name: "ada" } }] },
+//   { id: 2, name: "bob", posts: [] },
+// ]
+```
+
+## Declare relations
+
+`defineRelations(tables, build)` takes the tables by key, and a callback that returns the
+relations of each table by name. A relation names its target by that key, so two tables can point
+at each other, and a table can point at itself.
+
+| Builder | Loads | Result |
+| --- | --- | --- |
+| `r.many(target, { from, to })` | Every `target` row whose `to` column equals this row's `from` column | `ReadonlyArray<Row>` |
+| `r.one(target, { from, to })` | The first such row | `Row \| null` |
+
+The types check each declaration. The target must be a key of `tables`. `from` must be a column of
+the source table and `to` a column of the target. Both must decode to a `string`, `number` or
+`bigint`, because the loader matches keys by value. A relation name must not be a column name of
+its table, because the loaded row carries both.
+
+A `one` relation is always `| null`. A row can be missing even when a foreign key says it cannot:
+another statement can delete it between two levels.
+
+Relations are metadata only. They do not need a foreign key in the database, so they work on
+ClickHouse too.
+
+_(Backed by `src/database/relations.test-d.ts`, and `tests/relations.clickhouse.test.ts` on ClickHouse.)_
+
+## Read with `findMany`
+
+`findMany(schema.table, options?)` returns every column of the table, plus each relation that
+`with` names. `with` takes `true`, or the options of that level:
+
+| Option | Levels | Meaning |
+| --- | --- | --- |
+| `where` | All | More conditions, ANDed with the relation's key match |
+| `orderBy` | All | The order of the rows; nested rows keep it within each parent |
+| `with` | All | Relations to load under each row of this level |
+| `limit`, `offset` | Top only | Paging of the top level |
+| `crossTenant` | Top only | Declare every level `.crossTenant()` |
+
+Rows come back decoded through the codecs of their columns at every level. A `timestamptz` in a
+nested row is a `DateTime.Utc`, not a string.
+
+## One statement per level
+
+Each level of `with` is one ordinary builder query: every column of the target table,
+`WHERE <to> IN (<the from values of the level above>)`, plus the level's own `where` and
+`orderBy`. The loader groups the rows by key and puts them under their parents. A level with no
+keys sends no statement. A tree with three levels sends three statements, whatever the number of
+rows.
+
+The levels are separate statements. Outside a transaction, a write from another connection can
+land between two of them. Run `findMany` inside [`Db.transaction`](./database.md#transactions)
+with `isolationLevel: "repeatable read"` for one snapshot across all levels.
+
+All keys of a level go into one `IN` list. Postgres allows at most 65,535 bound values in one
+statement, so keep the top level paged with `limit`.
+
+## Tenant scope on every level
+
+Each level runs through `run`, so each level has its own
+[`tenantScope`](./tenant-scoping.md), and `requireTenantScope` checks each one.
+
+When the parent rows all carry one tenant, and both tables declare a `tenantColumn`, the nested
+query also gets `<tenant column> = <that tenant>`. That pins the level the same way a hand-written
+filter does. It also hides a child row whose tenant differs from its parent's.
+
+A level gets no pin when:
+
+- the parent table has no `tenantColumn`;
+- the parent rows carry more than one tenant.
+
+That level is then `"cross-tenant"`, and `requireTenantScope` refuses it. There is one exception:
+if the relation's `to` column is the target's tenant column and the level has one key, the
+one-value `IN` scopes it already. With `crossTenant: true`, every level is declared
+cross-tenant and runs.
+
+_(Backed by `src/database/relations.test.ts`.)_
+
+## Not yet
+
+- Choosing columns per level (`columns: { id: true }`). Every level reads every column.
+- A `limit` per parent on a nested level. That needs a window function or a lateral join.
+- Composite keys (`from` and `to` with more than one column).
+- `findFirst`. Use `findMany` with `limit: 1`.
+- Filters and counts over relations (`_count`, `some`, `every`, `none`).
+- `param.*` markers in a `where`: `findMany` takes no params, so write values in.
+
+See [design/relations.md](../design/relations.md) for the design and the alternatives.

src/database.ts

@@ -38,6 +38,21 @@ export {
 	TransactionUnsupported,
 	type TransactionError,
 } from "./database/errors"
+export {
+	defineRelations,
+	findMany,
+	type FindManyOptions,
+	type FindManyRow,
+	type KeyColumn,
+	type Relation,
+	type RelationalSchema,
+	type RelationalTable,
+	type RelationBuilder,
+	type RelationOptions,
+	type RelationsConfig,
+	type RelationTables,
+	type WithOf,
+} from "./database/relations"
 export { QuerySummary, withQuerySummary } from "./database/tracing"
 export { sql, type SqlIdentifier, type SqlRawText, type SqlTemplate } from "./database/sql"
 export type { DialectTransactions, IsolationLevel, TransactionSettings } from "./ch/dialect"

src/database/relations.test-d.ts

@@ -0,0 +1,82 @@
+// Type-level tests: relation declarations are checked against the tables, and
+// `findMany` rows follow the `with` tree.
+
+import type { DateTime, Effect } from "effect"
+import { expectTypeOf } from "expect-type"
+import * as PG from "../postgres"
+import * as Db from "../database"
+
+const Users = PG.table("users", { columns: { id: PG.int4, name: PG.text }, primaryKey: ["id"] })
+const Posts = PG.table("posts", {
+	columns: { id: PG.int4, authorId: PG.int4, title: PG.text, at: PG.timestamptz },
+	primaryKey: ["id"],
+})
+
+const schema = Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	users: { posts: r.many("posts", { from: "id", to: "authorId" }) },
+	posts: { author: r.one("users", { from: "authorId", to: "id" }) },
+}))
+
+type User = { readonly id: number; readonly name: string }
+type FindError = Db.DatabaseError | import("../postgres").QueryBuilderError | import("../postgres").CompiledQueryDecodeError
+
+// No `with`: the table's own row.
+expectTypeOf(Db.findMany(schema.users)).toEqualTypeOf<Effect.Effect<ReadonlyArray<User>, FindError, Db.Database>>()
+expectTypeOf(Db.findMany(schema.users, { where: ($) => [$.name.eq("ada")] })).toEqualTypeOf<
+	Effect.Effect<ReadonlyArray<User>, FindError, Db.Database>
+>()
+
+// `many` is an array, `one` is a row or null, nested to any depth.
+expectTypeOf(
+	Db.findMany(schema.users, {
+		with: { posts: { where: ($) => [$.title.eq("a")], with: { author: true } } },
+	}),
+).toEqualTypeOf<
+	Effect.Effect<
+		ReadonlyArray<{
+			readonly id: number
+			readonly name: string
+			readonly posts: ReadonlyArray<{
+				readonly id: number
+				readonly authorId: number
+				readonly title: string
+				readonly at: DateTime.Utc
+				readonly author: User | null
+			}>
+		}>,
+		FindError,
+		Db.Database
+	>
+>()
+
+// `where` and `orderBy` on a nested level see that level's columns.
+Db.findMany(schema.users, {
+	// @ts-expect-error -- `title` is a column of posts, not users
+	where: ($) => [$.title.eq("a")],
+})
+
+// A relation name `with` does not declare.
+// @ts-expect-error -- users has no relation `author`
+Db.findMany(schema.users, { with: { author: true } })
+
+// Declarations are checked: target table, `to` column, `from` column.
+Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	// @ts-expect-error -- no table `comments`
+	users: { posts: r.many("comments", { from: "id", to: "authorId" }) },
+}))
+Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	// @ts-expect-error -- posts has no column `userId`
+	users: { posts: r.many("posts", { from: "id", to: "userId" }) },
+}))
+Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	// @ts-expect-error -- users has no column `uid`
+	users: { posts: r.many("posts", { from: "uid", to: "authorId" }) },
+}))
+Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	// @ts-expect-error -- a timestamptz decodes to DateTime, which is not a key
+	posts: { author: r.one("users", { from: "at", to: "id" }) },
+}))
+Db.defineRelations({ users: Users, posts: Posts }, (r) => ({
+	// @ts-expect-error -- `name` is already a column of users
+	users: { name: r.many("posts", { from: "id", to: "authorId" }) },
+}))