Skip to content

refactor(db): move Postgres access from drizzle to effect-orm - #330

Merged
Makisuo merged 8 commits into
mainfrom
t3/adopt-effect-orm
Oct 12, 2026
Merged

Makisuo merged 8 commits into
mainfrom
t3/adopt-effect-orm

Conversation

@Makisuo

@Makisuo Makisuo commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Moves Hazel's Postgres access from drizzle-orm to effect-orm (@maple-dev/effect-orm 0.6.0) over @effect/sql-pg, following Maple's migration (maple#1353). drizzle-orm and drizzle-kit leave every Hazel package.

What changes

  • Tables: every table is a PG.table in packages/db/src/tables, keeping the drizzle names (schema.messagesTable, Message, …). Timestamps still read as Date. jsonb stays unvalidated and ids are type-only brands, as drizzle's $type was. The 18 enums are PG.enumType and the 7 unique keys push made constraints are PG.unique, so generate migrates both.
  • Database service:
    • db.run / db.first / db.query / db.execute / db.transaction.
    • Transactions pin every statement in their body through fiber context. The tx / TxFn / TransactionContext threading, makeQuery and transactionAwareExecute are gone, and a nested transaction is a savepoint.
    • DatabaseError keeps its type and gains constraint, sqlState and detail.
  • Clients:
    • Bun, the cluster and scripts use a pooled client that probes the database at startup, as before.
    • Workers (api, electric-proxy, bot-gateway) open one lazily dialed client per request, via Hyperdrive.
  • Call sites: all repositories, RPC handlers, routes, services, cluster crons and workflows, electric-proxy, bot-gateway and scripts use the PG builder. The Electric where-clause builder renders its own parameterized fragments with unqualified column names.
  • Migrations: drizzle-kit push is replaced by packages/db/migrations, written by effect-orm generate.
    • initial_schema is exactly what push produced, enums and unique constraints included; a fresh database pg_dumps identically to a pushed one.
    • bun run db:migrate adopts a pushed database by recording initial_schema as applied without running it, and builds a fresh one from scratch.
    • db:generate, db:check, db:status and db:verify come from effect-orm. db:verify reports no drift on a fresh database, a pushed one, or a copy of the local dev database.

Behavior notes

  • Outbox claim: locks candidate ids (FOR UPDATE SKIP LOCKED), then updates exactly those, in one transaction. It replaces the old CTE with the same guarantee.
  • Unique-conflict fallbacks (connect conversations, mounts, participants):
    • They now actually match the violated constraint, through error.constraint; the old check never matched.
    • They run the insert in a savepoint, so the recovery lookup works inside a transaction.
  • Upserts: they convert DateTime to Date before writing; the old as any hid this.
  • Raw SQL left (all values bound):
    • the startup SELECT 1
    • the RSS "feed due" interval condition
    • one COUNT(DISTINCT …) FILTER in the single-DM lookup
    • reset-all / merge-org-duplicates maintenance queries
    • test-harness TRUNCATE

Testing

  • 0.6.0 bump (latest commit): typecheck is clean, and the 263 tests that don't need Docker pass. The Docker-backed suites (db service, outbox, side effects, chat-sync integration) have not been re-run on this commit because local Docker stopped responding. The schema itself was verified against real Postgres: fresh, pushed and local-copy databases all verify with no drift.

  • Typecheck: clean in packages/{db,backend-core,auth,setup,domain,infra}, apps/{backend (both tsconfigs),cluster,electric-proxy,bot-gateway,web} and the alchemy config.

  • Tests: vitest passed 286 of 286 across 41 files before the 0.6.0 bump, including the chat-sync integration tests and the @hazel/db service tests against real Postgres 17 (Docker). The db tests cover rollback, savepoints, error classification and the recovery-after-conflict pattern.

  • Migrations:

    • A fresh database built by db:migrate pg_dumps identically to a drizzle-kit push database.
    • db:migrate adopted a copy of the local dev database and is idempotent.
    • generate reports no schema changes against the tables.
  • End to end, Bun: run against a copy of the local dev database, with this branch's backend and cluster.

    • A bot was created with the converted create-bot.ts.
    • The bot API was driven over HTTP: create (with reply and embed), list with cursor pagination, edit, reaction toggle on and off, and delete. All returned 200 with Electric transaction ids.
    • All outbox events were processed, including retries after a forced failure.
    • An @channel message ran the cluster notification workflow, which inserted the notification and incremented notificationCount.
  • End to end, Workers: a scratch Worker in local workerd (nodejs_compat) used Database.makeRequestConnection per request. Repositories, pagination, Date decoding, a transaction's txid and a rollback all worked across consecutive requests.

  • Not tested: the RPC/OAuth paths that need a real Clerk session, alchemy dev/deploy (real Hyperdrive), and Electric shape sync through the proxy in a browser.

Deploy notes

  • Run DATABASE_URL=<prd API role URL> bun run db:migrate once (packages/db). It records the existing pushed schema as applied and changes nothing else. db:migrate now refuses to run without DATABASE_URL.
  • Afterwards DATABASE_URL=<…> bun run db:verify should report no drift.

Known gaps

  • Not modeled by effect-orm: check constraints, and removing or reordering enum labels. Those go in a --custom migration.
  • Pre-existing bug: packages/setup's bots command never provides Database.Database (hidden by a cast), so it fails at runtime. Left as is.
  • Leftover lint warnings: unused identifiers that predate this branch.

🤖 Generated with Claude Code

Makisuo and others added 8 commits October 13, 2026 00:17
Every table as a PG.table in @hazel/db/tables, generated from drizzle's
runtime table config and checked against it: the DDL they render, applied
to an empty Postgres, pg_dumps identically to a drizzle-kit push, except
that drizzle's UNIQUE constraints are unique indexes of the same name
(effect-orm does not model unique constraints).

Timestamps read as Date and jsonb documents stay unvalidated, as drizzle
had them; ids are branded through the @hazel/schema schemas.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-orm

Database.Database is effect-orm over @effect/sql-pg: run/first/query/
execute/transaction, a pooled client for long-running processes and a
per-request client for Workers. Transactions pin every statement in
their body through fiber context, so the TxFn/TransactionContext
threading and makeQuery wrappers are gone. DatabaseError keeps its type
classification and gains constraint and sqlState.

Every backend-core repository and the proxy auth consumer query through
the PG builder over @hazel/db/tables. The Electric where-clause builder
renders its own parameterized fragments with unqualified column names.

Schema management moves from drizzle-kit push to effect-orm migrations:
an initial migration with the schema push left, then an effect-orm
baseline. db:migrate adopts a database push built by recording both as
applied, and migrates a fresh one from scratch.

Callers in apps/* are converted in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cluster crons and workflows, electric-proxy bot auth and access-context
cache query through the PG builder; drizzle-orm leaves both packages.
The cluster still probes the database at startup with the old retry
schedule, since the new pool dials lazily.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handlers query through db.run over the PG builder instead of
transactionAwareExecute and drizzle; unique-violation checks read
DatabaseError.constraint and the slug conflict reads its detail.
Maintenance scripts and the setup bots command follow, and drizzle-orm
and drizzle-kit leave apps/backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Chat-sync worker, side-effect, channel-access sync and connect services
query through the PG builder. The chat-sync DB harness migrates its
container with @hazel/db/migrate instead of db:push, and tests fake or
provide the new Database shape. The Bun backend probes the database at
startup with the old retry schedule, since the pool dials lazily.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Outbox claim locks the candidate ids (FOR UPDATE SKIP LOCKED), then
  updates exactly those, in one transaction: an UPDATE ... WHERE id IN
  (subquery ... LIMIT) may re-evaluate its subquery and over-claim.
- Unique-conflict fallbacks (connect conversations, mounts, participants)
  run their insert in a savepoint, so the recovery lookup does not hit an
  aborted transaction.
- The initial migration creates the seven unique keys drizzle-kit push
  made constraints as UNIQUE constraints: a fresh database now pg_dumps
  identically to a pushed one.
- Id columns are type-only brands again, as drizzle's $type was, so one
  stored id outside its schema's checks cannot fail a whole read.
- Database.layer probes the database at startup with the old retry
  schedule for every caller, replacing two copies of the probe; the
  transaction span is back; the redundant Reactivity layer is gone.
- db:migrate and the effect-orm driver require DATABASE_URL instead of
  falling back to localhost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Claude encountered an error —— View job


I'll analyze this and get back to you.

@coderabbitai

coderabbitai Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 161 files, which is 11 over the limit of 150.

To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: df8e3f95-daa7-4975-8bd2-7ce5bf901fb0
📥 Commits

Reviewing files that changed from the base of the PR and between 7a0865f and 611a36c.

⛔ Files ignored due to path filters (1)
  • bun.lock is excluded by !**/*.lock
📒 Files selected for processing (161)
  • CLAUDE.md
  • README.md
  • apps/backend/package.json
  • apps/backend/scripts/create-bot.ts
  • apps/backend/scripts/rebuild-channel-access.ts
  • apps/backend/scripts/reset-all.ts
  • apps/backend/scripts/seed-internal-bots.ts
  • apps/backend/src/lib/create-transactionId.ts
  • apps/backend/src/lib/transaction-aware-execute.ts
  • apps/backend/src/routes/api-v1/integrations.http.ts
  • apps/backend/src/routes/http-success-encoding.test.ts
  • apps/backend/src/rpc/handlers/bots.ts
  • apps/backend/src/rpc/handlers/channel-sections.ts
  • apps/backend/src/rpc/handlers/channels.ts
  • apps/backend/src/rpc/handlers/connect-shares.test.ts
  • apps/backend/src/rpc/handlers/connect-shares.ts
  • apps/backend/src/rpc/handlers/integration-requests.ts
  • apps/backend/src/rpc/handlers/organizations.ts
  • apps/backend/src/scripts/migrate-to-clerk/backfill-external-ids.ts
  • apps/backend/src/scripts/migrate-to-clerk/diagnose-org-duplicates.ts
  • apps/backend/src/scripts/migrate-to-clerk/merge-org-duplicates.ts
  • apps/backend/src/services/channel-access-sync.ts
  • apps/backend/src/services/chat-sync/chat-sync-core-worker.integration.test.ts
  • apps/backend/src/services/chat-sync/chat-sync-core-worker.ts
  • apps/backend/src/services/chat-sync/discord-sync-worker.test.ts
  • apps/backend/src/services/connect-conversation-service.test.ts
  • apps/backend/src/services/connect-conversation-service.ts
  • apps/backend/src/services/message-outbox-dispatcher.test.ts
  • apps/backend/src/services/message-side-effect-service.test.ts
  • apps/backend/src/services/message-side-effect-service.ts
  • apps/backend/src/test/chat-sync-db-harness.ts
  • apps/backend/src/test/message-outbox-repo.test.ts
  • apps/backend/src/worker/platform.ts
  • apps/bot-gateway/src/gateway/auth.ts
  • apps/cluster/package.json
  • apps/cluster/src/cron/presence-cleanup-cron.ts
  • apps/cluster/src/cron/rss-poll-cron.ts
  • apps/cluster/src/cron/status-expiration-cron.ts
  • apps/cluster/src/cron/typing-indicator-cleanup-cron.ts
  • apps/cluster/src/cron/upload-cleanup-cron.ts
  • apps/cluster/src/index.ts
  • apps/cluster/src/services/bot-user-service.ts
  • apps/cluster/src/workflows/cleanup-uploads-handler.ts
  • apps/cluster/src/workflows/github-installation-handler.ts
  • apps/cluster/src/workflows/github-webhook-handler.ts
  • apps/cluster/src/workflows/message-notification-handler.ts
  • apps/cluster/src/workflows/rss-feed-poll-handler.ts
  • apps/cluster/src/workflows/thread-naming-handler.ts
  • apps/electric-proxy/package.json
  • apps/electric-proxy/src/auth/bot-auth.ts
  • apps/electric-proxy/src/cache/access-context-service.ts
  • apps/electric-proxy/src/tables/bot-tables.ts
  • apps/electric-proxy/src/tables/user-tables.ts
  • apps/electric-proxy/src/tables/where-clause-builder.test.ts
  • apps/electric-proxy/src/tables/where-clause-builder.ts
  • apps/electric-proxy/src/worker.ts
  • bots/linear-bot/package.json
  • infra/README.md
  • libs/bot-sdk/tsdown.config.ts
  • packages/auth/src/consumers/proxy-auth.ts
  • packages/backend-core/package.json
  • packages/backend-core/src/repositories/bot-command-repo.ts
  • packages/backend-core/src/repositories/bot-installation-repo.ts
  • packages/backend-core/src/repositories/bot-repo.ts
  • packages/backend-core/src/repositories/channel-member-repo.ts
  • packages/backend-core/src/repositories/channel-repo.ts
  • packages/backend-core/src/repositories/channel-webhook-repo.ts
  • packages/backend-core/src/repositories/chat-sync-channel-link-repo.ts
  • packages/backend-core/src/repositories/chat-sync-connection-repo.ts
  • packages/backend-core/src/repositories/chat-sync-event-receipt-repo.ts
  • packages/backend-core/src/repositories/chat-sync-message-link-repo.ts
  • packages/backend-core/src/repositories/connect-conversation-channel-repo.ts
  • packages/backend-core/src/repositories/connect-conversation-repo.ts
  • packages/backend-core/src/repositories/connect-invite-repo.ts
  • packages/backend-core/src/repositories/connect-participant-repo.ts
  • packages/backend-core/src/repositories/custom-emoji-repo.ts
  • packages/backend-core/src/repositories/github-subscription-repo.ts
  • packages/backend-core/src/repositories/integration-connection-repo.ts
  • packages/backend-core/src/repositories/integration-token-repo.ts
  • packages/backend-core/src/repositories/message-outbox-repo.ts
  • packages/backend-core/src/repositories/message-reaction-repo.ts
  • packages/backend-core/src/repositories/message-repo.ts
  • packages/backend-core/src/repositories/notification-repo.ts
  • packages/backend-core/src/repositories/organization-member-repo.ts
  • packages/backend-core/src/repositories/organization-repo.ts
  • packages/backend-core/src/repositories/rss-subscription-repo.ts
  • packages/backend-core/src/repositories/typing-indicator-repo.ts
  • packages/backend-core/src/repositories/user-presence-status-repo.ts
  • packages/backend-core/src/repositories/user-repo.ts
  • packages/db/drizzle.config.ts
  • packages/db/effect-orm.config.ts
  • packages/db/migrations/20261012000000_initial_schema/migration.sql
  • packages/db/migrations/20261012222436_effect_orm_baseline/migration.sql
  • packages/db/migrations/20261012222436_effect_orm_baseline/snapshot.json
  • packages/db/package.json
  • packages/db/scripts/migrate.ts
  • packages/db/scripts/render-ddl.ts
  • packages/db/src/index.ts
  • packages/db/src/migrate.ts
  • packages/db/src/schema/attachments.ts
  • packages/db/src/schema/bot-commands.ts
  • packages/db/src/schema/bot-installations.ts
  • packages/db/src/schema/bots.ts
  • packages/db/src/schema/channel-access.ts
  • packages/db/src/schema/channel-sections.ts
  • packages/db/src/schema/channel-webhooks.ts
  • packages/db/src/schema/channels.ts
  • packages/db/src/schema/chat-sync.ts
  • packages/db/src/schema/connect.ts
  • packages/db/src/schema/custom-emojis.ts
  • packages/db/src/schema/github-subscriptions.ts
  • packages/db/src/schema/integration-connections.ts
  • packages/db/src/schema/integration-requests.ts
  • packages/db/src/schema/integration-tokens.ts
  • packages/db/src/schema/message-integration-links.ts
  • packages/db/src/schema/message-outbox.ts
  • packages/db/src/schema/messages.ts
  • packages/db/src/schema/notifications.ts
  • packages/db/src/schema/organizations.ts
  • packages/db/src/schema/pinned-messages.ts
  • packages/db/src/schema/relations.ts
  • packages/db/src/schema/rss-subscriptions.ts
  • packages/db/src/schema/typing-indicators.ts
  • packages/db/src/schema/user-presence-status.ts
  • packages/db/src/schema/users.ts
  • packages/db/src/services/database.test.ts
  • packages/db/src/services/database.ts
  • packages/db/src/services/drizzle-effect.ts
  • packages/db/src/services/index.ts
  • packages/db/src/services/repository.ts
  • packages/db/src/tables/attachments.ts
  • packages/db/src/tables/bot-commands.ts
  • packages/db/src/tables/bot-installations.ts
  • packages/db/src/tables/bots.ts
  • packages/db/src/tables/channel-access.ts
  • packages/db/src/tables/channel-sections.ts
  • packages/db/src/tables/channel-webhooks.ts
  • packages/db/src/tables/channels.ts
  • packages/db/src/tables/chat-sync.ts
  • packages/db/src/tables/columns.ts
  • packages/db/src/tables/connect.ts
  • packages/db/src/tables/custom-emojis.ts
  • packages/db/src/tables/github-subscriptions.ts
  • packages/db/src/tables/index.ts
  • packages/db/src/tables/integration-connections.ts
  • packages/db/src/tables/integration-requests.ts
  • packages/db/src/tables/integration-tokens.ts
  • packages/db/src/tables/message-outbox.ts
  • packages/db/src/tables/messages.ts
  • packages/db/src/tables/notifications.ts
  • packages/db/src/tables/organizations.ts
  • packages/db/src/tables/pinned-messages.ts
  • packages/db/src/tables/rss-subscriptions.ts
  • packages/db/src/tables/typing-indicators.ts
  • packages/db/src/tables/user-presence-status.ts
  • packages/db/src/tables/users.ts
  • packages/domain/package.json
  • packages/infra/src/cloudflare/hazel-db.ts
  • packages/setup/src/commands/bots.ts
  • packages/setup/src/commands/setup.ts
  • turbo.json

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

⚠️ Coverage report too large to inline, see the workflow summary.

@Makisuo Makisuo mentioned this pull request Oct 12, 2026
@Makisuo
Makisuo added this pull request to stack #332 October 12, 2026 23:31
@Makisuo
Makisuo merged commit e9517ab into main Oct 12, 2026
4 of 7 checks passed
@Makisuo
Makisuo deleted the t3/adopt-effect-orm branch October 12, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant