docs(ops): production runs on Supabase

[sshlg]

State of the migration

Schema is on Supabase — 65 tables, 200 indexes, 91 foreign keys, matching the source exactly (compared, not assumed). 356 statements applied, 0 failures.

Data is not, and that is the only thing left. It needs SUPABASE_DB_PASSWORD, which is set at project creation and which the Management API will not return — the one step no automation here can take.

The hardening went on before the schema, and that order is the point

Supabase serves public through the Data API, and ALTER DEFAULT PRIVILEGES grants anon and authenticated ALL privileges on everything created there:

schema=public  objtype=r
acl: anon=arwdDxtm/postgres
     authenticated=arwdDxtm/postgres

The anon key is designed to be public — it ships inside frontends. Restoring 65 tables into the default configuration would have made users, connections (Fernet-encrypted credentials), ssh_keys, mcp_api_keys and audit_logs readable and writable by anyone holding it, the moment the restore finished.

Three defences, each verified rather than assumed:

1. Data API no longer exposes public. Verified from outside with the real anon key against a canary row — 404 with the lock on, and HTTP 200 with the canary leaking when deliberately unlocked. The check is known to be able to fail.
2. Default privileges revoked from anon, authenticated, service_role — including PG17's MAINTAIN.
3. RLS on all 65 tables, every grant to public roles revoked. Measured: 0 tables without RLS, 0 grants. The application is unaffected because it connects as postgres, which has BYPASSRLS; anon and authenticated do not.

What the script does

scripts/migrate_to_supabase.sh dry-runs by default. It checks both ends, compares the schema shape, re-checks the hardening and refuses to copy if any of it has come undone, then copies, diffs row counts table by table, and fixes sequences.

That last one is not housekeeping: a restored table with its sequence left at 1 accepts exactly one insert and then fails on the primary key — the classic way a migration looks fine for an hour.

Two facts it carries because they are not guessable

• db.<ref>.supabase.co has an AAAA record and no A record, and Heroku dynos have no outbound IPv6. The pooler is the only route, not a preference.
• Port 5432 (session), not 6543 (transaction), because transaction mode breaks the named prepared statements SQLAlchemy's asyncpg dialect uses by default — and breaks them intermittently. At 9 of 20 connections in use the throughput argument for transaction mode is theoretical; the ways to be subtly wrong are not.

CB-KNOW2 struck, on evidence

before restart   34 038 vectors, visible to a fresh one-off dyno
restart both dynos
after restart    34 038 vectors — handle.count() = 34 038
repeated across a real deploy (v286), not only ps:restart

The rebuild reached pipeline_end, the checkpoint was deleted (which only happens on success), and the chained sync ran unprompted. The cost is stated too: the rebuild went 12 039 s → 15 051 s before the write batch was split from the embed batch in #243, and the ceiling followed the measurement.

Verification

pytest tests/unit/docs/ — 63 passed, including the ratchets over this board's own arithmetic.

[sshlg]

Done — the migration ran on 2026-08-29. This PR now records a completed move rather than a prepared one.

What was copied and how it was checked

247 MB, all 65 tables matching row for row, sequences advanced.

The row counts were the least interesting check. Two things could have failed silently:

The Fernet ciphertext. md5 over connections (db_password_encrypted, connection_string_encrypted, mcp_env_encrypted):

heroku    72ed7641a7c9b1710b568b57c9d755af
supabase  72ed7641a7c9b1710b568b57c9d755af

and over ssh_keys, 242c765decf6655cf25dda54175c1580 on both. Byte for byte, so every stored credential still decrypts.

asyncpg through the Supavisor session pooler — the thing transaction mode would have broken intermittently. Verified on a live dyno: alembic ran its DDL, a row was inserted, read back and deleted, and 20 parameterised round-trips succeeded.

Post-switch state from the worker:

app database host : aws-1-eu-west-1.pooler.supabase.com:5432
vectors visible   : 34 038      force_full? no
alembic (asyncpg) : 1d72054cd637
users / messages  : 9 / 131

Two things the migration hit that are not guessable

• heroku config:set DATABASE_URL is refused outright — "Cannot overwrite attachment values DATABASE_URL" — and detaching the add-on is refused too, because it is "the last attachment to billing app". The route is: attach the same add-on under a second name first, then detach DATABASE. That also leaves the old database reachable as HEROKU_PG_ROLLBACK_URL, which is a better rollback than a URL saved in a file.
• pg_dump --disable-triggers needs superuser, and Supabase's postgres is not one (rolsuper=false; only supabase_admin is). It failed with permission denied: "RI_ConstraintTrigger_a_18816" is a system trigger and rolled back to zero rows, which is what --single-transaction is there for. session_replication_role = replica defers the same checks and is available — set inside the copy transaction so it cannot outlive a failure.

A bug the switch created in this very script, found and fixed

Its source was DATABASE_URL. That variable now points at the target, so a re-run would have copied Supabase onto itself while every check passed — schema matching itself, row counts matching themselves. It reads the old attachment instead and refuses if both ends resolve to one host.

Downtime

~20 s, between detaching DATABASE and setting the new value, when DATABASE_URL was absent and config fell back to the SQLite default. The pgvector guard from #241 caught exactly that and refused to boot rather than failing later on a missing table.

Rollback

HEROKU_PG_ROLLBACK_URL still points at the Heroku database, which was read, never written. Two commands, recorded in the script. Anything written since the switch lives only on Supabase.