feat(mcp): write tools — compose, undo-send, mailbox ops, triage, settings

.gitignore

@@ -31,3 +31,5 @@ Thumbs.db
 
 .superpowers/
 output/
+
+# Search-eval query cache — paraphrases of real mail; the harness regenerates it

README.md

@@ -81,6 +81,172 @@ configurable per account, and accounts with GTD off behave exactly as before.
 
 ---
 
+## MCP server
+
+MailFlow exposes a Model Context Protocol endpoint at `POST /mcp`. Authenticate
+with an API token in the `Authorization: Bearer <token>` header. Tokens have one or
+more scopes:
+
+- `read` — search, inspect messages, and list mailbox state
+- `write` — change mailbox state; also implies `read`
+- `send` — send, queue, cancel, or recall mail; also implies `read`
+- `settings` — manage accounts, aliases, and rules; also implies `read`
+
+`stage_deletion` requires `write`, even though execution is a separate,
+session-authorized step. `run_rules` requires both `settings` and `write` because
+rule actions can move, archive, or delete messages.
+
+### Tools
+
+#### Search and read
+
+| Tool | Scope | Description |
+|---|---|---|
+| `ping` | `read` | Check transport and authentication health. |
+| `search_metadata` | `read` | Search message metadata with the supported Gmail-style query subset. |
+| `search_message_bodies` | `read` | Run keyword full-text search over message bodies. |
+| `semantic_search_messages` | `read` | Run vector or hybrid semantic search over indexed messages. |
+| `get_message` | `read` | Get message details and a pageable body slice. |
+| `list_messages` | `read` | List messages with account, participant, date, attachment, and conversation filters. |
+| `get_stats` | `read` | Get archive totals, attachment statistics, and accounts. |
+| `aggregate` | `read` | Group message statistics by sender, recipient, domain, label, or year. |
+| `search_by_domains` | `read` | Find messages involving any of a set of domains. |
+| `find_similar_messages` | `read` | Find messages closest to a seed message in the vector index. |
+| `search_in_message` | `read` | Find keyword or semantic matches inside one message. |
+
+#### Drafts
+
+| Tool | Scope | Description |
+|---|---|---|
+| `create_draft` | `write` | Create a draft without sending it. |
+| `update_draft` | `write` | Replace a draft and return its new IMAP UID. |
+| `list_drafts` | `read` | List drafts newest-first. |
+| `get_draft` | `read` | Get draft bodies, threading headers, and attachment metadata. |
+| `delete_draft` | `write` | Permanently delete a draft from IMAP. |
+
+#### Send, outbox, and undo-send
+
+| Tool | Scope | Description |
+|---|---|---|
+| `send_email` | `send` | Send immediately or queue within an undo-send window. |
+| `send_draft` | `send` | Send a stored draft and delete it only after delivery or enqueue succeeds. |
+| `reply_email` | `send` | Reply to a message sender. |
+| `reply_all_email` | `send` | Reply to the sender and stored To/Cc recipients. |
+| `forward_email` | `send` | Forward a message, including server-side attachment references. |
+| `unsend_email` | `send` | Cancel a queued message before its `send_at` time. |
+| `list_outbox` | `read` | List messages still waiting in the undo-send outbox. |
+| `recall_email` | `send` | Cancel a queued send, or clean up a Sent copy and prepare a follow-up draft. |
+
+#### Mailbox and folders
+
+| Tool | Scope | Description |
+|---|---|---|
+| `list_folders` | `read` | List scoped folders and live message counts. |
+| `create_folder` | `write` | Create a folder in one account. |
+| `rename_folder` | `write` | Rename the final component of a folder path. |
+| `delete_folder` | `write` | Delete a folder after confirming its live message count. |
+| `move_messages` | `write` | Move messages and return replacement IDs. |
+| `archive_messages` | `write` | Archive messages and return replacement IDs. |
+| `trash_messages` | `write` | Move messages to Trash without silently expunging existing Trash items. |
+| `mark_read` / `mark_unread` | `write` | Change read state for explicit message IDs. |
+| `star_message` / `unstar_message` | `write` | Change starred state for explicit message IDs. |
+| `mark_spam` / `mark_not_spam` | `write` | Move a message into or out of the configured spam folder. |
+| `snooze_message` / `unsnooze_message` | `write` | Snooze or restore a message and its reply-chain siblings. |
+| `set_category` | `write` | Set the category of one message. |
+| `gtd_classify` | `write` | Apply or remove one GTD state label. |
+| `gtd_done` | `write` | Remove GTD labels and archive the Inbox copy. |
+| `stage_deletion` | `write` | Stage a filtered deletion for separate session-authorized execution. |
+
+#### Triage
+
+| Tool | Scope | Description |
+|---|---|---|
+| `triage_inbox` | `read` | Page through untriaged Inbox messages with sender, thread, category, and optional semantic signals. |
+| `get_triage_context` | `read` | Get thread, sender-history, similar-message, and matched-rule context. |
+| `mark_triaged` | `write` | Checkpoint messages so later triage runs skip them. |
+
+#### Accounts and settings
+
+| Tool | Scope | Description |
+|---|---|---|
+| `list_accounts` | `read` | List connected accounts and aliases without credentials. |
+| `add_account` | `settings` | Stage non-secret account configuration and return a `stage_id`. |
+| `update_account_settings` | `settings` | Update non-secret account settings. |
+| `test_account_connection` | `settings` | Test stored IMAP and SMTP credentials without sending mail. |
+| `create_alias` / `update_alias` / `delete_alias` | `settings` | Manage send-as aliases on scoped accounts. |
+| `list_rules` | `settings` | List global and account-specific inbox rules. |
+| `create_rule` / `update_rule` / `delete_rule` | `settings` | Manage user-owned inbox rules. |
+| `run_rules` | `settings` + `write` | Run enabled rules against current Inbox messages. |
+
+### Undo-send and recall
+
+`send_email` and `send_draft` accept an undo window from 0 to 120 seconds. A
+zero-second window hands the message to SMTP immediately. A positive window puts
+it in the outbox until `send_at`; use `list_outbox` to inspect queued messages and
+`unsend_email` to cancel one before handoff.
+
+Recall is deliberately honest. `recall_email` can withdraw a message only while
+it is still queued. SMTP cannot claw back mail already delivered; for an
+already-sent message, recall can delete MailFlow's Sent copy and prepare a
+"please disregard" follow-up draft, but it never pretends recipients lost their
+copy and never sends the follow-up automatically.
+
+### Adding an account
+
+`add_account` accepts non-secret connection configuration only. It returns a
+`stage_id`; a human then supplies fresh credentials in **Settings → Accounts**,
+or through the session-authenticated
+`POST /api/mcp-account-stages/:id/execute` endpoint.
+
+Passwords, OAuth access tokens, and OAuth refresh tokens never cross the MCP
+bearer-token channel.
+
+### Agent cookbook
+
+A morning triage loop can page oldest-first and checkpoint every disposition:
+
+```text
+07:00  triage_inbox { limit: 25, unread_only: true }
+       → items, cursor, has_more, counts.untriaged_unread
+
+       Obvious newsletters/promotions:
+       archive_messages { message_ids: ["m1", "m2", ...] }
+       → archived: [{ id: "m1", new_id: "n1", ... }, ...]
+       mark_triaged {
+         message_ids: ["n1", "n2", ...],
+         action: "archived"
+       }
+
+       Ambiguous message with a known sender and active thread:
+       get_triage_context { message_id: "m17" }
+       gtd_classify { message_id: "m17", state: "todo" }
+       star_message { message_ids: ["m17"] }
+       mark_triaged {
+         message_ids: ["m17"],
+         action: "flagged_todo",
+         note: "Needs a reply"
+       }
+
+       Snooze a reply chain:
+       mark_triaged { message_ids: ["m22"], action: "snoozed" }
+       snooze_message {
+         message_id: "m22",
+         until: "2026-07-31T08:00:00Z"
+       }
+       → { ok: true, moved_count: 3, sibling_ids: [...], folder: "Snoozed" }
+
+07:04  triage_inbox { cursor: "<previous cursor>", limit: 25 }
+       → repeat until has_more is false
+```
+
+**Mark before move:** call `mark_triaged` before archiving, moving, or snoozing,
+or use each successful move/archive receipt's `new_id`. Message IDs change during
+a move; a stale pre-move ID resolves to `skipped`. Re-running `triage_inbox`
+without a cursor is safe: the checkpoint table, not the cursor, is what prevents
+already-triaged messages from being offered again.
+
+---
+
 ## Screenshots
 
 <table>
@@ -186,6 +352,14 @@ docker compose up -d
 
 To pin to a specific version instead of `latest`, add `MAILFLOW_VERSION=1.9.0` to your `.env`.
 
+> **Upgrading an existing install across the Postgres image change** (`postgres:16-alpine` → `pgvector/pgvector:pg16`): the new image uses a different C library (musl → glibc), which changes text collation order. Reindex once after the switch or text indexes can silently return wrong results — the backend also prints this warning at startup when it detects the mismatch:
+>
+> ```bash
+> docker compose exec postgres psql -U mailflow -d mailflow \
+>   -c 'REINDEX DATABASE "mailflow";' \
+>   -c 'ALTER DATABASE "mailflow" REFRESH COLLATION VERSION;'
+> ```
+
 ---
 
 ## Option B — Build from source

backend/migrations/0035_search_fts.sql

@@ -0,0 +1,55 @@
+-- Weighted full-text search column (search_fts) + version stamp, kept fresh by
+-- a BEFORE trigger. Fast, metadata-only DDL only (README D1): the nullable
+-- columns force no table rewrite on PG16, and the function/trigger are instant.
+-- Pre-existing rows are populated by the resumable drainer (ftsBackfill.js);
+-- the GIN index is built CONCURRENTLY in the separate no-transaction migration
+-- 0037 (a $$-quoted plpgsql body cannot survive the no-transaction ; splitter,
+-- so the trigger and the CONCURRENTLY index MUST live in different files).
+--
+-- The setweight(...) expression MUST stay identical to
+-- lexicalRepo.searchFtsExpr('NEW'); fts_version = 1 matches lexicalRepo.FTS_VERSION.
+-- Idempotent (IF NOT EXISTS / CREATE OR REPLACE / DROP IF EXISTS) because a
+-- crash before the schema_migrations INSERT retries this migration.
+
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS search_fts tsvector;
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS fts_version int;
+
+CREATE OR REPLACE FUNCTION messages_search_fts_refresh() RETURNS trigger AS $$
+BEGIN
+  -- Skip recompute on an UPDATE that changes none of the indexed source
+  -- columns (read/star flag flips, snippet-only writes), so the trigger does
+  -- not tax hot sync UPSERTs; this also lets the backfill's explicit SET win.
+  IF TG_OP = 'UPDATE'
+     AND NEW.subject      IS NOT DISTINCT FROM OLD.subject
+     AND NEW.from_name    IS NOT DISTINCT FROM OLD.from_name
+     AND NEW.from_email   IS NOT DISTINCT FROM OLD.from_email
+     AND NEW.to_addresses IS NOT DISTINCT FROM OLD.to_addresses
+     AND NEW.cc_addresses IS NOT DISTINCT FROM OLD.cc_addresses
+     AND NEW.body_text    IS NOT DISTINCT FROM OLD.body_text
+  THEN
+     RETURN NEW;
+  END IF;
+
+  BEGIN
+    NEW.search_fts := setweight(to_tsvector('english', coalesce(NEW.subject,'')), 'A') ||
+    setweight(to_tsvector('english', coalesce(NEW.from_name,'') || ' ' || coalesce(NEW.from_email,'')), 'B') ||
+    setweight(to_tsvector('english', coalesce(NEW.to_addresses::text,'') || ' ' || coalesce(NEW.cc_addresses::text,'')), 'C') ||
+    setweight(to_tsvector('english', LEFT(coalesce(NEW.body_text,''), 600000)), 'D');
+    NEW.fts_version := 1;
+  EXCEPTION WHEN program_limit_exceeded THEN
+    -- Even with the 600k LEFT cap, a pathologically dense/multibyte body can
+    -- exceed Postgres's ~1MB tsvector limit (SQLSTATE 54000). Never fail the
+    -- row write: leave search_fts NULL so the message still persists and stays
+    -- findable via the ILIKE fallback; the backfill's row-by-row skip stamps it.
+    NEW.search_fts := NULL;
+    NEW.fts_version := NULL;
+  END;
+
+  RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_messages_search_fts ON messages;
+CREATE TRIGGER trg_messages_search_fts
+  BEFORE INSERT OR UPDATE ON messages
+  FOR EACH ROW EXECUTE FUNCTION messages_search_fts_refresh();

backend/migrations/0036_background_jobs.sql

@@ -0,0 +1,19 @@
+-- Generic progress substrate for observable background drainers (first consumer:
+-- the FTS backfill). Plain table, fast DDL. One row per (kind, account); global
+-- jobs use a NULL account_id, COALESCE'd to '' in the unique index so the upsert
+-- has a single conflict target for both cases.
+
+CREATE TABLE IF NOT EXISTS background_jobs (
+  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  kind        VARCHAR(64) NOT NULL,
+  account_id  UUID REFERENCES email_accounts(id) ON DELETE CASCADE,
+  state       VARCHAR(20) NOT NULL DEFAULT 'idle',
+  processed   BIGINT NOT NULL DEFAULT 0,
+  total       BIGINT NOT NULL DEFAULT 0,
+  last_error  TEXT,
+  started_at  TIMESTAMPTZ,
+  updated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS ux_background_jobs_kind_account
+  ON background_jobs (kind, COALESCE(account_id::text, ''));

backend/migrations/0037_search_fts_index.sql

@@ -0,0 +1,24 @@
+-- no-transaction
+--
+-- GIN index that serves `search_fts @@ tsquery`, plus a partial btree that lets
+-- the backfill drainer (and any "needs backfill" probe) find not-yet-stamped
+-- rows without a full seq scan; it self-prunes as fts_version = 1 fills in.
+-- CONCURRENTLY must run outside a transaction, so these live apart from 0035's
+-- trigger. Idempotent via IF NOT EXISTS (retried if a crash precedes the
+-- schema_migrations INSERT).
+--
+-- DROP ... IF EXISTS before each CREATE: a cancelled or crashed CREATE INDEX
+-- CONCURRENTLY leaves an INVALID index under the target name. On retry, plain
+-- IF NOT EXISTS sees that name and silently skips the create, recording the
+-- migration as done while the scan stays unindexed forever. This file only re-runs
+-- after such a failure — in which case the index is either absent (drop is a no-op)
+-- or invalid (drop removes the dead stub) — so dropping first is safe and cheap.
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_search_fts;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_search_fts
+  ON messages USING GIN (search_fts);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_fts_stale_v1;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_fts_stale_v1
+  ON messages (date DESC)
+  WHERE fts_version IS DISTINCT FROM 1;

backend/migrations/0038_embed_watermark.sql

@@ -0,0 +1,44 @@
+-- Vector-substrate CAS columns + last_modified trigger (slice 04).
+-- Extension-independent, fast DDL only. NO vector-typed DDL here — the
+-- embeddings/index_generations/embed_watermark/embed_runs tables and the HNSW
+-- index are created by ensureVectorSchema() at startup (README invariant).
+-- Transactional migration (NOT -- no-transaction): the no-transaction runner
+-- splits on ';' and would break the dollar-quoted function below.
+
+-- last_modified: content-change CAS token the embed worker compares to detect a
+-- late-arriving body invalidating a stale subject-only embedding. NOT NULL DEFAULT
+-- now() is metadata-only on PG16 (now() is STABLE → one stored missing-value, no
+-- table rewrite).
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS last_modified TIMESTAMPTZ NOT NULL DEFAULT now();
+
+-- embed_gen: the index generation this row is embedded under. NULL = needs embedding.
+-- Plain BIGINT soft-stamp (no FK): a generation can be retired/deleted while stamps linger.
+ALTER TABLE messages ADD COLUMN IF NOT EXISTS embed_gen BIGINT;
+
+-- Bump last_modified AND clear embed_gen when an embedding-input column changes. The
+-- WHEN clause filters at the C level so unchanged-content UPDATEs (the hot re-sync path)
+-- and stamp-only UPDATEs (the worker setting embed_gen) skip the function entirely.
+-- Clearing embed_gen is what re-surfaces a late-arriving body: a row embedded
+-- subject-only and stamped, whose body later lands (phase-2 drainer / on-open fetch),
+-- has its stamp cleared here so the NULL-only embed scan re-finds it and the idempotent
+-- upsert replaces the stale chunks. The CAS only guards the read→stamp window; this
+-- trigger covers the post-stamp case (Mailflow's late-arriving bodies — msgvault's rows
+-- are immutable after ingest, so it never needed this). Together with createGeneration's
+-- stamp-reset (which handles generation rebuilds: unchanged content, new fingerprint),
+-- the invariant is exact: embed_gen IS NULL ⟺ the row needs embedding.
+CREATE OR REPLACE FUNCTION messages_bump_last_modified() RETURNS trigger AS $$
+BEGIN
+  NEW.last_modified := now();
+  NEW.embed_gen := NULL;
+  RETURN NEW;
+END;
+$$ LANGUAGE plpgsql;
+
+DROP TRIGGER IF EXISTS trg_messages_last_modified ON messages;
+CREATE TRIGGER trg_messages_last_modified
+  BEFORE UPDATE ON messages
+  FOR EACH ROW
+  WHEN (NEW.subject IS DISTINCT FROM OLD.subject
+        OR NEW.body_text IS DISTINCT FROM OLD.body_text
+        OR NEW.body_html IS DISTINCT FROM OLD.body_html)
+  EXECUTE FUNCTION messages_bump_last_modified();

backend/migrations/0039_embed_pending_index.sql

@@ -0,0 +1,17 @@
+-- no-transaction
+-- Partial index over the embed-scan's steady-state hot predicate. Once a generation
+-- reaches full coverage, the only live rows still needing work are newly-arrived
+-- messages with embed_gen IS NULL, so the 60s scheduler scan (scanForEmbedding) should
+-- touch O(pending), not O(mailbox). Built CONCURRENTLY (hence -- no-transaction) so it
+-- never blocks boot on a large messages table; extension-independent and cheap to
+-- maintain (only the sparse NULL set is indexed). Idempotent (IF NOT EXISTS) because a
+-- crash before the schema_migrations INSERT retries the migration.
+--
+-- DROP ... IF EXISTS before the CREATE: a cancelled or crashed CREATE INDEX
+-- CONCURRENTLY leaves an INVALID index under this name, and plain IF NOT EXISTS would
+-- then silently skip the create on retry — recording the migration as done while the
+-- embed scan stays unindexed forever. This file only re-runs after such a failure, so
+-- the index is either absent (drop is a no-op) or invalid (drop clears the dead stub).
+DROP INDEX CONCURRENTLY IF EXISTS idx_messages_embed_pending;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_messages_embed_pending
+  ON messages (id) WHERE embed_gen IS NULL;

backend/migrations/0040_api_tokens.sql

@@ -0,0 +1,12 @@
+-- MCP API tokens. We store only a SHA-256 hash of the token; the plaintext is
+-- shown to the operator exactly once at mint time and is never recoverable.
+CREATE TABLE IF NOT EXISTS api_tokens (
+  id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  user_id      UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  token_hash   TEXT NOT NULL UNIQUE,
+  name         TEXT NOT NULL,
+  created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  last_used_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_api_tokens_user_id ON api_tokens(user_id);

backend/migrations/0041_mcp_deletion_batches.sql

@@ -0,0 +1,21 @@
+-- Staged (not executed) MCP deletions. stage_deletion records a batch; a separate
+-- session-authenticated execute step flips messages.is_deleted (soft delete). No
+-- tool ever hard-deletes. Renumbered to 0041 (0040 is api_tokens) per the README
+-- migration-numbering rule.
+CREATE TABLE IF NOT EXISTS mcp_deletion_batches (
+  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  user_id       UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+  description   TEXT NOT NULL DEFAULT '',
+  status        TEXT NOT NULL DEFAULT 'staged',
+  message_count INTEGER NOT NULL DEFAULT 0,
+  created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+  executed_at   TIMESTAMPTZ
+);
+
+CREATE TABLE IF NOT EXISTS mcp_deletion_batch_messages (
+  batch_id   UUID NOT NULL REFERENCES mcp_deletion_batches(id) ON DELETE CASCADE,
+  message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
+  PRIMARY KEY (batch_id, message_id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_mcp_deletion_batches_user_id ON mcp_deletion_batches(user_id);