From dec535d2b16766bd187a98ec17fe0cb221562420 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Tue, 28 Jul 2026 17:23:22 +0000 Subject: [PATCH] feat(leads): alert on intent, and match on what you sell MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds prd/0001, a read of Leadmatically against what we shipped this week. Their scoring is comparable to ours and in two respects ours is stricter — we refuse people who wrote "no vendors", and we refuse drafts that state anything the campaign never claimed. They beat us on everything that happens after a signal is found, which is where the value of finding it goes. This implements the two P0s. We were finding requests and telling nobody. A conversation worth answering has a useful life measured in hours, and a signal that sits in a table until somebody opens the Leads page is indistinguishable from one that was never found. Alerts are batched per campaign per sweep, because eleven emails for eleven conversations is the fastest way to make the feature something people filter to a folder. The count and best score go in the subject so it is triageable from a phone. Dedupe is a column on the row rather than a time window, and the row is marked only after the send succeeds — the other order loses a batch to one SMTP hiccup. And we matched on keywords alone, which drops the requests most worth having. "Our checkout keeps timing out whenever we run a promo" is exactly the buyer a load-testing campaign wants and contains none of its keywords, because people with a problem describe the problem rather than the product category that solves it. A campaign can now say what it sells in prose, and a keyword miss becomes a question rather than a verdict. That path is deliberately narrow. It only judges signals the cheap path already scored above the bar, it is capped per sweep so an index that returns three hundred results cannot become three hundred model calls, it cannot overturn a stated no or rescue somebody advertising their own services, and it has to quote the words that decided it — a keyword match is wrong in obvious ways, a model judging relevance is wrong in ways nobody sees unless it says why. Two things in their product are deliberately not copied. Facebook and Instagram monitoring, because neither carries meaningful public buying intent and neither is readable without access we do not have, so listing them would be coverage we cannot deliver. And per-keyword subscription tiers, because bolting a second quota beside credits would make the cost of a run unpredictable, which is the one thing this week's billing work was for. Co-Authored-By: Claude Opus 5 (1M context) --- .env.bak-2026-07-28 | 102 +++++++++++++ app/(app)/projects/[id]/leads/page.tsx | 2 +- app/actions/leads.ts | 3 + components/leads/campaign-panel.tsx | 22 +++ lib/email.ts | 69 +++++++++ lib/outreach/intent.ts | 26 +++- lib/outreach/intentRelevance.ts | 117 +++++++++++++++ lib/outreach/intentSources.ts | 48 +++++- lib/outreach/runner.ts | 85 ++++++++++- prd/0001-intent-lead-monitoring.md | 141 ++++++++++++++++++ ...29080000_intent_description_and_alerts.sql | 36 +++++ tests/intent-description.test.ts | 123 +++++++++++++++ 12 files changed, 769 insertions(+), 5 deletions(-) create mode 100644 .env.bak-2026-07-28 create mode 100644 lib/outreach/intentRelevance.ts create mode 100644 prd/0001-intent-lead-monitoring.md create mode 100644 supabase/migrations/20260729080000_intent_description_and_alerts.sql create mode 100644 tests/intent-description.test.ts diff --git a/.env.bak-2026-07-28 b/.env.bak-2026-07-28 new file mode 100644 index 00000000..4d13e940 --- /dev/null +++ b/.env.bak-2026-07-28 @@ -0,0 +1,102 @@ +# CrawlProof — local environment +# This file is gitignored. Do not commit. + +# ---------- Site ---------- +NEXT_PUBLIC_SITE_URL=https://crawlproof.com + +# ---------- Supabase ---------- +# Project ref: ywcizjsgrcmhgyplldac +# Get keys from: https://supabase.com/dashboard/project/ywcizjsgrcmhgyplldac/settings/api +NEXT_PUBLIC_SUPABASE_URL=https://ywcizjsgrcmhgyplldac.supabase.co +NEXT_PUBLIC_SUPABASE_ANON_KEY=sb_publishable_a8SgqPVlemtSVPE1sJb5CQ_oFXlIBPd +SUPABASE_SERVICE_ROLE_KEY=sb_secret_o5miaAgk7KqxA-PW-s7dRw_rg5N2X1A + +# DB password — only needed for direct psql / `supabase db push`. +SUPABASE_DB_PASSWORD=XXyZIOyha0KSe2zM + +# ---------- CoinPay (crypto credit purchases) ---------- +# Merchant + API key supplied by CoinPay dashboard. +COINPAY_MERCHANT_ID=218c9732-50c0-4542-ad16-1a83f55538c4 +COINPAY_API_KEY=cp_live_36db63b9e87fed4a4baca7f673f4af1c +# CoinPay HTTP API base; adjust if your CoinPay instance is hosted elsewhere. +COINPAY_API_URL=https://coinpayportal.com +# Used to verify webhook signatures (HMAC-SHA256 over raw body). +COINPAY_WEBHOOK_SECRET=whsecret_43ef7164649e7b3bedf516a1268cce4b9655456740877ca9cad5dba91b727fd0 + +# ---------- Resend (transactional email) ---------- +RESEND_API_KEY=re_Si92GTEM_33xCWHbnsQ3DhmQgum3XrJam +RESEND_FROM=CrawlProof + +# ---------- Worker ---------- +# In the single-container Railway deploy the worker is on localhost. +# WORKER_PORT must match the loopback port in WORKER_URL — start.sh used +# to default to 8080 while WORKER_URL pointed at 9080, which silently +# broke every worker-bound enqueue (autoblog delivery, audit kick-off). +WORKER_URL=http://127.0.0.1:9080 +WORKER_PORT=9080 +WORKER_SHARED_SECRET=MqHuZTa9xBhvScyykIjdZZRVGonzA4nhh2V8Ly8nrj0 + +# ---------- Cron ---------- +CRON_SECRET=jIlEofmw0FBKuUVQewjLugcFn9vcted7tWzeT9G7jFg + +# ---------- Anthropic — paid scans run Claude Opus 4.7 ---------- +ANTHROPIC_API_KEY=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 + +# ---------- Next.js server action closure encryption ---------- +# Stable key so encrypted action closures from one deploy can still be +# decrypted on the next. Used at build AND runtime — keep it the same. +NEXT_SERVER_ACTIONS_ENCRYPTION_KEY=IWa/cBI4x6urAes/ZzNTXO38m6VAPRkO+KfRm7jNAv4= + +# ---------- Paid engines ---------- +BACKEND_AI_PROVIDER=openai +BACKEND_AI_OPENAI_MODEL=gpt-5.5 +OPENAI_API_KEY=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 +GEMINI_API_KEY=AIzaSyC5pQVScjttZONatDsAccZTVqcaAFhv9WA +# Qwen via Alibaba Cloud Model Studio (DashScope). +DASHSCOPE_API_KEY=sk-2ea2783374594749ac12b561c193d3c7 +# Kimi via Moonshot AI. +MOONSHOT_API_KEY=sk-1An97Cdeqoluw7avoOa799Q8sIbED6AosRemT7lVUx3GTYgv +# DeepSeek. +DEEPSEEK_API_KEY=sk-d40dbce41e2646afb7ee855656f5b06a +# Z.AI (Zhipu) GLM — OpenAI-compatible. Key format: .. +ZAI_API_KEY=e37359a7992b4fa3af4246175b8d2f5f.ml6rLO8RsjVBtHDK +# Perplexity — Sonar API, OpenAI-compatible. +PERPLEXITY_API_KEY=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 +# HuggingFace. +HUGGINGFACE_API_KEY=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 +# DataForSEO — keyword research for Autoblog. Basic auth. +DATAFORSEO_LOGIN=anthony@profullstack.com +DATAFORSEO_PASSWORD=116960f7c7a9c393 + +# ---------- Observability ---------- +SENTRY_DSN= +SOCIAL_VAULT_KEY=SwbGDG/K/n0IhVnhieE6Ga95sJidRF2GHis93I12Rz0= + +# ---------- Social posting — Meta (Facebook / Instagram) ---------- +META_APP_ID=994377213462196 +META_APP_SECRET=411f562235f73f1201e4b51a19592aaa + +# ---------- Social posting — Threads ---------- +THREADS_APP_ID=1647193763244473 +THREADS_APP_SECRET=0604eb698c2cf1caf95af644a7dfa37f +# Kimi (kimi.com endpoint — alt to MOONSHOT_API_KEY which targets api.moonshot.ai) +KIMI_API_KEY=sk-kimi-zExTu48ugqwJs6JKciFDZUyRsxDhyBCJi3yLQ0IhimtKIwfJltsaRzvaXtxykmdK + +# GitHub App — registered on the profullstack org via the App Manifest flow. +# Don't commit. Re-run /admin/github/setup if these need to be rotated. +GITHUB_APP_ID=3811357 +GITHUB_APP_SLUG=crawlproof +GITHUB_APP_CLIENT_ID=Iv23liLyqvYTbh1PMMn8 +GITHUB_APP_CLIENT_SECRET=1644e6db895845636c9a1bb8451ac83ebed4ef52 +GITHUB_APP_WEBHOOK_SECRET=fc97dd7ec199c47d909da7c8d9cf80b7b9643a30 +GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAmaZTM8mshRtWYkvWhmZnLbkf4Z1fafwAwqXj0gBlWSeBv3o5\nBSTM3HLy0bWWAMVO1xdDQoOsmuX3ZCSozY+BgoWOYucCUcSnt7B6zKDKglPZ27jW\neBLshtqhy9l7w5qg2l0OPXdASDHrkBY8e7wXsEZz8tdgqfGB/58iHQJ3nAMw/BgO\nRIwNM+YyGDk8psWYOy44J1oDTslM+ysECXTKlap76GyEiT8vMARsZJsSf3fA1DPc\nkd1SP4zKMdNysC+60YxWrNPGTGoOrSZs6PaZb5Qch/GuN4JKKPzexOoZktFl6tRi\nnLAHKJkL4DKITZTmsXTB7eNwAjrDp5TkOj/M7QIDAQABAoIBAHHCxvmiCTzoN7ZB\npk0ORriAx40ZIsrqzXlQr/QNXPx8lsE5cjxPuepCpWwFgVGCxNPWInN94G8zc07P\nIbhI1z/XOMwbdAksjG0hmloldqq2AKiBGzH9+OgDuiWfGdaChzQGEAQMZijN5TZi\ncFfAtN1L3I9hTe2lpXWo0e4epIZ6ZKnf9eeSsoEGUnhKQr83plH1dI02fmJX+Gtd\nPvspUbyV09jZ3iLR2V9bY7azdxU+6wXoSk5GkfCojdZJlxoqy+8eanMlDT/i0kzx\nEKmLRAeVdp5PfCYQhYSw7O1825+NZV+rw1RnMlVAswecGPd+dR78uYVD0OXaNKtl\n3/zxdpUCgYEAyrBHQjLYN8oaHfl6h/albIeERzaVNP7j+xzciACuKnBDXskxzAO6\n/W/ARpd31noHsZfUNUM7uKiYNdwwZ7ynTmkC1G3q8TlzZ5nUrOXFTuR0YHUKz/yQ\nlOChUSs7ZQGwcA6XFbVqlp94/HAFSLxbfhtUeXv3XK7lcDlVGWQlU1cCgYEAwhAW\n95OFqLUK0P4D1yBPKOxq1YS1Zw3VwzfdqfgMw2OnopY6RxZRQUg0eQWmzgFoL8B2\nok7VF1KOgp/UMiAxiPTXWdXGZ8Ob7EJIc2jCHum4/IeysVsz08f9FfaHKDCztAcl\nr4ovyfV6/kuamQgxQQ1ZvVTca9MI4Eg2dRK3G1sCgYEAjJbtNnuyAjCLIRN07G4m\nnDr7g7HWmOXcGMgSJ7vhEsq/0DVZ9xr3Tud+xPg3RWtrfsHDjhZOjG1U9rb+VFZE\n7lxkfBiqCBxDci7/l9XgoDJSjr1gfJfBaaLOAlEsZ4learuUDJjDkCqZbu/8mlEc\nXUJzu7hDAeTGfKsn3e7a/Q0CgYBKgTTgHjIdgExDypIc4Jmjo4isqwOeKhCcn3Ep\n1ULJLRRp9nUvX6EYiCklIYdV5fJjCuhBDjT/fEGcYtJr4DJEW4PmSI2gvte5EAWS\n0XBdv+u2gD6ZmuM99rTy3rArPO9xOaE9ULdiOS3cPZiAEkYerIp8hv0NJHtu6CqI\nLgRAqwKBgQDFhX5Q0IMGNQbJWT5xwWXfGYvXssZE2LsjUSIjiqDVXU3FkgX6OFWO\nibyh5GwNmZn+LBwHOJ3Cf5WRjllyfJwb78yEU6da33+cyHR5+8KeJ6xVpOMjWU9e\na3tvwJOb8bE5PNwQ5Dg3QoyRUid+LjtId6rO+it3fKx/xhjE0xLUoQ==\n-----END RSA PRIVATE KEY-----\n" + +# Peppers sha256 hashes of API tokens (sp_api_token + project_api_keys) +SP_TOKEN_PEPPER=RqYRUvLiBrqilHqIOOVJHtWl2T2ffK0LLd+VImjzGwY= + +# Audience Hub ingest key for this project ("prod" key, revealable in dashboard) +CRAWLPROOF_PROJECT_KEY=cpk_EhiJth_9UcD_T-X1KqzVkLJEtFc0K_wV-lZ-owHZCZ8 + +# CrawlProof Alerts — ValueSERP (Google SERP polling) +VALUESERP_API_KEY=5F0086E0C4D74B4A9113A89A945BA376 +ALERTS_FROM=CrawlProof Alerts diff --git a/app/(app)/projects/[id]/leads/page.tsx b/app/(app)/projects/[id]/leads/page.tsx index 26130eec..bef4d02d 100644 --- a/app/(app)/projects/[id]/leads/page.tsx +++ b/app/(app)/projects/[id]/leads/page.tsx @@ -92,7 +92,7 @@ export default async function LeadsPage({ supabase .from("outreach_campaigns") .select( - "id, name, active, auto_send, daily_send_limit, max_score, queries, seed_urls, last_run_at, last_run_note, auth_required_hosts, pitch_mode, pitch_intro, pitch_ask, pitch_facts, scan_prospects, min_intent, angle, sender_name, reply_to", + "id, name, active, auto_send, daily_send_limit, max_score, queries, seed_urls, last_run_at, last_run_note, auth_required_hosts, pitch_mode, pitch_intro, pitch_ask, pitch_facts, scan_prospects, min_intent, sells_description, angle, sender_name, reply_to", ) .eq("project_id", projectId) .order("updated_at", { ascending: false }) diff --git a/app/actions/leads.ts b/app/actions/leads.ts index 9ab0aa0e..c49953cc 100644 --- a/app/actions/leads.ts +++ b/app/actions/leads.ts @@ -511,6 +511,8 @@ export async function saveCampaignAction(input: { * campaign did before intent existed. */ minIntent?: number | null; + /** Plain prose: what this campaign sells. Enables description matching. */ + sellsDescription?: string; }): Promise | Err> { const auth = await requireLeadAccess(input.projectId); if (!auth.ok) return auth; @@ -584,6 +586,7 @@ export async function saveCampaignAction(input: { input.minIntent === null || input.minIntent === undefined ? null : Math.max(0, Math.min(100, Math.round(input.minIntent))), + sells_description: input.sellsDescription?.trim() || null, angle: input.angle?.trim() || null, sender_name: input.senderName?.trim() || null, reply_to: input.replyTo?.trim() || null, diff --git a/components/leads/campaign-panel.tsx b/components/leads/campaign-panel.tsx index e7d089d3..bcaff3dd 100644 --- a/components/leads/campaign-panel.tsx +++ b/components/leads/campaign-panel.tsx @@ -37,6 +37,7 @@ export type CampaignSummary = { pitch_facts: string[]; scan_prospects: boolean; min_intent: number | null; + sells_description: string | null; auth_required_hosts: string[]; runs: CampaignRun[]; }; @@ -89,6 +90,7 @@ export function CampaignPanel({ // the failure mode the whole feature exists to avoid, so it should be what // you turn off deliberately rather than what you remember to turn on. const [minIntent, setMinIntent] = useState(DEFAULT_MIN_INTENT); + const [sellsDescription, setSellsDescription] = useState(""); const [goal, setGoal] = useState(""); const [generating, setGenerating] = useState(false); const [pitchNote, setPitchNote] = useState(null); @@ -166,6 +168,7 @@ export function CampaignPanel({ setPitchFacts((c.pitch_facts ?? []).join("\n")); setScanProspects(c.scan_prospects); setMinIntent(c.min_intent ?? null); + setSellsDescription(c.sells_description ?? ""); // Every field the form submits has to be loaded, not just the ones that // changed recently: save() sends the whole form, so anything left blank // here is written back as blank and silently wipes the column. @@ -199,6 +202,7 @@ export function CampaignPanel({ pitchFacts, scanProspects, minIntent, + sellsDescription, }); if (result.ok) setEditing(null); return result; @@ -517,6 +521,24 @@ export function CampaignPanel({ + {minIntent !== null && ( +