Leads: find contact addresses, stop leads stranding at new - #124
Merged
Merged
Conversation
Every lead showed "no contact address found". Two separate causes.
1. Nothing came back after the scan finished.
Discovery queues a free scan and returns immediately, so a new lead has no
findings and no contact yet. A campaign tick revisits its own leads; leads
added by hand from the finder had nothing revisiting them, so they sat at
"new" beside a completed scan forever. All nine leads in production were in
exactly that state. Adds a "Check scans" button that re-researches whatever
is waiting, and shows how many that is.
2. Contact discovery was too naive for how sites actually publish addresses.
Measured against the nine real agency sites in the pipeline, it found 2/9.
Diagnosing those pages showed three causes, none of which were guesses:
- not one of them had a mailto: link; they all use contact forms
- two hid the address behind Cloudflare's data-cfemail encoding
- path guessing missed: /contact 404s where /contact-us works, and one
address existed only on /privacy-policy
So it now decodes Cloudflare addresses, reads "hello (at) example (dot) com"
and HTML-entity forms, and follows the site's own contact-ish links instead
of guessing paths — the homepage nav knows where its contact page is and we
do not. Links are ranked, not first-six-wins: the naive version spent its
whole budget on /about/are-we-fit and /company/block-inc while the address
sat on /privacy-policy. Legal pages rank high deliberately, since a site
that hides its address everywhere else still has to print it there.
Same nine sites: 2/9 -> 6/9. The three misses are directories rather than
agencies, so they are the wrong leads regardless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vu1nz Security Review0 finding(s) in PR #? No security issues found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every lead showed "no contact address found". Two causes.
1. Nothing revisited a lead after its scan finished
Discovery queues a scan and returns straight away, so a fresh lead has no findings and no contact. Campaign ticks revisit their own leads; hand-added leads had nothing revisiting them — all 9 leads in prod sat at
newnext to a completed scan.→ Adds a "Check scans (n)" button that re-researches whatever is waiting.
2. Contact discovery was too naive
Measured on the 9 real agency sites in the pipeline: 2/9. Diagnosing the pages showed why:
mailto:link — all contact formsdata-cfemail/contact404s where/contact-usworks, and one address lived only on/privacy-policy→ Decodes Cloudflare addresses, reads
hello (at) example (dot) comand HTML entities, and follows the site's own contact links instead of guessing paths — ranked, not first-six-wins, because the naive crawl burned its budget on/about/are-we-fitand/company/block-inc.Same 9 sites: 2/9 → 6/9. The 3 misses are directories, not agencies — wrong leads anyway.
642 tests pass (9 new), typecheck clean, build green. No migration.
🤖 Generated with Claude Code