perplexity-scraper

Ask Perplexity a question and get the cited answer back as structured data — the answer text plus the source domains it cited. Great for AI-answer / brand monitoring: tracking whether (and how) your brand shows up when people ask AI answer engines buyer-intent questions.

Powered by ScrapeUnblocker. Perplexity is a JavaScript-heavy, anti-bot-protected app — the answer is streamed into a real browser. This project uses ScrapeUnblocker's getPageSource endpoint with browser steps to load the page, wait for the answer to render, and return the final HTML, which it parses into a small, stable model.

Features

• 🔎 Cited answers — the answer prose plus the domains Perplexity cited.
• 🏷️ Brand monitoring — answer.mentions("YourBrand") / answer.cites("yourbrand.com"), and a CLI exit code you can alert on.
• 🧰 Library + CLI — use it in code or from the shell (json, text, csv).
• 🌍 Country targeting — run the query from a chosen proxy country.
• ♻️ Resilient — automatic retries on transient upstream/browser errors and on anonymous anti-bot / signup walls, so you get a real answer.
• ✅ Tested — offline unit tests that mock the API (no credit spent in CI).

Install

pip install .
# or, for development:
pip install -e ".[dev]"

Set your API key (get one from the ScrapeUnblocker dashboard):

cp .env.example .env      # then edit it, or just export the var:
export SCRAPEUNBLOCKER_KEY="your_key_here"

CLI usage

# JSON (default)
perplexity-scraper "What is the best web scraping API?"

# Plain text answer + sources
perplexity-scraper "How do I bypass anti-bot protection?" --format text

# Brand check: exit code 1 if "ScrapeUnblocker" is not mentioned
perplexity-scraper "Tools to scrape JavaScript sites" --brand ScrapeUnblocker

# Export cited sources as CSV, run the query from Germany
perplexity-scraper "beste web scraping tools" --format csv --country de

usage: perplexity-scraper [-h] [-b BRAND] [-f {json,text,csv}] [-c COUNTRY]
                          [--settle-ms SETTLE_MS] [--timeout-ms TIMEOUT_MS]
                          [--retries RETRIES]
                          query

Library usage

from perplexity_scraper import PerplexityScraper

with PerplexityScraper() as scraper:  # reads SCRAPEUNBLOCKER_KEY
    answer = scraper.ask("What is a headless browser?")

print(answer.answer)  # the cited answer text
print(answer.domains)  # ['example.com', 'wikipedia.org', ...]
print(answer.mentions("Selenium"))  # brand/term check -> bool
print(answer.cites("wikipedia.org"))  # cited-domain check -> bool

Example output

{
  "query": "What is the best web scraping API?",
  "url": "https://www.perplexity.ai/search?q=What%20is%20the%20best%20web%20scraping%20API%3F",
  "answer": "There are several strong web scraping APIs, including ...",
  "sources": [
    { "domain": "apify.com", "url": "https://apify.com/store/..." },
    { "domain": "scrapeunblocker.com", "url": null }
  ]
}

Perplexity's answers and citations are generated live, so exact wording and the set of cited sources vary between runs. sources[].url is populated when an outbound link for that domain is present, otherwise just the domain is returned.

Examples

Script	What it does
examples/ask.py	Ask one question, print the JSON answer.
examples/brand_monitor.py	Check a brand across buyer-intent questions; non-zero exit if missing.
examples/batch_to_csv.py	Run many questions, export every cited source to CSV.

Project layout

src/perplexity_scraper/
  __init__.py     package exports + __version__
  scraper.py      PerplexityScraper + Answer/Source models + HTML parsing
  cli.py          argparse CLI (json/text/csv, brand check)
examples/         runnable scripts (ask, brand monitor, batch CSV)
tests/            offline unit tests (SDK mocked)
.github/workflows/ci.yml   lint + tests on Python 3.9 and 3.12

How it works

1. Build the anonymous search URL https://www.perplexity.ai/search?q=<query>.
2. Call ScrapeUnblocker getPageSource with browser steps: wait for the answer to render (wait_for_text: "Sources"), then a short settle so late citations finish loading.
3. Parse the returned HTML: the answer prose from the .prose container, and the cited domains from each citation's favicon.

See the browser-steps guide and the getPageSource guide.

Development

make install    # editable install with dev extras
make lint       # ruff check + format check
make test       # run the offline test suite
make run        # ask a sample question (needs SCRAPEUNBLOCKER_KEY)

Links

• 🌐 Website: https://scrapeunblocker.com/?utm_source=github&utm_medium=integration&utm_campaign=example-repos
• 📚 Docs: https://docs.scrapeunblocker.com/?utm_source=github&utm_medium=integration&utm_campaign=example-repos
• 🐍 Python SDK: pip install scrapeunblocker

License

MIT © 2026 ScrapeUnblocker