crawler

This repository holds polite-crawler, a polite web crawler for research and for AI agents, and its site.

polite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.

• The package: packages/polite-crawler is on npm as polite-crawler.
• The site: packages/site is the documentation: https://mark1russell7.github.io/crawler/. Its interactive figures and tools start the real code: hybrid search, the recipes, the daemon and the robots.txt tester.
• The study: Who lets AI agents in? The crawler read the robots.txt files of the top 1,000 domains. The page shows the verdict for 15 agents, with the rule that decided it.

npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler mcp

What the crawler does for you and for the sites:

• It stops a host after a challenge, a 401, a 402 or a 403, for 1 to 24 hours. It does not try a different identity.
• By default, it fetches only public addresses. --allow-private permits loopback and private addresses.
• The MCP server gives page text in an <untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.
• The library is polite-crawler, and the MCP server is polite-crawler/mcp.

The package README tells the details.

Development

The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.

pnpm install
pnpm browser   # the headless shell for the renderer test
pnpm check     # typecheck, lint, test, lint:ste, build and the smoke test of the package

The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.

• CONTRIBUTING.md tells how to set up, test and write a change.
• SECURITY.md tells how to report a vulnerability privately.
• CHANGELOG.md records the changes in each version.

License

MIT