Scores how well AI agents can read, cite and operate any website.
Sites without machine-readable surfaces serve agents HTML soup, so they go uncited and agents can't operate them. agentready measures that gap with 15 checks. It ships as plain Node scripts with zero dependencies and needs no API keys, accounts or cloud.
- Evidence for every verdict. Each check records the URL it requested and the status it got back, and the report prints both.
- Soft-404s fail. A file check passes only when the response is OK, the body does not start as an HTML page, it differs from what a random nonexistent path on the same site returns, and it is valid for its type (
llms.txt: over 40 bytes). - Reproducible score. The score is
earned / 93weighted points, rounded. Grades run A ≥ 90, B ≥ 78, C ≥ 62, D ≥ 45, E ≥ 25, F otherwise. - No supply chain. No
node_modulesand no lockfile. It uses Node's globalfetchonly. - Read-only. The scanner only issues GET requests and writes nothing to the target.
git clone https://github.com/FetchDoge/agentready && cd agentready
node scan.mjs stripe.comSample output (recorded when this README was first written):
AI AGENT READINESS — stripe.com
────────────────────────────────────────────────────
SCORE 47/100 GRADE D (44/93 weighted points)
────────────────────────────────────────────────────
PASS 4pt Served over HTTPS
PASS 12pt llms.txt present → 200 (69878 bytes)
FAIL 4pt llms-full.txt present → 404
FAIL 10pt robots.txt addresses AI crawlers
robots.txt → 200; agents named: none
FAIL 8pt Markdown content negotiation
Accept: text/markdown → content-type: text/html; charset=utf-8
...
TOP FIXES:
1. llms-full.txt present (+4 pts available)
2. robots.txt addresses AI crawlers (+10 pts available)
3. Agent/MCP manifest present (+8 pts available)
The scan writes a standalone HTML report and a JSON record to ./agentready-out/. Pass --out <dir> to change the location or --json for machine-readable output.
| Condition | Behaviour |
|---|---|
| A request errors or exceeds the timeout | Each request is time-bounded (12 s by default, --timeout <ms> to change). It records status 0 and the check fails, but the scan does not crash |
| Site unreachable (DNS, TLS or connect error on both HTTPS and HTTP) | Reported as UNREACHABLE with the error, exit code 2, and no score |
Catch-all site (any unknown path returns 200) |
The scanner first requests a random path that cannot exist. A file whose body matches that page, or HTML served for a .txt/.json/.xml path, is reported as soft-404 and fails |
| Malformed JSON-LD block | The scanner skips that block and counts only parseable types |
| Issue bot: no URL in the issue | Replies asking for a URL and scans nothing |
Issue bot: localhost, *.local, *.internal, IP literals, non-http(s) |
Rejects the target and explains why. The filter checks hostnames only and does not resolve DNS, so it is hygiene, not a full SSRF defence |
| Issue bot: scan fails or times out | Replies that the site refused the request or timed out |
Open an issue titled "Scan my site" with a URL in the body — click here — and a GitHub Action scans the site and posts the full report as a comment. No backend, no signup, no API key. See a live one: issue #1.
# single site → HTML report + JSON
node scan.mjs example.com --out ./out
# machine-readable
node scan.mjs example.com --json
# show only the 5 biggest fixes (default: every failing check); per-request timeout in ms
node scan.mjs example.com --top 5 --timeout 8000
# white-label HTML (+ optional PDF via installed Chrome)
node scan.mjs example.com --brand brand.json --pdf --out ./out
# a cohort → ranked leaderboard + CSV
node batch.mjs my-list.txt --out ./out --cat "Calgary plumbers"
node batch.mjs my-list.txt --out ./out --anon # leaderboard shows Site A, Site B, … instead of hosts
node batch.mjs my-list.txt --out ./out --brand brand.json --pdf
# the fix, not the diagnosis → tailored llms.txt + robots AI block + schema + instructions
node fix.mjs example.com --out ./fixes
node fix.mjs jane-doe.dev --out ./fixes --type person # auto-detected by default: person | organization | localbusiness
node fix.mjs example.com --out ./fixes --brand brand.json
# regression tests (zero dependencies, local fake servers)
node --test test/scan.test.mjsExit codes: 0 scanned, 1 usage error, 2 site unreachable. An unreachable site is reported as
UNREACHABLE with the DNS/TLS/connect error and is not given a score. --pdf with no Chrome found
warns and still exits 0 after writing HTML and JSON.
Every file check is soft-404 aware: the scanner first requests a random path that cannot exist. If the
site answers it with 200 (a catch-all, e.g. an SPA host with no 404 page), any file whose body matches
that page, or any HTML body served for a .txt/.json/.xml path, is reported as soft-404 and fails.
Agencies can brand scan/batch/fix output with --brand <file.json>. Without a brand file, HTML stays
the default agentready look (plus print CSS). With a brand file, reports use a restrained light
typographic layout (score and grade first, plain-language summary, honest evidence, Letter print CSS),
poweredBy defaults to false, and brand text is HTML-escaped.
node scan.mjs client.com --brand brand.json --user-agent "Mozilla/5.0 (compatible; AcmeAudit/1.0; +https://acme.example)" --pdfExample brand.json:
{
"name": "Big Fox",
"url": "https://www.bigfox.ca",
"contact": "hello@example.com · 403-555-0100",
"logo": "./bigfox-logo.png",
"colors": {
"accent": "#ff6a00",
"bg": "#ffffff",
"card": "#f6f7f9",
"fg": "#111111",
"muted": "#5b6372",
"line": "#e3e6ec",
"good": "#1a9e5a",
"bad": "#d33a4c"
},
"theme": "light",
"reportTitle": "AI Agent Readiness Report",
"footer": "Prepared by Big Fox for {host}",
"poweredBy": false,
"userAgent": "Mozilla/5.0 (compatible; BigFoxAudit/1.0; +https://www.bigfox.ca)",
"probePrefix": "probe"
}| Field | Notes |
|---|---|
name, url, contact, reportTitle, footer |
Optional. {host} in footer is substituted. Strings are escaped in HTML |
logo |
Local PNG/JPEG/SVG path only (≤ 500 KB), inlined as a data: URI. Remote URLs rejected |
colors.* |
Hex colours (#rgb / #rrggbb / …) mapped to CSS variables |
theme |
light (default when branding) or dark |
poweredBy |
Default false with a brand file; set true to add “Methodology: open-source agentready (MIT)” |
userAgent |
Used for all requests; must include a contact http(s) URL. --user-agent overrides |
probePrefix |
Soft-404 probe path becomes /{probePrefix}-<hex>.txt (default agentready-probe) |
PDF: Reports always include @media print CSS (Letter page size). --pdf shells out to an already-installed
Chrome/Chromium (AGENTREADY_CHROME or --chrome <path>, then common install locations). No Puppeteer,
no node_modules. If Chrome is missing, the tool warns and still writes HTML + JSON.
| Weight | Check |
|---|---|
| 12 | llms.txt present (real content, not a soft-404) |
| 12 | JSON-LD structured data |
| 10 | robots.txt has an explicit User-agent group for at least one AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, …). Allow or Disallow counts; evidence lists which are allowed and which are blocked; comment-only mentions do not count |
| 8 | OpenAPI / Swagger spec discoverable |
| 8 | Agent manifest (/.well-known/mcp.json, ai-plugin.json, …) |
| 8 | Markdown content negotiation (Accept: text/markdown) |
| 5 | sitemap.xml reachable |
| 5 | High-value schema types (Organization, LocalBusiness, Product, FAQPage, Article, Person, ProfilePage, SoftwareSourceCode, …) |
| 5 | OpenGraph + meta description |
| 4 | HTTPS (a real TLS connection) |
| 4 | llms-full.txt present (real content, not a soft-404) |
| 3 | ai.txt present |
| 3 | security.txt present |
| 3 | Agent hint file present (/.well-known/agents.json, /agents.txt, /.well-known/ai.txt) |
| 3 | Sitemap declared in robots.txt (a robots.txt with only a Sitemap: record is a valid file under RFC 9309: it earns this point, but not the AI-crawler points) |
The table mirrors the add(...) calls in scan.mjs, which sum to 93. The weights are opinionated and public: edit them in scan.mjs and argue with them.
- Committed cohort scan:
examples/agent-readiness-2026-10-06.html(CSV), 20 sites anonymized as Site A–T. 0 of 20 sites grade C or better, the best score is 49/100, and 6 of 20 could not be read by the scanner. Full figures under Why it matters. - Live issue scan: issue #1.
- Quickstart output: the
stripe.comsample above, re-run on 2026-10-07. It scored 47/100, grade D, 44/93 points. Live sites change, so your numbers may differ. - Regression tests:
node --test test/scan.test.mjs(zero dependencies, local fake servers). CodeQL code scanning and the scan-on-issue workflow run on GitHub Actions.
Agents — ChatGPT, Claude, Perplexity, Gemini, Copilot — increasingly read sites through
machine-readable surfaces before they read the HTML. llms.txt, AI crawler rules in
robots.txt, schema.org JSON-LD, an OpenAPI spec, an MCP manifest, markdown content
negotiation. When a site lacks them, the agent gets an HTML soup of nav links and marketing
copy, or nothing at all.
Two things happen then: the site doesn't get cited when someone asks an assistant a question it should have answered, and it can't be operated by an agent that wanted to book, quote, or buy.
On 2026-10-06 we scanned the websites of 20 Calgary-area plumbing, heating and HVAC
companies with this version of the scanner (one pass of node batch.mjs <list> --anon,
user-agent AgentReadyBot/1.0, soft-404 aware). Full leaderboard:
examples/agent-readiness-2026-10-06.html (sites anonymized as Site A–T).
| Sites graded C (62+) or better | 0 of 20 |
| Best score | 49 / 100 (grade D; five sites tied) |
| Mean score | 25.9 / 100 across the 19 sites that answered (33.8 across the 14 the scanner could read) |
Sites publishing llms.txt |
5 of 20 |
| Sites with JSON-LD | 10 of 20 |
| Could not be read by the scanner | 6 of 20: 3 bot walls (403 / CAPTCHA), 2 domains with no site connected, 1 redirect loop (unreachable, not scored) |
The 6 unreadable sites stay in the counts. The 5 that answered are scored on what an agent actually received (4/100, HTTPS only): that is the honest answer for an agent, but it says nothing about the content behind the wall. A different network or user-agent may get through.
Three of the five that publish llms.txt do so by accident: Yoast SEO (v28.5 / v28.6) and Rank Math
now auto-generate it.
These figures replace an earlier 2026-10-04 run (mean 22.7). That run used a scanner that counted bot-challenge pages as real files and scored a domain it could not reach; both bugs are fixed here. Scores also move day to day: one site that blocked the scanner on Oct 4 scored 49 on Oct 6.
- Zero dependencies. No
node_modules, no lockfile, no supply chain. Globalfetchonly. - Readable in a sitting.
scan.mjsplus a small sharedbrand.mjsfor white-label/PDF helpers — you can audit every claim. - Evidence, not vibes. Every check prints the URL it hit and the status it got back.
- Soft-404 aware. A 200 that returns HTML is not a passing
llms.txt.
Working tool, early. The single-site scanner and the issue bot run in production on this repo. The weights are opinionated and open to argument.
Built by Marcus Richards, directing AI coding agents.
MIT. Copyright (c) 2026 Marcus Richards. See LICENSE.