docs: page descriptions, titles, structured data and crawler files for search and answer engines

[imran-siddique]

A crawl of every tests.agentrust-io.com sitemap URL on 2026-09-14 found 21 of 22 pages sharing one meta description, "Conformance test suite for TRACE: Trust Runtime Attestation and Compliance Evidence", because Material falls back to the site description.

• JSON-LD had no Organization @id, and every page carried two Organization and two WebSite nodes (44 of each), one of them inline as the SoftwareApplication author. Nothing tied the suite to the AgenTrust organization on agentrust-io.com, and no inner page had a BreadcrumbList.
• The SoftwareApplication node had no codeRepository, license or operatingSystem.
• The home description was 161 characters.
• robots.txt was never deployed: the host serves only Cloudflare's content-signals comment block, with no directives and no sitemap line.
• llms.txt sections were Getting started, Test modules and Reference, without LIMITATIONS or the self-verification report.
• The template comment said the homepage title was overridden. No such block existed.

The same fixes are going into every AgenTrust docs site from one shared kit.

Changes

• hooks/seo.py, registered under hooks: in mkdocs.yml, sets each page's description from its first paragraph of prose, capped at 155 characters. Front matter still wins. hooks/ is excluded from the docs.
• overrides/main.html now has the title block the old comment described: the home page uses its front-matter title alone, and a title that already names the site gets no suffix. og:title matches, og:site_name is AgenTrust. The JSON-LD graph has one Organization under https://agentrust-io.com/#organization, the WebSite, a WebPage or TechArticle per page with its own title, description and URL, and a BreadcrumbList on inner pages. The home page SoftwareApplication names the agentrust-trace-tests package with its PyPI URL, codeRepository, the Apache-2.0 license, operatingSystem, and the organization by @id.
• robots.txt is added with the ten crawlers agentrust-io.com names and the sitemap line, and docs.yml copies it into the build and triggers on it and on hooks/.
• llms.txt sections follow the tabs: Get started, Specification (levels, the eight modules, error codes), Guides (tutorials and the self-verification report) and Project (limitations, changelog, contributing). The description adds one sentence placing the suite in AgenTrust.
• The home description is 150 characters.

Checked, on a build that replicates docs.yml (allowlist copy including robots.txt, docs_dir rewrite, mkdocs build --strict with mkdocs 1.6.1, mkdocs-material 9.7.7, mkdocs-llmstxt 0.5.0)

• Strict build exits 0.
• 22 pages: 0 use the site description, 0 over 160 characters, 0 duplicate titles, every og:title equals the title. Each page's JSON-LD parses with exactly one Organization under the hub @id, a page node whose URL is the canonical and whose description matches the meta tag, and a BreadcrumbList on all 21 inner pages.
• robots.txt is at the built site root with 11 user-agent groups and the sitemap line. llms.txt has the four sections and 19 links, all resolving in the build.
• pytest tests/test_docs_match_the_modules.py tests/test_obligation_accounting.py: 82 passed.

Generated with Claude Code

https://claude.ai/code/session_013aK3gVWzNdcM3hZ2o2awK2