Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/tasks/P002-sitemap-robots-integrity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# P002 — Sitemap and robots crawl-discovery integrity

## Goal
Keep sitemap.xml and robots.txt consistent with published GitHub Pages URLs and prevent misleading per-page modification timestamps.

## Diagnosis (2026-10-26)
- Search Console showed "Site haritası okunamadı" with 0 discovered pages.
- Repository main contained a valid-looking XML sitemap with 49 URL entries, matching all 49 published index.html paths in the Git tree.
- robots.txt advertised the correct HTTPS sitemap and allowed crawling.
- Both offline documentation generators rewrote every sitemap `lastmod` to the generator execution date, regardless of which pages changed. This is not a verifiable per-page modification date.
- Search Console's fetch failure cannot be conclusively reproduced from the repository; public HTTP responses and Google's fetch diagnostics must be checked separately.

## Scope
- Preserve all 49 canonical URLs and the existing crawler access policy.
- Omit optional `lastmod` unless a future generator can derive an accurate per-URL significant-content modification date.
- Update both offline docs generators so they do not recreate inaccurate `lastmod`.
- Extend the existing dependency-free `scripts/verify-site.mjs` CI verifier to check sitemap XML envelope/entry structure, complete and unique canonical coverage, HTML canonical/noindex consistency, and the robots sitemap declaration.

## Acceptance criteria and verification
- Exactly one sitemap URL per public HTML page (root + both documentation trees).
- No stale URLs, duplicates, malformed sitemap entries or cross-origin URLs.
- robots.txt continues to allow general crawling and advertises exactly one canonical sitemap.
- `node scripts/verify-site.mjs` and `git diff --check` pass on the PR head.
- The required GitHub Actions `docs-integrity` check passes before squash merge to protected `main`.
- Verify PR merged and `main` contains the change. Preserve local uncommitted work.

## Evidence / operational follow-up
- Baseline repository tree inspection: 49 public HTML pages and 49 matching sitemap locations, with 49 entries dated 2026-10-02.
- Root cause of the Search Console fetch failure remains unconfirmed until the production endpoint responds with HTTP 200/XML from an external network and Google Search Console's live fetch diagnosis is available.
- After deployment, confirm `https://cynrath.github.io/sitemap.xml` returns HTTP 200 with XML, then resubmit `sitemap.xml` to Google Search Console and verify the report updates.
- Google guidance: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- Google troubleshooting: https://support.google.com/webmasters/answer/7451001
4 changes: 2 additions & 2 deletions scripts/sync-ackit-docs.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -200,13 +200,13 @@ async function main() {
// Merge sitemap: keep existing bridge URLs verbatim (owned by
// scripts/sync-ackit-spec-kit-bridge-docs.mjs) so generator order never
// drops product pages. Sorted+unique for deterministic output.
const today = new Date().toISOString().slice(0,10);
// Omit lastmod without a verifiable per-page modification timestamp.
const existingRaw = await readOptional(path.join(siteRoot,'sitemap.xml'));
const existingLocs = existingRaw ? [...existingRaw.matchAll(/<loc>([^<]+)<\/loc>/g)].map(m => m[1]) : [];
const bridgeLocs = existingLocs.filter(u => u === `${SITE}/ackit-spec-kit-bridge/` || u.startsWith(`${SITE}/ackit-spec-kit-bridge/`));
const urls = [...new Set([`${SITE}/`, ...list.map(p => `${SITE}/agent-context-kit/${p.slug ? `${p.slug}/` : ''}`), ...bridgeLocs])].sort();
let sitemap = '<?xml version="1.0" encoding="UTF-8"?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n';
for (const url of urls) sitemap += ` <url>\n <loc>${url}</loc>\n <lastmod>${today}</lastmod>\n </url>\n`;
for (const url of urls) sitemap += ` <url>\n <loc>${url}</loc>\n </url>\n`;
sitemap += '</urlset>\n';
await write(path.join(siteRoot,'sitemap.xml'), sitemap);

Expand Down
4 changes: 2 additions & 2 deletions scripts/sync-ackit-spec-kit-bridge-docs.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -189,13 +189,13 @@ async function main() {
// Merge sitemap: keep existing non-bridge URLs verbatim (root, ACKit),
// replace the bridge set with the freshly generated one. Sorted+unique
// so output is deterministic regardless of generator order.
const today = new Date().toISOString().slice(0,10);
// Omit lastmod without a verifiable per-page modification timestamp.
const existing = readSitemapLocs(await readOptional(path.join(siteRoot,'sitemap.xml')));
const keep = existing.filter(u => u !== `${SITE}/ackit-spec-kit-bridge/` && !u.startsWith(`${SITE}/ackit-spec-kit-bridge/`));
const bridgeUrls = list.map(p => `${SITE}/ackit-spec-kit-bridge/${p.slug ? `${p.slug}/` : ''}`);
const urls = [...new Set([...keep, ...bridgeUrls])].sort();
let sitemap = '<?xml version="1.0" encoding="UTF-8"?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n';
for (const url of urls) sitemap += ` <url>\n <loc>${url}</loc>\n <lastmod>${today}</lastmod>\n </url>\n`;
for (const url of urls) sitemap += ` <url>\n <loc>${url}</loc>\n </url>\n`;
sitemap += '</urlset>\n';
await write(path.join(siteRoot,'sitemap.xml'), sitemap);

Expand Down
65 changes: 65 additions & 0 deletions scripts/verify-site.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,71 @@ async function main() {
fail("ACKit generator does not preserve bridge sitemap URLs");
}


// 15. Crawl discovery: sitemap structure and canonical URL coverage, plus robots.
const seoXml = await fsp.readFile(path.join(siteRoot, "sitemap.xml"), "utf8");
const envelope = seoXml.match(
/^<\?xml version="1\.0" encoding="UTF-8"\?>\s*<urlset xmlns="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9">\s*([\s\S]*?)\s*<\/urlset>\s*$/,
);
if (!envelope) {
fail("SEO: sitemap XML declaration or urlset structure is invalid");
} else {
const entryPattern = /<url>\s*<loc>([^<]+)<\/loc>\s*(?:<lastmod>(\d{4}-\d{2}-\d{2})<\/lastmod>\s*)?<\/url>/g;
const entries = [...envelope[1].matchAll(entryPattern)];
if (envelope[1].replace(entryPattern, "").trim() !== "") {
fail("SEO: sitemap contains malformed or unsupported XML entries");
}
const found = new Set();
const expected = new Set([
SITE + "/",
...[...pages, ...bridgePages].map((page) => SITE + "/" + posix(page).replace(/index\.html$/, "")),
]);
for (const [, loc, lastmod] of entries) {
if (found.has(loc)) fail("SEO: duplicate sitemap URL " + loc);
found.add(loc);
try {
const parsed = new URL(loc);
if (parsed.origin !== SITE || parsed.href !== loc || !parsed.pathname.endsWith("/")) {
fail("SEO: invalid or non-canonical sitemap URL " + loc);
}
} catch {
fail("SEO: malformed sitemap URL " + loc);
}
if (lastmod && (!Number.isFinite(Date.parse(lastmod)) || lastmod > new Date().toISOString().slice(0, 10))) {
fail("SEO: invalid or future lastmod on " + loc);
}
if (!expected.has(loc)) fail("SEO: unexpected or stale sitemap URL " + loc);
}
for (const loc of expected) {
if (!found.has(loc)) fail("SEO: published HTML page missing from sitemap " + loc);
}
for (const page of [path.join(siteRoot, "index.html"), ...pages, ...bridgePages]) {
const html = await fsp.readFile(page, "utf8");
const canonical = html.match(/<link rel="canonical" href="([^"]+)"\s*\/?>/);
const expectedCanonical = page === path.join(siteRoot, "index.html")
? SITE + "/"
: SITE + "/" + posix(page).replace(/index\.html$/, "");
if (!canonical || canonical[1] !== expectedCanonical) {
fail("SEO: canonical mismatch on " + posix(page) + " (expected " + expectedCanonical + ")");
}
if (/<meta\s+name="robots"\s+content="[^"]*noindex/i.test(html)) {
fail("SEO: indexable sitemap page marked noindex: " + posix(page));
}
}
}

const crawlRobots = await fsp.readFile(path.join(siteRoot, "robots.txt"), "utf8");
const sitemapDirectives = [...crawlRobots.matchAll(/^Sitemap:\s*(\S+)\s*$/gm)];
if (sitemapDirectives.length !== 1 || sitemapDirectives[0][1] !== SITE + "/sitemap.xml") {
fail("SEO: robots.txt must advertise exactly the canonical sitemap URL");
}
if (!/^User-agent: \*$/m.test(crawlRobots) || !/^Allow: \/$/m.test(crawlRobots)) {
fail("SEO: robots.txt must allow general crawling");
}
if (/^Disallow:\s*\/\s*$/im.test(crawlRobots)) {
fail("SEO: robots.txt blocks the entire site");
}

if (failures.length > 0) {
for (const failure of failures) process.stdout.write(`FAIL ${failure}\n`);
process.stdout.write(`docs-integrity: ${failures.length} failure(s)\n`);
Expand Down
49 changes: 0 additions & 49 deletions sitemap.xml
Original file line number Diff line number Diff line change
Expand Up @@ -2,198 +2,149 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://cynrath.github.io/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/architecture/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/cli/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/compatibility/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/configuration/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/getting-started/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/lifecycle/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/profiles/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/release/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/security/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/spec-kit-extension/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/troubleshooting/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/trust-flow/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/ackit-spec-kit-bridge/verification/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/agents-md-validator/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/ai-agent-readiness/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/benchmarks/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/checkpoints/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/cli/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/context-optimization/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/dashboard/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/diagnostics/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/drift/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/evidence/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/getting-started/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/github-action/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/instruction-graph/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/intent/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/mcp-repository-tools/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/mcp/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/migration/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/optimize/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/policy/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/profiles/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/provider-surfaces/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/readiness/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/roles/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/rule-packs/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/sdk/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/security/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/skills/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/status/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/sync/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/trust-flow/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/verification/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/vscode/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://cynrath.github.io/agent-context-kit/workflows/</loc>
<lastmod>2026-10-02</lastmod>
</url>
</urlset>
Loading