feat(plugins): host-managed robots.txt and plugin-claimed root text files

[Mariomarquezt]

Summary

Closes #425 with Option B.

Two SEO standards authorize by file location, and plugin routes mount under
/admin/api/cms/plugins/<id>/runtime/*. The sitemaps.org protocol scopes a sitemap to its own
path and below, and indexnow.org scopes a key file the same way — so a plugin can register both
endpoints and have neither take effect. There was no root-level surface at all, and no way to add
a Sitemap: line to a robots.txt that did not exist.

/robots.txt is now host-managed. The host serves it whether or not a plugin contributes,
seeding the document with User-agent: * / Allow: / — the same instruction to a crawler that
the previous 404 already carried (RFC 9309 §2.3.1.3), so no crawler behaviour changes on a site
with no plugins. Plugins contribute directives, not text, through a new site.robots filter:

api.cms.hooks.filter('site.robots', (doc) => {
  doc.sitemaps.push('https://example.com/sitemap.xml')
  return doc
})

The host owns serialization, so a value carrying whitespace, a #, or a CR/LF cannot forge an
extra directive line. Handlers chain in registration order; groups are validated as a unit and
sitemap URLs one by one (filterArray, the same tolerance as a corrupt font entry); sitemaps are
de-duplicated; the host default group is re-inserted if the filtered document has none left.

A new site.rootFiles filter claims one root .txt path per file, which covers the IndexNow
key:

api.cms.hooks.filter('site.rootFiles', (doc) => {
  doc.files.push({ path: `/${key}.txt`, content: key })
  return doc
})

Claimable paths are an allowlist — one root segment, .txt, leading alphanumeric — so nested
paths, dot segments, percent escapes and every other extension are rejected, and /robots.txt
stays reserved for the host. Bodies are capped at 4 KiB and may not carry C0 control characters
other than tab and newline. A path claimed by two plugins is served by neither: resolving it
to one winner would silently authorize the wrong IndexNow submitter. The refusal is logged with
the candidate plugin ids, which is what the new hookBus.pluginsFor exists for.

Design notes

• No new permission. Both filters ride cms.hooks. publish.html already lets a cms.hooks
  plugin rewrite every published page, which is strictly more power than serving one inert root
  text file — a separate permission would add a consent line without adding safety, and would
  have required editing the locked EXPECTED_TARGET_PERMISSIONS table.
• Structured payloads, not raw text, following media.url.transform. TypeBox schemas in
  src/core/plugin-sdk/siteRootSchemas.ts are the source of truth; every list bound is enforced
  by a host-side slice as well as maxItems, because applyFilter only checks the value's
  runtime type category.
• Runtime registration, not manifest declaration. An IndexNow key is generated per install
  and lives in plugin settings, so a static manifest path cannot carry it. Riding hooks.filter
  also means hookBus.unregisterPlugin already handles teardown on disable, uninstall and crash
  recovery — no new call sites.
• Dispatcher position: both handlers sit after every host-owned namespace and before
  tryServePublicRoute. They cannot shadow content — pageSlugError rejects any page slug
  containing ., and a data-row route needs at least /<table>/<slug>, so no published URL is
  ever a root .txt path. An unclaimed .txt path returns null and keeps falling through to the
  site's 404.
• Not baked into the published slot. Layer A artefacts are .html files named from page
  routes and rewritten wholesale per publish; both of these documents depend on which plugins are
  active now, so a baked copy would keep serving a disabled plugin's sitemap line or key file.
  Responses go out text/plain with nosniff, default-src 'none' and no-store.

On the preparatory commit, and PR scope

server/router.ts was three lines under the 700-line ceiling module-size-budgets.test.ts
enforces, so any new route had to displace something first. The first commit moves serveSiteCss
— with its (bundle, hash) memo, in-flight de-duplication and published-snapshot rebuild walk —
into server/publish/siteCssServer.ts, next to the siteCssBundle.ts that builds what it
serves. Verbatim move, no behaviour change, separate commit so it reviews independently.

Flagging this against AGENTS.md's "keep PR scope coherent": the extraction is not opportunistic
cleanup, it is forced by your own size gate, and AGENTS.md also says not to justify a workaround
with "to keep this PR small". If you would rather have a GRANDFATHERED entry than the
extraction, that swap is trivial and I will make it.

Security

Every case below has a test in src/__tests__/server/siteRoot.test.ts: robots.txt directive
injection via CR/LF; # comment smuggling; response-header injection (plugins never supply
headers or raw text); path traversal, raw and percent-encoded (the pathname is matched raw and
never decoded, and a claimable path cannot contain %); a plugin claiming /index.html,
/favicon.svg, /sitemap.xml or a host-owned namespace; a plugin hijacking /robots.txt; one
plugin stealing another's path; unbounded content size (4 KiB per body, 20 claims, 20 groups, 50
sitemaps, 100 paths per group); control-character and terminal-escape payloads; content-type
confusion (forced text/plain, nosniff); a javascript:/file: sitemap URL; a buggy or
throwing plugin emptying robots.txt; and method confusion.

Known trade-offs

1. /robots.txt returns 200 where it returned 404 — the one intentional behaviour change,
   argued above as a no-op for conforming crawlers. If you disagree, the fix is one line:
   return null from serveRobotsTxt when !hasFiltersFor('site.robots').
2. Group-level validation granularity — one bad Disallow drops the whole User-agent
   group, not just that line. Matches filterArray granularity elsewhere.
3. No per-entry plugin attribution on conflicts. applyFilter chains handlers opaquely, so
   the duplicate-claim log names all plugins registered on site.rootFiles as candidates rather
   than the two actual claimants. This is the one place Option A would genuinely be better.
4. No memoisation. Every root .txt GET on a site with a site.rootFiles plugin costs one
   worker round-trip. A revision-keyed memo would go stale when a plugin's settings change the
   key it serves; reasoning is in the module header.
5. No Playwright coverage — the existing e2e specs do not cover public-route serving.

Docs updated in the same change: the filter table and a new "Site-root text files" section in
docs/features/plugin-system.md, the route table and ordering notes in docs/server.md, and the
module tables in docs/features/publisher.md.

Verification

• bun run build — tsc -b && vite build, ✓ built in 14.05s, no type errors
• bun test — 6867 pass / 0 fail across 739 files
• bun run lint — clean, exit 0
• Docker/deployment check — not run; this change adds no config, env var or deployment surface

Baseline on 3a9543ed was 6818 pass / 1 skip / 0 fail. The +48 reconciles as +31 new tests here
(30 in siteRoot.test.ts, 1 in pluginHookBus.test.ts) and +17 from bundle-size-budgets.test.ts,
which logs dist/assets/ missing — bundle gates skipped on a clean tree and runs its 18 tests once
bun run build has produced dist/.

One caveat worth stating: a single baseline run before this change failed with a QuickJS
Aborted(Assertion failed: list_empty(&rt->gc_obj_list)) in the plugin-VM tests. Subsequent runs
were clean, so that flake looks pre-existing and unrelated, but you may already know about it.

Checklist

• Tests cover behavior changes.
• Docs were updated when behavior, config, deployment, or public surfaces changed.
• No compatibility shim was added for old pre-release behavior.
• No secrets, local databases, uploads, or generated artifacts are included.