feat(plugins): host-managed robots.txt and plugin-claimed root text files

docs/features/plugin-system.md

@@ -39,6 +39,7 @@ A plugin is a zip package containing a `plugin.json` manifest and one or more bu
 | Route request/response I/O     | `server/plugins/host/routeIo.ts`          |
 | Media extension handlers       | `server/plugins/host/handlers/media.ts`, `src/core/plugins/mediaStorageRegistry.ts`, `src/core/plugins/mediaVariantDelegateRegistry.ts` |
 | Published-page asset injection | `server/publish/frontendInjections.ts`    |
+| Site-root text files (`/robots.txt`, `/<name>.txt`) | `server/siteRoot.ts`, `src/core/plugin-sdk/siteRootSchemas.ts` |
 | Dashboard widget registry      | `src/core/dashboard/registry.ts`          |
 | Plugin asset path containment      | `server/util/pathWithin.ts`            |
 | Plugin lifecycle (boot, install, activate, uninstall) | `server/plugins/runtime.ts`, `package.ts` |
@@ -543,12 +544,41 @@ const name = await api.cms.hooks.emit('sync.done', { /* … */ })
 // name === 'plugin.<your-plugin-id>.sync.done'
 ```
 
-**Host-emitted events** (the reserved core list, `CORE_HOOK_EVENTS` in `src/core/plugins/hookBus.ts`): `publish.before`, `publish.after`, `content.entry.created`, `content.entry.updated`, `content.entry.deleted`, `settings.changed`. **Filters**: `publish.html`, `publish.headers`, `content.entry.cells`.
+**Host-emitted events** (the reserved core list, `CORE_HOOK_EVENTS` in `src/core/plugins/hookBus.ts`): `publish.before`, `publish.after`, `content.entry.created`, `content.entry.updated`, `content.entry.deleted`, `settings.changed`. **Filters**: `publish.html`, `publish.headers`, `content.entry.cells`, `site.robots`, `site.rootFiles`.
 
 Every filter handler returns the same runtime value type it received. `src/core/plugins/hookBus.ts` checks each result before passing it to the next handler; a mismatched result keeps the previous value and logs the offending plugin ID. For example, `publish.html` returns a string and `content.entry.cells` returns an object, never `null`.
 
 **Plugin emits are namespaced.** The host rewrites every `emit('<name>', …)` to `plugin.<your-plugin-id>.<name>` (a name already in your own namespace passes through unchanged), so event provenance is unforgeable — a plugin cannot fire `content.entry.created` or any other core event at other listeners, and emitting a name in *another* plugin's namespace (`plugin.<other-id>.*`) is rejected with an error. `emit` resolves to the canonical namespaced name. Cross-plugin eventing still works: subscribing is unrestricted, so a plugin listens to another plugin's events by their full namespaced name, e.g. `api.cms.hooks.on('plugin.acme.analytics.page-view', …)`.
 
+### Site-root text files — requires `cms.hooks`
+
+Two SEO standards authorize by file *location*, and a plugin's routes mount under `/admin/api/cms/plugins/<id>/runtime/*`: the sitemaps.org protocol scopes a sitemap to its own directory and below, and indexnow.org scopes a key file the same way. Both surfaces below close that gap with hook-bus filters — no new permission, because `publish.html` already lets a `cms.hooks` plugin rewrite every published page.
+
+`/robots.txt` is **host-managed**. The host serves it whether or not a plugin contributes, seeding the chain with `User-agent: *` / `Allow: /` — the same instruction to a crawler that its previous 404 carried (RFC 9309 §2.3.1.3). Plugins contribute *directives*, not text, and the host renders the document:
+
+```js
+api.cms.hooks.filter('site.robots', (doc) => {
+  doc.sitemaps.push('https://example.com/sitemap.xml')
+  doc.groups.push({ userAgent: 'BadBot', allow: [], disallow: ['/'] })
+  return doc
+})
+```
+
+Handlers chain in registration order. Every list entry is validated against `src/core/plugin-sdk/siteRootSchemas.ts` — a group is accepted or dropped as a unit, sitemap URLs one by one — so a value carrying whitespace, a `#`, or a CR/LF can never forge an extra directive line. Sitemap URLs are de-duplicated; when the filtered document has no valid group left, the host default one is re-inserted.
+
+`site.rootFiles` claims one root `.txt` path per file — the IndexNow key case:
+
+```js
+api.cms.hooks.filter('site.rootFiles', (doc) => {
+  doc.files.push({ path: `/${key}.txt`, content: key })
+  return doc
+})
+```
+
+Claimable paths are an allowlist: one root segment, `.txt`, starting with an alphanumeric (`/a1b2c3.txt`). Nested paths, dot segments, percent escapes, and any other extension are rejected, and `/robots.txt` is reserved for the host. Bodies are capped at 4 KiB and may not carry C0 control characters other than tab and newline. **A path claimed by two plugins is served by neither** — resolving it to one winner would silently authorize the wrong submitter — and the refusal is logged with the candidate plugin ids. Both responses go out as `text/plain` with `nosniff`, `default-src 'none'`, and `no-store`.
+
+Both handlers sit directly before `tryServePublicRoute` in the dispatcher, so every host-owned namespace still wins. They cannot shadow content either: `pageSlugError` rejects any page slug containing `.` and a data-row route needs at least `/<table>/<slug>`, so no published URL is ever a root `.txt` path. Neither response is baked into the published slot — both depend on which plugins are active right now, not on the published snapshot.
+
 ### Loop sources — requires `loops.register`
 
 ```js

docs/features/publisher.md

@@ -52,6 +52,7 @@ server/publish/
 ├── publicRenderer.ts               — renderPublishedSnapshot, renderPublishedDataRowTemplate
 ├── publishedHtmlPipeline.ts        — post-process (sanitize + plugin filters + injections)
 ├── siteCssBundle.ts                — server-side hashing + file emission
+├── siteCssServer.ts                — serves `/_instatic/css/*` (disk-first, memoised DB-rebuild fallback)
 ├── frontendInjections.ts           — splice plugin <script>/<link>/<meta> into HTML
 ├── mediaPresentation.ts            — media URL materialization for originals + responsive variants
 ├── renderTreeWalk.ts               — walkRenderTree: visits every node that contributes to a rendered page (page nodes + VC definition trees, cycle-guarded); single source of truth for loop-prefetch and media-prefetch
@@ -313,10 +314,11 @@ statement right after the slot swap — so a baked `<instatic-hole data-instatic
 always matches what the hole endpoint expects (a mismatch would make the
 endpoint refuse to hydrate).
 
-The exclusive namespaces `/_instatic/css/*` (`serveSiteCss`) and `/_instatic/assets/*`
-(`tryServeRuntimeAsset`) are served **disk-first**, falling back to a rebuild
-(`serveSiteCss`) or the DB (`published_runtime_assets`) only for preview or a
-publish whose disk write failed. Unknown paths under either prefix 404 rather
+The exclusive namespaces `/_instatic/css/*` (`siteCssServer.ts`) and
+`/_instatic/assets/*` (`tryServeRuntimeAsset`) are served **disk-first**,
+falling back to a rebuild (`serveSiteCss`) or the DB
+(`published_runtime_assets`) only for preview or a publish whose disk write
+failed. Unknown paths under either prefix 404 rather
 than falling through.
 
 ---
@@ -389,6 +391,7 @@ Because `serializeCsp` sorts, the same plugins + adapters always emit a **byte-i
 | `server/publish/publicRenderer.ts`              | `renderPublishedSnapshot`, `renderPublishedDataRowTemplate` — thin wrappers (resolve + compose the template chain, seed the context) over one shared `renderMergedTemplate` (CSS bundle + loop/media prefetch + `publishPage` + publish-version stamping). The entry path also passes the row's `readEntrySeoOverride(...)` through as `documentMeta`. |
 | `server/publish/publishedHtmlPipeline.ts`       | Post-process: DOMPurify the final HTML, run plugin `publish.html` filter, splice in declarative tags from plugin manifests, inject runtime assets. Runs at publish time only — never per-request. |
 | `server/publish/siteCssBundle.ts`               | Hash the four CSS strings, write `uploads/css/...` files. The framework bundle's module-CSS half comes from the shared walk in `siteModuleAssets.ts`. |
+| `server/publish/siteCssServer.ts`               | `serveSiteCss` — answers `/_instatic/css/<bundle>-<hash>.css` disk-first, with a `(bundle, hash)`-memoised rebuild from the published snapshot as the preview fallback. |
 | `server/publish/siteModuleAssets.ts`            | `collectSiteModuleAssets` — the one full-site render walk whose accumulators feed BOTH the framework CSS bundle (`cssMap`) and the published module-JS map (`jsMap`). |
 | `server/publish/moduleJsBundle.ts`              | Module-JS channel: `buildSiteModuleJsMap` (fresh), `buildPublishedSiteModuleJsMap` (memoised per publishVersion + site, invalidated by `bumpPublishVersion()`), and `injectModuleScripts` (per-page `<script defer>` tags + CSP `script-src 'self'` relaxation). |
 | `server/publish/republish.ts`                   | Bulk re-publish on settings change (touches every page).            |

docs/server.md

@@ -89,11 +89,16 @@ const routes: readonly RouteHandler[] = [
   tryServePublicForm,              // /_instatic/form/*       → forms/handler.ts
   tryServeRuntimeAsset,            // /_instatic/assets/*     → published runtime assets
   tryServeRuntimePackageNamespace, // /_instatic/runtime/cache/<hash>/<...> → bun install workspace
-  tryServeSiteCssNamespace,        // /_instatic/css/*        → hashed CSS bundles
+  tryServeSiteCssNamespace,        // /_instatic/css/*        → publish/siteCssServer.ts
+                                   //   (hashed CSS bundles, disk-first)
   tryServeMediaRedirect,           // /_instatic/media/<adapterId>/<path> → 302 to signed read URL
   tryServeStaticAsset,             // /assets/* → dist/ (admin app)
   tryServeUpload,                  // /uploads/* → uploadsDir (with nosniff hardening)
   tryServeAdminApp,                // /admin/* → dist/index.html (SPA fallback)
+  tryServeRobotsTxt,               // /robots.txt → server/siteRoot.ts (host document
+                                   //   plus the site.robots filter chain)
+  tryServeRootFile,                // /<name>.txt → plugin-claimed root text file
+                                   //   (site.rootFiles filter); null when unclaimed
   tryServePublicRoute,             // /<slug> OR /<route-base>/<row-slug>
                                    //   → server/publish/publicRouter.ts
                                    //   resolves to page snapshot OR data row + template,
@@ -109,6 +114,7 @@ Order matters. Two examples:
 
 - `tryServeAi` is matched **before** `tryServeCmsApi` so the AI endpoints (`/admin/api/ai/*`) aren't swallowed by the broader CMS dispatcher (`/admin/api/cms/*`).
 - `tryServeUpload` is matched **before** `tryServeAdminApp` because `/uploads/...` is a sub-tree the SPA fallback would otherwise consume.
+- `tryServeRobotsTxt` / `tryServeRootFile` are matched **after** every host-owned namespace and **before** `tryServePublicRoute`, so a plugin-claimed root file can shadow neither an admin path nor a built asset. It cannot shadow content either: page slugs may not contain `.` and a data-row route needs two segments, so no published URL is ever a root `.txt` path. An unclaimed `.txt` path returns null and keeps falling through.
 
 Adding a new endpoint is a one-line edit to `routes` plus a focused `tryServeX` function.
 
@@ -580,6 +586,8 @@ Three static handlers, in order:
 | `tryServeUpload`       | `/uploads/*` from `uploadsDir` with `hardenUploadResponse` (nosniff, attachment for non-inert MIMEs, CORS for plugin bundles) |
 | `tryServeAdminApp`     | `/admin/*` — serves the admin shell from `dist/index.html` with path-specific injections (see below) |
 
+The site-root text files (`/robots.txt` and plugin-claimed `/<name>.txt`) are served by `server/siteRoot.ts`, not from disk — see [docs/features/plugin-system.md](features/plugin-system.md) → "Site-root text files".
+
 `server/static.ts` owns all three. Key behaviors:
 
 - **Range requests** are honored for media (`Range: bytes=...`).

server/publish/siteCssServer.ts

@@ -0,0 +1,153 @@
+/**
+ * Serving side of the per-site CSS bundles — `/_instatic/css/<bundle>-<hash>.css`.
+ *
+ * `siteCssBundle.ts` BUILDS the bundle files; this module answers HTTP
+ * requests for them. It was extracted from `server/router.ts`, which had no
+ * business owning a DB-fallback rebuild walk and its memo — the dispatcher
+ * just forwards the path here.
+ */
+
+import type { DbClient } from '../db/client'
+import { registry } from '@core/module-engine'
+import type { CssBundleFile, SiteCssBundleId } from '@core/publisher'
+import { toArrayBuffer } from '../binary'
+import { getLatestSnapshotForVersion } from './publishedSnapshotCache'
+import { getPublishVersion, registerVersionedCacheReset } from './publishState'
+import { prefetchMediaAssets } from './mediaPrefetch'
+import { buildPublishedSiteCssBundle } from './siteCssBundle'
+import { readStaticAsset } from './staticArtefact'
+
+/**
+ * Serve one of the three site CSS bundle files (reset / framework / style).
+ *
+ * The URL path is `/_instatic/css/<bundle>-<hash>.css` where `<bundle>` is the
+ * logical layer name and `<hash>` is the 12-hex SHA-256 prefix that
+ * `buildSiteCssBundle` produces.
+ *
+ * Disk-first: a full publish bakes every referenced CSS file into the active
+ * slot, so this handler reads it straight off disk — no DB, no rebuild. The
+ * DB rebuild below is a fallback for preview (pre-publish) or a publish whose
+ * disk write failed.
+ *
+ *  - Browsers / CDNs cache the response for a year (`immutable`).
+ *  - When a hash changes (the site, its classes, or a stylesheet was edited),
+ *    HTML pages re-render with the new `<link href>` and visitors fetch the
+ *    new file exactly once.
+ *
+ * Stale hash → 404 so the browser falls back to refetching the HTML, which
+ * carries the current hash. Returning the new content under the old name
+ * would defeat `immutable` caching by serving different bytes for the same
+ * URL across the cache lifetime.
+ *
+ * `reset`/`framework`/`style` are page-invariant; `userStyles` is page-scoped
+ * (each stylesheet targets a subset of pages), so the fallback walks the
+ * published pages until one produces the requested hash.
+ *
+ * The DB fallback is memoised by `(bundle, hash)` — the hash is content-derived
+ * so an entry can never go stale; it can only stop being requested. Negative
+ * results are cached too (a crafted stale-hash URL would otherwise force the
+ * full rebuild walk per request). The memo resets when the publish version
+ * moves and concurrent first-hits share one in-flight rebuild.
+ */
+const cssFallbackCache = new Map<string, string | null>()
+const CSS_FALLBACK_CACHE_MAX = 256
+const cssFallbackInFlight = new Map<string, Promise<string | null>>()
+let cssFallbackVersion = -1
+registerVersionedCacheReset(() => {
+  cssFallbackCache.clear()
+  cssFallbackInFlight.clear()
+  cssFallbackVersion = -1
+})
+
+export async function serveSiteCss(db: DbClient, pathname: string, uploadsDir?: string): Promise<Response | null> {
+  const filename = pathname.slice('/_instatic/css/'.length)
+  const match = filename.match(/^(reset|framework|style|userStyles)-([a-f0-9]{12})\.css$/)
+  if (!match) return null
+
+  const [, requestedBundle, requestedHash] = match
+  const bundleId = requestedBundle as SiteCssBundleId
+
+  // Disk-first.
+  if (uploadsDir) {
+    const bytes = await readStaticAsset(uploadsDir, pathname)
+    if (bytes) {
+      return cssResponse(toArrayBuffer(bytes), requestedHash)
+    }
+  }
+
+  // Memoised DB fallback.
+  const version = getPublishVersion()
+  if (version !== cssFallbackVersion) {
+    cssFallbackCache.clear()
+    cssFallbackVersion = version
+  }
+  const cacheKey = `${bundleId}:${requestedHash}`
+  const cached = cssFallbackCache.get(cacheKey)
+  if (cached !== undefined) {
+    return cached === null ? new Response('Not found', { status: 404 }) : cssResponse(cached, requestedHash)
+  }
+
+  const inflight = cssFallbackInFlight.get(cacheKey)
+  const promise = inflight ?? (async (): Promise<string | null> => {
+    try {
+      const content = await rebuildSiteCssFromSnapshot(db, bundleId, requestedHash, version)
+      if (cssFallbackCache.size >= CSS_FALLBACK_CACHE_MAX) cssFallbackCache.clear()
+      cssFallbackCache.set(cacheKey, content)
+      return content
+    } finally {
+      cssFallbackInFlight.delete(cacheKey)
+    }
+  })()
+  if (!inflight) cssFallbackInFlight.set(cacheKey, promise)
+
+  const content = await promise
+  return content === null ? new Response('Not found', { status: 404 }) : cssResponse(content, requestedHash)
+}
+
+/**
+ * Rebuild the requested CSS bundle file from the latest published snapshot.
+ * Returns the file body, or `null` when no page (nor the page-agnostic view)
+ * produces the requested hash. The page-invariant trio comes from the
+ * version-keyed memo, so only `userStyles` does per-page work here.
+ */
+async function rebuildSiteCssFromSnapshot(
+  db: DbClient,
+  bundleId: SiteCssBundleId,
+  requestedHash: string,
+  version: number,
+): Promise<string | null> {
+  const snapshot = await getLatestSnapshotForVersion(db, version)
+  if (!snapshot) return null
+
+  const pages = bundleId === 'userStyles' ? snapshot.site.pages : snapshot.site.pages.slice(0, 1)
+  for (const page of pages) {
+    const mediaAssets = await prefetchMediaAssets(page, snapshot.site, registry, db)
+    const file: CssBundleFile = buildPublishedSiteCssBundle(snapshot.site, registry, page, version, { mediaAssets })[bundleId]
+    if (file.hash === requestedHash) return file.content
+  }
+  // Page-agnostic view (every enabled stylesheet) — covers a hash that
+  // predates a scope change but is still referenced somewhere.
+  const fallbackMediaAssets = snapshot.site.pages[0]
+    ? await prefetchMediaAssets(snapshot.site.pages[0], snapshot.site, registry, db)
+    : undefined
+  const fallback: CssBundleFile = buildPublishedSiteCssBundle(
+    snapshot.site,
+    registry,
+    undefined,
+    version,
+    { mediaAssets: fallbackMediaAssets },
+  )[bundleId]
+  if (fallback.hash === requestedHash) return fallback.content
+
+  return null
+}
+
+function cssResponse(body: BodyInit, hash: string): Response {
+  return new Response(body, {
+    headers: {
+      'content-type': 'text/css; charset=utf-8',
+      'cache-control': 'public, max-age=31536000, immutable',
+      etag: `"${hash}"`,
+    },
+  })
+}