chore: clean up internal phase markers and reorganize scripts

The repo had grown two ingestion pipelines (PDF prose corpus and XSD
schema graph) without making the duality obvious. Header comments
referenced internal "Phase N" planning vocabulary that doesn't help
public readers, and a tool-output line shipped a forward reference
to a future phase to every MCP caller.

Reorganization:
  scripts/ingest/         -> scripts/ingest-pdf/   (was ambiguous)
  scripts/ingest-xsd/     stays
  scripts/fetch-xsd.ts    -> scripts/ingest-xsd/fetch.ts (sibling layout)
  scripts/sync-sources.ts -> scripts/sources-sync.ts (verb-style name)
  scripts/ingest-pdf/extract-pdf.py -> extract.py
  db/migrations/0003_phase3_metadata.sql -> 0003_xsd_metadata.sql
  scripts/ingest-xsd/smoke.ts            removed (debug-only, low value)

Renamed npm scripts to match the new directory layout:
  ingest          -> pdf:ingest
  ingest:chunk    -> pdf:chunk
  ingest:embed    -> pdf:embed
  ingest:upload   -> pdf:upload
  ingest:setup    -> pdf:setup
  db:sync-sources -> sources:sync
  xsd:smoke       removed

Strip "Phase N" markers from migration headers, source-file headers,
test-file headers, and inline comments. None of those references were
load-bearing; they were artifacts of the planning doc.

Drop the user-facing "_behavior notes: none yet (Phase 5)._" line that
shipped in every children/attributes/enum tool response. The line gave
no information when notes are absent and exposed an internal phase
label to the public.

Replace the lone PLAN.md reference in scripts/ingest-xsd/ingest.ts
with self-contained context. PLAN.md is gitignored; pointing at it
was a broken link for anyone reading the public repo.

Add scripts/ingest-pdf/README.md and scripts/ingest-xsd/README.md so
each pipeline is documented at the level that contributors land at,
and refresh CLAUDE.md to make the two corpora explicit and surface
both flavors of MCP tools.

41 / 0 across db / ingest / mcp-server.

Author: caiopizzol

CLAUDE.md

@@ -29,15 +29,23 @@ apps/
     src/data/docs.ts   ← All doc pages live here (single source of truth)
     src/components/    UI components (Sidebar, SuperDocPreview, etc.)
     src/pages/         Route pages (Home, Docs, SpecExplorer, Mcp)
-  mcp-server/          Cloudflare Worker — MCP server for AI spec search
+  mcp-server/          Cloudflare Worker - MCP server (semantic + structural tools)
 packages/
   shared/              Database client, embedding client, types
 scripts/
-  ingest/              PDF → chunks → embeddings → database pipeline
+  ingest-pdf/          ECMA PDF -> spec_content (semantic search corpus)
+  ingest-xsd/          ECMA XSDs -> schema graph (structural query corpus)
+  sources-sync.ts      data/sources.json -> reference_sources
+  db-migrate.ts        Apply db/migrations/*.sql in order
+  ooxml-call.ts        Local CLI harness for the structural MCP tools
 db/
-  schema.sql           PostgreSQL + pgvector schema
+  schema.sql           PostgreSQL + pgvector + XSD schema graph
+  migrations/          Numbered, idempotent SQL migrations
+data/
+  sources.json         Source manifest (artifact URLs, sha256, license notes)
+  xsd-cache/           Local-only XSD download cache (gitignored)
 dev/
-  data/                Extracted/chunked/embedded spec content
+  data/                Extracted/chunked/embedded PDF content
 ```
 
 ## Commands
@@ -97,23 +105,52 @@ The XML you provide is wrapped in a minimal `w:document > w:body` structure auto
 
 ## MCP Server
 
-Cloudflare Worker exposing three MCP tools for semantic spec search:
+Cloudflare Worker exposing two flavors of MCP tools backed by the same database:
 
-- `search_ecma_spec` — semantic vector search across 18,000+ spec chunks
-- `get_section` — fetch a specific section by ID (e.g., "17.3.1.24")
-- `list_parts` — browse the spec structure
+Always-on (semantic search over the spec PDF):
+
+- `search_ecma_spec` - semantic vector search across 18,000+ spec chunks
+- `get_section` - fetch a specific section by ID (e.g., "17.3.1.24")
+- `list_parts` - browse the spec structure
+
+Behind `ENABLE_OOXML_TOOLS` (structural queries over the XSD schema graph):
+
+- `ooxml_lookup_element` / `ooxml_lookup_type` - canonical symbol info
+- `ooxml_children` - legal children of an element/type/group, in document order
+- `ooxml_attributes` - attributes including those inherited and unfolded from attributeGroup refs
+- `ooxml_enum` - simpleType enumeration values
+- `ooxml_namespace_info` - vocabularies and per-profile symbol counts for a namespace URI
 
 Uses PostgreSQL with pgvector (Neon serverless in production, Docker locally).
 
-## Data Pipeline
+## Data Pipelines
+
+Two ingest paths feed the same database. Both are reproducible from `data/sources.json`.
 
-Ingests ECMA-376 PDFs into the vector database:
+**PDF (semantic corpus, into `spec_content`)**:
 
 ```
 PDF → extract (Python) → chunk (6KB) → embed (Voyage) → upload (PostgreSQL)
 ```
 
-Run the full pipeline: `bun scripts/ingest/pipeline.ts`
+```bash
+bun run pdf:ingest 1 ./pdfs/ECMA-376-Part1.pdf   # full pipeline for one part
+```
+
+See `scripts/ingest-pdf/README.md`.
+
+**XSD (structural corpus, into `xsd_*` tables)**:
+
+```
+ECMA Part 4 zip → fetch+verify (sha256) → parse → ingest (single transaction)
+```
+
+```bash
+bun run xsd:fetch  --url <part4-zip-url> --expected-sha256 <hex>
+bun run xsd:ingest
+```
+
+See `scripts/ingest-xsd/README.md`.
 
 ## Database
 

apps/mcp-server/src/index.ts

@@ -17,9 +17,10 @@ export interface Env {
 	DATABASE_URL: string;
 	VOYAGE_API_KEY: string;
 	/**
-	 * Phase 4 feature flag. Set to "true" to expose ooxml_lookup_element /
-	 * ooxml_lookup_type / ooxml_children / ooxml_attributes / ooxml_enum /
-	 * ooxml_namespace_info via tools/list and tools/call. Default off.
+	 * Feature flag for the OOXML structural tools. Set to "true" to expose
+	 * ooxml_lookup_element / ooxml_lookup_type / ooxml_children /
+	 * ooxml_attributes / ooxml_enum / ooxml_namespace_info via tools/list
+	 * and tools/call. Default off.
 	 */
 	ENABLE_OOXML_TOOLS?: string;
 }

apps/mcp-server/src/mcp.ts

@@ -162,9 +162,9 @@ async function handleToolsCall(
 	try {
 		let resultText: string;
 
-		// Phase 4 OOXML tools, feature-flagged. tools/list also gates on the same flag,
-		// so callers should not see these tool names unless the flag is on. Defensive
-		// check here in case a caller hand-crafts a request.
+		// OOXML tools are feature-flagged; tools/list filters them out when the flag
+		// is off, so callers should not see these tool names. Defensive check here in
+		// case a caller hand-crafts a request.
 		if (isOoxmlTool(name)) {
 			if (!ooxmlToolsEnabled(env)) {
 				return {

apps/mcp-server/src/ooxml-queries.ts

@@ -1,5 +1,5 @@
 /**
- * Read-only schema-graph queries powering the Phase 4 MCP tools:
+ * Read-only schema-graph queries powering the OOXML MCP tools:
  *   ooxml_lookup_element, ooxml_lookup_type, ooxml_children,
  *   ooxml_attributes, ooxml_enum, ooxml_namespace_info.
  *

apps/mcp-server/src/ooxml-tools.ts

@@ -1,14 +1,14 @@
 /**
- * Phase 4 read-only structural MCP tools. Behind ENABLE_OOXML_TOOLS env flag,
- * which gates both tools/list discovery and tools/call dispatch so the public
- * surface stays unchanged until the feature is intentionally enabled.
+ * Read-only structural MCP tools backed by the OOXML schema graph. Gated by
+ * ENABLE_OOXML_TOOLS, which filters both tools/list discovery and tools/call
+ * dispatch so the public surface stays unchanged until the flag is set.
  *
  * Tools:
  *   ooxml_lookup_element, ooxml_lookup_type, ooxml_children,
  *   ooxml_attributes, ooxml_enum, ooxml_namespace_info.
  *
- * Default profile is `transitional` until word-compatible-docx is composed
- * in Phase 6.
+ * Default profile is `transitional`. Future profiles (e.g. word-compatible-docx)
+ * will compose Transitional with Office extension schemas.
  */
 
 import { neon } from "@neondatabase/serverless";
@@ -315,8 +315,6 @@ function formatSymbolReport(label: string, hit: SymbolHit, profile: string): str
 	lines.push(`- namespace: ${hit.namespaceUri}`);
 	if (hit.typeRef) lines.push(`- type_ref: ${hit.typeRef}`);
 	if (hit.sourceName) lines.push(`- source: ${hit.sourceName}`);
-	lines.push("");
-	lines.push("_behavior notes: none yet (Phase 5)._");
 	return lines.join("\n");
 }
 
@@ -340,8 +338,6 @@ function formatChildrenReport(
 
 	if (children.length === 0) {
 		lines.push("_no direct or inherited children._");
-		lines.push("");
-		lines.push("_behavior notes: none yet (Phase 5)._");
 		return lines.join("\n");
 	}
 
@@ -359,8 +355,6 @@ function formatChildrenReport(
 	lines.push(
 		"_group entries are returned as-is; call `ooxml_children` on the group qname to expand them._",
 	);
-	lines.push("");
-	lines.push("_behavior notes: none yet (Phase 5)._");
 	return lines.join("\n");
 }
 
@@ -383,8 +377,6 @@ function formatAttributesReport(
 
 	if (attrs.length === 0) {
 		lines.push("_no attributes._");
-		lines.push("");
-		lines.push("_behavior notes: none yet (Phase 5)._");
 		return lines.join("\n");
 	}
 
@@ -401,8 +393,6 @@ function formatAttributesReport(
 			`| ${a.localName} | ${a.attrUse} | ${a.typeRef ?? "-"} | ${a.defaultValue ?? "-"} | ${a.fixedValue ?? "-"} | ${from} |`,
 		);
 	}
-	lines.push("");
-	lines.push("_behavior notes: none yet (Phase 5)._");
 	return lines.join("\n");
 }
 
@@ -420,8 +410,6 @@ function formatEnumReport(sym: SymbolHit, enums: EnumEntry[], profile: string):
 	} else {
 		for (const e of enums) lines.push(`- ${e.value}`);
 	}
-	lines.push("");
-	lines.push("_behavior notes: none yet (Phase 5)._");
 	return lines.join("\n");
 }
 

data/sources.json

@@ -1,5 +1,5 @@
 {
-  "$comment": "Source manifest. Human-edited; scripts/sync-sources.ts upserts these rows into reference_sources.",
+  "$comment": "Source manifest. Human-edited; scripts/sources-sync.ts upserts these rows into reference_sources.",
   "sources": [
     {
       "name": "ecma-376",

db/migrations/0001_reference_sources.sql

@@ -1,5 +1,4 @@
--- Phase 1: Provenance foundation
--- Adds reference_sources and source_id FK on spec_content.
+-- Provenance foundation: reference_sources catalog + source_id FK on spec_content.
 -- Idempotent: safe to run against fresh installs (matches db/schema.sql) or existing DBs.
 
 CREATE EXTENSION IF NOT EXISTS vector;

db/migrations/0002_xsd_schema.sql

@@ -1,5 +1,5 @@
--- Phase 2: XSD schema tables (empty)
--- Profile-scoped symbol graph. All tables empty after this migration; data lands in Phase 3+.
+-- Profile-scoped XSD schema graph. All tables empty after this migration; data
+-- lands when scripts/ingest-xsd/ingest.ts runs against a populated cache.
 -- Idempotent: safe to run against fresh installs (matches db/schema.sql) or existing DBs.
 
 CREATE TABLE IF NOT EXISTS xsd_profiles (
@@ -116,7 +116,8 @@ CREATE TABLE IF NOT EXISTS xsd_enums (
 );
 
 -- Curated Word/Office behavior claims keyed to symbols.
--- claim_type enum is locked now (Phase 5 will populate).
+-- claim_type enum is locked now; the table stays empty until curated behavior
+-- notes start landing.
 CREATE TABLE IF NOT EXISTS behavior_notes (
 	id SERIAL PRIMARY KEY,
 	symbol_id INT REFERENCES xsd_symbols(id) ON DELETE CASCADE,

db/migrations/0003_phase3_metadata.sql db/migrations/0003_xsd_metadata.sqldb/migrations/0003_phase3_metadata.sql renamed to db/migrations/0003_xsd_metadata.sql

@@ -1,4 +1,6 @@
--- Phase 3 review fix: preserve element/attribute @type and group-ref compositor context.
+-- Preserve element/attribute @type and group-ref compositor context so the
+-- structural lookup tools can resolve element-to-type chains and attach refs
+-- to their enclosing compositor.
 -- Idempotent.
 
 ALTER TABLE xsd_symbols

db/migrations/0004_local_element_scoping.sql

@@ -1,4 +1,4 @@
--- Phase 4 review fix: scope local element symbols by their owner.
+-- Scope local element symbols by their owner.
 --
 -- Before this migration, an inline <xsd:element name="X" type="..."/> declared
 -- inside two different complexTypes/groups collapsed to a single symbol keyed