diff --git a/docs/index.md b/docs/index.md index e2af04e..1039b0f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -64,6 +64,7 @@ explicit schemas and `(file, page, excerpt)` provenance for agents. - [`operations/hosting-personal.md`](operations/hosting-personal.md) — personal hosting checklist - [`operations/highsignal-integration.md`](operations/highsignal-integration.md) — fleet consumer integration notes - [`operations/jobs.md`](operations/jobs.md) — async ingestion (Queues + Workflows); no scheduled jobs +- [`operations/automation-inventory.md`](operations/automation-inventory.md) — automation inventory and Foundry evidence contract (auth-safe health, sanitized evidence, storage ownership, maintenance-only authority) ### Knowledge diff --git a/docs/operations/automation-inventory.md b/docs/operations/automation-inventory.md new file mode 100644 index 0000000..0df5a9b --- /dev/null +++ b/docs/operations/automation-inventory.md @@ -0,0 +1,168 @@ +--- +title: Automation inventory and Foundry evidence contract +description: Auth-safe health, sanitized API/search evidence, ingestion/index lifecycle, storage ownership, and maintenance-only authority for the Knowledge Base RAG Worker. Consumed by the Foundry coverage audit. +--- + +# Automation inventory and Foundry evidence contract + +> Project-side contract for the **`ai-infrastructure-toolbox-automation`** +> capability. Code is authoritative for behavior (`cloudflare/worker/src/`, +> `cloudflare/worker/wrangler.jsonc`); this page records the sanitized evidence +> and recovery ownership consumed by Foundry. The central capability +> specification moves with the Foundry control plane and is not duplicated +> here. + +Knowledge Base is **maintenance-only** Toolbox infrastructure and the only +fleet `RAG_SERVICE`. This inventory exists so fleet automation can verify +auth, ingestion/index, storage, and cited-search health without issuing +private corpus queries, logging retrieved chunks, or expanding product scope. + +## Maintenance-only authority + +| Action | Allowed without approval | Requires approval | +| --- | --- | --- | +| Read-only health/auth/metadata probes (`/healthz`, `/readyz`, `/metrics`, 401 probes) | yes | — | +| Bounded queue retry within existing policy | yes (already enforced in code) | — | +| Registry-approved idempotent re-ingest via existing routes | yes | — | +| Rotate `RAG_SERVICE_KEYS`, change providers, change spend | no | yes | +| Change rate limits, D1 schema, Vectorize index metadata, or migrations | no | yes | +| Production deploy or `wrangler d1 migrations apply` | no (manual `pnpm deploy`) | yes | +| New public corpus, schema template, or product surface | no | yes | +| Recreate a sibling `rag-service` | no (release gate `audit:sibling-rag-service --require-retired`) | n/a | + +Automation MUST record a PR + pending deploy approval when a critical patch +passes checks. It MUST NOT deploy, migrate, rotate credentials, or recreate +the retired sibling RAG service. + +## APIs, Worker, and auth + +| Surface | Route | Auth | Sanitized evidence | +| --- | --- | --- | --- | +| Health | `GET /v1/healthz`, `GET /healthz` | **public** (before `requireServiceKey`) | `ok`, `d1`, `d1_schema`, `vectorize`, `r2`, `version`, `deploy_fingerprint` — no corpus query | +| Readiness | `GET /readyz` | **public** | `status: ok\|degraded`, db/vector/object sub-status, worker version + fingerprint | +| Metrics | `GET /metrics` | **public** | Prometheus-format gauges: `kb_worker_info`, `kb_worker_ready`, `kb_d1_schema_ready`, isolate counters — no corpus content | +| Testing UI | `GET /`, `GET /ui` | **public** (operator HTML) | no private data | +| Legacy route aliases | `app.all('*')` → `/v1/kb/*` | public for unauthenticated 401 probe; `requireServiceKey` after | 401 without key, redirected to `/v1/kb/*` | +| Projects | `GET/POST /v1/kb/projects`, `GET /v1/kb/projects/:project/status` | `RAG_SERVICE_KEYS` (X-RAG-Key or Bearer) | project name, domain count, corpus status — no document contents | +| Indexes | `GET/POST /v1/indexes`, `DELETE /v1/indexes/:id`, `POST /v1/indexes/:id/ingest[-vectors]`, `GET /v1/indexes/:id/documents`, `POST /v1/indexes/:id/query[-vector\|-benchmark]` | service key | index metadata, doc counts, vector counts — no chunk text in list views | +| Embedding models | `GET /v1/embedding-models` | service key | model id, provider, dimensions, enabled flag | +| Domains | `GET/POST /v1/kb/domains` | service key | domain name, embedding model — no corpus text | +| Schemas | `GET/POST /v1/kb/schemas`, `GET /v1/kb/schemas/:domain/active`, `POST /v1/kb/schemas/:domain/reprocess`, `POST /v1/kb/schemas/infer[-upload]`, `GET/POST /v1/kb/schemas/drafts[/:draft_id/(apply\|discard)]` | service key | schema spec, draft status — no source document bodies | +| Files | `GET/POST /v1/kb/files`, `GET /v1/kb/files/:file_id`, `POST /v1/kb/files/:file_id/reprocess`, `DELETE /v1/kb/files/:file_id`, `POST /v1/kb/files/upload` | service key | file metadata, status, last_error (truncated 500 chars) — no chunk text | +| Status / jobs | `GET /v1/kb/status`, `GET /v1/kb/jobs`, `GET /v1/kb/ingest/jobs/:job_id`, `GET /v1/kb/ingest/runs/:run_id` | service key | job status, stage, attempts, `failure_classification` — no document contents | +| Sources | `GET /v1/kb/sources`, `POST /v1/kb/sources/import`, `GET/POST /v1/kb/source-sets[/:id/actions]` | service key | source metadata, action receipts — no corpus text | +| Ingest | `POST /v1/kb/ingest/record`, `POST /v1/kb/ingest/text`, `POST /v1/kb/ingest/run` | service key | job id, enqueued count, queue backlog, workflow instance id — no source text in durable evidence | +| Parse artifacts | `GET /v1/kb/parse-artifacts/:hash` | service key | parse output keyed by content hash (operator-only) | +| Entities / relationships | `GET/POST /v1/kb/entities[/:id/(lineage\|relationships\|search)]`, `GET/POST /v1/kb/relationships[/backfill]` | service key | entity metadata, relationship edges — no chunk text | +| Search / query | `POST /v1/kb/search`, `POST /v1/kb/query`, `POST /v1/kb/query/stream` | service key | cited `(file_id, page, excerpt)` per "cited or it didn't happen"; **durable evidence (job/trace routes) excludes retrieved chunk text** | +| Sessions | `GET/POST /v1/kb/sessions[/:id[/messages]]` | service key | session metadata, message metadata — no source chunks in trace exports | +| Traces / evals | `GET /v1/kb/query/traces[/export\|/compare\|/:id[/drilldown]]`, `POST /v1/kb/evals/(search\|parse\|query)`, `GET /v1/kb/evals/(reports\|summary\|reports/:id)` | service key | trace metadata, eval metrics (hit/citation rates, latency p95s) — no prompt or chunk text in eval reports | + +**Fail-closed auth:** `app.use('/v1/*', requireServiceKey)` runs after the +public health routes. Invalid/missing key → `401`. `RAG_SERVICE_KEYS` is a +JSON map of `{key: tenant}`; `RAG_SERVICE_KEYS_APPEND` and +`RAG_SERVICE_PROOF_KEYS` extend it. Constant-time comparison. + +**Cited-or-it-didn't-happen invariant:** every retrieval/answer route +terminates at a retrievable `(file_id, page, excerpt)` triple. Durable +evidence (job/trace/eval routes) stores metadata + latency + failure class +only — **no prompts, retrieved chunks, or corpus content**. + +## Background and index lifecycle + +Authoritative config: `cloudflare/worker/wrangler.jsonc` (`queues`, +`workflows`) and `cloudflare/worker/src/index.ts` (`KbIngestWorkflow`, +`processIngestQueue`). See also [`operations/jobs.md`](jobs.md). + +**No scheduled (cron) jobs.** The Worker defines no `[triggers]` / `scheduled` +handler. The only async path is ingestion via Queues + Workflows. + +| Job | Trigger | Bounds | Timeout | Concurrency | Retries | Idempotency / dedup | Failure state | Owner | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Ingest queue consumer | `POST /v1/kb/ingest/run` (default async) | `max_batch_size: 5`, `max_batch_timeout: 10` | parse 45s (`INGEST_PARSE_TIMEOUT_MS`); Workflow enqueue step 1m | 1 consumer | queue `max_retries: 3`; poison-input ack after `INGEST_QUEUE_MAX_ATTEMPTS: 5`; per-file `message.retry` exponential backoff capped at 300s | chunk IDs deterministic (`deterministicId('chk', tenant:indexId:external_id:i:content)`); D1 `INSERT OR REPLACE` by chunk ID; Vectorize upsert by vector ID; parse artifact keyed by content hash; `idempotency_key` + `idempotent_replay` on ingest routes | job `status='failed'` + `last_error` + `attempts` in D1; `failure_classification` on job detail route | `cloudflare/worker/src/index.ts` `processIngestQueue`, `runKbIngest` | +| Ingest workflow | same trigger, Workflow-backed | 1 enqueue step | 1m step timeout | 1 instance per run | Workflow step retries | durable run id in `kb_ingest_jobs.workflow_id` | workflow instance status | `KbIngestWorkflow` | +| Overlap control | per-file lease | `INGEST_JOB_LEASE_MS: 5m` | n/a | per locked_by | n/a | `isIngestJobLeaseActive` skips files with `status='running'` + `locked_by != current` + `locked_at` within 5m | skipped → `reason: 'lease_active'` | `runKbIngest` | +| Failure classification | every ingest failure | n/a | n/a | n/a | n/a | n/a | `classifyIngestFailure` → `{category, retryable, message (≤500 chars)}`: `embedding_readiness`, `missing_source_object` (not retryable), `parse_empty` (not retryable), `parse_timeout`, `validation` (not retryable), `unknown` | `classifyIngestFailure` | +| Replay | `POST /v1/kb/files/:file_id/reprocess`, `POST /v1/kb/schemas/:domain/reprocess`, `POST /v1/kb/source-sets/:id/actions` (`requeue_all\|requeue_failed\|requeue_pending`) | bounded by pending file set | n/a | n/a | re-queue | idempotent re-ingest by content hash | resets file to `pending` and re-queues | reprocess routes | +| Query cache | `RAG_SHARED_QUERY_CACHE_ENABLED`, `RAG_SHARED_EMBEDDING_CACHE_ENABLED` | `RAG_CACHE_TTL_SECONDS: 300`, `RAG_CACHE_MAX_ENTRIES: 1000` | n/a | per-isolate | n/a | lexical precheck before embedding/Vectorize | cache miss → cold path | `src/cache.ts` | +| CI gate | GitHub Actions | push/PR | job-level | 1 | n/a | n/a | red CI | `cloudflare/worker/.github/workflows/*` | +| Deploy | manual `pnpm deploy` (`wrangler deploy`) | manual | job-level | 1 | n/a | n/a | blocked deploy | operator | + +**Freshness:** the deployed corpus is **opt-in**. There is no scheduled +freshness window because corpora do not auto-refresh. `corpusStatus` reports +per-domain `state` (`no_schema`, `schema_ready`, `schema_draft`, +`files_staged`, `ready`, `ingesting`, `failed`). The Foundry snapshot treats +"freshness" as **on-demand by explicit ingestion run** — a domain in +`ready` with no recent ingest job is not stale, it is *stable*. A domain in +`ingesting` with no recent job progress is *stale* and MUST be reported. + +## Storage ownership and recovery + +| Store | Binding | Owner | Authoritative source | Backup / export / reconstruction | Migration guard | Last verification | +| --- | --- | --- | --- | --- | --- | --- | +| D1 `rag-db` | `DB` | this Worker | migrations `0001`–`0007` (`cloudflare/worker/migrations/`) | Cloudflare D1 dashboard export; metadata reconstructable from R2 raw docs + re-ingest | `pnpm check` + `pnpm run audit:d1-migrations` gate; `wrangler d1 migrations apply` is manual and requires approval | `/v1/healthz` `d1_schema: true` | +| R2 `rag-raw-docs` | `RAW_DOCS` | this Worker | uploaded source files | R2 is the authoritative source for raw documents; re-ingest from R2 rebuilds D1 metadata + Vectorize vectors | n/a (source of truth) | `/v1/healthz` `r2: true` | +| Vectorize `rag-gemini-1536` | `VECTORIZE` | this Worker | derived from R2 + D1 | **reconstructable**: re-ingest from R2 raw docs via `POST /v1/kb/files/:file_id/reprocess` or `POST /v1/kb/schemas/:domain/reprocess` | `pnpm run audit:vectorize-embedding-bindings` + `audit:vectorize-metadata-indexes` | `/v1/healthz` `vectorize: true` | +| Vectorize `rag-embedding-384/768/1024` | `VECTORIZE_384/768/1024` | this Worker | derived from R2 + D1 | same reconstruction path | same audit gates | `/readyz` `vector.ok` | +| Workers AI | `AI` | this Worker | Cloudflare | n/a (managed service) | n/a | `/v1/healthz` via embedding smoke | +| Free AI gateway | `FREE_AI` service binding | this Worker (consumer) | `free-ai` repo | n/a (delegated to free-ai) | `pnpm run audit:free-ai-embedding-contract` | `/v1/embedding-models` live | +| Analytics Engine `knowledgebase_rag_events` | `RAG_ANALYTICS` | this Worker | per-query events | ephemeral; not a source of truth | n/a | n/a | +| Service keys | `RAG_SERVICE_KEYS` secret | operator | wrangler secret | `wrangler secret put`; never commit | pre-push secret scan | `readiness:auth` | +| Demo corpora | `domains/sec/`, `domains/legal/` | operator (opt-in) | explicit ingestion run | re-run ingestion from source files | n/a | `smoke:rag-crud:embedding-model` | + +**Private data exclusion:** durable evidence (job/trace/eval routes, logs) +stores **no prompts, retrieved chunks, source document text, or authorization +headers**. `last_error` is truncated to 500 chars. Foundry snapshots MUST +NOT copy corpus content, retrieved chunks, or any credential-shaped value. + +## Deploy path + +- **Target:** Cloudflare Worker `knowledgebase`, + `https://knowledgebase.sarthakagrawal927.workers.dev` + custom domain + `search.sassmaker.com`. +- **Trigger:** manual `pnpm deploy` (`wrangler deploy`). `main` is releasable + and green but is **not** an automatic production trigger. +- **Pre-deploy gate:** `pnpm run predeploy:local` (the full local pre-deploy + gate). +- **Deploy fingerprint:** `WORKER_DEPLOY_FINGERPRINT = + 'knowledgebase-a-plus-evidence-2026-06-23'`; enforced by + `smoke:legacy-routes` and `readiness:full-port`. +- **Post-deploy smoke:** `curl /v1/healthz`; `pnpm run readiness:auth` with a + service key; `pnpm run smoke:legacy-routes --require-complete`. +- **Never** run `wrangler deploy` or `wrangler d1 migrations apply` from an + agent session. + +## Foundry evidence contract + +The Foundry coverage audit (`pnpm fleet:ai-infra-audit` in `saas-maker`) +reads this inventory and the live routes below to emit a sanitized snapshot. +The snapshot contains **only**: route name, HTTP status, dependency +readiness (D1/R2/Vectorize), deploy fingerprint, job lifecycle bounds, +ingest failure classification counts, corpus state, storage ownership, and +recovery owner. It contains **no** prompts, retrieved chunks, source +document text, corpus content, service keys, or any credential-shaped +string. + +| Contract field | Source | Freshness window | +| --- | --- | --- | +| `auth_safe_health` | `GET /v1/healthz` (public, no corpus query) + 401 probe on `/v1/indexes` | per-request | +| `structured_api_evidence` | `GET /v1/kb/jobs` + `GET /v1/kb/status` (failure_classification, no chunk text) | per-request | +| `provider_cost_and_degradation` | n/a (KB has no direct provider spend; embedding/synth delegated to `free-ai` and reported there) | n/a | +| `background_and_index_lifecycle` | this page + `GET /v1/kb/ingest/runs/:run_id` (workflow status) | per-request | +| `storage_ownership_and_recovery` | this page table + `/readyz` dependency flags | per-deploy | +| `maintenance_only_authority` | this page table | per-deploy | + +## Accepted exceptions and blockers + +- **S-grade proof** is blocked on `KARTE_SESSION_COOKIE` / + `STARBOARD_SESSION_COOKIE` for authenticated consumer smokes. Not an + automation gap; documented in `STATUS.md`. +- **AI Gateway cache** is implemented but not enabled (Wrangler OAuth blocked + gateway creation at the time). Revisit when unblocked. +- **Semantic p99 < 300 ms on completely unique cold misses** is deferred; + query/embedding caches cover hot/repeated questions, not first-seen misses. +- **Public multi-user hosting** is deferred until auth, per-project + authorization, upload limits, rate limits, job cancellation, backup drills, + and log redaction land. +- **`/api/ai` vs `/api-ai.json` mismatch** in `app/public/` is a known docs + issue (see `STATUS.md` Unresolved questions); not an automation gap.