diff --git a/AGENTS.md b/AGENTS.md index 686ca7322..f1bcd005f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,12 +14,23 @@ supporting literature and aggregate evidence. Event Lineage (reconstructed post-to-post parents) is distinct from the typed ontology neighborhood (ADR 0184); source-window continuation is ADR 0124. Do not mix those graphs. +ADRs are the normative source. Research notes, implementation matrices, +runtime-evidence files, and Storybook inventories stay supporting +documents unless an ADR explicitly promotes a decision from them -- +[`docs/adr/README.md`](docs/adr/README.md) maps each supporting document +to its governing ADR. Update research notes as literature changes; never +use them to introduce an untracked architecture decision. + ## Hard rule: no real data in repository artifacts This repository ships **synthetic fixtures only** (`lineageweave/fixtures.py`) and must never commit or expose, by name or otherwise identifiably, any real organization's records. Never add a real record to a fixture, test case, -screenshot, example, log, benchmark artifact, or documentation. +screenshot, example, log, benchmark artifact, or documentation. This includes +audit snapshots, gap baselines, PR/issue inventories, and runtime-evidence +documents (for example `docs/product-technical-gap-baseline.md`): aggregate +counts and PR numbers are fine; identifying post identifiers, organization +names, and production record keys are not (ADR 0001). The private runtime is different: the product is expected to read an authorized real PostgreSQL source through its configured import/data boundary. @@ -179,6 +190,39 @@ adjudication does -- never a raw LLM API. Demo TEPP seed goes through envelope is Failed (`tepp_not_available` / `tepp_result_not_persisted`), never a fabricated theta or a local psychometric substitute. +The lineage `text` channel follows [ADR 0190](docs/adr/0190-lineage-text-channel-embedding-swap.md): +when an embedding provider is configured, `reconstruct()` precomputes +batched label embeddings once per reconstruction and scores cosine +similarity; `difflib` character overlap is only the fallback for a pair +whose vector is missing. Clamp the raw cosine into `[0, 1]` -- do not +remap it from `[-1, 1]`, because real sentence embeddings occupy an +anisotropic cone (unrelated pairs already score a modestly positive +cosine) and the remap manufactures false weak positives. A missing +vector degrades that pair back to difflib; it never fabricates a score. + +## Measurement boundary + +- Channel-weight estimation stays unavailable until an independent + lineage anchor exists ([ADR 0145](docs/adr/0145-psychometric-channel-weight-estimation.md)). + Do not fit an unanchored IRT model over candidate pairs and promote its + latent factor to "relatedness"; `scripts/estimate_channel_weights.py` + must keep exiting without writing. The hand-picked constants in + `DEFAULT_CHANNEL_WEIGHTS` are an explicitly ungrounded compatibility + fallback -- never present them as calibrated or paper-grounded + measurement. +- Persisted channel-weight vectors fail closed: malformed, + mixed-provenance, or unsupported-anchor rows are ignored, not + renormalized or repaired. +- Caller-mapped grouping values stay raw in their source-provenance + columns across backfill and re-import; normalization happens only in + derived reconstruction fields. +- Per-edge channel breakdowns persist to nullable + `post_lineage_edge.channel_scores` jsonb ([ADR 0195](docs/adr/0195-lineage-edge-channel-scores-persistence.md)): + a missing breakdown means the edge predates the column -- an honest + unknown, never zero-filled or reconstructed. It exists for direct + database diagnosis only; exposing it through an API or UI needs its own + decision first. + ## Tests ```bash @@ -195,6 +239,25 @@ in the same spirit) -- never against real data, per the hard rule above. against a live local stack (`make up`) and self-skip without one -- see [README.md](README.md#local-product-stack-docker-compose). +`tests/test_public_docstrings.py` enforces repository-wide docstring +coverage: every public function and class under `lineageweave/` and +`backend/app/` (non-underscore names, `__init__.py` excluded) carries a +docstring. Ship new public definitions documented; the gate fails the PR +otherwise. + +### Schema migrations + +Migrations 0001–0011 are the non-idempotent image bootstrap and replay +only on an empty data directory. Everything numbered 0012 or later must be +safe to replay and is applied automatically by +`docker/postgres-init/migrate.sh` in sorted filename order -- one fixed +lower-bound pattern, no per-file allowlist ([ADR 0166](docs/adr/0166-idempotent-migration-replay-window.md)). +Prefer native idempotency (`IF NOT EXISTS`, `ON CONFLICT`); a migration +that cannot be made idempotent requires a migration-ledger ADR first. +Each accepted file runs with `psql -X -v ON_ERROR_STOP=1`, so a failure +stops startup instead of leaving a healthy-looking partial schema, and +application code must not compensate for a missing table. + Period leftover pairs (ADR 0017 / 0018 / 0048 / 0049 / 0119 / 0162 / 0163 / 0164 / 0182) are computed in `lineageweave/leftover_pairs.py` from the residual after a real GRM/GPCM score, never invented. Distances are @@ -220,9 +283,14 @@ floating Node version): ```bash cd frontend && pnpm install pnpm run lint && pnpm run test && pnpm run build -# Storybook inventory (ADR 0020 tokens): pnpm run build-storybook +# Storybook inventory: pnpm run build-storybook ``` +Repeated web objects use `frontend/src/styles/tokens.css`, not inline hex +(ADR 0099 badge/accent tokens, with dark-mode overrides guarded by +`tokens.test.ts`); new stories belong in the inventory at +`docs/storybook-inventory.md`. + A run-bearing analysis-run registry empties only after an unrevoked `analysis_run_retention_grant` and `GRANT analysis_run_retention_admin` (ADR 0020 / v0.87.0). The documented phrase is not a secret. Do not @@ -256,6 +324,7 @@ exist on the post. Do not weaken, skip, or `continue-on-error` a failing check -- fix the underlying cause or, for a genuine false positive in a third-party scanner, add a narrow, documented suppression referencing the specific finding. + ## W3C PROV-O boundary - Add standard provenance through `lineageweave.prov_o` and the @@ -266,3 +335,16 @@ add a narrow, documented suppression referencing the specific finding. - Appendix B inverse names normalize to the preferred W3C direction; do not proliferate private inverse vocabulary. - Keep `knowledge_graph_edge` an explicit navigation projection. + +## Ontology publication boundary + +The ontology namespace publishes as a deterministic GitHub Pages artifact +([ADR 0159](docs/adr/0159-published-ontology-pages.md)): render through +`scripts/build_ontology_site.py` (same source tree produces the same +bytes -- no build timestamps, source SHA-256 manifest) and deploy only +through the fail-closed `scripts/publish_ontology_site.py` from `main`. +A manual dispatch from another ref is not a publication path. The +lowercase and repository-case public namespace IRIs remain a tracked +interoperability gap ([ADR 0157](docs/adr/0157-public-ontology-namespace-identity.md), +issue #372): do not silently rewrite either form; any namespace migration +is a versioned ADR with compatibility mappings first. diff --git a/CLAUDE.md b/CLAUDE.md index 992753de8..f32325abc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,77 +1,56 @@ # CLAUDE.md Tool-specific pointer. Policy lives in [AGENTS.md](AGENTS.md) and the -ADRs under `docs/adr/`. Do not fork those rules here. +ADRs under `docs/adr/`. Do not fork those rules here. The sections below +are only the operational details Claude is asked for most often; every +rule they reference is stated once in AGENTS.md with its ADR. -## Analysis-run retention (v0.87.0) +## Analysis-run retention purge (ADR 0020) To empty a run-bearing registry, insert an unrevoked `analysis_run_retention_grant` for `session_user` and -`GRANT analysis_run_retention_admin` (ADR 0020). Then +`GRANT analysis_run_retention_admin`, then `select purge_analysis_run_registry('approved-retention-purge')`, export `analysis_run_retention_event`, delete those rows, and roll back 0020 then 0018. The published phrase is not a secret. Do not `DISABLE TRIGGER` as superuser. Do not grant the admin role or a retention grant to the application `DATABASE_URL` login. ADR 0019 -is the R&R catalog-id bind, not this purge. Person catalog identity -on that role row is ADR 0027 (`cataloged_person_id`). +is the R&R catalog-id bind, not this purge; person catalog identity +on that role row is ADR 0027 (`cataloged_person_id`). Purge never +sits on a public HTTP route. -## Analysis-run seed (v0.96.0) +## Analysis-run seed and run states (ADR 0013 / 0014 / 0024) `make seed` writes a Demo Corp lineage run, a TEPP run, and a Succeeded -period-report run on the same snapshot (ADR 0013 / ADR 0024). The TEPP path goes through `tepp_client`. A missing -transport or an unused accepted envelope is Failed -(`tepp_not_available` / `tepp_result_not_persisted`). Do not invent a -theta or a local psychometric substitute. The home list caption stays -`kind · status · entity`; the machine failure code is detail-only -(ADR 0014). Open a Failed TEPP row, then connect a live TEPP -transport. A failed lineage row retries reconstruction -- it does not -mention TEPP. A failed period-report row rebuilds the report. A -pending TEPP row does not claim a calibrated measurement. A pending -lineage row says reconstruction has not started yet. -Digest prefixes stay audible; hover a prefix to read the full digest. -Opening a cutoff title shows the live post. Titles marked updated -after cutoff were rewritten after the run; the opened body names -both clocks and shows **Body this run knew** beside the live -rewrite. Compare those two texts before treating the live body as -reconstructed evidence (ADR 0016 / 0025). -`POST /api/analysis-runs` records Pending lineage only on an -authorized cutoff capture (ADR 0017). TEPP and period-report kinds -are 422. The Request button waits until affiliated corps load; choose -a corp if the token walks more than one. `POST /api/analysis-runs/{id}/start` -commits Running plus a durable outbox row, then reconstructs that -frozen cutoff bag (ADR 0021 / ADR 0023) or submits TEPP through -`tepp_client` (ADR 0022). A missing transport or unused accepted -envelope is Failed. Failed TEPP is terminal — connect a TEPP -transport from that Failed row. Create does not invent a Pending -TEPP row. Do not invent a theta. Hover the Result prefix to read -the parent-choice digest. -After `make seed`, open **Period report · Succeeded · Demo Corp**, -then **Open period report 2026-W02**. The home week is already -2026-W02, so the grouping comparison strip lands on Demo Corp. Report -grouping is Corporate entity and Demo Corp is current. The focused -chip name contains `Corporate entity: Demo Corp` and the persisted -mean θ. The period-report panel says Demo Corp is the opened grouping -and to read its mean θ and member posts, then open a post. Those -members land immediately under that next action, ahead of Other Corp -and the week strip. After `make seed`, leftover closest/farthest pairs -sit above the member list with leftover-map rank (rank 0 names no -leftover structure) and unexplained leftover `U` next to leftover-map -distance `d`. Leftover-map axis share badges name Gabriel inertia -of axes 1 and 2; open a leftover pair to read the post–criterion cell. -The shares do not invent a leftover score. Opening Public post names the next action: read -Event Lineage, Keyman, and evaluation on that post. The popup Event -Lineage DAG marks that post current. After that current node, the -popup names Keyman and evaluation as the next read. After landed -evaluation, the popup names the first Keyman as the next read. After -landed Ada West related, the popup names the first related node as -the next read. After that next action, the popup lands Priya Nair -related nodes. After those related nodes land, the popup names Ask -about this lineage as the next read. After that next action, the -popup lands Ask about this lineage. After landed chat, the popup names -the first Ask. After that next action, the popup lands the first Ask -answer. After landed first Ask answer, the popup names the first -cited source. After that next action, the popup lands the first cited -evidence. Changing the week first still -focuses the report period field. Mean θ stays on the period-report -panel. +period-report run on one snapshot. The TEPP path goes through +`tepp_client`: a missing transport or an unused accepted envelope is +Failed (`tepp_not_available` / `tepp_result_not_persisted`). Never +invent a theta or a local psychometric substitute. + +- Failed TEPP is terminal -- open that row and connect a live TEPP + transport from it. +- A failed lineage row retries reconstruction and does not mention TEPP; + a failed period-report row rebuilds the report. +- Pending rows claim nothing: pending TEPP is not a calibrated + measurement, pending lineage has not started reconstructing yet. +- Home list captions stay `kind · status · entity`; machine failure + codes are detail-only (ADR 0014). + +## Cutoff knowledge (ADR 0016 / 0025) + +Opening a cutoff-rewritten title shows **Body this run knew** from +`source_post_revision` beside the live rewrite, with both clocks named. +Compare those two texts before treating the live body as reconstructed +evidence; do not invent an earlier sentence when no revision covers the +cutoff. + +## Where the rest lives + +Create/start endpoint rules (ADR 0017 / 0021), tie-vs-miss similarity +(ADR 0026), R&R catalog ids (ADR 0019 / 0027), leftover pairs +(ADR 0048–0164 / 0182), the text-channel embedding swap and cosine +clamp (ADR 0190), per-edge channel-score persistence (ADR 0195), +migration replay (ADR 0166), docstring coverage, and the measurement +boundary are all stated in [AGENTS.md](AGENTS.md) -- read it before +changing code, tests, or runtime policy rather than restating anything +here.