diff --git a/docs/adr/0064-lineage-evidence-and-tree-assembly.md b/docs/adr/0064-lineage-evidence-and-tree-assembly.md index aabe37b5e..6e6346699 100644 --- a/docs/adr/0064-lineage-evidence-and-tree-assembly.md +++ b/docs/adr/0064-lineage-evidence-and-tree-assembly.md @@ -2,6 +2,8 @@ - Status: Accepted - Date: 2026-08-19 +- Amended by: [ADR 0200](0200-channel-weight-reconciliation.md) and + [ADR 0205](0205-tepp-lineage-anchor.md) ## Context @@ -16,9 +18,10 @@ or promoting an inferred relation to fact. - Treat every input record as a fallible mention and every accepted edge as a lineage instance supported by evidence, not as a proven business fact. - Fuse independent temporal, secondary-key, text/embedding, and optional LLM - channels through the RankWeave weighted convex fusion contract. A missing - channel is dropped and weights are renormalized; it is never replaced with a - fabricated negative or score. + channels through the RankWeave weighted convex fusion contract. Resolve a + missing channel before loading the exact calibrated active-channel vector; + never repair or renormalize a vector estimated for another channel set, and + never replace a missing channel with a fabricated negative or score. - Keep the channel-score breakdown and provenance on every candidate decision. Candidates below the minimum fused-score floor remain roots rather than being force-attached. diff --git a/docs/adr/0172-event-lineage-channel-evidence.md b/docs/adr/0172-event-lineage-channel-evidence.md index 717fe61ec..b95ec5d91 100644 --- a/docs/adr/0172-event-lineage-channel-evidence.md +++ b/docs/adr/0172-event-lineage-channel-evidence.md @@ -3,6 +3,8 @@ **Status:** Accepted **Date:** 2026-08-21 **Issue:** [#274](https://github.com/ContextualWisdomLab/LineageWeave/issues/274) +**Amended by:** [ADR 0200](0200-channel-weight-reconciliation.md) and +[ADR 0205](0205-tepp-lineage-anchor.md) ## Context @@ -33,8 +35,9 @@ authoritative; PROV-O/RDF export is a projection. `lineage_signal_temporal`, `lineage_signal_secondary_key`, `lineage_signal_text`, `lineage_signal_llm`. The LLM row is omitted when the adjudication client is unavailable; it is never fabricated. -3. Weights are the normalized active weights actually used - (`reconstruct.active_weights`). Contribution is `weight * score` and +3. Weights are the exact calibrated active-channel vector validated by + `reconstruct.active_weights`; no different channel set is renormalized into + it. Contribution is `weight * score` and must reconcile with `fused_score` within `CHANNEL_EVIDENCE_TOLERANCE` (`1e-6`). 4. Live Event Lineage is replaced atomically. A singleton @@ -46,10 +49,10 @@ authoritative; PROV-O/RDF export is a projection. The administrator-triggered live rebuild and PostgreSQL import pass the configured contextual-orchestrator adjudication client through the same reconstruction boundary only when the exact candidate-pair count is at - most 5,000. Larger snapshots drop the LLM channel before any provider call - and renormalize the remaining weights. This is an operational work bound, - not a model-quality or provider-ranking heuristic. One rebuild never mixes - LLM and non-LLM weight profiles across edges. + most 5,000. Larger snapshots drop the LLM channel before loading the + separately calibrated three-channel profile. This is an operational work + bound, not a model-quality or provider-ranking heuristic. One rebuild never + mixes LLM and non-LLM weight profiles across edges. 5. `GET /api/lineage` returns an additive `channel_evidence` collection on each visible edge (`signal_code`, `signal_label`, `score`, `weight`, `contribution`, `rank`) ordered by contribution, then @@ -67,8 +70,9 @@ authoritative; PROV-O/RDF export is a projection. inference from source evidence. - A later rebuild rewrites live Event Lineage as a whole; historic meaning is not silently mutated in place. -- Completeness is lower when the LLM channel is unavailable, matching - ADR 0064: missing channels are dropped and weights renormalize. +- Completeness is lower when the LLM channel is unavailable; the exact + three-channel profile is loaded instead of renormalizing a four-channel + estimate. ## References diff --git a/docs/product-requirements.md b/docs/product-requirements.md index a3ef8f1a9..1dbbed05c 100644 --- a/docs/product-requirements.md +++ b/docs/product-requirements.md @@ -31,7 +31,8 @@ invented facts, scores, weights, or identities. - Reconstruct bounded post-to-post parentage from independent channels. - Keep Event Lineage separate from ontology/KG edges. - Persist reconstruction profile and participating channel evidence. -- Drop unavailable channels and fail closed on malformed calibrated weights. +- Resolve unavailable channels before loading the exact calibrated active-set + vector; never renormalize a vector calibrated for a different channel set. - Never label an inferred lineage edge causal or authoritative without separate evidence. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 6097dcd9b..83438d065 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -1,6 +1,6 @@ # Product & Technical Gap Baseline -> Dashboard delivery snapshot: 2026-08-25 23:05 KST. Protected `main` was +> Dashboard delivery snapshot: 2026-08-25 23:33 KST. Protected `main` was > `04e6b610655d0db91d5f7ba9486bdda1440e0b19`. This local branch is not > protected-main release evidence. @@ -60,16 +60,17 @@ only aggregate, non-identifying evidence to this repository. ### Exact open-PR boundary -At this snapshot there were 5 open PRs and 10 open issues. Exact observed heads -are listed below. PRs #579, #629, #631, #632, and #636 all target -protected `main`; each had zero unresolved review threads. Four were -`REVIEW_REQUIRED`; #629 had no formal review decision. All had queued required -workflows and remain blocked on hosted gates and independent review. These +At this snapshot there were 6 open PRs and 10 open issues. Exact observed heads +are listed below. PRs #579, #629, #631, #632, and #636 target protected +`main`; stacked PR #638 targets #636's branch. The five protected-main PRs had +zero unresolved review threads. PRs #579, #631, and #632 were +`REVIEW_REQUIRED`; #629, #636, and new #638 had no formal review decision. +Hosted gates and independent review remain required. These observations are not merge readiness. Re-fetch exact heads, unresolved threads, checks, approvals, rulesets, and merge SHA before any lifecycle claim. -> Audit snapshot: 2026-08-25 23:21 KST (refreshed by the autonomous merge +> Audit snapshot: 2026-08-25 23:33 KST (refreshed by the autonomous merge > loop). This repository records synthetic fixtures and aggregate, > non-identifying runtime evidence only. Open PRs and local checks are not > protected-default-branch release evidence. Identifying post identifiers, @@ -79,17 +80,18 @@ lifecycle claim. ## 1. Exact-head and governance evidence The protected default branch was `04e6b610655d0db91d5f7ba9486bdda1440e0b19` -when this baseline was refreshed. The live queue contained 5 open PRs and 10 +when this baseline was refreshed. The live queue contained 6 open PRs and 10 open issues. The exact-head inventory below supersedes older per-PR snapshots elsewhere in this document; those older rows remain useful historical delivery context only. | PR | Exact observed head | Merge/check state at this snapshot | | ---: | --- | --- | -| #636 | `78d87a18` | publishes the calibrated external lineage contract and fails closed when calibrated weights do not exactly match active channels; zero unresolved threads; queued checks and independent review remain required | +| #638 | `51c2be9d` | stacked on #636; makes the shared reconstruction boundary reject mismatched active-channel profiles instead of dropping and renormalizing calibrated weights; hosted checks and independent review remain required | +| #636 | `cc6d64a0` | publishes the calibrated external lineage contract and fails closed when calibrated weights do not exactly match active channels; zero unresolved threads; queued checks and independent review remain required | | #632 | `bb680329` | preserves graph-fact source provenance; zero unresolved threads; queued checks and independent review remain required | | #631 | `c0022c97` | refreshes product-gap and CI lifecycle governance; zero unresolved threads; queued checks and independent review remain required | -| #629 | `883d1ff7` | releases provider work and bounds landing reads; zero unresolved threads, no formal review decision, and queued checks remain | +| #629 | `ac38c652` | releases provider work and bounds landing reads; zero unresolved threads, no formal review decision, and queued checks remain | | #579 | `689a21b6` | persists leftover interaction-map coordinates and owns ADR 0211; zero unresolved threads; queued checks and independent review remain required | No row above is merge evidence. Immediately before any lifecycle action, @@ -362,7 +364,7 @@ this file per §3.5 of the prior snapshot). | Gap | Current evidence | Acceptance requirement | | --- | --- | --- | -| Protected release | 5 open PRs at the 2026-08-25 exact-head snapshot: #579, #629, #631, #632, and #636. All target `main`; each retains hosted checks and independent-review gates | Terminal exact-head checks, no unresolved threads, independent exact-head approvals, protected squash-merge SHA | +| Protected release | 6 open PRs at the 2026-08-25 exact-head snapshot: #579, #629, #631, #632, #636, and stacked #638. The first five target `main`; each retains hosted checks and independent-review gates | Terminal exact-head checks, no unresolved threads, independent exact-head approvals, protected squash-merge SHA | | Evidence-grounded operations workspace | Protected-main #614 delivers governed semantic Ask, live Similar VOC, disjoint pending/failed analysis metrics, full Storybook state inventory, and current desktop/mobile screenshot evidence. Authorized-corpus backfill acceptance remains unavailable | Perform authenticated authorized-corpus acceptance with aggregate evidence and retain fail-closed no-match behavior | | Shared frontend gate | The ADR 0109 login repair is on protected `main`; eight older branches carried the defect and received the same verified repair this loop (#521–#560) | Keep every future branch cut from post-repair bases; re-verify with frontend lint/test/build before push | | Identifying baseline regression | `main` gap file listed real post identifiers; separately, closed #506 and pre-existing public history contain a private runtime source-table identifier, while current `main` and #507 trees are clean | Land this non-identifying rewrite, then coordinate ADR 0001 history remediation with security/privacy owners; do not reproduce the value, force-push, or delete evidence ad hoc | @@ -374,10 +376,10 @@ this file per §3.5 of the prior snapshot). | Knowledge Graph readability | The black evidence-node root cause is an undefined-token fallback; the design-token repair and long-label/evidence-table coverage remain only on closed, unmerged #490, not protected `main` | Recreate the token repair on a current base and deliver it through protected `main`, then verify light/dark contrast, keyboard graph navigation, full labels, and evidence tables in the authenticated rendered surface | | Source-code lookup UX | Source state/detail codes remain evidence-bearing machine values and current detail presentation is dense | Catalog-backed display labels with raw-code provenance, compact 5W1H/source-detail hierarchy, keyboard access, and no unsupported customer/project binding | | Calendar / Naruon | #355 delivered the projection contract; v2.17.0 wires operator consumption without forwarding the end-user token. Naruon producer, provider/consumer fixtures, and protected merge remain open (#336) | Verify observed events against the published schema without invented events; keep commitments available when the channel is unwired | -| External email/project lineage | #636 recreates the historical non-main #343 work on current `main` as a bounded store-agnostic contract. It separates observed reply facts, inferred continuation, and proposed project projection; requires calibrated host weights that exactly match active channels without silent renormalization; and leaves the Naruon consumer disabled pending immutable release | Land #636 through protected gates, publish an immutable LineageWeave artifact, then add Naruon conformance and failure-path fixtures without transferring source credentials or stale stack evidence | +| External email/project lineage | #636 recreates the historical non-main #343 work on current `main` as a bounded store-agnostic contract. It separates observed reply facts, inferred continuation, and proposed project projection; requires calibrated host weights that exactly match active channels without silent renormalization; and leaves the Naruon consumer disabled pending immutable release. Stacked #638 applies the same exact-set rule to the shared reconstruction boundary | Land #638 into #636, then land #636 through protected gates, publish an immutable LineageWeave artifact, and add Naruon conformance and failure-path fixtures without transferring source credentials or stale stack evidence | | SKOS organization aliases | Catalog binding and chip caption live on #480 / #482 | One catalog row per corroborated org; companion caption is hint-only until bound | | Event Lineage evidence | Channel evidence and Allen relations live on #387 / #484 | Persist channel scores, explain them in the popup, never invent a fused score | -| Scientific measurement | Durable accepted TEPP receipts and LineageWeave #614's exact accepted snapshot/cutoff/run/pair-count consumer are protected; TEPP #237 remains open, so no registered producer artifact exists yet. #387 removes inferred/default persistence weights, but several older reconstruction tests still pass hand-authored numeric dictionaries that are not estimator evidence | Land TEPP #237 through its protected gate, then replace remaining reconstruction-test constants with provenance-bearing fast-mlsirm estimates over synthetic fixtures. Retain true-parameter RMSE recovery as the acceptance bar | +| Scientific measurement | Durable accepted TEPP receipts and LineageWeave #614's exact accepted snapshot/cutoff/run/pair-count consumer are protected; TEPP #237 remains open, so no registered producer artifact exists yet. Stacked #638 removes shared-runtime dropping and renormalization of a vector calibrated for a different active channel set. Several older reconstruction tests still pass hand-authored numeric dictionaries that are not estimator evidence | Land TEPP #237 and #638 through their protected dependency gates, then replace remaining reconstruction-test constants with provenance-bearing fast-mlsirm estimates over synthetic fixtures. Retain true-parameter RMSE recovery as the acceptance bar | | Asynchronous authorization | Protected `main` rebuilds Global Ask worker scope after the bearer token leaves the request; #468 now persists exact Keyverse organization/process-unit scope in 3NF child tables and intersects it with current affiliations | Land #468 through the protected gate; prove a second affiliation and a revoked process unit cannot widen delayed-job evidence | | Planned-facility intent | Planned-facility relationship intent remains only on closed, unmerged #490; earlier stack-only merges were not protected delivery | Recreate the evidence-backed slice on a current base and land through protected `main` before a release claim | | Accessibility and responsive UX | #602 delivered base post-detail modal semantics; #605 adds selected-post refocus, collapsed/hidden/inert/CSS-invisible focus exclusion across both modal types, readable evidence separators, focused tests, and desktop/mobile Storybook screenshots | Land #605 through the protected gate, then complete screen-reader and authenticated Playwright acceptance on the exact release head | diff --git a/lineageweave/embedding_client.py b/lineageweave/embedding_client.py index 3cb4d5975..d3525bc16 100644 --- a/lineageweave/embedding_client.py +++ b/lineageweave/embedding_client.py @@ -1,8 +1,8 @@ """Pluggable embedding channel. The default :class:`NullEmbeddingClient` makes the channel unavailable -rather than faking a score -- ``reconstruct.active_weights`` drops and -renormalizes around any channel whose client reports ``available = False``. +rather than faking a vector; callers must use the calibrated profile for the +channels they actually execute. :class:`ContextualOrchestratorEmbeddingClient` calls the authenticated contextual-orchestrator ``/v1/batch/embeddings`` boundary once a credential is set. No client in this repository calls a provider embedding endpoint directly. diff --git a/lineageweave/lineage_persistence.py b/lineageweave/lineage_persistence.py index 97cfdfce3..849a4421b 100644 --- a/lineageweave/lineage_persistence.py +++ b/lineageweave/lineage_persistence.py @@ -65,8 +65,8 @@ def lineage_edge_specs( ``llm`` defaults to ``None``, which ``reconstruct()`` treats as the unavailable :class:`~lineageweave.adjudication_client.NullAdjudicationClient` - (the llm channel is then dropped and the rest renormalized, not - faked) -- callers that want the highest-weighted reasoning channel + (the supplied calibrated vector must omit the llm channel) -- callers that + want the reasoning channel actually contributing to real reconstructions must pass a real one. ``weights`` is required and always a psychometric estimate (ADR diff --git a/lineageweave/reconstruct.py b/lineageweave/reconstruct.py index a52081369..5c4d74ffb 100644 --- a/lineageweave/reconstruct.py +++ b/lineageweave/reconstruct.py @@ -11,6 +11,7 @@ from __future__ import annotations +import math from collections import defaultdict import rankweave as rw @@ -25,8 +26,10 @@ # today, TEPP when integrated) -- product paths load the persisted # corpus estimate, and the library demo estimates from its declared # generative design (channel_weight_estimation.estimate_fixture_channel_weights). -# Every reconstruct() caller passes weights explicitly; the llm entry -# renormalizes away when no client is configured (see active_weights()). +# Every reconstruct() caller passes weights for the exact channel set it can +# execute; a vector estimated for another set is never repaired or renormalized. + +_CORE_WEIGHT_CHANNELS = frozenset({"temporal", "secondary_key", "text"}) # ponytail: only the most recent WINDOW prior records in a group are # considered as candidate parents, bounding per-group cost to O(n*window) @@ -47,12 +50,23 @@ def active_weights( llm: AdjudicationClient, weights: dict[str, float] ) -> dict[str, float]: - """Drop and renormalize the llm channel's weight when no client is configured.""" - active = dict(weights) - if not getattr(llm, "available", False): - active.pop("llm", None) - total = sum(active.values()) - return {channel: weight / total for channel, weight in active.items()} + """Validate and return the calibrated vector for the exact active channels.""" + + expected = set(_CORE_WEIGHT_CHANNELS) + if getattr(llm, "available", False): + expected.add("llm") + if set(weights) != expected: + raise ValueError("weights must exactly match the active lineage channels") + values = tuple(weights.values()) + if any( + isinstance(value, bool) + or not isinstance(value, (int, float)) + or not math.isfinite(float(value)) + or float(value) <= 0.0 + for value in values + ) or not math.isclose(sum(values), 1.0, abs_tol=1e-9): + raise ValueError("active lineage weights must be finite, positive, and sum to one") + return dict(weights) def _group_by(records: list[Record]) -> dict[str, list[Record]]: @@ -159,10 +173,10 @@ def reconstruct( records: every record across every group; grouping happens here. llm: adjudication channel client; defaults to :class:`~lineageweave.adjudication_client.NullAdjudicationClient` - (the llm channel is then dropped, not faked). - weights: per-channel fusion weights before llm-availability - renormalization. Required, and always a psychometric - estimate (ADR 0145, second amendment): the persisted + (the supplied vector must therefore omit the llm channel). + weights: fusion weights calibrated for exactly the channels this call + executes. Required, and always a psychometric estimate (ADR 0145, + second amendment): the persisted fast-mlsirm corpus estimate on product paths, or :func:`~lineageweave.channel_weight_estimation.estimate_fixture_channel_weights` for the library demo. No hand-picked default exists. diff --git a/tests/test_reconstruct.py b/tests/test_reconstruct.py index 1bb8647a9..5f52f17fe 100644 --- a/tests/test_reconstruct.py +++ b/tests/test_reconstruct.py @@ -2,6 +2,8 @@ from datetime import datetime +import pytest + from lineageweave import Record, reconstruct from lineageweave.fixtures import sample_records @@ -80,6 +82,22 @@ def test_llm_channel_is_used_and_scored_when_a_client_is_supplied() -> None: assert all("llm" in edge.channel_scores for edge in tree_a.edges) +def test_llm_weight_is_rejected_when_the_channel_is_unavailable() -> None: + """A four-channel estimate is not repaired into a three-channel vector.""" + + with pytest.raises(ValueError, match="exactly match"): + reconstruct(sample_records(), weights=_SYNTHETIC_WEIGHTS_WITH_LLM) + + +def test_missing_llm_weight_is_rejected_when_the_channel_is_available() -> None: + """An active channel requires its own exact calibrated profile.""" + + with pytest.raises(ValueError, match="exactly match"): + reconstruct( + sample_records(), llm=_StubAdjudicationClient(), weights=_SYNTHETIC_WEIGHTS + ) + + def test_candidate_window_bounds_which_priors_are_considered() -> None: records = [ Record(f"r{i}", "G", f"record {i}", datetime(2026, 1, 1, i), "") for i in range(5)