Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
766568b
feat: enforce source semantic coverage evidence
seonghobae Aug 26, 2026
e04249e
fix(ontology): make SHACL validation portable
Aug 26, 2026
8a23991
fix(test): mirror production migration execution
Aug 26, 2026
c57d4cd
Merge branch 'feat/source-semantic-coverage-audit' of https://github.…
Aug 26, 2026
e34e3a1
docs: track canonical orchestrator alias PR
seonghobae Aug 26, 2026
3aeb8cf
docs: refresh exact open PR evidence
Aug 26, 2026
7db5bdf
fix(semantic): require probability sample evidence
Aug 26, 2026
7d4d7dc
Merge branch 'feat/source-semantic-coverage-audit' of https://github.…
Aug 26, 2026
b439c8e
fix(semantic): bind selected sample membership
Aug 26, 2026
e4c6140
fix(import): remove evidence length heuristics
Aug 26, 2026
c488ae8
docs(gaps): refresh exact protected queue
Aug 26, 2026
e8fd06c
docs: record disjoint time-stratified sample
seonghobae Aug 26, 2026
02e453f
fix(ontology): align Unicode body availability
Aug 26, 2026
7d4c846
fix(audit): use internal orchestrator credential
seonghobae Aug 26, 2026
bee2cbb
fix(audit): validate probability stratum cardinality
Aug 26, 2026
5133184
docs(adr): remove semantic coverage identifier collisions
Aug 26, 2026
896430e
fix(import): preserve authoritative bodies on metadata refresh
Aug 26, 2026
2161343
fix(audit): bind inclusion probabilities to strata
Aug 26, 2026
039ad1a
fix(audit): reject unverifiable sample artifacts
Aug 26, 2026
0013b87
feat(ontology): add provenance-bound content semantics
seonghobae Aug 26, 2026
ebb1dfa
fix(import): require source classifications and trigger revisions
seonghobae Aug 26, 2026
3c551c2
fix(audit): include PROV semantic alignments
seonghobae Aug 26, 2026
77c45f7
fix(orchestrator): forward provider host allowlist
seonghobae Aug 26, 2026
78a1441
docs(math): freeze local scoring owner boundary (#712)
seonghobae Aug 26, 2026
af4642f
fix(audit): distinguish schema gaps from instances
seonghobae Aug 26, 2026
39ab844
fix(audit): verify Rust sampling design artifacts
Aug 26, 2026
4720085
Merge commit 'refs/pull/702/head' of https://github.com/ContextualWis…
Aug 26, 2026
73428f9
fix(audit): separate content classification from coverage
seonghobae Aug 26, 2026
019ca26
fix(docs): remove ADR trailing whitespace
Aug 26, 2026
e50fd5a
docs(adr): remove trailing whitespace
seonghobae Aug 26, 2026
0768e64
Merge commit 'refs/pull/702/head' of https://github.com/ContextualWis…
Aug 26, 2026
e75d564
fix(test): preserve audit import ordering
Aug 26, 2026
670d47d
Merge commit 'refs/pull/702/head' of https://github.com/ContextualWis…
Aug 26, 2026
66f064c
docs(audit): record governed-stratum sample
seonghobae Aug 26, 2026
f3add0f
test(orchestrator): verify provider host CLI pin
seonghobae Aug 26, 2026
b0c21d2
feat(audit): verify semantic document key coverage
seonghobae Aug 26, 2026
ae586b0
fix(audit): harden dynamic identifier boundary
seonghobae Aug 26, 2026
ff22ca2
feat(audit): distinguish direct evidence from PROV
seonghobae Aug 26, 2026
3d9e359
fix(audit): include PROV derivation table
seonghobae Aug 26, 2026
bf229e1
feat(audit): bind exact Rust inclusion ratios
Aug 26, 2026
5de66ab
Merge commit 'refs/codex/pr702-latest' into HEAD
Aug 26, 2026
7de6af0
docs(audit): record unavailable artifact references
seonghobae Aug 26, 2026
039b742
docs(audit): verify complete source artifact replay
seonghobae Aug 26, 2026
5cff76f
docs(audit): distinguish PROV deployment from linkage
seonghobae Aug 26, 2026
eda7083
fix(audit): require governed PROV support profile
Aug 26, 2026
2803651
feat(audit): bind terminal semantic coverage provenance
seonghobae Aug 26, 2026
9939eed
Merge remote-tracking branch 'origin/feat/source-semantic-coverage-au…
seonghobae Aug 26, 2026
05bdd5b
fix(audit): require structured multi-agent coverage
seonghobae Aug 26, 2026
8ffe11f
fix(audit): retain rejected attempt provenance
seonghobae Aug 26, 2026
4201bf6
fix(provenance): close reviewed trust boundaries
seonghobae Aug 26, 2026
9d17100
fix(audit): drain provider results before joining
Aug 26, 2026
9229422
fix(audit): correlate provider trace with attempt provenance
seonghobae Aug 26, 2026
8b51c73
docs(gap): refresh stacked estimator delivery state
seonghobae Aug 26, 2026
a4b52b4
fix(audit): bind spawned telemetry to attempt provenance
seonghobae Aug 26, 2026
9f247a6
fix(audit): bind terminal result to attempt provenance
seonghobae Aug 26, 2026
023f32d
docs: distinguish rare-category sampling gap
Aug 26, 2026
4325941
fix(import): avoid reprocessing unavailable source bodies
seonghobae Aug 27, 2026
f058c6b
feat(ontology): carry derived export-source place and region
seonghobae Aug 27, 2026
ebee952
Merge remote-tracking branch 'origin/feat/source-semantic-coverage-au…
seonghobae Aug 27, 2026
93e7b81
codex: address PR review feedback (#702)
Aug 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.d/0.100.0-export-source-ontology-coverage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Export-source ontology coverage (ADR 0246)

- Analyzed an aggregate-only, authorized export source against the published
ontology; all content of its rows is expressible through governed
`postTypeScheme` types, content-semantic classes, PROV attribution, and
raw-code-as-instance lifecycle fields (ADR 0246).
- Closed the one derived-semantic gap: `:Location` now carries the place its
content names (`:locationName`) and an ISO 3166-1 country/region code
(`:countryCode`) as instance data, with a SHACL `:LocationShape` failing
closed on non-code country values.
- No new lookup category is seeded; raw ERP codes remain instance data until
a caller governs their code system (ADR 0145 / 0241 / 0246).
- Aggregate-only reporting and the derived-node (not column-projection)
discipline preserve ADR 0001 / 0207 boundaries; supporting evidence in
`docs/doctoring/export-source-ontology-coverage.md`.
7 changes: 7 additions & 0 deletions CHANGELOG.d/local-scoring-owner-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Local scoring ownership contract

- Identified active Python lineage-channel, reconstruction-decision, and
corporate-entity similarity paths as migration debt.
- Defined the fail-closed owner envelopes required before those paths can be
removed, without assigning corporate identity or moving existing heuristics
to another repository by assumption.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Keep provider credentials outside the repository. Compose interpolation must
# read the same home env file as the orchestrator container's env_file.
COMPOSE := docker compose --env-file "$$HOME/.env"
COMPOSE := docker compose --env-file "$$HOME/.env" -p lineageweave

up:
$(COMPOSE) up -d
Expand Down
6 changes: 6 additions & 0 deletions backend/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -1967,6 +1967,9 @@ async def _load_post_semantic_hints(conn: asyncpg.Connection, post_id: str) -> s
source_customer.entity_name as source_customer_catalog_name,
post.source_project_code,
post.source_project_name,
post.voc_type_code,
post.source_stage_code,
post.source_detail_state_code,
post.secondary_grouping_key as project_field,
customer.entity_name as customer_name,
affiliated.entity_name as author_affiliation_name
Expand Down Expand Up @@ -2038,6 +2041,9 @@ async def _load_post_semantic_hints(conn: asyncpg.Connection, post_id: str) -> s
source_customer_catalog_name=first["source_customer_catalog_name"],
source_project_code=first["source_project_code"],
source_project_name=first["source_project_name"],
source_voc_type_code=first["voc_type_code"],
source_stage_code=first["source_stage_code"],
source_detail_state_code=first["source_detail_state_code"],
source_context_present=source_context_present,
)

Expand Down
17 changes: 16 additions & 1 deletion backend/tests/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import asyncio
import math
import os
import subprocess
import uuid
from contextlib import closing
from pathlib import Path
Expand Down Expand Up @@ -383,7 +384,21 @@ def seeded_db(demo_analyst_token):
cur.execute(_LEFTOVER_MAP_COVERAGE_MIGRATION.read_text())
cur.execute(_GLOBAL_ASK_JOB_MIGRATION.read_text())
cur.execute(_GLOBAL_ASK_SCOPE_MIGRATION.read_text())
cur.execute(_GLOBAL_ASK_EVIDENCE_SEARCH_MIGRATION.read_text())
# Match ADR 0166's production runner exactly: psql keeps
# concurrent indexes outside an implicit transaction and parses
# SQL literals/comments without a fixture-owned splitter.
subprocess.run(
[
"psql",
"-X",
"-v",
"ON_ERROR_STOP=1",
db_dsn,
"-f",
str(_GLOBAL_ASK_EVIDENCE_SEARCH_MIGRATION),
],
check=True,
)
Comment thread
seonghobae marked this conversation as resolved.
cur.execute(_GLOBAL_ASK_KNOWLEDGE_CUTOFF_MIGRATION.read_text())
cur.execute(_GLOBAL_ASK_PUBLIC_VERIFICATION_MIGRATION.read_text())
cur.execute(_EVENT_OCCURRED_AT_MIGRATION.read_text())
Expand Down
1 change: 1 addition & 0 deletions docker/contextual-orchestrator/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ RUN mkdir /tmp/contextual-orchestrator \
'opentelemetry-api>=1.30.0' \
'opentelemetry-sdk>=1.30.0' \
'opentelemetry-exporter-otlp-proto-http>=1.30.0' \
&& python -m contextual_orchestrator --help | grep -q -- '--allowed-provider-host' \
&& useradd --uid 10001 --no-create-home orchestrator

COPY agents.json /app/agents.json
Expand Down
28 changes: 28 additions & 0 deletions docker/contextual-orchestrator/start.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import sys
import json
from pathlib import Path
from urllib.parse import urlsplit


def _pop_first_env(*names: str) -> str:
Expand All @@ -23,6 +24,29 @@ def _pop_first_env(*names: str) -> str:
return first


def _allowed_provider_hosts(provider_url: str) -> tuple[str, ...]:
"""Require the configured gateway host in an explicit outbound allowlist."""
provider_host = (urlsplit(provider_url).hostname or "").rstrip(".").casefold()
allowed_hosts = tuple(
sorted(
{
value.strip().rstrip(".").casefold()
for value in os.environ.get(
"CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS", ""
).split(",")
if value.strip()
}
)
)
if not provider_host:
raise SystemExit("LLM_GATEWAY_API_URL must contain a hostname")
if not allowed_hosts:
raise SystemExit("CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS is required")
if provider_host not in allowed_hosts:
raise SystemExit("LLM_GATEWAY_API_URL hostname is not in the provider allowlist")
return allowed_hosts


def main() -> None:
"""Register the provider credential and delegate to the upstream server."""
gateway_key = _pop_first_env("LLM_GATEWAY_API_KEY", "LLM_API_KEY")
Expand All @@ -48,6 +72,7 @@ def main() -> None:
raise SystemExit("LLM_GATEWAY_API_URL or LLM_GATEWAY_URL is required to start the gateway")
if not provider_url.rstrip("/").endswith("/v1"):
provider_url = provider_url.rstrip("/") + "/v1"
allowed_provider_hosts = _allowed_provider_hosts(provider_url)
raw_limit = os.environ.pop("LLM_GATEWAY_MAX_OUTPUT_TOKENS", "4096").strip()
try:
max_output_tokens = int(raw_limit)
Expand All @@ -67,6 +92,7 @@ def main() -> None:
for agent in agents["agents"]:
agent["base_url"] = provider_url
agent["credential_key"] = "LLM_GATEWAY_API_KEY"
agent["provider_name"] = "configured_gateway"
agent.setdefault("provider_protocol", "auto")
os.environ.pop("LLM_GATEWAY_EMBEDDING_MODEL", None)
agents_path.write_text(json.dumps(agents), encoding="utf-8")
Expand Down Expand Up @@ -97,6 +123,8 @@ def main() -> None:
"--max-body-bytes",
str(max_body_bytes),
]
for allowed_host in allowed_provider_hosts:
sys.argv.extend(("--allowed-provider-host", allowed_host))
del provider_url
del auth_token
from contextual_orchestrator.__main__ import main as serve
Expand Down
3 changes: 3 additions & 0 deletions docs/adr/0030-external-llm-gateway-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ must never be returned through a buyer-facing API or persisted failure detail.
message that tells them to retry or restore the provider configuration.
- `CONTEXTUAL_ORCHESTRATOR_ALLOWED_PROVIDER_HOSTS` must explicitly allow the
hostname selected by `LLM_GATEWAY_API_URL`; wildcard allowlists are forbidden.
The Compose bootstrap forwards each normalized allowlisted hostname to the
orchestrator's `--allowed-provider-host` boundary so runtime model discovery
and provider calls enforce the same operator policy.
- Local Compose development permits only the explicitly enumerated
`host.docker.internal:8080` text gateway and `host.docker.internal:18082`
Vision gateway when `LINEAGEWEAVE_ALLOW_LOCAL_LLM_HTTP=1`; arbitrary local
Expand Down
11 changes: 7 additions & 4 deletions docs/adr/0208-externalize-local-mathematical-compute.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@
**Amends:** ADR 0003, ADR 0024, ADR 0064, ADR 0084, ADR 0132, ADR 0145,
ADR 0148, ADR 0167, ADR 0168, ADR 0182, ADR 0185, ADR 0200, ADR 0201, and
ADR 0205
**Extended by:** [ADR 0245](0245-lineage-scoring-and-entity-resolution-owner-contract.md),
which names the remaining channel, reconstruction-decision, and corporate-
entity similarity paths and defines their minimum owner envelopes.

## Context

Expand All @@ -28,9 +31,10 @@ The ecosystem product boundaries are already sufficient:
CPU/GPU implementation before LineageWeave treats a new result as governed
numerical evidence.

LineageWeave has no standalone canonical PRD file on this exact head. Until
one lands, `ARCHITECTURE.md` and the accepted ADR set are the product baseline;
this absence remains a product-documentation gap, not permission to infer a
At the time of this decision LineageWeave had no standalone canonical PRD.
`docs/product-requirements.md` has since landed as a supporting product
contract and confirms the same consumer-only measurement boundary; accepted
ADRs remain normative. The earlier absence was never permission to infer a
different responsibility.

## Decision
Expand Down Expand Up @@ -113,4 +117,3 @@ https://doi.org/10.1007/s11336-021-09762-5
Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for
structural topic models. *Journal of Statistical Software, 91*(2), 1–40.
https://doi.org/10.18637/jss.v091.i02

56 changes: 56 additions & 0 deletions docs/adr/0240-explicit-missing-body-import-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# ADR 0240: Explicit missing-body import boundary

**Status:** Accepted
**Date:** 2026-08-26
**Extends:** [ADR 0102](0102-semantic-source-unit-boundaries.md)

## Context

An authorized source export can contain titles, lifecycle fields, customer and
project codes, lineage keys, actors, timestamps, and source-artifact provenance
while exposing no record body. Requiring a non-empty body makes every such row
unimportable. Copying the title into the body would instead manufacture body
evidence and falsely imply semantic-unit coverage.

## Decision

1. The PostgreSQL importer accepts exactly one of a mapped body column or
`--no-body-dimension-evidence` containing a non-blank operator statement.
The importer records that attestation but does not use an arbitrary text-
length threshold as a proxy for evidence quality.
2. A missing body persists as the empty source representation. The title stays
`post_title`; it is never copied into `post_body` or emitted as a paragraph.
3. Content-unit, embedding, summary, VISION, and body-search coverage remain
unavailable until an authoritative body/file source is connected.
4. Structured source fields retain their existing raw provenance columns and
semantic-hint boundaries. Their presence does not prove an entity binding.
5. The import result repeats the evidence statement so an operator can retain
it with private runtime evidence. Repository artifacts contain aggregates
only.
6. `scripts/audit_source_semantic_coverage.py` reproduces availability counts
from caller-mapped columns and emits no source values.
7. RDF `bodyAvailable` and its published SHACL constraint use the same
whitespace predicate as the Python projector, including Unicode separator,
next-line, and legacy information-separator characters. A body containing
only those characters is unavailable; validators must not reinterpret it as
semantic evidence.
8. A no-body-dimension re-import preserves an already-populated target body
atomically in the source-post UPSERT. The preserved body is also the input
to revision and semantic-content persistence; an unavailable source
dimension must not erase evidence acquired from an authoritative body
source.

## Consequences

Title-only structured records can participate in explicitly supported
lineage and source-metadata views without fabricated prose. Semantic body
coverage remains honestly incomplete and can be retried after the owning
source supplies bodies.

## Evidence

A 2026-08-26 aggregate-only source inspection found 43,814 rows, 43,814
non-empty titles, zero non-empty bodies, 40,001 customer-code rows, 4,490
project-code rows, and complete process-unit, sales-pool, actor, and
source-artifact provenance fields. No source value, identifier, organization,
or artifact path was copied into this repository.
43 changes: 43 additions & 0 deletions docs/adr/0241-source-classification-semantic-hints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# ADR 0241: Source classification semantic hints

**Status:** Accepted
**Date:** 2026-08-26
**Extends:** [ADR 0004](0004-knowledge-graph-ontology.md),
[ADR 0117](0117-catalog-backed-semantic-hints.md),
[ADR 0159](0159-published-ontology-pages.md),
[ADR 0207](0207-repository-case-ontology-namespace-canonical.md), and
[ADR 0222](0222-project-nodes-in-ontology-neighborhood.md)

## Context

The importer preserves a governed VOC type plus caller-mapped source stage and
detail-state codes, but semantic extraction received neither classification.
An authorized source reference catalog currently provides examples, not a
complete code list or authoritative definitions for every observed value.
Dropping the fields loses source evidence; minting ontology concepts from
partial examples invents semantics.

## Decision

1. Pass `voc_type_code`, `source_stage_code`, and
`source_detail_state_code` to contextual-orchestrator as labeled raw source
hints with exact `source_post` column provenance.
2. Raw codes are context only. They do not assert a lifecycle transition,
inspection outcome, quality grade, entity relationship, or classified
ontology concept. An RDF source-code literal asserts only the observed raw
value and its predicate, not the value's business meaning.
3. RDF projects the governed five-value VOC type as `:hasPostType` to its
published SKOS concept. Stage and detail-state remain literal properties;
projecting a raw literal preserves evidence without minting a concept.
4. Promoting stage/detail values to ontology concepts requires a complete
source-owned code catalog, stable definitions, mapping provenance, and
SHACL fixtures. Partial screen examples are insufficient.
5. Missing codes remain `none`; no default classification is inferred. Every
RDF post projector therefore requires an explicit governed VOC type.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Consequences

Semantic extraction can consider classifications already preserved by the
import boundary without silently losing them or overclaiming their meaning.
The ontology remains intentionally incomplete for source-specific grades and
inspection states until their authority is available.
Loading
Loading