Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -864,7 +864,11 @@ still flows through unchanged. Cached in a new
(`migrations/0015_organization_name_resolution.sql`) keyed by the raw
name, so the same abbreviation across many posts is resolved once.
Grounded in SKOS `skos:altLabel`/`skos:prefLabel` (Miles & Bechhofer,
2009). Wired into `backend/app/keyman_ingestion.py`'s affiliation loop
2009). After a pair is search-corroborated, both labels compete as
virtual candidates for the **same** `corporate_entity_id`, so a later
mention of `AGP` or `Aurora Grid Power` reuses one catalog row instead
of inserting a second `AUTO-` identity (ADR 0160). Wired into
`backend/app/keyman_ingestion.py`'s affiliation loop
and the offline synthetic-batch script's paced re-implementation of it
(the batch script's own copy was also missing `role_title` persistence
entirely -- fixed alongside this).
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.d/skos-organization-alias-binding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Unreleased — SKOS organization alias catalog binding

## Added

- Corroborated SKOS `altLabel` / `prefLabel` pairs expand corporate
catalog candidates so a synthetic short form (`AGP`) and full form
(`Aurora Grid Power`) bind one `corporate_entity` row (ADR 0160).
Uncorroborated pairs, short near-matches, and tied scores stay unbound.
Real organization names are not used in fixtures.
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ All notable changes to this project are documented here. Format follows

### Added

- Corroborated SKOS `altLabel` / `prefLabel` pairs now expand corporate
catalog candidates so a synthetic short form (`AGP`) and full form
(`Aurora Grid Power`) bind one `corporate_entity` row instead of
creating a second `AUTO-` identity (ADR 0160). Tied scores still stay
unbound.
- ADRs 0150-0153 define the accepted boundaries for Korean relative-time
retrieval, multi-thread Event Lineage answers, persisted image-evidence
citations, and the focused evidence popup. Their implementations remain
Expand Down Expand Up @@ -36,6 +41,10 @@ All notable changes to this project are documented here. Format follows
- The public ontology now states its OWL 2 Full/RDF-Based semantics for the
ADR 0036 RDF-reified project evidence, and the PROV-O support profile uses
its canonical lowercase deployed IRI.
- Corporate-entity creation repeats normal similarity classification after
its lock to catch concurrent ties, while excluding the full resolved
ancestor path so no inferred ancestor can absorb its child. The separate
corroborated-alias recheck remains exact-only (ADR 0012 / ADR 0160).
- `make smoke` and `make seed` now run through the locked project `uv`
environment, so local OIDC and synthetic-data workflows resolve the same
pinned dependencies as CI.
Expand Down
69 changes: 58 additions & 11 deletions backend/app/corporate_entity_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
import asyncio
import hashlib

from collections.abc import Sequence

import asyncpg

from lineageweave.corporate_hierarchy_inference import (
Expand All @@ -27,6 +29,8 @@
RESOLUTION_TIE,
RESOLUTION_UNIQUE,
CorporateEntityCandidate,
OrganizationNameAlias,
expand_candidates_with_skos_aliases,
score_corporate_entity,
)
from lineageweave.http_client import HttpClientError
Expand All @@ -35,6 +39,10 @@
RelationVerificationClient,
)

from .organization_name_resolution_ingestion import (
load_corroborated_organization_name_aliases,
)

_AUTO_CODE_PREFIX = "AUTO-"
_MAX_HIERARCHY_DEPTH = 4
_CREATION_LOCK_KEY = "lineageweave:corporate_entity_creation"
Expand Down Expand Up @@ -111,27 +119,48 @@ async def get_or_create_corporate_entity(
verification_client: RelationVerificationClient,
candidates: list[CorporateEntityCandidate],
*,
aliases: Sequence[OrganizationNameAlias] | None = None,
_depth: int = 0,
_visited_names: frozenset[str] = frozenset(),
_ancestor_entity_ids: set[str] | None = None,
) -> str | None:
"""Return a verified catalog id, otherwise ``None``.

A unique similarity match is reused. A tied top score stays unbound
and does not create a third same-named row (ADR 0026). Only a genuine
miss -- no candidate at or above ``min_similarity`` -- may enter ADR
0010 inference. A proposed parent must independently corroborate and
resolve before the child can be inserted. Repeated names in the
recursion path are cycles, including multi-node cycles such as
A -> B -> A.
and does not create a third same-named row (ADR 0026). After a raw
miss, SKOS alt/pref pairs expand the candidate labels so a
synthetic short form and full form bind the same row (ADR 0160). Only
an alias-expanded miss may enter ADR 0010 inference. A proposed parent
must independently corroborate and resolve before the child can be
inserted. Repeated names in the recursion path are cycles, including
multi-node cycles such as A -> B -> A.
"""
normalized_name = organization_name.strip()
if not normalized_name:
return None
visit_key = normalized_name.casefold()
if visit_key in _visited_names:
return None
ancestor_entity_ids = (
_ancestor_entity_ids if _ancestor_entity_ids is not None else set()
)

existing = score_corporate_entity(normalized_name, candidates)
if existing.kind == RESOLUTION_UNIQUE and existing.catalog_id is not None:
return existing.catalog_id
if existing.kind == RESOLUTION_TIE:
return None
Comment thread
seonghobae marked this conversation as resolved.
Comment thread
seonghobae marked this conversation as resolved.

resolved_aliases: Sequence[OrganizationNameAlias]
if aliases is None:
resolved_aliases = await load_corroborated_organization_name_aliases(conn)
else:
resolved_aliases = aliases
existing = score_corporate_entity(
normalized_name,
expand_candidates_with_skos_aliases(candidates, resolved_aliases),
min_similarity=1.0,
)
Comment thread
seonghobae marked this conversation as resolved.
if existing.kind == RESOLUTION_UNIQUE and existing.catalog_id is not None:
return existing.catalog_id
if existing.kind == RESOLUTION_TIE:
Comment thread
seonghobae marked this conversation as resolved.
Expand Down Expand Up @@ -194,24 +223,42 @@ async def get_or_create_corporate_entity(
inference_client,
verification_client,
candidates,
aliases=resolved_aliases,
_depth=_depth + 1,
_visited_names=visited_names,
_ancestor_entity_ids=ancestor_entity_ids,
)
if parent_entity_id is None:
return None
ancestor_entity_ids.add(parent_entity_id)
Comment thread
seonghobae marked this conversation as resolved.

async with conn.transaction():
await conn.execute(
"select pg_advisory_xact_lock(hashtext($1))",
_CREATION_LOCK_KEY,
)
# ponytail: the lock recheck is exact-only; fuzzy matching here can
# mistake an inferred child for the parent just created above. The
# initial lookup remains fuzzy, while this check only prevents a
# concurrent insert of the same normalized name.
# ponytail: exclude the resolved ancestor path before repeating normal
# raw scoring, or an ancestor created by this recursion can absorb its child.
fresh_candidates = await _reload_candidates(conn)
if ancestor_entity_ids:
fresh_candidates = [
candidate
for candidate in fresh_candidates
if candidate.corporate_entity_id not in ancestor_entity_ids
]
fresh = score_corporate_entity(
normalized_name,
fresh_candidates,
)
if fresh.kind == RESOLUTION_UNIQUE and fresh.catalog_id is not None:
_remember_candidate(candidates, fresh.catalog_id, normalized_name)
return fresh.catalog_id
if fresh.kind == RESOLUTION_TIE:
return None
fresh_aliases = await load_corroborated_organization_name_aliases(conn)
fresh = score_corporate_entity(
normalized_name,
await _reload_candidates(conn),
expand_candidates_with_skos_aliases(fresh_candidates, fresh_aliases),
min_similarity=1.0,
Comment thread
seonghobae marked this conversation as resolved.
)
if fresh.kind == RESOLUTION_UNIQUE and fresh.catalog_id is not None:
Expand Down
11 changes: 10 additions & 1 deletion backend/app/keyman_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
from lineageweave.corporate_hierarchy_resolution import (
RESOLUTION_TIE,
CorporateEntityCandidate,
OrganizationNameAlias,
score_corporate_entity,
)
from lineageweave.keyman_extraction import KeymanExtractionClient, PersonMention
Expand All @@ -71,7 +72,10 @@

from .corporate_entity_ingestion import get_or_create_corporate_entity
from .knowledge_graph import persist_edges_for_post
from .organization_name_resolution_ingestion import resolve_organization_name
from .organization_name_resolution_ingestion import (
load_corroborated_organization_name_aliases,
resolve_organization_name,
)


async def _load_corporate_entity_candidates(conn: asyncpg.Connection) -> list[CorporateEntityCandidate]:
Expand Down Expand Up @@ -182,8 +186,10 @@ async def _resolve_affiliated_organization(
verification_client: RelationVerificationClient,
hierarchy_inference_client: CorporateHierarchyInferenceClient,
candidates: list[CorporateEntityCandidate],
aliases: list[OrganizationNameAlias] | None = None,
) -> tuple[str, str, str | None]:
"""Resolve one affiliation without rewriting a known raw-name tie."""
resolved_aliases = aliases if aliases is not None else []
Comment thread
seonghobae marked this conversation as resolved.
raw_outcome = score_corporate_entity(organization_name, candidates)
if raw_outcome.kind == RESOLUTION_TIE:
return organization_name, organization_name, None
Expand All @@ -202,6 +208,7 @@ async def _resolve_affiliated_organization(
hierarchy_inference_client,
verification_client,
candidates,
aliases=resolved_aliases,
)
return organization_name, resolved_name, corporate_entity_id

Expand Down Expand Up @@ -249,6 +256,7 @@ async def ingest_post_keymen(
else:
mentions = await asyncio.to_thread(client.extract, post_title, post_body)
candidates = await _load_corporate_entity_candidates(conn)
aliases = await load_corroborated_organization_name_aliases(conn)
Comment thread
seonghobae marked this conversation as resolved.
Comment thread
seonghobae marked this conversation as resolved.
resolved_by_mention: list[tuple[PersonMention, list[tuple[str, str, str | None]]]] = []
for mention in mentions:
resolved_orgs: list[tuple[str, str, str | None]] = []
Expand All @@ -262,6 +270,7 @@ async def ingest_post_keymen(
verification_client,
hierarchy_inference_client,
candidates,
aliases,
)
Comment thread
seonghobae marked this conversation as resolved.
)
resolved_by_mention.append((mention, resolved_orgs))
Expand Down
32 changes: 32 additions & 0 deletions backend/app/organization_name_resolution_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,43 @@
OrganizationNameResolutionClient,
resolve_and_verify_organization_name,
)
from lineageweave.corporate_hierarchy_resolution import OrganizationNameAlias
from lineageweave.relation_verification import (
STATUS_CORROBORATED,
RelationVerificationClient,
)

_CORROBORATED_ALIAS_SQL = (
"select raw_organization_name, resolved_organization_name "
"from organization_name_resolution "
"where verification_status_code = $1"
)


async def load_corroborated_organization_name_aliases(
conn: asyncpg.Connection,
) -> list[OrganizationNameAlias]:
"""Return search-corroborated SKOS alt/pref pairs, or an empty list.

Callers with a stub connection that has no ``fetch`` (the early-return
tie tests) get no aliases rather than raising. Only
``verify_corroborated`` rows are returned; pending or uncorroborated
guesses must not bind catalog identities (ADR 0008).
"""
fetch = getattr(conn, "fetch", None)
if not callable(fetch):
return []
rows = await fetch(_CORROBORATED_ALIAS_SQL, STATUS_CORROBORATED)
aliases: list[OrganizationNameAlias] = []
for row in rows:
aliases.append(
OrganizationNameAlias(
alt_label=row["raw_organization_name"],
pref_label=row["resolved_organization_name"],
)
)
return aliases


async def resolve_organization_name(
conn: asyncpg.Connection,
Expand Down
9 changes: 9 additions & 0 deletions backend/app/post_summary_ingestion.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,9 @@

from .corporate_entity_ingestion import get_or_create_corporate_entity
from .keyman_ingestion import _load_corporate_entity_candidates
from .organization_name_resolution_ingestion import (
load_corroborated_organization_name_aliases,
)
from .knowledge_graph import persist_edges_for_post
from .team_ingestion import upsert_team

Expand Down Expand Up @@ -249,6 +252,11 @@ async def persist_post_summary(
verification_client = verification_client or NullRelationVerificationClient()

context_text = post_body if post_body is not None else summary.korean_summary
aliases = (
await load_corroborated_organization_name_aliases(conn)
if summary.roles_and_responsibilities
else []
)
candidates = (
await _load_corporate_entity_candidates(conn)
if summary.roles_and_responsibilities
Expand All @@ -265,6 +273,7 @@ async def persist_post_summary(
hierarchy_inference_client,
verification_client,
candidates,
aliases=aliases,
)
if corporate_entity_id is not None:
resolved_organization_ids[role_index] = corporate_entity_id
Expand Down
5 changes: 5 additions & 0 deletions docs/adr/0008-organization-abbreviation-resolution.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ canonical name is returned to the caller as part of the normalized
`PersonMention`, so the same request's relationship classifier uses the
canonical form too rather than reintroducing the raw abbreviation.

[ADR 0160](0160-skos-organization-alias-catalog-binding.md) extends this
decision: a corroborated pair also expands catalog candidates so both
labels bind one `corporate_entity` row, including when the catalog was
first stored under the short form.

## Consequences

- The raw abbreviated form is not duplicated onto every row that
Expand Down
7 changes: 6 additions & 1 deletion docs/adr/0012-corporate-entity-creation-lock.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,12 @@ pg_advisory_xact_lock(
The transaction-scoped lock is acquired immediately before persistence and is released automatically by the enclosing transaction's commit or rollback (PostgreSQL Global Development Group, 2024).

1. The lock is acquired only after inference and verification complete. Holding it across network I/O would unnecessarily serialize unrelated workers.
2. Under the lock, candidates are reloaded and similarity matching is repeated. Another transaction may have committed the same entity after the caller's original snapshot was read.
2. Under the lock, candidates and corroborated organization aliases are
reloaded. Raw candidates repeat the normal similarity classification; a
full ancestor path resolved by the current recursion is excluded so an
ancestor cannot absorb its child. Alias-expanded candidates then repeat
ADR 0160's exact-label check. Another transaction may have committed the
same entity or alias after the caller's original snapshot was read.
3. The key is one fixed creation-path key rather than a per-name key. Per-name locking still permits the opposite-order multi-entity deadlock shape `A: [X, Y]` versus `B: [Y, X]`.
4. The already-cataloged resolution path remains lock-free.

Expand Down
Loading
Loading