Skip to content

feat: enforce source semantic coverage evidence - #702

Draft
seonghobae wants to merge 60 commits into
mainfrom
feat/source-semantic-coverage-audit
Draft

feat: enforce source semantic coverage evidence#702
seonghobae wants to merge 60 commits into
mainfrom
feat/source-semantic-coverage-audit

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make missing source-body evidence explicit and fail closed instead of copying titles into bodies
  • carry governed source classification literals through semantic hints and RDF/SHACL projection
  • require a probability-sample manifest, membership digests, retained failures, and a versioned fast-mlsirm Rust sample-size artifact before corpus inference
  • record pipeline and aggregate diagnostics without retaining identifying source data

Runtime evidence

  • source aggregate: 43,814 rows, 43,814 nonblank titles, zero nonblank bodies
  • pipeline acceptance audit: 80 ordered records, 0 fully covered, four orchestration trace steps per accepted batch; not a probability sample
  • disjoint structured diagnostic: five time strata x 100 records; every stratum had zero bodies and complete VOC/stage/detail-state values; not corpus-prevalence evidence
  • an additional 100-title semantic attempt produced no accepted batch after provider failures and contributes no classification counts

Persistence boundary

  • operational import already UPSERTs each stable source_post(post_id) and persists revisions/content per record
  • the audit binds selected membership and rejects cardinality/order/owner-token mismatches rather than misbinding batch-local indexes

Verification

  • uv run --extra dev --extra backend pytest -q (1496 passed, 17 skipped before the probability-sampling review repair)
  • uv run --extra dev --extra backend pytest -q tests/test_audit_source_content_semantics.py tests/test_import_postgresql_posts.py (36 passed at e8fd06cc)

Dependency

  • canonical contextual-orchestrator PR #868 owns the advertised gateway-default model contract used by this multi-agent audit path
  • superseded PR #870 was closed unmerged after its explicit-conduct regression was preserved in #868

Summary by CodeRabbit

  • 새로운 기능

    • 게시물의 VOC 유형, 단계, 세부 상태 정보를 의미론적 힌트와 RDF에 반영합니다.
    • 본문이 없는 소스 게시물도 명시적 증거와 함께 가져오며, 기존 본문은 보존됩니다.
    • 소스 콘텐츠 의미 커버리지 집계 및 감사 결과의 검증·추적성이 강화되었습니다.
    • 외부 모델 제공자 호스트 허용 목록을 설정하고 검증할 수 있습니다.
  • 문서

    • 본문 부재 가져오기, 분류 힌트, 비공개 콘텐츠 커버리지 감사 및 데이터 소유권 정책을 문서화했습니다.
  • 테스트

    • 게시물 가져오기, RDF 변환, 의미 힌트, 커버리지 감사 및 제공자 호스트 설정 검증을 확장했습니다.

@coderabbitai

coderabbitai Bot commented Aug 26, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

Next included review available in 48 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 38c763eb-2e16-4bbf-8c26-55719b5b69c8

📥 Commits

Reviewing files that changed from the base of the PR and between 4325941 and 93e7b81.

📒 Files selected for processing (12)
  • CHANGELOG.d/0.100.0-export-source-ontology-coverage.md
  • docs/adr/0246-export-source-ontology-coverage.md
  • docs/adr/README.md
  • docs/doctoring/export-source-ontology-coverage.md
  • docs/ontology/lineageweave-kg-shapes.ttl
  • docs/ontology/lineageweave-kg.ttl
  • docs/product-technical-gap-baseline.md
  • scripts/import_postgresql_posts.py
  • tests/test_audit_source_content_semantics.py
  • tests/test_import_postgresql_posts.py
  • tests/test_ontology.py
  • tests/test_ontology_shapes.py
📝 Walkthrough

Walkthrough

본문 부재 가져오기 경계와 RDF 검증 규칙을 확장했습니다. VOC 분류 코드를 의미 힌트와 RDF에 전달합니다. 소스 의미 커버리지와 비공개 콘텐츠 감사 CLI를 확장했습니다. 관련 ADR, 기준 문서와 테스트를 갱신했습니다.

Changes

의미 콘텐츠 처리 및 감사

Layer / File(s) Summary
본문 부재 가져오기 경계
scripts/import_postgresql_posts.py, tests/test_import_postgresql_posts.py, docs/adr/0240-explicit-missing-body-import-boundary.md
본문 컬럼과 --no-body-dimension-evidence를 상호 배타적으로 검증합니다. 본문이 없는 재가져오기에서 기존 본문을 보존합니다. 유효 본문만 후속 콘텐츠 처리에 사용합니다.
분류 힌트와 RDF 투영
lineageweave/ontology.py, lineageweave/semantic_hints.py, backend/app/main.py, scripts/backfill_post_summaries.py, docs/ontology/*, tests/test_ontology*.py, tests/test_semantic_hints.py, docs/adr/0241-source-classification-semantic-hints.md
VOC 유형, source stage, source detail-state 코드를 의미 힌트와 RDF에 전달합니다. bodyAvailable, hasPostType, 원본 분류 코드와 SHACL 제약을 추가합니다.
소스 의미 커버리지 집계
scripts/audit_source_semantic_coverage.py, tests/test_audit_source_semantic_coverage.py
행 수, 역할별 비공백 값 수, 키 커버리지, 주장 증거 상태와 provenance 스키마 완전성을 집계합니다. PostgreSQL 식별자를 검증합니다.
비공개 콘텐츠 의미 감사
scripts/audit_source_content_semantics.py, tests/test_audit_source_content_semantics.py, docs/adr/0242-private-content-semantic-coverage-audit.md, docs/doctoring/SEMANTIC_COVERAGE_SAMPLING_REFERENCES.md, docs/product-technical-gap-baseline.md
감사 시도 상태와 provenance를 기록합니다. 게이트웨이 요청에 deadline과 세션 ID를 적용합니다. 확률표본 매니페스트와 외부 Rust 설계 산출물을 검증합니다.
운영 실행 검증
Makefile, backend/tests/test_api.py, docker/contextual-orchestrator/*, tests/test_contextual_orchestrator_start.py, pyproject.toml, docs/adr/0030-external-llm-gateway-environment.md
Compose 프로젝트 이름을 lineageweave로 고정합니다. 테스트 migration을 psql subprocess로 실행합니다. provider host allowlist와 fast-mlsirm 커밋 고정을 검증합니다.
소유권 계약 문서화
docs/adr/0208-externalize-local-mathematical-compute.md, docs/adr/0245-lineage-scoring-and-entity-resolution-owner-contract.md, docs/doctoring/python-mathematical-compute-boundary-audit.md, CHANGELOG.d/local-scoring-owner-contract.md
로컬 수학 계산과 entity resolution의 owner contract, migration debt와 fail-closed 조건을 문서화합니다.

Estimated code review effort: 5 (Critical) | ~90+ minutes

Merge Risk: 🔵 Low · up to 43259

The change is mergeable with owner awareness: two spawned-process tests may intermittently time out on slower CI runners because their 5-second deadline includes child-process startup and imports.

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant audit_source_content_semantics
  participant PostgreSQL
  participant contextual_orchestrator
  Operator->>audit_source_content_semantics: 감사 CLI 실행
  audit_source_content_semantics->>audit_source_content_semantics: 매니페스트와 Rust 설계 산출물 검증
  audit_source_content_semantics->>PostgreSQL: 선택 토큰과 콘텐츠 조회
  audit_source_content_semantics->>contextual_orchestrator: conduct 모드 의미 차원 배치 전송
  contextual_orchestrator-->>audit_source_content_semantics: 구조화된 JSON 결과 반환
  audit_source_content_semantics-->>Operator: 비식별 집계 JSON 출력
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 63.04% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 138 functions across 16 files. (3 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 소스 의미론적 커버리지 증거를 강제하는 PR의 주요 변경 사항을 명확하고 간결하게 요약합니다.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 63.04% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 138 functions across 16 files. (3 skipped: 3 unsupported.)

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/source-semantic-coverage-audit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae
seonghobae enabled auto-merge (squash) August 26, 2026 09:35
devin-ai-integration[bot]

This comment was marked as resolved.

@opencode-agent
opencode-agent Bot disabled auto-merge August 26, 2026 09:40
@seonghobae
seonghobae enabled auto-merge (squash) August 26, 2026 09:42
devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Cycle probe 2026-08-26T10:10Z: write retry from seonghobae (MCP). Confirming comment scope before close/merge/push. Independent APPROVE still required on exact heads. Do not self-approve.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Cross-repository sampling-contract audit at exact predecessor head 896430e and repair head 2161343:

  • The recorded 80-item run is correctly labeled pipeline-only. For N=43,814, p=.5, and 95% confidence, n=80 has FPC 0.999098 and approximate margin +/-10.95%; +/-5% requires n=381. No corpus inference should be made from the observed 80.
  • Fixed one manifest-integrity defect in 2161343: each declared stratum inclusion probability must now agree with n_h / N_h (within 1e-12), with a focused regression. The 15-test audit suite passes.
  • Remaining blocker: rust_owner_artifact is self-attested. The caller may choose arbitrary artifact_version/source_sha256 and recompute input/output hashes locally; LineageWeave neither invokes a pinned fast-mlsirm implementation nor verifies a signature/content-addressed release artifact. fast-mlsirm PR #1445 returns sample-size/allocation values only and does not emit these hashes or selected-unit membership. Therefore this gate proves internal manifest consistency, not that Rust computed the declared design, and it is not yet sufficient for customer-facing corpus inference.

No approval or merge action was taken.

@seonghobae
seonghobae enabled auto-merge (squash) August 26, 2026 20:31
coderabbitai[bot]

This comment was marked as resolved.

@opencode-agent
opencode-agent Bot disabled auto-merge August 26, 2026 20:53
@seonghobae
seonghobae enabled auto-merge (squash) August 26, 2026 20:58
devin-ai-integration[bot]

This comment was marked as resolved.

@opencode-agent
opencode-agent Bot disabled auto-merge August 26, 2026 21:02
@seonghobae
seonghobae enabled auto-merge (squash) August 26, 2026 21:09
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@opencode-agent
opencode-agent Bot disabled auto-merge August 26, 2026 22:25
devin-ai-integration[bot]

This comment was marked as resolved.

Declare the RDF prefix used by the portable SHACL constraint and retain the existing semantic artifacts when a source export explicitly has no body dimension.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae
seonghobae enabled auto-merge (squash) August 27, 2026 04:40
coderabbitai[bot]

This comment was marked as resolved.

@opencode-agent
opencode-agent Bot disabled auto-merge August 27, 2026 05:34
ADR 0246: the export source audit (aggregate-only, ADR 0001/0242) shows full
expressibility except one derived-semantic gap. Close it: :Location gains
:locationName and ISO 3166-1 :countryCode instance-data datatype properties
with a closed-world SHACL :LocationShape (fails closed on non-code country
values). Raw ERP codes stay ungoverned instance literals per ADR 0145/0241;
no new lookup category seeded, so the code round-trip and column-only
discipline tests are untouched.
…dit' into feat/source-semantic-coverage-audit

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Devin Review

Comment thread scripts/import_postgresql_posts.py Outdated
Comment on lines +633 to +676
def _post_json_with_deadline(
endpoint: str,
payload: dict[str, object],
*,
headers: dict[str, str],
timeout: float,
) -> dict[str, Any]:
"""Enforce a wall-clock deadline even when a peer keeps a socket active."""
if timeout <= 0:
raise ValueError("timeout must be positive")
context = multiprocessing.get_context("spawn")
result_queue = context.Queue(maxsize=1)
process = context.Process(
target=_post_json_worker,
args=(result_queue, endpoint, payload, headers, timeout),
)
process.start()
try:
try:
succeeded, value = result_queue.get(timeout=timeout)
except queue.Empty as exc:
timed_out = process.is_alive()
if timed_out:
process.terminate()
process.join()
if timed_out:
raise TimeoutError(
"semantic audit provider request exceeded its deadline"
) from exc
raise RuntimeError(
"semantic audit provider process returned no result"
) from exc
if process.is_alive():
process.terminate()
process.join()
if not succeeded:
raise RuntimeError(f"semantic audit provider request failed: {value}")
if not isinstance(value, dict):
raise RuntimeError("semantic audit provider response must be an object")
return value
finally:
result_queue.close()
result_queue.join_thread()

@devin-ai-integration devin-ai-integration Bot Aug 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Per-request spawned process for each provider call

_post_json_with_deadline spawns a fresh spawn-context process per provider batch to enforce a wall-clock deadline. Each spawn re-imports the module, so a full audit incurs one process launch per batch. Acceptable for an operator script, but heavy.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants