From 72732f71fdcdfbde3a95cf7317ac227c1df62728 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 24 Aug 2026 05:52:38 +0900 Subject: [PATCH 1/5] feat(semantic): bind exact spans as units; refuse language as identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add semantic_core as the first ADR 0004/issue #168 slice. A unit is the exact evidence_core source span. Language is unresolved or a primary ISO 639 subtag with optional region. Unresolved metadata keeps the Korean 측정 byte span and does not retokenize. Language tags cannot become identity. Not concept alignment, not invariance, not a topic estimator. --- ARCHITECTURE.md | 1 + CHANGELOG.md | 1 + Cargo.lock | 7 + Cargo.toml | 2 + README.md | 3 +- crates/semantic_core/Cargo.toml | 20 +++ crates/semantic_core/src/error.rs | 47 ++++++ crates/semantic_core/src/lib.rs | 21 +++ crates/semantic_core/src/profile.rs | 129 +++++++++++++++ crates/semantic_core/src/unit.rs | 155 ++++++++++++++++++ crates/semantic_core/tests/crate_contract.rs | 7 + .../tests/language_profile_contract.rs | 54 ++++++ docs/TRACEABILITY.md | 4 +- .../0004-shared-multilingual-latent-space.md | 2 +- docs/adr/0020-span-grounded-semantic-units.md | 89 ++++++++++ docs/adr/README.md | 5 +- docs/research/span-grounded-semantic-units.md | 29 ++++ docs/research/standards-and-literature.md | 2 + scripts/check_workspace_contract.py | 1 + 19 files changed, 573 insertions(+), 6 deletions(-) create mode 100644 crates/semantic_core/Cargo.toml create mode 100644 crates/semantic_core/src/error.rs create mode 100644 crates/semantic_core/src/lib.rs create mode 100644 crates/semantic_core/src/profile.rs create mode 100644 crates/semantic_core/src/unit.rs create mode 100644 crates/semantic_core/tests/crate_contract.rs create mode 100644 crates/semantic_core/tests/language_profile_contract.rs create mode 100644 docs/adr/0020-span-grounded-semantic-units.md create mode 100644 docs/research/span-grounded-semantic-units.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ffe514db..9003df4c 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -52,6 +52,7 @@ boundaries above remain the target modular MSA architecture. | Rust crate | Initial responsibility | |---|---| | `evidence_core` | immutable evidence domain primitives | +| `semantic_core` | span-grounded semantic units; language is not identity | | `temporal_core` | typed clocks, intervals, and temporal reasoning | | `event_core` | event instances, mentions, roles, and provenance | | `relation_graph` | typed relations and forward-transition validation | diff --git a/CHANGELOG.md b/CHANGELOG.md index 36c2e8dd..4b1bd9cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ### Added +- `semantic_core` binds exact `evidence_core` source spans as semantic units. Language profiles are `unresolved` or a primary ISO 639 subtag with optional region (RFC 5646). Unresolved metadata keeps the caller-supplied Korean `측정` span and does not retokenize. `SemanticIdentity::from_language_tag` fails closed. Korean and English report sentences remain distinct units. Not concept alignment, not invariance, not a topic estimator (ADR 0020; issue #168). - `tepp_api` naruon live loopback HTTP/1.1 listener: `serve_one` installs a read/write deadline, requires a loopback `Host`, refuses `Transfer-Encoding` and NIM/proxy credential headers, parses `knowledge_cutoff` as RFC 3339 and refuses a future cutoff, keys analysis-run idempotency by tenant plus key, and proves both analysis-run and export POSTs over a real `TcpStream`. Not a production TLS/`$PORT` service (ADR 0011). - `tepp_api` adaptive orchestration router (ADR 0010): versioned `direct`/`verify`/`committee`/`conductor`/`abstain` selection from CPU `f64` risk, ambiguity, evidence, and token-budget inputs; recorded stages, recursion, decomposition, access lists, and role-specific reasoning effort; fail-closed document-controlled policy/access/credentials; LLM plans remain proposals under deterministic statistical authority; comparable-budget ablation requires a direct baseline; credential-free contextual-orchestrator binding. Live NIM HTTP remains accepted-target. - `tepp_api` purpose-bound provider-payload minimization: time-bounded `PurposeGrant` evaluation, fail-closed expired/not-yet-valid/inverted/cross-tenant/impossible-calendar denial, semantic UTC calendar validation, refusal to copy identity mappings into model-provider payloads or ordinary logs, preservation of opaque analytical identifiers and membership roles (no blanket PII mask), a separately authorized scientific re-identification path, and an internally bound FIPS 180-4 SHA-256 audit digest appended through `ReidentificationAuditSink` before disclosure. diff --git a/Cargo.lock b/Cargo.lock index fb502b9c..987689a6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -995,6 +995,13 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" +[[package]] +name = "semantic_core" +version = "0.1.0" +dependencies = [ + "evidence_core", +] + [[package]] name = "serde" version = "1.0.229" diff --git a/Cargo.toml b/Cargo.toml index 92565940..357af7ca 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,6 +2,7 @@ resolver = "2" members = [ "crates/evidence_core", + "crates/semantic_core", "crates/temporal_core", "crates/event_core", "crates/relation_graph", @@ -14,6 +15,7 @@ members = [ ] default-members = [ "crates/evidence_core", + "crates/semantic_core", "crates/temporal_core", "crates/event_core", "crates/relation_graph", diff --git a/README.md b/README.md index ae74015d..f4f71034 100644 --- a/README.md +++ b/README.md @@ -7,12 +7,13 @@ implemented in Rust. ## Current implementation state This branch establishes the Task 1 Rust workspace and quality-gate foundation. -The ten bounded crates compile independently but intentionally expose no +The eleven bounded crates compile independently but intentionally expose no placeholder production APIs. Domain behavior begins in Task 2 with immutable evidence identifiers and source records. ```text crates/evidence_core +crates/semantic_core crates/temporal_core crates/event_core crates/relation_graph diff --git a/crates/semantic_core/Cargo.toml b/crates/semantic_core/Cargo.toml new file mode 100644 index 00000000..6192fe98 --- /dev/null +++ b/crates/semantic_core/Cargo.toml @@ -0,0 +1,20 @@ +[package] +name = "semantic_core" +description = "Span-grounded semantic units whose identity is never a language tag." +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +homepage.workspace = true +readme.workspace = true +keywords.workspace = true +categories.workspace = true +publish = false + +[dependencies] +evidence_core = { path = "../evidence_core", version = "0.1.0" } + +[lints] +workspace = true diff --git a/crates/semantic_core/src/error.rs b/crates/semantic_core/src/error.rs new file mode 100644 index 00000000..147866f3 --- /dev/null +++ b/crates/semantic_core/src/error.rs @@ -0,0 +1,47 @@ +//! Fail-closed semantic-unit validation errors. + +use std::fmt; + +/// A fail-closed semantic-unit error. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +#[non_exhaustive] +pub enum SemanticError { + /// A language tag was offered as the unit identity. + LanguageIsNotIdentity, + /// A language tag was empty. + EmptyLanguageTag, + /// A language tag was not a primary ISO 639 subtag with optional region. + InvalidLanguageTag, +} + +impl fmt::Display for SemanticError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + let message = match self { + Self::LanguageIsNotIdentity => "language tag is not semantic-unit identity", + Self::EmptyLanguageTag => "empty language tag", + Self::InvalidLanguageTag => "invalid language tag", + }; + formatter.write_str(message) + } +} + +impl std::error::Error for SemanticError {} + +#[cfg(test)] +mod tests { + use super::SemanticError; + + #[test] + fn error_messages_are_stable() { + for (error, message) in [ + ( + SemanticError::LanguageIsNotIdentity, + "language tag is not semantic-unit identity", + ), + (SemanticError::EmptyLanguageTag, "empty language tag"), + (SemanticError::InvalidLanguageTag, "invalid language tag"), + ] { + assert_eq!(error.to_string(), message); + } + } +} diff --git a/crates/semantic_core/src/lib.rs b/crates/semantic_core/src/lib.rs new file mode 100644 index 00000000..fec29fe6 --- /dev/null +++ b/crates/semantic_core/src/lib.rs @@ -0,0 +1,21 @@ +#![forbid(unsafe_code)] +#![deny(missing_docs)] +//! Span-grounded semantic units whose identity is never a language tag. +//! +//! Language metadata may mark a unit unresolved or with a primary ISO 639 +//! subtag. Unresolved metadata does not retokenize or move the exact source +//! span. Equivalent Korean and English surfaces remain distinct units until a +//! later concept-alignment layer (ADR 0004 / ADR 0012) is validated. + +mod error; +mod profile; +mod unit; + +/// Fail-closed semantic-unit validation errors. +pub use error::SemanticError; +/// Language profile metadata, never unit identity. +pub use profile::LanguageProfile; +/// Exact-span identity of one semantic unit. +pub use unit::SemanticIdentity; +/// One exact-span semantic unit. +pub use unit::SemanticUnit; diff --git a/crates/semantic_core/src/profile.rs b/crates/semantic_core/src/profile.rs new file mode 100644 index 00000000..1565b06c --- /dev/null +++ b/crates/semantic_core/src/profile.rs @@ -0,0 +1,129 @@ +//! Language profiles are metadata, not unit identity. + +use crate::error::SemanticError; + +/// A language profile that may select tailoring without becoming identity. +#[derive(Clone, Debug, Eq, PartialEq)] +#[non_exhaustive] +pub enum LanguageProfile { + /// No language metadata; span bounds stay the supplied exact coordinates. + Unresolved, + /// A canonical primary ISO 639 subtag with optional ISO 3166-1 region. + Tagged { + /// Lowercase `language` or `language-region` label. + tag: String, + }, +} + +impl LanguageProfile { + /// Return the unresolved profile. + #[must_use] + pub const fn unresolved() -> Self { + Self::Unresolved + } + + /// Parse a primary language subtag with an optional region. + /// + /// Accepts two or three ASCII letters, optionally followed by a hyphen and + /// a two-letter region. The stored form is lowercase. Empty tags and any + /// other shape fail closed. The tag never becomes a [`crate::SemanticIdentity`]. + /// + /// # Errors + /// + /// Returns [`SemanticError::EmptyLanguageTag`] or + /// [`SemanticError::InvalidLanguageTag`]. + pub fn parse_bcp47(tag: &str) -> Result { + if tag.is_empty() { + return Err(SemanticError::EmptyLanguageTag); + } + let canonical = tag.to_ascii_lowercase(); + if !is_primary_language_tag(&canonical) { + return Err(SemanticError::InvalidLanguageTag); + } + Ok(Self::Tagged { tag: canonical }) + } + + /// Return whether this profile is unresolved. + #[must_use] + pub const fn is_unresolved(&self) -> bool { + matches!(self, Self::Unresolved) + } + + /// Return the stable profile label. + #[must_use] + pub fn as_str(&self) -> &str { + match self { + Self::Unresolved => "unresolved", + Self::Tagged { tag } => tag, + } + } +} + +fn is_primary_language_tag(tag: &str) -> bool { + match tag.split_once('-') { + None => is_letter_run(tag, 2, 3), + Some((language, region)) => is_letter_run(language, 2, 3) && is_letter_run(region, 2, 2), + } +} + +fn is_letter_run(value: &str, min: usize, max: usize) -> bool { + (min..=max).contains(&value.len()) && value.bytes().all(|byte| byte.is_ascii_lowercase()) +} + +#[cfg(test)] +mod tests { + use super::LanguageProfile; + use crate::error::SemanticError; + + #[test] + fn parse_accepts_primary_and_region_and_rejects_noise() { + let korean = LanguageProfile::parse_bcp47("KO").expect("ko"); + assert_eq!(korean.as_str(), "ko"); + assert!(!korean.is_unresolved()); + let tagged = LanguageProfile::parse_bcp47("en-US").expect("en-us"); + assert_eq!(tagged.as_str(), "en-us"); + assert_eq!( + LanguageProfile::parse_bcp47("yue").expect("yue").as_str(), + "yue" + ); + assert_eq!( + LanguageProfile::parse_bcp47("").unwrap_err(), + SemanticError::EmptyLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("english").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("en-US-x-private").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("e").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("en_us").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("e1").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("en-u1").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("en-u").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("e-us").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + let unresolved = LanguageProfile::unresolved(); + assert!(unresolved.is_unresolved()); + assert_eq!(unresolved.as_str(), "unresolved"); + } +} diff --git a/crates/semantic_core/src/unit.rs b/crates/semantic_core/src/unit.rs new file mode 100644 index 00000000..1aa22524 --- /dev/null +++ b/crates/semantic_core/src/unit.rs @@ -0,0 +1,155 @@ +//! Span-grounded semantic units. + +use crate::error::SemanticError; +use crate::profile::LanguageProfile; +use evidence_core::{EvidenceId, SourceSpan}; + +/// Exact-span identity of one semantic unit. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct SemanticIdentity { + document_id: EvidenceId, + byte_start: usize, + byte_end: usize, +} + +impl SemanticIdentity { + /// Construct identity from an exact source span. + #[must_use] + pub const fn from_span(span: SourceSpan) -> Self { + Self { + document_id: span.document_id(), + byte_start: span.byte_start(), + byte_end: span.byte_end(), + } + } + + /// Return the owning document identifier. + #[must_use] + pub const fn document_id(&self) -> EvidenceId { + self.document_id + } + + /// Return the inclusive byte start. + #[must_use] + pub const fn byte_start(&self) -> usize { + self.byte_start + } + + /// Return the exclusive byte end. + #[must_use] + pub const fn byte_end(&self) -> usize { + self.byte_end + } + + /// Refuse a language tag as semantic-unit identity. + /// + /// # Errors + /// + /// Always returns [`SemanticError::LanguageIsNotIdentity`]. + pub fn from_language_tag(_tag: &str) -> Result { + Err(SemanticError::LanguageIsNotIdentity) + } +} + +/// One exact-span semantic unit with optional language metadata. +#[derive(Clone, Debug)] +pub struct SemanticUnit { + span: SourceSpan, + language: LanguageProfile, +} + +impl PartialEq for SemanticUnit { + fn eq(&self, other: &Self) -> bool { + self.identity() == other.identity() + } +} + +impl Eq for SemanticUnit {} + +impl SemanticUnit { + /// Bind a language profile onto an exact source span. + /// + /// Unresolved and tagged profiles keep the same byte and scalar bounds. + /// The profile never becomes [`SemanticIdentity`]. + #[must_use] + pub fn bind(span: SourceSpan, language: LanguageProfile) -> Self { + Self { span, language } + } + + /// Return the exact source span. + #[must_use] + pub const fn span(&self) -> SourceSpan { + self.span + } + + /// Return the language profile metadata. + #[must_use] + pub const fn language(&self) -> &LanguageProfile { + &self.language + } + + /// Return span-grounded identity. + #[must_use] + pub const fn identity(&self) -> SemanticIdentity { + SemanticIdentity::from_span(self.span) + } + + /// Replace language metadata without moving the span. + #[must_use] + pub fn with_language(self, language: LanguageProfile) -> Self { + Self { + span: self.span, + language, + } + } +} + +#[cfg(test)] +mod tests { + use super::{SemanticIdentity, SemanticUnit}; + use crate::error::SemanticError; + use crate::profile::LanguageProfile; + use evidence_core::{DocumentRecord, SourceArtifact, SourceSpan}; + + fn span_over(text: &str, start: usize, end: usize) -> SourceSpan { + let artifact = SourceArtifact::from_bytes(b"src").expect("artifact"); + let document = DocumentRecord::from_text(artifact.id(), text).expect("document"); + let scalar_start = text[..start].chars().count(); + let scalar_end = scalar_start + text[start..end].chars().count(); + SourceSpan::new(&document, start, end, scalar_start, scalar_end, None).expect("span") + } + + #[test] + fn language_tag_cannot_become_identity() { + assert_eq!( + SemanticIdentity::from_language_tag("ko").unwrap_err(), + SemanticError::LanguageIsNotIdentity + ); + assert_eq!( + SemanticIdentity::from_language_tag("en-us").unwrap_err(), + SemanticError::LanguageIsNotIdentity + ); + } + + #[test] + fn unresolved_profile_keeps_supplied_korean_span() { + let korean = "측정 오차는 RMSE로 보고한다."; + let start = 0; + let end = "측정".len(); + let span = span_over(korean, start, end); + let unresolved = SemanticUnit::bind(span, LanguageProfile::unresolved()); + let tagged = unresolved + .clone() + .with_language(LanguageProfile::parse_bcp47("ko").expect("ko")); + assert_eq!(unresolved.identity(), tagged.identity()); + assert_eq!(unresolved.span().byte_start(), start); + assert_eq!(unresolved.span().byte_end(), end); + assert_eq!(tagged.span().byte_start(), start); + assert_eq!(tagged.span().byte_end(), end); + assert_ne!(unresolved.language(), tagged.language()); + assert_eq!(unresolved, tagged); + assert_eq!(unresolved.identity().document_id(), span.document_id()); + assert_eq!(unresolved.identity().byte_start(), start); + assert_eq!(unresolved.identity().byte_end(), end); + } +} diff --git a/crates/semantic_core/tests/crate_contract.rs b/crates/semantic_core/tests/crate_contract.rs new file mode 100644 index 00000000..55f665ff --- /dev/null +++ b/crates/semantic_core/tests/crate_contract.rs @@ -0,0 +1,7 @@ +//! Integration contract for the `semantic_core` package identity. + +#[test] +fn package_identity_is_stable() { + let observed = std::hint::black_box(env!("CARGO_PKG_NAME")); + assert_eq!(observed, "semantic_core"); +} diff --git a/crates/semantic_core/tests/language_profile_contract.rs b/crates/semantic_core/tests/language_profile_contract.rs new file mode 100644 index 00000000..931cc548 --- /dev/null +++ b/crates/semantic_core/tests/language_profile_contract.rs @@ -0,0 +1,54 @@ +//! Realistic Korean/English span identity is independent of language tags. + +use evidence_core::{DocumentRecord, SourceArtifact, SourceSpan}; +use semantic_core::{LanguageProfile, SemanticError, SemanticIdentity, SemanticUnit}; + +fn document(text: &str) -> DocumentRecord { + let artifact = SourceArtifact::from_bytes(b"corpus").expect("artifact"); + DocumentRecord::from_text(artifact.id(), text).expect("document") +} + +fn span(document: &DocumentRecord, start: usize, end: usize) -> SourceSpan { + let text = document.text(); + let scalar_start = text[..start].chars().count(); + let scalar_end = scalar_start + text[start..end].chars().count(); + SourceSpan::new(document, start, end, scalar_start, scalar_end, None).expect("span") +} + +#[test] +fn korean_and_english_surfaces_are_distinct_units() { + let korean_text = "측정 오차는 RMSE로 보고한다."; + let english_text = "Measurement error is reported as RMSE."; + let korean_doc = document(korean_text); + let english_doc = document(english_text); + let korean = SemanticUnit::bind( + span(&korean_doc, 0, "측정".len()), + LanguageProfile::parse_bcp47("ko").expect("ko"), + ); + let english = SemanticUnit::bind( + span(&english_doc, 0, "Measurement".len()), + LanguageProfile::parse_bcp47("en").expect("en"), + ); + assert_ne!(korean.identity(), english.identity()); + assert_ne!(korean, english); + assert_eq!(korean.language().as_str(), "ko"); + assert_eq!(english.language().as_str(), "en"); +} + +#[test] +fn missing_language_does_not_retokenize_or_steal_identity() { + let text = "측정 오차는 RMSE로 보고한다."; + let document = document(text); + let exact = span(&document, 0, "측정".len()); + let unresolved = SemanticUnit::bind(exact, LanguageProfile::unresolved()); + let tagged = unresolved + .clone() + .with_language(LanguageProfile::parse_bcp47("ko").expect("ko")); + assert_eq!(unresolved.identity(), tagged.identity()); + assert_eq!(unresolved.span().byte_start(), tagged.span().byte_start()); + assert_eq!(unresolved.span().byte_end(), tagged.span().byte_end()); + assert_eq!( + SemanticIdentity::from_language_tag("ko"), + Err(SemanticError::LanguageIsNotIdentity) + ); +} diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index a3e674cf..09997aab 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -1,7 +1,7 @@ # TEPP Requirements, Research, and Evidence Traceability **Status:** Accepted cross-cutting traceability baseline -**Last reviewed:** 2026-08-13 +**Last reviewed:** 2026-08-24 The full APA 7th standards/literature register remains `docs/research/standards-and-literature.md`. This matrix links durable requirements to their owning decisions and implementation/evidence maturity without duplicating the bibliography. @@ -21,7 +21,7 @@ The full APA 7th standards/literature register remains `docs/research/standards- | known-truth temporal/event simulation manifests | PRD; TRD; Test Strategy | `tepp_simulation` on protected main; recovery metrics in `validation_core` | implemented-main | | versioned service/API contracts and exports | PRD; API contract; ADR 0011/0013 | `tepp_api` analysis-run/export/JSON-LD/GraphML contracts on protected main (PR #21); HTTP service remaining accepted-target | partial | | immutable split/run/reproducibility manifests | ADR 0013; ERD | `tepp_api` reproducibility manifest contract on protected main; `persistence_postgres` append-only SQL insert/lookup for `reproducibility_manifest`, `corpus_split_manifest`, `model_run`, and `model_artifact` (migration `0003`); full physical ERD constraints remaining | partial | -| multilingual shared latent semantic space | PRD; ADR 0004 | future semantic/concept/topic crates | accepted-target | +| multilingual shared latent semantic space | PRD; ADR 0004; ADR 0020 | `semantic_core` span-grounded units (active-PR); concept dictionary and shared latent estimator remaining | active-PR | | TRSL-TM temporal/relational topic posterior and backend compatibility | ADR 0012; ADR 0004 | future `topic_measurement` | accepted-target | | global P0 topic identity with activity/dormancy/reactivation | ADR 0012 | future topic lineage/activity state | accepted-target | | no default stopword deletion / no TF-IDF-BM25 inferential weighting | ADR 0004/0012; PRD/TRD | future semantic/method-source model | accepted-target | diff --git a/docs/adr/0004-shared-multilingual-latent-space.md b/docs/adr/0004-shared-multilingual-latent-space.md index c8da25b5..6958a9e4 100644 --- a/docs/adr/0004-shared-multilingual-latent-space.md +++ b/docs/adr/0004-shared-multilingual-latent-space.md @@ -3,7 +3,7 @@ **Decision status:** Accepted **Implementation maturity:** accepted-target **Date:** 2026-08-05 -**Supersedes:** None. ADR 0012 governs the complete topic-estimator/backend/global-topic contract built on this multilingual measurement decision. +**Supersedes:** None. ADR 0012 governs the complete topic-estimator/backend/global-topic contract built on this multilingual measurement decision. ADR 0020 owns the first span-grounded unit-identity slice as an active-PR; it is not shared-latent estimation. ## Context diff --git a/docs/adr/0020-span-grounded-semantic-units.md b/docs/adr/0020-span-grounded-semantic-units.md new file mode 100644 index 00000000..8f0cec8f --- /dev/null +++ b/docs/adr/0020-span-grounded-semantic-units.md @@ -0,0 +1,89 @@ +# ADR 0020 — Span-grounded semantic units + +**Decision status:** Accepted +**Implementation maturity:** active-PR +**Date:** 2026-08-24 +**Supersedes:** None. Complements ADR 0004 and ADR 0008. Does not replace ADR 0012 topic estimation or ADR 0005 psychometrics. +**Figma File ID:** N/A — this increment is a Rust domain crate with no user-interface surface. +**Storybook inventory:** N/A — no reusable web object or interaction changed. + +## Context + +Issue #168 requires multilingual documents to enter a shared latent space while +preserving exact native-language evidence. Protected `main` already stores +immutable byte and Unicode-scalar spans (`evidence_core`, ADR 0008). It does not +yet bind those spans into semantic units, and language metadata must not become +a substitute identity or silently retokenize unresolved text. + +## Decision + +Add standalone crate `semantic_core` as the first ADR 0004 production slice: + +- a semantic unit is identified by exact `SourceSpan` coordinates + (document identity plus byte start/end); +- a language profile is optional metadata (`unresolved` or a primary ISO 639 + subtag with optional region); +- unresolved language keeps the caller-supplied span and does not switch + segmentation heuristic; +- `SemanticIdentity::from_language_tag` fails closed; +- Korean and English surfaces of comparable meaning remain distinct units. + +This slice does not unitize raw text, align concepts, estimate topics, or claim +measurement invariance. + +## Non-goals + +- do not tokenize or morphologically analyze text; +- do not treat BCP 47 as a complete language-identification system; +- do not merge cross-language units into one concept; +- do not grant LLM-proposed units authority without exact spans. + +## Alternatives considered + +1. **Use the language tag as the unit key** — rejected because language is + identifying metadata, not evidence identity, and mixed-language documents + would collide or split incorrectly. +2. **Retokenize unresolved language with a default whitespace heuristic** — + rejected because missing metadata must not silently change span bounds + (ADR 0004). +3. **Span-grounded units with language as profile only** — accepted. + +## Consequences + +Operators can bind Korean and English exact spans without collapsing them. +Later concept-dictionary and invariance work (ADR 0004 / #84) can consume these +units without inheriting language-as-identity. + +## Failure and recovery + +Empty or malformed language tags fail closed. Offering a language tag as +identity fails closed. Recovery supplies a valid exact span and optional +profile; it does not rewrite historical artifacts. + +## Security, privacy, and governance impact + +Language tags and native lexical spans can be identifying. Purpose-bound access +under ADR 0009 applies. Documents remain untrusted input. + +## Compatibility and migration + +Standalone crate with no persistence schema. No database object names. Future +concept-alignment versions must not change span identity without a superseding +ADR. + +## Verification + +Integration tests bind a realistic Korean report sentence and an English +counterpart, prove distinct identities, prove unresolved vs `ko` keeps the +`측정` byte span, and prove language tags cannot become identity. + +## Rollback and supersession + +Remove the crate from the workspace if the slice is rejected. Supersede only +with a decision that keeps exact-span identity and explicit language-profile +status. + +## Authority links + +PRD multilingual measurement; ADR 0004; ADR 0008; issue #168; Phillips & Davis +(2009, RFC 5646). diff --git a/docs/adr/README.md b/docs/adr/README.md index 258eb7f3..7fdfacff 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -9,7 +9,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0001](0001-rust-first-modular-msa.md) | Rust-first numerical core and CPU `f64` reference | Accepted | partial | ADR 0011 owns cross-service/MSA authority; 0001 retains numerical/backend authority. | | [0002](0002-six-clock-temporal-semantics.md) | Six-clock temporal semantics and fail-closed historical leakage prevention | Accepted | active-PR | Unmerged PR #8 is the canonical Task 3 replacement implementing typed clocks/intervals against the current protected-main lineage; conflicted PR #5 is superseded lineage. Later graph/split enforcement remains target work. | | [0003](0003-relational-event-multiple-membership.md) | Relational event ontology and time-varying cross-classified multiple membership | Accepted | partial | Weighted time-varying membership network/roles are active-PR (PR #12); full multilevel estimators, graph ontology, and persistence remain accepted-target. ADR 0016 owns event-intelligence tasks. | -| [0004](0004-shared-multilingual-latent-space.md) | One shared multilingual latent space with explicit invariance status | Accepted | accepted-target | ADR 0012 owns the full topic-estimator/backend/global-topic contract. | +| [0004](0004-shared-multilingual-latent-space.md) | One shared multilingual latent space with explicit invariance status | Accepted | accepted-target | ADR 0020 owns the first span-grounded unit-identity slice; ADR 0012 owns the topic estimator. | | [0005](0005-posterior-esem-dsem.md) | Posterior-aware ESEM/DSEM and valid compositional coordinates | Accepted | accepted-target | Downstream psychometric authority; upstream topic/network model is clarified by ADR 0012. | | [0006](0006-vram-gpu-nvidia-orchestration.md) | VRAM-adaptive GPU compute and model-credential boundary | Accepted | accepted-target | LLM orchestration policy superseded by ADR 0010; autonomous development authority governed by ADR 0015. | | [0007](0007-rust-workspace-quality-gates.md) | Explicit Rust workspace, pinned toolchains, and exact quality gates | Accepted | implemented-main | ADR 0014 governs scientific/product claim promotion beyond repository-quality tooling. | @@ -22,6 +22,7 @@ Read [`ADR_POLICY.md`](ADR_POLICY.md) first. **Decision status and implementatio | [0014](0014-scientific-claim-promotion-and-release-evidence.md) | Scientific claim promotion and release evidence authority | Accepted | partial | Separates design, implementation, scientific/product claim, and release authority; repository SBOM/provenance generator implemented, full release bundle remaining. | | [0015](0015-autonomous-development-review-and-merge-authority.md) | Autonomous development, review, and merge authority separation | Accepted | active-PR | Separates model proposal, deterministic verification, publication, independent review, and merge/release authority. | | [0016](0016-tdt-chronos-event-intelligence-boundary.md) | TDT, CHRONOS, and Event Ontology intelligence boundary | Accepted | accepted-target | Separates observed evidence, detection/tracking, prediction/schema inference, temporal consistency, and promoted transition authority. | +| [0020](0020-span-grounded-semantic-units.md) | Span-grounded semantic units; language tags are not identity | Accepted | active-PR | First ADR 0004 production slice. Does not claim concept alignment, invariance, or a topic estimator. | ## Decision ownership summary @@ -30,7 +31,7 @@ Use the narrowest owning ADR when decisions overlap: - **numerical implementation / reference backend:** ADR 0001; - **clock/time eligibility:** ADR 0002; - **event ontology / relation / membership semantics:** ADR 0003; -- **multilingual semantic alignment:** ADR 0004; +- **multilingual semantic alignment:** ADR 0004; span-grounded unit identity: ADR 0020; - **ESEM/DSEM and psychometric interpretation:** ADR 0005; - **GPU/VRAM and model-credential boundary:** ADR 0006; - **repository quality tooling:** ADR 0007; diff --git a/docs/research/span-grounded-semantic-units.md b/docs/research/span-grounded-semantic-units.md new file mode 100644 index 00000000..1881b33f --- /dev/null +++ b/docs/research/span-grounded-semantic-units.md @@ -0,0 +1,29 @@ +# Span-grounded semantic units + +This note traces the first `semantic_core` slice to primary sources. It does not +claim concept alignment, measurement invariance, or a topic estimator. + +## Sources + +Phillips, A., & Davis, M. (Eds.). (2009). *Tags for identifying languages* +(RFC 5646). Internet Engineering Task Force. https://doi.org/10.17487/RFC5646 + +The Unicode Consortium. (2023). *The Unicode Standard, Version 15.1.0*. +https://www.unicode.org/versions/Unicode15.1.0/ + +Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009). +Polylingual topic models. In *Proceedings of the 2009 Conference on Empirical +Methods in Natural Language Processing* (pp. 880–889). Association for +Computational Linguistics. + +## Application + +RFC 5646 licenses a primary language subtag with an optional region as +*metadata*. TEPP stores that label on `LanguageProfile` and refuses it as +`SemanticIdentity`. Unicode scalar/byte spans remain the evidence coordinates +from ADR 0008. Polylingual topic identity is a later shared-latent claim +(ADR 0004 / ADR 0012), not this slice. + +Korean `측정` and English `Measurement` in matched report sentences are distinct +units on realistic native text. Unresolved language does not retokenize those +spans. diff --git a/docs/research/standards-and-literature.md b/docs/research/standards-and-literature.md index 28e62d5c..061a8a81 100644 --- a/docs/research/standards-and-literature.md +++ b/docs/research/standards-and-literature.md @@ -24,6 +24,8 @@ Chang, J., & Blei, D. M. (2009). Relational topic models for document networks. Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009). Polylingual topic models. In *Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing* (pp. 880–889). Association for Computational Linguistics. +Phillips, A., & Davis, M. (Eds.). (2009). *Tags for identifying languages* (RFC 5646). Internet Engineering Task Force. https://doi.org/10.17487/RFC5646 + Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 diff --git a/scripts/check_workspace_contract.py b/scripts/check_workspace_contract.py index c7b1ecf5..3269cbc6 100644 --- a/scripts/check_workspace_contract.py +++ b/scripts/check_workspace_contract.py @@ -14,6 +14,7 @@ EXPECTED_CRATES: tuple[str, ...] = ( "evidence_core", + "semantic_core", "temporal_core", "event_core", "relation_graph", From 5344729219a386078bcb77b7f42a78039a5910f9 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 24 Aug 2026 05:56:22 +0900 Subject: [PATCH 2/5] test(quality): compare crate roots to EXPECTED_CRATES by name The live docstring discovery test still asserted ten crate roots, so semantic_core failed the repository-contracts job 11 != 10. Name-set comparison requires the workspace contract list and rejects extras. --- CHANGELOG.md | 1 + tests/quality/test_check_docstrings.py | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b1bd9cb..5552e8d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,6 +76,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ### Changed +- The docstring discovery test compares crate-root names to `EXPECTED_CRATES` instead of a hardcoded count of 10, so `semantic_core` is required and an unapproved extra crate fails closed. - Clarified ADR 0001 so it owns Rust-first numerical/reference-backend authority while ADR 0011 owns cross-service MSA/service authority. - Clarified ADR 0006 so it owns GPU/VRAM and model-credential boundaries; ADR 0010 now owns LLM orchestration policy and ADR 0015 owns autonomous repository-write/review/merge authority. - Expanded ADR 0002–0005 and 0009–0011 with explicit implementation maturity, alternatives, failure/recovery, compatibility/migration, verification, and rollback/supersession boundaries where they were previously implicit. diff --git a/tests/quality/test_check_docstrings.py b/tests/quality/test_check_docstrings.py index 2c11f7a5..911221b4 100644 --- a/tests/quality/test_check_docstrings.py +++ b/tests/quality/test_check_docstrings.py @@ -11,6 +11,7 @@ from unittest import mock from scripts import check_docstrings as docstrings +from scripts.check_workspace_contract import EXPECTED_CRATES REPOSITORY_ROOT = Path(__file__).resolve().parents[2] @@ -24,7 +25,10 @@ def test_live_repository_is_documented(self) -> None: sources = docstrings.rust_sources(REPOSITORY_ROOT) crate_roots = sorted(REPOSITORY_ROOT.glob("crates/*/src/lib.rs")) - self.assertEqual(len(crate_roots), 10) + self.assertEqual( + sorted(path.parent.parent.name for path in crate_roots), + sorted(EXPECTED_CRATES), + ) self.assertTrue(set(crate_roots).issubset(sources)) self.assertGreaterEqual(len(sources), len(crate_roots)) self.assertEqual(docstrings.validate_repository(REPOSITORY_ROOT), []) From 6afd650667e1011242bb9b0437ce07dc0f6a2431 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 24 Aug 2026 06:45:16 +0900 Subject: [PATCH 3/5] docs(research): cite RFC 5646 once in the APA register The Unicode/language-tags section already recorded Phillips & Davis (2009). Drop the duplicate under topic models. Slice-specific application stays in docs/research/span-grounded-semantic-units.md. --- CHANGELOG.md | 2 +- docs/research/standards-and-literature.md | 2 -- tests/quality/test_check_workspace_contract.py | 8 ++++++++ 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5552e8d5..fa12288f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ### Added -- `semantic_core` binds exact `evidence_core` source spans as semantic units. Language profiles are `unresolved` or a primary ISO 639 subtag with optional region (RFC 5646). Unresolved metadata keeps the caller-supplied Korean `측정` span and does not retokenize. `SemanticIdentity::from_language_tag` fails closed. Korean and English report sentences remain distinct units. Not concept alignment, not invariance, not a topic estimator (ADR 0020; issue #168). +- `semantic_core` binds exact `evidence_core` source spans as semantic units. Language profiles are `unresolved` or a primary ISO 639 subtag with optional region (RFC 5646). Unresolved metadata keeps the caller-supplied Korean `측정` span and does not retokenize. `SemanticIdentity::from_language_tag` fails closed. Korean and English report sentences remain distinct units. Not concept alignment, not invariance, not a topic estimator (ADR 0020; issue #168). The APA register cites RFC 5646 once, in the Unicode/language-tags section; the slice-specific note remains `docs/research/span-grounded-semantic-units.md`. - `tepp_api` naruon live loopback HTTP/1.1 listener: `serve_one` installs a read/write deadline, requires a loopback `Host`, refuses `Transfer-Encoding` and NIM/proxy credential headers, parses `knowledge_cutoff` as RFC 3339 and refuses a future cutoff, keys analysis-run idempotency by tenant plus key, and proves both analysis-run and export POSTs over a real `TcpStream`. Not a production TLS/`$PORT` service (ADR 0011). - `tepp_api` adaptive orchestration router (ADR 0010): versioned `direct`/`verify`/`committee`/`conductor`/`abstain` selection from CPU `f64` risk, ambiguity, evidence, and token-budget inputs; recorded stages, recursion, decomposition, access lists, and role-specific reasoning effort; fail-closed document-controlled policy/access/credentials; LLM plans remain proposals under deterministic statistical authority; comparable-budget ablation requires a direct baseline; credential-free contextual-orchestrator binding. Live NIM HTTP remains accepted-target. - `tepp_api` purpose-bound provider-payload minimization: time-bounded `PurposeGrant` evaluation, fail-closed expired/not-yet-valid/inverted/cross-tenant/impossible-calendar denial, semantic UTC calendar validation, refusal to copy identity mappings into model-provider payloads or ordinary logs, preservation of opaque analytical identifiers and membership roles (no blanket PII mask), a separately authorized scientific re-identification path, and an internally bound FIPS 180-4 SHA-256 audit digest appended through `ReidentificationAuditSink` before disclosure. diff --git a/docs/research/standards-and-literature.md b/docs/research/standards-and-literature.md index 061a8a81..28e62d5c 100644 --- a/docs/research/standards-and-literature.md +++ b/docs/research/standards-and-literature.md @@ -24,8 +24,6 @@ Chang, J., & Blei, D. M. (2009). Relational topic models for document networks. Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., & McCallum, A. (2009). Polylingual topic models. In *Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing* (pp. 880–889). Association for Computational Linguistics. -Phillips, A., & Davis, M. (Eds.). (2009). *Tags for identifying languages* (RFC 5646). Internet Engineering Task Force. https://doi.org/10.17487/RFC5646 - Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for structural topic models. *Journal of Statistical Software, 91*(2), 1–40. https://doi.org/10.18637/jss.v091.i02 Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., & Rand, D. G. (2014). Structural topic models for open-ended survey responses. *American Journal of Political Science, 58*(4), 1064–1082. https://doi.org/10.1111/ajps.12103 diff --git a/tests/quality/test_check_workspace_contract.py b/tests/quality/test_check_workspace_contract.py index 6f3d9c20..40c187f8 100644 --- a/tests/quality/test_check_workspace_contract.py +++ b/tests/quality/test_check_workspace_contract.py @@ -38,6 +38,14 @@ def test_live_repository_satisfies_contract(self) -> None: """The committed workspace satisfies every repository contract.""" self.assertEqual(contract.validate_workspace(REPOSITORY_ROOT), []) + + def test_standards_register_cites_rfc_5646_once(self) -> None: + """The APA register must not duplicate Phillips & Davis RFC 5646.""" + + text = ( + REPOSITORY_ROOT / "docs" / "research" / "standards-and-literature.md" + ).read_text(encoding="utf-8") + self.assertEqual(text.count("RFC 5646"), 1) self.assertEqual( contract.expected_member_paths(), [f"crates/{name}" for name in contract.EXPECTED_CRATES], From 650ea4a311d79304ad46e4e4b954c56abf4f5c91 Mon Sep 17 00:00:00 2001 From: opencode-agent Date: Mon, 24 Aug 2026 12:13:07 +0900 Subject: [PATCH 4/5] fix(semantic): accept RFC 5646 numeric regions; split contract tests - Accept three-digit UN M.49 region subtags so es-419 resolves per Phillips & Davis (2009) sections 2.2.1 and 2.2.4 instead of failing closed on valid regional variants. - Document why SemanticIdentity keys on byte coordinates only. - Split the absorbed workspace-contract assertions into focused tests. --- crates/semantic_core/src/profile.rs | 44 +++++++++++++++++-- crates/semantic_core/src/unit.rs | 6 +++ .../quality/test_check_workspace_contract.py | 12 +++++ 3 files changed, 58 insertions(+), 4 deletions(-) diff --git a/crates/semantic_core/src/profile.rs b/crates/semantic_core/src/profile.rs index 1565b06c..11ff3203 100644 --- a/crates/semantic_core/src/profile.rs +++ b/crates/semantic_core/src/profile.rs @@ -24,9 +24,12 @@ impl LanguageProfile { /// Parse a primary language subtag with an optional region. /// - /// Accepts two or three ASCII letters, optionally followed by a hyphen and - /// a two-letter region. The stored form is lowercase. Empty tags and any - /// other shape fail closed. The tag never becomes a [`crate::SemanticIdentity`]. + /// Accepts a two- or three-letter primary language subtag, optionally + /// followed by a hyphen and either an ISO 3166-1 alpha-2 region or a + /// three-digit UN M.49 numeric region (Phillips & Davis, 2009, section + /// 2.2.4; for example `es-419`). The stored form is lowercase. Empty tags + /// and any other shape fail closed. The tag never becomes a + /// [`crate::SemanticIdentity`]. /// /// # Errors /// @@ -59,13 +62,26 @@ impl LanguageProfile { } } +/// Validate a primary language tag per RFC 5646 sections 2.2.1 and 2.2.4. +/// +/// A region may be an ISO 3166-1 alpha-2 code or a UN M.49 three-digit +/// numeric subtag; both are accepted here so regional variants such as +/// Latin-American Spanish resolve instead of failing closed. fn is_primary_language_tag(tag: &str) -> bool { match tag.split_once('-') { None => is_letter_run(tag, 2, 3), - Some((language, region)) => is_letter_run(language, 2, 3) && is_letter_run(region, 2, 2), + Some((language, region)) => { + is_letter_run(language, 2, 3) + && (is_letter_run(region, 2, 2) || is_numeric_region(region)) + } } } +/// Return whether `region` is exactly three ASCII digits. +fn is_numeric_region(region: &str) -> bool { + region.len() == 3 && region.bytes().all(|byte| byte.is_ascii_digit()) +} + fn is_letter_run(value: &str, min: usize, max: usize) -> bool { (min..=max).contains(&value.len()) && value.bytes().all(|byte| byte.is_ascii_lowercase()) } @@ -82,6 +98,14 @@ mod tests { assert!(!korean.is_unresolved()); let tagged = LanguageProfile::parse_bcp47("en-US").expect("en-us"); assert_eq!(tagged.as_str(), "en-us"); + let numeric = LanguageProfile::parse_bcp47("es-419").expect("es-419"); + assert_eq!(numeric.as_str(), "es-419"); + assert_eq!( + LanguageProfile::parse_bcp47("ES-419") + .expect("upper") + .as_str(), + "es-419" + ); assert_eq!( LanguageProfile::parse_bcp47("yue").expect("yue").as_str(), "yue" @@ -122,6 +146,18 @@ mod tests { LanguageProfile::parse_bcp47("e-us").unwrap_err(), SemanticError::InvalidLanguageTag ); + assert_eq!( + LanguageProfile::parse_bcp47("es-41a").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("es-41").unwrap_err(), + SemanticError::InvalidLanguageTag + ); + assert_eq!( + LanguageProfile::parse_bcp47("es-4199").unwrap_err(), + SemanticError::InvalidLanguageTag + ); let unresolved = LanguageProfile::unresolved(); assert!(unresolved.is_unresolved()); assert_eq!(unresolved.as_str(), "unresolved"); diff --git a/crates/semantic_core/src/unit.rs b/crates/semantic_core/src/unit.rs index 1aa22524..48125ee4 100644 --- a/crates/semantic_core/src/unit.rs +++ b/crates/semantic_core/src/unit.rs @@ -5,6 +5,12 @@ use crate::profile::LanguageProfile; use evidence_core::{EvidenceId, SourceSpan}; /// Exact-span identity of one semantic unit. +/// +/// Identity is `(document_id, byte_start, byte_end)`. Scalar character +/// coordinates are derived deterministically from byte coordinates inside one +/// document encoding, so they add no distinguishing power; page or layout +/// positions are presentation metadata that may vary across renderings and +/// therefore stay outside identity. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub struct SemanticIdentity { document_id: EvidenceId, diff --git a/tests/quality/test_check_workspace_contract.py b/tests/quality/test_check_workspace_contract.py index 40c187f8..a57bd627 100644 --- a/tests/quality/test_check_workspace_contract.py +++ b/tests/quality/test_check_workspace_contract.py @@ -46,10 +46,18 @@ def test_standards_register_cites_rfc_5646_once(self) -> None: REPOSITORY_ROOT / "docs" / "research" / "standards-and-literature.md" ).read_text(encoding="utf-8") self.assertEqual(text.count("RFC 5646"), 1) + + def test_member_paths_match_expected_crates(self) -> None: + """Workspace members resolve to the approved crate roots by name.""" + self.assertEqual( contract.expected_member_paths(), [f"crates/{name}" for name in contract.EXPECTED_CRATES], ) + + def test_placeholder_api_detection_boundaries(self) -> None: + """Real APIs pass and placeholder or todo bodies are refused.""" + self.assertFalse(contract._contains_placeholder_api("//! documented\n")) self.assertFalse( contract._contains_placeholder_api("/// Real API.\npub struct EvidenceId;\n") @@ -63,6 +71,10 @@ def test_standards_register_cites_rfc_5646_once(self) -> None: self.assertTrue( contract._contains_placeholder_api("fn private() { unimplemented!() }\n") ) + + def test_mapping_normalizes_only_toml_tables(self) -> None: + """TOML tables map to dictionaries; other shapes fail to empty maps.""" + self.assertEqual(contract._mapping({"key": "value"}), {"key": "value"}) self.assertEqual(contract._mapping("not-a-table"), {}) From 57be20fc361aa0a812b0ed4e0eaffc70680d7123 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 24 Aug 2026 12:26:23 +0900 Subject: [PATCH 5/5] fix(semantic): reject unregistered language regions --- CHANGELOG.md | 2 +- crates/semantic_core/src/profile.rs | 45 +++++++++++++++---- .../tests/language_profile_contract.rs | 17 +++++++ docs/adr/0020-span-grounded-semantic-units.md | 6 ++- docs/research/standards-and-literature.md | 2 + 5 files changed, 61 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa12288f..c45c67b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang ### Added -- `semantic_core` binds exact `evidence_core` source spans as semantic units. Language profiles are `unresolved` or a primary ISO 639 subtag with optional region (RFC 5646). Unresolved metadata keeps the caller-supplied Korean `측정` span and does not retokenize. `SemanticIdentity::from_language_tag` fails closed. Korean and English report sentences remain distinct units. Not concept alignment, not invariance, not a topic estimator (ADR 0020; issue #168). The APA register cites RFC 5646 once, in the Unicode/language-tags section; the slice-specific note remains `docs/research/span-grounded-semantic-units.md`. +- `semantic_core` binds exact `evidence_core` source spans as semantic units. Language profiles are `unresolved` or a primary ISO 639 subtag with an IANA-registered ISO 3166-1 alpha-2 or UN M.49 region (RFC 5646; IANA File-Date 2026-08-08); private-use and unknown regions fail closed. Unresolved metadata keeps the caller-supplied Korean `측정` span and does not retokenize. `SemanticIdentity::from_language_tag` fails closed. Korean and English report sentences remain distinct units. Not concept alignment, not invariance, not a topic estimator (ADR 0020; issue #168). The APA register cites RFC 5646 once, in the Unicode/language-tags section; the slice-specific note remains `docs/research/span-grounded-semantic-units.md`. - `tepp_api` naruon live loopback HTTP/1.1 listener: `serve_one` installs a read/write deadline, requires a loopback `Host`, refuses `Transfer-Encoding` and NIM/proxy credential headers, parses `knowledge_cutoff` as RFC 3339 and refuses a future cutoff, keys analysis-run idempotency by tenant plus key, and proves both analysis-run and export POSTs over a real `TcpStream`. Not a production TLS/`$PORT` service (ADR 0011). - `tepp_api` adaptive orchestration router (ADR 0010): versioned `direct`/`verify`/`committee`/`conductor`/`abstain` selection from CPU `f64` risk, ambiguity, evidence, and token-budget inputs; recorded stages, recursion, decomposition, access lists, and role-specific reasoning effort; fail-closed document-controlled policy/access/credentials; LLM plans remain proposals under deterministic statistical authority; comparable-budget ablation requires a direct baseline; credential-free contextual-orchestrator binding. Live NIM HTTP remains accepted-target. - `tepp_api` purpose-bound provider-payload minimization: time-bounded `PurposeGrant` evaluation, fail-closed expired/not-yet-valid/inverted/cross-tenant/impossible-calendar denial, semantic UTC calendar validation, refusal to copy identity mappings into model-provider payloads or ordinary logs, preservation of opaque analytical identifiers and membership roles (no blanket PII mask), a separately authorized scientific re-identification path, and an internally bound FIPS 180-4 SHA-256 audit digest appended through `ReidentificationAuditSink` before disclosure. diff --git a/crates/semantic_core/src/profile.rs b/crates/semantic_core/src/profile.rs index 11ff3203..4803e2fc 100644 --- a/crates/semantic_core/src/profile.rs +++ b/crates/semantic_core/src/profile.rs @@ -24,7 +24,7 @@ impl LanguageProfile { /// Parse a primary language subtag with an optional region. /// - /// Accepts a two- or three-letter primary language subtag, optionally + /// Accepts a syntactically valid two- or three-letter primary language subtag, /// followed by a hyphen and either an ISO 3166-1 alpha-2 region or a /// three-digit UN M.49 numeric region (Phillips & Davis, 2009, section /// 2.2.4; for example `es-419`). The stored form is lowercase. Empty tags @@ -71,21 +71,37 @@ fn is_primary_language_tag(tag: &str) -> bool { match tag.split_once('-') { None => is_letter_run(tag, 2, 3), Some((language, region)) => { - is_letter_run(language, 2, 3) - && (is_letter_run(region, 2, 2) || is_numeric_region(region)) + is_letter_run(language, 2, 3) && is_registered_region_subtag(region) } } } -/// Return whether `region` is exactly three ASCII digits. -fn is_numeric_region(region: &str) -> bool { - region.len() == 3 && region.bytes().all(|byte| byte.is_ascii_digit()) -} - fn is_letter_run(value: &str, min: usize, max: usize) -> bool { (min..=max).contains(&value.len()) && value.bytes().all(|byte| byte.is_ascii_lowercase()) } +// IANA Language Subtag Registry, File-Date 2026-08-08. Private-use region +// ranges and private-use records are intentionally excluded because this +// profile accepts only reproducible registered region metadata. +const REGISTERED_ALPHA2_REGION_SUBTAGS: &str = "ac ad ae af ag ai al am an ao aq ar as at au aw ax az ba bb bd be bf bg bh bi bj bl bm bn bo bq br bs bt bu bv bw by bz ca cc cd cf cg ch ci ck cl cm cn co cp cq cr cs cu cv cw cx cy cz dd de dg dj dk dm do dz ea ec ee eg eh er es et eu ez fi fj fk fm fo fr fx ga gb gd ge gf gg gh gi gl gm gn gp gq gr gs gt gu gw gy hk hm hn hr ht hu ic id ie il im in io iq ir is it je jm jo jp ke kg kh ki km kn kp kr kw ky kz la lb lc li lk lr ls lt lu lv ly ma mc md me mf mg mh mk ml mm mn mo mp mq mr ms mt mu mv mw mx my mz na nc ne nf ng ni nl no np nr nt nu nz om pa pe pf pg ph pk pl pm pn pr ps pt pw py qa re ro rs ru rw sa sb sc sd se sg sh si sj sk sl sm sn so sr ss st su sv sx sy sz ta tc td tf tg th tj tk tl tm tn to tp tr tt tv tw tz ua ug um un us uy uz va vc ve vg vi vn vu wf ws yd ye yt yu za zm zr zw"; + +const REGISTERED_NUMERIC_REGION_SUBTAGS: &[&str] = &[ + "001", "002", "003", "005", "009", "011", "013", "014", "015", "017", "018", "019", "021", + "029", "030", "034", "035", "039", "053", "054", "057", "061", "142", "143", "145", "150", + "151", "154", "155", "202", "419", +]; + +fn is_registered_region_subtag(region: &str) -> bool { + if region.len() == 2 { + return REGISTERED_ALPHA2_REGION_SUBTAGS + .split_ascii_whitespace() + .any(|candidate| candidate == region); + } + region.len() == 3 + && region.bytes().all(|byte| byte.is_ascii_digit()) + && REGISTERED_NUMERIC_REGION_SUBTAGS.contains(®ion) +} + #[cfg(test)] mod tests { use super::LanguageProfile; @@ -106,6 +122,12 @@ mod tests { .as_str(), "es-419" ); + assert_eq!( + LanguageProfile::parse_bcp47("en-GB") + .expect("registered alpha-2 region") + .as_str(), + "en-gb" + ); assert_eq!( LanguageProfile::parse_bcp47("yue").expect("yue").as_str(), "yue" @@ -158,6 +180,13 @@ mod tests { LanguageProfile::parse_bcp47("es-4199").unwrap_err(), SemanticError::InvalidLanguageTag ); + for tag in ["en-aa", "en-XX", "en-QM", "en-abc", "en-999"] { + assert_eq!( + LanguageProfile::parse_bcp47(tag), + Err(SemanticError::InvalidLanguageTag), + "private or unknown region must fail closed: {tag}" + ); + } let unresolved = LanguageProfile::unresolved(); assert!(unresolved.is_unresolved()); assert_eq!(unresolved.as_str(), "unresolved"); diff --git a/crates/semantic_core/tests/language_profile_contract.rs b/crates/semantic_core/tests/language_profile_contract.rs index 931cc548..32538691 100644 --- a/crates/semantic_core/tests/language_profile_contract.rs +++ b/crates/semantic_core/tests/language_profile_contract.rs @@ -52,3 +52,20 @@ fn missing_language_does_not_retokenize_or_steal_identity() { Err(SemanticError::LanguageIsNotIdentity) ); } + +#[test] +fn language_profiles_use_registered_iana_region_subtags() { + assert_eq!( + LanguageProfile::parse_bcp47("es-419") + .expect("registered numeric region") + .as_str(), + "es-419" + ); + for tag in ["en-aa", "en-XX", "en-QM", "en-abc", "en-999"] { + assert_eq!( + LanguageProfile::parse_bcp47(tag), + Err(SemanticError::InvalidLanguageTag), + "private or unknown region must fail closed: {tag}" + ); + } +} diff --git a/docs/adr/0020-span-grounded-semantic-units.md b/docs/adr/0020-span-grounded-semantic-units.md index 8f0cec8f..fa5cf19d 100644 --- a/docs/adr/0020-span-grounded-semantic-units.md +++ b/docs/adr/0020-span-grounded-semantic-units.md @@ -22,7 +22,8 @@ Add standalone crate `semantic_core` as the first ADR 0004 production slice: - a semantic unit is identified by exact `SourceSpan` coordinates (document identity plus byte start/end); - a language profile is optional metadata (`unresolved` or a primary ISO 639 - subtag with optional region); + subtag with an optional region validated against the pinned IANA Language + Subtag Registry snapshot dated 2026-08-08); - unresolved language keeps the caller-supplied span and does not switch segmentation heuristic; - `SemanticIdentity::from_language_tag` fails closed; @@ -56,7 +57,8 @@ units without inheriting language-as-identity. ## Failure and recovery -Empty or malformed language tags fail closed. Offering a language tag as +Empty, malformed, private-use, or unknown-region language tags fail closed. +Offering a language tag as identity fails closed. Recovery supplies a valid exact span and optional profile; it does not rewrite historical artifacts. diff --git a/docs/research/standards-and-literature.md b/docs/research/standards-and-literature.md index 28e62d5c..4dfe0f0d 100644 --- a/docs/research/standards-and-literature.md +++ b/docs/research/standards-and-literature.md @@ -76,6 +76,8 @@ Davis, M., Iancu, L., & Whistler, K. (Eds.). (2024). *Unicode Standard Annex #29 Phillips, A., & Davis, M. (2009). *Tags for identifying languages* (RFC 5646). Internet Engineering Task Force. https://doi.org/10.17487/RFC5646 +Internet Assigned Numbers Authority. (2026). *Language subtag registry* (File-Date 2026-08-08). https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry + Nivre, J., de Marneffe, M.-C., Ginter, F., Hajič, J., Manning, C. D., Pyysalo, S., Schuster, S., Tyers, F., & Zeman, D. (2020). Universal Dependencies v2: An evergrowing multilingual treebank collection. In *Proceedings of the 12th Language Resources and Evaluation Conference* (pp. 4034–4043). European Language Resources Association. The original source is preserved. NFC is used for canonical analysis views; compatibility normalization is limited to explicit auxiliary keys. Segmentation and morphology are language-tailored. Universal POS informs source priors rather than irreversible deletion.