diff --git a/docs/adr/0006-language-agnostic-semantic-span-embedding.md b/docs/adr/0006-language-agnostic-semantic-span-embedding.md new file mode 100644 index 000000000..f2c362831 --- /dev/null +++ b/docs/adr/0006-language-agnostic-semantic-span-embedding.md @@ -0,0 +1,167 @@ +# ADR 0006 — Language-agnostic semantic spans for token-safe embeddings + +**Decision status:** Accepted +**Date:** 2026-08-15 + +## Context + +LineageWeave already embeds paragraph-level units so a short relevant passage is +not diluted by a complete document. The existing sentence helper, however, +contains script and capitalization assumptions, and neither the paragraph nor +sentence path proves that a final request fits the selected embedding model's +input context. Fixed token windows would solve overflow but cut authored meaning +and depend on duplicate overlap to restore context. Language-routed NLP stacks +would fail on code-switching and would make language classification an +unnecessary operational dependency. + +The product requirement is stricter: segmentation must work without caring +which language produced the text, must remain below the +`text-embedding-3-large` context limit, and must not use TF-IDF. + +OpenAI currently documents an 8,192-token input maximum for +`text-embedding-3-large`, recommends Tiktoken with `cl100k_base` for +third-generation embedding token counts, and normalizes embedding vectors so +cosine similarity is an appropriate ranking measure. Unicode NFC provides a +stable canonical representation without erasing compatibility distinctions. + +## Decision + +Add `lineageweave.semantic_spans` and make it the planned successor to the +legacy paragraph/sentence-only embedding segmentation path. + +The module: + +- never identifies a language or dispatches by language/script; +- normalizes input to NFC, standardizes line endings, and preserves ZWJ/ZWNJ; +- creates micro-units from authored structure and script-diverse terminal + punctuation; +- accepts an exact model `TokenCodec`, with a lazy `cl100k_base` Tiktoken + adapter supplied for `text-embedding-3-large` deployments; +- uses exact token windows only as the last-resort split for one structurally + indivisible unit; +- combines structural boundary strength, dense adjacent semantic drop, and + current-length pressure; +- defaults to a 700-token target, 1,200-token leaf ceiling, and 256-token final + request reserve beneath the provider maximum; +- caches each unique micro-unit embedding during adjacency comparisons; +- continues with structure plus token budget when the dense provider is absent, + rather than fabricating similarity or falling back to TF-IDF; +- renders only high-signal metadata and re-counts the final payload before the + provider call; and +- exposes previous/next indices and a `Chunk` adapter so the existing + `chunked_max_similarity` API can consume the new spans. + +The accepted boundary score is: + +```text +B_i = ( + 0.35 * structure_break + + 0.45 * (1 - adjacent_dense_similarity) + + 0.20 * min(1, current_tokens / target_tokens) +) +``` + +A semantic boundary is taken only after the current span reaches its minimum +size and `B_i >= 0.55`; an impending token-ceiling violation always takes a +boundary. All weights, thresholds, and token targets are policy-versioned and +must be tuned on retrieval evidence before default-on production rollout. + +Generative LLM work—section/document summaries, synthetic evaluation queries, +and ambiguous-case adjudication—must go through contextual-orchestrator. It is +not part of the deterministic leaf-packing hot path. + +The detailed product, persistence, evaluation, rollout, and governance plan is +in [`../language-agnostic-semantic-span-plan.md`](../language-agnostic-semantic-span-plan.md). + +## Consequences + +### Positive + +- One path handles arbitrary and mixed Unicode scripts. +- Exact model token accounting makes provider overflow preventable and + testable. +- Dense semantic changes can create boundaries without TF-IDF or morphology. +- Existing embedding clients and `chunked_max_similarity` remain compatible. +- Policy and codec injection allow model changes without rewriting the packer. +- Neighbor metadata prepares hierarchical context restoration without making + fixed overlap the primary design. + +### Costs and limitations + +- Dense adjacency scoring adds embedding work during indexing; production must + batch and cache calls. +- Punctuation remains an imperfect weak boundary cue, especially for + abbreviations; dense similarity and span packing mitigate rather than erase + that ambiguity. +- The hosted OpenAI API does not expose token-level hidden states required for + true late chunking, so this implementation performs pre-embedding semantic + segmentation. +- The baseline PR does not yet add persistence migrations, parent summaries, or + a default-on feature flag. Those are explicit rollout phases, not implied to + exist. +- `TiktokenTokenCodec` imports Tiktoken lazily; a deployment choosing that + adapter must include Tiktoken in the embedding worker or inject another exact + codec. + +## Alternatives rejected + +### Language-specific morphology and sentence tokenizers + +Rejected as a required control path. They introduce language identification, +code-switching failure modes, and per-language maintenance. They may be optional +research signals later but cannot determine the safety budget. + +### TF-IDF/TextTiling lexical cohesion + +Rejected for this feature. TextTiling motivates topic-boundary thinking, but +TF-IDF/lexical cohesion is not the requested semantic signal. Dense adjacency +similarity supplies the semantic component. + +### Fixed token windows with global overlap + +Rejected as the primary design because they split meaning mechanically and +multiply storage. Exact windows remain only the final safety fallback; +parent/neighbor links restore context. + +### Translate everything into one language + +Rejected because it increases cost and latency, can alter names and domain +meaning, and makes source-grounded offsets difficult to audit. + +### Whole-document embedding + +Rejected because it can exceed the provider limit and dilute short relevant +passages. + +## Verification + +This change includes synthetic tests covering: + +- Korean, Japanese, Arabic, and English in one no-language-label input; +- punctuation without whitespace or capitalization assumptions; +- decimal preservation; +- structural unit types; +- dense semantic boundaries and embedding-cache reuse; +- provider-free deterministic fallback; +- oversized punctuation-free token windows; +- contiguous neighbor links; +- final metadata-plus-content overflow rejection; +- invalid policy configurations; and +- the existing `Chunk` adapter contract. + +The new module is covered at 100% in the focused test run and makes no network +calls. + +## References (APA 7th) + +Günther, M., Mohr, I., Williams, D. J., Wang, B., & Xiao, H. (2024). *Late chunking: Contextual chunk embeddings using long-context embedding models* [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2409.04701 + +Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages. *Computational Linguistics, 23*(1), 33–64. https://aclanthology.org/J97-1003/ + +OpenAI. (2026). *Embeddings guide*. OpenAI API documentation. Retrieved August 15, 2026, from https://platform.openai.com/docs/guides/embeddings + +OpenAI. (2026). *Embeddings FAQ*. OpenAI Help Center. Retrieved August 15, 2026, from https://help.openai.com/en/articles/6824809-embeddings-faq + +The Unicode Consortium. (2025). *Unicode Standard Annex #15: Unicode normalization forms* (Revision 57). https://www.unicode.org/reports/tr15/ + +The Unicode Consortium. (2025). *Unicode Standard Annex #29: Unicode text segmentation* (Revision 47). https://www.unicode.org/reports/tr29/ diff --git a/docs/language-agnostic-semantic-span-plan.md b/docs/language-agnostic-semantic-span-plan.md new file mode 100644 index 000000000..fa02b786e --- /dev/null +++ b/docs/language-agnostic-semantic-span-plan.md @@ -0,0 +1,557 @@ +# Language-Agnostic Semantic Span Embedding Plan + +**Product:** LineageWeave +**Decision owner:** ContextualWisdomLab +**Date:** 2026-08-15 +**Status:** Implementation baseline proposed in this change + +## 1. Executive decision + +LineageWeave will stop treating language identification as a prerequisite for +embedding segmentation. The ingestion path will accept arbitrary Unicode text, +including documents that mix scripts inside the same paragraph, and will build +embedding inputs from: + +1. authored document structure; +2. exact token counts from the selected embedding model's codec; and +3. optional dense-embedding similarity between adjacent micro-units. + +The system will not use TF-IDF, whitespace-delimited word counts, +translation-first preprocessing, or language-specific morphology as a control +path. A language label may be stored for analytics, but it must not select a +chunking algorithm or change the token budget. + +The first implementation is `lineageweave.semantic_spans`. It is directly +compatible with the existing `chunked_max_similarity` contract through +`make_semantic_span_chunker`. + +## 2. Why LineageWeave owns this capability + +LineageWeave already has all of the adjacent responsibilities: + +- an OpenAI-compatible embedding client; +- paragraph, sentence, DOM, image, and conversation-turn chunking; +- chunk-level maximum similarity for lineage reconstruction; +- contextual-orchestrator as the approved path for paid model calls; and +- tests for embedding and chunk behavior. + +The capability therefore belongs beside the existing embedding channel rather +than in TEPP, which owns temporal-relational measurement, or in +contextual-orchestrator, which owns provider routing and policy rather than +document semantics. The algorithm remains provider-neutral so other +ContextualWisdomLab products can import it later instead of reimplementing it. + +## 3. Problem statement + +A document-wide embedding can over-compress several topics into one vector and +can exceed the provider's input context. A fixed token window prevents overflow +but cuts through authored meaning, duplicates content through overlap, and loses +section context. Language-routed tokenizers add another failure mode: +code-switching, translated quotations, product names, source code, and mixed +CJK/Latin/Arabic content can occur in the same span. + +For `text-embedding-3-large`, OpenAI currently documents a maximum input of +8,192 tokens and recommends `cl100k_base`/Tiktoken for token accounting. The +absolute limit is a guardrail, not a desirable retrieval unit. This plan keeps a +256-token final-payload reserve and defaults leaf spans to a 700-token target +with a 1,200-token ceiling. + +## 4. Product outcomes + +The product must provide: + +- **zero provider overflow:** no submitted embedding payload may exceed its + configured model budget; +- **language-independent behavior:** identical code paths for every Unicode + script and for mixed-script content; +- **meaning-preserving retrieval units:** short related units may merge, while + a dense semantic drop can create a boundary before the token ceiling; +- **traceable context restoration:** every leaf span has stable order and + previous/next links, with parent levels added during hierarchical indexing; +- **reproducibility:** policy version, model, tokenizer/codec, dimensions, + source offsets, and content hash can be persisted; and +- **graceful degradation:** provider unavailability removes the dense signal + and continues with structure plus exact token budget; it never invents a + similarity score or falls back to TF-IDF. + +## 5. Non-goals + +This work does not: + +- infer a language and select an NLP pipeline; +- translate all content into English before embedding; +- define tokens by characters, bytes, words, or whitespace; +- use TF-IDF/BM25 as a semantic-boundary surrogate; +- claim that punctuation alone is a perfect sentence segmenter; +- expose a raw model API key from LineageWeave; or +- implement late chunking inside a hosted model whose token-level hidden states + are not exposed. + +BM25 may still exist independently as a retrieval channel elsewhere, but it is +not a segmentation dependency and is not a fallback for this feature. + +## 6. User stories + +### US-1: mixed-script analyst search + +As an analyst, I can search records containing any mixture of scripts and +receive the passage that carries the relevant meaning rather than a diluted +whole-document vector. + +**Acceptance:** the runtime receives no language label, still returns token-safe +semantic spans, and preserves source order. + +### US-2: ingestion operator safety + +As an ingestion operator, I can change the embedding model's tokenizer and +context limit through an explicit policy/codec without editing the segmentation +algorithm. + +**Acceptance:** an invalid policy fails at startup; final metadata-plus-content +payloads are checked before the provider call. + +### US-3: lineage investigator context + +As a lineage investigator, I can expand a matching span to its previous, next, +and parent context without re-embedding the complete document. + +**Acceptance:** leaf spans expose contiguous neighbor indices; persistence adds +parent and sibling relationships without duplicating source text. + +### US-4: auditor reproducibility + +As an auditor, I can identify which policy, tokenizer, model, vector dimension, +and source content produced a stored vector. + +**Acceptance:** persisted records are immutable by version and identified by +content hashes. + +## 7. Functional requirements + +| ID | Requirement | +| --- | --- | +| FR-01 | Normalize input as Unicode NFC and standardize line endings without compatibility folding. | +| FR-02 | Preserve ZWJ/ZWNJ; remove only transport-noise zero-width space/BOM. | +| FR-03 | Generate micro-units from paragraph, line, heading, list, table, code-fence, and script-diverse terminal-punctuation signals. | +| FR-04 | Do not call language identification or language-specific morphological analyzers. | +| FR-05 | Count tokens with an injected codec authoritative for the selected embedding model. | +| FR-06 | Recursively reduce an oversized unit, ending with exact token-window splitting as the last resort. | +| FR-07 | Score adjacent boundaries with structure, dense semantic drop, and length pressure. | +| FR-08 | Split unconditionally before adding a unit that would exceed `max_span_tokens`. | +| FR-09 | Validate the final metadata-plus-content payload against `model_max_tokens - request_reserve_tokens`. | +| FR-10 | Return source-unit membership and previous/next span indices. | +| FR-11 | Adapt spans to the existing `Chunk` interface. | +| FR-12 | Cache each micro-unit embedding during adjacent comparisons. | +| FR-13 | Continue deterministically without a dense provider using structure and token budget only. | + +## 8. Non-functional requirements + +- New algorithm unit coverage: **100%**. +- Provider overflow rate: **0%**. +- Deterministic result for a fixed normalized input, codec, embedding vectors, + and policy. +- No network access in unit tests. +- No real customer or employer data in fixtures. +- Linear span packing after micro-unit embeddings are available. +- Bounded memory proportional to one document's micro-units and vectors. +- All future database object names use two-or-more-word `snake_case` names and + remain in third normal form. + +## 9. Processing architecture + +```text +Unicode input + -> NFC/transport normalization + -> authored-structure parser + -> language-agnostic micro-units + -> exact model-token accounting + -> optional cached dense adjacency similarity + -> semantic boundary score + -> token-safe leaf span packing + -> metadata payload guard + -> embedding provider via contextual-orchestrator + -> leaf/section/document vector indexes + -> candidate retrieval + parent/neighbor expansion +``` + +### 9.1 Micro-units + +A micro-unit is not the final retrieval chunk. It is a small ordered piece from +which spans are packed. Boundary strengths are structural priors, not language +rules: + +- first unit: `1.00`; +- Markdown heading/list/table/code boundary: at least `0.90`; +- new authored paragraph: `0.75`; +- new non-empty line: `0.45`; and +- subsequent terminal-punctuation unit on the same line: `0.25`. + +A period between two digits is preserved as a decimal rather than split. +Terminal punctuation covers several Unicode scripts and does not require a +following space or an uppercase next character. + +### 9.2 Boundary equation + +For candidate boundary `i`: + +```text +B_i = ( + w_structure * S_i + + w_semantic * (1 - similarity(E_(i-1), E_i)) + + w_length * min(1, current_tokens / target_tokens) +) / sum(weights) +``` + +Default weights: + +```text +w_structure = 0.35 +w_semantic = 0.45 +w_length = 0.20 +threshold = 0.55 +``` + +The packer starts a new span when either: + +1. adding the unit would exceed the leaf ceiling; or +2. the current span has reached the minimum size and `B_i >= threshold`. + +These defaults are hypotheses and must be tuned against a labeled retrieval +set. They are versioned configuration, not universal constants. + +### 9.3 Token policy + +```text +model_max_tokens = 8192 +request_reserve = 256 +usable_request_budget = 7936 +minimum_leaf = 120 +leaf_target = 700 +leaf_ceiling = 1200 +micro_unit_ceiling = 320 +``` + +The leaf ceiling is deliberately far below the provider maximum. Metadata is +rendered only from short, high-signal fields such as title, heading path, +block type, speaker, and event date. The final renderer counts those fields and +the content together; it rejects overflow before any HTTP request. + +### 9.4 Dense boundary provider + +`CachedEmbeddingSimilarity` accepts the existing LineageWeave embedding-client +shape (`embed(text) -> vector`). Production calls continue through +contextual-orchestrator's OpenAI-compatible endpoint. The cache ensures one +vector per unique micro-unit within a document. + +The next optimization is a batched embedding call for all micro-units in a +document. It changes transport efficiency, not segmentation semantics. + +### 9.5 LLM responsibilities + +A generative LLM is not in the deterministic leaf-packing hot path. Through +contextual-orchestrator it may later: + +- produce section/document summaries; +- generate synthetic evaluation queries; +- adjudicate ambiguous offline boundary examples; and +- explain why a retrieved leaf belongs to a lineage candidate. + +It must not silently rewrite source content, invent a missing signal, or make a +raw provider call from this repository. + +## 10. Hierarchical retrieval design + +### Level 0: leaf spans + +- primary precision index; +- original source text; +- stable source offsets and content hash; +- previous/next links; +- default 120–1,200 tokens. + +### Level 1: section spans + +- parent of contiguous leaf spans under an authored heading or inferred section; +- concise extractive or LLM-assisted summary; +- used for candidate narrowing and context restoration. + +### Level 2: document spans + +- one document-level descriptor and summary; +- used for coarse routing, filters, and document-level ranking. + +Recommended retrieval flow: + +1. embed the query once; +2. retrieve document/section candidates; +3. search leaf vectors inside those candidates; +4. rerank; +5. expand selected leaves with parent and bounded neighbor context; and +6. deduplicate parallel or overlapping evidence before generation. + +Fixed overlap is reserved for the final token-window fallback. Hierarchical and +neighbor links are the default context-restoration mechanism. + +## 11. Proposed normalized persistence model + +No migration is included in this baseline PR. The implementation phase should +introduce these third-normal-form objects: + +### `embedding_policy_version` + +- `embedding_policy_id` (PK) +- `policy_version_code` (unique) +- `model_name_text` +- `token_codec_name` +- `model_token_limit` +- `request_reserve_count` +- `target_span_count` +- `maximum_span_count` +- `minimum_span_count` +- `vector_dimension_count` +- `policy_payload_json` +- `created_at` + +### `embedding_document_record` + +- `embedding_document_id` (PK) +- `tenant_account_id` (FK where tenancy applies) +- `source_record_id` (FK to the owning source record) +- `source_content_hash` +- `document_title_text` +- `created_at` + +### `semantic_span_record` + +- `semantic_span_id` (PK) +- `embedding_document_id` (FK) +- `parent_span_id` (nullable self-FK) +- `embedding_policy_id` (FK) +- `span_level_code` +- `span_order_number` +- `source_start_offset` +- `source_end_offset` +- `content_token_count` +- `content_hash_value` +- `content_text` +- `created_at` + +Unique constraint: +`(embedding_document_id, span_level_code, span_order_number)`. + +### `semantic_span_edge` + +- `semantic_span_edge_id` (PK) +- `source_span_id` (FK) +- `target_span_id` (FK) +- `edge_type_code` (`edge_previous`, `edge_next`, `edge_parent`, + `edge_parallel`) +- `created_at` + +### `embedding_vector_record` + +- `embedding_vector_id` (PK) +- `semantic_span_id` (FK) +- `embedding_policy_id` (FK) +- `vector_dimension_count` +- `vector_value` +- `created_at` + +This separates source identity, span structure, versioned policy, and generated +vectors. A new vector model does not overwrite source spans or historical +vectors. + +## 12. API and code usage + +```python +from lineageweave.embedding_client import ( + OpenAiCompatibleEmbeddingClient, + chunked_max_similarity, +) +from lineageweave.semantic_spans import ( + TiktokenTokenCodec, + make_semantic_span_chunker, +) + +client = OpenAiCompatibleEmbeddingClient( + base_url=orchestrator_url, + api_key=service_token, + model="text-embedding-3-large", +) +chunker = make_semantic_span_chunker( + codec=TiktokenTokenCodec("cl100k_base"), + embedder=client, +) +score, left_span, right_span = chunked_max_similarity( + client, + left_text, + right_text, + chunker=chunker, +) +``` + +A deployment may inject another exact `TokenCodec`, including a tokenizer +service owned by `pg-llm-batch`, without changing the packer. + +## 13. Evaluation plan + +### 13.1 Baselines + +1. current paragraph chunker; +2. fixed 800-token window with 100-token overlap; and +3. structure-only semantic spans without dense boundary scoring. + +### 13.2 Proposed treatment + +Structure + exact token budget + dense adjacency similarity + hierarchical +retrieval. + +### 13.3 Corpus + +Use synthetic and public/licensed documents that contain: + +- single-script prose; +- mixed scripts within one sentence and paragraph; +- CJK text without whitespace; +- right-to-left text; +- translated quotations; +- headings, lists, tables, code, and dialogue; +- one extremely long punctuation-free unit; and +- metadata large enough to test final-payload overflow. + +Evaluation labels may describe script composition for analysis. Runtime must +not consume those labels or route on a language classification. + +### 13.4 Metrics + +- overflow rate; +- Recall@5/10; +- nDCG@10; +- MRR; +- boundary precision/recall/F1 on annotated topic transitions; +- human-rated span coherence; +- context-restoration success; +- duplicate evidence rate; +- embedding tokens and calls per source token; +- p50/p95 indexing latency; +- p50/p95 query latency; and +- storage per document at 3,072/1,536/1,024 dimensions. + +### 13.5 Promotion gates + +- `overflow_rate == 0` across adversarial tests; +- Recall@10 no worse than current paragraph baseline; +- statistically meaningful improvement over fixed windows on at least one of + Recall@10 or nDCG@10 without material regression in the other; +- mixed-script slice within five percentage points of the overall retrieval + score; +- no unit-test network calls; and +- no real organization data in repository fixtures. + +## 14. Observability + +Emit structured metrics by policy version, not by inferred language: + +- `semantic_span_documents_total` +- `semantic_span_units_total` +- `semantic_span_token_count` +- `semantic_span_boundary_score` +- `semantic_span_forced_split_total` +- `semantic_span_provider_fallback_total` +- `semantic_span_payload_rejection_total` +- `semantic_span_embedding_cache_hit_total` +- `semantic_span_index_latency_seconds` + +Sample only synthetic or redacted content. Production logs should carry hashes, +counts, span IDs, and policy IDs rather than full text. + +## 15. Security, privacy, and governance + +- Treat source text as untrusted data, never executable instructions. +- Keep provider credentials in contextual-orchestrator or secret storage. +- Enforce tenant filters before vector search and neighbor expansion. +- Include all metadata in the token guard to prevent an oversized-prefix bypass. +- Hash normalized content for idempotency and audit without logging plaintext. +- Apply retention/deletion to source spans, parent summaries, vectors, caches, + and evaluation exports together. +- Record model and policy provenance so regulated decisions can be reproduced. +- Do not infer or persist language, ethnicity, nationality, or other sensitive + traits merely to make segmentation work. + +## 16. Delivery plan + +### Phase 0 — baseline in this PR + +- `semantic_spans.py` core; +- exact-token codec protocol and optional Tiktoken adapter; +- dense-adjacency cache; +- final metadata payload guard; +- adapter to existing `Chunk` interface; +- mixed-script and adversarial unit tests at 100% module coverage; and +- ADR plus this implementation plan. + +### Phase 1 — embedding-channel integration + +- feature flag: `LINEAGEWEAVE_SEMANTIC_SPANS_ENABLED`; +- batch micro-unit embeddings through contextual-orchestrator; +- policy configuration validation at service startup; +- current paragraph chunker retained for controlled A/B comparison; +- structured metrics and cost counters. + +### Phase 2 — persistence and hierarchy + +- migrations for the four normalized objects; +- section/document summary generation through contextual-orchestrator; +- parent/neighbor expansion API; +- content-hash incremental reindexing; +- policy-versioned reindex command. + +### Phase 3 — evaluation and promotion + +- public/synthetic benchmark pack; +- offline report for all baselines and dimension settings; +- canary by tenant/project, not language; +- rollback by policy version; +- default-on only after promotion gates pass. + +## 17. Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| Dense boundary calls increase indexing cost | Embed micro-units in one batch, cache within the document, and reuse leaf vectors where possible. | +| Punctuation segmentation over/under-splits abbreviations | Treat punctuation as a weak structural prior; dense similarity and packing can merge related units. | +| A single unpunctuated unit exceeds the provider limit | Exact token-window fallback guarantees safety. | +| Metadata pushes a safe leaf over the request limit | Count and reject the final rendered payload before HTTP. | +| Short final span loses context | Store parent and bounded neighbor links; do not solve primarily with duplicate overlap. | +| Hosted API cannot provide late chunking | Keep current pre-chunk design; evaluate late chunking only for self-hosted models exposing token representations. | +| Model/tokenizer changes invalidate counts | Version the codec and policy and reindex by content hash. | +| Runtime language inference creates sensitive metadata | No language inference is required or stored by this control path. | + +## 18. Definition of done + +The capability is production-ready when: + +- all promotion gates pass; +- the final request guard is active on every embedding call; +- batch transport and idempotent reindexing are implemented; +- hierarchical persistence and tenant filtering are verified; +- runbooks document provider outage, reindex, rollback, and deletion; +- architecture and API docs link the accepted ADR; and +- released versions update `CHANGELOG.md`, package versions, and deployment + manifests together. + +## References (APA 7th) + +Günther, M., Mohr, I., Williams, D. J., Wang, B., & Xiao, H. (2024). *Late chunking: Contextual chunk embeddings using long-context embedding models* [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2409.04701 + +Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages. *Computational Linguistics, 23*(1), 33–64. https://aclanthology.org/J97-1003/ + +OpenAI. (2026). *Embeddings guide*. OpenAI API documentation. Retrieved August 15, 2026, from https://platform.openai.com/docs/guides/embeddings + +OpenAI. (2026). *Embeddings FAQ*. OpenAI Help Center. Retrieved August 15, 2026, from https://help.openai.com/en/articles/6824809-embeddings-faq + +OpenAI. (2026). *Text-embedding-3-large model*. OpenAI API documentation. Retrieved August 15, 2026, from https://developers.openai.com/api/docs/models/text-embedding-3-large + +The Unicode Consortium. (2025). *Unicode Standard Annex #15: Unicode normalization forms* (Revision 57). https://www.unicode.org/reports/tr15/ + +The Unicode Consortium. (2025). *Unicode Standard Annex #29: Unicode text segmentation* (Revision 47). https://www.unicode.org/reports/tr29/ diff --git a/lineageweave/semantic_spans.py b/lineageweave/semantic_spans.py new file mode 100644 index 000000000..da6021c43 --- /dev/null +++ b/lineageweave/semantic_spans.py @@ -0,0 +1,548 @@ +"""Language-agnostic, token-safe semantic-span construction. + +This module deliberately does not identify a language, dispatch to a +language-specific tokenizer, count whitespace-delimited words, translate the +input, or use TF-IDF. It treats the input as Unicode text and combines three +signals instead: + +1. authored structure (paragraph, line, heading, list, and punctuation cues), +2. exact token accounting supplied by the embedding model's token codec, and +3. an optional dense-embedding similarity score between adjacent micro-units. + +The resulting spans can be passed directly to LineageWeave's existing +``chunked_max_similarity`` function through :func:`make_semantic_span_chunker`. +The default policy is intentionally much smaller than an embedding provider's +absolute context limit; the provider limit is a safety boundary, not a target +chunk size. +""" + +from __future__ import annotations + +import math +import re +import unicodedata +from dataclasses import dataclass, replace +from typing import Callable, Protocol, Sequence + +from .chunking import Chunk + + +class TokenCodec(Protocol): + """Encode and decode text exactly as the selected embedding model does.""" + + def encode(self, text: str) -> list[int]: ... + + def decode(self, tokens: Sequence[int]) -> str: ... + + +class VectorEmbedder(Protocol): + """Minimal adapter implemented by LineageWeave embedding clients.""" + + def embed(self, text: str) -> list[float]: ... + + +AdjacentSimilarity = Callable[[str, str], float] + + +class TokenBudgetExceededError(ValueError): + """Raised when a final embedding payload exceeds its configured budget.""" + + +class TiktokenTokenCodec: + """A lazy ``cl100k_base`` codec for OpenAI third-generation embeddings. + + ``tiktoken`` is imported only when this adapter is instantiated. This + keeps the core semantic-span algorithm provider-neutral and lets callers + inject an authoritative codec from another service. Deployments that use + this adapter must install ``tiktoken`` in their embedding worker image. + """ + + def __init__(self, encoding_name: str = "cl100k_base") -> None: + try: + import tiktoken # type: ignore[import-not-found] + except ModuleNotFoundError as exc: # pragma: no cover - environment dependent + raise RuntimeError( + "TiktokenTokenCodec requires the optional 'tiktoken' package; " + "install it in the embedding worker or inject another exact TokenCodec" + ) from exc + self._encoding = tiktoken.get_encoding(encoding_name) + + def encode(self, text: str) -> list[int]: + """Return model tokens without allowing provider-specific specials.""" + return self._encoding.encode(text, disallowed_special=()) + + def decode(self, tokens: Sequence[int]) -> str: + """Decode a token slice back into text.""" + return self._encoding.decode(list(tokens)) + + +@dataclass(frozen=True) +class SemanticSpanPolicy: + """Controls token safety and boundary decisions. + + ``model_max_tokens`` is the provider's absolute input limit. + ``request_reserve_tokens`` protects room for short metadata prefixes and + future wire-format changes. ``max_span_tokens`` is the much smaller + retrieval-quality ceiling for one leaf span. + """ + + model_max_tokens: int = 8192 + request_reserve_tokens: int = 256 + target_span_tokens: int = 700 + max_span_tokens: int = 1200 + min_span_tokens: int = 120 + max_micro_unit_tokens: int = 320 + boundary_threshold: float = 0.55 + structure_weight: float = 0.35 + semantic_weight: float = 0.45 + length_weight: float = 0.20 + + def __post_init__(self) -> None: + integer_fields = { + "model_max_tokens": self.model_max_tokens, + "request_reserve_tokens": self.request_reserve_tokens, + "target_span_tokens": self.target_span_tokens, + "max_span_tokens": self.max_span_tokens, + "min_span_tokens": self.min_span_tokens, + "max_micro_unit_tokens": self.max_micro_unit_tokens, + } + for name, value in integer_fields.items(): + if value < 0 or (name != "request_reserve_tokens" and value == 0): + raise ValueError(f"{name} must be positive (reserve may be zero)") + if self.request_reserve_tokens >= self.model_max_tokens: + raise ValueError("request_reserve_tokens must be smaller than model_max_tokens") + if not self.min_span_tokens <= self.target_span_tokens <= self.max_span_tokens: + raise ValueError("expected min_span_tokens <= target_span_tokens <= max_span_tokens") + if self.max_micro_unit_tokens > self.max_span_tokens: + raise ValueError("max_micro_unit_tokens cannot exceed max_span_tokens") + if self.max_span_tokens > self.usable_input_tokens: + raise ValueError("max_span_tokens cannot exceed the model input budget after reserve") + if not 0.0 <= self.boundary_threshold <= 1.0: + raise ValueError("boundary_threshold must be between 0 and 1") + weights = (self.structure_weight, self.semantic_weight, self.length_weight) + if any(weight < 0.0 for weight in weights) or sum(weights) <= 0.0: + raise ValueError("boundary weights must be non-negative and sum to more than zero") + + @property + def usable_input_tokens(self) -> int: + """Maximum final request size after the metadata safety reserve.""" + return self.model_max_tokens - self.request_reserve_tokens + + +@dataclass(frozen=True) +class MicroUnit: + """One small, ordered unit from which semantic spans are packed.""" + + text: str + index: int + token_count: int + boundary_before: float + unit_type: str + + +@dataclass(frozen=True) +class SemanticSpan: + """A token-safe leaf span plus lineage-friendly adjacency metadata.""" + + text: str + index: int + token_count: int + source_indices: tuple[int, ...] + source_unit_types: tuple[str, ...] + boundary_score: float + previous_index: int | None = None + next_index: int | None = None + + +@dataclass(frozen=True) +class EmbeddingMetadata: + """Short, high-signal context that may prefix a leaf span.""" + + title: str = "" + heading_path: tuple[str, ...] = () + block_type: str = "" + speaker: str = "" + occurred_at: str = "" + + +# Terminal punctuation is intentionally script-diverse and does not depend on +# knowing which language produced the input. A decimal point between digits +# is treated as data rather than a sentence boundary. +_TERMINATORS = frozenset(".!?。！？｡؟۔։።᙮꘎꛳…") +_CLOSERS = frozenset("\"'”’»›）)]}】〕〗〙〛」』〉》") +_ZERO_WIDTH_TRANSLATION = str.maketrans({"\u200b": None, "\ufeff": None}) +_PARAGRAPH_BREAK = re.compile(r"\n[ \t]*\n+") +_MARKDOWN_HEADING = re.compile(r"^#{1,6}(?:\s|$)") +_LIST_ITEM = re.compile(r"^(?:[-*+]\s+|\d+[.)]\s+)") + + +def normalize_unicode_text(text: str) -> str: + """Normalize transport noise while preserving language and content. + + NFC normalization composes canonically equivalent sequences. CRLF/CR + become LF, and only the zero-width space and BOM are removed; ZWJ/ZWNJ are + preserved because they may be meaningful in scripts and emoji sequences. + """ + + normalized = text.replace("\r\n", "\n").replace("\r", "\n") + normalized = unicodedata.normalize("NFC", normalized) + return normalized.translate(_ZERO_WIDTH_TRANSLATION).strip() + + +def _token_count(codec: TokenCodec, text: str) -> int: + return len(codec.encode(text)) + + +def _line_unit_type(line: str) -> str: + if _MARKDOWN_HEADING.match(line): + return "heading" + if _LIST_ITEM.match(line): + return "list_item" + if line.startswith("```") or line.startswith("~~~"): + return "code_fence" + if "|" in line and line.count("|") >= 2: + return "table_row" + return "text" + + +def _split_at_universal_terminators(line: str) -> list[str]: + """Split a line without language, script, or capitalization assumptions.""" + + pieces: list[str] = [] + start = 0 + cursor = 0 + while cursor < len(line): + char = line[cursor] + if char not in _TERMINATORS: + cursor += 1 + continue + if ( + char == "." + and cursor > 0 + and cursor + 1 < len(line) + and line[cursor - 1].isdigit() + and line[cursor + 1].isdigit() + ): + cursor += 1 + continue + end = cursor + 1 + while end < len(line) and line[end] in _TERMINATORS: + end += 1 + while end < len(line) and line[end] in _CLOSERS: + end += 1 + piece = line[start:end].strip() + if piece: + pieces.append(piece) + start = end + cursor = end + remainder = line[start:].strip() + if remainder: + pieces.append(remainder) + return pieces + + +def _token_window_units( + text: str, + *, + codec: TokenCodec, + window_size: int, + inherited_boundary: float, + inherited_type: str, +) -> list[tuple[str, float, str]]: + """Last-resort exact-token split for a structurally indivisible unit.""" + + tokens = codec.encode(text) + units: list[tuple[str, float, str]] = [] + for offset in range(0, len(tokens), window_size): + decoded = codec.decode(tokens[offset : offset + window_size]).strip() + if not decoded: + continue + units.append( + ( + decoded, + inherited_boundary if not units else 0.15, + inherited_type if not units else "token_window", + ) + ) + return units + + +def build_micro_units( + text: str, + *, + codec: TokenCodec, + policy: SemanticSpanPolicy = SemanticSpanPolicy(), +) -> list[MicroUnit]: + """Create Unicode, structure, and token-aware units without language ID.""" + + normalized = normalize_unicode_text(text) + if not normalized: + return [] + + pending: list[tuple[str, float, str]] = [] + paragraphs = [paragraph for paragraph in _PARAGRAPH_BREAK.split(normalized) if paragraph.strip()] + for paragraph_index, paragraph in enumerate(paragraphs): + lines = [line.strip() for line in paragraph.split("\n") if line.strip()] + for line_index, line in enumerate(lines): + unit_type = _line_unit_type(line) + if paragraph_index == 0 and line_index == 0: + structural_boundary = 1.0 + elif line_index == 0: + structural_boundary = 0.75 + else: + structural_boundary = 0.45 + if unit_type in {"heading", "list_item", "code_fence", "table_row"}: + structural_boundary = max(structural_boundary, 0.90) + + pieces = _split_at_universal_terminators(line) or [line] + for piece_index, piece in enumerate(pieces): + boundary = structural_boundary if piece_index == 0 else 0.25 + piece_type = unit_type if len(pieces) == 1 else f"{unit_type}_sentence" + if _token_count(codec, piece) <= policy.max_micro_unit_tokens: + pending.append((piece, boundary, piece_type)) + else: + pending.extend( + _token_window_units( + piece, + codec=codec, + window_size=policy.max_micro_unit_tokens, + inherited_boundary=boundary, + inherited_type=piece_type, + ) + ) + + return [ + MicroUnit( + text=unit_text, + index=index, + token_count=_token_count(codec, unit_text), + boundary_before=boundary, + unit_type=unit_type, + ) + for index, (unit_text, boundary, unit_type) in enumerate(pending) + if unit_text + ] + + +def _join_micro_units(units: Sequence[MicroUnit]) -> str: + """Retain visible structure without reintroducing language assumptions.""" + + return "\n\n".join(unit.text for unit in units) + + +def _clamp_similarity(value: float) -> float: + if not math.isfinite(value): + raise ValueError("adjacent similarity must be finite") + return min(1.0, max(0.0, value)) + + +def _score_boundary( + current_units: Sequence[MicroUnit], + current_tokens: int, + next_unit: MicroUnit, + *, + adjacent_similarity: AdjacentSimilarity | None, + policy: SemanticSpanPolicy, +) -> float: + structure = next_unit.boundary_before + if adjacent_similarity is None: + semantic_drop = 0.0 + else: + similarity = _clamp_similarity( + adjacent_similarity(current_units[-1].text, next_unit.text) + ) + semantic_drop = 1.0 - similarity + length_pressure = min(1.0, current_tokens / policy.target_span_tokens) + numerator = ( + policy.structure_weight * structure + + policy.semantic_weight * semantic_drop + + policy.length_weight * length_pressure + ) + denominator = policy.structure_weight + policy.semantic_weight + policy.length_weight + return numerator / denominator + + +def _materialize_span( + units: Sequence[MicroUnit], + *, + codec: TokenCodec, + index: int, + boundary_score: float, +) -> SemanticSpan: + text = _join_micro_units(units) + return SemanticSpan( + text=text, + index=index, + token_count=_token_count(codec, text), + source_indices=tuple(unit.index for unit in units), + source_unit_types=tuple(unit.unit_type for unit in units), + boundary_score=boundary_score, + ) + + +def build_semantic_spans( + text: str, + *, + codec: TokenCodec, + adjacent_similarity: AdjacentSimilarity | None = None, + policy: SemanticSpanPolicy = SemanticSpanPolicy(), +) -> list[SemanticSpan]: + """Pack micro-units into coherent spans while proving token safety. + + The algorithm is greedy and deterministic for a fixed codec, similarity + function, and policy. Dense similarity is optional so ingestion can fail + open to structure-plus-budget chunking when an embedding provider is + unavailable; it never falls back to TF-IDF or language-specific logic. + """ + + units = build_micro_units(text, codec=codec, policy=policy) + if not units: + return [] + + spans: list[SemanticSpan] = [] + current: list[MicroUnit] = [] + current_start_score = 1.0 + + def flush() -> None: + nonlocal current + if not current: # pragma: no cover - internal invariant guard + return + span = _materialize_span( + current, + codec=codec, + index=len(spans), + boundary_score=current_start_score, + ) + if span.token_count > policy.max_span_tokens: # pragma: no cover - codec invariant + raise AssertionError("semantic span exceeded max_span_tokens") + if span.token_count > policy.usable_input_tokens: # pragma: no cover - policy invariant + raise AssertionError("semantic span exceeded provider input budget") + spans.append(span) + current = [] + + for unit in units: + if not current: + current = [unit] + current_start_score = 1.0 if not spans else unit.boundary_before + continue + + current_text = _join_micro_units(current) + current_tokens = _token_count(codec, current_text) + candidate_tokens = _token_count(codec, _join_micro_units([*current, unit])) + boundary_score = _score_boundary( + current, + current_tokens, + unit, + adjacent_similarity=adjacent_similarity, + policy=policy, + ) + exceeds_maximum = candidate_tokens > policy.max_span_tokens + crosses_semantic_boundary = ( + current_tokens >= policy.min_span_tokens + and boundary_score >= policy.boundary_threshold + ) + + if exceeds_maximum or crosses_semantic_boundary: + flush() + current_start_score = 1.0 if exceeds_maximum else boundary_score + current = [unit] + else: + current.append(unit) + + flush() + + last_index = len(spans) - 1 + return [ + replace( + span, + previous_index=span.index - 1 if span.index > 0 else None, + next_index=span.index + 1 if span.index < last_index else None, + ) + for span in spans + ] + + +class CachedEmbeddingSimilarity: + """Convert an existing embedding client into an adjacent-similarity scorer.""" + + def __init__(self, embedder: VectorEmbedder) -> None: + self._embedder = embedder + self._cache: dict[str, list[float]] = {} + + def _vector(self, text: str) -> list[float]: + if text not in self._cache: + self._cache[text] = self._embedder.embed(text) + return self._cache[text] + + def __call__(self, left: str, right: str) -> float: + left_vector = self._vector(left) + right_vector = self._vector(right) + if len(left_vector) != len(right_vector): + raise ValueError("embedding vectors must have the same dimension") + left_norm = math.sqrt(sum(value * value for value in left_vector)) + right_norm = math.sqrt(sum(value * value for value in right_vector)) + if left_norm == 0.0 or right_norm == 0.0: + return 0.0 + cosine = sum(a * b for a, b in zip(left_vector, right_vector)) / ( + left_norm * right_norm + ) + return min(1.0, max(0.0, (cosine + 1.0) / 2.0)) + + +def make_semantic_span_chunker( + *, + codec: TokenCodec, + embedder: VectorEmbedder | None = None, + policy: SemanticSpanPolicy = SemanticSpanPolicy(), +) -> Callable[[str], list[Chunk]]: + """Return a ``chunked_max_similarity``-compatible semantic chunker.""" + + similarity = CachedEmbeddingSimilarity(embedder) if embedder is not None else None + + def chunker(text: str) -> list[Chunk]: + spans = build_semantic_spans( + text, + codec=codec, + adjacent_similarity=similarity, + policy=policy, + ) + return [ + Chunk( + text=span.text, + unit_type="semantic_span", + index=span.index, + label=f"tokens:{span.token_count}", + ) + for span in spans + ] + + return chunker + + +def render_embedding_input( + span: SemanticSpan, + *, + codec: TokenCodec, + policy: SemanticSpanPolicy = SemanticSpanPolicy(), + metadata: EmbeddingMetadata = EmbeddingMetadata(), +) -> str: + """Render high-signal metadata plus content and enforce the final limit.""" + + prefix: list[str] = [] + if metadata.title: + prefix.append(f"[title] {metadata.title}") + if metadata.heading_path: + prefix.append(f"[heading_path] {' > '.join(metadata.heading_path)}") + if metadata.block_type: + prefix.append(f"[block_type] {metadata.block_type}") + if metadata.speaker: + prefix.append(f"[speaker] {metadata.speaker}") + if metadata.occurred_at: + prefix.append(f"[occurred_at] {metadata.occurred_at}") + payload = "\n".join([*prefix, "[content]", span.text]) if prefix else span.text + token_count = _token_count(codec, payload) + if token_count > policy.usable_input_tokens: + raise TokenBudgetExceededError( + f"embedding payload has {token_count} tokens; budget is {policy.usable_input_tokens}" + ) + return payload diff --git a/tests/test_semantic_spans.py b/tests/test_semantic_spans.py new file mode 100644 index 000000000..541666cf5 --- /dev/null +++ b/tests/test_semantic_spans.py @@ -0,0 +1,411 @@ +from __future__ import annotations + +import pytest + +from lineageweave.semantic_spans import ( + CachedEmbeddingSimilarity, + EmbeddingMetadata, + SemanticSpanPolicy, + TokenBudgetExceededError, + build_micro_units, + build_semantic_spans, + make_semantic_span_chunker, + normalize_unicode_text, + render_embedding_input, +) + + +class _CodePointCodec: + """Deterministic test codec: one Unicode code point equals one token.""" + + def encode(self, text: str) -> list[int]: + return [ord(character) for character in text] + + def decode(self, tokens: list[int]) -> str: + return "".join(chr(token) for token in tokens) + + +class _KeywordEmbedder: + available = True + + def __init__(self) -> None: + self.calls: list[str] = [] + + def embed(self, text: str) -> list[float]: + self.calls.append(text) + if "different" in text.lower() or "다른" in text: + return [0.0, 1.0] + return [1.0, 0.0] + + +def _policy(**overrides: object) -> SemanticSpanPolicy: + values: dict[str, object] = { + "model_max_tokens": 256, + "request_reserve_tokens": 16, + "target_span_tokens": 80, + "max_span_tokens": 120, + "min_span_tokens": 1, + "max_micro_unit_tokens": 100, + "boundary_threshold": 0.55, + "structure_weight": 0.35, + "semantic_weight": 0.45, + "length_weight": 0.20, + } + values.update(overrides) + return SemanticSpanPolicy(**values) + + +def test_normalization_preserves_script_content_and_joiners() -> None: + text = "e\u0301\r\n한\u200b글\u200d🙂\ufeff" + + normalized = normalize_unicode_text(text) + + assert normalized == "é\n한글\u200d🙂" + + +def test_micro_units_split_mixed_scripts_without_language_detection_or_spaces() -> None: + text = "한국어 문장입니다。日本語です！مرحبا بالعالم؟English sentence." + + units = build_micro_units(text, codec=_CodePointCodec(), policy=_policy()) + + assert [unit.text for unit in units] == [ + "한국어 문장입니다。", + "日本語です！", + "مرحبا بالعالم؟", + "English sentence.", + ] + assert [unit.index for unit in units] == [0, 1, 2, 3] + + +def test_decimal_point_is_not_treated_as_a_sentence_boundary() -> None: + units = build_micro_units( + "The score was 3.14. 다음 값은 2.71입니다。", + codec=_CodePointCodec(), + policy=_policy(), + ) + + assert units[0].text == "The score was 3.14." + assert units[1].text == "다음 값은 2.71입니다。" + + +def test_dense_semantic_drop_splits_related_units_from_a_new_topic() -> None: + codec = _CodePointCodec() + embedder = _KeywordEmbedder() + similarity = CachedEmbeddingSimilarity(embedder) + text = ( + "Alpha idea continues. Alpha detail follows.\n\n" + "A completely different topic starts here." + ) + + spans = build_semantic_spans( + text, + codec=codec, + adjacent_similarity=similarity, + policy=_policy(), + ) + + assert len(spans) == 2 + assert "Alpha detail follows." in spans[0].text + assert spans[1].text == "A completely different topic starts here." + # Three unique micro-units are embedded once each even though they form + # two adjacent comparisons. + assert len(embedder.calls) == 3 + + +def test_related_short_paragraphs_can_merge_despite_authored_break() -> None: + spans = build_semantic_spans( + "First related observation.\n\nSecond related observation.", + codec=_CodePointCodec(), + adjacent_similarity=lambda _left, _right: 1.0, + policy=_policy(), + ) + + assert len(spans) == 1 + assert spans[0].source_indices == (0, 1) + + +def test_oversized_unpunctuated_text_uses_exact_token_windows() -> None: + codec = _CodePointCodec() + policy = _policy( + model_max_tokens=32, + request_reserve_tokens=2, + target_span_tokens=8, + max_span_tokens=10, + max_micro_unit_tokens=6, + ) + text = "가나다라마바사아자차카타파하ABCDEFGHIJK" + + spans = build_semantic_spans(text, codec=codec, policy=policy) + + assert spans + assert all(span.token_count <= 10 for span in spans) + assert "".join(span.text.replace("\n", "") for span in spans) == text + + +def test_span_neighbors_are_contiguous_for_context_restoration() -> None: + policy = _policy( + model_max_tokens=64, + request_reserve_tokens=4, + target_span_tokens=6, + max_span_tokens=8, + max_micro_unit_tokens=6, + ) + spans = build_semantic_spans( + "abcdefghiABCDEFGHIabcdefghi", + codec=_CodePointCodec(), + policy=policy, + ) + + assert len(spans) > 2 + assert spans[0].previous_index is None + assert spans[0].next_index == 1 + assert spans[1].previous_index == 0 + assert spans[-1].next_index is None + + +def test_semantic_span_chunker_plugs_into_existing_chunk_contract() -> None: + chunker = make_semantic_span_chunker( + codec=_CodePointCodec(), + embedder=_KeywordEmbedder(), + policy=_policy(), + ) + + chunks = chunker("Related note. A different subject appears.") + + assert chunks + assert all(chunk.unit_type == "semantic_span" for chunk in chunks) + assert [chunk.index for chunk in chunks] == list(range(len(chunks))) + assert all(chunk.label.startswith("tokens:") for chunk in chunks) + + +def test_embedding_metadata_is_budgeted_with_the_content() -> None: + codec = _CodePointCodec() + policy = _policy( + model_max_tokens=120, + request_reserve_tokens=20, + target_span_tokens=40, + max_span_tokens=80, + max_micro_unit_tokens=60, + ) + span = build_semantic_spans("short content.", codec=codec, policy=policy)[0] + + payload = render_embedding_input( + span, + codec=codec, + policy=policy, + metadata=EmbeddingMetadata(title="T", heading_path=("A",), block_type="paragraph"), + ) + + assert "[title] T" in payload + assert "[heading_path] A" in payload + assert "[content]" in payload + + +def test_embedding_payload_overflow_is_rejected_before_provider_call() -> None: + codec = _CodePointCodec() + policy = _policy( + model_max_tokens=30, + request_reserve_tokens=5, + target_span_tokens=15, + max_span_tokens=20, + max_micro_unit_tokens=20, + ) + span = build_semantic_spans("content.", codec=codec, policy=policy)[0] + + with pytest.raises(TokenBudgetExceededError, match="budget is 25"): + render_embedding_input( + span, + codec=codec, + policy=policy, + metadata=EmbeddingMetadata(title="X" * 30), + ) + + +def test_policy_rejects_a_leaf_limit_above_the_safe_provider_budget() -> None: + with pytest.raises(ValueError, match="model input budget"): + SemanticSpanPolicy( + model_max_tokens=100, + request_reserve_tokens=20, + min_span_tokens=10, + target_span_tokens=50, + max_span_tokens=90, + max_micro_unit_tokens=50, + ) + + +def test_similarity_must_be_finite() -> None: + with pytest.raises(ValueError, match="finite"): + build_semantic_spans( + "One sentence. Another sentence.", + codec=_CodePointCodec(), + adjacent_similarity=lambda _left, _right: float("nan"), + policy=_policy(), + ) + + +def test_tiktoken_adapter_uses_requested_encoding(monkeypatch: pytest.MonkeyPatch) -> None: + import sys + from types import SimpleNamespace + + from lineageweave.semantic_spans import TiktokenTokenCodec + + requested: list[str] = [] + + class _FakeEncoding: + def encode(self, text: str, *, disallowed_special: tuple[()] = ()) -> list[int]: + assert disallowed_special == () + return [ord(character) for character in text] + + def decode(self, tokens: list[int]) -> str: + return "".join(chr(token) for token in tokens) + + fake_module = SimpleNamespace( + get_encoding=lambda name: requested.append(name) or _FakeEncoding() + ) + monkeypatch.setitem(sys.modules, "tiktoken", fake_module) + + codec = TiktokenTokenCodec("cl100k_base") + + assert requested == ["cl100k_base"] + assert codec.encode("A한") == [ord("A"), ord("한")] + assert codec.decode([ord("A"), ord("한")]) == "A한" + + +@pytest.mark.parametrize( + ("overrides", "message"), + [ + ({"model_max_tokens": 0}, "must be positive"), + ( + {"model_max_tokens": 100, "request_reserve_tokens": 100}, + "smaller than model_max_tokens", + ), + ( + {"min_span_tokens": 90, "target_span_tokens": 80}, + "min_span_tokens <= target_span_tokens", + ), + ( + {"max_micro_unit_tokens": 121}, + "max_micro_unit_tokens cannot exceed", + ), + ({"boundary_threshold": 1.1}, "between 0 and 1"), + ({"semantic_weight": -0.1}, "non-negative"), + ( + {"structure_weight": 0.0, "semantic_weight": 0.0, "length_weight": 0.0}, + "sum to more than zero", + ), + ], +) +def test_policy_validates_all_invariants( + overrides: dict[str, object], message: str +) -> None: + with pytest.raises(ValueError, match=message): + _policy(**overrides) + + +def test_structural_unit_types_are_script_neutral() -> None: + units = build_micro_units( + "# 제목\n- عنصر\n```\n| 日本 | 한국 |", + codec=_CodePointCodec(), + policy=_policy(), + ) + + assert [unit.unit_type for unit in units] == [ + "heading", + "list_item", + "code_fence", + "table_row", + ] + assert units[1].boundary_before == 0.9 + + +def test_repeated_terminal_marks_and_closing_quotes_remain_with_the_unit() -> None: + units = build_micro_units( + 'Really?!" 다음입니다。』', + codec=_CodePointCodec(), + policy=_policy(), + ) + + assert [unit.text for unit in units] == ['Really?!"', "다음입니다。』"] + + +def test_empty_input_produces_no_micro_units_or_spans() -> None: + codec = _CodePointCodec() + + assert build_micro_units("\u200b \n\n", codec=codec, policy=_policy()) == [] + assert build_semantic_spans("", codec=codec, policy=_policy()) == [] + + +def test_empty_token_window_decode_is_ignored() -> None: + class _DroppingCodec(_CodePointCodec): + def decode(self, tokens: list[int]) -> str: + if tokens and chr(tokens[0]) == "a": + return "" + return super().decode(tokens) + + units = build_micro_units( + "abcdefghi", + codec=_DroppingCodec(), + policy=_policy(max_micro_unit_tokens=3), + ) + + assert [unit.text for unit in units] == ["def", "ghi"] + assert units[0].index == 0 + + +def test_embedding_similarity_rejects_dimension_mismatch() -> None: + class _VariableDimensionEmbedder: + def embed(self, text: str) -> list[float]: + return [1.0] if text == "left" else [1.0, 0.0] + + similarity = CachedEmbeddingSimilarity(_VariableDimensionEmbedder()) + + with pytest.raises(ValueError, match="same dimension"): + similarity("left", "right") + + +def test_embedding_similarity_handles_zero_vectors() -> None: + class _ZeroEmbedder: + def embed(self, _text: str) -> list[float]: + return [0.0, 0.0] + + assert CachedEmbeddingSimilarity(_ZeroEmbedder())("a", "b") == 0.0 + + +def test_similarity_values_are_clamped_to_the_contract_range() -> None: + policy = _policy(boundary_threshold=0.99) + + spans = build_semantic_spans( + "One. Two.", + codec=_CodePointCodec(), + adjacent_similarity=lambda _left, _right: 4.0, + policy=policy, + ) + + assert len(spans) == 1 + + +def test_metadata_supports_speaker_date_and_content_only_payloads() -> None: + codec = _CodePointCodec() + policy = _policy(model_max_tokens=300, request_reserve_tokens=20) + span = build_semantic_spans("content.", codec=codec, policy=policy)[0] + + content_only = render_embedding_input(span, codec=codec, policy=policy) + enriched = render_embedding_input( + span, + codec=codec, + policy=policy, + metadata=EmbeddingMetadata(speaker="Speaker 1", occurred_at="2026-08-15"), + ) + + assert content_only == "content." + assert "[speaker] Speaker 1" in enriched + assert "[occurred_at] 2026-08-15" in enriched + + +def test_chunker_can_run_without_dense_provider() -> None: + chunker = make_semantic_span_chunker( + codec=_CodePointCodec(), + policy=_policy(), + ) + + assert chunker("One. Two.")