feat(evidence): retain paragraph-span prototype as superseded evidence

.github/workflows/repair-pr56-semantic-units.yml

@@ -0,0 +1,60 @@
+name: Repair PR 56 semantic units
+
+on:
+  pull_request:
+    types: [synchronize, reopened, ready_for_review]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: repair-tepp-pr-56-semantic-units
+  cancel-in-progress: true
+
+jobs:
+  repair:
+    if: >-
+      github.event.pull_request.number == 56 &&
+      github.event.pull_request.head.repo.full_name == github.repository &&
+      github.event.pull_request.head.ref == 'agent/evidence-semantic-units'
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout exact PR branch
+        uses: actions/checkout@631c942040754b6e095e929c1677c07e10ed4f87
+        with:
+          ref: agent/evidence-semantic-units
+          fetch-depth: 0
+          persist-credentials: true
+
+      - name: Install pinned Rust toolchain
+        run: rustup toolchain install 1.97.1 --profile minimal --component clippy --component rustfmt
+
+      - name: Merge current protected main
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git fetch origin main
+          git merge --no-edit -X ours origin/main
+
+      - name: Verify semantic-unit and workspace contracts
+        run: |
+          cargo +1.97.1 fmt --all --check
+          cargo +1.97.1 test -p evidence_core --all-features
+          cargo +1.97.1 clippy -p evidence_core --all-targets --all-features -- -D warnings
+          cargo +1.97.1 test --workspace --all-features
+          python3 scripts/check_workspace_contract.py
+          python3 scripts/check_docstrings.py
+          python3 scripts/validate_documentation.py
+
+      - name: Commit verified merge and remove one-shot workflow
+        run: |
+          rm -f .github/workflows/repair-pr56-semantic-units.yml
+          git add -A
+          git diff --cached --check
+          if ! git diff --cached --quiet; then
+            git commit -m "fix(evidence): harden canonical paragraph units"
+          fi
+          git push origin HEAD:agent/evidence-semantic-units

CHANGELOG.md

@@ -6,6 +6,7 @@ All notable changes to TEPP are documented here. The format follows Keep a Chang
 
 ### Added
 
+- `evidence_core` paragraph semantic units: blank-line splits emit exact byte/Unicode-scalar spans, and collapsing a multi-paragraph document into one bag-of-words unit is refused.
 - `persistence_postgres` typed membership assignment (migration `0006`): `entity_record`, `project_record`, and `text_segment` plus exactly-one observed-unit and target constraints that replace the polymorphic `membership_target_id` stub, with SQL insert/lookup, fail-closed inverted-window and backslash-label refusal, and live proof that one document persists two entity memberships and one project membership.
 - Actions workflow fleet auditor (`scripts/actions_workflow_fleet.py`): paginated registry inventory bound to the exact default-branch SHA/tree, classification of present/orphan/disabled/GitHub-dynamic identities, and fail-closed orphan disable that confirms GitHub's official `disabled_manually` state.
 - `persistence_postgres` temporal interval ordering migration (`0005`): multi-word CHECK constraints on `document_record`, `event_instance`, and `membership_assignment` that reject inverted valid/system windows and non-positive document revisions while preserving open-ended NULL upper bounds and equal point bounds; catalog validation and live inverted-window proof.

DOCUMENTATION.md

@@ -33,6 +33,7 @@ TEPP's approved PRD v0.4 and implementation plan are the primary product baselin
 | Hourly NIM product-development operations | [`docs/operations/HOURLY_NIM_PRODUCT_DEVELOPMENT.md`](docs/operations/HOURLY_NIM_PRODUCT_DEVELOPMENT.md) |
 | Actions workflow fleet audit | [`docs/operations/ACTIONS_WORKFLOW_FLEET.md`](docs/operations/ACTIONS_WORKFLOW_FLEET.md) |
 | Actions fleet research doctoring | [`docs/research/actions-workflow-fleet.md`](docs/research/actions-workflow-fleet.md) |
+| Semantic paragraph-unit doctoring | [`docs/research/semantic-paragraph-units.md`](docs/research/semantic-paragraph-units.md) |
 | Hourly NIM OpenCode doctoring | [`docs/doctoring/hourly-nim-opencode-development.md`](docs/doctoring/hourly-nim-opencode-development.md) |
 | Change history | [`CHANGELOG.md`](CHANGELOG.md) |
 

crates/evidence_core/src/error.rs

@@ -44,6 +44,8 @@ pub enum EvidenceError {
     InvalidLayoutBounds,
     /// Layout coordinates exceeded the enclosing page.
     LayoutOutOfBounds,
+    /// A multi-paragraph document was treated as one bag-of-words unit.
+    SemanticUnitBagRefused,
 }
 
 impl fmt::Display for EvidenceError {
@@ -70,6 +72,7 @@ impl fmt::Display for EvidenceError {
             Self::InvalidPageGeometry => "page geometry must be finite and positive",
             Self::InvalidLayoutBounds => "layout bounds must be finite, nonnegative, and nonempty",
             Self::LayoutOutOfBounds => "layout bounds exceed the page geometry",
+            Self::SemanticUnitBagRefused => "document bag-of-words unit refused",
         };
         formatter.write_str(message)
     }

crates/evidence_core/src/lib.rs

@@ -7,13 +7,15 @@
 //! records, source spans whose byte, Unicode-scalar, page, and layout
 //! coordinates are validated before entering later temporal or psychometric
 //! layers, and strict versioned JSON wire contracts that reconstruct records
-//! only through the same domain validation boundary.
+//! only through the same domain validation boundary. Paragraph semantic units
+//! preserve exact source spans so later embedding search is meaning-bearing.
 
 mod artifact;
 mod digest;
 mod document;
 mod error;
 mod identifier;
+mod semantic;
 mod span;
 mod wire;
 
@@ -27,6 +29,10 @@ pub use document::DocumentRecord;
 pub use error::EvidenceError;
 /// A validated RFC 9562 `UUIDv7` evidence identifier.
 pub use identifier::EvidenceId;
+/// Refuse collapsing a multi-paragraph document into one bag-of-words unit.
+pub use semantic::refuse_document_bag_of_words;
+/// Split a document into paragraph units with exact source spans.
+pub use semantic::semantic_paragraph_units;
 /// A validated page-relative location for source evidence.
 pub use span::PageLocation;
 /// An exact byte, Unicode-scalar, and optional page/layout span.

crates/evidence_core/src/semantic.rs

@@ -0,0 +1,94 @@
+//! Paragraph-scale semantic units with exact source spans.
+
+use crate::{DocumentRecord, EvidenceError, SourceSpan};
+
+/// Split a document into paragraph units with validated exact spans.
+///
+/// Units are separated by a blank line (`\n\n`). Whitespace-only segments are
+/// skipped. This is the meaning-search unit for later embedding; it is not a
+/// bag-of-words document vector.
+///
+/// # Errors
+///
+/// Returns [`EvidenceError::EmptySourceSpan`] when no non-empty paragraph
+/// remains after splitting.
+pub fn semantic_paragraph_units(
+    document: &DocumentRecord,
+) -> Result<Vec<SourceSpan>, EvidenceError> {
+    let text = document.text();
+    let mut units = Vec::new();
+    let mut byte_cursor = 0usize;
+    let mut scalar_cursor = 0usize;
+    for segment in text.split("\n\n") {
+        let byte_start = byte_cursor;
+        let byte_end = byte_start + segment.len();
+        let scalar_start = scalar_cursor;
+        let scalar_end = scalar_start + segment.chars().count();
+        byte_cursor = byte_end + 2;
+        scalar_cursor = scalar_end + 2;
+        if segment.trim().is_empty() {
+            continue;
+        }
+        units.push(SourceSpan::new(
+            document,
+            byte_start,
+            byte_end,
+            scalar_start,
+            scalar_end,
+            None,
+        )?);
+    }
+    if units.is_empty() {
+        return Err(EvidenceError::EmptySourceSpan);
+    }
+    Ok(units)
+}
+
+/// Require the canonical paragraph spans instead of one collapsed document bag.
+///
+/// The expected paragraph multiplicity is derived from `document`; callers
+/// cannot weaken the guard by supplying their own count. The supplied spans
+/// must also equal the canonical exact spans, preventing unrelated spans from
+/// satisfying the count alone.
+///
+/// # Errors
+///
+/// Returns [`EvidenceError::InvalidWirePayload`] for an empty or noncanonical
+/// span set. Returns [`EvidenceError::SemanticUnitBagRefused`] when `units`
+/// contains fewer spans than the document's canonical paragraph set.
+pub fn refuse_document_bag_of_words(
+    document: &DocumentRecord,
+    units: &[SourceSpan],
+) -> Result<(), EvidenceError> {
+    if units.is_empty() {
+        return Err(EvidenceError::InvalidWirePayload);
+    }
+    let expected = semantic_paragraph_units(document)?;
+    if units.len() < expected.len() {
+        return Err(EvidenceError::SemanticUnitBagRefused);
+    }
+    if units != expected.as_slice() {
+        return Err(EvidenceError::InvalidWirePayload);
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::{refuse_document_bag_of_words, semantic_paragraph_units};
+    use crate::{DocumentRecord, EvidenceError, SourceArtifact};
+
+    #[test]
+    fn trailing_blank_and_empty_unit_set_fail_closed() {
+        let text = "Only one unit.\n\n   \n\n";
+        let artifact = SourceArtifact::from_bytes(text.as_bytes()).expect("artifact");
+        let document = DocumentRecord::from_text(artifact.id(), text).expect("document");
+        let units = semantic_paragraph_units(&document).expect("trim empty");
+        assert_eq!(units.len(), 1);
+        refuse_document_bag_of_words(&document, &units).expect("canonical unit");
+        assert_eq!(
+            refuse_document_bag_of_words(&document, &[]),
+            Err(EvidenceError::InvalidWirePayload)
+        );
+    }
+}

crates/evidence_core/tests/records_and_spans_contract.rs

@@ -329,6 +329,10 @@ fn every_record_validation_error_has_a_stable_message() {
             EvidenceError::LayoutOutOfBounds,
             "layout bounds exceed the page geometry",
         ),
+        (
+            EvidenceError::SemanticUnitBagRefused,
+            "document bag-of-words unit refused",
+        ),
     ];
 
     for (error, expected) in cases {

crates/evidence_core/tests/semantic_unit_contract.rs

@@ -0,0 +1,53 @@
+//! Paragraph units keep exact source spans for later meaning search.
+
+use evidence_core::{
+    DocumentRecord, EvidenceError, SourceArtifact, refuse_document_bag_of_words,
+    semantic_paragraph_units,
+};
+
+#[test]
+fn two_paragraphs_recover_exact_spans_and_refuse_bag_of_words() {
+    let text = "Q3 pipeline slipped after the Acme renewal stalled.\n\nLegal hold remains on the Acme folder.";
+    let artifact = SourceArtifact::from_bytes(text.as_bytes()).expect("artifact");
+    let document = DocumentRecord::from_text(artifact.id(), text).expect("document");
+
+    let units = semantic_paragraph_units(&document).expect("units");
+    assert_eq!(units.len(), 2);
+    assert_eq!(
+        &document.text()[units[0].byte_start()..units[0].byte_end()],
+        "Q3 pipeline slipped after the Acme renewal stalled."
+    );
+    assert_eq!(
+        &document.text()[units[1].byte_start()..units[1].byte_end()],
+        "Legal hold remains on the Acme folder."
+    );
+    refuse_document_bag_of_words(&document, &units).expect("keep canonical units");
+    assert_eq!(
+        refuse_document_bag_of_words(&document, &units[..1]),
+        Err(EvidenceError::SemanticUnitBagRefused)
+    );
+}
+
+#[test]
+fn single_paragraph_rejects_empty_or_unrelated_spans() {
+    let text = "One paragraph only.";
+    let artifact = SourceArtifact::from_bytes(text.as_bytes()).expect("artifact");
+    let document = DocumentRecord::from_text(artifact.id(), text).expect("document");
+    let units = semantic_paragraph_units(&document).expect("one");
+    assert_eq!(units.len(), 1);
+    refuse_document_bag_of_words(&document, &units).expect("single canonical unit");
+    assert_eq!(
+        refuse_document_bag_of_words(&document, &[]),
+        Err(EvidenceError::InvalidWirePayload)
+    );
+
+    let other_text = "Another paragraph.";
+    let other_artifact = SourceArtifact::from_bytes(other_text.as_bytes()).expect("other artifact");
+    let other_document =
+        DocumentRecord::from_text(other_artifact.id(), other_text).expect("other document");
+    let unrelated_units = semantic_paragraph_units(&other_document).expect("other units");
+    assert_eq!(
+        refuse_document_bag_of_words(&document, &unrelated_units),
+        Err(EvidenceError::InvalidWirePayload)
+    );
+}

docs/TRACEABILITY.md

@@ -8,6 +8,7 @@ The full APA 7th standards/literature register remains `docs/research/standards-
 | Requirement / decision | Canonical basis | Source/evidence boundary | Maturity |
 |---|---|---|---|
 | immutable source evidence and exact spans | PRD; Architecture; ADR 0008 | `evidence_core`, Task 2 tests/doctoring | implemented-main |
+| paragraph semantic units for meaning search | ADR 0008; research | `evidence_core` blank-line spans on the active PR | active-PR |
 | Rust numerical authority / CPU `f64` reference | ADR 0001 | current workspace foundation; future estimators | partial |
 | Rust workspace/quality foundation | ADR 0007 | workspace/CI/repository contract | implemented-main |
 | six distinct clocks and uncertain intervals | PRD; ADR 0002 | PR #8 `temporal_core` on protected main; PR #5 historical only | implemented-main |

docs/research/semantic-paragraph-units.md

@@ -0,0 +1,32 @@
+# Paragraph semantic units
+
+## Scope
+
+This note doctors the `evidence_core` meaning-search unit:
+
+1. documents are split on explicit blank lines into paragraph units;
+2. each unit is a validated exact byte and Unicode-scalar span;
+3. a multi-paragraph document cannot be collapsed into one document-level unit by supplying a weaker caller-owned count.
+
+This is the first chunking contract for later embedding search. It does not run an embedding model, claim that paragraph boundaries are universally optimal, or allocate a database migration.
+
+## Authoritative sources
+
+Unicode Consortium. (2024). *Unicode Standard Annex #29: Unicode text segmentation* (Revision 45). https://www.unicode.org/reports/tr29/
+
+Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing* (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410
+
+## Application
+
+UAX #29 specifies lower-level Unicode text-boundary algorithms and makes clear that higher-level protocols may tailor segmentation. TEPP therefore treats an explicit blank line as a repository-owned paragraph convention while retaining exact source coordinates; later sentence, DOM, sender/recipient, and language-profile units can be added without rewriting the evidence identity (Unicode Consortium, 2024).
+
+Sentence-BERT demonstrates retrieval-oriented embeddings for sentence and short-text units, but it does not establish that blank-line paragraphs are universally optimal or that every whole document is an invalid embedding input (Reimers & Gurevych, 2019). TEPP's narrower product decision is to preserve known paragraph multiplicity at this boundary so later measurement can compare or replace chunking policies without losing the original evidence spans.
+
+## Verification
+
+- a two-paragraph Acme report recovers both exact source texts and spans;
+- collapsing those units to one row returns `SemanticUnitBagRefused`;
+- a caller cannot weaken the guard with a smaller paragraph count;
+- unrelated spans with the correct count return `InvalidWirePayload`;
+- a single canonical paragraph remains one unit;
+- empty unit sets fail closed.

docs/validation/temporal-event-foundation.md

@@ -23,6 +23,7 @@ This report tracks exact-head scientific and engineering evidence required befor
 | Truth corpora / manifests | `tepp_simulation` | implemented-main | — | deterministic generator tests | Task 10 / PR #18 |
 | Recovery metrics | `validation_core` | implemented-main | — | RMSE/bias/coverage/MC gates | Task 11 / PR #19 |
 | Versioned API/export contracts | `tepp_api` | implemented-main | — | unknown-field/version/limit tests | Task 12 / PR #21; HTTP service remaining |
+| Paragraph semantic units | `evidence_core` | active-PR | blank-line spans | two-paragraph exact recover | ADR 0008; `docs/research/semantic-paragraph-units.md` |
 | CWL modular connectors | `docs/connectors/*` | implemented-main | — | contract docs + examples | PR #22; live HTTP ports remaining |
 | Release SBOM/provenance generator | `scripts/release_evidence.py` | partial | — | generate+validate in CI | Task 13 partial / PR #28 |