Verify Morris source reconstruction

.github/workflows/validate.yml

@@ -43,6 +43,11 @@ jobs:
           python3 scripts/translation_process_status.py --check
           python3 scripts/test_translation_process_status.py
 
+      - name: Verify source reconstruction
+        run: |
+          python3 scripts/source_reconstruction.py
+          python3 scripts/test_source_reconstruction.py
+
       - name: Check phonetic-neighbor baseline is in sync
         run: |
           python3 scripts/audit_phonetic_neighbors.py --output /tmp/phonetic_neighbors_baseline.txt

README.md

@@ -46,9 +46,10 @@ python3 scripts/validate_examples.py
 python3 scripts/validate_sentences.py
 python3 scripts/test_sentence_validator.py
 python3 scripts/translation_process_status.py --check
+python3 scripts/source_reconstruction.py
 ```
 
-Install the pinned dependency once for each Python environment. The lexical validator checks every entry against the [executable schema](vocabulary/schema.json) before applying Phi's sound, layout, and corpus rules. It also checks lexical use across active documents and compares supported citations with their stored sources. The sentence parser is a separate gate for complete examples in the lexicon and maintained teaching corpus. Machines can reject a particle in the wrong place; they cannot decide whether a translation has understood Morris. Literary translations therefore use the [isolated process](documents/reference/translation_process.md) and its generated [certification register](documents/evaluation/translation_process_status.md).
+Install the pinned dependency once for each Python environment. The lexical validator checks every entry against the [executable schema](vocabulary/schema.json) before applying Phi's sound, layout, and corpus rules. It also checks lexical use across active documents. The sentence parser is a separate gate for complete examples in the lexicon and maintained teaching corpus. For the Morris chapters, the source-reconstruction checker selects each chapter from the stored novel and requires the ordered citations to reproduce it exactly. Machines can reject a particle in the wrong place or find a missing source phrase; they cannot decide whether a translation has understood Morris. Literary translations therefore use the [isolated process](documents/reference/translation_process.md) and its generated [certification register](documents/evaluation/translation_process_status.md).
 
 The [CI workflow](.github/workflows/validate.yml) contains the complete recipe. If you change the vocabulary, regenerate the derived reference with `python3 scripts/generate_reference.py`. Settled language decisions live in [canon.md](canon.md); the current sequence lives in the [near-term development plan](project/near_term_development_plan.md); broader work and evidence gates live in the [status roadmap](project/roadmap.md). The [development protocol](project/development_protocol.md) governs language work, while the [publishing strategy](project/publishing.md) holds the longer view.
 

documents/reference/translation_process.md

@@ -58,6 +58,14 @@ python3 scripts/translation_layers.py texts/north_wind_and_sun.md --phase source
 
 That view contains numbered Phi units and their decoded source citations. It omits glosses, derived English, notes, and limits. A new translation begins directly in the same numbered Phi-and-source form under `/tmp`; there is no English layer to extract. Before the freeze is accepted, the source citations must still reconstruct the chosen source passage exactly.
 
+Where the repository holds an independent witness and a source manifest, check that reconstruction mechanically:
+
+```bash
+python3 scripts/source_reconstruction.py
+```
+
+The manifest names the witness, citation label, selected boundaries, and normalization rule for each translation it covers. For the Project Gutenberg Morris source, normalization rejoins a hyphenated word split by a line wrap and collapses the remaining whitespace. Spelling, punctuation, and case stay exact. The checker then compares the complete selected passage with the citations in order; matching character counts alone cannot make it pass.
+
 ## 4. Make compact anonymous bundles
 
 The frozen source-to-Phi view becomes the input to the second phase. For a work of ordinary length, make a directory of bounded packets:

project/README.md

@@ -11,6 +11,7 @@ This directory records how Phi is maintained. The index below gathers its workin
 | [Content vocabulary coverage](content_vocabulary_coverage.md) | Status-tracked semantic coverage audits and the review gate for content vocabulary batches. |
 | [Content vocabulary decisions](content_vocabulary_decisions.md) | Generated, readable view of every candidate carried forward from the content-vocabulary audit and its present decision state. |
 | [Translation certification register](translation_process_status.json) | Machine-readable status and checksum evidence for every translation document under the isolated process. |
+| [Source reconstruction manifest](source_reconstruction_manifest.json) | Stored source, citation label, chapter boundaries, and normalization rule for each mechanically reconstructed translation. |
 | [Deferred questions](deferred_questions.md) | Parked questions and decisions, with the condition for returning to each one. |
 | [Model continuation handoff](handoff/README.md) | Dated transfer package with the exact work state, repository utilities, active vocabulary method, and standing maintainer instructions. |
 | [Publishing strategy](publishing.md) | Order of operations for public access and eventual print publication. |

project/development_log.md

@@ -131,6 +131,7 @@ Implementation status, dependencies, optional evaluation work, and the next deve
 | D113 | Accepted | Certify Gibran's *On Children* under the isolated translation process. | Eighteen aligned units reconstruct all 980 normalized Gibran characters. The source-only pass gives the framing exchange explicit participants, establishes `ne musatapha`, reserves reflexive `miso` for a true object, states thought possession and plurality directly, and unfolds the archery image through visible materials and roles. Anonymous derivation and independent source-blind review expose three invalid possessive structures, an incorrectly attached request topic, and ambiguous strength and gladness relations. Each affected English layer is discarded before the Phi is repaired and freshly derived. The final stream freezes at SHA-256 `028a864fb7bb333245159dcf9e2048b9021c3dab04a34f0e299506da9d059455`. No root, module membership, registered compound, function word, or grammar is added. |
 | D114 | Accepted | Give certified translations a public mark and explanation generated from the process register. | The website links every certified translation page to a public account of the two isolated directions, shows compact marks in the short-work and collection catalogues, and lists the current certified works from `project/translation_process_status.json`. The explanation states prominently that certified texts can still contain errors. The build validates the register before rendering and stops when a translation page, catalogue, or public list disagrees with it. Pending translations receive no mark. |
 | D115 | Accepted | Close the completed lexicon prose migration at the schema boundary. | Every lexicon entry must carry `description`, `articulatory_notes`, and structured `examples`. The schema rejects the retired `concept` and `grammatical_notes` properties instead of accepting them as fallbacks. Regression tests prove both requirements and both rejections, while the completed migration ledger remains available as evidence. No vocabulary entry or generated lexicon reference changes. |
+| D116 | Accepted | Verify ordered source reconstruction against an independently stored witness. | A manifest selects each of the six current *News from Nowhere* chapters from the complete Morris source, names the `morris` citation stream, and applies a stated Project Gutenberg line-wrap normalization. The checker proves that 1,054 ordered citations reconstruct 75,243 normalized characters exactly and reports missing, duplicated, reordered, or altered spans directly. A required chapter pattern stops later chapters from bypassing the check, and synthetic regression tests exercise every failure class in CI. No translation layer changes. |
 
 ## Completed discretionary lexical migrations
 

project/near_term_development_plan.md

@@ -25,8 +25,8 @@ Community engagement is outside this plan. Phi can strengthen its language, corp
 | ID | Status | Work package | Completion gate |
 |---|---|---|---|
 | NTP-01 | **DONE** | Close the migrated vocabulary schema | Current entries pass a schema that requires the target prose fields and no longer accepts the retired fields. |
-| NTP-02 | **NEXT** | Add independent source-reconstruction checking | At least the locally stored Morris source can be compared mechanically with each chapter's ordered citation stream. |
-| NTP-03 | **READY** | Establish the *News from Nowhere* continuity record | Recurring source-to-Phi choices have one maintained home that is excluded from Phi-to-English derivation. |
+| NTP-02 | **DONE** | Add independent source-reconstruction checking | At least the locally stored Morris source can be compared mechanically with each chapter's ordered citation stream. |
+| NTP-03 | **NEXT** | Establish the *News from Nowhere* continuity record | Recurring source-to-Phi choices have one maintained home that is excluded from Phi-to-English derivation. |
 | NTP-04 | **READY** | Reconcile stale project records | The roadmap, development protocol, and deferred records describe the actual completed corpus and closed grammar boundary. |
 | NTP-05 | **READY** | Finish the short-work certification queue | The present Gibran, Taoist, and Buddhist short works have completed the isolated process and the generated register passes. |
 | NTP-06 | **PENDING** | Certify *News from Nowhere* chapter 1 as a pilot | The chapter has a current certification record and its findings have been classified as local or series-wide. |
@@ -45,9 +45,11 @@ Completed under D115. The target fields are direct requirements, the retired pro
 
 ## NTP-02: Verify source reconstruction independently
 
-The certification register verifies unit counts, digests, aligned layers, and recorded normalized source counts. It does not by itself compare an ordered citation stream with an independently stored source witness. That leaves a gap between a strong process and the strongest reading of the public reconstruction claim.
+The certification register verified unit counts, digests, aligned layers, and recorded normalized source counts. It did not by itself compare an ordered citation stream with an independently stored source witness. That left a gap between a strong process and the strongest reading of the public reconstruction claim.
 
-Begin with *News from Nowhere*, whose source witness is stored in the repository. A small manifest should identify the source file, citation label, selected chapter boundary, and normalization rule. A checker should join the chapter's decoded citations in order and compare the result character for character with the selected normalized source. It should report a missing span, duplicated span, reordering, or altered text directly. The design may later support other works with independently stored witnesses, but it should not pretend that every source has one when it does not.
+The package begins with *News from Nowhere*, whose source witness is stored in the repository. A small manifest identifies the source file, citation label, selected chapter boundary, and normalization rule. A checker joins the chapter's decoded citations in order and compares the result character for character with the selected normalized source. It reports a missing span, duplicated span, reordering, or altered text directly. The design may later support other works with independently stored witnesses, but it does not pretend that every source has one when it does not.
+
+Completed under D116. All six current chapters reconstruct 75,243 normalized source characters across 1,054 citations. The manifest's required chapter pattern prevents a later Morris chapter from bypassing the check, and CI exercises the four named failure classes with synthetic mutations.
 
 ## NTP-03: Preserve Morris continuity
 

project/source_reconstruction_manifest.json

@@ -0,0 +1,68 @@
+{
+  "format": "phi-source-reconstruction-v1",
+  "required_translation_globs": [
+    "texts/news_from_nowhere/chapter_*.md"
+  ],
+  "documents": [
+    {
+      "translation": "texts/news_from_nowhere/chapter_01.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER I: DISCUSSION AND BED",
+        "end_before": "CHAPTER II: A MORNING BATH"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    },
+    {
+      "translation": "texts/news_from_nowhere/chapter_02.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER II: A MORNING BATH",
+        "end_before": "CHAPTER III: THE GUEST HOUSE AND BREAKFAST THEREIN"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    },
+    {
+      "translation": "texts/news_from_nowhere/chapter_03.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER III: THE GUEST HOUSE AND BREAKFAST THEREIN",
+        "end_before": "CHAPTER IV: A MARKET BY THE WAY"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    },
+    {
+      "translation": "texts/news_from_nowhere/chapter_04.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER IV: A MARKET BY THE WAY",
+        "end_before": "CHAPTER V: CHILDREN ON THE ROAD"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    },
+    {
+      "translation": "texts/news_from_nowhere/chapter_05.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER V: CHILDREN ON THE ROAD",
+        "end_before": "CHAPTER VI: A LITTLE SHOPPING"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    },
+    {
+      "translation": "texts/news_from_nowhere/chapter_06.md",
+      "source": "texts/news_from_nowhere/source.txt",
+      "citation_label": "morris",
+      "selection": {
+        "start_after": "CHAPTER VI: A LITTLE SHOPPING",
+        "end_before": "CHAPTER VII: TRAFALGAR SQUARE"
+      },
+      "normalization": "gutenberg-wrapped-prose-v1"
+    }
+  ]
+}

scripts/README.md

@@ -136,6 +136,17 @@ python3 scripts/translation_layers.py texts/north_wind_and_sun.md --digest-only
 python3 scripts/test_translation_layers.py
 ```
 
+## source_reconstruction.py
+
+Checks the translation and source relationships declared in `project/source_reconstruction_manifest.json`. Each entry names the stored witness, citation label, source boundaries, and normalization rule. The current manifest selects the first six chapters of *News from Nowhere* from the complete Morris source and compares each chapter with its ordered `morris` citations. Its required path pattern also makes an unregistered new chapter an error.
+
+The `gutenberg-wrapped-prose-v1` rule rejoins a hyphenated word split by a Project Gutenberg line wrap, then collapses the remaining whitespace. It does not change spelling, punctuation, or case. A mismatch is reported as a missing span, duplicated span, reordered span, or alteration at the first citation where the streams diverge.
+
+```bash
+python3 scripts/source_reconstruction.py
+python3 scripts/test_source_reconstruction.py
+```
+
 ## translation_process_status.py
 
 Validates the complete D102 certification queue and generates its readable ledger. Standalone translations and Gibran selections come from their catalogues; every catalogued book must declare whether the process applies, and the current *News from Nowhere* chapter glob supplies that book's documents. A certified row records both the frozen Phi digest and a digest over all four published layers, so later drift in Phi, gloss, derived English, or source citations stops CI.