Convert USFM versification before updating from rows

[claude]

Quick summary

ParatextProjectTextUpdaterBase.update_usfm now converts the source USFM to the rows' versification before applying the update, so rows in one versification update a project in another. The conversion runs only when the two differ, so same-versification updates behave as before. Nothing in machine/jobs is touched.

Where to look

• Chapter, book and verse-range remapping, plus heading and paragraph placement across chapter breaks: the 25 tests in tests/corpora/test_convert_usfm_versification_handler.py.
• The updater switching versification only when it differs: test_update_usfm_converts_to_rows_versification.
• DefaultParatextProjectSettings in the test utils now defaults to English, not Original, matching ScriptureRef. With the old default, 8 existing update tests started converting and failed.
• ConvertUsfmVersificationHandler is a new export of machine.corpora. get_rows_versification is a new module-level function in update_usfm_parser_handler.py.

Validation

• ./local_check.sh --agent-strict -- 873 passed, 3 skipped (local run, not CI-equivalent)

Issue / porting context

Closes #369
Closes #382

Ports sillsdev/machine#472 and sillsdev/machine#521.

🤖 Generated with Claude Code

---

This change is 

[ddaspit]

@claude The CI failed for this PR. Fix it.

[ddaspit]

@claude The CI build failed for this PR. Fix it.

[claude]

Claude finished @ddaspit's task in 2m 16s —— View job

---

CI fix pushed

The Lint with pyright step failed. ConvertUsfmVersificationHandler.chapter declares alt_number and pub_number as Optional[str]. It calls ScriptureRefUsfmParserHandlerBase.chapter, which declared them as str.

I changed the base signature to Optional[str]. That matches UsfmParserHandler.chapter and the base class's own verse. The fix is commit 515e732.

Verification

• pyright machine/corpora tests/corpora reports 0 errors.
• black --check and flake8 are clean.
• pytest tests/corpora gives 299 passed, 3 skipped.
• ./local_check.sh --agent-strict did not complete here. It stopped while collecting the huggingface tests, with an ImportError for is_offline_mode in the local environment. Those tests are not touched by this change. The full pyright run also lists errors in the huggingface modules, which come from the same environment mismatch. The only pyright errors in this PR's files were the two I fixed.

The CI run was cancelled after the pyright failure, so I haven't seen the other CI jobs finish. Please check the new run.
· branch claude/issue-369-20261001-1520

[codecov-commenter]

Codecov Report

❌ Patch coverage is 99.68051% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 92.16%. Comparing base (8d3767c) to head (aa0d660).

Files with missing lines	Patch %	Lines
...hine/corpora/convert_usfm_versification_handler.py	99.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
+ Coverage   92.07%   92.16%   +0.09%     
==========================================
  Files         394      396       +2     
  Lines       24896    25202     +306     
==========================================
+ Hits        22922    23228     +306     
  Misses       1974     1974              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ddaspit]

@claude Address all of the review comments, except those that would result in the PR diverging from the original C# code.

[claude]

Claude finished @ddaspit's task in 17s —— View job

---

No code changes made

I made no changes and pushed no commits. Every review comment falls under your exception, because a fix would make the Python port differ from the C# code.

• F1 (convert_usfm_versification_handler.py:170): a split range at the end of the input writes the verse text twice. The review says C# EndUsfm has the same shape, so I left it. A fix here would diverge from C#.
• F2 (:163): markers after the last verse are dropped. The review says C# shares this, so I left it.
• F3 (:58): a chapter with no verses loses its \c. The review says C# shares this, so I left it.
• F4 (paratext_project_text_updater_base.py:68): with no rows, get_rows_versification returns English. This matches C# GetRowsVersification. Skipping the conversion would be a deliberate divergence, so I left it.

Not verified: I could not read the C# source in this session, because the gh api and WebFetch calls were denied. I am relying on the review's statement that C# behaves the same way for all four. The Python code has not been run for F1–F3 either, since those findings were hand traces.

Suggested follow-up: F1–F3 look like real defects. If you want them fixed, they could go upstream in sillsdev/machine first, and then be ported here. Tell me if you would rather fix them here and accept the divergence.
· branch claude/issue-369-20261001-1520

[ddaspit]

@claude You should now be able to read the C# source.

[claude]

Claude finished @ddaspit's task in 18s —— View job

---

No code changes made

I read the C# source (ConvertUsfmVersificationHandler.cs and ParatextProjectTextUpdaterBase.cs in sillsdev/machine) and compared it with this PR. The Python matches it on all four review findings, so fixing any of them here would make the port diverge. I pushed no commits.

• F1 (convert_usfm_versification_handler.py:170): C# EndUsfm (lines 229-235) appends state.Token unconditionally when _skip is false and the token is not a chapter or verse. The Python end_usfm does the same. C# Verse also adds the next text token in the split-range branches (lines 156-165, 177-186), as _add_next_text_token does. The double write on an input that ends on a split range is therefore in C# too. I did not run it.
• F2 (:163): C# EndUsfm calls ProcessTokens and never AddTrailingTokens, so markers after the last verse are dropped there as well.
• F3 (:58): C# Chapter only sets _insertChapterIndex = _tokens.Count (line 58). It emits \c only from Verse, so a chapter with no verses loses its \c in C# too.
• F4 (paratext_project_text_updater_base.py:68): C# UpdateUsfmParserHandler.GetRowsVersification(rows) feeds the same rowsVersification != _settings.Versification check (lines 69-77). The Python is the same.

F1-F3 look like real defects in the C# original. I'd fix them upstream in sillsdev/machine and then port the fix here. If you'd rather fix them in this PR and accept the divergence, tell me and I will. I would also add an empty-chapter test for F3 in either case.
· branch claude/issue-369-20261001-1520

[claude]

FYI: F5. A merged chapter leaves its \ca/\cp behind as stray markers: the parser treats them as special tokens, but _verse_boundary stops at the \c, so the next _process_tokens copies them. English to Original MAL \c 4 \ca 5\ca* \p \v 1 a creates no \c 4, yet \ca 5\ca* lands between 3:18 and 3:19. Unverified hand trace; C# ProcessTokens is identical, so this is parity.

[claude]

1. Verdict: approve. Every open finding is a parity defect shared with sillsdev/machine, and the author deferred those.
2. Most important: F1, machine/corpora/convert_usfm_versification_handler.py:170. A split range at the end of the input still writes its text twice. It is accepted for C# parity and should be fixed upstream first.
3. Counts: 0 Critical, 0 Important open, 1 Low new (F5).
4. Ran: gh api compare between 7bc8382 and aa0d660 returned diverged. That was a rebase, and every file blob in the PR is identical to the last round. Ran /code-review high 390. Read C# ConvertUsfmVersificationHandler.cs, ParatextProjectTextUpdaterBase.cs, UpdateUsfmParserHandler.GetRowsVersification and the test DefaultParatextProjectSettings on sillsdev/machine main. The converter matches line for line, the updater does the same conversion, and the C# test default is English. Searching sillsdev/machine issues and PRs for ConvertUsfmVersification found nothing.
5. Not verified: I could not run python, pytest or ./local_check.sh --agent-strict, because execution needed approval this session lacked. F1-F3 and F5 remain hand traces. Dropped as parity or duplicates: the semantics change and the chapters filter order (both match C#), the English default for empty rows (F4), and the Original-to-English test default (C# also uses English). Also dropped: a lenient get_rows_versification for rows whose refs are all empty (C# throws, and Python only fails later), and _change_versification vs VerseRef.to_versification, which raises and returns self, so it is not equivalent.

API surface: ConvertUsfmVersificationHandler is newly exported from machine.corpora. ScriptureRefUsfmParserHandlerBase.chapter now takes Optional[str], which is compatible. ParatextProjectTextUpdaterBase.update_usfm returns USFM in the rows' versification whenever it differs from the project's, including the English fallback with no rows. That matches sillsdev/machine#472/#521, so it is parity. Wheel users who write the result back should see it in the release notes.

Findings: F1 accepted, unverified. F2 accepted, unverified. F3 accepted, unverified. F4 accepted. F5 new, unverified.

Reviewed at aa0d660

[ddaspit]

@claude Fix F5 if it doesn't diverge from the C# source.

[claude]

Claude finished @ddaspit's task in 21s —— View job

---

No code changes made for F5

I left F5 alone because fixing it would make the Python port differ from the C# code. I pushed no commits.

• C# ConvertUsfmVersificationHandler.ProcessTokens (ConvertUsfmVersificationHandler.cs:243-271) matches Python _process_tokens line for line. Both copy every token up to state.Index and then set _verseBoundary = state.Index + 1. Neither treats \ca/\cp specially, so a merged chapter's \ca 5\ca* is copied at the next call.
• C# Chapter only sets _insertChapterIndex, the same as Python. It never drops the alternate-number tokens.

I read the C# source and compared it by hand. I did not run the Python on the F5 input.

F5 looks like a real defect in C#. I'd fix it upstream in sillsdev/machine and then port the fix here. If you'd rather fix it in this PR and accept the divergence, tell me and I will. I'd add a test for the \ca case either way.
· branch claude/issue-369-20261001-1520