Keep emoji whole in digest change runs, so a changed emoji does not print "�"

[HMarzban]

Summary

In the Change digest, an edit that swaps one emoji for a similar one can print replacement characters ("�") instead of the emoji. The diff splits the emoji in half, and the mail cannot encode half an emoji. The same can happen when a length cap cuts through an emoji.

Land after #448. It widens the change ranges that #448's reordered runsAround slices. #448 also edits runsAround and capRuns in diffSections.ts, so re-read the line numbers below.

Where

• prosemirror-changeset (via @tiptap/pm) compares text one UTF-16 code unit at a time (tokens(), charCodeAt). An emoji outside the Basic Multilingual Plane is two code units, a "surrogate pair".
• simplifyChanges keeps a one-unit swap as it is. Its letter test does not match half a pair, so nothing widens the change back to the whole emoji.
• apps/hocuspocus.server/src/modules/document-changes/domain/diffSections.ts:
  • quantify calls simplifyChanges once (:151-158). The word loop and both excerpts (:165-172) and runsAround (:183) all read that one list.
  • runsAround (:97-117) slices docA and docB at those raw positions.
  • contextBefore (:57) uses slice(-100), which can start on the second half of a pair.
  • capRuns (:88-89) cuts the run through sanitizeText at :88, then slices again after adding lead at :89. Both can end on the first half.
• apps/hocuspocus.server/src/lib/sanitizePlainText.ts:9: .slice(0, maxChars) cuts by code unit. It is reached through sanitizeText (domain/sanitizeText.ts:9-10) at diffSections.ts:88,223-224 and segmentSections.ts:11,39. So EXCERPT_MAX_CHARS (140) and SECTION_TEXT_MAX_CHARS (200) can split a pair.
• sanitizePlainText is shared. src/lib/documentGridPreview.ts (:73, :116, :123) and src/lib/email/digestDocuments.ts:100 (chat message_preview, 200) get the same fix. That is intended: they only lose a half that would print "�".
• Precedent: trimShared in apps/hocuspocus.server/src/modules/document-content/domain/applyContentToDoc.ts:124-136 steps back one unit at a split pair. Its two predicates (:116-117) are private there. Write the ranges inline; do not export them.

What happens

Read from code, not measured.

A writer changes "📍 House London" to "📌 House London". Both emoji share the first half (\uD83D). The change covers only the second half, so the runs become: grey \uD83D, red struck \uDCCD, green \uDCCC. UTF-8 turns each lone half into U+FFFD, and the reader sees three "�" marks.

Fix

• In quantify, widen the simplifyChanges result once, right after :151-158 and before the word loop. Use one local function in diffSections.ts.
  • If the code unit before fromB is a first half (0xD800-0xDBFF), move fromA and fromB back by one.
  • If the code unit at toB is a second half (0xDC00-0xDFFF), move toA and toB on by one.
  • Read the units with docB.textBetween, inside 0..docB.content.size. The text outside a change is equal on both sides, so checking docB is enough.
  • runsAround gets the same list, so it does not change.
• In sanitizePlainText, after the slice, drop a leading second half and a trailing first half. For example: .replace(/^[\uDC00-\uDFFF]|[\uD800-\uDBFF]$/g, '').
  • Every non-gap run passes through it in capRuns (:88). So it also covers the slice(-100) in contextBefore, which needs no change.
  • This relies on Show red, green or a label on every changed heading in the digest email #448: after its guard, a same run never merges into another same run, so its first unit reaches this check.
• In capRuns, pass room - lead.length to sanitizeText on :88. The .slice(0, room) on :89 can then drop only the trailing space, never half an emoji.

Out of scope

• The History preview cut .slice(0, DIFF_PREVIEW_CHARS) in apps/hocuspocus.server/src/modules/document-versions/domain/diffBlocks.ts:17, and the webapp Compare view. Not checked here.
• truncate in packages/email-templates/src/helpers.ts:198-201, which cuts chat previews in the mail.

Acceptance criteria

• Swapping one emoji for another that shares its first half gives a removed run with the whole old emoji. The added run holds the whole new one. No run text holds a lone half.
• When a cap falls inside an emoji, excerpt, removedExcerpt and run text end before the emoji, never on its first half.
• Two unit tests in apps/hocuspocus.server/src/modules/document-changes/__tests__/unit/diffSections.test.ts, observed green. One pins the "📍" to "📌" swap through runs. One pins an excerpt whose 140th code unit is the first half of an emoji, next to caps the excerpt. Both fail on main. No new test file.

Verify

• cd apps/hocuspocus.server && bun test src/modules/document-changes/__tests__/unit/diffSections.test.ts, then bun test and bun run typecheck.
• Undo the widening in quantify and confirm the swap test fails.