fix(vt): keep grapheme clusters intact across Write calls - #955
Open
Rohilalala wants to merge 1 commit into
Open
fix(vt): keep grapheme clusters intact across Write calls#955Rohilalala wants to merge 1 commit into
Rohilalala wants to merge 1 commit into
Conversation
The screen depended on where the writer happened to chop up its input. A pipe, socket, PTY or tmux control client may split one visible character across two reads, and "❤️ vs ❤" written whole and written in two pieces did not produce the same cells or leave the cursor in the same column. The emulator cannot wait to find out whether a cluster is finished, because that is only known once the rune after it arrives. So the base is written as soon as it is seen, and a continuation arriving later — in the same write or a later one — is now folded into the cell already on screen. That also fixes a combining mark following an ASCII base, which never attached at all: the ASCII fast path in handlePrint had already committed the base, so "e" plus U+0301 stayed two cells instead of becoming "é". ansi.FirstGraphemeCluster decides what may be folded; nothing else does. An allowlist of continuation classes looks tempting and is wrong — the unicode tables and the grapheme-break data disagree on thousands of runes, U+0897 and U+0E33 among them — so the only cheap gate here is to keep ASCII and Latin-1 away and defer everything else. The candidate is built in a buffer the emulator keeps, since asking the question would otherwise allocate once per cell; a screenful of CJK holds at 355 allocations, and costs about 13% more time for being asked at all. The base is tracked rather than inferred from the cursor. At the last column the cursor stops advancing — pending wrap with autowrap on, clamped without it — so the cell before the cursor is not the cell just written, and inferring it there folds the continuation into the wrong one. Any sequence that runs ends the cluster on screen as well as the one buffered, because it may move the cursor or rewrite that cell; a cleared cell holding a blank would otherwise match the remembered content by coincidence. Merging is bounded at 256 bytes, which is far above the longest real cluster and stops a stream of joiners from growing one cell without limit. Not addressed: a cluster whose finished width does not fit the last column still lands differently depending on the split, because the unsplit path writes an oversized cell there and the screen blanks it. That happens on main too — "ab" followed by a two-column emoji on a four-column screen loses the emoji — and wants its own fix. Fixes charmbracelet#935
Rohilalala
force-pushed
the
fix/vt-grapheme-across-writes
branch
from
August 24, 2026 16:52
5b59ae9 to
33005ba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #935.
Thanks @marcus for the writeup — the repro is used verbatim below.
Emulator.Writeforce-flushed the pending grapheme at the end of every call (|| i == len(p)-1), so the write boundary terminated the cluster. Feeding the same bytes in two pieces produced different cells and a different cursor column:The second case in the issue is a separate defect with the same shape:
handlePrint's ASCII fast path commits the base immediately, so a combining mark can never attach to it ande+ U+0301 stayed two cells.Approach
Not flushing at the end of a write would make the last character of every write invisible until more input arrived, which is presumably why that clause is there. So the base is still written as soon as it is seen, and a continuation arriving afterwards — in the same write or a later one — is folded into the cell already on screen. That is what terminals do, and it fixes both defects without deferring anything.
ansi.FirstGraphemeClusterdecides what may be folded; nothing else does.Four things worth flagging in review
An allowlist of continuation classes is unsound. It looks like the obvious cheap gate, and it silently loses merges: sweeping every rune from U+0300 to U+10FFFF against
FirstGraphemeClusterfound 12,351 pairs that aunicode.Mn/Me/Mc-based allowlist rejects but that really do continue a cluster — U+0897 and Thai U+0E33 among them, because theunicodetables and the grapheme-break data do not agree. The only sound cheap gate isr >= 0x0300, deferring everything else to the exact test.The candidate is built in a reusable buffer.
prev.Content + contentcosts an allocation per cell for every character above Latin-1: 14,363 allocs/op on a CJK screenful versus 355 with the buffer.FirstGraphemeClusterbeing generic over[]bytemakes this free. It still costs about 13% more time on pure CJK, which is the price of asking the authority per character.The base is tracked, not inferred from the cursor. At the last column the cursor stops advancing — pending wrap with autowrap on, clamped without it — so "the cell before the cursor" is not the cell just written, and inferring it there merges into the wrong one.
Any sequence ends the cluster on screen as well as the one buffered. A cleared cell holding a blank matches a remembered
" "by coincidence, so without this a mark folds into a cell thatED, a scroll or an alt-screen switch has already changed.endClusterdoes both at the eight sites that already calledflushGrapheme.Merging is bounded at 256 bytes: far above the longest real cluster, and without it a stream of joiners grows one cell without limit while every continuation rescans what is already there.
Not addressed
A cluster whose finished width does not fit the last column still lands differently depending on the split, because the unsplit path writes an oversized cell there and the screen blanks it. That happens on
maintoo — a single write of"ab"plus a two-column emoji on a four-column screen loses the emoji — and it is a wrapping bug rather than a clustering one. I have a separate branch for it and will open that as its own PR.Tests
TestGraphemeClusterAcrossWritessplits each case at every byte boundary and compares against the unsplit result, over variation selectors, combining marks on ASCII, skin tones, ZWJ sequences, regional indicators, conjoining jamo, and a widening cluster at the wrap point. The rest cover the merge's guards individually: wide bases, a moved cursor, an overwritten anchor, a clamped cursor, chained continuations, the cluster break after a joiner, and the size bound.gofmt,go vet,golangci-lint --config ../.golangci.yml(matchesmain's two pre-existing findings), andgo test ./... -race -count=3 -shuffle=on.