Skip to content

fix(vt): keep grapheme clusters intact across Write calls - #955

Open
Rohilalala wants to merge 1 commit into
charmbracelet:mainfrom
Rohilalala:fix/vt-grapheme-across-writes
Open

fix(vt): keep grapheme clusters intact across Write calls#955
Rohilalala wants to merge 1 commit into
charmbracelet:mainfrom
Rohilalala:fix/vt-grapheme-across-writes

Conversation

@Rohilalala

Copy link
Copy Markdown

Fixes #935.

Thanks @marcus for the writeup — the repro is used verbatim below.

Emulator.Write force-flushed the pending grapheme at the end of every call (|| i == len(p)-1), so the write boundary terminated the cluster. Feeding the same bytes in two pieces produced different cells and a different cursor column:

whole  cursor=(7,0)  ["❤️" w=2]["" w=0][" "]["v"]["s"][" "]["❤"]
split  cursor=(6,0)  ["❤" w=1][" "]["v"]["s"][" "]["❤"]            <- U+FE0F dropped entirely

The second case in the issue is a separate defect with the same shape: handlePrint's ASCII fast path commits the base immediately, so a combining mark can never attach to it and e + U+0301 stayed two cells.

Approach

Not flushing at the end of a write would make the last character of every write invisible until more input arrived, which is presumably why that clause is there. So the base is still written as soon as it is seen, and a continuation arriving afterwards — in the same write or a later one — is folded into the cell already on screen. That is what terminals do, and it fixes both defects without deferring anything.

ansi.FirstGraphemeCluster decides what may be folded; nothing else does.

Four things worth flagging in review

An allowlist of continuation classes is unsound. It looks like the obvious cheap gate, and it silently loses merges: sweeping every rune from U+0300 to U+10FFFF against FirstGraphemeCluster found 12,351 pairs that a unicode.Mn/Me/Mc-based allowlist rejects but that really do continue a cluster — U+0897 and Thai U+0E33 among them, because the unicode tables and the grapheme-break data do not agree. The only sound cheap gate is r >= 0x0300, deferring everything else to the exact test.

The candidate is built in a reusable buffer. prev.Content + content costs an allocation per cell for every character above Latin-1: 14,363 allocs/op on a CJK screenful versus 355 with the buffer. FirstGraphemeCluster being generic over []byte makes this free. It still costs about 13% more time on pure CJK, which is the price of asking the authority per character.

The base is tracked, not inferred from the cursor. At the last column the cursor stops advancing — pending wrap with autowrap on, clamped without it — so "the cell before the cursor" is not the cell just written, and inferring it there merges into the wrong one.

Any sequence ends the cluster on screen as well as the one buffered. A cleared cell holding a blank matches a remembered " " by coincidence, so without this a mark folds into a cell that ED, a scroll or an alt-screen switch has already changed. endCluster does both at the eight sites that already called flushGrapheme.

Merging is bounded at 256 bytes: far above the longest real cluster, and without it a stream of joiners grows one cell without limit while every continuation rescans what is already there.

Not addressed

A cluster whose finished width does not fit the last column still lands differently depending on the split, because the unsplit path writes an oversized cell there and the screen blanks it. That happens on main too — a single write of "ab" plus a two-column emoji on a four-column screen loses the emoji — and it is a wrapping bug rather than a clustering one. I have a separate branch for it and will open that as its own PR.

Tests

TestGraphemeClusterAcrossWrites splits each case at every byte boundary and compares against the unsplit result, over variation selectors, combining marks on ASCII, skin tones, ZWJ sequences, regional indicators, conjoining jamo, and a widening cluster at the wrap point. The rest cover the merge's guards individually: wide bases, a moved cursor, an overwritten anchor, a clamped cursor, chained continuations, the cluster break after a joiner, and the size bound.

gofmt, go vet, golangci-lint --config ../.golangci.yml (matches main's two pre-existing findings), and go test ./... -race -count=3 -shuffle=on.

The screen depended on where the writer happened to chop up its input. A pipe,
socket, PTY or tmux control client may split one visible character across two
reads, and "❤️ vs ❤" written whole and written in two pieces did not produce
the same cells or leave the cursor in the same column.

The emulator cannot wait to find out whether a cluster is finished, because
that is only known once the rune after it arrives. So the base is written as
soon as it is seen, and a continuation arriving later — in the same write or a
later one — is now folded into the cell already on screen. That also fixes a
combining mark following an ASCII base, which never attached at all: the ASCII
fast path in handlePrint had already committed the base, so "e" plus U+0301
stayed two cells instead of becoming "é".

ansi.FirstGraphemeCluster decides what may be folded; nothing else does. An
allowlist of continuation classes looks tempting and is wrong — the unicode
tables and the grapheme-break data disagree on thousands of runes, U+0897 and
U+0E33 among them — so the only cheap gate here is to keep ASCII and Latin-1
away and defer everything else. The candidate is built in a buffer the emulator
keeps, since asking the question would otherwise allocate once per cell; a
screenful of CJK holds at 355 allocations, and costs about 13% more time for
being asked at all.

The base is tracked rather than inferred from the cursor. At the last column
the cursor stops advancing — pending wrap with autowrap on, clamped without it
— so the cell before the cursor is not the cell just written, and inferring it
there folds the continuation into the wrong one. Any sequence that runs ends
the cluster on screen as well as the one buffered, because it may move the
cursor or rewrite that cell; a cleared cell holding a blank would otherwise
match the remembered content by coincidence. Merging is bounded at 256 bytes,
which is far above the longest real cluster and stops a stream of joiners from
growing one cell without limit.

Not addressed: a cluster whose finished width does not fit the last column
still lands differently depending on the split, because the unsplit path writes
an oversized cell there and the screen blanks it. That happens on main too —
"ab" followed by a two-column emoji on a four-column screen loses the emoji —
and wants its own fix.

Fixes charmbracelet#935
@Rohilalala
Rohilalala force-pushed the fix/vt-grapheme-across-writes branch from 5b59ae9 to 33005ba Compare August 24, 2026 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vt: keep grapheme clusters intact across Write calls

1 participant