Repository navigation
fix: wrap PO strings with GNU gettext's UAX #14 algorithm - #116
Conversation
|
Thanks a lot for this. On your open questions:
I've also pushed a test to the branch test/gnu-wrapping. It compiles You'll get a small conflict in |
Port the line wrapping used by the GNU gettext tools: wrap() in gettext-tools/src/write-po.c plus the libunistring unilbrk (UAX #14 line breaking) and uniwidth code it relies on. Instead of folding only the string value at a fixed length, the whole logical line (key prefix, quotes and escape sequences) is counted, strings are split into portions at real newlines, and multi-portion or overflowing strings use the same 'msgid ""' quoted-piece style the GNU tools emit. Default foldLength changes from 76 to 79 to match gettext's page width.
Port u8_possible_linebreaks_loop() as of gettext 1.0 (Unicode 17.0.0): east asian width aware quotation marks (LB19a), word-initial hyphens (LB20a), numbers (LB25), Brahmic syllables (LB28a) and the HTML tag exception to LB29. The line break classes, pair table and width tables now live in lib/unicode-data.js, written by scripts/generate-unicode-data.js from the gnulib tables in a gettext source release. Regenerate with: npm run generate-unicode-data -- path/to/gettext-1.0
b81130d to
4c9f46d
Compare
|
I rebased onto test/gnu-wrapping and kept both tests. The wrapping now follows gettext 1.0 (Unicode 17.0.0). The tables moved to |
|
Thanks! Does the Windows failure tell you anything? |
|
Yes, that one was mine. |
|
Merged and released as |
Fixes #51.
po.compile()now wraps strings the way the GNU gettext tools do, so its output matchesmsgcat1.0 line for line instead of producing re-wrap diffs.The old fold used a substring-and-backtrack heuristic at a fixed width of 76 and ignored the
msgid "/msgid_plural "/#~prefix.lib/shared.jsnow portswrap()fromgettext-tools/src/write-po.c, plus the libunistringunilbrk(UAX #14) anduniwidthcode it calls, as of gettext 1.0. The port is pure JavaScript and adds no dependency.#~entries.key ""+ quoted-pieces style, as GNU does.foldLengthis now 79, the gettext page width.0/falsestill disables folding. TheescapeCharacters: false+foldLength: 0output does not change.This is a breaking output change: existing
.pooutput re-wraps once on its next compile, andfoldLength: Nnow means page width N.flowchart LR A["escaped value"] --> B["split into portions at \n"] B --> C["UAX 14 break candidates<br/>no breaks inside escapes"] C --> D["greedy packing to<br/>foldLength - 1 - prefix columns"] D --> E{"wraps or several portions?"} E -- no --> F["key #quot;value#quot;"] E -- yes --> G["key #quot;#quot;<br/>#quot;piece#quot;<br/>#quot;piece#quot;"]Unicode tables
The line-break and width tables live in
lib/unicode-data.js, written byscripts/generate-unicode-data.js. The generator has no dependencies. It reads the gnulib tables bundled in an extracted gettext source release and records the gettext and Unicode versions in the file header (currently gettext 1.0, Unicode 17.0.0):Testing
The branch is rebased onto
test/gnu-wrapping, and both compiler tests are kept. Regeneratingtest/fixtures/wrapping-msgcat.powith msgcat 1.0 (npm run test-generate-po) gives a byte-identical file.Compared against
msgcat --width=N1.0 on five seeded random corpora (HTML, escapes, tabs, newlines, CJK, Hangul, emoji, Hebrew, Devanagari, Thai, Arabic, URLs, plurals, msgctxt, obsolete entries) at widths 20, 40, 60, 79 and 100, this branch differs on 0 of about 171,000 lines. Comment lines (#:,#|, ...) are outside this change and were excluded.Not covered, as before: legacy CJK encodings (EUC-JP, GBK, BIG5) use the non-CJK width rules, and format-directive break overrides are not modelled.