Skip to content

fix: wrap PO strings with GNU gettext's UAX #14 algorithm - #116

Merged
smhg merged 4 commits into
smhg:mainfrom
cpruijsen:fix/issue-51-21cafd10
Oct 22, 2026
Merged

smhg merged 4 commits into
smhg:mainfrom
cpruijsen:fix/issue-51-21cafd10

Conversation

@cpruijsen

@cpruijsen cpruijsen commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #51.

po.compile() now wraps strings the way the GNU gettext tools do, so its output matches msgcat 1.0 line for line instead of producing re-wrap diffs.

The old fold used a substring-and-backtrack heuristic at a fixed width of 76 and ignored the msgid " / msgid_plural " / #~ prefix. lib/shared.js now ports wrap() from gettext-tools/src/write-po.c, plus the libunistring unilbrk (UAX #14) and uniwidth code it calls, as of gettext 1.0. The port is pure JavaScript and adds no dependency.

  • The whole logical line counts toward the width (key, quotes, escape sequences), with 4 extra columns for obsolete #~ entries.
  • Strings split into portions at real newlines.
  • Break positions come from the UAX Silent error doesnt compile my .po #14 pair table and line-break classes. Wide characters count as 2 columns.
  • A string that wraps or has more than one portion is written in the key "" + quoted-pieces style, as GNU does.
  • The default foldLength is now 79, the gettext page width. 0/false still disables folding. The escapeCharacters: false + foldLength: 0 output does not change.

This is a breaking output change: existing .po output re-wraps once on its next compile, and foldLength: N now means page width N.

flowchart LR
  A["escaped value"] --> B["split into portions at \n"]
  B --> C["UAX 14 break candidates<br/>no breaks inside escapes"]
  C --> D["greedy packing to<br/>foldLength - 1 - prefix columns"]
  D --> E{"wraps or several portions?"}
  E -- no --> F["key #quot;value#quot;"]
  E -- yes --> G["key #quot;#quot;<br/>#quot;piece#quot;<br/>#quot;piece#quot;"]
Loading

Unicode tables

The line-break and width tables live in lib/unicode-data.js, written by scripts/generate-unicode-data.js. The generator has no dependencies. It reads the gnulib tables bundled in an extracted gettext source release and records the gettext and Unicode versions in the file header (currently gettext 1.0, Unicode 17.0.0):

npm run generate-unicode-data -- path/to/gettext-1.0

Testing

The branch is rebased onto test/gnu-wrapping, and both compiler tests are kept. Regenerating test/fixtures/wrapping-msgcat.po with msgcat 1.0 (npm run test-generate-po) gives a byte-identical file.

Compared against msgcat --width=N 1.0 on five seeded random corpora (HTML, escapes, tabs, newlines, CJK, Hangul, emoji, Hebrew, Devanagari, Thai, Arabic, URLs, plurals, msgctxt, obsolete entries) at widths 20, 40, 60, 79 and 100, this branch differs on 0 of about 171,000 lines. Comment lines (#:, #|, ...) are outside this change and were excluded.

Not covered, as before: legacy CJK encodings (EUC-JP, GBK, BIG5) use the non-CJK width rules, and format-directive break overrides are not modelled.

@smhg

smhg commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Thanks a lot for this.

On your open questions:

  • Output change: keeping foldLength with GNU's page-width meaning is good. We'll release this as a new major.
  • gettext version: please go with the latest (1.0).
  • Tables: yes, please add the generator. Ideally it writes the tables to their own file (e.g. lib/unicode-data.js), uses no extra dependencies, and states the gettext/Unicode version it was built from.

I've also pushed a test to the branch test/gnu-wrapping. It compiles test/fixtures/wrapping.po (including the #51 examples) and compares the result with msgcat output. Could you rebase onto it?

git fetch https://github.com/smhg/gettext-parser.git test/gnu-wrapping
git rebase FETCH_HEAD

You'll get a small conflict in test/po-compiler-test.js; keep both tests. I generated the reference with msgcat 0.23.2, so please regenerate it with 1.0 (npm run test-generate-po) as part of your changes.

Port the line wrapping used by the GNU gettext tools: wrap() in
gettext-tools/src/write-po.c plus the libunistring unilbrk
(UAX #14 line breaking) and uniwidth code it relies on.

Instead of folding only the string value at a fixed length, the
whole logical line (key prefix, quotes and escape sequences) is
counted, strings are split into portions at real newlines, and
multi-portion or overflowing strings use the same 'msgid ""'
quoted-piece style the GNU tools emit. Default foldLength changes
from 76 to 79 to match gettext's page width.
Port u8_possible_linebreaks_loop() as of gettext 1.0 (Unicode 17.0.0):
east asian width aware quotation marks (LB19a), word-initial hyphens
(LB20a), numbers (LB25), Brahmic syllables (LB28a) and the HTML tag
exception to LB29.

The line break classes, pair table and width tables now live in
lib/unicode-data.js, written by scripts/generate-unicode-data.js from
the gnulib tables in a gettext source release. Regenerate with:

  npm run generate-unicode-data -- path/to/gettext-1.0
@cpruijsen
cpruijsen force-pushed the fix/issue-51-21cafd10 branch from b81130d to 4c9f46d Compare October 18, 2026 23:31
@cpruijsen

Copy link
Copy Markdown
Contributor Author

I rebased onto test/gnu-wrapping and kept both tests. The wrapping now follows gettext 1.0 (Unicode 17.0.0). The tables moved to lib/unicode-data.js. npm run generate-unicode-data -- <extracted gettext source> writes that file with no dependencies and records the gettext and Unicode versions in its header. When I regenerated wrapping-msgcat.po with msgcat 1.0, the file was byte-identical, so 0.23.2 and 1.0 agree on that fixture.

@smhg

smhg commented Oct 19, 2026

Copy link
Copy Markdown
Owner

Thanks! Does the Windows failure tell you anything?

@cpruijsen

Copy link
Copy Markdown
Contributor Author

Yes, that one was mine. .gitattributes checks the fixtures out with CRLF on Windows, and the new test compiled with a hardcoded \n. It now uses EOL like the other compiler tests (263be71).

@smhg
smhg merged commit 75fe5c4 into smhg:main Oct 22, 2026
6 checks passed
@smhg

smhg commented Oct 22, 2026

Copy link
Copy Markdown
Owner

Merged and released as 10.0.0. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adjust line wrapping algorithm to be closer to the GNU gettext tooling

2 participants