Fix Unicode normalization on PyPy - #74
Merged
Merged
Conversation
Port normalization conformance tests from CPython 9ab004d41e into test_unicodedata2.py and remove the stale reference to test_normalization.py. Also add coverage for long combining-mark runs, private-use names, newer character properties, aliases and named sequences, and check numeric consistency across all code points. Download the current database's normalization corpus and the frozen Unicode 3.2 corpus during CI and tox setup. Select the current version from the generated database header, validate the downloaded file headers, and fail on download errors instead of skipping conformance checks. Keep downloaded data out of git and distributions. On unfixed PyPy, both normalization conformance tests and both combining-mark run tests fail. The following canonical-ordering backport makes them pass. Tests for APIs or behavior not yet backported from CPython are left out.
Port the normalization sorting changes from CPython 991224b1e8 and 90748760d3 (gh-149079). Already-sorted combining-character runs are skipped; unsorted runs use stable insertion sort below 20 characters and stable counting sort for longer runs, avoiding quadratic canonical ordering. Both paths sort the temporary Py_UCS4 buffer before creating the Python string. This also fixes lost mark reordering on PyPy, where writes to PyUnicode_DATA after materialization were not reflected in the returned string. Keep this backport's self null check and resize a temporary scratch-buffer pointer, so allocation failure preserves the old pointer for cleanup. Otherwise the helpers and normalization function follow CPython 9ab004d41e. All 34 tests pass on CPython 3.9, CPython 3.14 and PyPy 3.11, including the Unicode 17 and 3.2 conformance corpora. Mixed-mark and threshold-boundary probes match CPython's normalization results.
Member
Author
|
cibuildwheel 3.0 made PyPy an opt-in, so PyPy wheels stopped shipping in 17.0.0. They are enabled again on master (#73), so this fixes them before the next Unicode 18 release goes out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes incorrect combining-mark ordering on PyPy by backporting CPython's faster normalization sort (#149080, #150782).
On PyPy, NFD and NFKD returned combining marks in the wrong order, and NFC/NFKC inherited it because they normalize via the decomposed form. The old code created the result string and then reordered the marks in place through
PyUnicode_DATA; under pypy's cpyext that pointer is a temporary copy, so the reordering went to a buffer nobody read. CPython's latest version sorts the decomposition buffer before constructing the string, so the writes land on PyPy as well.I also ported upstream normalization, character-property and name-lookup tests. CI and tox download version-matched conformance data; download errors fail the run.