Skip to content

Fix Unicode regular expressions#33

Open
sutherland wants to merge 2 commits into
h4cc:masterfrom
sutherland:fix-unicode-regex
Open

Fix Unicode regular expressions#33
sutherland wants to merge 2 commits into
h4cc:masterfrom
sutherland:fix-unicode-regex

Conversation

@sutherland

Copy link
Copy Markdown

Slugifying strings containing Unicode characters can lead to invalid results:

iex(1)> Slugger.slugify("abc 😀")
<<97, 98, 99, 45, 159, 152, 128>>

Regular expressions do not match Unicode by default. Adding the unicode modifier seems to resolve the issue.

@etiennealbert

Copy link
Copy Markdown

+1 for the merge.
@sutherland you should use ~r/([^a-z0-9\p{Hangul}])+/u to correctly handle the Korean alphabet.

@inhji

inhji commented Jan 5, 2022

Copy link
Copy Markdown

I would love to see this merged. It causes Postgrex to fail with

** (Postgrex.Error) ERROR 22021 (character_not_in_repertoire) invalid byte sequence for encoding "UTF8": 0x9f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants