Preserve backslashes in generated Python stub docstrings#2218
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses SyntaxWarning: invalid escape sequence errors raised by Python 3.12+ when compiling the generated bindings/python/py_src/tokenizers/pre_tokenizers.pyi, caused by regex examples containing \w / \s inside non-raw Python docstrings.
Changes:
- Escapes backslashes in two Rust binding doc-comment regex examples so the generated Python stub no longer contains invalid escape sequences.
- Updates the generated
pre_tokenizers.pyito reflect the new escaped docstrings.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| bindings/python/src/pre_tokenizers.rs | Updates doc-comment regex examples by doubling backslashes. |
| bindings/python/py_src/tokenizers/pre_tokenizers.pyi | Regenerates stubs so docstrings compile cleanly under -W error::SyntaxWarning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /// This pre-tokenizer splits on word boundaries according to the ``\\w+|[^\\w\\s]+`` | ||
| /// regex pattern. It splits on word characters or characters that aren't words or | ||
| /// whitespaces (punctuation such as hyphens, apostrophes, commas, etc.). |
There was a problem hiding this comment.
Good catch. I moved the escaping to the stub-generation boundary in 2a5950c and restored the Rust doc comments. The generator now escapes introspected docstrings only before emitting ordinary Python string literals, so runtime .__doc__ still shows a copyable single-backslash regex while the generated stub contains the required source escape. I also covered module/class/function/attribute docstrings and refreshed other affected examples such as \n and \u0301.
| /// [('one', (0, 3)), ('two', (4, 7)), ('three', (8, 13))] | ||
| /// >>> # Split using a regex, keeping the delimiter isolated | ||
| /// >>> Split(Regex(r"\s+"), behavior="isolated").pre_tokenize_str("hello world") | ||
| /// >>> Split(Regex(r"\\s+"), behavior="isolated").pre_tokenize_str("hello world") |
There was a problem hiding this comment.
Good catch. I moved the escaping to the stub-generation boundary in 2a5950c and restored the Rust doc comments. The generator now escapes introspected docstrings only before emitting ordinary Python string literals, so runtime .__doc__ still shows a copyable single-backslash regex while the generated stub contains the required source escape. I also covered module/class/function/attribute docstrings and refreshed other affected examples such as \n and \u0301.
cupkk
changed the title
The stub generator writes introspected runtime docstrings as ordinary Python triple-quoted strings. Backslash sequences are therefore interpreted a second time: regex escapes such as
\sraiseSyntaxWarning, while valid sequences such as\nsilently change the generated docstring.This escapes backslashes in module, class, function, and attribute docstrings immediately before stub emission. Runtime docstrings remain human-readable, while parsed stub docstrings preserve the same text. The affected generated normalizer and pre-tokenizer stubs are refreshed.
Validation:
cargo test --manifest-path bindings/python/tools/stub-gen/Cargo.tomlpython -W error::SyntaxWarning -m compileall -q bindings/python/py_src/tokenizerspytest --import-mode=importlib tests/bindings -q(171 passed)Fixes #1791