Add missing return type hints to BaseTokenizer.save, save_model, to_str#2211
Open
RudrenduPaul wants to merge 1 commit into
Open
Add missing return type hints to BaseTokenizer.save, save_model, to_str#2211RudrenduPaul wants to merge 1 commit into
RudrenduPaul wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds missing return type hints to three methods in
bindings/python/py_src/tokenizers/implementations/base_tokenizer.py:save_model(self, directory: str, prefix: Optional[str] = None) -> List[str]save(self, path: str, pretty: bool = True) -> Noneto_str(self, pretty: bool = False) -> strThis is a self-identified typing-consistency fix (no linked issue). Every
other method in this class already carries a return type hint; these three
were the only ones missing one.
to_str's own docstring already documents"Returns: str", so the annotation was simply never added to the signature.
Why these types are correct
Each method is a thin wrapper delegating to the compiled Rust bindings, and
the return type was verified against the auto-generated
.pyistubs(
bindings/python/py_src/tokenizers/__init__.pyiandbindings/python/py_src/tokenizers/models.pyi), which are the source oftruth for the underlying Rust-exposed signatures:
save_modelreturnsself._tokenizer.model.save(directory, prefix=prefix).Model.save(self, /, folder: str, prefix: str | None = None, name: str | None = None) -> "list[str]"per
models.pyi— henceList[str](the file already importsListfromtyping).savereturnsself._tokenizer.save(path, pretty).Tokenizer.save(self, /, path: str, pretty: bool = True) -> "None"per__init__.pyi— henceNone.to_strreturnsself._tokenizer.to_str(pretty).Tokenizer.to_str(self, /, pretty: bool = False) -> "str"per__init__.pyi— hencestr, matching the existing docstring.Scope note: not a duplicate of #1928 / issue #1927
#1928 (merged, closing issue #1927) added type hints
and an automatic stub generator for the compiled PyO3 bindings (
.pyifiles,
stub.py,stub-gentool). It did not touch the pure-Pythonimplementations/package, sobase_tokenizer.py's three unannotatedmethods were left untouched by that effort. Open PR #1932 ("fix: added type
hints in .py files") likewise only touches benches/examples/scripts/tests,
not
base_tokenizer.py. Confirmed viagh pr viewfile lists beforestarting this change.
No behaviour change
Type-hint-only change. No logic was modified.
Testing
ruff check bindings/python/py_src/tokenizers/implementations/base_tokenizer.py— all checks passed.ruff format --check bindings/python/py_src/tokenizers/implementations/base_tokenizer.py— passes, file already formatted.python3 -c "import ast; ast.parse(...)"on the modified file — passes.