Repository navigation
fix: UniversalDetector crashes with concurrent calls and after a weighted reset - #182
Merged
Merged
Conversation
With language weights set, UniversalDetector.close() after reset() with no new data returned freed memory as the encoding, usually raising UnicodeDecodeError. uchardet's Reset() freed the candidate strings while its weighed_candidates still borrowed them. Bump the uchardet submodule to PyYoshi/uchardet@1702c11, which clears them on reset and adds a lifecycle test that the Native ASan / UBSan job runs, and add a Python regression test.
UniversalDetector runs uchardet without the GIL, so concurrent calls on one detector could enter the same native state at once. That corrupted memory and crashed the process on both regular and free-threaded builds, for example with free(): invalid pointer in uchardet_reset() or a segfault in close(). Hold a cython.pymutex in every method and property. It stays held across the nogil sections, unlike cython.critical_section, and waiting for it releases the GIL or detaches the thread state. Because it is not reentrant, feed() completes early detection through a final _close_locked(), and Python objects are converted or built outside the lock so that finalizers or user mappings cannot re-enter it. Add a test that drives one shared detector from eight threads, document the threading behavior, and bound hangs in CI with a job timeout and pytest's faulthandler_timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes two memory-safety bugs in
UniversalDetector. Both can be reproduced with the public API in cChardet 2.4.0, the current PyPI release.1. A detector shared between threads crashes the process, even on GIL builds.
feed(),reset(), andclose()run uchardet insidewith nogil:. While one thread is inside uchardet, another thread can call into the same native detector. A stress script that shares one detector across 8 threads crashed in 17 of 20 runs on CPython 3.15.2 and 3.15.2t:free(): invalid pointerinuchardet_reset(SIGABRT)close()The other 3 runs either completed or returned corrupted encoding names.
The detector now holds a per-instance
cython.pymutexin every method and property.cython.pymutex: it stays held across thenogilsections. Unlikecython.critical_section, it is not suspended when the thread detaches, and it is not a no-op on GIL builds.PyMutexon 3.13+ and aPyThread_type_lockon 3.11/3.12.feed()now finishes early completion through a@cython.final_close_locked()instead of callingclose().result/doneobjects are built, and theRuntimeErrorfromfeed()is raised, after the lock is released.2.
reset()leaves dangling weighted candidates (bundled uchardet).HandleUniversalDetector::Reset()freed the candidate strings but keptweighed_candidates, which borrows them. Withlanguage_weightsset,close()afterreset()with no new data returned freed memory as the encoding. The result usually raisedUnicodeDecodeErrorwith different garbage bytes each run. The bug dates from 2.2.0, and upstream uchardet has the same code.cchardetbranch).Native ASan / UBSanjob catch this deterministically.Behavior
close(),reset()afterclose(), empty feeds,feed(None), and invalid weights. The only difference is in the private_cchardet.UniversalDetector: a subclass that overridesclose()is no longer called on early completion. The publiccchardet.UniversalDetectorwraps the native class instead of subclassing it, so it is unaffected.reset()/close()affects every caller.doneandresultwait for a feed that is in progress. The README now says so and recommends one detector per input.Tests and CI
test_shared_detector_survives_concurrent_calls: 8 threads cyclefeed/reset/close/resulton one detector, with and without language weights. It takes about 0.3 s.test_language_weights_survive_reset_without_new_data: a single-threaded regression test for bug 2.uchardet-lifecycle-tests(in the submodule): the same reset path in C, run byctest.testjob getstimeout-minutes: 20, and pytest getsfaulthandler_timeout = 300. A lock regression would otherwise hang a runner until GitHub's 6-hour limit. The longesttestjob currently takes 110 s. Job names are unchanged.Validation
Before the fix:
heap-use-after-freeunder the CI's GCC ASan/UBSan configuration and under clang.After the fix:
ctestcases pass under GCC 16 ASan/UBSan with the CI flags, including-Werror=maybe-uninitialized.Before the fix, the same loop crashed in 17 of 20 runs.
3.15.2, and 3.15.2t.
_cchardet.pyxcythonizes with-Werrorunder both Cython 3.2.9 (thebuild-systemminimum) and 3.3.1. The test builds use 3.3.1, which is what isolated builds pick.resulton 3.11. The other is a mapping whoseitems()callsreset()during__init__.Single-threaded overhead (pyperf, both builds compiled with the same interpreter):
PyThread_type_lock)PyMutex)feed()of 17 bytesreset+ 50-linefeed+close+resultdetect()anddetect_all()create their own detector per call and do not take the lock.