Skip to content

fix: UniversalDetector crashes with concurrent calls and after a weighted reset - #182

Merged
PyYoshi merged 2 commits into
masterfrom
fix/universal-detector-thread-safety
Nov 13, 2026
Merged

PyYoshi merged 2 commits into
masterfrom
fix/universal-detector-thread-safety

Conversation

@PyYoshi

@PyYoshi PyYoshi commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Summary

This PR fixes two memory-safety bugs in UniversalDetector. Both can be reproduced with the public API in cChardet 2.4.0, the current PyPI release.

1. A detector shared between threads crashes the process, even on GIL builds.

feed(), reset(), and close() run uchardet inside with nogil:. While one thread is inside uchardet, another thread can call into the same native detector. A stress script that shares one detector across 8 threads crashed in 17 of 20 runs on CPython 3.15.2 and 3.15.2t:

  • free-threaded builds: free(): invalid pointer in uchardet_reset (SIGABRT)
  • GIL builds: SIGSEGV in close()

The other 3 runs either completed or returned corrupted encoding names.

The detector now holds a per-instance cython.pymutex in every method and property.

  • Why cython.pymutex: it stays held across the nogil sections. Unlike cython.critical_section, it is not suspended when the thread detaches, and it is not a no-op on GIL builds.
  • No GIL deadlock: acquiring it releases the GIL or detaches the thread state while waiting. It is a PyMutex on 3.13+ and a PyThread_type_lock on 3.11/3.12.
  • No reentrancy: the lock is not reentrant, so feed() now finishes early completion through a @cython.final _close_locked() instead of calling close().
  • No Python code under the lock: user code that runs while the lock is held could call back into the detector. The weights are therefore converted before the lock is taken. The result/done objects are built, and the RuntimeError from feed() is raised, after the lock is released.

2. reset() leaves dangling weighted candidates (bundled uchardet).

HandleUniversalDetector::Reset() freed the candidate strings but kept weighed_candidates, which borrows them. With language_weights set, close() after reset() with no new data returned freed memory as the encoding. The result usually raised UnicodeDecodeError with different garbage bytes each run. The bug dates from 2.2.0, and upstream uchardet has the same code.

  • Fixed in PyYoshi/uchardet@1702c11 (cchardet branch).
  • The submodule is bumped.
  • A new C lifecycle test lets the existing Native ASan / UBSan job catch this deterministically.

Behavior

  • Single-threaded use is unchanged. 18 scenarios compared between the old and new builds give identical results: early completion, feeds after done, repeated close(), reset() after close(), empty feeds, feed(None), and invalid weights. The only difference is in the private _cchardet.UniversalDetector: a subclass that overrides close() is no longer called on early completion. The public cchardet.UniversalDetector wraps the native class instead of subclassing it, so it is unaffected.
  • Sharing a detector between threads no longer crashes, but the calls interleave. All feeds go into one stream, and any thread's reset()/close() affects every caller. done and result wait for a feed that is in progress. The README now says so and recommends one detector per input.
  • The free-threaded builds keep the GIL disabled.

Tests and CI

  • test_shared_detector_survives_concurrent_calls: 8 threads cycle feed/reset/close/result on one detector, with and without language weights. It takes about 0.3 s.
  • test_language_weights_survive_reset_without_new_data: a single-threaded regression test for bug 2.
  • uchardet-lifecycle-tests (in the submodule): the same reset path in C, run by ctest.
  • The test job gets timeout-minutes: 20, and pytest gets faulthandler_timeout = 300. A lock regression would otherwise hang a runner until GitHub's 6-hour limit. The longest test job currently takes 110 s. Job names are unchanged.

Validation

Before the fix:

  • The new concurrency test crashed or failed in 25 of 25 runs on CPython 3.11, 3.13, 3.14, 3.15.2, and 3.15.2t.
  • The weighted-reset test failed on 3.13, 3.15.2, and 3.15.2t.
  • The C lifecycle test fails with heap-use-after-free under the CI's GCC ASan/UBSan configuration and under clang.

After the fix:

  • The full suite (31 tests) passes on 3.11, 3.12, 3.13, 3.14, 3.14t, 3.15.2, and 3.15.2t. ruff and strict mypy pass.
  • All 154 uchardet ctest cases pass under GCC 16 ASan/UBSan with the CI flags, including -Werror=maybe-uninitialized.
  • Stress:
    • The crash-reproducing loop completes in 70 of 70 runs (10 per interpreter).
      Before the fix, the same loop crashed in 17 of 20 runs.
    • 20-second mixed-workload runs complete without errors on 3.11, 3.13,
      3.15.2, and 3.15.2t.
  • _cchardet.pyx cythonizes with -Werror under both Cython 3.2.9 (the build-system minimum) and 3.3.1. The test builds use 3.3.1, which is what isolated builds pick.
  • Reentrancy probes complete without deadlocking. One is a finalizer that touches the detector during a GC run triggered by an allocation in result on 3.11. The other is a mapping whose items() calls reset() during __init__.

Single-threaded overhead (pyperf, both builds compiled with the same interpreter):

Case 3.11 (PyThread_type_lock) 3.15.2 (PyMutex) 3.15.2t
feed() of 17 bytes 80.7 → 103 ns 75.5 → 76.0 ns (not significant) 55.0 → 54.1 ns (not significant)
reset + 50-line feed + close + result 586 → 585 µs (not significant) 587 → 587 µs (not significant) 570 → 571 µs (not significant)

detect() and detect_all() create their own detector per call and do not take the lock.

With language weights set, UniversalDetector.close() after reset() with
no new data returned freed memory as the encoding, usually raising
UnicodeDecodeError. uchardet's Reset() freed the candidate strings while
its weighed_candidates still borrowed them.

Bump the uchardet submodule to PyYoshi/uchardet@1702c11, which clears
them on reset and adds a lifecycle test that the Native ASan / UBSan job
runs, and add a Python regression test.
UniversalDetector runs uchardet without the GIL, so concurrent calls on
one detector could enter the same native state at once. That corrupted
memory and crashed the process on both regular and free-threaded
builds, for example with free(): invalid pointer in uchardet_reset() or
a segfault in close().

Hold a cython.pymutex in every method and property. It stays held
across the nogil sections, unlike cython.critical_section, and waiting
for it releases the GIL or detaches the thread state. Because it is not
reentrant, feed() completes early detection through a final
_close_locked(), and Python objects are converted or built outside the
lock so that finalizers or user mappings cannot re-enter it.

Add a test that drives one shared detector from eight threads, document
the threading behavior, and bound hangs in CI with a job timeout and
pytest's faulthandler_timeout.
@PyYoshi
PyYoshi merged commit ef85862 into master Nov 13, 2026
47 checks passed
@PyYoshi
PyYoshi deleted the fix/universal-detector-thread-safety branch November 13, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant