Skip to content

LazyString is corrupted by copy, deepcopy and pickle on Python 3.11+ #130

Description

@DerevenetsArtyom

Affected versions

  • transifex-python 3.7.0 (latest on PyPI) and current devel
  • Python 3.11 and 3.12 (works correctly on 3.10 and below)
  • Independent of the Django version

Steps to reproduce

import copy, pickle, sys
from transifex.common.strings import LazyString

def render():
    return "hello"

v = f"{sys.version_info.major}.{sys.version_info.minor}"
s = LazyString(render)
print(f"py{v}  str(original)      -> {str(s)!r}")
for name, fn in (("copy.copy", copy.copy), ("copy.deepcopy", copy.deepcopy),
                 ("pickle round-trip", lambda x: pickle.loads(pickle.dumps(x)))):
    try:
        print(f"py{v}  {name:18} -> {str(fn(s))!r}")
    except Exception as e:
        print(f"py{v}  {name:18} -> {type(e).__name__}: {e}")

A module-level function is used rather than a lambda so the pickle case is meaningful on every version.

uv run --no-project --python 3.10 --with 'transifex-python==3.7.0' python repro.py
uv run --no-project --python 3.11 --with 'transifex-python==3.7.0' python repro.py
uv run --no-project --python 3.12 --with 'transifex-python==3.7.0' python repro.py

Observed behavior

py3.10  str(original)      -> 'hello'
py3.10  copy.copy          -> 'hello'
py3.10  copy.deepcopy      -> 'hello'
py3.10  pickle round-trip  -> 'hello'

py3.11  str(original)      -> 'hello'
py3.11  copy.copy          -> TypeError: 'str' object is not callable
py3.11  copy.deepcopy      -> TypeError: 'str' object is not callable
py3.11  pickle round-trip  -> TypeError: 'str' object is not callable

py3.12  (same as 3.11)

The original object is fine — only copies are corrupted, and the exception is raised later, when the copy is evaluated.

Expected behavior

A copied or unpickled LazyString should still evaluate to the same text, as it does on Python 3.10.

Why this is a problem

It is easy to hit through Django without doing anything unusual. django/forms/forms.py does self.fields = copy.deepcopy(self.base_fields) on every form instantiation, and ChoiceField.__deepcopy__ deepcopies its choices. So a lazy string used as a choice label — for example on a TextChoices — is corrupted on every request that builds such a form.

It is also hard to trace back to this library. The traceback surfaces inside copy and django.forms, and the only frame belonging to transifex is a generated wrapper named b, with no mention of __getstate__:

django/forms/forms.py:107        self.fields = copy.deepcopy(self.base_fields)
django/forms/fields.py:867       result._choices = copy.deepcopy(self._choices, memo)
copy.py:151                      rv = reductor(4)
transifex/common/strings.py:136  return func(str(self), *args, **kwargs)

Cause

Python 3.11 added object.__getstate__ (gh-70766 / bpo-26579), so '__getstate__' in dir(str) is False on 3.10 and True from 3.11 on.

lazy_str_meta rewrites every name in set(dir(str)) minus an exclusion set. That set deliberately protects the serialization protocol — it lists __reduce__, __reduce_ex__ and __getnewargs__ — but it predates 3.11 and therefore does not list __getstate__.

As a result LazyString.__getstate__ returns the evaluated text as the object's state rather than _func / _args / _kwargs. copy and pickle then apply that string as the new object's state, overwriting _func, and the next evaluation runs return self._func(*self._args, **self._kwargs) with self._func being a str.

object.__reduce_ex__ consults __getstate__ from C, which is why no intermediate Python frame appears in the traceback.

Suggested fix

Add "__getstate__" to the exclusion set in lazy_str_meta, alongside the other serialization-protocol entries.

__getstate__ is the only name added to dir(str) between 3.10 and 3.12, so this single entry covers the regression rather than being the first of several:

added to dir(str) in 3.12 vs 3.10: ['__getstate__']
removed: []

Note on CI

The test matrix already covers py3.11-dj4.1 and py3.12-dj4.2 and is green, because nothing in the current suite copies or pickles a LazyString.

I have a patch ready with the one-line change plus a regression test in tests/common/test_strings.py, verified to fail without the fix and to pass with it, and green on your py3.12-dj4.2 and py3.9-dj3.2 images. Happy to open the PR if that is welcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions