PYTHON-5965 Add abi3/abi3t wheel support with BSON performance parity by blink1073 · Pull Request #28 · blink1073/mongo-python-driver

blink1073 · 2026-09-14T20:55:36Z

Adds abi3 (and prepares for abi3t) wheel support: build PyMongo's C extensions against CPython's limited API so one cp311-abi3 wheel covers GIL-enabled Python 3.11+, with cp310 and cp314t kept per-version. The datetime and BSON dispatch hot paths were optimized so the limited-API build stays at or above non-abi3 throughput.

Changes in this PR

Build bson._cbson and pymongo._cmessage against Py_LIMITED_API=0x030B0000, producing a cp311-abi3 wheel; keep per-version wheels for cp310 and cp314t.
Decode datetimes via the frozen PyDateTime_CAPI capsule instead of a 7-arg Python constructor.
Encode datetimes by reading the datetime's raw field layout instead of per-field getattr.
Move both the limited and non-limited datetime paths onto Hinnant's O(1) days_from_civil/civil_from_days and remove the time64 library.
Replace the value dispatch's subclass checks with a Py_TYPE-based exact-type fast path so string, dict, and datetime values no longer hit out-of-line PyType_GetFlags calls under the limited API.
Encode string keys and values with PyUnicode_AsUTF8AndSize, avoiding a per-element temporary bytes allocation.
Add a threshold-gated BSON regression benchmark and CI jobs that install the abi3 wheel on CPython 3.11-3.14 plus the next pre-release, so a change to the non-stable PyDateTime_CAPI layout is caught during a release's beta phase.

Test Plan

test_bson.py + test_bson_corpus.py: 125 passed (abi3 build), 94 passed (non-limited build).
Differential correctness: abi3 encode output is byte-identical to the non-limited baseline; 0 decode mismatches across ~6000 datetime values; year-1/year-9999 boundaries, tz-aware, DATETIME_CLAMP/AUTO/MS, the out-of-range InvalidBSON path, and subclasses (SON, Int64, user list/tuple/dict/str/bytes) all verified.
Performance on a dedicated host, interleaved A/B via test/performance/bench_abi3.py against the extended_bson fixtures. A positive delta means the abi3 build is faster than the non-limited baseline. All within the <3% overall / <10% datetime / <3% _cmessage targets.

Comparison across main / non-limited / abi3 (MB/s, sum of MB/s)

Measured on an EC2 m5.metal perf host (gcc, CPython 3.11) against the extended_bson fixtures. "non-limited" is this branch built without the limited API, carrying all the optimization changes.

fixture	main	non-limited	abi3
deep_bson.encode	193.46	259.55	272.92
deep_bson.decode	141.24	138.66	142.99
flat_bson.encode	410.86	517.75	529.39
flat_bson.decode	327.91	333.34	324.28
full_bson.encode	232.09	264.68	259.45
full_bson.decode	113.75	113.89	114.24
Overall (sum)	1419.31	1627.87	1643.26

Deltas: abi3 vs main +15.78% overall (deep encode +41.07%, flat encode +28.85%, full encode +11.79%; decode at parity), abi3 vs non-limited +0.95% overall (the pure limited-API cost), non-limited vs main +14.69% overall.

BSON fixtures (MB/s)

fixture	baseline	abi3	delta
deep_bson.encode	211.97	222.64	+5.03%
deep_bson.decode	105.02	108.40	+3.22%
flat_bson.encode	438.14	436.50	-0.38%
flat_bson.decode	279.26	276.93	-0.84%
full_bson.encode	232.84	234.53	+0.73%
full_bson.decode	91.09	95.12	+4.42%
Overall (sum)	1358.34	1374.12	+1.16%

Datetime encode (thousands of datetimes/s)

path	baseline	abi3	delta
naive	12,490	12,790	+2.4%
tz-aware	3,010	2,760	-8.3%

The naive datetime path was the largest regression in the earlier abi3 build (a redundant utcoffset() call was charged to every naive datetime on encode); it is now faster than the non-limited baseline. tz-aware datetimes still need a utcoffset() call, which the limited API cannot route through _PyObject_GetMethod, so that path remains the smallest gap.

just typing (mypy + pyright): passes. just lint currently fails on pre-existing ruff errors and synchro drift in the base tree, unrelated to this change.

Checklist

Checklist for Author

Did you update the changelog (if necessary)?
Is there test coverage?
Is any followup work tracked in a JIRA ticket? If so, add link(s). (PYTHON-5965; the time64 removal is tracked as the prior "Improve time usage" ticket.)

Checklist for Reviewer

Does the title of the PR reference a JIRA Ticket?
Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
Is all relevant documentation (README or docstring) updated?

Build the C extensions against CPython's limited API so a single cp311-abi3 wheel covers GIL-enabled Python 3.11+, keeping cp310/cp314t per-version. Replace the non-limited datetime C API with limited-API-compatible paths and move both paths onto Hinnant's O(1) calendar math, dropping time64. Optimize BSON encode (exact-type dispatch, PyUnicode_AsUTF8AndSize) so abi3 encode/decode and _cmessage stay within the performance targets. Add a threshold-gated BSON regression benchmark and cross-version CI jobs for the abi3 wheel.

+    name: Test abi3 wheel on CPython ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    needs: build_wheels
+    strategy:
+      fail-fast: false
+      matrix:
+        # The cp311-abi3 wheel claims support for every GIL-enabled Python
+        # >= 3.11. Its decode path relies on CPython's frozen PyDateTime_CAPI
+        # (re-declared in bson/_cbsonmodule.c), so we must run the BSON suite
+        # against every version we ship it for, not just the build Python.
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - name: Checkout pymongo
+        uses: actions/checkout@v7.0.1
+        with:
+          persist-credentials: false
+          ref: ${{ inputs.ref }}
+
+      - uses: actions/setup-python@v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+          allow-prereleases: true
+
+      - name: Download abi3 wheel
+        uses: actions/download-artifact@v8
+        with:
+          name: wheel-manylinux_x86_64
+          path: wheelhouse
+
+      - name: Install abi3 wheel and run BSON suite
+        run: |
+          python -m pip install wheelhouse/*cp311-abi3*.whl
+          python tools/fail_if_no_c.py
+          python -m pytest test/test_bson.py test/test_bson_corpus.py -q
+
+  test_abi3_next_release:


Port the buffer_owner / raw_bson_view_threshold zero-copy from the base into the limited-API build: elements_to_dict slices the owner buffer instead of copying, and _raw_to_dict + _prepare_input_buffer + _contains_view_eligible_doc are added for RawBSONDocument inflation. PyBytes_AS_STRING/GET_SIZE are replaced with PyBytes_AsStringAndSize so the path compiles under Py_LIMITED_API.

cibuildwheel rejects a build selector that appends a platform to an abi3 tag (e.g. cp311-abi3-manylinux_i686), so use the bare cp311-abi3 selector and pass the target arch via CIBW_ARCHS. Regenerate uv.lock after the requires-python >=3.10 bump so the lockfile check passes.

…uild Match the base's RawBSONDocument behavior: - _cbson_bson_to_dict / _cbson_decode_all run _prepare_input_buffer so memoryview/bytearray inputs yield an owned bytes buffer for zero-copy slices. - write_raw_doc accepts bytes and memoryview (via _get_buffer) and rejects other buffer types with the expected message (no opaque tp_name access under Py_LIMITED_API). - _cbson_dict_to_bson converts a non-bytes raw to bytes via PyBytes_FromObject so encode always yields bytes. Release the prepared bson buffer on cleanup.

Revert the requires-python drop to avoid triggering UP036 ruff errors across the codebase (the static check is pre-existing lint debt). cibuildwheel does not accept a bare cp311-abi3 selector or an abi3+manylinux selector for i686/ppc64le/s390x, so build per-version cp311 wheels for those archs and keep the platform-qualified selector for x86_64/aarch64/win/macos. Regenerate the lockfile for the restored 3.9 minimum.

cibuildwheel 3.4.1 rejects both platform-qualified ('cp311-abi3-manylinux_*') and bare ('cp311-abi3') abi3 build selectors. Use the plain 'cp311' interpreter selector (the abi3 tag is inferred from py_limited_api) plus CIBW_ARCHS for cross-arch, keeping PYMONGO_BUILD_ABI3=1 for the abi3 rows.

cibuildwheel 3.4.1 rejects bare 'cp311', bare 'cp311-abi3', and platform-qualified 'cp311-abi3-manylinux_*' build selectors. Strip the '-abi3' suffix so abi3 rows build with the accepted per-version selector (e.g. 'cp311-manylinux_x86_64'); the abi3 tag comes from py_limited_api with PYMONGO_BUILD_ABI3=1.

cibuildwheel 3.4.1 rejects bare 'cp311', bare 'cp311-abi3', and platform-qualified 'cp311-abi3-manylinux_*' build selectors, and GitHub Actions has no replace() expression function, so earlier attempts failed to parse or select. Add a fourth matrix column holding the accepted per-version interpreter selector (e.g. cp311-manylinux_x86_64) for each row; the abi3 tag comes from py_limited_api with PYMONGO_BUILD_ABI3=1.

…3 build PyObject_Vectorcall and PyObject_VectorcallMethod are PEP-590 optimizations that are not part of the PEP-384 stable ABI, so CPython does not export them from its import library on Windows. The limited-API build declared them as PyAPI_FUNC and failed to link abi3 wheels on Windows/macOS with unresolved externals. Route the limited-API build through stable PyObject_Call/PyObject_CallMethod helpers (which build a positional-args tuple); the non-limited build keeps the inlined vectorcall fast path. Fixes the abi3 Windows/macOS wheel build.

cibuildwheel's default macOS arch did not match delocate's --require-archs, causing the repaired wheel to fail (no x86_64 binary). Build arm64 explicitly on the Apple Silicon macos-14 runner so the wheel and delocate agree.

download-artifact compares the artifact digest against the previous run's, so the test_abi3 jobs aborted with a digest mismatch and installed no wheel. Pin to the current run and warn instead of failing on the digest change.

Multiple rows shared a platform (e.g. manylinux_x86_64 builds cp311-abi3, cp310 and cp314t), so same-name uploads overwrote and the abi3 wheel was lost. Name each artifact by its build selector (buildplat[2]) and download the abi3 one by that unique name; drop the macosx_* wildcard from buildplat[2] (illegal in artifact names - the wildcard stays on the buildplat[3] cibuildwheel selector).

Importing the installed wheel from the repo root resolves pymongo/bson to the checked-out source (no C extension) and fails fail_if_no_c. Leave the source tree after install so the wheel's _cmessage/_cbson are used, and run pytest with importlib mode from the runner temp dir.

+      - name: Install abi3 wheel and run BSON suite
+        run: |
+          python -m pip install pytest wheelhouse/*cp311-abi3*.whl
+          # Leave the source tree so the freshly-installed wheel (with the C
+          # extension) is imported, not the checkout's pure-Python fallback.
+          cd "$RUNNER_TEMP"
+          python "$GITHUB_WORKSPACE/tools/fail_if_no_c.py"
+          python -m pytest --import-mode=importlib "$GITHUB_WORKSPACE/test/test_bson.py" "$GITHUB_WORKSPACE/test/test_bson_corpus.py" -q
+
+  test_abi3_next_release:


The repo pyproject pytest options need pytest-asyncio and the live-server test markers (--strict-config, asyncio_default_fixture_loop_scope). Using them for two pure-BSON files against an installed wheel fails at config load. Give those files an empty pytest config so they run with defaults.

+  test_abi3_compat:
+    name: Test abi3 wheel on CPython ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    needs: build_wheels
+    strategy:
+      fail-fast: false
+      matrix:
+        # The cp311-abi3 wheel claims support for every GIL-enabled Python
+        # >= 3.11. Its decode path relies on CPython's frozen PyDateTime_CAPI
+        # (re-declared in bson/_cbsonmodule.c), so we must run the BSON suite
+        # against every version we ship it for, not just the build Python.
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - name: Checkout pymongo
+        uses: actions/checkout@v7.0.1
+        with:
+          persist-credentials: false
+          ref: ${{ inputs.ref }}
+
+      - uses: actions/setup-python@v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+          allow-prereleases: true
+
+      - name: Download abi3 wheel
+        uses: actions/download-artifact@v8
+        with:
+          # Pin to this run - the wheel is rebuilt for every commit, so a
+          # digest check against a prior run's artifact fails spurious.
+          run-id: ${{ github.run_id }}
+          name: wheel-cp311-abi3-manylinux_x86_64
+          path: wheelhouse
+          digest-mismatch: warn
+
+      - name: Install abi3 wheel and run BSON suite
+        run: |
+          python -m pip install pytest wheelhouse/*cp311-abi3*.whl
+          # Leave the source tree so the freshly-installed wheel (with the C
+          # extension) is imported, not the checkout's pure-Python fallback.
+          cd "$RUNNER_TEMP"
+          python "$GITHUB_WORKSPACE/tools/fail_if_no_c.py"
+          # The repo pyproject pytest config needs pytest-asyncio and the
+          # live-server test markers (--strict-config, asyncio fixture scope);
+          # these two sync BSON files run fine under an empty config.
+          printf '[pytest]\n' > pytest.ini
+          python -m pytest -c "$RUNNER_TEMP/pytest.ini" --import-mode=importlib "$GITHUB_WORKSPACE/test/test_bson.py" "$GITHUB_WORKSPACE/test/test_bson_corpus.py" -q


+      - name: Install abi3 wheel and run BSON suite
+        run: |
+          python -m pip install pytest wheelhouse/*cp311-abi3*.whl
+          # Leave the source tree so the freshly-installed wheel (with the C
+          # extension) is imported, not the checkout's pure-Python fallback.
+          cd "$RUNNER_TEMP"
+          python "$GITHUB_WORKSPACE/tools/fail_if_no_c.py"
+          # The repo pyproject pytest config needs pytest-asyncio and the
+          # live-server test markers (--strict-config, asyncio fixture scope);
+          # these two sync BSON files run fine under an empty config.
+          printf '[pytest]\n' > pytest.ini
+          python -m pytest -c "$RUNNER_TEMP/pytest.ini" --import-mode=importlib "$GITHUB_WORKSPACE/test/test_bson.py" "$GITHUB_WORKSPACE/test/test_bson_corpus.py" -q


+    name: Test abi3 wheel on the next CPython pre-release
+    runs-on: ubuntu-latest
+    needs: build_wheels
+    # Best-effort backstop for the non-stable-ABI PyDateTime_CAPI usage: if
+    # the next CPython (e.g. 3.15 dev/beta) resolves or changes that layout,
+    # we want to catch it during its beta phase. allow-prereleases + a "-dev"
+    # specifier tracks the in-development version; the job is non-blocking so
+    # a version that has not yet been published on the setup-python mirror
+    # does not fail the release.
+    continue-on-error: true
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.15-dev"]
+    steps:
+      - name: Checkout pymongo
+        uses: actions/checkout@v7.0.1
+        with:
+          persist-credentials: false
+          ref: ${{ inputs.ref }}
+
+      - uses: actions/setup-python@v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+          allow-prereleases: true
+
+      - name: Download abi3 wheel
+        uses: actions/download-artifact@v8
+        with:
+          run-id: ${{ github.run_id }}
+          name: wheel-cp311-abi3-manylinux_x86_64
+          path: wheelhouse
+          digest-mismatch: warn
+
+      - name: Install abi3 wheel and run BSON suite
+        run: |
+          python -m pip install pytest wheelhouse/*cp311-abi3*.whl
+          # Leave the source tree so the freshly-installed wheel (with the C
+          # extension) is imported, not the checkout's pure-Python fallback.
+          cd "$RUNNER_TEMP"
+          python "$GITHUB_WORKSPACE/tools/fail_if_no_c.py"
+          # The repo pyproject pytest config needs pytest-asyncio and the
+          # live-server test markers (--strict-config, asyncio fixture scope);
+          # these two sync BSON files run fine under an empty config.
+          printf '[pytest]\n' > pytest.ini
+          python -m pytest -c "$RUNNER_TEMP/pytest.ini" --import-mode=importlib "$GITHUB_WORKSPACE/test/test_bson.py" "$GITHUB_WORKSPACE/test/test_bson_corpus.py" -q


handle_datetime calls utcoffset() for every datetime on encode, even naive ones where it always returns None. The call is charged to all builds (not just abi3) and costs ~25% of naive datetime encode time. Shortcut the exact, naive case by reading the hastzinfo/tzinfo-is-None flag and writing the millis directly, keeping the utcoffset() path for aware datetimes and subclasses that may override utcoffset(). Eliminates ~27% of naive datetime encode cost in the full-API build and makes the abi3 naive path ~30% faster than the previous abi3 build.

This reverts commit 14f715b.

Py_IS_TYPE was applied to the dict class object and always returned false, so every decode took the slow PyObject_SetItem path instead of the fast PyDict_SetItem path. Compare document_class against PyDict_Type by identity, matching main.