Skip to content

Fix #1116: Drop the numpy constraints and make pyarrow a single range - #1117

Open
javihern98 wants to merge 3 commits into
1.9.Xfrom
cr-1116
Open

Fix #1116: Drop the numpy constraints and make pyarrow a single range#1117
javihern98 wants to merge 3 commits into
1.9.Xfrom
cr-1116

Conversation

@javihern98

@javihern98 javihern98 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1116.

Three commits. The first replaces six numpy/pyarrow constraint lines with three different Python boundaries (3.10, 3.11, 3.14) by a single one; the second regenerates poetry.lock from scratch; the third drops the now-stale numpy line from the JupyterLite lock patch.

-"pyarrow>=14.0,<20.0; python_version < '3.10'",
-"pyarrow>=14.0,<25.0; python_version >= '3.10' and python_version < '3.14'",
-"pyarrow>=22.0,<25.0; python_version >= '3.14'",
-"numpy>=2.0.2,<2.1; python_version < '3.10'",
-"numpy>=2.2.0,<2.5; python_version == '3.10'",
-"numpy>=2.4.3,<2.5; python_version >= '3.11'",
+"pyarrow>=14.0,<26.0",
  • pyarrow needs no split. 24.0.0 and 25.0.1 cover Python 3.10–3.14 with wheels, and requires-python handles 3.9 (→ 21.0.0). Poetry already locks a single pyarrow entry. The cap moves from <25.0 to <26.0 so the 25.x line (25.0.1, Python >= 3.10) is reachable.
  • numpy is not our dependency. Nothing under src/ or tests/ imports it: it is pandas' dependency, and pandas already pins what it needs per Python (>=1.26.0 below 3.14, >=2.3.3 from 3.14). Declaring it bought nothing — fresh installs resolve identically without it — and the marker split never protected the lock from the Dependabot collapse it was meant to guard against: Dependabot rewrites poetry.lock in violation of the pyproject markers and poetry install trusts the lock (PR Bump coverage from 7.14.3 to 7.15.0 #876, superseded by Update coverage to 7.15.0 (supersedes #876) #885). Reviewing Dependabot lock diffs remains the actual protection.
  • Dev group: the python_version < '3.10' pandas-stubs line goes (the Poetry environment starts at 3.10, so it never resolved) and the remaining line loses its redundant >= '3.10' marker.

Resolution per interpreter

Verified with uv pip compile pyproject.toml --python-version <X>:

Python numpy (via pandas) pyarrow pandas
3.9 2.0.2 21.0.0 2.3.3
3.10 2.2.6 25.0.1 2.3.3
3.11 2.4.6 25.0.1 3.0.5
3.12 – 3.14 2.5.3 25.0.1 3.0.5

What changes for users — everything else resolves identically:

  • Python 3.9: pyarrow ceiling 19.0.1 → 21.0.0 (20.x/21.x ship cp39 wheels).
  • Python 3.10–3.14: pyarrow ceiling 24.0.0 → 25.0.1.
  • vtlengine no longer imposes a numpy floor (previously 2.0.2 / 2.2.0 / 2.4.3 by Python); pandas' own floors apply. The Ubuntu 24.04 job already runs the APT numpy 1.26.4 with pandas 2.1.4.
  • numpy 2.5.x becomes reachable on Python 3.12+ (it was capped at <2.5).
  • Python 3.14: no explicit pyarrow>=22.0 floor any more; pip still resolves the highest in range.

Lock

The first commit only moves the lock's content-hash. The second is a clean poetry lock --regenerate against the committed constraints, so transitive pins stop drifting (Dependabot only touches the packages it bumps); poetry check --lock passes. Every locked version change versus 1.9.X — 74 packages before and after, none added or removed; the numpy row follows from dropping the declared dependency (Poetry then partitions it the way pip resolves it), everything else from the regeneration:

Package 1.9.X This PR
anyio 4.14.2 4.15.1
ast-serialize 0.8.0 0.11.0
coverage 7.15.4 7.16.0
duckdb 1.4.5 1.5.5
lxml 6.1.2 6.1.3
pyarrow 24.0.0 25.0.1
numpy 2.2.6 (python_version == "3.10")
2.4.6 (python_version >= "3.11")
2.2.6 (python_version == "3.10")
2.4.6 (python_version == "3.11")
2.5.3 (python_version >= "3.12")
ruff 0.16.5 0.16.6
typing-extensions 4.16.0 (main group: python_version < "3.13") 4.16.0 (main group: python_version <= "3.14") — marker only, anyio 4.15.1 now needs it up to 3.14

As a result the Testing matrix exercises numpy 2.5.3 on 3.12–3.14, pyarrow 25.0.1 on every 3.10+ job and duckdb 1.5.5 everywhere. The Ubuntu 24.04 workflow keeps its hand-maintained pins: they only need to stay inside the pyproject ranges, and pyarrow==24.0.0 still does.

Checklist

  • Code quality checks pass (ruff format, ruff check, mypy — mypy run against numpy 2.5.3 and pyarrow 25.0.1)
  • Tests pass (pytest) — full suite locally on Python 3.12 against the regenerated lock (pyarrow 25.0.1, duckdb 1.5.5, numpy 2.5.3, pandas 3.0.5): pandas backend 5138 passed / 639 skipped, duckdb backend 6085 passed / 20 skipped; CI on 880df6b green, 22/22 checks
  • Documentation updated (if applicable) — n/a

Impact / Risk

  • Breaking changes? None in API or behavior; only the constraint edges above move. numpy 2.5.x on 3.12+ is new and covered by the local run and the 3.12–3.14 CI jobs; the numpy 2.5 removals (row_stack, 2-D cross, recfrom*, overflow-raising where/datetime64 arithmetic, tuple meshgrid, complex linalg.eig) have no call sites here since vtlengine does not import numpy. pyarrow 25.0.1 is likewise new for 3.10+ and covered locally and by every 3.10+ CI job. The cp39 wheel smoke test at release resolves pyarrow 21.0.0 instead of 19.0.1.
  • Data/SDMX compatibility concerns? None.
  • Notes for release/changelog? Dependency-metadata cleanup: numpy is no longer declared, pyarrow is a single range up to <26.0; worth a line about the pyarrow ceilings (21.0.0 on 3.9, 25.0.1 on 3.10+).

Notes

  • .github/workflows/ubuntu_test_24_04.yml is unaffected: pyarrow==24.0.0 with --no-dependencies stays inside <26.0, and its APT numpy 1.26.4 + pandas 2.1.4 combination was always exercised there.
  • Wasm wheel: numpy comes from the Pyodide distribution through pandas (2.4.3 / 2.4.6), pyarrow 22.0.0 is in range; the micropip check in release.yml covers it.
  • docs/jupyterlite/patch_lock.py mirrors vtlengine's declared dependencies in the patched Pyodide lockfile, so its numpy entry for vtlengine goes too; nothing changes at load time because the distribution's pandas and pyarrow entries both depend on numpy. pysdmx keeps its numpy entry (pysdmx declares numpy under its data extra).
  • main receives this through the periodic 1.9.X merge.

@javihern98
javihern98 marked this pull request as ready for review September 9, 2026 11:50
@javihern98
javihern98 requested a review from a team September 9, 2026 11:50
@javihern98
javihern98 enabled auto-merge September 9, 2026 11:51
@javihern98 javihern98 changed the title Fix #1116: Simplify numpy and pyarrow constraints to a single Python 3.11 split Fix #1116: Drop the numpy constraints and make pyarrow a single range Sep 9, 2026
pyarrow becomes one unmarked range (>=14.0,<26.0): 24.0.0 covers Python
3.10-3.14 with wheels and requires-python handles 3.9 (-> 21.0.0), so the
three Python-bounded lines bought nothing. The cap moves from <25.0 to
<26.0 so the 25.x line (25.0.1, Python >= 3.10) is reachable; the
Ubuntu 24.04 job's hand-maintained pyarrow==24.0.0 stays inside the range.

numpy leaves [project].dependencies altogether: nothing under src/ or
tests/ imports it, so it is pandas' dependency, not ours, and pandas
already pins what it needs per Python. Fresh installs resolve exactly as
before (3.9 -> 2.0.2, 3.10 -> 2.2.6, 3.11 -> 2.4.6, 3.12+ -> 2.5.3); the
only effects are that pip no longer forces numpy>=2.0.2 onto older stacks
(the Ubuntu 24.04 job already runs the APT numpy 1.26.4 with pandas 2.1.4)
and that numpy 2.5.x becomes reachable on 3.12+. The old marker split
never protected the lock from Dependabot's collapse either: Dependabot
rewrites poetry.lock in violation of the pyproject markers and
`poetry install` trusts the lock (PR #876, superseded by #885).

The dev group loses the python_version < '3.10' pandas-stubs line (the
Poetry environment starts at 3.10, so it never resolved) and the
redundant marker on the remaining one.

poetry.lock only changes its content-hash here; the next commit
regenerates it from scratch, after which numpy is partitioned the way pip
resolves it -- 2.2.6 (3.10), 2.4.6 (3.11), 2.5.3 (>=3.12).
A clean `poetry lock --regenerate` against the committed constraints, so
transitive pins stop drifting (Dependabot only touches the packages it
bumps): anyio 4.14.2 -> 4.15.1, ast-serialize 0.8.0 -> 0.11.0,
duckdb 1.4.5 -> 1.5.5, lxml 6.1.2 -> 6.1.3, pyarrow 24.0.0 -> 25.0.1,
coverage 7.15.4 -> 7.16.0, ruff 0.16.5 -> 0.16.6. numpy, now transitive,
is partitioned the way pip resolves it: 2.2.6 (3.10), 2.4.6 (3.11),
2.5.3 (>=3.12).

The Ubuntu 24.04 workflow keeps its hand-maintained pins: they only need
to stay inside the pyproject ranges, and pyarrow==24.0.0 still does.
The patched Pyodide lockfile mirrors vtlengine's declared dependencies,
which no longer include numpy. Nothing changes at runtime: the
distribution's pandas and pyarrow entries both depend on numpy, so it is
loaded with them. pysdmx keeps its numpy entry because pysdmx does declare
numpy (under its data extra).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simplify the numpy and pyarrow constraints: drop numpy, single range for pyarrow

1 participant