Support pandas 3.0#1745
Open
filippsatverily wants to merge 15 commits into
Open
Conversation
Moves dependency constraints to pyproject.toml. Makes requirements.txt a lockfile.
Fixes an incompatibility caused by click 8.3.0, which passes the default value as-is.
Fixes an incompatibility caused by pyreadstat 1.2.9, which changed original_variable_type from 'NULL' to None
Works around an behavior change in jsonpath-ng 1.8.0 where Child.str gets wrapped in parenthesis.
Fixes tokenization errors when using dask 2024.8.1+. Starting with this version, dask enforces that tokens remain stable across pickle round-trips (dask/dask#11320). Capturing self in a lambda fails this check because instance objects can have non-deterministic pickle representations. Since calculate_variable_value_length is already a static method, replacing self with the class name is enough to remove the capture.
Dask 2025.4.0 optimizes multiple DataFrames together, which exposes division mismatches and causes dask to throw an error. This change removes a source of repartitioning, preserving the divisions when assigning a pandas series to a dask dataframe
Fixes a unit test to support pandas 2.2.0+. The pandas release fixes an sorting bug with pandas-dev/pandas#54611. This commit changes the expected results accordingly.
Collaborator
|
@filippsatverily Could you please resolve the conflicts on the branch so we can move towards validation and merge? |
SFJohnson24
reviewed
Jun 10, 2026
| Pympler==1.1 | ||
| pyreadstat==1.2.7 | ||
| python-dotenv==1.0.0 | ||
| pytz==2026.2 |
Collaborator
There was a problem hiding this comment.
@filippsatverily what this dependency declaration intentional?
Collaborator
|
@filippsatverily we have merged your other PR with some tweaks--we are now using a pyproject.toml for the dependency installation. I suspect this PR will need to be reworked given the changes--I have thus moved this back to In-Progress. I am happy to rereview once this is ready |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades cdisc-rules-engine to support pandas 3.0. Stacked on top of #1713 (relax dependency constraints) — please merge that first.
pandas <3.0upper bound and addpytz(no longer a pandas transitive dep)applymap()withmap()(removed in pandas 3.0)inplace=Truemutation patterns (pandas 3.0 Copy-on-Write)StringDtypein comparison operatorsDaskDataset.__setitem__dd.DataFrametype annotation inparquet_readermethod=anddowncast=kwargs removed in pandas 3.0.apply(set)path inDistinctoperation