polars 2.0 changed three things specsolve leaned on:
- A select of no columns is a frame of height 0, so every reader of a
declaration with no dimensions came back empty, and an expression
reading its dual read 0.0. Labelled.valued attaches the share before it
projects.
- Cost-based join reordering put a constraint's rows between a shift's two
joins, keyed on one dimension of the two. Every collect now goes through
relational.collect.collected, with join_order off.
- The streaming cross join picks its buffered side from the table sizes,
so the coordinate product lost label order and had to be sorted. Each
cross join now maintains left-then-right order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EpzN6XKNByxsfi3GhX3v3D
Note
The following content was generated by AI.
specsolve now requires
polars>=2.0, and lifts the<2cap from #1852. On polars 2.0.0 it reads a scalar declaration correctly again, and it builds every ladder rung in the memory and time it took on 1.44.2. Three polars 2 changes caused the breakage.What broke on polars 2.0, and the fix for each
select()with no columns gives a frame of height 0. Every reader of a declaration with no dimensions didselect(*dims).with_columns(share). Soprimal,dualandactivitycame back empty, andevaluate('dual(row)')read 0.0 where the answer is 2.0. This is the dual failure in fix(deps): specsolve installs a polars older than 2.0, on which a row's dual can come back empty and a model can take nine times the memory #1852. The fix isLabelled.valued, which attaches the share first and then projects.storage. polars 2 adds cost-based join reordering (QueryOptFlags.join_order). On the cyclicshift, it moves the constraint rows' join between the shift's two joins, keyed onstorealone. That intermediate holds every pair of snapshots per store. The same thing reproduces in plain polars with no specsolve involved: 20,000 result rows peak at 557 MB with reordering and 66 MB without. specsolve orders its joins by hand, so every collect now goes throughrelational.collect.collected, which runs withjoin_order=False. An architecture test refuses a direct.collect()/collect_all()anywhere else insrc/.transport. The streaming cross join now picks which side it buffers from the table sizes (perf: Use stats to decide cross join buffering side pola-rs/polars#29270). The coordinate product therefore lost the label order thatScope.productrelied on, and each label frame had to be sorted (7M rows: 0.36 s). Each cross join now passesmaintain_order='left_right'and folds forward, so the order is guaranteed rather than incidental.The floor.
join_orderdoes not exist in polars 1.x, so the floor moves to 2.0. A version branch was the alternative.floorsnow pinspolars==2.0. The pypsa-parity line moves topolars>=2.0; pypsa 1.3.0 resolves with polars 2.0.0.What polars 2 gives us. The engine changes are mostly free speed. With the three fixes,
dispatch/lbuilds 12% faster andtransport/l5% faster than on 1.44.2, with lower peaks. Out-of-core spilling is on by default in polars 2, and no code here needs it. The new knobs (POLARS_OOC_*, hot-table size, runtime join filters, engine affinity) did not movestorage/sat all; onlyjoin_orderdid.Benchmarks: build seconds (minimum) / peak RSS (median), 3 rounds, fresh process per cell
Method:
bench.arms.specsolve.build_and_emit('highs', …)(build and HiGHS load, no solve) in a fresh process per cell, with configurations alternated cell by cell on an idle 4-core machine.mainis79258b1.main-2.0.0ismainwith polars swapped (installed--no-depspast the cap). A watchdog kills a process above 10 GB RSS.storage/m(+14%) andstorage/l(+4%) are the cells where this PR builds slower than 1.44.2. An earlier run of the same cells gave 0.297 s and 2.533 s against 0.307 s and 2.615 s, so I read both as noise. This is not thepixi run ladderharness and not CodSpeed.xl/2xlwere not run.Mutation table: hand mutations, full suite with
-xEach mutation is a substitution rather than a deletion, so it was taken by hand: committed tree,
git checkout --restore,__pycache__dropped on both sides, tree checked clean afterwards.labels.pyvalued: select the dims before attaching the sharetest_a_declaration_with_no_dimensions_reads_back_its_one_value[primal]collect.py:QueryOptFlags(), so join reordering is ontest_a_join_runs_in_the_order_it_was_writtenscope.pyproduct: nomaintain_ordertest_a_coordinate_product_arrives_in_label_orderassumptions.py: one bare.collect()test_every_frame_is_collected_through_one_functionThe scalar-reader test was written first as a strict
xfail. It failed on polars 2.0.0 ([] == [{'value': 10.0}]) and XPASSed on 1.44.2. The marker came off with the fix.Gates, and what was not done
ruff check .,ruff format --check .andpyreflyare clean. The suite on polars 2.0.0 gives 5002 passed, 564 skipped, 1 xfailed; before the fixes it gave the two failures of fix(deps): specsolve installs a polars older than 2.0, on which a row's dual can come back empty and a model can take nine times the memory #1852.pixi run -e floors test-floorson polars 2.0.0 gives 3271 passed.pixi run -e docs docs-buildis strict and clean.uv.lock. Relocked withuv lock(uv 0.11.32). Besides polars 2.0.0, it drops platform markers on the textual / rich / jinja2 dependents; I did not hand-edit them out.relational/collect.pyrow indocs/about/architecture.mdand the "Collect engine" entry indocs/reference/internals.mdnow say every collect goes throughcollected, with join reordering off.tests/test_collect.py::_shift_shaped.sink_parquetcalls do not go throughcollected. Their frames are results with no join chain.2 * slack) raisesColumnNotFoundError: __unit__, and so doessave()with such an expression declared. It is left for its own PR.join_orderflag needs polars 2.🤖 Generated with Claude Code
https://claude.ai/code/session_01EpzN6XKNByxsfi3GhX3v3D
Generated by Claude Code