Skip to content

refactor(api): specsolve no longer builds a linopy model, and its linopy extra and LaneError are gone - #1755

Merged
FBumann merged 9 commits into
claude/confident-goodall-2iuw8nfrom
claude/linopy-oracle-in-tests
Sep 25, 2026
Merged

FBumann merged 9 commits into
claude/confident-goodall-2iuw8nfrom
claude/linopy-oracle-in-tests

Conversation

@FBumann

@FBumann FBumann commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Prompt: We should work on removing linopy entirely from lpspec. But keep it as a testing oracle!

Later, after CI failed: "Drop the xarray extra for now! Delete LaneError".

Note

The following content was generated by AI.

The linopy lane moves to tests/linopy_lane/ and stays the differential oracle. Nothing under src/ imports linopy. The package has no extra for linopy, pandas or xarray, and LaneError is deleted. No direct reference is left in the package metadata.

What changed, coverage moved, guards, gates, not done

The package

  • specsolve.linopy, check(spec, sink='linopy'), lanes.LANES and lane_cannot_build_message are gone.
  • The linopy extra is gone, and no xarray extra replaces it. to_pandas and to_dataarray import pandas and xarray lazily as before. When one is missing, the message says pip install pandas or pip install xarray. to_dataarray reads through pandas first, so an install with neither package is told pandas.
  • allow-direct-references is gone: mathspec comes from PyPI (build(deps): specsolve depends on the mathspec 0.1.0 release from PyPI rather than a git tag #1754), and linopy is in a dependency group, which is not package metadata.
  • LaneError is deleted. The engine's one construct that the language accepts and the engine cannot build (a constant beside a term, under an operator along a dimension, a constant summed beside a term is accepted by check and the linopy lane, refused by the engine #1137) now raises SpecsolveError. Its message names only the rewrite. The public surface drops from 28 names to 27, and architecture.md now says so.

The oracle

  • tests/linopy_lane/ holds the eight modules, with their own CAPABILITIES table and a test-side OracleCannotBuildError(SpecsolveError).
  • The dev group carries linopy @ git+…@master, xarray and pandas.
  • pyrefly checks the oracle under strict, from project-includes, with search-path = ["src", "."].
  • The PyPSA parity job takes linopy's pin from pyproject.toml with grep, so the pin has one home. Without it, pypsa pulled released linopy 0.9.1 into uv run's --with layer, and the oracle's import failed.

Coverage that moved

was now
test_the_linopy_lane_stays_two_verbs deleted: the oracle has no public surface
the _in_linopy_lane exemption in the two import fences gone: the fences cover every module under src/
check(SPEC, sink='linopy') in test_quadratic_constraint gone with the sink. The oracle's own refusal is still asserted
pytest.raises(LaneError) for the engine's gap (test_relational, test_compiler) SpecsolveError, and still asserted not to be a LanguageError
LaneError from the oracle (test_corpus_parity xfail, test_conditioned_relations, test_linopy_lane, test_quadratic_constraint) OracleCannotBuildError
the expression sweep's except LaneError except OracleCannotBuildError, plus a SpecsolveError whose message says specsolve cannot build. Any other error is re-raised
bridge tests expecting pip install "specsolve[linopy]" expect the missing package by name, and pandas where neither is installed

Guards, probed by hand after the commit: each mutation was made, the fence tests were run, the tree was restored with git checkout --, and __pycache__ was dropped on both sides.

mutation result
import linopy at module level in src/specsolve/api.py caught (2 fence tests)
import linopy inside check() caught (the lazy-import fence)

Gates. pixi is not installable here, so the gates ran in uv environments:

  • ruff check . and ruff format --check . pass.
  • pyrefly check: 0 errors.
  • Full environment (dev + docs groups, gurobi and xpress extras): pytest -n auto, 4139 passed, 250 skipped, 1 xfailed, 0 failed.
  • A bare install (the built package, pytest, pytest-xdist and pyyaml; no linopy, xarray or pandas) on f65ad79: 2811 passed, 216 skipped. This is the run that reproduced the floors failure on the bridge test before the fix.
  • zensical build --strict: "No issues found".
  • The PyPSA parity job's command, run locally against mathspec v0.1.0: every rung matches, and the committed certificate is unchanged. CI's parity is green too.
  • Not run: test-bench, and test-floors with its exact pinned floors.

Branch history: this branch merges #1751 after #1751 took in main (#1754, #1756, alpha.359). Another session had pushed the same merge, but its tree kept the [xarray] extra and the failing parity line. It is recorded with -s ours, so its content is superseded; nothing was force-pushed.

Not done: the internal docstrings that say "both lanes" (sources.py, assumptions.py, expressions.py) still hold, because the oracle enters through the same door.

Stacked on #1751, which adds ## Upcoming version.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw

…das and xarray bridges move to an xarray extra

The linopy lane moves from src/specsolve/linopy to tests/linopy_lane,
where it stays the differential-test oracle. Nothing under src/ imports
linopy now, and tests/test_architecture.py holds that for every module.

- The `linopy` extra is gone. linopy comes from the `dev` group, still
  from master; a dependency group is not package metadata, so PyPI never
  sees the reference. The `xarray` extra carries xarray and pandas for
  Result.to_pandas and to_dataarray.
- check(spec, sink='linopy') is gone, with lanes.LANES and
  lane_cannot_build_message. The oracle declares its own capabilities.
- The engine's LaneError for a constant beside a term names the rewrite
  only, since no other lane ships.
- pyrefly checks tests/linopy_lane as before, from the repository root.
- The PyPSA parity harness imports the oracle from tests/, and the
  workflow installs the `xarray` extra; `uv run` brings the dev group.
- The docs say linopy is the oracle and not a lane a caller picks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw
@FBumann FBumann mentioned this pull request Sep 25, 2026
19 of 33 tasks
@codspeed

codspeed Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 5.66%

⚡ 1 improved benchmark
✅ 23 untouched benchmarks
⏩ 82 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_window[commitment-s-specsolve-highs] 1.6 MB 1.6 MB +5.66%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/linopy-oracle-in-tests (f65ad79) with claude/confident-goodall-2iuw8n (bca5a67)

Open in CodSpeed

Footnotes

  1. 82 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@read-the-docs-community

read-the-docs-community Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

… into claude/linopy-oracle-in-tests

# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/pypsa-parity.yml
#	RELEASING.md
#	pyproject.toml
#	uv.lock
pandas and xarray are no extra of the package: `to_pandas` and
`to_dataarray` name the package to install when it is missing, and the
dev group carries both for the suite and the oracle. With no direct
reference left in the package metadata, allow-direct-references goes.

LaneError is deleted. The engine's one construct the language accepts and
it cannot build raises SpecsolveError, naming the rewrite. The oracle
raises its own OracleCannotBuildError, in tests/linopy_lane.

The quadratic-constraint test no longer asks check(sink='linopy'), and
the PyPSA parity job takes the dev group's linopy pin from pyproject.toml,
since pypsa otherwise pulls released linopy into the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw
…cle-in-tests

The base trimmed the configuration comments and moved mathspec to its
PyPI release. With the linopy extra gone as well, no direct reference
stands in the package metadata, so allow-direct-references goes: the
wheel builds without it. uv.lock was regenerated with `uv lock`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LKoYyV812fByVyk4ijZfQ2
@FBumann FBumann changed the title refactor(api): specsolve no longer builds a linopy model, and the pandas and xarray bridges move to an xarray extra refactor(api): specsolve no longer builds a linopy model, and its linopy extra and LaneError are gone Sep 25, 2026
@FBumann
FBumann added this pull request to stack #1762 September 25, 2026 11:41
…ld to naming pandas

`to_dataarray` reads through pandas first, so the floors job, which has
neither package, reports pandas when the test hides xarray.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw
@FBumann
FBumann merged commit a4be39a into main Sep 25, 2026
15 checks passed
@FBumann
FBumann deleted the claude/linopy-oracle-in-tests branch September 25, 2026 11:47
FBumann added a commit that referenced this pull request Sep 25, 2026
> **Prompt:** And prepare a todo list for me the same way. Also start
with 0.1.0rc1

> [!NOTE]
> The following content was generated by AI.

This is the release PR for `0.1.0rc1`, the release candidate for the
first PyPI upload. It is stacked on #1751. It stays a draft until PyPI
can take the package. One direct reference is left, the `linopy` extra,
and #1755 removes it (step 1).

## The first PyPI release, step by step

### 1. What blocks the upload

PyPI refuses a distribution that names a git URL in its metadata.

- [x] mathspec is on PyPI, and `pyproject.toml` depends on
`mathspec>=0.1.0` (#1754). The parity job reads the corpus tag from that
floor.
- [x] Merge #1755. It takes linopy out of the package and keeps it as
the test oracle: the `linopy @ git+…@master` reference moves to the
`dev` group, which PyPI never sees. It also deletes
`allow-direct-references`, so hatchling refuses a direct reference at
build time, before PyPI would.

### 2. One-time setup

**PyPI**
- [x] On pypi.org, go to Account → Publishing → "Add a new pending
publisher" → GitHub, and fill in:
  - PyPI project name: `specsolve`
  - Owner: `fluxopt`
  - Repository name: `specsolve`
  - Workflow name: `release.yaml`
  - Environment name: `pypi`

**GitHub**, under fluxopt/specsolve Settings
- [x] Environments: create or edit `pypi`. Under "Deployment branches
and tags", choose "Selected branches and tags" and add `main`. Add the
people who may approve a release as required reviewers.
- [x] After #1751 merges: Rules → the `main` ruleset. Require `ci`,
`Conventional commit subject` and `Changelog line`. A required check
must run on `main` once before you can require it.
- [x] Issues → Labels: create `no changelog`.

**Clean-up that the release does not need**
- [ ] Delete the repository variables `AUTO_RELEASE` and
`PUBLISH_TO_PYPI` if they exist. Turn off "Allow auto-merge" unless you
want it for other PRs.
- [x] Remove `specsolve` from the release app's (`fluxopt-release-bot`)
repository access. If no other repository uses the app, uninstall it and
delete the secrets `APP_CLIENT_ID` and `APP_PRIVATE_KEY`.
- [x] Delete the branches
`release-please--branches--main--components--farkas` and
`release-please--branches--main--components--linopy-yaml`.
- [x] Close #1675 and #1676. #1751 replaces both.
- [ ] Open PRs owe a changelog line after #1751 merges: #1737, #1527,
#1516 and #1541. #1541 (drop the linopy lane for linopy's own
`from_spec`) conflicts with #1755; close it or rebase it.

### 3. Before this PR merges: what the PyPI page shows

- [x] `[project.urls]` has only `repository`. Add `Documentation`,
`Issues` and `Changelog`, as energy-models/mathspec#707 did.
- [x] The README has two relative links, `CONTRIBUTING.md` and
`docs/about/prior-art.md`. On PyPI they do not resolve. Make them
absolute.
- [x] `src/specsolve/relational/parquet.py` has two error messages that
say "while the package is on 0.0.1aN". `tests/test_archive.py` asserts
the same. Reword them to "before 1.0".
- [x] RELEASING.md, "Relaxed while in early development": it says to
tighten CI before 0.1.0, starting with a Python matrix behind the gate
job. Do that, or change the sentence.

### 4. This PR

- [x] Merge #1751, then #1755. Then this PR's base must be `main`.
GitHub retargets it when #1751's branch is deleted; if the branch stays,
retarget by hand.
- [x] Mark the PR ready.
- [x] Set the date in `## 0.1.0rc1 (2026-09-25)` to the day you merge,
if that is not the 25th.
- [x] Edit the notes under the heading. Add the lines of the PRs merged
since. The text becomes the GitHub release notes, as written.
- [x] Check that `ci` is green. Its step "Check the changelog headings"
prints `releases 0.1.0rc1 on merge`.
- [x] Review, then squash-merge.

### 5. After the merge

- [ ] Actions → **Release**, the run for the merge commit:
- [ ] `Tag the version the changelog names` created the tag `v0.1.0rc1`
and the GitHub release `0.1.0rc1`, marked as a pre-release.
  - [ ] `Build` passed, and its check printed `specsolve-0.1.0rc1`.
- [ ] `Publish to PyPI` waits for approval: "Review deployments" →
`pypi` → Approve.
- [ ] https://pypi.org/project/specsolve/0.1.0rc1/ exists. The README
renders, and the project links work.
- [ ] Smoke test in a fresh environment:
  ```bash
  python -m venv /tmp/ss && /tmp/ss/bin/pip install specsolve==0.1.0rc1
/tmp/ss/bin/python -c "import specsolve, mathspec;
print(specsolve.__version__, mathspec.__version__)" # 0.1.0rc1 0.1.0
/tmp/ss/bin/pip install specsolve # finds no stable version until 0.1.0
exists
  ```
- [ ] If a job failed, fix the cause, then use "Re-run failed jobs" on
that run. If `0.1.0rc1` reached PyPI broken, release `0.1.0rc2`.

### 6. Then 0.1.0

- [ ] Open the next release PR, titled `chore: release 0.1.0`. It
renames the next `## Upcoming version` to `## 0.1.0 (date)`, with notes
that say this is the first release on PyPI. Merge it and approve the
upload as above.
- [ ] Smoke test `pip install specsolve` without a version. It should
install `0.1.0`.

### 7. Follow-ups

- [ ] 83 links in the tree point at `math-spec.readthedocs.io`. The
redirect keeps them working. Point them at `mathspec.readthedocs.io`.
- [ ] Optional: a conda-forge recipe.
- [ ] Patch releases to an older version line. No path exists yet;
mathspec tracks the same gap in energy-models/mathspec#710.

<details><summary>What this PR changes, what merging does,
gates</summary>

**The changes:**
- `CHANGELOG.md`: an empty `## Upcoming version`, then `## 0.1.0rc1
(2026-09-25)`, with a paragraph and #1748's line.
- `AGENTS.md`: "The project is `0.0.1aN` and holds no compatibility
promise" becomes "The project holds no compatibility promise before
1.0". One sentence is added: a release that breaks a model file or an
import raises the minor version, and its notes name the break.
- `CONTRIBUTING.md`: the same rule, in the same words.

**Why a release candidate:** it goes through the same tag, GitHub
release, build, approval and trusted-publishing upload as `0.1.0`. The
only addition is `--prerelease` on the GitHub release. pip skips it
unless someone asks for it, so a broken first upload reaches no user.
The version sorts after `0.0.1a358` and before `0.1.0`.

**What merging does:** `release.yaml` finds `0.1.0rc1` with no tag. It
creates `v0.1.0rc1` and a GitHub pre-release, builds from the tag,
checks that the wheel and the sdist are `0.1.0rc1`, and waits for
approval on `pypi` before the upload.

**Gates:** `python -m tools.changelog check` prints "releases 0.1.0rc1
on merge", and `notes 0.1.0rc1` prints the section above. The suite on
#1751's commit is in that PR. This commit changes only prose and the
changelog.

**Not verified:** the name `specsolve` on PyPI. The PyPI API cannot be
reached from this container, so check that the name is free when you add
the pending publisher.

The type is `chore`, which owes no changelog line.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016Pv7LzSgzt7Yn2K3ioyJXw

Co-authored-by: Claude <noreply@anthropic.com>
FBumann added a commit that referenced this pull request Oct 6, 2026
)

> **Prompt:** The benchmark ran. Can you update our published benchmark
with a PR?

> [!NOTE]
> The following content was generated by AI.

The benchmark pages now show run
[37298997947](https://github.com/fluxopt/specsolve/actions/runs/37298997947),
measured on `aa902393` (0.6.0). It covers the same five cases as the
page it replaces. The page's sentences about the run are recomputed from
the new files, and `bench/reproduce.py` is pinned to that run.

<details><summary>What changed</summary>

- **Results.** Five cases: `highs` dispatch and fleet, and `gurobi`
dispatch, transport and fleet. The other three cases were stopped by the
memory watchdog, as on the previous run (#1498).
- **Where the files come from.** The network policy of the session
blocks the artifact host, so the files come from the eight per-case
artifacts, uploaded by hand. Every results file is byte-identical in
each artifact that holds it. `results-gurobi-fleet` holds all ten files.
- **`casualties.json` is not committed.** It was never committed, and
`bench/results.py` skips it. Its three entries are in the page text.
- **Generated content.** The tables come from `pixi run -e bench report`
and the chart data from `pixi run -e bench plot`. Their output,
fingerprint included, is the same as the run's own `report` and `plot`
steps.
- **Hand-written sentences, recomputed.** Before I used each method on
the new files, I checked that it reproduces the old sentence exactly
from `main`'s results.
- **Mean against median:** 14 cells are above 1.10x, against 4 before.
The worst is 3.41x, against 1.21x before.
- **The cell the median flips:** now `transport/w1` on `gurobi` against
gurobipy-loop, in specsolve's favour. Before, it was `dispatch/s`
against linopy, against specsolve. With it goes the #1288 sentence:
specsolve's rounds on that cell are now 88 to 94 ms.
  - **The three casualties:** 24.6, 26.3 and 23.9 GB.
- **Removed:** the sentence on why `storage` survived on `gurobi`
earlier. The previous run already contradicted it.
- **`bench/reproduce.py`.**
`test_the_lock_installs_what_the_published_numbers_were_taken_on` failed
on the new files, because the lock installed `lpspec` at `8d27e88b`.
Changes:
- The script now installs `specsolve[gurobi]` at `aa90239393`, and
linopy from `master`, because the `linopy` extra is gone (#1755).
  - The lock moves pandas to 3.0.6, as measured.
  - The docstring line about the `lpspec 0.0.1a61` numbers is removed.
- **`main` is merged in, with #1852's polars cap.** The cap was ported
here first, and the merge makes it identical to `main`. The only
conflict was in `CHANGELOG.md`, which now keeps both lines.

</details>

<details><summary>Found, and not fixed here</summary>

- **A cold first round.** In 13 cells, the first of the nine rounds of
an `xs` cell on specsolve, linopy or pyomo is 1.9 to 22.6 times the
median of the other eight. Pyomo `dispatch/xs` on `highs` takes 1524 ms,
then about 67 ms. `main`'s results from 14 September have no such round.
Published medians do not move, because that round is always the largest
of nine. The q3 of a band may move a little. `_rounds` had
`warmup_rounds=0` at the previous run too, so the cause is elsewhere in
what changed after `8d27e88b`. I did not look further.
- **The 84-rounds sentence.** `benchmarks-scaling.html` says "a quick
cell here took 84 rounds and a slow one 9". That sentence is
hand-written and was already false, because rounds are pinned at 9.
- **linopy's commit.** The run does not record which linopy commit it
measured, so the lock pins today's `master` (`f665a260`). The guard
checks only the versions the run records.

</details>

<details><summary>Gates</summary>

- `pixi run check`, before the `main` merge, on polars 1.44.2: 4996
passed, 564 skipped, 1 xfailed. After the merge: `tools.changelog
check`, `tests/test_changelog.py` and `tests/test_tooling_pins.py` pass.
- `pytest bench/test_harness.py`: 262 passed, 16 skipped. This is
`test-bench`, including
`test_the_report_renders_from_the_committed_results` and the lock guard.
- `pixi run docs-build`, the command Read the Docs runs: clean.
- `uv lock --script bench/reproduce.py --check`: clean. `pixi run lint`
and `format-check`: clean.
- **Not run:** `uv run --locked bench/reproduce.py` itself.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018nKGXp8uKnMeJACjFYwbdY

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants