Skip to content

feat(api): a solve or a sweep asked for outputs={'variable_basis', 'constraint_basis'} reads the basis it ended on, in the same words on every solver - #1876

Merged
FBumann merged 6 commits into
claude/festive-carson-8x46sq-slackfrom
claude/festive-carson-8x46sq-basis
Oct 8, 2026
Merged

FBumann merged 6 commits into
claude/festive-carson-8x46sq-slackfrom
claude/festive-carson-8x46sq-basis

Conversation

@FBumann

@FBumann FBumann commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Prompt: Add stacked PRs about the new things we discussed

The discussion it refers to began with: "What about basis, warmstaer initial solution etc? What kind of things would be reasonable to add to boost sovler performance?"

Note

The following content was generated by AI.

outputs={'variable_basis', 'constraint_basis'} makes a solve or a sweep carry the basis status it ended on. The words are the same on HiGHS, Gurobi and Xpress. This is #1869 part 1, and the warm start stacked on it reads it.

What it is
  • One vocabulary (BASIS_STATUSES), read back as an Enum: basic, at_lower, at_upper, fixed, superbasic. A row's bound is its right-hand side, so a binding <= row is at_upper, a binding >= row at_lower, and a binding == row fixed. A column at equal bounds is fixed, whichever bound its solver names.
  • Two outputs, not one, because a constraint may share a variable's name (mathspec's flat namespace leaves constraints out). Each output holds one frame per declaration, as every other output does.
  • How each member reads its basis: _basis() maps the solver's own codes to the shared ones. It reads only basic or not, and which bound a column is at. Solver.run(handoff, basis=True) then reads each nonbasic status against the model (settled): a row's bound from its sense, fixed from equal bounds. That is also why all three sinks agree, even though Gurobi says only "nonbasic" for a row and Xpress names a row's slack.
  • Read only when asked: the engine passes basis=True only where outputs names one of the two.
  • Where there is no basis — a mixed-integer model, a quadratic constraint, interior point with crossover off — the reader raises NO_BASIS, which names the solver options that give one. A sweep raises it only where some slice solved. Otherwise it gives its "nothing to read" message.
  • An output under feat(api): every answer holds an optional output only when outputs= asks for it, and a sweep can hold the activity too #1865's rule, stacked on feat(api): a solve or a sweep asked for outputs={'slack'} reads how far each constraint is from binding #1868: Result.variable_basis / constraint_basis, the same on Sweep, kind= on the bridges, <output>/<name>.parquet in a save, a spill and an archive, and a catalog row per declaration. No LAYOUT bump.
Verification

Gates: ruff check ., ruff format --check ., pyrefly check, the full suite (pytest -n auto, with HiGHS, Gurobi and Xpress installed), zensical build --strict, test_docstring_links.py and test_docs_math.py.

The local runs above were taken while one test of the base, test_a_solution_is_read_back_in_label_order_without_sorting_for_it, failed there, so they deselect it. The base has fixed it since, it is merged in, and CI runs the whole suite.

tests/test_basis.py covers the following:

  • The vertex of a five-variable, four-row LP, worked out by hand so that every status but superbasic appears, read in the same words on every sink.
  • The Enum dtype.
  • A save that loads back.
  • The refusal when not asked for, per output.
  • No basis after a mixed-integer solve and after interior point without crossover, on each sink.
  • No basis with a quadratic constraint, on Gurobi.
  • A sweep held, on a process pool, spilled and archived.
  • A sweep whose every slice ended on no basis.
  • A basis a member reads short, refused.

Mutation table, from python -m tools.mutate against the full suite (that one test deselected through PYTEST_ADDOPTS), except one row taken by hand with the same three precautions:

guard result
a basis that does not span the model refused (base.py:271-272) caught
a column at equal bounds reads fixed (base.py:274) caught
a nonbasic row reads its bound off its sense (base.py:277) caught
a result that ended on no basis says so (result.py:666-667) caught
a sweep whose slices ended on no basis says so (sweep.py:733-734) caught, on a second run; the first was interrupted by the load of another run on the box and proved nothing
HiGHS: an invalid basis is none (highs.py:244-245) caught
Xpress: no basis after a mixed-integer solve (xpress.py:140-141) caught
Xpress: no basis after barrier without crossover (by hand: the try/except SolverError removed) caught (1 failed)

The tree was clean after each run. One guard is not in the table: run reads a basis only after a run that left a primal. Reading one after a failed run would be harmless and invisible, because the reader refuses on the status first.

Deliberately not done
  • Gurobi's _basis has no guard for a quadratic constraint: Gurobi refuses VBasis there itself, which the new test shows.
  • No fixed read from a solver for a row: every row is single-sided, so its nonbasic status follows from its sense.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1

claude added 4 commits October 7, 2026 07:00
…onstraint_basis'} reads the basis it ended on, in the same words on every solver

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1
…onstraint ends on none

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1
@FBumann
FBumann added this pull request to stack #1874 October 7, 2026 07:18
@read-the-docs-community

read-the-docs-community Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

@codspeed

codspeed Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 67 untouched benchmarks
⏩ 307 skipped benchmarks1


Comparing claude/festive-carson-8x46sq-basis (452789c) with claude/festive-carson-8x46sq-slack (5b54514)

Open in CodSpeed

Footnotes

  1. 307 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…x46sq-basis

Brings main through the stack: polars 2.0, the 0.7.0 release and
load_inputs. The collects this branch added go through collected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014wLrXpeSGDqoBFxyrs8NEz
@FBumann
FBumann merged commit a95747e into main Oct 8, 2026
13 checks passed
@FBumann
FBumann deleted the claude/festive-carson-8x46sq-basis branch October 8, 2026 14:16
FBumann added a commit that referenced this pull request Oct 8, 2026
…hed by coordinate, so a model that gained a cut starts from where it was (#1877)

> **Prompt:** Add stacked PRs about the new things we discussed

The discussion it refers to began with: "What about basis, warmstaer
initial solution etc? What kind of things would be reasonable to add to
boost sovler performance?"

> [!NOTE]
> The following content was generated by AI.

`solve(start=earlier_answer)` starts an LP from the basis an earlier
answer ended on. The basis is matched by coordinate, so a master that
gained a cut starts where it was. On the Benders bench this saved 66–73%
of the simplex iterations. This is #1869 part 3 and closes the gap #382
left.

<details><summary>What it is</summary>

- **Matched by coordinate** (`readback.matched_basis`). A coordinate
both builds hold keeps its status. A column only the new build holds
starts at a bound. A row only the new build holds starts basic, so a
gained cut enters without moving the vertex. A declaration whose dims
changed is new.
- **Made takeable.** `_counted` makes it one basic entry per row: a
surplus leaves from the last basic columns, and a shortfall is made up
by the last nonbasic rows. `settled` (now shared with the read side)
then puts every nonbasic column at a bound the new build has. Each
solver repairs a singular basis itself.
- **Any answer, any solver**: live, from `load_result`, or from an
archive, and read on one solver to warm another, because #1876's
statuses are one vocabulary. The answer must carry both basis outputs.
The refusal names the `outputs=` to solve it with, before the solver
loads.
- **A mixed-integer model is refused**: it has no basis to start from.
The PR stacked on this one makes `start=` on a MIP take values instead.
- **The sink family is simpler.** `WarmStart`, `warm_start()`, and the
refusal of another solver's statuses are gone. A member now implements
`_warm(basis)` in its own statuses and `_start(values)` for an
incumbent, and the base checks spans. No caller used `warm_start()`
outside tests and the bench.

</details>

<details><summary>Measured: bench/warm_payoff.py, the Benders
master</summary>

Method: `python -m bench.warm_payoff xs s m --wall`, HiGHS, at
`ecdefd1`. At every rebuild the master is solved cold, and with `start=`
the previous started answer (`keep='nothing'` both, so each solve has a
fresh solver). Simplex iterations are read off HiGHS and are
deterministic. Wall seconds count the whole `solve` call, including
reading the basis back and matching it.

| generators | rebuilds | iterations cold | iterations started | saved |
wall cold | wall started |
|---|---|---|---|---|---|---|
| 6 | 28 | 112 | 35 | 68.8% | 0.099 s | 0.339 s |
| 100 | 34 | 283 | 97 | 65.7% | 0.164 s | 0.465 s |
| 1,000 | 47 | 856 | 235 | 72.5% | 0.809 s | 0.839 s |

The wall columns were taken at load averages between 1.7 and 2.0 on 4
cores. They are a direction, not a measurement: below 1,000 generators
the extra work of a started solve (reading the basis back and matching
it, a polars join per declaration, about 9 ms a solve here) costs more
than the iterations save. A run at 10,000 generators reached the
30-minute limit of a background job before it printed, so it is not in
the table. A wall measurement on an idle machine is still owed before
the how-to claims time. The conclusion that goes into the warm-start
how-to is the iteration count only.

</details>

<details><summary>Verification</summary>

Gates: `ruff check .`, `ruff format --check .`, `pyrefly check`, the
full suite (`pytest -n auto`, with HiGHS, Gurobi and Xpress installed),
`pytest bench/test_harness.py -k started_master`,
`test_docstring_links.py`, `test_assertions.py`, `zensical build
--strict`.

`tests/test_start.py` covers the following, on every sink unless a row
says otherwise:
- An unchanged model started from its own answer does no simplex work.
- A model that gained or lost snapshots takes fewer iterations than cold
and reaches the same objective.
- A constraint gained: the #382 case.
- HiGHS's basis starts Gurobi and Xpress.
- A saved answer and an archived one start a solve.
- A declaration whose dims changed starts as new.
- The refusals: an answer without its basis, one that ended on no basis,
a MIP, a closed answer.
- `settled` and `_counted`, case by case.

`tests/test_warm_start.py`'s sink half moved onto `warm(basis)` and
`start(values)`. Its test of the other-solver refusal is gone with the
refusal, and `test_start.py` asserts the opposite now.

Mutation table, from `python -m tools.mutate` against the full suite,
except the rows marked by hand, which were taken with its three
precautions (a clean tree, `git checkout --` to restore, `__pycache__`
dropped on both sides):

| guard | result |
|---|---|
| a mixed-integer model takes no basis (`engine.py:228-229`) |
**caught** |
| a declaration whose dims changed is new (`readback.py:362-363`) |
**caught** |
| an answer without its basis refused (`result.py:660-665`) | **caught**
|
| `sps.solve` checks the start before the build (`api.py:563-564`) |
**caught** |
| a basis that does not span the model refused (`base.py:171-172`) |
**caught** |
| a column at equal bounds is `fixed` (`base.py:332`) | **caught** |
| a nonbasic column moves to a bound it has (`base.py:333`) | **caught**
|
| a surplus leaves from the columns (by hand: `if surplus > 0` → `if
False`) | **caught** (1 failed) |
| a shortfall enters as rows (by hand: `elif surplus < 0` → `elif
False`) | **caught** (1 failed) |
| Xpress: a nonbasic row names its slack's side (by hand: rows mapped as
columns) | **still green**. A probe was added
(`test_a_basis_is_loaded_in_the_row_statuses_xpress_itself_reports`),
and it is green too: Xpress puts a nonbasic slack at its one finite
bound when it loads the basis, whichever side it was given. So the side
cannot be observed. The mapping stays the one Xpress reports, and the
probe holds the rest of the round trip. |

The tree was clean after each run.

</details>

<details><summary>Deliberately not done</summary>

- No `start=` on `solve_over`: `keep='progress'` already carries a
sweep's work from slice to slice. A sweep that rebuilds between slices
is a separate case.
- No repair choice beyond the count. Which basic column leaves is
arbitrary, and each solver repairs a singular basis. A smarter choice
waits for a model that shows it matters.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1

---
_Generated by [Claude
Code](https://claude.ai/code/session_01GLBvwp6NU7YjszFz5QMRf1)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
FBumann added a commit that referenced this pull request Oct 8, 2026
…is statuses, and a sweep from an earlier sweep or from the slice before it (#1882)

> **Prompt:** Review the R stack with 1878. Look for room for
simplification, refactoring, better naming, better seams and if better
encapsulation.

> **Prompt:** Id like to unify the start for lp and mip. But we need to
find a simple and intuitive api. Users need to be able to pass basis
values easily too!

> **Prompt:** I think start on a sweep is prefectly reasonable! And it
makes no sense for it to not exist if its availlable on regular solve!

> [!NOTE]
> The following content was generated by AI.

`start=` is one shape for an LP and a mixed-integer model: an earlier
answer, or tables keyed by reader (`primal`, `variable_basis`,
`constraint_basis`). `solve_over` also takes an earlier `Sweep` or
`'previous'`, cut by the axis as a source is. `outputs={'basis'}` asks
for the basis.

<details><summary>The API</summary>

```python
start: Result | Start | None                              # solve, Model.solve
start: Result | Sweep | Start | Literal['previous'] | None   # solve_over
```

- **`Start` is a `TypedDict`** in `specsolve.types`: `{'primal': {name:
table}, 'variable_basis': {...}, 'constraint_basis': {...}}`. Each table
takes every shape a parameter's source takes (#1878), read by the same
reader. Basis tables are read as strings and held to `basic`,
`at_lower`, `at_upper`, `fixed` and `superbasic`.
- **An LP starts from a basis where one is given, else from values. A
mixed-integer model starts from values.** An LP answer solved without
`outputs={'basis'}` now starts a solve from its primal rather than being
refused.
- **Values for an LP follow each solver's `lp_values`.** HiGHS uses a
complete primal and warns of a partial one. Gurobi warns of either.
Xpress warns of a complete one and refuses a partial one, which
`loadLPSol` cannot take. A warning asks the user to report a gain at the
issue tracker.
- **A sweep's start is cut by its axis, as a source is.** A table over
the sliced dimension gives each slice its rows: a scenario its
scenario's, a window the coordinates it covers, lookahead included, over
its local index. A table without it reaches every slice whole. An
earlier sweep is its answer, so an archive without kept windows starts a
sweep too. Each `Slice` carries the cut it applied to its sources, and
the start goes through the same function.
- **A slice its start leaves no row starts from nothing, with a
warning.** That covers a table short of the slice's key, and an earlier
sweep whose slice left no values. A missing start costs time, never the
answer, so it no longer refuses the sweep. This changed after review:
"For empty tables we should warn, not error".
- **`'previous'`** starts each slice from the answer of the one before
it, matched by the slice model's own coordinates (local `t` for a
window). Each slice is solved with its basis for the next to start from,
whether or not `outputs=` asks for it; the answer, the spill and the
archive keep only what `outputs=` asks for. The first slice, and a slice
after one that left no values, starts cold. Refused under an executor. A
slice read back off a spill starts the next from what the spill holds.
(#1884 changes this.)
- **`start=` combines with `keep='solver'` only.** `keep='progress'` and
`keep='nothing'` also say what the solve begins from, so the two
together are refused, on `solve` and `solve_over`.
- **`outputs={'basis'}`** replaces `{'variable_basis',
'constraint_basis'}`. The readers keep their names, and the files stay
under `variable_basis/` and `constraint_basis/`.

</details>

<details><summary>From the review of #1865–#1878</summary>

- The basis vocabulary (`BASIS_STATUSES`, the `BASIS` dtype, the codes)
moved from the sinks to `answer_layout`, beside the outputs read back in
it.
- The engine matches a start to a `Basis` or to values and branches on
it, rather than returning a closure. It reads an earlier answer through
`Result._start` rather than its private frames.
- `Result` and `Sweep` dispatch an output kind explicitly rather than by
`getattr`, and the output readers share one path.
- Each sink's write-back table names the status it maps rather than
listing bare codes.
- The spill manifest check compares the key first, then the outputs.
- `Result.dual` no longer says slacks are not read.

</details>

<details><summary>Values for an LP: the probe behind
<code>lp_values</code></summary>

A random 300×400 LP (5% density, seed 0), solved once cold and once
started from its own optimum, outside specsolve. This is the best case
for a values start, one run each, and it was not taken on a machine
checked for idleness.

| | cold | from the full primal | from half the columns |
|---|---|---|---|
| HiGHS `setSolution` (simplex iterations) | 661 | 0 | 661 |
| Gurobi `PStart` | 877 | 877 | — |
| Xpress `loadLPSol` | 760 | 760 | refused: needs every column |

Separately, on HiGHS, `DISPATCH` at 40 snapshots started from a
`variable_basis` alone (rows left at their default) took 40 iterations
against 29 cold. The test asserts only that such a start is taken.

</details>

<details><summary>Mutation tables</summary>

`python -m tools.mutate`, scoped to `tests/test_start.py` (and
`test_basis`, `test_warm_start`, `test_outputs` for the first group),
rather than the full suite. A guard the subset catches, the full suite
catches too.

| guard | result |
|---|---|
| a key that names no reader (`sources.py`) | **caught** |
| a name its reader has no declaration for (`sources.py`) | **caught** |
| a basis status outside the five words (`sources.py`) | **caught** |
| `lp_values` refused (`engine.py`) | **caught** |
| `lp_values` no_gain warns (`engine.py`, body → `pass`) | **caught** |
| nothing the model starts from (`engine.py`) | **caught** |
| a start that lands nowhere (`readback.py`) | **caught** |
| start beside keep progress or nothing (`result.py`) | **caught** |
| a word other than `previous` (`strategy.py`) | **caught** |
| `previous` under an executor (`strategy.py`) | **caught** |
| a table over the local index alone (`strategy.py`) | **caught** |
| the warning for a slice its start leaves no row
(`strategy.py:407-413`) | **caught** |
| by hand: that slice's empty start passed on rather than none |
**caught** (3 failed) |
| a slice after one with no values starts cold (`strategy.py`) |
**caught** |
| a word on one solve (`api.py`, taken before the merge in `engine.py`)
| **caught** |
| a constraint named as a variable keeps its own basis (by hand: both
halves in one directory) | **caught** |
| a slice keeps only the outputs asked for (by hand: every output the
result holds copied) | **caught** |

Three guards first survived, and each got a purpose-built probe before
the table was taken: the up-front check on a sweep's start and the
up-front windows check, which now spy on the build, and the outputs
filter, whose test now looks for a basis in the spill. The tree was
clean after every run.

</details>

<details><summary>Verification</summary>

- `ruff check .`, `ruff format --check .`, `pyrefly check`, and the full
suite (`pytest -n 4`, with HiGHS, Gurobi and Xpress): 5301 passed, 0
failed, on the tree of the last source change. The collected tests
(5867) are the same as at the commit before it. An earlier run on the
merge reported 5311 passed; I have not found why the two counts differ.
- On ef01170 (the empty-start warning): `pixi run check` gave 5329
passed, 566 skipped, 1 xfailed. `docs-test` gave 112 passed. The new
test was a strict `xfail` first, and failed on the refusal.
- `pixi run test-bench`: 263 passed. An earlier version of this line
said it passed, which was wrong: it had last run before the
`keep`/`start=` rule, which refused `bench/warm_payoff.py`'s warm arm
(`keep='nothing'` beside `start=`). CI caught it. The warm arm now asks
for `keep='solver'` and runs first after each update, where the gained
cut loads a fresh solver anyway, and an assertion on `Result.kept` holds
both arms to a fresh solver.
- `pixi run -e docs docs-test` and `docs-build --strict` pass.
- #1878's branch moved while this was open (any-shape values). It is
merged in here, not rebased, and its tests now pass their tables under
`'primal'`.

</details>

<details><summary>Deliberately not done, and departures</summary>

- **One PR rather than a stack.** The review refactors, the basis
output, the unified start and the sweep start all reshape #1876–#1878
before release, and each builds on the one before.
- **`keep` is unchanged** apart from the refusal beside `start=`.
Whether `start=` replaces `keep='progress'`, and which alignment
`'previous'` uses for windows, needs a measurement first: a follow-up.
- **No dual values in a start.** Gurobi's `PStart` showed no gain, and
duals would need the sign conversion reversed per solver. Measure first.
- **The other PRs' changelog lines are unchanged.** #1876's line still
reads `outputs={'variable_basis', 'constraint_basis'}`, and its title
would need the same edit.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_014wLrXpeSGDqoBFxyrs8NEz

---------

Co-authored-by: Claude <noreply@anthropic.com>
FBumann added a commit that referenced this pull request Oct 9, 2026
> **Prompt:** Prepare the latest release of specsolve.
>
> **Prompt:** Im wondering if we bumped the layout stuff multiple times
in this release. We should only increment once since the last release!

> [!NOTE]
> The following content was generated by AI.

Releases 0.8.0 on merge. The answer layout rose twice since 0.7.0, and
the new inputs layout once after it was added. Each now rises once:
`ANSWER_LAYOUT` 3 → 4, `INPUTS_LAYOUT` 1.

<details><summary>The layout numbers</summary>

`v0.7.0..origin/main` on `archive_layout.py` and
`relational/answer_layout.py`:

| PR | change |
|---|---|
| #1883 | `LAYOUT` renamed `ANSWER_LAYOUT` (3); `INPUTS_LAYOUT = 1`
added |
| #1865 | `ANSWER_LAYOUT` 3 → 4 |
| #1904 | `ANSWER_LAYOUT` 4 → 5 |
| #1913 | `INPUTS_LAYOUT` 1 → 2 |

No release wrote the intermediate numbers, so this PR sets
`ANSWER_LAYOUT = 4` and `INPUTS_LAYOUT = 1`. The `format.json` example
in `Result.save`'s docstring follows. A 0.7.0 answer stamps 3 and a
0.7.0 archive has no root stamp, so both stay refused.

The cost is on unreleased files only. An answer written from `main`
between #1865 and #1904 stamps 4 and is no longer refused, though it
still carries `model_digest`. An archive written from `main` between
#1883 and #1913 stamps inputs 1 and is no longer refused, though it may
hold `sources.parquet`.

No test asserts a layout number; the refusal tests use 0, 99 and no
stamp.

</details>

<details><summary>Release notes and gates</summary>

- `## Upcoming version` → `## 0.8.0 (2026-10-09)`, with a paragraph, a
**Breaks** list and the lines grouped under Solving, Reading an answer,
Archives, Data and Examples. An empty `## Upcoming version` stays above
it. All 21 lines are kept; #1876's line says `outputs={'basis'}`, the
spelling #1882 gave it.
- **Minor, not patch**: #1884 removes `keep=` and `Result.kept`, #1913
removes the digests, and the layout refuses 0.7.0 files.
- `python -m tools.changelog check` → `releases 0.8.0 on merge`.
- `pixi run check`: 5523 passed, 566 skipped, 4 xfailed.
- Not run: `pixi run docs-test`.
- Departure: the layout fix rides in this PR rather than its own,
because the release must not ship the doubled numbers. It is its own
commit.

</details>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01SUzrD57WeiMZtKUnw7wsG8

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants