Repository navigation
refac!: land the spec/specsolve stack (#274 … #357) in one merge - #358
Merged
Merged
Conversation
fluxopt's layers are Elements -> ModelData -> Model. This adds a second implementation of the last step: instead of building a linopy model in memory, it binds the same ModelData to a declarative YAML program executed by farkas, which streams the model to the solver under a fixed memory budget. The seam already existed, so nothing in the element or data layers changes. Two properties carry the design: - Sparsity is row absence. A parameter keeps its declared rank while its table holds only live entries, so the (flow, effect) coefficient block travels as ~4% of its dense grid instead of being materialised. - Feature membership is a `where` mask, not a subset dimension. The status/sizing cross-product that needs four code paths in model.py is four masked equations here. Covered: carrier balance, converters, storage (cyclic and acyclic, sizing, final-level bounds, charge/discharge exclusion), flow sizing, status (uptime/downtime, transitions, pre-horizon carry-over), ramps, effects with cross-effect folding, effect limits, and multi-period. Investment, piecewise and component status raise UnsupportedFeatureError rather than being dropped — a missing constraint still solves, just to the wrong answer. Both backends are held to the same answer by tests/test_relational_parity.py, which drives each to a proven optimum and fails if the relational lane ever beats linopy's, since a minimisation cannot do that without having lost a constraint. farkas is not on PyPI yet, so the `relational` extra resolves through [tool.uv.sources] and must become a normal constraint before release.
Investment is the last feature that needed no upstream primitive. `build` and `size_at_build` live on a `build_period` axis, and the lifetime window becomes a precomputed (period, build_period) matrix — so `active[p] = sum_b build[b] * window[p, b]` is a plain sum rather than a per-flow Python loop over periods. At-build effects are diagonal in (period, build_period): a one-time cost belongs to the period the build happened in, not to every period the unit is alive. Getting that wrong is invisible in the decision variables — build, active and size all matched linopy exactly while the objective was 50% high, because the cost landed in every period. Piecewise conversion and component status both need the same missing primitive: an indexed lookup, the adjoint of `group_sum`, so a per-component quantity can be read on each of its flows. Planned upstream as `at(x, over=dim, index=map)` (farkas ROADMAP Track 1 item 2). Both keep raising UnsupportedFeatureError, now with that reason recorded. Tests cover investment parity over 2 and 3 periods, asserting the build decision agrees and not only its cost, plus the guard that investment without periods is rejected rather than built.
A storage sized with mandatory=True has no build indicator, so cap_ind_coeff is empty — and the empty-table fallback keyed every absent lump term by `flow`, which fails to bind against a storage-keyed parameter. Found by running the backend over the benchmark systems rather than the test suite: green_city has exactly that shape and no parity test did. lump_terms now carries the entity dim alongside the coefficients.
Three passes over core.yaml, each held to the parity suite. Element-layer vocabulary. The program spoke a third dialect — neither the fields users type in Python nor plain math, but ModelData's internal encoding names. `sab_coeff` is unreadable; `effects_per_size_at_build` is a field they already know. 61 names renamed, and the coefficient families now follow one scheme (`effects_per_*` / `effects_fixed_*`) instead of four. Companion flags dropped. A bare parameter name in a `where` already means "defined: non-null and finite", so `has_uptime_min` alongside `uptime_min` was restating what the data says. 12 removed; `has_size_min_pos` became the comparison it always was, `size_min > 0`. The structural masks that carry no value of their own (is_first, is_cyclic, has_status) stay. Effect accounting is a ledger. Twelve mechanical `sum(sum(x * c, over=...))` terms collapse behind three macros — per_step, per_entity, at_build — so the equation reads as the list of contributions it is. Macros are pure AST substitution; neither backend sees them. 100 parameters -> 88, of which boolean scaffolding 27 -> 14.
…ed copies dt_previous and dt_at_start were encodings of a quantity the program already has. shift(dt, time=1) works on a parameter, and inside a block masked to is_first the plain dt already means 'dt at the first step'.
farkas reads YAML 1.2, where `on` is a string, so the program worked. Any consumer loading the same file with stock pyyaml gets `True` as the key instead — and pyyaml collapses on/off/yes/no to two boolean keys, so a file using more than one of them silently loses entries. `running` is safe under both revisions and matches the element-layer name already used by effects_per_running_hour.
The math is a file, not a code path: `relational/core.yaml` becomes the whole model and `model.py` is deleted rather than ported. linopy moves behind an optional lane. Records the decisions so they are not re-argued — one monolithic program rather than merged templates, topology as data, no Python escape hatch, lpspec pinned to a tag — and the order of the work that follows. Also records two findings that cost time to make: `at(x, by=lookup)` exists and closes both features `relational/__init__.py` names as blocked, and declaring the linopy extra pulls an unreleased linopy branch into the base environment, because uv resolves one version per package across the whole lock. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`farkas` was renamed to `lpspec` and the repo moved to fluxopt/lpspec. Pinned to v0.0.1-alpha.213 rather than tracking the branch: the language surface is pre-1.0 and still moves, so an unpinned ref would let a `uv sync` change what core.yaml means. lpspec becomes a core dependency rather than an extra — it is the modeling layer now, not an alternative backend. The `linopy` extra is written but left commented, and the reason is worth keeping: uv resolves one version per package across the whole lock, extras included, so declaring `lpspec[linopy]` pulls lpspec's unreleased linopy branch into the *base* environment, where model.py builds against release semantics and emits ~2900 LinopySemanticsWarnings. It is uncommented in the commit that deletes model.py, when nothing in core imports linopy at all. The engine no longer takes a memory budget — it holds the model it builds — so `memory_limit` goes and `solver_options` is now in the solver's own vocabulary rather than linopy's shape. core.yaml is still written against the pre-lpspec surface and does not load against the pin, so the parity module is skipped with a reason naming the step that rewrites it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The program was written against the pre-rename surface and does not load against the pinned tag. What changed, and what each change is: Mechanical: `group_sum(x, over=d, by=l)` -> `sum(x, by=l)`; `binary: true` -> `domain: binary`; `roll(x, time=1)` -> `shift(x, over=time, offset=1, edge='wrap')`; `objectives:` (named, plural) -> `objective:`; a dimension's `coords:` -> a top-level `lookups:` block. Structural: `equations:` is refused rather than migrated, because a constraint is one row-rule. Every guarded alternative becomes its own named constraint carrying the conjunction of the block's `where` and its own — 26 constraints become 60. They read better than the lists did: a name per rule, and the masks of a family visibly disjoint. Two lookups are renamed `carrier_of` / `converter_of`. The namespace is flat and refuses a lookup that shadows the dimension it maps into. Semantic, and the only one that changes what is built: shifting a variable-free expression must now say what the vacated edge holds, so the four `shift(dt, ...)` calls carry `edge=0`. Their rows are masked by `NOT is_first` either way; the language declines to assume that. On the data side, three things pandas cannot express are now stated: - an empty table has no dtypes to carry, and pandas types an empty column float64 — which reads as a numeric label space and is refused. `_empty` and `_flags` construct typed empties, and one stamping pass covers the frames `_tidy` and the period re-indexing derive; - an empty label space has nothing to infer from, so dimension labels travel as string arrays; - a lookup column no system happens to fill is all-null and infers as a null column, so the flow index states its schema. That makes polars a direct dependency, which is right: it is the interchange type with lpspec, not an implementation detail of it. `effects_fixed_mandatory` is now dense over `effect`. It is the constant side of `effect_accounting`, where a missing row is not an absence but a zero that binds — lpspec refuses the ambiguity, and it is right to. Parity is green across all ten cases, so the program says what `model.py` says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Component-level status is expressed, and not by writing the status family a second time. `flows.status` and `flows.cstatus` carry the same fields and obey the same math — they differ only in which rows read the binary. So they bind to one `status_entity` dimension and the program states transition, exclusivity, uptime and downtime tracking, and pre-horizon carry-over once. What separates them is a lookup: `status_of` maps a flow to the decision it answers to, itself where the flow carries its own Status and its component where a component's governs it. The gating rows read `at(running, by=status_of)` and cannot tell the two apart. That is the whole feature. The alternative was ~150 lines of near identical YAML and the drift that invites, which is the thing this migration is meant to remove rather than reproduce. Two consequences worth naming: - `has_status` is gone from the entity side. Membership of `status_entity` *is* the mask — an entity is in the dimension because it carries a Status, and there is no second way to say so. What remains is `is_gated`, a flow-side mask, because whether a rate answers to a binary is a fact about the flow. - the status effect ledger is keyed by the decision rather than by the flow it gates, so `per_step_status` sums over the entity axis. A component's startup cost is charged once, not once per governed flow. Parity covers it: a storage with Status gates its charging and discharging flows together, and the test asserts the binary itself agrees with the linopy lane, not only its cost — a dropped gate would still solve, and cheaper. Piecewise is still refused, and is the last feature that is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last feature, and the only one where lpspec's own sugar did not reach. `piecewise:` declares its links as a static list, so the number of expressions a curve ties together is fixed in the file — and in fluxopt that number is data: a 2-flow boiler curve and a 3-flow CHP curve are ordinary in the same system. Raised upstream as fluxopt/specsolve#1101. So the lambda formulation is written out, and the move that makes it general is that a link becomes a *row* keyed on `flow` rather than an entry in a list: pw_link_equal: foreach: [flow, time, period] where: "is_piecewise AND pw_equal" expression: rate == sum(at(lam, by=converter_of) * pw_bp_value, over=bp) Arity is data after that, and so is breakpoint count — `pw_bp_present` masks a narrower curve's tail away, and the `<=` / `>=` links are two more constraints under complementary masks rather than a sign in a tuple. `absence: zero` on the segment binary is what makes the adjacency row read correctly at the last breakpoint, where no segment starts. `method` becomes advisory: adjacency is exact for every method fluxopt's `auto` would pick, so sos2 and incremental agree with it. `lp` does not and is refused — it is a tangent-line *relaxation*, and answering it with an exact formulation would be answering a different question. Four bugs found on the way there, each of which would have solved to a wrong answer rather than failing: - `_tidy` keeps False for a bool array, so routing a mask through it marked every breakpoint as carrying a segment; - a piecewise-only converter states no linear equation and so never reached `converter_of`, which is the lookup its own curve rows are keyed through; - the flow index was frozen into `sources` before the piecewise block filled that column; - the `converter` axis listed only converters with linear equations, so a curve's converter was not a coordinate of the dimension its rows are keyed on. Parity covers two curves of different arity in one system, which is the case a static link list cannot express. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`solve` returns a `fluxopt.results.Result` rather than the engine's own object, so parity compares answers instead of shapes — and deleting the linopy builder becomes a deletion rather than a migration. Almost all of it is relabelling. The program indexes time and period by ordinal because the engine joins on them; a `Result` is read by humans and indexes them by the element layer's labels. Everything else is one program variable to one `Var` name, except `running`/`startup`/ `shutdown`, which split back into their flow and component halves — the program keeps them on one axis because they obey the same math, a solution keeps them apart because they are different things to read. `contributions.py` needs no change at all. It reads a solution Dataset and a ModelData, never a linopy model, so producing the same Dataset makes `result.stats` work as it stands. Both lanes agree to ~1e-12 on effect contributions and on the reconstructed per-timestep effects. The effect ledger is now two named expressions, `effect_temporal` and `effect_lump`, with the accounting row reading `effect_total == effect_temporal + effect_lump`. That reads better and it is how the lump half is recovered after a solve — through the same compiler that built the row, so the split cannot drift from the total. Three real differences surfaced, each of which would have been a wrong answer rather than a failure: - the engine returns labels sorted, and both `Result`'s NetCDF round trip and the contributions check are positional on the element layer's own order. The check caught it, which is what it is for; - the penalty effect is added to the objective at 1.0 unless the caller names it. Only the linopy lane did that, so the two lanes were minimising different objectives for the same request; - `to_dataarray` is dense over the declared dimensions, so a variable masked down to one flow came back spanning every flow. Padding is indistinguishable from a solved zero once it is in a Dataset — an uninvested flow would read as invested-at-zero — so membership is read off the padding, which is NaN where a solved value never is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`optimize()` now solves through lpspec. Running the 515-test math suite through it is what this PR is really about — it is a far stronger gate than the 13 parity cases, and it found things they could not. **A silent gap.** Flow aggregates — `flow_hours_min/max` and `load_factor_min/max` — were never ported and nothing rejected them, so the lane had been dropping those constraints and solving to a wrong answer. This is precisely the failure mode `UnsupportedFeatureError` exists to prevent, and it slipped through because the omission was in the program rather than in a rejection list. They are ported here, with a sized flow's bound travelling as a coefficient rather than a product because its size is a decision. Two more wrong answers, both in code added earlier in this stack: - `dict.fromkeys(gated)` maps each gated curve to `None`, not to itself, so `pw_status_of` read as "no Status" and every gated piecewise curve ran ungated. Eight tests caught it by asserting flow was zero when the binary was off; - the build axis is a second period axis and a solution names it `period`. Keeping the program's `build_period` out through the solution misaligned the at-build coefficients in the contributions decomposition. `customize` is gone and gets no Python replacement. `FlowSystem.math()` returns the program as an `lpspec.Model` — readable and typesettable before any data exists — and `optimize(math=..., parameters=...)` solves an edited one. A caller may not supply a name the program already declares: silently replacing `rate_max` would change the model without editing it, which is the property the file exists to have. The time-varying-contribution-into-lump refusal moves to `validation.py`. It is a fact about the model, so both lanes owe it. `build_model()` becomes `build_data()` and `FlowSystemModel` leaves the public API. `model.py` stays for now as the parity oracle and the benchmark's build target; deleting it is the next step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`model.py` (1693 lines) and `constraints/` (465) are gone. Nothing in fluxopt builds math in Python any more; `relational/core.yaml` is the whole model, and the `linopy` extra is declared now that no core module imports linopy. `src/` goes from 9.2k lines to 7.7k, and the shape of what is left is the point: the math is a file, so it is reviewed, diffed, typeset and extended as one artefact rather than reconstructed from a builder. Two things in the deleted tree were not builder code and moved rather than went: `compute_previous_duration` (prior rates -> a duration, pure data) into `model_data.py`, and `_lump_bearing_effects` into `effect_terms.py`, beside the terms it reads. `Result.from_model` goes with the model it read. The benchmark builds through lpspec and reports `nonzeros` where it reported `binaries` — the engine counts columns, and integrality is a property of each rather than a second total. **The parity test becomes a differential test between engines.** Same `core.yaml`, same sources, one built relationally and one evaluated onto a `linopy.Model` by `lpspec.linopy`. That is a sharper instrument than it was: the two sides used to be two hand-written models, so a disagreement could be a bug in either; now they are two engines on one file, and a disagreement is an engine bug. It does not run yet. The eager lane refuses a reduction over an empty dimension, which is the ordinary shape of a system not using some feature — reported as fluxopt/specsolve#1108, minimal repro included. The oracle-dependent cases carry `needs_oracle` and turn back on when it lands; the ones that assert something on their own still run. What stands meanwhile is `tests/math` and `tests/math_port`: 587 tests, all of them on the lane fluxopt ships. That is the stronger gate anyway — it was the math suite, not parity, that caught flow aggregates going missing. A limitation is gone rather than ported: a governed flow may now be sized. `_constrain_flow_rates_component_status` raised NotImplementedError there; one status axis makes the case ordinary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`fluxopt.relational` was named for a distinction that no longer exists. There is no second backend to be relational *rather than*; it is the math and the code that binds data to it. So `fluxopt/relational/` is `fluxopt/math/`, `core.yaml` is `program.yaml`, `MATH_PROGRAM` is `PROGRAM`, and `test_relational_parity.py` is `test_engines_agree.py` — which is what it now tests. "Relational" survives only where it is still true: lpspec's engine, as against its eager lane. Two deletions: - `IdList` and the `Identified` protocol, 74 lines. Nothing has constructed one since flows became plain lists; what was left were isinstance branches for a container no code produces, and a public export nobody could use. - `Var.STORAGE_PRIOR`, declared and never produced or read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lpspec ships the folds fluxopt would otherwise write: scenario sweeps, myopic pathways and rolling horizon are all `solve_over` over an axis, with `carry` copying one slice's answer into the next slice's data. Two things in our sources refused to be cut, and both were ours to fix. `prior_level` gains a `period` axis. A carry reduces exactly one dimension, and `level` over `(storage, time, period)` into `prior_level` over `(storage)` asked it to collapse two — so a rolling horizon over time was refused before it read a row. Carrying the period makes the collapse `time` alone, which is also the more honest shape: a prior level is per period. Dimension labels travel as one-column frames rather than bare sequences. A strategy filters every source carrying the axis it cuts on, and a list is not a table it can cut. Neither costs anything at build, and together they mean a rolling horizon over fluxopt's math is `lps.solve_over(...)` rather than a driver we maintain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`math/sources.py` imported `_leontief`/`_apply_leontief` from `contributions.py` — the binder reaching into the post-solve reader for its math, which is the dependency backwards. Both need it and neither is the other's: the binder folds it into the coefficients it hands the solver, and the breakdown applies it to charge each contributor the full priced-in cost. So it lives in `fluxopt/leontief.py`, and the names lose their underscore because two modules is not private. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every contribution is a named expression now, declared with the entity it came from and reduced only where `effect_accounting` adds it up. The ledger sums those very expressions, so `result.stats.effect_contributions` is not a reconstruction of the model's arithmetic — it is that arithmetic, read one step before the sum. That was the design doc's plan, and the objection I raised against it was wrong. I had costed *declaring the attribution separately*, which would have meant carrying raw coefficients beside the folded ones and undoing #270. Summing the named expression means there is only one declaration, and the three obstacles dissolve: - the aggregation weight leaves the coefficient and multiplies inside the sum, so a contribution reads as the physical per-step quantity and the ledger still totals what it totalled. dt stays, being part of what a per-flow-hour rate means; - mandatory sizing keeps its build indicator instead of a pre-summed constant, pinned to 1 by a row. A constant could not be attributed to the entity that owed it, and on the constant side of the ledger it had to be covered for every row — which an empty `storage` dimension could not do (the same shape as fluxopt/specsolve#1108); - the direct view is `(I - C) . charged`, a forward multiply and the exact inverse of the fold the binder applied. Verified against the old implementation to 5.7e-14 before it was deleted. `effect_terms.py` goes: it existed to keep two consumers on one declaration, and there is one consumer now. `_lump_bearing_effects` moves to `validation.py` and reads the containers directly — what makes a cost one-time is which container declares it. The `Contribution` vocabulary goes with it; the program's expression names are the vocabulary. The breakdown is solve-time only. A named expression is evaluated at a solution, so a Result read back from disk carries one or has none — and saying so is better than the old fallback, which re-derived with today's logic against yesterday's numbers and warned that it might differ. Two bugs the change surfaced, both in the assembly rather than the math: the contributor axis and the solver totals disagreed on *dim order*, which a positional comparison reads as a transpose; and the public accessors' documented `(contributor, effect, ...)` order was whatever the expression declared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last change made the effect breakdown solve-time only, because a
named expression is evaluated against a solution. That is true of *every*
named expression, so the answer should carry them all rather than the
handful fluxopt happens to read.
`Result.expressions` replaces `Result.contributions`, and the breakdown
becomes a view over it — assembled on access from the stored quantities
rather than stored a second time beside them. Same bytes on disk, one
fewer bespoke group, and `result.expression(name)` reaches anything.
The point is what that does for a caller. `optimize(math=...)` lets
someone add an expression; naming a quantity is how you ask for it, and
now the answer brings it home and it survives export:
math.expressions['grid_energy'] = ...('sum(rate * dt, over=flow)')
result = system.optimize(math=math)
result.expression('grid_energy') # theirs
Result.from_netcdf(path).expression('grid_energy') # still theirs
`solve` loads the program once and hands it to the reader, so what is
evaluated is what was solved — a caller's own declarations included,
which is the only way this could work for them.
One wrinkle the Dataset introduces: expressions whose entity axis is
narrower than another's are aligned to the union and padded. A solved
product is never NaN, so padding there means the entity had no row, and
the assembly reads it as the zero it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of its claims did not survive the work, and a design note that quietly edits itself is worth less than one that says what it learned — so the original text stands and a new section corrects it. The parity test did not retire with `model.py`; it became a differential between engines, which is sharper than what it replaced. And `contributions.py` shrank rather than went, by a better route than the page described: name the contribution and have the ledger sum that very expression, rather than declaring the attribution separately and paying for a second set of coefficients. The general lesson is one the ceiling notes already state and this page restated badly — a named expression costs nothing where it is referenced, so naming a quantity the model already computes is free. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`CarriersData.flow_coeff` was a dense `(carrier, flow)` matrix of +1/-1/NaN holding exactly one entry per column, because a flow balances on exactly one carrier. It becomes the two columns it always was: `carrier_of` and `sign`. The tell was that nobody wanted the matrix. `build` was *given* a per-flow sign and expanded it; the binder collapsed it back with an `argmax` over the carrier axis; `Result.topology` walked it with a `dropna` per carrier; `stats.carrier_balance` multiplied it against the rates to produce a mostly-empty `(carrier, flow, time)` array. Four readers, four collapses, one column. Not a measured memory win — on the largest reference system the matrix is 0.02 MB, and honesty is better than a benchmark that flatters. It is quadratic in flows x carriers, so it would matter on a model far larger than any here; what it costs *today* is code, and that is the reason. `carrier_balance` changes shape with it: `(flow, time)` signed rates carrying the carrier as a coordinate rather than an axis. Group by it to get the balance, which is the thing the model holds at zero — and which the old shape made you sum a mostly-NaN array to see. This is the first slice of the `ModelData` re-cut, taken on its own because it needs none of the rest: the representation was wrong on its own terms, not just wrong for lpspec. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`StatusData.governed_flows` was a `(component, governed_idx)` string array padded with `''` to the widest row — a function stored as a ragged matrix. Both readers ravelled it and filtered the padding back out, and one of them only ever wanted the first entry per row. It becomes `FlowsData.governed_by`, a `(flow,)` column holding the component whose Status governs that flow, `''` where none does. A flow is governed by at most one component, so that is the direction the relation actually has, and it is the direction both readers wanted: the binder builds `status_of` from it by lookup rather than by walking rows, and the breakdown takes the first flow per component with a `setdefault` over one pass. Moving it also takes a dead field off a reused type. `StatusData` serves both a flow's own Status and a component's; `governed_flows` was meaningful for the second and `None` for the first, which is the shape that argues for a separate type — and the shape that goes away when the field sits on the thing it describes. The `ModelData` id check gets stronger on the way through: it used to verify the governed ids were flows, and now also verifies the component named actually has a Status. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ConvertersData.eq_mask` was a dense `(converter, eq_idx)` boolean, and `build` filled it with `mask_row[i] = True for i in range(len( conv.conversion_factors))` — which is to say it stored `eq_idx < n` as a matrix. It becomes `n_equations`, a `(converter,)` count. Everything that read it read it back as a count: the binder tidied the mask to rows and asserted True on every one, took the converter list off its coordinate, and took the `eq_idx` width off its size. All three are the count, said three ways. The validation gets sharper for free. A boolean matrix can only be checked for dtype; a count can be checked for being a positive integer, which is what "a converter states at least one equation" actually means. Third of the same shape in this series, after the carrier matrix (#303) and the governed-flow padding (#304): a fact with one value per entity, stored across a second axis that every reader immediately collapsed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`FlowsData.effect_coeff` was dense over `(flow, effect, time, period)`. On the stress reference system at 8760 steps that is 442.8 MB at 3.98% live — 252 real (flow, effect) pairs out of 6324 — built in full and thrown away one line later by `drop_zero`. It becomes the pair table the codebase already uses twice, for converter and piecewise coefficients: `effect_pair_flow`, `effect_pair_effect`, and `effect_pair_coeff` carrying each live pair's own `(time[, period])` series. 442.8 MB -> 17.7 MB, and 56.1 -> 6.2 on energy_transition. `flow_id` arrives with it. The roster used to be readable off the dense array's coordinate; a pair table names only the flows that charge something, so who exists has to be said rather than inferred — which it should have been anyway. The binder no longer densifies to fold the Leontief inverse. That contraction ran as an `xr.dot` over the effect axis, which is why the dense form existed at all; it now runs on the rows, joining each live pair against the effects it feeds and summing back per (flow, effect). Same numbers, and the widest thing built is the result. Two things that join has to get right, both caught by tests rather than by reading: the pair table indexes time by ordinal and the factor table did not, and a `contribution_from` may vary by period as well as by timestep — so the join keys follow whatever axes the factor actually carries, rather than a fixed pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Sizing(size_min=-5, size_max=9)` constructed happily. The rule existed,
but as a check on a materialised array several hundred lines later, where
it had to reconstruct which entity failed from a coordinate:
ValueError: Sizing.size_min < 0 on [np.str_('f')]
It is a pydantic constraint now, so it fires on the value the user wrote
and names the field. Same for `size_max >= size_min`, `Investment`'s
lifetime and prior size, and `Status`'s durations. The array-level checks
stay, because a hand-edited netCDF never passed through the element at
all — and their docstrings now say that is the job they are left doing.
Seven checks come out entirely: `Unknown effect {k!r} in ...`, repeated
in five container builders. `validate_system` walks the same element
models and raises on any undeclared effect before a container is built,
on every public path — so none of the seven could fire. Verified by
reaching one through the private container API, which is the only way
left to get to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fluxopt refuses a bad model in four places — the element, the system, the data and the bind — and nothing said which owns what. That is how a rule about one field's value ended up enforced on a materialised array three layers away, and how seven checks came to sit somewhere they could never fire. The page names the four, gives the question each answers that the others cannot, and gives the test for layer 3 that would have caught both: could a caller reach this without an earlier layer already refusing it? If not, the check is dead. Two smells worth having a name for, both of which had happened here: a rule that has to reconstruct which entity failed is in too deep a layer, because it had the entity when the user wrote it; and a check nothing can reach is a rule that moved up and left a copy behind. Layer 4 gets the shortest section and the firmest advice — lpspec checks dtypes, unknown labels and coverage against the program's own declarations, so anything we write there is a weaker second copy of a check that runs anyway. The guards that stay now say in their docstrings which job they are doing, so the next reader does not mistake a reload guard for the enforcement. `StoragesData` is labelled as the one that is still the enforcement, and should move up when convenient. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last check that was enforcement rather than a guard. Capacity, efficiencies and the loss rate were validated on materialised arrays, naming the storages that failed by reconstructing them from a coordinate; they are refused at construction now, naming the field and the value. The wrinkle is that these are `Variate` fields, so a value may be a scalar, a series, or a `ProfileRef` pointing at numbers supplied later. `variate_out_of_range` checks the first two where they are written and declines to guess at the third, which is exactly what the element can know: `eta_charge=1.5` is refused immediately, `eta_charge=ProfileRef(...)` cannot be until profiles resolve. So the array-level check stays, and for a better reason than "reload" — it is where a resolved reference's values first exist. There is a test for that path now, because it is the only thing keeping the check alive and a future reader would otherwise delete it as a duplicate. That is a third legitimate reason for layer 3 and the validation-layers page did not have it, so it does now: the decision rule reads "only violable by a reload, or by a resolved ref", and the closing section says the reload half retires with the tidy-frames re-cut while the ProfileRef half does not and should not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`EffectsData.cf_temporal` was a dense `(effect, source_effect, time,
period)` matrix. On the stress reference system that is 67.3 MB holding
**22 declared factors** — and most of the waste is not sparsity but
repetition: a scalar `contribution_from={'co2': 45.0}` broadcast across
every one of 547 timesteps and 16 periods.
It becomes pairs — `cf_pair_effect`, `cf_pair_source`, `cf_pair_factor`,
each factor carrying its own `(time[, period])` series. **67.3 MB ->
1.54 MB stored.**
The square matrix still exists, because inverting it is what the Leontief
fold needs and an inverse wants a square. But it is built by
`cf_matrix()` at the moment of inversion and dropped after, rather than
carried in the model data and serialized to disk. That is the whole
trade: the thing that needs a matrix is transient, and the thing that is
kept is what was declared.
The cycle and self-reference checks read the edge list directly now,
which is what they were reconstructing from the matrix anyway — `!= 0`,
reduce over the extra dims, then walk the survivors. One thing that
reconstruction gave for free and the edge list does not: a node for every
effect, edges or not. The cycle walk looks up each neighbour's state, so
a reachable-but-unlisted node is a `KeyError` — seeded explicitly now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First container across, and the one that proves the migration is incremental rather than atomic. The obstacle looked structural: `ModelData` serializes by reflecting over dataclass fields into an `xr.Dataset`, so a container holding frames and a container holding arrays cannot share a netCDF file — which made the whole conversion one change of ~3000 lines. It is not, because netCDF is a perfectly good container for a table. `frame_to_dataset` writes each column as a variable over a row index and `dataset_to_frame` reverses it exactly, so a converted container serializes into the same file, the same format, and the same public `to_netcdf`. `SizingData`'s five arrays become two frames: `bounds`, one row per sized entity, and `effects`, one row per (entity, effect) actually charged — the shape #306 and #311 established for coefficients. `_fold_lump_rows` is written once and is most of the remaining work already done. Every lump container needs the Leontief contraction, and doing it as a join and a group-by in polars is both the same arithmetic and the thing a dataframe is for. Two things measured rather than assumed. Building the frame row-wise regressed `ModelData.build` from 15.6s to 21.3s on the stress system at 8760 steps — 9216 dicts cost more than the data in them. Column-wise it is **11.6s**, faster than the arrays it replaces. And the lump tables leave `PERIOD_PARAMS`: they carry their own period column now, so cross-joining one onto every period would multiply what is already there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second container, and the first to spend what the frames buy: a value that is not there has no row, rather than a row holding a sentinel. `lifetime` was `NaN = forever` on a dense per-entity array, so every reader tested `np.isnan` to mean "no expiry". It is its own frame now holding only the entities that expire, and forever is the lookup's default — the absence *is* the meaning, which is also how the program already reads a missing parameter row. The at-build coefficients lose their `eye`. A one-time cost belongs to the period the build happened in, which as a dense array meant building a square `(period, build_period)` matrix and multiplying by an identity to erase everything off the diagonal. As rows the diagonal is just the same period named twice. Both tables leave `PERIOD_PARAMS` and `BUILD_PERIOD_PARAMS` with the sizing ones: they carry their own period column, so a cross-join would multiply what is already there. That is the third time this has come up, so it is worth stating as the rule — a frame-backed table is never in those lists. `_frames_to_dataset` / `_frames_from_dataset` generalise the bridge from #312 to any number of named frames, and `SizingData` uses them too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 29, 2026
Closed
refac!: a rule with several regimes is written once, and the regimes are cases of what it reads
#343
Closed
Closed
refac!: effect coefficients bind as declared, and a cross-effect factor is one value per period
#346
Closed
Closed
Closed
FBumann
pushed a commit
that referenced
this pull request
Sep 29, 2026
The stack (#358) removed Dims, FlowSystemModel, IdList, ModelData and Result from fluxopt.__all__; a solve now answers with specsolve's Result. The README table, the pinned test set, the API index page and the internal-module list in gen_ref_pages.py follow the current src/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182CG6AhGfcNcWshCsR8Rt5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Intent: to be written by the author.)
Note
The following content was generated by AI.
Description
The stack #274 → #357 in one PR, so it lands with one merge and one CI run instead of 62. Each commit below was already reviewed in its own PR. This PR adds only the merges that bring the stack up to date with
main, plus one doc fix.Included PRs, bottom to top: #274, #275, #289, #290, #291, #293, #294, #295, #296, #297, #298, #299, #300, #301, #302, #303, #304, #305, #306, #307, #308, #310, #311, #312, #313, #314, #315, #316, #317, #318, #319, #320, #321, #322, #323, #324, #325, #326, #327, #328, #329, #330, #331, #332, #333, #336, #341, #342, #343, #344, #345, #346, #347, #348, #349, #350, #351, #352, #353, #354, #355, #357.
How
mainwas brought in: main was merged into #274 and then carried up the stack one branch at a time. The branch that first ran into each conflict got the fix:pyproject.toml(chore: pin lpspec to a tag, drop farkas #290, chore(deps)!: move to specsolve and mathspec 0.2, on the composition branch #341): kept the stack's dependency block and applied main'slinopy==0.9.0pin. refac!: drop the dead piecewise method, linopy and netcdf4, and refuse unknown fields #354 later dropslinopyandnetcdf4, and that wins.README.md(refac: delete the netCDF bridge ModelData no longer crosses #319, refac!: a system is a spec and its sources, solved and archived by specsolve #351): kept the stack's text and restyled the code blocks the way ruff 0.16 formats them (main's lint bump).benchmark/uv.lock(chore(deps)!: move to specsolve and mathspec 0.2, on the composition branch #341): regenerated withuv lock.main(refac!: revert the flat time axis and ragged periods (#213) so the spec stack can land #356).main's tree equals the revert commit inside the stack (ef32b05), so the merge withmaintakes the stack's tree unchanged.T201ignores keep main'sbenchmark/flixopt_stress.pyand swapsrc/fluxopt/benchmark.pyforbenchmark/reference.py. The twopython -m fluxopt.benchmark stresscommands that main's docs(benchmark): measured comparison with flixopt on the stress system #271 added (benchmark/flixopt_stress.py,docs/benchmark.md) now point atreference.py.These merge commits were never pushed to the stack branches, so those PRs still show conflicts on GitHub. Once this merges, they can be closed.
Type of Change
Related Issues
Supersedes #274 … #357 (the list above).
Testing
Gates
On this branch:
At the branches where a conflict was resolved (before #357):
#274 and #319 could not be tested: their pinned git dependencies no longer resolve (
farkasis now published asspecsolve). That affects only those intermediate states, not this branch. The docs build and notebooks were left to CI.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_0182CG6AhGfcNcWshCsR8Rt5
Generated by Claude Code