feat(language): a parameter and a lookup each say whether their data must be complete, so a row lost in preparation is not read as a mask

[FBumann]

Prompt: "Do you want to combine them into one PR now?" and, on an lpspec review of it, "Yes" — #243 and #245 were the two halves of one issue, and the review took the bound/divisor guard out of it.

Prompt: "Please rebase #245 and #429" — rebased onto main at eac1930, where #422 landed.

Note

The following content was generated by AI.

Closes #296.

parameters: and lookups: each take coverage: total | masked, default total — one word, one vocabulary. A total declaration short of a row is a bind-time error naming what is missing; masked says the gap is the point.

parameters:
  cost: { dims: [generator] } # total: every generator has one
  ramp_limit: { dims: [generator], coverage: masked } # no row means no limit

lookups:
  gen_bus: { over: generator, into: bus } # total: every generator is on a bus
  line_to: { over: line, into: bus, coverage: masked } # an open end is meant

Nothing here checks a table against data, because nothing here holds data. The language states the rule and names the vocabulary; the consumer that binds enforces it. One refusal stays at load, and it is a question the file alone settles: coverage: on a parameter a piecewise: block consumes.

What the rebase onto eac1930 changed

#422 removed the label-space kind of lookup — every lookup now maps into: a declared dimension — so the second of this branch's two refusals has nothing left to refuse and is gone with it, along with the test that pinned it and the sentence in dimensions.md that stated it. LookupBlock.into is required again and LookupDeclaration.coverage is Coverage rather than Coverage | None: with one kind of lookup, every map answers for its coverage.

Three semantic conflicts git did not flag, each carried into the commit that introduced the assertion:

• SMALL_MODEL's tag is a parameter on main, where it used to be a label-space lookup, so the lowering test's parameter table gains a 'tag': 'total' row.
• TestRulesDecidedWithoutData pins the closed schema's valid-key list for a lookup, which this branch adds coverage to.
• dimensions.md was rewritten around this branch's edit; the surviving sentence took it.

What changed after review

An earlier revision also refused coverage: masked at load wherever it stood as a bound or a divisor. That is gone. It refused examples/ports/pypsa_losses.yaml in the lpspec corpus, whose loss variable is masked on loss_max and bounded by it:

loss: { foreach: [snapshot, line], where: loss_max, bounds: { upper: loss_max } }

Reproduced here before removing it — and the message told the author to "move the mask onto the declaration that wants it, as a where:", which that model already does. Narrowing it to the case where the where: names the same parameter is decidable (the lowered where is Mask(root=ParameterDefinedNode('loss_max'))), but it only moves the false refusal: a variable masked indirectly is still refused, and the file cannot say whether it is covered. Whether a bound has a row is a question about rows, so it belongs to whatever binds — where the check is row-precise and already exists.

That guard was the one place this PR broke its own division. Removing it is most of the src diff.

In its place, the refusal points: was already the answer to. A ragged curve — three breakpoints for one generator, five for another — is not something a parameter declares:

parameter 'bx': 'coverage:' is not for a parameter a piecewise block consumes — 'curve'
already owns the shape of its curve. A breakpoint 'points:' leaves out declares no weight
and its values are not asked for, so the table is total over the points the block admits.
Drop the 'coverage:' line, and say how far the curve runs with 'points:'.

Both spellings are refused, not just masked: total there would claim every coordinate the dims reach, which is exactly what a short curve does not carry. Without this, a file could say "the missing row is meant" while points: and the curve-extent check still refuse it — a contradiction the review found, and the language already had the third state.

The two spellings of coverage: are now one. ParameterBlock and LookupBlock share _CoveredBlock, so the field is Coverage | None = None on both and coverage_or_default is what consumers ask. Needed anyway: the piecewise refusal has to know whether the file wrote the key.

What changed after the second review

• The six pypsa set-points declare coverage: masked. Generator_p_set, Link_p_set, StorageUnit_p_set, StorageUnit_state_of_charge_set, Store_e_set and Line_s_set are consumed only as where: masks and their descriptions already said a component without one has no row; under the default the example would refuse to bind on any network where most components carry no schedule. Only examples/pypsa.yaml declares them.
• absence.md, declarations.md and dimensions.md say one thing. absence.md still said, unconditionally, that a missing parameter row reads as the value that contributes nothing; it now says that is the masked reading and a bind-time error naming the coordinate is the total one. declarations.md said a masked gap "is absence", which under absence.md's own rules would spread through arithmetic and take the row — the earlier revision of this PR tested a-term-reads-as-zero, and lpspec reads it that way, so the page now says "reads as the value that contributes nothing" and leaves the table of what creates absence as it was. dimensions.md agrees at both places it spoke.
• The schema enum test gains a row for each coverage field, and the unread COVERAGE set is deleted.

What changed after the third review

A parameter a curve owns answers for no coverage. Load refuses coverage: on a parameter a piecewise: block consumes — points: says how far each curve runs — but that refusal is a fact about the file, and whatever binds the table reads the program. Lowering wrote the unwritten default there, so the program said a ragged breakpoint table must carry every coordinate its dims reach. Against lpspec's examples/piecewise_ragged.yaml, which declares bp_x and bp_y over [generator, bp] with no two curves the same length, before and after:

before  {'p_max': 'total', 'load': 'total', 'bp_x': 'total', 'bp_y': 'total', 'cost_curve_points': 'total'}
after   {'p_max': 'total', 'load': 'total', 'bp_x': None,    'bp_y': None,    'cost_curve_points': None}

A binder that trusted the first refuses a model the block admits. ParameterDeclaration.coverage is Coverage | None, and lowering passes None for the names a block consumes or emits — the emitted mask is derived from a values parameter and is exactly as ragged as it is, so it needs the same answer. The consumed set is one property on PiecewiseBlock, read by the load-time refusal and by lowering rather than spelled twice. Reproduced as a failing test first: the three names came back 'total'.

_NO_READING_FOR_ABSENCE is deleted — the last trace of the bound/divisor guard this branch removed, defined and read nowhere, and its comment still citing a rule the guard no longer applies.

Verified after the rebase

pixi is not installed in the session that rebased this, so the gates were run tool by tool in a plain venv at the pinned ruff==0.16.1 and pyrefly==1.2.0, on df2da95, clean worktree, base eac1930:

Gate	Result
ruff check / ruff format --check	pass, 127 files
pyrefly check	0 errors (9 suppressed)
reuse lint	compliant, 203 / 203
pytest -q	1163 passed, 6 skipped — origin/main in the same venv: 1155 passed, 6 skipped
mkdocs build --strict	pass
prettier --check on every changed page	pass

schema/math-spec.schema.json and tests/typesetting/golden/*.out were regenerated and are byte-identical to what the branch already carried. The schema diff is the parameter's coverage taking the shape the lookup's already had — anyOf [enum, null], default: null. The published schema therefore no longer advertises default: "total" for it; the reading lives in coverage_or_default and in the language reference, as it already did for lookups.

Not run: compile-tex — tectonic and typst are absent here, and the golden output did not move. The 6 skips are those two. The strict docs build had the one intersphinx inventory (https://docs.python.org/3/objects.inv) removed from mkdocs.yml for the run and restored after: the proxy answers it 403, and it is the only thing that failed the gate. pixi run ci itself was not run.

Every guard deleted in turn — measured before the rebase, at badd60f

Applied to a clean tree, reverted between. The counts are the pre-rebase suite, and the label-space row is the refusal #422 has since made unreachable:

Mutation	Result
the curve-parameter refusal never runs	2 failed, 1026 passed
the label-space refusal never fires	1 failed, 1027 passed
the unwritten default flips to masked	3 failed, 1025 passed
lowering drops the lookup's declaration	1 failed, 1027 passed
lowering drops the parameter's declaration	1 failed, 1027 passed
restored	1028 passed

The last one survived on the first attempt at 1028 passed — ruff-format had reflowed the call across lines, so the edit silently matched nothing. Re-applied against the formatted source it fails as it should. Worth recording: a mutation that "survives" is a claim about the edit before it is a claim about the suite.

What lpspec has to do

Per the review on that side:

• The check goes in sources.py, not the engine. _short_parameters lives in the polars engine only, so a check there would be lane-asymmetric; tidy_sources is the door both lanes enter.
• lookup_relations gains one check after _check_keys_are_labels, which already holds both the labels and the rows.
• diagnostics().sparse_parameters dies. It reports rather than judges precisely because nothing said which parameters could be either; with total an error and masked declared there is no middle rung left. That field, its engine block, its docs row and its tests go — the one place the stack makes lpspec smaller.
• The corpus: 28 of 288 parameter declarations across 15 of 43 referenced models, plus bench/ and differential/pypsa/, unmeasured. Three of the 28 are block-owned — piecewise_ragged's bp_x, bp_y and cost_curve_points — and declare nothing at all. The other 25 write coverage: masked, piecewise_conversion's bp_rate and genx_piecewise_fuel's fuel_slope/fuel_intercept among them: they read like breakpoints and no piecewise: block consumes them, so the exemption does not reach them.
• The lookup side costs one line, re-checked against the bound data rather than the file: the fixture's relation() helper turns each null into a missing row, and of the four lines line_from maps all four while line_to maps three. So line_to declares coverage: masked and line_from is left as it is — 1 of 63 into: lookups in that corpus.

Deliberately not done: the bind-time check itself, which is the consumer's. coverage is not typeset: two files differing only in it render identically in all three formats, which is the precedent dtype already sets — the legend states the model, not the contract its tables must meet. A row in the legend is a follow-up if a reader of the rendered document turns out to need it. Base: main; #242 has merged, and its overclaim on unsupplied dimensions — also from this review — is #378, not this PR. #244 and #249 still need retargeting off claude/parameter-coverage.

[read-the-docs-community]

Documentation build overview

📚 math-spec | 🛠️ Build #34579598 | 📁 Comparing 00d31ae against latest (a9237ec)

  🔍 Preview build  

58 files changed · + 10 added · ± 44 modified · - 4 deleted

+ Added

• about/limits/index.html
• about/what-counts-as-public-api/index.html
• examples/commitment/index.html
• howto/check/index.html
• howto/installation/index.html
• howto/print/index.html
• howto/regimes/index.html
• reference/language/reported/index.html
• reference/math_spec/exclusivity/index.html
• reference/math_spec/separability/index.html

± Modified

• index.html
• CHANGELOG/index.html
• contributing/index.html
• examples/index.html
• about/what-counts-as-language/index.html
• examples/dispatch/index.html
• examples/operators/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• examples/pypsa_losses/index.html
• and 34 more...

- Deleted

• installation/index.html
• about/ceiling/index.html
• reference/math_spec/expression_parser/index.html
• reference/math_spec/where_parser/index.html

[FBumann]

@brynpickering @FabianHofmann opinions?

[brynpickering]

@FBumann I'm not sure it's needed for numeric parameters. I expect them to always be at least partially masked. I agree that some cases you want all data to be there (a boolean membership array, a string lookup that maps assets to buses). I wonder whether the more generalised checks top-level key that I would like to reintroduce (#11) would be sufficient for users to catch these cases as necessary. It would allow for more power over the data checks made, as it could also enforce data ranges as well as existence of data.

checks
  all_gen_bus:
    where: not gen_bus
    errors: raise
    message: "all generators must map to a bus"
  p_max_min:
    where: p_max < 0
    errors: raise
    message: "Cannot have negative `p_max`"

Perhaps for total coverage it could become cumbersome.

If we have multi-dims then total coverage would make more sense to be able to validate against. The multi-dim itself (e.g. connection) would be a subset of other dims ([entity, bus]) but anything indexed over it would need to have total coverage (e.g. sign).

[FBumann]

@brynpickering Such a check syntax is kind of another mini lang inside the model, especially the error stuff etc. Not sure about wether this should be in scope.

How would we treat that in typeseting? total/masked is pretty straight forward.

[brynpickering]

How would we treat that in typesetting?

The where can be parsed exactly as other where, so only the message and errors keys are new. You could have a checks section of the typesetting that writes out the where in the same way as for the other math blocks and just have a description that says "if ... |warn/error| |message|".

[brynpickering]

On where total/masked can break down is where >1 lookups/parameters together define total. This could be managed with a check:

dimensions:
  asset: ...
lookups:
  gen_bus: {over: asset, into: bus}  # will be masked to just generators
  store_bus: {over: asset, into: bus}  # will be masked to just storage devices
checks
  all_bus:
    where: not (gen_bus or store_bus)
    errors: raise
    message: "all assets must map to a bus"

[FBumann]

On where total/masked can break down is where >1 lookups/parameters together define total. This could be managed with a check:

dimensions:
  asset: ...
lookups:
  gen_bus: {over: asset, into: bus}  # will be masked to just generators
  store_bus: {over: asset, into: bus}  # will be masked to just storage devices
checks
  all_bus:
    where: not (gen_bus or store_bus)
    errors: raise
    message: "all assets must map to a bus"

Good point!

Could we open an issue discussing the scope of this checks feature more explicitly? Which operations need to be possible?
How much would it extend the surface of Spec and Program?
How would we handle it in typesetting?

I would close this PR for now if you agree @brynpickering

[brynpickering]

I would close this PR for now if you agree @brynpickering

Sure, keep the branch around in case we decide it is actually the best approach after diving deep on checks.

I'll work on a fleshed out checks issue.

[FBumann]

@brynpickering I dived deeper into checks. #471
I think having the coverage on the parameter buys a lot of strictness and silent errors. If we assume coverage=total by default, users need to opt into sparsity.

With checks/ assumptions we have the logic the other way round: Its always assumed sparse until someone declares an assumption of it being total.

[FBumann]

Prompt: "Resolve #245's conflicts"

Note

The following content was generated by AI.

Merged origin/main at 5656b11 into the branch (16034ce). A merge rather than a rebase, because this is not a branch I created. Three reference pages conflicted, each where main had rewritten the same passage this branch edits. Main's prose wins in every case; this branch's semantics are folded into it.

What each conflict was, and how it was settled

absence.md — main had rewritten the "nothing else creates absence" paragraph into two, naming the four positions explicitly. Kept that shape, carrying this branch's coverage split into the first paragraph and its "the bind is refused" into the second. Main still said "loading is refused" there, which this branch deliberately corrects: whether a bound has a row is a question about rows.

declarations.md — main had rewritten the parameter field table and expanded dtype into a table plus three paragraphs, which supersedes this branch's single dtype paragraph. Took main's table and dtype section, added the coverage row, and kept this branch's four coverage paragraphs and the worked example between them.

dimensions.md — three hunks. Main had reorganised "where the members come from" into three numbered rules, which say what this branch's four paragraphs said, so main's list stands. Rule 3 needed one amendment and it is the load-bearing one:

At most once, not exactly once: a coordinate with no row is how a parameter declared coverage: masked masks, and an error at the bind for one declared total. Which of the two it is, the declaration says and the table does not.

Main's sentence there read "a coordinate with no row is absence, and absence is how a model masks" — which absence.md stops saying on this branch, so leaving it would have left the two pages disagreeing. The other two hunks took main's side outright: its select-only-dimension paragraph, and its "a partial map is exactly the rows it has". The partial-lookup paragraph then carries this branch's lookup example.

One thing added that was not a conflict: the lookup field table did not list coverage, though LookupBlock takes it. It does now, matching the parameter table.

One semantic conflict git did not flag. tests/test_lowering.py and tests/test_validation.py both declare variables.s with foreach:, which #429 renamed to dims: on main. The merge was clean and four tests failed:

FAILED tests/test_lowering.py::test_a_parameter_a_curve_owns_answers_for_no_coverage
FAILED tests/test_validation.py::TestLookupCoverage::test_a_curve_and_its_parameters_load_when_neither_declares_coverage
FAILED tests/test_validation.py::TestLookupCoverage::test_coverage_on_a_parameter_a_curve_consumes_is_refused
FAILED tests/test_validation.py::TestLookupCoverage::test_total_is_refused_there_too_rather_than_read_as_agreement

Both fixtures now say dims:.

Verified on the merged tree at 16034ce, clean worktree

pixi is not installable in this session, so the gates were run tool by tool in a uv venv at the pinned ruff==0.16.1 and pyrefly==1.2.0:

Gate	Result
pytest -q -n auto	1214 passed, 6 skipped
ruff check / ruff format --check	pass, 131 files
pyrefly check	0 errors (9 suppressed)
prettier --check on the changed pages	pass
mkdocs build --strict	pass — no dead link, despite main's docs restructure

pyrefly reports 3 errors in a venv without types-PyYAML, which pixi's environment provides; with it installed the count is 0.

The strict docs build needed the docs.python.org inventory dropped into a throwaway config, because this session's proxy answers that host 403. mkdocs.yml is untouched, and the throwaway was deleted. The schema and the typeset goldens did not need regenerating — test_schema.py and test_docs.py pass as merged.

Not run: compile-tex, reuse, typos, taplo, zizmor. prettier --check over the whole tree flags mkdocs.yml, which is equally unformatted on main and is not this branch's to fix.

Worth deciding before this lands, and not a merge question. #471 stacks assumptions: onto the same ground: a missing row reads as false there too, so any assumption naming a parameter refuses a short table, and where: on an assumption says a gap was meant. The two mechanisms differ in default — coverage: total catches every short table with no author action, while an assumption catches only what somebody wrote a predicate about, and a predicate that folds to a literal is refused, so "this table is complete" has no spelling of its own there. Neither PR mentions the other. Both close work on #296.

---

Generated by Claude Code

[FBumann]

Note

The following content was generated by AI.

Brought up to the alpha.90 release as c5601e7. One part was rewritten rather than merged, and it takes a reading you should confirm.

#437 replaced lookups: with keyed relations:, so the over:/into: map this branch declared coverage: on no longer exists. coverage: now sits on RelationBlock through the same _CoveredBlock the parameter uses, and RelationDeclaration carries it into the program.

The reading I had to choose. A relation declares no direction, so total can no longer mean "every label of the dimension it is over". As merged it means: the table carries a row for every coordinate of its key:, and where there is no key, for every combination of its columns' dimensions.

The keyless case is the one worth arguing about. A bare relation is many-to-many, so a total claim over the full product is rarely what a file means — refusing coverage: on a keyless relation may be the better answer. I did not add that refusal, because it is new language surface and this was meant to be a merge.

The parameter half is untouched by the rename and merged as it stood.

Coverage moved, and what was verified

• TestLookupCoverage is TestRelationCoverage; it declares columns:/key: and reads program.dimensions['g'].relations.
• The schema enum row is RelationBlock, not LookupBlock.
• dimensions.md gains a coverage row in the relation field table and a section stating the key-based reading; the absence paragraph says relation rather than lookup.
• examples/pypsa.yaml keeps its six masked set-points.

pixi is not installable in this session, so gates ran from a uv venv on Python 3.12 at the pinned ruff==0.16.1 and pyrefly==1.2.0:

• pytest -q --ignore=tests/test_docs.py: 1240 passed, 6 skipped.
• ruff check, ruff format --check (131 files), pyrefly check (0 errors, 9 suppressed): clean.
• Schema, goldens and all four page generators regenerated to no drift.

Not run: pixi run ci as one command, reuse, typos, taplo, zizmor, docs-build --strict, compile-tex, and tests/test_docs.py, whose site-render cases need mkdocs plugins this venv lacks.

The merge is against 0b4f046 rather than main's tip, because the tip then carried #474, which #481 has since reverted. From 0b4f046 to main today the only content change is CHANGELOG.md and the version manifest, and this branch merges main cleanly.

---

Generated by Claude Code

[FBumann]

Superseeded by #671