feat(language)!: missing: says what a missing row means, a table short of a row is refused unless the file says otherwise, and a variable's absence: is now missing:

[FBumann]

Prompt: "I think we should unify the concept under missing:

The options are:

1. error
2. absent
3. neutral (zero)
4. a default value (scalar/bool)"

"I think we should do all of this in one combined PR."

Then: "I think "refused" is better! It also allows the engine etc to warn raise or wathever...?", and "Do we expose the missing literals properly ?"

Follow-up: 810: fix it. If we can typeset in a meaningful way we should, but we probably can't. It's more about the datacontract I think

Follow-up: why not demand the user to declare it instead? No smart default! Error instead.

Follow-up: Add it too

Note

The following content was generated by AI.

One key, missing:, says what a missing row means. It replaces a variable's absence: and #671's coverage:. Parameters and relations default to refused, so a row lost in data prep is refused when the data is attached.

Closes #773. Closes #790. Closes #296. Supersedes #671 (closed). #824 and #826 are stacked on this PR.

What this changes, breaks, gate output, departures

What this changes

missing:	A missing row	Parameter	Relation with values:	Variable
refused	is refused when the data is attached	default	default	refused
absent	is absence: it takes the row, and is one summand fewer in a sum	yes	yes	default
neutral	reads 0 as a coefficient and false in a where	yes	refused	yes, needs where:
a value	reads that value wherever a value is read	yes, of its dtype	refused	refused

refused completes "a missing row is ___" as absent and neutral do, and is the docs' word for data a consumer rejects. "Error" stays the word for what to_spec raises. A consumer does not build the model from data with a refused row; how it says so (raise at the first gap, list every coordinate) is its own.

• Loader (spec.py). ParameterBlock.missing, with .reading giving refused where the file wrote nothing. RelationBlock.missing, with .reading giving None for a bare relation, whose rows are its membership; missing: on a bare relation is refused. VariableBlock.missing replaces absence: (undefined → absent, zero → neutral). A value has the dtype: any number for float, an integer for int (no inf), true/false for bool, none for str. inf and -inf load as the same floats as .inf and -.inf. missing: null (on all three kinds), NaN, a quoted number, an unknown word, and a reading a kind does not take are refused at load, each naming the fix. A given: parameter takes no missing:.

• Program. ParameterDeclaration.missing, RelationDeclaration.missing, VariableDeclaration.missing; the types Missing, MissingReading, RelationMissing, VariableMissing replace VariableAbsence in mathspec.program.__all__, beside VariableDomain and ParameterDtype. names_under is public, because lowering reads it too.

• Schema. Each kind publishes its readings as an enum and its unwritten reading as the default: refused for a parameter and a relation, absent for a variable. A parameter's inf/-inf spellings are their own branch. The closed-vocabulary test covers all three.

• Piecewise (ported from feat(language): a parameter and a relation each say whether their data must be complete, so a row lost in preparation is not read as a mask #671, then changed by the second follow-up). A curve's parameter takes missing: like any other. Where a declaration outside every curve also reads it and the file writes no missing:, the load is refused:

  parameter 'bp_y': piecewise 'cost_curve' reads it as breakpoints and constraint 'cost_cap' reads it too, so the file says what a missing row of it means. Declare missing: refused, absent, neutral, or a value of its dtype.
  

  The assumptions a method implies are the curve's and do not count; an assumptions: entry the file writes does. One reading is still inferred, because the curve fixes it: a values parameter that only curves with points: read, with no missing: written, is neutral. The curve stops where its mask stops, so the table has no rows past it by design and the curve reads none of them; refused would refuse the curve the block declares. piecewise.ragged names these; lowering reports them and the expansion declares them. A boolean points: mask is no longer inferred: a full true/false table is a mask as well, so it reads the default, refused. The <block>_complete assumption says "a missing row does not shorten the curve".

• Exclusivity prover. A comparison at a missing row reads the value. Before this change, efficiency == 1 and not efficiency were proved apart under missing: 1, although a missing row is in both. A bare name keeps the null cell, and so does a refused parameter, which can only add witnesses. A witness says efficiency has no row, not is absent, so it does not use the name of a reading. The soundness fuzz draws a value per parameter.

• Typesetter. The legend prints every reading but the default beside its declaration, in all three formats: a parameter's value as math (…, ∞ where the data has no row), a parameter's reading in monospace (…, `neutral` where the data has no row), a map with missing: absent beside each set it joins, and a variable's missing: neutral (…, `neutral` where the mask leaves it out). refused and a variable's absent print nothing, as an unwritten key does. format.number is shared by the walk and the legend, and prints -∞.

• Examples. Under the new default, every sparse table is marked. In examples/pypsa.yaml: 71 parameters missing: neutral, the ones whose description says a row may be missing (*_set, ramp limits including the start-up and shut-down ones, p_init, *_nom_mod, v_ang_max, maintenance durations, cycle and constraint weights, BODF, big_m, Carrier_max_growth), and the two partial outage maps Outage_line and Outage_transformer missing: absent. Six in pypsa_linearized_uc.yaml. The 24 fragments are regenerated with tools.pypsa_split, and check passes. No example has a curve parameter read outside its curve; the curve on reading.md declares bp_y missing: refused, because an assumption reads it.

• Docs. absence.md, declarations.md ("A missing row"), relations.md (data contract), expressions.md, piecewise.md ("Missing breakpoints"), reading.md, the glossary, the compose how-to and rule 8 on the language index. Schema, goldens and generated pages regenerated.

What breaks

A release with this raises the minor version.

• absence: on a variable is now missing:: undefined → absent, zero → neutral. No alias.
• A parameter or relation with no missing: is now refused: a consumer refuses a table short of a row. A file with a deliberately sparse table needs missing: neutral (the old reading) or missing: absent.
• A parameter a curve reads and a declaration outside the curve reads too needs missing:, or the load is refused.
• ParameterDeclaration and RelationDeclaration take missing before description, so a positional description moves one place.

Bare names on a refused parameter in the PyPSA example

A bare numeric name asks "has a row and is finite". Under refused every row is there, so it asks only "finite?". The six start-up and shut-down ramp tables (Generator_, Link_, Process_ramp_limit_start_up and _shut_down) were refused, so otherwise: 1 in *_start_up_rate was dead and (ramp_limit_up OR ramp_limit_start_up) was always true. They are now missing: neutral, the meaning they had on main and the one PyPSA gives a NaN ramp limit (no limit). The bare names left on a refused parameter are *_nom_max, Generator_e_sum_min and Generator_e_sum_max; PyPSA writes infinity where no cap is meant, so "finite?" is what they mean.

Gates

Run on 475ac3e5:

• pixi run lint: passed.
• pixi run test: 2710 passed.
• Every generator re-run (tests.typesetting.golden, tools.gallery, tools.notation, tools.home_math, tools.spec_math, tools.expansion_math, tools.pypsa_split).
• pixi run docs-build: blocked here. The proxy refuses docs.python.org/3/objects.inv, and the only warning is the pathlib.Path autoref that inventory resolves. CI runs it.
• pixi run compile-tex: blocked here. Tectonic cannot download its bundle. CI runs it.

Guards, each deleted in turn (on 5f63706 unless noted):

guard deleted	suite
the prover reading a value	1 failed
the refusal of a curve parameter read outside the curve with no missing: (on 03ba7392)	2 failed
lowering reading a points: values parameter as neutral (on 03ba7392)	2 failed
the expansion writing neutral	3 failed
the unwritten default being refused (flipped to neutral)	4 failed
a bare relation reading None	1 failed
the refusal of missing: on a bare relation	1 failed
the refusal of neutral and values on a relation	2 failed
the refusal of refused and values on a variable	2 failed
the refusal of missing: null on a parameter / on a variable	1 failed / 1 failed
the refusal of missing: null on a relation (on 7a2449af)	1 failed
the schema publishing refused as the default (on 7a2449af)	2 failed
the refusal of an unknown word	2 failed
inf as a number	6 failed
the witness saying has no row (on bf743d07)	1 failed
the legend printing a reading (on 714aa67c)	3 failed
the legend printing a partial map (on 714aa67c)	9 failed
the legend skipping the default reading (on 475ac3e5)	31 failed

The fuzz does not catch a deleted value read in the prover: it checks that the cells cover the values, and both of its sides read through the same function. TestAMissingRow catches it.

Departures and things deliberately not done

• One PR for three issues, as you decided, against "one issue, one PR".
• missing: null is refused, although upper: null loads because every open field takes null. Here null could mean the default (refused) or an absent row, so the refusal names both words.
• The examples are marked by their descriptions, not by attaching data. No tool in this repository attaches PyPSA data, so a sparse table whose description does not say so is still refused. feat(language): a parameter and a relation each say whether their data must be complete, so a row lost in preparation is not read as a mask #671's list was the starting point; this branch's examples have grown since.
• The outside-read refusal also covers a curve with no points:, where the default refused would serve the curve and the outside row alike. The rule is stated once, for every curve.
• No load-time refusal of an inf value on a coefficient. Whether a row is missing depends on the data.
• The missing types are not exported from the top-level mathspec package, as no vocabulary type is; they live in mathspec.program.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CsaSWMofYqJiKmnSiPycV9

[read-the-docs-community]

Documentation build overview

📚 mathspec | 🛠️ Build #34908951 | 📁 Comparing 475ac3e against latest (69f673d)

  🔍 Preview build  

39 files changed · ± 39 modified

± Modified

• changelog/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• howto/compose/index.html
• howto/see-an-expansion/index.html
• reference/glossary/index.html
• reference/language/index.html
• reference/notation/index.html
• reference/program/index.html
• reference/reading/index.html
• and 29 more...

[FabianHofmann]

nice!

[FBumann]

@FabianHofmann We need to agree on a default! "error" will the most strict/safe, but might be cumbersome.
Haven't made my mind up yet. Maybe we should rebase after you merged the pypsa docs and check how it fits...?

EDIT: I think refused is the best word here. It leaves some room for how a refusal is done (raise, warn, collect then raise combined etc). It seems more neutral in behavior of the engine