docs(examples): the PyPSA file states a constraint once where PyPSA builds one row set, rather than a block per regime

[FBumann]

Prompt: We should do a stacked PR updating all pypsa models that need this feature to actually use it. To bring the remaining pypsa ladder work close to 0. See open issues #124, #123

Note

The following content was generated by AI.

Stacked on #168 — it adds cases:; this is the port that needed it. Both were rebased onto main at e412e0b, after #271 and #273 landed in the same file; the counts below were re-derived on that base and are unchanged.

Twelve PyPSA names — sixteen, counted once per file they appear in — were stated by two to four blocks apiece, split by a regime the language could not name inside a quantity. Now each is one block, and the two examples lose 18 and 4 constraints without losing a row. That is #123's failure class — same math, different bookkeeping — closed for every name cases: can reach.

	pypsa.yaml	pypsa_linearized_uc.yaml
constraint blocks	116 → 98	25 → 21
names stated by more than one block	17 → 5	4 → 0

The five that remain are the GlobalConstraint sense-splits, where the split is in the operator rather than in a value. cases: is value-level and cannot reach them; that is #239, untouched here.

What replaced each split

Eight cased quantities, and the regime moves into them:

StorageUnit_charge_carried_in:
  foreach: [snapshot, storage_unit]
  cases:
    cyclic:
      when: StorageUnit_cyclic_state_of_charge
      expression: StorageUnit_retention * shift(StorageUnit_state_of_charge, over=snapshot, offset=1, edge='wrap')
    opening:
      when: not StorageUnit_cyclic_state_of_charge AND position(snapshot) == 0
      expression: StorageUnit_state_of_charge_initial
  otherwise: StorageUnit_retention * shift(StorageUnit_state_of_charge, over=snapshot, offset=1)

so StorageUnit-energy_balance is one block with no where: at all, where it was three. #124's opening question — "what about storages having cyclic and non cyclic?" — is that case list.

The others: Generator_previous_status and Generator_previous_p carry the boundary, Generator_p_nom_effective and Link_p_nom_effective carry extendable-or-not, and Generator_ramp_up_allowance / _down_allowance carry committed-or-not — which is what lets Generator-p-ramp_limit_up go from four blocks to one.

Verified

pixi run ci green — lint, 886 tests, mkdocs build --strict, 27 compiled documents.

Beyond that, the collapse is only correct if the rows did not move, so both halves were checked exhaustively against HEAD rather than argued. For each PyPSA name, its blocks' where: strings are parsed with the repo's own parse_where, every assignment of their atoms is enumerated (× the first snapshot and any other), and a row is taken to exist where the mask holds and no term is absent — absence being a bare shift(… over=snapshot …) at the first snapshot, followed through each cased expression into the region that world selects.

• The masks. All 16 names build a row in exactly the same worlds as before. The check also asserts each world is claimed by exactly one block, which is what says the old blocks never double-built a row.
• The coefficients. In every world that carries a row, each cased quantity is substituted by the value its region gives there, shift() is held as an opaque symbol, and both sides are compared as polynomials rather than as text. Every row is the row the old blocks built.

Both runs

=== examples/pypsa.yaml ===
  same rows Generator-com-transition-shut-down: 2 block(s) -> 1, 2/4 worlds carry a row
  same rows Generator-com-transition-start-up: 2 block(s) -> 1, 2/4 worlds carry a row
  same rows Generator-p-ramp_limit_down: 4 block(s) -> 1, 7/32 worlds carry a row
  same rows Generator-p-ramp_limit_down-run-bigM: 2 block(s) -> 1, 3/32 worlds carry a row
  same rows Generator-p-ramp_limit_down-shut-bigM: 2 block(s) -> 1, 3/32 worlds carry a row
  same rows Generator-p-ramp_limit_up: 4 block(s) -> 1, 7/32 worlds carry a row
  same rows Generator-p-ramp_limit_up-run-bigM: 2 block(s) -> 1, 3/32 worlds carry a row
  same rows Generator-p-ramp_limit_up-start-bigM: 2 block(s) -> 1, 3/32 worlds carry a row
  same rows Link-p-ramp_limit_down: 2 block(s) -> 1, 2/8 worlds carry a row
  same rows Link-p-ramp_limit_up: 2 block(s) -> 1, 2/8 worlds carry a row
  same rows StorageUnit-energy_balance: 3 block(s) -> 1, 4/4 worlds carry a row
  same rows Store-energy_balance: 3 block(s) -> 1, 4/4 worlds carry a row
  12 names checked
=== examples/pypsa_linearized_uc.yaml ===
  same rows Generator-com-transition-shut-down: 2 block(s) -> 1, 2/4 worlds carry a row
  same rows Generator-com-transition-start-up: 2 block(s) -> 1, 2/4 worlds carry a row
  same rows Generator-p-ramp_limit_down: 2 block(s) -> 1, 3/16 worlds carry a row
  same rows Generator-p-ramp_limit_up: 2 block(s) -> 1, 3/16 worlds carry a row
  4 names checked
every PyPSA name builds the same rows as before

=== examples/pypsa.yaml ===
  same math Generator-com-transition-shut-down: 2 row-worlds, 2 block(s) -> 1
  same math Generator-com-transition-start-up: 2 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_down: 7 row-worlds, 4 block(s) -> 1
  same math Generator-p-ramp_limit_down-run-bigM: 3 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_down-shut-bigM: 3 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_up: 7 row-worlds, 4 block(s) -> 1
  same math Generator-p-ramp_limit_up-run-bigM: 3 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_up-start-bigM: 3 row-worlds, 2 block(s) -> 1
  same math Link-p-ramp_limit_down: 2 row-worlds, 2 block(s) -> 1
  same math Link-p-ramp_limit_up: 2 row-worlds, 2 block(s) -> 1
  same math StorageUnit-energy_balance: 4 row-worlds, 3 block(s) -> 1
  same math Store-energy_balance: 4 row-worlds, 3 block(s) -> 1
=== examples/pypsa_linearized_uc.yaml ===
  same math Generator-com-transition-shut-down: 2 row-worlds, 2 block(s) -> 1
  same math Generator-com-transition-start-up: 2 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_down: 3 row-worlds, 2 block(s) -> 1
  same math Generator-p-ramp_limit_up: 3 row-worlds, 2 block(s) -> 1
16 names checked
every row the new blocks build is the row the old blocks built

rowsets.py — the masks

"""Prove the collapse kept every row: old union of masks == new, per PyPSA name.

A block builds a row where its `where:` holds and every term it leans on has a
value there. The only term without one is a bare `shift(... over=snapshot ...)`
at the first snapshot — which is what the `_initial` blocks existed to supply,
and what a cased expression's `opening:` region supplies instead.
"""
import itertools, re, subprocess, sys, yaml, collections
from math_spec.where_parser import (parse_where, BooleanLiteralNode, UnresolvedNameNode,
    UnresolvedComparisonNode, UnresolvedPositionNode, NotNode, AndNode, OrNode)

POSITIONS = (0, 1)                       # the first snapshot, and any other
BARE_SHIFT = re.compile(r"shift\((?![^()]*edge=)[^()]*over=snapshot[^()]*\)")
OPS = {'==': lambda a, b: a == b, '!=': lambda a, b: a != b, '>': lambda a, b: a > b,
       '<': lambda a, b: a < b, '>=': lambda a, b: a >= b, '<=': lambda a, b: a <= b}

def atoms(node, acc):
    if isinstance(node, UnresolvedNameNode): acc.add(('name', node.name))
    elif isinstance(node, UnresolvedComparisonNode): acc.add(('cmp', node.name, node.op, node.value))
    elif isinstance(node, (UnresolvedPositionNode, BooleanLiteralNode)): pass
    elif isinstance(node, NotNode): atoms(node.operand, acc)
    elif isinstance(node, (AndNode, OrNode)): atoms(node.left, acc); atoms(node.right, acc)
    else: raise TypeError(f'unhandled {type(node).__name__}')
    return acc

def ev(node, env, pos):
    if isinstance(node, BooleanLiteralNode): return node.value
    if isinstance(node, UnresolvedNameNode): return env[('name', node.name)]
    if isinstance(node, UnresolvedComparisonNode): return env[('cmp', node.name, node.op, node.value)]
    if isinstance(node, UnresolvedPositionNode):
        assert node.dimension == 'snapshot' and node.by is None
        op = node.op.value if hasattr(node.op, 'value') else str(node.op)
        return OPS[op](pos, node.position)
    if isinstance(node, NotNode): return not ev(node.operand, env, pos)
    if isinstance(node, AndNode): return ev(node.left, env, pos) and ev(node.right, env, pos)
    if isinstance(node, OrNode): return ev(node.left, env, pos) or ev(node.right, env, pos)
    raise TypeError(type(node))

def absent(text, exprs, env, pos, seen=()):
    """Whether a term written as `text` has no value in this world."""
    text = str(text)
    if pos == 0 and BARE_SHIFT.search(text): return True
    for name, e in exprs.items():
        if name in seen or not re.search(rf'\b{name}\b', text): continue
        if isinstance(e, str): 
            if absent(e, exprs, env, pos, seen + (name,)): return True
            continue
        if e.get('cases'):
            for case in e['cases'].values():
                if ev(parse_where(case['when']), env, pos):
                    value = case['expression']; break
            else:
                value = e['otherwise']
        else:
            value = e.get('expression', '')
        if absent(value, exprs, env, pos, seen + (name,)): return True
    return False

def blocks(spec_text):
    d = yaml.safe_load(spec_text)
    exprs = {k: v for k, v in (d.get('expressions') or {}).items()}
    out = collections.defaultdict(list)
    for name, b in (d['constraints'] or {}).items():
        m = re.search(r'`([^`]+)`', b.get('description', '') or '')
        if m:
            w = parse_where(b['where']) if b.get('where') else BooleanLiteralNode(True)
            out[m.group(1)].append((name, w, b['expression'], exprs))
    return out

def truth(group, universe):
    got = set()
    for _, w, text, exprs in group:
        for env, pos in universe:
            if ev(w, env, pos) and not absent(text, exprs, env, pos):
                got.add((tuple(sorted(env.items())), pos))
    return got

def main():
    fail = 0
    for f in ('examples/pypsa.yaml', 'examples/pypsa_linearized_uc.yaml'):
        old = blocks(subprocess.run(['git', 'show', f'HEAD:{f}'], capture_output=True, text=True, check=True).stdout)
        new = blocks(open(f).read())
        assert set(old) == set(new), f'{f}: PyPSA names differ: {set(old) ^ set(new)}'
        print(f'=== {f} ===')
        checked = 0
        for name in sorted(old):
            onames, nnames = [b[0] for b in old[name]], [b[0] for b in new[name]]
            if onames == nnames and old[name][0][2] == new[name][0][2]: continue
            every = sorted(set().union(*(atoms(b[1], set()) for b in old[name] + new[name])))
            universe = [(dict(zip(every, vals)), pos)
                        for vals in itertools.product([False, True], repeat=len(every))
                        for pos in POSITIONS]
            o, n = truth(old[name], universe), truth(new[name], universe)
            checked += 1
            if o != n:
                fail += 1
                print(f'  DIFFERS   {name}: only-old={sorted(o - n)[:2]} only-new={sorted(n - o)[:2]}')
            else:
                print(f'  same rows {name}: {len(onames)} block(s) -> {len(nnames)}, '
                      f'{len(o)}/{len(universe)} worlds carry a row')
        print(f'  {checked} names checked')
    print('FAILED' if fail else 'every PyPSA name builds the same rows as before')
    sys.exit(1 if fail else 0)


if __name__ == '__main__':
    main()

algebra.py — the coefficients

"""Prove the collapse kept the math: in every world, the one new row is the old row.

Each cased expression is substituted by the value its region gives in that
world, `shift()` is held as an opaque symbol, and both sides are compared as
polynomials rather than as text.
"""
import ast, itertools, re, subprocess, sys, yaml, collections, fractions
sys.path.insert(0, '/private/tmp/claude-501/-Users-felix-PycharmProjects-math-spec/0a8dbda6-e2f9-453d-a6d1-4f142cabb7f9/scratchpad')
from rowsets import (atoms, ev, absent, blocks, POSITIONS, parse_where)

SHIFT = re.compile(r"shift\(\s*([A-Za-z_]\w*)\s*,[^()]*?\)")

def hold(text):
    """Hold every shift as one symbol — wrap and plain are different terms."""
    return SHIFT.sub(lambda m: ('wrap__' if 'wrap' in m.group(0) else 'prev__') + m.group(1), str(text))

def poly(node):
    """{multiset of symbols: coefficient} for an arithmetic node."""
    if isinstance(node, ast.Constant): return {(): fractions.Fraction(node.value)}
    if isinstance(node, ast.Name): return {(node.id,): fractions.Fraction(1)}
    if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub):
        return {k: -v for k, v in poly(node.operand).items()}
    if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.UAdd): return poly(node.operand)
    if isinstance(node, ast.BinOp):
        l, r = poly(node.left), poly(node.right)
        if isinstance(node.op, ast.Add):
            out = collections.Counter(l); out.update(r); return {k: v for k, v in out.items() if v}
        if isinstance(node.op, ast.Sub):
            out = collections.Counter(l); out.subtract(r); return {k: v for k, v in out.items() if v}
        if isinstance(node.op, ast.Mult):
            out = collections.Counter()
            for a, av in l.items():
                for b, bv in r.items(): out[tuple(sorted(a + b))] += av * bv
            return {k: v for k, v in out.items() if v}
        if isinstance(node.op, ast.Div):
            assert len(r) == 1, ast.dump(node)
            (rk, rv), = r.items()
            inv = tuple(sorted('inv__' + s for s in rk))
            return {tuple(sorted(a + inv)): av / rv for a, av in l.items()}
    raise TypeError(ast.dump(node))

REL = re.compile(r'(<=|>=|==)')

def relation(text):
    lhs, op, rhs = REL.split(hold(text), maxsplit=1)
    d = collections.Counter(poly(ast.parse(lhs.strip(), mode='eval').body))
    d.subtract(poly(ast.parse(rhs.strip(), mode='eval').body))
    return op, frozenset((k, v) for k, v in d.items() if v)

def expand(text, exprs, env, pos, seen=()):
    """Substitute every named expression by the value its region gives here."""
    text = str(text)
    for name, e in exprs.items():
        if name in seen or not re.search(rf'\b{name}\b', text): continue
        if isinstance(e, str): value = e
        elif e.get('cases'):
            for case in e['cases'].values():
                if ev(parse_where(case['when']), env, pos): value = case['expression']; break
            else: value = e['otherwise']
        else: value = e['expression']
        value = expand(value, exprs, env, pos, seen + (name,))
        text = re.sub(rf'\b{name}\b', f'({value})', text)
    return text

fail = checked = 0
for f in ('examples/pypsa.yaml', 'examples/pypsa_linearized_uc.yaml'):
    old = blocks(subprocess.run(['git', 'show', f'HEAD:{f}'], capture_output=True, text=True, check=True).stdout)
    new = blocks(open(f).read())
    print(f'=== {f} ===')
    for name in sorted(old):
        onames, nnames = [b[0] for b in old[name]], [b[0] for b in new[name]]
        if onames == nnames and old[name][0][2] == new[name][0][2]: continue
        every = sorted(set().union(*(atoms(b[1], set()) for b in old[name] + new[name])))
        worlds = [(dict(zip(every, vals)), pos)
                  for vals in itertools.product([False, True], repeat=len(every))
                  for pos in POSITIONS]
        rows = bad = 0
        for env, pos in worlds:
            o = [b for b in old[name] if ev(b[1], env, pos) and not absent(b[2], b[3], env, pos)]
            n = [b for b in new[name] if ev(b[1], env, pos) and not absent(b[2], b[3], env, pos)]
            if not o and not n: continue
            rows += 1
            assert len(o) == 1 and len(n) == 1, f'{name}: {len(o)} old / {len(n)} new rows in one world'
            ro = relation(expand(o[0][2], o[0][3], env, pos))
            rn = relation(expand(n[0][2], n[0][3], env, pos))
            if ro != rn:
                bad += 1
                if bad == 1:
                    print(f'  DIFFERS {name} at pos={pos} {env}')
                    print(f'     old {ro[0]}  {sorted(map(str, ro[1] ^ rn[1]))}')
        checked += 1
        fail += bool(bad)
        print(f'  {"BAD " if bad else "same"} math {name}: {rows} row-worlds, '
              f'{len(onames)} block(s) -> {len(nnames)}')
print(f'{checked} names checked')
print('FAILED' if fail else 'every row the new blocks build is the row the old blocks built')
sys.exit(1 if fail else 0)

What I could not check, and what it costs

The dual-parity gate did not run. It lives in lpspec's differential/pypsa/, which is not in this tree and is not checked out on this machine, so nothing here compares a solved objective or a row dual against pypsa 1.3.0. The two checks above prove the rows are the same rows; they do not prove an engine builds them.

lpspec needs a companion change, because ten constraint blocks are gone and four are renamed — Generator_p_ramp_limit_up_fix / _ext / _com become Generator_p_ramp_limit_up, and so on. Anything keyed on those names in references.json or deviations.yaml has to move with them. Worth landing before this, or beside it.

One thing the feature cannot say, worth knowing before more ports use it. expressions: takes no where:, so a cased quantity is total over its foreach: — Generator_previous_status is declared for every generator, and for a non-committable one it has no value, Generator_status being masked. Nothing here reads it there, and the constraints that use it all carry where: Generator_committable. But a consumer that materialises a named expression over its whole frame — which is what result.expression(name) promises — would meet that absence. Not filed; raising it here first.

Deliberately not done

• The five GlobalConstraint sense-splits (A global constraint's sense is data, so one requirement is three declarations differing only by an operator #239). Three blocks per type because <=, >= and == are data, and cases: gives a value per region, not an operator. A different feature.
• two bookkeeping claims about PyPSA's rows are still unproven: the meshed split, and a modular build that is not extendable #123's two standing items are untouched: the Bus-meshed-* split still has no fixture, and a fixed modular build still cannot be stated. Neither is a case-split.
• regimes no fixture reaches: multi-link ports past bus2, and the per-conjunct sweep that would find the rest #124's standing item — multi-link ports past bus2 — is untouched for the same reason.
• No renaming beyond the collapse. Blocks that already stood alone kept their names, so the diff is the fusion and nothing else.

[read-the-docs-community]

Documentation build overview

📚 math-spec | 🛠️ Build #34309156 | 📁 Comparing ecb755f against latest (a9237ec)

  🔍 Preview build  

24 files changed · + 2 added · ± 22 modified

+ Added

• examples/commitment/index.html
• reference/math_spec/exclusivity/index.html

± Modified

• CHANGELOG/index.html
• examples/index.html
• examples/pypsa/index.html
• examples/pypsa_linearized_uc/index.html
• examples/pypsa_losses/index.html
• examples/pypsa_multi_period/index.html
• examples/pypsa_quadratic/index.html
• examples/pypsa_stochastic/index.html
• reference/notation/index.html
• reference/language/expressions/index.html
• and 12 more...

[FBumann]

@FabianHofmann Do you want to review? I dont think its necessary. Just ping me quickly