[DISCUSS] differentiated masking within expressions

[brynpickering]

One of the hardest things we found in the implementation of the calliope math spec was how to handle application of macros/helpers in math when the expression's mask would mask out some of the data. Sometimes you want the mask to be applied to an expression before an operation is applied, sometimes you don't want the mask to be applied. Examples below.

The main questions I have are:

• How can we best flag protection of an expression from the mask in a macro/helper? Calliope approach isn't particularly satisfactory.
• Do we allow a mask macro/helper in the expressions to enable per sub-expression masking within a single larger expression? calliope doesn't yet allow full arbitrary where strings within the expression helper as it proved to be a pain to implement, but it does allow masking with a lookup.

You want a mask applied throughout

For instance, you might be summing over all technologies at nodes limited in a mask:

constraints:
  special_limit:
    foreach: [node]
    mask: special_nodes
    expression: sum(capacity, over=tech) < 5

In Calliope, if we now want to also mask the techs, we could mask or slice directly within the expression:

constraints:
  special_limit:
    foreach: [node]
    mask: special_nodes
    expression: sum(capacity[tech=special_techs], over=tech) < 5

constraints:
  special_limit:
    foreach: [node]
    mask: special_nodes
    expression: sum(mask(capacity, a_more_complex_lookup), over=tech) < 5

You don't want the mask applied

If you are shifting the timesteps or using a lookup to map between timesteps you want to do the shifting/mapping first and then apply the mask. E.g., if you are applying a constraint to the last timestep in each day, but it needs to take the value of the first timestep of that day in the applying the constraint, you can't mask with an input array final_ts_each_day as that would nullify the first timestep that we will ultimately map to.

Or, perhaps you want to apply a constraint over a subset of technologies and compare it to a sum of all technologies:

constraints:
  special_limit:
    foreach: [tech, node]
    mask: special_techs
    expression: capacity < sum(capacity, over=tech) * 0.5  # the sum will be limited to only `special_techs`, but we mean for it to be for _all_ techs

The way we circumvented it in calliope was to have a per-helper flag for whether it allowed pre-masking the array before applying the helper or not. Or, you create a separate expression with the operation applied, and then use that later:

expressions:
  capacity_sum_all:
    foreach: [node]
    expression: sum(capacity, over=tech)
constraints:
  special_limit:
    foreach: [tech, node]
    mask: special_techs
    expression: capacity < capacity_sum_all * 0.5

I never found either approach particularly satisfactory.

[FBumann]

I think this problem doesnt need a solution here. It just doesn't arise with the closed ast.

The ordering question only exists if a mask can reach an operand. Here it can't: a constraint's where: deletes rows of the frame and nothing else — the absence page calls it out as the one construct that "deletes a row directly, with nothing absent in it". No before, no after, because the mask is never in the expression. That dissolves your first question: nothing to protect an expression from, no per-helper flag to design.

Your second — a mask helper — isn't needed either. What Calliope does with one flag is four things here, each attached to a different object, which is why ordering can't be ambiguous:

What you mean	Spelling	What happens
this row shouldn't be built	where: on the constraint	the row is deleted
this variable doesn't exist here	where: on the variable	absent — and absence does not travel out of a reduction
this term shouldn't count in a sum	multiply by an indicator	0 is the identity of a sum, so the term contributes nothing
these terms should land per group	sum(x, by=lookup)	one row per group

Which one you want follows from what a row is:

• Bound each tech → the row is per (tech, node) → the subset is a row property → where:.
• Bound them together → the row is per node, no tech row left → the subset is an operand property → an indicator coefficient for one subset, a lookup for every group at once.

Your examples

One sparse column covers both, and the sparsity is the mask:

parameters:
  is_special: { dims: [tech], dtype: int } # 1 per special tech, no row otherwise

constraints:
  each_special: # your "don't apply the mask" case
    foreach: [tech, node]
    where: is_special # picks rows; the inner sum is untouched
    expression: capacity <= sum(capacity, over=tech) * 0.5 # sums ALL techs

  special_total: # your "apply it throughout" case, at your original foreach
    foreach: [node]
    expression: sum(capacity * is_special, over=tech) <= 5 # sums special only

Same column, two mechanisms: in a where: a bare numeric name is true where the table has a row; as a coefficient a missing row reads as zero. So capacity[tech=special_techs] and mask(capacity, ...) both become multiplication — more honest, since the exclusion is arithmetic rather than a flag, and the sparse column means those terms are never built. The two compose in one constraint (where: is_special for rows, * is_special for terms).

If the subset is really a partition, that's a lookup instead:

lookups:
  group_of: { over: tech, into: tech_group }

constraints:
  group_share:
    foreach: [node, tech_group]
    expression: sum(capacity, by=group_of) <= sum(capacity, over=tech) * 0.5

The last snapshot of each day is a lookup too, not a mask: where: "position(snapshot, by=day_of) == -1" — negatives count from the end, by= makes it per-group. final_ts_each_day never exists as an array, so it can't nullify the timestep you map to.

---

AI findings on top of that:

• The loader already states the row/operand rule as an error. Frame [node] with a mask on tech: "where-parameter 'is_special' has dims ['tech'] outside the frame ['node']. Reducing a mask over an unlisted dim would silently widen it."
• index() is gone — my snapshot snippet first used snapshot == index(snapshot, -1, by=day_of), the exact shape An ordering comparison against index(dim, i) loads with no defined meaning #32 refused and feat: position(dim) replaces index(dim, i), converting on the left #31 replaced. Corrected above.
• dtype is unchecked in arithmetic, and probably shouldn't be. declarations.md scopes it to three things and arithmetic is not one; a dtype: str parameter loads clean as a coefficient and as a divisor, neither of which any engine can build. A missing divisor row is already a documented refusal, so the string one being silent looks like the within= class of gap. Own issue.
• That decides the indicator's dtype. bool is right for a where:-only flag ("a bool is its own answer") but is explicitly not a 1/0 column, so it is the wrong thing to multiply by — hence sparse int, correct in both positions.
• Loader-verified against 0.0.0a10: everything above loads, including is_special in both positions. capacity[tech=special_techs] and sum(capacity * (cap > 5), over=tech) are parse errors; mask(...) is "Unknown operator 'mask'. Available: ['at', 'shift', 'sum', 'sum_back']".

[brynpickering]

@FBumann OK, so what I'm reading here is that you are basically keeping the inner mask concept by creating a boolean lookup that you multiply by the thing you want to mask before applying an operation (capacity * is_special). That is what I would have done before as well and is fine but should be documented.