diff --git a/AGENTS.md b/AGENTS.md index df8a23a9..b5d68a13 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -81,7 +81,9 @@ answer here is the mistake. the ones nothing calls. Where a file does not determine the answer, loading fails, and the message names the rewrite. - **Triage a new construct first: is it a primitive, a macro, or refused?** - A primitive is admissible when it is relational and local. Read the + A primitive is admissible when it is relational. Locality prices a primitive + rather than barring it. An operator that reads the whole table costs one full + pass over the data. Read the deliberate non-primitives in [limits.md](docs/about/limits.md) first. The argument for admitting one of those is the argument that page is already making, not a new argument. diff --git a/docs/about/limits.md b/docs/about/limits.md index 59ecbe90..00b7b042 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -40,13 +40,20 @@ the value of a keyword argument, such as `over=snapshot`, never in the key. A macro can write `over=d` and let the caller supply `d`. It could not do that if the dimension were the keyword itself. -**Each output row reads a bounded number of input rows.** `sum(p, over=g)` reads -one row per generator. `shift(p, over=t, offset=1)` reads one row, the one -before it. `x * y * a` reads the rows of `a` that pair an `x` with a `y`. An -operator that reads the whole table to produce one row, or that calls itself, is -refused, because an engine cannot then build the model one chunk of rows at a -time. Reading only the coordinate labels does not count: "the last snapshot" -looks at the list of snapshots, not at the data, and is allowed. +**An operator may read the whole table. It pays one full pass over the data.** +`sum(p, over=g)` reads one row per generator. `shift(p, over=t, offset=1)` reads +one row, the one before it. `x * y * a` reads the rows of `a` that pair an `x` +with a `y`. Each reads a bounded number of rows per output row, so an engine +builds the model one chunk of rows at a time. + +One kind of operator reads every row to produce one row. The engine then reads +the whole table before it builds any chunk, and the chunks stop being +independent. That is the price, and a request for such an operator names it. +Reading only the coordinate labels costs nothing. "The last snapshot" looks at +the list of snapshots, not at the data. + +**An operator that calls itself is refused.** Nothing bounds how far it expands, +so no number of passes over the data is enough. | The operator | Allowed? | | ---------------------------------------------------- | ----------------------------------------------- | @@ -54,7 +61,8 @@ looks at the list of snapshots, not at the data, and is allowed. | joins each row against a parameter or a lookup table | yes | | reads a fixed number of neighbouring rows | yes | | reads only the coordinate labels | yes | -| reads every row, or calls itself | no, and the message names what to write instead | +| reads every row | yes, at one full pass before any chunk builds | +| calls itself | no, and the message names what to write instead | **Degree is not a third test.** `p * q` at one coordinate is a join of a table with itself, so the objective and the constraints take it. Two things limit the @@ -74,11 +82,11 @@ the same model written out by hand. ### Three kinds of refusal -| The language refuses it because… | Examples | Can it change? | -| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| **one solver cannot take it** | indicator constraints (#220); a quadratic constraint. `sos:` was in this group, and entered: a solver with sets takes it as one, and a solver without gets binaries | yes, solver by solver | -| **no engine could build it one chunk of rows at a time** | an operator that reads a whole table; arbitrary Python | not today. A capped block of Python for this is planned as [#38](https://github.com/fluxopt/lpspec/issues/38) | -| **this project puts the work elsewhere** | data preparation such as resampling; helpers for one domain; Python that decides which declarations exist | it could; this project does not want it to | +| The language refuses it because… | Examples | Can it change? | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | +| **one solver cannot take it** | indicator constraints (#220); a quadratic constraint. `sos:` was in this group, and entered: a solver with sets takes it as one, and a solver without gets binaries | yes, solver by solver | +| **the file would stop being the artifact** | arbitrary Python, whose content no loader can check and no typesetter can print | no | +| **this project puts the work elsewhere** | data preparation such as resampling; helpers for one domain; Python that decides which declarations exist | it could; this project does not want it to | Three things never appear inside one model: an `if`, a loop, and a set of declarations that depends on the data. `dims: [snapshot]` does not know how diff --git a/docs/reference/language/errors.md b/docs/reference/language/errors.md index 12add9e8..8b7bf518 100644 --- a/docs/reference/language/errors.md +++ b/docs/reference/language/errors.md @@ -91,7 +91,7 @@ and [the limits](../../about/limits.md) gives the reasons. | time-series processing (resample, cluster, interpolate, align), file IO, units | Data preparation. Pass a parameter | | indicator constraints | What a solver can take is a question of its own, and `sos:` is where it landed ([#220](https://github.com/fluxopt/lpspec/issues/220)) | | multi-objective | There is one `objective:` block. Weight the goals into one expression | -| arbitrary array operations (`merge`, `reindex`, `apply_ufunc`) | Data preparation. The closed operator set is what lets a build stream its terms | +| arbitrary array operations (`merge`, `reindex`, `apply_ufunc`) | Data preparation. The operator set is closed so that every tool reads the file the same way | | filling a missing value (`.fillna`) | Data preparation, or a `where` if the coordinate should not exist. Inside the language, only `shift(..., edge=)` fills ([absence](absence.md)) | | schema migrations | — | @@ -101,7 +101,7 @@ The arrays it holds would build the same model, but an `expression:` string and reviewer could read. A library that wants a file passes a `dict` with the file's keys to `to_spec`, and calls `to_yaml()`. -For math the language cannot express, a block of Python named in the file, with -a cap on how many rows and columns it may emit, is planned as -[#38](https://github.com/fluxopt/lpspec/issues/38). It has not shipped, and no -key for it exists yet. +The language has no escape hatch. Math it cannot express is a gap in the +language, and a gap closes as a macro, a primitive or a formulation +([the limits](../../about/limits.md)). Where the table above has a row, that row +names what to write instead.