From bc33b71e3ed16301b1e52bcb39c7a0c0ce513a4f Mon Sep 17 00:00:00 2001 From: Felix Bumann Date: Tue, 15 Sep 2026 08:28:47 +0000 Subject: [PATCH] docs(limits): a whole-table operator is priced at a barrier rather than refused MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stated reason for refusing an operator that reads a whole table was that no engine could build the model one chunk of rows at a time. That is not what it costs. Such an operator makes the engine read the whole table before it builds any chunk, and the chunks stop being independent. The page now says that, and admits the operator at that price. Self-reference keeps its refusal, on its own ground: nothing bounds how far it expands. The escape hatch is withdrawn. There is no capped block of Python, so the refusal table names the reason that remains — the file would stop being the artifact — and the errors page says the language closes its own gaps as a macro, a primitive or a formulation. Sentence length on the changed pages, measured with the docs-writing skill's script: limits.md avg 18.7 to 17.9, median 16, 20 sentences over 25 words before and after, all of them pre-existing; errors.md avg 17.2 to 16.7, median 14, over-25 5 to 4. No sentence this change adds exceeds 25 words. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01T18AdG3hoawcwuqXxr4dnM --- AGENTS.md | 4 +++- docs/about/limits.md | 34 +++++++++++++++++++------------ docs/reference/language/errors.md | 10 ++++----- 3 files changed, 29 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df8a23a9..b5d68a13 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -81,7 +81,9 @@ answer here is the mistake. the ones nothing calls. Where a file does not determine the answer, loading fails, and the message names the rewrite. - **Triage a new construct first: is it a primitive, a macro, or refused?** - A primitive is admissible when it is relational and local. Read the + A primitive is admissible when it is relational. Locality prices a primitive + rather than barring it. An operator that reads the whole table costs one full + pass over the data. Read the deliberate non-primitives in [limits.md](docs/about/limits.md) first. The argument for admitting one of those is the argument that page is already making, not a new argument. diff --git a/docs/about/limits.md b/docs/about/limits.md index 59ecbe90..00b7b042 100644 --- a/docs/about/limits.md +++ b/docs/about/limits.md @@ -40,13 +40,20 @@ the value of a keyword argument, such as `over=snapshot`, never in the key. A macro can write `over=d` and let the caller supply `d`. It could not do that if the dimension were the keyword itself. -**Each output row reads a bounded number of input rows.** `sum(p, over=g)` reads -one row per generator. `shift(p, over=t, offset=1)` reads one row, the one -before it. `x * y * a` reads the rows of `a` that pair an `x` with a `y`. An -operator that reads the whole table to produce one row, or that calls itself, is -refused, because an engine cannot then build the model one chunk of rows at a -time. Reading only the coordinate labels does not count: "the last snapshot" -looks at the list of snapshots, not at the data, and is allowed. +**An operator may read the whole table. It pays one full pass over the data.** +`sum(p, over=g)` reads one row per generator. `shift(p, over=t, offset=1)` reads +one row, the one before it. `x * y * a` reads the rows of `a` that pair an `x` +with a `y`. Each reads a bounded number of rows per output row, so an engine +builds the model one chunk of rows at a time. + +One kind of operator reads every row to produce one row. The engine then reads +the whole table before it builds any chunk, and the chunks stop being +independent. That is the price, and a request for such an operator names it. +Reading only the coordinate labels costs nothing. "The last snapshot" looks at +the list of snapshots, not at the data. + +**An operator that calls itself is refused.** Nothing bounds how far it expands, +so no number of passes over the data is enough. | The operator | Allowed? | | ---------------------------------------------------- | ----------------------------------------------- | @@ -54,7 +61,8 @@ looks at the list of snapshots, not at the data, and is allowed. | joins each row against a parameter or a lookup table | yes | | reads a fixed number of neighbouring rows | yes | | reads only the coordinate labels | yes | -| reads every row, or calls itself | no, and the message names what to write instead | +| reads every row | yes, at one full pass before any chunk builds | +| calls itself | no, and the message names what to write instead | **Degree is not a third test.** `p * q` at one coordinate is a join of a table with itself, so the objective and the constraints take it. Two things limit the @@ -74,11 +82,11 @@ the same model written out by hand. ### Three kinds of refusal -| The language refuses it because… | Examples | Can it change? | -| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| **one solver cannot take it** | indicator constraints (#220); a quadratic constraint. `sos:` was in this group, and entered: a solver with sets takes it as one, and a solver without gets binaries | yes, solver by solver | -| **no engine could build it one chunk of rows at a time** | an operator that reads a whole table; arbitrary Python | not today. A capped block of Python for this is planned as [#38](https://github.com/fluxopt/lpspec/issues/38) | -| **this project puts the work elsewhere** | data preparation such as resampling; helpers for one domain; Python that decides which declarations exist | it could; this project does not want it to | +| The language refuses it because… | Examples | Can it change? | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | +| **one solver cannot take it** | indicator constraints (#220); a quadratic constraint. `sos:` was in this group, and entered: a solver with sets takes it as one, and a solver without gets binaries | yes, solver by solver | +| **the file would stop being the artifact** | arbitrary Python, whose content no loader can check and no typesetter can print | no | +| **this project puts the work elsewhere** | data preparation such as resampling; helpers for one domain; Python that decides which declarations exist | it could; this project does not want it to | Three things never appear inside one model: an `if`, a loop, and a set of declarations that depends on the data. `dims: [snapshot]` does not know how diff --git a/docs/reference/language/errors.md b/docs/reference/language/errors.md index 12add9e8..8b7bf518 100644 --- a/docs/reference/language/errors.md +++ b/docs/reference/language/errors.md @@ -91,7 +91,7 @@ and [the limits](../../about/limits.md) gives the reasons. | time-series processing (resample, cluster, interpolate, align), file IO, units | Data preparation. Pass a parameter | | indicator constraints | What a solver can take is a question of its own, and `sos:` is where it landed ([#220](https://github.com/fluxopt/lpspec/issues/220)) | | multi-objective | There is one `objective:` block. Weight the goals into one expression | -| arbitrary array operations (`merge`, `reindex`, `apply_ufunc`) | Data preparation. The closed operator set is what lets a build stream its terms | +| arbitrary array operations (`merge`, `reindex`, `apply_ufunc`) | Data preparation. The operator set is closed so that every tool reads the file the same way | | filling a missing value (`.fillna`) | Data preparation, or a `where` if the coordinate should not exist. Inside the language, only `shift(..., edge=)` fills ([absence](absence.md)) | | schema migrations | — | @@ -101,7 +101,7 @@ The arrays it holds would build the same model, but an `expression:` string and reviewer could read. A library that wants a file passes a `dict` with the file's keys to `to_spec`, and calls `to_yaml()`. -For math the language cannot express, a block of Python named in the file, with -a cap on how many rows and columns it may emit, is planned as -[#38](https://github.com/fluxopt/lpspec/issues/38). It has not shipped, and no -key for it exists yet. +The language has no escape hatch. Math it cannot express is a gap in the +language, and a gap closes as a macro, a primitive or a formulation +([the limits](../../about/limits.md)). Where the table above has a row, that row +names what to write instead.