diff --git a/docs/guides/batches.md b/docs/guides/batches.md index ed59af3f..05978aeb 100644 --- a/docs/guides/batches.md +++ b/docs/guides/batches.md @@ -157,7 +157,7 @@ The violation's message matches the server's in its code, the constraint's name ### Too many partitions: a batch-shape 252 -A schema whose `CREATE TABLE` declares a `PARTITION BY` answers which partition each stored row lands in (`partition_id` on each row) and how many partitions the batch spans (`partition_count`). There is no separate partition setter in 1.0: the key is part of the one statement you compile ([`reference/bindings-v1.md` §7](../reference/bindings-v1.md#7-what-v0-api-is-deleted-and-why)). A body that would split into more partitions than the call's `max_partitions_per_insert_block` allows — `100` unless a setting says otherwise, `0` meaning unlimited — is refused the way the server refuses it: the batch `outcome` is `rejected` with ClickHouse's own **code 252**, and the rows stay itemized, exactly the shape a `CHECK` violation has. Nothing in any row is wrong; the batch is. The remedy is to split the body by partition, and grouping an accepted batch's export spans by `partition_id` gives exactly those per-partition bodies. +A schema whose `CREATE TABLE` declares a `PARTITION BY` answers which partition each stored row lands in (`partition_id` on each row) and how many partitions the batch spans (`partition_count`). There is no separate partition setter in 1.0: the key is part of the one statement you compile ([`reference/bindings-v1.md` §7](../reference/bindings-v1.md#7-what-v0-api-is-deleted-and-why)). A body that would split into more partitions than the call's `max_partitions_per_insert_block` allows — `100` unless a setting says otherwise, `0` meaning unlimited — is refused the way the server refuses it: the batch `outcome` is `rejected` with ClickHouse's own **code 252**, and the rows stay itemized, exactly the shape a `CHECK` violation has. Nothing in any row is wrong; the batch is. The remedy is to split the body by partition, and grouping an accepted batch's export spans by `partition_id` gives exactly those per-partition bodies. On ABI v2 the limit, and `throw_on_max_partitions_per_insert_block`, come from the per-call settings, the server profile and the setup defaults only: one in the compile profile does not reach the INSERT, as a server's CREATE-time settings never reach its INSERT ([`settings.md`](settings.md#the-compile-profile-at-the-insert-abi-v2)). **Branch on code 252, never on the message text.** The message is ClickHouse's own wording and moves between lines; the code is the stable part. And the name a code carries is the loaded build's own, so if you want it for a log line, ask the library rather than writing it down: its error-code table (`lib.ErrorCodes()` in Go, `lib.error_codes()` in Python and Rust, `lib.errorCodes()` in TypeScript) answers the name for 252 on that line ([`ErrorCodeTable`](../reference/bindings-v1.md#errorcodetable-from-the-error_code_table-document)). diff --git a/docs/guides/filters.md b/docs/guides/filters.md index 0c9b7a7c..0c36bd9e 100644 --- a/docs/guides/filters.md +++ b/docs/guides/filters.md @@ -310,7 +310,7 @@ The export mechanics you already know from the plain export channel carry over u ## A server's settings, and the ones a filter declines (ABI v2) -A filter's `WHERE` sees the settings layered under it: the build's own, the process defaults, the server profile, the schema's own settings and the filter's own, highest last, with `session_timezone` at every layer on the zone path above. Each one is honored (applied to the `WHERE`), passed through (accepted and ignored), or declined, and the library decides which, never a binding. Every layer known when the schema compiles (the process defaults as the compile captured them, the server profile and the schema's own settings) is judged then: a setting there that this build's filters do not honor is listed in the schema description's top-level `filter_declined_settings` (Go `SchemaDescription.FilterDeclinedSettings`, Python `filter_declined_settings`, TypeScript `filterDeclinedSettings`, Rust `filter_declined_settings`), once per name, with its tier from [the WHERE-settings lists](../../spec/abi-v2/where-settings/README.md) and its layer (`defaults`, `server` or `schema`, the highest that sets it), and compiling any filter on that schema raises the binding's `UnsupportedError` (`CHS_DECLINED`), naming each setting and its layer. A declined setting from the defaults or the schema's own settings used to be named only at evaluation, and now fails the compile. A declined setting from the filter's own settings is still named at evaluation: in the filter result's `unsupported_settings`, with every verdict `d`, and in the batch's own `unsupported_settings` when the filter rides along on `rows`, which makes that batch `unsupported`. An evaluation's own settings are the INSERT session's: they govern parsing the body and never reach the `WHERE`, so they are never named as WHERE settings. So a filter evaluated in the INSERT's session, a materialized view's `WHERE` for example, is compiled with that session's WHERE-tier settings in the filter's own settings. A consumer that streams rows instead of asking the server must refuse a tenant if `filter_declined_settings` is non-empty, or a `result-content` setting is changed. +A filter's `WHERE` sees the settings layered under it: the build's own, the process defaults, the server profile, the schema's own settings and the filter's own, highest last, with `session_timezone` at every layer on the zone path above. Each one is honored (applied to the `WHERE`), passed through (accepted and ignored), or declined, and the library decides which, never a binding. Every layer known when the schema compiles (the process defaults as the compile captured them, the server profile and the schema's own settings) is judged then: a setting there that this build's filters do not honor is listed in the schema description's top-level `filter_declined_settings` (Go `SchemaDescription.FilterDeclinedSettings`, Python `filter_declined_settings`, TypeScript `filterDeclinedSettings`, Rust `filter_declined_settings`), once per name, with its tier from [the WHERE-settings lists](../../spec/abi-v2/where-settings/README.md) and its layer (`defaults`, `server` or `schema`, the highest that sets it), and compiling any filter on that schema raises the binding's `UnsupportedError` (`CHS_DECLINED`), naming each setting and its layer. A declined setting from the defaults or the schema's own settings used to be named only at evaluation, and now fails the compile. A declined setting from the filter's own settings is still named at evaluation: in the filter result's `unsupported_settings`, with every verdict `d`, and in the batch's own `unsupported_settings` when the filter rides along on `rows`, which makes that batch `unsupported`. An evaluation's own settings are the INSERT session's: they govern parsing the body and the expressions the INSERT itself evaluates ([`settings.md`](settings.md#the-compile-profile-at-the-insert-abi-v2)), and never reach the `WHERE`, so they are never named as WHERE settings. So a filter evaluated in the INSERT's session, a materialized view's `WHERE` for example, is compiled with that session's WHERE-tier settings in the filter's own settings. A consumer that streams rows instead of asking the server must refuse a tenant if `filter_declined_settings` is non-empty, or a `result-content` setting is changed. ## Lifetimes diff --git a/docs/guides/settings.md b/docs/guides/settings.md index 57b74b54..3e06bb61 100644 --- a/docs/guides/settings.md +++ b/docs/guides/settings.md @@ -83,6 +83,14 @@ A gate named in the **compile profile** binds at compile. A refusing value fails This is the one place the precedence table above does not hold, and it holds the way it does because a table that exists is a table that exists: the gate settled at CREATE, and no later INSERT reopens the question. +## The compile profile at the INSERT (ABI v2) + +On ABI v2 the order above has a second limit, and it is the server's own too: a server's CREATE-time settings never reach its INSERT. At the INSERT that a row, batch or filter-evaluation call models, the compile profile reaches only the parse. The settings that shape how a body is read (the ones ClickHouse's own `FormatFactory::getFormatSettings` reads, such as `input_format_*`, `input_format_allow_errors_*` and `date_time_input_format`) keep the order above, with the server profile's `settings` under the compile profile, except a type gate among them, which binds the compile only. A filter's `WHERE` is not the INSERT, and keeps its own layers ([`filters.md`](filters.md#a-servers-settings-and-the-ones-a-filter-declines-abi-v2)). + +Everything else the INSERT does runs under the per-call settings, the server profile's `settings` and the setup defaults only, over ClickHouse's own defaults, with every type gate at ClickHouse's default unless one of those three sets it. That covers every expression the INSERT evaluates (a `DEFAULT` or `MATERIALIZED` fill, a `CHECK`, a TTL, a partition key, a projection, an expression in a `Values` body) and its limits, `max_partitions_per_insert_block` and `throw_on_max_partitions_per_insert_block` among them. The sorting key and the skip indices take the server profile and the setup defaults only. + +So a gated `CAST` or a gated aggregate function whose gate is only in the compile profile is refused as the server refuses it (455, 44 or 63), and a partition limit only in the compile profile does not reach the INSERT. If your real INSERT session carries such a setting, pass it per call or in the server profile, as that session does. The full rule is the "Settings." paragraph of the [ABI v2 reference](../reference/abi-v2.md)'s preamble. + ## Values cross as strings. Always This has cost real bugs, so v1 makes it one rule in all four bindings: **a settings value is a string, and only a string.** The binding serializes the map into a JSON object of string values with the language's stock encoder and passes it verbatim, and it never rewrites a value: no boolean or integer spelling, no float. A non-string value is the language's own type error. @@ -161,7 +169,7 @@ A zone name is valid exactly when ClickHouse's own `DateLUT` loads it, and a nam The zone follows the settings precedence above: a per-call `session_timezone` beats the one in the compile profile, which beats the one in the setup defaults, which beats the image zone. A zone in the compile profile is a default for later calls on that schema; a compiled type always takes the image zone, never the profile's. -**A filter's zone is its own**, fixed when it is compiled: its `WHERE` runs in the zone of the settings the filter was compiled with, and an evaluation's settings decide only how the body is parsed. [`reference/bindings-v1.md` §2](../reference/bindings-v1.md#the-call-options) has the rule and the per-binding spellings. +**A filter's zone is its own**, fixed when it is compiled: its `WHERE` runs in the zone of the settings the filter was compiled with, and an evaluation's settings are the INSERT's: they decide how the body is parsed and the expressions the INSERT evaluates, never the `WHERE`. [`reference/bindings-v1.md` §2](../reference/bindings-v1.md#the-call-options) has the rule and the per-binding spellings. 0.x declined a per-call `session_timezone` that named a different zone than the process's, because zone was resolved once at `chs_init`. That decline is gone with the model it protected. diff --git a/docs/reference/abi-v2.md b/docs/reference/abi-v2.md index 35546d4f..fb23985a 100644 --- a/docs/reference/abi-v2.md +++ b/docs/reference/abi-v2.md @@ -2262,7 +2262,7 @@ Everything is validated here, once, and the server never changes after it. A key The macros are the server's whole `system.macros` when `macros` is present, even as `{}`: a macro a Replicated engine's arguments name that the set lacks is then the server's own refusal (`CHS_REJECTED`, 139, NO_ELEMENTS_IN_CONFIG). When `macros` is absent the server's macros are unknown, and a schema whose engine reads one is declined, as it is without a server, except under `schema_options`' `posture` `load`, where a stored Replicated definition compiles without expanding them (`input:schema_options`). A build that validates a profile's macros but does not yet apply them declines such a schema in both cases. -A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. So does `chs_schema_create` on the server, because a profile never changes a schema compile: it lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer `server`. +A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. So does `chs_schema_create` on the server: it lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer `server`. The call reads no defaults and leaves `chs_set_defaults`' latch open. Its init class is `required`, so before `chs_initialize` it refuses (Initialization, in the Preamble). @@ -2413,7 +2413,7 @@ On a schema whose `filter_declined_settings` is not empty (`chs_schema_describe` **Changed (public issue #588).** A declined setting from the `chs_set_defaults` or the schema layer moves from evaluation time to create time: it was named at evaluation (`filter_result`'s `unsupported_settings`), and it now declines this call. A server-profile setting already declined it. -A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here but for one case, and it is never in `schema_description`. The case is a cell of the line's `where-settings/documented-pathological.tsv`, a value under which the server refuses every query: set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115. +A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here but for one case, and it is never in `schema_description`. The case is a cell of the line's `where-settings/documented-pathological.tsv`, a value under which the server refuses every query: set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and the expressions the INSERT itself evaluates (Settings, above), and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115. A filter under a per-call `session_timezone` answers as the artifact producer has measured against live servers, including how the zone in an evaluation's settings combines with the zone the filter was compiled under. Until a path is measured to match a server it declines, which is never a wrong answer. The filter holds a counted reference to the schema. @@ -2446,7 +2446,7 @@ CHS_API chs_status chs_filter_eval_body(const chs_filter *filter, chs_format for - `out`: out_handle, `chs_buf`, `document:filter_result`. - `err`: out_error. -Evaluates the filter over every row of a body, returning one verdict per row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that names it, and with none the INSERT that names none. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. +Evaluates the filter over every row of a body, returning one verdict per row. `settings` is the INSERT session's: it governs parsing the body and the expressions the INSERT evaluates (Settings, above), and `session_timezone` there is this call's zone. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that names it, and with none the INSERT that names none. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. #### `chs_block_create` @@ -2466,7 +2466,7 @@ CHS_API chs_status chs_block_create(const chs_schema *schema, chs_format format, - `out`: out_handle, `chs_block`. - `err`: out_error. -Parses a body once under the schema, for evaluating many filters over it. `settings` governs the parse, `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal error. An accepted body is read once, for the block and that verdict together. `options` (`input:insert_options`) is read as `chs_preview_batch` reads it. +Parses a body once under the schema, for evaluating many filters over it. `settings` is the INSERT session's: it governs the parse and the expressions the INSERT evaluates (Settings, above), `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal error. An accepted body is read once, for the block and that verdict together. `options` (`input:insert_options`) is read as `chs_preview_batch` reads it. #### `chs_block_free` diff --git a/docs/reference/rust.md b/docs/reference/rust.md index bbfb1c7e..adec823e 100644 --- a/docs/reference/rust.md +++ b/docs/reference/rust.md @@ -67,7 +67,7 @@ The crate ships a `chtypes` binary over the fetch layer (`cargo install chtypes` Every option struct has public fields and `Default`, and is built as `RowOptions { session_timezone: Some(z), ..Default::default() }`. Settings are `Vec<(String, String)>`: values are strings, and only strings, and are never rewritten. The per-call zone is the `session_timezone` key of the settings; passing the option and a settings key of the same name is `Error::Usage`. -A filter's zone is its own, fixed when it is compiled (`FilterOptions::session_timezone`); an evaluation's settings (`EvalOptions`, or a block's `RowOptions`) are the body's parse settings only. +A filter's zone is its own, fixed when it is compiled (`FilterOptions::session_timezone`); an evaluation's settings (`EvalOptions`, or a block's `RowOptions`) are the body's INSERT settings: they govern its parse and the expressions the INSERT evaluates, never the WHERE. ## Generated defaults diff --git a/go/CHANGELOG.md b/go/CHANGELOG.md index 941a2d01..fece7322 100644 --- a/go/CHANGELOG.md +++ b/go/CHANGELOG.md @@ -78,6 +78,7 @@ The four bindings in this repository are released together and give one answer, - **Breaking (ABI v2 dev): `BatchResult.Spans` is `[]*Span`**, one entry per `Rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which the entry reads as nil. It was `[]Span` of the exported rows only. **This change compiles silently:** code written for `[]Span` that reads `r.Spans[i].Off` still builds, because Go dereferences the pointer for you, and it panics at run time on a nil entry. Check each entry for nil before reading it. - **Breaking (ABI v2 dev): `AtMergeEntry.Row` is `AtMergeEntry.PartRow`**, the document's `part_row`. - A document's `code` can be `CodeDeclined` (-2) for a row, batch, filter result or verdict this build declined, and every decline names its `Reason`; a `Source` the description does not list reads the description's fallback (`skipped`), not stored. +- The ABI v2 spec now states the INSERT-time settings layering the library already uses, since artifact build `20261021.025722` (#632): at the INSERT, the schema's own settings (the compile profile) reach only the parse, minus a type gate, and every expression the INSERT evaluates and its limits (`max_partitions_per_insert_block` among them) run under the call's own settings, the server profile and the setup defaults only. A setting your INSERT session carries for those goes in the call's settings or the server profile (`docs/guides/settings.md`). No API change. ### Removed diff --git a/go/chtypes/schema.go b/go/chtypes/schema.go index 6ddd40b3..ca071e88 100644 --- a/go/chtypes/schema.go +++ b/go/chtypes/schema.go @@ -258,7 +258,8 @@ func (f *Filter) enter() error { } // Rows evaluates the filter over a body (chs_filter_eval_body). The settings -// are the body's parse settings only; the filter's own zone governs its WHERE. +// are the body's INSERT settings: they govern its parse and the expressions the +// INSERT evaluates, never the WHERE, which runs in the filter's own zone. // WithColumns and the insert options model the same INSERT as on Schema.Rows. func (f *Filter) Rows(format Format, body []byte, opts ...EvalOption) (FilterResult, error) { c := evalConfig(opts) diff --git a/include/v2/chtypes.h b/include/v2/chtypes.h index a2e080bf..bc6600d6 100644 --- a/include/v2/chtypes.h +++ b/include/v2/chtypes.h @@ -119,12 +119,32 @@ * server profile, the schema's own, and the filter's own (`chs_filter_create`), with * `session_timezone` at every layer staying on the zone path (Time zones, above). One exception: a * type gate in the schema's own settings binds the CREATE only, and a WHERE's CAST validates under - * the session layers, every layer but the schema's own. Each setting that reaches a WHERE is - * honored (applied to it), passed through (accepted and ignored), or declined (`chs_filter_create` - * says how). A server resource limit (an `execution`-tier setting, e.g. a small - * `max_columns_to_read`) can make the server refuse a query the library answers; that is resource - * policy, not row access, and the library does not model it. A server's settings are the ones its - * profile applies to every query, and they never change how a schema compiles. No call reads a + * the session layers, every layer but the schema's own. At the INSERT, the schema's own settings + * reach only the parse, as a server's CREATE-time settings never reach its INSERT: the settings + * ClickHouse's own `FormatFactory::getFormatSettings` reads, which shape how a body is read, keep + * all five layers, except a type gate among them (`Nullable(Tuple)`'s, for one), which binds the + * CREATE only. Everything else the INSERT does runs under the call's own settings, its server's + * `settings` and `chs_set_defaults` only, over the build's own, with every type gate at the build's + * default unless one of those three sets it. That covers every expression it evaluates (a `DEFAULT` + * or `MATERIALIZED` fill, a `CHECK`, a TTL, a partition key the server rebuilds at INSERT, one with + * a modulo, a projection, an expression a `Values` body evaluates while it is read, and the creator + * of any aggregate function in them) and its limits, `max_partitions_per_insert_block` and + * `throw_on_max_partitions_per_insert_block` among them. The sorting key and the skip indices, + * which a server builds under its global context, run under its server's `settings` and + * `chs_set_defaults` only, though an aggregate function's creator there still reads the call's own + * settings. So a gated `CAST` or a gated aggregate function in such an expression, a `Values` + * body's included, with its gate only in the schema's own settings, is refused as the server + * refuses it (455, 44 or 63), and a partition limit in the schema's own settings does not reach the + * INSERT; a caller whose real INSERT session carries such a setting passes it in the call's + * settings or the server profile, as that INSERT does. `session_timezone` keeps the zone path. Each + * setting that reaches a WHERE is honored (applied to it), passed through (accepted and ignored), + * or declined (`chs_filter_create` says how). A server resource limit (an `execution`-tier setting, + * e.g. a small `max_columns_to_read`) can make the server refuse a query the library answers; that + * is resource policy, not row access, and the library does not model it. A server's settings are + * the ones its profile applies to every query. At posture `create`, they are layered under the + * schema's own settings for the whole CREATE, so they shape how the schema compiles + * (`flatten_nested`, the type gates and the aggregate gates among them). At posture `load`, the + * stored definition is taken as written, as a server attaches it at startup. No call reads a * server's version or its changed settings itself: that discovery is a recorded gap in this * generation, so a caller that wants a server's settings passes them explicitly, in a server * profile, through `chs_set_defaults` or in each call's settings. @@ -1214,9 +1234,8 @@ CHS_API chs_status chs_quote_string( * yet apply them declines such a schema in both cases. * * A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. - * So does `chs_schema_create` on the server, because a profile never changes a schema compile: it - * lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer - * `server`. + * So does `chs_schema_create` on the server: it lists the setting in the schema's + * `filter_declined_settings` (`schema_description`), at layer `server`. * * The call reads no defaults and leaves `chs_set_defaults`' latch open. Its init class is * `required`, so before `chs_initialize` it refuses (Initialization, in the Preamble). @@ -1535,10 +1554,11 @@ CHS_API chs_status chs_preview_batch( * defaults, the server profile, the schema's own settings and this call's), as a subscriber's * SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, * `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern - * parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a - * filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created - * with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's - * own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115. + * parsing the body and the expressions the INSERT itself evaluates (Settings, above), and never + * reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's + * own session, a materialized view's WHERE for example, is created with that session's WHERE-tier + * settings in its own `settings`. Every key still meets the server's own SET check first, so a name + * the server does not know is `CHS_REJECTED` with ch_code 115. * * A filter under a per-call `session_timezone` answers as the artifact producer has measured * against live servers, including how the zone in an evaluation's settings combines with the zone @@ -1587,17 +1607,18 @@ CHS_API void chs_filter_free(chs_filter *filter); /* * chs_filter_eval_body: Evaluates the filter over every row of a body, returning one verdict per - * row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. - * `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and - * empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the - * INSERT that names it, and with none the INSERT that names none. When a server would not accept an - * INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column - * list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that - * verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch - * verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a - * refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. - * `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the - * verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. + * row. `settings` is the INSERT session's: it governs parsing the body and the expressions the + * INSERT evaluates (Settings, above), and `session_timezone` there is this call's zone. `columns` + * is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for + * none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that + * names it, and with none the INSERT that names none. When a server would not accept an INSERT of + * the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, + * with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict + * (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict + * together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or + * decline of the call itself in the document (`refused_at` `call`), never as a status. `options` + * (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict + * mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. * * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, * CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its options document, @@ -1633,8 +1654,9 @@ CHS_API chs_status chs_filter_eval_body( /* * chs_block_create: Parses a body once under the schema, for evaluating many filters over it. - * `settings` governs the parse, `session_timezone` there included, and `columns` is read as - * `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records + * `settings` is the INSERT session's: it governs the parse and the expressions the INSERT evaluates + * (Settings, above), `session_timezone` there included, and `columns` is read as `chs_preview_row` + * reads it. The block holds a counted reference to the schema. It also records * `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, * which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is * recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal diff --git a/python/CHANGELOG.md b/python/CHANGELOG.md index 360a0132..66752b1b 100644 --- a/python/CHANGELOG.md +++ b/python/CHANGELOG.md @@ -80,6 +80,7 @@ The four bindings in this repository are released together and give one answer, - **Enforcement gate lifted** for `26.8` on `darwin-arm64`. - **Enforcement gate lifted** for `26.9` on `darwin-arm64`. - `docs/limitations.md` is re-checked against artifact build `20261012.142240` (v2-dev.13), the library half measured on that build: functions that need a server, in a column expression, and a body the server may read serially or in parallel under a non-zero error budget are listed under "Declined by design"; the known divergence for a column expression that calls a function needing a server is retired, since that build declines those schemas (on 26.8 and 26.9 the `AI*` and `ai*` functions compile, as the page says); and the `lossy` raw-NUL TSV gap is marked fixed on that build and still present in the 1.0 libraries. `docs/guides/filters.md` gains a section on subcolumn filters and a JSON value typed by its batch. +- The ABI v2 spec now states the INSERT-time settings layering the library already uses, since artifact build `20261021.025722` (#632): at the INSERT, the schema's own settings (the compile profile) reach only the parse, minus a type gate, and every expression the INSERT evaluates and its limits (`max_partitions_per_insert_block` among them) run under the call's own settings, the server profile and the setup defaults only. A setting your INSERT session carries for those goes in the call's settings or the server profile (`docs/guides/settings.md`). No API change. ### Removed diff --git a/python/src/chtypes/library.py b/python/src/chtypes/library.py index 61378c8f..17a42e41 100644 --- a/python/src/chtypes/library.py +++ b/python/src/chtypes/library.py @@ -212,7 +212,8 @@ def rows( insert_options_extra: Mapping[str, str] | None = None, ) -> FilterResult: """Evaluate over a body. `settings` and `session_timezone` are the body's - PARSE settings only: the filter's own zone was fixed when it compiled. + INSERT settings: they govern its parse and the expressions the INSERT + evaluates, never the WHERE, whose zone was fixed when the filter compiled. `columns` is the INSERT column list and the keywords after it the `insert_options` members, as for `Schema.rows`.""" payload = body_bytes(body) diff --git a/rust/CHANGELOG.md b/rust/CHANGELOG.md index 4de4f0c8..ad564fd9 100644 --- a/rust/CHANGELOG.md +++ b/rust/CHANGELOG.md @@ -80,6 +80,7 @@ The four bindings in this repository are released together and give one answer, - **Enforcement gate lifted** for `26.8` on `darwin-arm64`. - **Enforcement gate lifted** for `26.9` on `darwin-arm64`. - `docs/limitations.md` is re-checked against artifact build `20261012.142240` (v2-dev.13), the library half measured on that build: functions that need a server, in a column expression, and a body the server may read serially or in parallel under a non-zero error budget are listed under "Declined by design"; the known divergence for a column expression that calls a function needing a server is retired, since that build declines those schemas (on 26.8 and 26.9 the `AI*` and `ai*` functions compile, as the page says); and the `lossy` raw-NUL TSV gap is marked fixed on that build and still present in the 1.0 libraries. `docs/guides/filters.md` gains a section on subcolumn filters and a JSON value typed by its batch. +- The ABI v2 spec now states the INSERT-time settings layering the library already uses, since artifact build `20261021.025722` (#632): at the INSERT, the schema's own settings (the compile profile) reach only the parse, minus a type gate, and every expression the INSERT evaluates and its limits (`max_partitions_per_insert_block` among them) run under the call's own settings, the server profile and the setup defaults only. A setting your INSERT session carries for those goes in the call's settings or the server profile (`docs/guides/settings.md`). No API change. ### Removed diff --git a/rust/src/schema.rs b/rust/src/schema.rs index 74410137..dd8c7391 100644 --- a/rust/src/schema.rs +++ b/rust/src/schema.rs @@ -21,7 +21,8 @@ //! A filter's zone is its own, fixed when it is compiled //! ([`FilterOptions`]): its WHERE runs in that zone for every evaluation. An //! evaluation's settings ([`EvalOptions`], [`RowOptions`] for a block) are the -//! body's PARSE settings only. Differing zones are an ordinary input. +//! body's INSERT settings: its parse and the expressions the INSERT evaluates, +//! never the WHERE. Differing zones are an ordinary input. use std::sync::Arc; diff --git a/spec/abi-v2/docs.md b/spec/abi-v2/docs.md index dda130d1..2e08d63f 100644 --- a/spec/abi-v2/docs.md +++ b/spec/abi-v2/docs.md @@ -94,7 +94,7 @@ Stacks. A call's answer doesn't depend on the calling thread's stack. Every call Time zones. A library image has one zone of its own, set once by `chs_initialize`: the image zone. A schema has a home zone, its server's `timezone` (`chs_server_create`), or else the image zone. The schema's types bind its home zone, as a server's table binds the server's zone after a restart, and so do the functions its DEFAULT, MATERIALIZED, ALIAS and CHECK expressions apply to its columns, its PARTITION BY expressions, and a Date-valued TTL under a merge. Each call runs in a zone too, the first of these that is set: the call's own `session_timezone` (a key in its settings, which ClickHouse applies through its own query context); the `session_timezone` in the schema's own settings; the one in its server's `settings`; the one set by `chs_set_defaults`; its server's `timezone`; the image zone. An empty `session_timezone` at any of these means the server's zone, the last two. The call's zone governs parsing and rendering a zone-less `DateTime` or `DateTime64`, export, a DEFAULT literal's fold, `toDateTimeOrZero` and a filter's literals. A `Date` partition id is the civil date in every zone. Every zone name is validated by ClickHouse's own `DateLUT`, which reads the library host's time zone database, on every platform. On darwin the file system is case-insensitive, so `DateLUT` there accepts a spelling such as `utc` that a Linux server refuses; that difference is documented, not patched. -Settings. Every settings input is a JSON object whose values are JSON strings, or the object form for bytes (Input values, below), and a binding never rewrites one. A call's settings are layered, highest first: the call's own `settings`; the schema's own (`chs_schema_create`); its server's `settings` (`chs_server_create`); the defaults set by `chs_set_defaults`; the build's own. A filter's WHERE takes the same layers, lowest first: the build's own, `chs_set_defaults`, the server profile, the schema's own, and the filter's own (`chs_filter_create`), with `session_timezone` at every layer staying on the zone path (Time zones, above). One exception: a type gate in the schema's own settings binds the CREATE only, and a WHERE's CAST validates under the session layers, every layer but the schema's own. Each setting that reaches a WHERE is honored (applied to it), passed through (accepted and ignored), or declined (`chs_filter_create` says how). A server resource limit (an `execution`-tier setting, e.g. a small `max_columns_to_read`) can make the server refuse a query the library answers; that is resource policy, not row access, and the library does not model it. A server's settings are the ones its profile applies to every query, and they never change how a schema compiles. No call reads a server's version or its changed settings itself: that discovery is a recorded gap in this generation, so a caller that wants a server's settings passes them explicitly, in a server profile, through `chs_set_defaults` or in each call's settings. +Settings. Every settings input is a JSON object whose values are JSON strings, or the object form for bytes (Input values, below), and a binding never rewrites one. A call's settings are layered, highest first: the call's own `settings`; the schema's own (`chs_schema_create`); its server's `settings` (`chs_server_create`); the defaults set by `chs_set_defaults`; the build's own. A filter's WHERE takes the same layers, lowest first: the build's own, `chs_set_defaults`, the server profile, the schema's own, and the filter's own (`chs_filter_create`), with `session_timezone` at every layer staying on the zone path (Time zones, above). One exception: a type gate in the schema's own settings binds the CREATE only, and a WHERE's CAST validates under the session layers, every layer but the schema's own. At the INSERT, the schema's own settings reach only the parse, as a server's CREATE-time settings never reach its INSERT: the settings ClickHouse's own `FormatFactory::getFormatSettings` reads, which shape how a body is read, keep all five layers, except a type gate among them (`Nullable(Tuple)`'s, for one), which binds the CREATE only. Everything else the INSERT does runs under the call's own settings, its server's `settings` and `chs_set_defaults` only, over the build's own, with every type gate at the build's default unless one of those three sets it. That covers every expression it evaluates (a `DEFAULT` or `MATERIALIZED` fill, a `CHECK`, a TTL, a partition key the server rebuilds at INSERT, one with a modulo, a projection, an expression a `Values` body evaluates while it is read, and the creator of any aggregate function in them) and its limits, `max_partitions_per_insert_block` and `throw_on_max_partitions_per_insert_block` among them. The sorting key and the skip indices, which a server builds under its global context, run under its server's `settings` and `chs_set_defaults` only, though an aggregate function's creator there still reads the call's own settings. So a gated `CAST` or a gated aggregate function in such an expression, a `Values` body's included, with its gate only in the schema's own settings, is refused as the server refuses it (455, 44 or 63), and a partition limit in the schema's own settings does not reach the INSERT; a caller whose real INSERT session carries such a setting passes it in the call's settings or the server profile, as that INSERT does. `session_timezone` keeps the zone path. Each setting that reaches a WHERE is honored (applied to it), passed through (accepted and ignored), or declined (`chs_filter_create` says how). A server resource limit (an `execution`-tier setting, e.g. a small `max_columns_to_read`) can make the server refuse a query the library answers; that is resource policy, not row access, and the library does not model it. A server's settings are the ones its profile applies to every query. At posture `create`, they are layered under the schema's own settings for the whole CREATE, so they shape how the schema compiles (`flatten_nested`, the type gates and the aggregate gates among them). At posture `load`, the stored definition is taken as written, as a server attaches it at startup. No call reads a server's version or its changed settings itself: that discovery is a recorded gap in this generation, so a caller that wants a server's settings passes them explicitly, in a server profile, through `chs_set_defaults` or in each call's settings. Documents. Every output document is valid JSON that a stock parser reads: every ClickHouse rendering (a stored value, an input value, an engine's row) is a JSON string, a number written bare is within plus or minus 2^53 (anything larger is a string), and there is never a bare `inf`, `-inf` or `nan`. Every column entry of a row carries `null`, the library's verdict that the stored value is NULL, poisoned cells included. @@ -521,7 +521,7 @@ Everything is validated here, once, and the server never changes after it. A key The macros are the server's whole `system.macros` when `macros` is present, even as `{}`: a macro a Replicated engine's arguments name that the set lacks is then the server's own refusal (`CHS_REJECTED`, 139, NO_ELEMENTS_IN_CONFIG). When `macros` is absent the server's macros are unknown, and a schema whose engine reads one is declined, as it is without a server, except under `schema_options`' `posture` `load`, where a stored Replicated definition compiles without expanding them (`input:schema_options`). A build that validates a profile's macros but does not yet apply them declines such a schema in both cases. -A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. So does `chs_schema_create` on the server, because a profile never changes a schema compile: it lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer `server`. +A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. So does `chs_schema_create` on the server: it lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer `server`. The call reads no defaults and leaves `chs_set_defaults`' latch open. Its init class is `required`, so before `chs_initialize` it refuses (Initialization, in the Preamble). @@ -583,7 +583,7 @@ On a schema whose `filter_declined_settings` is not empty (`chs_schema_describe` **Changed (public issue #588).** A declined setting from the `chs_set_defaults` or the schema layer moves from evaluation time to create time: it was named at evaluation (`filter_result`'s `unsupported_settings`), and it now declines this call. A server-profile setting already declined it. -A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here but for one case, and it is never in `schema_description`. The case is a cell of the line's `where-settings/documented-pathological.tsv`, a value under which the server refuses every query: set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115. +A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here but for one case, and it is never in `schema_description`. The case is a cell of the line's `where-settings/documented-pathological.tsv`, a value under which the server refuses every query: set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and the expressions the INSERT itself evaluates (Settings, above), and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115. A filter under a per-call `session_timezone` answers as the artifact producer has measured against live servers, including how the zone in an evaluation's settings combines with the zone the filter was compiled under. Until a path is measured to match a server it declines, which is never a wrong answer. The filter holds a counted reference to the schema. @@ -593,11 +593,11 @@ Releases the caller's reference to a filter. Freeing NULL does nothing. ### chs_filter_eval_body -Evaluates the filter over every row of a body, returning one verdict per row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that names it, and with none the INSERT that names none. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. +Evaluates the filter over every row of a body, returning one verdict per row. `settings` is the INSERT session's: it governs parsing the body and the expressions the INSERT evaluates (Settings, above), and `session_timezone` there is this call's zone. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that names it, and with none the INSERT that names none. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options. ### chs_block_create -Parses a body once under the schema, for evaluating many filters over it. `settings` governs the parse, `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal error. An accepted body is read once, for the block and that verdict together. `options` (`input:insert_options`) is read as `chs_preview_batch` reads it. +Parses a body once under the schema, for evaluating many filters over it. `settings` is the INSERT session's: it governs the parse and the expressions the INSERT evaluates (Settings, above), `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal error. An accepted body is read once, for the block and that verdict together. `options` (`input:insert_options`) is read as `chs_preview_batch` reads it. ### chs_block_free diff --git a/ts/CHANGELOG.md b/ts/CHANGELOG.md index 7c43ecfd..bb4343e3 100644 --- a/ts/CHANGELOG.md +++ b/ts/CHANGELOG.md @@ -86,6 +86,7 @@ The four bindings in this repository are released together and give one answer, - **Breaking (ABI v2 dev): `AtMergeEntry.row` is `AtMergeEntry.partRow`**, the document's `part_row`. - **Breaking (ABI v2 dev): `Settings` values and `sessionTimezone` take `string | Uint8Array`**, and a string holding a lone surrogate (no bytes) is still refused as a `UsageError`; the refusal of a value that is not UTF-8 is gone for a `Uint8Array`, while a key that is not valid UTF-8 is still refused. - A document's `code` can be -2 for a row, batch, filter result or verdict this build declined, and every decline names its `reason`; a `Source` the description does not list reads the description's fallback (`skipped`), not stored. +- The ABI v2 spec now states the INSERT-time settings layering the library already uses, since artifact build `20261021.025722` (#632): at the INSERT, the schema's own settings (the compile profile) reach only the parse, minus a type gate, and every expression the INSERT evaluates and its limits (`max_partitions_per_insert_block` among them) run under the call's own settings, the server profile and the setup defaults only. A setting your INSERT session carries for those goes in the call's settings or the server profile (`docs/guides/settings.md`). No API change. ### Removed diff --git a/ts/src/schema.ts b/ts/src/schema.ts index 9144ac04..33491de5 100644 --- a/ts/src/schema.ts +++ b/ts/src/schema.ts @@ -109,7 +109,8 @@ export interface FilterOptions { } /** - * Options of `Filter.rows`. The settings are PARSE settings only: they decide how the body is parsed, never the filter's WHERE. + * Options of `Filter.rows`. The settings are the body's INSERT settings: they decide how the body is parsed and the + * expressions the INSERT evaluates, never the filter's WHERE. * `columns` and the `insert_options` members model the same INSERT as on `Schema.rows` (ABI v2). */ export interface EvalOptions {