diff --git a/docs/guides/batches.md b/docs/guides/batches.md
index 8e99130f..9421c4a2 100644
--- a/docs/guides/batches.md
+++ b/docs/guides/batches.md
@@ -291,6 +291,57 @@ Three payload states, and they are distinct: absent means no export was requeste
Attaching a compiled filter to this same call narrows the export further, to the rows the filter admits, from the same one parse: [`filters.md` → Exporting only the rows a filter admits](filters.md#exporting-only-the-rows-a-filter-admits).
+## Overriding a column's DEFAULT for one call (ABI v2)
+
+`column_defaults` is a per-call list of `{name, value}`. In a cell the server would have filled from the column's DEFAULT, the call uses your `value` instead. The library decides which cells those are, by ClickHouse's own reader and the call's format settings: a field the record omits, a `null` for a non-Nullable column under `input_format_null_as_default`, an empty CSV field under `input_format_csv_empty_as_default`. A field the record supplied is never overridden, and `""` is a value like any other. A cell an override fills reads `default_override` (`Source` in each binding), which is stored, and the export carries it as a present value, so inserting the export needs no override.
+
+The binding only passes the list: `name` and `value` are byte strings (UTF-8, else the `name_b64` and `value_b64` forms), the order is kept, and an empty list is sent as `[]`. It validates nothing. An entry that names an unknown, non-insertable, MATERIALIZED or ALIAS column, or a value that does not read as the column's type, is `CHS_INVALID_ARGUMENT` at the call, naming `column_defaults[i]`. A build that does not implement the member declines it. The option exists on the four calls that model an INSERT of a body: row, rows, parse a block and filter rows.
+
+Go
+
+```go
+batch, err := schema.Rows(chtypes.JSONEachRow, body,
+ chtypes.WithColumnDefaults([]chtypes.ColumnDefault{{Name: "region", Value: "eu"}}))
+// a cell filled from the override: batch.Rows[i].Columns[j].Source == chtypes.SourceDefaultOverride
+```
+
+
+
+Python
+
+```python
+batch = schema.rows(
+ Format.JSON_EACH_ROW, body, column_defaults=[ColumnDefault("region", "eu")]
+)
+# a cell filled from the override has source == Source.DEFAULT_OVERRIDE
+```
+
+
+
+TypeScript
+
+```ts
+const batch = schema.rows(Format.JSONEachRow, body, {
+ columnDefaults: [{ name: 'region', value: 'eu' }],
+});
+// a cell filled from the override has source Source.DefaultOverride
+```
+
+
+
+Rust
+
+```rust
+let options = RowsOptions {
+ column_defaults: Some(vec![ColumnDefault { name: b"region".to_vec(), value: b"eu".to_vec() }]),
+ ..Default::default()
+};
+let batch = schema.rows(Format::JsonEachRow, body, &options)?;
+// a cell filled from the override has source Source::DefaultOverride
+```
+
+
+
## Next
- [`transformations.md`](transformations.md) — what the accepted rows were silently changed into.
diff --git a/docs/guides/settings.md b/docs/guides/settings.md
index 0cbd44f2..18774227 100644
--- a/docs/guides/settings.md
+++ b/docs/guides/settings.md
@@ -98,7 +98,16 @@ This has cost real bugs, so v1 makes it one rule in all four bindings: **a setti
The reason is one setting in particular. `chtypes_now_epoch_nanos` is a 19-digit nanosecond epoch, and 19 digits do not survive an IEEE double: `1700000000123456789` becomes `1.7e+18`. Sent as a JSON number, the setting would be **silently ignored** — the batch keeps stamping the real wall clock, and nothing anywhere says so. A loud `TypeError` is the whole point.
-A value whose bytes are not valid UTF-8 cannot be passed as a setting or a query parameter in 1.0. Inline it in the SQL as `unhex('')`, built from hex digits only ([`limitations.md`](../limitations.md#binary-input-parameters-must-be-utf-8)).
+A value whose bytes are not valid UTF-8 is sent in the object form `{"b64": ""}` (ABI v2): a setting value, a query-parameter value, a `Setup` default and a value of a server profile's `settings`. The binding writes it in place of the string, so the library sees the exact bytes; a value that is valid UTF-8 stays a plain string. Three of the bindings can send one:
+
+| Binding | How a non-UTF-8 value is passed |
+| ---------- | -------------------------------------------------------------------------------------------------- |
+| Go | a `string` holding the bytes (a Go `string` is a byte string) |
+| Python | a `str` carrying the bytes with `surrogateescape` |
+| TypeScript | a `Uint8Array` (a `string` holding a lone surrogate has no bytes and is refused, as a key is) |
+| Rust | **cannot**: settings and query parameters take `String`, so a non-UTF-8 value is unreachable there |
+
+A key that is not valid UTF-8 cannot be carried in any binding (a JSON object has no byte form for a key), and neither can a macro's name or value. Each is refused before any call. Where a binding cannot send the bytes, inline them in the SQL as `unhex('')`, built from hex digits only ([`limitations.md`](../limitations.md#binary-input-parameters-must-be-utf-8)).
## An unknown name refuses the whole call
diff --git a/docs/reference/abi-v2.md b/docs/reference/abi-v2.md
index f15cda09..17bd90bf 100644
--- a/docs/reference/abi-v2.md
+++ b/docs/reference/abi-v2.md
@@ -37,17 +37,17 @@ The v2 header has its own include guard and refuses, with `#error`, to follow AB
## Reference
-
+
### Identity
| item | value |
| --------------------- | ----------------------------------------------------------------------------------- |
| `CHS_ABI_VERSION` | 2 |
-| `CHS_ABI_FINGERPRINT` | `sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4` |
+| `CHS_ABI_FINGERPRINT` | `sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2` |
| library | `libchtypes`, exporting only `chs_*` |
-| functions | 40: 33 api, 3 handshake, 1 tombstone, 3 tooling |
-| FIRM functions | 40 |
+| functions | 41: 34 api, 3 handshake, 1 tombstone, 3 tooling |
+| FIRM functions | 41 |
| provisional functions | 0 |
| `reuse_v0_names` | true |
| stability | UNSTABLE: the fingerprint moves with every change to the description until the lock |
@@ -63,12 +63,12 @@ These rules bind generation 2 from its first draft: the library, every binding a
**(r3) Every enum has an `unknown(n)` member.** Every enum the description defines, an `int32` enum and a string vocabulary alike (`chs_status`, `chs_format`, `value_src`, `default_kind`, the three outcomes, `filter_verdict`, `transform_reason`, every one), MUST have in every binding an explicit member `unknown(n)` carrying the raw value `n`: the integer of an `int32` enum, the exact string of a vocabulary. A reader MUST map a value its description does not list to that member, for that field alone, and go on decoding. An unlisted value never fails the document, the row or the batch that carries it. Checked by `gen.py`: no described value is spelled `unknown`, `CHS_UNKNOWN` or `CHS__UNKNOWN`, the name readers reserve for that member.
- **A `fallback` changes meaning.** It no longer names the value an unlisted one is read as: the reader keeps `unknown(n)`. It names whose per-value facts `unknown(n)` reports (`lossy` for `transform_reason`, `answered` for `filter_verdict`) and how a question about the value is answered, so the fail-closed reading stays: an unknown outcome is never accepted, and an unknown verdict is never answered.
-- **Facts without a fallback.** An enum whose values carry facts and which names no `fallback` MUST say what `unknown(n)` reports before the description is locked. Today that is `value_src`, whose `is_stored` has no answer for an unknown source.
+- **Facts need a fallback.** An enum whose values carry facts MUST name a `fallback`, so that what `unknown(n)` reports is decided before the description is locked. `value_src` names `skipped`: an unknown source reports `is_stored` false, which is how every dev binding already read it.
- **A call status.** A status outside the closed set reads as `unknown(n)` too. The call still fails, as an internal error naming `n` (the `unknown` entry of `errors.status` in `spec/abi-v2/sdk.json`), because a status a binding cannot read is never a success.
- **Schemas constrain the writer.** An `enum` inside a document schema (`default_kind` in `schema_description`, `framing.container` in `batch`) constrains what the library writes, never what a reader accepts.
- **Why.** Measured on the released 1.0.4 bindings, all four on 12 CI legs ([`v1-abi` run 37506595472](https://github.com/Wave-RF/chtypes/actions/runs/37506595472), the conformance legs of public pull request #509): a reader survives an unlisted value only where its vocabulary has a fallback (an outcome reads as `unsupported`, a verdict as `d`, a transform reason stays raw and reads as lossy), and an unknown call status is an internal error. An unlisted `value_src` fails the WHOLE document with an internal error, so one row's unknown source loses the entire batch; an unlisted `default_kind` fails `schema_description` the same way.
-**(r4) Error codes are stable once published.** An error code a caller can match on keeps its name and its number from the first release that carries it, a `2.0.0-dev.N` pre-release included, in every later release of every generation, and is never reused for another meaning. That covers the `chs_status` values, the `CHTYPES_*` codes (`errors.codes` and `errors.fetch_codes` in `sdk.json`, and the fetch layer's table in `docs/guides/fetch-v1.md` §8) and the exit status each code maps to. A new failure gets a new code, with an exit status no published code has. Checked by `gen.py`: every `chs_status` value generation 1 published keeps its name and number, and every `errors.codes` entry of `spec/abi-v1/sdk.json` is kept and none is reused under another key. The fetch layer's table is checked against `spec/fetch-v1/constants.json`, the file §8 is generated from: `errors.fetch_codes` lists every code in it with the same exit status and no code it lacks, every `errors.codes` entry names a listed code, and every code in `spec/abi-v2/published-fetch-codes.json` (what releases have published) is kept under its exit status in both files, and a new code takes a status none of them has. The baseline is a committed file the check reads, never git history.
+**(r4) Error codes are stable once published.** An error code a caller can match on keeps its name and its number from the first release that carries it, a `2.0.0-dev.N` pre-release included, in every later release of every generation, and is never reused for another meaning. That covers the `chs_status` values, the `CHTYPES_*` codes (`errors.codes` and `errors.fetch_codes` in `sdk.json`, and the fetch layer's table in `docs/guides/fetch-v1.md` §8) and the exit status each code maps to. A new failure gets a new code, with an exit status no published code has. Checked by `gen.py`: every `chs_status` value generation 1 published keeps its name and number, and every `errors.codes` entry of `spec/abi-v1/sdk.json` is kept and none is reused under another key. The fetch layer's table is checked against `spec/fetch-v1/constants.json`, the file §8 is generated from: `errors.fetch_codes` lists every code in it with the same exit status and no code it lacks, every `errors.codes` entry names a listed code, and every code in `spec/abi-v2/published-fetch-codes.json` (what releases have published) is kept under its exit status in both files, and a new code takes a status none of them has. The baseline is a committed file the check reads, never git history. The same holds for the decline catalog's reason ids (`spec/abi-v2/declines/declines.json`'s `id`, which `chs_error_reason` and every document's `reason` and `verdict_reason` carry) and for the sentinel `CHS_CODE_DECLINED`: an id keeps its meaning from the first release that carries it, a retired entry stays in the catalog under its id with a `retirement` naming its successors, and an id is never reused for another meaning. A new cause gets a new id. `scripts/abi-v2/check-declines.py` is where the check belongs, against the published catalog.
**(r5) The cache.** A generation-2 binding's cache MUST be one no released 1.x reader ever reads. The normative fetch specification for generation 2 is [`docs/guides/fetch-v1.md` §"Generation 2 after the lock"](../../docs/guides/fetch-v1.md#generation-2-after-the-lock), and r5 and r6 point at it rather than restate it.
@@ -96,6 +96,10 @@ These rules bind generation 2 from its first draft: the library, every binding a
Checked by two shared conformance cases (`tests/fixtures/abi-v2/setup-cases.json`, generated by `scripts/abi-v1/emit/setup_cases.py`), which every binding that can express them runs through its public API over the stub library: `server.a_closed_server_is_never_passed_as_null` and `filter.a_closed_filter_is_never_passed_as_null`. Each compiles or previews with an open handle (the stub sees a non-NULL handle), with no handle (the stub sees NULL), and with a closed handle (the binding's usage error, where a NULL passed in its place would have succeeded). The server case reads the stub's live-handle counts; a batch call mints no handle, so the filter case runs on the stub variant `filter-observable`, which answers `rows_passed` 1 when a filter reached it and 0 when it received NULL. Rust declares both cases not expressible, with the reason, in the case itself. `gen.py` checks only that this rule is stated (`GENERATION_RULES`), since which parameters give NULL a meaning is prose, not structure.
+**(r8) Crashes.** The library runs ClickHouse's own code on the caller's input, unmodified. A crash inside that code is a defect of security severity, like an over-accept, and is fixed or declined with a catalog entry in a later build; it is never documented as acceptable. Until then this generation does **not** promise that no input can terminate the process. What it promises is the following. (1) **The known-crash list.** Every build's release notes carry the producer's known-crash list: each entry names the line, the entry point, the input shape and the issue, and the producer's release gate counts those entries and fails on any crash not on the list. A list that is not empty is stated in the notes of every build that ships the line. (2) **The breaker.** A compile-class call (`chs_schema_create`, `chs_filter_create`, `chs_server_create`, `chs_set_defaults`, `chs_type_validate`, `chs_reference_type`, `chs_discover_columns`) records its input before it runs and clears the record after; a process that dies during the call leaves the record, and the next process of the same build on the same host declines that exact input with `CHS_DECLINED` and the reason id `vendored-crash-previously`, so a crashing input terminates a process at most once per host where the record can be written. Where it cannot (a read-only cache, no writable directory), the breaker is off and the library says so once at `chs_initialize`; `chs_build_info.capabilities.features` lists `crash_breaker`. An input whose shape is on the build's own known-crash list never reaches the code that crashes: `chs_schema_create` and `chs_filter_create` decline it at create, `CHS_DECLINED` with the reason id `known-crash-declined` and a message naming the list's rule id, the line, the clause and what to change. (3) **No wrong answer.** A breaker decline, like every decline, is never an accept and never a server's own refusal; an input the breaker does not know is answered exactly as without it. (4) **The exceptions** r8 does not cover in any generation: misuse the ABI forbids, exhaustion of the process's memory outside the admission budgets, and signals from outside the process. The full guarantee, no input terminates the process through any `chs_*` call, is generation 3's. `gen.py` checks only that this rule is stated.
+
+**(r9) The description evolves by reserve and decline.** After the lock a new input member, document member or enum value moves the fingerprint, so a feature that may follow the lock is reserved in the description before it: its input member is defined and a build that does not implement it declines it at any value but its default (Input documents, in the Preamble), and its output members are defined and absent until a build writes them. Implementing a reserved member is a behavior change, never a fingerprint change, and a build advertises each implemented reserved member in `chs_build_info`'s `capabilities.features`, spelled as the member's path (`insert_options.defer_defaults`). A reader treats an absent reserved output member as not reported. `gen.py` checks only that this rule is stated.
+
At the lock (public issue #511 says when), the description's `stability` becomes `locked`. That moves the fingerprint one last time, so a released 2.0.0 SDK never accepts a dev build and a dev SDK never accepts a 2.0.0 library. SDK 2.0.0 then ships from the production `chtypes/v2` repository, signed with the release key.
### Provisional markers
@@ -112,6 +116,13 @@ None: every entry in the description is FIRM.
| `process_once` | process setup, once per image: the first call sets process state, and a repeat with the same arguments is a no-op that is safe at any time |
| `process_serial` | process-level: must not overlap any other call into the same library image |
+### Init classes
+
+| class | when a call is valid |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `any` | valid before and after `chs_initialize` |
+| `required` | valid only once `chs_initialize` has succeeded: until then the call changes nothing and returns `CHS_INVALID_ARGUMENT`, ch_code 0, and with arguments that are otherwise valid its message starts `: call chs_initialize first` |
+
### Parameter kinds
| kind | C spelling | rule |
@@ -128,18 +139,18 @@ None: every entry in the description is FIRM.
Every counted input and every `chs_buf` is a byte string; the content says what it carries.
-| content | carries |
-| --------------------------- | ------------------------------------------------------------------------------------------- |
-| `bytes` | arbitrary bytes: a body, a string value, an export |
-| `name` | a ClickHouse column name: NUL and invalid UTF-8 are legal, so never assume UTF-8 |
-| `sql` | SQL text (a statement, an expression, a type, a quoted name or literal): may carry any byte |
-| `message` | a ClickHouse message: may quote input bytes, so may carry any byte |
-| `ascii` | guaranteed ASCII |
-| `json_object_string_values` | a JSON object whose values are JSON strings (settings, query parameters) |
-| `json_array_names` | a JSON array of column names, each an object carrying the name as byte_strings says |
-| `timezone` | a time zone name, validated by ClickHouse's own DateLUT; length 0 means UTC |
-| `document:` | a JSON document, listed under Documents below |
-| `input:` | a JSON object the library validates, listed under Input documents below |
+| content | carries |
+| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `bytes` | arbitrary bytes: a body, a string value, an export |
+| `name` | a ClickHouse column name: NUL and invalid UTF-8 are legal, so never assume UTF-8 |
+| `sql` | SQL text (a statement, an expression, a type, a quoted name or literal): may carry any byte |
+| `message` | a ClickHouse message: may quote input bytes, so may carry any byte |
+| `ascii` | guaranteed ASCII |
+| `json_object_byte_values` | a JSON object whose values are JSON strings, or the object `{"b64": ""}` for a value whose bytes are not valid UTF-8 (settings, query parameters) |
+| `json_array_names` | a JSON array of column names, each an object carrying the name as byte_strings says |
+| `timezone` | a time zone name, validated by ClickHouse's own DateLUT; length 0 means UTC |
+| `document:` | a JSON document, listed under Documents below |
+| `input:` | a JSON object the library validates, listed under Input documents below |
### Byte strings in JSON
@@ -147,15 +158,15 @@ Every data-derived string a JSON document carries (a column name, a value's rend
The data-derived fields and the entries carrying `value_b64`, by document (a path joins members with `.`, and `[]` steps into an array):
-| document | data-derived fields | entries with `value_b64` |
-| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- |
-| `live_handles` | - | - |
-| `error_code_table` | - | - |
-| `schema_description` | `columns[].name`, `columns[].type`, `columns[].default_expression`, `filter_declined_settings[].name`, `replicated.zookeeper_path`, `replicated.replica_name` | - |
-| `row` | `err`, `verdict_err`, `partition_id`, `unknown_fields[].name`, `unsupported_settings[].name`, `cols[].name`, `cols[].type`, `cols[].base`, `cols[].input`, `cols[].stored`, `cols[].ref`, `cols[].ref_type`, `cols[].wire`, `computed[].name`, `computed[].stored`, `transformed[].column`, `transformed[].input`, `transformed[].stored` | `cols[]`, `computed[]` |
-| `batch` | `err`, `export_declined`, `rows[].err`, `rows[].verdict_err`, `rows[].partition_id`, `rows[].unknown_fields[].name`, `rows[].unsupported_settings[].name`, `rows[].cols[].name`, `rows[].cols[].type`, `rows[].cols[].base`, `rows[].cols[].input`, `rows[].cols[].stored`, `rows[].cols[].ref`, `rows[].cols[].ref_type`, `rows[].cols[].wire`, `rows[].computed[].name`, `rows[].computed[].stored`, `rows[].transformed[].column`, `rows[].transformed[].input`, `rows[].transformed[].stored`, `engine_rows[][].name`, `engine_rows[][].stored`, `storage_transforms[].column`, `storage_transforms[].stored`, `at_merge[].column`, `at_merge[].stored`, `transformed[].column`, `transformed[].input`, `transformed[].stored`, `unsupported_settings[].name`, `framing.header.names[].name` | `rows[].cols[]`, `rows[].computed[]`, `engine_rows[][]`, `storage_transforms[]` |
-| `filter_result` | `err`, `unsupported_settings[].name`, `errors[].err` | - |
-| `discovery` | `columns[].name`, `columns[].type`, `columns[].default_kind`, `columns[].default_expression`, `columns[].declaration`, `columns_sql` | - |
+| document | data-derived fields | entries with `value_b64` |
+| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
+| `live_handles` | - | - |
+| `error_code_table` | - | - |
+| `schema_description` | `columns[].name`, `columns[].type`, `columns[].default_expression`, `columns[].reads[].name`, `filter_declined_settings[].name`, `replicated.zookeeper_path`, `replicated.replica_name` | - |
+| `row` | `err`, `verdict_err`, `partition_id`, `unknown_fields[].name`, `cols[].name`, `cols[].type`, `cols[].base`, `cols[].input`, `cols[].stored`, `cols[].ref`, `cols[].ref_type`, `cols[].wire`, `computed[].name`, `computed[].stored`, `transformed[].column`, `transformed[].input`, `transformed[].stored` | `cols[]`, `computed[]` |
+| `batch` | `err`, `export_declined`, `rows[].err`, `rows[].verdict_err`, `rows[].partition_id`, `rows[].unknown_fields[].name`, `rows[].cols[].name`, `rows[].cols[].type`, `rows[].cols[].base`, `rows[].cols[].input`, `rows[].cols[].stored`, `rows[].cols[].ref`, `rows[].cols[].ref_type`, `rows[].cols[].wire`, `rows[].computed[].name`, `rows[].computed[].stored`, `rows[].transformed[].column`, `rows[].transformed[].input`, `rows[].transformed[].stored`, `engine_rows[][].name`, `engine_rows[][].stored`, `storage_transforms[].column`, `storage_transforms[].stored`, `at_merge[].column`, `at_merge[].stored`, `transformed[].column`, `transformed[].input`, `transformed[].stored`, `unsupported_settings[].name`, `export_columns[].name`, `framing.header.names[].name` | `rows[].cols[]`, `rows[].computed[]`, `engine_rows[][]`, `storage_transforms[]` |
+| `filter_result` | `err`, `unsupported_settings[].name`, `errors[].err` | - |
+| `discovery` | `columns[].name`, `columns[].type`, `columns[].default_kind`, `columns[].default_expression`, `columns[].declaration`, `columns_sql` | - |
### Handles
@@ -199,8 +210,8 @@ A body parsed once under a schema, for evaluating many filters without parsing a
The verdict of every fallible call, in five values whose numbering is frozen.
- `CHS_OK`: the call succeeded and its outputs are set.
-- `CHS_REJECTED`: ClickHouse's own refusal, with its own code, name and message, which a server would also give. `CHS_REJECTED` with ch_code 306, TOO_DEEP_RECURSION, never `CHS_INTERNAL`, is returned for a deep argument the call compiles (an expression, a type, or a settings, columns or options JSON), at `chs_set_defaults`, `chs_server_create`, `chs_schema_create`, `chs_schema_describe`, `chs_preview_row`, `chs_preview_batch`, `chs_filter_create`, `chs_filter_eval_body`, `chs_block_create`, `chs_filter_eval_block`, `chs_type_validate`, `chs_reference_type` and `chs_discover_columns`. A body's depth refusal is reported in its document (`CHS_OK`, with code 306), as every refused body is.
-- `CHS_DECLINED`: this build will not answer; a server might accept. Never scored as agreement.
+- `CHS_REJECTED`: ClickHouse's own refusal, with its own code, name and message, which a server would also give. `CHS_REJECTED` with ch_code 306, TOO_DEEP_RECURSION, never `CHS_INTERNAL`, is returned for a deep argument the call compiles (an expression, a type, or a settings, columns or options JSON), at `chs_set_defaults`, `chs_server_create`, `chs_schema_create`, `chs_filter_create`, `chs_type_validate`, `chs_reference_type` and `chs_discover_columns`. A call that answers with a document (`chs_preview_row`, `chs_preview_batch`, `chs_filter_eval_body`, `chs_block_create` and `chs_filter_eval_block`) returns a status other than `CHS_OK` only for caller misuse (`CHS_INVALID_ARGUMENT`) or a library bug (`CHS_INTERNAL`): every refusal or decline of the call itself, of its settings, its column list or its options, a deep argument (306) included, is reported in its document with `refused_at` `call`, and a body's refusal with `refused_at` `row` or `body_step`, as every refused body is. `chs_schema_describe` never refuses a schema `chs_schema_create` returned: the statement's depth is judged at compile.
+- `CHS_DECLINED`: this build will not answer; a server might accept. Never scored as agreement. The error names the reason (`chs_error_reason`). A call that answers with a document never returns it: its decline is the document's `unsupported` with `refused_at` `call`.
- `CHS_INVALID_ARGUMENT`: caller misuse, such as a NULL pointer with a nonzero length, a wrong-kind, freed or cross-library handle, or a NULL required out-parameter.
- `CHS_INTERNAL`: a guarded exception inside the library, which is a library bug.
@@ -285,11 +296,15 @@ FIRM.
#### `value_src`
-Where a column's value came from. `is_stored` says whether the row as stored carries a value for the column: a column the input named but the table never stores (an EPHEMERAL column, a skipped MATERIALIZED or ALIAS column) and a DEFAULT the library could not resolve carry none.
+Where a column's value came from. `is_stored` says whether the row as stored carries a value for the column: a column the input named but the table never stores (an EPHEMERAL column, a skipped MATERIALIZED or ALIAS column) and a DEFAULT the library could not resolve carry none. A source this description does not list reports its fallback's facts, `skipped`'s: `is_stored` false (rule r3).
`default_generated` marks a DEFAULT column whose expression calls a random or ID generator the build admits; a build that does this lists `default_generators` in `chs_build_info`'s `capabilities.features`. The label means: generated by chtypes, and it becomes the stored value when you insert this output. The library drew the value itself, with ClickHouse's own vendored function. So insert the library's output (the export of `chs_preview_batch`, or the stored values its documents report), not your original input: a server evaluates a DEFAULT only for a column the INSERT does not supply, and the original input, which omits the column, would make the server draw a different value. Generators are admitted only in DEFAULT expressions: a MATERIALIZED column is always computed by the server, and an EPHEMERAL column is not stored. No binding logic is involved; a binding reports the source as it reads it.
-FIRM.
+`default_override` marks a column whose value came from `insert_options`' `column_defaults`: the caller's value in a cell the server would have filled from the column's DEFAULT. It is stored, and the export carries it as a present value. A build that does not implement `column_defaults` never writes it.
+
+`default_deferred` marks a DEFAULT column the call deferred to the server (`insert_options`' `defer_defaults`): the library computed no value, the export omits the column from every line and from the batch document's `export_columns` (a body whose rows do not all defer it is declined: `defer_defaults`), and the server fills it on INSERT. A build that does not implement `defer_defaults` never writes it.
+
+FIRM; an unrecognized value is read as `skipped`.
| value | is_stored |
| ----------------------------- | --------- |
@@ -304,6 +319,8 @@ FIRM.
| `default_expr_unsupported` | false |
| `default_volatile_unresolved` | false |
| `default_pending` | false |
+| `default_deferred` | false |
+| `default_override` | true |
#### `row_outcome`
@@ -333,7 +350,7 @@ The batch answers the first of these that applies:
A skipped row never changes the verdict. Steps 2 and 3 depart from "rejected if any row is refused" for one reason: never a refusal that is not definite. Both answer `unsupported`, which a caller never scores as agreement.
-A filter's verdicts follow the batch. When the batch's outcome is not `accepted`, by any of steps 1 to 4, every row's filter verdict is `d`. A row the parse accepted carries `verdict_code` -2 and `verdict_err` `batch outcome is '': only a fully accepted batch has filter verdicts`; a refused row carries its own code and message. Nothing from such a batch is stored for a WHERE to read, exported or counted, so no verdict of it is an answer.
+A filter's verdicts follow the batch. When the batch's outcome is not `accepted`, by any of steps 1 to 4, every row's filter verdict is `d`. A row the parse accepted carries `verdict_code` -2, its `verdict_reason`, and `verdict_err` `batch outcome is '': only a fully accepted batch has filter verdicts`; a refused row carries its own code and message. Nothing from such a batch is stored for a WHERE to read, exported or counted, so no verdict of it is an answer.
FIRM; an unrecognized value is read as `unsupported`.
@@ -346,9 +363,9 @@ FIRM; an unrecognized value is read as `unsupported`.
#### `filter_outcome`
-Whether a filter evaluation answers its body. `ok`: every row carries its own verdict, and per-row failures are verdicts, not outcomes. Any non-`ok` outcome, including `unsupported`, means: treat every verdict as `d`. A caller MUST do so whatever the verdict string holds.
+Whether a filter evaluation answers its body. `ok`: every row carries its own verdict, and per-row failures are verdicts, not outcomes. Under any non-`ok` outcome, including `unsupported`, the library writes `d` for every verdict, on `chs_filter_eval_body` and `chs_filter_eval_block` alike, so the rule holds in the data; a caller MUST still treat every verdict as `d` whatever the verdict string holds. The document's `refused_at`, `refused_row` and `reason` say where the refusal sits and why.
-`chs_filter_eval_body` and `chs_filter_eval_block` answer `ok` only for a body a server would accept, that is, when `chs_preview_batch`'s verdict over the same body and settings, with no filter, is `accepted`. Otherwise the outcome mirrors that verdict: `rejected` with its code and message; `unsupported` with its code and message, a call-level decline included; and `unsupported` with code -2 for `accepted_poisoned`, which this vocabulary does not hold and which is no refusal of the INSERT. A body the parse itself refuses answers `rejected` or `unsupported` with no verdicts.
+`chs_filter_eval_body` and `chs_filter_eval_block` answer `ok` only for a body a server would accept, that is, when `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is `accepted`. Otherwise the outcome mirrors that verdict: `rejected` with its code and message; `unsupported` with its code and message, a call-level decline included; and `unsupported` with code -2 for `accepted_poisoned`, which this vocabulary does not hold and which is no refusal of the INSERT. A body the parse itself refuses answers `rejected` or `unsupported`, with a `d` for each record read (`rows_read`, 0 when nothing was read).
FIRM; an unrecognized value is read as `unsupported`.
@@ -393,6 +410,19 @@ FIRM; an unrecognized value is read as `d`.
| `e` | false |
| `d` | false |
+#### `refusal_site`
+
+Where a refusal or decline sits, as `refused_at` reports it on a batch document, a filter result and `chs_preview_row`'s row whenever the outcome is `rejected` or `unsupported` (Conditional members, in the Preamble): `call`, the call itself (its settings, its column list, its options, or a decline of the call such as a zone mismatch), before a byte of the body was read; `body_read`, the body outside any row and before the whole-body steps, so no record is to blame: a header the format reads first (a `WithNames` or `WithNamesAndTypes` header line, a header field naming a column the INSERT does not list under `input_format_skip_unknown_fields` 0), the framing (a JSON array's brackets, a separator or trailing content after the last record) or a byte-order mark the reader refuses; `row`, one row of the body, whose index is `refused_row`; `body_step`, a whole-body step over rows the parse accepted (the partition split, the writer's index expressions, the engine merge, the TTL, the projections, a CHECK over a merged block). `refused_row` is present exactly for `row`. A site added later is a new value, which a reader reads as its `unknown(n)` (rule r3).
+
+FIRM.
+
+| value |
+| ----------- |
+| `call` |
+| `body_read` |
+| `row` |
+| `body_step` |
+
#### `discover_query_param`
The query parameters in the SQL `chs_discover_query` returns, each with its ClickHouse type. The caller binds them when it runs the query (over HTTP, as `param_database` and `param_table`), so no binding builds SQL.
@@ -418,9 +448,22 @@ FIRM.
| `ALIAS` |
| `EPHEMERAL` |
+#### `default_support`
+
+What this build does with a column's DEFAULT, as `schema_description` reports it per column (reserved: a build that does not report it writes no `default_support`): `computed`, the library evaluates the expression itself (a literal, a deterministic expression of the row, an admitted clock read); `generated`, the library draws the value with an admitted generator (`default_generated`); `deferred`, the library cannot compute it and `insert_options`' `defer_defaults` `server` would defer it; `declined`, the library cannot compute it, so a body that omits the column is declined, with the catalog id in `default_support_reason`. The honor set behind it is published per line as data beside `where-settings/.json`. A value added later is a new value, which a reader reads as its `unknown(n)` (rule r3).
+
+FIRM.
+
+| value |
+| ----------- |
+| `computed` |
+| `generated` |
+| `deferred` |
+| `declined` |
+
#### `declined_tier`
-The tier of a setting a settings layer sets and this build's filters do not honor in a WHERE, as `schema_description`'s `filter_declined_settings` reports it, spelled as this build line's WHERE-settings list (`spec/abi-v2/where-settings/.json`) spells it: `predicate` (measured to change a verdict), `predicate-unflipped` (read in the WHERE's scope, though no value the flip test tried changed a verdict) or `result-content` (selects rows by content). The lists' other tiers (`execution`, `result-truncate` and `output`) never appear here: those settings are passed through and ignored. A tier added later is a new value, which a reader reads as its `unknown(n)` (rule r3).
+The tier of a setting a settings layer sets and this build's filters do not honor in a WHERE, as `schema_description`'s `filter_declined_settings` reports it, spelled as this build line's WHERE-settings list (`spec/abi-v2/where-settings/.json`) spells it: `predicate` (measured to change a verdict), `predicate-unflipped` (read in the WHERE's scope, though no value the flip test tried changed a verdict) `result-content` (selects rows by content), or `server-refuses` (a cell of `where-settings/documented-pathological.tsv`: a value under which the server refuses every query, so no row verdict can stand). The lists' other tiers (`execution`, `result-truncate` and `output`) never appear here: those settings are passed through and ignored. A tier added later is a new value, which a reader reads as its `unknown(n)` (rule r3).
FIRM.
@@ -429,6 +472,7 @@ FIRM.
| `predicate` |
| `predicate-unflipped` |
| `result-content` |
+| `server-refuses` |
#### `declined_layer`
@@ -442,12 +486,24 @@ FIRM.
| `server` |
| `schema` |
+#### `posture`
+
+How `chs_schema_create` compiles its statement, as `schema_options`' `posture` asks and `schema_description` reports: `create`, as a server runs a `CREATE TABLE` (ClickHouse's `LoadingStrictnessLevel::CREATE`), the default; `load`, as a server loads the stored definition of a table it already holds when it starts (`LoadingStrictnessLevel::FORCE_ATTACH`). `schema_description` carries the member only for `load`: absent is `create`. A posture added later is a new value, which a reader reads as its `unknown(n)` (rule r3).
+
+FIRM.
+
+| value |
+| -------- |
+| `create` |
+| `load` |
+
### Constants
| constant | type | value | status |
| ---------------------------- | ------ | ----- | ------ |
| `CHS_ABI_REVISION_TOMBSTONE` | int32 | 1001 | FIRM |
| `CHS_EXPORT_NONE` | int32 | -1 | FIRM |
+| `CHS_CODE_DECLINED` | int32 | -2 | FIRM |
| `CHS_DOC_VALUES` | uint32 | 1 | FIRM |
| `CHS_DOC_TRANSFORMS` | uint32 | 2 | FIRM |
| `CHS_DOC_DEFAULTS` | uint32 | 4 | FIRM |
@@ -494,11 +550,13 @@ The fields the description fixes, as JSON Schema; the document may carry more:
#### document `schema_description`
-A schema's columns, in declared order, with each column's canonical type, its `default_kind` (a value of the `default_kind` vocabulary) and default expression, and the facts a caller needs to build a row. The document is `{"columns": [...]}`, and each entry carries `name`, `type`, `default_kind` (a value of the `default_kind` vocabulary), `default_expression` (empty when there is none) and `default_is_literal`; the name, the type and the expression follow the rule for byte strings in JSON.
+A schema's columns, in declared order, with each column's canonical type, its `default_kind` (a value of the `default_kind` vocabulary) and default expression, and the facts a caller needs to build a row. The document is `{"columns": [...]}`, and each entry carries `name`, `type` and `default_kind` (a value of the `default_kind` vocabulary), always, and `default_expression` (empty when there is none) and `default_is_literal`; the name, the type and the expression follow the rule for byte strings in JSON. `type` is present on every column: a column whose type the compile cannot settle (a column declared without a type whose own DEFAULT or ALIAS this build cannot type) is declined by `chs_schema_create` (`CHS_DECLINED`, reason `column-type-unsettled`), so no schema describes a column without its type. The decline is a limit of this build, never a claim about whether a server accepts the table. `default_support`, a `default_support` value with `default_support_reason` for `declined`, says what this build does with the column's DEFAULT; reserved, absent until a build writes it. `reads`, a list of name objects, names the columns the column's DEFAULT, MATERIALIZED, ALIAS or EPHEMERAL expression reads, as ClickHouse's own analysis of the expression's required source columns reports them, each once, in the table's declaration order: a build that writes it writes it on every column, `[]` for a column with no expression or one that reads none. Reserved (rule r9): absent until a build writes it, which that build advertises in `chs_build_info`'s `capabilities.features`; a reader treats it as not reported until then. A schema `chs_schema_create` returned always describes: the walk describing needs runs at compile, so a compiled schema is never refused here.
+
+`posture`, a `posture` value, is present only on a schema compiled in a posture other than `create`, so the document of a schema compiled in `create` is byte for byte what it was before postures existed.
Two members report the server a schema was created on. Both are absent on a schema created with `server` NULL, so that document is byte for byte what it was before servers existed.
-- `server`, on every schema created on a server, even one described by `{}`: `timezone` is the zone the schema's types bind (the profile's, or else the image zone, exactly as `chs_initialize` spelled it), so a reader never needs the image zone separately; `settings` is the server's settings as the profile gave them, `{}` when it gave none; `macros` is the server's macro set as the library holds it, present exactly when the profile carried `macros`. These are the caller's own JSON strings given back, not data-derived, so they are plain strings. The member no longer carries `filter_declined_settings`: that list is the top-level one, below, over every layer.
+- `server`, on every schema created on a server, even one described by `{}`: `timezone` is the zone the schema's types bind (the profile's, or else the image zone, exactly as `chs_initialize` spelled it), so a reader never needs the image zone separately; `settings` is the server's settings as the profile gave them, `{}` when it gave none; `macros` is the server's macro set as the library holds it, present exactly when the profile carried `macros`. These are the caller's own values given back, not data-derived, so the rule for byte strings in JSON does not apply to them: the zone and the macros are plain strings, and each settings value is a JSON string when its bytes are valid UTF-8 and otherwise the object form `{"b64": …}` (Input values), so a settings value takes the object form only when its bytes are not UTF-8. The member no longer carries `filter_declined_settings`: that list is the top-level one, below, over every layer.
- `replicated`, on a schema created on a server whose engine is a Replicated one: `zookeeper_path` and `replica_name`, the path and the replica name ClickHouse's own `TableZnodeInfo` resolved, fully expanded. The database and table names expanded into them are DDL bytes, so both follow the rule for byte strings in JSON. A caller can compare them with the server's own `system.replicas`.
`filter_declined_settings` is required on every schema description, with or without a server: an array, `[]` when nothing is declined. It lists the settings the schema compiles under that this build's filters do not honor in a WHERE, over every layer known when `chs_schema_create` ran, and `chs_schema_create` computes it. The layers, lowest first, are the `declined_layer` values:
@@ -509,14 +567,18 @@ Two members report the server a schema was created on. Both are absent on a sche
Each entry is `{name, tier, layer}`: `name` follows the rule for byte strings in JSON (`name` or `name_b64`), `tier` is a `declined_tier` value, the tier this build line's WHERE-settings list gives the name, and `layer` is a `declined_layer` value. The entries are:
-- every name set at any of the three layers that the line's list carries at tier `predicate`, `predicate-unflipped` or `result-content` and that is not in this build's honor set (the settings it applies to a WHERE, each with a measured case where the server's verdict moves and the library's tracks it);
-- `compatibility`, at tier `predicate`, when its value at the highest layer that sets it moves a listed setting outside the honor set.
+- every name set at any of the three layers that the line's list carries at tier `predicate` or `predicate-unflipped` and that is not in this build's honor set (the settings it applies to a WHERE, each with a measured case where the server's verdict moves and the library's tracks it);
+- `compatibility`, at tier `predicate`, when its value at the highest layer that sets it moves a listed setting outside the honor set;
+- every `result-content` name set at any of the three layers, honored or not: a setting that selects rows by content beyond the WHERE is outside a row verdict's scope whatever the library applies;
+- every cell of the line's `where-settings/documented-pathological.tsv` present at any of the three layers, at tier `server-refuses`: the server refuses every query under that value, so no row verdict stands.
+
+So the consumer rule is one check: refuse when the list is not empty. No consumer copies the WHERE-settings lists or the pathological cells.
A name the list does not carry is never an entry, and neither is a setting of the `execution`, `result-truncate` or `output` tiers: those are passed through and ignored. The test is presence, not value: any value at any layer counts, a declined name set to its default value included, because some of ClickHouse's reads are `isChanged()`, which asks only whether a setting was set. A higher layer that sets the same name never takes the entry away; it only moves the entry's `layer`.
The order is deterministic. There is one entry per name, at the highest layer that sets it (`defaults`, then `server`, then `schema`). The entries are ordered by layer, `defaults` first, and within a layer by name as raw bytes, ascending; a `name_b64` entry sorts by its decoded bytes.
-The list holds only the layers known at schema compile. A filter's own `settings` (`chs_filter_create`) are never in it: a declined one there is named at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`). The settings an evaluation brings (the call's, and the body's) are never in it either: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings (`chs_filter_create`). A schema whose list is not empty is one `chs_filter_create` declines.
+The list holds only the layers known at schema compile. A filter's own `settings` (`chs_filter_create`) are never in it: a declined one there is named at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`), and a cell of `documented-pathological.tsv` there declines `chs_filter_create` itself (`setting-server-refuses`). The settings an evaluation brings (the call's, and the body's) are never in it either: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings (`chs_filter_create`). A schema whose list is not empty is one `chs_filter_create` declines.
The fields the description fixes, as JSON Schema; the document may carry more:
@@ -532,15 +594,21 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"type": "string",
"pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
},
- "tier": {"type": "string"},
- "layer": {"type": "string"}
+ "tier": {
+ "vocabulary": "declined_tier",
+ "enum": ["predicate", "predicate-unflipped", "result-content", "server-refuses"]
+ },
+ "layer": {"vocabulary": "declined_layer", "enum": ["defaults", "server", "schema"]}
},
"allOf": [{"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]}]
},
"column": {
"type": "object",
"properties": {
- "default_kind": {"enum": ["", "DEFAULT", "MATERIALIZED", "ALIAS", "EPHEMERAL"]},
+ "default_kind": {
+ "vocabulary": "default_kind",
+ "enum": ["", "DEFAULT", "MATERIALIZED", "ALIAS", "EPHEMERAL"]
+ },
"default_is_literal": {"type": "boolean"},
"name": {"type": "string"},
"name_b64": {
@@ -556,17 +624,14 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"default_expression_b64": {
"type": "string",
"pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
- }
+ },
+ "default_support": {"vocabulary": "default_support", "enum": ["computed", "generated", "deferred", "declined"]},
+ "default_support_reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
+ "reads": {"type": "array", "items": {"$ref": "#/$defs/named"}}
},
"allOf": [
{"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]},
- {
- "oneOf": [
- {"required": ["type"], "properties": {"type_b64": false}},
- {"required": ["type_b64"], "properties": {"type": false}},
- {"properties": {"type": false, "type_b64": false}}
- ]
- },
+ {"oneOf": [{"required": ["type"]}, {"required": ["type_b64"]}]},
{
"oneOf": [
{"required": ["default_expression"], "properties": {"default_expression_b64": false}},
@@ -574,7 +639,29 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"default_expression": false, "default_expression_b64": false}}
]
}
- ]
+ ],
+ "required": ["default_kind"]
+ },
+ "bytes_value": {
+ "type": "object",
+ "required": ["b64"],
+ "properties": {
+ "b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ }
+ }
+ },
+ "named": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string"},
+ "name_b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ }
+ },
+ "allOf": [{"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]}]
}
},
"type": "object",
@@ -587,7 +674,10 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"required": ["timezone", "settings"],
"properties": {
"timezone": {"type": "string"},
- "settings": {"type": "object", "additionalProperties": {"type": "string"}},
+ "settings": {
+ "type": "object",
+ "additionalProperties": {"oneOf": [{"type": "string"}, {"$ref": "#/$defs/bytes_value"}]}
+ },
"macros": {"type": "object", "additionalProperties": {"type": "string"}}
}
},
@@ -609,14 +699,15 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"oneOf": [{"required": ["zookeeper_path"]}, {"required": ["zookeeper_path_b64"]}]},
{"oneOf": [{"required": ["replica_name"]}, {"required": ["replica_name_b64"]}]}
]
- }
+ },
+ "posture": {"vocabulary": "posture", "enum": ["load"]}
}
}
```
#### document `row`
-One row's verdict and, as the flags ask, its columns' stored values, provenance and transformations. Every entry of `cols` carries its name, `null` (whether the stored value is NULL, poisoned cells included), its renderings (`input`, `stored`, `ref`, `wire`) and its types (`type`, `base`, `ref_type`), all by the rule for byte strings in JSON, and `value_b64` for a String or FixedString value. `unknown_fields` and `unsupported_settings` are lists of name objects; `computed` entries carry a name, a kind and a stored value; `err`, `verdict_err` and `partition_id` follow the rule too. The transformations come from the library, never from a binding: each is the `transformed` entry the SDK's parity fixtures pin, with its `reason` from `transform_reason`. `input_span` is `{off, len}`, the bytes of the input body the reader consumed for this record, read from the vendored reader's own position.
+One row's verdict and, as the flags ask, its columns' stored values, provenance and transformations. `outcome` (a `row_outcome` value), `code` (Codes, in the Preamble) and `input_span` are always present, `err` names the refusal, and `reason` the catalog id of a decline. With a filter attached (`chs_preview_batch`), `verdict` (a `filter_verdict` value), `verdict_code`, `verdict_err` and, for `d`, `verdict_reason` are present on every row; with none they are absent. On `chs_preview_row`'s document, which stands for the whole call, `refused_at` and `refused_row` say where a refusal sits (`refusal_site`); the rows of a batch carry neither. Every entry of `cols` carries its name, `null` (whether the stored value is NULL, poisoned cells included), `src` (a `value_src` value), its renderings (`input`, `stored`, `ref`, `wire`) and its types (`type`, `base`, `ref_type`), all by the rule for byte strings in JSON, `value_b64` for a String or FixedString value, and the booleans `nullable` (the column's type is Nullable), `poison` (the stored value is one the server cannot read back), `null_input` (the input field was the format's NULL literal) and `dup_dropped` (a duplicate key the reader dropped), with `ref_unclassified` present, true, when the reference type could not be classified. `unknown_fields` is a list of name objects (the call's declined settings are the batch document's `unsupported_settings` only, which every row of the batch shares); `computed` entries carry a name, a `kind` (the column's default kind, lower case) and a stored value; `err`, `verdict_err` and `partition_id` follow the rule too. The transformations come from the library, never from a binding: each is the `transformed` entry the SDK's parity fixtures pin, with its `reason` from `transform_reason` and its `lossy` fact. `input_span` is `{off, len}`, the bytes of the input body the reader consumed for this record, read from the vendored reader's own position. A `rejected` row always carries a nonzero ClickHouse code, and an `unsupported` one `CHS_CODE_DECLINED` with its `reason`.
The fields the description fixes, as JSON Schema; the document may carry more:
@@ -644,9 +735,32 @@ The fields the description fixes, as JSON Schema; the document may carry more:
},
"col": {
"type": "object",
- "required": ["null"],
+ "required": ["null", "src"],
"properties": {
"null": {"type": "boolean"},
+ "src": {
+ "vocabulary": "value_src",
+ "enum": [
+ "input",
+ "default",
+ "default_substituted",
+ "default_generated",
+ "absent",
+ "materialized_input",
+ "skipped",
+ "ephemeral_input",
+ "default_expr_unsupported",
+ "default_volatile_unresolved",
+ "default_pending",
+ "default_deferred",
+ "default_override"
+ ]
+ },
+ "nullable": {"type": "boolean"},
+ "poison": {"type": "boolean"},
+ "null_input": {"type": "boolean"},
+ "dup_dropped": {"type": "boolean"},
+ "ref_unclassified": {"type": "boolean"},
"name": {"type": "string"},
"name_b64": {
"type": "string",
@@ -753,6 +867,7 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"type": "string",
"pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
},
+ "kind": {"type": "string"},
"stored": {"type": "string"},
"stored_b64": {
"type": "string",
@@ -772,12 +887,43 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"stored": false, "stored_b64": false}}
]
}
- ]
+ ],
+ "required": ["kind"]
},
"transform": {
"type": "object",
"properties": {
"row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "reason": {
+ "vocabulary": "transform_reason",
+ "enum": [
+ "overflow_wrap",
+ "null_to_default",
+ "null_loss",
+ "decimal_truncate",
+ "date_clamp",
+ "datetime_wrap",
+ "date_shift",
+ "uuid_mangle",
+ "ip_mangle",
+ "float_precision",
+ "lossy_numeric",
+ "fixedstring_pad",
+ "emptied",
+ "element_changed",
+ "enum_coerce",
+ "value_changed",
+ "poisoned",
+ "duplicate_key_dropped",
+ "reformat",
+ "default_filled",
+ "zero_filled",
+ "default_materialized",
+ "ttl_expired",
+ "ttl_column_expired"
+ ]
+ },
+ "lossy": {"type": "boolean"},
"column": {"type": "string"},
"column_b64": {
"type": "string",
@@ -810,15 +956,23 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"stored": false, "stored_b64": false}}
]
}
- ]
+ ],
+ "required": ["reason", "lossy"]
}
},
"type": "object",
- "required": ["input_span"],
+ "required": ["outcome", "code", "input_span"],
"properties": {
+ "outcome": {
+ "vocabulary": "row_outcome",
+ "enum": ["accepted", "accepted_poisoned", "rejected", "skipped", "unsupported"]
+ },
+ "code": {"type": "integer", "minimum": -2},
+ "verdict": {"vocabulary": "filter_verdict", "enum": ["t", "f", "e", "d"]},
+ "verdict_code": {"type": "integer", "minimum": -2},
+ "verdict_reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
"input_span": {"$ref": "#/$defs/span"},
"unknown_fields": {"type": "array", "items": {"$ref": "#/$defs/named"}},
- "unsupported_settings": {"type": "array", "items": {"$ref": "#/$defs/named"}},
"cols": {"type": "array", "items": {"$ref": "#/$defs/col"}},
"computed": {"type": "array", "items": {"$ref": "#/$defs/computed"}},
"transformed": {"type": "array", "items": {"$ref": "#/$defs/transform"}},
@@ -827,7 +981,10 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"verdict_err": {"type": "string"},
"verdict_err_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"},
"partition_id": {"type": "string"},
- "partition_id_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"}
+ "partition_id_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"},
+ "reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
+ "refused_at": {"vocabulary": "refusal_site", "enum": ["call", "body_read", "row", "body_step"]},
+ "refused_row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991}
},
"allOf": [
{
@@ -857,13 +1014,19 @@ The fields the description fixes, as JSON Schema; the document may carry more:
#### document `batch`
-A body's verdict, counts and per-row documents (each a row document, with its own `input_span`), and, when an export was asked for, where each accepted row sits in the export bytes. `engine_rows`, present when the table's engine merges rows at insert, is a list of rows, each a list of cells: `{name, stored, null}`, plus `value_b64` for a String or FixedString value. `storage_transforms` entries carry `row`, `column`, `reason` and `stored`, plus `value_b64`; `transformed` entries carry `row`, `column`, `input`, `stored`, `reason` and `lossy`. Every name, rendering, `err` and `export_declined` follows the rule for byte strings in JSON.
+A body's verdict, counts and per-row documents, and, when an export ran, where each row sits in the export bytes. Always present: `outcome` (a `batch_outcome` value), `code` (Codes, in the Preamble), `err`, `rows` (each a row document, with its own `input_span`), `rows_read` (the records the reader read, skipped ones included), `rows_skipped` (the records skipped under the server's error allowance), `unsupported_settings`, `unconsumed` and `framing`. Present with a filter attached: `rows_passed` and `rows_cut`, which partition the accepted rows of an `accepted` batch by verdict (`t` against the rest) and read 0 and 0 on any other outcome. Present on a schema with a partition key: `partition_count`, the distinct partitions the stored rows span, which with each row's `partition_id` lets a caller cap the partitions one INSERT touches (`max_partitions_per_insert_block`). Present whenever the outcome is not `accepted` or `accepted_poisoned`: `refused_at` (a `refusal_site` value), `refused_row` for `row`, and `reason` for a decline; `rows_read` is 0 when nothing was read. With an export asked for, exactly one of `row_spans` and `export_declined` is present: `row_spans` when the export ran, one entry per `rows[]` entry, index-aligned, `{off, len}` into `out_export` for an exported row (its whole output line, `\n` included, so slicing the spans out of the bytes is the per-row payload and concatenating them reproduces it) and `null` for a row not exported; `export_declined`, naming the cause, with `export_declined_reason`, the cause's catalog id, when it did not (Conditional members, in the Preamble). When the export ran, `export_columns` is present too: the positional column list of every line in `out_export`, as name objects: the stored non-computed columns in declaration order, independent of the call's `columns` list, a column outside `insertable_columns` included, with the value the library drew, resolved or filled for it, less the columns every row defers (`defer_defaults`). A caller inserts the export naming `export_columns` as the column list; it never derives a list of its own. `export_columns` is the export's only column-list member, so no second list can disagree with it.
+
+The export's truth table, by outcome, export asked for and filter verdict: with no export asked for, `out_export` is NULL and neither `row_spans` nor `export_declined` is present. With an export asked for and the outcome `accepted`, the export runs: `out_export` is the bytes (an empty buffer when no row is exported) and `row_spans` is present, with a span for every row whose outcome is `accepted` and, with a filter attached, whose verdict is `t`, and `null` for every other row (`skipped`, and a `f`, `e` or `d` row under a filter); the rows are exported in body order. With an export asked for and any other outcome (`accepted_poisoned`, `rejected`, `unsupported`), nothing is exported: `out_export` is NULL and `export_declined` names the cause (the outcome, a poisoned value), with `export_declined_reason` `export-batch-not-accepted`. With an export asked for and the outcome `accepted` but the serialization failing, or the export's format unable to carry a stored value, `out_export` is NULL, `export_declined` names the cause and `export_declined_reason` its catalog id; the batch's `outcome` stays `accepted`, with code 0 and no `reason`, so an export decline is keyed by `export_declined_reason`, never by the batch's `reason`. So a caller tells a declined export from an empty one by `export_declined`, never by the buffer, and never slices the payload by an index other than the row's own.
+
+A computed value the export cannot carry. A column the export omits is recomputed by the server that inserts the export. A DEFAULT that reads an EPHEMERAL value is stored, so the export carries its value and the stored row is the same. A MATERIALIZED expression that reads an EPHEMERAL value the row supplied is not: the export carries no EPHEMERAL column, so the inserting server computes the MATERIALIZED value from the EPHEMERAL's type default instead, and the stored row differs from the one the document reports. Such a body's export is declined (`export_declined`, with `export_declined_reason` `export-computed-value-not-carried`): a caller that needs that stored row inserts the body.
-`unsupported_settings`, at the top level, is a list of name objects with the same shape as a row document's: the call's own settings that the library declined. It is empty when none was, and when it is not empty the batch's outcome is at least `unsupported` (see `batch_outcome`).
+Composition. A caller that concatenates accepted bodies into one INSERT (a worker's batch, async inserts) keeps every per-row verdict: a row's `outcome`, its stored values, its `partition_id` and its filter verdict are functions of the row and the schema alone. The whole-body steps are not compositional, and can refuse the concatenation after each body was accepted: the partition split (`max_partitions_per_insert_block`, code 252, bounded by the sum of the bodies' `partition_count`), an insert-time engine merge across bodies, `rowNumberInAllBlocks()` and the like, and a CHECK over a merged async block, which fails every INSERT in it. A caller that must not see those caps its INSERT by `partition_count` and keeps CHECK-bearing bodies isolated. `engine_rows`, present when the table's engine merges rows at insert, is a list of rows, each a list of cells: `{name, stored, null}`, plus `value_b64` for a String or FixedString value. `storage_transforms` entries carry `row`, `column`, `reason` and `stored`, plus `value_b64`; `transformed` entries carry `row`, `column`, `input`, `stored`, `reason` and `lossy`. Every name, rendering, `err` and `export_declined` follows the rule for byte strings in JSON.
-`at_merge` lists what an `OPTIMIZE TABLE ... FINAL` of the part this `INSERT` writes would do, at the call's clock instant. It is absent when nothing would happen. `engine_rows` stays exactly what the `INSERT` writer produces, so the two never contradict each other. Each entry is `{row, reason, column, stored, input_rows}`:
+`unsupported_settings`, at the top level, is always present: a list of name objects, the call's own settings that the library declined, which every row of the batch shares. It is empty when none was, and when it is not empty the batch's outcome is at least `unsupported` (see `batch_outcome`).
-- `row` indexes `engine_rows`: the part's row, not the input body's. An engine's insert-time merge (Collapsing, Replacing, Summing, Aggregating) runs first, inside the `INSERT` writer, so `engine_rows` already excludes the rows it merged away, and after a Summing or Aggregating merge the part's rows do not map one to one to the input's.
+`at_merge` lists what an `OPTIMIZE TABLE ... FINAL` of the part this `INSERT` writes would do, at the call's clock instant. It is absent when nothing would happen. `engine_rows` stays exactly what the `INSERT` writer produces, so the two never contradict each other. Each entry is `{part_row, reason, column, stored, input_rows}`:
+
+- `part_row` indexes `engine_rows`: the part's row, not the input body's. Every other `row` member of this document (`transformed[].row`, `storage_transforms[].row`, `refused_row`) and of `filter_result` (`errors[].row`) indexes `rows[]`, the input body's rows, and `input_rows` here does too. An engine's insert-time merge (Collapsing, Replacing, Summing, Aggregating) runs first, inside the `INSERT` writer, so `engine_rows` already excludes the rows it merged away, and after a Summing or Aggregating merge the part's rows do not map one to one to the input's.
- `reason` is a `merge_reason` value: `ttl_delete` (the rows TTL removes the row) or `ttl_column_reset` (a column TTL resets the column's value). A row that is removed and also carries column resets lists only `ttl_delete`: it is removed in the same merge pass, so its column values are never observable.
- `column` names the column, by the rule for byte strings in JSON, and is present only for `ttl_column_reset`.
- `stored` is the column's value after the reset, by the same rule. It is present when that value is decided at the `INSERT`: the column's `DEFAULT` is a constant or a deterministic expression of the row, or the column has no `DEFAULT` (the type's default). It is absent when the `DEFAULT` reads the clock (`now()`) or a generator (`rand()`, `generateUUIDv4()`), and the entry then means that the value is reset at the merge and decided then.
@@ -902,9 +1065,32 @@ The fields the description fixes, as JSON Schema; the document may carry more:
},
"col": {
"type": "object",
- "required": ["null"],
+ "required": ["null", "src"],
"properties": {
"null": {"type": "boolean"},
+ "src": {
+ "vocabulary": "value_src",
+ "enum": [
+ "input",
+ "default",
+ "default_substituted",
+ "default_generated",
+ "absent",
+ "materialized_input",
+ "skipped",
+ "ephemeral_input",
+ "default_expr_unsupported",
+ "default_volatile_unresolved",
+ "default_pending",
+ "default_deferred",
+ "default_override"
+ ]
+ },
+ "nullable": {"type": "boolean"},
+ "poison": {"type": "boolean"},
+ "null_input": {"type": "boolean"},
+ "dup_dropped": {"type": "boolean"},
+ "ref_unclassified": {"type": "boolean"},
"name": {"type": "string"},
"name_b64": {
"type": "string",
@@ -1011,6 +1197,7 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"type": "string",
"pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
},
+ "kind": {"type": "string"},
"stored": {"type": "string"},
"stored_b64": {
"type": "string",
@@ -1030,12 +1217,43 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"stored": false, "stored_b64": false}}
]
}
- ]
+ ],
+ "required": ["kind"]
},
"transform": {
"type": "object",
"properties": {
"row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "reason": {
+ "vocabulary": "transform_reason",
+ "enum": [
+ "overflow_wrap",
+ "null_to_default",
+ "null_loss",
+ "decimal_truncate",
+ "date_clamp",
+ "datetime_wrap",
+ "date_shift",
+ "uuid_mangle",
+ "ip_mangle",
+ "float_precision",
+ "lossy_numeric",
+ "fixedstring_pad",
+ "emptied",
+ "element_changed",
+ "enum_coerce",
+ "value_changed",
+ "poisoned",
+ "duplicate_key_dropped",
+ "reformat",
+ "default_filled",
+ "zero_filled",
+ "default_materialized",
+ "ttl_expired",
+ "ttl_column_expired"
+ ]
+ },
+ "lossy": {"type": "boolean"},
"column": {"type": "string"},
"column_b64": {
"type": "string",
@@ -1068,15 +1286,23 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"stored": false, "stored_b64": false}}
]
}
- ]
+ ],
+ "required": ["reason", "lossy"]
},
"row": {
"type": "object",
- "required": ["input_span"],
+ "required": ["outcome", "code", "input_span"],
"properties": {
+ "outcome": {
+ "vocabulary": "row_outcome",
+ "enum": ["accepted", "accepted_poisoned", "rejected", "skipped", "unsupported"]
+ },
+ "code": {"type": "integer", "minimum": -2},
+ "verdict": {"vocabulary": "filter_verdict", "enum": ["t", "f", "e", "d"]},
+ "verdict_code": {"type": "integer", "minimum": -2},
+ "verdict_reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
"input_span": {"$ref": "#/$defs/span"},
"unknown_fields": {"type": "array", "items": {"$ref": "#/$defs/named"}},
- "unsupported_settings": {"type": "array", "items": {"$ref": "#/$defs/named"}},
"cols": {"type": "array", "items": {"$ref": "#/$defs/col"}},
"computed": {"type": "array", "items": {"$ref": "#/$defs/computed"}},
"transformed": {"type": "array", "items": {"$ref": "#/$defs/transform"}},
@@ -1094,7 +1320,10 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"partition_id_b64": {
"type": "string",
"pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
- }
+ },
+ "reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
+ "refused_at": {"vocabulary": "refusal_site", "enum": ["call", "body_read", "row", "body_step"]},
+ "refused_row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991}
},
"allOf": [
{
@@ -1183,10 +1412,10 @@ The fields the description fixes, as JSON Schema; the document may carry more:
},
"at_merge": {
"type": "object",
- "required": ["row", "reason"],
+ "required": ["part_row", "reason"],
"properties": {
- "row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
- "reason": {"type": "string"},
+ "part_row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "reason": {"vocabulary": "merge_reason", "enum": ["ttl_delete", "ttl_column_reset"]},
"column": {"type": "string"},
"column_b64": {
"type": "string",
@@ -1218,8 +1447,22 @@ The fields the description fixes, as JSON Schema; the document may carry more:
}
},
"type": "object",
- "required": ["rows", "unconsumed", "framing"],
+ "required": [
+ "outcome",
+ "code",
+ "rows",
+ "rows_read",
+ "rows_skipped",
+ "unsupported_settings",
+ "unconsumed",
+ "framing"
+ ],
"properties": {
+ "outcome": {
+ "vocabulary": "batch_outcome",
+ "enum": ["accepted", "accepted_poisoned", "rejected", "unsupported"]
+ },
+ "code": {"type": "integer", "minimum": -2},
"rows": {"type": "array", "items": {"$ref": "#/$defs/row"}},
"unsupported_settings": {"type": "array", "items": {"$ref": "#/$defs/named"}},
"engine_rows": {"type": "array", "items": {"type": "array", "items": {"$ref": "#/$defs/cell"}}},
@@ -1251,8 +1494,19 @@ The fields the description fixes, as JSON Schema; the document may carry more:
},
"err": {"type": "string"},
"err_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"},
+ "rows_read": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "rows_skipped": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "rows_passed": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "rows_cut": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "partition_count": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "row_spans": {"type": "array", "items": {"oneOf": [{"$ref": "#/$defs/span"}, {"type": "null"}]}},
+ "export_columns": {"type": "array", "items": {"$ref": "#/$defs/named"}},
+ "reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
+ "refused_at": {"vocabulary": "refusal_site", "enum": ["call", "body_read", "row", "body_step"]},
+ "refused_row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
"export_declined": {"type": "string"},
- "export_declined_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"}
+ "export_declined_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"},
+ "export_declined_reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"}
},
"allOf": [
{
@@ -1275,9 +1529,9 @@ The fields the description fixes, as JSON Schema; the document may carry more:
#### document `filter_result`
-A filter evaluation: the call's outcome, one verdict character per row (`filter_verdict`), and each error or declined row itemized as `{row, code, err}`. `err`, at the top level and per row, follows the rule for byte strings in JSON, and `unsupported_settings` is a list of name objects.
+A filter evaluation. Always present: `outcome` (a `filter_outcome` value), `code` (Codes, in the Preamble), `err`, `rows_read`, `verdicts` (one `filter_verdict` character per row read, so its length is `rows_read`, and every character `d` under any outcome but `ok`), `unsupported_settings` (a list of name objects) and `errors`, each error or declined row itemized as `{row, code, reason, err}` with `row` indexing the body's rows, `code` nonzero (ClickHouse's for `e`, `CHS_CODE_DECLINED` for `d`) and `reason` the catalog id of a `d`. Present whenever the outcome is not `ok`: `refused_at` (a `refusal_site` value), `refused_row` for `row`, and `reason` for a decline. `err`, at the top level and per row, follows the rule for byte strings in JSON.
-`unsupported_settings` (after `rows_read`, before `verdicts`) names the settings the parse itself declined, and then, after them and without repeats, every setting of the filter's own `settings` that this build's filters do not honor in a WHERE. When it names one of those, every verdict is `d` and every row is itemized in `errors` as `{row, code, err}` with `code` -2. `outcome` follows its own rules all the same: `ok` for a body a server would accept, and the mirror of `chs_preview_batch`'s verdict otherwise (`filter_outcome`). A declined setting from the `chs_set_defaults`, server or schema layer never reaches an evaluation, because `chs_filter_create` declines first (`schema_description`'s `filter_declined_settings`).
+`unsupported_settings` (after `rows_read`, before `verdicts`) names the settings the parse itself declined, and then, after them and without repeats, every setting of the filter's own `settings` that this build's filters do not honor in a WHERE. When it names one of those, every verdict is `d` and every row is itemized in `errors` as `{row, code, reason, err}` with `code` -2. `outcome` follows its own rules all the same: `ok` for a body a server would accept, and the mirror of `chs_preview_batch`'s verdict otherwise (`filter_outcome`). A declined setting from the `chs_set_defaults`, server or schema layer never reaches an evaluation, because `chs_filter_create` declines first (`schema_description`'s `filter_declined_settings`).
The fields the description fixes, as JSON Schema; the document may carry more:
@@ -1299,6 +1553,8 @@ The fields the description fixes, as JSON Schema; the document may carry more:
"type": "object",
"properties": {
"row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "code": {"type": "integer", "minimum": -2},
+ "reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
"err": {"type": "string"},
"err_b64": {
"type": "string",
@@ -1313,15 +1569,23 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"err": false, "err_b64": false}}
]
}
- ]
+ ],
+ "required": ["row", "code"]
}
},
"type": "object",
"properties": {
+ "outcome": {"vocabulary": "filter_outcome", "enum": ["ok", "rejected", "unsupported"]},
+ "code": {"type": "integer", "minimum": -2},
+ "rows_read": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "verdicts": {"type": "string"},
"unsupported_settings": {"type": "array", "items": {"$ref": "#/$defs/named"}},
"errors": {"type": "array", "items": {"$ref": "#/$defs/filter_error"}},
"err": {"type": "string"},
- "err_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"}
+ "err_b64": {"type": "string", "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"},
+ "reason": {"type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"},
+ "refused_at": {"vocabulary": "refusal_site", "enum": ["call", "body_read", "row", "body_step"]},
+ "refused_row": {"type": "integer", "minimum": 0, "maximum": 9007199254740991}
},
"allOf": [
{
@@ -1331,7 +1595,8 @@ The fields the description fixes, as JSON Schema; the document may carry more:
{"properties": {"err": false, "err_b64": false}}
]
}
- ]
+ ],
+ "required": ["outcome", "code", "rows_read", "unsupported_settings", "verdicts", "errors"]
}
```
@@ -1429,18 +1694,20 @@ The fields the description fixes, as JSON Schema; the document may carry more:
Each is a JSON object a caller passes in a `bytes_in` parameter whose content is `input:`. The library refuses a key the document does not define, so each schema closes every object (rule r1). An options document is the one a growable function takes the inputs it gains in.
-| input document | options | taken by |
-| ---------------- | ------- | ----------------------------- |
-| `server_profile` | no | `chs_server_create` `profile` |
-| `server_options` | yes | `chs_server_create` `options` |
-| `schema_options` | yes | `chs_schema_create` `options` |
+| input document | options | taken by |
+| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
+| `server_profile` | no | `chs_server_create` `profile` |
+| `server_options` | yes | `chs_server_create` `options` |
+| `schema_options` | yes | `chs_schema_create` `options` |
+| `insert_options` | yes | `chs_preview_row` `options`, `chs_preview_batch` `options`, `chs_filter_eval_body` `options`, `chs_block_create` `options` |
+| `filter_options` | yes | `chs_filter_create` `options` |
#### input `server_profile`
A server's profile, `{"timezone": "", "settings": {"": ""}, "macros": {"": ""}}`. Every member is optional, so `{}` describes a server with nothing known about it. The profile holds server facts only, in the shape a server's own tables report them, so a caller can fill it from the server it describes; library knobs go in `chs_server_create`'s options instead.
- `timezone`: the server's own zone, the zone its tables bind. An empty string, or no member, means the image zone. The name is validated by ClickHouse's own `DateLUT`, which reads the library host's time zone database.
-- `settings`: the settings the server's profile applies to every query, each value a JSON string, never rewritten. The library accepts exactly what the server's own SET check accepts. `session_timezone` here is the default session zone of the server's user.
+- `settings`: the settings the server's profile applies to every query, each value a JSON string or the object form for bytes (Input values), never rewritten. A `chtypes_*` knob is never one of them (`chs_set_defaults`). The library accepts exactly what the server's own SET check accepts. `session_timezone` here is the default session zone of the server's user.
- `macros`: the server's macros, as its `system.macros` lists them (`macro`, `substitution`). Absent, they are unknown; present, even as `{}`, they are the complete set (see `chs_server_create`). A macro name that ClickHouse's own macro reader refuses (for example `a.b`) is refused with `CHS_INVALID_ARGUMENT` naming it.
Its members, as JSON Schema:
@@ -1451,12 +1718,28 @@ Its members, as JSON Schema:
"additionalProperties": false,
"properties": {
"timezone": {"type": "string"},
- "settings": {"type": "object", "additionalProperties": {"type": "string"}},
+ "settings": {
+ "type": "object",
+ "additionalProperties": {"oneOf": [{"type": "string"}, {"$ref": "#/$defs/bytes_value"}]}
+ },
"macros": {
"type": "object",
"propertyNames": {"minLength": 1},
"additionalProperties": {"type": "string"}
}
+ },
+ "$defs": {
+ "bytes_value": {
+ "type": "object",
+ "additionalProperties": false,
+ "required": ["b64"],
+ "properties": {
+ "b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ }
+ }
+ }
}
}
```
@@ -1473,12 +1756,97 @@ Its members, as JSON Schema:
#### input `schema_options`
-The options of `chs_schema_create` (rule r1). It defines no member yet, so `{}` (or length 0) is the only valid document, and any key is `CHS_INVALID_ARGUMENT`, naming the key. A choice about how one statement is compiled lands here, as a new member: for example, compiling a server's own `create_table_query`, read back from the server, rather than a caller's CREATE.
+The options of `chs_schema_create` (rule r1): how one statement is compiled. Its member is `posture`, a `posture` value, and `{}` (or length 0) compiles in `create`, as before the member existed. A key it does not define is `CHS_INVALID_ARGUMENT`, naming the key, and so is a `posture` that is not one of its values, naming the value.
+
+- `create`: the statement is a `CREATE TABLE` a server would run now, with every check a server's CREATE runs, under the settings layers (`chs_schema_create`).
+- `load`: the statement is the stored definition of a table the server already holds, as that server reports it (`system.tables.create_table_query`, `SHOW CREATE TABLE`, or the table's metadata file), compiled as the server loads it when it starts. It is one switch, and ClickHouse's own: every check on the compile's path that reads the loading posture reads `FORCE_ATTACH` instead of `CREATE`, in ClickHouse's own code, and nothing else changes. So the checks a server applies only to a new table are not applied: the type gates (`allow_suspicious_low_cardinality_types`, `allow_suspicious_fixed_string_types` and the rest, at the compile and at every call on the schema), the codec, statistics and suspicious-index checks, the DEFAULT validation of a typed column, the TTL checks, the MergeTree settings' constraint and sanity checks, and the floating-point partition-key, nullable-key and deprecated-syntax gates. `flatten_nested` and `data_type_default_nullable` are not applied either: a stored definition is already flattened and typed. Which checks these are on each supported line is the artifact producer's enumeration from the vendored source; no list is kept here. A projection that a starting server skips because it no longer parses is declined, because whether a given server still holds it depends on that server's history. A Replicated engine's ZooKeeper path and replica name never change a row verdict, so under `load` a stored definition whose engine arguments read a macro the server's `macros` cannot expand (or carry none, as a Replicated database's table does) compiles as the server loads it: `schema_description` then carries no `replicated` member, and only an answer that depends on the path (none today) is declined.
+
+**The caller's obligation.** `load` asserts that the server holds a table defined by exactly this statement. The library cannot check that assertion, because it never connects to the server. A statement a server would refuse at `CREATE` compiles under `load`, and every answer on the schema is the answer for INSERTs into that stored table. If no server holds such a table, every answer describes a table that does not exist, and an accepted row is evidence of nothing. A statement a server reported for a table it holds (its `create_table_query`, `SHOW CREATE TABLE`, or its metadata file) MUST be compiled in `load`, and a statement a caller wrote or edited in `create`. A server's `create_table_query` compiled in `create` can be wrong in both directions, because the server created the table under its own CREATE's settings, which the statement does not carry: an over-reject where a gate was set (a `LowCardinality(UInt8)` column) and an over-accept where a shape setting was (a `Nested` column of a table created with `flatten_nested = 0`).
Its members, as JSON Schema:
```json
-{"type": "object", "additionalProperties": false}
+{
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {"posture": {"enum": ["create", "load"]}}
+}
+```
+
+#### input `insert_options`
+
+The options of every call that models one INSERT of a body: `chs_preview_row`, `chs_preview_batch`, `chs_block_create` and `chs_filter_eval_body` (rule r1). Its members: `insertable_columns`, `column_defaults` and `verdict_basis`, `defer_defaults`.
+
+- `insertable_columns`: the columns an INSERT may supply, as name objects: the server's column-level INSERT grant (`GRANT INSERT(a, b) ON t`), checked as the server checks it, on the INSERT's header and never on the fields a body supplies. The header is the call's `columns` list, or with none the server's own no-list header: every ordinary and DEFAULT column, in declaration order (so a positional CSV, TSV, Values or JSONCompactEachRow body with no list supplies all of them). The check runs after the list's own refusals (an unknown or ALIAS column, 16; a duplicate, 15; a MATERIALIZED column, 44, unless `insert_allow_materialized_columns` is set, when it needs the grant like any other) and before any byte of the body is read. A header naming a column outside the set is refused 497 (ACCESS_DENIED), with `refused_at` `call`, in every format and whatever the body holds, an empty or a malformed body included. So `columns` MUST be a subset of the set, and a caller whose set does not cover every ordinary and DEFAULT column passes `columns`, at every call that takes this document, `chs_filter_eval_body` included: with a list a call follows the listed INSERT's rules, and with none the no-list rule. A body field naming a column the header omits (a JSON key, a `WithNames` header field) is an unknown field, never a grant question: skipped, or refused 117 under `input_format_skip_unknown_fields` 0 (`refused_at` `row` for a JSON key, `body_read` for a header field), and never stored. An omitted column keeps its declared kind, exactly as with no set: a generator DEFAULT draws (`default_generated`), a clock DEFAULT resolves, a literal DEFAULT fills, a MATERIALIZED column computes and an EPHEMERAL column takes its type default. The export carries every stored non-computed column, those outside the set included (`document:batch`), so the INSERT of the export needs a grant that covers its column list. A Nested column is granted per subcolumn (`n.x`). The set is a grant, not a column list: like a grant it may name an EPHEMERAL or MATERIALIZED column, or a name the table lacks. The 497's message is ClickHouse's own, naming the user `default` and the table `default.t`, since a schema carries neither a user nor a table name; it is informative. Absent is every column. Implemented in the first build that carries this description.
+- `column_defaults`: per-call DEFAULT overrides, a list of `{name, value}` (each by the rule for byte strings in JSON). It is one rule: in a cell the server would fill from the column's DEFAULT, the call uses the caller's `value` instead. A supplied value wins: the override fills exactly the cells ClickHouse's own input format marks missing for that column under the call's format settings, as the vendored reader decides it, and no other: a field the record omits, a `null` for a non-Nullable column under `input_format_null_as_default` 1, and an empty CSV field under `input_format_csv_empty_as_default` 1. An empty JSON string `""` is a supplied value. A Nullable column's `null` stays `NULL` where the server keeps it (the implementing build confirms this against the server's own missing mark). Such a cell reads `default_override`, and everything that reads the column sees the value as an INSERT that supplied it would: a CHECK, the sorting, primary and partition keys, a skip index, a projection, a TTL, a MATERIALIZED or DEFAULT expression that reads it, and a filter. The export carries it as a stored, present value at the column's position in `export_columns`, so the INSERT of the export needs no override, and a column the override fills is not deferred in those rows (`defer_defaults`). An entry may name an ordinary or a DEFAULT column, and an EPHEMERAL column where the implementing build measures it equal to a supplied EPHEMERAL value; an entry that names an unknown column, a column that is not insertable, or a MATERIALIZED or ALIAS column is `CHS_INVALID_ARGUMENT` at `call`, naming `column_defaults[i]`. When `insertable_columns` is given, an entry's column MUST be in it, because the INSERT the override models names the column in its header; one outside it is the same `CHS_INVALID_ARGUMENT`. `value` is read in the body format's own field encoding, as if the record supplied this field (the implementing build confirms); a value that does not read as the column's type is `CHS_INVALID_ARGUMENT` at `call`, naming `column_defaults[i]`, before any byte of the body is read. Built before the lock: the build that implements it lists `insert_options.column_defaults` in `capabilities.features` (rule r9), and a build that does not declines it as Input documents (in the Preamble) says.
+- `verdict_basis`: `input`, the default, or `export`: what row a filter verdict answers for. Under `input`, a filter that reads a value the library drew or resolved itself is `d` (`drawn-value`), because the server would draw its own. Under `export`, the verdict answers for the row the export would store, so a drawn or resolved value is answerable, and the caller MUST insert the export. Reserved: declined at `export` until implemented.
+- A member whose value is a list or an object is refused `CHS_INVALID_ARGUMENT` when an entry does not match its shape, naming the key. `{}` (or length 0) is the INSERT these calls modeled before the document existed, byte for byte. Each member means the same at every call that takes the document. A key it does not define is `CHS_INVALID_ARGUMENT`, naming the key, and so is a value that is not one of the member's, naming the value; a member this build does not implement declines as Input documents (in the Preamble) says.
+
+- `defer_defaults`: `none`, the default, or `server` (public issue #479). `server` defers to the server each DEFAULT this build would otherwise decline because its value is the server's to compute (a generator or a server function the build does not admit). Such a column reads `default_deferred` in the rows that omit it, and the export omits it from those rows' lines. What reads it there is not answered (the deferred-column rule): a filter that reads it makes the row's verdict `d` (`verdict_code` `CHS_CODE_DECLINED`, `verdict_reason` `defer-defaults-reader`); a CHECK, a sorting, primary or partition key, a skip index, a projection, a TTL, or a MATERIALIZED or other DEFAULT expression that reads it declines the row (`unsupported`, `reason` `defer-defaults-reader`); and such a row carries no `partition_id`. Admitted generators keep drawing (`default_generated`), so a drawn id stays usable for deduplication. A body is mixed when its exported rows' lines would not all omit the same columns (some rows supply a deferred column and others omit it). The export has one column list, so a mixed body is declined, and only when an export is asked for: the batch is `unsupported`, `refused_at` `body_step`, `reason` `defer-defaults-mixed-body`; with no export asked for no line omits anything, and the batch is answered. A body whose rows all defer the same columns, or none, is never mixed: its export omits those columns from every line and from `export_columns`. Declined until implemented.
+
+Its members, as JSON Schema:
+
+```json
+{
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "defer_defaults": {"enum": ["none", "server"]},
+ "insertable_columns": {"type": "array", "items": {"$ref": "#/$defs/named"}},
+ "column_defaults": {"type": "array", "items": {"$ref": "#/$defs/column_default"}},
+ "verdict_basis": {"enum": ["input", "export"]}
+ },
+ "$defs": {
+ "named": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "name": {"type": "string"},
+ "name_b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ }
+ },
+ "allOf": [{"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]}]
+ },
+ "column_default": {
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {
+ "name": {"type": "string"},
+ "name_b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ },
+ "value": {"type": "string"},
+ "value_b64": {
+ "type": "string",
+ "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ }
+ },
+ "allOf": [
+ {"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]},
+ {"oneOf": [{"required": ["value"]}, {"required": ["value_b64"]}]}
+ ]
+ }
+ }
+}
+```
+
+#### input `filter_options`
+
+The options of `chs_filter_create` (rule r1): how one filter is compiled. `{}` (or length 0) compiles as before the document existed. A key it does not define is `CHS_INVALID_ARGUMENT`, naming the key, and so is a value that is not one of the member's, naming the value; a member this build does not implement declines as Input documents (in the Preamble) says.
+
+- `param_encoding`: `escaped`, the default, or `raw`: how a query parameter's value is read. Under `escaped`, the value is decoded by ClickHouse's escaped-text reader, exactly as a server decodes an HTTP `param_` value, so a backslash, a tab or a newline must be escaped as that reader expects; the object form `{"b64": ...}` is decoded from base64 first and the bytes then read the same way. Under `raw`, the bytes are the value. Reserved: declined at `raw` until implemented (rule r9).
+
+Its members, as JSON Schema:
+
+```json
+{
+ "type": "object",
+ "additionalProperties": false,
+ "properties": {"param_encoding": {"enum": ["escaped", "raw"]}}
+}
```
### build_info
@@ -1502,48 +1870,49 @@ Its members, as JSON Schema:
### Functions
-| function | class | thread | returns | status |
-| -------------------------- | --------- | -------------- | ----------------- | ------ |
-| `chs_abi_version` | handshake | any | `int32_t` | FIRM |
-| `chs_build_info` | handshake | any | `const char *` | FIRM |
-| `chs_clickhouse_version` | handshake | any | `const char *` | FIRM |
-| `chs_abi_revision` | tombstone | any | `int` | FIRM |
-| `chs_buf_data` | api | shared | `const uint8_t *` | FIRM |
-| `chs_buf_len` | api | shared | `size_t` | FIRM |
-| `chs_buf_free` | api | handle_serial | `void` | FIRM |
-| `chs_error_status` | api | shared | `chs_status` | FIRM |
-| `chs_error_ch_code` | api | shared | `int32_t` | FIRM |
-| `chs_error_ch_name` | api | shared | `chs_buf *` | FIRM |
-| `chs_error_message` | api | shared | `chs_buf *` | FIRM |
-| `chs_error_column` | api | shared | `chs_buf *` | FIRM |
-| `chs_error_free` | api | handle_serial | `void` | FIRM |
-| `chs_live_handles` | api | any | status | FIRM |
-| `chs_error_codes` | api | any | status | FIRM |
-| `chs_registered_families` | tooling | process_serial | status | FIRM |
-| `chs_function_flags` | tooling | process_serial | status | FIRM |
-| `chs_reference_type` | tooling | process_serial | status | FIRM |
-| `chs_initialize` | api | process_once | status | FIRM |
-| `chs_set_defaults` | api | process_serial | status | FIRM |
-| `chs_shutdown` | api | process_serial | `void` | FIRM |
-| `chs_type_validate` | api | any | status | FIRM |
-| `chs_back_quote` | api | any | status | FIRM |
-| `chs_back_quote_if_needed` | api | any | status | FIRM |
-| `chs_quote_string` | api | any | status | FIRM |
-| `chs_server_create` | api | any | status | FIRM |
-| `chs_server_free` | api | handle_serial | `void` | FIRM |
-| `chs_schema_create` | api | shared | status | FIRM |
-| `chs_schema_free` | api | handle_serial | `void` | FIRM |
-| `chs_schema_describe` | api | shared | status | FIRM |
-| `chs_preview_row` | api | shared | status | FIRM |
-| `chs_preview_batch` | api | shared | status | FIRM |
-| `chs_filter_create` | api | shared | status | FIRM |
-| `chs_filter_free` | api | handle_serial | `void` | FIRM |
-| `chs_filter_eval_body` | api | shared | status | FIRM |
-| `chs_block_create` | api | shared | status | FIRM |
-| `chs_block_free` | api | handle_serial | `void` | FIRM |
-| `chs_filter_eval_block` | api | shared | status | FIRM |
-| `chs_discover_query` | api | any | status | FIRM |
-| `chs_discover_columns` | api | any | status | FIRM |
+| function | class | thread | init | returns | status |
+| -------------------------- | --------- | -------------- | -------- | ----------------- | ------ |
+| `chs_abi_version` | handshake | any | any | `int32_t` | FIRM |
+| `chs_build_info` | handshake | any | any | `const char *` | FIRM |
+| `chs_clickhouse_version` | handshake | any | any | `const char *` | FIRM |
+| `chs_abi_revision` | tombstone | any | any | `int` | FIRM |
+| `chs_buf_data` | api | shared | any | `const uint8_t *` | FIRM |
+| `chs_buf_len` | api | shared | any | `size_t` | FIRM |
+| `chs_buf_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_error_status` | api | shared | any | `chs_status` | FIRM |
+| `chs_error_ch_code` | api | shared | any | `int32_t` | FIRM |
+| `chs_error_ch_name` | api | shared | any | `chs_buf *` | FIRM |
+| `chs_error_message` | api | shared | any | `chs_buf *` | FIRM |
+| `chs_error_column` | api | shared | any | `chs_buf *` | FIRM |
+| `chs_error_reason` | api | shared | any | `chs_buf *` | FIRM |
+| `chs_error_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_live_handles` | api | any | required | status | FIRM |
+| `chs_error_codes` | api | any | required | status | FIRM |
+| `chs_registered_families` | tooling | process_serial | required | status | FIRM |
+| `chs_function_flags` | tooling | process_serial | required | status | FIRM |
+| `chs_reference_type` | tooling | process_serial | required | status | FIRM |
+| `chs_initialize` | api | process_once | any | status | FIRM |
+| `chs_set_defaults` | api | process_serial | required | status | FIRM |
+| `chs_shutdown` | api | process_serial | any | `void` | FIRM |
+| `chs_type_validate` | api | any | required | status | FIRM |
+| `chs_back_quote` | api | any | required | status | FIRM |
+| `chs_back_quote_if_needed` | api | any | required | status | FIRM |
+| `chs_quote_string` | api | any | required | status | FIRM |
+| `chs_server_create` | api | any | required | status | FIRM |
+| `chs_server_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_schema_create` | api | shared | required | status | FIRM |
+| `chs_schema_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_schema_describe` | api | shared | required | status | FIRM |
+| `chs_preview_row` | api | shared | required | status | FIRM |
+| `chs_preview_batch` | api | shared | required | status | FIRM |
+| `chs_filter_create` | api | shared | required | status | FIRM |
+| `chs_filter_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_filter_eval_body` | api | shared | required | status | FIRM |
+| `chs_block_create` | api | shared | required | status | FIRM |
+| `chs_block_free` | api | handle_serial | any | `void` | FIRM |
+| `chs_filter_eval_block` | api | shared | required | status | FIRM |
+| `chs_discover_query` | api | any | required | status | FIRM |
+| `chs_discover_columns` | api | any | required | status | FIRM |
#### `chs_abi_version`
@@ -1551,7 +1920,7 @@ Its members, as JSON Schema:
CHS_API int32_t chs_abi_version(void);
```
-- Class `handshake`, thread `any`, FIRM.
+- Class `handshake`, thread `any`, init `any`, FIRM.
The ABI generation this library implements: always 2 for this header. A loader resolves and calls it right after opening the library. A library without the symbol is not a chtypes ABI artifact of generation 1 or later, and any other value is refused, naming both values. An absent handshake symbol other than this one is refused as `missing_symbol:` instead, at whichever step first resolves it — only this symbol's absence means the library is not such an artifact at all.
@@ -1561,7 +1930,7 @@ The ABI generation this library implements: always 2 for this header. A loader r
CHS_API const char *chs_build_info(void);
```
-- Class `handshake`, thread `any`, FIRM.
+- Class `handshake`, thread `any`, init `any`, FIRM.
What this library is, as static, NUL-terminated, ASCII-only JSON in the image's read-only data: never NULL, never freed, the same bytes on every call, and callable straight after opening the library. Its fields are listed under `build_info` in the reference, and `abi_fingerprint` is `CHS_ABI_FINGERPRINT` as the library was built, copied, never recomputed.
@@ -1575,7 +1944,7 @@ A loader parses it (ASCII only, duplicate keys refused, `schema` equal to 1, eve
CHS_API const char *chs_clickhouse_version(void);
```
-- Class `handshake`, thread `any`, FIRM.
+- Class `handshake`, thread `any`, init `any`, FIRM.
The ClickHouse release this library was built from, spelled as every v0 library spelled it (for example `26.8.15.10-lts`): static, never freed. It keeps its v0 signature and spelling so a v0 binding reaches its own refusal. A binding of generation 1 or later reads `clickhouse_version` from `chs_build_info` instead.
@@ -1585,7 +1954,7 @@ The ClickHouse release this library was built from, spelled as every v0 library
CHS_API int chs_abi_revision(void);
```
-- Class `tombstone`, thread `any`, FIRM.
+- Class `tombstone`, thread `any`, init `any`, FIRM.
- Always returns `CHS_ABI_REVISION_TOMBSTONE`.
The v0 tombstone. It always returns `CHS_ABI_REVISION_TOMBSTONE` (1001). It must never be removed, and its name must never be given another meaning.
@@ -1598,7 +1967,7 @@ A sweep of every released binding (every minor from 0.1 to 0.5, in all four lang
CHS_API const uint8_t *chs_buf_data(const chs_buf *buf);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- The pointer is borrowed from `buf`.
- `buf`: handle, `chs_buf`.
@@ -1610,7 +1979,7 @@ The buffer's bytes. The pointer may be NULL when `chs_buf_len` is 0, and a bindi
CHS_API size_t chs_buf_len(const chs_buf *buf);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- `buf`: handle, `chs_buf`.
The buffer's length in bytes. A NULL, freed, wrong-kind or cross-library buffer gives 0.
@@ -1621,7 +1990,7 @@ The buffer's length in bytes. A NULL, freed, wrong-kind or cross-library buffer
CHS_API void chs_buf_free(chs_buf *buf);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `buf`: handle, `chs_buf`, nullable.
Releases the caller's reference to a buffer. Freeing NULL does nothing.
@@ -1632,7 +2001,7 @@ Releases the caller's reference to a buffer. Freeing NULL does nothing.
CHS_API chs_status chs_error_status(const chs_error *error);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- `error`: handle, `chs_error`.
The error's status, never `CHS_OK` for an error a call set. A NULL, freed, wrong-kind or cross-library error gives `CHS_INVALID_ARGUMENT`.
@@ -1643,7 +2012,7 @@ The error's status, never `CHS_OK` for an error a call set. A NULL, freed, wrong
CHS_API int32_t chs_error_ch_code(const chs_error *error);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- `error`: handle, `chs_error`.
ClickHouse's own error code, nonzero only when the status is `CHS_REJECTED`, and 0 otherwise or for an invalid error.
@@ -1654,7 +2023,7 @@ ClickHouse's own error code, nonzero only when the status is `CHS_REJECTED`, and
CHS_API chs_buf *chs_error_ch_name(const chs_error *error);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- Returns a new `chs_buf` the caller owns (ascii), freed with `chs_buf_free`.
- `error`: handle, `chs_error`.
@@ -1666,11 +2035,11 @@ A new buffer holding the name this build's vendored error table gives the code,
CHS_API chs_buf *chs_error_message(const chs_error *error);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- Returns a new `chs_buf` the caller owns (message), freed with `chs_buf_free`.
- `error`: handle, `chs_error`.
-A new buffer holding the message: ClickHouse's own text, verbatim, for `CHS_REJECTED`, and the library's explanation otherwise. A message may quote input bytes, so it is a byte string, not text.
+A new buffer holding the message: ClickHouse's own text, verbatim, for `CHS_REJECTED`, and the library's explanation otherwise. A message may quote input bytes, so it is a byte string, not text. It is informative: a caller keys on the status, the code and the reason (`chs_error_reason`), never on message text.
#### `chs_error_column`
@@ -1678,19 +2047,31 @@ A new buffer holding the message: ClickHouse's own text, verbatim, for `CHS_REJE
CHS_API chs_buf *chs_error_column(const chs_error *error);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `any`, FIRM.
- Returns a new `chs_buf` the caller owns (name), freed with `chs_buf_free`.
- `error`: handle, `chs_error`.
A new buffer holding the name of the column the error concerns, as raw bytes, empty when there is none. A column name is a byte string: NUL and invalid UTF-8 are legal in a name.
+#### `chs_error_reason`
+
+```c
+CHS_API chs_buf *chs_error_reason(const chs_error *error);
+```
+
+- Class `api`, thread `shared`, init `any`, FIRM.
+- Returns a new `chs_buf` the caller owns (ascii), freed with `chs_buf_free`.
+- `error`: handle, `chs_error`.
+
+A new buffer holding the reason: the catalog id (`spec/abi-v2/declines/declines.json`, ASCII) of a decline or a library-side refusal, empty when the error carries none (ClickHouse's own refusal, an internal error). It is the member a caller keys on, never the message. Like the other text accessors, it returns NULL only for a NULL or invalid error.
+
#### `chs_error_free`
```c
CHS_API void chs_error_free(chs_error *error);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `error`: handle, `chs_error`, nullable.
Releases the caller's reference to an error. Freeing NULL does nothing.
@@ -1701,7 +2082,7 @@ Releases the caller's reference to an error. Freeing NULL does nothing.
CHS_API chs_status chs_live_handles(chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `out`: out_handle, `chs_buf`, `document:live_handles`.
- `err`: out_error.
@@ -1714,7 +2095,7 @@ A JSON document counting the live handles of each kind this image has made, take
CHS_API chs_status chs_error_codes(chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `out`: out_handle, `chs_buf`, `document:error_code_table`.
- `err`: out_error.
@@ -1727,7 +2108,7 @@ ClickHouse's own error-code table for this build, as a JSON array of `{"code", "
CHS_API chs_status chs_registered_families(chs_buf **out, chs_error **err);
```
-- Class `tooling`, thread `process_serial`, FIRM.
+- Class `tooling`, thread `process_serial`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `out`: out_handle, `chs_buf`, `ascii`.
- `err`: out_error.
@@ -1740,7 +2121,7 @@ Every type family in this build's own type registry, one per line. A tooling exp
CHS_API chs_status chs_function_flags(chs_buf **out, chs_error **err);
```
-- Class `tooling`, thread `process_serial`, FIRM.
+- Class `tooling`, thread `process_serial`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `out`: out_handle, `chs_buf`, `ascii`.
- `err`: out_error.
@@ -1753,7 +2134,7 @@ A tab-separated audit of every registered function's volatility flags, one funct
CHS_API chs_status chs_reference_type(const uint8_t *type_expr, size_t type_expr_len, chs_buf **out, chs_error **err);
```
-- Class `tooling`, thread `process_serial`, FIRM.
+- Class `tooling`, thread `process_serial`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `type_expr`: bytes_in, `sql`.
- `out`: out_handle, `chs_buf`, `sql`.
@@ -1767,7 +2148,7 @@ The widened reference type this build pairs with a type expression. A tooling ex
CHS_API chs_status chs_initialize(const uint8_t *timezone, size_t timezone_len, chs_error **err);
```
-- Class `api`, thread `process_once`, FIRM.
+- Class `api`, thread `process_once`, init `any`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `timezone`: bytes_in, `timezone`.
- `err`: out_error.
@@ -1782,12 +2163,14 @@ The zone is process state, not a per-call parameter, because ClickHouse's own Me
CHS_API chs_status chs_set_defaults(const uint8_t *settings, size_t settings_len, chs_error **err);
```
-- Class `api`, thread `process_serial`, FIRM.
+- Class `api`, thread `process_serial`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
- `err`: out_error.
-Seeds the settings every later call starts from, as a JSON object whose values are JSON strings (a binding never rewrites a value, a boolean included). Each successful call replaces the previous defaults whole. A setting name the server would refuse is refused here, with ClickHouse's own code and message, and nothing is committed. `session_timezone` here is the default zone for later calls; it does not change the image zone that compiled types bind.
+Seeds the settings every later call starts from, as a JSON object whose values are JSON strings, or the object form for bytes (Input values), and a binding never rewrites a value, a boolean included. Each successful call replaces the previous defaults whole. A setting name the server would refuse is refused here, with ClickHouse's own code and message, and nothing is committed. `session_timezone` here is the default zone for later calls; it does not change the image zone that compiled types bind.
+
+The library's own knobs are `chtypes_*` settings, and this is the only channel for the process-level ones. Each is an integer or a string given as a JSON string, and no server knows one, so a server profile refuses it (115) and a call's settings carry only the per-call ones. The knobs, their layer, type, default and effect: `chtypes_now_epoch_nanos` (per call, int64 nanoseconds since the epoch, default the wall clock: pins the call's clock instant); `chtypes_clock_offset_nanos` (per call, int64, default 0: shifts the instant); `chtypes_max_clock_skew_nanos` (per call, int64, default 0: the skew budget a clock-reading expression may resolve under); `chtypes_default_eval_memory_bytes` (process, int64 bytes, default 268435456, 0 disables: the admission memory budget one DEFAULT, CHECK, TTL or filter evaluation may use); `chtypes_default_eval_cpu_nanos` (process, int64, default 1000000000, 0 disables: the admission time budget, the evaluating thread's CPU time, detected not interrupted; the earlier name `chtypes_default_eval_wall_nanos` is refused on v2 with `CHS_INVALID_ARGUMENT` naming this one); `chtypes_custom_settings_prefixes` (process, string, default none: the server's `custom_settings_prefixes`, comma-separated). A process knob in a call's settings is declined, named in `unsupported_settings` (`call-setting-process-only-key`), and a clock knob whose value is not an integer likewise (`call-setting-clock-value`). An evaluation that exceeds a budget is declined with that budget's own reason id, never answered: `admission-memory-budget` for the memory budget and `admission-time-budget` for the time budget; the earlier single id `admission-envelope` is retired, its catalog entry naming both successors (rule r4). The time budget is the evaluating thread's CPU time, not wall time. Each catalog entry states its load dependence. A memory-budget decline is deterministic (`measured`). A time-budget decline depends on load only near the budget: CPU time rose up to x1.03 under load on linux-arm64 and up to x2.0 on linux-amd64 when two busy hardware threads share a core, so a time-budget decline is retry-stable when the evaluation costs more than the budget on an idle core and never happens under about half of it; between half and one times the budget the host's load can decide. Normal evaluations cost about 1000 times less than the 1 s default. a schema whose DEFAULT exceeds a budget at admission is declined at compile. The defaults are this description's, not a build's: a build reports nothing about them.
Setup only. The defaults are immutable once this image has created its first `chs_schema` (a filter or a block needs one): from then on the call changes nothing and answers `CHS_INVALID_ARGUMENT`. No call ever reads defaults that change under it, so no binding needs a lock around them.
@@ -1797,7 +2180,7 @@ Setup only. The defaults are immutable once this image has created its first `ch
CHS_API void chs_shutdown(void);
```
-- Class `api`, thread `process_serial`, FIRM.
+- Class `api`, thread `process_serial`, init `any`, FIRM.
Stops what `chs_initialize` started (its background threads) and joins them. It is idempotent and safe before `chs_initialize`. After it, no call is valid except `chs_shutdown` again. Unloading the library, or initializing it again in the same process, is unsupported: a loader never unloads a library.
@@ -1807,7 +2190,7 @@ Stops what `chs_initialize` started (its background threads) and joins them. It
CHS_API chs_status chs_type_validate(const uint8_t *type_expr, size_t type_expr_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `type_expr`: bytes_in, `sql`.
- `out`: out_handle, `chs_buf`, `sql`.
@@ -1821,7 +2204,7 @@ Parses one type expression with ClickHouse's own parser and returns its canonica
CHS_API chs_status chs_back_quote(const uint8_t *name, size_t name_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `name`: bytes_in, `name`.
- `out`: out_handle, `chs_buf`, `sql`.
@@ -1835,7 +2218,7 @@ A name quoted the way ClickHouse's own `backQuote` quotes it: always quoted, eve
CHS_API chs_status chs_back_quote_if_needed(const uint8_t *name, size_t name_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `name`: bytes_in, `name`.
- `out`: out_handle, `chs_buf`, `sql`.
@@ -1849,7 +2232,7 @@ A name quoted only where this build's own `backQuoteIfNeed` says it must be. Whi
CHS_API chs_status chs_quote_string(const uint8_t *text, size_t text_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `text`: bytes_in, `bytes`.
- `out`: out_handle, `chs_buf`, `sql`.
@@ -1863,7 +2246,7 @@ A byte string spelled as a ClickHouse string literal by the vendored `quoteStrin
CHS_API chs_status chs_server_create(const uint8_t *profile, size_t profile_len, const uint8_t *options, size_t options_len, chs_server **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `profile`: bytes_in, `input:server_profile`.
@@ -1877,11 +2260,11 @@ Everything is validated here, once, and the server never changes after it. A key
`CHS_DECLINED` is reserved: a build that will not describe a profile declines it, which is never a wrong answer. The one decline foreseen is a setting outside the set that shapes an input, at a value other than its default, once that set is generated from the vendored source.
-The macros are the server's whole `system.macros` when `macros` is present, even as `{}`: a macro a Replicated engine's arguments name that the set lacks is then the server's own refusal (`CHS_REJECTED`, 139, NO_ELEMENTS_IN_CONFIG). When `macros` is absent the server's macros are unknown, and a schema whose engine reads one is declined, as it is without a server. A build that validates a profile's macros but does not yet apply them declines such a schema in both cases.
+The macros are the server's whole `system.macros` when `macros` is present, even as `{}`: a macro a Replicated engine's arguments name that the set lacks is then the server's own refusal (`CHS_REJECTED`, 139, NO_ELEMENTS_IN_CONFIG). When `macros` is absent the server's macros are unknown, and a schema whose engine reads one is declined, as it is without a server, except under `schema_options`' `posture` `load`, where a stored Replicated definition compiles without expanding them (`input:schema_options`). A build that validates a profile's macros but does not yet apply them declines such a schema in both cases.
A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds. So does `chs_schema_create` on the server, because a profile never changes a schema compile: it lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer `server`.
-The call reads no defaults and leaves `chs_set_defaults`' latch open. Like every call that is not the handshake, it is valid only after `chs_initialize`.
+The call reads no defaults and leaves `chs_set_defaults`' latch open. Its init class is `required`, so before `chs_initialize` it refuses (Initialization, in the Preamble).
#### `chs_server_free`
@@ -1889,7 +2272,7 @@ The call reads no defaults and leaves `chs_set_defaults`' latch open. Like every
CHS_API void chs_server_free(chs_server *server);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `server`: handle, `chs_server`, nullable.
Releases the caller's reference to a server. Schemas created on it keep it alive. Freeing NULL does nothing.
@@ -1900,17 +2283,19 @@ Releases the caller's reference to a server. Schemas created on it keep it alive
CHS_API chs_status chs_schema_create(const chs_server *server, const uint8_t *create_table, size_t create_table_len, const uint8_t *settings, size_t settings_len, const uint8_t *options, size_t options_len, chs_schema **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `server`: handle, `chs_server`, nullable.
- `create_table`: bytes_in, `sql`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
- `options`: bytes_in, `input:schema_options`.
- `out`: out_handle, `chs_schema`.
- `err`: out_error.
-Compiles exactly one `CREATE TABLE` statement (columns, engine, keys, TTL, settings and constraints) with ClickHouse's own parser and the checks a server's CREATE runs, on `server`, under a profile of settings given as a JSON object of string values. A trailing semicolon is allowed. A second statement is refused the way ClickHouse's own parser refuses one in a single query (`CHS_REJECTED`, with its code and message), and a statement of any other kind is `CHS_INVALID_ARGUMENT`.
+Compiles exactly one `CREATE TABLE` statement (columns, engine, keys, TTL, settings and constraints) with ClickHouse's own parser and the checks a server's CREATE runs, on `server`, under a profile of settings given as a JSON object of string values. A trailing semicolon is allowed. A second statement is refused the way ClickHouse's own parser refuses one in a single query (`CHS_REJECTED`, with its code and message), and a statement of any other kind is `CHS_INVALID_ARGUMENT`, except under `posture` `load`: there a `CREATE VIEW`, `CREATE MATERIALIZED VIEW` or `CREATE DICTIONARY` statement, the stored definition of an object a server lists in `system.tables` and whose columns it lists in `system.columns`, is `CHS_DECLINED` with the reason `engine-not-modeled`, so a caller that compiles a server's stored definitions falls back for it as it does for an engine this build does not model. The `ATTACH` spellings of those kinds that a server refuses with 62 stay `CHS_REJECTED` with that code, as the server answers. An `ATTACH TABLE` statement is not one of the other kinds (`options`, below). The library models a table, not its database's engine: `ATTACH TABLE` with a column list and an engine but no UUID is accepted under `posture` `load`, which a server's Atomic database refuses (80), and under `create` it is declined (`attach-statement-under-create`).
+
+A table whose engine this build does not model (a `Distributed` or `Buffer` table, for example) is `CHS_DECLINED` with the reason `engine-not-modeled`, which is returned only when the engine itself is the cause: a decline for anything else in the statement carries that thing's own reason.
`server` is the server the table is on (`chs_server_create`), or NULL. On a server, the schema's home zone is the server's `timezone`: its zone-less `DateTime` and `DateTime64` columns bind it, as a restarted server binds its own, and every call on the schema that names no `session_timezone` runs in it. The server's `settings` layer under the schema's own, and a Replicated engine's ZooKeeper path and replica name expand the server's `macros`. With `server` NULL the schema is on the image's own server, exactly as before servers existed: the image zone, no server settings layer, and macros unknown. A freed server, a handle of another kind or a server from another library image is `CHS_INVALID_ARGUMENT`. The schema holds a counted reference to its server, so the caller may free the server at any time.
@@ -1918,7 +2303,7 @@ Compiles exactly one `CREATE TABLE` statement (columns, engine, keys, TTL, setti
A setting this build's filters do not honor in a WHERE, from the `chs_set_defaults` snapshot this call captures, the server profile or this call's own `settings`, is no decline here: the call succeeds and lists it in the schema's `filter_declined_settings` (`schema_description`), and `chs_filter_create` on the schema declines.
-`options` (`input:schema_options`) defines no member yet, so any key in it is `CHS_INVALID_ARGUMENT`, naming the key.
+`options` (`input:schema_options`) sets the `posture` the statement compiles in: `create`, the default, or `load` for a server's own stored definition (see `input:schema_options`, and the caller's obligation there). A `posture` this build does not know, or any other key, is `CHS_INVALID_ARGUMENT`, naming it. An `ATTACH TABLE` statement is a stored definition, never a statement of another kind: under `create` it is `CHS_DECLINED`, ch_code 0, with exactly the message `chs_schema_create: an ATTACH TABLE statement is a stored definition; compile it with schema_options {"posture":"load"}; declined rather than answered`, and under `load` it compiles as the same definition spelled `CREATE TABLE`.
A build that does not yet compile on a server declines a non-NULL `server`, and a schema whose compile needs its server's zone is declined on a host thread that already runs a ClickHouse query of its own. Each is `CHS_DECLINED`, which is never a wrong answer.
@@ -1930,7 +2315,7 @@ The result is immutable: nothing changes a schema after this call, so it replace
CHS_API void chs_schema_free(chs_schema *schema);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `schema`: handle, `chs_schema`, nullable.
Releases the caller's reference to a schema. Filters and blocks made from it keep it alive. Freeing NULL does nothing.
@@ -1941,84 +2326,94 @@ Releases the caller's reference to a schema. Filters and blocks made from it kee
CHS_API chs_status chs_schema_describe(const chs_schema *schema, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `schema`: handle, `chs_schema`.
- `out`: out_handle, `chs_buf`, `document:schema_description`.
- `err`: out_error.
A JSON document describing the schema's columns, owned by the caller; it replaces v0's borrowed column accessors.
-A deep input the server's stack check refuses is `CHS_REJECTED` with ch_code 306. Here the input is the schema's own statement, which describing walks again; it is never `CHS_INTERNAL`.
+A schema compiled in `load` carries `"posture": "load"`; one compiled in `create` carries no `posture`.
+
+A schema `chs_schema_create` returned always describes: the walk this document needs runs at compile, so a deep statement is refused there (`CHS_REJECTED`, 306) and this call never refuses or declines a compiled schema. It is never `CHS_INTERNAL`.
#### `chs_preview_row`
```c
-CHS_API chs_status chs_preview_row(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, chs_buf **out, chs_error **err);
+CHS_API chs_status chs_preview_row(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, const uint8_t *options, size_t options_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `schema`: handle, `chs_schema`.
- `format`: enum, `chs_format`.
- `body`: bytes_in, `bytes`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
- `columns`: bytes_in, `json_array_names`.
+- `options`: bytes_in, `input:insert_options`.
- `out`: out_handle, `chs_buf`, `document:row`.
- `err`: out_error.
Validates and coerces one row of `body` under the schema, as a server's INSERT would, and returns the row's document. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none. `session_timezone` in `settings` is this call's zone.
-`body` holds exactly one row, and the library reads all of it, with the same reader as `chs_preview_batch`. A malformed tail, or trailing content the format's reader does not itself skip, is rejected with the reader's own code, as a server's INSERT rejects it; what the reader skips, such as a trailing newline, is fine. A body holding no row, or more than one, is `CHS_INVALID_ARGUMENT`, naming the call: `chs_preview_batch` answers a body of many rows. The row's document equals the single row of `chs_preview_batch`'s document for the same body.
+`body` holds exactly one row, and the library reads all of it, with the same reader as `chs_preview_batch`. A malformed tail, or trailing content the format's reader does not itself skip, is rejected with the reader's own code, as a server's INSERT rejects it; what the reader skips, such as a trailing newline, is fine. A body holding no row, or more than one, is `CHS_INVALID_ARGUMENT`, naming the call: `chs_preview_batch` answers a body of many rows. The row's document equals the single row of `chs_preview_batch`'s document for the same body, and stands for the call: a refusal or decline of the call, its settings, its column list or its options is this document's `outcome` with `refused_at` `call`, never a status.
An INSERT whose input block has no column at all (every insertable column EPHEMERAL, and no column list) is declined: a server answers it (code 90, EMPTY_LIST_OF_COLUMNS_PASSED) from a statement inside its INSERT interpreter that the library cannot call, and a decline is never a wrong answer.
+`options` (`input:insert_options`) shapes the INSERT the call models, and `{}` is the INSERT it modeled before the document existed.
+
#### `chs_preview_batch`
```c
-CHS_API chs_status chs_preview_batch(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, const chs_filter *filter, int32_t export_format, uint32_t doc_flags, chs_buf **out, chs_buf **out_export, chs_error **err);
+CHS_API chs_status chs_preview_batch(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, const chs_filter *filter, int32_t export_format, uint32_t doc_flags, const uint8_t *options, size_t options_len, chs_buf **out, chs_buf **out_export, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `schema`: handle, `chs_schema`.
- `format`: enum, `chs_format`.
- `body`: bytes_in, `bytes`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
- `columns`: bytes_in, `json_array_names`.
- `filter`: handle, `chs_filter`, nullable.
- `export_format`: scalar, `int32`.
- `doc_flags`: scalar, `uint32`.
+- `options`: bytes_in, `input:insert_options`.
- `out`: out_handle, `chs_buf`, `document:batch`.
- `out_export`: out_handle, `chs_buf`, `bytes`, nullable.
- `err`: out_error.
-Validates and coerces a whole body, which may hold many rows, and returns the batch document, with the same declines as `chs_preview_row`. Row separation and the server's error allowance are ClickHouse's own, and the document's `unconsumed` and `framing` say what the reader skipped and how it framed the body. `filter`, when given, is evaluated over each stored row in the same parse. When that filter's WHERE would read a setting this build's filters do not honor, from the filter's own `settings`, the name joins the batch document's top-level `unsupported_settings` and the batch is `unsupported` by `batch_outcome`'s step 2: every row `unsupported`, filter rows `d`, and no export. The same call without the filter is unchanged. `export_format` is `CHS_EXPORT_NONE` or a `chs_format` the build can write; with an export that runs, `out_export` receives the accepted rows serialized once, an empty buffer when the batch holds no row. When the export is declined (the document carries `export_declined`), `out_export` is NULL, as it may be when no export is asked for, so a caller tells a declined export from an empty one by `export_declined`, never by the buffer. `doc_flags` chooses which groups the per-row documents carry.
+Validates and coerces a whole body, which may hold many rows, and returns the batch document, with the same declines as `chs_preview_row`. Row separation and the server's error allowance are ClickHouse's own, and the document's `unconsumed` and `framing` say what the reader skipped and how it framed the body. `filter`, when given, is evaluated over each stored row in the same parse. When that filter's WHERE would read a setting this build's filters do not honor, from the filter's own `settings`, the name joins the batch document's top-level `unsupported_settings` and the batch is `unsupported` by `batch_outcome`'s step 2: every row `unsupported`, filter rows `d`, and no export. The same call without the filter is unchanged. `export_format` is `CHS_EXPORT_NONE` or a `chs_format` the build can write; with an export that runs, `out_export` receives the accepted rows serialized once, an empty buffer when the batch holds no row. When the export is declined (the document carries `export_declined` and `export_declined_reason`), `out_export` is NULL, as it may be when no export is asked for, so a caller tells a declined export from an empty one by `export_declined`, never by the buffer. `doc_flags` chooses which groups the per-row documents carry. `options` (`input:insert_options`) shapes the INSERT the call models, and `{}` is the INSERT it modeled before the document existed. `filter` must have been compiled on `schema`: a filter from another schema handle is `CHS_INVALID_ARGUMENT`, as it is at `chs_filter_eval_block`. The call returns a status other than `CHS_OK` only for misuse or an internal error: a decline of the call (the filter's zone differing from the batch's, `preview-batch-zone-mismatch`) and a refusal of its arguments are the document's, with `refused_at` `call` and `reason`. The export's truth table is in `document:batch`.
#### `chs_filter_create`
```c
-CHS_API chs_status chs_filter_create(const chs_schema *schema, const uint8_t *expr, size_t expr_len, const uint8_t *query_params, size_t query_params_len, const uint8_t *settings, size_t settings_len, chs_filter **out, chs_error **err);
+CHS_API chs_status chs_filter_create(const chs_schema *schema, const uint8_t *expr, size_t expr_len, const uint8_t *query_params, size_t query_params_len, const uint8_t *settings, size_t settings_len, const uint8_t *options, size_t options_len, chs_filter **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
+- Class `api`, thread `shared`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `schema`: handle, `chs_schema`.
- `expr`: bytes_in, `sql`.
-- `query_params`: bytes_in, `json_object_string_values`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `query_params`: bytes_in, `json_object_byte_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
+- `options`: bytes_in, `input:filter_options`.
- `out`: out_handle, `chs_filter`.
- `err`: out_error.
-Compiles a boolean SQL expression over the schema's columns, binding its query parameters (a JSON object of string values) by ClickHouse's own substitution, so a value is never SQL text. `settings` is the profile the expression is compiled under, so `session_timezone` there is the zone of its literals. Non-deterministic expressions are declined.
+Compiles a boolean SQL expression over the schema's columns, binding its query parameters (a JSON object of string values) by ClickHouse's own substitution, so a value is never SQL text. A parameter's value is decoded by ClickHouse's escaped-text reader, exactly as a server decodes an HTTP `param_` value (`input:filter_options`' `param_encoding`, `escaped`): `\b` is an escape, so the parameter `a\b` reads as `a`, the byte 0x08 and `b`, and a literal backslash is written `\\`; `\N` reads as the empty value; `\x41`, `\'` and `\0` decode as that reader decodes them; a raw tab or newline is `CHS_REJECTED` with 457 and a trailing backslash with 25, as a server refuses them; and the bytes an escape produces are kept, valid UTF-8 or not. The object form `{"b64": ...}` (Input values) is decoded from base64 first, and its bytes are then read the same way. `options` (`input:filter_options`) shapes the compile, and `{}` is the compile before the document existed. `settings` is the profile the expression is compiled under, so `session_timezone` there is the zone of its literals. Non-deterministic expressions are declined.
The WHERE's settings are layered, lowest first: the build's own, `chs_set_defaults`, the schema's server profile, the schema's own settings, and then this call's `settings`; `session_timezone` at every layer stays on the zone path. A type gate in the schema's own settings binds the CREATE only, so a WHERE's CAST validates under the session layers, every layer but the schema's own.
-On a schema whose `filter_declined_settings` is not empty (`chs_schema_describe`), from any layer, the call is `CHS_DECLINED` with ch_code 0. It is checked after the argument checks and before the expression is parsed, because `dialect` can change the parse, and the message names each setting and its layer, in the list's order: `chs_filter_create: the settings this schema was compiled under set (), (), which this build's filters do not honor in a WHERE (listed in chs_schema_describe's filter_declined_settings); declined rather than answered`, where each `` is the entry's `declined_layer` value. The entries are joined by a comma and a space, in the list's order, and each name renders verbatim. A `name_b64` entry renders as `b64:`; it does not occur in practice, because a setting name ClickHouse does not know is refused earlier, with 115. The server and the schema were created all the same: no settings layer changes a schema compile.
+On a schema whose `filter_declined_settings` is not empty (`chs_schema_describe`), from any layer, the call is `CHS_DECLINED` with ch_code 0. It is checked after the argument checks and before the expression is parsed, because `dialect` can change the parse, and the message names each setting and its layer, in the list's order: `chs_filter_create: the settings this schema was compiled under set (), (), which this build's filters do not honor in a WHERE (listed in chs_schema_describe's filter_declined_settings); declined rather than answered`, where each `` is the entry's `declined_layer` value, with the reason `server-profile-setting`. The message is informative; the entries are joined by a comma and a space, in the list's order, and each name renders verbatim. A `name_b64` entry renders as `b64:`; it does not occur in practice, because a setting name ClickHouse does not know is refused earlier, with 115. The server and the schema were created all the same: no settings layer changes a schema compile.
**Changed (public issue #588).** A declined setting from the `chs_set_defaults` or the schema layer moves from evaluation time to create time: it was named at evaluation (`filter_result`'s `unsupported_settings`), and it now declines this call. A server-profile setting already declined it.
-A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here, and it is never in `schema_description`. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115.
+A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here but for one case, and it is never in `schema_description`. The case is a cell of the line's `where-settings/documented-pathological.tsv`, a value under which the server refuses every query: set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115.
A filter under a per-call `session_timezone` answers as the artifact producer has measured against live servers, including how the zone in an evaluation's settings combines with the zone the filter was compiled under. Until a path is measured to match a server it declines, which is never a wrong answer. The filter holds a counted reference to the schema.
@@ -2028,7 +2423,7 @@ A filter under a per-call `session_timezone` answers as the artifact producer ha
CHS_API void chs_filter_free(chs_filter *filter);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `filter`: handle, `chs_filter`, nullable.
Releases the caller's reference to a filter. Freeing NULL does nothing.
@@ -2036,37 +2431,42 @@ Releases the caller's reference to a filter. Freeing NULL does nothing.
#### `chs_filter_eval_body`
```c
-CHS_API chs_status chs_filter_eval_body(const chs_filter *filter, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, chs_buf **out, chs_error **err);
+CHS_API chs_status chs_filter_eval_body(const chs_filter *filter, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, const uint8_t *options, size_t options_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `filter`: handle, `chs_filter`.
- `format`: enum, `chs_format`.
- `body`: bytes_in, `bytes`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
+- `columns`: bytes_in, `json_array_names`.
+- `options`: bytes_in, `input:insert_options`.
- `out`: out_handle, `chs_buf`, `document:filter_result`.
- `err`: out_error.
-Evaluates the filter over every row of a body, returning one verdict per row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body and settings, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together.
+Evaluates the filter over every row of a body, returning one verdict per row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the INSERT that names it, and with none the INSERT that names none. When a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a refusal or decline of the call itself in the document (`refused_at` `call`), never as a status. `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options.
#### `chs_block_create`
```c
-CHS_API chs_status chs_block_create(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, chs_block **out, chs_error **err);
+CHS_API chs_status chs_block_create(const chs_schema *schema, chs_format format, const uint8_t *body, size_t body_len, const uint8_t *settings, size_t settings_len, const uint8_t *columns, size_t columns_len, const uint8_t *options, size_t options_len, chs_block **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Growable (rule r1): an input it gains is a new member of its options document, `options`.
- `schema`: handle, `chs_schema`.
- `format`: enum, `chs_format`.
- `body`: bytes_in, `bytes`.
-- `settings`: bytes_in, `json_object_string_values`.
+- `settings`: bytes_in, `json_object_byte_values`.
- `columns`: bytes_in, `json_array_names`.
+- `options`: bytes_in, `input:insert_options`.
- `out`: out_handle, `chs_block`.
- `err`: out_error.
-Parses a body once under the schema, for evaluating many filters over it. `settings` governs the parse, `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings and columns, with no filter, which `chs_filter_eval_block` answers by. An accepted body is read once, for the block and that verdict together.
+Parses a body once under the schema, for evaluating many filters over it. `settings` governs the parse, `session_timezone` there included, and `columns` is read as `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter, which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal error. An accepted body is read once, for the block and that verdict together. `options` (`input:insert_options`) is read as `chs_preview_batch` reads it.
#### `chs_block_free`
@@ -2074,7 +2474,7 @@ Parses a body once under the schema, for evaluating many filters over it. `setti
CHS_API void chs_block_free(chs_block *block);
```
-- Class `api`, thread `handle_serial`, FIRM.
+- Class `api`, thread `handle_serial`, init `any`, FIRM.
- `block`: handle, `chs_block`, nullable.
Releases the caller's reference to a block. Freeing NULL does nothing.
@@ -2085,14 +2485,14 @@ Releases the caller's reference to a block. Freeing NULL does nothing.
CHS_API chs_status chs_filter_eval_block(const chs_filter *filter, const chs_block *block, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `shared`, FIRM.
-- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
+- Class `api`, thread `shared`, init `required`, FIRM.
+- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `filter`: handle, `chs_filter`.
- `block`: handle, `chs_block`.
- `out`: out_handle, `chs_buf`, `document:filter_result`.
- `err`: out_error.
-Evaluates the filter over a parsed block. Over a body a server would accept, it gives the same answers `chs_filter_eval_body` gives for that body. Over one it would not (the verdict `chs_block_create` recorded is not `accepted`), the outcome mirrors that verdict as for `chs_filter_eval_body`, and every verdict from the first refused row on is `d`. The first refused row is the first row of that batch whose outcome is neither `accepted` nor `skipped`. When the refusal belongs to no row (a whole-body step, a poisoned value, the call's own settings), it is the first row of the block. The rows before it keep their evaluated verdicts. A caller treats those as `d` too: any non-`ok` outcome, including `unsupported`, means every verdict is `d`. It takes no settings: the filter brings the settings it was compiled under and the block those it was parsed under. The filter and the block must come from the same schema; a pair from two schemas is `CHS_INVALID_ARGUMENT`.
+Evaluates the filter over a parsed block. Over a body a server would accept, it gives the same answers `chs_filter_eval_body` gives for that body. Over one it would not (the verdict `chs_block_create` recorded is not `accepted`), the outcome mirrors that verdict as for `chs_filter_eval_body`, and every verdict is written `d`, the rows before the refused one included: the two calls write the same document for the same body, and the rule that any non-`ok` outcome means every verdict is `d` holds in the data. `refused_at` and `refused_row` say where the refusal sat (`row` for a refused row, `body_step` for a whole-body step or a poisoned value, `call` for the block's own settings). It takes no settings: the filter brings the settings it was compiled under and the block those it was parsed under. The filter and the block must come from the same schema; a pair from two schemas is `CHS_INVALID_ARGUMENT`.
#### `chs_discover_query`
@@ -2100,7 +2500,7 @@ Evaluates the filter over a parsed block. Over a body a server would accept, it
CHS_API chs_status chs_discover_query(chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `out`: out_handle, `chs_buf`, `sql`.
- `err`: out_error.
@@ -2113,7 +2513,7 @@ The query a caller runs against a server to read a table's `system.columns` rows
CHS_API chs_status chs_discover_columns(const uint8_t *rows, size_t rows_len, chs_buf **out, chs_error **err);
```
-- Class `api`, thread `any`, FIRM.
+- Class `api`, thread `any`, init `required`, FIRM.
- May return `CHS_OK`, `CHS_REJECTED`, `CHS_DECLINED`, `CHS_INVALID_ARGUMENT`, `CHS_INTERNAL`.
- `rows`: bytes_in, `bytes`.
- `out`: out_handle, `chs_buf`, `document:discovery`.
diff --git a/docs/reference/bindings-v1.md b/docs/reference/bindings-v1.md
index f39646a3..3d23924d 100644
--- a/docs/reference/bindings-v1.md
+++ b/docs/reference/bindings-v1.md
@@ -95,7 +95,7 @@ One row per public operation. `request` is a version spelling under [`fetch-v1.m
A `ServerProfile` carries: `timezone` (a zone name), `settings` (settings) and `macros` (settings), each optional (input:server_profile, [`abi-v2.md`](abi-v2.md)).
-**The server profile (ABI v2).** `new_server` describes one ClickHouse server with `chs_server_create`, and the compile option `server` compiles a table on it (`chs_schema_create`'s `server`): the schema's home zone is the server's `timezone`, the server's `settings` layer under the schema's own, and a Replicated engine's ZooKeeper path and replica name expand its `macros`. On a server schema, `timezone()` answers the server's zone when no session zone is in force; a session zone that would change the answer is declined (named in `unsupported_settings`). That applies equally to a profile whose `settings` carry `session_timezone`. The binding serializes the profile with the stock encoder and validates nothing in it: a zone `DateLUT` cannot load, or a setting the server's `SET` check refuses, is the library's own refusal (`SchemaError`), a malformed macro set is its `UsageError`, and a build that does not compile on a server yet declines (`UnsupportedError`). An empty zone and an absent map are omitted from the document; a present map is sent as given, `{}` when empty. **Absent `macros` is not empty `macros`:** absent means the server's macros are unknown, and present, even `{}`, is the server's complete set (Go nil against a non-nil map, Python and TypeScript absent against `{}`, Rust `None` against `Some`). No server option and no schema option is defined yet, so each options document is empty. The server is immutable; a schema holds a counted reference to it inside the library, so the two close in any order, and a closed server is a `UsageError` raised before any call, never NULL in its place (rule r7: NULL means the image's own server, so the library cannot tell a freed server from none); Rust meets the rule by construction, since a `Server` is freed only on its last drop and a compile borrows it. A server from another library is that library's own `CHS_INVALID_ARGUMENT`.
+**The server profile (ABI v2).** `new_server` describes one ClickHouse server with `chs_server_create`, and the compile option `server` compiles a table on it (`chs_schema_create`'s `server`): the schema's home zone is the server's `timezone`, the server's `settings` layer under the schema's own, and a Replicated engine's ZooKeeper path and replica name expand its `macros`. On a server schema, `timezone()` answers the server's zone when no session zone is in force; a session zone that would change the answer is declined (named in `unsupported_settings`). That applies equally to a profile whose `settings` carry `session_timezone`. The binding serializes the profile with the stock encoder and validates nothing in it: a zone `DateLUT` cannot load, or a setting the server's `SET` check refuses, is the library's own refusal (`SchemaError`), a malformed macro set is its `UsageError`, and a build that does not compile on a server yet declines (`UnsupportedError`). An empty zone and an absent map are omitted from the document; a present map is sent as given, `{}` when empty. **Absent `macros` is not empty `macros`:** absent means the server's macros are unknown, and present, even `{}`, is the server's complete set (Go nil against a non-nil map, Python and TypeScript absent against `{}`, Rust `None` against `Some`). No server option is defined yet, so its options document is empty; `schema_options` has `posture` (§2, the call options). The server is immutable; a schema holds a counted reference to it inside the library, so the two close in any order, and a closed server is a `UsageError` raised before any call, never NULL in its place (rule r7: NULL means the image's own server, so the library cannot tell a freed server from none); Rust meets the rule by construction, since a `Server` is freed only on its last drop and a compile borrows it. A server from another library is that library's own `CHS_INVALID_ARGUMENT`.
### A schema
@@ -121,18 +121,29 @@ A `ServerProfile` carries: `timezone` (a zone name), `settings` (settings) and `
One entry point per operation, with its optional inputs spelled in the language's own idiom, as the 0.x binding contract's "One compile function" already ruled: Go functional options, Python keyword-only arguments, a TypeScript options object, and a Rust options struct with `Default` (or a positional slice where the call has only that one optional input).
-| option | C parameter | applies to | Go | Python | TypeScript | Rust |
-| ---------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------------- | ------------------------------------------------ |
-| settings | `settings` | compile (the profile), row, rows, parse a block, filter compile (the filter's own zone and profile), filter rows (the body's parse settings) | `WithSettings(map[string]string)`: one function whose value implements `CompileOption`, `RowOption`, `RowsOption`, `BlockOption`, `FilterOption` and `EvalOption` | `settings=` | `settings` | `settings: Vec<(String, String)>` |
-| the per-call zone | the `session_timezone` key inside `settings` | the same six calls; on a filter compile it is the filter's zone, on a filter evaluation the body's parse zone | `WithSessionTimezone(name string)`, implementing the same six option interfaces | `session_timezone=` | `sessionTimezone` | `session_timezone: Option` |
-| the INSERT column list | `columns` | row, rows, parse a block | `WithColumns(names []string)`: a `RowOption`, `RowsOption` and `BlockOption` | `columns=` | `columns` | `columns: Vec>` |
-| an attached filter | `filter` | rows | `WithRowFilter(*Filter)` | `row_filter=` | `rowFilter` | `filter: Option` |
-| an export | `export_format`, `out_export` | rows | `WithExport(Format)`; absent means `CHS_EXPORT_NONE` | `export=` | `exportFormat` | `export: Option` |
-| document groups | `doc_flags` | rows | `WithDocFlags(DocFlags)`; absent means `DocAll` | `doc_flags=` | `docFlags` | `doc_flags: Option`, `None` means all |
-| query parameters | `query_params` | filter compile | `WithFilterParams(map[string]string)` | `params=` | `params` | `params: Vec<(String, String)>` |
-| the server | `server` | compile (ABI v2) | `OnServer(*Server)`; `OnServer(nil)` is no server | `server=` | `server` | `server: Option`, `None` means no server |
-
-The TypeScript option types are `CompileOptions { settings?, sessionTimezone?, server? }`, `RowOptions { settings?, sessionTimezone?, columns? }`, `RowsOptions extends RowOptions { rowFilter?, exportFormat?, docFlags? }`, `FilterOptions { params?, settings?, sessionTimezone? }` and `EvalOptions { settings?, sessionTimezone? }`. The Rust ones are `CompileOptions { settings, session_timezone, server }`, `RowOptions { settings, session_timezone, columns }`, `RowsOptions { settings, session_timezone, columns, filter, export, doc_flags }`, `FilterOptions { params, settings, session_timezone }` and `EvalOptions { settings, session_timezone }`, each with public fields and `Default`, built as `RowOptions { session_timezone: Some(z), ..Default::default() }`. They are deliberately not `#[non_exhaustive]`, which would forbid that construction outside the crate; the result types are.
+| option | C parameter | applies to | Go | Python | TypeScript | Rust |
+| ---------------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | -------------------- | ------------------------------------------------ |
+| settings | `settings` | compile (the profile), row, rows, parse a block, filter compile (the filter's own zone and profile), filter rows (the body's parse settings) | `WithSettings(map[string]string)`: one function whose value implements `CompileOption`, `RowOption`, `RowsOption`, `BlockOption`, `FilterOption` and `EvalOption` | `settings=` | `settings` | `settings: Vec<(String, String)>` |
+| the per-call zone | the `session_timezone` key inside `settings` | the same six calls; on a filter compile it is the filter's zone, on a filter evaluation the body's parse zone | `WithSessionTimezone(name string)`, implementing the same six option interfaces | `session_timezone=` | `sessionTimezone` | `session_timezone: Option` |
+| the INSERT column list | `columns` | row, rows, parse a block, filter rows (ABI v2) | `WithColumns(names []string)`: a `RowOption`, `RowsOption`, `BlockOption` and `EvalOption` | `columns=` | `columns` | `columns: Vec>` |
+| an attached filter | `filter` | rows | `WithRowFilter(*Filter)` | `row_filter=` | `rowFilter` | `filter: Option` |
+| an export | `export_format`, `out_export` | rows | `WithExport(Format)`; absent means `CHS_EXPORT_NONE` | `export=` | `exportFormat` | `export: Option` |
+| document groups | `doc_flags` | rows | `WithDocFlags(DocFlags)`; absent means `DocAll` | `doc_flags=` | `docFlags` | `doc_flags: Option`, `None` means all |
+| query parameters | `query_params` | filter compile | `WithFilterParams(map[string]string)` | `params=` | `params` | `params: Vec<(String, String)>` |
+| the server | `server` | compile (ABI v2) | `OnServer(*Server)`; `OnServer(nil)` is no server | `server=` | `server` | `server: Option`, `None` means no server |
+| insertable columns | `options`: `insert_options.insertable_columns` | row, rows, parse a block, filter rows (ABI v2) | `WithInsertableColumns(names []string)`: an `InsertOption`; nil leaves the member out | `insertable_columns=` | `insertableColumns` | `insertable_columns: Option>>` |
+| column defaults | `options`: `insert_options.column_defaults` | row, rows, parse a block, filter rows (ABI v2) | `WithColumnDefaults(defaults []ColumnDefault)`: an `InsertOption`; `ColumnDefault{Name, Value string}` | `column_defaults=` | `columnDefaults` | `column_defaults: Option>` |
+| verdict basis | `options`: `insert_options.verdict_basis` | row, rows, parse a block, filter rows (ABI v2) | `WithVerdictBasis(VerdictBasis)`: `VerdictBasisInput`, `VerdictBasisExport` | `verdict_basis=` | `verdictBasis` | `verdict_basis: Option` |
+| defer defaults | `options`: `insert_options.defer_defaults` | row, rows, parse a block, filter rows (ABI v2) | `WithDeferDefaults(DeferDefaults)`: `DeferDefaultsNone`, `DeferDefaultsServer` | `defer_defaults=` | `deferDefaults` | `defer_defaults: Option` |
+| an insert option with no name here | `options`: `insert_options.` | row, rows, parse a block, filter rows (ABI v2) | `WithInsertOption(key string, valueJSON []byte)`: a member passed through as raw JSON | `insert_options_extra=` | `insertOptionsExtra` | `insert_options_extra: Vec<(String, String)>` |
+| parameter encoding | `options`: `filter_options.param_encoding` | filter compile (ABI v2) | `WithParamEncoding(ParamEncoding)`: `ParamEncodingEscaped`, `ParamEncodingRaw` | `param_encoding=` | `paramEncoding` | `param_encoding: Option` |
+| a filter option with no name here | `options`: `filter_options.` | filter compile (ABI v2) | `WithFilterOption(key string, valueJSON []byte)`: passed through as raw JSON | `filter_options_extra=` | `filterOptionsExtra` | `filter_options_extra: Vec<(String, String)>` |
+| posture | `options`: `schema_options.posture` | compile (ABI v2) | `WithPosture(Posture)`: `PostureCreate`, `PostureLoad` | `posture=` | `posture` | `posture: Option` |
+| a schema option with no name here | `options`: `schema_options.` | compile (ABI v2) | `WithSchemaOption(key string, valueJSON []byte)`: passed through as raw JSON | `schema_options_extra=` | `schemaOptionsExtra` | `schema_options_extra: Vec<(String, String)>` |
+
+The TypeScript option types are `CompileOptions { settings?, sessionTimezone?, server?, posture?, schemaOptionsExtra? }`, `RowOptions { settings?, sessionTimezone?, columns?, insertableColumns?, columnDefaults?, verdictBasis?, deferDefaults?, insertOptionsExtra? }`, `RowsOptions extends RowOptions { rowFilter?, exportFormat?, docFlags? }`, `FilterOptions { params?, settings?, sessionTimezone?, paramEncoding?, filterOptionsExtra? }` and `EvalOptions { settings?, sessionTimezone?, columns?, insertableColumns?, columnDefaults?, verdictBasis?, deferDefaults?, insertOptionsExtra? }`. The Rust ones are `CompileOptions { settings, session_timezone, server, posture, schema_options_extra }`, `RowOptions { settings, session_timezone, columns, insertable_columns, column_defaults, verdict_basis, defer_defaults, insert_options_extra }`, `RowsOptions { settings, session_timezone, columns, insertable_columns, column_defaults, verdict_basis, defer_defaults, insert_options_extra, filter, export, doc_flags }`, `FilterOptions { params, settings, session_timezone, param_encoding, filter_options_extra }` and `EvalOptions { settings, session_timezone, columns, insertable_columns, column_defaults, verdict_basis, defer_defaults, insert_options_extra }`, each with public fields and `Default`, built as `RowOptions { session_timezone: Some(z), ..Default::default() }`. They are deliberately not `#[non_exhaustive]`, which would forbid that construction outside the crate; the result types are.
+
+**The options documents (ABI v2).** Three calls take an options document (rule r1): `insert_options` on the four calls that model an INSERT of a body (`chs_preview_row`, `chs_preview_batch`, `chs_block_create`, `chs_filter_eval_body`), `filter_options` on `chs_filter_create`, and `schema_options` on `chs_schema_create`. Each member is its own option in the table above, and the binding builds the document from the members the caller set: a document with no member set is not sent (length 0, which the library reads as `{}`, the call before the document existed). The binding validates no member and no value: the library refuses a key it does not know, or a value that is not one of the member's, with `CHS_INVALID_ARGUMENT`, naming it, and a member a build does not implement yet (`defer_defaults` `server`, `verdict_basis` `export`, `param_encoding` `raw`, and `column_defaults`, until a build lists it in `capabilities.features`) declines. A member the binding has no name for is passed through as JSON text under its key, so a caller can set an option the library implements before the binding names it; a key set twice keeps the last value, and text that is not valid JSON is a `UsageError` before any call. `insertable_columns` and the `columns` list are name lists: a name that is not valid UTF-8 is its `name_b64` form, as for `columns`. A `column_defaults` entry is `{name, value}`, each a byte string in the same two forms (`name_b64`, `value_b64`). An absent list is the member left out (every column; no override), and a present empty list is sent as `[]`.
**The per-call zone is a settings key, and nothing more.** The ABI has no zone parameter on any call: the per-call zone is `"session_timezone": ""` inside the call's settings object. The option exists so the zone is visible in every signature it affects; the binding writes its value into the settings object as that one key, verbatim, and does nothing else with it: no default, no validation, no canonicalization. The library validates it with `DateLUT` and answers a bad name with ClickHouse's own refusal. Passing the option and a `session_timezone` key in `settings` together is a `UsageError`, raised before the call whether or not the two agree, so a call never has two spellings of its zone. In a compile profile it is a default for later calls on that schema; a compiled type always takes the image zone, never the profile's.
@@ -188,7 +199,7 @@ The two zones, spelled in each binding:
- **A Go `string` is a byte string.** Go has no separate byte-string type, and a `string` holds any bytes, NUL included. The rule is only that it is built from copied bytes (`string(b)`), never through `C.GoString`, which stops at the first NUL.
- **A `str` or `string` input is encoded as UTF-8**, because that is the only way a text value becomes bytes. That is not an assumption about the payload: a caller with non-UTF-8 bytes passes them as `bytes`, `Uint8Array` or `&[u8]`. A body never accepts the text type, as in v0.
- **Rust keeps v0's `RawText`** (the bytes, `as_bytes()`, `as_str() -> Option<&str>` and a lossy `Display`) for every byte output, so it is the one byte-out type in that binding.
-- **Settings values are strings, and only strings.** The binding serializes the map into a JSON object of string values with the language's stock encoder and passes it verbatim. It never rewrites a value: no boolean or integer spelling, no float (the v0 Python and TypeScript encoders are deleted, §7). A non-string value is the language's own type error: a compile error in Go and Rust, `TypeError` in Python and TypeScript. A value whose bytes are not valid UTF-8 cannot be passed as a setting or a query parameter in 1.0; the workaround is to inline it in the SQL as `unhex('')`, built from hex digits only, which cannot break out of the quoted literal. The bindings refuse one before any call, as a usage error naming the input (settings, query parameters, the zone) and the key in escaped form (a Python `str` or a TypeScript string holding a lone surrogate counts); stock encoders would have replaced it with U+FFFD, so the library would have seen a different value. Rust takes `String` throughout, so a non-UTF-8 value is unreachable there.
+- **Settings values are strings, and only strings.** The binding serializes the map into a JSON object with the language's stock encoder and passes each value verbatim. It never rewrites a value: no boolean or integer spelling, no float (the v0 Python and TypeScript encoders are deleted, §7). A non-string value is the language's own type error: a compile error in Go and Rust, `TypeError` in Python and TypeScript. **A value whose bytes are not valid UTF-8 is the object form `{"b64": ""}`** (ABI v2, `json_object_byte_values`: a settings value, a query-parameter value, a `chs_set_defaults` default, and a value of a server profile's `settings`), written by the binding in place of the string, so the library sees the exact bytes; a value that is valid UTF-8 stays a JSON string. (In TypeScript a value is a `string` or a `Uint8Array`: the bytes of a `Uint8Array` that are not valid UTF-8 are the object form, and a string holding a lone surrogate has no bytes, so it is refused like a key; the macros of a server profile are text only.) A key that is not valid UTF-8 cannot be carried (a JSON object has no byte form for a key, and a macro name or value and a zone have none either): the bindings refuse one before any call, as a usage error naming the input (settings, query parameters, the profile) and the key in escaped form (a Python `str` or a TypeScript string holding a lone surrogate counts); stock encoders would have replaced it with U+FFFD, so the library would have seen a different key. Rust takes `String` throughout, so a non-UTF-8 value is unreachable there. A per-call zone is a settings value, so a zone that is not valid UTF-8 is written in the object form too, and the library validates it. What a schema description reports back is bytes, though: a server profile's settings value in the object form decodes to its bytes, so Rust's `SchemaServer::settings` holds `RawText` values.
- **A zone name is text**, because ClickHouse's zone names are, and a name `DateLUT` cannot load is refused by the library whatever its bytes. It crosses the ABI as counted bytes (`chs_initialize`) or as a JSON string (`session_timezone`), UTF-8 encoded.
- **A list of column names** (the `columns` option) is serialized as a JSON array of names under the name encoding of §5.
@@ -244,22 +255,26 @@ TypeScript runs every call synchronously on one thread, so within an isolate the
Every vocabulary is generated from the description into all four bindings, with the description's numbers and spellings. No binding keeps a copy of its own.
-| vocabulary | from | Go | Python | TypeScript | Rust | notes |
-| ------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------- | -------------------------- | ------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| formats | `chs_format` | `Format` (`int32`): `JSONEachRow`, `CSV`, …, `TSVWithNames` | `Format` (`IntEnum`) | `Format` | `Format` (`#[repr(i32)]`) | FIRM and frozen (D1.4). Each carries `ch_name`, ClickHouse's own name, which is how `capabilities` lists a format |
-| call statuses | `chs_status` | `Status` (`int32`): `StatusOK` … `StatusInternal` | `Status` (`IntEnum`) | `Status` | `chtypes::status::{OK, …}` (`i32`) | FIRM (D3) |
-| row and batch outcomes | `row_outcome`, `batch_outcome` | `Outcome`: `Accepted`, `AcceptedPoisoned`, `Rejected`, `Skipped`, `Unsupported` | `Outcome` | `Outcome` | `Outcome`, `#[non_exhaustive]` | FIRM. An unknown value reads as the description's fallback, `unsupported` |
-| filter outcomes | `filter_outcome` | `FilterOutcome`: `FilterOK`, `FilterRejected`, `FilterUnsupported` | `FilterOutcome` | `FilterOutcome` | `FilterOutcome` | FIRM. Unknown reads as `unsupported` |
-| filter verdicts | `filter_verdict` | `Verdict`: `VerdictTrue`, `VerdictFalse`, `VerdictError`, `VerdictDecline`, and `Answered()` | `Verdict`, `.answered` | `Verdict`, `answered` | `Verdict`, `answered()` | FIRM. Unknown reads as `d`. `answered` is the description's own fact, never a list kept here |
-| transform reasons | `transform_reason` | `ReasonOverflowWrap` … | `Reason.OVERFLOW_WRAP` … | `Reason.OverflowWrap` … | `reason::OVERFLOW_WRAP` … | FIRM. The spellings are v0's. Each carries its `lossy` fact, which `Transform.lossy` reads |
-| value sources | `value_src` | `SourceInput` … | `Source.INPUT` … | `Source.Input` … | `source::INPUT` … | FIRM. Every value now, not v0's two. Each carries its `is_stored` fact, which `Value.is_stored` reads |
-| `default_generated` | a `value_src` value | `SourceDefaultGenerated` | `Source.DEFAULT_GENERATED` | `Source.DefaultGenerated` | `source::DEFAULT_GENERATED` | FIRM. `is_stored` is true: the value is stored once the caller inserts the library's output (§5, Generated defaults). Generated from the description like every other value, never hand-added |
-| merge reasons | `merge_reason` | `MergeReason`: `MergeTTLDelete`, `MergeTTLColumnReset` | `MergeReason` | `MergeReason` | `MergeReason`, `#[non_exhaustive]` | ABI v2 dev. The reason of a batch's `at_merge` entry (§5). The spellings are the description's |
-| declined tiers | `declined_tier` | `DeclinedTier`: `TierPredicate`, `TierPredicateUnflipped`, `TierResultContent` | `DeclinedTier` | `DeclinedTier` | `DeclinedTier`, `#[non_exhaustive]` | ABI v2 dev. The tier of an entry of a schema description's `filter_declined_settings` (§5). The spellings are the description's; each binding reads a hyphen as a word boundary in its identifiers |
-| declined layers | `declined_layer` | `DeclinedLayer`: `LayerDefaults`, `LayerServer`, `LayerSchema` | `DeclinedLayer` | `DeclinedLayer` | `DeclinedLayer`, `#[non_exhaustive]` | ABI v2 dev. The settings layer of an entry of a schema description's `filter_declined_settings` (§5), the highest that sets the name: `defaults`, `server` or `schema`, lowest first. The spellings are the description's |
-| discover query parameters | `discover_query_param` | `DiscoverQueryParam`: `QueryParamDatabase`, `QueryParamTable` | `DiscoverQueryParam` | `DiscoverQueryParam` | `DiscoverQueryParam`, `#[non_exhaustive]` | ABI v2 dev. The discovery query's two parameters (§2, the discovery query), which a caller binds when it runs the query against its server. The spellings are the description's |
-| default kinds | `default_kind` | `DefaultKind`: `KindNone`, `KindDefault`, `KindMaterialized`, `KindAlias`, `KindEphemeral` | `DefaultKind` | `DefaultKind` | `DefaultKind` | FIRM; joins the description in the regeneration that freezes the fingerprint (§8 question 3). The spellings are v0's |
-| document groups | `CHS_DOC_*` | `DocFlags`: `DocValues`, `DocTransforms`, `DocDefaults`, `DocAll` | `DocFlags` | `DocFlags` | `DocFlags` | FIRM |
+| vocabulary | from | Go | Python | TypeScript | Rust | notes |
+| ------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | -------------------------- | ------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| formats | `chs_format` | `Format` (`int32`): `JSONEachRow`, `CSV`, …, `TSVWithNames` | `Format` (`IntEnum`) | `Format` | `Format` (`#[repr(i32)]`) | FIRM and frozen (D1.4). Each carries `ch_name`, ClickHouse's own name, which is how `capabilities` lists a format |
+| call statuses | `chs_status` | `Status` (`int32`): `StatusOK` … `StatusInternal` | `Status` (`IntEnum`) | `Status` | `chtypes::status::{OK, …}` (`i32`) | FIRM (D3) |
+| row and batch outcomes | `row_outcome`, `batch_outcome` | `Outcome`: `Accepted`, `AcceptedPoisoned`, `Rejected`, `Skipped`, `Unsupported` | `Outcome` | `Outcome` | `Outcome`, `#[non_exhaustive]` | FIRM. An unknown value reads as the description's fallback, `unsupported` |
+| filter outcomes | `filter_outcome` | `FilterOutcome`: `FilterOK`, `FilterRejected`, `FilterUnsupported` | `FilterOutcome` | `FilterOutcome` | `FilterOutcome` | FIRM. Unknown reads as `unsupported` |
+| filter verdicts | `filter_verdict` | `Verdict`: `VerdictTrue`, `VerdictFalse`, `VerdictError`, `VerdictDecline`, and `Answered()` | `Verdict`, `.answered` | `Verdict`, `answered` | `Verdict`, `answered()` | FIRM. Unknown reads as `d`. `answered` is the description's own fact, never a list kept here |
+| transform reasons | `transform_reason` | `ReasonOverflowWrap` … | `Reason.OVERFLOW_WRAP` … | `Reason.OverflowWrap` … | `reason::OVERFLOW_WRAP` … | FIRM. The spellings are v0's. Each carries its `lossy` fact, which `Transform.lossy` reads |
+| value sources | `value_src` | `SourceInput` …, `SourceDefaultDeferred`, `SourceDefaultOverride` | `Source.INPUT` … | `Source.Input` … | `source::INPUT` … | FIRM. Every value now, not v0's two. Each carries its `is_stored` fact, which `Value.is_stored` reads; an unlisted value reads the description's fallback (`skipped`), not stored. `default_deferred` (not stored) and `default_override` (stored) are ABI v2 |
+| `default_generated` | a `value_src` value | `SourceDefaultGenerated` | `Source.DEFAULT_GENERATED` | `Source.DefaultGenerated` | `source::DEFAULT_GENERATED` | FIRM. `is_stored` is true: the value is stored once the caller inserts the library's output (§5, Generated defaults). Generated from the description like every other value, never hand-added |
+| merge reasons | `merge_reason` | `MergeReason`: `MergeTTLDelete`, `MergeTTLColumnReset` | `MergeReason` | `MergeReason` | `MergeReason`, `#[non_exhaustive]` | ABI v2 dev. The reason of a batch's `at_merge` entry (§5). The spellings are the description's |
+| declined tiers | `declined_tier` | `DeclinedTier`: `TierPredicate`, `TierPredicateUnflipped`, `TierResultContent`, `TierServerRefuses` | `DeclinedTier` | `DeclinedTier` | `DeclinedTier`, `#[non_exhaustive]` | ABI v2 dev. The tier of an entry of a schema description's `filter_declined_settings` (§5). The spellings are the description's; each binding reads a hyphen as a word boundary in its identifiers |
+| declined layers | `declined_layer` | `DeclinedLayer`: `LayerDefaults`, `LayerServer`, `LayerSchema` | `DeclinedLayer` | `DeclinedLayer` | `DeclinedLayer`, `#[non_exhaustive]` | ABI v2 dev. The settings layer of an entry of a schema description's `filter_declined_settings` (§5), the highest that sets the name: `defaults`, `server` or `schema`, lowest first. The spellings are the description's |
+| refusal sites | `refusal_site` | `RefusalSite`: `RefusedAtCall`, `RefusedAtBodyRead`, `RefusedAtRow`, `RefusedAtBodyStep` | `RefusalSite` | `RefusalSite` | `RefusalSite`, `#[non_exhaustive]` | ABI v2 dev. Where a batch, a row or a filter result's refusal sits (`refused_at`, §5). The spellings are the description's; each binding reads an underscore as a word boundary in its identifiers |
+| postures | `posture` | `Posture`: `PostureCreate`, `PostureLoad` | `Posture` | `Posture` | `Posture`, `#[non_exhaustive]` | ABI v2 dev. How a schema's statement was compiled (`schema_options`' `posture`, and a schema description's `posture`). The spellings are the description's |
+| default support | `default_support` | `DefaultSupport`: `DefaultSupportComputed`, `DefaultSupportGenerated`, `DefaultSupportDeferred`, `DefaultSupportDeclined` | `DefaultSupport` | `DefaultSupport` | `DefaultSupport`, `#[non_exhaustive]` | ABI v2 dev, reserved. What a build does with a column's DEFAULT (`default_support`, §5). The spellings are the description's |
+| the decline code | `CHS_CODE_DECLINED` | `CodeDeclined` (`int32`, -2) | `CODE_DECLINED` | `CODE_DECLINED` | `CODE_DECLINED` (`i32`, crate root) | ABI v2 dev. The `code` of a declined row, outcome or verdict, outside ClickHouse's code table; 0 is written only on a success |
+| discover query parameters | `discover_query_param` | `DiscoverQueryParam`: `QueryParamDatabase`, `QueryParamTable` | `DiscoverQueryParam` | `DiscoverQueryParam` | `DiscoverQueryParam`, `#[non_exhaustive]` | ABI v2 dev. The discovery query's two parameters (§2, the discovery query), which a caller binds when it runs the query against its server. The spellings are the description's |
+| default kinds | `default_kind` | `DefaultKind`: `KindNone`, `KindDefault`, `KindMaterialized`, `KindAlias`, `KindEphemeral` | `DefaultKind` | `DefaultKind` | `DefaultKind` | FIRM; joins the description in the regeneration that freezes the fingerprint (§8 question 3). The spellings are v0's |
+| document groups | `CHS_DOC_*` | `DocFlags`: `DocValues`, `DocTransforms`, `DocDefaults`, `DocAll` | `DocFlags` | `DocFlags` | `DocFlags` | FIRM |
**A fact in TypeScript is a free function.** TypeScript's vocabularies are const objects (`Format.JSONEachRow` is a number, not an object), which cannot carry a member, so each fact the table names (`ch_name`, `answered`, `lossy`, `is_stored`) is a free function named for its vocabulary: `formatChName(f)`, `verdictAnswered(v)`, `reasonLossy(r)` and `sourceIsStored(s)`, generated from the description like the facts of the other three. This is the language's idiom, not a convenience of its own; the parity gate carries the four as spelling entries.
@@ -286,7 +301,7 @@ The class for each status and each loader refusal is [`spec/abi-v1/sdk.json`](..
### The fields
-**Every call error carries the five fields of `chs_error`, verbatim, and nothing synthesized:**
+**Every call error carries the six fields of `chs_error`, verbatim, and nothing synthesized:**
| field | meaning | Go | Python | TypeScript | Rust |
| --------- | ------------------------------------------------------------------------- | ---------------- | ---------------- | ---------------------- | ------------------ |
@@ -295,6 +310,7 @@ The class for each status and each loader refusal is [`spec/abi-v1/sdk.json`](..
| `ch_name` | the name this build's vendored table gives the code; ASCII; empty if none | `ChName string` | `ch_name: str` | `chName: string` | `ch_name: String` |
| `message` | ClickHouse's own message for a refusal, the library's otherwise; bytes | `Message string` | `message: bytes` | `messageBytes: Buffer` | `message: RawText` |
| `column` | the column concerned, empty if none; bytes | `Column string` | `column: bytes` | `column: Buffer` | `column: RawText` |
+| `reason` | the catalog id (`chs_error_reason`); ASCII; empty if none | `Reason string` | `reason: str` | `reason: string` | `reason: String` |
A misuse the binding detects itself carries `status` `CHS_INVALID_ARGUMENT`, `ch_code` 0, an empty `ch_name` and `column`, and a message naming the misuse, so a handler sees one shape whichever side caught it.
@@ -370,22 +386,25 @@ Every table below lists a document's fields as the regenerated description draws
### `RowResult`, from the `row` document
-| field | from | type |
-| ---------------------- | ------------------------------------------------------------------ | ------------------- |
-| `outcome` | `outcome`, already final: the library applies every promotion (D6) | `Outcome` |
-| `err_code` | `code` | int32 |
-| `err_msg` | `err` (rule 5) | bytes |
-| `columns` | `cols`, every entry, in document order | list of `Value` |
-| `values` | the entries of `cols` whose `src` is stored, in order | list of `Value` |
-| `transformed` | `transformed`, computed by the library (D6) | list of `Transform` |
-| `unknown_fields` | `unknown_fields`, name objects (rule 5) | list of bytes |
-| `unsupported_settings` | `unsupported_settings`, name objects (rule 5) | list of bytes |
-| `computed` | `computed` | list of `Computed` |
-| `verdict` | `verdict`, present only with an attached filter | optional `Verdict` |
-| `verdict_code` | `verdict_code` | int32 |
-| `verdict_err` | `verdict_err` (rule 5) | bytes |
-| `partition_id` | `partition_id` (rule 5) | optional bytes |
-| `input_span` | `input_span`: the bytes the reader consumed for this record | optional `Span` |
+| field | from | type |
+| ---------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------ |
+| `outcome` | `outcome`, already final: the library applies every promotion (D6) | `Outcome` |
+| `err_code` | `code`: ClickHouse's own code, 0 only on a success, `CHS_CODE_DECLINED` (-2) for a decline | int32 |
+| `err_msg` | `err` (rule 5) | bytes |
+| `reason` | `reason`: the decline's catalog id, present exactly where `code` is `CHS_CODE_DECLINED` | optional ASCII text |
+| `refused_at` | `refused_at`: where a single-row result's refusal sits; the rows of a batch carry neither this nor `refused_row` | optional `RefusalSite` |
+| `refused_row` | `refused_row`, present exactly when `refused_at` is `row` | optional unsigned 64-bit |
+| `columns` | `cols`, every entry, in document order | list of `Value` |
+| `values` | the entries of `cols` whose `src` is stored, in order | list of `Value` |
+| `transformed` | `transformed`, computed by the library (D6) | list of `Transform` |
+| `unknown_fields` | `unknown_fields`, name objects (rule 5) | list of bytes |
+| `computed` | `computed` | list of `Computed` |
+| `verdict` | `verdict`, present only with an attached filter | optional `Verdict` |
+| `verdict_code` | `verdict_code` | int32 |
+| `verdict_reason` | `verdict_reason`: the catalog id, present exactly where `verdict_code` is `CHS_CODE_DECLINED` | optional ASCII text |
+| `verdict_err` | `verdict_err` (rule 5) | bytes |
+| `partition_id` | `partition_id` (rule 5) | optional bytes |
+| `input_span` | `input_span`: the bytes the reader consumed for this record | optional `Span` |
`values` filters on `Value.is_stored`, which is the description's own `is_stored` fact for the entry's `src`; it consults no list the binding keeps, which is the whole difference from v0's "exclude `skipped` and `ephemeral_input`" rule. `columns` is new, and it is how a caller still sees the entries that are never stored (an `ephemeral_input` column, an unresolved DEFAULT).
@@ -401,30 +420,34 @@ The transform entry keeps v0's shape (`column`, `input`, `stored`, `reason`, `ro
### `BatchResult`, from the `batch` document and the export buffer
-| field | from | type |
-| ------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- |
-| `outcome` | `outcome`, final | `Outcome` |
-| `err_code`, `err_msg` | `code`, `err` (rule 5) | int32, bytes |
-| `rows` | `rows`, each decoded as a `RowResult` | list of `RowResult` |
-| `rows_read` | `rows_read` | unsigned 64-bit |
-| `rows_skipped` | `rows_skipped` | unsigned 64-bit |
-| `transformed` | `transformed`: every transform in the batch with its `row`, storage transforms (TTL) included, as the library lists them | list of `Transform` |
-| `unsupported_settings` | `unsupported_settings`, name objects (rule 5): the call's own settings the library declined (ABI v2) | list of bytes |
-| `engine_rows` | `engine_rows`: each stored row after the engine's insert-time merge, a list of cells | optional list of rows, each a list of `EngineCell` |
-| `at_merge` | `at_merge`: what an `OPTIMIZE TABLE ... FINAL` of the part would do to its rows (ABI v2) | list of `AtMergeEntry` |
-| `payload` | the `out_export` buffer, present only for an export that ran (absent when `export_declined` is non-empty) | optional payload |
-| `spans` | `row_spans`: each exported row's place in `payload` | optional list of `Span` |
-| `export_declined` | `export_declined` (rule 5) | bytes |
-| `rows_passed`, `rows_cut` | `rows_passed`, `rows_cut`, with an attached filter | unsigned 64-bit |
-| `partition_count` | `partition_count` | optional unsigned 64-bit |
-| `unconsumed` | `unconsumed`: the byte ranges the reader's error recovery skipped | list of `Span` |
-| `framing` | `framing`: what the reader decided about the body's framing | optional `Framing` |
-
-| type | fields, each from the same-named key unless noted |
-| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `AtMergeEntry` | `row` (int: an index into `engine_rows`, the part's row), `reason` (`MergeReason`, a `merge_reason` value), `column` (optional bytes, from `column` or `column_b64`: rule 5), `stored` (optional bytes, from `stored` or `stored_b64`: rule 5), `input_rows` (optional list of int) |
-
-**`at_merge` and the batch's own `unsupported_settings` (ABI v2).** `at_merge` lists what an `OPTIMIZE TABLE ... FINAL` of the part the `INSERT` writes would do to its rows, at the call's clock instant: `ttl_delete` (the rows TTL removes the row) or `ttl_column_reset` (a column TTL resets the column's value). `engine_rows` stays exactly what the `INSERT` writer produces, so the two never contradict each other. An entry's `row` indexes `engine_rows`, not the input body, and `input_rows`, when present, names the input rows, each by its index in the body, that formed that row after an engine's insert-time merge. A row both removed and reset lists only `ttl_delete`. `column` is present only for `ttl_column_reset`, and `stored` is absent when the column's `DEFAULT` reads the clock or a generator, so the value is decided at the merge. Absent is never empty, in any binding: Go `nil`, Python `None`, TypeScript `undefined`, Rust `None`. The list is "at least these", not "only these": a later background merge can remove more rows as more of them expire. The batch's own `unsupported_settings` names the call's settings the library declined. Both lists are empty when the document carries none. [`spec/abi-v2/docs.md`](../../spec/abi-v2/docs.md) is normative for both.
+| field | from | type |
+| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
+| `outcome` | `outcome`, final | `Outcome` |
+| `err_code`, `err_msg` | `code`, `err` (rule 5) | int32, bytes |
+| `reason` | `reason`: the catalog id of a declined batch, present exactly where `code` is `CHS_CODE_DECLINED` | optional ASCII text |
+| `refused_at`, `refused_row` | `refused_at` (a `refusal_site` value) and `refused_row`: where a refusal or decline sits, present whenever the outcome is not `accepted` or `accepted_poisoned`; `refused_row` for `row`, below `rows_read` | optional `RefusalSite`, optional unsigned 64-bit |
+| `rows` | `rows`, each decoded as a `RowResult` | list of `RowResult` |
+| `rows_read` | `rows_read` | unsigned 64-bit |
+| `rows_skipped` | `rows_skipped` | unsigned 64-bit |
+| `transformed` | `transformed`: every transform in the batch with its `row`, storage transforms (TTL) included, as the library lists them | list of `Transform` |
+| `unsupported_settings` | `unsupported_settings`, name objects (rule 5): the call's own settings the library declined (ABI v2) | list of bytes |
+| `engine_rows` | `engine_rows`: each stored row after the engine's insert-time merge, a list of cells | optional list of rows, each a list of `EngineCell` |
+| `at_merge` | `at_merge`: what an `OPTIMIZE TABLE ... FINAL` of the part would do to its rows (ABI v2) | list of `AtMergeEntry` |
+| `payload` | the `out_export` buffer, present only for an export that ran (absent when `export_declined` is non-empty) | optional payload |
+| `spans` | `row_spans`: one entry per `rows` entry, index-aligned, each exported row's place in `payload` and null for a row not exported | optional list of optional `Span` |
+| `export_columns` | `export_columns`: the export's one column list, name objects (rule 5) | optional list of bytes |
+| `export_declined` | `export_declined` (rule 5) | bytes |
+| `export_declined_reason` | `export_declined_reason`: the catalog id of a declined export | optional ASCII text |
+| `rows_passed`, `rows_cut` | `rows_passed`, `rows_cut`, with an attached filter | unsigned 64-bit |
+| `partition_count` | `partition_count` | optional unsigned 64-bit |
+| `unconsumed` | `unconsumed`: the byte ranges the reader's error recovery skipped | list of `Span` |
+| `framing` | `framing`: what the reader decided about the body's framing | optional `Framing` |
+
+| type | fields, each from the same-named key unless noted |
+| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `AtMergeEntry` | `part_row` (int: an index into `engine_rows`, the part's row), `reason` (`MergeReason`, a `merge_reason` value), `column` (optional bytes, from `column` or `column_b64`: rule 5), `stored` (optional bytes, from `stored` or `stored_b64`: rule 5), `input_rows` (optional list of int) |
+
+**`at_merge` and the batch's own `unsupported_settings` (ABI v2).** `at_merge` lists what an `OPTIMIZE TABLE ... FINAL` of the part the `INSERT` writes would do to its rows, at the call's clock instant: `ttl_delete` (the rows TTL removes the row) or `ttl_column_reset` (a column TTL resets the column's value). `engine_rows` stays exactly what the `INSERT` writer produces, so the two never contradict each other. An entry's `part_row` indexes `engine_rows`, not the input body, and `input_rows`, when present, names the input rows, each by its index in the body, that formed that row after an engine's insert-time merge. A row both removed and reset lists only `ttl_delete`. `column` is present only for `ttl_column_reset`, and `stored` is absent when the column's `DEFAULT` reads the clock or a generator, so the value is decided at the merge. Absent is never empty, in any binding: Go `nil`, Python `None`, TypeScript `undefined`, Rust `None`. The list is "at least these", not "only these": a later background merge can remove more rows as more of them expire. The batch's own `unsupported_settings` names the call's settings the library declined. Both lists are empty when the document carries none. [`spec/abi-v2/docs.md`](../../spec/abi-v2/docs.md) is normative for both.
`Framing` is `bom_skipped`, `container` and `header`, each the vendored reader's own state:
@@ -450,25 +473,28 @@ A column whose DEFAULT calls one of the random or UUID generators this build adm
### `FilterResult`, from the `filter_result` document
-| field | from | type |
-| ---------------------- | -------------------------------------------------------------------------- | ------------------------ |
-| `outcome` | `outcome` | `FilterOutcome` |
-| `err_code`, `err_msg` | `code`, `err` (rule 5) | int32, bytes |
-| `rows_read` | `rows_read` | unsigned 64-bit |
-| `unsupported_settings` | `unsupported_settings`, name objects (rule 5) | list of bytes |
-| `verdicts` | `verdicts`: one character per row, each mapped through the generated table | list of `Verdict` |
-| `errors` | `errors`: `row` (int), `code` (int32), `msg` (bytes, from `err`, rule 5) | list of `FilterRowError` |
+| field | from | type |
+| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
+| `outcome` | `outcome` | `FilterOutcome` |
+| `err_code`, `err_msg` | `code`, `err` (rule 5) | int32, bytes |
+| `reason` | `reason`: the catalog id of a declined evaluation, present exactly where `code` is `CHS_CODE_DECLINED` | optional ASCII text |
+| `refused_at`, `refused_row` | `refused_at` and `refused_row`, as the batch's: where a refusal or decline sits | optional `RefusalSite`, optional unsigned 64-bit |
+| `rows_read` | `rows_read` | unsigned 64-bit |
+| `unsupported_settings` | `unsupported_settings`, name objects (rule 5) | list of bytes |
+| `verdicts` | `verdicts`: one character per row, each mapped through the generated table | list of `Verdict` |
+| `errors` | `errors`: `row` (int), `code` (int32), `reason` (optional ASCII text, present where `code` is `CHS_CODE_DECLINED`), `msg` (bytes, from `err`, rule 5) | list of `FilterRowError` |
`e` and `d` are never answers, and a caller enforcing visibility fails closed on both; `Verdict.answered` says which. A non-ok outcome's verdicts are whatever the document holds: v0's rule that a binding empties them is the library's now.
### `SchemaDescription`, from the `schema_description` document
-| field | from | type |
-| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
-| `columns` | `columns`, in declared order. Each `Column`: `name` (bytes, rule 5), `type` (bytes, rule 5: the canonical type), `default_kind` (`DefaultKind`), `default_expr` (bytes, rule 5, from `default_expression`) | list of `Column` |
-| `filter_declined_settings` | `filter_declined_settings`: the settings the schema compiles under that this build's filters do not honor in a WHERE, over every layer known at schema compile, as the library wrote them (one entry per name, at the highest layer that sets it, by layer and then by name bytes), empty when it lists none (ABI v2) | list of `DeclinedSetting` |
-| `server` | `SchemaServer`: `timezone` (text: the zone the schema's types bind, the profile's or else the image zone), `settings` (settings, `{}` when the profile gave none) and `macros` (settings, absent exactly when the profile carried none). From `server`, which the document carries exactly when the schema was compiled on a server (ABI v2) | optional `SchemaServer`: absent exactly when the schema was compiled without a server |
-| `replicated` | `SchemaReplicated`: `zookeeper_path` (bytes, rule 5) and `replica_name` (bytes, rule 5), fully expanded. From `replicated`: what ClickHouse's own `TableZnodeInfo` resolved for a Replicated engine on a server, to compare with the server's `system.replicas` (ABI v2) | optional `SchemaReplicated` |
+| field | from | type |
+| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
+| `columns` | `columns`, in declared order. Each `Column`: `name` (bytes, rule 5), `type` (bytes, rule 5: the canonical type), `default_kind` (`DefaultKind`), `default_expr` (bytes, rule 5, from `default_expression`), `default_support` (optional `DefaultSupport`) and `default_support_reason` (optional ASCII text), reserved, and `reads` (optional list of bytes, name objects, rule 5), reserved | list of `Column` |
+| `posture` | `posture` (a `posture` value): how the statement was compiled, present only for `load`; absent is `create` (ABI v2) | optional `Posture` |
+| `filter_declined_settings` | `filter_declined_settings`: the settings the schema compiles under that this build's filters do not honor in a WHERE, over every layer known at schema compile, as the library wrote them (one entry per name, at the highest layer that sets it, by layer and then by name bytes), empty when it lists none (ABI v2) | list of `DeclinedSetting` |
+| `server` | `SchemaServer`: `timezone` (text: the zone the schema's types bind, the profile's or else the image zone), `settings` (settings, `{}` when the profile gave none) and `macros` (settings, absent exactly when the profile carried none). From `server`, which the document carries exactly when the schema was compiled on a server (ABI v2) | optional `SchemaServer`: absent exactly when the schema was compiled without a server |
+| `replicated` | `SchemaReplicated`: `zookeeper_path` (bytes, rule 5) and `replica_name` (bytes, rule 5), fully expanded. From `replicated`: what ClickHouse's own `TableZnodeInfo` resolved for a Replicated engine on a server, to compare with the server's `system.replicas` (ABI v2) | optional `SchemaReplicated` |
| type | fields, each from the same-named key unless noted |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
diff --git a/docs/reference/go.md b/docs/reference/go.md
index 7c0fe0b1..d0f1f8d9 100644
--- a/docs/reference/go.md
+++ b/docs/reference/go.md
@@ -31,7 +31,7 @@ Row and batch verdicts are **never** Go errors: they are `Outcome` in the result
| `*InternalError` | `CHS_INTERNAL`, or a document that does not decode |
| `*ArtifactError` | a fetch or load failure, with a `Code` (an `ErrorCode`, one of the generated `Code…` constants) and a sentinel per code for `errors.Is`, such as `ErrArtifactMissing` |
-All four call types embed `CallError`, whose five fields (`Status`, `ChCode`, `ChName`, `Message`, `Column`) read directly; `AsCallError` reads them from any of the four. `Message` and `Column` are byte strings, and `Error()` is the one lossy display form.
+All four call types embed `CallError`, whose six fields (`Status`, `ChCode`, `ChName`, `Message`, `Column`, `Reason`) read directly; `AsCallError` reads them from any of the four. `Reason` is the catalog id (ASCII). `Message` and `Column` are byte strings, and `Error()` is the one lossy display form.
## The Go spellings
diff --git a/docs/reference/python.md b/docs/reference/python.md
index b5b8dc9d..8b542d97 100644
--- a/docs/reference/python.md
+++ b/docs/reference/python.md
@@ -42,15 +42,17 @@ A library is never unloaded and never closed: there is no `Registry.close` or `L
## Schema, Filter and Block
-| Symbol | C entry point | Returns |
-| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------- |
-| `Schema.describe()` | `chs_schema_describe` | `SchemaDescription` |
-| `Schema.row(format, body, *, settings=None, session_timezone=None, columns=None)` | `chs_preview_row` | `RowResult` |
-| `Schema.rows(format, body, *, settings=None, session_timezone=None, columns=None, row_filter=None, export=None, doc_flags=DocFlags.ALL)` | `chs_preview_batch` | `BatchResult` |
-| `Schema.compile_filter(expr, *, params=None, settings=None, session_timezone=None)` | `chs_filter_create` | `Filter` |
-| `Schema.parse_block(format, body, *, settings=None, session_timezone=None, columns=None)` | `chs_block_create` | `Block` |
-| `Filter.rows(format, body, *, settings=None, session_timezone=None)`, `Filter.eval(block)` | `chs_filter_eval_body`, `chs_filter_eval_block` | `FilterResult` |
-| `Schema.close()`, `Filter.close()`, `Block.close()` | `chs_*_free` | `None`; idempotent, any order is safe |
+| Symbol | C entry point | Returns |
+| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------- |
+| `Schema.describe()` | `chs_schema_describe` | `SchemaDescription` |
+| `Schema.row(format, body, *, settings=None, session_timezone=None, columns=None, insertable_columns=None, column_defaults=None, verdict_basis=None, defer_defaults=None, insert_options_extra=None)` | `chs_preview_row` | `RowResult` |
+| `Schema.rows(format, body, *, settings=None, session_timezone=None, columns=None, insertable_columns=None, column_defaults=None, verdict_basis=None, defer_defaults=None, insert_options_extra=None, row_filter=None, export=None, doc_flags=DocFlags.ALL)` | `chs_preview_batch` | `BatchResult` |
+| `Schema.compile_filter(expr, *, params=None, settings=None, session_timezone=None, param_encoding=None, filter_options_extra=None)` | `chs_filter_create` | `Filter` |
+| `Schema.parse_block(format, body, *, settings=None, session_timezone=None, columns=None, insertable_columns=None, column_defaults=None, verdict_basis=None, defer_defaults=None, insert_options_extra=None)` | `chs_block_create` | `Block` |
+| `Filter.rows(format, body, *, settings=None, session_timezone=None, columns=None, insertable_columns=None, column_defaults=None, verdict_basis=None, defer_defaults=None, insert_options_extra=None)`, `Filter.eval(block)` | `chs_filter_eval_body`, `chs_filter_eval_block` | `FilterResult` |
+| `Schema.close()`, `Filter.close()`, `Block.close()` | `chs_*_free` | `None`; idempotent, any order is safe |
+
+**The options documents (ABI v2).** The insert options are keyword-only on the four calls that model an INSERT of a body (`Schema.row`, `Schema.rows`, `Schema.parse_block`, `Filter.rows`): `insertable_columns` (names), `column_defaults` (a sequence of `ColumnDefault(name, value)`), `verdict_basis` (`VerdictBasis`), `defer_defaults` (`DeferDefaults`). `compile_filter` takes `param_encoding` (`ParamEncoding`) and `compile_table` takes `posture` (`Posture`). Each `*_options_extra` mapping passes a member the binding has no name for, as JSON text by key, so the library (which refuses an unknown key) is the only judge. `None` leaves a member out and an empty sequence is sent as `[]`; no document is sent when no member is set. A settings or query-parameter value that is not UTF-8 is a `str` holding the bytes as `surrogateescape` makes them (`b'\xff'.decode('utf-8', 'surrogateescape')`), and travels as `{"b64": ...}`.
`Schema`, `Filter` and `Block` are context managers and are freed by a finalizer when abandoned. Using a closed object is a `UsageError`. `BytesIn` is `bytes | str` (a `str` is encoded as UTF-8); a body is `bytes` and never accepts `str`. The per-call zone is the `session_timezone` argument, which is written into `settings` as that one key, and giving both is a `UsageError`. The export channel is the keyword-only `export` of `rows`; there is no separate method.
@@ -60,11 +62,11 @@ A library is never unloaded and never closed: there is no `Registry.close` or `L
`Value.is_stored` and `Transform.lossy` are the description's own facts, read from the generated tables. `BatchResult.engine_rows` is `None` or a tuple of rows, each a tuple of `EngineCell`; `RowResult.partition_id` and `BatchResult.partition_count` are `None` unless the statement declared a partition key.
-**Vocabularies**, generated from the description ([§3](bindings-v1.md)): `Format` (an `IntEnum`, with `.ch_name`), `Status`, `Outcome` and `FilterOutcome` (`StrEnum`), `Verdict` (with `.answered`), `DefaultKind`, `DocFlags`, and the plain-string vocabularies `Reason` and `Source` (a `Source` of `DEFAULT_GENERATED` means insert the library's export, never the original body). `Reason.lossy(reason)` and `Source.is_stored(source)` are the same facts the result fields read.
+**Vocabularies**, generated from the description ([§3](bindings-v1.md)): `Format` (an `IntEnum`, with `.ch_name`), `Status`, `Outcome` and `FilterOutcome` (`StrEnum`), `Verdict` (with `.answered`), `DefaultKind`, `RefusalSite`, `Posture`, `DefaultSupport`, `DocFlags`, and the plain-string vocabularies `Reason` and `Source` (a `Source` of `DEFAULT_GENERATED` means insert the library's export, never the original body). `Reason.lossy(reason)` and `Source.is_stored(source)` are the same facts the result fields read.
## Errors
-One table ([§4](bindings-v1.md)). The four call errors are peers under `CallError`, which carries `status`, `ch_code`, `ch_name`, `message` (bytes) and `column` (bytes):
+One table ([§4](bindings-v1.md)). The four call errors are peers under `CallError`, which carries `status`, `ch_code`, `ch_name`, `message` (bytes), `column` (bytes) and `reason` (the catalog id, ASCII, empty if none):
| Class | Raised for |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
diff --git a/examples/go/main.go b/examples/go/main.go
index 9ca643e7..35233ebd 100644
--- a/examples/go/main.go
+++ b/examples/go/main.go
@@ -609,7 +609,6 @@ func section6(lib *chtypes.Library) {
blank()
kv("UnknownFields[]", fmt.Sprintf("%q", r.UnknownFields))
- kv("UnsupportedSettings[]", fmt.Sprintf("%q", r.UnsupportedSettings))
note("unknown fields are reported, not judged: whether to 400 on them is")
note("gateway policy")
blank()
@@ -1190,15 +1189,20 @@ func section15(lib *chtypes.Library) {
note("bytes are directly INSERT-able with no column list; DEFAULTs (seq=7,")
note("label='unknown') are already applied: preview == stored")
for i, sp := range res.Spans {
- l := "(no bytes: row not accepted)"
- if sp.Len > 0 {
- l = fmt.Sprintf("%q", res.Payload[sp.Off:sp.Off+sp.Len])
+ l := "(no bytes: row not exported)"
+ sd := "nil"
+ if sp != nil {
+ sd = fmt.Sprintf("{off:%d len:%d}", sp.Off, sp.Len)
+ if sp.Len > 0 {
+ l = fmt.Sprintf("%q", res.Payload[sp.Off:sp.Off+sp.Len])
+ }
}
- kv(fmt.Sprintf(" span[%d] {off:%d len:%d}", i, sp.Off, sp.Len), fmt.Sprintf("%s -> %s", res.Rows[i].Outcome, l))
+ kv(fmt.Sprintf(" span[%d] %s", i, sd), fmt.Sprintf("%s -> %s", res.Rows[i].Outcome, l))
}
- note("spans are INDEX-ALIGNED with rows; slicing spans out of the payload")
- note("IS the per-row payload, and concatenating non-zero spans reproduces")
- note("it exactly: batches merge by byte concatenation")
+ note("spans are INDEX-ALIGNED with rows, nil for a row not exported;")
+ note("slicing spans out of the payload IS the per-row payload, and")
+ note("concatenating non-nil spans reproduces it exactly: batches merge")
+ note("by byte concatenation")
blank()
// The lean document: fewer doc flags keep the whole verdict channel and
diff --git a/examples/python/demo.py b/examples/python/demo.py
index 546c5d7d..9041c25d 100644
--- a/examples/python/demo.py
+++ b/examples/python/demo.py
@@ -472,7 +472,6 @@ def section6(lib: chtypes.Library) -> None:
blank()
kv("unknown_fields[]", str([show(n) for n in r.unknown_fields]))
- kv("unsupported_settings[]", str([show(n) for n in r.unsupported_settings]))
blank()
with lib.compile_table(table("a UInt8, d UInt8 DEFAULT a + 1")) as dep:
@@ -1019,15 +1018,16 @@ def section15(lib: chtypes.Library) -> None:
note("wire order = declared minus MATERIALIZED/ALIAS/EPHEMERAL, so these bytes are")
note("INSERT-able with no column list; DEFAULTs are already applied")
for i, span in enumerate(batch.spans or ()):
- line = "(no bytes - row not accepted)"
- if span.len:
+ line = "(no bytes - row not exported)"
+ if span is not None and span.len:
line = repr(payload[span.off : span.off + span.len])
kv(
- f" span[{i}] {{off:{span.off} len:{span.len}}}",
+ f" span[{i}] "
+ + ("None" if span is None else f"{{off:{span.off} len:{span.len}}}"),
f"{batch.rows[i].outcome.value} -> {line}",
)
- note("spans are INDEX-ALIGNED with rows; concatenating the non-zero spans")
- note("reproduces the payload exactly")
+ note("spans are INDEX-ALIGNED with rows, None for a row not exported;")
+ note("concatenating the non-None spans reproduces the payload exactly")
blank()
lean = schema.rows(
diff --git a/examples/rust/src/main.rs b/examples/rust/src/main.rs
index dce640c0..445f9524 100644
--- a/examples/rust/src/main.rs
+++ b/examples/rust/src/main.rs
@@ -636,10 +636,6 @@ fn section6(lib: &Arc) {
}
blank();
kv("unknown_fields[]", &format!("{:?}", r.unknown_fields));
- kv(
- "unsupported_settings[]",
- &format!("{:?}", r.unsupported_settings),
- );
blank();
let dep = compile(lib, "a UInt8, d UInt8 DEFAULT a + 1");
@@ -1221,21 +1217,27 @@ fn section15(lib: &Arc) {
),
);
for (i, span) in batch.spans.iter().flatten().enumerate() {
- let line = if span.len > 0 {
+ let line = if let Some(span) = span.as_ref().filter(|s| s.len > 0) {
let (off, len) = (span.off as usize, span.len as usize);
format!(
"{:?}",
String::from_utf8_lossy(payload.get(off..off + len).unwrap_or_default())
)
} else {
- "(no bytes: row not accepted)".to_string()
+ "(no bytes: row not exported)".to_string()
};
let outcome = batch
.rows
.get(i)
.map_or("?".to_string(), |r| r.outcome.to_string());
kv(
- &format!(" span[{i}] {{off:{} len:{}}}", span.off, span.len),
+ &format!(
+ " span[{i}] {}",
+ span.as_ref().map_or("None".to_string(), |s| format!(
+ "{{off:{} len:{}}}",
+ s.off, s.len
+ ))
+ ),
&format!("{outcome} -> {line}"),
);
}
diff --git a/examples/ts/demo.ts b/examples/ts/demo.ts
index a69f164f..606d4194 100644
--- a/examples/ts/demo.ts
+++ b/examples/ts/demo.ts
@@ -462,7 +462,6 @@ function section6(lib: Library): void {
blank();
kv('unknownFields[]', JSON.stringify(r.unknownFields.map((b) => b.toString())));
- kv('unsupportedSettings[]', JSON.stringify(r.unsupportedSettings.map((b) => b.toString())));
note('unknown fields are reported, not judged: whether to 400 on them is gateway policy');
blank();
@@ -914,10 +913,10 @@ function section15(lib: Library): void {
note('wire order = declared minus MATERIALIZED/ALIAS/EPHEMERAL, so these bytes are directly');
note("INSERT-able with no column list; DEFAULTs (seq=7, label='unknown') are already applied");
(batch.spans ?? []).forEach((span, i) => {
- const line = span.len ? JSON.stringify(payload.subarray(span.off, span.off + span.len).toString()) : '(no bytes: row not accepted)';
- kv(` span[${i}] {off:${span.off} len:${span.len}}`, `${batch.rows[i]?.outcome} -> ${line}`);
+ const line = span?.len ? JSON.stringify(payload.subarray(span.off, span.off + span.len).toString()) : '(no bytes: row not exported)';
+ kv(` span[${i}] ${span === null ? 'null' : `{off:${span.off} len:${span.len}}`}`, `${batch.rows[i]?.outcome} -> ${line}`);
});
- note('spans are INDEX-ALIGNED with rows; concatenating the non-zero spans reproduces the payload');
+ note('spans are INDEX-ALIGNED with rows, null for a row not exported; concatenating the non-null spans reproduces the payload');
blank();
const lean = schema.rows(Format.JSONEachRow, body, { settings, exportFormat, docFlags: 0 });
@@ -1306,7 +1305,7 @@ async function openOrNote(registry: Registry, line: string): Promise[0] {
- return { status: 0, chCode: 0, chName: '', messageBytes: Buffer.alloc(0), column: Buffer.alloc(0) };
+ return { status: 0, chCode: 0, chName: '', messageBytes: Buffer.alloc(0), column: Buffer.alloc(0), reason: '' };
}
function errName(err: unknown): string {
diff --git a/go/CHANGELOG.md b/go/CHANGELOG.md
index 6c972166..47de7371 100644
--- a/go/CHANGELOG.md
+++ b/go/CHANGELOG.md
@@ -8,6 +8,8 @@ The four bindings in this repository are released together and give one answer,
### Added
+- `column_defaults` is covered by unit tests (ABI v2): the exact `insert_options` JSON for a UTF-8 value, a non-UTF-8 value (`value_b64`), an empty value (sent as `""`) and two entries in order, and the decode of a `default_override` cell (`SourceDefaultOverride`, stored). The batches guide gains a usage section. No behavior changed.
+- `CallError.Reason` (ABI v2): the error's catalog id, read through `chs_error_reason` by the generated layer and passed through verbatim (ASCII, empty if none), on all four call classes.
- A fetch-only call (#492): `(*Registry).Fetch(ctx, request) (Resolved, error)` resolves, fetches, verifies and installs the build a request names into the cache, as `chtypes fetch` does, and opens nothing: no library is loaded and the process setup is untouched, so it needs no `Setup`. It honors the registry's fetch options (offline, the cache, and under the production channel the bases, the trust and the lock), and a concurrent `Fetch` and open of one request share one fetch. The `Resolved` names the installed version, build, digests and library path; errors are the fetch codes, and a done `ctx` is `ctx.Err()`.
- `chtypes resolve [--json]` (#493): what a line or version resolves to on every platform the registry offers, one `resolved ` line each, or with `--json` one JSON array of `{platform, version, build, manifest}` objects, each signed statement verified exactly as a fetch verifies it and no layer downloaded; `--offline` answers from the cache. The four CLIs print the same bytes, so one parser reads them (a CI cache key, for example).
- `chtypes prune [--line ] [--keep ] [--dry-run]` (#494): removes the installed builds of each line and platform that at least n newer installed builds supersede (n defaults to 1), each with its record, its blobs and its `index.json` entry; `--dry-run` prints exactly what would go. It never removes a build a running process holds: a registry now holds every build it opens or fetches, with a shared `flock` on the entry's `verified.json` until the process exits, and prune removes an entry only after taking the exclusive lock without waiting, reporting it `in-use` otherwise. It never touches a system directory, nor, on the dev channel, another fingerprint's builds. The rules are `docs/guides/fetch-v1.md` sections 1 and 3, and the `resolve-build-*` and `prune-*` conformance cases hold all four bindings to them.
@@ -24,6 +26,10 @@ The four bindings in this repository are released together and give one answer,
- Build-level revocation for generation 2 (#625; `docs/guides/fetch-v1.md`, "Build-level revocation"). A build its signed revocation refuses is the new code `CHTYPES_ARTIFACT_REVOKED`, exit status 11 (`CodeArtifactRevoked`, `ErrArtifactRevoked`), whose message names the advisory and whose `ArtifactError.Revocation` carries it for programs (`Revocation`, `Advisory`). An install reads a revocation from the trust listing it already makes and refuses before the layer; `--frozen` makes one listing, warm or cold, so a warm frozen fetch makes 1 request instead of 0 (`--frozen --offline` stays at 0); a registry open, `preload` and a fetch that finds the build installed check it at most once per build per process, one attempt per base, no backoff and one 5 s deadline (`ocifetch.CheckInstalled`); `--lock` and `update` refuse a revoked build on every platform they pin. A verified revocation is stored in the build's cache entry (`revocation.json`) and refuses it from then on with no request, offline too; `chtypes verify` reports it and exits 11. A revocation that does not verify is ignored with one warning; a status that cannot be read is one warning, or with `FetchOptions.RequireRevocationCheck` (`--require-revocation-check`) the transport's error. `FetchOptions.AllowRevoked` (`--allow-revoked`) uses a revoked build for one call, with a warning, never for a lock or an update. A `410` whose body's `errors[0].detail.revoked` is an object is `CHTYPES_ARTIFACT_REVOKED`; every other `410` stays `CHTYPES_SOURCE_RETIRED`. The v1 contract is unchanged.
- A `410 Gone` is a retired repository, and permanent (#571). Any request the fetch layer makes (a tag, an alias, a digest, a blob, `tags/list`, the referrers API or its fallback tag, after a redirect too) that answers `410` is now the new code `CHTYPES_SOURCE_RETIRED` (exit status 10), never retried and never sent to the next base, nor from the dev channel's alias to its tag. It was `CHTYPES_SOURCE_UNREACHABLE` (exit 3) with the registry's message dropped, and Go, Python and Rust moved on to the next base, so a mirror's 404 could hide the retirement. The error names the URL that answered and carries the registry's own message, with control and bidirectional-formatting characters removed and cut to 256 code points: Go `CodeSourceRetired` (`errors.Is(err, ErrSourceRetired)`), Python and TypeScript `SourceRetiredError`, Rust `Error::SourceRetired`. The rule is `docs/guides/fetch-v1.md` section 2, "A retired repository", and the `retired-410-*` and `list-tags-retired-410` conformance cases and one shared message table (`tests/fixtures/retired-message/cases.json`) hold all four bindings to it.
- A 2.0.0-dev SDK prefers the newest build of its own fingerprint (#511). A version request resolves `--fp-` first, the alias the staging dev registry serves beside every dev tag, with the binding's own generated ABI fingerprint, and the tag itself only when every base answers the alias 404; any other failure of the alias request is that failure, with no fallback, and an alias whose index lacks the platform is `CHTYPES_ARTIFACT_UNPUBLISHED`. The dev channel's cache lookups (the installed-build lookup a registry open makes first, the offline answer, and the monotonic rule of a fetch) now see only records whose signed `abi_fingerprint` is the SDK's own, so a newer build of another fingerprint, installed by another dev SDK in the shared cache, is never returned or kept over this one's own, and is never removed or rewritten. So a dev build of a newer fingerprint no longer strands an older dev SDK until its consumers upgrade. Trust is unchanged (the signed predicate and the load checks decide), a listing never shows an alias, nothing records one, `list` and `where` are unchanged, and there is no option: both steps are the dev channel's alone, and the v1 contract and every production channel never take them. The rule is `docs/guides/fetch-v1.md` section 3, and the `dev-alias-*`, `resolve-installed-dev-alias-*` and `list-tags-dev-alias` conformance cases hold all four bindings to it.
+- Spec batch 2, the options documents (ABI v2; moves the dev fingerprint): `WithInsertableColumns`, `WithColumnDefaults` (with `ColumnDefault`), `WithVerdictBasis` (`VerdictBasisInput`, `VerdictBasisExport`) and `WithDeferDefaults` (`DeferDefaultsNone`, `DeferDefaultsServer`) build `insert_options` for `Schema.Row`, `Schema.Rows`, `Schema.ParseBlock` and `Filter.Rows`; `WithParamEncoding` (`ParamEncodingEscaped`, `ParamEncodingRaw`) builds `filter_options` for `CompileFilter`; `WithPosture` (`PostureCreate`, `PostureLoad`) builds `schema_options` for `CompileTable`. `WithInsertOption`, `WithFilterOption` and `WithSchemaOption` pass a member the binding has no name for through as JSON text, so the library, which refuses an unknown key (rule r1), is the only judge. A document with no member set is not sent. `WithColumns` now applies to `Filter.Rows` too (`chs_filter_eval_body` takes the column list).
+- Spec batch 2, a value that is not UTF-8 (ABI v2): a settings value, a query-parameter value, a `Setup` default and a server profile's settings value is sent as `{"b64": ...}` (the content `json_object_byte_values`), where the binding used to refuse it; and `SchemaServer.Settings` decodes the same form back to its bytes.
+- Spec batch 2, the decline and refusal members (ABI v2): `RowResult`, `BatchResult` and `FilterResult` carry `Reason` (the catalog id, present where the code is `CodeDeclined`, -2), `RefusedAt` (a `RefusalSite`: `RefusedAtCall`, `RefusedAtBodyRead`, `RefusedAtRow`, `RefusedAtBodyStep`) and `RefusedRow`; `RowResult.VerdictReason`; `FilterRowError.Reason`; `BatchResult.ExportColumns` and `BatchResult.ExportDeclinedReason`.
+- Spec batch 2, the schema description (ABI v2): `SchemaDescription.Posture`, and `Column.DefaultSupport` (a `DefaultSupport`), `Column.DefaultSupportReason` and `Column.Reads`, reserved members a build reports when it implements them. `SourceDefaultDeferred`, `SourceDefaultOverride` (stored) and `TierServerRefuses` join the vocabularies.
### Changed
@@ -68,9 +74,14 @@ The four bindings in this repository are released together and give one answer,
- **Enforcement gate lifted** for `26.8` on `darwin-arm64`.
- **Enforcement gate lifted** for `26.9` on `darwin-arm64`.
- `docs/limitations.md` is re-checked against artifact build `20261012.142240` (v2-dev.13), the library half measured on that build: functions that need a server, in a column expression, and a body the server may read serially or in parallel under a non-zero error budget are listed under "Declined by design"; the known divergence for a column expression that calls a function needing a server is retired, since that build declines those schemas (on 26.8 and 26.9 the `AI*` and `ai*` functions compile, as the page says); and the `lossy` raw-NUL TSV gap is marked fixed on that build and still present in the 1.0 libraries. `docs/guides/filters.md` gains a section on subcolumn filters and a JSON value typed by its batch.
+- **Breaking (ABI v2 dev): `BatchResult.Spans` is `[]*Span`**, one entry per `Rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which the entry reads as nil. It was `[]Span` of the exported rows only.
+- **Breaking (ABI v2 dev): `AtMergeEntry.Row` is `AtMergeEntry.PartRow`**, the document's `part_row`.
+- A document's `code` can be `CodeDeclined` (-2) for a row, batch, filter result or verdict this build declined, and every decline names its `Reason`; a `Source` the description does not list reads the description's fallback (`skipped`), not stored.
### Removed
+- **Breaking (ABI v2 dev): `RowResult.UnsupportedSettings`**: the row document no longer carries `unsupported_settings`; the batch's and the filter result's lists stay.
+- The refusal of a settings, query-parameter, default or server-settings value that is not valid UTF-8 (it is the `{"b64": ...}` form now); a key that is not valid UTF-8 is still refused.
- `SchemaServer.FilterDeclinedSettings` (a breaking change inside v2 dev, #588): the server member no longer carries the list. Read `SchemaDescription.FilterDeclinedSettings`, whose `server`-layer entries are the profile's.
### Fixed
diff --git a/go/README.md b/go/README.md
index cee14956..c356b7d5 100644
--- a/go/README.md
+++ b/go/README.md
@@ -108,7 +108,7 @@ A bad **row** is a verdict, not an error: `Outcome` becomes `Rejected` with Clic
- `*InternalError`: a library bug, or a document that does not decode.
- `*ArtifactError`: fetch and load failures, with a `Code` and a sentinel per code for `errors.Is`.
-`chtypes.AsCallError` reads the five fields every call error carries.
+`chtypes.AsCallError` reads the six fields every call error carries.
## Threads
diff --git a/go/chtypes/capture_batch_test.go b/go/chtypes/capture_batch_test.go
index f5ccd738..0e313623 100644
--- a/go/chtypes/capture_batch_test.go
+++ b/go/chtypes/capture_batch_test.go
@@ -52,7 +52,7 @@ func TestCaptureBatchDocuments(t *testing.T) {
defer sch.Close()
body := captureBody()
for _, flags := range []DocFlags{0, DocAll} {
- docBuf, exportBuf, cerr := sch.lib.tbl.PreviewBatch(sch.h, int32(JSONEachRow), body, nil, nil, nil, int32(ExportNone), uint32(flags))
+ docBuf, exportBuf, cerr := sch.lib.tbl.PreviewBatch(sch.h, int32(JSONEachRow), body, nil, nil, nil, int32(ExportNone), uint32(flags), nil)
if cerr != nil {
t.Fatal(callError(cerr))
}
@@ -67,12 +67,12 @@ func TestCaptureBatchDocuments(t *testing.T) {
// The third document is the shape a filtering caller gets: a filter
// attached, so the document also carries rows_passed, rows_cut and one
// row_spans entry per row, and an export asked for.
- flt, cerr := sch.lib.tbl.FilterCreate(sch.h, []byte("small < {m:Int8}"), []byte(`{"m":"50"}`), nil)
+ flt, cerr := sch.lib.tbl.FilterCreate(sch.h, []byte("small < {m:Int8}"), []byte(`{"m":"50"}`), nil, nil)
if cerr != nil {
t.Fatal(callError(cerr))
}
defer flt.Close()
- docBuf, exportBuf, cerr := sch.lib.tbl.PreviewBatch(sch.h, int32(JSONEachRow), body, nil, nil, flt, int32(JSONCompactEachRow), 0)
+ docBuf, exportBuf, cerr := sch.lib.tbl.PreviewBatch(sch.h, int32(JSONEachRow), body, nil, nil, flt, int32(JSONCompactEachRow), 0, nil)
if cerr != nil {
t.Fatal(callError(cerr))
}
diff --git a/go/chtypes/decode.go b/go/chtypes/decode.go
index 0e3fc011..bb7bc6a0 100644
--- a/go/chtypes/decode.go
+++ b/go/chtypes/decode.go
@@ -162,12 +162,26 @@ func (r *reader) stringMap(m map[string]any, key, path string) map[string]string
}
out := make(map[string]string, len(om))
for k, e := range om {
- s, ok := e.(string)
- if !ok {
- r.fail(path+"."+key+"."+k, "want a JSON string")
+ switch s := e.(type) {
+ case string:
+ out[k] = s
+ case map[string]any:
+ // The object form of a value whose bytes are not UTF-8: {"b64": ...}.
+ enc, ok := s["b64"].(string)
+ if !ok || len(s) != 1 {
+ r.fail(path+"."+key+"."+k, `want a JSON string or {"b64": "..."}`)
+ return nil
+ }
+ raw, err := base64.StdEncoding.DecodeString(enc)
+ if err != nil {
+ r.fail(path+"."+key+"."+k, "b64 is not standard base64: %v", err)
+ return nil
+ }
+ out[k] = string(raw)
+ default:
+ r.fail(path+"."+key+"."+k, `want a JSON string or {"b64": "..."}`)
return nil
}
- out[k] = s
}
return out
}
@@ -259,6 +273,16 @@ func (r *reader) u64Of(v any, path string) uint64 {
return u
}
+// optU64 reads an unsigned 64-bit integer that may be absent: nil when absent.
+func (r *reader) optU64(m map[string]any, key, path string) *uint64 {
+ v := r.field(m, key)
+ if v == nil {
+ return nil
+ }
+ n := r.u64Of(v, path+"."+key)
+ return &n
+}
+
func (r *reader) i32(m map[string]any, key, path string) int32 {
v := r.field(m, key)
if v == nil {
@@ -361,6 +385,24 @@ func (r *reader) span(v any, path string) *Span {
return &Span{Off: r.u64(m, "off", path), Len: r.u64(m, "len", path)}
}
+// rowSpans reads row_spans: one entry per row, a span for an exported row and
+// JSON null for a row not exported (nil here).
+func (r *reader) rowSpans(m map[string]any, key, path string) []*Span {
+ arr := r.array(r.field(m, key), path+"."+key)
+ if arr == nil {
+ return nil
+ }
+ out := make([]*Span, 0, len(arr))
+ for i, e := range arr {
+ if e == nil {
+ out = append(out, nil)
+ continue
+ }
+ out = append(out, r.span(e, fmt.Sprintf("%s.%s[%d]", path, key, i)))
+ }
+ return out
+}
+
func (r *reader) spans(m map[string]any, key, path string) []Span {
arr := r.array(r.field(m, key), path+"."+key)
if arr == nil {
@@ -417,7 +459,7 @@ func (r *reader) atMerge(m map[string]any, key, path string) []AtMergeEntry {
return nil
}
out = append(out, AtMergeEntry{
- Row: r.int(em, "row", p),
+ PartRow: r.int(em, "part_row", p),
Reason: MergeReason(r.text(em, "reason", p)),
Column: r.optBytes(em, "column", p),
Stored: r.optBytes(em, "stored", p),
@@ -456,14 +498,17 @@ func (r *reader) declinedSettings(m map[string]any, key, path string) []Declined
// batch's rows).
func (r *reader) row(m map[string]any, path string) RowResult {
res := RowResult{
- Outcome: parseRowOutcome(r.text(m, "outcome", path)),
- ErrCode: r.i32(m, "code", path),
- ErrMsg: r.bytes(m, "err", path),
- Transformed: r.transforms(m, "transformed", path),
- UnsupportedSettings: r.nameList(m, "unsupported_settings", path),
- UnknownFields: r.nameList(m, "unknown_fields", path),
- VerdictCode: r.i32(m, "verdict_code", path),
- VerdictErr: r.bytes(m, "verdict_err", path),
+ Outcome: parseRowOutcome(r.text(m, "outcome", path)),
+ ErrCode: r.i32(m, "code", path),
+ ErrMsg: r.bytes(m, "err", path),
+ Reason: r.text(m, "reason", path),
+ RefusedAt: RefusalSite(r.text(m, "refused_at", path)),
+ RefusedRow: r.optU64(m, "refused_row", path),
+ Transformed: r.transforms(m, "transformed", path),
+ UnknownFields: r.nameList(m, "unknown_fields", path),
+ VerdictCode: r.i32(m, "verdict_code", path),
+ VerdictReason: r.text(m, "verdict_reason", path),
+ VerdictErr: r.bytes(m, "verdict_err", path),
}
if arr := r.array(r.field(m, "cols"), path+".cols"); arr != nil {
for i, e := range arr {
@@ -568,18 +613,23 @@ func decodeBatchGeneric(raw []byte, payload []byte) (BatchResult, error) {
r := &reader{doc: "batch"}
m := r.object(v, "$")
res := BatchResult{
- Outcome: parseBatchOutcome(r.text(m, "outcome", "$")),
- ErrCode: r.i32(m, "code", "$"),
- ErrMsg: r.bytes(m, "err", "$"),
- RowsRead: r.u64(m, "rows_read", "$"),
- RowsSkipped: r.u64(m, "rows_skipped", "$"),
- Transformed: r.transforms(m, "transformed", "$"),
- Payload: payload,
- Spans: r.spans(m, "row_spans", "$"),
- ExportDeclined: r.bytes(m, "export_declined", "$"),
- RowsPassed: r.u64(m, "rows_passed", "$"),
- RowsCut: r.u64(m, "rows_cut", "$"),
- Unconsumed: r.spans(m, "unconsumed", "$"),
+ Outcome: parseBatchOutcome(r.text(m, "outcome", "$")),
+ ErrCode: r.i32(m, "code", "$"),
+ ErrMsg: r.bytes(m, "err", "$"),
+ Reason: r.text(m, "reason", "$"),
+ RefusedAt: RefusalSite(r.text(m, "refused_at", "$")),
+ RefusedRow: r.optU64(m, "refused_row", "$"),
+ RowsRead: r.u64(m, "rows_read", "$"),
+ RowsSkipped: r.u64(m, "rows_skipped", "$"),
+ Transformed: r.transforms(m, "transformed", "$"),
+ Payload: payload,
+ Spans: r.rowSpans(m, "row_spans", "$"),
+ ExportColumns: r.nameList(m, "export_columns", "$"),
+ ExportDeclined: r.bytes(m, "export_declined", "$"),
+ ExportDeclinedReason: r.text(m, "export_declined_reason", "$"),
+ RowsPassed: r.u64(m, "rows_passed", "$"),
+ RowsCut: r.u64(m, "rows_cut", "$"),
+ Unconsumed: r.spans(m, "unconsumed", "$"),
// The call's own declined settings, and what a merge of the part would
// do (ABI v2): both absent on a document that predates them.
UnsupportedSettings: r.nameList(m, "unsupported_settings", "$"),
@@ -660,6 +710,9 @@ func decodeFilterResult(raw []byte) (FilterResult, error) {
Outcome: parseFilterOutcome(r.text(m, "outcome", "$")),
ErrCode: r.i32(m, "code", "$"),
ErrMsg: r.bytes(m, "err", "$"),
+ Reason: r.text(m, "reason", "$"),
+ RefusedAt: RefusalSite(r.text(m, "refused_at", "$")),
+ RefusedRow: r.optU64(m, "refused_row", "$"),
RowsRead: r.u64(m, "rows_read", "$"),
UnsupportedSettings: r.nameList(m, "unsupported_settings", "$"),
}
@@ -674,9 +727,10 @@ func decodeFilterResult(raw []byte) (FilterResult, error) {
break
}
res.Errors = append(res.Errors, FilterRowError{
- Row: r.int(em, "row", p),
- Code: r.i32(em, "code", p),
- Msg: r.bytes(em, "err", p),
+ Row: r.int(em, "row", p),
+ Code: r.i32(em, "code", p),
+ Reason: r.text(em, "reason", p),
+ Msg: r.bytes(em, "err", p),
})
}
}
@@ -705,9 +759,17 @@ func decodeSchemaDescription(raw []byte) (SchemaDescription, error) {
Type: r.bytes(cm, "type", p),
DefaultKind: kind,
DefaultExpr: r.bytes(cm, "default_expression", p),
+ // Reserved members: absent until a build reports them. An unlisted
+ // default_support is its unknown(n), kept (r3).
+ DefaultSupport: DefaultSupport(r.text(cm, "default_support", p)),
+ DefaultSupportReason: r.text(cm, "default_support_reason", p),
+ Reads: r.nameList(cm, "reads", p),
})
}
}
+ // Present only for a schema compiled in the load posture; an unlisted
+ // posture is its unknown(n), kept (r3).
+ res.Posture = Posture(r.text(m, "posture", "$"))
// Over every layer known at schema compile, with or without a server.
res.FilterDeclinedSettings = r.declinedSettings(m, "filter_declined_settings", "$")
// Both are absent on a schema compiled without a server, so that document
diff --git a/go/chtypes/decode_batch.go b/go/chtypes/decode_batch.go
index d75b7a9b..4c113a81 100644
--- a/go/chtypes/decode_batch.go
+++ b/go/chtypes/decode_batch.go
@@ -505,6 +505,7 @@ type colDoc struct {
RefTypeB64 fIgnore `json:"ref_type_b64"`
Wire fIgnore `json:"wire"`
WireB64 fIgnore `json:"wire_b64"`
+ RefUnclass fIgnore `json:"ref_unclassified"`
Nullable fIgnore `json:"nullable"`
Poison fIgnore `json:"poison"`
NullInput fIgnore `json:"null_input"`
@@ -532,7 +533,7 @@ type cellDoc struct {
// atMergeDoc is one entry of at_merge (ABI v2).
type atMergeDoc struct {
- Row fInt `json:"row"`
+ PartRow fInt `json:"part_row"`
Reason fWord `json:"reason"`
Column fWord `json:"column"`
ColumnB64 fB64 `json:"column_b64"`
@@ -577,19 +578,26 @@ type framingDoc struct {
// batchRowDoc is one entry of rows. The lists keep their backing arrays between
// rows (reset), the scalars are cleared.
type batchRowDoc struct {
- Outcome fWord `json:"outcome"`
- Code fI32 `json:"code"`
- Err fString `json:"err"`
- ErrB64 fB64 `json:"err_b64"`
- Verdict fString `json:"verdict"`
- VerdictCode fI32 `json:"verdict_code"`
- VerdictErr fString `json:"verdict_err"`
- VerdictErrB64 fB64 `json:"verdict_err_b64"`
- PartitionID fString `json:"partition_id"`
- PartitionIDB64 fB64 `json:"partition_id_b64"`
- InputSpan fObj[spanDoc] `json:"input_span"`
- UnknownFields fList[nameDoc] `json:"unknown_fields"`
- UnsupportedSetting fList[nameDoc] `json:"unsupported_settings"`
+ Outcome fWord `json:"outcome"`
+ Code fI32 `json:"code"`
+ Err fString `json:"err"`
+ ErrB64 fB64 `json:"err_b64"`
+ Reason fWord `json:"reason"`
+ RefusedAt fWord `json:"refused_at"`
+ RefusedRow fU64 `json:"refused_row"`
+ Verdict fString `json:"verdict"`
+ VerdictCode fI32 `json:"verdict_code"`
+ VerdictReason fWord `json:"verdict_reason"`
+ VerdictErr fString `json:"verdict_err"`
+ VerdictErrB64 fB64 `json:"verdict_err_b64"`
+ PartitionID fString `json:"partition_id"`
+ PartitionIDB64 fB64 `json:"partition_id_b64"`
+ InputSpan fObj[spanDoc] `json:"input_span"`
+ UnknownFields fList[nameDoc] `json:"unknown_fields"`
+ // unsupported_settings was removed from the row document (spec batch 2); a
+ // library that still writes it is read as the fast path always has, the
+ // member ignored (rule r2), rather than sent to the slower generic reader.
+ UnsupportedSetting fIgnore `json:"unsupported_settings"`
Computed fList[computedDoc] `json:"computed"`
Cols fList[colDoc] `json:"cols"`
Transformed fList[transformDoc] `json:"transformed"`
@@ -600,7 +608,6 @@ func newBatchRowDoc() *batchRowDoc {
r := &batchRowDoc{}
r.InputSpan.sub = sub
r.UnknownFields.sub = sub
- r.UnsupportedSetting.sub = sub
r.Computed.sub = sub
r.Cols.sub = sub
r.Transformed.sub = sub
@@ -611,12 +618,13 @@ func newBatchRowDoc() *batchRowDoc {
// emptied with its backing array kept.
func (d *batchRowDoc) reset() {
d.Outcome, d.Code, d.Err, d.ErrB64 = fWord{}, fI32{}, fString{}, fB64{}
- d.Verdict, d.VerdictCode = fString{}, fI32{}
+ d.Reason, d.RefusedAt, d.RefusedRow = fWord{}, fWord{}, fU64{}
+ d.Verdict, d.VerdictCode, d.VerdictReason = fString{}, fI32{}, fWord{}
d.VerdictErr, d.VerdictErrB64 = fString{}, fB64{}
d.PartitionID, d.PartitionIDB64 = fString{}, fB64{}
d.InputSpan.seen, d.InputSpan.null, d.InputSpan.v = false, false, spanDoc{}
+ d.UnsupportedSetting = fIgnore{}
d.UnknownFields.reset()
- d.UnsupportedSetting.reset()
d.Computed.reset()
d.Cols.reset()
d.Transformed.reset()
@@ -733,7 +741,7 @@ func atMergeOf(list []atMergeDoc, present bool) ([]AtMergeEntry, bool) {
out := make([]AtMergeEntry, 0, len(list))
for i := range list {
a := &list[i]
- e := AtMergeEntry{Row: a.Row.v, Reason: MergeReason(a.Reason.s)}
+ e := AtMergeEntry{PartRow: a.PartRow.v, Reason: MergeReason(a.Reason.s)}
column, present, ok := byteOf(&a.Column.fString, &a.ColumnB64)
if !ok {
return nil, false
@@ -793,6 +801,55 @@ func decodeSpanList(dec *json.Decoder, hint int) ([]Span, error) {
return list, nil
}
+// decodeRowSpans reads row_spans as decodeSpanList reads a list of spans, but
+// a null element is a row not exported (a nil entry). The entries share chunks
+// of Span rather than allocating one each.
+func decodeRowSpans(dec *json.Decoder, hint int) ([]*Span, error) {
+ t, err := dec.Token()
+ if err != nil {
+ return nil, errDeclined
+ }
+ if t == nil {
+ return nil, nil
+ }
+ if t != json.Delim('[') {
+ return nil, errDeclined
+ }
+ list := make([]*Span, 0, hint)
+ var chunk []Span
+ slot := new(spanSlot)
+ for dec.More() {
+ *slot = spanSlot{}
+ if err := dec.Decode(slot); err != nil {
+ return nil, errDeclined
+ }
+ if slot.null {
+ list = append(list, nil)
+ continue
+ }
+ if len(chunk) == 0 {
+ chunk = make([]Span, slabChunk)
+ }
+ p := &chunk[0]
+ chunk = chunk[1:]
+ *p = Span{Off: slot.v.Off.v, Len: slot.v.Len.v}
+ list = append(list, p)
+ }
+ if t, err := dec.Token(); err != nil || t != json.Delim(']') {
+ return nil, errDeclined
+ }
+ return list, nil
+}
+
+// optU64Of is reader.optU64: nil when the member is absent or null.
+func optU64Of(f *fU64) *uint64 {
+ if !f.seen || f.null {
+ return nil
+ }
+ n := f.v
+ return &n
+}
+
// spanHint is the count a document has already announced (one span per row),
// or 0 when none was or it is not credible for a document of docLen bytes.
func spanHint(n fU64, docLen int) int {
@@ -813,9 +870,9 @@ func (d *batchRowDoc) result(slabs *rowSlabs) (RowResult, bool) {
if res.Transformed, ok = transformsOf(d.Transformed.v, d.Transformed.present()); !ok {
return res, false
}
- if res.UnsupportedSettings, ok = namesOf(&d.UnsupportedSetting); !ok {
- return res, false
- }
+ res.Reason, res.RefusedAt = d.Reason.s, RefusalSite(d.RefusedAt.s)
+ res.RefusedRow = optU64Of(&d.RefusedRow)
+ res.VerdictReason = d.VerdictReason.s
if res.UnknownFields, ok = namesOf(&d.UnknownFields); !ok {
return res, false
}
@@ -963,6 +1020,11 @@ const (
kStorageTransforms
kUnsupportedSettings
kAtMerge
+ kReason
+ kRefusedAt
+ kRefusedRow
+ kExportDeclinedReason
+ kExportColumns
)
// decodeBatchFast reads a batch document; ok is false when the generic reader
@@ -988,7 +1050,12 @@ func decodeBatchFast(raw, payload []byte) (res BatchResult, ok bool) {
exportDeclinedB64 fB64
partitionCount fU64
transformed []transformDoc
- rowSpans, unconsumed []Span
+ rowSpans []*Span
+ unconsumed []Span
+ reason, refusedAt fWord
+ refusedRow fU64
+ exportDeclinedReason fWord
+ exportColumns fList[nameDoc]
engineRows [][]cellDoc
framing *framingDoc
storageTransforms fIgnore
@@ -1038,7 +1105,7 @@ func decodeBatchFast(raw, payload []byte) (res BatchResult, ok bool) {
case "row_spans":
// One span per row, so rows_read, when it came first, sizes the list.
bit = kRowSpans
- top.rowSpans, err = decodeSpanList(dec, spanHint(top.rowsRead, len(raw)))
+ top.rowSpans, err = decodeRowSpans(dec, spanHint(top.rowsRead, len(raw)))
case "unconsumed":
bit = kUnconsumed
top.unconsumed, err = decodeSpanList(dec, 0)
@@ -1052,6 +1119,16 @@ func decodeBatchFast(raw, payload []byte) (res BatchResult, ok bool) {
bit, err = kUnsupportedSettings, dec.Decode(&top.unsupportedSettings)
case "at_merge":
bit, err = kAtMerge, dec.Decode(&top.atMerge)
+ case "reason":
+ bit, err = kReason, dec.Decode(&top.reason)
+ case "refused_at":
+ bit, err = kRefusedAt, dec.Decode(&top.refusedAt)
+ case "refused_row":
+ bit, err = kRefusedRow, dec.Decode(&top.refusedRow)
+ case "export_declined_reason":
+ bit, err = kExportDeclinedReason, dec.Decode(&top.exportDeclinedReason)
+ case "export_columns":
+ bit, err = kExportColumns, dec.Decode(&top.exportColumns)
case "rows":
bit = kRows
hint := 0
@@ -1080,6 +1157,9 @@ func decodeBatchFast(raw, payload []byte) (res BatchResult, ok bool) {
if res.ErrMsg, good = bytesOf(&top.errText, &top.errB64); !good {
return res, false
}
+ res.Reason, res.RefusedAt = top.reason.s, RefusalSite(top.refusedAt.s)
+ res.RefusedRow = optU64Of(&top.refusedRow)
+ res.ExportDeclinedReason = top.exportDeclinedReason.s
res.RowsRead, res.RowsSkipped = top.rowsRead.v, top.rowsSkipped.v
if res.Transformed, good = transformsOf(top.transformed, top.transformed != nil); !good {
return res, false
@@ -1095,6 +1175,9 @@ func decodeBatchFast(raw, payload []byte) (res BatchResult, ok bool) {
res.Payload = payload
res.Spans = top.rowSpans
}
+ if res.ExportColumns, good = namesOf(&top.exportColumns); !good {
+ return res, false
+ }
res.RowsPassed, res.RowsCut = top.rowsPassed.v, top.rowsCut.v
res.Unconsumed = top.unconsumed
if res.UnsupportedSettings, good = namesOf(&top.unsupportedSettings); !good {
diff --git a/go/chtypes/decode_equivalence_test.go b/go/chtypes/decode_equivalence_test.go
index 6eeb554c..550751bf 100644
--- a/go/chtypes/decode_equivalence_test.go
+++ b/go/chtypes/decode_equivalence_test.go
@@ -67,9 +67,15 @@ var batchFixtures = map[string]string{
"byte-safe batch fields": `{"outcome":"rejected","code":27,"err_b64":"/w==","export_declined_b64":"/gA=","rows_read":"7"}`,
"storage transforms": `{"outcome":"accepted","storage_transforms":[{"row":0,"column":"a","stored":"1"}]}`,
"at_merge, every shape": atMergeBatchDoc,
- "at_merge, nulls": `{"outcome":"accepted","at_merge":[{"row":null,"reason":null,"column":null,"stored":null,"input_rows":null}],"unsupported_settings":null}`,
+ "at_merge, nulls": `{"outcome":"accepted","at_merge":[{"part_row":null,"reason":null,"column":null,"stored":null,"input_rows":null}],"unsupported_settings":null}`,
"at_merge, empty": `{"outcome":"accepted","at_merge":[],"unsupported_settings":[]}`,
"at_merge, null list": `{"outcome":"accepted","at_merge":null}`,
+ // Spec batch 2: the refusal members, the export's column list and reason, and
+ // a row_spans with a null entry for a row not exported.
+ "refusal members": `{"outcome":"unsupported","code":-2,"reason":"preview-batch-zone-mismatch","refused_at":"body_read","refused_row":3,"rows_read":0,"rows":[]}`,
+ "refusal, unlisted site": `{"outcome":"rejected","code":27,"refused_at":"a_site_nobody_listed","refused_row":"9007199254740991"}`,
+ "export, spans with nulls": `{"outcome":"accepted","rows_read":3,"rows":[],"row_spans":[{"off":0,"len":5},null,{"off":5,"len":6}],"export_columns":[{"name":"a"},{"name_b64":"/w=="}]}`,
+ "export declined with a reason": `{"outcome":"unsupported","export_declined":"x","export_declined_reason":"export-batch-not-accepted","export_columns":[]}`,
}
// rowFixtures are documents of one row, as decodeRow reads them; each also
@@ -77,8 +83,9 @@ var batchFixtures = map[string]string{
var rowFixtures = map[string]string{
"row, every field": rowDoc,
"row, minimal": `{"outcome":"accepted"}`,
- "row, empty lists": `{"outcome":"accepted","cols":[],"computed":[],"transformed":[],"unknown_fields":[],"unsupported_settings":[]}`,
- "row, null lists": `{"outcome":"accepted","cols":null,"computed":null,"transformed":null,"unknown_fields":null,"unsupported_settings":null,"input_span":null,"verdict":null,"partition_id":null}`,
+ "row, declined": `{"outcome":"unsupported","code":-2,"reason":"drawn-value","refused_at":"row","refused_row":0,"verdict":"d","verdict_code":-2,"verdict_reason":"defer-defaults-reader"}`,
+ "row, empty lists": `{"outcome":"accepted","cols":[],"computed":[],"transformed":[],"unknown_fields":[]}`,
+ "row, null lists": `{"outcome":"accepted","cols":null,"computed":null,"transformed":null,"unknown_fields":null,"input_span":null,"verdict":null,"partition_id":null}`,
"row, partition id bytes": `{"outcome":"accepted","partition_id_b64":"/w==","err_b64":"/gA=","verdict_err_b64":"/w==","verdict_code":3}`,
"row, string value": `{"outcome":"accepted","cols":[{"name":"s","src":"input","null":false,"stored":"\"a\\\"b\"","value_b64":"YSJi","type":"String","ref":"x"}]}`,
"row, unknown outcome": `{"outcome":"something_new"}`,
diff --git a/go/chtypes/decode_reference_test.go b/go/chtypes/decode_reference_test.go
index 255649b5..9416f80f 100644
--- a/go/chtypes/decode_reference_test.go
+++ b/go/chtypes/decode_reference_test.go
@@ -17,18 +17,23 @@ func decodeBatchReference(raw []byte, payload []byte) (BatchResult, error) {
r := &reader{doc: "batch"}
m := r.object(v, "$")
res := BatchResult{
- Outcome: parseBatchOutcome(r.text(m, "outcome", "$")),
- ErrCode: r.i32(m, "code", "$"),
- ErrMsg: r.bytes(m, "err", "$"),
- RowsRead: r.u64(m, "rows_read", "$"),
- RowsSkipped: r.u64(m, "rows_skipped", "$"),
- Transformed: r.transforms(m, "transformed", "$"),
- Payload: payload,
- Spans: r.spans(m, "row_spans", "$"),
- ExportDeclined: r.bytes(m, "export_declined", "$"),
- RowsPassed: r.u64(m, "rows_passed", "$"),
- RowsCut: r.u64(m, "rows_cut", "$"),
- Unconsumed: r.spans(m, "unconsumed", "$"),
+ Outcome: parseBatchOutcome(r.text(m, "outcome", "$")),
+ ErrCode: r.i32(m, "code", "$"),
+ ErrMsg: r.bytes(m, "err", "$"),
+ Reason: r.text(m, "reason", "$"),
+ RefusedAt: RefusalSite(r.text(m, "refused_at", "$")),
+ RefusedRow: r.optU64(m, "refused_row", "$"),
+ RowsRead: r.u64(m, "rows_read", "$"),
+ RowsSkipped: r.u64(m, "rows_skipped", "$"),
+ Transformed: r.transforms(m, "transformed", "$"),
+ Payload: payload,
+ Spans: r.rowSpans(m, "row_spans", "$"),
+ ExportColumns: r.nameList(m, "export_columns", "$"),
+ ExportDeclinedReason: r.text(m, "export_declined_reason", "$"),
+ ExportDeclined: r.bytes(m, "export_declined", "$"),
+ RowsPassed: r.u64(m, "rows_passed", "$"),
+ RowsCut: r.u64(m, "rows_cut", "$"),
+ Unconsumed: r.spans(m, "unconsumed", "$"),
// Added with ABI v2's at_merge (public issue #544), through the shared
// reader helpers, so the comparison covers the two new members too.
UnsupportedSettings: r.nameList(m, "unsupported_settings", "$"),
diff --git a/go/chtypes/decode_test.go b/go/chtypes/decode_test.go
index a337a4df..bd46d398 100644
--- a/go/chtypes/decode_test.go
+++ b/go/chtypes/decode_test.go
@@ -37,6 +37,24 @@ const rowDoc = `{
"partition_id": "all"
}`
+func TestDecodeRowDefaultOverride(t *testing.T) {
+ r, err := decodeRow([]byte(`{"outcome":"accepted","code":0,"cols":[
+ {"name":"a","null":false,"src":"default_override","stored":"5"},
+ {"name":"b","null":false,"src":"default_deferred","stored":""}]}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if v := r.Columns[0]; v.Source != SourceDefaultOverride || !v.IsStored || v.Text != "5" {
+ t.Errorf("default_override entry = %+v, want stored with text 5", v)
+ }
+ if r.Columns[1].IsStored {
+ t.Errorf("default_deferred read as stored")
+ }
+ if len(r.Values) != 1 || r.Values[0].Column != "a" {
+ t.Errorf("Values = %+v, want only the override cell", r.Values)
+ }
+}
+
func TestDecodeRow(t *testing.T) {
r, err := decodeRow([]byte(rowDoc))
if err != nil {
@@ -212,13 +230,13 @@ const atMergeBatchDoc = `{
"outcome": "accepted",
"unsupported_settings": [{"name": "session_timezone"}, {"name_b64": "/w=="}],
"at_merge": [
- {"row": 0, "reason": "ttl_delete"},
- {"row": 1, "reason": "ttl_column_reset", "column": "c", "stored": "0"},
- {"row": 2, "reason": "ttl_column_reset", "column": "t"},
- {"row": 3, "reason": "ttl_column_reset", "column_b64": "/wA=", "stored_b64": "/w=="},
- {"row": 4, "reason": "ttl_delete", "input_rows": [4, 7]},
- {"row": 5, "reason": "a_reason_nobody_listed"},
- {"row": 6, "reason": "ttl_column_reset", "column": "e", "stored": "", "input_rows": []}
+ {"part_row": 0, "reason": "ttl_delete"},
+ {"part_row": 1, "reason": "ttl_column_reset", "column": "c", "stored": "0"},
+ {"part_row": 2, "reason": "ttl_column_reset", "column": "t"},
+ {"part_row": 3, "reason": "ttl_column_reset", "column_b64": "/wA=", "stored_b64": "/w=="},
+ {"part_row": 4, "reason": "ttl_delete", "input_rows": [4, 7]},
+ {"part_row": 5, "reason": "a_reason_nobody_listed"},
+ {"part_row": 6, "reason": "ttl_column_reset", "column": "e", "stored": "", "input_rows": []}
]
}`
@@ -258,9 +276,9 @@ func TestDecodeBatchAtMerge(t *testing.T) {
{MergeTTLColumnReset, "e", "", []int{}, false, true, true}, // an empty stored is not an absent one
} {
e := b.AtMerge[i]
- if e.Row != i || e.Reason != tc.reason || e.Reason.Known() != tc.known || str(e.Column) != tc.column || str(e.Stored) != tc.stored {
- t.Errorf("entry %d = {Row %d Reason %q Known %v Column %q Stored %q}, want {%d %q %v %q %q}",
- i, e.Row, e.Reason, e.Reason.Known(), str(e.Column), str(e.Stored), i, tc.reason, tc.known, tc.column, tc.stored)
+ if e.PartRow != i || e.Reason != tc.reason || e.Reason.Known() != tc.known || str(e.Column) != tc.column || str(e.Stored) != tc.stored {
+ t.Errorf("entry %d = {PartRow %d Reason %q Known %v Column %q Stored %q}, want {%d %q %v %q %q}",
+ i, e.PartRow, e.Reason, e.Reason.Known(), str(e.Column), str(e.Stored), i, tc.reason, tc.known, tc.column, tc.stored)
}
if (e.Column != nil) != tc.columnPresent {
t.Errorf("entry %d: Column present = %v, want %v", i, e.Column != nil, tc.columnPresent)
@@ -279,7 +297,7 @@ func TestDecodeBatchAtMerge(t *testing.T) {
}
// An unknown member, in an entry and beside the list, is ignored (r2).
- b, err = decodeBatch([]byte(`{"outcome":"accepted","at_merge":[{"row":0,"reason":"ttl_delete","x_future":{"a":[1]},"x_future_b64":"/w=="}],"x_future_list":[1]}`), nil)
+ b, err = decodeBatch([]byte(`{"outcome":"accepted","at_merge":[{"part_row":0,"reason":"ttl_delete","x_future":{"a":[1]},"x_future_b64":"/w=="}],"x_future_list":[1]}`), nil)
if err != nil || len(b.AtMerge) != 1 || b.AtMerge[0].Reason != MergeTTLDelete || b.AtMerge[0].Column != nil {
t.Errorf("at_merge with unknown members = %+v, %v", b.AtMerge, err)
}
@@ -298,11 +316,11 @@ func TestDecodeBatchAtMerge(t *testing.T) {
// A document that breaks its schema is an InternalError naming the key.
for _, bad := range []string{
- `{"at_merge":[{"row":0,"reason":"ttl_column_reset","column":"c","column_b64":"Yw=="}]}`,
- `{"at_merge":[{"row":0,"reason":"ttl_column_reset","column":"c","stored_b64":"!!"}]}`,
- `{"at_merge":[{"row":0,"reason":"ttl_delete","input_rows":["x"]}]}`,
- `{"at_merge":[{"row":0,"reason":"ttl_delete","input_rows":[-1]}]}`,
- `{"at_merge":[{"row":0,"reason":7}]}`,
+ `{"at_merge":[{"part_row":0,"reason":"ttl_column_reset","column":"c","column_b64":"Yw=="}]}`,
+ `{"at_merge":[{"part_row":0,"reason":"ttl_column_reset","column":"c","stored_b64":"!!"}]}`,
+ `{"at_merge":[{"part_row":0,"reason":"ttl_delete","input_rows":["x"]}]}`,
+ `{"at_merge":[{"part_row":0,"reason":"ttl_delete","input_rows":[-1]}]}`,
+ `{"at_merge":[{"part_row":0,"reason":7}]}`,
`{"at_merge":{"row":0}}`,
`{"unsupported_settings":["session_timezone"]}`,
} {
@@ -494,3 +512,104 @@ func TestDeclinedExportPayloadIsAbsent(t *testing.T) {
})
}
}
+
+// Spec batch 2: a refusal or decline names its code (CodeDeclined is -2), its
+// catalog id and where it sits; a batch fast path and the generic reader agree.
+func TestDecodeBatchRefusalMembers(t *testing.T) {
+ doc := `{"outcome":"unsupported","code":-2,"reason":"preview-batch-zone-mismatch","refused_at":"body_read","refused_row":3,"rows_read":0,"rows":[]}`
+ for name, dec := range map[string]func([]byte, []byte) (BatchResult, error){"production": decodeBatch, "generic": decodeBatchGeneric} {
+ b, err := dec([]byte(doc), nil)
+ if err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ if b.ErrCode != CodeDeclined || b.Reason != "preview-batch-zone-mismatch" || b.RefusedAt != RefusedAtBodyRead || !b.RefusedAt.Known() ||
+ b.RefusedRow == nil || *b.RefusedRow != 3 {
+ t.Errorf("%s: batch = %+v", name, b)
+ }
+ b, err = dec([]byte(`{"outcome":"accepted","code":0,"rows_read":1}`), nil)
+ if err != nil || b.Reason != "" || b.RefusedAt != "" || b.RefusedRow != nil {
+ t.Errorf("%s: an accepted batch carries no refusal members: %+v, %v", name, b, err)
+ }
+ // An unlisted site is its unknown(n): kept, never a failure (r3).
+ b, err = dec([]byte(`{"outcome":"rejected","refused_at":"a_site_nobody_listed"}`), nil)
+ if err != nil || b.RefusedAt != "a_site_nobody_listed" || b.RefusedAt.Known() {
+ t.Errorf("%s: unlisted site = %+v, %v", name, b.RefusedAt, err)
+ }
+ }
+}
+
+func TestDecodeBatchExportMembers(t *testing.T) {
+ doc := `{"outcome":"accepted","rows_read":3,"row_spans":[{"off":0,"len":5},null,{"off":5,"len":6}],"export_columns":[{"name":"a"},{"name_b64":"/w=="}]}`
+ for name, dec := range map[string]func([]byte, []byte) (BatchResult, error){"production": decodeBatch, "generic": decodeBatchGeneric} {
+ b, err := dec([]byte(doc), []byte("0123456789a"))
+ if err != nil {
+ t.Fatalf("%s: %v", name, err)
+ }
+ if len(b.Spans) != 3 || b.Spans[0] == nil || b.Spans[1] != nil || b.Spans[2] == nil || b.Spans[2].Off != 5 || b.Spans[2].Len != 6 {
+ t.Errorf("%s: spans = %v: a row not exported is a nil entry, aligned with Rows", name, b.Spans)
+ }
+ if len(b.ExportColumns) != 2 || b.ExportColumns[0] != "a" || b.ExportColumns[1] != "\xff" {
+ t.Errorf("%s: export_columns = %q", name, b.ExportColumns)
+ }
+ d, err := dec([]byte(`{"outcome":"unsupported","export_declined":"nope","export_declined_reason":"export-batch-not-accepted","row_spans":[null]}`), []byte("x"))
+ if err != nil || d.ExportDeclined != "nope" || d.ExportDeclinedReason != "export-batch-not-accepted" || d.Payload != nil || d.Spans != nil || d.ExportColumns != nil {
+ t.Errorf("%s: declined export = %+v, %v", name, d, err)
+ }
+ }
+}
+
+func TestDecodeRowRefusalMembers(t *testing.T) {
+ r, err := decodeRow([]byte(`{"outcome":"unsupported","code":-2,"reason":"drawn-value","refused_at":"row","refused_row":0,"verdict":"d","verdict_code":-2,"verdict_reason":"defer-defaults-reader"}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if r.ErrCode != CodeDeclined || r.Reason != "drawn-value" || r.RefusedAt != RefusedAtRow || r.RefusedRow == nil || *r.RefusedRow != 0 ||
+ r.Verdict == nil || *r.Verdict != VerdictDecline || r.VerdictCode != CodeDeclined || r.VerdictReason != "defer-defaults-reader" {
+ t.Errorf("row = %+v", r)
+ }
+}
+
+func TestDecodeFilterResultRefusalMembers(t *testing.T) {
+ f, err := decodeFilterResult([]byte(`{"outcome":"unsupported","code":-2,"reason":"filter-zone","refused_at":"call","rows_read":0,"verdicts":"","errors":[{"row":0,"code":-2,"reason":"drawn-value","err":"x"}],"unsupported_settings":[]}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if f.ErrCode != CodeDeclined || f.Reason != "filter-zone" || f.RefusedAt != RefusedAtCall || f.RefusedRow != nil ||
+ len(f.Errors) != 1 || f.Errors[0].Code != CodeDeclined || f.Errors[0].Reason != "drawn-value" {
+ t.Errorf("filter result = %+v", f)
+ }
+}
+
+func TestDecodeSchemaDescriptionBatch2(t *testing.T) {
+ d, err := decodeSchemaDescription([]byte(`{"columns":[
+ {"name":"a","type":"Int32","default_kind":"","default_support":"deferred","default_support_reason":"r-x","reads":[{"name":"b"},{"name_b64":"/w=="}]},
+ {"name":"b","type":"Int32","default_kind":"DEFAULT","default_support":"a_support_nobody_listed"}],
+ "filter_declined_settings":[],"posture":"load",
+ "server":{"timezone":"UTC","settings":{"s":"v","t":{"b64":"/w=="}}}}`))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if d.Posture != PostureLoad || len(d.Columns) != 2 {
+ t.Fatalf("description = %+v", d)
+ }
+ if c := d.Columns[0]; c.DefaultSupport != DefaultSupportDeferred || c.DefaultSupportReason != "r-x" || len(c.Reads) != 2 || c.Reads[1] != "\xff" {
+ t.Errorf("column 0 = %+v", c)
+ }
+ if c := d.Columns[1]; c.DefaultSupport != "a_support_nobody_listed" || c.DefaultSupport.Known() || c.Reads != nil {
+ t.Errorf("column 1 = %+v: an unlisted support is its unknown(n), an absent reads is nil", c)
+ }
+ if d.Server == nil || d.Server.Settings["s"] != "v" || d.Server.Settings["t"] != "\xff" {
+ t.Errorf("server = %+v: a settings value in the b64 form is its bytes", d.Server)
+ }
+ // Absent members: a create-posture schema, and a build that reports no default_support.
+ d, err = decodeSchemaDescription([]byte(`{"columns":[{"name":"a","type":"Int32","default_kind":""}],"filter_declined_settings":[]}`))
+ if err != nil || d.Posture != "" || d.Columns[0].DefaultSupport != "" || d.Columns[0].Reads != nil {
+ t.Errorf("absent members = %+v, %v", d, err)
+ }
+ // A b64 settings value that is not the one-member object form is an InternalError.
+ _, err = decodeSchemaDescription([]byte(`{"columns":[],"server":{"settings":{"t":{"b64":"/w==","x":1}}}}`))
+ var ie *InternalError
+ if !errors.As(err, &ie) {
+ t.Errorf("a malformed b64 value = %v, want an *InternalError", err)
+ }
+}
diff --git a/go/chtypes/doc.go b/go/chtypes/doc.go
index 6262d972..493e041b 100644
--- a/go/chtypes/doc.go
+++ b/go/chtypes/doc.go
@@ -56,7 +56,7 @@
// answer is a *UnsupportedError, a PEER of the first that errors.As never
// confuses with it. *UsageError is misuse (a closed object, a refused
// version spelling, a conflicting setup) and *InternalError a library bug.
-// AsCallError reads the five fields every one of the four carries. A
+// AsCallError reads the six fields every one of the four carries. A
// loader or fetch failure is an *ArtifactError whose Code is the shared
// vocabulary, with a sentinel per code for errors.Is.
//
diff --git a/go/chtypes/errors.go b/go/chtypes/errors.go
index e6065c8a..2c55f287 100644
--- a/go/chtypes/errors.go
+++ b/go/chtypes/errors.go
@@ -1,7 +1,7 @@
package chtypes
// errors.go — the error classes (bindings-v1.md section 4). Every call error
-// carries the five fields of chs_error verbatim; the class of a status is
+// carries the six fields of chs_error verbatim; the class of a status is
// spec/abi-v2/sdk.json's table, generated into the abi2 layer. A status the
// description does not list is its unknown(n) and an InternalError (rule r3).
@@ -15,7 +15,7 @@ import (
"github.com/wave-rf/chtypes/go/v2/internal/ocifetch"
)
-// CallError is the five fields every call error carries, verbatim and with
+// CallError is the six fields every call error carries, verbatim and with
// nothing synthesized. It is embedded in *SchemaError, *UnsupportedError,
// *UsageError and *InternalError, so the fields read directly
// (se.ChCode); AsCallError reads them from any of the four.
@@ -33,6 +33,9 @@ type CallError struct {
Message string
// Column is the column concerned, empty if none. It is bytes.
Column string
+ // Reason is the error's catalog id, read through chs_error_reason: ASCII,
+ // empty if none.
+ Reason string
}
// display is the one lossy display form: invalid UTF-8 replaced, for printing
@@ -125,7 +128,7 @@ func callError(c *abi2.CallError) error {
if c == nil {
return nil
}
- ce := CallError{ChCode: c.ChCode, ChName: c.ChName, Message: c.Message, Column: c.Column}
+ ce := CallError{ChCode: c.ChCode, ChName: c.ChName, Message: c.Message, Column: c.Column, Reason: c.Reason}
status, known := statusOfName(c.Status)
if !known {
raw := int64(StatusInternal)
diff --git a/go/chtypes/errors_test.go b/go/chtypes/errors_test.go
index 38f86227..16945224 100644
--- a/go/chtypes/errors_test.go
+++ b/go/chtypes/errors_test.go
@@ -22,13 +22,13 @@ func TestCallErrorClasses(t *testing.T) {
{"CHS_INTERNAL", "*chtypes.InternalError", StatusInternal},
}
for _, c := range cases {
- err := callError(&abi2.CallError{Status: c.status, ChCode: 53, ChName: "TYPE_MISMATCH", Message: "m\xff", Column: "c\x00"})
+ err := callError(&abi2.CallError{Status: c.status, ChCode: 53, ChName: "TYPE_MISMATCH", Message: "m\xff", Column: "c\x00", Reason: "option-not-implemented"})
if got := fmt.Sprintf("%T", err); got != c.want {
t.Errorf("%s -> %s, want %s", c.status, got, c.want)
}
ce, ok := AsCallError(err)
- if !ok || ce.Status != c.code || ce.ChCode != 53 || ce.ChName != "TYPE_MISMATCH" || ce.Message != "m\xff" || ce.Column != "c\x00" {
- t.Errorf("%s: AsCallError = %+v, %v: the five fields are verbatim", c.status, ce, ok)
+ if !ok || ce.Status != c.code || ce.ChCode != 53 || ce.ChName != "TYPE_MISMATCH" || ce.Message != "m\xff" || ce.Column != "c\x00" || ce.Reason != "option-not-implemented" {
+ t.Errorf("%s: AsCallError = %+v, %v: the six fields are verbatim", c.status, ce, ok)
}
// Wrapped, still reachable.
if _, ok := AsCallError(fmt.Errorf("wrapped: %w", err)); !ok {
diff --git a/go/chtypes/goldens_v1_runner_test.go b/go/chtypes/goldens_v1_runner_test.go
index 766a76d2..43f4eb18 100644
--- a/go/chtypes/goldens_v1_runner_test.go
+++ b/go/chtypes/goldens_v1_runner_test.go
@@ -428,7 +428,7 @@ func runGoldensCase(t *testing.T, lib *Library, c goldensCase, rec map[string]an
}
doc = tbl.Take(buf)
case "preview_row":
- buf, ce := tbl.PreviewRow(sch, c.Body.Format, c.Body.Body, c.Body.Settings, c.Body.Columns)
+ buf, ce := tbl.PreviewRow(sch, c.Body.Format, c.Body.Body, c.Body.Settings, c.Body.Columns, nil)
if ce != nil {
stopAt(rec, "call", ce)
return
@@ -442,7 +442,7 @@ func runGoldensCase(t *testing.T, lib *Library, c goldensCase, rec map[string]an
if c.Body.DocFlags != nil {
df = *c.Body.DocFlags
}
- dbuf, ebuf, ce := tbl.PreviewBatch(sch, c.Body.Format, c.Body.Body, c.Body.Settings, c.Body.Columns, nil, ef, uint32(df))
+ dbuf, ebuf, ce := tbl.PreviewBatch(sch, c.Body.Format, c.Body.Body, c.Body.Settings, c.Body.Columns, nil, ef, uint32(df), nil)
if ce != nil {
stopAt(rec, "call", ce)
return
@@ -455,13 +455,13 @@ func runGoldensCase(t *testing.T, lib *Library, c goldensCase, rec map[string]an
}
}
case "filter_eval_body":
- f, ce := tbl.FilterCreate(sch, c.Filter.Expr, c.Filter.QueryParams, c.Filter.Settings)
+ f, ce := tbl.FilterCreate(sch, c.Filter.Expr, c.Filter.QueryParams, c.Filter.Settings, nil)
if ce != nil {
stopAt(rec, "filter_create", ce)
return
}
defer f.Close()
- buf, ce := tbl.FilterEvalBody(f, c.Body.Format, c.Body.Body, c.Body.Settings)
+ buf, ce := tbl.FilterEvalBody(f, c.Body.Format, c.Body.Body, c.Body.Settings, c.Body.Columns, nil)
if ce != nil {
stopAt(rec, "call", ce)
return
diff --git a/go/chtypes/library.go b/go/chtypes/library.go
index 21bbae1d..ff78a32a 100644
--- a/go/chtypes/library.go
+++ b/go/chtypes/library.go
@@ -225,8 +225,13 @@ func (l *Library) CompileTable(createTable string, opts ...CompileOption) (*Sche
defer c.server.g.leave()
server = c.server.h
}
- // No schema option is defined yet: length 0, which the library reads as {}.
- h, cerr := l.tbl.SchemaCreate(server, []byte(createTable), settings, nil)
+ // The options document (posture, and any member passed through), or length
+ // 0, which the library reads as {}.
+ options, err := c.schemaOpts.json("schema options")
+ if err != nil {
+ return nil, err
+ }
+ h, cerr := l.tbl.SchemaCreate(server, []byte(createTable), settings, options)
if cerr != nil {
return nil, callError(cerr)
}
diff --git a/go/chtypes/options.go b/go/chtypes/options.go
index 7c4ddb91..ef3564e0 100644
--- a/go/chtypes/options.go
+++ b/go/chtypes/options.go
@@ -1,5 +1,10 @@
package chtypes
+import (
+ "encoding/base64"
+ "unicode/utf8"
+)
+
// options.go — the call options (bindings-v1.md section 2, "The call options").
// One entry point per operation, with its optional inputs as Go functional
// options. Each operation takes its own option interface, and an option
@@ -16,6 +21,8 @@ type callConfig struct {
docFlags DocFlags
params map[string]string
server *Server
+ // The three options documents (rule r1), built member by member.
+ schemaOpts, insertOpts, filterOpts optionDoc
}
func newCallConfig() *callConfig {
@@ -51,12 +58,23 @@ type SettingsOption interface {
EvalOption
}
-// ColumnsOption is an option that applies to the three calls that parse an
-// INSERT body.
+// ColumnsOption is an option that applies to the four calls that model an
+// INSERT of a body: Row, Rows, ParseBlock and Filter.Rows.
type ColumnsOption interface {
RowOption
RowsOption
BlockOption
+ EvalOption
+}
+
+// InsertOption is an option of the insert_options document: it applies to
+// the four calls that model an INSERT of a body, Row, Rows, ParseBlock and
+// Filter.Rows, and means the same at each.
+type InsertOption interface {
+ RowOption
+ RowsOption
+ BlockOption
+ EvalOption
}
// The options are small named types, one per option, rather than closures: a
@@ -90,12 +108,11 @@ func (o timezoneOpt) applyEval(c *callConfig) { o.apply(c) }
type columnsOpt []string
-func (o columnsOpt) apply(c *callConfig) { c.columns, c.haveColumns = o, true }
-func (o columnsOpt) applyRow(c *callConfig) { o.apply(c) }
-func (o columnsOpt) applyRows(c *callConfig) { o.apply(c) }
-func (o columnsOpt) applyBlock(c *callConfig) {
- o.apply(c)
-}
+func (o columnsOpt) apply(c *callConfig) { c.columns, c.haveColumns = o, true }
+func (o columnsOpt) applyRow(c *callConfig) { o.apply(c) }
+func (o columnsOpt) applyRows(c *callConfig) { o.apply(c) }
+func (o columnsOpt) applyBlock(c *callConfig) { o.apply(c) }
+func (o columnsOpt) applyEval(c *callConfig) { o.apply(c) }
type rowFilterOpt struct{ f *Filter }
@@ -118,7 +135,8 @@ type serverOpt struct{ s *Server }
func (o serverOpt) applyCompile(c *callConfig) { c.server = o.s }
// WithSettings sets the call's settings: a map of strings to strings,
-// serialized as a JSON object and passed verbatim. No value is ever rewritten.
+// serialized as a JSON object and passed verbatim. No value is ever rewritten:
+// a value that is not valid UTF-8 is sent as {"b64": ...}, the object form.
// On a compile it is the profile; on a filter compile the filter's own profile;
// on the calls that parse a body, the body's parse settings.
func WithSettings(settings map[string]string) SettingsOption {
@@ -133,7 +151,8 @@ func WithSessionTimezone(name string) SettingsOption {
return timezoneOpt(name)
}
-// WithColumns sets the INSERT column list.
+// WithColumns sets the INSERT column list. It applies to every call that models
+// an INSERT of a body, Filter.Rows included.
func WithColumns(names []string) ColumnsOption {
return columnsOpt(names)
}
@@ -233,3 +252,177 @@ func evalConfig(opts []EvalOption) *callConfig {
}
return c
}
+
+// ---- the options documents (rule r1). Each member is its own option, and a
+// key the binding has no name for goes through WithInsertOption, WithFilterOption
+// or WithSchemaOption as raw JSON. The binding builds the document from the
+// members the caller set, sends nothing when none was set, and never validates a
+// member: the library refuses a key it does not know, or a value that is not one
+// of the member's, with CHS_INVALID_ARGUMENT, naming it.
+
+// VerdictBasis is insert_options' verdict_basis: what row a filter verdict answers
+// for.
+type VerdictBasis string
+
+const (
+ // VerdictBasisInput is the default: the verdict answers for the input row.
+ VerdictBasisInput VerdictBasis = "input"
+ // VerdictBasisExport answers for the row the export would store. A build
+ // that does not implement it declines it.
+ VerdictBasisExport VerdictBasis = "export"
+)
+
+// DeferDefaults is insert_options' defer_defaults.
+type DeferDefaults string
+
+const (
+ // DeferDefaultsNone is the default: no DEFAULT is deferred.
+ DeferDefaultsNone DeferDefaults = "none"
+ // DeferDefaultsServer defers to the server each DEFAULT the build would
+ // otherwise decline. A build that does not implement it declines it.
+ DeferDefaultsServer DeferDefaults = "server"
+)
+
+// ParamEncoding is filter_options' param_encoding: how a query parameter's value
+// is read.
+type ParamEncoding string
+
+const (
+ // ParamEncodingEscaped is the default: ClickHouse's escaped-text reader.
+ ParamEncodingEscaped ParamEncoding = "escaped"
+ // ParamEncodingRaw takes the bytes as the value. A build that does not
+ // implement it declines it.
+ ParamEncodingRaw ParamEncoding = "raw"
+)
+
+// ColumnDefault is one entry of insert_options' column_defaults: the value the
+// call uses in a cell the server would fill from the column's DEFAULT. Both
+// fields are byte strings: a Name or Value that is not valid UTF-8 is sent in
+// its _b64 form.
+type ColumnDefault struct {
+ Name string
+ Value string
+}
+
+type insertableColumnsOpt []string
+
+func (o insertableColumnsOpt) apply(c *callConfig) {
+ c.insertOpts.set("insertable_columns", nameObjects(o))
+}
+func (o insertableColumnsOpt) applyRow(c *callConfig) { o.apply(c) }
+func (o insertableColumnsOpt) applyRows(c *callConfig) { o.apply(c) }
+func (o insertableColumnsOpt) applyBlock(c *callConfig) { o.apply(c) }
+func (o insertableColumnsOpt) applyEval(c *callConfig) { o.apply(c) }
+
+type columnDefaultsOpt []ColumnDefault
+
+func (o columnDefaultsOpt) apply(c *callConfig) {
+ list := make([]map[string]string, 0, len(o))
+ for _, d := range o {
+ e := nameObjects([]string{d.Name})[0]
+ if utf8.ValidString(d.Value) {
+ e["value"] = d.Value
+ } else {
+ e["value_b64"] = base64.StdEncoding.EncodeToString([]byte(d.Value))
+ }
+ list = append(list, e)
+ }
+ c.insertOpts.set("column_defaults", list)
+}
+func (o columnDefaultsOpt) applyRow(c *callConfig) { o.apply(c) }
+func (o columnDefaultsOpt) applyRows(c *callConfig) { o.apply(c) }
+func (o columnDefaultsOpt) applyBlock(c *callConfig) { o.apply(c) }
+func (o columnDefaultsOpt) applyEval(c *callConfig) { o.apply(c) }
+
+// insertMember is any other member of insert_options: a key and its value.
+type insertMember struct {
+ key string
+ v any
+}
+
+func (o insertMember) apply(c *callConfig) {
+ if o.key != "" { // the zero insertMember is an option that sets nothing
+ c.insertOpts.set(o.key, o.v)
+ }
+}
+func (o insertMember) applyRow(c *callConfig) { o.apply(c) }
+func (o insertMember) applyRows(c *callConfig) { o.apply(c) }
+func (o insertMember) applyBlock(c *callConfig) { o.apply(c) }
+func (o insertMember) applyEval(c *callConfig) { o.apply(c) }
+
+type filterMember struct {
+ key string
+ v any
+}
+
+func (o filterMember) applyFilter(c *callConfig) { c.filterOpts.set(o.key, o.v) }
+
+type schemaMember struct {
+ key string
+ v any
+}
+
+func (o schemaMember) applyCompile(c *callConfig) { c.schemaOpts.set(o.key, o.v) }
+
+// WithInsertableColumns sets insert_options' insertable_columns: the columns an
+// INSERT may supply (the server's column-level INSERT grant), checked on the
+// INSERT's header before any byte of the body is read. nil leaves the member out
+// (every column); a non-nil empty slice is sent as [] (no column).
+func WithInsertableColumns(names []string) InsertOption {
+ if names == nil {
+ return insertMember{}
+ }
+ return insertableColumnsOpt(names)
+}
+
+// WithColumnDefaults sets insert_options' column_defaults: per-call DEFAULT
+// overrides. nil leaves the member out; a non-nil empty slice is sent as [].
+func WithColumnDefaults(defaults []ColumnDefault) InsertOption {
+ if defaults == nil {
+ return insertMember{}
+ }
+ return columnDefaultsOpt(defaults)
+}
+
+// WithVerdictBasis sets insert_options' verdict_basis.
+func WithVerdictBasis(b VerdictBasis) InsertOption {
+ return insertMember{"verdict_basis", string(b)}
+}
+
+// WithDeferDefaults sets insert_options' defer_defaults.
+func WithDeferDefaults(d DeferDefaults) InsertOption {
+ return insertMember{"defer_defaults", string(d)}
+}
+
+// WithInsertOption sets a member of insert_options this binding has no name
+// for, with its value as JSON text, passed through as given. The library
+// refuses a key it does not know (CHS_INVALID_ARGUMENT, naming it); a value that
+// is not valid JSON is a *UsageError raised before any call.
+func WithInsertOption(key string, valueJSON []byte) InsertOption {
+ return insertMember{key, rawMember(valueJSON)}
+}
+
+// WithParamEncoding sets filter_options' param_encoding.
+func WithParamEncoding(e ParamEncoding) FilterOption {
+ return filterMember{"param_encoding", string(e)}
+}
+
+// WithFilterOption sets a member of filter_options this binding has no name
+// for, as WithInsertOption does for insert_options.
+func WithFilterOption(key string, valueJSON []byte) FilterOption {
+ return filterMember{key, rawMember(valueJSON)}
+}
+
+// WithPosture sets schema_options' posture: how the statement is compiled,
+// PostureCreate (the default) as a server's CREATE TABLE, or PostureLoad as a
+// server loads the stored definition of a table it holds. The caller's
+// obligation under PostureLoad is spec/abi-v2/docs.md's (input:schema_options).
+func WithPosture(p Posture) CompileOption {
+ return schemaMember{"posture", string(p)}
+}
+
+// WithSchemaOption sets a member of schema_options this binding has no name
+// for, as WithInsertOption does for insert_options.
+func WithSchemaOption(key string, valueJSON []byte) CompileOption {
+ return schemaMember{key, rawMember(valueJSON)}
+}
diff --git a/go/chtypes/reader_rules_test.go b/go/chtypes/reader_rules_test.go
index f2e38ae7..d1c99a10 100644
--- a/go/chtypes/reader_rules_test.go
+++ b/go/chtypes/reader_rules_test.go
@@ -57,7 +57,7 @@ func TestR2UnknownMembersAreIgnored(t *testing.T) {
if r, err := schema.Row(JSONEachRow, []byte(`{"x":1}`)); err != nil {
t.Errorf("row with unknown members: %v", err)
} else if r.Outcome != Accepted || len(r.Columns) != 1 || r.Columns[0].Text != "abc" || r.InputSpan == nil || r.InputSpan.Len != 3 ||
- len(r.Computed) != 1 || len(r.Transformed) != 1 || len(r.UnknownFields) != 1 || len(r.UnsupportedSettings) != 1 {
+ len(r.Computed) != 1 || len(r.Transformed) != 1 || len(r.UnknownFields) != 1 {
t.Errorf("row = %+v", r)
} else if tr := r.Transformed[0]; tr.Column != "s" || tr.Input != "1" || tr.Stored != "2" || tr.Reason != ReasonValueChanged ||
!tr.Lossy || tr.Row != 0 {
diff --git a/go/chtypes/results.go b/go/chtypes/results.go
index d722d18d..5520d5c2 100644
--- a/go/chtypes/results.go
+++ b/go/chtypes/results.go
@@ -69,8 +69,9 @@ type EngineCell struct {
// the batch's at_merge list. EngineRows stays exactly what the INSERT writer
// produces, so the two never contradict each other.
type AtMergeEntry struct {
- // Row indexes EngineRows: the part's row, not the input body's.
- Row int
+ // PartRow indexes EngineRows: the part's row, not the input body's (the
+ // document's part_row).
+ PartRow int
// Reason is a merge_reason value. One the description does not list is its
// unknown(n), kept as it came (rule r3).
Reason MergeReason
@@ -91,20 +92,35 @@ type AtMergeEntry struct {
type RowResult struct {
// Outcome is final: the library applies every promotion.
Outcome Outcome
+ // ErrCode is the document's code: ClickHouse's own error code, 0 only on a
+ // success, and CodeDeclined (-2) where this build declined the row.
ErrCode int32
ErrMsg string
+ // Reason is the decline's catalog id (spec/abi-v2/declines/declines.json),
+ // from reason; "" when the document carries none, which it does exactly where
+ // ErrCode is CodeDeclined. Key on it, never on ErrMsg.
+ Reason string
+ // RefusedAt is where a single-row result's refusal sits, from refused_at: ""
+ // for an accepted or skipped row. The rows of a batch carry neither this nor
+ // RefusedRow: the batch's own say where.
+ RefusedAt RefusalSite
+ // RefusedRow is the row index of a RefusedAtRow refusal, from refused_row; nil
+ // when the document carries none.
+ RefusedRow *uint64
// Columns is every entry of cols, in document order; Values is the subset
// whose IsStored is true.
- Columns []Value
- Values []Value
- Transformed []Transform
- UnknownFields []string
- UnsupportedSettings []string
- Computed []Computed
+ Columns []Value
+ Values []Value
+ Transformed []Transform
+ UnknownFields []string
+ Computed []Computed
// Verdict is present only with an attached filter.
Verdict *Verdict
VerdictCode int32
- VerdictErr string
+ // VerdictReason is the catalog id of a declined verdict (VerdictCode is
+ // CodeDeclined), from verdict_reason; "" when the document carries none.
+ VerdictReason string
+ VerdictErr string
// PartitionID is nil when the document carries none; it is bytes.
PartitionID *string
// InputSpan is the bytes the reader consumed for this record.
@@ -130,9 +146,18 @@ type Framing struct {
// BatchResult is a body's verdict, counts and per-row documents.
type BatchResult struct {
- Outcome Outcome
- ErrCode int32
- ErrMsg string
+ Outcome Outcome
+ // ErrCode is the document's code, as RowResult.ErrCode.
+ ErrCode int32
+ ErrMsg string
+ // Reason is the catalog id of a declined batch (ErrCode is CodeDeclined),
+ // from reason; "" when the document carries none.
+ Reason string
+ // RefusedAt is where a refusal or decline sits, from refused_at: "" for an
+ // accepted batch. A RefusedAtRow one names its row in RefusedRow (below
+ // RowsRead); nil otherwise.
+ RefusedAt RefusalSite
+ RefusedRow *uint64
Rows []RowResult
RowsRead uint64
RowsSkipped uint64
@@ -156,13 +181,23 @@ type BatchResult struct {
// 20261011.105019 returned a zero-length buffer), non-nil and empty for an
// accepted batch with zero rows.
Payload []byte
- // Spans is each exported row's place in Payload; nil when absent, and
- // always nil when the export was declined.
- Spans []Span
- ExportDeclined string
- RowsPassed uint64
- RowsCut uint64
- PartitionCount *uint64
+ // Spans is each row's place in Payload, from row_spans: one entry per
+ // Rows entry, index-aligned, nil for a row not exported (a skipped row, or a
+ // row a filter cut). The slice itself is nil when absent, and always nil when
+ // the export was declined.
+ Spans []*Span
+ // ExportColumns is the export's one column list, in the order every line
+ // carries its columns, from export_columns; nil when the document carries
+ // none (no export ran).
+ ExportColumns []string
+ // ExportDeclined names the cause of a declined export, and ExportDeclinedReason
+ // its catalog id, from export_declined_reason; both "" when the export ran or
+ // none was asked for.
+ ExportDeclined string
+ ExportDeclinedReason string
+ RowsPassed uint64
+ RowsCut uint64
+ PartitionCount *uint64
// Unconsumed is the byte ranges the reader's error recovery skipped. It
// does not account for every record: a skipped row's InputSpan can cover
// more than one input record, so verdicts can be fewer than records while
@@ -177,14 +212,23 @@ type BatchResult struct {
type FilterRowError struct {
Row int
Code int32
- Msg string
+ // Reason is the catalog id where Code is CodeDeclined, from reason; "" when
+ // the entry carries none.
+ Reason string
+ Msg string
}
// FilterResult is a filter evaluation.
type FilterResult struct {
- Outcome FilterOutcome
- ErrCode int32
- ErrMsg string
+ Outcome FilterOutcome
+ // ErrCode is the document's code, as RowResult.ErrCode.
+ ErrCode int32
+ ErrMsg string
+ // Reason, RefusedAt and RefusedRow are the batch's (see BatchResult): where a
+ // refused or declined evaluation sits, and its catalog id.
+ Reason string
+ RefusedAt RefusalSite
+ RefusedRow *uint64
RowsRead uint64
UnsupportedSettings []string
// Verdicts is one verdict per row. A verdict counts only when Outcome is
@@ -203,12 +247,23 @@ type Column struct {
Type string
DefaultKind DefaultKind
DefaultExpr string
+ // DefaultSupport is what this build does with the column's DEFAULT, from
+ // default_support (reserved: "" when the build reports none), and
+ // DefaultSupportReason the catalog id behind a DefaultSupportDeclined one.
+ DefaultSupport DefaultSupport
+ DefaultSupportReason string
+ // Reads is the columns the column's expression reads, from reads (reserved:
+ // nil when the build reports none).
+ Reads []string
}
// SchemaDescription is a schema's columns, in declared order, the settings
// its filters decline, and the server it was compiled on.
type SchemaDescription struct {
Columns []Column
+ // Posture is how the schema's statement was compiled, from posture, which the
+ // document carries only for PostureLoad: "" is PostureCreate.
+ Posture Posture
// FilterDeclinedSettings is the settings the schema compiles under that
// this build's filters do not honor in a WHERE, over every layer known at
// schema compile (the defaults Setup set, the server profile and the
@@ -234,8 +289,9 @@ type SchemaServer struct {
// Timezone is the zone the schema's types bind: the profile's, or else the
// image zone (Setup's), exactly as chs_initialize spelled it.
Timezone string
- // Settings is the server's settings as the profile gave them; the library
- // reports {} (an empty map) when it gave none.
+ // Settings is the server's settings as the profile gave them (a value that
+ // was not UTF-8 comes back as its bytes); the library reports {} (an empty
+ // map) when it gave none.
Settings map[string]string
// Macros is the server's macro set: nil exactly when the profile carried
// no macros (unknown), and non-nil, even empty, when it did (the complete
diff --git a/go/chtypes/rows_pool_test.go b/go/chtypes/rows_pool_test.go
index 17ccddc5..7049d668 100644
--- a/go/chtypes/rows_pool_test.go
+++ b/go/chtypes/rows_pool_test.go
@@ -112,7 +112,7 @@ func TestStubTakeIntoMatchesTake(t *testing.T) {
}
defer schema.Close()
call := func() *abi2.Buf {
- doc, exp, cerr := lib.tbl.PreviewBatch(schema.h, int32(JSONEachRow), []byte(`{"x":1}`), nil, nil, nil, int32(ExportNone), 0)
+ doc, exp, cerr := lib.tbl.PreviewBatch(schema.h, int32(JSONEachRow), []byte(`{"x":1}`), nil, nil, nil, int32(ExportNone), 0, nil)
if cerr != nil {
t.Fatal(cerr)
}
diff --git a/go/chtypes/schema.go b/go/chtypes/schema.go
index 50fef02e..6ddd40b3 100644
--- a/go/chtypes/schema.go
+++ b/go/chtypes/schema.go
@@ -95,7 +95,7 @@ func (s *Schema) Describe() (SchemaDescription, error) {
// Row previews one row (chs_preview_row).
func (s *Schema) Row(format Format, body []byte, opts ...RowOption) (RowResult, error) {
c := rowConfig(opts)
- settings, columns, err := callInputs(c)
+ settings, columns, options, err := callInputs(c)
if err != nil {
return RowResult{}, err
}
@@ -104,7 +104,7 @@ func (s *Schema) Row(format Format, body []byte, opts ...RowOption) (RowResult,
}
defer s.g.leave()
raw, err := s.lib.document(func() (*abi2.Buf, *abi2.CallError) {
- return s.lib.tbl.PreviewRow(s.h, int32(format), body, settings, columns)
+ return s.lib.tbl.PreviewRow(s.h, int32(format), body, settings, columns, options)
})
if err != nil {
return RowResult{}, err
@@ -116,7 +116,7 @@ func (s *Schema) Row(format Format, body []byte, opts ...RowOption) (RowResult,
// attached and an export of the accepted rows.
func (s *Schema) Rows(format Format, body []byte, opts ...RowsOption) (BatchResult, error) {
c := rowsConfig(opts)
- settings, columns, err := callInputs(c)
+ settings, columns, options, err := callInputs(c)
if err != nil {
return BatchResult{}, err
}
@@ -132,7 +132,7 @@ func (s *Schema) Rows(format Format, body []byte, opts ...RowsOption) (BatchResu
defer c.filter.g.leave()
fh = c.filter.h
}
- docBuf, exportBuf, cerr := s.lib.tbl.PreviewBatch(s.h, int32(format), body, settings, columns, fh, int32(c.export), uint32(c.docFlags))
+ docBuf, exportBuf, cerr := s.lib.tbl.PreviewBatch(s.h, int32(format), body, settings, columns, fh, int32(c.export), uint32(c.docFlags), options)
if cerr != nil {
return BatchResult{}, callError(cerr)
}
@@ -175,11 +175,15 @@ func (s *Schema) CompileFilter(expr string, opts ...FilterOption) (*Filter, erro
if err != nil {
return nil, err
}
+ options, err := c.filterOpts.json("filter options")
+ if err != nil {
+ return nil, err
+ }
if err := s.enter(); err != nil {
return nil, err
}
defer s.g.leave()
- h, cerr := s.lib.tbl.FilterCreate(s.h, []byte(expr), params, settings)
+ h, cerr := s.lib.tbl.FilterCreate(s.h, []byte(expr), params, settings, options)
if cerr != nil {
return nil, callError(cerr)
}
@@ -190,7 +194,7 @@ func (s *Schema) CompileFilter(expr string, opts ...FilterOption) (*Filter, erro
// evaluated over it with no re-parse.
func (s *Schema) ParseBlock(format Format, body []byte, opts ...BlockOption) (*Block, error) {
c := blockConfig(opts)
- settings, columns, err := callInputs(c)
+ settings, columns, options, err := callInputs(c)
if err != nil {
return nil, err
}
@@ -198,23 +202,26 @@ func (s *Schema) ParseBlock(format Format, body []byte, opts ...BlockOption) (*B
return nil, err
}
defer s.g.leave()
- h, cerr := s.lib.tbl.BlockCreate(s.h, int32(format), body, settings, columns)
+ h, cerr := s.lib.tbl.BlockCreate(s.h, int32(format), body, settings, columns, options)
if cerr != nil {
return nil, callError(cerr)
}
return newBlock(s.lib, h), nil
}
-// callInputs serializes the settings (the per-call zone written into them) and
-// the column list of a call.
-func callInputs(c *callConfig) (settings, columns []byte, err error) {
+// callInputs serializes the settings (the per-call zone written into them), the
+// column list and the insert options document of a call that models an INSERT.
+func callInputs(c *callConfig) (settings, columns, options []byte, err error) {
if settings, err = callSettings(c.settings, c.timezone); err != nil {
- return nil, nil, err
+ return nil, nil, nil, err
}
if columns, err = columnsJSON(c.columns, c.haveColumns); err != nil {
- return nil, nil, err
+ return nil, nil, nil, err
}
- return settings, columns, nil
+ if options, err = c.insertOpts.json("insert options"); err != nil {
+ return nil, nil, nil, err
+ }
+ return settings, columns, options, nil
}
// Filter is a compiled filter. It holds its schema inside the library.
@@ -252,9 +259,10 @@ func (f *Filter) enter() error {
// Rows evaluates the filter over a body (chs_filter_eval_body). The settings
// are the body's parse settings only; the filter's own zone governs its WHERE.
+// WithColumns and the insert options model the same INSERT as on Schema.Rows.
func (f *Filter) Rows(format Format, body []byte, opts ...EvalOption) (FilterResult, error) {
c := evalConfig(opts)
- settings, err := callSettings(c.settings, c.timezone)
+ settings, columns, options, err := callInputs(c)
if err != nil {
return FilterResult{}, err
}
@@ -263,7 +271,7 @@ func (f *Filter) Rows(format Format, body []byte, opts ...EvalOption) (FilterRes
}
defer f.g.leave()
raw, err := f.lib.document(func() (*abi2.Buf, *abi2.CallError) {
- return f.lib.tbl.FilterEvalBody(f.h, int32(format), body, settings)
+ return f.lib.tbl.FilterEvalBody(f.h, int32(format), body, settings, columns, options)
})
if err != nil {
return FilterResult{}, err
diff --git a/go/chtypes/server.go b/go/chtypes/server.go
index 593daf30..d133dad8 100644
--- a/go/chtypes/server.go
+++ b/go/chtypes/server.go
@@ -42,25 +42,31 @@ type ServerProfile struct {
// makes normative for macros ("absent = unknown, present even {} = complete").
type serverProfileDoc struct {
Timezone string `json:"timezone,omitzero"`
- Settings map[string]string `json:"settings,omitzero"`
+ Settings map[string]any `json:"settings,omitzero"`
Macros map[string]string `json:"macros,omitzero"`
}
// serverProfileJSON is the profile document, with the stock encoder settings
// use (keys in sorted order, no HTML escaping, no value rewritten).
func serverProfileJSON(p ServerProfile) ([]byte, error) {
- // A non-UTF-8 string would be replaced with U+FFFD by the encoder, and the
+ // A non-UTF-8 zone would be replaced with U+FFFD by the encoder, and the
// library would see a different profile: refuse it instead.
if !utf8.ValidString(p.Timezone) {
return nil, usageError("server profile: the timezone is not valid UTF-8, and %s", nonUTF8Hint)
}
- if err := checkUTF8Map("server profile settings", p.Settings); err != nil {
- return nil, err
+ // A setting's value that is not valid UTF-8 is the object form {"b64": ...};
+ // a setting's name, and a macro's name or value, cannot be carried.
+ var settings map[string]any
+ if p.Settings != nil {
+ var err error
+ if settings, err = byteValues("server profile settings", p.Settings); err != nil {
+ return nil, err
+ }
}
if err := checkUTF8Map("server profile macros", p.Macros); err != nil {
return nil, err
}
- return marshalNoEscape(serverProfileDoc(p))
+ return marshalNoEscape(serverProfileDoc{Timezone: p.Timezone, Settings: settings, Macros: p.Macros})
}
// Server is one ClickHouse server (chs_server), made by Library.NewServer and
diff --git a/go/chtypes/settings.go b/go/chtypes/settings.go
index 5adad9fd..b2cde551 100644
--- a/go/chtypes/settings.go
+++ b/go/chtypes/settings.go
@@ -18,28 +18,75 @@ import (
// sessionTimezoneKey is the settings key that carries the per-call zone.
const sessionTimezoneKey = "session_timezone"
-// stringMapJSON serializes a map of strings as a JSON object, or returns nil
-// for an empty map (the ABI reads length 0 as "none"). A key or value that is
-// not valid UTF-8 is a UsageError naming what (the input) and the key: stock
-// encoding/json would replace it with U+FFFD, and the library would then see a
-// different value with no error raised.
+// stringMapJSON serializes a map of strings as a JSON object of byte values
+// (the content json_object_byte_values), or returns nil for an empty map (the
+// ABI reads length 0 as "none"). A value that is valid UTF-8 is a JSON string,
+// and any other value is the object {"b64": ""} (spec/abi-v2/docs.md,
+// "Input values"), so no value is ever rewritten. A key that is not valid UTF-8
+// is a UsageError naming what (the input) and the key: a JSON object cannot
+// carry a key's bytes, and stock encoding/json would replace it with U+FFFD, so
+// the library would see a different key with no error raised.
func stringMapJSON(what string, m map[string]string) ([]byte, error) {
if len(m) == 0 {
return nil, nil
}
- if err := checkUTF8Map(what, m); err != nil {
+ doc, err := byteValues(what, m)
+ if err != nil {
return nil, err
}
- return marshalNoEscape(m)
+ return marshalNoEscape(doc)
}
-// nonUTF8Hint is the tail of every refusal: a JSON input cannot carry the
-// bytes in this ABI generation (spec/abi-v2/docs.md, "Input values").
-const nonUTF8Hint = "this ABI generation cannot carry a non-UTF-8 value in a JSON input; write the bytes as unhex('') in the expression instead"
+// b64Value is the object form of a value whose bytes are not valid UTF-8.
+type b64Value struct {
+ B64 string `json:"b64"`
+}
+
+// byteValue is one value of json_object_byte_values: the string itself when it is
+// valid UTF-8, else the object form.
+func byteValue(v string) any {
+ if utf8.ValidString(v) {
+ return v
+ }
+ return b64Value{B64: base64.StdEncoding.EncodeToString([]byte(v))}
+}
+
+// byteValues is the document of a map of byte values; it refuses a key that is
+// not valid UTF-8, the first in sorted order so the message does not depend on
+// map iteration.
+func byteValues(what string, m map[string]string) (map[string]any, error) {
+ if err := checkUTF8Keys(what, m); err != nil {
+ return nil, err
+ }
+ out := make(map[string]any, len(m))
+ for k, v := range m {
+ out[k] = byteValue(v)
+ }
+ return out, nil
+}
+
+// nonUTF8Hint is the tail of every refusal: a JSON input cannot carry these
+// bytes (spec/abi-v2/docs.md, "Input values").
+const nonUTF8Hint = "a JSON input cannot carry these bytes here (only a settings or query-parameter value has the {\"b64\": ...} object form)"
-// checkUTF8Map refuses the first key (in sorted order, so the message does not
-// depend on map iteration) or value that is not valid UTF-8. The key appears
-// in the message escaped, never as raw bytes.
+// checkUTF8Keys refuses the first key (in sorted order) that is not valid
+// UTF-8. The key appears in the message escaped, never as raw bytes.
+func checkUTF8Keys(what string, m map[string]string) error {
+ var bad []string
+ for k := range m {
+ if !utf8.ValidString(k) {
+ bad = append(bad, k)
+ }
+ }
+ if len(bad) == 0 {
+ return nil
+ }
+ sort.Strings(bad)
+ return usageError("%s: the key %s is not valid UTF-8, and %s", what, strconv.QuoteToASCII(bad[0]), nonUTF8Hint)
+}
+
+// checkUTF8Map refuses the first key or value (in sorted order) that is not
+// valid UTF-8, for an input with no object form for a value (a server's macros).
func checkUTF8Map(what string, m map[string]string) error {
var bad []string
for k, v := range m {
@@ -96,12 +143,9 @@ func callSettings(settings map[string]string, timezone *string) ([]byte, error)
if _, dup := settings[sessionTimezoneKey]; dup {
return nil, usageError("the zone is given twice: a session timezone option and a %s settings key", sessionTimezoneKey)
}
- if err := checkUTF8Map("settings", settings); err != nil {
+ if err := checkUTF8Keys("settings", settings); err != nil {
return nil, err
}
- if !utf8.ValidString(*timezone) {
- return nil, usageError("the zone: the value is not valid UTF-8, and %s", nonUTF8Hint)
- }
merged := make(map[string]string, len(settings)+1)
for k, v := range settings {
merged[k] = v
@@ -110,13 +154,9 @@ func callSettings(settings map[string]string, timezone *string) ([]byte, error)
return stringMapJSON("settings", merged)
}
-// columnsJSON serializes the INSERT column list as a JSON array of name
-// objects: "name" for a valid UTF-8 name, "name_b64" (standard base64, with
-// padding) for any other bytes.
-func columnsJSON(names []string, have bool) ([]byte, error) {
- if !have {
- return nil, nil
- }
+// nameObjects is a list of names as name objects: "name" for a valid UTF-8
+// name, "name_b64" (standard base64, with padding) for any other bytes.
+func nameObjects(names []string) []map[string]string {
list := make([]map[string]string, 0, len(names))
for _, n := range names {
if utf8.ValidString(n) {
@@ -125,5 +165,50 @@ func columnsJSON(names []string, have bool) ([]byte, error) {
list = append(list, map[string]string{"name_b64": base64.StdEncoding.EncodeToString([]byte(n))})
}
}
- return marshalNoEscape(list)
+ return list
+}
+
+// columnsJSON serializes the INSERT column list as a JSON array of name
+// objects (nameObjects).
+func columnsJSON(names []string, have bool) ([]byte, error) {
+ if !have {
+ return nil, nil
+ }
+ return marshalNoEscape(nameObjects(names))
+}
+
+// optionDoc is an options document under construction: the members a caller
+// set, by key. An option the binding has no name for is set with its raw JSON
+// value and passed through as given (rule r1: the library refuses a key it does
+// not know, naming it; the binding never drops or rewrites one). Setting a key
+// twice keeps the last value.
+type optionDoc map[string]any
+
+func (d *optionDoc) set(key string, v any) {
+ if *d == nil {
+ *d = optionDoc{}
+ }
+ (*d)[key] = v
+}
+
+// json is the document, or nil when no member was set (length 0 is "none", the
+// document the call took before it had one).
+func (d optionDoc) json(what string) ([]byte, error) {
+ if len(d) == 0 {
+ return nil, nil
+ }
+ for k, v := range d {
+ if raw, ok := v.(rawMember); ok && !json.Valid(raw) {
+ return nil, usageError("%s: the value for key %s is not valid JSON", what, strconv.QuoteToASCII(k))
+ }
+ if !utf8.ValidString(k) {
+ return nil, usageError("%s: the key %s is not valid UTF-8, and %s", what, strconv.QuoteToASCII(k), nonUTF8Hint)
+ }
+ }
+ return marshalNoEscape(map[string]any(d))
}
+
+// rawMember is a member's value given as JSON text, passed through verbatim.
+type rawMember json.RawMessage
+
+func (r rawMember) MarshalJSON() ([]byte, error) { return []byte(r), nil }
diff --git a/go/chtypes/settings_test.go b/go/chtypes/settings_test.go
index b81b9757..48f53641 100644
--- a/go/chtypes/settings_test.go
+++ b/go/chtypes/settings_test.go
@@ -41,8 +41,9 @@ func TestCallSettings(t *testing.T) {
}
}
-// A non-UTF-8 key or value is refused before any call, naming the input and the
-// key in escaped form; stock encoding/json would have replaced it with U+FFFD.
+// A non-UTF-8 key, and a non-UTF-8 value where the input has no object form for
+// one, is refused before any call, naming the input and the key in escaped form;
+// stock encoding/json would have replaced it with U+FFFD.
func TestNonUTF8InputIsRefused(t *testing.T) {
z := "UTC"
bad := "\xff"
@@ -51,13 +52,10 @@ func TestNonUTF8InputIsRefused(t *testing.T) {
call func() ([]byte, error)
want []string
}{
- {"settings value", func() ([]byte, error) { return callSettings(map[string]string{"k": bad}, nil) }, []string{"settings", `the value for key "k"`}},
{"settings key", func() ([]byte, error) { return callSettings(map[string]string{"k" + bad: "v"}, nil) }, []string{"settings", `the key "k\xff"`}},
- {"settings value beside a zone", func() ([]byte, error) { return callSettings(map[string]string{"k": bad}, &z) }, []string{"settings", `the value for key "k"`}},
- {"zone", func() ([]byte, error) { return callSettings(nil, &bad) }, []string{"the zone"}},
- {"query parameters value", func() ([]byte, error) { return stringMapJSON("query parameters", map[string]string{"p": bad}) }, []string{"query parameters", `the value for key "p"`}},
+ {"settings key beside a zone", func() ([]byte, error) { return callSettings(map[string]string{"k" + bad: "v"}, &z) }, []string{"settings", `the key "k\xff"`}},
{"query parameters key", func() ([]byte, error) { return stringMapJSON("query parameters", map[string]string{bad: "v"}) }, []string{"query parameters", `the key "\xff"`}},
- {"default settings", func() ([]byte, error) { return stringMapJSON("default settings", map[string]string{"d": bad}) }, []string{"default settings", `"d"`}},
+ {"default settings key", func() ([]byte, error) { return stringMapJSON("default settings", map[string]string{"d" + bad: "1"}) }, []string{"default settings", `"d\xff"`}},
{"server profile macro", func() ([]byte, error) { return serverProfileJSON(ServerProfile{Macros: map[string]string{"m": bad}}) }, []string{"server profile macros", `"m"`}},
{"server profile settings key", func() ([]byte, error) { return serverProfileJSON(ServerProfile{Settings: map[string]string{bad: "1"}}) }, []string{"server profile settings", "key"}},
{"server profile zone", func() ([]byte, error) { return serverProfileJSON(ServerProfile{Timezone: bad}) }, []string{"timezone"}},
@@ -70,7 +68,7 @@ func TestNonUTF8InputIsRefused(t *testing.T) {
continue
}
msg := err.Error()
- for _, w := range append(c.want, "not valid UTF-8", "unhex('')") {
+ for _, w := range append(c.want, "not valid UTF-8", "cannot carry") {
if !strings.Contains(msg, w) {
t.Errorf("%s: message %q lacks %q", c.name, msg, w)
}
@@ -81,6 +79,31 @@ func TestNonUTF8InputIsRefused(t *testing.T) {
}
}
+// A settings or query-parameter value that is not valid UTF-8 is the object form
+// {"b64": ...} (content json_object_byte_values), standard base64 with padding;
+// a valid one stays a JSON string. The per-call zone is a value too.
+func TestNonUTF8ValueIsTheB64Form(t *testing.T) {
+ z := "\xffZ"
+ for _, c := range []struct {
+ name string
+ got func() ([]byte, error)
+ want string
+ }{
+ {"settings", func() ([]byte, error) { return callSettings(map[string]string{"k": "\xff\xfe", "a": "ok"}, nil) }, `{"a":"ok","k":{"b64":"//4="}}`},
+ {"the zone", func() ([]byte, error) { return callSettings(nil, &z) }, `{"session_timezone":{"b64":"/1o="}}`},
+ {"query parameters", func() ([]byte, error) { return stringMapJSON("query parameters", map[string]string{"p": "\xff"}) }, `{"p":{"b64":"/w=="}}`},
+ {"defaults", func() ([]byte, error) { return stringMapJSON("default settings", map[string]string{"d": "\x00\xff"}) }, `{"d":{"b64":"AP8="}}`},
+ {"server profile settings", func() ([]byte, error) {
+ return serverProfileJSON(ServerProfile{Settings: map[string]string{"s": "\xff"}})
+ }, `{"settings":{"s":{"b64":"/w=="}}}`},
+ } {
+ b, err := c.got()
+ if err != nil || string(b) != c.want {
+ t.Errorf("%s = %s, %v, want %s", c.name, b, err, c.want)
+ }
+ }
+}
+
// Valid UTF-8 serializes byte for byte as before: non-ASCII is written as
// itself, in keys and values, and U+FFFD that the caller really wrote stays.
func TestValidNonASCIIInputIsUnchanged(t *testing.T) {
@@ -272,3 +295,81 @@ func TestAFailedOpenUnlocksOnlyItsOwnRecordAndNeverALatchedOne(t *testing.T) {
t.Errorf("the latched setup again = %v, want a no-op", err)
}
}
+
+func TestColumnDefaultsDocument(t *testing.T) {
+ cases := []struct {
+ name string
+ in []ColumnDefault
+ want string
+ }{
+ {"utf8", []ColumnDefault{{Name: "a", Value: "héllo"}}, `{"column_defaults":[{"name":"a","value":"héllo"}]}`},
+ {"non-utf8 value is value_b64", []ColumnDefault{{Name: "a", Value: "\xff\x00"}}, `{"column_defaults":[{"name":"a","value_b64":"/wA="}]}`},
+ {"non-utf8 name is name_b64", []ColumnDefault{{Name: "\xfe", Value: "v"}}, `{"column_defaults":[{"name_b64":"/g==","value":"v"}]}`},
+ {"empty value is sent", []ColumnDefault{{Name: "a", Value: ""}}, `{"column_defaults":[{"name":"a","value":""}]}`},
+ {"order is kept", []ColumnDefault{{Name: "z", Value: "1"}, {Name: "a", Value: "2"}}, `{"column_defaults":[{"name":"z","value":"1"},{"name":"a","value":"2"}]}`},
+ {"empty list is sent", []ColumnDefault{}, `{"column_defaults":[]}`},
+ }
+ for _, tc := range cases {
+ b, err := rowConfig([]RowOption{WithColumnDefaults(tc.in)}).insertOpts.json("insert options")
+ if err != nil || string(b) != tc.want {
+ t.Errorf("%s: got %s, %v, want %s", tc.name, b, err, tc.want)
+ }
+ }
+}
+
+func TestInsertOptionsDocument(t *testing.T) {
+ // No member set: no document (length 0).
+ if b, err := rowConfig(nil).insertOpts.json("insert options"); err != nil || b != nil {
+ t.Errorf("empty = %q, %v", b, err)
+ }
+ // A nil list sets nothing; a non-nil empty list is sent.
+ if b, _ := rowConfig([]RowOption{WithInsertableColumns(nil), WithColumnDefaults(nil)}).insertOpts.json("x"); b != nil {
+ t.Errorf("nil lists set a member: %s", b)
+ }
+ c := rowsConfig([]RowsOption{
+ WithInsertableColumns([]string{"a", "\xff"}),
+ WithColumnDefaults([]ColumnDefault{{Name: "n", Value: "v"}, {Name: "\xfe", Value: "\xff"}}),
+ WithVerdictBasis(VerdictBasisExport),
+ WithDeferDefaults(DeferDefaultsServer),
+ WithInsertOption("future_member", []byte(`{"x":[1,2]}`)),
+ })
+ b, err := c.insertOpts.json("insert options")
+ want := `{"column_defaults":[{"name":"n","value":"v"},{"name_b64":"/g==","value_b64":"/w=="}],"defer_defaults":"server","future_member":{"x":[1,2]},"insertable_columns":[{"name":"a"},{"name_b64":"/w=="}],"verdict_basis":"export"}`
+ if err != nil || string(b) != want {
+ t.Errorf("insert options = %s, %v, want %s", b, err, want)
+ }
+ // The same option applies to all four calls that model an INSERT.
+ for name, cfg := range map[string]*callConfig{
+ "row": rowConfig([]RowOption{WithDeferDefaults(DeferDefaultsNone)}),
+ "block": blockConfig([]BlockOption{WithDeferDefaults(DeferDefaultsNone)}),
+ "eval": evalConfig([]EvalOption{WithDeferDefaults(DeferDefaultsNone), WithColumns([]string{"a"})}),
+ } {
+ if b, _ := cfg.insertOpts.json("x"); string(b) != `{"defer_defaults":"none"}` {
+ t.Errorf("%s: insert options = %s", name, b)
+ }
+ }
+ if c := evalConfig([]EvalOption{WithColumns([]string{"a"})}); !c.haveColumns || len(c.columns) != 1 {
+ t.Errorf("an eval does not take the column list: %+v", c)
+ }
+ // A value that is not JSON is misuse, raised before any call.
+ bad := rowConfig([]RowOption{WithInsertOption("k", []byte(`{`))})
+ if _, err := bad.insertOpts.json("insert options"); err == nil {
+ t.Error("invalid JSON passed through")
+ } else if _, ok := err.(*UsageError); !ok {
+ t.Errorf("invalid JSON = %T, want *UsageError", err)
+ }
+}
+
+func TestFilterAndSchemaOptionsDocuments(t *testing.T) {
+ f := filterConfig([]FilterOption{WithParamEncoding(ParamEncodingRaw), WithFilterOption("k", []byte(`1`))})
+ if b, err := f.filterOpts.json("filter options"); err != nil || string(b) != `{"k":1,"param_encoding":"raw"}` {
+ t.Errorf("filter options = %s, %v", b, err)
+ }
+ c := compileConfig([]CompileOption{WithPosture(PostureLoad), WithSchemaOption("z", []byte(`"v"`))})
+ if b, err := c.schemaOpts.json("schema options"); err != nil || string(b) != `{"posture":"load","z":"v"}` {
+ t.Errorf("schema options = %s, %v", b, err)
+ }
+ if b, _ := compileConfig(nil).schemaOpts.json("x"); b != nil {
+ t.Errorf("an empty schema options document = %s", b)
+ }
+}
diff --git a/go/chtypes/testdata/goldens-v1-stub.json b/go/chtypes/testdata/goldens-v1-stub.json
index 57f9082b..016fbb97 100644
--- a/go/chtypes/testdata/goldens-v1-stub.json
+++ b/go/chtypes/testdata/goldens-v1-stub.json
@@ -68,12 +68,12 @@
"status": "CHS_OK",
"document": {
"outcome": "accepted",
+ "code": 0,
"input_span": {
"off": 0,
"len": 6
},
"unknown_fields": [],
- "unsupported_settings": [],
"cols": [
{
"name": "s",
@@ -112,12 +112,12 @@
"status": "CHS_OK",
"document": {
"outcome": "accepted",
+ "code": 0,
"input_span": {
"off": 0,
"len": 6
},
"unknown_fields": [],
- "unsupported_settings": [],
"cols": [
{
"name": "s",
@@ -164,12 +164,12 @@
"status": "CHS_OK",
"document": {
"outcome": "accepted",
+ "code": 0,
"input_span": {
"off": 0,
"len": 6
},
"unknown_fields": [],
- "unsupported_settings": [],
"cols": [
{
"name": "s",
diff --git a/go/chtypes/vocab_gen.go b/go/chtypes/vocab_gen.go
index db1c9128..00d79ec2 100644
--- a/go/chtypes/vocab_gen.go
+++ b/go/chtypes/vocab_gen.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
package chtypes
@@ -110,6 +110,10 @@ const TSVWithNames Format = 11
// ExportNone is CHS_EXPORT_NONE: no export was asked for.
const ExportNone Format = -1
+// CodeDeclined is CHS_CODE_DECLINED: the code a declined row, outcome or verdict carries, outside
+// ClickHouse's code table (the Codes paragraph of the reference).
+const CodeDeclined int32 = -2
+
// Known reports whether the description lists the format. A value it does not list is the
// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
func (f Format) Known() bool {
@@ -304,9 +308,14 @@ const SourceDefaultVolatileUnresolved Source = "default_volatile_unresolved"
// SourceDefaultPending is the Source value "default_pending".
const SourceDefaultPending Source = "default_pending"
+// SourceDefaultDeferred is the Source value "default_deferred".
+const SourceDefaultDeferred Source = "default_deferred"
+
+// SourceDefaultOverride is the Source value "default_override".
+const SourceDefaultOverride Source = "default_override"
+
// IsStored is the description's own is_stored fact for the source: whether the value is stored.
-// The description does not yet say what an unknown(n) source reports (rule r3, "Facts without a
-// fallback"); until it does, IsStored is false for one.
+// An unknown(n) source reports its fallback's (skipped) fact (rule r3).
func (x Source) IsStored() bool {
switch x {
case SourceInput:
@@ -321,6 +330,8 @@ func (x Source) IsStored() bool {
return true
case SourceMaterializedInput:
return true
+ case SourceDefaultOverride:
+ return true
}
return false
}
@@ -329,7 +340,7 @@ func (x Source) IsStored() bool {
// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
func (x Source) Known() bool {
switch x {
- case SourceInput, SourceDefault, SourceDefaultSubstituted, SourceDefaultGenerated, SourceAbsent, SourceMaterializedInput, SourceSkipped, SourceEphemeralInput, SourceDefaultExprUnsupported, SourceDefaultVolatileUnresolved, SourceDefaultPending:
+ case SourceInput, SourceDefault, SourceDefaultSubstituted, SourceDefaultGenerated, SourceAbsent, SourceMaterializedInput, SourceSkipped, SourceEphemeralInput, SourceDefaultExprUnsupported, SourceDefaultVolatileUnresolved, SourceDefaultPending, SourceDefaultDeferred, SourceDefaultOverride:
return true
}
return false
@@ -522,11 +533,14 @@ const TierPredicateUnflipped DeclinedTier = "predicate-unflipped"
// TierResultContent is the DeclinedTier value "result-content".
const TierResultContent DeclinedTier = "result-content"
+// TierServerRefuses is the DeclinedTier value "server-refuses".
+const TierServerRefuses DeclinedTier = "server-refuses"
+
// Known reports whether the description lists the tier. A value it does not list is the
// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
func (x DeclinedTier) Known() bool {
switch x {
- case TierPredicate, TierPredicateUnflipped, TierResultContent:
+ case TierPredicate, TierPredicateUnflipped, TierResultContent, TierServerRefuses:
return true
}
return false
@@ -556,9 +570,83 @@ func (x DeclinedLayer) Known() bool {
return false
}
+// Posture is posture: how a schema's statement was compiled, as a server's CREATE TABLE (create)
+// or as a server loads the stored definition of a table it holds (load). The spelling is the
+// description's.
+type Posture string
+
+// PostureCreate is the Posture value "create".
+const PostureCreate Posture = "create"
+
+// PostureLoad is the Posture value "load".
+const PostureLoad Posture = "load"
+
+// Known reports whether the description lists the posture. A value it does not list is the
+// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
+func (x Posture) Known() bool {
+ switch x {
+ case PostureCreate, PostureLoad:
+ return true
+ }
+ return false
+}
+
+// RefusalSite is refusal_site: where a batch or filter document's refusal sits, the call (its settings
+// or its column list), the body outside any row (a header, the framing), one row (RefusedRow) or a
+// whole-body step. The spelling is the description's.
+type RefusalSite string
+
+// RefusedAtCall is the RefusalSite value "call".
+const RefusedAtCall RefusalSite = "call"
+
+// RefusedAtBodyRead is the RefusalSite value "body_read".
+const RefusedAtBodyRead RefusalSite = "body_read"
+
+// RefusedAtRow is the RefusalSite value "row".
+const RefusedAtRow RefusalSite = "row"
+
+// RefusedAtBodyStep is the RefusalSite value "body_step".
+const RefusedAtBodyStep RefusalSite = "body_step"
+
+// Known reports whether the description lists the site. A value it does not list is the
+// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
+func (x RefusalSite) Known() bool {
+ switch x {
+ case RefusedAtCall, RefusedAtBodyRead, RefusedAtRow, RefusedAtBodyStep:
+ return true
+ }
+ return false
+}
+
+// DefaultSupport is default_support: what this build does with a column's DEFAULT, as a schema
+// description reports it per column. The spelling is the description's.
+type DefaultSupport string
+
+// DefaultSupportComputed is the DefaultSupport value "computed".
+const DefaultSupportComputed DefaultSupport = "computed"
+
+// DefaultSupportGenerated is the DefaultSupport value "generated".
+const DefaultSupportGenerated DefaultSupport = "generated"
+
+// DefaultSupportDeferred is the DefaultSupport value "deferred".
+const DefaultSupportDeferred DefaultSupport = "deferred"
+
+// DefaultSupportDeclined is the DefaultSupport value "declined".
+const DefaultSupportDeclined DefaultSupport = "declined"
+
+// Known reports whether the description lists the support. A value it does not list is the
+// vocabulary's unknown(n) member, carrying the raw value (rule r3): a reader keeps it and decodes on.
+func (x DefaultSupport) Known() bool {
+ switch x {
+ case DefaultSupportComputed, DefaultSupportGenerated, DefaultSupportDeferred, DefaultSupportDeclined:
+ return true
+ }
+ return false
+}
+
// describedVocabularyNames is every enum the description defines, in its own order.
func describedVocabularyNames() []string {
- return []string{"chs_status", "chs_format", "transform_reason", "merge_reason", "value_src", "row_outcome", "batch_outcome", "filter_outcome", "filter_verdict", "discover_query_param", "default_kind", "declined_tier", "declined_layer"}
+ return []string{"chs_status", "chs_format", "transform_reason", "merge_reason", "value_src", "row_outcome", "batch_outcome", "filter_outcome", "filter_verdict", "refusal_site", "discover_query_param", "default_kind", "default_support", "declined_tier", "declined_layer", "posture"}
}
// describedVocabulary builds the value of the enum `name` whose raw spelling is raw (an integer for an
@@ -601,18 +689,27 @@ func describedVocabulary(name, raw string) (known bool, back string, ok bool) {
case "filter_verdict":
v := Verdict(raw)
return v.Known(), string(v), true
+ case "refusal_site":
+ v := RefusalSite(raw)
+ return v.Known(), string(v), true
case "discover_query_param":
v := DiscoverQueryParam(raw)
return v.Known(), string(v), true
case "default_kind":
v := DefaultKind(raw)
return v.Known(), string(v), true
+ case "default_support":
+ v := DefaultSupport(raw)
+ return v.Known(), string(v), true
case "declined_tier":
v := DeclinedTier(raw)
return v.Known(), string(v), true
case "declined_layer":
v := DeclinedLayer(raw)
return v.Known(), string(v), true
+ case "posture":
+ v := Posture(raw)
+ return v.Known(), string(v), true
}
return false, "", false
}
diff --git a/go/internal/abi2/abi2_table.h b/go/internal/abi2/abi2_table.h
index 74cd6d5e..a211434c 100644
--- a/go/internal/abi2/abi2_table.h
+++ b/go/internal/abi2/abi2_table.h
@@ -1,4 +1,4 @@
-/* GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT */
+/* GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT */
#ifndef CHTYPES_ABI2_TABLE_H
#define CHTYPES_ABI2_TABLE_H
@@ -25,6 +25,7 @@ typedef int32_t (*chtypes_abi2_fn_chs_error_ch_code)(const void * error);
typedef void * (*chtypes_abi2_fn_chs_error_ch_name)(const void * error);
typedef void * (*chtypes_abi2_fn_chs_error_message)(const void * error);
typedef void * (*chtypes_abi2_fn_chs_error_column)(const void * error);
+typedef void * (*chtypes_abi2_fn_chs_error_reason)(const void * error);
typedef void (*chtypes_abi2_fn_chs_error_free)(void * error);
typedef int32_t (*chtypes_abi2_fn_chs_live_handles)(void ** out, void ** err);
typedef int32_t (*chtypes_abi2_fn_chs_error_codes)(void ** out, void ** err);
@@ -43,12 +44,12 @@ typedef void (*chtypes_abi2_fn_chs_server_free)(void * server);
typedef int32_t (*chtypes_abi2_fn_chs_schema_create)(const void * server, const uint8_t * create_table, size_t create_table_len, const uint8_t * settings, size_t settings_len, const uint8_t * options, size_t options_len, void ** out, void ** err);
typedef void (*chtypes_abi2_fn_chs_schema_free)(void * schema);
typedef int32_t (*chtypes_abi2_fn_chs_schema_describe)(const void * schema, void ** out, void ** err);
-typedef int32_t (*chtypes_abi2_fn_chs_preview_row)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, void ** out, void ** err);
-typedef int32_t (*chtypes_abi2_fn_chs_preview_batch)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const void * filter, int32_t export_format, uint32_t doc_flags, void ** out, void ** out_export, void ** err);
-typedef int32_t (*chtypes_abi2_fn_chs_filter_create)(const void * schema, const uint8_t * expr, size_t expr_len, const uint8_t * query_params, size_t query_params_len, const uint8_t * settings, size_t settings_len, void ** out, void ** err);
+typedef int32_t (*chtypes_abi2_fn_chs_preview_row)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err);
+typedef int32_t (*chtypes_abi2_fn_chs_preview_batch)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const void * filter, int32_t export_format, uint32_t doc_flags, const uint8_t * options, size_t options_len, void ** out, void ** out_export, void ** err);
+typedef int32_t (*chtypes_abi2_fn_chs_filter_create)(const void * schema, const uint8_t * expr, size_t expr_len, const uint8_t * query_params, size_t query_params_len, const uint8_t * settings, size_t settings_len, const uint8_t * options, size_t options_len, void ** out, void ** err);
typedef void (*chtypes_abi2_fn_chs_filter_free)(void * filter);
-typedef int32_t (*chtypes_abi2_fn_chs_filter_eval_body)(const void * filter, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, void ** out, void ** err);
-typedef int32_t (*chtypes_abi2_fn_chs_block_create)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, void ** out, void ** err);
+typedef int32_t (*chtypes_abi2_fn_chs_filter_eval_body)(const void * filter, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err);
+typedef int32_t (*chtypes_abi2_fn_chs_block_create)(const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err);
typedef void (*chtypes_abi2_fn_chs_block_free)(void * block);
typedef int32_t (*chtypes_abi2_fn_chs_filter_eval_block)(const void * filter, const void * block, void ** out, void ** err);
typedef int32_t (*chtypes_abi2_fn_chs_discover_query)(void ** out, void ** err);
@@ -67,6 +68,7 @@ struct chtypes_abi2_table {
chtypes_abi2_fn_chs_error_ch_name chs_error_ch_name;
chtypes_abi2_fn_chs_error_message chs_error_message;
chtypes_abi2_fn_chs_error_column chs_error_column;
+ chtypes_abi2_fn_chs_error_reason chs_error_reason;
chtypes_abi2_fn_chs_error_free chs_error_free;
chtypes_abi2_fn_chs_live_handles chs_live_handles;
chtypes_abi2_fn_chs_error_codes chs_error_codes;
diff --git a/go/internal/abi2/abi_gen.go b/go/internal/abi2/abi_gen.go
index cff68eb8..f67fd027 100644
--- a/go/internal/abi2/abi_gen.go
+++ b/go/internal/abi2/abi_gen.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
package abi2
@@ -94,6 +94,8 @@ static const char *chtypes_abi2_resolve_all(void *h, struct chtypes_abi2_table *
if (!t->chs_error_message) return "chs_error_message";
t->chs_error_column = (chtypes_abi2_fn_chs_error_column) dlsym(h, "chs_error_column");
if (!t->chs_error_column) return "chs_error_column";
+ t->chs_error_reason = (chtypes_abi2_fn_chs_error_reason) dlsym(h, "chs_error_reason");
+ if (!t->chs_error_reason) return "chs_error_reason";
t->chs_error_free = (chtypes_abi2_fn_chs_error_free) dlsym(h, "chs_error_free");
if (!t->chs_error_free) return "chs_error_free";
t->chs_live_handles = (chtypes_abi2_fn_chs_live_handles) dlsym(h, "chs_live_handles");
@@ -165,6 +167,7 @@ static int32_t chtypes_abi2_call_chs_error_ch_code(struct chtypes_abi2_table *t,
static void * chtypes_abi2_call_chs_error_ch_name(struct chtypes_abi2_table *t, const void * error) { return t->chs_error_ch_name(error); }
static void * chtypes_abi2_call_chs_error_message(struct chtypes_abi2_table *t, const void * error) { return t->chs_error_message(error); }
static void * chtypes_abi2_call_chs_error_column(struct chtypes_abi2_table *t, const void * error) { return t->chs_error_column(error); }
+static void * chtypes_abi2_call_chs_error_reason(struct chtypes_abi2_table *t, const void * error) { return t->chs_error_reason(error); }
static void chtypes_abi2_call_chs_error_free(struct chtypes_abi2_table *t, void * error) { t->chs_error_free(error); }
static int32_t chtypes_abi2_call_chs_live_handles(struct chtypes_abi2_table *t, void ** out, void ** err) { return t->chs_live_handles(out, err); }
static int32_t chtypes_abi2_call_chs_error_codes(struct chtypes_abi2_table *t, void ** out, void ** err) { return t->chs_error_codes(out, err); }
@@ -183,12 +186,12 @@ static void chtypes_abi2_call_chs_server_free(struct chtypes_abi2_table *t, void
static int32_t chtypes_abi2_call_chs_schema_create(struct chtypes_abi2_table *t, const void * server, const uint8_t * create_table, size_t create_table_len, const uint8_t * settings, size_t settings_len, const uint8_t * options, size_t options_len, void ** out, void ** err) { return t->chs_schema_create(server, create_table, create_table_len, settings, settings_len, options, options_len, out, err); }
static void chtypes_abi2_call_chs_schema_free(struct chtypes_abi2_table *t, void * schema) { t->chs_schema_free(schema); }
static int32_t chtypes_abi2_call_chs_schema_describe(struct chtypes_abi2_table *t, const void * schema, void ** out, void ** err) { return t->chs_schema_describe(schema, out, err); }
-static int32_t chtypes_abi2_call_chs_preview_row(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, void ** out, void ** err) { return t->chs_preview_row(schema, format, body, body_len, settings, settings_len, columns, columns_len, out, err); }
-static int32_t chtypes_abi2_call_chs_preview_batch(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const void * filter, int32_t export_format, uint32_t doc_flags, void ** out, void ** out_export, void ** err) { return t->chs_preview_batch(schema, format, body, body_len, settings, settings_len, columns, columns_len, filter, export_format, doc_flags, out, out_export, err); }
-static int32_t chtypes_abi2_call_chs_filter_create(struct chtypes_abi2_table *t, const void * schema, const uint8_t * expr, size_t expr_len, const uint8_t * query_params, size_t query_params_len, const uint8_t * settings, size_t settings_len, void ** out, void ** err) { return t->chs_filter_create(schema, expr, expr_len, query_params, query_params_len, settings, settings_len, out, err); }
+static int32_t chtypes_abi2_call_chs_preview_row(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err) { return t->chs_preview_row(schema, format, body, body_len, settings, settings_len, columns, columns_len, options, options_len, out, err); }
+static int32_t chtypes_abi2_call_chs_preview_batch(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const void * filter, int32_t export_format, uint32_t doc_flags, const uint8_t * options, size_t options_len, void ** out, void ** out_export, void ** err) { return t->chs_preview_batch(schema, format, body, body_len, settings, settings_len, columns, columns_len, filter, export_format, doc_flags, options, options_len, out, out_export, err); }
+static int32_t chtypes_abi2_call_chs_filter_create(struct chtypes_abi2_table *t, const void * schema, const uint8_t * expr, size_t expr_len, const uint8_t * query_params, size_t query_params_len, const uint8_t * settings, size_t settings_len, const uint8_t * options, size_t options_len, void ** out, void ** err) { return t->chs_filter_create(schema, expr, expr_len, query_params, query_params_len, settings, settings_len, options, options_len, out, err); }
static void chtypes_abi2_call_chs_filter_free(struct chtypes_abi2_table *t, void * filter) { t->chs_filter_free(filter); }
-static int32_t chtypes_abi2_call_chs_filter_eval_body(struct chtypes_abi2_table *t, const void * filter, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, void ** out, void ** err) { return t->chs_filter_eval_body(filter, format, body, body_len, settings, settings_len, out, err); }
-static int32_t chtypes_abi2_call_chs_block_create(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, void ** out, void ** err) { return t->chs_block_create(schema, format, body, body_len, settings, settings_len, columns, columns_len, out, err); }
+static int32_t chtypes_abi2_call_chs_filter_eval_body(struct chtypes_abi2_table *t, const void * filter, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err) { return t->chs_filter_eval_body(filter, format, body, body_len, settings, settings_len, columns, columns_len, options, options_len, out, err); }
+static int32_t chtypes_abi2_call_chs_block_create(struct chtypes_abi2_table *t, const void * schema, int32_t format, const uint8_t * body, size_t body_len, const uint8_t * settings, size_t settings_len, const uint8_t * columns, size_t columns_len, const uint8_t * options, size_t options_len, void ** out, void ** err) { return t->chs_block_create(schema, format, body, body_len, settings, settings_len, columns, columns_len, options, options_len, out, err); }
static void chtypes_abi2_call_chs_block_free(struct chtypes_abi2_table *t, void * block) { t->chs_block_free(block); }
static int32_t chtypes_abi2_call_chs_filter_eval_block(struct chtypes_abi2_table *t, const void * filter, const void * block, void ** out, void ** err) { return t->chs_filter_eval_block(filter, block, out, err); }
static int32_t chtypes_abi2_call_chs_discover_query(struct chtypes_abi2_table *t, void ** out, void ** err) { return t->chs_discover_query(out, err); }
@@ -208,7 +211,7 @@ const ChsAbiVersion = 2
// ChsAbiFingerprint is CHS_ABI_FINGERPRINT: sha256 over the RFC 8785 canonical
// form of spec/abi-v2/abi.json. chs_build_info()'s abi_fingerprint field is
// this value, copied from the compiled-in macro, never recomputed.
-const ChsAbiFingerprint = "sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4"
+const ChsAbiFingerprint = "sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2"
// ChsAbiStability is the description's stability: "unstable" while this generation
// is being designed, so ChsAbiFingerprint moves with every change, and "locked"
@@ -289,7 +292,7 @@ func (t *Table) ResolveAll() string {
}
// CallError is D3's per-call error shape, read verbatim off a non-OK status:
-// the chs_status NAME (never the bare int), ch_code, ch_name, message and column.
+// the chs_status NAME (never the bare int), ch_code, ch_name, message and column and, from chs_error_reason, reason.
// It carries no class of its own -- StatusClass(e.Status) is sdk.json's table,
// generated into errmap_gen.go, and the public API (wave C) is what turns this
// into an idiomatic *SchemaError/*UsageError/...
@@ -299,6 +302,7 @@ type CallError struct {
ChName string
Message string
Column string
+ Reason string
}
func (e *CallError) Error() string {
@@ -376,6 +380,7 @@ func makeCallError(t *Table, status int32, err *ErrorHandle) *CallError {
ce.ChName = string(readAndFreeBuf(t, t.ErrorChName(err)))
ce.Message = string(readAndFreeBuf(t, t.ErrorMessage(err)))
ce.Column = string(readAndFreeBuf(t, t.ErrorColumn(err)))
+ ce.Reason = string(readAndFreeBuf(t, t.ErrorReason(err)))
err.Close()
}
return ce
@@ -689,6 +694,11 @@ func (t *Table) ErrorColumn(errorArg *ErrorHandle) *Buf {
return newBuf(t, C.chtypes_abi2_call_chs_error_column(&t.tbl, errorArg.rawPtr()))
}
+// ErrorReason is the generated call wrapper for chs_error_reason.
+func (t *Table) ErrorReason(errorArg *ErrorHandle) *Buf {
+ return newBuf(t, C.chtypes_abi2_call_chs_error_reason(&t.tbl, errorArg.rawPtr()))
+}
+
// ErrorFree is the generated call wrapper for chs_error_free.
func (t *Table) ErrorFree(errorArg *ErrorHandle) {
C.chtypes_abi2_call_chs_error_free(&t.tbl, errorArg.rawPtr())
@@ -832,36 +842,39 @@ func (t *Table) SchemaDescribe(schema *Schema) (*Buf, *CallError) {
}
// PreviewRow is the generated call wrapper for chs_preview_row.
-func (t *Table) PreviewRow(schema *Schema, format int32, body []byte, settings []byte, columns []byte) (*Buf, *CallError) {
+func (t *Table) PreviewRow(schema *Schema, format int32, body []byte, settings []byte, columns []byte, options []byte) (*Buf, *CallError) {
bodyPtr, bodyLen := bytesPtr(body)
settingsPtr, settingsLen := bytesPtr(settings)
columnsPtr, columnsLen := bytesPtr(columns)
+ optionsPtr, optionsLen := bytesPtr(options)
var outOut unsafe.Pointer
var errOut unsafe.Pointer
- status := C.chtypes_abi2_call_chs_preview_row(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, &outOut, &errOut)
+ status := C.chtypes_abi2_call_chs_preview_row(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, optionsPtr, optionsLen, &outOut, &errOut)
return newBuf(t, outOut), makeCallError(t, int32(status), newErrorHandle(t, errOut))
}
// PreviewBatch is the generated call wrapper for chs_preview_batch.
-func (t *Table) PreviewBatch(schema *Schema, format int32, body []byte, settings []byte, columns []byte, filter *Filter, exportFormat int32, docFlags uint32) (*Buf, *Buf, *CallError) {
+func (t *Table) PreviewBatch(schema *Schema, format int32, body []byte, settings []byte, columns []byte, filter *Filter, exportFormat int32, docFlags uint32, options []byte) (*Buf, *Buf, *CallError) {
bodyPtr, bodyLen := bytesPtr(body)
settingsPtr, settingsLen := bytesPtr(settings)
columnsPtr, columnsLen := bytesPtr(columns)
+ optionsPtr, optionsLen := bytesPtr(options)
var outOut unsafe.Pointer
var outExportOut unsafe.Pointer
var errOut unsafe.Pointer
- status := C.chtypes_abi2_call_chs_preview_batch(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, filter.rawPtr(), C.int32_t(exportFormat), C.uint32_t(docFlags), &outOut, &outExportOut, &errOut)
+ status := C.chtypes_abi2_call_chs_preview_batch(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, filter.rawPtr(), C.int32_t(exportFormat), C.uint32_t(docFlags), optionsPtr, optionsLen, &outOut, &outExportOut, &errOut)
return newBuf(t, outOut), newBuf(t, outExportOut), makeCallError(t, int32(status), newErrorHandle(t, errOut))
}
// FilterCreate is the generated call wrapper for chs_filter_create.
-func (t *Table) FilterCreate(schema *Schema, expr []byte, queryParams []byte, settings []byte) (*Filter, *CallError) {
+func (t *Table) FilterCreate(schema *Schema, expr []byte, queryParams []byte, settings []byte, options []byte) (*Filter, *CallError) {
exprPtr, exprLen := bytesPtr(expr)
queryParamsPtr, queryParamsLen := bytesPtr(queryParams)
settingsPtr, settingsLen := bytesPtr(settings)
+ optionsPtr, optionsLen := bytesPtr(options)
var outOut unsafe.Pointer
var errOut unsafe.Pointer
- status := C.chtypes_abi2_call_chs_filter_create(&t.tbl, schema.rawPtr(), exprPtr, exprLen, queryParamsPtr, queryParamsLen, settingsPtr, settingsLen, &outOut, &errOut)
+ status := C.chtypes_abi2_call_chs_filter_create(&t.tbl, schema.rawPtr(), exprPtr, exprLen, queryParamsPtr, queryParamsLen, settingsPtr, settingsLen, optionsPtr, optionsLen, &outOut, &errOut)
return newFilter(t, outOut), makeCallError(t, int32(status), newErrorHandle(t, errOut))
}
@@ -871,23 +884,26 @@ func (t *Table) FilterFree(filter *Filter) {
}
// FilterEvalBody is the generated call wrapper for chs_filter_eval_body.
-func (t *Table) FilterEvalBody(filter *Filter, format int32, body []byte, settings []byte) (*Buf, *CallError) {
+func (t *Table) FilterEvalBody(filter *Filter, format int32, body []byte, settings []byte, columns []byte, options []byte) (*Buf, *CallError) {
bodyPtr, bodyLen := bytesPtr(body)
settingsPtr, settingsLen := bytesPtr(settings)
+ columnsPtr, columnsLen := bytesPtr(columns)
+ optionsPtr, optionsLen := bytesPtr(options)
var outOut unsafe.Pointer
var errOut unsafe.Pointer
- status := C.chtypes_abi2_call_chs_filter_eval_body(&t.tbl, filter.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, &outOut, &errOut)
+ status := C.chtypes_abi2_call_chs_filter_eval_body(&t.tbl, filter.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, optionsPtr, optionsLen, &outOut, &errOut)
return newBuf(t, outOut), makeCallError(t, int32(status), newErrorHandle(t, errOut))
}
// BlockCreate is the generated call wrapper for chs_block_create.
-func (t *Table) BlockCreate(schema *Schema, format int32, body []byte, settings []byte, columns []byte) (*Block, *CallError) {
+func (t *Table) BlockCreate(schema *Schema, format int32, body []byte, settings []byte, columns []byte, options []byte) (*Block, *CallError) {
bodyPtr, bodyLen := bytesPtr(body)
settingsPtr, settingsLen := bytesPtr(settings)
columnsPtr, columnsLen := bytesPtr(columns)
+ optionsPtr, optionsLen := bytesPtr(options)
var outOut unsafe.Pointer
var errOut unsafe.Pointer
- status := C.chtypes_abi2_call_chs_block_create(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, &outOut, &errOut)
+ status := C.chtypes_abi2_call_chs_block_create(&t.tbl, schema.rawPtr(), C.int32_t(format), bodyPtr, bodyLen, settingsPtr, settingsLen, columnsPtr, columnsLen, optionsPtr, optionsLen, &outOut, &errOut)
return newBlock(t, outOut), makeCallError(t, int32(status), newErrorHandle(t, errOut))
}
diff --git a/go/internal/abi2/conformance_lifecycle_test.go b/go/internal/abi2/conformance_lifecycle_test.go
index 267605ee..cf7fe93e 100644
--- a/go/internal/abi2/conformance_lifecycle_test.go
+++ b/go/internal/abi2/conformance_lifecycle_test.go
@@ -157,38 +157,38 @@ func callResolved(t *Table, fn string, a []resolved) (*InvokeResult, error) {
o, ce := t.SchemaDescribe(a[0].schema())
return finishResult(t, fn, ce, map[string]*Buf{"out": o})
case "chs_preview_row":
- if err := need(5); err != nil {
+ if err := need(6); err != nil {
return nil, err
}
- o, ce := t.PreviewRow(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b())
+ o, ce := t.PreviewRow(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b(), a[5].b())
return finishResult(t, fn, ce, map[string]*Buf{"out": o})
case "chs_preview_batch":
- if err := need(8); err != nil {
+ if err := need(9); err != nil {
return nil, err
}
- o, x, ce := t.PreviewBatch(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b(), a[5].filter(), int32(a[6].i()), uint32(a[7].i()))
+ o, x, ce := t.PreviewBatch(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b(), a[5].filter(), int32(a[6].i()), uint32(a[7].i()), a[8].b())
return finishResult(t, fn, ce, map[string]*Buf{"out": o, "out_export": x})
case "chs_filter_create":
- if err := need(4); err != nil {
+ if err := need(5); err != nil {
return nil, err
}
- h, ce := t.FilterCreate(a[0].schema(), a[1].b(), a[2].b(), a[3].b())
+ h, ce := t.FilterCreate(a[0].schema(), a[1].b(), a[2].b(), a[3].b(), a[4].b())
r, err := finishResult(t, fn, ce, nil)
if r != nil {
r.Handle = h
}
return r, err
case "chs_filter_eval_body":
- if err := need(4); err != nil {
+ if err := need(6); err != nil {
return nil, err
}
- o, ce := t.FilterEvalBody(a[0].filter(), int32(a[1].i()), a[2].b(), a[3].b())
+ o, ce := t.FilterEvalBody(a[0].filter(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b(), a[5].b())
return finishResult(t, fn, ce, map[string]*Buf{"out": o})
case "chs_block_create":
- if err := need(5); err != nil {
+ if err := need(6); err != nil {
return nil, err
}
- h, ce := t.BlockCreate(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b())
+ h, ce := t.BlockCreate(a[0].schema(), int32(a[1].i()), a[2].b(), a[3].b(), a[4].b(), a[5].b())
r, err := finishResult(t, fn, ce, nil)
if r != nil {
r.Handle = h
diff --git a/go/internal/abi2/errmap_gen.go b/go/internal/abi2/errmap_gen.go
index 48efc93d..547ef000 100644
--- a/go/internal/abi2/errmap_gen.go
+++ b/go/internal/abi2/errmap_gen.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
package abi2
diff --git a/go/internal/abi2/invoke_gen_test.go b/go/internal/abi2/invoke_gen_test.go
index 45e4f155..cdca10df 100644
--- a/go/internal/abi2/invoke_gen_test.go
+++ b/go/internal/abi2/invoke_gen_test.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
package abi2
@@ -345,6 +345,23 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
return nil, fmt.Errorf("chs_error_column: %w", err)
}
return &InvokeResult{Outputs: map[string]interface{}{"return": doc}}, nil
+ case "chs_error_reason":
+ if len(args) != 1 {
+ return nil, fmt.Errorf("chs_error_reason: want %d args, got %d", 1, len(args))
+ }
+ h0, err := mintErrorHandle(t, args[0])
+ if err != nil {
+ return nil, fmt.Errorf("chs_error_reason: arg %d: %w", 0, err)
+ }
+ v := t.ErrorReason(h0)
+ if v == nil {
+ return &InvokeResult{}, nil
+ }
+ doc, err := bufJSON(t, v)
+ if err != nil {
+ return nil, fmt.Errorf("chs_error_reason: %w", err)
+ }
+ return &InvokeResult{Outputs: map[string]interface{}{"return": doc}}, nil
case "chs_error_free":
if len(args) != 1 {
return nil, fmt.Errorf("chs_error_free: want %d args, got %d", 1, len(args))
@@ -688,8 +705,8 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
}
return res, nil
case "chs_preview_row":
- if len(args) != 5 {
- return nil, fmt.Errorf("chs_preview_row: want %d args, got %d", 5, len(args))
+ if len(args) != 6 {
+ return nil, fmt.Errorf("chs_preview_row: want %d args, got %d", 6, len(args))
}
h0, err := mintSchema(t, args[0])
if err != nil {
@@ -711,7 +728,11 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
if err != nil {
return nil, fmt.Errorf("chs_preview_row: arg %d: %w", 4, err)
}
- o0, callErr := t.PreviewRow(h0, int32(v1), b2, b3, b4)
+ b5, err := argBytes(args[5])
+ if err != nil {
+ return nil, fmt.Errorf("chs_preview_row: arg %d: %w", 5, err)
+ }
+ o0, callErr := t.PreviewRow(h0, int32(v1), b2, b3, b4, b5)
res := &InvokeResult{Error: callErr}
if callErr != nil {
res.Status = callErr.Status
@@ -728,8 +749,8 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
}
return res, nil
case "chs_preview_batch":
- if len(args) != 8 {
- return nil, fmt.Errorf("chs_preview_batch: want %d args, got %d", 8, len(args))
+ if len(args) != 9 {
+ return nil, fmt.Errorf("chs_preview_batch: want %d args, got %d", 9, len(args))
}
h0, err := mintSchema(t, args[0])
if err != nil {
@@ -763,7 +784,11 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
if err != nil {
return nil, fmt.Errorf("chs_preview_batch: arg %d: %w", 7, err)
}
- o0, o1, callErr := t.PreviewBatch(h0, int32(v1), b2, b3, b4, h5, int32(v6), uint32(v7))
+ b8, err := argBytes(args[8])
+ if err != nil {
+ return nil, fmt.Errorf("chs_preview_batch: arg %d: %w", 8, err)
+ }
+ o0, o1, callErr := t.PreviewBatch(h0, int32(v1), b2, b3, b4, h5, int32(v6), uint32(v7), b8)
res := &InvokeResult{Error: callErr}
if callErr != nil {
res.Status = callErr.Status
@@ -787,8 +812,8 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
}
return res, nil
case "chs_filter_create":
- if len(args) != 4 {
- return nil, fmt.Errorf("chs_filter_create: want %d args, got %d", 4, len(args))
+ if len(args) != 5 {
+ return nil, fmt.Errorf("chs_filter_create: want %d args, got %d", 5, len(args))
}
h0, err := mintSchema(t, args[0])
if err != nil {
@@ -806,7 +831,11 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
if err != nil {
return nil, fmt.Errorf("chs_filter_create: arg %d: %w", 3, err)
}
- o0, callErr := t.FilterCreate(h0, b1, b2, b3)
+ b4, err := argBytes(args[4])
+ if err != nil {
+ return nil, fmt.Errorf("chs_filter_create: arg %d: %w", 4, err)
+ }
+ o0, callErr := t.FilterCreate(h0, b1, b2, b3, b4)
res := &InvokeResult{Error: callErr}
if callErr != nil {
res.Status = callErr.Status
@@ -826,8 +855,8 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
t.FilterFree(h0)
return &InvokeResult{}, nil
case "chs_filter_eval_body":
- if len(args) != 4 {
- return nil, fmt.Errorf("chs_filter_eval_body: want %d args, got %d", 4, len(args))
+ if len(args) != 6 {
+ return nil, fmt.Errorf("chs_filter_eval_body: want %d args, got %d", 6, len(args))
}
h0, err := mintFilter(t, args[0])
if err != nil {
@@ -845,7 +874,15 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
if err != nil {
return nil, fmt.Errorf("chs_filter_eval_body: arg %d: %w", 3, err)
}
- o0, callErr := t.FilterEvalBody(h0, int32(v1), b2, b3)
+ b4, err := argBytes(args[4])
+ if err != nil {
+ return nil, fmt.Errorf("chs_filter_eval_body: arg %d: %w", 4, err)
+ }
+ b5, err := argBytes(args[5])
+ if err != nil {
+ return nil, fmt.Errorf("chs_filter_eval_body: arg %d: %w", 5, err)
+ }
+ o0, callErr := t.FilterEvalBody(h0, int32(v1), b2, b3, b4, b5)
res := &InvokeResult{Error: callErr}
if callErr != nil {
res.Status = callErr.Status
@@ -862,8 +899,8 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
}
return res, nil
case "chs_block_create":
- if len(args) != 5 {
- return nil, fmt.Errorf("chs_block_create: want %d args, got %d", 5, len(args))
+ if len(args) != 6 {
+ return nil, fmt.Errorf("chs_block_create: want %d args, got %d", 6, len(args))
}
h0, err := mintSchema(t, args[0])
if err != nil {
@@ -885,7 +922,11 @@ func invoke(t *Table, fn string, args []Arg) (*InvokeResult, error) {
if err != nil {
return nil, fmt.Errorf("chs_block_create: arg %d: %w", 4, err)
}
- o0, callErr := t.BlockCreate(h0, int32(v1), b2, b3, b4)
+ b5, err := argBytes(args[5])
+ if err != nil {
+ return nil, fmt.Errorf("chs_block_create: arg %d: %w", 5, err)
+ }
+ o0, callErr := t.BlockCreate(h0, int32(v1), b2, b3, b4, b5)
res := &InvokeResult{Error: callErr}
if callErr != nil {
res.Status = callErr.Status
diff --git a/go/internal/abi2/linked_gen.go b/go/internal/abi2/linked_gen.go
index 433f02ad..20d06030 100644
--- a/go/internal/abi2/linked_gen.go
+++ b/go/internal/abi2/linked_gen.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//go:build chtypes_linked
@@ -34,6 +34,7 @@ static void chtypes_abi2_linked_fill(struct chtypes_abi2_table *t) {
t->chs_error_ch_name = (chtypes_abi2_fn_chs_error_ch_name) &chs_error_ch_name;
t->chs_error_message = (chtypes_abi2_fn_chs_error_message) &chs_error_message;
t->chs_error_column = (chtypes_abi2_fn_chs_error_column) &chs_error_column;
+ t->chs_error_reason = (chtypes_abi2_fn_chs_error_reason) &chs_error_reason;
t->chs_error_free = (chtypes_abi2_fn_chs_error_free) &chs_error_free;
t->chs_live_handles = (chtypes_abi2_fn_chs_live_handles) &chs_live_handles;
t->chs_error_codes = (chtypes_abi2_fn_chs_error_codes) &chs_error_codes;
diff --git a/go/internal/ocifetch/abi_fingerprint_gen.go b/go/internal/ocifetch/abi_fingerprint_gen.go
index 6f648f8e..1681c3d7 100644
--- a/go/internal/ocifetch/abi_fingerprint_gen.go
+++ b/go/internal/ocifetch/abi_fingerprint_gen.go
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
package ocifetch
@@ -6,4 +6,4 @@ package ocifetch
// module speaks: the same value as the ABI layer's ChsAbiFingerprint. The dev
// channel names its alias tags with it, as 64 lowercase hex characters
// without the sha256: prefix (docs/guides/fetch-v1.md section 3).
-const DevABIFingerprint = "sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4"
+const DevABIFingerprint = "sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2"
diff --git a/include/v2/chtypes.h b/include/v2/chtypes.h
index a7d54e6a..ac9ef0b0 100644
--- a/include/v2/chtypes.h
+++ b/include/v2/chtypes.h
@@ -1,10 +1,10 @@
-/* GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT */
+/* GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT */
/*
* chtypes.h: the chtypes C ABI, generation 2.
*
* This header is generated from `spec/abi-v2/abi.json` by `scripts/abi-v1/gen.py --major 2`. Edit
* the description and regenerate; never edit this file. `docs/reference/abi-v2.md` is the normative
- * reference, and the rules generation 2 is written under (r1 to r7) are in it.
+ * reference, and the rules generation 2 is written under (r1 to r9) are in it.
*
* Stability. Generation 2 is UNSTABLE: its description is still being designed, so
* `CHS_ABI_FINGERPRINT` moves with every change until the description is locked. Nothing built from
@@ -29,8 +29,43 @@
* drops only the caller's reference, so any free order is safe and freeing NULL does nothing.
*
* Errors. A fallible call returns a `chs_status` and, through an optional last `chs_error **err`,
- * an owned error carrying the status, ClickHouse's own code and name, the message and the column.
- * The status alone carries the verdict, so `err` may be NULL.
+ * an owned error carrying the status, ClickHouse's own code and name, the message, the column and,
+ * for a decline or a library-side refusal, the reason: the catalog id (`chs_error_reason`,
+ * `spec/abi-v2/declines/declines.json`) a caller keys on instead of matching message text. The
+ * status alone carries the verdict, so `err` may be NULL.
+ *
+ * Codes. One code space serves the error and every document's `code` and `verdict_code`: a positive
+ * value is ClickHouse's own error code, as the server's table names it; `CHS_CODE_DECLINED` (-2) is
+ * this build's decline; and 0 is written only on a success (`accepted`, `accepted_poisoned`, `ok`,
+ * a `t` or `f` verdict). A refused or declined row, outcome or verdict never carries 0, and no
+ * other negative or library-private number is ever written. Every decline and library-side refusal
+ * also names its reason, the catalog id, in the error and in the document (`reason`,
+ * `verdict_reason`), and ids and the sentinel are stable once published (rule r4). A message is
+ * informative: once a reason is structured no message text is normative, the pinned texts of
+ * generation 2's earlier drafts included.
+ *
+ * Conditional members. A document's schema states the members it always carries (`required`). These
+ * members are present under a condition the schema cannot see, and each condition is a MUST on the
+ * library, held by a conformance case in the SDK's suites (named in brackets) that drives it both
+ * ways; a binding MAY check them and, when it does, fails closed on a violation.
+ *
+ * - **The export.** When `chs_preview_batch` is asked for an export (`export_format` is not
+ * `CHS_EXPORT_NONE`), exactly one of `export_declined` and `row_spans` is present; with
+ * `export_declined`, `export_declined_reason`, its catalog id; and with `row_spans`,
+ * `export_columns`; with no export asked for, none of them is [`batch.export_members_xor`].
+ * - **The filter.** A row's `verdict`, `verdict_code` and `verdict_err`, and the batch's
+ * `rows_passed` and `rows_cut`, are present exactly when the call carries a filter
+ * (`chs_preview_batch` with a non-NULL `filter`) [`batch.verdict_iff_filter`].
+ * - **The reason.** `reason` is present exactly where its `code` is `CHS_CODE_DECLINED` (a row, a
+ * batch, a filter result, an `errors[]` entry), and `verdict_reason` exactly where its
+ * `verdict_code` is [`doc.reason_iff_declined`].
+ * - **The site.** `refused_at` is present exactly when the outcome of a batch, a filter result or
+ * `chs_preview_row`'s row is a refusal or a decline (`rejected` or `unsupported`), and never for
+ * `accepted`, `accepted_poisoned`, `skipped` or `ok`; `refused_row` is present exactly when
+ * `refused_at` is `row`, and on a batch or a filter result it is less than `rows_read`. The rows
+ * of a batch carry neither: the batch's own say where [`doc.refused_at_iff_refused`].
+ * - **The verdicts.** A filter result's `verdicts` has exactly `rows_read` characters
+ * [`filter.verdicts_len_is_rows_read`].
*
* Threads. Every function has a thread class, listed in the reference. A handle is immutable once
* made, so every call that only reads handles is `shared`: any number of threads may call it at
@@ -38,6 +73,18 @@
* a handle that call uses. Process setup (`chs_initialize`, then `chs_set_defaults`) runs before
* the traffic it configures.
*
+ * Initialization. Every function has an init class, listed in the reference and in its own comment:
+ * `any`, valid before and after `chs_initialize`; or `required`, valid only once `chs_initialize`
+ * has succeeded. Until then a `required` call changes nothing and returns `CHS_INVALID_ARGUMENT`,
+ * ch_code 0, and when its arguments are otherwise valid its message starts `: call
+ * chs_initialize first`; it never crashes. The `any` calls are the handshake and the tombstone,
+ * `chs_initialize`, `chs_shutdown`, the error and buffer accessors, and every free. Each of them
+ * but `chs_initialize` returns no `chs_status`, so it has nothing to refuse with. The error and
+ * buffer accessors read the refusal itself (`chs_error_message` returns a `chs_buf`), and a free
+ * given NULL does nothing, which is all a server, schema, filter or block free can be given before
+ * `chs_initialize`, because nothing makes one. Every other call is `required`, unless its own
+ * section says it is valid before `chs_initialize`.
+ *
* Stacks. A call's answer doesn't depend on the calling thread's stack. Every call that returns a
* `chs_status` runs with an 8 MiB stack's worth of room: on the calling thread when half its stack,
* less what it has used, is between 3.75 MiB and 4 MiB (a default 8 MiB thread: glibc's, a Go cgo
@@ -64,20 +111,21 @@
* the file system is case-insensitive, so `DateLUT` there accepts a spelling such as `utc` that a
* Linux server refuses; that difference is documented, not patched.
*
- * Settings. Every settings input is a JSON object whose values are JSON strings, and a binding
- * never rewrites one. A call's settings are layered, highest first: the call's own `settings`; the
- * schema's own (`chs_schema_create`); its server's `settings` (`chs_server_create`); the defaults
- * set by `chs_set_defaults`; the build's own. A filter's WHERE takes the same layers, lowest first:
- * the build's own, `chs_set_defaults`, the server profile, the schema's own, and the filter's own
- * (`chs_filter_create`), with `session_timezone` at every layer staying on the zone path (Time
- * zones, above). One exception: a type gate in the schema's own settings binds the CREATE only, and
- * a WHERE's CAST validates under the session layers, every layer but the schema's own. Each setting
- * that reaches a WHERE is honored (applied to it), passed through (accepted and ignored), or
- * declined (`chs_filter_create` says how). A server resource limit (an `execution`-tier setting,
- * e.g. a small `max_columns_to_read`) can make the server refuse a query the library answers; that
- * is resource policy, not row access, and the library does not model it. A server's settings are
- * the ones its profile applies to every query, and they never change how a schema compiles. No call
- * reads a server's version or its changed settings itself: that discovery is a recorded gap in this
+ * Settings. Every settings input is a JSON object whose values are JSON strings, or the object form
+ * for bytes (Input values, below), and a binding never rewrites one. A call's settings are layered,
+ * highest first: the call's own `settings`; the schema's own (`chs_schema_create`); its server's
+ * `settings` (`chs_server_create`); the defaults set by `chs_set_defaults`; the build's own. A
+ * filter's WHERE takes the same layers, lowest first: the build's own, `chs_set_defaults`, the
+ * server profile, the schema's own, and the filter's own (`chs_filter_create`), with
+ * `session_timezone` at every layer staying on the zone path (Time zones, above). One exception: a
+ * type gate in the schema's own settings binds the CREATE only, and a WHERE's CAST validates under
+ * the session layers, every layer but the schema's own. Each setting that reaches a WHERE is
+ * honored (applied to it), passed through (accepted and ignored), or declined (`chs_filter_create`
+ * says how). A server resource limit (an `execution`-tier setting, e.g. a small
+ * `max_columns_to_read`) can make the server refuse a query the library answers; that is resource
+ * policy, not row access, and the library does not model it. A server's settings are the ones its
+ * profile applies to every query, and they never change how a schema compiles. No call reads a
+ * server's version or its changed settings itself: that discovery is a recorded gap in this
* generation, so a caller that wants a server's settings passes them explicitly, in a server
* profile, through `chs_set_defaults` or in each call's settings.
*
@@ -111,18 +159,38 @@
* value nested in another type (`Array(String)`, `Map`, `Tuple`, and the like) carries no
* `value_b64`: that is a recorded gap in this generation, and its rendering is the only form.
*
- * Input values. A JSON input (`settings`, `query_params`, an input document) cannot carry a value
- * that is not valid UTF-8 in this generation. The workaround is to inline the value in the SQL as
- * `unhex('')`, built from hex digits only, which keeps it injection-safe because hex digits
- * cannot break out of the quoted literal. A later version would add an object form, `{"":
- * {"b64": ...}}`, never a `_b64` member, which could collide with a real key.
+ * Input values. A value in a settings or query-parameters input (every `settings` and
+ * `query_params`, and a server profile's `settings`: the content `json_object_byte_values`) is a
+ * JSON string when its bytes are valid UTF-8, and otherwise the object `{"b64": ""}`, which carries the raw bytes: `{"format_csv_delimiter": {"b64": "/w=="}}` sets the one
+ * byte 0xFF. It is never a `_b64` member, which could collide with a real key. The two forms
+ * are the same value when the bytes are valid UTF-8, so a binding may send either, and it MUST send
+ * the object form for any other value. Any other object, and a value of any other JSON type, is
+ * `CHS_INVALID_ARGUMENT`, naming the key. A key never takes the object form: setting and
+ * query-parameter names are ClickHouse identifiers. Every JSON input is valid UTF-8 JSON (RFC
+ * 8259), so a raw byte that is not UTF-8 inside a JSON string is `CHS_INVALID_ARGUMENT`, and the
+ * object form is the one way to carry such a value. The other inputs take no object form: an
+ * options document's values are closed vocabularies, a server profile's `timezone` and `macros` are
+ * names a server's configuration spells, and the input column list carries a name that is not UTF-8
+ * as `name_b64`, by the rule for byte strings in JSON.
*
* Input documents. A counted input whose content is `input:` is a JSON object that the
* library validates against the document of that name (rule r1). A key the document does not define
* is `CHS_INVALID_ARGUMENT`, naming the key, and so is a document that is not a JSON object or a
* member of the wrong JSON type. Length 0 is `{}`, and a repeated member keeps its last value. The
* library never ignores a key, so a caller that sets an option a build does not implement is told
- * so instead of silently getting the old behavior.
+ * so instead of silently getting the old behavior. The whole document is validated before any
+ * member is acted on, so a document with a key it does not define is `CHS_INVALID_ARGUMENT` even
+ * when it also sets a member this build declines. A member a document defines but this build does
+ * not implement, set to any value but its default, is `CHS_DECLINED` with ch_code 0 and exactly the
+ * message `: sets to "", which this build does not implement;
+ * declined rather than answered` (for example `chs_preview_batch: insert_options sets verdict_basis
+ * to "export", which this build does not implement; declined rather than answered`), reason
+ * `option-not-implemented`, which is never a wrong answer. At a call that answers with a document
+ * (`chs_preview_row`, `chs_preview_batch`, `chs_filter_eval_body`, `chs_block_create` and
+ * `chs_filter_eval_block`), that decline is the document's, not a status: outcome `unsupported`,
+ * `code` `CHS_CODE_DECLINED`, `refused_at` `call` and `reason` `option-not-implemented`, with the
+ * status `CHS_OK` (`refusal_site`).
*
* Teardown. `chs_shutdown` stops what `chs_initialize` started. A loader never unloads a library:
* unloading or initializing an image again in one process is unsupported. Before `chs_initialize`,
@@ -131,6 +199,17 @@
* NULL. After `chs_shutdown` has run, every such call refuses the same way, its message saying that
* `chs_shutdown` has run. The handshake, `chs_abi_revision`, `chs_shutdown`, the buffer and error
* accessors and the frees answer at any time.
+ *
+ * Images. (Pending the confirmation runs with the time budget at 0, and the remaining arm64 cells
+ * at the default budget and at 0.) Any number of library images, one per ClickHouse line, may be
+ * loaded and used concurrently in one process, from any number of threads, each initialized once
+ * with its own zone, and every answer is the one a single image on a single thread gives. The one
+ * exception is the admission time budget (`chs_set_defaults`): it measures the evaluating thread's
+ * CPU time, and an evaluation that costs more than about half of the budget on an idle core may be
+ * declined (`CHS_CODE_DECLINED`, reason `admission-time-budget`) when the process's load raises its
+ * CPU time, never answered differently: above the budget on an idle core it always declines, below
+ * about half of it never. Its catalog entry states this load dependence. A conformance case drives
+ * two lines, then four, concurrently in one process.
*/
#ifndef CHTYPES_V2_H
#define CHTYPES_V2_H
@@ -164,7 +243,7 @@ extern "C" {
* spec/abi-v2/abi.json. chs_build_info()'s `abi_fingerprint` is this macro, copied; a loader
* compares the two byte for byte and refuses any difference.
*/
-#define CHS_ABI_FINGERPRINT "sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4"
+#define CHS_ABI_FINGERPRINT "sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2"
/*
* UNSTABLE: spec/abi-v2/abi.json declares `stability` "unstable". Generation 2 is still being
@@ -188,6 +267,18 @@ extern "C" {
* - process_serial: process-level: must not overlap any other call into the same library image.
*/
+/* --------------------------------------------------------------------------------- init classes */
+
+/*
+ * Every function below names its init class: whether it is valid before `chs_initialize`
+ * (Initialization, above).
+ *
+ * - any: valid before and after `chs_initialize`.
+ * - required: valid only once `chs_initialize` has succeeded: until then the call changes nothing
+ * and returns `CHS_INVALID_ARGUMENT`, ch_code 0, and with arguments that are otherwise valid its
+ * message starts `: call chs_initialize first`.
+ */
+
/* ------------------------------------------------------------------------- byte strings in JSON */
/*
@@ -204,19 +295,20 @@ extern "C" {
* The data-derived fields, by document:
*
* - schema_description: columns[].name, columns[].type, columns[].default_expression,
- * filter_declined_settings[].name, replicated.zookeeper_path, replicated.replica_name.
- * - row: err, verdict_err, partition_id, unknown_fields[].name, unsupported_settings[].name,
- * cols[].name, cols[].type, cols[].base, cols[].input, cols[].stored, cols[].ref,
- * cols[].ref_type, cols[].wire, computed[].name, computed[].stored, transformed[].column,
- * transformed[].input, transformed[].stored.
+ * columns[].reads[].name, filter_declined_settings[].name, replicated.zookeeper_path,
+ * replicated.replica_name.
+ * - row: err, verdict_err, partition_id, unknown_fields[].name, cols[].name, cols[].type,
+ * cols[].base, cols[].input, cols[].stored, cols[].ref, cols[].ref_type, cols[].wire,
+ * computed[].name, computed[].stored, transformed[].column, transformed[].input,
+ * transformed[].stored.
* - batch: err, export_declined, rows[].err, rows[].verdict_err, rows[].partition_id,
- * rows[].unknown_fields[].name, rows[].unsupported_settings[].name, rows[].cols[].name,
- * rows[].cols[].type, rows[].cols[].base, rows[].cols[].input, rows[].cols[].stored,
- * rows[].cols[].ref, rows[].cols[].ref_type, rows[].cols[].wire, rows[].computed[].name,
- * rows[].computed[].stored, rows[].transformed[].column, rows[].transformed[].input,
- * rows[].transformed[].stored, engine_rows[][].name, engine_rows[][].stored,
- * storage_transforms[].column, storage_transforms[].stored, at_merge[].column, at_merge[].stored,
- * transformed[].column, transformed[].input, transformed[].stored, unsupported_settings[].name,
+ * rows[].unknown_fields[].name, rows[].cols[].name, rows[].cols[].type, rows[].cols[].base,
+ * rows[].cols[].input, rows[].cols[].stored, rows[].cols[].ref, rows[].cols[].ref_type,
+ * rows[].cols[].wire, rows[].computed[].name, rows[].computed[].stored,
+ * rows[].transformed[].column, rows[].transformed[].input, rows[].transformed[].stored,
+ * engine_rows[][].name, engine_rows[][].stored, storage_transforms[].column,
+ * storage_transforms[].stored, at_merge[].column, at_merge[].stored, transformed[].column,
+ * transformed[].input, transformed[].stored, unsupported_settings[].name, export_columns[].name,
* framing.header.names[].name.
* - filter_result: err, unsupported_settings[].name, errors[].err.
* - discovery: columns[].name, columns[].type, columns[].default_kind,
@@ -242,9 +334,10 @@ extern "C" {
* - `timezone`: the server's own zone, the zone its tables bind. An empty string, or no member,
* means the image zone. The name is validated by ClickHouse's own `DateLUT`, which reads the
* library host's time zone database.
- * - `settings`: the settings the server's profile applies to every query, each value a JSON string,
- * never rewritten. The library accepts exactly what the server's own SET check accepts.
- * `session_timezone` here is the default session zone of the server's user.
+ * - `settings`: the settings the server's profile applies to every query, each value a JSON string
+ * or the object form for bytes (Input values), never rewritten. A `chtypes_*` knob is never one
+ * of them (`chs_set_defaults`). The library accepts exactly what the server's own SET check
+ * accepts. `session_timezone` here is the default session zone of the server's user.
* - `macros`: the server's macros, as its `system.macros` lists them (`macro`, `substitution`).
* Absent, they are unknown; present, even as `{}`, they are the complete set (see
* `chs_server_create`). A macro name that ClickHouse's own macro reader refuses (for example
@@ -258,12 +351,28 @@ extern "C" {
* "additionalProperties": false,
* "properties": {
* "timezone": {"type": "string"},
- * "settings": {"type": "object", "additionalProperties": {"type": "string"}},
+ * "settings": {
+ * "type": "object",
+ * "additionalProperties": {"oneOf": [{"type": "string"}, {"$ref": "#/$defs/bytes_value"}]}
+ * },
* "macros": {
* "type": "object",
* "propertyNames": {"minLength": 1},
* "additionalProperties": {"type": "string"}
* }
+ * },
+ * "$defs": {
+ * "bytes_value": {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "required": ["b64"],
+ * "properties": {
+ * "b64": {
+ * "type": "string",
+ * "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ * }
+ * }
+ * }
* }
* }
* ```
@@ -282,16 +391,215 @@ extern "C" {
*/
/*
- * input:schema_options: The options of `chs_schema_create` (rule r1). It defines no member yet, so
- * `{}` (or length 0) is the only valid document, and any key is `CHS_INVALID_ARGUMENT`, naming the
- * key. A choice about how one statement is compiled lands here, as a new member: for example,
- * compiling a server's own `create_table_query`, read back from the server, rather than a caller's
- * CREATE.
+ * input:schema_options: The options of `chs_schema_create` (rule r1): how one statement is
+ * compiled. Its member is `posture`, a `posture` value, and `{}` (or length 0) compiles in
+ * `create`, as before the member existed. A key it does not define is `CHS_INVALID_ARGUMENT`,
+ * naming the key, and so is a `posture` that is not one of its values, naming the value.
+ *
+ * - `create`: the statement is a `CREATE TABLE` a server would run now, with every check a server's
+ * CREATE runs, under the settings layers (`chs_schema_create`).
+ * - `load`: the statement is the stored definition of a table the server already holds, as that
+ * server reports it (`system.tables.create_table_query`, `SHOW CREATE TABLE`, or the table's
+ * metadata file), compiled as the server loads it when it starts. It is one switch, and
+ * ClickHouse's own: every check on the compile's path that reads the loading posture reads
+ * `FORCE_ATTACH` instead of `CREATE`, in ClickHouse's own code, and nothing else changes. So the
+ * checks a server applies only to a new table are not applied: the type gates
+ * (`allow_suspicious_low_cardinality_types`, `allow_suspicious_fixed_string_types` and the rest,
+ * at the compile and at every call on the schema), the codec, statistics and suspicious-index
+ * checks, the DEFAULT validation of a typed column, the TTL checks, the MergeTree settings'
+ * constraint and sanity checks, and the floating-point partition-key, nullable-key and
+ * deprecated-syntax gates. `flatten_nested` and `data_type_default_nullable` are not applied
+ * either: a stored definition is already flattened and typed. Which checks these are on each
+ * supported line is the artifact producer's enumeration from the vendored source; no list is kept
+ * here. A projection that a starting server skips because it no longer parses is declined,
+ * because whether a given server still holds it depends on that server's history. A Replicated
+ * engine's ZooKeeper path and replica name never change a row verdict, so under `load` a stored
+ * definition whose engine arguments read a macro the server's `macros` cannot expand (or carry
+ * none, as a Replicated database's table does) compiles as the server loads it:
+ * `schema_description` then carries no `replicated` member, and only an answer that depends on
+ * the path (none today) is declined.
+ *
+ * **The caller's obligation.** `load` asserts that the server holds a table defined by exactly this
+ * statement. The library cannot check that assertion, because it never connects to the server. A
+ * statement a server would refuse at `CREATE` compiles under `load`, and every answer on the schema
+ * is the answer for INSERTs into that stored table. If no server holds such a table, every answer
+ * describes a table that does not exist, and an accepted row is evidence of nothing. A statement a
+ * server reported for a table it holds (its `create_table_query`, `SHOW CREATE TABLE`, or its
+ * metadata file) MUST be compiled in `load`, and a statement a caller wrote or edited in `create`.
+ * A server's `create_table_query` compiled in `create` can be wrong in both directions, because the
+ * server created the table under its own CREATE's settings, which the statement does not carry: an
+ * over-reject where a gate was set (a `LowCardinality(UInt8)` column) and an over-accept where a
+ * shape setting was (a `Nested` column of a table created with `flatten_nested = 0`).
*
* An options document (rule r1). Its members, as JSON Schema:
*
* ```json
- * {"type": "object", "additionalProperties": false}
+ * {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "properties": {"posture": {"enum": ["create", "load"]}}
+ * }
+ * ```
+ */
+
+/*
+ * input:insert_options: The options of every call that models one INSERT of a body:
+ * `chs_preview_row`, `chs_preview_batch`, `chs_block_create` and `chs_filter_eval_body` (rule r1).
+ * Its members: `insertable_columns`, `column_defaults` and `verdict_basis`, `defer_defaults`.
+ *
+ * - `insertable_columns`: the columns an INSERT may supply, as name objects: the server's
+ * column-level INSERT grant (`GRANT INSERT(a, b) ON t`), checked as the server checks it, on the
+ * INSERT's header and never on the fields a body supplies. The header is the call's `columns`
+ * list, or with none the server's own no-list header: every ordinary and DEFAULT column, in
+ * declaration order (so a positional CSV, TSV, Values or JSONCompactEachRow body with no list
+ * supplies all of them). The check runs after the list's own refusals (an unknown or ALIAS
+ * column, 16; a duplicate, 15; a MATERIALIZED column, 44, unless
+ * `insert_allow_materialized_columns` is set, when it needs the grant like any other) and before
+ * any byte of the body is read. A header naming a column outside the set is refused 497
+ * (ACCESS_DENIED), with `refused_at` `call`, in every format and whatever the body holds, an
+ * empty or a malformed body included. So `columns` MUST be a subset of the set, and a caller
+ * whose set does not cover every ordinary and DEFAULT column passes `columns`, at every call that
+ * takes this document, `chs_filter_eval_body` included: with a list a call follows the listed
+ * INSERT's rules, and with none the no-list rule. A body field naming a column the header omits
+ * (a JSON key, a `WithNames` header field) is an unknown field, never a grant question: skipped,
+ * or refused 117 under `input_format_skip_unknown_fields` 0 (`refused_at` `row` for a JSON key,
+ * `body_read` for a header field), and never stored. An omitted column keeps its declared kind,
+ * exactly as with no set: a generator DEFAULT draws (`default_generated`), a clock DEFAULT
+ * resolves, a literal DEFAULT fills, a MATERIALIZED column computes and an EPHEMERAL column takes
+ * its type default. The export carries every stored non-computed column, those outside the set
+ * included (`document:batch`), so the INSERT of the export needs a grant that covers its column
+ * list. A Nested column is granted per subcolumn (`n.x`). The set is a grant, not a column list:
+ * like a grant it may name an EPHEMERAL or MATERIALIZED column, or a name the table lacks. The
+ * 497's message is ClickHouse's own, naming the user `default` and the table `default.t`, since a
+ * schema carries neither a user nor a table name; it is informative. Absent is every column.
+ * Implemented in the first build that carries this description.
+ * - `column_defaults`: per-call DEFAULT overrides, a list of `{name, value}` (each by the rule for
+ * byte strings in JSON). It is one rule: in a cell the server would fill from the column's
+ * DEFAULT, the call uses the caller's `value` instead. A supplied value wins: the override fills
+ * exactly the cells ClickHouse's own input format marks missing for that column under the call's
+ * format settings, as the vendored reader decides it, and no other: a field the record omits, a
+ * `null` for a non-Nullable column under `input_format_null_as_default` 1, and an empty CSV field
+ * under `input_format_csv_empty_as_default` 1. An empty JSON string `""` is a supplied value. A
+ * Nullable column's `null` stays `NULL` where the server keeps it (the implementing build
+ * confirms this against the server's own missing mark). Such a cell reads `default_override`, and
+ * everything that reads the column sees the value as an INSERT that supplied it would: a CHECK,
+ * the sorting, primary and partition keys, a skip index, a projection, a TTL, a MATERIALIZED or
+ * DEFAULT expression that reads it, and a filter. The export carries it as a stored, present
+ * value at the column's position in `export_columns`, so the INSERT of the export needs no
+ * override, and a column the override fills is not deferred in those rows (`defer_defaults`). An
+ * entry may name an ordinary or a DEFAULT column, and an EPHEMERAL column where the implementing
+ * build measures it equal to a supplied EPHEMERAL value; an entry that names an unknown column, a
+ * column that is not insertable, or a MATERIALIZED or ALIAS column is `CHS_INVALID_ARGUMENT` at
+ * `call`, naming `column_defaults[i]`. When `insertable_columns` is given, an entry's column MUST
+ * be in it, because the INSERT the override models names the column in its header; one outside it
+ * is the same `CHS_INVALID_ARGUMENT`. `value` is read in the body format's own field encoding, as
+ * if the record supplied this field (the implementing build confirms); a value that does not read
+ * as the column's type is `CHS_INVALID_ARGUMENT` at `call`, naming `column_defaults[i]`, before
+ * any byte of the body is read. Built before the lock: the build that implements it lists
+ * `insert_options.column_defaults` in `capabilities.features` (rule r9), and a build that does
+ * not declines it as Input documents (in the Preamble) says.
+ * - `verdict_basis`: `input`, the default, or `export`: what row a filter verdict answers for.
+ * Under `input`, a filter that reads a value the library drew or resolved itself is `d`
+ * (`drawn-value`), because the server would draw its own. Under `export`, the verdict answers for
+ * the row the export would store, so a drawn or resolved value is answerable, and the caller MUST
+ * insert the export. Reserved: declined at `export` until implemented.
+ * - A member whose value is a list or an object is refused `CHS_INVALID_ARGUMENT` when an entry
+ * does not match its shape, naming the key. `{}` (or length 0) is the INSERT these calls modeled
+ * before the document existed, byte for byte. Each member means the same at every call that takes
+ * the document. A key it does not define is `CHS_INVALID_ARGUMENT`, naming the key, and so is a
+ * value that is not one of the member's, naming the value; a member this build does not implement
+ * declines as Input documents (in the Preamble) says.
+ *
+ * - `defer_defaults`: `none`, the default, or `server` (public issue #479). `server` defers to the
+ * server each DEFAULT this build would otherwise decline because its value is the server's to
+ * compute (a generator or a server function the build does not admit). Such a column reads
+ * `default_deferred` in the rows that omit it, and the export omits it from those rows' lines.
+ * What reads it there is not answered (the deferred-column rule): a filter that reads it makes
+ * the row's verdict `d` (`verdict_code` `CHS_CODE_DECLINED`, `verdict_reason`
+ * `defer-defaults-reader`); a CHECK, a sorting, primary or partition key, a skip index, a
+ * projection, a TTL, or a MATERIALIZED or other DEFAULT expression that reads it declines the row
+ * (`unsupported`, `reason` `defer-defaults-reader`); and such a row carries no `partition_id`.
+ * Admitted generators keep drawing (`default_generated`), so a drawn id stays usable for
+ * deduplication. A body is mixed when its exported rows' lines would not all omit the same
+ * columns (some rows supply a deferred column and others omit it). The export has one column
+ * list, so a mixed body is declined, and only when an export is asked for: the batch is
+ * `unsupported`, `refused_at` `body_step`, `reason` `defer-defaults-mixed-body`; with no export
+ * asked for no line omits anything, and the batch is answered. A body whose rows all defer the
+ * same columns, or none, is never mixed: its export omits those columns from every line and from
+ * `export_columns`. Declined until implemented.
+ *
+ * An options document (rule r1). Its members, as JSON Schema:
+ *
+ * ```json
+ * {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "properties": {
+ * "defer_defaults": {"enum": ["none", "server"]},
+ * "insertable_columns": {"type": "array", "items": {"$ref": "#/$defs/named"}},
+ * "column_defaults": {"type": "array", "items": {"$ref": "#/$defs/column_default"}},
+ * "verdict_basis": {"enum": ["input", "export"]}
+ * },
+ * "$defs": {
+ * "named": {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "properties": {
+ * "name": {"type": "string"},
+ * "name_b64": {
+ * "type": "string",
+ * "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ * }
+ * },
+ * "allOf": [{"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]}]
+ * },
+ * "column_default": {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "properties": {
+ * "name": {"type": "string"},
+ * "name_b64": {
+ * "type": "string",
+ * "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ * },
+ * "value": {"type": "string"},
+ * "value_b64": {
+ * "type": "string",
+ * "pattern": "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$"
+ * }
+ * },
+ * "allOf": [
+ * {"oneOf": [{"required": ["name"]}, {"required": ["name_b64"]}]},
+ * {"oneOf": [{"required": ["value"]}, {"required": ["value_b64"]}]}
+ * ]
+ * }
+ * }
+ * }
+ * ```
+ */
+
+/*
+ * input:filter_options: The options of `chs_filter_create` (rule r1): how one filter is compiled.
+ * `{}` (or length 0) compiles as before the document existed. A key it does not define is
+ * `CHS_INVALID_ARGUMENT`, naming the key, and so is a value that is not one of the member's, naming
+ * the value; a member this build does not implement declines as Input documents (in the Preamble)
+ * says.
+ *
+ * - `param_encoding`: `escaped`, the default, or `raw`: how a query parameter's value is read.
+ * Under `escaped`, the value is decoded by ClickHouse's escaped-text reader, exactly as a server
+ * decodes an HTTP `param_` value, so a backslash, a tab or a newline must be escaped as that
+ * reader expects; the object form `{"b64": ...}` is decoded from base64 first and the bytes then
+ * read the same way. Under `raw`, the bytes are the value. Reserved: declined at `raw` until
+ * implemented (rule r9).
+ *
+ * An options document (rule r1). Its members, as JSON Schema:
+ *
+ * ```json
+ * {
+ * "type": "object",
+ * "additionalProperties": false,
+ * "properties": {"param_encoding": {"enum": ["escaped", "raw"]}}
+ * }
* ```
*/
@@ -361,11 +669,18 @@ typedef struct chs_block chs_block;
* would also give. `CHS_REJECTED` with ch_code 306, TOO_DEEP_RECURSION, never `CHS_INTERNAL`, is
* returned for a deep argument the call compiles (an expression, a type, or a settings, columns
* or options JSON), at `chs_set_defaults`, `chs_server_create`, `chs_schema_create`,
- * `chs_schema_describe`, `chs_preview_row`, `chs_preview_batch`, `chs_filter_create`,
- * `chs_filter_eval_body`, `chs_block_create`, `chs_filter_eval_block`, `chs_type_validate`,
- * `chs_reference_type` and `chs_discover_columns`. A body's depth refusal is reported in its
- * document (`CHS_OK`, with code 306), as every refused body is.
+ * `chs_filter_create`, `chs_type_validate`, `chs_reference_type` and `chs_discover_columns`. A
+ * call that answers with a document (`chs_preview_row`, `chs_preview_batch`,
+ * `chs_filter_eval_body`, `chs_block_create` and `chs_filter_eval_block`) returns a status other
+ * than `CHS_OK` only for caller misuse (`CHS_INVALID_ARGUMENT`) or a library bug
+ * (`CHS_INTERNAL`): every refusal or decline of the call itself, of its settings, its column list
+ * or its options, a deep argument (306) included, is reported in its document with `refused_at`
+ * `call`, and a body's refusal with `refused_at` `row` or `body_step`, as every refused body is.
+ * `chs_schema_describe` never refuses a schema `chs_schema_create` returned: the statement's
+ * depth is judged at compile.
* - `CHS_DECLINED`: this build will not answer; a server might accept. Never scored as agreement.
+ * The error names the reason (`chs_error_reason`). A call that answers with a document never
+ * returns it: its decline is the document's `unsupported` with `refused_at` `call`.
* - `CHS_INVALID_ARGUMENT`: caller misuse, such as a NULL pointer with a nonzero length, a
* wrong-kind, freed or cross-library handle, or a NULL required out-parameter.
* - `CHS_INTERNAL`: a guarded exception inside the library, which is a library bug.
@@ -418,6 +733,13 @@ typedef int32_t chs_format;
*/
#define CHS_EXPORT_NONE (-1)
+/*
+ * CHS_CODE_DECLINED: The code a declined row, outcome or verdict carries (Codes, in the Preamble):
+ * -2, outside ClickHouse's code table. It is a sentinel, stable under rule r4, and never a
+ * ClickHouse code.
+ */
+#define CHS_CODE_DECLINED (-2)
+
/*
* CHS_DOC_VALUES: A `doc_flags` bit: the per-row documents carry each column's stored value and
* provenance.
@@ -451,7 +773,7 @@ typedef int32_t chs_format;
* instead, at whichever step first resolves it — only this symbol's absence means the library is
* not such an artifact at all.
*
- * Class: handshake. Thread: any.
+ * Class: handshake. Thread: any. Init: any.
*/
CHS_API int32_t chs_abi_version(void);
@@ -476,7 +798,7 @@ CHS_API int32_t chs_abi_version(void);
* file's own size, digest and glibc floor are not in it, because the file cannot carry facts
* measured on itself; they are in the signed statement.
*
- * Class: handshake. Thread: any.
+ * Class: handshake. Thread: any. Init: any.
*/
CHS_API const char *chs_build_info(void);
@@ -486,7 +808,7 @@ CHS_API const char *chs_build_info(void);
* and spelling so a v0 binding reaches its own refusal. A binding of generation 1 or later reads
* `clickhouse_version` from `chs_build_info` instead.
*
- * Class: handshake. Thread: any.
+ * Class: handshake. Thread: any. Init: any.
*/
CHS_API const char *chs_clickhouse_version(void);
@@ -503,7 +825,7 @@ CHS_API const char *chs_clickhouse_version(void);
* it. A binding of generation 1 or later never calls it; the loader resolves it only to prove it is
* present.
*
- * Class: tombstone. Thread: any. Always returns CHS_ABI_REVISION_TOMBSTONE (1001).
+ * Class: tombstone. Thread: any. Init: any. Always returns CHS_ABI_REVISION_TOMBSTONE (1001).
*/
CHS_API int chs_abi_revision(void);
@@ -511,8 +833,8 @@ CHS_API int chs_abi_revision(void);
* chs_buf_data: The buffer's bytes. The pointer may be NULL when `chs_buf_len` is 0, and a binding
* never dereferences it then. A NULL, freed, wrong-kind or cross-library buffer gives NULL.
*
- * Class: api. Thread: shared. The pointer is borrowed from `buf` and valid until it is freed; copy
- * before freeing.
+ * Class: api. Thread: shared. Init: any. The pointer is borrowed from `buf` and valid until it is
+ * freed; copy before freeing.
*
* - `buf`: required.
*/
@@ -522,7 +844,7 @@ CHS_API const uint8_t *chs_buf_data(const chs_buf *buf);
* chs_buf_len: The buffer's length in bytes. A NULL, freed, wrong-kind or cross-library buffer
* gives 0.
*
- * Class: api. Thread: shared.
+ * Class: api. Thread: shared. Init: any.
*
* - `buf`: required.
*/
@@ -531,7 +853,7 @@ CHS_API size_t chs_buf_len(const chs_buf *buf);
/*
* chs_buf_free: Releases the caller's reference to a buffer. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `buf`: may be NULL.
*/
@@ -541,7 +863,7 @@ CHS_API void chs_buf_free(chs_buf *buf);
* chs_error_status: The error's status, never `CHS_OK` for an error a call set. A NULL, freed,
* wrong-kind or cross-library error gives `CHS_INVALID_ARGUMENT`.
*
- * Class: api. Thread: shared.
+ * Class: api. Thread: shared. Init: any.
*
* - `error`: required.
*/
@@ -551,7 +873,7 @@ CHS_API chs_status chs_error_status(const chs_error *error);
* chs_error_ch_code: ClickHouse's own error code, nonzero only when the status is `CHS_REJECTED`,
* and 0 otherwise or for an invalid error.
*
- * Class: api. Thread: shared.
+ * Class: api. Thread: shared. Init: any.
*
* - `error`: required.
*/
@@ -563,8 +885,8 @@ CHS_API int32_t chs_error_ch_code(const chs_error *error);
* `CHS_REJECTED`. Like the other two text accessors, it returns NULL only for a NULL or invalid
* error.
*
- * Class: api. Thread: shared. Returns a new chs_buf the caller owns and frees with chs_buf_free;
- * its content is guaranteed ASCII; NULL only for a NULL or invalid argument.
+ * Class: api. Thread: shared. Init: any. Returns a new chs_buf the caller owns and frees with
+ * chs_buf_free; its content is guaranteed ASCII; NULL only for a NULL or invalid argument.
*
* - `error`: required.
*/
@@ -573,11 +895,12 @@ CHS_API chs_buf *chs_error_ch_name(const chs_error *error);
/*
* chs_error_message: A new buffer holding the message: ClickHouse's own text, verbatim, for
* `CHS_REJECTED`, and the library's explanation otherwise. A message may quote input bytes, so it
- * is a byte string, not text.
+ * is a byte string, not text. It is informative: a caller keys on the status, the code and the
+ * reason (`chs_error_reason`), never on message text.
*
- * Class: api. Thread: shared. Returns a new chs_buf the caller owns and frees with chs_buf_free;
- * its content is a ClickHouse message: may quote input bytes, so may carry any byte; NULL only for
- * a NULL or invalid argument.
+ * Class: api. Thread: shared. Init: any. Returns a new chs_buf the caller owns and frees with
+ * chs_buf_free; its content is a ClickHouse message: may quote input bytes, so may carry any byte;
+ * NULL only for a NULL or invalid argument.
*
* - `error`: required.
*/
@@ -588,18 +911,32 @@ CHS_API chs_buf *chs_error_message(const chs_error *error);
* empty when there is none. A column name is a byte string: NUL and invalid UTF-8 are legal in a
* name.
*
- * Class: api. Thread: shared. Returns a new chs_buf the caller owns and frees with chs_buf_free;
- * its content is a ClickHouse column name: NUL and invalid UTF-8 are legal, so never assume UTF-8;
- * NULL only for a NULL or invalid argument.
+ * Class: api. Thread: shared. Init: any. Returns a new chs_buf the caller owns and frees with
+ * chs_buf_free; its content is a ClickHouse column name: NUL and invalid UTF-8 are legal, so never
+ * assume UTF-8; NULL only for a NULL or invalid argument.
*
* - `error`: required.
*/
CHS_API chs_buf *chs_error_column(const chs_error *error);
+/*
+ * chs_error_reason: A new buffer holding the reason: the catalog id
+ * (`spec/abi-v2/declines/declines.json`, ASCII) of a decline or a library-side refusal, empty when
+ * the error carries none (ClickHouse's own refusal, an internal error). It is the member a caller
+ * keys on, never the message. Like the other text accessors, it returns NULL only for a NULL or
+ * invalid error.
+ *
+ * Class: api. Thread: shared. Init: any. Returns a new chs_buf the caller owns and frees with
+ * chs_buf_free; its content is guaranteed ASCII; NULL only for a NULL or invalid argument.
+ *
+ * - `error`: required.
+ */
+CHS_API chs_buf *chs_error_reason(const chs_error *error);
+
/*
* chs_error_free: Releases the caller's reference to an error. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `error`: may be NULL.
*/
@@ -611,7 +948,8 @@ CHS_API void chs_error_free(chs_error *error);
* them and then asserts every count is zero, which is how all four bindings prove their finalizers
* release everything.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `out`: receives the JSON document `live_handles`, owned by the caller, freed with chs_buf_free;
* required.
@@ -626,7 +964,8 @@ CHS_API chs_status chs_live_handles(chs_buf **out, chs_error **err);
* caller that needs several lines asks each library. The name is v0's, with the v1 call shape; the
* tombstone keeps every v0 binding from reaching it.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `out`: receives the JSON document `error_code_table`, owned by the caller, freed with
* chs_buf_free; required.
@@ -639,8 +978,8 @@ CHS_API chs_status chs_error_codes(chs_buf **out, chs_error **err);
* tooling export for the artifact producer's build gates: no binding wraps it, and a loader
* resolves it only for presence.
*
- * Class: tooling. Thread: process_serial. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
- * CHS_INTERNAL.
+ * Class: tooling. Thread: process_serial. Init: required. Returns one of: CHS_OK,
+ * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `out`: receives guaranteed ASCII, owned by the caller, freed with chs_buf_free; required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
@@ -652,8 +991,8 @@ CHS_API chs_status chs_registered_families(chs_buf **out, chs_error **err);
* function per line, read off this build's own registry. A tooling export for the artifact
* producer's build gates, which derive the statelessness evidence from it; no binding wraps it.
*
- * Class: tooling. Thread: process_serial. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
- * CHS_INTERNAL.
+ * Class: tooling. Thread: process_serial. Init: required. Returns one of: CHS_OK,
+ * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `out`: receives guaranteed ASCII, owned by the caller, freed with chs_buf_free; required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
@@ -664,8 +1003,8 @@ CHS_API chs_status chs_function_flags(chs_buf **out, chs_error **err);
* chs_reference_type: The widened reference type this build pairs with a type expression. A tooling
* export; no binding wraps it.
*
- * Class: tooling. Thread: process_serial. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: tooling. Thread: process_serial. Init: required. Returns one of: CHS_OK, CHS_REJECTED,
+ * CHS_DECLINED, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `type_expr`, `type_expr_len`: SQL text (a statement, an expression, a type, a quoted name or
* literal): may carry any byte; length 0 is none, and the pointer may then be NULL.
@@ -694,8 +1033,8 @@ CHS_API chs_status chs_reference_type(
* before any other call; the list of type families the build must refuse is embedded when the
* library is built, so nothing else is passed in.
*
- * Class: api. Thread: process_once. Returns one of: CHS_OK, CHS_REJECTED, CHS_INVALID_ARGUMENT,
- * CHS_INTERNAL.
+ * Class: api. Thread: process_once. Init: any. Returns one of: CHS_OK, CHS_REJECTED,
+ * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `timezone`, `timezone_len`: a time zone name, validated by ClickHouse's own DateLUT; length 0
* means UTC; length 0 is none, and the pointer may then be NULL.
@@ -705,20 +1044,51 @@ CHS_API chs_status chs_initialize(const uint8_t *timezone, size_t timezone_len,
/*
* chs_set_defaults: Seeds the settings every later call starts from, as a JSON object whose values
- * are JSON strings (a binding never rewrites a value, a boolean included). Each successful call
- * replaces the previous defaults whole. A setting name the server would refuse is refused here,
- * with ClickHouse's own code and message, and nothing is committed. `session_timezone` here is the
- * default zone for later calls; it does not change the image zone that compiled types bind.
+ * are JSON strings, or the object form for bytes (Input values), and a binding never rewrites a
+ * value, a boolean included. Each successful call replaces the previous defaults whole. A setting
+ * name the server would refuse is refused here, with ClickHouse's own code and message, and nothing
+ * is committed. `session_timezone` here is the default zone for later calls; it does not change the
+ * image zone that compiled types bind.
+ *
+ * The library's own knobs are `chtypes_*` settings, and this is the only channel for the
+ * process-level ones. Each is an integer or a string given as a JSON string, and no server knows
+ * one, so a server profile refuses it (115) and a call's settings carry only the per-call ones. The
+ * knobs, their layer, type, default and effect: `chtypes_now_epoch_nanos` (per call, int64
+ * nanoseconds since the epoch, default the wall clock: pins the call's clock instant);
+ * `chtypes_clock_offset_nanos` (per call, int64, default 0: shifts the instant);
+ * `chtypes_max_clock_skew_nanos` (per call, int64, default 0: the skew budget a clock-reading
+ * expression may resolve under); `chtypes_default_eval_memory_bytes` (process, int64 bytes, default
+ * 268435456, 0 disables: the admission memory budget one DEFAULT, CHECK, TTL or filter evaluation
+ * may use); `chtypes_default_eval_cpu_nanos` (process, int64, default 1000000000, 0 disables: the
+ * admission time budget, the evaluating thread's CPU time, detected not interrupted; the earlier
+ * name `chtypes_default_eval_wall_nanos` is refused on v2 with `CHS_INVALID_ARGUMENT` naming this
+ * one); `chtypes_custom_settings_prefixes` (process, string, default none: the server's
+ * `custom_settings_prefixes`, comma-separated). A process knob in a call's settings is declined,
+ * named in `unsupported_settings` (`call-setting-process-only-key`), and a clock knob whose value
+ * is not an integer likewise (`call-setting-clock-value`). An evaluation that exceeds a budget is
+ * declined with that budget's own reason id, never answered: `admission-memory-budget` for the
+ * memory budget and `admission-time-budget` for the time budget; the earlier single id
+ * `admission-envelope` is retired, its catalog entry naming both successors (rule r4). The time
+ * budget is the evaluating thread's CPU time, not wall time. Each catalog entry states its load
+ * dependence. A memory-budget decline is deterministic (`measured`). A time-budget decline depends
+ * on load only near the budget: CPU time rose up to x1.03 under load on linux-arm64 and up to x2.0
+ * on linux-amd64 when two busy hardware threads share a core, so a time-budget decline is
+ * retry-stable when the evaluation costs more than the budget on an idle core and never happens
+ * under about half of it; between half and one times the budget the host's load can decide. Normal
+ * evaluations cost about 1000 times less than the 1 s default. a schema whose DEFAULT exceeds a
+ * budget at admission is declined at compile. The defaults are this description's, not a build's: a
+ * build reports nothing about them.
*
* Setup only. The defaults are immutable once this image has created its first `chs_schema` (a
* filter or a block needs one): from then on the call changes nothing and answers
* `CHS_INVALID_ARGUMENT`. No call ever reads defaults that change under it, so no binding needs a
* lock around them.
*
- * Class: api. Thread: process_serial. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: process_serial. Init: required. Returns one of: CHS_OK, CHS_REJECTED,
+ * CHS_DECLINED, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
* - `err`: may be NULL; set only when the status is not CHS_OK.
*/
@@ -730,7 +1100,7 @@ CHS_API chs_status chs_set_defaults(const uint8_t *settings, size_t settings_len
* again. Unloading the library, or initializing it again in the same process, is unsupported: a
* loader never unloads a library.
*
- * Class: api. Thread: process_serial.
+ * Class: api. Thread: process_serial. Init: any.
*/
CHS_API void chs_shutdown(void);
@@ -738,7 +1108,7 @@ CHS_API void chs_shutdown(void);
* chs_type_validate: Parses one type expression with ClickHouse's own parser and returns its
* canonical spelling, or ClickHouse's own refusal.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
* CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `type_expr`, `type_expr_len`: SQL text (a statement, an expression, a type, a quoted name or
@@ -758,7 +1128,8 @@ CHS_API chs_status chs_type_validate(
* every special byte escaped. A passthrough over the vendored function, and named after it; the
* name is a byte string.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `name`, `name_len`: a ClickHouse column name: NUL and invalid UTF-8 are legal, so never assume
* UTF-8; length 0 is none, and the pointer may then be NULL.
@@ -777,7 +1148,8 @@ CHS_API chs_status chs_back_quote(
* must be. Which names stay bare is a property of the ClickHouse release, so a caller that needs
* one answer for several releases asks each library.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `name`, `name_len`: a ClickHouse column name: NUL and invalid UTF-8 are legal, so never assume
* UTF-8; length 0 is none, and the pointer may then be NULL.
@@ -795,7 +1167,8 @@ CHS_API chs_status chs_back_quote_if_needed(
* chs_quote_string: A byte string spelled as a ClickHouse string literal by the vendored
* `quoteString`. The input may contain NUL; the answer escapes it.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `text`, `text_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
* the pointer may then be NULL.
@@ -834,18 +1207,20 @@ CHS_API chs_status chs_quote_string(
* The macros are the server's whole `system.macros` when `macros` is present, even as `{}`: a macro
* a Replicated engine's arguments name that the set lacks is then the server's own refusal
* (`CHS_REJECTED`, 139, NO_ELEMENTS_IN_CONFIG). When `macros` is absent the server's macros are
- * unknown, and a schema whose engine reads one is declined, as it is without a server. A build that
- * validates a profile's macros but does not yet apply them declines such a schema in both cases.
+ * unknown, and a schema whose engine reads one is declined, as it is without a server, except under
+ * `schema_options`' `posture` `load`, where a stored Replicated definition compiles without
+ * expanding them (`input:schema_options`). A build that validates a profile's macros but does not
+ * yet apply them declines such a schema in both cases.
*
* A setting this build's filters do not honor in a WHERE is no decline here, and the call succeeds.
* So does `chs_schema_create` on the server, because a profile never changes a schema compile: it
* lists the setting in the schema's `filter_declined_settings` (`schema_description`), at layer
* `server`.
*
- * The call reads no defaults and leaves `chs_set_defaults`' latch open. Like every call that is not
- * the handshake, it is valid only after `chs_initialize`.
+ * The call reads no defaults and leaves `chs_set_defaults`' latch open. Its init class is
+ * `required`, so before `chs_initialize` it refuses (Initialization, in the Preamble).
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
* CHS_INVALID_ARGUMENT, CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its
* options document, `options`.
*
@@ -868,7 +1243,7 @@ CHS_API chs_status chs_server_create(
* chs_server_free: Releases the caller's reference to a server. Schemas created on it keep it
* alive. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `server`: may be NULL.
*/
@@ -880,7 +1255,21 @@ CHS_API void chs_server_free(chs_server *server);
* `server`, under a profile of settings given as a JSON object of string values. A trailing
* semicolon is allowed. A second statement is refused the way ClickHouse's own parser refuses one
* in a single query (`CHS_REJECTED`, with its code and message), and a statement of any other kind
- * is `CHS_INVALID_ARGUMENT`.
+ * is `CHS_INVALID_ARGUMENT`, except under `posture` `load`: there a `CREATE VIEW`, `CREATE
+ * MATERIALIZED VIEW` or `CREATE DICTIONARY` statement, the stored definition of an object a server
+ * lists in `system.tables` and whose columns it lists in `system.columns`, is `CHS_DECLINED` with
+ * the reason `engine-not-modeled`, so a caller that compiles a server's stored definitions falls
+ * back for it as it does for an engine this build does not model. The `ATTACH` spellings of those
+ * kinds that a server refuses with 62 stay `CHS_REJECTED` with that code, as the server answers. An
+ * `ATTACH TABLE` statement is not one of the other kinds (`options`, below). The library models a
+ * table, not its database's engine: `ATTACH TABLE` with a column list and an engine but no UUID is
+ * accepted under `posture` `load`, which a server's Atomic database refuses (80), and under
+ * `create` it is declined (`attach-statement-under-create`).
+ *
+ * A table whose engine this build does not model (a `Distributed` or `Buffer` table, for example)
+ * is `CHS_DECLINED` with the reason `engine-not-modeled`, which is returned only when the engine
+ * itself is the cause: a decline for anything else in the statement carries that thing's own
+ * reason.
*
* `server` is the server the table is on (`chs_server_create`), or NULL. On a server, the schema's
* home zone is the server's `timezone`: its zone-less `DateTime` and `DateTime64` columns bind it,
@@ -905,8 +1294,14 @@ CHS_API void chs_server_free(chs_server *server);
* succeeds and lists it in the schema's `filter_declined_settings` (`schema_description`), and
* `chs_filter_create` on the schema declines.
*
- * `options` (`input:schema_options`) defines no member yet, so any key in it is
- * `CHS_INVALID_ARGUMENT`, naming the key.
+ * `options` (`input:schema_options`) sets the `posture` the statement compiles in: `create`, the
+ * default, or `load` for a server's own stored definition (see `input:schema_options`, and the
+ * caller's obligation there). A `posture` this build does not know, or any other key, is
+ * `CHS_INVALID_ARGUMENT`, naming it. An `ATTACH TABLE` statement is a stored definition, never a
+ * statement of another kind: under `create` it is `CHS_DECLINED`, ch_code 0, with exactly the
+ * message `chs_schema_create: an ATTACH TABLE statement is a stored definition; compile it with
+ * schema_options {"posture":"load"}; declined rather than answered`, and under `load` it compiles
+ * as the same definition spelled `CREATE TABLE`.
*
* A build that does not yet compile on a server declines a non-NULL `server`, and a schema whose
* compile needs its server's zone is declined on a host thread that already runs a ClickHouse query
@@ -917,14 +1312,15 @@ CHS_API void chs_server_free(chs_server *server);
* use it at once. Its types bind its home zone, never the `session_timezone` in its settings, which
* is the zone of this call and of the later calls on the schema that name none of their own.
*
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
* CHS_INVALID_ARGUMENT, CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its
* options document, `options`.
*
* - `server`: may be NULL.
* - `create_table`, `create_table_len`: SQL text (a statement, an expression, a type, a quoted name
* or literal): may carry any byte; length 0 is none, and the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
* - `options`, `options_len`: the input document `schema_options`; length 0 is `{}`, and the
* pointer may then be NULL.
@@ -946,7 +1342,7 @@ CHS_API chs_status chs_schema_create(
* chs_schema_free: Releases the caller's reference to a schema. Filters and blocks made from it
* keep it alive. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `schema`: may be NULL.
*/
@@ -956,10 +1352,14 @@ CHS_API void chs_schema_free(chs_schema *schema);
* chs_schema_describe: A JSON document describing the schema's columns, owned by the caller; it
* replaces v0's borrowed column accessors.
*
- * A deep input the server's stack check refuses is `CHS_REJECTED` with ch_code 306. Here the input
- * is the schema's own statement, which describing walks again; it is never `CHS_INTERNAL`.
+ * A schema compiled in `load` carries `"posture": "load"`; one compiled in `create` carries no
+ * `posture`.
+ *
+ * A schema `chs_schema_create` returned always describes: the walk this document needs runs at
+ * compile, so a deep statement is refused there (`CHS_REJECTED`, 306) and this call never refuses
+ * or declines a compiled schema. It is never `CHS_INTERNAL`.
*
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_INVALID_ARGUMENT,
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
* CHS_INTERNAL.
*
* - `schema`: required.
@@ -980,23 +1380,32 @@ CHS_API chs_status chs_schema_describe(const chs_schema *schema, chs_buf **out,
* skip, is rejected with the reader's own code, as a server's INSERT rejects it; what the reader
* skips, such as a trailing newline, is fine. A body holding no row, or more than one, is
* `CHS_INVALID_ARGUMENT`, naming the call: `chs_preview_batch` answers a body of many rows. The
- * row's document equals the single row of `chs_preview_batch`'s document for the same body.
+ * row's document equals the single row of `chs_preview_batch`'s document for the same body, and
+ * stands for the call: a refusal or decline of the call, its settings, its column list or its
+ * options is this document's `outcome` with `refused_at` `call`, never a status.
*
* An INSERT whose input block has no column at all (every insertable column EPHEMERAL, and no
* column list) is declined: a server answers it (code 90, EMPTY_LIST_OF_COLUMNS_PASSED) from a
* statement inside its INSERT interpreter that the library cannot call, and a decline is never a
* wrong answer.
*
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * `options` (`input:insert_options`) shapes the INSERT the call models, and `{}` is the INSERT it
+ * modeled before the document existed.
+ *
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its options document,
+ * `options`.
*
* - `schema`: required.
* - `body`, `body_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
* the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
* - `columns`, `columns_len`: a JSON array of column names, each an object carrying the name as
* byte_strings says; length 0 is none, and the pointer may then be NULL.
+ * - `options`, `options_len`: the input document `insert_options`; length 0 is `{}`, and the
+ * pointer may then be NULL.
* - `out`: receives the JSON document `row`, owned by the caller, freed with chs_buf_free;
* required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
@@ -1010,6 +1419,8 @@ CHS_API chs_status chs_preview_row(
size_t settings_len,
const uint8_t *columns,
size_t columns_len,
+ const uint8_t *options,
+ size_t options_len,
chs_buf **out,
chs_error **err);
@@ -1024,22 +1435,33 @@ CHS_API chs_status chs_preview_row(
* `unsupported`, filter rows `d`, and no export. The same call without the filter is unchanged.
* `export_format` is `CHS_EXPORT_NONE` or a `chs_format` the build can write; with an export that
* runs, `out_export` receives the accepted rows serialized once, an empty buffer when the batch
- * holds no row. When the export is declined (the document carries `export_declined`), `out_export`
- * is NULL, as it may be when no export is asked for, so a caller tells a declined export from an
- * empty one by `export_declined`, never by the buffer. `doc_flags` chooses which groups the per-row
- * documents carry.
- *
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * holds no row. When the export is declined (the document carries `export_declined` and
+ * `export_declined_reason`), `out_export` is NULL, as it may be when no export is asked for, so a
+ * caller tells a declined export from an empty one by `export_declined`, never by the buffer.
+ * `doc_flags` chooses which groups the per-row documents carry. `options` (`input:insert_options`)
+ * shapes the INSERT the call models, and `{}` is the INSERT it modeled before the document existed.
+ * `filter` must have been compiled on `schema`: a filter from another schema handle is
+ * `CHS_INVALID_ARGUMENT`, as it is at `chs_filter_eval_block`. The call returns a status other than
+ * `CHS_OK` only for misuse or an internal error: a decline of the call (the filter's zone differing
+ * from the batch's, `preview-batch-zone-mismatch`) and a refusal of its arguments are the
+ * document's, with `refused_at` `call` and `reason`. The export's truth table is in
+ * `document:batch`.
+ *
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its options document,
+ * `options`.
*
* - `schema`: required.
* - `body`, `body_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
* the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
* - `columns`, `columns_len`: a JSON array of column names, each an object carrying the name as
* byte_strings says; length 0 is none, and the pointer may then be NULL.
* - `filter`: may be NULL.
+ * - `options`, `options_len`: the input document `insert_options`; length 0 is `{}`, and the
+ * pointer may then be NULL.
* - `out`: receives the JSON document `batch`, owned by the caller, freed with chs_buf_free;
* required.
* - `out_export`: receives arbitrary bytes: a body, a string value, an export, owned by the caller,
@@ -1058,6 +1480,8 @@ CHS_API chs_status chs_preview_batch(
const chs_filter *filter,
int32_t export_format,
uint32_t doc_flags,
+ const uint8_t *options,
+ size_t options_len,
chs_buf **out,
chs_buf **out_export,
chs_error **err);
@@ -1065,8 +1489,16 @@ CHS_API chs_status chs_preview_batch(
/*
* chs_filter_create: Compiles a boolean SQL expression over the schema's columns, binding its query
* parameters (a JSON object of string values) by ClickHouse's own substitution, so a value is never
- * SQL text. `settings` is the profile the expression is compiled under, so `session_timezone` there
- * is the zone of its literals. Non-deterministic expressions are declined.
+ * SQL text. A parameter's value is decoded by ClickHouse's escaped-text reader, exactly as a server
+ * decodes an HTTP `param_` value (`input:filter_options`' `param_encoding`, `escaped`): `\b` is an
+ * escape, so the parameter `a\b` reads as `a`, the byte 0x08 and `b`, and a literal backslash is
+ * written `\\`; `\N` reads as the empty value; `\x41`, `\'` and `\0` decode as that reader decodes
+ * them; a raw tab or newline is `CHS_REJECTED` with 457 and a trailing backslash with 25, as a
+ * server refuses them; and the bytes an escape produces are kept, valid UTF-8 or not. The object
+ * form `{"b64": ...}` (Input values) is decoded from base64 first, and its bytes are then read the
+ * same way. `options` (`input:filter_options`) shapes the compile, and `{}` is the compile before
+ * the document existed. `settings` is the profile the expression is compiled under, so
+ * `session_timezone` there is the zone of its literals. Non-deterministic expressions are declined.
*
* The WHERE's settings are layered, lowest first: the build's own, `chs_set_defaults`, the schema's
* server profile, the schema's own settings, and then this call's `settings`; `session_timezone` at
@@ -1079,8 +1511,9 @@ CHS_API chs_status chs_preview_batch(
* each setting and its layer, in the list's order: `chs_filter_create: the settings this schema was
* compiled under set (), (), which this build's filters do not honor in a
* WHERE (listed in chs_schema_describe's filter_declined_settings); declined rather than answered`,
- * where each `` is the entry's `declined_layer` value. The entries are joined by a comma and
- * a space, in the list's order, and each name renders verbatim. A `name_b64` entry renders as
+ * where each `` is the entry's `declined_layer` value, with the reason
+ * `server-profile-setting`. The message is informative; the entries are joined by a comma and a
+ * space, in the list's order, and each name renders verbatim. A `name_b64` entry renders as
* `b64:`; it does not occur in practice, because a setting name ClickHouse does not know is
* refused earlier, with 115. The server and the schema were created all the same: no settings layer
* changes a schema compile.
@@ -1090,34 +1523,42 @@ CHS_API chs_status chs_preview_batch(
* `unsupported_settings`), and it now declines this call. A server-profile setting already declined
* it.
*
- * A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here,
- * and it is never in `schema_description`. A key in this build's honor set is applied to the WHERE;
- * a key passed through is accepted and ignored; and a key this build's filters do not honor is
- * reported at evaluation (`filter_result`'s `unsupported_settings`, every verdict `d`, and
- * `chs_preview_batch`'s). The WHERE runs under this call's layers only (the defaults, the server
- * profile, the schema's own settings and this call's), as a subscriber's SELECT in its own session
- * would. An evaluation's `settings` (`chs_filter_eval_body`, `chs_preview_batch` with a filter,
- * `chs_block_create`) are the INSERT session's: they govern parsing the body and never reach the
- * WHERE, so they are never named as WHERE settings. So a filter evaluated in the INSERT's own
- * session, a materialized view's WHERE for example, is created with that session's WHERE-tier
- * settings in its own `settings`. Every key still meets the server's own SET check first, so a name
- * the server does not know is `CHS_REJECTED` with ch_code 115.
+ * A key of this call's own `settings` other than `session_timezone` is never `CHS_DECLINED` here
+ * but for one case, and it is never in `schema_description`. The case is a cell of the line's
+ * `where-settings/documented-pathological.tsv`, a value under which the server refuses every query:
+ * set in this call's own `settings`, it is `CHS_DECLINED` with ch_code 0 and the reason
+ * `setting-server-refuses`, because no verdict under it could stand. A key in this build's honor
+ * set is applied to the WHERE; a key passed through is accepted and ignored; and a key this build's
+ * filters do not honor is reported at evaluation (`filter_result`'s `unsupported_settings`, every
+ * verdict `d`, and `chs_preview_batch`'s). The WHERE runs under this call's layers only (the
+ * defaults, the server profile, the schema's own settings and this call's), as a subscriber's
+ * SELECT in its own session would. An evaluation's `settings` (`chs_filter_eval_body`,
+ * `chs_preview_batch` with a filter, `chs_block_create`) are the INSERT session's: they govern
+ * parsing the body and never reach the WHERE, so they are never named as WHERE settings. So a
+ * filter evaluated in the INSERT's own session, a materialized view's WHERE for example, is created
+ * with that session's WHERE-tier settings in its own `settings`. Every key still meets the server's
+ * own SET check first, so a name the server does not know is `CHS_REJECTED` with ch_code 115.
*
* A filter under a per-call `session_timezone` answers as the artifact producer has measured
* against live servers, including how the zone in an evaluation's settings combines with the zone
* the filter was compiled under. Until a path is measured to match a server it declines, which is
* never a wrong answer. The filter holds a counted reference to the schema.
*
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
+ * CHS_INVALID_ARGUMENT, CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its
+ * options document, `options`.
*
* - `schema`: required.
* - `expr`, `expr_len`: SQL text (a statement, an expression, a type, a quoted name or literal):
* may carry any byte; length 0 is none, and the pointer may then be NULL.
- * - `query_params`, `query_params_len`: a JSON object whose values are JSON strings (settings,
+ * - `query_params`, `query_params_len`: a JSON object whose values are JSON strings, or the object
+ * `{"b64": ""}` for a value whose bytes are not valid UTF-8 (settings,
* query parameters); length 0 is none, and the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
+ * - `options`, `options_len`: the input document `filter_options`; length 0 is `{}`, and the
+ * pointer may then be NULL.
* - `out`: receives a new chs_filter, owned by the caller, freed with chs_filter_free; required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
*/
@@ -1129,13 +1570,15 @@ CHS_API chs_status chs_filter_create(
size_t query_params_len,
const uint8_t *settings,
size_t settings_len,
+ const uint8_t *options,
+ size_t options_len,
chs_filter **out,
chs_error **err);
/*
* chs_filter_free: Releases the caller's reference to a filter. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `filter`: may be NULL.
*/
@@ -1143,20 +1586,32 @@ CHS_API void chs_filter_free(chs_filter *filter);
/*
* chs_filter_eval_body: Evaluates the filter over every row of a body, returning one verdict per
- * row. `settings` governs parsing the body, and `session_timezone` there is this call's zone. When
- * a server would not accept an INSERT of the body, because `chs_preview_batch`'s verdict over the
- * same body and settings, with no filter, is not `accepted`, every verdict is `d` and the outcome
- * mirrors that verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts
- * and that batch verdict together.
- *
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * row. `settings` governs parsing the body, and `session_timezone` there is this call's zone.
+ * `columns` is the INSERT column list, a JSON array of name objects (`{name}` or `{name_b64}`), and
+ * empty for none, read exactly as `chs_preview_batch` reads it: with a list the call models the
+ * INSERT that names it, and with none the INSERT that names none. When a server would not accept an
+ * INSERT of the body, because `chs_preview_batch`'s verdict over the same body, settings and column
+ * list, with no filter, is not `accepted`, every verdict is `d` and the outcome mirrors that
+ * verdict (`filter_outcome`). The call reads an accepted body once, for its verdicts and that batch
+ * verdict together, and under any outcome but `ok` writes `d` for every verdict, answering a
+ * refusal or decline of the call itself in the document (`refused_at` `call`), never as a status.
+ * `options` (`input:insert_options`) is that INSERT's, read as `chs_preview_batch` reads it, so the
+ * verdict mirrored is `chs_preview_batch`'s over the same body, settings, columns and options.
+ *
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its options document,
+ * `options`.
*
* - `filter`: required.
* - `body`, `body_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
* the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
+ * - `columns`, `columns_len`: a JSON array of column names, each an object carrying the name as
+ * byte_strings says; length 0 is none, and the pointer may then be NULL.
+ * - `options`, `options_len`: the input document `insert_options`; length 0 is `{}`, and the
+ * pointer may then be NULL.
* - `out`: receives the JSON document `filter_result`, owned by the caller, freed with
* chs_buf_free; required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
@@ -1168,6 +1623,10 @@ CHS_API chs_status chs_filter_eval_body(
size_t body_len,
const uint8_t *settings,
size_t settings_len,
+ const uint8_t *columns,
+ size_t columns_len,
+ const uint8_t *options,
+ size_t options_len,
chs_buf **out,
chs_error **err);
@@ -1175,20 +1634,26 @@ CHS_API chs_status chs_filter_eval_body(
* chs_block_create: Parses a body once under the schema, for evaluating many filters over it.
* `settings` governs the parse, `session_timezone` there included, and `columns` is read as
* `chs_preview_row` reads it. The block holds a counted reference to the schema. It also records
- * `chs_preview_batch`'s verdict over the same body, settings and columns, with no filter, which
- * `chs_filter_eval_block` answers by. An accepted body is read once, for the block and that verdict
- * together.
+ * `chs_preview_batch`'s verdict over the same body, settings, columns and options, with no filter,
+ * which `chs_filter_eval_block` answers by; a refusal or decline of the call or the body is
+ * recorded there, so this call returns a status other than `CHS_OK` only for misuse or an internal
+ * error. An accepted body is read once, for the block and that verdict together. `options`
+ * (`input:insert_options`) is read as `chs_preview_batch` reads it.
*
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL. Growable (rule r1): an input it gains is a new member of its options document,
+ * `options`.
*
* - `schema`: required.
* - `body`, `body_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
* the pointer may then be NULL.
- * - `settings`, `settings_len`: a JSON object whose values are JSON strings (settings, query
+ * - `settings`, `settings_len`: a JSON object whose values are JSON strings, or the object `{"b64":
+ * ""}` for a value whose bytes are not valid UTF-8 (settings, query
* parameters); length 0 is none, and the pointer may then be NULL.
* - `columns`, `columns_len`: a JSON array of column names, each an object carrying the name as
* byte_strings says; length 0 is none, and the pointer may then be NULL.
+ * - `options`, `options_len`: the input document `insert_options`; length 0 is `{}`, and the
+ * pointer may then be NULL.
* - `out`: receives a new chs_block, owned by the caller, freed with chs_block_free; required.
* - `err`: may be NULL; set only when the status is not CHS_OK.
*/
@@ -1201,13 +1666,15 @@ CHS_API chs_status chs_block_create(
size_t settings_len,
const uint8_t *columns,
size_t columns_len,
+ const uint8_t *options,
+ size_t options_len,
chs_block **out,
chs_error **err);
/*
* chs_block_free: Releases the caller's reference to a block. Freeing NULL does nothing.
*
- * Class: api. Thread: handle_serial.
+ * Class: api. Thread: handle_serial. Init: any.
*
* - `block`: may be NULL.
*/
@@ -1217,17 +1684,16 @@ CHS_API void chs_block_free(chs_block *block);
* chs_filter_eval_block: Evaluates the filter over a parsed block. Over a body a server would
* accept, it gives the same answers `chs_filter_eval_body` gives for that body. Over one it would
* not (the verdict `chs_block_create` recorded is not `accepted`), the outcome mirrors that verdict
- * as for `chs_filter_eval_body`, and every verdict from the first refused row on is `d`. The first
- * refused row is the first row of that batch whose outcome is neither `accepted` nor `skipped`.
- * When the refusal belongs to no row (a whole-body step, a poisoned value, the call's own
- * settings), it is the first row of the block. The rows before it keep their evaluated verdicts. A
- * caller treats those as `d` too: any non-`ok` outcome, including `unsupported`, means every
- * verdict is `d`. It takes no settings: the filter brings the settings it was compiled under and
- * the block those it was parsed under. The filter and the block must come from the same schema; a
- * pair from two schemas is `CHS_INVALID_ARGUMENT`.
- *
- * Class: api. Thread: shared. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
- * CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * as for `chs_filter_eval_body`, and every verdict is written `d`, the rows before the refused one
+ * included: the two calls write the same document for the same body, and the rule that any non-`ok`
+ * outcome means every verdict is `d` holds in the data. `refused_at` and `refused_row` say where
+ * the refusal sat (`row` for a refused row, `body_step` for a whole-body step or a poisoned value,
+ * `call` for the block's own settings). It takes no settings: the filter brings the settings it was
+ * compiled under and the block those it was parsed under. The filter and the block must come from
+ * the same schema; a pair from two schemas is `CHS_INVALID_ARGUMENT`.
+ *
+ * Class: api. Thread: shared. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `filter`: required.
* - `block`: required.
@@ -1250,7 +1716,8 @@ CHS_API chs_status chs_filter_eval_block(
* `{table:String}`), which the caller binds when it runs the query; it selects exactly the
* `system.columns` fields `chs_discover_columns` reads, `FORMAT JSONEachRow`.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT, CHS_INTERNAL.
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_INVALID_ARGUMENT,
+ * CHS_INTERNAL.
*
* - `out`: receives SQL text (a statement, an expression, a type, a quoted name or literal): may
* carry any byte, owned by the caller, freed with chs_buf_free; required.
@@ -1263,7 +1730,7 @@ CHS_API chs_status chs_discover_query(chs_buf **out, chs_error **err);
* `chs_discover_query`'s query) with ClickHouse's own reader and returns the column declarations,
* formatted by ClickHouse's own formatter.
*
- * Class: api. Thread: any. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
+ * Class: api. Thread: any. Init: required. Returns one of: CHS_OK, CHS_REJECTED, CHS_DECLINED,
* CHS_INVALID_ARGUMENT, CHS_INTERNAL.
*
* - `rows`, `rows_len`: arbitrary bytes: a body, a string value, an export; length 0 is none, and
diff --git a/python/CHANGELOG.md b/python/CHANGELOG.md
index 09b151af..2128fdda 100644
--- a/python/CHANGELOG.md
+++ b/python/CHANGELOG.md
@@ -8,6 +8,13 @@ The four bindings in this repository are released together and give one answer,
### Added
+- `column_defaults` is covered by unit tests (ABI v2): the exact `insert_options` JSON for a UTF-8 value, a non-UTF-8 value (`value_b64`), an empty value (sent as `""`) and two entries in order, and the decode of a `default_override` cell (`Source.DEFAULT_OVERRIDE`, stored). The batches guide gains a usage section. No behavior changed.
+- `CODE_DECLINED` (ABI v2, -2): the code a declined row, outcome or verdict carries, generated from the description and exported from `chtypes`, as `bindings-v1.md` section 3 promises.
+- `CallError.reason` (ABI v2): the error's catalog id, read through `chs_error_reason` and passed through verbatim (ASCII, empty if none), on all four call classes.
+- Spec batch 2, the options documents (ABI v2; moves the dev fingerprint): keyword-only `insertable_columns`, `column_defaults` (a sequence of `ColumnDefault`), `verdict_basis` (`VerdictBasis`: `INPUT`, `EXPORT`) and `defer_defaults` (`DeferDefaults`: `NONE`, `SERVER`) build `insert_options` for `Schema.row`, `Schema.rows`, `Schema.parse_block` and `Filter.rows`; `param_encoding` (`ParamEncoding`: `ESCAPED`, `RAW`) builds `filter_options` for `Schema.compile_filter`; `posture` (`Posture`: `CREATE`, `LOAD`) builds `schema_options` for `Library.compile_table`. `insert_options_extra`, `filter_options_extra` and `schema_options_extra` pass a member the binding has no name for through as JSON text, so the library, which refuses an unknown key (rule r1), is the only judge. A document with no member set is not sent. `Filter.rows` takes `columns` (`chs_filter_eval_body` takes the column list).
+- Spec batch 2, a value that is not UTF-8 (ABI v2): a settings value, a query-parameter value, a `setup` default and a server profile's settings value is sent as `{"b64": ...}` (the content `json_object_byte_values`), where the binding used to refuse it, and `SchemaServer.settings` decodes the same form back (as `surrogateescape` text, the spelling the input side takes).
+- Spec batch 2, the decline and refusal members (ABI v2): `RowResult`, `BatchResult` and `FilterResult` carry `reason` (the catalog id, present where the code is -2, `CHS_CODE_DECLINED`), `refused_at` (a `RefusalSite`: `CALL`, `BODY_READ`, `ROW`, `BODY_STEP`) and `refused_row`; `RowResult.verdict_reason`; `FilterRowError.reason`; `BatchResult.export_columns` and `BatchResult.export_declined_reason`.
+- Spec batch 2, the schema description (ABI v2): `SchemaDescription.posture`, and `Column.default_support` (a `DefaultSupport`), `Column.default_support_reason` and `Column.reads`, reserved members a build reports when it implements them. `Source.DEFAULT_DEFERRED`, `Source.DEFAULT_OVERRIDE` (stored) and `DeclinedTier.SERVER_REFUSES` join the vocabularies.
- A fetch-only call (#492): `Registry.fetch(request) -> Resolved` resolves, fetches, verifies and installs the build a request names into the cache, as `chtypes fetch` does, and opens nothing: no library is loaded and the process setup is untouched, so it needs no `setup`. It honors the registry's fetch options (offline, the cache, and under the production channel the bases, the trust and the lock), and a concurrent `fetch` and open of one request share one fetch. The `Resolved` names the installed version, build, digests and library path; errors are the fetch codes' classes.
- `chtypes resolve [--json]` (#493): what a line or version resolves to on every platform the registry offers, one `resolved ` line each, or with `--json` one JSON array of `{platform, version, build, manifest}` objects, each signed statement verified exactly as a fetch verifies it and no layer downloaded; `--offline` answers from the cache. The four CLIs print the same bytes, so one parser reads them (a CI cache key, for example).
- `chtypes prune [--line ] [--keep ] [--dry-run]` (#494): removes the installed builds of each line and platform that at least n newer installed builds supersede (n defaults to 1), each with its record, its blobs and its `index.json` entry; `--dry-run` prints exactly what would go. It never removes a build a running process holds: a registry now holds every build it opens or fetches, with a shared `flock` on the entry's `verified.json` until the process exits, and prune removes an entry only after taking the exclusive lock without waiting, reporting it `in-use` otherwise. It never touches a system directory, nor, on the dev channel, another fingerprint's builds. The rules are `docs/guides/fetch-v1.md` sections 1 and 3, and the `resolve-build-*` and `prune-*` conformance cases hold all four bindings to them.
@@ -27,6 +34,10 @@ The four bindings in this repository are released together and give one answer,
### Changed
+- **Breaking (ABI v2 dev): `BatchResult.spans` is a tuple of `Span | None`**, one entry per `rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which reads as `None`. It was the exported rows' spans only.
+- **Breaking (ABI v2 dev): `AtMergeEntry.row` is `AtMergeEntry.part_row`**, the document's `part_row`.
+- A document's `code` can be -2 (`CHS_CODE_DECLINED`) for a row, batch, filter result or verdict this build declined, and every decline names its `reason`.
+- The refusal of a settings, query-parameter, default or server-settings value that is not valid UTF-8 is gone (it is the `{"b64": ...}` form now); a key that is not valid UTF-8, a macro and a server timezone are still refused, as a `UsageError` naming the input.
- `chtypes list` and the tag listing name the published lines only (two-part, numeric order), as Go's always did; `chtypes resolve ` prints the exact version a line resolves to (#613).
- **Breaking: the public surface the parity gate carried as exceptions is resolved, before the v2 lock (#499, #500).** A convenience exists in every binding or in none (`docs/reference/bindings-v1.md` §1, principle 2). Migration, old → new:
- `chtypes.errors.internal` and `chtypes.errors.misuse` → private (#499): they were the binding's own construction helpers, never documented. Catch `InternalError` and `UsageError`, as before.
@@ -71,6 +82,7 @@ The four bindings in this repository are released together and give one answer,
### Removed
+- **Breaking (ABI v2 dev): `RowResult.unsupported_settings`**: the row document no longer carries `unsupported_settings`; the batch's and the filter result's lists stay.
- `SchemaServer.filter_declined_settings` (a breaking change inside v2 dev, #588): the server member no longer carries the list. Read `SchemaDescription.filter_declined_settings`, whose `server`-layer entries are the profile's.
### Fixed
diff --git a/python/README.md b/python/README.md
index bfe9e872..a28f41b7 100644
--- a/python/README.md
+++ b/python/README.md
@@ -77,7 +77,7 @@ A bad **row** is a verdict, not an exception: `outcome` becomes `Outcome.REJECTE
**`ctypes` releases the GIL for the whole of a foreign call**, so calls on compiled handles run in parallel across threads. The package takes no lock around a call; it keeps a close guard per handle and one setup guard.
-**The INSERT column list is a keyword-only `columns`** on `schema.row`, `schema.rows` and `schema.parse_block`: `schema.row(Format.JSON_EACH_ROW, body, columns=["id", "e"])`. `None` or an empty sequence is the no-list shape; a list makes the data supply exactly those columns, with a listed `EPHEMERAL` value read and in scope for the DEFAULTs that reference it but never stored.
+**The INSERT column list is a keyword-only `columns`** on `schema.row`, `schema.rows`, `schema.parse_block` and `filter.rows`: `schema.row(Format.JSON_EACH_ROW, body, columns=["id", "e"])`. `None` or an empty sequence is the no-list shape; a list makes the data supply exactly those columns, with a listed `EPHEMERAL` value read and in scope for the DEFAULTs that reference it but never stored.
**Known gaps in 1.0** are listed, with workarounds, in [`docs/limitations.md`](https://github.com/wave-rf/chtypes/blob/main/docs/limitations.md#known-gaps-in-10).
diff --git a/python/src/chtypes/__init__.py b/python/src/chtypes/__init__.py
index 1201d31b..49685997 100644
--- a/python/src/chtypes/__init__.py
+++ b/python/src/chtypes/__init__.py
@@ -47,22 +47,34 @@
from importlib.metadata import version as _package_version
from ._abi2._vocab import (
+ CODE_DECLINED,
EXPORT_NONE,
DeclinedLayer,
DeclinedTier,
DefaultKind,
+ DefaultSupport,
DiscoverQueryParam,
DocFlags,
FilterOutcome,
Format,
MergeReason,
Outcome,
+ Posture,
Reason,
+ RefusalSite,
Source,
Status,
Verdict,
)
-from ._input import BytesIn, ServerProfile, Settings
+from ._input import (
+ BytesIn,
+ ColumnDefault,
+ DeferDefaults,
+ ParamEncoding,
+ ServerProfile,
+ Settings,
+ VerdictBasis,
+)
from ._setup import setup
from .errors import (
CODE_ARTIFACT_CORRUPT,
@@ -143,6 +155,7 @@
"CODE_SOURCE_RETIRED",
"CODE_SOURCE_UNAUTHORIZED",
"CODE_SOURCE_UNREACHABLE",
+ "CODE_DECLINED",
"EXPORT_NONE",
"Advisory",
"ArtifactCorruptError",
@@ -163,11 +176,14 @@
"Capabilities",
"ChtypesError",
"Column",
+ "ColumnDefault",
"Computed",
"DeclinedLayer",
"DeclinedSetting",
"DeclinedTier",
+ "DeferDefaults",
"DefaultKind",
+ "DefaultSupport",
"DiscoverQueryParam",
"DiscoveredColumn",
"Discovery",
@@ -187,8 +203,11 @@
"Library",
"MergeReason",
"Outcome",
+ "ParamEncoding",
+ "Posture",
"Reason",
"Registry",
+ "RefusalSite",
"Resolved",
"Revocation",
"RowResult",
@@ -213,6 +232,7 @@
"UsageError",
"Value",
"Verdict",
+ "VerdictBasis",
"cache_root",
"open_unverified",
"search_dirs",
diff --git a/python/src/chtypes/_abi2/_decls.py b/python/src/chtypes/_abi2/_decls.py
index cd1d4312..33bf02a7 100644
--- a/python/src/chtypes/_abi2/_decls.py
+++ b/python/src/chtypes/_abi2/_decls.py
@@ -1,4 +1,4 @@
-# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT # noqa: E501
+# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT # noqa: E501
"""The Python binding's generated ABI v2 layer: ctypes signatures, the handle
classes, the two-phase resolve functions and invoke_by_name(), the test-only
dispatcher the conformance suite drives from tests/fixtures/abi-v2/cases.json.
@@ -19,7 +19,7 @@
from . import _errmap
CHS_ABI_VERSION = 2
-CHS_ABI_FINGERPRINT = "sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4"
+CHS_ABI_FINGERPRINT = "sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2"
# The description's stability: "unstable" while this generation is being
# designed, so CHS_ABI_FINGERPRINT moves with every change, and "locked" after
@@ -87,6 +87,7 @@
"chs_error_column",
"chs_error_free",
"chs_error_message",
+ "chs_error_reason",
"chs_error_status",
"chs_filter_create",
"chs_filter_eval_block",
@@ -248,6 +249,7 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_add("chs_error_ch_name", "handle", "chs_buf", _P("error", "handle", "chs_error"))
_add("chs_error_message", "handle", "chs_buf", _P("error", "handle", "chs_error"))
_add("chs_error_column", "handle", "chs_buf", _P("error", "handle", "chs_error"))
+_add("chs_error_reason", "handle", "chs_buf", _P("error", "handle", "chs_error"))
_add("chs_error_free", "void", None, _P("error", "handle", "chs_error", nullable=True))
_add(
"chs_live_handles",
@@ -371,6 +373,7 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_P("body", "bytes_in", None),
_P("settings", "bytes_in", None),
_P("columns", "bytes_in", None),
+ _P("options", "bytes_in", None),
_P("out", "out_handle", "chs_buf"),
_P("err", "out_error", "chs_error", nullable=True),
)
@@ -386,6 +389,7 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_P("filter", "handle", "chs_filter", nullable=True),
_P("export_format", "scalar", "int32"),
_P("doc_flags", "scalar", "uint32"),
+ _P("options", "bytes_in", None),
_P("out", "out_handle", "chs_buf"),
_P("out_export", "out_handle", "chs_buf", nullable=True),
_P("err", "out_error", "chs_error", nullable=True),
@@ -398,6 +402,7 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_P("expr", "bytes_in", None),
_P("query_params", "bytes_in", None),
_P("settings", "bytes_in", None),
+ _P("options", "bytes_in", None),
_P("out", "out_handle", "chs_filter"),
_P("err", "out_error", "chs_error", nullable=True),
)
@@ -410,6 +415,8 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_P("format", "enum", "chs_format"),
_P("body", "bytes_in", None),
_P("settings", "bytes_in", None),
+ _P("columns", "bytes_in", None),
+ _P("options", "bytes_in", None),
_P("out", "out_handle", "chs_buf"),
_P("err", "out_error", "chs_error", nullable=True),
)
@@ -422,6 +429,7 @@ def _add(name: str, returns_kind: str, returns_type: str | None, *params: _P) ->
_P("body", "bytes_in", None),
_P("settings", "bytes_in", None),
_P("columns", "bytes_in", None),
+ _P("options", "bytes_in", None),
_P("out", "out_handle", "chs_block"),
_P("err", "out_error", "chs_error", nullable=True),
)
@@ -563,6 +571,7 @@ def _check(self, status: int, err: ctypes.c_void_p | None) -> None:
info.ch_name.decode("ascii", "replace") if info else "",
message,
info.column if info else b"",
+ reason=info.reason.decode("ascii", "replace") if info else "",
)
def live_handles(self) -> bytes:
@@ -801,6 +810,7 @@ def preview_row(
body: bytes | None,
settings: bytes | None,
columns: bytes | None,
+ options: bytes | None,
) -> bytes:
"""The generated call wrapper for chs_preview_row (thread class shared).
@@ -813,6 +823,7 @@ def preview_row(
*_bytes_in(body),
*_bytes_in(settings),
*_bytes_in(columns),
+ *_bytes_in(options),
ctypes.byref(out),
ctypes.byref(err),
)
@@ -829,6 +840,7 @@ def preview_batch(
filter: FilterHandle | None,
export_format: int,
doc_flags: int,
+ options: bytes | None,
) -> tuple[bytes, bytes | None]:
"""The generated call wrapper for chs_preview_batch (thread class shared).
@@ -845,6 +857,7 @@ def preview_batch(
None if filter is None else filter.value,
int(export_format),
int(doc_flags),
+ *_bytes_in(options),
ctypes.byref(out),
ctypes.byref(out_export),
ctypes.byref(err),
@@ -861,6 +874,7 @@ def filter_create(
expr: bytes | None,
query_params: bytes | None,
settings: bytes | None,
+ options: bytes | None,
) -> FilterHandle:
"""The generated call wrapper for chs_filter_create (thread class shared).
@@ -872,6 +886,7 @@ def filter_create(
*_bytes_in(expr),
*_bytes_in(query_params),
*_bytes_in(settings),
+ *_bytes_in(options),
ctypes.byref(out),
ctypes.byref(err),
)
@@ -884,6 +899,8 @@ def filter_eval_body(
format: int,
body: bytes | None,
settings: bytes | None,
+ columns: bytes | None,
+ options: bytes | None,
) -> bytes:
"""The generated call wrapper for chs_filter_eval_body (thread class shared).
@@ -895,6 +912,8 @@ def filter_eval_body(
int(format),
*_bytes_in(body),
*_bytes_in(settings),
+ *_bytes_in(columns),
+ *_bytes_in(options),
ctypes.byref(out),
ctypes.byref(err),
)
@@ -908,6 +927,7 @@ def block_create(
body: bytes | None,
settings: bytes | None,
columns: bytes | None,
+ options: bytes | None,
) -> BlockHandle:
"""The generated call wrapper for chs_block_create (thread class shared).
@@ -920,6 +940,7 @@ def block_create(
*_bytes_in(body),
*_bytes_in(settings),
*_bytes_in(columns),
+ *_bytes_in(options),
ctypes.byref(out),
ctypes.byref(err),
)
@@ -1007,6 +1028,7 @@ class ErrorInfo:
ch_name: bytes
message: bytes
column: bytes
+ reason: bytes = b""
@dataclass
@@ -1068,6 +1090,7 @@ def _decode_error(api: Api, err_ptr: int) -> ErrorInfo:
ch_name = _read_buf(api, api._raw["chs_error_ch_name"](err_ptr))
message = _read_buf(api, api._raw["chs_error_message"](err_ptr))
column = _read_buf(api, api._raw["chs_error_column"](err_ptr))
+ reason = _read_buf(api, api._raw["chs_error_reason"](err_ptr))
api._raw["chs_error_free"](err_ptr)
return ErrorInfo(
status=STATUS_BY_VALUE.get(status_val, f"UNKNOWN:{status_val}"),
@@ -1075,6 +1098,7 @@ def _decode_error(api: Api, err_ptr: int) -> ErrorInfo:
ch_name=ch_name,
message=message,
column=column,
+ reason=reason,
)
diff --git a/python/src/chtypes/_abi2/_errmap.py b/python/src/chtypes/_abi2/_errmap.py
index 664cd253..c9c32fba 100644
--- a/python/src/chtypes/_abi2/_errmap.py
+++ b/python/src/chtypes/_abi2/_errmap.py
@@ -1,4 +1,4 @@
-# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT # noqa: E501
+# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT # noqa: E501
"""spec/abi-v2/sdk.json's error-mapping tables, as references to the
public classes (python/src/chtypes/errors.py, re-exported by
python/src/chtypes/_abi2/_errors.py). Only the MAPPING is generated: the
diff --git a/python/src/chtypes/_abi2/_vocab.py b/python/src/chtypes/_abi2/_vocab.py
index 039cc061..cab9a22f 100644
--- a/python/src/chtypes/_abi2/_vocab.py
+++ b/python/src/chtypes/_abi2/_vocab.py
@@ -1,4 +1,4 @@
-# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT # noqa: E501
+# GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT # noqa: E501
"""The Python binding's generated vocabularies, from spec/abi-v2/abi.json: the C
enums (`Format`, `Status`), the document vocabularies (`Outcome`, `FilterOutcome`,
`Verdict`, `DefaultKind`, `MergeReason`, `DeclinedTier`, `DeclinedLayer`, `Reason`,
@@ -256,6 +256,7 @@ class DeclinedTier(StrEnum):
PREDICATE = "predicate"
PREDICATE_UNFLIPPED = "predicate-unflipped"
RESULT_CONTENT = "result-content"
+ SERVER_REFUSES = "server-refuses"
@classmethod
def _missing_(cls, value: object) -> DeclinedTier | None:
@@ -299,6 +300,83 @@ def of(cls, text: str) -> DeclinedLayer:
return cls(text)
+class Posture(StrEnum):
+ """How a schema's statement was compiled: `create`, as a server's CREATE TABLE,
+ or `load`, as a server loads the stored definition of a table it holds."""
+
+ CREATE = "create"
+ LOAD = "load"
+
+ @classmethod
+ def _missing_(cls, value: object) -> Posture | None:
+ return _unknown_str(cls, value)
+
+ @property
+ def known(self) -> bool:
+ """Whether the description lists this value: False for exactly the unknown(n)
+ member an unlisted value reads as (rule r3)."""
+ return type(self)._value2member_map_.get(self._value_) is self
+
+ @classmethod
+ def of(cls, text: str) -> Posture:
+ """Map a document's spelling. One the description does not list is kept as
+ this vocabulary's unknown(n) member, carrying it (rule r3)."""
+ return cls(text)
+
+
+class RefusalSite(StrEnum):
+ """Where a batch or filter document's refusal sits: `call` (the call, its settings or its
+ column list), `body_read` (the body outside any row: a header, the framing), `row` (one
+ row, `refused_row`) or `body_step` (a whole-body step)."""
+
+ CALL = "call"
+ BODY_READ = "body_read"
+ ROW = "row"
+ BODY_STEP = "body_step"
+
+ @classmethod
+ def _missing_(cls, value: object) -> RefusalSite | None:
+ return _unknown_str(cls, value)
+
+ @property
+ def known(self) -> bool:
+ """Whether the description lists this value: False for exactly the unknown(n)
+ member an unlisted value reads as (rule r3)."""
+ return type(self)._value2member_map_.get(self._value_) is self
+
+ @classmethod
+ def of(cls, text: str) -> RefusalSite:
+ """Map a document's spelling. One the description does not list is kept as
+ this vocabulary's unknown(n) member, carrying it (rule r3)."""
+ return cls(text)
+
+
+class DefaultSupport(StrEnum):
+ """What this build does with a column's DEFAULT, as `schema_description` reports it per
+ column: `computed`, `generated`, `deferred` or `declined` (with a reason)."""
+
+ COMPUTED = "computed"
+ GENERATED = "generated"
+ DEFERRED = "deferred"
+ DECLINED = "declined"
+
+ @classmethod
+ def _missing_(cls, value: object) -> DefaultSupport | None:
+ return _unknown_str(cls, value)
+
+ @property
+ def known(self) -> bool:
+ """Whether the description lists this value: False for exactly the unknown(n)
+ member an unlisted value reads as (rule r3)."""
+ return type(self)._value2member_map_.get(self._value_) is self
+
+ @classmethod
+ def of(cls, text: str) -> DefaultSupport:
+ """Map a document's spelling. One the description does not list is kept as
+ this vocabulary's unknown(n) member, carrying it (rule r3)."""
+ return cls(text)
+
+
class DiscoverQueryParam(StrEnum):
"""A query parameter of the SQL `discover_query` returns, which the caller binds
when it runs the query."""
@@ -409,6 +487,8 @@ class Source:
DEFAULT_EXPR_UNSUPPORTED: Final = "default_expr_unsupported"
DEFAULT_VOLATILE_UNRESOLVED: Final = "default_volatile_unresolved"
DEFAULT_PENDING: Final = "default_pending"
+ DEFAULT_DEFERRED: Final = "default_deferred"
+ DEFAULT_OVERRIDE: Final = "default_override"
@staticmethod
def known(source: str) -> bool:
@@ -417,9 +497,8 @@ def known(source: str) -> bool:
@staticmethod
def is_stored(source: str) -> bool:
- """The description's `is_stored` fact for `source`. The description does not yet
- say what an unknown(n) source reports (rule r3, "Facts without a fallback");
- until it does, an unlisted source is not stored."""
+ """The description's `is_stored` fact for `source`; an unknown(n) source
+ reports its fallback's (`skipped`), as rule r3 says."""
return _SOURCE_IS_STORED.get(source, False)
@@ -435,6 +514,8 @@ def is_stored(source: str) -> bool:
"default_expr_unsupported": False,
"default_volatile_unresolved": False,
"default_pending": False,
+ "default_deferred": False,
+ "default_override": True,
}
@@ -450,6 +531,10 @@ class DocFlags(IntFlag):
# `chs_preview_batch`'s `export_format` for no export (`CHS_EXPORT_NONE`).
EXPORT_NONE: Final = -1
+# The code a declined row, outcome or verdict carries (`CHS_CODE_DECLINED`), outside
+# ClickHouse's code table.
+CODE_DECLINED: Final = -2
+
# Every enum the description defines, by its own name, and the Python type that
# spells it: rule r3's tests reach every one from here, so a new enum is covered
# without a hand-kept list.
@@ -463,10 +548,13 @@ class DocFlags(IntFlag):
"batch_outcome": Outcome,
"filter_outcome": FilterOutcome,
"filter_verdict": Verdict,
+ "refusal_site": RefusalSite,
"discover_query_param": DiscoverQueryParam,
"default_kind": DefaultKind,
+ "default_support": DefaultSupport,
"declined_tier": DeclinedTier,
"declined_layer": DeclinedLayer,
+ "posture": Posture,
}
diff --git a/python/src/chtypes/_decode.py b/python/src/chtypes/_decode.py
index 76879430..d494dabd 100644
--- a/python/src/chtypes/_decode.py
+++ b/python/src/chtypes/_decode.py
@@ -32,10 +32,13 @@
DeclinedLayer,
DeclinedTier,
DefaultKind,
+ DefaultSupport,
FilterOutcome,
MergeReason,
Outcome,
+ Posture,
Reason,
+ RefusalSite,
Source,
Verdict,
)
@@ -291,6 +294,20 @@ def _spans(obj: Mapping[str, Any], key: str, where: str) -> tuple[Span, ...]:
return _objects(obj, key, where, _span)
+def _row_spans(obj: Mapping[str, Any], key: str, where: str) -> tuple[Span | None, ...]:
+ """`row_spans`: one entry per row, a span or null for a row not exported."""
+ out: list[Span | None] = []
+ for i, item in enumerate(_list(obj, key, where)):
+ out.append(None if item is None else _span(_obj(item, where, key), f"{where}.{key}[{i}]"))
+ return tuple(out)
+
+
+def _refused_at(obj: Mapping[str, Any], where: str) -> RefusalSite | None:
+ # An unlisted site is its unknown(n): kept, never a failure (rule r3).
+ value = _opt_text(obj, "refused_at", where)
+ return None if value is None else RefusalSite.of(value)
+
+
def _value(obj: dict[str, Any], where: str) -> Value:
# An unlisted src is its unknown(n): kept, never a failure (rule r3), and
# not stored until the description says what one reports. An ABSENT src is
@@ -353,7 +370,7 @@ def _int_list(obj: Mapping[str, Any], key: str, where: str) -> tuple[int, ...] |
def _at_merge(obj: dict[str, Any], where: str) -> AtMergeEntry:
# An unlisted reason is its unknown(n): kept, never a failure (rule r3).
return AtMergeEntry(
- row=_int(obj, "row", where),
+ part_row=_int(obj, "part_row", where),
reason=MergeReason.of(_text(obj, "reason", where)),
column=_bytes_or_none(obj, "column", where),
stored=_bytes_or_none(obj, "stored", where),
@@ -386,14 +403,17 @@ def decode_row(obj: Mapping[str, Any], where: str = "row") -> RowResult:
outcome=Outcome.of(_text(obj, "outcome", where)),
err_code=_int(obj, "code", where),
err_msg=_bytes(obj, "err", where),
+ reason=_opt_text(obj, "reason", where),
+ refused_at=_refused_at(obj, where),
+ refused_row=_opt_u64(obj, "refused_row", where),
columns=columns,
values=tuple(v for v in columns if v.is_stored),
transformed=_objects(obj, "transformed", where, _transform),
unknown_fields=_name_list(obj, "unknown_fields", where),
- unsupported_settings=_name_list(obj, "unsupported_settings", where),
computed=_objects(obj, "computed", where, _computed),
verdict=None if verdict is None else Verdict.of(verdict),
verdict_code=_int(obj, "verdict_code", where),
+ verdict_reason=_opt_text(obj, "verdict_reason", where),
verdict_err=_bytes(obj, "verdict_err", where),
partition_id=partition,
input_span=_opt_span(obj, "input_span", where),
@@ -427,7 +447,10 @@ def decode_batch(raw: bytes, export: bytes | None) -> BatchResult:
obj = _obj(strict_loads(raw, where), where)
framing = obj.get("framing")
engine_rows = None if obj.get("engine_rows") is None else _engine_rows(obj, where)
- spans = None if obj.get("row_spans") is None else _spans(obj, "row_spans", where)
+ spans = None if obj.get("row_spans") is None else _row_spans(obj, "row_spans", where)
+ export_columns = (
+ None if obj.get("export_columns") is None else _name_list(obj, "export_columns", where)
+ )
export_declined = _bytes(obj, "export_declined", where)
if export_declined:
# A declined export has no payload and no spans, whatever buffer the call
@@ -439,6 +462,9 @@ def decode_batch(raw: bytes, export: bytes | None) -> BatchResult:
outcome=Outcome.of(_text(obj, "outcome", where)),
err_code=_int(obj, "code", where),
err_msg=_bytes(obj, "err", where),
+ reason=_opt_text(obj, "reason", where),
+ refused_at=_refused_at(obj, where),
+ refused_row=_opt_u64(obj, "refused_row", where),
rows=_objects(obj, "rows", where, decode_row),
rows_read=_u64(obj, "rows_read", where),
rows_skipped=_u64(obj, "rows_skipped", where),
@@ -446,7 +472,9 @@ def decode_batch(raw: bytes, export: bytes | None) -> BatchResult:
engine_rows=engine_rows,
payload=export,
spans=spans,
+ export_columns=export_columns,
export_declined=export_declined,
+ export_declined_reason=_opt_text(obj, "export_declined_reason", where),
rows_passed=_u64(obj, "rows_passed", where),
rows_cut=_u64(obj, "rows_cut", where),
partition_count=_opt_u64(obj, "partition_count", where),
@@ -467,6 +495,7 @@ def _filter_row_error(obj: dict[str, Any], where: str) -> FilterRowError:
return FilterRowError(
row=_int(obj, "row", where),
code=_int(obj, "code", where),
+ reason=_opt_text(obj, "reason", where),
msg=_bytes(obj, "err", where),
)
@@ -479,6 +508,9 @@ def decode_filter_result(raw: bytes) -> FilterResult:
outcome=FilterOutcome.of(_text(obj, "outcome", where)),
err_code=_int(obj, "code", where),
err_msg=_bytes(obj, "err", where),
+ reason=_opt_text(obj, "reason", where),
+ refused_at=_refused_at(obj, where),
+ refused_row=_opt_u64(obj, "refused_row", where),
rows_read=_u64(obj, "rows_read", where),
unsupported_settings=_name_list(obj, "unsupported_settings", where),
verdicts=tuple(Verdict.of(c) for c in verdicts),
@@ -487,12 +519,17 @@ def decode_filter_result(raw: bytes) -> FilterResult:
def _column(obj: dict[str, Any], where: str) -> Column:
- # An unlisted default_kind is its unknown(n): kept, never a failure (rule r3).
+ # An unlisted default_kind or default_support is its unknown(n): kept, never a
+ # failure (rule r3).
+ support = _opt_text(obj, "default_support", where)
return Column(
name=_name(obj, "name", where),
type=_bytes(obj, "type", where),
default_kind=DefaultKind.of(_text(obj, "default_kind", where)),
default_expr=_bytes(obj, "default_expression", where),
+ default_support=None if support is None else DefaultSupport.of(support),
+ default_support_reason=_opt_text(obj, "default_support_reason", where),
+ reads=None if obj.get("reads") is None else _name_list(obj, "reads", where),
)
@@ -506,6 +543,14 @@ def _string_map(obj: Mapping[str, Any], key: str, where: str) -> dict[str, str]
inner = _obj(value, where, key)
out: dict[str, str] = {}
for name, item in inner.items():
+ if isinstance(item, dict):
+ # The object form of a value that was not UTF-8: its bytes, spelled
+ # with surrogateescape (the spelling the input side takes back).
+ if set(item) != {"b64"}:
+ raise _wrong(f"{where}.{key}", name, "a string or a b64 object", item)
+ item = _decode_b64(item["b64"], f"{where}.{key}", name).decode(
+ "utf-8", "surrogateescape"
+ )
if not isinstance(item, str):
raise _wrong(f"{where}.{key}", name, "a string", item)
out[name] = item
@@ -543,6 +588,7 @@ def decode_schema_description(raw: bytes) -> SchemaDescription:
# decodes exactly as it did before servers existed.
server = obj.get("server")
replicated = obj.get("replicated")
+ posture = _opt_text(obj, "posture", where)
return SchemaDescription(
columns=_objects(obj, "columns", where, _column),
server=None
@@ -557,6 +603,7 @@ def decode_schema_description(raw: bytes) -> SchemaDescription:
filter_declined_settings=_objects(
obj, "filter_declined_settings", where, _declined_setting
),
+ posture=None if posture is None else Posture.of(posture),
)
diff --git a/python/src/chtypes/_input.py b/python/src/chtypes/_input.py
index 5ab6f8b0..4b7a3b06 100644
--- a/python/src/chtypes/_input.py
+++ b/python/src/chtypes/_input.py
@@ -13,10 +13,20 @@
import json
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
+from enum import StrEnum
+from ._abi2._vocab import Posture
from .errors import _misuse as misuse
-__all__ = ["BytesIn", "ServerProfile", "Settings"]
+__all__ = [
+ "BytesIn",
+ "ColumnDefault",
+ "DeferDefaults",
+ "ParamEncoding",
+ "ServerProfile",
+ "Settings",
+ "VerdictBasis",
+]
BytesIn = bytes | str
Settings = Mapping[str, str]
@@ -46,8 +56,8 @@ def body_bytes(value: bytes, what: str = "body") -> bytes:
_NON_UTF8_HINT = (
- "this ABI generation cannot carry a non-UTF-8 value in a JSON input; "
- "write the bytes as unhex('') in the expression instead"
+ "a JSON input cannot carry these bytes here "
+ '(only a settings or query-parameter value has the {"b64": ...} object form)'
)
@@ -70,7 +80,40 @@ def _refuse_non_utf8(what: str, part: str, key: str) -> Exception:
)
-def _string_map(mapping: Mapping[str, str], what: str) -> dict[str, str]:
+def _byte_value(text: str, what: str, key: str) -> str | dict[str, str]:
+ """One value of `json_object_byte_values`: the string itself when it is valid
+ UTF-8, else the object form `{"b64": }` of its bytes (a `str`
+ holding the lone surrogates `surrogateescape` makes of undecodable bytes
+ carries those bytes). A surrogate that is not one of those has no bytes to
+ carry and is refused."""
+ if _is_utf8(text):
+ return text
+ try:
+ raw = text.encode("utf-8", "surrogateescape")
+ except UnicodeEncodeError:
+ raise _refuse_non_utf8(what, "value", key) from None
+ return {"b64": base64.b64encode(raw).decode("ascii")}
+
+
+def _string_map(mapping: Mapping[str, str], what: str) -> dict[str, str | dict[str, str]]:
+ out: dict[str, str | dict[str, str]] = {}
+ for key, value in mapping.items():
+ if not isinstance(key, str) or not isinstance(value, str):
+ raise TypeError(
+ f"{what} maps strings to strings only; got {type(key).__name__} -> "
+ f"{type(value).__name__} for {key!r} (a value is never rewritten)"
+ )
+ # json.dumps would write a lone surrogate as a \udXXX escape, a key the
+ # library does not read as the bytes the caller meant: refuse.
+ if not _is_utf8(key):
+ raise _refuse_non_utf8(what, "key", key)
+ out[key] = _byte_value(value, what, key)
+ return out
+
+
+def _text_map(mapping: Mapping[str, str], what: str) -> dict[str, str]:
+ """A map whose values have no object form (a server's macros): a key or a
+ value that is not valid UTF-8 is refused."""
out: dict[str, str] = {}
for key, value in mapping.items():
if not isinstance(key, str) or not isinstance(value, str):
@@ -78,8 +121,6 @@ def _string_map(mapping: Mapping[str, str], what: str) -> dict[str, str]:
f"{what} maps strings to strings only; got {type(key).__name__} -> "
f"{type(value).__name__} for {key!r} (a value is never rewritten)"
)
- # json.dumps would write a lone surrogate as a \udXXX escape, a value
- # the library does not read as the bytes the caller meant: refuse.
if not _is_utf8(key):
raise _refuse_non_utf8(what, "key", key)
if not _is_utf8(value):
@@ -96,24 +137,25 @@ def settings_json(
settings: Settings | None,
session_timezone: str | None = None,
) -> bytes | None:
- """The settings object, as a JSON object of string values, or None for none.
+ """The settings object, as a JSON object of byte values (a string, or `{"b64": ...}`
+ for a value that is not UTF-8), or None for none.
`session_timezone` is written into it as that one key, verbatim. Giving it
alongside a settings key of the same name is a `UsageError`, whether or not
the two agree: a call never carries two spellings of its zone.
"""
- merged = {} if settings is None else _string_map(settings, "settings")
+ merged: dict[str, str | dict[str, str]] = (
+ {} if settings is None else _string_map(settings, "settings")
+ )
if session_timezone is not None:
if not isinstance(session_timezone, str):
raise TypeError(f"session_timezone must be str, not {type(session_timezone).__name__}")
- if not _is_utf8(session_timezone):
- raise misuse(f"the zone: the value is not valid UTF-8, and {_NON_UTF8_HINT}")
if _ZONE_KEY in merged:
raise misuse(
f"the {_ZONE_KEY} option and a {_ZONE_KEY} settings key were both given; "
"a call carries one spelling of its zone"
)
- merged[_ZONE_KEY] = session_timezone
+ merged[_ZONE_KEY] = _byte_value(session_timezone, "the zone", _ZONE_KEY)
return _dump(merged) if merged else None
@@ -124,6 +166,17 @@ def string_map_json(mapping: Mapping[str, str] | None, what: str) -> bytes | Non
return _dump(_string_map(mapping, what))
+def _name_objects(names: Sequence[BytesIn]) -> list[dict[str, str]]:
+ out = []
+ for name in names:
+ raw = to_bytes(name, "a column name")
+ try:
+ out.append({"name": raw.decode("utf-8")})
+ except UnicodeDecodeError:
+ out.append({"name_b64": base64.b64encode(raw).decode("ascii")})
+ return out
+
+
def columns_json(columns: Sequence[BytesIn] | None) -> bytes | None:
"""The INSERT column list: a JSON array of name objects, `{"name": ...}`
for a name that is valid UTF-8 and `{"name_b64": ...}` otherwise."""
@@ -133,14 +186,152 @@ def columns_json(columns: Sequence[BytesIn] | None) -> bytes | None:
raise TypeError("columns must be a sequence of names, not a single name")
if len(columns) == 0:
return None
- names = []
- for column in columns:
- raw = to_bytes(column, "a column name")
+ return _dump(_name_objects(columns))
+
+
+# ---- the options documents (rule r1). Each member is its own keyword, and a key
+# the binding has no name for goes through the `*_options_extra` mapping as JSON
+# text. The binding builds the document from the members the caller set, sends
+# nothing when none was set, and never validates a member: the library refuses a
+# key it does not know, or a value that is not one of the member's, with
+# CHS_INVALID_ARGUMENT, naming it.
+
+
+class VerdictBasis(StrEnum):
+ """`insert_options.verdict_basis`: what row a filter verdict answers for."""
+
+ INPUT = "input" # the default: the input row
+ EXPORT = "export" # the row the export would store; a build without it declines
+
+
+class DeferDefaults(StrEnum):
+ """`insert_options.defer_defaults`."""
+
+ NONE = "none" # the default: no DEFAULT is deferred
+ SERVER = "server" # defer to the server each DEFAULT the build would decline
+
+
+class ParamEncoding(StrEnum):
+ """`filter_options.param_encoding`: how a query parameter's value is read."""
+
+ ESCAPED = "escaped" # the default: ClickHouse's escaped-text reader
+ RAW = "raw" # the bytes are the value; a build without it declines
+
+
+@dataclass(frozen=True, slots=True)
+class ColumnDefault:
+ """One entry of `insert_options.column_defaults`: the value the call uses in
+ a cell the server would fill from the column's DEFAULT. Both fields are byte
+ strings (`str` is UTF-8 encoded); one that is not valid UTF-8 is sent in its
+ `_b64` form."""
+
+ name: BytesIn
+ value: BytesIn
+
+
+def _enum_text(value: object, what: str) -> str:
+ if isinstance(value, str): # a StrEnum member is a str: its value is the spelling
+ return value.value if isinstance(value, StrEnum) else value
+ raise TypeError(f"{what} must be a str or its enum, not {type(value).__name__}")
+
+
+def _column_default_objects(defaults: Sequence[ColumnDefault]) -> list[dict[str, str]]:
+ if isinstance(defaults, str | bytes):
+ raise TypeError("column_defaults must be a sequence of ColumnDefault")
+ out = []
+ for entry in defaults:
+ if not isinstance(entry, ColumnDefault):
+ raise TypeError(
+ f"column_defaults entries must be ColumnDefault, not {type(entry).__name__}"
+ )
+ item = _name_objects([entry.name])[0]
+ raw = to_bytes(entry.value, "a column default value")
try:
- names.append({"name": raw.decode("utf-8")})
+ item["value"] = raw.decode("utf-8")
except UnicodeDecodeError:
- names.append({"name_b64": base64.b64encode(raw).decode("ascii")})
- return _dump(names)
+ item["value_b64"] = base64.b64encode(raw).decode("ascii")
+ out.append(item)
+ return out
+
+
+def _options_json(
+ what: str, members: dict[str, object], extra: Mapping[str, str] | None
+) -> bytes | None:
+ """The document from the members set (value already JSON-encodable) and the
+ extra members (JSON text, spliced verbatim). A key set twice keeps the last
+ value, and an extra member is set after the named ones. Nothing set is None
+ (length 0, which the library reads as `{}`)."""
+ parts: dict[str, str] = {key: json.dumps(value) for key, value in members.items()}
+ if extra is not None:
+ if not isinstance(extra, Mapping):
+ raise TypeError(f"{what}_extra must map keys to JSON text")
+ for key, text in extra.items():
+ if not isinstance(key, str) or not isinstance(text, str):
+ raise TypeError(f"{what}_extra maps strings to JSON text only")
+ if not _is_utf8(key):
+ raise _refuse_non_utf8(f"{what} options", "key", key)
+ try:
+ json.loads(text)
+ except ValueError:
+ raise misuse(
+ f"{what} options: the value for key {json.dumps(key)} is not valid JSON"
+ ) from None
+ parts[key] = text
+ if not parts:
+ return None
+ body = ",".join(f"{json.dumps(key)}:{text}" for key, text in parts.items())
+ try:
+ return ("{" + body + "}").encode("utf-8")
+ except UnicodeEncodeError:
+ raise misuse(f"{what} options: a value is not valid UTF-8 JSON text") from None
+
+
+def insert_options_json(
+ *,
+ insertable_columns: Sequence[BytesIn] | None = None,
+ column_defaults: Sequence[ColumnDefault] | None = None,
+ verdict_basis: VerdictBasis | str | None = None,
+ defer_defaults: DeferDefaults | str | None = None,
+ insert_options_extra: Mapping[str, str] | None = None,
+) -> bytes | None:
+ """`insert_options`, from the members set. None leaves a member out; an empty
+ list is sent as `[]`."""
+ members: dict[str, object] = {}
+ if insertable_columns is not None:
+ if isinstance(insertable_columns, str | bytes):
+ raise TypeError("insertable_columns must be a sequence of names, not a single name")
+ members["insertable_columns"] = _name_objects(insertable_columns)
+ if column_defaults is not None:
+ members["column_defaults"] = _column_default_objects(column_defaults)
+ if verdict_basis is not None:
+ members["verdict_basis"] = _enum_text(verdict_basis, "verdict_basis")
+ if defer_defaults is not None:
+ members["defer_defaults"] = _enum_text(defer_defaults, "defer_defaults")
+ return _options_json("insert", members, insert_options_extra)
+
+
+def filter_options_json(
+ *,
+ param_encoding: ParamEncoding | str | None = None,
+ filter_options_extra: Mapping[str, str] | None = None,
+) -> bytes | None:
+ """`filter_options`, from the members set."""
+ members: dict[str, object] = {}
+ if param_encoding is not None:
+ members["param_encoding"] = _enum_text(param_encoding, "param_encoding")
+ return _options_json("filter", members, filter_options_extra)
+
+
+def schema_options_json(
+ *,
+ posture: Posture | str | None = None,
+ schema_options_extra: Mapping[str, str] | None = None,
+) -> bytes | None:
+ """`schema_options`, from the members set."""
+ members: dict[str, object] = {}
+ if posture is not None:
+ members["posture"] = _enum_text(posture, "posture")
+ return _options_json("schema", members, schema_options_extra)
@dataclass(frozen=True, slots=True)
@@ -181,5 +372,5 @@ def server_profile_json(profile: ServerProfile) -> bytes:
if profile.settings is not None:
doc["settings"] = _string_map(profile.settings, "server profile settings")
if profile.macros is not None:
- doc["macros"] = _string_map(profile.macros, "server profile macros")
+ doc["macros"] = _text_map(profile.macros, "server profile macros")
return _dump(doc)
diff --git a/python/src/chtypes/errors.py b/python/src/chtypes/errors.py
index c4ef7edb..5d28a4e7 100644
--- a/python/src/chtypes/errors.py
+++ b/python/src/chtypes/errors.py
@@ -103,6 +103,7 @@ class CallError(ChtypesError):
message: ClickHouse's own message for a refusal, the library's
otherwise; bytes.
column: the column concerned, empty if none; bytes.
+ reason: the catalog id (`chs_error_reason`), ASCII; empty if none.
"""
def __init__(
@@ -112,6 +113,7 @@ def __init__(
ch_name: str = "",
message: bytes = b"",
column: bytes = b"",
+ reason: str = "",
) -> None:
super().__init__(message.decode("utf-8", "replace"))
self.status = status
@@ -119,6 +121,7 @@ def __init__(
self.ch_name = ch_name
self.message = message
self.column = column
+ self.reason = reason
class SchemaError(CallError):
diff --git a/python/src/chtypes/library.py b/python/src/chtypes/library.py
index a3a4d152..61378c8f 100644
--- a/python/src/chtypes/library.py
+++ b/python/src/chtypes/library.py
@@ -13,20 +13,27 @@
import contextlib
import os
import threading
-from collections.abc import Iterator, Sequence
+from collections.abc import Iterator, Mapping, Sequence
from pathlib import Path
from types import TracebackType
from . import _decode, _setup
from ._abi2 import _decls, _loader
-from ._abi2._vocab import EXPORT_NONE, DocFlags, Format
+from ._abi2._vocab import EXPORT_NONE, DocFlags, Format, Posture
from ._guard import CloseGuard
from ._input import (
BytesIn,
+ ColumnDefault,
+ DeferDefaults,
+ ParamEncoding,
ServerProfile,
Settings,
+ VerdictBasis,
body_bytes,
columns_json,
+ filter_options_json,
+ insert_options_json,
+ schema_options_json,
server_profile_json,
settings_json,
string_map_json,
@@ -197,14 +204,30 @@ def rows(
*,
settings: Settings | None = None,
session_timezone: str | None = None,
+ columns: Sequence[BytesIn] | None = None,
+ insertable_columns: Sequence[BytesIn] | None = None,
+ column_defaults: Sequence[ColumnDefault] | None = None,
+ verdict_basis: VerdictBasis | str | None = None,
+ defer_defaults: DeferDefaults | str | None = None,
+ insert_options_extra: Mapping[str, str] | None = None,
) -> FilterResult:
"""Evaluate over a body. `settings` and `session_timezone` are the body's
- PARSE settings only: the filter's own zone was fixed when it compiled."""
+ PARSE settings only: the filter's own zone was fixed when it compiled.
+ `columns` is the INSERT column list and the keywords after it the
+ `insert_options` members, as for `Schema.rows`."""
payload = body_bytes(body)
settings_bytes = settings_json(settings, session_timezone)
+ columns_bytes = columns_json(columns)
+ options = insert_options_json(
+ insertable_columns=insertable_columns,
+ column_defaults=column_defaults,
+ verdict_basis=verdict_basis,
+ defer_defaults=defer_defaults,
+ insert_options_extra=insert_options_extra,
+ )
with _hold(self._guard):
raw = self._library._api.filter_eval_body(
- self._handle, int(format), payload, settings_bytes
+ self._handle, int(format), payload, settings_bytes, columns_bytes, options
)
return _decode.decode_filter_result(raw)
@@ -239,14 +262,30 @@ def row(
settings: Settings | None = None,
session_timezone: str | None = None,
columns: Sequence[BytesIn] | None = None,
+ insertable_columns: Sequence[BytesIn] | None = None,
+ column_defaults: Sequence[ColumnDefault] | None = None,
+ verdict_basis: VerdictBasis | str | None = None,
+ defer_defaults: DeferDefaults | str | None = None,
+ insert_options_extra: Mapping[str, str] | None = None,
) -> RowResult:
- """One row, as a server's INSERT would validate and coerce it."""
+ """One row, as a server's INSERT would validate and coerce it. The keywords
+ after `columns` are the `insert_options` members (rule r1), passed through
+ unvalidated: `insertable_columns`, `column_defaults`, `verdict_basis`,
+ `defer_defaults`, and `insert_options_extra` (JSON text by key) for a
+ member this binding has no name for."""
payload = body_bytes(body)
settings_bytes = settings_json(settings, session_timezone)
columns_bytes = columns_json(columns)
+ options = insert_options_json(
+ insertable_columns=insertable_columns,
+ column_defaults=column_defaults,
+ verdict_basis=verdict_basis,
+ defer_defaults=defer_defaults,
+ insert_options_extra=insert_options_extra,
+ )
with _hold(self._guard):
raw = self._library._api.preview_row(
- self._handle, int(format), payload, settings_bytes, columns_bytes
+ self._handle, int(format), payload, settings_bytes, columns_bytes, options
)
return _decode.decode_row_document(raw)
@@ -258,6 +297,11 @@ def rows(
settings: Settings | None = None,
session_timezone: str | None = None,
columns: Sequence[BytesIn] | None = None,
+ insertable_columns: Sequence[BytesIn] | None = None,
+ column_defaults: Sequence[ColumnDefault] | None = None,
+ verdict_basis: VerdictBasis | str | None = None,
+ defer_defaults: DeferDefaults | str | None = None,
+ insert_options_extra: Mapping[str, str] | None = None,
row_filter: Filter | None = None,
export: Format | int | None = None,
doc_flags: DocFlags = DocFlags.ALL,
@@ -268,6 +312,13 @@ def rows(
payload = body_bytes(body)
settings_bytes = settings_json(settings, session_timezone)
columns_bytes = columns_json(columns)
+ options = insert_options_json(
+ insertable_columns=insertable_columns,
+ column_defaults=column_defaults,
+ verdict_basis=verdict_basis,
+ defer_defaults=defer_defaults,
+ insert_options_extra=insert_options_extra,
+ )
if row_filter is not None and not isinstance(row_filter, Filter):
raise TypeError(f"row_filter must be a Filter, not {type(row_filter).__name__}")
guards = [self._guard] + ([] if row_filter is None else [row_filter._guard])
@@ -281,6 +332,7 @@ def rows(
None if row_filter is None else row_filter._handle,
EXPORT_NONE if export is None else int(export),
int(doc_flags),
+ options,
)
return _decode.decode_batch(raw, exported if export is not None else None)
@@ -291,16 +343,24 @@ def compile_filter(
params: Settings | None = None,
settings: Settings | None = None,
session_timezone: str | None = None,
+ param_encoding: ParamEncoding | str | None = None,
+ filter_options_extra: Mapping[str, str] | None = None,
) -> Filter:
"""Compile a boolean expression over this schema's columns. The filter's
zone is its own, fixed here: its WHERE runs in the session this call's
- `session_timezone` (or `settings`) names, on every evaluation."""
+ `session_timezone` (or `settings`) names, on every evaluation.
+ `param_encoding` and `filter_options_extra` (JSON text by key, for a
+ member this binding has no name for) are the `filter_options` members,
+ passed through unvalidated (rule r1)."""
expr_bytes = to_bytes(expr, "expr")
params_bytes = string_map_json(params, "query parameters")
settings_bytes = settings_json(settings, session_timezone)
+ options = filter_options_json(
+ param_encoding=param_encoding, filter_options_extra=filter_options_extra
+ )
with _hold(self._guard):
handle = self._library._api.filter_create(
- self._handle, expr_bytes, params_bytes, settings_bytes
+ self._handle, expr_bytes, params_bytes, settings_bytes, options
)
return Filter(self._library, handle)
@@ -312,14 +372,27 @@ def parse_block(
settings: Settings | None = None,
session_timezone: str | None = None,
columns: Sequence[BytesIn] | None = None,
+ insertable_columns: Sequence[BytesIn] | None = None,
+ column_defaults: Sequence[ColumnDefault] | None = None,
+ verdict_basis: VerdictBasis | str | None = None,
+ defer_defaults: DeferDefaults | str | None = None,
+ insert_options_extra: Mapping[str, str] | None = None,
) -> Block:
- """Parse a body once, for evaluating many filters over it."""
+ """Parse a body once, for evaluating many filters over it. The keywords
+ after `columns` are the `insert_options` members, as for `row`."""
payload = body_bytes(body)
settings_bytes = settings_json(settings, session_timezone)
columns_bytes = columns_json(columns)
+ options = insert_options_json(
+ insertable_columns=insertable_columns,
+ column_defaults=column_defaults,
+ verdict_basis=verdict_basis,
+ defer_defaults=defer_defaults,
+ insert_options_extra=insert_options_extra,
+ )
with _hold(self._guard):
handle = self._library._api.block_create(
- self._handle, int(format), payload, settings_bytes, columns_bytes
+ self._handle, int(format), payload, settings_bytes, columns_bytes, options
)
return Block(self._library, handle)
@@ -421,6 +494,8 @@ def compile_table(
settings: Settings | None = None,
session_timezone: str | None = None,
server: Server | None = None,
+ posture: Posture | str | None = None,
+ schema_options_extra: Mapping[str, str] | None = None,
) -> Schema:
"""Compile exactly one `CREATE TABLE` statement. `settings` is the profile;
a compiled type always takes the image zone, never the profile's.
@@ -431,17 +506,22 @@ def compile_table(
replica name expand the server's macros. A closed server is a
`UsageError`, raised before any call; a server from another library is
that library's own refusal, also a `UsageError`. `None` is no server: the
- schema is on the image's own server."""
+ schema is on the image's own server.
+
+ `posture` (`Posture.CREATE`, the default, or `Posture.LOAD`) and
+ `schema_options_extra` (JSON text by key, for a member this binding has no
+ name for) are the `schema_options` members, passed through unvalidated
+ (rule r1)."""
statement = to_bytes(create_table, "create_table")
settings_bytes = settings_json(settings, session_timezone)
if server is not None and not isinstance(server, Server):
raise TypeError(f"server must be a Server, not {type(server).__name__}")
- # No schema option is defined yet: none, which the library reads as `{}`.
+ options = schema_options_json(posture=posture, schema_options_extra=schema_options_extra)
# A closed server is refused here, before the call, because its freed
# handle would cross as NULL and the library would compile on the
# image's server instead of refusing.
with _hold(*(() if server is None else (server._guard,))):
handle = self._api.schema_create(
- None if server is None else server._handle, statement, settings_bytes, None
+ None if server is None else server._handle, statement, settings_bytes, options
)
return Schema(self, handle)
diff --git a/python/src/chtypes/results.py b/python/src/chtypes/results.py
index c08521f6..f8628259 100644
--- a/python/src/chtypes/results.py
+++ b/python/src/chtypes/results.py
@@ -18,9 +18,12 @@
DeclinedLayer,
DeclinedTier,
DefaultKind,
+ DefaultSupport,
FilterOutcome,
MergeReason,
Outcome,
+ Posture,
+ RefusalSite,
Verdict,
)
@@ -125,7 +128,7 @@ class AtMergeEntry:
`at_merge` list (ABI v2). `engine_rows` stays exactly what the INSERT writer
produces, so the two never contradict each other.
- `row` indexes `engine_rows`: the part's row, not the input body's. `reason` is
+ `part_row` indexes `engine_rows`: the part's row, not the input body's. `reason` is
a `MergeReason`; one the description does not list is its unknown(n) member
(rule r3). `column` is the column a `ttl_column_reset` resets (from `column`
or `column_b64`), None when the entry names none. `stored` is the column's
@@ -136,7 +139,7 @@ class AtMergeEntry:
the document carries none.
"""
- row: int
+ part_row: int
reason: MergeReason
column: bytes | None = None
stored: bytes | None = None
@@ -146,19 +149,32 @@ class AtMergeEntry:
@dataclass(frozen=True, slots=True)
class RowResult:
"""The `row` document. `columns` is every `cols` entry in document order;
- `values` is the subset whose `is_stored` is true, in order."""
+ `values` is the subset whose `is_stored` is true, in order.
+
+ `err_code` is the document's `code`: ClickHouse's own code, 0 only on a
+ success, and -2 (`CHS_CODE_DECLINED`) where this build declined the row.
+ `reason` is the decline's catalog id (ASCII), present exactly where `err_code`
+ is -2, else None; key on it, never on `err_msg`. `refused_at` is where a
+ single-row result's refusal sits (the rows of a batch carry neither this nor
+ `refused_row`: the batch's own say where), None for an accepted or skipped
+ row. `refused_row` is the row index of a `RefusalSite.ROW` refusal, else None.
+ `verdict_reason` is the catalog id of a declined verdict (`verdict_code` is
+ -2), else None."""
outcome: Outcome
err_code: int
err_msg: bytes
+ reason: str | None
+ refused_at: RefusalSite | None
+ refused_row: int | None
columns: tuple[Value, ...]
values: tuple[Value, ...]
transformed: tuple[Transform, ...]
unknown_fields: tuple[bytes, ...]
- unsupported_settings: tuple[bytes, ...]
computed: tuple[Computed, ...]
verdict: Verdict | None
verdict_code: int
+ verdict_reason: str | None
verdict_err: bytes
partition_id: bytes | None
input_span: Span | None
@@ -196,6 +212,16 @@ class BatchResult:
a body with any skipped row or any `unconsumed` range
(docs/guides/batches.md).
+ `reason`, `refused_at` and `refused_row` are the row's (see `RowResult`) for the
+ batch: the catalog id of a declined batch, and where a refusal or decline
+ sits, None when the document carries none. `spans` is `row_spans`: one entry
+ per `rows` entry, index-aligned, None for a row not exported (a skipped row, or
+ a row a filter cut); the tuple itself is None when absent, and always None when
+ the export was declined. `export_columns` is the export's one column list, in
+ the order every line carries its columns (None when no export ran), and
+ `export_declined_reason` the catalog id of a declined export (None when the
+ export ran or none was asked for); `export_declined` stays the cause text.
+
`unsupported_settings` is the call's own settings the library declined, from
the batch's top-level list (ABI v2). `at_merge` is what an `OPTIMIZE TABLE ...
FINAL` of the part would do to its rows, at least: a later background merge
@@ -206,14 +232,19 @@ class BatchResult:
outcome: Outcome
err_code: int
err_msg: bytes
+ reason: str | None
+ refused_at: RefusalSite | None
+ refused_row: int | None
rows: tuple[RowResult, ...]
rows_read: int
rows_skipped: int
transformed: tuple[Transform, ...]
engine_rows: tuple[tuple[EngineCell, ...], ...] | None
payload: bytes | None
- spans: tuple[Span, ...] | None
+ spans: tuple[Span | None, ...] | None
+ export_columns: tuple[bytes, ...] | None
export_declined: bytes
+ export_declined_reason: str | None
rows_passed: int
rows_cut: int
partition_count: int | None
@@ -225,21 +256,30 @@ class BatchResult:
@dataclass(frozen=True, slots=True)
class FilterRowError:
+ """One entry of a filter result's `errors`. `reason` is the catalog id where
+ `code` is -2 (`CHS_CODE_DECLINED`), else None."""
+
row: int
code: int
+ reason: str | None
msg: bytes
@dataclass(frozen=True, slots=True)
class FilterResult:
"""The `filter_result` document. A verdict counts only when `outcome` is `ok`;
- otherwise treat every verdict as `d`, whatever the verdict string holds. Under
+ otherwise treat every verdict as `d`, whatever the verdict string holds.
+ `reason`, `refused_at` and `refused_row` are the batch's (see `BatchResult`):
+ where a refused or declined evaluation sits, and its catalog id. Under
`ok`, `e` and `d` are never answers, and a caller enforcing visibility fails
closed on both; `Verdict.answered` says which."""
outcome: FilterOutcome
err_code: int
err_msg: bytes
+ reason: str | None
+ refused_at: RefusalSite | None
+ refused_row: int | None
rows_read: int
unsupported_settings: tuple[bytes, ...]
verdicts: tuple[Verdict, ...]
@@ -250,12 +290,20 @@ class FilterResult:
class Column:
"""One column of a compiled schema. `type` is the canonical type; it can
differ between ClickHouse lines, so a cross-line hash is taken over the
- caller's own statement, never over `columns`."""
+ caller's own statement, never over `columns`.
+
+ `default_support` is what this build does with the column's DEFAULT and
+ `default_support_reason` the catalog id behind a `DECLINED` one; `reads` is
+ the columns the column's expression reads. All three are reserved: None when
+ the build reports none."""
name: bytes
type: bytes
default_kind: DefaultKind
default_expr: bytes
+ default_support: DefaultSupport | None = None
+ default_support_reason: str | None = None
+ reads: tuple[bytes, ...] | None = None
@dataclass(frozen=True, slots=True)
@@ -280,7 +328,9 @@ class SchemaServer:
data-derived. `timezone` is the zone the schema's types bind: the profile's,
or else the image zone. `settings` is the profile's settings (`{}` when it
gave none). `macros` is None exactly when the profile carried no macros
- (unknown), and a mapping, even an empty one, when it did (the complete set)."""
+ (unknown), and a mapping, even an empty one, when it did (the complete set). A
+ setting value that was not UTF-8 comes back as its bytes decoded with
+ `surrogateescape`, the spelling that goes back in unchanged."""
timezone: str
settings: Mapping[str, str]
@@ -308,12 +358,16 @@ class SchemaDescription:
the highest layer that sets it, ordered by layer and then by name bytes, as
the library wrote it. A schema that lists any is one `compile_filter`
declines (`UnsupportedError`), and a consumer that streams rows refuses
- such a tenant (spec/abi-v2/where-settings/README.md)."""
+ such a tenant (spec/abi-v2/where-settings/README.md).
+
+ `posture` is how the statement was compiled, which the document carries only
+ for `Posture.LOAD`: None is `Posture.CREATE`."""
columns: tuple[Column, ...]
server: SchemaServer | None = None
replicated: SchemaReplicated | None = None
filter_declined_settings: tuple[DeclinedSetting, ...] = ()
+ posture: Posture | None = None
@dataclass(frozen=True, slots=True)
diff --git a/python/tests/abi2/test_public_library.py b/python/tests/abi2/test_public_library.py
index ed556d99..914ba70c 100644
--- a/python/tests/abi2/test_public_library.py
+++ b/python/tests/abi2/test_public_library.py
@@ -15,13 +15,18 @@
from chtypes import (
BatchResult,
CallError,
+ ColumnDefault,
+ DeferDefaults,
DocFlags,
Format,
InternalError,
Outcome,
+ ParamEncoding,
+ Posture,
SchemaError,
UnsupportedError,
UsageError,
+ VerdictBasis,
)
from chtypes._abi2 import _decls
@@ -149,8 +154,8 @@ def spy(*args):
# lists, reads as its unknown(n) member, never as accepted (rule r3).
assert result.outcome == "" and not result.outcome.known and result.payload is not None
assert result.outcome is not Outcome.ACCEPTED
- (_, fmt, body, settings, columns, handle, export, flags), plain = calls
- assert fmt == 0 and body == b'{"a":1}\n' and settings is None
+ (_, fmt, body, settings, columns, handle, export, flags, options), plain = calls
+ assert fmt == 0 and body == b'{"a":1}\n' and settings is None and options is None
assert json.loads(columns) == [{"name": "a"}, {"name_b64": "/w=="}]
assert handle is not None and export == int(Format.CSV) and flags == 3
assert plain[5] is None and plain[6] == -1 and plain[7] == 7 # no export, all groups
@@ -169,12 +174,12 @@ def test_filter_zone_and_parse_zone_are_distinct_arguments(lib, monkeypatch) ->
monkeypatch.setattr(
lib._api,
"filter_create",
- lambda s, e, p, st: (created.append(st), real_create(s, e, p, st))[1],
+ lambda s, e, p, st, o: (created.append(st), real_create(s, e, p, st, o))[1],
)
monkeypatch.setattr(
lib._api,
"filter_eval_body",
- lambda f, fmt, b, st: (evaluated.append(st), real_eval(f, fmt, b, st))[1],
+ lambda f, fmt, b, st, c, o: (evaluated.append(st), real_eval(f, fmt, b, st, c, o))[1],
)
with lib.compile_table(CREATE) as schema:
with schema.compile_filter("a > 1", params={"p": "1"}, session_timezone="Asia/Tokyo") as f:
@@ -327,3 +332,105 @@ def fake():
table = lib.error_codes()
assert table.name(62) == "SYNTAX_ERROR"
assert lib.error_codes() is table and len(calls) == 2
+
+
+def _spy(lib, monkeypatch, name):
+ calls: list[tuple] = []
+ real = getattr(lib._api, name)
+
+ def spy(*args):
+ calls.append(args)
+ return real(*args)
+
+ monkeypatch.setattr(lib._api, name, spy)
+ return calls
+
+
+def test_insert_options_reach_the_four_insert_calls_and_nothing_else(lib, monkeypatch) -> None:
+ """Spec batch 2: `options` on preview_row, preview_batch, block_create and
+ filter_eval_body, the `columns` of filter_eval_body, and no document at all
+ when no member is set. What the library did with each is the stub's echo."""
+ row = _spy(lib, monkeypatch, "preview_row")
+ batch = _spy(lib, monkeypatch, "preview_batch")
+ block = _spy(lib, monkeypatch, "block_create")
+ body = _spy(lib, monkeypatch, "filter_eval_body")
+ with lib.compile_table(CREATE) as schema, schema.compile_filter("a > 1") as flt:
+ kw = {
+ "insertable_columns": ["a"],
+ "column_defaults": [ColumnDefault("a", "1")],
+ "verdict_basis": VerdictBasis.EXPORT,
+ "defer_defaults": DeferDefaults.SERVER,
+ }
+ schema.row(Format.CSV, b"1", **kw)
+ schema.rows(Format.CSV, b"1", **kw)
+ schema.parse_block(Format.CSV, b"1", **kw).close()
+ flt.rows(Format.CSV, b"1", columns=["a"], **kw)
+ schema.row(Format.CSV, b"1")
+ want = {
+ "insertable_columns": [{"name": "a"}],
+ "column_defaults": [{"name": "a", "value": "1"}],
+ "verdict_basis": "export",
+ "defer_defaults": "server",
+ }
+ assert json.loads(row[0][-1]) == want and row[1][-1] is None
+ assert json.loads(batch[0][-1]) == want
+ assert json.loads(block[0][-1]) == want
+ assert json.loads(body[0][-1]) == want and json.loads(body[0][-2]) == [{"name": "a"}]
+
+
+def test_filter_and_schema_options_reach_their_creates(lib, monkeypatch) -> None:
+ created = _spy(lib, monkeypatch, "filter_create")
+ schemas = _spy(lib, monkeypatch, "schema_create")
+ with lib.compile_table(CREATE, posture=Posture.LOAD) as schema:
+ schema.compile_filter("a > 1", param_encoding=ParamEncoding.RAW).close()
+ schema.compile_filter("a > 1").close()
+ assert json.loads(schemas[0][-1]) == {"posture": "load"}
+ assert json.loads(created[0][-1]) == {"param_encoding": "raw"} and created[1][-1] is None
+
+
+def test_an_unnamed_member_reaches_the_library_which_refuses_it_by_name(lib) -> None:
+ """Rule r1: the binding never drops a key it has no name for; the library
+ refuses one it does not know, naming it (here the stub, which enforces r1)."""
+ with lib.compile_table(CREATE) as schema:
+ with pytest.raises(UsageError, match='unknown key "future"'):
+ schema.row(Format.CSV, b"1", insert_options_extra={"future": "[1]"})
+ with pytest.raises(UsageError, match='unknown key "future"'):
+ schema.compile_filter("a > 1", filter_options_extra={"future": "1"})
+ with pytest.raises(UsageError, match='unknown key "future"'):
+ lib.compile_table(CREATE, schema_options_extra={"future": "1"})
+
+
+def test_a_call_error_carries_the_librarys_reason(monkeypatch) -> None:
+ """`CallError.reason` is `chs_error_reason`, read with the other five fields.
+ The stub writes an empty reason, so a fake raw table (real memory behind each
+ buffer) supplies one: the verdict is what the generated `_check` raised."""
+ import ctypes
+
+ keep: list[ctypes.Array] = []
+ length: dict[int, int] = {}
+
+ def buf(text: bytes) -> int:
+ b = ctypes.create_string_buffer(text, max(len(text), 1))
+ keep.append(b)
+ length[ctypes.addressof(b)] = len(text)
+ return ctypes.addressof(b)
+
+ ch_name, message, column, reason = buf(b"CH"), buf(b"msg"), buf(b""), buf(b"zone-declined")
+ raw = {
+ "chs_error_status": lambda e: 2,
+ "chs_error_ch_code": lambda e: 0,
+ "chs_error_ch_name": lambda e: ch_name,
+ "chs_error_message": lambda e: message,
+ "chs_error_column": lambda e: column,
+ "chs_error_reason": lambda e: reason,
+ "chs_error_free": lambda e: None,
+ "chs_buf_data": lambda p: p,
+ "chs_buf_len": lambda p: length[p],
+ "chs_buf_free": lambda p: None,
+ }
+ api = object.__new__(_decls.Api)
+ api._raw = raw
+ with pytest.raises(Exception) as info:
+ api._check(2, ctypes.c_void_p(1))
+ assert info.value.reason == "zone-declined" and info.value.ch_name == "CH"
+ assert info.value.message == b"msg"
diff --git a/python/tests/abi2/test_reader_rules.py b/python/tests/abi2/test_reader_rules.py
index 60615ab4..bd7f5a19 100644
--- a/python/tests/abi2/test_reader_rules.py
+++ b/python/tests/abi2/test_reader_rules.py
@@ -86,7 +86,7 @@ def test_r2_unknown_members_are_ignored(open_variant) -> None:
entry0 = row.transformed[0]
assert (entry0.column, entry0.input, entry0.stored) == (b"s", b"1", b"2")
assert entry0.reason == Reason.VALUE_CHANGED and entry0.lossy is True and entry0.row == 0
- assert row.unknown_fields == (b"u",) and row.unsupported_settings == (b"st",)
+ assert row.unknown_fields == (b"u",)
batch = schema.rows(JSON, b'{"x":1}', export=JSON)
assert batch.outcome is Outcome.ACCEPTED and batch.rows_read == 1
diff --git a/python/tests/api/test_decode.py b/python/tests/api/test_decode.py
index fd96a6ea..17d3a618 100644
--- a/python/tests/api/test_decode.py
+++ b/python/tests/api/test_decode.py
@@ -7,7 +7,17 @@
import pytest
-from chtypes import DefaultKind, FilterOutcome, InternalError, MergeReason, Outcome, Verdict
+from chtypes import (
+ DefaultKind,
+ DefaultSupport,
+ FilterOutcome,
+ InternalError,
+ MergeReason,
+ Outcome,
+ Posture,
+ RefusalSite,
+ Verdict,
+)
from chtypes._decode import (
decode_batch,
decode_discovery,
@@ -42,7 +52,7 @@ def dumps(doc) -> bytes:
{"column_b64": b64(b"\xff"), "input_b64": b64(b"\x80"), "stored": "x", "reason": "zzz"},
],
"unknown_fields": [{"name": "extra"}, {"name_b64": b64(b"\xff\x00")}],
- "unsupported_settings": [{"name": "foo"}],
+ "unsupported_settings": [{"name": "removed_at_row_level"}], # skipped: not a member now
"computed": [{"name": "c", "kind": "materialized", "stored": "9", "value_b64": b64(b"9")}],
"verdict": "t",
"verdict_code": 0,
@@ -67,7 +77,9 @@ def test_a_row_decodes_one_to_one_with_names_and_values_as_bytes() -> None:
assert row.transformed[1].column == b"\xff" and row.transformed[1].input == b"\x80"
assert row.transformed[1].reason == "zzz" and row.transformed[1].lossy is True
assert row.transformed[1].row == 0
- assert row.unknown_fields == (b"extra", b"\xff\x00") and row.unsupported_settings == (b"foo",)
+ assert row.unknown_fields == (b"extra", b"\xff\x00") and not hasattr(
+ row, "unsupported_settings"
+ )
assert row.computed[0].text == b"9" and row.computed[0].value == b"9"
assert row.verdict is Verdict.TRUE
assert row.partition_id == b"p1" and row.input_span.off == 0 and row.input_span.len == 5
@@ -80,6 +92,8 @@ def test_absent_is_the_default() -> None:
assert row.outcome == "" and not row.outcome.known and row.outcome is not Outcome.ACCEPTED
assert row.columns == () and row.verdict is None and row.input_span is None
assert row.err_msg == b"" and row.partition_id is None
+ assert row.reason is None and row.refused_at is None and row.refused_row is None
+ assert row.verdict_reason is None
@pytest.mark.parametrize(
@@ -232,30 +246,38 @@ def test_a_batch_decodes_rows_payload_framing_and_spans() -> None:
"outcome": "accepted",
"unsupported_settings": [{"name": "session_timezone"}, {"name_b64": b64(b"\xff")}],
"at_merge": [
- {"row": 0, "reason": "ttl_delete"},
- {"row": 1, "reason": "ttl_column_reset", "column": "c", "stored": "0"},
- {"row": 2, "reason": "ttl_column_reset", "column": "t"},
+ {"part_row": 0, "reason": "ttl_delete"},
+ {"part_row": 1, "reason": "ttl_column_reset", "column": "c", "stored": "0"},
+ {"part_row": 2, "reason": "ttl_column_reset", "column": "t"},
{
- "row": 3,
+ "part_row": 3,
"reason": "ttl_column_reset",
"column_b64": b64(b"\xff\x00"),
"stored_b64": b64(b"\xff"),
},
- {"row": 4, "reason": "ttl_delete", "input_rows": [4, 7]},
+ {"part_row": 4, "reason": "ttl_delete", "input_rows": [4, 7]},
{
- "row": 5,
+ "part_row": 5,
"reason": "a_reason_nobody_listed",
"x_future": {"a": [1]},
"x_future_b64": "/w==",
},
- {"row": 6, "reason": "ttl_column_reset", "column": "e", "stored": "", "input_rows": []},
+ {
+ "part_row": 6,
+ "reason": "ttl_column_reset",
+ "column": "e",
+ "stored": "",
+ "input_rows": [],
+ },
],
}
def test_a_batch_decodes_at_merge_one_to_one() -> None:
entries = decode_batch(dumps(AT_MERGE_BATCH), None).at_merge
- got = [(e.row, e.reason, e.reason.known, e.column, e.stored, e.input_rows) for e in entries]
+ got = [
+ (e.part_row, e.reason, e.reason.known, e.column, e.stored, e.input_rows) for e in entries
+ ]
assert got == [
(0, MergeReason.TTL_DELETE, True, None, None, None),
(1, MergeReason.TTL_COLUMN_RESET, True, b"c", b"0", None),
@@ -298,14 +320,18 @@ def test_absent_at_merge_and_unsupported_settings_are_empty(doc) -> None:
[
{
"at_merge": [
- {"row": 0, "reason": "ttl_column_reset", "column": "c", "column_b64": "Yw=="}
+ {"part_row": 0, "reason": "ttl_column_reset", "column": "c", "column_b64": "Yw=="}
]
},
- {"at_merge": [{"row": 0, "reason": "ttl_column_reset", "column": "c", "stored_b64": "!!"}]},
- {"at_merge": [{"row": 0, "reason": "ttl_delete", "input_rows": ["x"]}]},
- {"at_merge": [{"row": 0, "reason": "ttl_delete", "input_rows": [-1]}]},
- {"at_merge": [{"row": 0, "reason": 7}]},
- {"at_merge": {"row": 0}},
+ {
+ "at_merge": [
+ {"part_row": 0, "reason": "ttl_column_reset", "column": "c", "stored_b64": "!!"}
+ ]
+ },
+ {"at_merge": [{"part_row": 0, "reason": "ttl_delete", "input_rows": ["x"]}]},
+ {"at_merge": [{"part_row": 0, "reason": "ttl_delete", "input_rows": [-1]}]},
+ {"at_merge": [{"part_row": 0, "reason": 7}]},
+ {"at_merge": {"part_row": 0}},
{"unsupported_settings": ["session_timezone"]},
],
)
@@ -415,7 +441,6 @@ def test_error_code_table_and_live_handles() -> None:
{"unknown_fields": ["bare string"]},
{"unknown_fields": [{}]},
{"unknown_fields": [{"name": "a", "name_b64": "YQ=="}]},
- {"unsupported_settings": ["x"]},
{"partition_id": "p", "partition_id_b64": "cA=="},
{"cols": [{"name": "a", "null": False, "src": "input", "value_b64": "YQ"}]}, # unpadded
{"computed": [{"name": "c", "value_b64": "!!"}]},
@@ -431,3 +456,169 @@ def test_an_engine_row_must_be_a_list_of_cells_never_an_object() -> None:
decode_batch(dumps({"engine_rows": [{"a": "1"}]}), None)
with pytest.raises(InternalError):
decode_batch(dumps({"engine_rows": [[{"stored": "1", "null": False}]]}), None) # no name
+
+
+# ABI v2 spec batch 2: the refusal members, the export members, the null row span
+# and the schema description's new members.
+
+
+def test_a_declined_batch_names_its_code_reason_and_site() -> None:
+ batch = decode_batch(
+ dumps(
+ {
+ "outcome": "unsupported",
+ "code": -2,
+ "reason": "preview-batch-zone-mismatch",
+ "refused_at": "body_read",
+ "rows_read": 0,
+ "rows_skipped": 0,
+ "unsupported_settings": [],
+ }
+ ),
+ None,
+ )
+ assert batch.err_code == -2 and batch.reason == "preview-batch-zone-mismatch"
+ assert batch.refused_at is RefusalSite.BODY_READ and batch.refused_at.known
+ assert batch.refused_row is None
+
+
+def test_a_row_refusal_names_its_row_and_a_future_site_is_kept() -> None:
+ row = decode_row_document(
+ dumps(
+ {
+ "outcome": "unsupported",
+ "code": -2,
+ "reason": "drawn-value",
+ "refused_at": "row",
+ "refused_row": 0,
+ "verdict": "d",
+ "verdict_code": -2,
+ "verdict_reason": "defer-defaults-reader",
+ }
+ )
+ )
+ assert row.reason == "drawn-value" and row.refused_at is RefusalSite.ROW
+ assert row.refused_row == 0 # present zero is not absent
+ assert row.verdict is Verdict.DECLINE and row.verdict_code == -2
+ assert row.verdict_reason == "defer-defaults-reader"
+ future = decode_row_document(dumps({"refused_at": "a_site_nobody_listed"}))
+ assert future.refused_at == "a_site_nobody_listed" and not future.refused_at.known
+
+
+def test_a_filter_result_carries_the_batchs_refusal_members_and_error_reasons() -> None:
+ res = decode_filter_result(
+ dumps(
+ {
+ "outcome": "unsupported",
+ "code": -2,
+ "reason": "filter-zone",
+ "refused_at": "call",
+ "rows_read": 1,
+ "verdicts": "d",
+ "errors": [
+ {"row": 0, "code": -2, "reason": "drawn-value"},
+ {"row": 1, "code": 386},
+ ],
+ }
+ )
+ )
+ assert res.err_code == -2 and res.reason == "filter-zone"
+ assert res.refused_at is RefusalSite.CALL and res.refused_row is None
+ assert [(e.code, e.reason) for e in res.errors] == [(-2, "drawn-value"), (386, None)]
+
+
+def test_the_export_members_and_the_null_row_span() -> None:
+ batch = decode_batch(
+ dumps(
+ {
+ "outcome": "accepted",
+ "row_spans": [{"off": 0, "len": 4}, None, {"off": 4, "len": 2}],
+ "export_columns": [{"name": "a"}, {"name_b64": b64(b"\xff")}],
+ "export_declined": "",
+ }
+ ),
+ b"abcdef",
+ )
+ assert batch.spans is not None
+ assert [s and (s.off, s.len) for s in batch.spans] == [(0, 4), None, (4, 2)]
+ assert batch.export_columns == (b"a", b"\xff") and batch.export_declined_reason is None
+ declined = decode_batch(
+ dumps(
+ {
+ "outcome": "rejected",
+ "row_spans": [None],
+ "export_columns": [{"name": "a"}],
+ "export_declined": "no export",
+ "export_declined_reason": "export-after-refusal",
+ }
+ ),
+ None,
+ )
+ assert declined.spans is None and declined.payload is None
+ assert declined.export_declined == b"no export"
+ assert declined.export_declined_reason == "export-after-refusal"
+ assert decode_batch(dumps({}), None).export_columns is None
+
+
+def test_a_schema_description_decodes_posture_default_support_reads_and_b64_settings() -> None:
+ desc = decode_schema_description(
+ dumps(
+ {
+ "posture": "load",
+ "columns": [
+ {
+ "name": "a",
+ "type": "UInt8",
+ "default_kind": "DEFAULT",
+ "default_support": "declined",
+ "default_support_reason": "drawn-value",
+ "reads": [{"name": "b"}, {"name_b64": b64(b"\xff")}],
+ },
+ {"name": "b", "type": "UInt8", "default_kind": ""},
+ ],
+ "server": {
+ "timezone": "UTC",
+ "settings": {"plain": "1", "raw": {"b64": b64(b"\xff\x00")}},
+ },
+ }
+ )
+ )
+ assert desc.posture is Posture.LOAD
+ a, b = desc.columns
+ assert (
+ a.default_support is DefaultSupport.DECLINED and a.default_support_reason == "drawn-value"
+ )
+ assert a.reads == (b"b", b"\xff")
+ assert b.default_support is None and b.default_support_reason is None and b.reads is None
+ assert desc.server is not None
+ assert desc.server.settings == {"plain": "1", "raw": "\udcff\x00"}
+ assert desc.server.settings["raw"].encode("utf-8", "surrogateescape") == b"\xff\x00"
+ assert decode_schema_description(dumps({"columns": []})).posture is None
+
+
+@pytest.mark.parametrize(
+ "settings",
+ [{"k": {"b64": "!!"}}, {"k": {"b64": "YQ==", "x": 1}}, {"k": 5}],
+)
+def test_a_bad_server_settings_value_is_an_internal_error(settings) -> None:
+ with pytest.raises(InternalError):
+ decode_schema_description(dumps({"server": {"settings": settings}}))
+
+
+def test_default_override_decodes_as_a_stored_value_source() -> None:
+ row = decode_row_document(
+ dumps(
+ {
+ "outcome": "accepted",
+ "code": 0,
+ "cols": [
+ {"name": "a", "null": False, "src": "default_override", "stored": "5"},
+ {"name": "b", "null": False, "src": "default_deferred", "stored": ""},
+ ],
+ }
+ )
+ )
+ assert row.columns[0].source == "default_override" and row.columns[0].is_stored is True
+ assert row.columns[0].text == b"5"
+ assert row.columns[1].is_stored is False
+ assert [c.column for c in row.values] == [b"a"]
diff --git a/python/tests/api/test_input.py b/python/tests/api/test_input.py
index b006a850..021ad537 100644
--- a/python/tests/api/test_input.py
+++ b/python/tests/api/test_input.py
@@ -7,8 +7,18 @@
import pytest
-from chtypes import UsageError
-from chtypes._input import body_bytes, columns_json, settings_json, string_map_json, to_bytes
+from chtypes import ColumnDefault, ParamEncoding, Posture, ServerProfile, UsageError, VerdictBasis
+from chtypes._input import (
+ body_bytes,
+ columns_json,
+ filter_options_json,
+ insert_options_json,
+ schema_options_json,
+ server_profile_json,
+ settings_json,
+ string_map_json,
+ to_bytes,
+)
def test_settings_are_a_json_object_of_string_values_written_verbatim() -> None:
@@ -69,36 +79,160 @@ def test_text_inputs_are_utf8_and_bodies_never_accept_text() -> None:
_LONE = "\udcff" # what surrogateescape makes of the byte 0xff
+def test_a_non_utf8_value_travels_as_the_b64_object_form() -> None:
+ # ABI v2 json_object_byte_values: the bytes are exact; a UTF-8 value stays a string.
+ got = json.loads(settings_json({"k": _LONE, "ok": "é"}) or b"")
+ assert got == {"k": {"b64": "/w=="}, "ok": "é"}
+ assert json.loads(settings_json({"k": _LONE}, "UTC") or b"") == {
+ "k": {"b64": "/w=="},
+ "session_timezone": "UTC",
+ }
+ # the zone is a settings value too
+ assert json.loads(settings_json(None, _LONE) or b"") == {"session_timezone": {"b64": "/w=="}}
+ assert json.loads(string_map_json({"p": _LONE + "a"}, "query parameters") or b"") == {
+ "p": {"b64": base64.b64encode(b"\xffa").decode()}
+ }
+
+
@pytest.mark.parametrize(
("call", "words"),
[
- (lambda: settings_json({"k": _LONE}), ("settings", 'the value for key "k"')),
(lambda: settings_json({"k" + _LONE: "v"}), ("settings", 'the key "k\\udcff"')),
- (lambda: settings_json({"k": _LONE}, "UTC"), ("settings", 'the value for key "k"')),
- (lambda: settings_json(None, _LONE), ("the zone",)),
- (
- lambda: string_map_json({"p": _LONE}, "query parameters"),
- ("query parameters", 'the value for key "p"'),
- ),
(
lambda: string_map_json({_LONE: "v"}, "query parameters"),
("query parameters", 'the key "\\udcff"'),
),
+ # a surrogate that is not a surrogateescape byte has no bytes to carry
+ (lambda: settings_json({"k": "\ud800"}), ("settings", 'the value for key "k"')),
+ # a macro (name or value) and a server timezone have no object form
+ (
+ lambda: server_profile_json(ServerProfile(macros={"m": _LONE})),
+ ("macros", 'the value for key "m"'),
+ ),
+ (
+ lambda: server_profile_json(ServerProfile(macros={_LONE: "v"})),
+ ("macros", 'the key "\\udcff"'),
+ ),
+ (lambda: server_profile_json(ServerProfile(timezone=_LONE)), ("timezone",)),
],
)
-def test_a_non_utf8_key_or_value_is_a_usage_error_naming_the_input_and_the_key(call, words) -> None:
+def test_a_key_or_value_with_no_byte_form_is_a_usage_error_naming_the_input_and_the_key(
+ call, words
+) -> None:
with pytest.raises(UsageError) as info:
call()
message = info.value.message.decode("utf-8")
- for word in (*words, "not valid UTF-8", "unhex('')"):
+ for word in (*words, "not valid UTF-8", "cannot carry these bytes"):
assert word in message
assert info.value.status == 3 and info.value.ch_code == 0
assert message.isascii() # the key is escaped, never raw
+def test_server_profile_settings_take_the_b64_form_too() -> None:
+ doc = json.loads(server_profile_json(ServerProfile(settings={"k": _LONE, "j": "v"})))
+ assert doc == {"settings": {"k": {"b64": "/w=="}, "j": "v"}}
+
+
+def test_no_option_set_sends_no_document() -> None:
+ assert insert_options_json() is None
+ assert filter_options_json() is None
+ assert schema_options_json() is None
+
+
+def test_insert_options_are_the_members_set_and_nothing_else() -> None:
+ raw = insert_options_json(
+ insertable_columns=["a", b"\xff"],
+ column_defaults=[ColumnDefault("a", "1"), ColumnDefault(b"\xfe", b"\xfd")],
+ verdict_basis=VerdictBasis.EXPORT,
+ defer_defaults="server",
+ )
+ assert json.loads(raw or b"") == {
+ "insertable_columns": [{"name": "a"}, {"name_b64": "/w=="}],
+ "column_defaults": [
+ {"name": "a", "value": "1"},
+ {"name_b64": "/g==", "value_b64": "/Q=="},
+ ],
+ "verdict_basis": "export",
+ "defer_defaults": "server",
+ }
+
+
+def test_an_absent_list_is_left_out_and_a_present_empty_one_is_sent() -> None:
+ assert json.loads(insert_options_json(insertable_columns=[]) or b"") == {
+ "insertable_columns": []
+ }
+ assert json.loads(insert_options_json(column_defaults=[]) or b"") == {"column_defaults": []}
+ assert insert_options_json(insertable_columns=None, column_defaults=None) is None
+
+
+def test_an_unnamed_member_is_passed_through_as_json_text_verbatim() -> None:
+ raw = insert_options_json(
+ verdict_basis="input", insert_options_extra={"future": '{"a": [1, 2.50]}', "n": "7"}
+ )
+ # the JSON text is spliced as given (no re-serialization), after the named members
+ assert raw == b'{"verdict_basis":"input","future":{"a": [1, 2.50]},"n":7}'
+ # a key set twice keeps the last value: the extra member wins
+ assert json.loads(
+ insert_options_json(
+ verdict_basis="input", insert_options_extra={"verdict_basis": '"export"'}
+ )
+ or b""
+ ) == {"verdict_basis": "export"}
+ with pytest.raises(UsageError) as info:
+ insert_options_json(insert_options_extra={"k": "{not json"})
+ assert b"not valid JSON" in info.value.message
+ with pytest.raises(TypeError):
+ insert_options_json(insert_options_extra={"k": 1}) # type: ignore[dict-item]
+
+
+def test_filter_and_schema_options() -> None:
+ assert json.loads(filter_options_json(param_encoding=ParamEncoding.RAW) or b"") == {
+ "param_encoding": "raw"
+ }
+ assert json.loads(
+ schema_options_json(posture=Posture.LOAD, schema_options_extra={"x": "true"}) or b""
+ ) == {"posture": "load", "x": True}
+
+
+@pytest.mark.parametrize("bad", [5, b"input", None.__class__])
+def test_an_enum_option_is_a_str_or_its_enum(bad) -> None:
+ with pytest.raises(TypeError):
+ insert_options_json(verdict_basis=bad)
+
+
+def test_the_options_never_validate_a_member() -> None:
+ # The library refuses an unknown value, naming it; the binding sends it.
+ assert json.loads(insert_options_json(verdict_basis="nonsense") or b"") == {
+ "verdict_basis": "nonsense"
+ }
+
+
def test_valid_non_ascii_serializes_byte_for_byte_as_before() -> None:
# Measured before the change on the same inputs: json.dumps, ASCII escapes.
assert string_map_json({"p": "é日本"}, "query parameters") == b'{"p":"\\u00e9\\u65e5\\u672c"}'
assert settings_json({"é": "\ufffd"}, "Asia/Tokyo") == (
b'{"\\u00e9":"\\ufffd","session_timezone":"Asia/Tokyo"}'
)
+
+
+def test_column_defaults_wire_form_utf8_b64_empty_and_order() -> None:
+ def doc(*entries: ColumnDefault) -> dict[str, object]:
+ return json.loads(insert_options_json(column_defaults=list(entries)) or b"")
+
+ assert doc(ColumnDefault("a", "h\u00e9llo")) == {
+ "column_defaults": [{"name": "a", "value": "h\u00e9llo"}]
+ }
+ assert doc(ColumnDefault("a", b"\xff\x00")) == {
+ "column_defaults": [{"name": "a", "value_b64": "/wA="}]
+ }
+ assert doc(ColumnDefault(b"\xfe", "v")) == {
+ "column_defaults": [{"name_b64": "/g==", "value": "v"}]
+ }
+ # An empty value is a supplied value: sent as "", never omitted.
+ assert doc(ColumnDefault("a", "")) == {"column_defaults": [{"name": "a", "value": ""}]}
+ assert doc(ColumnDefault("a", b"")) == {"column_defaults": [{"name": "a", "value": ""}]}
+ # The list keeps its order (a JSON list compares in order).
+ raw = insert_options_json(column_defaults=[ColumnDefault("z", "1"), ColumnDefault("a", "2")])
+ assert json.loads(raw or b"") == {
+ "column_defaults": [{"name": "z", "value": "1"}, {"name": "a", "value": "2"}]
+ }
diff --git a/python/tests/goldens_v1/data/stub-goldens.json b/python/tests/goldens_v1/data/stub-goldens.json
index 43f46547..07810968 100644
--- a/python/tests/goldens_v1/data/stub-goldens.json
+++ b/python/tests/goldens_v1/data/stub-goldens.json
@@ -43,12 +43,12 @@
"status": "CHS_OK",
"document": {
"outcome": "accepted",
+ "code": 0,
"input_span": {
"off": 0,
"len": 8
},
"unknown_fields": [],
- "unsupported_settings": [],
"cols": [
{
"name": "s",
@@ -120,12 +120,12 @@
"status": "CHS_OK",
"document": {
"outcome": "accepted",
+ "code": 0,
"input_span": {
"off": 0,
"len": 9
},
"unknown_fields": [],
- "unsupported_settings": [],
"cols": [
{
"name": "s",
diff --git a/python/tests/goldens_v1/test_runner.py b/python/tests/goldens_v1/test_runner.py
index 1959612b..4a7d531b 100644
--- a/python/tests/goldens_v1/test_runner.py
+++ b/python/tests/goldens_v1/test_runner.py
@@ -147,6 +147,7 @@ def decoded(call: Any) -> bool:
_b64d(body["body_b64"]),
_b64d(body["settings_b64"]),
_b64d(body["columns_b64"]) if "columns_b64" in body else None,
+ None,
),
None,
)
@@ -162,6 +163,7 @@ def decoded(call: Any) -> bool:
None,
requested,
body["doc_flags"],
+ None,
)
if requested == -1:
export = None
@@ -174,6 +176,7 @@ def decoded(call: Any) -> bool:
_b64d(f["expr_b64"]),
_b64d(f["query_params_b64"]),
_b64d(f["settings_b64"]),
+ None,
)
except CallError as err:
return failed("filter_create", err)
@@ -183,6 +186,8 @@ def decoded(call: Any) -> bool:
body["format"],
_b64d(body["body_b64"]),
_b64d(body["settings_b64"]),
+ _b64d(body["columns_b64"]) if "columns_b64" in body else None,
+ None,
),
None,
)
diff --git a/rust/CHANGELOG.md b/rust/CHANGELOG.md
index 140b92be..63c1f3f0 100644
--- a/rust/CHANGELOG.md
+++ b/rust/CHANGELOG.md
@@ -8,6 +8,8 @@ The four bindings in this repository are released together and give one answer,
### Added
+- `column_defaults` is covered by unit tests (ABI v2): the exact `insert_options` JSON for a UTF-8 value, a non-UTF-8 value (`value_b64`), an empty value (sent as `""`) and two entries in order, and the decode of a `default_override` cell (`Source::DefaultOverride`, stored). The batches guide gains a usage section. No behavior changed.
+- `chtypes::CODE_DECLINED` (ABI v2, `i32`, -2): the code a declined row, outcome or verdict carries, generated from the description. It is at the crate root, not in `code`, which is the fetch-code table.
- A fetch-only call (#492): `Registry::fetch(&self, request: &str) -> Result` resolves, fetches, verifies and installs the build a request names into the cache, as `chtypes fetch` does, and opens nothing: no library is loaded and the process setup is untouched, so it needs no `setup`. It honors the registry's fetch options (offline, the cache, and under the production channel the bases, the trust and the lock), whatever `autofetch` says, and a concurrent `fetch` and open of one request share one fetch. The `Resolved` names the installed version, build, digests and library path; errors are the fetch codes, and a refused spelling or pin is `Error::Usage`, as for an open.
- `chtypes resolve [--json]` (#493): what a line or version resolves to on every platform the registry offers, one `resolved ` line each, or with `--json` one JSON array of `{platform, version, build, manifest}` objects, each signed statement verified exactly as a fetch verifies it and no layer downloaded; `--offline` answers from the cache. The four CLIs print the same bytes, so one parser reads them (a CI cache key, for example).
- `chtypes prune [--line ] [--keep ] [--dry-run]` (#494): removes the installed builds of each line and platform that at least n newer installed builds supersede (n defaults to 1), each with its record, its blobs and its `index.json` entry; `--dry-run` prints exactly what would go. It never removes a build a running process holds: a `Registry` now holds every build it opens or fetches, with a shared `flock` on the entry's `verified.json` until the process exits, and prune removes an entry only after taking the exclusive lock without waiting, reporting it `in-use` otherwise. It never touches a system directory, nor, on the dev channel, another fingerprint's builds. The rules are `docs/guides/fetch-v1.md` sections 1 and 3, and the `resolve-build-*` and `prune-*` conformance cases hold all four bindings to them. The crate now depends on `libc` directly, for `flock`.
@@ -24,10 +26,18 @@ The four bindings in this repository are released together and give one answer,
- The v2 spec's decline catalog adds `server-dependent-function` (a column's DEFAULT, MATERIALIZED, ALIAS or EPHEMERAL expression, or a CHECK, that calls a function whose resolver needs a server is declined at `chs_schema_create`), `function-name-as-lambda` (a bare function name written where a higher-order function takes a lambda, such as `arrayExists(isNull, arr)`, is declined at `chs_filter_create` on 26.7, 26.8 and 26.9) and `skipped-key-in-squashed-block` (a batch whose answer depends on how the INSERT's squashing step groups its chunks is unsupported), and `call-setting-session-timezone` names only a per-call `session_timezone` the call cannot attach (55 reasons).
- A `410 Gone` is a retired repository, and permanent (#571). Any request the fetch layer makes (a tag, an alias, a digest, a blob, `tags/list`, the referrers API or its fallback tag, after a redirect too) that answers `410` is now the new code `CHTYPES_SOURCE_RETIRED` (exit status 10), never retried and never sent to the next base, nor from the dev channel's alias to its tag. It was `CHTYPES_SOURCE_UNREACHABLE` (exit 3) with the registry's message dropped, and Go, Python and Rust moved on to the next base, so a mirror's 404 could hide the retirement. The error names the URL that answered and carries the registry's own message, with control and bidirectional-formatting characters removed and cut to 256 code points: Go `CodeSourceRetired` (`errors.Is(err, ErrSourceRetired)`), Python and TypeScript `SourceRetiredError`, Rust `Error::SourceRetired`. The rule is `docs/guides/fetch-v1.md` section 2, "A retired repository", and the `retired-410-*` and `list-tags-retired-410` conformance cases and one shared message table (`tests/fixtures/retired-message/cases.json`) hold all four bindings to it.
- A 2.0.0-dev SDK prefers the newest build of its own fingerprint (#511). A version request resolves `--fp-` first, the alias the staging dev registry serves beside every dev tag, with the binding's own generated ABI fingerprint, and the tag itself only when every base answers the alias 404; any other failure of the alias request is that failure, with no fallback, and an alias whose index lacks the platform is `CHTYPES_ARTIFACT_UNPUBLISHED`. The dev channel's cache lookups (the installed-build lookup a registry open makes first, the offline answer, and the monotonic rule of a fetch) now see only records whose signed `abi_fingerprint` is the SDK's own, so a newer build of another fingerprint, installed by another dev SDK in the shared cache, is never returned or kept over this one's own, and is never removed or rewritten. So a dev build of a newer fingerprint no longer strands an older dev SDK until its consumers upgrade. Trust is unchanged (the signed predicate and the load checks decide), a listing never shows an alias, nothing records one, `list` and `where` are unchanged, and there is no option: both steps are the dev channel's alone, and the v1 contract and every production channel never take them. The rule is `docs/guides/fetch-v1.md` section 3, and the `dev-alias-*`, `resolve-installed-dev-alias-*` and `list-tags-dev-alias` conformance cases hold all four bindings to it.
+- `CallError::reason` (ABI v2): the error's catalog id, read through `chs_error_reason` by the generated layer and copied into the public `CallError` (ASCII, empty if none), on all four call classes. A misuse the binding detects itself carries an empty one.
+- Spec batch 2, the options documents (ABI v2; moves the dev fingerprint): `RowOptions`, `RowsOptions` and `EvalOptions` gain `insertable_columns` (`Option>>`), `column_defaults` (`Option>`), `verdict_basis` (`Option`: `Input`, `Export`), `defer_defaults` (`Option`: `None`, `Server`) and `insert_options_extra`, which build `insert_options` for `Schema::row`, `Schema::rows`, `Schema::parse_block` and `Filter::rows`; `FilterOptions` gains `param_encoding` (`Option`: `Escaped`, `Raw`) and `filter_options_extra`; `CompileOptions` gains `posture` (`Option`: `Create`, `Load`) and `schema_options_extra`. Each `*_extra` is a list of `(key, JSON text)` passed through verbatim, so the library, which refuses an unknown key (rule r1), is the only judge; text that is not valid JSON is `Error::Usage` before any call. A document with no member set is not sent. `EvalOptions` gains `columns`, because `chs_filter_eval_body` takes the column list now.
+- Spec batch 2, the decline and refusal members (ABI v2): `RowResult`, `BatchResult` and `FilterResult` carry `reason` (the catalog id, present where the code is -2, `CHS_CODE_DECLINED`), `refused_at` (an `Option`: `Call`, `BodyRead`, `Row`, `BodyStep`) and `refused_row`; `RowResult::verdict_reason`; `FilterRowError::reason`; `BatchResult::export_columns` and `BatchResult::export_declined_reason`. A document's `code` can be -2 for a row, batch, filter result or verdict this build declined.
+- Spec batch 2, the schema description (ABI v2): `SchemaDescription::posture`, and `Column::default_support` (an `Option`), `Column::default_support_reason` and `Column::reads`, reserved members a build reports when it implements them. `Source::DefaultDeferred`, `Source::DefaultOverride` (stored) and `DeclinedTier::ServerRefuses` join the generated vocabularies, and `RefusalSite`, `Posture` and `DefaultSupport` are exported.
+- Spec batch 2, a settings value that is not UTF-8 (ABI v2): `SchemaServer::settings` decodes the `{"b64": ...}` form back to its bytes. Its values are `RawText` now (see Changed). Rust takes `String` for every value it sends, so sending a non-UTF-8 value stays unreachable.
- Build-level revocation for generation 2 (#625; `docs/guides/fetch-v1.md`, "Build-level revocation"). A build its signed revocation refuses is the new code `CHTYPES_ARTIFACT_REVOKED`, exit status 11 (`code::ARTIFACT_REVOKED`, `Error::ArtifactRevoked`), whose message names the advisory and whose `Revocation` carries it for programs (`Revocation`, `Advisory`). An install reads a revocation from the trust listing it already makes and refuses before the layer; `--frozen` makes one listing, warm or cold, so a warm frozen fetch makes 1 request instead of 0 (`--frozen --offline` stays at 0); a registry open, `preload` and a fetch that finds the build installed check it at most once per build per process, one attempt per base, no backoff and one 5 s deadline (`check_installed`); `--lock` and `update` refuse a revoked build on every platform they pin. A verified revocation is stored in the build's cache entry (`revocation.json`) and refuses it from then on with no request, offline too; `chtypes verify` reports it and exits 11. A revocation that does not verify is ignored with one warning; a status that cannot be read is one warning, or with `FetchOptions::require_revocation_check` (`--require-revocation-check`) the transport's error. `FetchOptions::allow_revoked` (`--allow-revoked`) uses a revoked build for one call, with a warning, never for a lock or an update. A `410` whose body's `errors[0].detail.revoked` is an object is `CHTYPES_ARTIFACT_REVOKED`; every other `410` stays `CHTYPES_SOURCE_RETIRED`. The v1 contract is unchanged.
### Changed
+- **Breaking (ABI v2 dev): `BatchResult::spans` is `Option>>`**, one entry per `rows` entry and index-aligned with it: `row_spans` carries `null` for a row the export did not carry (a skipped row, or a row a filter cut), which the entry reads as `None`. It was `Option>` of the exported rows only.
+- **Breaking (ABI v2 dev): `AtMergeEntry::row` is `AtMergeEntry::part_row`**, the document's `part_row`.
+- **Breaking (ABI v2 dev): `SchemaServer::settings` is `BTreeMap`** (was `BTreeMap`), because a server profile's setting value can be bytes that are not valid UTF-8.
- `chtypes list` and the tag listing name the published lines only (two-part, numeric order), as Go's always did; `chtypes resolve ` prints the exact version a line resolves to (#613).
- **Breaking: the public surface the parity gate carried as exceptions is resolved, before the v2 lock (#500).** A convenience exists in every binding or in none (`docs/reference/bindings-v1.md` §1, principle 2). Migration, old → new:
- `Refusal.detail` → `Refusal.got`: what a one-sided refusal found (the `dlopen` error, what a malformed `build_info` lacks) is in `got`, with `want` `None`, as in the other bindings; the display still prints it. `Refusal` is `#[non_exhaustive]`, so build one only through the crate.
@@ -72,6 +82,7 @@ The four bindings in this repository are released together and give one answer,
### Removed
+- **Breaking (ABI v2 dev): `RowResult::unsupported_settings`**: the row document no longer carries `unsupported_settings`; the batch's and the filter result's lists stay.
- `SchemaServer::filter_declined_settings` (a breaking change inside v2 dev, #588): the server member no longer carries the list. Read `SchemaDescription::filter_declined_settings`, whose `server`-layer entries are the profile's.
### Fixed
diff --git a/rust/src/abi2/calls_gen.rs b/rust/src/abi2/calls_gen.rs
index ff006ac6..b0760d2a 100644
--- a/rust/src/abi2/calls_gen.rs
+++ b/rust/src/abi2/calls_gen.rs
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//
// The typed, memory-safe call layer over `decls.rs`: one method per
// status-returning `api` call, named without the `chs_` prefix, that copies
@@ -22,6 +22,7 @@ pub(crate) struct RawCallError {
pub(crate) ch_name: String,
pub(crate) message: Vec,
pub(crate) column: Vec,
+ pub(crate) reason: String,
}
/// Copy one owned `chs_buf *` into a `Vec` and free it. NULL reads as
@@ -63,6 +64,7 @@ unsafe fn read_free_error(api: &Api, status: i32, err: *mut ChsError) -> RawCall
ch_name: String::new(),
message: b"the library returned a non-OK status with no error object".to_vec(),
column: Vec::new(),
+ reason: String::new(),
};
}
// SAFETY: the caller's `# Safety` clause; every accessor reads the live
@@ -76,6 +78,11 @@ unsafe fn read_free_error(api: &Api, status: i32, err: *mut ChsError) -> RawCall
.into_owned();
let message = copy_free_buf(api, (api.chs_error_message)(err as *const ChsError));
let column = copy_free_buf(api, (api.chs_error_column)(err as *const ChsError));
+ let reason = String::from_utf8_lossy(©_free_buf(
+ api,
+ (api.chs_error_reason)(err as *const ChsError),
+ ))
+ .into_owned();
(api.chs_error_free)(err);
RawCallError {
status,
@@ -83,6 +90,7 @@ unsafe fn read_free_error(api: &Api, status: i32, err: *mut ChsError) -> RawCall
ch_name,
message,
column,
+ reason,
}
}
}
@@ -97,6 +105,7 @@ fn missing_output(status: i32, what: &str) -> RawCallError {
ch_name: String::new(),
message: format!("the library answered OK and returned no {what}").into_bytes(),
column: Vec::new(),
+ reason: String::new(),
}
}
@@ -466,6 +475,7 @@ impl Api {
body: &[u8],
settings: &[u8],
columns: &[u8],
+ options: &[u8],
) -> Result, RawCallError> {
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
@@ -481,6 +491,8 @@ impl Api {
settings.len(),
columns.as_ptr(),
columns.len(),
+ options.as_ptr(),
+ options.len(),
&mut out_out,
&mut err,
)
@@ -506,6 +518,7 @@ impl Api {
filter: Option<&FilterHandle>,
export_format: i32,
doc_flags: u32,
+ options: &[u8],
) -> Result<(Vec, Option>), RawCallError> {
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut out_out_export: *mut ChsBuf = std::ptr::null_mut();
@@ -525,6 +538,8 @@ impl Api {
filter.map_or(std::ptr::null(), |h| h.as_ptr()),
export_format,
doc_flags,
+ options.as_ptr(),
+ options.len(),
&mut out_out,
&mut out_out_export,
&mut err,
@@ -553,6 +568,7 @@ impl Api {
expr: &[u8],
query_params: &[u8],
settings: &[u8],
+ options: &[u8],
) -> Result {
let mut out_out: *mut ChsFilter = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
@@ -567,6 +583,8 @@ impl Api {
query_params.len(),
settings.as_ptr(),
settings.len(),
+ options.as_ptr(),
+ options.len(),
&mut out_out,
&mut err,
)
@@ -592,6 +610,8 @@ impl Api {
format: i32,
body: &[u8],
settings: &[u8],
+ columns: &[u8],
+ options: &[u8],
) -> Result, RawCallError> {
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
@@ -605,6 +625,10 @@ impl Api {
body.len(),
settings.as_ptr(),
settings.len(),
+ columns.as_ptr(),
+ columns.len(),
+ options.as_ptr(),
+ options.len(),
&mut out_out,
&mut err,
)
@@ -627,6 +651,7 @@ impl Api {
body: &[u8],
settings: &[u8],
columns: &[u8],
+ options: &[u8],
) -> Result {
let mut out_out: *mut ChsBlock = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
@@ -642,6 +667,8 @@ impl Api {
settings.len(),
columns.as_ptr(),
columns.len(),
+ options.as_ptr(),
+ options.len(),
&mut out_out,
&mut err,
)
diff --git a/rust/src/abi2/decls.rs b/rust/src/abi2/decls.rs
index 229ea88c..22647b71 100644
--- a/rust/src/abi2/decls.rs
+++ b/rust/src/abi2/decls.rs
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//
// The opaque handle types, the `extern "C"` function-pointer type for
// every described function, the resolved symbol tables (`Handshake`, the
@@ -21,7 +21,7 @@ pub(crate) const CHS_ABI_VERSION: i32 = 2;
/// form of spec/abi-v2/abi.json (D1.1). Compared byte for byte against
/// `chs_build_info()`'s `abi_fingerprint` field at loader step 4.
pub(crate) const CHS_ABI_FINGERPRINT: &str =
- "sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4";
+ "sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2";
/// `CHS_ABI_STABILITY`: the description's stability, `"unstable"` while this
/// generation is being designed (so `CHS_ABI_FINGERPRINT` moves with every
@@ -174,6 +174,7 @@ pub(crate) type FnChsErrorChCode = unsafe extern "C" fn(*const ChsError) -> i32;
pub(crate) type FnChsErrorChName = unsafe extern "C" fn(*const ChsError) -> *mut ChsBuf;
pub(crate) type FnChsErrorMessage = unsafe extern "C" fn(*const ChsError) -> *mut ChsBuf;
pub(crate) type FnChsErrorColumn = unsafe extern "C" fn(*const ChsError) -> *mut ChsBuf;
+pub(crate) type FnChsErrorReason = unsafe extern "C" fn(*const ChsError) -> *mut ChsBuf;
pub(crate) type FnChsErrorFree = unsafe extern "C" fn(*mut ChsError);
pub(crate) type FnChsLiveHandles =
unsafe extern "C" fn(*mut *mut ChsBuf, *mut *mut ChsError) -> i32;
@@ -228,6 +229,8 @@ pub(crate) type FnChsPreviewRow = unsafe extern "C" fn(
usize,
*const u8,
usize,
+ *const u8,
+ usize,
*mut *mut ChsBuf,
*mut *mut ChsError,
) -> i32;
@@ -243,6 +246,8 @@ pub(crate) type FnChsPreviewBatch = unsafe extern "C" fn(
*const ChsFilter,
i32,
u32,
+ *const u8,
+ usize,
*mut *mut ChsBuf,
*mut *mut ChsBuf,
*mut *mut ChsError,
@@ -255,6 +260,8 @@ pub(crate) type FnChsFilterCreate = unsafe extern "C" fn(
usize,
*const u8,
usize,
+ *const u8,
+ usize,
*mut *mut ChsFilter,
*mut *mut ChsError,
) -> i32;
@@ -266,6 +273,10 @@ pub(crate) type FnChsFilterEvalBody = unsafe extern "C" fn(
usize,
*const u8,
usize,
+ *const u8,
+ usize,
+ *const u8,
+ usize,
*mut *mut ChsBuf,
*mut *mut ChsError,
) -> i32;
@@ -278,6 +289,8 @@ pub(crate) type FnChsBlockCreate = unsafe extern "C" fn(
usize,
*const u8,
usize,
+ *const u8,
+ usize,
*mut *mut ChsBlock,
*mut *mut ChsError,
) -> i32;
@@ -346,6 +359,7 @@ pub(crate) struct Api {
pub(crate) chs_error_ch_name: Symbol,
pub(crate) chs_error_message: Symbol,
pub(crate) chs_error_column: Symbol,
+ pub(crate) chs_error_reason: Symbol,
pub(crate) chs_error_free: Symbol,
pub(crate) chs_live_handles: Symbol,
pub(crate) chs_error_codes: Symbol,
@@ -423,6 +437,9 @@ impl Api {
chs_error_column: lib
.get(concat!("chs_error_column", "\0").as_bytes())
.map_err(|_| "chs_error_column")?,
+ chs_error_reason: lib
+ .get(concat!("chs_error_reason", "\0").as_bytes())
+ .map_err(|_| "chs_error_reason")?,
chs_error_free: lib
.get(concat!("chs_error_free", "\0").as_bytes())
.map_err(|_| "chs_error_free")?,
diff --git a/rust/src/abi2/errmap_gen.rs b/rust/src/abi2/errmap_gen.rs
index da776a0e..6dc1545d 100644
--- a/rust/src/abi2/errmap_gen.rs
+++ b/rust/src/abi2/errmap_gen.rs
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//
// spec/abi-v2/sdk.json's error table: which public error class each call
// status and each loader refusal maps to. See scripts/abi-v1/emit/rust.py.
diff --git a/rust/src/abi2/invoke_gen.rs b/rust/src/abi2/invoke_gen.rs
index 5a67f211..ce8a1694 100644
--- a/rust/src/abi2/invoke_gen.rs
+++ b/rust/src/abi2/invoke_gen.rs
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//
// The test-only invoke-by-name dispatcher: `invoke(api, name, args)` marshals
// an already-resolved argument list (`ResolvedArg`, rust/src/abi2/conformance.rs)
@@ -517,6 +517,7 @@ pub(crate) unsafe fn invoke(
let a_body = cur.bytes()?;
let a_settings = cur.bytes()?;
let a_columns = cur.bytes()?;
+ let a_options = cur.bytes()?;
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
// SAFETY: every input is marshaled from the case's own recipe above, matching chs_preview_row's declared shape; every output pointer is this call's own local.
@@ -530,6 +531,8 @@ pub(crate) unsafe fn invoke(
a_settings.len(),
a_columns.as_ptr(),
a_columns.len(),
+ a_options.as_ptr(),
+ a_options.len(),
&mut out_out,
&mut err,
)
@@ -551,6 +554,7 @@ pub(crate) unsafe fn invoke(
a_filter.map_or(std::ptr::null_mut(), |p| p as *mut ChsFilter) as *const ChsFilter;
let a_export_format = cur.int()? as i32;
let a_doc_flags = cur.int()? as u32;
+ let a_options = cur.bytes()?;
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut out_out_export: *mut ChsBuf = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
@@ -568,6 +572,8 @@ pub(crate) unsafe fn invoke(
a_filter,
a_export_format,
a_doc_flags,
+ a_options.as_ptr(),
+ a_options.len(),
&mut out_out,
&mut out_out_export,
&mut err,
@@ -585,6 +591,7 @@ pub(crate) unsafe fn invoke(
let a_expr = cur.bytes()?;
let a_query_params = cur.bytes()?;
let a_settings = cur.bytes()?;
+ let a_options = cur.bytes()?;
let mut out_out: *mut ChsFilter = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
// SAFETY: every input is marshaled from the case's own recipe above, matching chs_filter_create's declared shape; every output pointer is this call's own local.
@@ -597,6 +604,8 @@ pub(crate) unsafe fn invoke(
a_query_params.len(),
a_settings.as_ptr(),
a_settings.len(),
+ a_options.as_ptr(),
+ a_options.len(),
&mut out_out,
&mut err,
)
@@ -612,6 +621,8 @@ pub(crate) unsafe fn invoke(
let a_format = cur.int()? as i32;
let a_body = cur.bytes()?;
let a_settings = cur.bytes()?;
+ let a_columns = cur.bytes()?;
+ let a_options = cur.bytes()?;
let mut out_out: *mut ChsBuf = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
// SAFETY: every input is marshaled from the case's own recipe above, matching chs_filter_eval_body's declared shape; every output pointer is this call's own local.
@@ -623,6 +634,10 @@ pub(crate) unsafe fn invoke(
a_body.len(),
a_settings.as_ptr(),
a_settings.len(),
+ a_columns.as_ptr(),
+ a_columns.len(),
+ a_options.as_ptr(),
+ a_options.len(),
&mut out_out,
&mut err,
)
@@ -639,6 +654,7 @@ pub(crate) unsafe fn invoke(
let a_body = cur.bytes()?;
let a_settings = cur.bytes()?;
let a_columns = cur.bytes()?;
+ let a_options = cur.bytes()?;
let mut out_out: *mut ChsBlock = std::ptr::null_mut();
let mut err: *mut ChsError = std::ptr::null_mut();
// SAFETY: every input is marshaled from the case's own recipe above, matching chs_block_create's declared shape; every output pointer is this call's own local.
@@ -652,6 +668,8 @@ pub(crate) unsafe fn invoke(
a_settings.len(),
a_columns.as_ptr(),
a_columns.len(),
+ a_options.as_ptr(),
+ a_options.len(),
&mut out_out,
&mut err,
)
diff --git a/rust/src/abi2/vocab_gen.rs b/rust/src/abi2/vocab_gen.rs
index b0544f8a..3603d5fa 100644
--- a/rust/src/abi2/vocab_gen.rs
+++ b/rust/src/abi2/vocab_gen.rs
@@ -1,4 +1,4 @@
-// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d60a681ea0e249ab9e2c83b50640ad9cef040261ddf61805106c93e2c68774e4) — DO NOT EDIT
+// GENERATED by scripts/abi-v1/gen.py --major 2 from spec/abi-v2/abi.json (CHS_ABI_FINGERPRINT sha256:d53239933966e15a3c363e631a94fbba1231d84c4300337115d3ae5377bb41c2) — DO NOT EDIT
//
// Every vocabulary of the description, as Rust: the `chs_format` and
// `chs_status` numbers, the document vocabularies (outcomes, verdicts,
@@ -534,6 +534,8 @@ pub enum DeclinedTier {
PredicateUnflipped,
/// The wire value `result-content`.
ResultContent,
+ /// The wire value `server-refuses`.
+ ServerRefuses,
/// A value the description does not list, carrying its exact spelling (rule r3,
/// spec/abi-v2/docs.md): a reader keeps it, for this field alone, and decodes on.
Unknown(String),
@@ -546,6 +548,7 @@ impl DeclinedTier {
DeclinedTier::Predicate => "predicate",
DeclinedTier::PredicateUnflipped => "predicate-unflipped",
DeclinedTier::ResultContent => "result-content",
+ DeclinedTier::ServerRefuses => "server-refuses",
DeclinedTier::Unknown(raw) => raw.as_str(),
}
}
@@ -557,6 +560,7 @@ impl DeclinedTier {
"predicate" => DeclinedTier::Predicate,
"predicate-unflipped" => DeclinedTier::PredicateUnflipped,
"result-content" => DeclinedTier::ResultContent,
+ "server-refuses" => DeclinedTier::ServerRefuses,
_ => DeclinedTier::Unknown(s.to_string()),
}
}
@@ -626,6 +630,165 @@ impl std::fmt::Display for DeclinedLayer {
}
}
+/// How a schema's statement was compiled: `create`, as a server's CREATE TABLE, or `load`, as a server
+/// loads the stored definition of a table it holds. A posture the description does not list is `Unknown`,
+/// and the document decodes on (rule r3).
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum Posture {
+ /// The wire value `create`.
+ Create,
+ /// The wire value `load`.
+ Load,
+ /// A value the description does not list, carrying its exact spelling (rule r3,
+ /// spec/abi-v2/docs.md): a reader keeps it, for this field alone, and decodes on.
+ Unknown(String),
+}
+
+impl Posture {
+ /// The wire spelling: the description's, or the raw value `Unknown` carries.
+ pub fn as_str(&self) -> &str {
+ match self {
+ Posture::Create => "create",
+ Posture::Load => "load",
+ Posture::Unknown(raw) => raw.as_str(),
+ }
+ }
+
+ /// Read one wire value. One the description does not list is `Unknown`,
+ /// carrying it (rule r3): it never fails the document.
+ pub fn from_wire(s: &str) -> Self {
+ match s {
+ "create" => Posture::Create,
+ "load" => Posture::Load,
+ _ => Posture::Unknown(s.to_string()),
+ }
+ }
+
+ /// Whether the description lists this value: false exactly for `Unknown`.
+ pub fn is_known(&self) -> bool {
+ !matches!(self, Posture::Unknown(_))
+ }
+}
+
+/// The wire spelling (`as_str`), padded to a width the format asks for.
+impl std::fmt::Display for Posture {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.pad(self.as_str())
+ }
+}
+
+/// Where a batch or filter document's refusal sits: the call, the body outside any row (a header, the
+/// framing), one row (`refused_row`) or a whole-body step. A site the description does not list is
+/// `Unknown`, and the document decodes on (rule r3).
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum RefusalSite {
+ /// The wire value `call`.
+ Call,
+ /// The wire value `body_read`.
+ BodyRead,
+ /// The wire value `row`.
+ Row,
+ /// The wire value `body_step`.
+ BodyStep,
+ /// A value the description does not list, carrying its exact spelling (rule r3,
+ /// spec/abi-v2/docs.md): a reader keeps it, for this field alone, and decodes on.
+ Unknown(String),
+}
+
+impl RefusalSite {
+ /// The wire spelling: the description's, or the raw value `Unknown` carries.
+ pub fn as_str(&self) -> &str {
+ match self {
+ RefusalSite::Call => "call",
+ RefusalSite::BodyRead => "body_read",
+ RefusalSite::Row => "row",
+ RefusalSite::BodyStep => "body_step",
+ RefusalSite::Unknown(raw) => raw.as_str(),
+ }
+ }
+
+ /// Read one wire value. One the description does not list is `Unknown`,
+ /// carrying it (rule r3): it never fails the document.
+ pub fn from_wire(s: &str) -> Self {
+ match s {
+ "call" => RefusalSite::Call,
+ "body_read" => RefusalSite::BodyRead,
+ "row" => RefusalSite::Row,
+ "body_step" => RefusalSite::BodyStep,
+ _ => RefusalSite::Unknown(s.to_string()),
+ }
+ }
+
+ /// Whether the description lists this value: false exactly for `Unknown`.
+ pub fn is_known(&self) -> bool {
+ !matches!(self, RefusalSite::Unknown(_))
+ }
+}
+
+/// The wire spelling (`as_str`), padded to a width the format asks for.
+impl std::fmt::Display for RefusalSite {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.pad(self.as_str())
+ }
+}
+
+/// What this build does with a column's DEFAULT, as a schema description reports it per column. A value
+/// the description does not list is `Unknown`, and the column decodes on (rule r3).
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+#[non_exhaustive]
+pub enum DefaultSupport {
+ /// The wire value `computed`.
+ Computed,
+ /// The wire value `generated`.
+ Generated,
+ /// The wire value `deferred`.
+ Deferred,
+ /// The wire value `declined`.
+ Declined,
+ /// A value the description does not list, carrying its exact spelling (rule r3,
+ /// spec/abi-v2/docs.md): a reader keeps it, for this field alone, and decodes on.
+ Unknown(String),
+}
+
+impl DefaultSupport {
+ /// The wire spelling: the description's, or the raw value `Unknown` carries.
+ pub fn as_str(&self) -> &str {
+ match self {
+ DefaultSupport::Computed => "computed",
+ DefaultSupport::Generated => "generated",
+ DefaultSupport::Deferred => "deferred",
+ DefaultSupport::Declined => "declined",
+ DefaultSupport::Unknown(raw) => raw.as_str(),
+ }
+ }
+
+ /// Read one wire value. One the description does not list is `Unknown`,
+ /// carrying it (rule r3): it never fails the document.
+ pub fn from_wire(s: &str) -> Self {
+ match s {
+ "computed" => DefaultSupport::Computed,
+ "generated" => DefaultSupport::Generated,
+ "deferred" => DefaultSupport::Deferred,
+ "declined" => DefaultSupport::Declined,
+ _ => DefaultSupport::Unknown(s.to_string()),
+ }
+ }
+
+ /// Whether the description lists this value: false exactly for `Unknown`.
+ pub fn is_known(&self) -> bool {
+ !matches!(self, DefaultSupport::Unknown(_))
+ }
+}
+
+/// The wire spelling (`as_str`), padded to a width the format asks for.
+impl std::fmt::Display for DefaultSupport {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.pad(self.as_str())
+ }
+}
+
/// The `transform_reason` vocabulary: why a stored value differs from the supplied one, and the `lossy`
/// fact the description gives each. A reason the description does not list is `Unknown`, and reports
/// the fallback's (`value_changed`) facts, so it reads as lossy.
@@ -855,8 +1018,8 @@ pub mod reason {
}
/// The `value_src` vocabulary: where a column's value came from, and whether the description says it is
-/// stored. A source the description does not list is `Unknown`; the description does not yet say what
-/// its `is_stored` reports (rule r3, "Facts without a fallback"), and until it does, `false`.
+/// stored. A source the description does not list is `Unknown`, and reports its fallback's (`skipped`)
+/// `is_stored` (rule r3).
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
#[non_exhaustive]
pub enum Source {
@@ -882,6 +1045,10 @@ pub enum Source {
DefaultVolatileUnresolved,
/// The wire value `default_pending`.
DefaultPending,
+ /// The wire value `default_deferred`.
+ DefaultDeferred,
+ /// The wire value `default_override`.
+ DefaultOverride,
/// A value the description does not list, carrying its exact spelling (rule r3,
/// spec/abi-v2/docs.md): a reader keeps it, for this field alone, and decodes on.
Unknown(String),
@@ -902,6 +1069,8 @@ impl Source {
Source::DefaultExprUnsupported => "default_expr_unsupported",
Source::DefaultVolatileUnresolved => "default_volatile_unresolved",
Source::DefaultPending => "default_pending",
+ Source::DefaultDeferred => "default_deferred",
+ Source::DefaultOverride => "default_override",
Source::Unknown(raw) => raw.as_str(),
}
}
@@ -921,6 +1090,8 @@ impl Source {
"default_expr_unsupported" => Source::DefaultExprUnsupported,
"default_volatile_unresolved" => Source::DefaultVolatileUnresolved,
"default_pending" => Source::DefaultPending,
+ "default_deferred" => Source::DefaultDeferred,
+ "default_override" => Source::DefaultOverride,
_ => Source::Unknown(s.to_string()),
}
}
@@ -930,8 +1101,7 @@ impl Source {
!matches!(self, Source::Unknown(_))
}
- /// The description's `is_stored` fact for this source; `false` for `Unknown` until the description
- /// says otherwise.
+ /// The description's `is_stored` fact for this source; `Unknown` reports the fallback's (`skipped`).
pub fn is_stored(&self) -> bool {
match self {
Source::Input => true,
@@ -945,6 +1115,8 @@ impl Source {
Source::DefaultExprUnsupported => false,
Source::DefaultVolatileUnresolved => false,
Source::DefaultPending => false,
+ Source::DefaultDeferred => false,
+ Source::DefaultOverride => true,
Source::Unknown(_) => false,
}
}
@@ -983,6 +1155,10 @@ pub mod source {
pub const DEFAULT_VOLATILE_UNRESOLVED: &str = "default_volatile_unresolved";
/// `default_pending`.
pub const DEFAULT_PENDING: &str = "default_pending";
+ /// `default_deferred`.
+ pub const DEFAULT_DEFERRED: &str = "default_deferred";
+ /// `default_override`.
+ pub const DEFAULT_OVERRIDE: &str = "default_override";
/// The description's `is_stored` fact for a source spelling; one the
/// description does not list is `Source::Unknown`, `false` until the
@@ -1103,6 +1279,10 @@ pub(crate) const BYTES_SUFFIX: &str = "_b64";
/// value, beside its rendering.
pub(crate) const VALUE_BYTES: &str = "value_b64";
+/// The code a declined row, outcome or verdict carries (`CHS_CODE_DECLINED`), outside
+/// ClickHouse's code table.
+pub const CODE_DECLINED: i32 = -2;
+
/// Every enum the description defines, in its own order: rule r3's tests reach
/// each one through [`described_vocabulary`], so a new enum is covered without
/// a hand-kept list.
@@ -1116,10 +1296,13 @@ pub(crate) const DESCRIBED_VOCABULARIES: &[&str] = &[
"batch_outcome",
"filter_outcome",
"filter_verdict",
+ "refusal_site",
"discover_query_param",
"default_kind",
+ "default_support",
"declined_tier",
"declined_layer",
+ "posture",
];
/// Build the value of the described enum `name` whose raw spelling is `raw` (an
@@ -1164,6 +1347,10 @@ pub(crate) fn described_vocabulary(name: &str, raw: &str) -> Option<(bool, Strin
let v = Verdict::from_wire(raw);
Some((v.is_known(), v.as_str().to_string()))
}
+ "refusal_site" => {
+ let v = RefusalSite::from_wire(raw);
+ Some((v.is_known(), v.as_str().to_string()))
+ }
"discover_query_param" => {
let v = DiscoverQueryParam::from_wire(raw);
Some((v.is_known(), v.as_str().to_string()))
@@ -1172,6 +1359,10 @@ pub(crate) fn described_vocabulary(name: &str, raw: &str) -> Option<(bool, Strin
let v = DefaultKind::from_wire(raw);
Some((v.is_known(), v.as_str().to_string()))
}
+ "default_support" => {
+ let v = DefaultSupport::from_wire(raw);
+ Some((v.is_known(), v.as_str().to_string()))
+ }
"declined_tier" => {
let v = DeclinedTier::from_wire(raw);
Some((v.is_known(), v.as_str().to_string()))
@@ -1180,6 +1371,10 @@ pub(crate) fn described_vocabulary(name: &str, raw: &str) -> Option<(bool, Strin
let v = DeclinedLayer::from_wire(raw);
Some((v.is_known(), v.as_str().to_string()))
}
+ "posture" => {
+ let v = Posture::from_wire(raw);
+ Some((v.is_known(), v.as_str().to_string()))
+ }
_ => None,
}
}
diff --git a/rust/src/decode.rs b/rust/src/decode.rs
index e7fcd15b..20825203 100644
--- a/rust/src/decode.rs
+++ b/rust/src/decode.rs
@@ -26,8 +26,8 @@ use serde::de::{Deserialize, Deserializer, MapAccess, SeqAccess, Visitor};
use serde_json::{Map, Value as Json};
use crate::abi2::vocab_gen::{
- BYTES_SUFFIX, DeclinedLayer, DeclinedTier, DefaultKind, FilterOutcome, MergeReason, Outcome,
- Reason, Source, VALUE_BYTES, Verdict,
+ BYTES_SUFFIX, DeclinedLayer, DeclinedTier, DefaultKind, DefaultSupport, FilterOutcome,
+ MergeReason, Outcome, Posture, Reason, RefusalSite, Source, VALUE_BYTES, Verdict,
};
use crate::error::{Error, Result};
use crate::raw::RawText;
@@ -273,6 +273,46 @@ impl<'a> Obj<'a> {
.map(Some)
}
+ /// An object whose values are byte values (`json_object_byte_values`): a
+ /// JSON string, or the object `{"b64": ""}` for bytes that
+ /// are not valid UTF-8. `None` when absent, `Some`, even of an empty map,
+ /// when present.
+ fn byte_value_map(&self, key: &str) -> Result