From 4479d26bf7d38b87a8466da28ec3da76f33e7970 Mon Sep 17 00:00:00 2001 From: toporek <363280+toporek@users.noreply.github.com> Date: Mon, 17 Aug 2026 06:48:54 +0000 Subject: [PATCH] chore: sync Zero docs from upstream --- SOURCE.md | 2 +- skills/zero-docs/INDEX.md | 3 + .../references/connecting-to-postgres.md | 14 ++ skills/zero-docs/references/connection.md | 4 +- skills/zero-docs/references/mutators.md | 31 ++-- skills/zero-docs/references/otel.md | 160 +++++++++++++----- .../zero-docs/references/release-notes/1.7.md | 62 +++++++ .../zero-docs/references/release-notes/1.8.md | 82 +++++++++ .../zero-docs/references/release-notes/1.9.md | 55 ++++++ .../references/release-notes/index.md | 3 + skills/zero-docs/references/schema.md | 14 +- skills/zero-docs/references/self-host.md | 31 ++-- .../zero-docs/references/zero-cache-config.md | 64 ++++++- skills/zero-docs/references/zql.md | 14 +- 14 files changed, 450 insertions(+), 89 deletions(-) create mode 100644 skills/zero-docs/references/release-notes/1.7.md create mode 100644 skills/zero-docs/references/release-notes/1.8.md create mode 100644 skills/zero-docs/references/release-notes/1.9.md diff --git a/SOURCE.md b/SOURCE.md index 044537d..254941e 100644 --- a/SOURCE.md +++ b/SOURCE.md @@ -1,5 +1,5 @@ # Source Doc list and metadata from https://github.com/rocicorp/zero-docs -Upstream commit: b37d0de26315c8bc51989ada51021755be249a6c +Upstream commit: 6d3c69978b2b662f886bbc949916fcd037104384 Page bodies fetched from https://zero.rocicorp.dev/docs/{path} (build-rendered markdown) diff --git a/skills/zero-docs/INDEX.md b/skills/zero-docs/INDEX.md index a534bba..4acf0c0 100644 --- a/skills/zero-docs/INDEX.md +++ b/skills/zero-docs/INDEX.md @@ -84,4 +84,7 @@ - [Zero 1.4](references/release-notes/1.4.md) — Performance and Reliability Improvements - [Zero 1.5](references/release-notes/1.5.md) — Schema Change Improvements and Client Group Auth - [Zero 1.6](references/release-notes/1.6.md) — PlanetScale Failover Support +- [Zero 1.7](references/release-notes/1.7.md) — Query Correctness and Performance +- [Zero 1.8](references/release-notes/1.8.md) — Observability and Reliability +- [Zero 1.9](references/release-notes/1.9.md) — Stability and Query Correctness - [Release Notes](references/release-notes/index.md) diff --git a/skills/zero-docs/references/connecting-to-postgres.md b/skills/zero-docs/references/connecting-to-postgres.md index d40988e..9e12035 100644 --- a/skills/zero-docs/references/connecting-to-postgres.md +++ b/skills/zero-docs/references/connecting-to-postgres.md @@ -55,6 +55,20 @@ After your server restarts, show the `wal_level` again to ensure it has changed: psql -c 'SHOW wal_level' ``` +### Socket Inactivity Timeout + +`zero-cache` monitors wire activity on its Postgres connections so it can recover when a proxy or network failure leaves a half-open socket. The watchdog samples each connection every 120,000 milliseconds and resets it after one to two intervals without any bytes read or written. In-flight queries on a reset connection are rejected and can recover through their normal retry or restart paths. + +Wire activity resets the watchdog, so streaming operations such as `COPY` remain active. A statement that legitimately computes without sending any data for several minutes can be interrupted. + +### WAL Sender Timeout + +`zero-cache` uses Postgres's `wal_sender_timeout` setting to monitor its replication connection. When the timeout is greater than `0`, Zero sends keepalives and reconnects if the replication stream stops responding. The inbound timeout defaults to twice `wal_sender_timeout`. + +A healthy WAL sender can sometimes remain silent longer than this while decoding WAL from unpublished tables or assembling a large transaction. Set [`ZERO_UPSTREAM_PG_STREAM_INBOUND_TIMEOUT_MS`](zero-cache-config.md#upstream-pg-stream-inbound-timeout) to widen Zero's inbound threshold without changing the server's timeout. Manual keepalive timing remains derived from `wal_sender_timeout`. + +Setting `wal_sender_timeout` to `0` disables the timeout in Postgres and the related keepalive and reconnect checks in Zero, even when an inbound timeout override is configured. Other connection failure detection remains active. + ### Bounding WAL Size For development databases, you can set a `max_slot_wal_keep_size` value in Postgres. This will help limit the amount of WAL kept around. diff --git a/skills/zero-docs/references/connection.md b/skills/zero-docs/references/connection.md index 2b06d65..ae57a00 100644 --- a/skills/zero-docs/references/connection.md +++ b/skills/zero-docs/references/connection.md @@ -153,9 +153,9 @@ Reads are allowed while `disconnected`, but writes are rejected and return an of ### Error -If `zero-cache` itself crashes, or if the [mutate](mutators.md) or [query](queries.md) endpoints return a network or HTTP error, Zero transitions to the `error` state. +If `zero-cache` crashes, or [mutate](mutators.md) or [query](queries.md) endpoints fail, Zero enters the `error` state. If the response code is `5xx`, `zero-cache` will retry up to four times. -This type of error is unlikely to resolve just by retrying, so Zero doesn't try. The app can retry the connection manually by calling `zero.connection.connect()`. +Zero does not retry from the `error` state. Call `zero.connection.connect()` to retry manually. Reads are allowed while in the `error` state, but writes are rejected. diff --git a/skills/zero-docs/references/mutators.md b/skills/zero-docs/references/mutators.md index 2c8dd9c..a9cc552 100644 --- a/skills/zero-docs/references/mutators.md +++ b/skills/zero-docs/references/mutators.md @@ -93,6 +93,8 @@ tx.mutate.user.insert({ }) ``` +If the Zero primary key already exists, `insert` will succeed without changing the row - use `upsert` to update an existing row. + Optional fields can be set to `null` to explicitly set the new field to `null`. They can also be set to `undefined` to take the default value (which is often `null` but can also be some generated value server-side): ```tsx @@ -782,7 +784,7 @@ app.post('/api/zero/mutate', async c => { }) ``` -If Zero receives any response from the mutate endpoint other than HTTP 200, 401, or 403, it will disconnect and enter the [error state](connection.md#error). +Responses other than 200, 401, or 403 enter the [error state](connection.md#error). `zero-cache` will retry on `5xx` up to four times before returning an error. If Zero receives HTTP 401 or 403, the client will enter the needs auth state and require a manual reconnect. Use `zero.connection.connect()` for cookie auth or `zero.connection.connect({auth: newToken})` for token auth, then Zero will retry all queued mutations. @@ -936,7 +938,7 @@ const read2 = await zero.run( ) ``` -You can also wait for the server write to succeed: +You can also await `.server` for the server result: ```ts const write = zero.mutate( @@ -948,10 +950,9 @@ const write = zero.mutate( const clientRes = await write.client if (clientRes.type === 'error') { - throw new Error( - `Mutator failed on client`, - clientRes.error - ) + throw new Error(`Mutator failed on client`, { + cause: clientRes.error + }) } // optimistic write guaranteed to be present here, but not @@ -960,25 +961,23 @@ const read1 = await zero.run( queries.issue.byId('issue-123').one() ) -// Await server write – this involves a round-trip. +// Await the server result/acknowledgment. This requires a round trip. const serverRes = await write.server if (serverRes.type === 'error') { - throw new Error( - `Mutator failed on server`, - serverRes.error - ) + throw new Error(`Mutator failed on server`, { + cause: serverRes.error + }) } -// issue-123 is written to server and any results are -// synced to this client. -// read2 could potentially be undefined here, for example if the -// server mutator rejected the write. +// The server acknowledged the mutation, but its Postgres changes +// may not have replicated to this client yet. This read can still +// reflect optimistic rather than authoritative state. const read2 = await zero.run( queries.issue.byId('issue-123').one() ) ``` -If the client-side mutator fails, the `.server` promise is also rejected with the same error. You don't have to listen to both promises, the server promise covers both cases. +If the client-side mutator fails, `.server` also resolves to an error result. Awaiting `.server` therefore covers both client- and server-side failures. > **Returning data from mutators**: There is not yet a way to return data from mutators in the success case. [Let us know](https://discord.rocicorp.dev/)if you need this. diff --git a/skills/zero-docs/references/otel.md b/skills/zero-docs/references/otel.md index c975339..7478d66 100644 --- a/skills/zero-docs/references/otel.md +++ b/skills/zero-docs/references/otel.md @@ -95,65 +95,133 @@ This callback is called before sending WebSocket messages that trigger API serve ## Metrics Reference +> **Histogram support**: `zero_sync_view_syncer_lag`, `zero_sync_view_syncer_hydration`, and `zero_sync_e2e_serving_lag` require OpenTelemetry exponential histogram support. Prometheus users must enable native histograms. Use the existing `zero_sync_serving_lag` gauges if your backend does not support them. + ### zero.server -| Metric | Type | Unit | Description | -| -------- | ----- | ---- | --------------------------------------------------------- | -| `uptime` | Gauge | s | Cumulative uptime, starting from when requests are served | +| Metric | Type | Unit | Description | +| ------------------------------------- | ------------- | ---- | ------------------------------------------------------------------------------------- | +| `zero_server_uptime` | Gauge | s | Cumulative uptime, starting from when requests are served | +| `zero_server_api_requests` | Counter | | Calls to user mutate and query APIs, including cleanup and auth-validation operations | +| `zero_server_api_request_duration` | Histogram | s | End-to-end user API request duration, including retries | +| `zero_server_api_attempts` | Counter | | HTTP fetch attempts made while calling user API endpoints | +| `zero_server_api_attempt_duration` | Histogram | s | Duration of each API HTTP attempt, excluding retry delays | +| `zero_server_api_in_flight` | UpDownCounter | | API requests currently in flight | +| `zero_server_startup_duration` | Histogram | s | Time from starting `zero-cache` until it is ready | +| `zero_server_worker_startup_duration` | Histogram | s | Time from starting a worker until it is ready | ### zero.replica -| Metric | Type | Unit | Description | -| ------------ | ----- | ----- | ----------------------------------------------------------------------------------------------------------------------- | -| `db_size` | Gauge | bytes | Size of the replica's main db file (excludes WAL) | -| `wal_size` | Gauge | bytes | Size of the replica's WAL file | -| `wal2_size` | Gauge | bytes | Size of the replica's WAL2 file (only if using wal2 mode) | -| `backup_lag` | Gauge | ms | Time since last litestream backup. Expected to sawtooth from 0 to `ZERO_LITESTREAM_INCREMENTAL_BACKUP_INTERVAL_MINUTES` | +| Metric | Type | Unit | Description | +| ------------------------------------------------------- | --------- | ----- | ----------------------------------------------------------------------------------------------------------------------- | +| `zero_replica_db_size` | Gauge | bytes | Size of the replica's main db file (excludes WAL) | +| `zero_replica_wal_size` | Gauge | bytes | Size of the replica's WAL file | +| `zero_replica_wal2_size` | Gauge | bytes | Size of the replica's WAL2 file (only if using wal2 mode) | +| `zero_replica_backup_lag` | Gauge | ms | Time since last litestream backup. Expected to sawtooth from 0 to `ZERO_LITESTREAM_INCREMENTAL_BACKUP_INTERVAL_MINUTES` | +| `zero_replica_purge_blocked` | Counter | | Number of change-log purges blocked because the actual backup state could not be verified or is stale | +| `zero_replica_litestream_restore_runs` | Counter | | Litestream restore runs | +| `zero_replica_litestream_restore_attempts` | Counter | | Litestream restore subprocess attempts | +| `zero_replica_litestream_restore_db_bytes` | Counter | bytes | SQLite database bytes restored by successful Litestream restores | +| `zero_replica_litestream_restore_duration` | Histogram | s | Wall-clock duration of Litestream restore runs | +| `zero_replica_litestream_restore_wait_duration` | Histogram | s | Time spent waiting for replication-manager snapshot status before restoring | +| `zero_replica_litestream_restore_process_duration` | Histogram | s | Wall-clock duration of Litestream restore subprocesses | +| `zero_replica_litestream_restore_validation_duration` | Histogram | s | Time spent validating restored replica databases | +| `zero_replica_litestream_backup_process_runs` | Counter | | Litestream backup process exits | +| `zero_replica_litestream_backup_process_duration` | Histogram | s | Runtime of Litestream backup subprocesses before exit | +| `zero_replica_litestream_backup_list_duration` | Histogram | s | Time to list the Litestream backup destination | +| `zero_replica_litestream_backup_verification_duration` | Histogram | s | Time to verify backup state in the destination | +| `zero_replica_litestream_snapshot_reservation_duration` | Histogram | s | Time snapshot reservations are held while view-syncers restore and subscribe | ### zero.replication -| Metric | Type | Unit | Description | -| ---------------------- | --------- | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `upstream_lag` | Gauge | ms | Latency from sending a replication report to receiving it in the stream | -| `replica_lag` | Gauge | ms | Latency from receiving a replication report to it reaching the replica | -| `total_lag` | Gauge | ms | End-to-end replication latency. Grows as an estimate if the next report hasn't arrived | -| `events` | Counter | | Number of replication events processed | -| `transactions` | Counter | | Count of replicated transactions | -| `shadow-sync-runs` | Counter | | Number of [shadow initial-sync](zero-cache-config.md#shadow-sync-enabled) runs. Has a `result` attribute: `success`, `error` | -| `shadow-sync-duration` | Histogram | s | Wall-clock duration of a shadow initial-sync run. Has a `result` attribute: `success`, `error` | +| Metric | Type | Unit | Description | +| ----------------------------------------------------- | --------- | ----- | ------------------------------------------------------------------------------------------------------------------------------- | +| `zero_replication_upstream_lag` | Gauge | ms | Latency from sending a replication report to receiving it in the stream | +| `zero_replication_replica_lag` | Gauge | ms | Latency from receiving a replication report to it reaching the replica | +| `zero_replication_total_lag` | Gauge | ms | Measured end-to-end latency of the most recently received replication report; does not grow if reports stop arriving | +| `zero_replication_last_total_lag` | Gauge | ms | Alias of `zero_replication_total_lag`, retained for dashboards that explicitly use the non-extrapolated metric | +| `zero_replication_upstream_clock_skew` | Gauge | ms | Estimated offset of the upstream database clock relative to `zero-cache`; positive values mean upstream is ahead | +| `zero_replication_lag_report_retries` | Counter | | Replication lag reports retried because an expected report did not arrive before the next report interval | +| `zero_replication_events` | Counter | | Number of replication events processed | +| `zero_replication_transactions` | Counter | | Count of replicated transactions | +| `zero_replication_changes` | Counter | | Count of replicated changes, including DML and DDL statements | +| `zero_replication_slot_health` | Gauge | 1 | One-hot status for the active logical replication slot: `ok`, `unreserved`, `lost`, `missing`, or `unknown` | +| `zero_replication_slot_retained_wal_bytes` | Gauge | bytes | WAL bytes retained by the active logical replication slot | +| `zero_replication_slot_safe_wal_bytes` | Gauge | bytes | Remaining WAL capacity before the active logical replication slot is lost; omitted when Postgres reports no value | +| `zero_replication_initial_sync_runs` | Counter | | Number of initial-sync runs | +| `zero_replication_initial_sync_duration` | Histogram | s | Wall-clock duration of an initial-sync run | +| `zero_replication_initial_sync_copy_duration` | Histogram | s | Wall-clock duration of the COPY phase for a successful initial-sync run | +| `zero_replication_initial_sync_copy_other_duration` | Histogram | s | Initial-sync duration excluding SQLite flush and index time for a successful run | +| `zero_replication_initial_sync_flush_duration` | Histogram | s | Total SQLite flush time for a successful initial-sync run | +| `zero_replication_initial_sync_index_duration` | Histogram | s | SQLite index creation time for a successful initial-sync run | +| `zero_replication_initial_sync_rows` | Counter | | Rows copied during successful initial-sync runs | +| `zero_replication_initial_sync_copy_stream` | Counter | bytes | PostgreSQL COPY stream bytes, including failed runs; reported in approximately 8 MiB batches and flushed when the stream ends | +| `zero_replication_initial_sync_completed_copy_stream` | Counter | bytes | PostgreSQL COPY stream bytes processed during successful initial-sync runs | +| `zero_replication_initial_sync_copy_chunks` | Counter | | PostgreSQL COPY stream chunks processed during initial sync; batched with COPY-stream updates and flushed when the stream ends | +| `zero_replication_shadow_sync_runs` | Counter | | Number of [shadow initial-sync](zero-cache-config.md#shadow-sync-enabled) runs, labeled by `result` | +| `zero_replication_shadow_sync_duration` | Histogram | s | Wall-clock duration of a shadow initial-sync run, labeled by `result` | +| `zero_replication_flow_control_active_subscribers` | Gauge | | Active change-stream subscribers receiving live changes | +| `zero_replication_flow_control_queued_subscribers` | Gauge | | Change-stream subscribers waiting for the current transaction to finish before activation | +| `zero_replication_flow_control_pending_messages` | Gauge | | Downstream change-stream messages not yet acknowledged by subscribers | +| `zero_replication_flow_control_backlog_messages` | Gauge | | Live change-stream messages buffered while subscribers catch up | +| `zero_replication_flow_control_backlog_bytes` | Gauge | bytes | Live change-stream bytes buffered while subscribers catch up | +| `zero_replication_flow_control_max_backlog_bytes` | Gauge | bytes | Maximum live change-stream bytes buffered by a single subscriber | +| `zero_replication_flow_control_waits` | Counter | | Completed flow-control checkpoints | +| `zero_replication_flow_control_wait_duration` | Histogram | s | Time replication waits at flow-control checkpoints | + +`zero_replication_total_lag` and `zero_replication_last_total_lag` now report the same latest measured round trip and do not grow when reports stop arriving. Use `zero_replication_lag_report_retries` to detect a stalled or missing report stream. ### zero.sync -| Metric | Type | Unit | Description | -| ----------------------------------- | ------------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `max-protocol-version` | Gauge | | Highest sync protocol version seen from connecting clients | -| `active-clients` | UpDownCounter | | Number of currently connected sync clients | -| `active-client-groups` | Gauge | | Number of active ViewSyncerService instances in a syncer worker | -| `queries` | Gauge | | Active IVM pipelines across all client groups in a syncer worker | -| `rows` | Gauge | | CVR-tracked rows across all client groups in a syncer worker | -| `lock-wait-time` | Histogram | s | Time spent waiting to acquire the ViewSyncerService lock per operation | -| `pipeline-resets` | Counter | | Count of pipeline resets. Has a `reason` attribute: `advancement-timeout`, `scalar-subquery`, `schema-change`, `truncation`, `permissions-change` | -| `hydration` | Counter | | Number of query hydrations | -| `hydration-time` | Histogram | s | Time to hydrate a query | -| `advance-time` | Histogram | s | Time to advance all queries for a client group after applying a transaction | -| `poke.time` | Histogram | s | Time per poke transaction (excludes canceled/noop pokes) | -| `poke.transactions` | Counter | | Count of poke transactions | -| `poke.rows` | Counter | | Count of poked rows | -| `cvr.flush-time` | Histogram | s | Time to flush a CVR transaction | -| `cvr.rows-flushed` | Counter | | Number of changed rows flushed to a CVR | -| `ivm.advance-time` | Histogram | s | Time to advance IVM queries in response to a single change | -| `ivm.conflict-rows-deleted` | Counter | | Rows deleted because they conflicted with an added row | -| `query.transformations` | Counter | | Number of query transformations performed | -| `query.transformation-time` | Histogram | s | Time to transform custom queries via API server | -| `query.transformation-hash-changes` | Counter | | Times a query transformation hash changed | -| `query.transformation-no-ops` | Counter | | Times a query transformation was a no-op | +| Metric | Type | Unit | Description | +| ---------------------------------------------------- | ------------- | ---- | --------------------------------------------------------------------------------------------------------- | +| `zero_sync_max_protocol_version` | Gauge | | Highest sync protocol version seen from connecting clients | +| `zero_sync_active_clients` | UpDownCounter | | Number of currently connected sync clients | +| `zero_sync_active_client_groups` | Gauge | | Number of active ViewSyncerService instances in a syncer worker | +| `zero_sync_queries` | Gauge | | Active IVM pipelines across all client groups in a syncer worker | +| `zero_sync_rows` | Gauge | | CVR-tracked rows across all client groups in a syncer worker | +| `zero_sync_serving_lag` | Gauge | ms | Longest time locally ready replica changes have remained unserved across eligible active client groups | +| `zero_sync_serving_lag_stats` | Gauge | ms | Distribution of serving lag across eligible active client groups | +| `zero_sync_serving_lagging_client_groups` | Gauge | | Eligible active client groups with locally ready replica changes not yet served to clients | +| `zero_sync_view_syncer_lag` | Histogram | s | Time from replica changes becoming ready to ViewSyncer output, sampled once per minute per eligible group | +| `zero_sync_view_syncer_hydration` | Histogram | s | Time from a ViewSyncer query sync requiring hydration until output, per client group | +| `zero_sync_e2e_serving_lag` | Histogram | s | Completion latency from the upstream transaction commit through ViewSyncer output | +| `zero_sync_e2e_serving_lag_clamps` | Counter | | Negative end-to-end lag observations clamped to zero because the upstream clock was ahead | +| `zero_sync_lock_wait_time` | Histogram | s | Time spent waiting to acquire the ViewSyncerService lock per operation | +| `zero_sync_pipeline_resets` | Counter | | Count of pipeline resets, labeled by `reason` | +| `zero_sync_hydration` | Counter | | Number of query hydrations | +| `zero_sync_hydration_time` | Histogram | s | Time to hydrate a query | +| `zero_sync_advance_time` | Histogram | s | Time to advance all queries for a client group after applying a transaction | +| `zero_sync_poke_time` | Histogram | s | Time per poke transaction (excludes canceled/noop pokes) | +| `zero_sync_poke_transactions` | Counter | | Count of poke transactions | +| `zero_sync_poke_rows` | Counter | | Count of poked rows | +| `zero_sync_cvr_load_attempts` | Counter | | CVR load attempts | +| `zero_sync_cvr_load_duration` | Histogram | s | Time to load a CVR | +| `zero_sync_cvr_flush_attempts` | Counter | | CVR flush attempts | +| `zero_sync_cvr_flush_time` | Histogram | s | Time to flush a CVR transaction | +| `zero_sync_cvr_rows_flushed` | Counter | | Number of changed rows flushed to a CVR | +| `zero_sync_websocket_open_connections` | UpDownCounter | | Open client WebSocket connections | +| `zero_sync_websocket_connection_attempts` | Counter | | Client WebSocket connection attempts | +| `zero_sync_websocket_connection_successes` | Counter | | Client WebSocket connections successfully initialized | +| `zero_sync_websocket_connection_failures` | Counter | | Client WebSocket connection attempts that failed before initialization | +| `zero_sync_websocket_errors` | Counter | | Client WebSocket error events | +| `zero_sync_ivm_advance_time` | Histogram | s | Time to advance IVM queries in response to a single change | +| `zero_sync_ivm_conflict_rows_deleted` | Counter | | Rows deleted because they conflicted with an added row | +| `zero_sync_query_transformations` | Counter | | Number of query transformations performed | +| `zero_sync_query_transformation_time` | Histogram | s | Time to transform custom queries via API server | +| `zero_sync_query_transformation_hash_changes` | Counter | | Times a query transformation hash changed | +| `zero_sync_query_transformation_no_ops` | Counter | | Times a query transformation was a no-op | +| `zero_sync_query_row_set_signature_drifts` | Counter | | Unchanged query rehydrations whose row-set signature differs from the CVR, forcing a config-version bump | +| `zero_sync_query_same_hash_rehydrations_forced_bump` | Counter | | Same-hash query rehydrations that force a config-version bump so changed rows are delivered | + +Serving-lag metrics include only client groups with at least one connected client and a validated background connection context. Retained groups without an eligible connection do not contribute lag. ### zero.mutation -| Metric | Type | Unit | Description | -| -------- | ------- | ---- | ------------------------------------ | -| `crud` | Counter | | Number of CRUD mutations processed | -| `custom` | Counter | | Number of custom mutations processed | -| `pushes` | Counter | | Number of pushes processed | +| Metric | Type | Unit | Description | +| ---------------------- | ------- | ---- | ------------------------------------ | +| `zero_mutation_crud` | Counter | | Number of CRUD mutations processed | +| `zero_mutation_custom` | Counter | | Number of custom mutations processed | +| `zero_mutation_pushes` | Counter | | Number of pushes processed | **For AI agents**: to view all the available documentation, visit https://zero.rocicorp.dev/llms.txt diff --git a/skills/zero-docs/references/release-notes/1.7.md b/skills/zero-docs/references/release-notes/1.7.md new file mode 100644 index 0000000..438d803 --- /dev/null +++ b/skills/zero-docs/references/release-notes/1.7.md @@ -0,0 +1,62 @@ +# Zero 1.7 + +Query Correctness and Performance + +# Zero 1.7 + +## Installation + +```bash +npm install @rocicorp/zero@1.7 +``` + +## Overview + +Zero 1.7 includes improvements to query correctness, performance, and operational safety. + +## Features + +* [**Immutable query result updates:**](../zql.md#immutability) Query result updates now preserve references for unchanged subtrees, improving compatibility with React memoization and Solid reactivity without the earlier eager-expansion regression. ([#6093](https://github.com/rocicorp/mono/pull/6093)) +* [**Additional Postgres scalar types:**](../postgres-support.md#column-types) Zero now syncs selected text-represented Postgres scalar types, including network/address types, `pg_lsn`, and the `isn` extension family, as `string` columns. These columns can also participate in Zero primary keys. ([#6099](https://github.com/rocicorp/mono/pull/6099)) +* [**Replication lag diagnostics:**](../otel.md#zeroreplication) Added a `replication.last_total_lag` gauge so operators can distinguish actual replication lag from a stalled lag-report stream. ([#6042](https://github.com/rocicorp/mono/pull/6042)) + +## Performance + +Zero 1.7 improves the performance of replication and `exists` queries. + +### Replication + +Replication in Zero 1.7 is about **1.8x faster** than Zero 1.6 in benchmarks. Initial sync is also modestly faster. + +In real-world workloads we saw better results. One customer workload on Cloud Zero saw max sustainable replication increase from \~750 writes/second to \~2500 (> 3x faster). + +### Flipped Exists Queries + +In a query like `doc.where("id", id).whereExists('comment').limit(10)`, the order that tables are considered matters for performance. If there are only a few child rows per parent, it's much faster to find children first, then find the corresponding parents, then sort and limit. Zero does this automatically using a process called *[join flipping](../zql.md#manually-flipping-joins)*. + +Flipped joins are very common since `exists` is used in permissions, and in many systems each user has access to only a small subset of total rows. + +These kinds of queries got faster in Zero 1.7. In simple cases where each child has a unique parent (e.g., `doc -> comment`), Zero 1.7 is roughly 3x faster. In cases where each child has multiple parents (e.g., `user -> doc_acl`), Zero 1.7 is dramatically faster – reaching \~100x faster at 10k results. + +## Fixes + +* [WebSocket close code `1009` could trap a client in a reconnect loop by repeatedly resending the same oversized message.](https://github.com/rocicorp/mono/pull/5982) +* [Inspector/analyze-query could show a query plan different from the actual plan generated by the table source.](https://github.com/rocicorp/mono/pull/5990) +* [Duplicate bundled copies of runtime peer dependencies could break customer-observable behavior such as `Pool instanceof` checks and React single-instance assumptions.](https://github.com/rocicorp/mono/pull/6046) +* [Singular and plural queries with the same AST could incorrectly share a client view.](https://github.com/rocicorp/mono/pull/6065) +* [React and Solid queries with nested relationships using `one()` vs `limit(1)` could share a view cache entry and return the wrong nested shape.](https://github.com/rocicorp/mono/pull/6104) +* [Removing a query could trigger an assertion.](https://github.com/rocicorp/mono/pull/6066) +* [`LIKE`/`ILIKE` matching in the JS query engine used multiline regex behavior instead of dotall behavior, causing mismatches for strings containing newlines.](https://github.com/rocicorp/mono/pull/6083) (thanks [@sravan27](https://github.com/sravan27)!) +* [Range filters (`<`, `<=`, `>`, `>=`) could use ordering that differed from SQLite/Postgres ordering.](https://github.com/rocicorp/mono/pull/6088) (thanks [@sravan27](https://github.com/sravan27)!) +* [SQLite replica `LIKE`/`ILIKE` behavior could diverge from Postgres and the in-memory JS matcher.](https://github.com/rocicorp/mono/pull/6095) +* [SQLite replica `ILIKE` did not match non-ASCII case variants such as `MÜLLER` and `müller`.](https://github.com/rocicorp/mono/pull/6098) +* [Litestream-backed deployments could purge change-log entries based on stale claimed backup progress rather than actual backup state.](https://github.com/rocicorp/mono/pull/6110) [#6123](https://github.com/rocicorp/mono/pull/6123) +* [Query hydration and pipeline failures now include the query hash, transformation hash, and custom query name when available.](https://github.com/rocicorp/mono/pull/6128) +* ["vended rows" inspector feature was slow for large numbers of rows](https://github.com/rocicorp/mono/pull/5991) +* [Logging code created gc pressure throughout system](https://github.com/rocicorp/mono/pull/6125) + +## Breaking Changes + +None. + +**For AI agents**: to view all the available documentation, visit https://zero.rocicorp.dev/llms.txt diff --git a/skills/zero-docs/references/release-notes/1.8.md b/skills/zero-docs/references/release-notes/1.8.md new file mode 100644 index 0000000..70e8f2d --- /dev/null +++ b/skills/zero-docs/references/release-notes/1.8.md @@ -0,0 +1,82 @@ +# Zero 1.8 + +Observability and Reliability + +# Zero 1.8 + +## Installation + +```bash +npm install @rocicorp/zero@1.8 +``` + +You can now use `zero-cache` from GHCR: + +```bash +docker pull rocicorp/zero:1.8.0 +# or +docker pull ghcr.io/rocicorp/zero:1.8.0 +``` + +## Overview + +Zero 1.8 improves observability, performance, and reliability. + +## Features + +* [**Request-header forwarding:**](../zero-cache-config.md#mutate-allowed-request-headers) `zero-cache` can forward selected WebSocket upgrade headers to custom APIs using [`ZERO_MUTATE_ALLOWED_REQUEST_HEADERS`](../zero-cache-config.md#mutate-allowed-request-headers) and [`ZERO_QUERY_ALLOWED_REQUEST_HEADERS`](../zero-cache-config.md#query-allowed-request-headers). ([#6144](https://github.com/rocicorp/mono/pull/6144), thanks [@tjenkinson](https://github.com/tjenkinson)!) +* [**GHCR Docker images:**](../self-host.md#docker-images) Zero images are now published to `ghcr.io/rocicorp/zero` as well as Docker Hub. ([#6161](https://github.com/rocicorp/mono/pull/6161)) +* [**Mutator result type:**](../mutators.md#waiting-for-results) `MutatorResult` is now exported from `@rocicorp/zero` for typing helpers that await `.client` or `.server`. ([#6223](https://github.com/rocicorp/mono/pull/6223)) +* **Operational metrics:** `zero-cache` adds metrics for [API calls and startup](../otel.md#zeroserver), [initial sync and replication slots](../otel.md#zeroreplication), and [Litestream backup and restore](../otel.md#zeroreplica). ([#6203](https://github.com/rocicorp/mono/pull/6203), [#6208](https://github.com/rocicorp/mono/pull/6208), [#6191](https://github.com/rocicorp/mono/pull/6191), [#6199](https://github.com/rocicorp/mono/pull/6199), [#6210](https://github.com/rocicorp/mono/pull/6210)) +* **Stability metrics:** New [serving-lag, CVR, and WebSocket metrics](../otel.md#zerosync) and [replication flow-control metrics](../otel.md#zeroreplication) help diagnose delayed updates, reconnects, and backpressure. ([#6157](https://github.com/rocicorp/mono/pull/6157), [#6214](https://github.com/rocicorp/mono/pull/6214), [#6207](https://github.com/rocicorp/mono/pull/6207)) + +## Performance + +Zero 1.8 speeds up replication of large transactions, maintenance of queries that use `limit()`, and client-side query hydration. + +### Replicating Large Transactions + +Bulk imports, backfills, or migrations often change thousands of rows in a single Postgres transaction. These large transactions replicate about **1.5x faster in Zero 1.8**. + +### Maintaining `limit()` Queries + +Consider a query like this: + +```ts +zql.issue + .where('status', 'open') + .orderBy('created', 'asc') + .limit(50) +``` + +Zero can fulfill this query using an index on either `status` or `created`. If it decides to use the `created` index, Zero might have to consider many rows before it finds 50 matches. That is unavoidable. + +But when changes to the data move rows in or out of the first 50 results, Zero 1.7 repeated the work to find the first 50 results, making incremental updates slower than necessary. Zero 1.8 fixes this. + +In benchmarks, when the last returned row was 50,000 rows into the index, incremental updates were **2x faster in Zero 1.8**. When it was 100,000 rows in, updates were **over 50x faster in Zero 1.8**. + +### Client-Side Hydration + +Zero runs queries first on the client, then on the server. The initial client-side hydration got faster in Zero 1.8. For example, this query returns initial data from client about **1.3x faster in Zero 1.8**: + +```ts +zql.issue.related('creator').related('comments') +``` + +## Fixes + +* [Logical replication now reconnects when the inbound Postgres stream goes silent.](https://github.com/rocicorp/mono/pull/6047) +* [Postgres writes no longer use sockets after disconnection.](https://github.com/rocicorp/mono/pull/6193) +* [The Drizzle adapter now handles array-mode results from Drizzle 1.0 RC `prepareQuery`.](https://github.com/rocicorp/mono/pull/6154) (thanks [@typedrat](https://github.com/typedrat)!) +* [z2s now compiles queries using `start`, and SQLite fetches handle `null` start-cursor fields.](https://github.com/rocicorp/mono/pull/6189) +* [Queries no longer appear `complete` with stale or empty results after reconnect.](https://github.com/rocicorp/mono/pull/6172) +* [React Native reads now work with `op-sqlite` v17.](https://github.com/rocicorp/mono/pull/6180) +* [View-syncers no longer fail while the first backup is uploading](https://github.com/rocicorp/mono/pull/6134) or [retry before a restorable backup exists on cold start](https://github.com/rocicorp/mono/pull/6135). +* [Zero Docker images now choose the correct default sync-worker count.](https://github.com/rocicorp/mono/pull/6198) +* [Change-stream catch-up now respects flow control, preventing unbounded in-memory backlogs.](https://github.com/rocicorp/mono/pull/6186) + +## Breaking Changes + +None. + +**For AI agents**: to view all the available documentation, visit https://zero.rocicorp.dev/llms.txt diff --git a/skills/zero-docs/references/release-notes/1.9.md b/skills/zero-docs/references/release-notes/1.9.md new file mode 100644 index 0000000..b3f4881 --- /dev/null +++ b/skills/zero-docs/references/release-notes/1.9.md @@ -0,0 +1,55 @@ +# Zero 1.9 + +Stability and Query Correctness + +# Zero 1.9 + +## Installation + +```bash +npm install @rocicorp/zero@1.9 +``` + +## Overview + +Zero 1.9 improves query/mutation correctness and contains numerous reliability improvements. + +## Features + +* [`zero_sync_e2e_serving_lag`](../otel.md#zerosync) measures completed replicated work from the upstream transaction commit through view-syncer poke. [`zero_replication_upstream_clock_skew`](../otel.md#zeroreplication) tries to identify measurements biased by clock differences. ([#6312](https://github.com/rocicorp/mono/pull/6312)) + +## Performance + +### Cold Mutation Latency + +Before running mutations, Zero Server fetches and caches PostgreSQL schema metadata. This is now **2.7x faster** in Zero 1.9 (done in [#6292](https://github.com/rocicorp/mono/pull/6292), thanks [@diegopereira99](https://github.com/diegopereira99)!). This is most noticeable with cold-starts in serverless environments like AWS Lambda. + +## Fixes + +* [Restores now use Litestream 0.5.15 for legacy-format compatibility](https://github.com/rocicorp/mono/pull/6260), [retry transient failures](https://github.com/rocicorp/mono/pull/6347), [clean up temporary databases and staged WAL files after failed or interrupted attempts](https://github.com/rocicorp/mono/pull/6355), and [retain the previous snapshot generation during active restores](https://github.com/rocicorp/mono/pull/6267). +* [`insert` now succeeds without changing the row when its Zero primary key already exists; before 1.9, the server returned an error.](https://github.com/rocicorp/mono/pull/6251) +* [Ordered queries now return correct results when cursor fields contain `NULL`.](https://github.com/rocicorp/mono/pull/6121) (thanks [@YevheniiKotyrlo](https://github.com/YevheniiKotyrlo)!) +* [Rebuilt queries now deliver changed rows instead of occasionally leaving clients with stale results.](https://github.com/rocicorp/mono/pull/6196) +* [Queries no longer drop rows or emit invalid SQL when given an inapplicable scalar hint, and scalar `NOT EXISTS` now handles empty or `NULL` results.](https://github.com/rocicorp/mono/pull/6306) +* [Schema construction, CRUD mutators, and materialized views now preserve a key named `__proto__` as user data.](https://github.com/rocicorp/mono/pull/6185) (thanks [@tjenkinson](https://github.com/tjenkinson)!) +* SQLite statement caches now [retain at most 1,000 idle entries each](https://github.com/rocicorp/mono/pull/6202). +* Terminated client groups now [release custom-query timers and caches](https://github.com/rocicorp/mono/pull/6228). +* Large replica transactions [can spill dirty pages to WAL instead of retaining the complete write set in native memory](https://github.com/rocicorp/mono/pull/6311). +* [Missing replication-lag reports are retried and `total_lag` no longer grows when reports stop arriving](https://github.com/rocicorp/mono/pull/6187), while [serving-lag metrics exclude disconnected or not-yet-validated client groups](https://github.com/rocicorp/mono/pull/6219). +* `zero-cache` now recovers from [half-open PostgreSQL sockets](https://github.com/rocicorp/mono/pull/6220), [including over TLS](https://github.com/rocicorp/mono/pull/6221), and [the official image applies the bundled postgres.js disconnect patch](https://github.com/rocicorp/mono/pull/6310). +* [With PostgreSQL `wal_sender_timeout=0`, replication no longer enters a continuous reconnect loop.](https://github.com/rocicorp/mono/pull/6244) See [WAL Sender Timeout](../connecting-to-postgres.md#wal-sender-timeout). +* [Client connection attempts now time out across setup and the server handshake, abandon late sockets, and retry normally.](https://github.com/rocicorp/mono/pull/6299) +* [Reconnect confirmations no longer produce false slow-query warnings or inflated materialization metrics.](https://github.com/rocicorp/mono/pull/6308) +* [Different integration versions in a pnpm workspace no longer create peer-qualified duplicate copies of `@rocicorp/zero`, fixing cross-package type and module-augmentation failures.](https://github.com/rocicorp/mono/pull/6231) +* [Replicated PostgreSQL type and nullability changes now preserve compound-index column order in SQLite replicas.](https://github.com/rocicorp/mono/pull/6225) To repair an affected replica, resync it from Postgres or recreate the PostgreSQL index. +* [Expected schema and replica resets now log warnings instead of errors](https://github.com/rocicorp/mono/pull/6248), and [`zero-cache` skips Litestream restore when backups are not configured](https://github.com/rocicorp/mono/pull/6259). (thanks [@asterikx](https://github.com/asterikx)!) +* [Server CRUD updates and upserts no longer assign primary-key columns, avoiding PostgreSQL locks that could block concurrent foreign-key inserts.](https://github.com/rocicorp/mono/pull/6280) (thanks [@shayonj](https://github.com/shayonj)!) +* [Mutation and query API calls now retry all `5xx` responses using the existing four-attempt limit and backoff; `4xx` responses still fail without retry.](https://github.com/rocicorp/mono/pull/6315) (thanks [@shayonj](https://github.com/shayonj)!) +* [SQLite corruption failures now log diagnostics](https://github.com/rocicorp/mono/pull/6215), [delete the corrupted replica before exit](https://github.com/rocicorp/mono/pull/6342), and support [extended corruption errors](https://github.com/rocicorp/mono/pull/6339), with [deeper checks available as an opt-in](https://github.com/rocicorp/mono/pull/6341). +* [Oversized replication updates now identify the transaction, affected column, and value type without logging the value.](https://github.com/rocicorp/mono/pull/6318) +* [Fatal replica-writer failures now surface as replication errors and cause `zero-cache` to exit with a failure instead of silently stopping replication.](https://github.com/rocicorp/mono/pull/6326) +* [Replication now recovers from upstream disconnects](https://github.com/rocicorp/mono/pull/6346) or [stalled PostgreSQL writes](https://github.com/rocicorp/mono/pull/6348) while flow control is blocked. +* [Mutations from multiple tabs in the same client group are no longer skipped or sent out of order.](https://github.com/rocicorp/mono/pull/6340) +* Zero's replication-stream inbound timeout can now be [configured separately from PostgreSQL's `wal_sender_timeout`](https://github.com/rocicorp/mono/pull/6351), avoiding unnecessary reconnects during long gaps in WAL output (thanks [@gerardatkonvo](https://github.com/gerardatkonvo)!) + +**For AI agents**: to view all the available documentation, visit https://zero.rocicorp.dev/llms.txt diff --git a/skills/zero-docs/references/release-notes/index.md b/skills/zero-docs/references/release-notes/index.md index ed6eb11..8be8fdb 100644 --- a/skills/zero-docs/references/release-notes/index.md +++ b/skills/zero-docs/references/release-notes/index.md @@ -1,5 +1,8 @@ # Release Notes +* [Zero 1.9: Stability and Query Correctness](1.9.md) +* [Zero 1.8: Observability and Reliability](1.8.md) +* [Zero 1.7: Query Correctness and Performance](1.7.md) * [Zero 1.6: PlanetScale Failover Support](1.6.md) * [Zero 1.5: Schema Change Improvements and Client Group Auth](1.5.md) * [Zero 1.4: Performance and Reliability Improvements](1.4.md) diff --git a/skills/zero-docs/references/schema.md b/skills/zero-docs/references/schema.md index 73a4956..5fac5c2 100644 --- a/skills/zero-docs/references/schema.md +++ b/skills/zero-docs/references/schema.md @@ -587,10 +587,20 @@ Zero handles both these cases through a process called *backfilling*. Zero backfills existing data to the replica in the background after detecting a new column. The new column is not exposed to the client until all data has been backfilled, which may take some time depending on the amount of data. +> **Wait for backfill before deploying code that uses the new data**: During backfill, the new column is hidden from the replica. If you deploy app or API code that queries the column before backfill completes, those queries can fail even though the column exists in Postgres. + ### Monitoring Backfill Progress -To track backfill progress, check your `zero-cache` logs for messages about backfilling status. +You can detect backfill status several ways: + +* **Cloud Zero**: Backfill progress is displayed directly in the dashboard. + +* **ChangeDB**: Query the `backfilling` table in the ChangeDB. When any rows are returned, Zero is still backfilling: + + ```sql + SELECT * FROM "/cdc"."backfilling" WHERE "schema" = 'public'; + ``` -If you're using [Cloud Zero](https://zerosync.dev/#pricing), backfill progress is displayed directly in the dashboard. +* **Client observation**: Observe rows synced to a client. After backfill completes, the new column is visible on each row's JavaScript value, regardless of whether the column is present in `schema.ts`. **For AI agents**: to view all the available documentation, visit https://zero.rocicorp.dev/llms.txt diff --git a/skills/zero-docs/references/self-host.md b/skills/zero-docs/references/self-host.md index a2d8756..a034379 100644 --- a/skills/zero-docs/references/self-host.md +++ b/skills/zero-docs/references/self-host.md @@ -23,6 +23,13 @@ You will also need to deploy a Postgres database, your frontend, and your API se Before setting up Postgres, read [Connecting to Postgres](connecting-to-postgres.md) for provider-specific notes. +## Docker Images + +The examples below use Docker Hub, but the Zero container image is available from: + +* Docker Hub: `rocicorp/zero:1.9.0` +* GHCR: `ghcr.io/rocicorp/zero:1.9.0` + ## Minimum Viable Strategy The simplest way to deploy Zero is to run everything on a single node. This is the least expensive way to run Zero, and it can take you surprisingly far. @@ -36,7 +43,7 @@ Here are equivalent single-node configurations for a few common deployment targe ```yaml services: zero-cache: - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 ports: - 4848:4848 stop_grace_period: 10m @@ -86,7 +93,7 @@ primary_region = "iad" kill_timeout = 300 [build] - image = "rocicorp/zero:1.6.2" + image = "rocicorp/zero:1.9.0" [http_service] internal_port = 4848 @@ -139,7 +146,7 @@ export default $config({ new sst.aws.Service('ZeroCache', { cluster, - image: 'rocicorp/zero:1.6.2', + image: 'rocicorp/zero:1.9.0', cpu: '1 vCPU', memory: '2 GB', volumes: [{efs, path: '/data'}], @@ -209,7 +216,7 @@ spec: terminationGracePeriodSeconds: 600 containers: - name: zero-cache - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 ports: - name: http containerPort: 4848 @@ -284,7 +291,7 @@ Here are equivalent multi-node configurations for the same topology on a few com ```yaml services: replication-manager: - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 # Do not expose the RM to the public internet - only view-syncers expose: - 4849 @@ -308,7 +315,7 @@ services: start_period: 10m view-syncer: - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 ports: - 4848:4848 stop_grace_period: 10m @@ -354,7 +361,7 @@ primary_region = "iad" kill_timeout = 300 [build] - image = "rocicorp/zero:1.6.2" + image = "rocicorp/zero:1.9.0" # Do not add [http_service] or [[services]] to this app. The # replication-manager serves Zero's internal replication protocol and should @@ -391,7 +398,7 @@ primary_region = "iad" kill_timeout = 300 [build] - image = "rocicorp/zero:1.6.2" + image = "rocicorp/zero:1.9.0" # If you run more than one view-syncer on Fly, add sticky routing # (for example Fly Replay / replay_cache) so clients stay on one machine. @@ -462,7 +469,7 @@ export default $config({ 'ReplicationManager', { cluster, - image: 'rocicorp/zero:1.6.2', + image: 'rocicorp/zero:1.9.0', cpu: '1 vCPU', memory: '2 GB', environment: { @@ -503,7 +510,7 @@ export default $config({ 'ViewSyncer', { cluster, - image: 'rocicorp/zero:1.6.2', + image: 'rocicorp/zero:1.9.0', cpu: '2 vCPU', memory: '4 GB', environment: { @@ -573,7 +580,7 @@ spec: terminationGracePeriodSeconds: 600 containers: - name: replication-manager - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 ports: - name: http containerPort: 4849 @@ -645,7 +652,7 @@ spec: terminationGracePeriodSeconds: 600 containers: - name: view-syncer - image: rocicorp/zero:1.6.2 + image: rocicorp/zero:1.9.0 ports: - name: http containerPort: 4848 diff --git a/skills/zero-docs/references/zero-cache-config.md b/skills/zero-docs/references/zero-cache-config.md index 5173097..8642b8d 100644 --- a/skills/zero-docs/references/zero-cache-config.md +++ b/skills/zero-docs/references/zero-cache-config.md @@ -269,6 +269,26 @@ Path to the litestream executable. This must be built from the `rocicorp/litestr flag: `--litestream-executable`env: `ZERO_LITESTREAM_EXECUTABLE` +### Litestream V5 Executable + +Path to the official Litestream v0.5.x executable used for restores when `ZERO_LITESTREAM_RESTORE_USING_V5` is enabled. Litestream v0.5.8 and later can restore both legacy WAL backups and LTX backups, choosing the format with the latest data. The official Zero Docker image includes Litestream 0.5.15 at this path. + +flag: `--litestream-executable-v5`env: `ZERO_LITESTREAM_EXECUTABLE_V5` + +### Litestream Restore Using V5 + +Use `ZERO_LITESTREAM_EXECUTABLE_V5` for restores when that executable is configured. If it is unavailable, Zero falls back to the legacy executable. Set this to `false` to force legacy restore behavior. + +Litestream v0.5 cannot restore legacy backups encrypted with Age. Keep legacy restore enabled for those backups or migrate them before enabling v5 restore. + +flag: `--litestream-restore-using-v5`env: `ZERO_LITESTREAM_RESTORE_USING_V5`default: `true` + +### Litestream Backup Using V5 + +Write LTX backups with Litestream v0.5.x. This is disabled by default to continue writing legacy WAL backups. Enabling it requires v5 restore and makes rollback difficult because older versions cannot restore an LTX-only backup. + +flag: `--litestream-backup-using-v5`env: `ZERO_LITESTREAM_BACKUP_USING_V5`default: `false` + ### Litestream Incremental Backup Interval Minutes The interval between incremental backups of the replica. Shorter intervals reduce the amount of change history that needs to be replayed when catching up a new view-syncer, at the expense of increasing the number of files needed to download for the initial litestream restore. @@ -323,7 +343,7 @@ flag: `--litestream-restore-parallelism`env: `ZERO_LITESTREAM_RESTORE_PARALLELIS ### Litestream Snapshot Backup Interval Hours -The interval between snapshot backups of the replica. Snapshot backups make a full copy of the database to a new litestream generation. This improves restore time at the expense of bandwidth. Applications with a large database and low write rate can increase this interval to reduce network usage for backups (litestream defaults to 24 hours). +The interval between snapshot backups of the replica. Snapshot backups make a full copy of the database to a new litestream generation. Zero retains the previous generation for six additional hours so an active restore can finish before its snapshot and WAL files are removed. This improves restore time and safety at the expense of bandwidth and temporary backup storage. Applications with a large database and low write rate can increase this interval to reduce network usage for backups (litestream defaults to 24 hours). flag: `--litestream-snapshot-backup-interval-hours`env: `ZERO_LITESTREAM_SNAPSHOT_BACKUP_INTERVAL_HOURS`default: `12` @@ -365,12 +385,24 @@ flag: `--mutate-api-key`env: `ZERO_MUTATE_API_KEY` ### Mutate Allowed Client Headers -Comma-separated list of custom request headers that zero-cache is allowed to forward to your mutate endpoint. Header names are matched case-insensitively. +Comma-separated allowlist of client-provided custom headers to forward to your mutate endpoint. Header names are matched case-insensitively. By default, no client-provided custom headers are forwarded. flag: `--mutate-allowed-client-headers`env: `ZERO_MUTATE_ALLOWED_CLIENT_HEADERS`default: `none` +### Mutate Allowed Request Headers + +Comma-separated allowlist of HTTP headers from the request that opened the WebSocket to forward to your mutate endpoint. Use this for proxy- or load-balancer-injected headers such as `x-forwarded-for` or `cf-ray`. + +Unlike [mutate allowed client headers](#mutate-allowed-client-headers), these values come from the request that established the connection. Header names are matched case-insensitively. + +Values are retained for the WebSocket's lifetime, so clients must reconnect to receive changes. The allowlist does not verify the header source - only allow headers that a trusted proxy overwrites or removes from untrusted requests. + +No request headers are forwarded by default. + +flag: `--mutate-allowed-request-headers`env: `ZERO_MUTATE_ALLOWED_REQUEST_HEADERS`default: `none` + ### Mutate Forward Cookies If true, zero-cache will forward cookies from the request to zero-cache to your mutate endpoint. This is useful for passing authentication cookies to the API server. If false, cookies are not forwarded. @@ -423,7 +455,7 @@ flag: `--per-user-mutation-limit-window-ms`env: `ZERO_PER_USER_MUTATION_LIMIT_WI ### PG Replication Slot Failover -For upstream Postgres 17+, creates replication slots with the `failover` flag enabled so they can be synchronized to a standby and survive a failover. This requires additional Postgres-side configuration on your provider; see [High Availability and Failover](connecting-to-postgres.md#high-availability-and-failover). Has no effect on Postgres versions before 17. +For upstream Postgres 17+, creates replication slots with the `failover` flag enabled so they can be synchronized to a standby and survive a failover. This requires additional Postgres-side configuration on your provider; see [High Availability](connecting-to-postgres.md#high-availability). Has no effect on Postgres versions before 17. flag: `--upstream-pg-replication-slot-failover`env: `ZERO_UPSTREAM_PG_REPLICATION_SLOT_FAILOVER`default: `false` @@ -441,12 +473,24 @@ flag: `--query-api-key`env: `ZERO_QUERY_API_KEY` ### Query Allowed Client Headers -Comma-separated list of custom request headers that zero-cache is allowed to forward to your query endpoint. Header names are matched case-insensitively. +Comma-separated allowlist of client-provided custom headers to forward to your query endpoint. Header names are matched case-insensitively. By default, no client-provided custom headers are forwarded. flag: `--query-allowed-client-headers`env: `ZERO_QUERY_ALLOWED_CLIENT_HEADERS`default: `none` +### Query Allowed Request Headers + +Comma-separated allowlist of HTTP headers from the request that opened the WebSocket to forward to your query endpoint. Use this for proxy- or load-balancer-injected headers such as `x-forwarded-for` or `cf-ray`. + +Unlike [query allowed client headers](#query-allowed-client-headers), these values come from the request that established the connection. Header names are matched case-insensitively. + +Values are retained for the WebSocket's lifetime, so clients must reconnect to receive changes. The allowlist does not verify the header source - only allow headers that a trusted proxy overwrites or removes from untrusted requests. + +No request headers are forwarded by default. + +flag: `--query-allowed-request-headers`env: `ZERO_QUERY_ALLOWED_REQUEST_HEADERS`default: `none` + ### Query Forward Cookies If true, zero-cache will forward cookies from the request to zero-cache to your query endpoint. This is useful for passing authentication cookies to the API server. If false, cookies are not forwarded. @@ -501,7 +545,7 @@ flag: `--replica-vacuum-interval-hours`env: `ZERO_REPLICA_VACUUM_INTERVAL_HOURS` ### Replication Lag Report Interval (ms) -The minimum interval at which replication lag reports are written upstream and reported via the `zero.replication.total_lag` [OpenTelemetry metric](otel.md). Because replication lag reports are only issued after the previous one was received, the actual interval between reports may be longer when there is a backlog in the replication stream. +The minimum interval at which replication lag reports are written upstream and reported via the `zero.replication.total_lag` [OpenTelemetry metric](otel.md). If an expected report is not received before the next interval, Zero emits a new report and increments `zero.replication.lag_report_retries`. This feature requires write access to upstream Postgres (uses `pg_logical_emit_message()`). For PostgreSQL 17+, lag measurements accurately reflect committed write latency (single-digit milliseconds). For PostgreSQL 16 and earlier, measurements may appear 50-100ms longer due to flush behavior. A negative or 0 value disables lag reporting. @@ -567,6 +611,16 @@ See the PostgreSQL docs for details: [https://www.postgresql.org/docs/current/lo flag: `--upstream-pg-replication-slot-failover`env: `ZERO_UPSTREAM_PG_REPLICATION_SLOT_FAILOVER`default: `false` +### Upstream PG Stream Inbound Timeout + +The time, in milliseconds, without an inbound message from the upstream WAL sender after which `zero-cache` tears down the replication stream to force a reconnect. By default, the threshold is twice the server's `wal_sender_timeout`. + +Increase this value when a healthy WAL sender can remain silent while decoding unpublished WAL or assembling a large transaction. This changes only Zero's inbound timeout; keepalive timing remains derived from `wal_sender_timeout`. The option has no effect when `wal_sender_timeout` is `0`, which disables inbound liveness detection. + +See [WAL Sender Timeout](connecting-to-postgres.md#wal-sender-timeout). + +flag: `--upstream-pg-stream-inbound-timeout-ms`env: `ZERO_UPSTREAM_PG_STREAM_INBOUND_TIMEOUT_MS` + ### Websocket Compression Enable WebSocket per-message deflate compression. Compression can reduce bandwidth usage for sync traffic but increases CPU usage on both client and server. Disabled by default. See: [https://github.com/websockets/ws#websocket-compression](https://github.com/websockets/ws#websocket-compression) diff --git a/skills/zero-docs/references/zql.md b/skills/zero-docs/references/zql.md index 1623238..3589838 100644 --- a/skills/zero-docs/references/zql.md +++ b/skills/zero-docs/references/zql.md @@ -42,9 +42,9 @@ zql.issue This is a design tradeoff that allows Zero to better reuse the row locally for future queries. This also makes it easier to share types between different parts of the code. -> 🧑‍🏫 **Data returned from ZQL should be considered immutable**: This means you should not modify the data directly. Instead, clone the data and modify the clone. +> 🧑‍🏫 **Data returned from ZQL should be considered immutable**: Do not directly modify JavaScript values returned from ZQL queries. > -> ZQL caches values and returns them multiple times. If you modify a value returned from ZQL, you will modify it everywhere it is used. This can lead to subtle bugs. +> ZQL caches values across queries to improve performance and reduce re-renders in frameworks like React and Solid. If you modify a value returned from ZQL, you will modify it everywhere it is used. This can lead to subtle bugs. > > JavaScript and TypeScript lack true immutable types so we use `readonly` to help enforce it. But it's easy to cast away the `readonly` accidentally. @@ -200,19 +200,23 @@ The first parameter is always a column name from the table being queried. TypeSc ### Comparison Operators -Where supports the following comparison operators: +`where()` supports the following comparison operators: | Operator | Allowed Operand Types | Description | | ---------------------------------------- | ----------------------------- | ------------------------------------------------------------------------ | | `=` , `!=` | boolean, number, string | JS strict equal (===) semantics | -| `<` , `<=`, `>`, `>=` | number | JS number compare semantics | +| `<` , `<=`, `>`, `>=` | number, string | Numeric or string ordering | | `LIKE`, `NOT LIKE`, `ILIKE`, `NOT ILIKE` | string | SQL-compatible `LIKE` / `ILIKE` | | `IN` , `NOT IN` | boolean, number, string | RHS must be array. Returns true if rhs contains lhs by JS strict equals. | | `IS` , `IS NOT` | boolean, number, string, null | Same as `=` but also works for `null` | TypeScript will restrict you from using operators with types that don’t make sense – you can’t use `>` with `boolean` for example. -> **Don't see the operator you need?**: [Let us know](https://discord.rocicorp.dev/)! Many are easy to add. +> **ILIKE and collation**: Zero can evaluate an `ILIKE` filter in three places: in-memory on the client, in ZQLite against the replica, or as SQL in Postgres. +> +> ZQLite and the in-memory query engine use Unicode-aware lowercasing before matching. This means common non-ASCII case variants, such as `MÜLLER` and `müller`, match in both. +> +> If you need non-ASCII text in PG to work with `ILIKE`, use a Unicode-aware collation for those text columns, such as the database's default UTF-8 locale. ### Equals is the Default Comparison Operator