doc: design decoupled coordination - #38696
Draft
aljoscha wants to merge 31 commits into
Draft
Conversation
Define catalog authority for maintained lifecycle and separate it from request-scoped query execution. Record the ownership and readability boundaries needed for independent clients, while leaving implementation mechanisms open and full multi-adapter operation outside this deliverable. Include an append-only implementation log for work across sessions.
Make sink export installation an implication of the committed catalog diff, not a sequencer-side lifecycle call. Create exports after source and table inputs, then initialize their default read policies. Existing read holds protect inputs through installation, and SQL success still waits for it. Extend sink compatibility coverage with no-op creation and rename operations. Independent review found no blocking issue. CI Clippy and formatting passed before a documentation push superseded the run. Runtime validation remains pending, and local builds were blocked by missing artifacts and tools. This is an incremental ownership migration. Cluster-side catalog following, MV/index creation implications, and direct committed-update tests remain open.
Separate index reconstruction from bootstrap orchestration so committed catalog implications can reuse it without inheriting bootstrap timestamp selection or installation. Keep cache validation, dependency ordering, and CREATE INDEX behavior unchanged. This is preparatory work, not index lifecycle decoupling. Runtime cache reads and committed index installation remain to be implemented. Validation: adapter cargo check and Rust formatting passed. Independent review found no issues. Full formatting and lint were blocked by local tooling and dependency build failures. Runtime tests and PR CI are pending.
Make index-add implications acquire compatible cached plans or reconstruct from catalog definitions, then install after cluster and storage creation. Retain precommit optimization and success-only session notices, without a sequencer installation closure. Reuse the existing read-hold handover. Expose best-effort expression-cache reads through its task and validate cached dependencies against committed state and available compute imports. Filter session notices for optimizer dependencies dropped during planning. Extend cache-handle tests and add cache-disabled SQL restart coverage for index use, EXPLAIN, notices, and drop cleanup. Same-batch external catalog application remains a test gap until a production subscriber exists. Adapter and cache-test compilation, Rust and Python formatting, and Ruff passed. Full local formatting and lint are blocked by missing tools. Runtime validation is delegated to the draft PR CI and remains pending.
Register materialized view storage collections from committed implications before dependent sinks and indexes acquire read holds. Share descriptor construction with bootstrap to preserve initial visibility and replacement shard ownership. Apply read policies through the deferred batch. Keep compute installation and write enablement in the sequencer for this incremental step. Extend the existing MV test with a dependent index. Adapter cargo check and Rust/Python formatting pass. Independent review found no issue. Full local formatting and lint are blocked by missing tools and a doctest dependency build. Regular PR CI is pending.
Reuse bootstrap MV optimization and notice rendering as a catalog-based planning boundary for compute implications. Preserve cache policy, SQL typing, item versions, and timestamp selection. Compute installation and input read protection remain unchanged. Extend cache-disabled restart coverage to MV results, stored EXPLAIN plans, and continued maintenance after restart. Validation: adapter cargo check and Rust/Python formatting passed. Independent review found no issue. Full formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime validation and regular PR CI are pending.
Keep a REFRESH AT CREATION materialized view unexecuted across restart, change its input after creation, and verify reconstruction uses the committed refresh timestamp rather than current input state. Extend the existing cache-disabled restart workflow without a test-only coordinator API. Record the input-protection obstacle to moving runtime MV installation. Broader creator holds can change historical readability and do not cover all access paths exposed by catalog changes. Validation: Python formatting and Ruff passed. Full local formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime coverage is pending regular PR CI.
Record explicit catalog bounds as the compaction authority for maintained collections. Define the catalog transaction, application, and recovery boundaries while leaving representation, granularity, and publication cadence to implementation and measurement. Document delegated advancement as an alternative and explain the tradeoff between ongoing catalog publication and owner-backed DDL admission. Preserve the exploration log and append the agreed decision. Documentation checks passed, including preservation of earlier log entries. Full formatting and lint remain blocked by missing tools and the Python doctest OpenSSL dependency build. No runtime behavior changes.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 7, 2026 12:02
9705b3c to
d1afd3d
Compare
aljoscha
changed the title
aljoscha
force-pushed
the
decoupled-coordination
branch
7 times, most recently
from
September 7, 2026 14:31
b77723e to
be76324
Compare
Protect all logical collection inputs so recovery can reconstruct from catalog SQL without preserving optimizer decisions. Record the storage boundaries, progress-driven protection, and accepted retention cost. Automatic creation timestamps account for every logical input. Reject explicit historical refresh requests incompatible with committed compaction permission. Existing objects retain their promised results and convert only once their remaining recovery needs are protected. The conversion mechanism and rollout policy remain open. Document integrity checks passed, with independent review of the recovery boundary. Full formatting and lint are blocked by missing local tools and an OpenSSL dependency build. No runtime recovery experiments or retention measurements were run.
aljoscha
force-pushed
the
decoupled-coordination
branch
2 times, most recently
from
September 8, 2026 14:15
42629f6 to
48a42d1
Compare
Expand unmaterialized views through their relational name references and raw HIR reads, independently of optimization and available indexes. Stop at persisted collection outputs and logs, preserving exact version IDs. This establishes dependency discovery for logical recovery protection without changing admission, read holds, or runtime installation. Add catalog-boundary tests for eliminated and implicit inputs, shared view subgraphs, index independence, MV boundaries, and collection versions. Adapter library and test compilation pass. Rust formatting passes. Full formatting and lint are blocked by missing tools and an OpenSSL dependency build. Runtime validation remains pending in draft PR CI.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 8, 2026 14:28
48a42d1 to
565e789
Compare
…y tests Strip validation churn from the design's implementation log: pending CI, passed formatting checks, local tooling gaps, and review outcomes are all reconstructible from the PR and buried the findings and decisions. Update the session prompt so future handoffs record only findings, decisions, open questions, and the next step, and note which local checks work. Reduce the logical_collection_inputs tests to the promises that matter: name resolution survives planning-time elimination, raw-HIR reads count, indexes do not affect traversal, and upstream MV outputs are leaves. The dropped version test mutated catalog maps directly, and the replacement-MV case cannot run in the debug catalog because it has no shard metadata for the target.
Focus implementation on observable recovery, lifecycle, and query-client outcomes in environments initialized under the new protection rules. Defer existing-environment conversion without weakening its result preservation contract. Direct implementers to the active milestone and retain rejected alternatives as decision context rather than pending work.
Account for committed compaction permission independently of retention policies and execution holds. Preserve it during initialization, shared shard version changes, and drops, and apply monotonic advances in batches. Recovery uses protected readability rather than policy frontiers, and initialization stops if a fencing compare fails. This is the storage consumer boundary for decoupled coordination. Durable catalog production, maintained admission, and MV recovery wiring remain unimplemented, so SQL-created collections do not yet use these bounds. Five focused storage tests cover persist advancement, recovery, aliases, and drops. Recovery regressions were verified with the fixes removed. Formatting, cheap lint checks, and all-target checks through adapter and compute-client pass locally. Broad validation remains with draft PR CI.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 8, 2026 16:16
b471041 to
540dceb
Compare
Include logical collection inputs in MV refresh preparation and creation timestamp selection, even when planning or optimization removes their reads. Preserve early holds and acquire missing planner-introduced inputs before installation, rejecting explicit refreshes that those inputs cannot support. Carry query-scoped name references separately from statement dependencies so replacement targets do not constrain input readability. Add SQL coverage for eliminated inputs, automatic timestamps, refresh-at-creation, replacement targets, and implicit SQL-function reads. This prepares catalog-backed recovery protection but does not implement durable bounds or maintained requirements. Admission currently uses held readability. Validation: bin/fmt, cargo check for mz-adapter and mz-sqllogictest with all targets, and selected Cargo, formatting, and Python-doc lint checks passed. SQL execution and broader validation are delegated to draft PR CI.
Accept committed removal of collection metadata during creation without releasing the installed compaction bound. Catalog implications create collections before applying drops, so requiring the removed bound in the committed snapshot rejects valid mixed batches. Keep requiring bounds for collections that remain cataloged and retain shared-shard protection until explicit drop application. Extend the version/drop boundary test to cover create-before-drop and rejection of missing bounds for still-cataloged collections. The regression failed before the fix. Targeted storage-bound tests, cargo check, formatting, and selected repository lint checks passed. Regular PR CI remains pending. This unblocks catalog-bound delivery but does not add its durable producer.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 8, 2026 18:40
d2dfb67 to
fff2033
Compare
Persist storage-backed compaction bounds separately from maintained logical read requirements. Validate final transaction state before batch extraction and return admission failures before the adapter fatal commit boundary. Deliver committed bounds through catalog implications and preserve collection lifetimes in dry runs and drops. This connects catalog authority to storage enforcement without enabling SQL production of the records. Creation, recovery, and progress publication remain work toward the first protected MV. Add durable reopen and admission coverage, plus adapter coverage for committed updates and staged collection lifetimes.
Keep valid client protection independent of the components enforcing compaction, while query execution remains ephemeral. Record the volatile alternative and its failure tradeoff without prescribing a component, schema, or lease mechanism. Add independent-client recovery evidence to milestone 3 and retain the protected-MV production path as the immediate integration target.
Allow any component to apply committed monotonic permission without exclusive ownership. Keep coordination at authorization, read admission, and reclamation boundaries, with collection identity and all valid read protection preserved. Record concurrent and delayed application as milestone evidence without removing fencing from controllers that still rely on private accounting.
Commit logical MV requirements and collection compaction permission before installation, including fresh builtin collections. Advance requirements from durable output progress and publish compatible bounds in batches, while retaining valid execution and creation holds. Recover using actual protected readability, co-register shared-shard aliases, and preserve leased readability during read-only handover. Retry finite persist since delivery through rate limits and primary changes. Separate DDL conflict tracking from planning-cache freshness so background protection and cadence controls preserve existing SQL behavior. Enable the path at fresh-environment initialization, off by default in production and on in tests. Existing-environment conversion and builtin schema migration remain outside milestone 1. Add a production-path demonstration of optimized-away input protection, actual compaction, historical admission, uncached pending-refresh recovery, and progress-driven history release. Keep catalog consistency comparisons complete by quiescing publication while acquiring their snapshots. Validation: formatting and targeted compilation checks pass. The milestone demonstration passes in draft-PR CI. See the PR for broad check outcomes.
Record the architectural review priorities in the implementer prompt: publication scaling, consistency-check coupling, and remaining ownership dependencies. Keep the design unchanged and retire resolved steering rather than accumulating a permanent checklist.
Keep milestone 1 active beyond the protected-MV checkpoint. Cover source and sink recovery requirements and compute compaction before moving to independent lifecycle execution. Retain catalog-local authorization and its traffic tradeoff without prescribing duplicate records or stronger historical recovery guarantees.
Keep builtin schema migration and protected-environment version upgrades outside the fresh-environment decoupling deliverable, alongside existing environment conversion. Demonstrate the ownership model before tackling those follow-on concerns. Preserve fresh builtin initialization, same-version recovery, and ownership handover within the milestones.
Stage changing recovery requirements and compaction bounds as one catalog operation. This avoids rescanning accumulated transaction updates once per record while preserving final-state admission and committed implications. Exercise mixed publication batches, atomic rejection, speculative replay, and unchanged updates at the catalog boundary. Add a generated catalog microbenchmark comparing scalar and batched transactions across collection counts. Full publication, DDL latency, and retention measurement remain separate acceptance work.
Checkpoint source/sink catalog requirements, ingestion birth-floor delivery, and frozen diagnostic catalog reconstruction. Remove checker-driven DDL revision semantics and retain input execution protection for metric sinks. Index bound ownership and controller accounting are implemented, but production index publication, delivery, and installation authorization are not connected. Read-only fenced reconstruction beyond saved permission needs a boundary decision before that integration is completed. The extended source/sink production demonstration is not yet exercised. This checkpoint is intentionally local and is not milestone completion.
Publish compute bounds with storage recovery progress and deliver committed permission before installation and compaction. Read-only prewarming follows durable catalog bounds independently of its SQL savepoint, without granting permission through local reconstruction. Extend source/sink recovery evidence for the committed SNAPSHOT=false cutoff and add a configurable production publication measurement workflow. Milestone 1 remains under integration and verification.
Cover publication pause and restart for maintained indexes and committed-bound following during read-only prewarming, including a dropped sibling index. Complete storage trait implementations in compute test fixtures and preserve the upstream catalog-validation skip.
aljoscha
force-pushed
the
decoupled-coordination
branch
from
September 9, 2026 06:43
cfbf09d to
9d5e1a9
Compare
Contributor
Author
|
Commit 9d5e1a9 connects storage-backed catalog read protection to committed delivery. Bounds and maintained requirements have separate durable records, final-state transaction validation rejects incompatibility and regression, and the adapter returns admission errors before its fatal commit boundary. Committed bounds reach storage through implications. Dry runs model collection births and drops without controller effects. This is an intermediate milestone 1 change, not activation. SQL creation still needs to produce initial bounds and logical-input requirements with secured readability, including shared-shard initialization. Recovery and durable-progress publication remain unwired. Validation: formatting, targeted compilation, and the durable admission/reopen and adapter admission/dry-run tests pass. Independent review findings are resolved. Regular PR CI is still pending: https://buildkite.com/materialize/test/builds/134033 |
aljoscha
force-pushed
the
decoupled-coordination
branch
2 times, most recently
from
September 9, 2026 14:32
a13ac74 to
7c30792
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Make the catalog authoritative for maintained storage and compute lifecycle,
independent of the adapter that initiated a change. Query-local execution stays
on a separate fast protocol. The design records the ownership, read-protection,
recovery, and query-ordering boundaries.
Description
Milestone 1 remains active while the integrated production path is verified.
The target is fresh environments and same-version recovery.
Admission accounts for optimized-away inputs and committed permission, not
lagging physical compaction. Pending first refreshes and builtin inputs are protected.
including remap protection, source-export birth floors, and sink input alteration.
Fresh sinks use committed input permission for their automatic cutoff, including
SNAPSHOT = false. Existing pending output remains protected.storage and compute compaction proposals. Controller proposals preserve creation
and execution holds. Committed bounds govern storage and readable compute exports.
depending on it. Read-only prewarming follows a real durable catalog subscriber
alongside its SQL reconstruction savepoint. The savepoint cannot grant permission.
publication settings or DDL conflict semantics.
Protection defaults off in production and on in tests. Existing unprotected
objects are not converted. Sink exports and indexes install through catalog
implications. MV storage registration follows implications, but MV compute
installation still uses a sequencer closure. Controllers still live in the adapter.
Metric sinks require input execution protection, not historical output guarantees.
The restart demo covers actual persist compaction, an optimizer-eliminated input,
uncached pending-refresh recovery, system-catalog inputs, source/remap recovery,
pending sink output, and the fresh no-snapshot sink cutoff. A separate configurable
workflow measures production publication, concurrent DDL latency, catalog traffic,
and retained history at increasing collection counts. These harnesses are not
completion evidence until their current runs pass and measurements are inspected.
Remaining milestone-1 work is production proof, representative publication/retention
measurements, and resolution of integration or CI findings. Later milestones cover
independent maintained-lifecycle following, independent query clients and durable
client protection, and catalog/query ordering. The permission subscriber is not yet
an independent maintained-lifecycle executor. Existing-environment conversion,
builtin schema migration, and protected-environment version upgrades remain
follow-on work after fresh-environment decoupling.
Validation
Current integration head:
4786ae2b. Regular PR CI is pending. The PR remains adraft, with nightly intentionally disabled.
Local formatting, the targeted cargo/formatting/Python-documentation check scripts,
and compilation for adapter, environmentd, and sqllogictest across all targets
passed. Targeted publication and durable-subscriber tests are running. The extended
production demo and scaling measurements have not yet run on this head.
The prior published head
7c307928passed regular PR CI inbuild 134061. Its transaction
microbenchmark measured approximately 5.17 ms, 33.7 ms, and 426 ms for batched updates
at 100, 1,000, and 10,000 collections. Those in-memory persist transaction costs do
not establish end-to-end publication scalability or a suitable publication cadence.
Reproduce the production workflows in a disposable composition project:
Both workflows reset their project volumes.
Replaces #38694 under the outcome-focused name Decoupled coordination.