Wire the initial snapshot into startup - #146
Open
lukashes wants to merge 3 commits into
Open
Conversation
Run the snapshot before START_REPLICATION so it reads under the slot's exported snapshot, gated on a freshly created slot, snapshot.mode=initial, and at least one stream listing read.
Contributor
📊 Benchmark ResultsCurrent run is the minimum over 3 passes, compared against the base branch (
Summary: 🟢 1 faster · ➡️ 9 neutral · ⚪ 3 ignored (sub-μs) Thresholds: <1μs ignore · 1–20μs 15% · 20–50μs 10% · ≥50μs 5%. Measured on a shared CI runner — treat small deltas as noise. Informational only; this check never fails the build. |
Detect an interrupted snapshot on startup and redo the whole bootstrap from a fresh consistent point, instead of streaming past unread rows. - A marker publication <pub>_snapshotting flags an in-progress snapshot: created before the slot, dropped once the snapshot is flushed. On restart a slot with the marker present is dropped and re-created; a slot without it resumes. - The streaming publication stays before the slot, so during-snapshot changes still decode (pgoutput resolves the publication by name in the historical catalog per change). The marker is never passed to START_REPLICATION, so steady state keeps a single publication. - source.connect takes want_snapshot to drive the reconciliation; runInitialSnapshot reports completion so a signal-interrupted snapshot leaves the marker for the next start.
The mode knob was redundant with the per-stream operations set: whether to snapshot is already decided by a stream listing read. Remove SnapshotConfig, the [snapshot] section, and the mode validation; wantsInitialSnapshot now checks the streams only.
Owner
Author
|
/bench |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the #49 series (PR3 of 3): drives the snapshot reader from #145 at startup and makes an interrupted snapshot safe.
Problem
Streaming only emits changes made after the slot was created, so a new consumer never sees pre-existing rows. The reader and the READ event landed in #145 but nothing called them.
Solution
Wiring:
main.zig: connect and ensure the slot without streaming, run the snapshot, thenSTART_REPLICATION. Order matters:START_REPLICATIONinvalidates the slot's exported snapshot, so the reads happen first on a second regular connection bound to that snapshot.[snapshot].modeisinitial(default;no-snapshotdisables), and at least one stream listsread. Only the distinct read-opted resources are snapshotted.processor.runInitialSnapshotproduces READ events through the same match/serialize/partition/produce path as streaming (factored intoproduceEvent). READ rows carry the slot's start LSN, so snapshot and stream share one boundary.pending_lsnstays 0, so the slot advances only once the stream is confirmed.replication_protocol.zigcaptures the slot'ssnapshot_namealongsideconsistent_point.Interrupted-snapshot recovery:
<pub>_snapshottingflags an in-progress snapshot: created before the slot, dropped once the snapshot is flushed. On restart a slot with the marker present is dropped and the whole bootstrap is redone from a fresh consistent point; a slot without it resumes. The marker is never passed toSTART_REPLICATION, so steady state keeps a single publication.READas an upsert.Tests
snapshot.modevalidation,wantsInitialSnapshot,Stream.hasReadOperation.Unit, integration, and e2e pass locally.