Add the READ operation and a snapshot reader for existing rows - #145
Open
lukashes wants to merge 1 commit into
Open
Add the READ operation and a snapshot reader for existing rows#145lukashes wants to merge 1 commit into
lukashes wants to merge 1 commit into
Conversation
Emit pre-existing table rows as READ events so a new consumer can bootstrap current state before the stream begins (#49). Not wired into startup yet.
Contributor
📊 Benchmark ResultsCurrent run is the minimum over 3 passes, compared against the base branch (
Summary: ➡️ 9 neutral · 🔴 1 slower · ⚪ 3 ignored (sub-μs) Thresholds: <1μs ignore · 1–20μs 15% · 20–50μs 10% · ≥50μs 5%. Measured on a shared CI runner — treat small deltas as noise. Informational only; this check never fails the build. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second PR toward the initial snapshot (#49). Groundwork only: the reader is not wired into startup yet.
Problem
Replication starts at the slot's LSN, so a new consumer never sees rows that existed before the slot was created. #49 asks for those rows to be emitted once, up front, consistent with where the stream then begins.
Solution
READtoChangeOperationand"read"to the allowed stream operations. A READ event reuses the insert payload, so serialization,matchStreams, and partitioning are unchanged; a stream opts into the snapshot by listingreadin its operations.SnapshotReader(source/postgres/snapshot.zig): on a regular connection it enters aREPEATABLE READtransaction bound to an exported snapshot (SET TRANSACTION SNAPSHOT), reads each table with a cursor +FETCHto bound memory, and emits each row as a READ event stamped with the slot's consistent point asmeta.lsn.converter.mapValuepublic so a READ row and a streamed change of the same column map (oid, text) to the same JSON type.Tests
pg_export_snapshot(), then checks the reader returns the pre-export rows only (a row inserted after export is not read) and that values are typed like the streamed path (int/bool/float). Plus an empty-table case."op":"READ",readpasses config validation, and a purematchStreamscheck that a READ event routes only to streams listingread.Startup wiring, the
snapshot.modegate, and e2e come in the next PR.