Skip to content

Add the READ operation and a snapshot reader for existing rows - #145

Open
lukashes wants to merge 1 commit into
mainfrom
feat/snapshot-reader
Open

Add the READ operation and a snapshot reader for existing rows#145
lukashes wants to merge 1 commit into
mainfrom
feat/snapshot-reader

Conversation

@lukashes

@lukashes lukashes commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Second PR toward the initial snapshot (#49). Groundwork only: the reader is not wired into startup yet.

Problem

Replication starts at the slot's LSN, so a new consumer never sees rows that existed before the slot was created. #49 asks for those rows to be emitted once, up front, consistent with where the stream then begins.

Solution

  • Add READ to ChangeOperation and "read" to the allowed stream operations. A READ event reuses the insert payload, so serialization, matchStreams, and partitioning are unchanged; a stream opts into the snapshot by listing read in its operations.
  • New SnapshotReader (source/postgres/snapshot.zig): on a regular connection it enters a REPEATABLE READ transaction bound to an exported snapshot (SET TRANSACTION SNAPSHOT), reads each table with a cursor + FETCH to bound memory, and emits each row as a READ event stamped with the slot's consistent point as meta.lsn.
  • Make converter.mapValue public so a READ row and a streamed change of the same column map (oid, text) to the same JSON type.

Tests

  • Integration: exports a snapshot with pg_export_snapshot(), then checks the reader returns the pre-export rows only (a row inserted after export is not read) and that values are typed like the streamed path (int/bool/float). Plus an empty-table case.
  • Unit: READ serializes to "op":"READ", read passes config validation, and a pure matchStreams check that a READ event routes only to streams listing read.

Startup wiring, the snapshot.mode gate, and e2e come in the next PR.

Emit pre-existing table rows as READ events so a new consumer can bootstrap
current state before the stream begins (#49). Not wired into startup yet.
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

📊 Benchmark Results

Current run is the minimum over 3 passes, compared against the base branch (main @ 8fda403), built on the same runner.

Benchmark Baseline Current Δ Time Allocs Status
Converter DELETE 141.90μs 142.13μs +0.2% 13 → 13 ➡️
Converter INSERT 141.81μs 142.41μs +0.4% 13 → 13 ➡️
Converter UPDATE 171.21μs 171.41μs +0.1% 22 → 22 ➡️
JsonSerializer 50.21μs 49.77μs -0.9% 3 → 3 ➡️
KafkaProducer produce 215.05μs 218.20μs +1.5% 1 → 1 ➡️
KafkaProducer send 253.90μs 278.15μs +9.5% 0 → 0 🔴 slower
PgOutputDecoder 95.48μs 94.88μs -0.6% 6 → 6 ➡️
getPartitionKeyValue boolean 15.40μs 15.35μs -0.3% 1 → 1 ➡️
getPartitionKeyValue integer 0.04μs 0.04μs +5.0% 0 → 0 ⚪ noise
getPartitionKeyValue not found 0.03μs 0.03μs +3.1% 0 → 0 ⚪ noise
getPartitionKeyValue string 17.23μs 17.10μs -0.7% 1 → 1 ➡️
matchStreams found 17.43μs 17.20μs -1.3% 1 → 1 ➡️
matchStreams not found 0.06μs 0.05μs -1.8% 0 → 0 ⚪ noise

Summary: ➡️ 9 neutral · 🔴 1 slower · ⚪ 3 ignored (sub-μs)

Thresholds: <1μs ignore · 1–20μs 15% · 20–50μs 10% · ≥50μs 5%. Measured on a shared CI runner — treat small deltas as noise. Informational only; this check never fails the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant