Repository navigation
fix(source): refuse a replication slot that was replaced under the same name (#442, Tier 1, v0.17.5) - #447
Merged
Conversation
Design for detecting a replication slot that was dropped and created again under the same name: Postgres exposes no identity for a slot incarnation (measured on 13 to 18), so the connector keeps one. Positions carry a slot identity (format version 4), the connector writes it into the WAL as a logical decoding message, never advances an idle slot past the last one it received, and on resume peeks the slot for it. Covers the research per version, the alternatives rejected, failure modes, upgrade from positions without an identity, rollback, the v0.14.x backport, and observability. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…me name (#442) A slot dropped and created again under the same name exists, is healthy, and is past the saved position, like a slot the connector advanced while idle, so a resume on it skipped every change in between without a word (invariant 3). Positions now carry a slot identity (format version 4). The connector writes it into the WAL as a logical decoding message in a transaction of its own (a beacon, pg_logical_emit_message, synchronous commit), and never advances an idle slot past the last beacon the stream delivered back. Advancing for acked records is unchanged. On resume, a slot at or below the saved position's commit LSN is accepted; a slot past it is accepted only if a peek finds a beacon with the position's identity, and is otherwise refused with postgres.source.slot_replaced (logrepl.allowWALGap accepts it with one DATA LOSS warning). The slot is checked again once START_REPLICATION holds it, which also closes the check-to-CREATE window when possible and otherwise taken over once with postgres.source.slot_identity_unverified. A role that cannot call pg_logical_emit_message fails the start with postgres.source.slot_beacon_failed. Postgres 13 and standbys cannot carry beacons and keep the v0.17.4 check with a warning. Tests: drop and recreate between runs, recreate after the identity check, forged and copied beacons, a half-finished create with an injected failure, the #445 case, snapshot positions, the connector's own idle advance (no false positive), upgrade from a v0.17.3 position (golden from a real v0.17.3 run), a revoked pg_logical_emit_message, a server without beacons, VerifyHeld against a slot swapped in the subscription's own window, property tests for the position round trip with the identity, and a Postgres 17+ failover test (test/failover, make test-failover). TestCombinedIterator_IntactSlot_ResumesAfterIdle stood in for the idle advance with pg_replication_slot_advance; another client's advance is now refused, so it becomes TestCombinedIterator_ExternallyAdvancedSlot_Refused. Beacons are authenticated: the identity is a 256-bit secret key kept only in the positions, and a beacon carries a nonce, the WAL insert position read before it, and HMAC-SHA256(key, slot, nonce, position), verified in constant time and only if its position is at or after the slot's confirmed_flush_lsn. A role with CONNECT (pg_logical_emit_message is PUBLIC) cannot forge one, and a genuine beacon copied into a replaced slot is refused. The key is never logged (logs carry an 8-digit SHA-256 fingerprint); the one debug log that printed a raw position now prints its length. The connector logs postgres.source.slot_beacon_public while PUBLIC may call pg_logical_emit_message. Every slot the connector accepts as a gap (logrepl.allowWALGap) gets a new key, so an attempt that created and stamped the slot and failed before saving anything never passes for the original on retry. The peek's rounds are clamped before the query. An idle slot now reaches the WAL end one beacon round trip after the last ack, so a stop right after an ack leaves the slot below that transaction's commit and the next start decodes it again (the change key skips it, as after a SIGKILL). TestB1_AC7_RestartHaltTruthfulMessage relied on the teardown's uncapped advance to start run 2 past the pre-DDL relation message; its run 1 now waits for the slot to pass the baseline's commit (PGCHAOS_SETTLE_SLOT) and its assertions are unchanged. The test helper's ports can be moved with PG_TEST_PORT and PGBOUNCER_TEST_PORT (also read by test/docker-compose.yml). Design: docs/design-documents/20261021-slot-identity.md. v0.17.5. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Beacons are written only into the slot this run holds (an xid first, then active_pid or the pre-stream limit checked inside the beacon's transaction, with why a slot created after the check cannot decode it). VerifyHeld runs on every start, with the creation point as the limit for a slot this run created. Positions name their WAL history (system identifier and timeline), and a resume on another history is refused with wal_history_changed; a legacy position warns with wal_history_unverified. Failing beacons give up after beaconFailureTimeout instead of holding WAL forever, with the retention math. synchronous_commit=local, a configurable check timeout, servers that cannot carry beacons, the threat model for positions that reach destinations, the upgrade release notes, and the v0.14.x reservation (#449). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…-N5) B1: a beacon written after the stream lost its walsender vouched for a slot recreated in between. Emit now takes an xid, checks inside its own transaction that the slot is held by this run's walsender (or, before the stream, inactive and not past the limit this run checked or created it at), and only then writes; a slot created after the check cannot decode that transaction. TestSlotIdentity_BeaconAfterWalsenderLoss (the review's repro) failed before and passes. B2: on a fresh start nothing holds a newly created slot (Postgres 17 shows it inactive right after CREATE), so a slot replaced mid-snapshot streamed from the replacement. VerifyHeld now runs on every start but an import, with the creation point as the limit. TestSlotIdentity_FreshStartSlot- ReplacedMidSnapshot (the repro) failed before and passes. B3: positions carry system_identifier and timeline (IDENTIFY_SYSTEM). A resume is refused with postgres.source.wal_history_changed on another cluster, an earlier timeline, a timeline that left the position's before its bound (TIMELINE_HISTORY), or WAL behind the position; allowWALGap accepts it with a "WAL HISTORY CHANGE ACCEPTED" warning and a pg_upgrade runbook. A legacy position learns its history and logs postgres.source.wal_history_unverified once. S1: beacons that keep failing for logrepl.slotIdentity.beaconFailureTimeout (5m) are abandoned: the idle advance is uncapped again, an error with retained_wal_bytes is logged, and the next start is refused unless a record was saved. Beacons commit with synchronous_commit=local, and a delivered beacon counts even when its Emit returned an error. S2: the check's peek timeout is logrepl.slotIdentity.checkTimeout (10m). S3/S4/S6: README and release-note wording, a pre-upgrade privilege query, unproven_wal_bytes in slot_identity_unverified. N1: no raw positions in errors. N4: an unsettled AC7 chaos variant. N5: a position with an identity is never accepted unproven on a server that cannot carry beacons (slot_identity_unprovable); such servers write positions without one. The README description lives in connector.yaml (readmegen), so it now carries the section. #445's TestCDCIterator_AllowWALGap_WarnsWhenCreateSucceedsThenFails now expects the retry to warn again: the slot the failed attempt created is not the position's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarismeroxa
force-pushed
the
fix/slot-identity
branch
from
October 21, 2026 23:20
48d16b7 to
645f23e
Compare
…story_unverified test The legacy start takes the slot over with a new key, after which the v4 position's key proves nothing and its start is (correctly) refused. Check the v4 position first, and that the legacy warning is logged at warn level. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…acons On Postgres 13 (and standbys) positions carry no slot identity since the #447 review (N5), so the drift marker format test checks the identity only where beacons are supported (and the WAL history everywhere), and #445's create-then-fail test expects the pre-#442 single warning there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
pg_control_checkpoint still shows the standby's last restartpoint right after a promotion, which is before the switch, so the "position past the switch point" case built its position below it and was not refused. Read the switch point from the new timeline's history file instead, and check that the saved position was below it (why the real resume passes). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
The position's transaction (xid, commit time) closes the same-timeline restore case for covered resumes, with pg_walinspect when installed and the Postgres 15+ false positive documented. A standby never proves an identity. Abandoned beacons are retried and re-arm, so the outage heals itself. The guard identifies the walsender by PID and backend_start, and a start whose walsender cannot be identified fails. The xid argument and the committed- but-errored beacon are now tested, so they leave "Not verified". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
S2: a same-timeline restore (a volume snapshot) whose new WAL had passed the position's bound resumed "covered" and skipped the rows written after it. v4 positions now carry tx_xid and tx_commit_time from BEGIN, and a covered resume checks that the commit LSN still names that transaction: pg_walinspect's commit record when installed and usable, otherwise the first transaction the slot decodes from the bound (same commit LSN, xid and commit time; a restored server reuses the xids too). Otherwise wal_history_changed. test/restore/run.sh snapshots and restores a data volume (make test-restore): refused on 14, 17 and 18, resumed before. S3: legacy positions get the "WAL behind the position" check too. S1: a standby never proves an identity (a slot it decodes is never the primary's); refused with slot_identity_unprovable. Reproduced on 17 with a standby-local slot, now refused (in test/failover/run.sh). S4: abandoned beacons keep being written; while one is on its way the idle advance waits at its start, and the first one delivered past the reported flush re-arms the identity. Wording fixed: a restart is refused until then, and the outage heals itself. S5: tests for the xid before the guard (a slot created while the beacon's transaction is open does not decode it, via a seam after the guard) and for a beacon that committed although Emit errored (unit, and a seam after the commit with a running stream). N1: the guard joins pg_stat_activity and compares the walsender's backend_start recorded at the stream start, so a reused PID is refused. N2: a stream start whose walsender cannot be identified fails with slot_beacon_failed instead of skipping the holder. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…t markers The chaos suite caught the transaction check refusing resumes from a drift marker: its key can be one below its change's (#338), which names no transaction. Positions now carry tx_xid and tx_commit_time only when the key's commit LSN is the transaction's, and a resume from a marker skips the check (TestCDCPosition_TxIdentityOnlyForRealCommits). With the guard, a stamp at open is safe on every start (a slot swapped in after the check is past the limit and refused), so the deferred-stamp path for existing slots is gone; with allowWALGap a refused stamp waits for VerifyHeld, which warns, rotates the key and stamps. Beacons are recognised only as transactional messages, as Emit writes them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…he stream-holder test slotIDLegacy now also drops tx_xid and tx_commit_time, as a real v0.17.3 position has neither; with them kept, the legacy current-WAL test was refused by the transaction check instead of the check it is about (the mutation that removes that check survived). The N2 test runs the stream under a timeout, so a regression fails it instead of hanging the package. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
CI on bc5daa0 failed SRC-01: the acceptance test stopped the source during the snapshot-to-CDC handoff, the stream-holder query (#447 N1) ran on the canceled context, and the start failed with slot_beacon_failed, which the graceful stop reported as a plugin error. Any start step that queries the server had the same race. A start step that fails after the run's context ended now returns the context error with the step's error in the chain (TestSubscription_StartCanceledReportsTheContextError, fails without it). The walsender-unknown error no longer advises a GRANT: a role always sees its own backends, so that failure is the connection or the server. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Brings in #446 (record keys and row data out of trace and debug logs). The one conflict is the debug log for an unparsable position in combined.go: both sides log only its length; #446's field name is kept and the comment names the slot identity key as a second reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
B3: the pipeline's own slot, consumed by another client past a transaction the pipeline had received but not acked, was accepted because it still decoded the stamp written after that transaction. Beacons (v3) now certify r, a WAL position before which the run needed nothing more: serverWALEnd for a beacon requested while every record is acked, the checked slot position for the stamp. The idle advance stops at the r of the last received beacon, and a slot past the position is accepted only on a beacon written after its confirmed_flush_lsn whose r covers it. A slot moved over delivered changes only is still accepted; one moved past undelivered ones is refused with slot_replaced. B1, S1: the covered-resume transaction check no longer reads pg_walinspect. pg_get_wal_record_info returns the next record for an LSN inside one and its commit time is only in the description (a restored history that reused the xid was accepted), and COMMIT PREPARED has xid 0 there (a prepared transaction's position was refused). The check always reads the slot: same commit LSN, xid and commit time. B2: a drift marker's position names the transaction it halted before (its key is one below that commit), and the check looks for it there. A position without an xid (pre-v0.17.5) must at least have a commit at its commit LSN. Tests, each failing on 85da517: the walinspect and drift scenarios of test/restore/run.sh, ConsumedByAnotherClientBelowTheLastBeacon_Refused, PreparedTransactionPosition_ResumesCovered (test/conf.d enables prepared transactions). Plus TestSameTransaction, the marker and v3 beacon unit tests, and the legacy no-commit refusal. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarismeroxa
added a commit
that referenced
this pull request
Oct 22, 2026
…tore) (#451) * ci: add the external-state chaos suite (slot, privileges, network, restore) Adds test/externalstate, a case manifest, a fail-closed runner, `make test-external-state` and an external-state workflow (PRs touching source/**, push to main, nightly). Cases that other packages already prove are cited and run, not copied; cases blocked on #447 are pending and flip to required automatically-by-failure once the blocking test exists. The volume snapshot restore case is a known gap: it shows silent loss and stays red on purpose. No data-path code changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH * ci(external-state): own compose projects and ports, pinned refusal codes, SILENT evidence for known gaps The base cases no longer use port 5433 or take down the shared stack: the base stack runs as project pgextbase on 56153/56154 (test/docker-compose.yml and test/helper.go read CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT). requireLoudOrDelivered takes the refusal codes acceptable for the case; a case that lists none must deliver. A known-gap case must fail with SILENT loss output. The workflow gets permissions: contents: read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH * ci(external-state): a loud read error is a refusal, not silent loss; every published port is checked against the default stacks Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH --------- Co-authored-by: DeVaris Brown <devaris@devaris.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
devarismeroxa
added a commit
that referenced
this pull request
Oct 22, 2026
* ci: guard mutation gate for source/logrepl Mark the slot, import and delete-key refusals with `// guard: <name>`. tools/guardmut deletes each in turn and fails the job when the package tests stay green. The guards added by #447 are marked in a follow-up after it merges. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH * ci(guardmut): own compose project, tests= on every marker, guard list, invalidation guards run alone make guardmut runs as project pgguardmut on ports 56170/56171 and tears down only that project. test/docker-compose.yml and test/helper.go read CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT. The two 300 s mutants get tests= and now die in seconds. A name list and a shrink-only baseline are enforced in the workflow; actions need permissions: contents: read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH * ci(guardmut): strip multi-line and unclosed HTML comments before reading the waiver; detector covers Makefile and tools/go.mod Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH --------- Co-authored-by: DeVaris Brown <devaris@devaris.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The security review of 453669f asked whether a keepalive's WAL end can be past committed changes the walsender has not sent yet, which would let the idle advance (on main already) and a beacon's r (#447) skip them. It cannot, measured on 13, 14, 15, 16, 17 and 18: while it sends a large transaction, the walsender asks for a reply after wal_sender_timeout/2, and the keepalive's WAL end is sentPtr, the start of the commit record being replayed, which is that transaction's own commit LSN. A slot confirmed at a commit LSN decodes that transaction again. So everything that commits before a keepalive's WAL end was sent, and with every received record acked, reporting it loses nothing. TestWalsender_KeepaliveInsideATransactionNeverPassesItsCommit pins that with a raw client: a 100000-row transaction, a keepalive inside it at its commit LSN, that LSN reported as flushed, a crash, and the slot decodes all 100000 rows again. TestSlotIdentity_LargeTransactionInFlight_SlotNeverPassesIt runs the connector through such a transaction (a short wal_sender_timeout), checks that neither the slot nor the beacon cap passes its commit while it is in flight, stops in the middle, and gets the rest after the restart. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Brings in #451 (external-state chaos suite) and #452 (guard mutation gate). Conflicts: the test port override, which both sides added under different names. main's CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT are kept (test/helper.go, test/docker-compose.yml); the Makefile keeps both sides' targets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
The external-state suite (#451) failed on this PR by design: three pending cases name tests #447 adds (slot recreated under the same name, the pg_logical_emit_message privilege revoked, failover), and the snapshot-restore known gap now fails loudly instead of losing rows. The three are required now, and snapshot-restore is required with postgres.source.wal_history_changed as an accepted refusal (it was refused with it; the case listed only slot_wal_gap and slot_invalidated). All ten cases pass locally (test/externalstate/run.sh). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Brings in #450. Merges clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…-S3, NITs) S1: a transaction of at least findBeaconMaxRows rows between the slot's confirmed_flush_lsn and what the check looks for made it give up for good: a change limit stops only at a transaction boundary, so every round returned that transaction whole until ErrBeaconSearchExhausted, an uncoded refusal on every retry that allowWALGap did not help. FindBeacon now reads to the end of the WAL without a limit after the round at the clamp, and the covered-resume transaction check (TxCommittingAt, was FirstTxFrom) is bounded by upto_lsn = the commit LSN + 1 instead of a change count. A check that still cannot read the slot (timeout, peek error) is refused with postgres.source.slot_identity_check_failed and a fix; allowWALGap accepts the slot unproven. Tests at the review's scale (a 10000-row transaction, the clamp at 4096) fail on 76fdea1. S2: guard markers on the idle-advance beacon cap, verifyBeacon's r >= confirmed, the beacon's xid-first step (assignXid, a call of its own so the gate can remove it), the walsender check in checkHeld (PID and backend_start as one guard: alone the PID check is an equivalent mutant), VerifyHeld's limit and the covered-transaction check. guardmut: all killed. S3: make test runs go test with -timeout 20m; source/logrepl takes about 8 minutes on CI. NITs: a position whose bound is the timeline switch point is refused (leftTimelineBy, <=); the slot_replaced message names a rollback and re-upgrade and abandoned beacons as causes that lose nothing; the external-state snapshot restore pins wal_history_changed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
… allowWALGap (#447) A transient failure of the identity check (FindBeacon) or the covered-resume transaction check (TxCommittingAt) — a query error, a timeout, a connection drop — is not a verdict: the check did not run, so the slot could still be the position's. Before this change the code treated such a failure as "unproven" and, under logrepl.allowWALGap, accepted it like a verified slot_replaced and resumed, silently dropping the identity guarantee. Now a transient failure always fails loudly with postgres.source.slot_identity_check_failed, whatever allowWALGap says: the flag accepts only a slot the check proved is not the position's, never a slot the check could not read. The start is refused and retried by the engine; the verified recreated-slot opt-in is unchanged. Adds regression tests that inject a transient error at each check and assert the coded refusal (they fail without the fix), updates the checkTimeout test that asserted the old accept-unproven behaviour, and marks both refusals as guards (logrepl-transient-check-refused, logrepl-covered-transient-refused) so the mutation gate covers them.
Contributor
Author
|
Sign-off @ e1ffca6 |
Contributor
Author
|
Review: approve @ e1ffca6 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #442. Becomes v0.17.5. Rebased onto
mainafter #445 merged (squash2edf779;git rebase --onto origin/main bccd617), base ismain. Roadmap: Phase 0 data-integrity floor (invariant 3).Risk tier: Tier 1 (data path: slot advance, resume check, serialized position format). Needs DeVaris's sign-off in a separate session. Design doc:
docs/design-documents/20261021-slot-identity.md.Commits:
b6f8539design doc and6005085implementation (both as first reviewed, rebased);87e759bdesign-doc changes and645f23efixes for the #447 review;32ea7ac,df5899band69231d4test-only follow-ups;1050436design-doc changes and39dae13fixes for the re-review of69231d4;d5011cf,bc5daa0andd8cd266follow-ups found by chaos, mutation and CI;85da517mergesmainfor #446 (one conflict, the unparsable-position debug log incombined.go: both sides log only its length, #446's field name kept). A merge rather than a rebase so the push stays fast-forward.453669faddresses the round-4 review of85da517;3901c9fadds the keepalive tests for the security review of453669f;e2e3e61mergesmainfor #451 and #452 (conflict: the test port override both sides added; main'sCONDUIT_PG_TEST_PORT/CONDUIT_PG_TEST_BOUNCER_PORTkept);76fdea1flips the external-state cases this PR closes to required;26741c6mergesmainfor #450 (clean);3ae5fb2addresses the fresh-context review of76fdea1. Head:3ae5fb2.Problem
A slot dropped and created again under the same name exists, is healthy, and is past the saved position, exactly like a slot the connector advanced while idle. A resume on it skipped every change in between without a word. #441 catches a missing or invalidated slot, not this.
Design (summary; the doc has the research per Postgres version and the alternatives)
slot_identity, a 256-bit secret key, andsystem_identifier+timeline(the WAL history).pg_logical_emit_message, HMAC-SHA256 over slot, nonce and the WAL position read before it) and never advances an idle slot past the last beacon the stream delivered back. Acked-record reports are unchanged.wal_history_changed); accept a slot at or below the saved commit LSN; past it, accept only a slot that still decodes a beacon of the position's key written after itsconfirmed_flush_lsn; otherwiseslot_replaced.VerifyHeldchecks the slot again once the stream holds it.slot_identity_unverified(now carryingunproven_wal_bytes) andwal_history_unverified. Protection begins at the first record saved after the upgrade.#447 review: what changed
Blockers (each with the reviewer's repro as a test: fails before the fix, passes after):
B1, a beacon written after the slot was replaced vouched for it.
Emitnow takes an xid first (pg_current_xact_id()), then checks inside the same transaction that the slot is held by this run's walsender (active_pid= the replication connection's backend PID) or, before the stream, inactive and not past the limit this run checked or created it at. Only then does it read the insert LSN and write. Why a slot created after the check cannot decode that xid is in the doc, Decision 2. Test:TestSlotIdentity_BeaconAfterWalsenderLoss_DoesNotVouchForReplacement. Before: the emit succeeded and the restart delivered only[after]. After: the emit fails withErrSlotNotHeldand the restart is refused withslot_replaced. Unit:TestBeacons_EmitOnlyIntoTheHeldSlot.B2, a fresh start's slot was unchecked until the stream.
VerifyHeldnow runs on every start but an import, with theconfirmed_flush_lsnread right after CREATE/open as the limit. Comments and doc corrected: nothing holds the slot in between, and on PG17 it is inactive right after CREATE. Test:TestSlotIdentity_FreshStartSlotReplacedMidSnapshot_Refused. Before:SILENT SKIP ... delivered [after], never during-snapshot and lost-while-gone. After:slot_replacedat the handoff.B3, the bound rule across WAL histories.
IDENTIFY_SYSTEMvalues are stored in v4 positions. A resume is refused withpostgres.source.wal_history_changedon:TIMELINE_HISTORYleft the position's timeline before its bound, or does not descend from it;allowWALGapaccepts it withWAL HISTORY CHANGE ACCEPTED, not the DATA LOSS text, and the README has a pg_upgrade runbook. Legacy positions learn the history and warn withwal_history_unverified(sysid fingerprint and timeline).Tests:
TestSlotIdentity_WALHistoryChanged_Refused(other cluster, earlier timeline, WAL behind the position; all three were SILENT SKIP with the check removed); the failover test (a position past the timeline switch point is refused after the promotion, and the legitimate resume still passes);TestParseSwitchPoint;TestReadWALHistory;TestSlotIdentity_WALHistoryUnverified_WarnsOnlyForLegacy.SHOULDs:
logrepl.slotIdentity.beaconFailureTimeout(default 5m) of failures, or of a written beacon that never comes back, the cap is abandoned: the idle advance is uncapped again, an errorpostgres.source.slot_beacon_unavailableis logged withretained_wal_bytes(each failure warns with it first), and the next start is refused unless a record was saved since. That is an outage, never a loss. The SDK has no metrics API, so the log fields are the metric.synchronous_commit = local; the doc explains why that is durable enough.Emitreturned an error (the last 16 attempts are kept).TestBeacons_AbandonedAfterFailureTimeout.logrepl.slotIdentity.checkTimeout(default 10m, configurable), and the timeout error names both causes.allowWALGaponly after a record has been delivered.has_function_privilege('<role>', 'pg_logical_emit_message(...)', 'EXECUTE')) and release note.unproven_wal_bytes, and the release note (re-snapshot pipelines whose slot was recreated, or whose database was restored or replaced, since their last delivered record).NITs:
test/chaos/child.gono longer prints a raw position, and neither does the debug log incombined.go.TestB1_AC7_RestartHaltUnsettledadded.postgres.source.slot_identity_unprovable. Servers that cannot carry beacons write positions without an identity.Also found while fixing:
connector.yaml. The README description is generated fromconnector.yaml(readmegen), so the first version's README text would have been overwritten (and failedvalidate-generated-files). The section now lives inconnector.yaml.TestSource_FailedCDCStartInOpenReleasesTheConnectionpassing.TestCDCIterator_AllowWALGap_WarnsWhenCreateSucceedsThenFailsnow expects the retry to warn again. The slot that the failed attempt created is not the position's; without the opt-in it is refused.#447 re-review of
69231d4: what changedtx_xidandtx_commit_timefrom pgoutput BEGIN. A covered resume checks that the commit LSN still names that transaction: throughpg_walinspectwhen it is installed and usable, otherwise by peeking the inactive slot (undercheckTimeout) up to the first BEGIN whose final LSN is at or past the bound. It needs the same final LSN, xid and commit time; anything else iswal_history_changed. A restored server reuses xids too, which is why the commit time is compared. Documented false positive: on Postgres 15+, a publication or row filter changed since the position can hide the transaction;allowWALGapaccepts it. Drift markers, whose key can be one below their change's (Stop dropping the boundary row on drift-halt approval (marker carries the key one below the deciding change) #338), carry no transaction and skip the check (d5011cf, found by chaos).test/restore/run.sh(make test-restore) snapshots a data volume, restores it, advances the new WAL past the bound, and expects the refusal. Refused on 14 to 18; resumed silently before the fix. Unit/integration:TestSlotIdentity_CoveredResume_VerifiesTheTransaction(with and withoutpg_walinspect),TestCDCPosition_TxIdentityOnlyForRealCommits.TestSlotIdentity_LegacyPositionPastCurrentWAL_Refused. Legacy positions in that test carry no tx identity, as a real v0.17.3 position does (bc5daa0, the S3 mutation survived before).canPeekMessagesis gone. A standby never proves an identity (a slot it decodes is the standby's, not the primary's):slot_identity_unprovable. Advice text andconnector.yamlfixed. Reproduced on 17 with a standby-local slot, now refused (TestSlotIdentity_StandbyLocalSlot_Refused, intest/failover/run.sh), plusTestSlotIdentity_StandbyNeverProvesAnIdentity.lis past the reported flush re-arms the identity. A restart before re-arm is refused. Wording fixed inbeacon.go,config.go,connector.yaml, README and the design doc. Tests:TestBeacons_AbandonedBeaconsRearm(unit) andTestSlotIdentity_BeaconsAbandonedThenRecover(revoke, abandon, restore the grant, beacon, restart accepted; a restart before re-arm refused).TestBeacons_SlotCreatedAfterTheGuardDoesNotDecodeTheBeacon, a seam between the guard and the insert-LSN read creates a slot while the beacon's transaction is open; the creation waits for the xid and the new slot does not decode the beacon. M1:TestBeacons_CommittedButEmitErrored_StillRaisesCap(unit) andTestSlotIdentity_CommittedButEmitErrored_StreamRaisesCap(a seam after the commit makesEmiterror; the running stream still raises the cap).pg_stat_activityand compares the walsender'sbackend_startrecorded at the stream start. Test:TestSlotIdentity_PIDReuse_Refused(an impostor with the same PID and a differentbackend_start).slot_beacon_failed(TestSubscription_StreamStartedErrorStopsTheStart). Its fix text no longer suggests a GRANT: a role always sees its own backends, so that failure is the connection or the server.d8cd266. CI onbc5daa0failedTestAcceptanceV2/SRC-01. The test stopped the source during the snapshot-to-CDC handoff, the stream-holder query ran on the canceled context, and the start failed withslot_beacon_failed, which the graceful stop reported as a plugin error. Any start step that queries the server had the same race (N1 made the window bigger). A start step that fails after the run's context ended now returns the context error with the step's error still in the chain, so a refusal keeps its code. Test:TestSubscription_StartCanceledReportsTheContextError, fails without the fix. This does not hide a refusal: nothing is sent or acked, and the next start checks again.#447 round-4 review of
85da517: what changedEvery blocker was reproduced as silent loss on
85da517. The reviewer's reproductions are now real tests, and each one fails on85da517and passes on453669f.start_lsnand parsing the commit time out ofdescription: that ties a data-integrity check to a text format that changes between versions, and the slot already gives the same answer on 13 to 18. Tests:walinspectscenario oftest/restore/run.sh(the reviewer's construction: the next xid is P1's, and the row's heap record straddles P1's commit LSN);TestSameTransaction(same xid and LSN with another commit time; a commit just after the LSN);CoveredResume_VerifiesTheTransactionnow refuses "same xid and LSN, other commit time" and "commit LSN one byte early" with pg_walinspect installed too.COMMIT PREPAREDwith its own xid and that commit's time, which is what the position recorded. Test:TestSlotIdentity_PreparedTransactionPosition_ResumesCovered. It was refused on85da517with pg_walinspect installed.test/conf.dnow setsmax_prepared_transactions=10so CI runs it; it skips on servers without prepared transactions.(C-1, MaxUint64), sonamesCurrentTxtakes the current transaction, andsavedTransactionlooks for that transaction atC.driftscenario oftest/restore/run.sh,TestCDCPosition_TxIdentityNamesTheDecidingTransaction, andTestSlotIdentity_UpgradeFromV0173_CoveredButNoCommitThere_Refused.v=3) certifyr: a WAL position before which the run needed nothing more. For a beacon requested while every emitted record is acked,ris theserverWALEndof that moment: the walsender had sent everything committing before it, and all of it was acked. For the stamp written when a run opens the slot,ris the slot position the run checked or created.rof the last received beacon (Cap).confirmed_flush_lsnwhoseris at or past it.ris the point run 2 started from, before R, so the restart is refused withslot_replaced.TestSlotIdentity_ConsumedByAnotherClientBelowTheLastBeacon_Refused, the reviewer's scratch test. It holds R unacked for 12 s, so the stream has a status interval in which it could ask for a beacon. It was accepted on85da517and is refused now.TestSlotIdentity_LateBeaconFromAStoppedRun_Refused.confirmed_flush_lsnwith pg_walinspect would have to map heap records to published relations (TOAST, partitions, subtransactions, transactions that started before the bound). I judged that too fragile after B1.rin the beacon covers the graceful stop, kill -9, and any client that moves the slot past what was delivered. It needs no extension and no extra WAL. The cost: the idle slot rests at therof the last beacon instead of its start, which keeps the beacon itself and the few bytes before it.slot_replacedmessage (slot_identity.go) and the design doc. The walinspect text is gone.beaconFailureTimeout. The restart is refused (an outage), as before.Fresh-context review of
76fdea1: what changedS1: a large transaction made the slot check fail permanently.
pg_logical_slot_peek_binary_changesstops at a change limit only at a transaction boundary. A transaction of at leastfindBeaconMaxRows(2^24) rows, sitting betweenconfirmed_flush_lsnand what the check looks for, came back whole in every round. The search then gave up withErrBeaconSearchExhausted: an uncoded refusal on every retry, whichallowWALGapdid not get past.FindBeaconkeeps its growing rounds, but after the round at the clamp it reads to the end of the WAL without a limit.TxCommittingAt, which replacesFirstTxFrom) is bounded byupto_lsn = commit LSN + 1and has no change limit.postgres.source.slot_identity_check_failed, with a fix that says to raisecheckTimeout.allowWALGapaccepts the slot unproven.TestFindBeacon_LargeTransactionInFrontOfTheBeaconandTestTxCommittingAt_LargeTransactionInFrontuse the review's scale: a 10,000-row transaction with the clamp at 4,096.TestSlotIdentity_CheckCannotReadTheSlot_CodedRefusalcovers the coded refusal.76fdea1: the FindBeacon test fails with "the slot holds too many changes to search for a beacon (read 10003 changes)". The review's FirstTxFrom probe fails the same way. TheTxCommittingAtand coded-refusal tests cannot run on76fdea1, because the function and the code are new.S2: guard markers, and
guardmutpasses.logrepl-idle-advance-beacon-cap(the cap inreportedPositions);logrepl-beacon-covers-confirmed(verifyBeacon'sr >= confirmed);logrepl-beacon-xid-first(assignXid, made a call of its own so the gate can remove it);logrepl-beacon-held-by-this-walsender(the PID andbackend_startchecks incheckHeld);logrepl-verifyheld-limit;logrepl-covered-transaction-checked.backend_startis read for the slot'sactive_pid, so a different holder already fails thebackend_startcomparison.make guardmut-equivalent run (own compose project, ports 56172/56173): all 12 guards killed, none survived. The baseline stays empty.This round's verification on
3ae5fb2: full suite (-race -timeout 20m, fresh volumes,CONDUIT_PG_TEST_PORT=56133;source/logrepltook 460 s) plus the invalidation test alone, chaos (port 56135), external-state (all 10 cases), failover on 17 and 18, restore on 17 (basic, walinspect, drift), and lint with all build tags: all pass.S3:
make testruns bothgo testinvocations with-timeout 20m.NITs:
leftTimelineBy,<=). The switch point is the first LSN of the new timeline.TestLeftTimelineByfails with<.slot_replacedmessage. It now names two causes that lose nothing: a rollback followed by the upgrade again, and beacons abandoned afterbeaconFailureTimeout.postgres.source.wal_history_changed.Security review of
453669f: keepalive WAL end vs undelivered changesThe concern: a beacon requested while every record is acked certifies
r = serverWALEndfrom the last keepalive. If a keepalive's WAL end could be past committed changes the stream has not delivered, the beacon'srand the idle advance would skip them. The idle advance is the sameallAcked → serverWALEndthatmainalready has.I measured it against real servers, on 13, 14, 15, 16, 17 and 18. There is no hole, and no code change.
wal_sender_timeout/2has passed without one, even mid-transaction.sentPtr, the start of the commit record being replayed. It is exactly the transaction's own commit LSN, never past it. It arrived after 14k to 30k of the transaction's changes.confirmed_flush_lsnat the commit LSN. The slot then decodes the whole transaction again (all 100,000 or 200,000 inserts plus the next transaction). Postgres skips only commits that start beforeconfirmed_flush_lsn.rare safe, onmainand here.Tests (
3901c9f):TestWalsender_KeepaliveInsideATransactionNeverPassesItsCommit(internal, raw client) pins this server behaviour on every version. It uses one large transaction and no replies, so a keepalive arrives inside the transaction. It asserts the keepalive is not past the commit, reports that LSN as flushed, crashes, and requires the slot to decode every row again. Passes on 13 to 18.TestSlotIdentity_LargeTransactionInFlight_SlotNeverPassesItruns the connector through a 100,000-row transaction withwal_sender_timeout=8s. While the transaction is in flight it checks that neitherconfirmed_flush_lsnnor the beacon cap passes its commit. It stops after 60,000 acked rows and requires the restart to deliver exactly the remaining 40,000. In my runs the connector's slot did not reach the commit LSN itself, becauseallAckedwas false whenever a keepalive arrived. So this test covers the outcome; the internal test covers the boundary.synchronous_standby_namesset) carry that empty transaction's end LSN, after every earlier transaction was sent. That is from reading the Postgres source; no test runs synchronous replication.External-state suite (#451)
#451's suite failed on this PR, as designed. Three pending cases named tests this PR adds, and the snapshot-restore known gap now failed loudly instead of losing rows.
76fdea1makesslot-recreated-same-name,privilege-revoked-emit-messageandfailoverrequired. It also makessnapshot-restorerequired withpostgres.source.wal_history_changedas an accepted refusal; the case had listed onlyslot_wal_gapandslot_invalidated. All ten cases pass locally and in CI.Evidence
Docker,
-race. Every stack ran on a moved port (never 5433/5434) and was torn down withdown -v. Head453669funless noted.453669f: all 8 checks pass. CI on3ae5fb2: all 10 checks pass, including ci: external-state chaos suite (slot, privileges, network, volume restore) #451'sexternal-stateand ci: guard mutation gate for source/logrepl #452'sguardmut: test, golangci-lint, validate-generated-files (the three required), chaos, chaos-build, debezium, test-timezones, tls-channel-binding.go test -race ./...on the compose stack (bitnami 17.5, fresh volumes,PG_TEST_PORT=56133 PGBOUNCER_TEST_PORT=56134), then the slot invalidation test alone, asmake testdoes: passes.-tags conduitchaos -race, chaos stack on port 56135 in a scratch worktree: passes. Every halt scenario resumes from a drift marker, which now goes through the transaction check.test/failover/run.shon 17 and 18 passes, includingStandbyLocalSlot_Refused. This ran on the round-4 tree before the last test-only changes.test/restore/run.shruns three scenarios (basic, walinspect, drift).wal_history_changed.85da517, the walinspect and drift scenarios fail on 17 with SILENT SKIP. This ran on the round-4 tree before the last test-only changes.TestFindBeaconfailed; its expectation predatedr(it advanced the slot to the beacon's start and expected the beacon to vouch). It is fixed to assert the new rule and passes on 14 to 18.pg_log_standby_snapshot()import tests, which are unchanged and fail the same way on the base.453669f(-count=2). One failure on 17 was fix: keep record keys and row data out of trace and debug logs #446'sTestRelationSet_DecodeErrorDoesNotContainTheValue/interval; it passed when re-run, on the base and on this head. One failure on 13 wasTestRelationSetTemporalTypes, which fails on the base too.golangci-lint, also withrestore,failover,conduitchaos,debezium: 0 issues.Mutation checks on
453669f(42, each reverted afterwards). All are caught; none survive.RecreatedBetweenRuns_Refused,SnapshotPosition_RecreatedRefusedIdleAdvancedSlot_ResumesWithoutFalsePositiveVerifyHeldnever setAllowWALGap_WarnsWhenSlotRecreatedAfterCheck,FreshStartSlotReplacedMidSnapshot_RefusedMissingAtIdentityCheckThenBack_RefusedForgedBeacons_StillRefused,RefusesForgedAndCopiedBeaconsrabovelacceptedTestBeaconContent_VerifyRecreatedBetweenRuns_Refused,HalfFinishedCreate_RefusedExhaustedAtClampedLimitBeaconAfterWalsenderLoss_DoesNotVouchForReplacement,EmitOnlyIntoTheHeldSlotVerifyHeldonly for saved positionsFreshStartSlotReplacedMidSnapshot_RefusedWALHistoryChanged_Refused,LegacyPositionPastCurrentWAL_RefusedWALHistoryChanged_Refusedwal_history_unverifiedwarningWALHistoryUnverified_WarnsOnlyForLegacyAbandonedAfterFailureTimeoutUnsupportedServer_WarnsAndResumes(on 13)StandbyNeverProvesAnIdentityCoveredResume_VerifiesTheTransactionLegacyPositionPastCurrentWAL_Refused(now asserts which check refused)AbandonedAfterFailureTimeout,AbandonedBeaconsRearmAbandonedBeaconsRearmEmiterror (stream side /Emitside)CommittedButEmitErrored_StillRaisesCap/ newTestBeacons_EmitErrorAfterCommitKeepsTheAttemptSlotCreatedAfterTheGuardDoesNotDecodeTheBeaconbackend_startignoredPIDReuse_RefusedStreamStartedErrorStopsTheStartIdleAdvancedSlot_ResumesWithoutFalsePositive,TestFindBeaconCoveredResume_VerifiesTheTransactionStartCanceledReportsTheContextErrorTxIdentityNamesTheDecidingTransactionTestSameTransaction,CoveredResume_VerifiesTheTransactionTestSameTransaction,CoveredResume_VerifiesTheTransactionTxIdentityNamesTheDecidingTransaction(and the restoredriftscenario, by the85da517run)TestDrift338_*(Approval, CrashBeforeAck, ReformattedMarkerAck, InterleavedBoundary)TestSameTransaction,UpgradeFromV0173_CoveredButNoCommitThere_RefusedUpgradeFromV0173_CoveredButNoCommitThere_Refusedrnot checkedConsumedByAnotherClientBelowTheLastBeacon_Refused,TestBeaconContent_VerifyConsumedByAnotherClientBelowTheLastBeacon_RefusedTestBeacons_ObserveConsumedByAnotherClientBelowTheLastBeacon_RefusedSnapshotPosition_RecreatedRefused(its intact resume)The old "no
lbound" mutation is now equivalent:r <= landr >= confirmedimply it, so the redundant check was removed and ther <= lcheck is mutated instead.Failure-mode analysis
synchronized_standby_slots(including the timeline change), a walsender still releasing the slot, and an in-use slot on a fresh start.pg_replication_slot_advance, a rollback that idled, a failover withoutsynchronized_standby_slotswhile the standby lags, beacons abandoned after the failure timeout, and positions with an identity on servers that cannot check them.allowWALGap; pre-v0.17.5 positions now take a peek on a covered resume too.rof the last received beacon), lifted after the failure timeout. Beacon transactions never become records or change keys. Beacons arev=3; nothing before this PR wrote beacons, so there is no compatibility to keep.replication slot verified ...withproof. WARN codes:slot_identity_unverified,wal_history_unverified,slot_identity_unsupported,slot_beacon_public,slot_beacon_unavailable. ERROR:slot_beacon_unavailable(abandoned). Refusals:slot_replaced,slot_beacon_failed,slot_identity_unprovable,wal_history_changed.Not verified
TestBeacons_EmitErrorAfterCommitKeepsTheAttempt|TestBeaconContent_Verify|TestFindBeaconfailed on the test stack early in this round and printed no reason in my filtered output; 6-vruns, 8 + 8 counted runs and 25 binary runs since did not reproduce it. That was probably theTestFindBeaconexpectation fixed above: it failed only when WAL landed between the slot's creation and the stamp. I could not confirm that.synchronized_standby_slotsunder standby lag: not reproduced.make test-failover,make test-restore).TestImport_RetreatedSlotIsRefusedand the twopg_log_standby_snapshot()import tests: unchanged, fail on the base too.🤖 Generated with Claude Code
https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH