Skip to content

fix(source): refuse a replication slot that was replaced under the same name (#442, Tier 1, v0.17.5) - #447

Merged
devarismeroxa merged 20 commits into
mainfrom
fix/slot-identity
Oct 24, 2026
Merged

devarismeroxa merged 20 commits into
mainfrom
fix/slot-identity

Conversation

@devarismeroxa

@devarismeroxa devarismeroxa commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #442. Becomes v0.17.5. Rebased onto main after #445 merged (squash 2edf779; git rebase --onto origin/main bccd617), base is main. Roadmap: Phase 0 data-integrity floor (invariant 3).

Risk tier: Tier 1 (data path: slot advance, resume check, serialized position format). Needs DeVaris's sign-off in a separate session. Design doc: docs/design-documents/20261021-slot-identity.md.

Commits: b6f8539 design doc and 6005085 implementation (both as first reviewed, rebased); 87e759b design-doc changes and 645f23e fixes for the #447 review; 32ea7ac, df5899b and 69231d4 test-only follow-ups; 1050436 design-doc changes and 39dae13 fixes for the re-review of 69231d4; d5011cf, bc5daa0 and d8cd266 follow-ups found by chaos, mutation and CI; 85da517 merges main for #446 (one conflict, the unparsable-position debug log in combined.go: both sides log only its length, #446's field name kept). A merge rather than a rebase so the push stays fast-forward. 453669f addresses the round-4 review of 85da517; 3901c9f adds the keepalive tests for the security review of 453669f; e2e3e61 merges main for #451 and #452 (conflict: the test port override both sides added; main's CONDUIT_PG_TEST_PORT/CONDUIT_PG_TEST_BOUNCER_PORT kept); 76fdea1 flips the external-state cases this PR closes to required; 26741c6 merges main for #450 (clean); 3ae5fb2 addresses the fresh-context review of 76fdea1. Head: 3ae5fb2.

Problem

A slot dropped and created again under the same name exists, is healthy, and is past the saved position, exactly like a slot the connector advanced while idle. A resume on it skipped every change in between without a word. #441 catches a missing or invalidated slot, not this.

Design (summary; the doc has the research per Postgres version and the alternatives)

  • Positions (format version 4) carry slot_identity, a 256-bit secret key, and system_identifier + timeline (the WAL history).
  • The connector writes authenticated beacons into the WAL (pg_logical_emit_message, HMAC-SHA256 over slot, nonce and the WAL position read before it) and never advances an idle slot past the last beacon the stream delivered back. Acked-record reports are unchanged.
  • On resume: refuse another WAL history (wal_history_changed); accept a slot at or below the saved commit LSN; past it, accept only a slot that still decodes a beacon of the position's key written after its confirmed_flush_lsn; otherwise slot_replaced. VerifyHeld checks the slot again once the stream holds it.
  • Pre-v0.17.5 positions: learned on the first resume (Decision 6, kept per the review), with slot_identity_unverified (now carrying unproven_wal_bytes) and wal_history_unverified. Protection begins at the first record saved after the upgrade.

#447 review: what changed

Blockers (each with the reviewer's repro as a test: fails before the fix, passes after):

  • B1, a beacon written after the slot was replaced vouched for it. Emit now takes an xid first (pg_current_xact_id()), then checks inside the same transaction that the slot is held by this run's walsender (active_pid = the replication connection's backend PID) or, before the stream, inactive and not past the limit this run checked or created it at. Only then does it read the insert LSN and write. Why a slot created after the check cannot decode that xid is in the doc, Decision 2. Test: TestSlotIdentity_BeaconAfterWalsenderLoss_DoesNotVouchForReplacement. Before: the emit succeeded and the restart delivered only [after]. After: the emit fails with ErrSlotNotHeld and the restart is refused with slot_replaced. Unit: TestBeacons_EmitOnlyIntoTheHeldSlot.

  • B2, a fresh start's slot was unchecked until the stream. VerifyHeld now runs on every start but an import, with the confirmed_flush_lsn read right after CREATE/open as the limit. Comments and doc corrected: nothing holds the slot in between, and on PG17 it is inactive right after CREATE. Test: TestSlotIdentity_FreshStartSlotReplacedMidSnapshot_Refused. Before: SILENT SKIP ... delivered [after], never during-snapshot and lost-while-gone. After: slot_replaced at the handoff.

  • B3, the bound rule across WAL histories. IDENTIFY_SYSTEM values are stored in v4 positions. A resume is refused with postgres.source.wal_history_changed on:

    • another system identifier;
    • an earlier timeline;
    • a later timeline whose TIMELINE_HISTORY left the position's timeline before its bound, or does not descend from it;
    • a bound past the server's current WAL position.

    allowWALGap accepts it with WAL HISTORY CHANGE ACCEPTED, not the DATA LOSS text, and the README has a pg_upgrade runbook. Legacy positions learn the history and warn with wal_history_unverified (sysid fingerprint and timeline).

    Tests: TestSlotIdentity_WALHistoryChanged_Refused (other cluster, earlier timeline, WAL behind the position; all three were SILENT SKIP with the check removed); the failover test (a position past the timeline switch point is refused after the promotion, and the legitimate resume still passes); TestParseSwitchPoint; TestReadWALHistory; TestSlotIdentity_WALHistoryUnverified_WarnsOnlyForLegacy.

SHOULDs:

  • S1, failing beacons.
    • After logrepl.slotIdentity.beaconFailureTimeout (default 5m) of failures, or of a written beacon that never comes back, the cap is abandoned: the idle advance is uncapped again, an error postgres.source.slot_beacon_unavailable is logged with retained_wal_bytes (each failure warns with it first), and the next start is refused unless a record was saved since. That is an outage, never a loss. The SDK has no metrics API, so the log fields are the metric.
    • Beacons commit with synchronous_commit = local; the doc explains why that is durable enough.
    • A beacon is recognised from the stream by its nonce even when its Emit returned an error (the last 16 attempts are kept).
    • The doc has the retention math. Test: TestBeacons_AbandonedAfterFailureTimeout.
  • S2. logrepl.slotIdentity.checkTimeout (default 10m, configurable), and the timeout error names both causes.
  • S3. Neither fix is possible: the cap cannot go below a new slot's creation point, and Conduit has no position-only writes. The README now says: remove allowWALGap only after a record has been delivered.
  • S4. README pre-upgrade query (has_function_privilege('<role>', 'pg_logical_emit_message(...)', 'EXECUTE')) and release note.
  • S5. Issue release/v0.14.x: reserve position format version 4 (slot identity, WAL history) #449 opened. v0.14.x reads v4 like v2 and ignores the new fields; it does not reject them, per its "read, not rejected" contract.
  • S6. Decision 6 wording ("protection begins at the first record saved after the upgrade"), unproven_wal_bytes, and the release note (re-snapshot pipelines whose slot was recreated, or whose database was restored or replaced, since their last delivered record).

NITs:

  • N1: test/chaos/child.go no longer prints a raw position, and neither does the debug log in combined.go.
  • N2: the mutation list below is re-run on this head.
  • N3: the threat model says plainly that positions reach destinations and the API, so the key is readable there, and that forging still needs slot-drop privileges plus WAL writes.
  • N4: TestB1_AC7_RestartHaltUnsettled added.
  • N5: done. A position with an identity is never accepted unproven: a standby on 16+ peeks for the primary's beacons, and anything else refuses with postgres.source.slot_identity_unprovable. Servers that cannot carry beacons write positions without an identity.

Also found while fixing:

  • README was not generated from connector.yaml. The README description is generated from connector.yaml (readmegen), so the first version's README text would have been overwritten (and failed validate-generated-files). The section now lives in connector.yaml.
  • Fresh start on an in-use slot. On a fresh start whose slot is in use by another connection, the stamp is deferred to the stream start, which fails with its own error. This keeps TestSource_FailedCDCStartInOpenReleasesTheConnection passing.
  • fix(source): warn when allowWALGap recreates a slot dropped after the check #445's warn-twice test changed. TestCDCIterator_AllowWALGap_WarnsWhenCreateSucceedsThenFails now expects the retry to warn again. The slot that the failed attempt created is not the position's; without the opt-in it is refused.

#447 re-review of 69231d4: what changed

  • S2, same-timeline restore (reproduced silent loss). v4 positions now carry tx_xid and tx_commit_time from pgoutput BEGIN. A covered resume checks that the commit LSN still names that transaction: through pg_walinspect when it is installed and usable, otherwise by peeking the inactive slot (under checkTimeout) up to the first BEGIN whose final LSN is at or past the bound. It needs the same final LSN, xid and commit time; anything else is wal_history_changed. A restored server reuses xids too, which is why the commit time is compared. Documented false positive: on Postgres 15+, a publication or row filter changed since the position can hide the transaction; allowWALGap accepts it. Drift markers, whose key can be one below their change's (Stop dropping the boundary row on drift-halt approval (marker carries the key one below the deciding change) #338), carry no transaction and skip the check (d5011cf, found by chaos).
    • Test: test/restore/run.sh (make test-restore) snapshots a data volume, restores it, advances the new WAL past the bound, and expects the refusal. Refused on 14 to 18; resumed silently before the fix. Unit/integration: TestSlotIdentity_CoveredResume_VerifiesTheTransaction (with and without pg_walinspect), TestCDCPosition_TxIdentityOnlyForRealCommits.
  • S3. Legacy positions get the "bound past the server's current WAL" check too. Test: TestSlotIdentity_LegacyPositionPastCurrentWAL_Refused. Legacy positions in that test carry no tx identity, as a real v0.17.3 position does (bc5daa0, the S3 mutation survived before).
  • S1. canPeekMessages is gone. A standby never proves an identity (a slot it decodes is the standby's, not the primary's): slot_identity_unprovable. Advice text and connector.yaml fixed. Reproduced on 17 with a standby-local slot, now refused (TestSlotIdentity_StandbyLocalSlot_Refused, in test/failover/run.sh), plus TestSlotIdentity_StandbyNeverProvesAnIdentity.
  • S4. After abandonment, beacons keep being written, uncapped. The first delivered beacon whose l is past the reported flush re-arms the identity. A restart before re-arm is refused. Wording fixed in beacon.go, config.go, connector.yaml, README and the design doc. Tests: TestBeacons_AbandonedBeaconsRearm (unit) and TestSlotIdentity_BeaconsAbandonedThenRecover (revoke, abandon, restore the grant, beacon, restart accepted; a restart before re-arm refused).
  • S5. M3: TestBeacons_SlotCreatedAfterTheGuardDoesNotDecodeTheBeacon, a seam between the guard and the insert-LSN read creates a slot while the beacon's transaction is open; the creation waits for the xid and the new slot does not decode the beacon. M1: TestBeacons_CommittedButEmitErrored_StillRaisesCap (unit) and TestSlotIdentity_CommittedButEmitErrored_StreamRaisesCap (a seam after the commit makes Emit error; the running stream still raises the cap).
  • N1. The guard joins pg_stat_activity and compares the walsender's backend_start recorded at the stream start. Test: TestSlotIdentity_PIDReuse_Refused (an impostor with the same PID and a different backend_start).
  • N2. A stream start whose walsender cannot be identified fails with slot_beacon_failed (TestSubscription_StreamStartedErrorStopsTheStart). Its fix text no longer suggests a GRANT: a role always sees its own backends, so that failure is the connection or the server.
  • CI regression from N1/N2, fixed in d8cd266. CI on bc5daa0 failed TestAcceptanceV2/SRC-01. The test stopped the source during the snapshot-to-CDC handoff, the stream-holder query ran on the canceled context, and the start failed with slot_beacon_failed, which the graceful stop reported as a plugin error. Any start step that queries the server had the same race (N1 made the window bigger). A start step that fails after the run's context ended now returns the context error with the step's error still in the chain, so a refusal keeps its code. Test: TestSubscription_StartCanceledReportsTheContextError, fails without the fix. This does not hide a refusal: nothing is sent or acked, and the next start checks again.

#447 round-4 review of 85da517: what changed

Every blocker was reproduced as silent loss on 85da517. The reviewer's reproductions are now real tests, and each one fails on 85da517 and passes on 453669f.

  • B1, pg_walinspect accepted a restored history (15, 17). The walinspect path is gone; the covered-resume check always reads the slot and compares the commit LSN, the xid and the commit time. I did not fix it by checking start_lsn and parsing the commit time out of description: that ties a data-integrity check to a text format that changes between versions, and the slot already gives the same answer on 13 to 18. Tests:
    • the walinspect scenario of test/restore/run.sh (the reviewer's construction: the next xid is P1's, and the row's heap record straddles P1's commit LSN);
    • TestSameTransaction (same xid and LSN with another commit time; a commit just after the LSN);
    • CoveredResume_VerifiesTheTransaction now refuses "same xid and LSN, other commit time" and "commit LSN one byte early" with pg_walinspect installed too.
  • S1, 2PC false positive. Gone with walinspect. The slot decodes a prepared transaction at COMMIT PREPARED with its own xid and that commit's time, which is what the position recorded. Test: TestSlotIdentity_PreparedTransactionPosition_ResumesCovered. It was refused on 85da517 with pg_walinspect installed. test/conf.d now sets max_prepared_transactions=10 so CI runs it; it skips on servers without prepared transactions.
  • B2, drift-marker positions skipped the check (17).
    • A marker's position now names the transaction it halted before: its key is (C-1, MaxUint64), so namesCurrentTx takes the current transaction, and savedTransaction looks for that transaction at C.
    • Any position with a commit LSN but no xid (v2/v3, the reviewer's S2) must at least have a commit starting exactly there.
    • Tests: the drift scenario of test/restore/run.sh, TestCDCPosition_TxIdentityNamesTheDecidingTransaction, and TestSlotIdentity_UpgradeFromV0173_CoveredButNoCommitThere_Refused.
    • The chaos suite resumes from markers on every halt scenario and passes, so no false positive there.
  • B3, the pipeline's own slot consumed below its last beacon. This can be detected without pg_walinspect, and that is what I implemented.
    • Beacons (now v=3) certify r: a WAL position before which the run needed nothing more. For a beacon requested while every emitted record is acked, r is the serverWALEnd of that moment: the walsender had sent everything committing before it, and all of it was acked. For the stamp written when a run opens the slot, r is the slot position the run checked or created.
    • The idle advance stops at the r of the last received beacon (Cap).
    • A slot past the saved position is accepted only on a beacon written after its confirmed_flush_lsn whose r is at or past it.
    • In the reviewer's scenario the only beacon left is run 2's stamp, whose r is the point run 2 started from, before R, so the restart is refused with slot_replaced.
    • Test: TestSlotIdentity_ConsumedByAnotherClientBelowTheLastBeacon_Refused, the reviewer's scratch test. It holds R unacked for 12 s, so the stream has a status interval in which it could ask for a beacon. It was accepted on 85da517 and is refused now.
    • Also added from the reviewer's scratch file: TestSlotIdentity_LateBeaconFromAStoppedRun_Refused.
    • On the questions asked:
      • Whether run 2 had unacked records when it stopped is not stored anywhere a restart can read. Conduit has no position-only writes. A "stopped with records pending" beacon would cover only graceful stops, and finding it would mean reading the whole backlog.
      • Reading the WAL between the position and confirmed_flush_lsn with pg_walinspect would have to map heap records to published relations (TOAST, partitions, subtransactions, transactions that started before the bound). I judged that too fragile after B1.
      • Certifying r in the beacon covers the graceful stop, kill -9, and any client that moves the slot past what was delivered. It needs no extension and no extra WAL. The cost: the idle slot rests at the r of the last beacon instead of its start, which keeps the beacon itself and the few bytes before it.
  • Wording. README and connector.yaml now say exactly this: a slot another client moved past changes the pipeline had not delivered is refused, including changes it had received but not acked; one moved only over delivered changes loses nothing and is accepted. The same goes for the slot_replaced message (slot_identity.go) and the design doc. The walinspect text is gone.
  • What stays undetectable (DeVaris decides):
    • A pre-v0.17.5 position on a restored server whose new history commits a transaction at exactly the position's commit LSN. That position has no xid to compare. The next position the run saves has one.
    • A client that moves the slot only up to what a beacon certifies. This is not a loss, since nothing undelivered is skipped.
    • Beacons abandoned after beaconFailureTimeout. The restart is refused (an outage), as before.

Fresh-context review of 76fdea1: what changed

S1: a large transaction made the slot check fail permanently.

  • Cause: pg_logical_slot_peek_binary_changes stops at a change limit only at a transaction boundary. A transaction of at least findBeaconMaxRows (2^24) rows, sitting between confirmed_flush_lsn and what the check looks for, came back whole in every round. The search then gave up with ErrBeaconSearchExhausted: an uncoded refusal on every retry, which allowWALGap did not get past.
  • Fixes:
    • FindBeacon keeps its growing rounds, but after the round at the clamp it reads to the end of the WAL without a limit.
    • The covered-resume check (TxCommittingAt, which replaces FirstTxFrom) is bounded by upto_lsn = commit LSN + 1 and has no change limit.
    • A check that still cannot read the slot (timeout, peek error) refuses with the new code postgres.source.slot_identity_check_failed, with a fix that says to raise checkTimeout. allowWALGap accepts the slot unproven.
  • Tests: TestFindBeacon_LargeTransactionInFrontOfTheBeacon and TestTxCommittingAt_LargeTransactionInFront use the review's scale: a 10,000-row transaction with the clamp at 4,096. TestSlotIdentity_CheckCannotReadTheSlot_CodedRefusal covers the coded refusal.
  • Revert check on 76fdea1: the FindBeacon test fails with "the slot holds too many changes to search for a beacon (read 10003 changes)". The review's FirstTxFrom probe fails the same way. The TxCommittingAt and coded-refusal tests cannot run on 76fdea1, because the function and the code are new.

S2: guard markers, and guardmut passes.

  • New markers:
    • logrepl-idle-advance-beacon-cap (the cap in reportedPositions);
    • logrepl-beacon-covers-confirmed (verifyBeacon's r >= confirmed);
    • logrepl-beacon-xid-first (assignXid, made a call of its own so the gate can remove it);
    • logrepl-beacon-held-by-this-walsender (the PID and backend_start checks in checkHeld);
    • logrepl-verifyheld-limit;
    • logrepl-covered-transaction-checked.
  • The walsender check is one guard, not two. Marked separately, the PID check survived: backend_start is read for the slot's active_pid, so a different holder already fails the backend_start comparison.
  • Local make guardmut-equivalent run (own compose project, ports 56172/56173): all 12 guards killed, none survived. The baseline stays empty.

This round's verification on 3ae5fb2: full suite (-race -timeout 20m, fresh volumes, CONDUIT_PG_TEST_PORT=56133; source/logrepl took 460 s) plus the invalidation test alone, chaos (port 56135), external-state (all 10 cases), failover on 17 and 18, restore on 17 (basic, walinspect, drift), and lint with all build tags: all pass.

S3: make test runs both go test invocations with -timeout 20m.

NITs:

  • Timeline switch point. A bound at the timeline switch point is now refused (leftTimelineBy, <=). The switch point is the first LSN of the new timeline. TestLeftTimelineBy fails with <.
  • slot_replaced message. It now names two causes that lose nothing: a rollback followed by the upgrade again, and beacons abandoned after beaconFailureTimeout.
  • External-state snapshot restore pins postgres.source.wal_history_changed.

Security review of 453669f: keepalive WAL end vs undelivered changes

The concern: a beacon requested while every record is acked certifies r = serverWALEnd from the last keepalive. If a keepalive's WAL end could be past committed changes the stream has not delivered, the beacon's r and the idle advance would skip them. The idle advance is the same allAcked → serverWALEnd that main already has.

I measured it against real servers, on 13, 14, 15, 16, 17 and 18. There is no hole, and no code change.

  • During a 100,000–200,000-row transaction, the walsender asks for a reply once wal_sender_timeout/2 has passed without one, even mid-transaction.
  • That keepalive's WAL end is sentPtr, the start of the commit record being replayed. It is exactly the transaction's own commit LSN, never past it. It arrived after 14k to 30k of the transaction's changes.
  • Reporting that LSN as flushed and then dropping the connection leaves confirmed_flush_lsn at the commit LSN. The slot then decodes the whole transaction again (all 100,000 or 200,000 inserts plus the next transaction). Postgres skips only commits that start before confirmed_flush_lsn.
  • So every transaction that commits before a keepalive's WAL end has been sent. With every received record acked, both the idle advance and r are safe, on main and here.

Tests (3901c9f):

  • TestWalsender_KeepaliveInsideATransactionNeverPassesItsCommit (internal, raw client) pins this server behaviour on every version. It uses one large transaction and no replies, so a keepalive arrives inside the transaction. It asserts the keepalive is not past the commit, reports that LSN as flushed, crashes, and requires the slot to decode every row again. Passes on 13 to 18.
  • TestSlotIdentity_LargeTransactionInFlight_SlotNeverPassesIt runs the connector through a 100,000-row transaction with wal_sender_timeout=8s. While the transaction is in flight it checks that neither confirmed_flush_lsn nor the beacon cap passes its commit. It stops after 60,000 acked rows and requires the restart to deliver exactly the remaining 40,000. In my runs the connector's slot did not reach the commit LSN itself, because allAcked was false whenever a keepalive arrived. So this test covers the outcome; the internal test covers the boundary.
  • Not done: sync-rep keepalives for skipped empty transactions (Postgres 15+, sent only with synchronous_standby_names set) carry that empty transaction's end LSN, after every earlier transaction was sent. That is from reading the Postgres source; no test runs synchronous replication.

External-state suite (#451)

#451's suite failed on this PR, as designed. Three pending cases named tests this PR adds, and the snapshot-restore known gap now failed loudly instead of losing rows. 76fdea1 makes slot-recreated-same-name, privilege-revoked-emit-message and failover required. It also makes snapshot-restore required with postgres.source.wal_history_changed as an accepted refusal; the case had listed only slot_wal_gap and slot_invalidated. All ten cases pass locally and in CI.

Evidence

Docker, -race. Every stack ran on a moved port (never 5433/5434) and was torn down with down -v. Head 453669f unless noted.

  • CI on 453669f: all 8 checks pass. CI on 3ae5fb2: all 10 checks pass, including ci: external-state chaos suite (slot, privileges, network, volume restore) #451's external-state and ci: guard mutation gate for source/logrepl #452's guardmut: test, golangci-lint, validate-generated-files (the three required), chaos, chaos-build, debezium, test-timezones, tls-channel-binding.
  • Full suite. go test -race ./... on the compose stack (bitnami 17.5, fresh volumes, PG_TEST_PORT=56133 PGBOUNCER_TEST_PORT=56134), then the slot invalidation test alone, as make test does: passes.
  • Chaos: -tags conduitchaos -race, chaos stack on port 56135 in a scratch worktree: passes. Every halt scenario resumes from a drift marker, which now goes through the transaction check.
  • Failover: test/failover/run.sh on 17 and 18 passes, including StandbyLocalSlot_Refused. This ran on the round-4 tree before the last test-only changes.
  • Restore: test/restore/run.sh runs three scenarios (basic, walinspect, drift).
    • 15, 16, 17 and 18: all refused with wal_history_changed.
    • 14: basic and drift refused; walinspect skips, because pg_walinspect needs 15+.
    • On 85da517, the walinspect and drift scenarios fail on 17 with SILENT SKIP. This ran on the round-4 tree before the last test-only changes.
  • Version matrix (official 13.23, 14.24, 15.19, 16.15, 17.11, 18.6; acceptance, slot identity, beacons, subscription, WAL history, CDC iterator, drift, positions):
    • All green on 16 and 17.
    • On 18, TestFindBeacon failed; its expectation predated r (it advanced the slot to the beacon's start and expected the beacon to vouch). It is fixed to assert the new rule and passes on 14 to 18.
    • On 13 to 15 the only failures are the two pg_log_standby_snapshot() import tests, which are unchanged and fail the same way on the base.
    • The whole internal package was re-run on 13 to 18 at 453669f (-count=2). One failure on 17 was fix: keep record keys and row data out of trace and debug logs #446's TestRelationSet_DecodeErrorDoesNotContainTheValue/interval; it passed when re-run, on the base and on this head. One failure on 13 was TestRelationSetTemporalTypes, which fails on the base too.
  • Lint: golangci-lint, also with restore,failover,conduitchaos,debezium: 0 issues.

Mutation checks on 453669f (42, each reverted afterwards). All are caught; none survive.

Mutation Tests that fail
Beacon check always passes RecreatedBetweenRuns_Refused, SnapshotPosition_RecreatedRefused
No idle cap IdleAdvancedSlot_ResumesWithoutFalsePositive
VerifyHeld never set AllowWALGap_WarnsWhenSlotRecreatedAfterCheck, FreshStartSlotReplacedMidSnapshot_Refused
Gone-then-back slot not refused MissingAtIdentityCheckThenBack_Refused
MAC not checked ForgedBeacons_StillRefused, RefusesForgedAndCopiedBeacons
r above l accepted TestBeaconContent_Verify
No key rotation on an accepted gap RecreatedBetweenRuns_Refused, HalfFinishedCreate_Refused
Unclamped peek rounds ExhaustedAtClampedLimit
B1 guard removed BeaconAfterWalsenderLoss_DoesNotVouchForReplacement, EmitOnlyIntoTheHeldSlot
VerifyHeld only for saved positions FreshStartSlotReplacedMidSnapshot_Refused
No WAL history check WALHistoryChanged_Refused, LegacyPositionPastCurrentWAL_Refused
No sysid/timeline check WALHistoryChanged_Refused
No wal_history_unverified warning WALHistoryUnverified_WarnsOnlyForLegacy
Never abandon failing beacons AbandonedAfterFailureTimeout
Unprovable accepted UnsupportedServer_WarnsAndResumes (on 13)
Standby may prove StandbyNeverProvesAnIdentity
Covered transaction not checked CoveredResume_VerifiesTheTransaction
Legacy skips the current-WAL check LegacyPositionPastCurrentWAL_Refused (now asserts which check refused)
Abandoned beacons not retried AbandonedAfterFailureTimeout, AbandonedBeaconsRearm
No re-arm / no probe cap AbandonedBeaconsRearm
Attempts dropped after an Emit error (stream side / Emit side) CommittedButEmitErrored_StillRaisesCap / new TestBeacons_EmitErrorAfterCommitKeepsTheAttempt
No xid before the guard SlotCreatedAfterTheGuardDoesNotDecodeTheBeacon
backend_start ignored PIDReuse_Refused
Stream holder skipped StreamStartedErrorStopsTheStart
Non-transactional beacon IdleAdvancedSlot_ResumesWithoutFalsePositive, TestFindBeacon
Tx info not wired CoveredResume_VerifiesTheTransaction
Canceled start reported as a refusal (two variants) StartCanceledReportsTheContextError
Tx identity on unrelated keys TxIdentityNamesTheDecidingTransaction
B1: commit time not compared TestSameTransaction, CoveredResume_VerifiesTheTransaction
B1: commit LSN not compared TestSameTransaction, CoveredResume_VerifiesTheTransaction
B2: markers name no transaction TxIdentityNamesTheDecidingTransaction (and the restore drift scenario, by the 85da517 run)
B2: marker transaction looked for at the key TestDrift338_* (Approval, CrashBeforeAck, ReformattedMarkerAck, InterleavedBoundary)
B2: positions without an xid: commit LSN not compared TestSameTransaction, UpgradeFromV0173_CoveredButNoCommitThere_Refused
B2: legacy transaction check skipped UpgradeFromV0173_CoveredButNoCommitThere_Refused
B3: r not checked ConsumedByAnotherClientBelowTheLastBeacon_Refused, TestBeaconContent_Verify
B3: beacons certify their start ConsumedByAnotherClientBelowTheLastBeacon_Refused
B3: cap at the beacon's start TestBeacons_Observe
B3: beacons requested with records in flight ConsumedByAnotherClientBelowTheLastBeacon_Refused
B3: stamp certifies nothing (false positive) SnapshotPosition_RecreatedRefused (its intact resume)

The old "no l bound" mutation is now equivalent: r <= l and r >= confirmed imply it, so the redundant check was removed and the r <= l check is mutated instead.

Failure-mode analysis

  • What could break: a false positive (an outage).
    • Covered by tests: idle advance, quick restart, snapshot resume, legacy upgrade, failover with synchronized_standby_slots (including the timeline change), a walsender still releasing the slot, and an in-use slot on a fresh start.
    • Refused by design (all documented): pg_upgrade, manual pg_replication_slot_advance, a rollback that idled, a failover without synchronized_standby_slots while the standby lags, beacons abandoned after the failure timeout, and positions with an identity on servers that cannot check them.
  • What is now caught: silent loss. The replaced slot (B1, B2, B3 of the first review), the same-timeline restore (S2), and from round 4: a restore with pg_walinspect installed, a restore after a drift halt, and the pipeline's own slot moved by another client past changes it never delivered. The transaction check's false positive, a publication or row filter changed since the position on 15+, is documented and accepted by allowWALGap; pre-v0.17.5 positions now take a peek on a covered resume too.
  • Data path. The only change to what is reported is a lower idle-advance flush (the cap, now the r of the last received beacon), lifted after the failure timeout. Beacon transactions never become records or change keys. Beacons are v=3; nothing before this PR wrote beacons, so there is no compatibility to keep.
  • Metrics and alerts. INFO replication slot verified ... with proof. WARN codes: slot_identity_unverified, wal_history_unverified, slot_identity_unsupported, slot_beacon_public, slot_beacon_unavailable. ERROR: slot_beacon_unavailable (abandoned). Refusals: slot_replaced, slot_beacon_failed, slot_identity_unprovable, wal_history_changed.
  • Rollback. Revert. v0.17.4 reads v4 positions (additive). Coming forward after a rollback that idled is refused once (documented).

Not verified

  • One run of TestBeacons_EmitErrorAfterCommitKeepsTheAttempt|TestBeaconContent_Verify|TestFindBeacon failed on the test stack early in this round and printed no reason in my filtered output; 6 -v runs, 8 + 8 counted runs and 25 binary runs since did not reproduce it. That was probably the TestFindBeacon expectation fixed above: it failed only when WAL landed between the slot's creation and the stamp. I could not confirm that.
  • The S2 publication/row-filter false positive on 15+ is reasoned from pgoutput and documented; no test changes a publication between runs.
  • pg_upgrade (system identifier, slot migration): reasoned, not run.
  • Failover without synchronized_standby_slots under standby lag: not reproduced.
  • Managed services (RDS, Cloud SQL, Azure).
  • The failover and restore tests are not in CI (make test-failover, make test-restore).
  • TestImport_RetreatedSlotIsRefused and the two pg_log_standby_snapshot() import tests: unchanged, fail on the base too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

@devarismeroxa
devarismeroxa requested a review from a team as a code owner October 21, 2026 22:13
Base automatically changed from fix/allow-wal-gap-warn to main October 21, 2026 22:16
devarispbrown and others added 2 commits October 21, 2026 15:38
Design for detecting a replication slot that was dropped and created again
under the same name: Postgres exposes no identity for a slot incarnation
(measured on 13 to 18), so the connector keeps one. Positions carry a slot
identity (format version 4), the connector writes it into the WAL as a
logical decoding message, never advances an idle slot past the last one it
received, and on resume peeks the slot for it. Covers the research per
version, the alternatives rejected, failure modes, upgrade from positions
without an identity, rollback, the v0.14.x backport, and observability.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…me name (#442)

A slot dropped and created again under the same name exists, is healthy,
and is past the saved position, like a slot the connector advanced while
idle, so a resume on it skipped every change in between without a word
(invariant 3).

Positions now carry a slot identity (format version 4). The connector
writes it into the WAL as a logical decoding message in a transaction of
its own (a beacon, pg_logical_emit_message, synchronous commit), and never
advances an idle slot past the last beacon the stream delivered back.
Advancing for acked records is unchanged. On resume, a slot at or below the
saved position's commit LSN is accepted; a slot past it is accepted only if
a peek finds a beacon with the position's identity, and is otherwise
refused with postgres.source.slot_replaced (logrepl.allowWALGap accepts it
with one DATA LOSS warning). The slot is checked again once
START_REPLICATION holds it, which also closes the check-to-CREATE window
when possible and otherwise taken over once with
postgres.source.slot_identity_unverified. A role that cannot call
pg_logical_emit_message fails the start with
postgres.source.slot_beacon_failed. Postgres 13 and standbys cannot carry
beacons and keep the v0.17.4 check with a warning.

Tests: drop and recreate between runs, recreate after the identity check,
forged and copied beacons, a half-finished create with an injected failure,
the #445 case, snapshot positions, the connector's own idle advance (no
false positive), upgrade from a v0.17.3 position (golden from a real
v0.17.3 run), a revoked pg_logical_emit_message, a server without beacons,
VerifyHeld against a slot swapped in the subscription's own window,
property tests for the position round trip with the identity, and a
Postgres 17+ failover test (test/failover, make test-failover).
TestCombinedIterator_IntactSlot_ResumesAfterIdle stood in for the idle
advance with pg_replication_slot_advance; another client's advance is now
refused, so it becomes TestCombinedIterator_ExternallyAdvancedSlot_Refused.

Beacons are authenticated: the identity is a 256-bit secret key kept only
in the positions, and a beacon carries a nonce, the WAL insert position read
before it, and HMAC-SHA256(key, slot, nonce, position), verified in constant
time and only if its position is at or after the slot's confirmed_flush_lsn.
A role with CONNECT (pg_logical_emit_message is PUBLIC) cannot forge one, and
a genuine beacon copied into a replaced slot is refused. The key is never
logged (logs carry an 8-digit SHA-256 fingerprint); the one debug log that
printed a raw position now prints its length. The connector logs
postgres.source.slot_beacon_public while PUBLIC may call
pg_logical_emit_message. Every slot the connector accepts as a gap
(logrepl.allowWALGap) gets a new key, so an attempt that created and stamped
the slot and failed before saving anything never passes for the original on
retry. The peek's rounds are clamped before the query.

An idle slot now reaches the WAL end one beacon round trip after the last
ack, so a stop right after an ack leaves the slot below that transaction's
commit and the next start decodes it again (the change key skips it, as
after a SIGKILL). TestB1_AC7_RestartHaltTruthfulMessage relied on the
teardown's uncapped advance to start run 2 past the pre-DDL relation
message; its run 1 now waits for the slot to pass the baseline's commit
(PGCHAOS_SETTLE_SLOT) and its assertions are unchanged.

The test helper's ports can be moved with PG_TEST_PORT and
PGBOUNCER_TEST_PORT (also read by test/docker-compose.yml).

Design: docs/design-documents/20261021-slot-identity.md. v0.17.5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarispbrown and others added 2 commits October 21, 2026 16:19
Beacons are written only into the slot this run holds (an xid first, then
active_pid or the pre-stream limit checked inside the beacon's
transaction, with why a slot created after the check cannot decode it).
VerifyHeld runs on every start, with the creation point as the limit for a
slot this run created. Positions name their WAL history (system identifier
and timeline), and a resume on another history is refused with
wal_history_changed; a legacy position warns with wal_history_unverified.
Failing beacons give up after beaconFailureTimeout instead of holding WAL
forever, with the retention math. synchronous_commit=local, a configurable
check timeout, servers that cannot carry beacons, the threat model for
positions that reach destinations, the upgrade release notes, and the
v0.14.x reservation (#449).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…-N5)

B1: a beacon written after the stream lost its walsender vouched for a slot
recreated in between. Emit now takes an xid, checks inside its own
transaction that the slot is held by this run's walsender (or, before the
stream, inactive and not past the limit this run checked or created it
at), and only then writes; a slot created after the check cannot decode
that transaction. TestSlotIdentity_BeaconAfterWalsenderLoss (the review's
repro) failed before and passes.

B2: on a fresh start nothing holds a newly created slot (Postgres 17 shows
it inactive right after CREATE), so a slot replaced mid-snapshot streamed
from the replacement. VerifyHeld now runs on every start but an import,
with the creation point as the limit. TestSlotIdentity_FreshStartSlot-
ReplacedMidSnapshot (the repro) failed before and passes.

B3: positions carry system_identifier and timeline (IDENTIFY_SYSTEM). A
resume is refused with postgres.source.wal_history_changed on another
cluster, an earlier timeline, a timeline that left the position's before
its bound (TIMELINE_HISTORY), or WAL behind the position; allowWALGap
accepts it with a "WAL HISTORY CHANGE ACCEPTED" warning and a pg_upgrade
runbook. A legacy position learns its history and logs
postgres.source.wal_history_unverified once.

S1: beacons that keep failing for logrepl.slotIdentity.beaconFailureTimeout
(5m) are abandoned: the idle advance is uncapped again, an error with
retained_wal_bytes is logged, and the next start is refused unless a record
was saved. Beacons commit with synchronous_commit=local, and a delivered
beacon counts even when its Emit returned an error. S2: the check's peek
timeout is logrepl.slotIdentity.checkTimeout (10m). S3/S4/S6: README and
release-note wording, a pre-upgrade privilege query, unproven_wal_bytes in
slot_identity_unverified. N1: no raw positions in errors. N4: an unsettled
AC7 chaos variant. N5: a position with an identity is never accepted
unproven on a server that cannot carry beacons (slot_identity_unprovable);
such servers write positions without one. The README description lives in
connector.yaml (readmegen), so it now carries the section.

#445's TestCDCIterator_AllowWALGap_WarnsWhenCreateSucceedsThenFails now
expects the retry to warn again: the slot the failed attempt created is not
the position's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarispbrown and others added 6 commits October 21, 2026 16:33
…story_unverified test

The legacy start takes the slot over with a new key, after which the v4
position's key proves nothing and its start is (correctly) refused. Check
the v4 position first, and that the legacy warning is logged at warn level.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…acons

On Postgres 13 (and standbys) positions carry no slot identity since the
#447 review (N5), so the drift marker format test checks the identity only
where beacons are supported (and the WAL history everywhere), and #445's
create-then-fail test expects the pre-#442 single warning there.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
pg_control_checkpoint still shows the standby's last restartpoint right
after a promotion, which is before the switch, so the "position past the
switch point" case built its position below it and was not refused. Read
the switch point from the new timeline's history file instead, and check
that the saved position was below it (why the real resume passes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
The position's transaction (xid, commit time) closes the same-timeline
restore case for covered resumes, with pg_walinspect when installed and the
Postgres 15+ false positive documented. A standby never proves an identity.
Abandoned beacons are retried and re-arm, so the outage heals itself. The
guard identifies the walsender by PID and backend_start, and a start whose
walsender cannot be identified fails. The xid argument and the committed-
but-errored beacon are now tested, so they leave "Not verified".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
S2: a same-timeline restore (a volume snapshot) whose new WAL had passed
the position's bound resumed "covered" and skipped the rows written after
it. v4 positions now carry tx_xid and tx_commit_time from BEGIN, and a
covered resume checks that the commit LSN still names that transaction:
pg_walinspect's commit record when installed and usable, otherwise the first
transaction the slot decodes from the bound (same commit LSN, xid and commit
time; a restored server reuses the xids too). Otherwise wal_history_changed.
test/restore/run.sh snapshots and restores a data volume (make
test-restore): refused on 14, 17 and 18, resumed before.

S3: legacy positions get the "WAL behind the position" check too.

S1: a standby never proves an identity (a slot it decodes is never the
primary's); refused with slot_identity_unprovable. Reproduced on 17 with a
standby-local slot, now refused (in test/failover/run.sh).

S4: abandoned beacons keep being written; while one is on its way the idle
advance waits at its start, and the first one delivered past the reported
flush re-arms the identity. Wording fixed: a restart is refused until then,
and the outage heals itself.

S5: tests for the xid before the guard (a slot created while the beacon's
transaction is open does not decode it, via a seam after the guard) and for
a beacon that committed although Emit errored (unit, and a seam after the
commit with a running stream).

N1: the guard joins pg_stat_activity and compares the walsender's
backend_start recorded at the stream start, so a reused PID is refused. N2:
a stream start whose walsender cannot be identified fails with
slot_beacon_failed instead of skipping the holder.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…t markers

The chaos suite caught the transaction check refusing resumes from a drift
marker: its key can be one below its change's (#338), which names no
transaction. Positions now carry tx_xid and tx_commit_time only when the
key's commit LSN is the transaction's, and a resume from a marker skips the
check (TestCDCPosition_TxIdentityOnlyForRealCommits).

With the guard, a stamp at open is safe on every start (a slot swapped in
after the check is past the limit and refused), so the deferred-stamp path
for existing slots is gone; with allowWALGap a refused stamp waits for
VerifyHeld, which warns, rotates the key and stamps. Beacons are recognised
only as transactional messages, as Emit writes them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarispbrown and others added 2 commits October 21, 2026 18:33
…he stream-holder test

slotIDLegacy now also drops tx_xid and tx_commit_time, as a real v0.17.3
position has neither; with them kept, the legacy current-WAL test was
refused by the transaction check instead of the check it is about (the
mutation that removes that check survived). The N2 test runs the stream
under a timeout, so a regression fails it instead of hanging the package.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
CI on bc5daa0 failed SRC-01: the acceptance test stopped the source during
the snapshot-to-CDC handoff, the stream-holder query (#447 N1) ran on the
canceled context, and the start failed with slot_beacon_failed, which the
graceful stop reported as a plugin error. Any start step that queries the
server had the same race. A start step that fails after the run's context
ended now returns the context error with the step's error in the chain
(TestSubscription_StartCanceledReportsTheContextError, fails without it).

The walsender-unknown error no longer advises a GRANT: a role always sees
its own backends, so that failure is the connection or the server.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarispbrown and others added 2 commits October 21, 2026 20:33
Brings in #446 (record keys and row data out of trace and debug logs). The
one conflict is the debug log for an unparsable position in combined.go:
both sides log only its length; #446's field name is kept and the comment
names the slot identity key as a second reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
B3: the pipeline's own slot, consumed by another client past a transaction
the pipeline had received but not acked, was accepted because it still
decoded the stamp written after that transaction. Beacons (v3) now certify
r, a WAL position before which the run needed nothing more: serverWALEnd
for a beacon requested while every record is acked, the checked slot
position for the stamp. The idle advance stops at the r of the last
received beacon, and a slot past the position is accepted only on a beacon
written after its confirmed_flush_lsn whose r covers it. A slot moved over
delivered changes only is still accepted; one moved past undelivered ones
is refused with slot_replaced.

B1, S1: the covered-resume transaction check no longer reads
pg_walinspect. pg_get_wal_record_info returns the next record for an LSN
inside one and its commit time is only in the description (a restored
history that reused the xid was accepted), and COMMIT PREPARED has xid 0
there (a prepared transaction's position was refused). The check always
reads the slot: same commit LSN, xid and commit time.

B2: a drift marker's position names the transaction it halted before
(its key is one below that commit), and the check looks for it there. A
position without an xid (pre-v0.17.5) must at least have a commit at its
commit LSN.

Tests, each failing on 85da517: the walinspect and drift scenarios of
test/restore/run.sh, ConsumedByAnotherClientBelowTheLastBeacon_Refused,
PreparedTransactionPosition_ResumesCovered (test/conf.d enables prepared
transactions). Plus TestSameTransaction, the marker and v3 beacon unit
tests, and the legacy no-commit refusal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
devarismeroxa added a commit that referenced this pull request Oct 22, 2026
…tore) (#451)

* ci: add the external-state chaos suite (slot, privileges, network, restore)

Adds test/externalstate, a case manifest, a fail-closed runner, `make
test-external-state` and an external-state workflow (PRs touching source/**,
push to main, nightly). Cases that other packages already prove are cited and
run, not copied; cases blocked on #447 are pending and flip to required
automatically-by-failure once the blocking test exists. The volume snapshot
restore case is a known gap: it shows silent loss and stays red on purpose.

No data-path code changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

* ci(external-state): own compose projects and ports, pinned refusal codes, SILENT evidence for known gaps

The base cases no longer use port 5433 or take down the shared stack: the base
stack runs as project pgextbase on 56153/56154 (test/docker-compose.yml and
test/helper.go read CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT).
requireLoudOrDelivered takes the refusal codes acceptable for the case; a case
that lists none must deliver. A known-gap case must fail with SILENT loss
output. The workflow gets permissions: contents: read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

* ci(external-state): a loud read error is a refusal, not silent loss; every published port is checked against the default stacks

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

---------

Co-authored-by: DeVaris Brown <devaris@devaris.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
devarismeroxa added a commit that referenced this pull request Oct 22, 2026
* ci: guard mutation gate for source/logrepl

Mark the slot, import and delete-key refusals with `// guard: <name>`.
tools/guardmut deletes each in turn and fails the job when the package tests
stay green. The guards added by #447 are marked in a follow-up after it merges.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

* ci(guardmut): own compose project, tests= on every marker, guard list, invalidation guards run alone

make guardmut runs as project pgguardmut on ports 56170/56171 and tears down
only that project. test/docker-compose.yml and test/helper.go read
CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT. The two 300 s mutants
get tests= and now die in seconds. A name list and a shrink-only baseline are
enforced in the workflow; actions need permissions: contents: read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

* ci(guardmut): strip multi-line and unclosed HTML comments before reading the waiver; detector covers Makefile and tools/go.mod

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH

---------

Co-authored-by: DeVaris Brown <devaris@devaris.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
devarispbrown and others added 6 commits October 22, 2026 09:36
The security review of 453669f asked whether a keepalive's WAL end can be
past committed changes the walsender has not sent yet, which would let the
idle advance (on main already) and a beacon's r (#447) skip them.

It cannot, measured on 13, 14, 15, 16, 17 and 18: while it sends a large
transaction, the walsender asks for a reply after wal_sender_timeout/2,
and the keepalive's WAL end is sentPtr, the start of the commit record
being replayed, which is that transaction's own commit LSN. A slot
confirmed at a commit LSN decodes that transaction again. So everything
that commits before a keepalive's WAL end was sent, and with every
received record acked, reporting it loses nothing.

TestWalsender_KeepaliveInsideATransactionNeverPassesItsCommit pins that
with a raw client: a 100000-row transaction, a keepalive inside it at
its commit LSN, that LSN reported as flushed, a crash, and the slot decodes
all 100000 rows again. TestSlotIdentity_LargeTransactionInFlight_SlotNeverPassesIt
runs the connector through such a transaction (a short wal_sender_timeout),
checks that neither the slot nor the beacon cap passes its commit while it
is in flight, stops in the middle, and gets the rest after the restart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Brings in #451 (external-state chaos suite) and #452 (guard mutation gate).
Conflicts: the test port override, which both sides added under different
names. main's CONDUIT_PG_TEST_PORT and CONDUIT_PG_TEST_BOUNCER_PORT are kept
(test/helper.go, test/docker-compose.yml); the Makefile keeps both sides'
targets.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
The external-state suite (#451) failed on this PR by design: three pending
cases name tests #447 adds (slot recreated under the same name, the
pg_logical_emit_message privilege revoked, failover), and the
snapshot-restore known gap now fails loudly instead of losing rows. The
three are required now, and snapshot-restore is required with
postgres.source.wal_history_changed as an accepted refusal (it was refused
with it; the case listed only slot_wal_gap and slot_invalidated). All ten
cases pass locally (test/externalstate/run.sh).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
Brings in #450. Merges clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
…-S3, NITs)

S1: a transaction of at least findBeaconMaxRows rows between the slot's
confirmed_flush_lsn and what the check looks for made it give up for good:
a change limit stops only at a transaction boundary, so every round
returned that transaction whole until ErrBeaconSearchExhausted, an uncoded
refusal on every retry that allowWALGap did not help. FindBeacon now reads
to the end of the WAL without a limit after the round at the clamp, and
the covered-resume transaction check (TxCommittingAt, was FirstTxFrom) is
bounded by upto_lsn = the commit LSN + 1 instead of a change count. A
check that still cannot read the slot (timeout, peek error) is refused with
postgres.source.slot_identity_check_failed and a fix; allowWALGap accepts
the slot unproven. Tests at the review's scale (a 10000-row transaction,
the clamp at 4096) fail on 76fdea1.

S2: guard markers on the idle-advance beacon cap, verifyBeacon's
r >= confirmed, the beacon's xid-first step (assignXid, a call of its own
so the gate can remove it), the walsender check in checkHeld (PID and
backend_start as one guard: alone the PID check is an equivalent mutant),
VerifyHeld's limit and the covered-transaction check. guardmut: all
killed.

S3: make test runs go test with -timeout 20m; source/logrepl takes about
8 minutes on CI.

NITs: a position whose bound is the timeline switch point is refused
(leftTimelineBy, <=); the slot_replaced message names a rollback and
re-upgrade and abandoned beacons as causes that lose nothing; the
external-state snapshot restore pins wal_history_changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SvPfP7eL7SskkoptXvAYCH
… allowWALGap (#447)

A transient failure of the identity check (FindBeacon) or the
covered-resume transaction check (TxCommittingAt) — a query error, a
timeout, a connection drop — is not a verdict: the check did not run, so
the slot could still be the position's. Before this change the code
treated such a failure as "unproven" and, under logrepl.allowWALGap,
accepted it like a verified slot_replaced and resumed, silently dropping
the identity guarantee.

Now a transient failure always fails loudly with
postgres.source.slot_identity_check_failed, whatever allowWALGap says:
the flag accepts only a slot the check proved is not the position's,
never a slot the check could not read. The start is refused and retried
by the engine; the verified recreated-slot opt-in is unchanged.

Adds regression tests that inject a transient error at each check and
assert the coded refusal (they fail without the fix), updates the
checkTimeout test that asserted the old accept-unproven behaviour, and
marks both refusals as guards (logrepl-transient-check-refused,
logrepl-covered-transient-refused) so the mutation gate covers them.
@devarismeroxa

Copy link
Copy Markdown
Contributor Author

Sign-off @ e1ffca6

@devarismeroxa

Copy link
Copy Markdown
Contributor Author

Review: approve @ e1ffca6

@devarismeroxa
devarismeroxa merged commit 15a0865 into main Oct 24, 2026
11 of 13 checks passed
@devarismeroxa
devarismeroxa deleted the fix/slot-identity branch October 24, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

source: detect a replication slot dropped and recreated under the same name between runs

2 participants