Add skip-empty-xacts option#293
Open
adrijshikhar wants to merge 1 commit into
Open
Conversation
Logical decoding emits a BEGIN/COMMIT pair even for transactions that contain no decodable changes (DDL, VACUUM FULL, REFRESH MATERIALIZED VIEW) or whose changes were all filtered out (filter-tables, add-tables, filter-msg-prefixes). Consumers are flooded with empty changesets; see issue eulerto#106 for reports of thousands of them. Add a skip-empty-xacts option (default false, preserving the current behavior). When enabled, the BEGIN output is deferred until the first change that survives filtering is emitted and the COMMIT output is suppressed if no change was emitted. Transactional messages count as changes; non-transactional messages are not affected. On PostgreSQL 15 and later, skipped transactions are reported to the progress machinery (skipped_xact) so the walsender keeps sending keepalives during long runs of skipped transactions. This is the same approach used by test_decoding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #106. Supersedes #248, which can no longer be reopened (the fork that submitted it was deleted).
Logical decoding emits a BEGIN/COMMIT pair even for transactions that contain no decodable changes (DDL,
VACUUM FULL,REFRESH MATERIALIZED VIEW) or whose changes were all filtered out (filter-tables,add-tables,filter-msg-prefixes). Consumers are flooded with empty changesets; #106 reports thousands of them.This adds a
skip-empty-xactsoption (defaultfalse, existing behavior unchanged). Same approach astest_decoding.How it works
JsonDecodingData— no per-transaction allocation.add-tables,filter-tables, actions) is emitted, and the COMMIT output is suppressed if no change was emitted.skipped_xacttoOutputPluginUpdateProgress(16+) /update_replication_progress(15), resolving the twoXXXcomments inpg_decode_commit_txn, so the walsender keeps sending keepalives during long runs of skipped transactions.write-in-chunks(the change callback prepares its chunk after the deferred BEGIN chunk is written).Tests
Regression test built from the reproduction cases reported in #106:
add-tablesfiltering, partitioned tables, DDL-only transactions,TRUNCATE,VACUUM FULL,REFRESH MATERIALIZED VIEW, plus transactional / non-transactional / prefix-filtered messages. Covers format versions 1 and 2 andwrite-in-chunks.VACUUM FULLandREFRESH MATERIALIZED VIEWassert on counts rather than raw output because their transaction counts vary across server versions.make installcheck: all 31 tests pass (PostgreSQL 17).