Skip to content

pulsar: support debezium protocol#5054

Open
HGHNice wants to merge 2 commits into
pingcap:masterfrom
HGHNice:pulsar/support-debezium-protocol
Open

pulsar: support debezium protocol#5054
HGHNice wants to merge 2 commits into
pingcap:masterfrom
HGHNice:pulsar/support-debezium-protocol

Conversation

@HGHNice
Copy link
Copy Markdown

@HGHNice HGHNice commented May 15, 2026

What problem does this PR solve?

Issue Number: close #5056

The Pulsar sink currently only supports canal-json. Users who consume
TiCDC events via Pulsar with Debezium-compatible consumers (e.g. Flink CDC)
have no way to use the standard Debezium message format.

What is changed and how it works?

  • Extend IsPulsarSupportedProtocols() in pkg/config/sink_protocol.go
    to include ProtocolDebezium.
  • Update the error message in downstreamadapter/sink/pulsar/helper.go
    to reflect the expanded protocol list.
  • The Debezium codec is already fully implemented and shared with the Kafka
    sink via the common codec builder, so no additional encoding logic is needed.
  • Add unit test TestIsPulsarSupportedProtocols.

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?

No. Existing canal-json behavior is unchanged.

Do you need to update user documentation, design documentation or monitoring documentation?

The Pulsar sink docs should note that debezium is now a valid protocol value.

Release note

The Pulsar sink now supports the `debezium` protocol. Set `protocol=debezium`
in the Pulsar sink URI to produce Debezium-format change events.


## Summary by CodeRabbit

## Release Notes

* **New Features**
  * Pulsar sink now supports debezium protocol in addition to canal-json protocol.

* **Tests**
  * Added validation tests for Pulsar sink protocol compatibility.

<!-- review_stack_entry_start -->

[![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/pingcap/ticdc/pull/5054)

<!-- review_stack_entry_end -->

Extend IsPulsarSupportedProtocols to include ProtocolDebezium so that
Pulsar changefeeds can use the debezium message format. The debezium
codec is already implemented and shared with the Kafka sink via the
common codec builder, so no additional encoding logic is needed.
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 15, 2026
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented May 15, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign flowbehappy for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. labels May 15, 2026
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented May 15, 2026

Hi @HGHNice. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added the needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. label May 15, 2026
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented May 15, 2026

Welcome @HGHNice!

It looks like this is your first PR to pingcap/ticdc 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/ticdc. 😃

@ti-chi-bot ti-chi-bot Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 15, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 15, 2026

📝 Walkthrough

Walkthrough

This PR extends Pulsar sink protocol support from canal-json only to include debezium as well. The validation logic is updated with corresponding test coverage, and user-facing error messages and comments in the Pulsar helper are refreshed to reflect the expanded protocol support.

Changes

Pulsar Debezium Protocol Support

Layer / File(s) Summary
Protocol validation logic and test
pkg/config/sink_protocol.go, pkg/config/sink_protocol_test.go
IsPulsarSupportedProtocols now returns true for both ProtocolCanalJSON and ProtocolDebezium. A new parallel test case validates the function returns true for those two protocols and false for ProtocolAvro, ProtocolOpen, ProtocolSimple, and ProtocolCanal.
Pulsar sink error message and documentation
downstreamadapter/sink/pulsar/helper.go
The unsupported protocol error message is updated to list [canal-json, debezium] instead of [canal-json], and an obsolete inline comment claiming Pulsar only supports canal-json is removed.

🎯 1 (Trivial) | ⏱️ ~3 minutes

🐰 A rabbit hops with glee, both canals and bees,
No longer just JSON, now Debezium flows free!
Comments updated, protocols sound true,
Tests verify both—good work through and through! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding Debezium protocol support to the Pulsar sink.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description follows the required template with all essential sections properly completed including issue number, problem statement, changes explanation, test verification, compatibility assessment, and release notes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@pingcap-cla-assistant
Copy link
Copy Markdown

pingcap-cla-assistant Bot commented May 15, 2026

CLA assistant check
All committers have signed the CLA.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the Debezium protocol to the Pulsar sink by updating the protocol validation logic and error messages. Feedback includes a recommendation to replace a hardcoded boolean with a dynamic protocol check in the event router, a request for additional test coverage in existing protocol parsing and string conversion tests, and a minor grammatical correction in a user-facing error message.

Comment thread downstreamadapter/sink/pulsar/helper.go
Comment thread pkg/config/sink_protocol_test.go
Comment thread downstreamadapter/sink/pulsar/helper.go
- Make isAvro parameter dynamic in NewEventRouter call
- Add debezium and simple cases to protocol parsing and string tests
@ti-chi-bot ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/needs-triage-completed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pulsar: support debezium protocol

1 participant