Add batch processing documentation for Kafka Inbound Endpoint

[chathuranga-jayanath-99]

Purpose

Documents the batch processing feature for the Kafka Inbound Endpoint.

• Added batch.processing.enabled and kafka.dlq.topic parameters to the configuration reference table.
• Added a new Batch mediation section covering:
  • How to enable batch mode
  • Message payload format for JSON, XML, and plain text content types
  • Poison pill handling and DLQ forwarding (with DLQ header reference table)
  • Manual offset control and retry behavior differences in batch mode

Please refer below PDF for the updated page.
Kafka Inbound Endpoint Reference - WSO2 Micro Integrator Documentation 4.4.0.pdf

Screenshots of the updated content is below as well.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d2596f45-e14b-4b83-8d98-879da84ab6f9

📥 Commits

Reviewing files that changed from the base of the PR and between fb940fa and de3ed4b.

📒 Files selected for processing (1)

• en/docs/reference/connectors/kafka-connector/kafka-inbound-endpoint-config.md

---

📝 Walkthrough

Summary

This pull request adds comprehensive documentation for the batch processing feature of the Kafka Inbound Endpoint in the configuration reference guide.

Changes

Documentation Updates:

• Added two new configuration parameters to the reference table:

  • batch.processing.enabled: Enables batch mode, which bundles all valid records from a single Kafka poll into one injected message instead of processing records individually
  • kafka.dlq.topic: Forwards undeserializable records (poison pills) to a Dead Letter Queue topic with provenance metadata

• New "Batch mediation" section covering:

  • Message payload formats based on configured content type:
    • JSON: Records combined into a JSON array
    • XML: Records wrapped in a <messages> root element
    • Plain text: Records wrapped in <text> elements under <messages> root with escaped special characters
  • Poison pill handling: Invalid records are skipped individually from batch payloads with warnings logged; optional DLQ forwarding includes provenance headers (original topic, partition, offset, timestamp, and exception details)
  • Manual offset control and retry behavior: Offsets are committed at batch boundaries; on failure with remaining retries, the consumer seeks back to the first valid record's offset per partition; once retries are exhausted, offsets are committed and an error is injected

Impact

Provides users with detailed guidance on configuring and understanding batch processing behavior, including edge cases around error handling and offset management, enabling more reliable batch ingestion from Kafka topics.

Walkthrough

This documentation update adds two new optional parameters—batch.processing.enabled and kafka.dlq.topic—to the Kafka Inbound Endpoint configuration reference table. It also introduces a new Batch mediation section under the manual offset control guidance. The section documents injected message payload formats per contentType (JSON array, XML under <messages>, plaintext under <messages><text>), poison pill handling behavior (individual skipping with warnings and optional DLQ forwarding with provenance headers), and batch-specific offset control and retry semantics including seek-back behavior on failure and the RETRY_EXHAUSTED error code 700511 injected into onError when retries are exhausted.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers the Purpose section with clear documentation of the batch processing feature and linked reference materials, but omits most other required template sections (Goals, Approach, User stories, Release note, etc.).	Complete the missing template sections including Goals, Approach, User stories, Release note, Documentation, Training, Certification, Marketing, Automation tests, Security checks, Samples, Related PRs, Migrations, Test environment, and Learning as applicable.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically summarizes the main change: adding documentation for the batch processing feature in Kafka Inbound Endpoint.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chathuranga-jayanath-99]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[chanikag]

Suggested change

	The shape of the injected message depends on the configured `contentType`.
	The content type of the injected message depends on the configured `contentType`.

[chanikag]

Let's define these as tabs.

[chanikag]

Let's add a note about the all the messages are built to the message context, and can utilize memory, so fine tune the batch size accordingly.