Long Running Process

[renatojava]

If you had a long-running process you get the error:

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

How to solve this?

[CodeSmell]

Kafka is not really designed to handle long running processes within the poll loop. When a consumer within the consumer group is not responding in a pre-defined amount of time, the group coordinator will remove that consumer from the group and rebalance the partitions among the remaining consumers.

That appears to be what is happening from the error message:

group has already rebalanced and assigned the partitions to another member.

This feature allows Kafka to seamlessly make sure that all of the partitions on a topic are being consumed within the group.

How to solve depends on your application needs, what you have configured now, and how long is your process.

You can do a couple different things (some of which are suggested in the error message):

• increase the max.poll.interval.ms interval
• decrease the max.poll.records
• change the architecture so that long running processes are not in the poll loop

It it likely you will want to explore that last option.
That might mean something like this:

Camel Route A

• Read message(s) from Kafka topic
• Publish them to JMS queue
• if no error publishing to JMS queue, commit offset

Camel Route B

• Read message from JMS queue
• Perform long running process
• if no errors then acknowledge/commit the message