Parallel consumer in Go

[Alvin Rizki (cikupin)]

We have kafka client library for Go here https://github.com/confluentinc/confluent-kafka-go.

Can we achieve this parallel consumer using that Go client?

[Antony Stubbs (astubbs)]

Alvin Rizki (@cikupin) - not with confluent-kafka-go on its own. The client gives you records and offsets; what makes this library work is the part above the client - the per-key shards, the retry queue, and the encoded map of completed offsets that goes into each commit so nothing is replayed - and none of that exists in the Go client.

The maintained fork's answer for Go is most likely in-process: the engine runs inside your Go binary as a GraalVM shared library, called over FFI, with no gRPC and no JVM to install, so your Go function gets key-ordered concurrency, retries and the offset accounting from the same engine the Java library runs (astubbs#340, which carries the Go, Python and C path). The alternative is a sidecar your Go program talks to over a frozen gRPC contract - the contract (astubbs#383), the sidecar process (astubbs#384), one conformance suite every client must pass (astubbs#387) and the per-language clients (astubbs#390). All of it is on open pull requests, not in a release, and which route Go gets first is not decided. The tracking issue is astubbs#242, this tracker's #154.

So: nothing you can use today; the "What comes next" section of v0.6.0.0 lists it among the open pull requests that follow the release. If Go is your case, a comment on astubbs#242 saying so is useful - it is what decides which client gets finished first.

The coordinates changed with the fork - on Maven Central it is now:

<dependency>
    <groupId>bz.stub.parallelconsumer</groupId>
    <artifactId>parallel-consumer-core</artifactId>
    <version>0.6.0.0</version>
</dependency>

(parallel-consumer-vertx, parallel-consumer-reactor and parallel-consumer-mutiny likewise.) The Java package is bz.stub.parallelconsumer too, so imports change with it - one command rewrites them (GNU sed; on macOS use sed -i ''):

find . -name '*.java' -exec sed -i 's/io\.confluent\.parallelconsumer/bz.stub.parallelconsumer/g' {} +