Feature: Kafka module - di.kafka

[alowrydi]

Kafka consumer/producer module, di.kafka

Extracts code/common/kafka.q from TorQ and packages it as a standalone kdb-x module. Wraps the kafkaq native C shared library to provide Kafka consumer initialisation, producer initialisation, topic subscription, and message publishing, with incoming messages delivered to a configurable kupd callback.

Trello ticket - https://trello.com/c/eDFHYhSx/103-kdb-x-kafka-module

Files created

File	Description
di/kafka/init.q	Loads kafka.q, defines export of 8 functions
di/kafka/kafka.q	Full implementation - module state, native function stubs, public API, init
di/kafka/kafka.md	Full module documentation - see below
di/kafka/test.csv	25 k4unit tests covering log dependency validation, disabled-mode stubs, missing-library handling, and config behaviour

How to test

k4unit:use`di.k4unit
k4unit.moduletest`di.kafka

Unit test example:

q)k4unit.moduletest`di.kafka;
2026.06.16T13:40:01.201 start
2026.06.16T13:40:01.201 :/home/alowry/bin/local-kdbx-modules/di/kafka/test.csv 25 test(s)
2026.06.16T13:40:01.202 end
Test results:
action ms bytes lang code                                                                                                         repeat file                                                   msx bytesx ok okms okbytes valid timestamp
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
true   0  0     q    .[kafka.init;(()!();()!());{x}] like "*log dependency*"                                                      1      :/home/alowry/bin/local-kdbx-modules/di/kafka/test.csv 0   0      1  1    1       1     2026.06.16T13:40:01.201
true   0  0     q    .[kafka.init;(()!();(::));{x}] like "*log dependency*"                                                       1      :/home/alowry/bin/local-kdbx-modules/di/kafka/test.csv 0   0      1  1    1       1     2026.06.16T13:40:01.201
true   0  0     q    .[kafka.init;(()!();42);{x}] like "*log dependency*"                                                         1      :/home/alowry/bin/local-kdbx-modules/di/kafka/test.csv 0   0      1  1    1       1     2026.06.16T13:40:01.201
. .
q)exec sum ok from k4unit.getresults[] // 25/25 pass
25i

Coverage includes all invalid log-dependency shapes, all native function stub error messages, the enabled:0b path proven via a capturing mock logger, the missing-library path, setkupd, and multiple consecutive init calls.

Design decisions

1. enabled variable matches TorQ's platform-detection default - enabled defaults to `.z.o in `l64`, matching TorQ's own kafka.q exactly rather than a hardcoded value. On a non-l64 host the module loads disabled unless explicitly overridden.

2. Missing library will log and continue, does not throw - If enabled:1b and the library file is not found, init logs an error via the injected logger and leaves all native functions as stubs, rather than throwing. This matches TorQ's original log-and-degrade behaviour - a missing native library should not halt the calling process.

3. Injected logging dependency - init now takes [config;deps] and requires a log dependency with info/warn/error functions, throwing a clear, module-prefixed error if it is missing or incomplete. This logging support consistent with how the other new modules in this project are being built.

4. Default kupd logs through the injected logger, not stdout - TorQ's original default kupd writes raw bytes to stdout. Since this module now requires logging to be wired up regardless, the default handler logs via the injected logger instead, so there is one consistent output path rather than two.

5. Native function stubs in disabled mode - TorQ leaves the native functions undefined when disabled, so calling them produces a generic, uninformative error. This module instead gives each one a stub that throws 'kafka not enabled, a clear, distinguishable error rather than an undefined-variable error.

Checklist

• 25/25 k4unit tests passing
• Follows consistency.md and style.md
• kafka.md documents all exported functions, config, disabled-mode behaviour, usage example and TorQ migration table

Documentation

See kafka.md for full reference including config table, disabled-mode behaviour, exported function documentation, usage example, global side effect, and TorQ migration pattern table.