Tabellarium

Tabellarium is a resilient Logback appender that ships structured log events to Apache Kafka. Named after the Roman letter-carrier, it never blocks the sender: per-topic-class circuit breakers stop hammering a broken route, mandatory overrides pin the strictest producer-side delivery settings for audit-class topics (acks=all, idempotence), and a fallback appender catches what cannot be shipped. Delivery is best-effort transport with visible loss - see Delivery guarantees for the exact scope.

Documentation: inqudium.github.io/tabellarium — configuration guide, metrics overview, and Grafana dashboards.

Features

Delivery

• The sender is never made to wait. The hot path never blocks (UnsynchronizedAppenderBase - no synchronized doAppend, no waits, no I/O) and never calls producer.send itself: the caller only routes, encodes and enriches, then hands the record to a bounded per-topic-class send queue in O(1). A dedicated worker per class performs the send, so a stalled broker neither pins carrier threads on virtual threads nor stalls a Reactor event loop - and a stuck AUDIT route never delays TECHNICAL delivery. max.block.ms is additionally capped per class (500 ms; 200 ms for PERFORMANCE), bounding each worker's worst case.
• Undeliverable events take the side road, not the ditch. An optional fallback appender receives what Kafka refuses, fed through a bounded queue and its own worker thread — the Kafka I/O thread is never blocked by a slow file appender, and dropped events are counted rather than silently lost.

Circuit breaking

• A broken route is not hammered. One Resilience4j circuit breaker per topic class, so a stuck audit broker never throttles technical logging. Deterministic payload errors (RecordTooLargeException and friends) are deliberately excluded from the failure rate — a buggy log statement must not silence a healthy pipeline.
• Recovery probes are spread over time. In half-open state a throttle admits one probe per interval instead of letting a high-volume logger burn every permitted call in microseconds.

Routing & service levels

• Quality of service per log stream. Each topic class carries its own producer tuning and its own circuit breaker: AUDIT buys producer-side durability (acks=all, idempotence, retries), PERFORMANCE buys throughput (larger batches, longer linger, tighter block budget), with FUNCTIONAL and TECHNICAL in between. Compliance-graded classes additionally enforce their producer settings over any conflicting operator value — and report every override at startup instead of applying it silently.
• Marker-based routing. <mapping> elements route by SLF4J marker to their own topic and class; one producer, breaker and client.id per active class, and none for dormant ones.

Traceability

• Every record says where it came from. meta.component, meta.cmdbId, meta.environment and meta.agent.* ride on every record as headers, encoded once at startup rather than per event — so a consumer can filter by service, instance or stage without parsing the payload.
• Trace affinity, attributable producers. The record key is the MDC trace id, so the records of one trace share a partition and keep their relative order; each producer announces itself to the broker as tabellarium-<component>-<class>, so connections, quotas and kafka.producer.* metrics name the service and its service level instead of a generic producer-N.

Operations

• Misconfiguration fails at startup, not per event. Blank identity fields, invalid Kafka topic names, unknown topic classes, duplicate markers and idempotence-incompatible tuning all abort start() with a named error.
• Metrics are opt-in and complete. Counters, timers and queue gauges for a Micrometer registry, plus Grafana dashboards and a Spring binding helper — and nothing at all until you bind a registry.

Footprint & security

• A lean dependency tree. Micrometer, Spring and the Logstash encoder are all optional; consumers who do not want them do not get them.
• Security-conscious defaults. Diagnostics never echo your producer configuration, compliance-graded topics warn when shipped over cleartext, the partitioning key is length-bounded, and the appender ignores its own producer's log output instead of feeding it back.

The name

Tabellarium is named after the tabellarius, the letter-carrier of the Roman world. His load was the tabella — a wax tablet, a small written record of fixed form — and his craft was not writing but delivery: taking the tablet off the sender's hands at the door, and getting it to its destination even when the usual road was closed.

That is precisely this project's job, transposed to logging. Every log event is a tabella — one encoded, structured record — and the appender is the carrier that accepts it at the moment of logging and delivers it to Kafka. The craft lies in how it carries: the sender is never made to wait (the hot path never blocks, and the delivery outcome is reported asynchronously through the send callback), a broken road is not hammered (a circuit breaker per topic class suspends dispatch while the route is down), and an undeliverable tablet takes the side road rather than the ditch (the fallback appender). Dispatches of rank travel under stricter carriage rules the sender cannot waive: the mandatory overrides of the AUDIT class — acks=all, idempotence — are the seal the carrier applies whatever the configuration asked for. The name deliberately refers to the carrier, not the road: Kafka is route and destination, the broker infrastructure someone else operates; Tabellarium is only ever the one carrying.

The form follows the naming of chemical elements. Real elements take their names from places, figures and ideas — rhenium after the Rhine, promethium after a myth — and tabellarius + the element suffix -ium yields a plausible entry in that series. This places Tabellarium in the same fictional periodic table as Inqudium (the eu.inqudium group it is published under) and Limesium: an element-style name for one well-defined capability, here the element of reliable carriage. The two neighbours even share a story — Limesium is the watchtower that records each crossing at the service's own boundary; Tabellarium is the courier who carries the records away.

Installation

Releases are published to Maven Central (GPG-signed, with sources, javadoc, and a CycloneDX SBOM) — no repository configuration needed:

<dependency>
    <groupId>eu.inqudium</groupId>
    <artifactId>tabellarium</artifactId>
    <version>1.0.0</version>
</dependency>

Mirrors: the GitHub Packages Maven registry (https://maven.pkg.github.com/Inqudium/tabellarium; needs a token with read:packages even for public packages), and the jar plus SBOM attached to each GitHub release.

Quick start

1. Declare the appender in your logback-spring.xml:

   <appender name="KAFKA" class="eu.inqudium.tabellarium.KafkaAppender">

2. Fill in the required elements — see Configuration and the complete example at docs/config/example-logback-spring.xml.

3. Optionally add a fallback appender (recommended) — see Resilience below.

4. Deploy.

Configuration

A complete configuration example lives at docs/config/example-logback-spring.xml. Reference for every supported element:

Element	Required	Type	Notes
<encoder>	Yes	nested	Any standard Logback encoder. LogstashEncoder recommended (and optional, so opt-in) — JSON escaping also prevents log forging via attacker-influenced message text; any JSON encoder does.
<kafkaProducerProperties>	Yes	text	Multi-line key=value Kafka producer config. Comments with # supported.
<topicMapping>	Yes	nested	<defaultTopic> plus any number of <mapping> elements (marker → topic → topic class) — see Topic routing.
<environment>	Yes	string	Deployment environment (e.g. prod, staging).
<component>	Yes	string	Service component identifier (typically ${spring.application.name}).
<cmdbId>	Yes	string	CMDB identifier of the deploying instance.
<debug>	No	boolean	Startup diagnostics only: logs active topic classes, fallback configuration, and the generated producer settings (derived client.id, applied class overrides) to Logback's status manager. No per-event effect.
<sendQueueCapacity>	No	int	Capacity of each per-topic-class send queue (default 1024). Overflow diverts to the fallback (reason queue.full) instead of blocking.
<includeCallerData>	No	boolean	Captures caller data on the logging thread before the asynchronous hand-off (default false); only relevant when a fallback layout uses %caller.
<appender-ref ref="..."/>	No	ref	Single fallback appender — see Resilience.

Missing or blank values for the five required elements cause the appender to refuse startup with an explicit addError on Logback's status manager. The error message identifies which element is missing.

Delivery guarantees

Tabellarium is a best-effort transport with visible loss, not a durable audit store. The scope of every guarantee in this document:

• What the topic classes guarantee: the Kafka producer policy of a send that reaches the broker path. AUDIT enforces acks=all and idempotence, so a record the producer has accepted is not silently lost to a leader failover or duplicated by a retry.
• What they do not guarantee: end-to-end completeness. Ahead of the producer sit bounded in-memory queues drained by daemon workers. A full send queue, an open circuit breaker, a send failure, an expired shutdown budget, or a JVM crash loses events — counted and (except for a crash) routed to the optional fallback appender, but lost to Kafka nonetheless. The fallback path itself is again a bounded queue that drops (counted) on overflow, and is optional.
• Consequence: a deployment whose compliance requirement is "every audit event is durably recorded" needs a durable record ahead of or beside this pipeline (e.g. a transactional outbox in the emitting service). Use AUDIT to make the transport as safe as a logging pipeline can be — configure a fallback appender and alert on the kafka.appender.events.fallback / kafka.appender.fallback.dropped metrics to make the residual loss observable.

Mandatory override policy

A single Kafka producer shared by every topic would mean an audit topic and a debug topic share the same acks value — and if the operator configured acks=1 for throughput, audit records accepted by the producer could silently be lost on a Kafka leader failover.

This module classifies every topic into one of four classes and enforces per-class producer configuration:

Class	Producer durability	Mandatory overrides	Use case
AUDIT	Strictest	acks=all, enable.idempotence=true	Audit-relevant log streams (e.g. BaFin/MaRisk contexts)
FUNCTIONAL	Strict	acks=all	Operationally important logs
TECHNICAL	Best-effort	(none — operator-tunable)	Debug and diagnostic logs
PERFORMANCE	Best-effort	(none — operator-tunable)	High-volume metric logs

A mandatory override is non-negotiable: if the operator's <kafkaProducerProperties> specifies acks=1 for an audit topic, the appender forces acks=all at startup and emits a status warning naming the property, the operator-supplied value, and the enforced value. Auditors and operators see the override in the Logback startup log:

WARN  Mandatory override applied for AUDIT: acks forced from '1' to 'all'.
      This is a non-negotiable topic-class requirement; see TopicClass.AUDIT for rationale.

With the minimal configuration (only <defaultTopic>) all topics are treated as TECHNICAL — no mandatory overrides apply. The compliance differentiation activates per class through <mapping> elements.

Topic routing

<topicMapping> routes events by SLF4J marker and classifies each mapped topic:

<topicMapping>
  <defaultTopic>my-application.logs</defaultTopic>
  <mapping>
    <marker>SECURITY</marker>
    <topic>audit.security</topic>
    <topicClass>AUDIT</topicClass>
  </mapping>
</topicMapping>

Events whose markers match no <mapping> (including marker-less events) go to <defaultTopic>, classified via the optional <defaultTopicClass> (default: TECHNICAL) — set it to AUDIT etc. when the default stream itself carries that compliance grade. Each active class gets its own producer and circuit breaker with the class's overrides. Misconfiguration — unknown class names, a marker mapped twice, one topic with two classes, Kafka-invalid topic names — aborts start() with a named error. Full resolution rules and validation live in the configuration guide.

Resilience

Three resilience mechanisms run independently per topic class:

1. Per-class circuit breaker. A Resilience4j CircuitBreaker is instantiated per active topic class. A stuck audit-topic broker does not throttle technical-log delivery, and vice versa. Default thresholds (tuned for logging volume, canonical in the configuration guide): 50% failure rate over a sliding window of 20 calls, 30 second cooldown in open state, 10 probe calls in half-open, probes spread 5 ms apart. Operators can override per class by pre-registering a CircuitBreakerConfig under the name kafka-appender-audit / -functional / -technical / -performance on the registry.

2. Asynchronous delivery with callback-driven outcome tracking. Kafka's producer.send is invoked with a callback that feeds the circuit breaker (onSuccess / onError). The Future returned by send is deliberately not retained — the callback is the single source of truth for delivery outcome, so delivery failures are never invisible.

3. Fallback appender. When the circuit is open or a send fails synchronously, the original ILoggingEvent is routed to the configured fallback appender. Standard Logback <appender-ref> syntax is supported:

   <appender name="KAFKA_FALLBACK_FILE" class="ch.qos.logback.core.FileAppender">
     <file>/var/log/myapp/kafka-fallback.log</file>
     <encoder>...</encoder>
   </appender>
   
   <appender name="KAFKA" class="eu.inqudium.tabellarium.KafkaAppender">
     ...
     <appender-ref ref="KAFKA_FALLBACK_FILE"/>
   </appender>

   When no fallback is configured, records are silently dropped on failure. This is the deliberate operator choice: configuring a fallback says "loss is unacceptable here"; leaving it out says "best-effort is fine".

In addition, a self-logging guard keeps the appender out of feedback loops: log events originating from the appender's own Kafka producer threads (recognizable because the Kafka client names them after the producer's client.id) are ignored entirely — the producer's internal logging is never shipped through the producer itself.

Should I wrap this in a Logback AsyncAppender?

Short answer: no. A common pattern with Kafka appenders is to wrap them in ch.qos.logback.classic.AsyncAppender to keep application threads from blocking on Kafka I/O. This module makes that pattern unnecessary and, with default AsyncAppender settings, counterproductive.

Why it is not needed

AsyncAppender exists to absorb caller-thread blocking. This module eliminates caller-thread blocking through four layered defenses:

1. producer.send never runs on the caller. Each topic class has its own bounded send queue and worker thread (SendDispatcher); the logging thread only routes, encodes, enriches and enqueues in O(1). A full queue diverts to the fallback (metric reason queue.full) instead of blocking.
2. max.block.ms is capped at 500 ms per topic class (200 ms for PERFORMANCE). The cap is enforced: a lower operator value wins, a higher one is clamped with a startup warning. It bounds how long a send worker can be held per event.
3. The circuit breaker trips after ~10 failures (50% failure rate in a 20-call sliding window). Once open, subsequent events are routed to the fallback in O(1).
4. The fallback uses an asynchronous FallbackDispatcher — a bounded queue with its own daemon worker, so the Kafka I/O thread and the send workers are never held hostage by a slow fallback appender (e.g. a FileAppender on saturated disk).

Worst case for a service whose Kafka cluster has just gone down: caller threads keep logging in microseconds; each class's send worker absorbs at most max.block.ms per event until its breaker opens (~10 × 500 ms = 5 s, on the worker, not on your request threads), and queue overflow flows to the fallback, counted.

Why wrapping in AsyncAppender now hurts

Wrapping in AsyncAppender with its default settings introduces worse loss semantics than this module's built-in mechanisms:

• neverBlock=false (the default) is a trap. When the AsyncAppender queue (queueSize=256 by default) fills up, doAppend blocks the caller — defeating the entire point of using the wrapper. The 500 ms max.block.ms protection is bypassed because the wait happens in the queue offer, not in producer.send.
• discardingThreshold=queueSize/5 (the default) silently drops INFO/DEBUG/TRACE once the queue is 80% full. These dropped events do not reach the fallback appender; they vanish.
• AsyncAppender adds an extra thread, an extra queue, and extra indirection between the application and the appender for no benefit this module does not already provide.

Recommendation

Use the KafkaAppender directly:

<appender name="KAFKA" class="eu.inqudium.tabellarium.KafkaAppender">
    ...
    <appender-ref ref="KAFKA_FALLBACK_FILE"/>
</appender>

<root level="INFO">
    <appender-ref ref="KAFKA"/>
</root>

If an existing logback-spring.xml wraps the Kafka appender in an AsyncAppender, drop the wrapper:

- <appender name="ASYNC_KAFKA" class="ch.qos.logback.classic.AsyncAppender">
-   <appender-ref ref="KAFKA"/>
- </appender>
- <root level="INFO">
-   <appender-ref ref="ASYNC_KAFKA"/>
- </root>
+ <root level="INFO">
+   <appender-ref ref="KAFKA"/>
+ </root>

When AsyncAppender is still justified

The one cost that remains on the logging thread is the synchronous encoding/enrichment work in append(). Latency-critical paths with hard sub-millisecond budgets (trading, real-time risk) that cannot afford even that may still want to move it off the request thread. In that case, use AsyncAppender with these non-default settings:

<appender name="ASYNC_KAFKA" class="ch.qos.logback.classic.AsyncAppender">
    <appender-ref ref="KAFKA"/>
    <queueSize>2048</queueSize>
    <discardingThreshold>0</discardingThreshold>
    <neverBlock>true</neverBlock>
    <includeCallerData>false</includeCallerData>
</appender>

Be aware that neverBlock=true drops events silently without routing them to the fallback — a weaker loss guarantee than the appender's built-in resilience.

Reactive applications

This module is safe to use in Reactor-Netty / WebFlux services and in code that runs on JDK virtual threads. The largest reactive hazard — synchronized blocks in the appender hot path, which cause carrier-thread pinning on virtual threads and Reactor-Netty event-loop stalls — does not arise: the appender extends UnsynchronizedAppenderBase. There are no locks in the hot path; only atomics and volatiles.

Two reactive-specific concerns remain that are worth tuning per service.

max.block.ms bounds the send worker, not the event loop

KafkaProducer.send() is asynchronous per the Kafka spec, but it can synchronously block for up to max.block.ms when the producer buffer is full, metadata is stale, or buffer allocation contends with other producers. That block lands on the topic class's dedicated send worker — the event-loop (or virtual) thread that called log.info() only encodes and enqueues, and returns in microseconds regardless of this setting.

max.block.ms therefore no longer needs reactive-specific tuning for latency. What it still governs is queue drain speed during broker trouble: with the 500 ms class default, a stalled cluster lets each worker absorb at most ~10 × 500 ms before its breaker opens, during which the bounded send queue may fill and overflow to the fallback (reason queue.full). A service that prefers to shed to the fallback faster can lower the value in <kafkaProducerProperties>:

<kafkaProducerProperties>
    bootstrap.servers=...
    max.block.ms=50
</kafkaProducerProperties>

Trade-off: with 50 ms the worker gives up earlier under transient buffer pressure (not a cluster outage, just a brief backlog) and those events divert to the fallback. This is the intended use of the fallback. The circuit breaker still trips after roughly 10 failures, after which all further events route to the fallback in O(1) regardless of max.block.ms.

Everything else (acks, enable.idempotence, linger.ms) is decided by topic class, for servlet and reactive services alike.

BlockHound

Services that run BlockHound (io.projectreactor.tools:blockhound) in their integration tests will see the appender's internal operations flagged as blocking — most notably the LinkedBlockingQueue.offer() in the [FallbackDispatcher] and the internals of KafkaProducer.send(). These are not true blocks in the harmful sense (the queue offer is non-blocking on a non-full queue; the producer send is the operator's accepted max.block.ms budget), but BlockHound's heuristics don't know that.

Add an allow-list entry in the test setup:

BlockHound.builder()
    .allowBlockingCallsInside(
        "ch.qos.logback.classic.Logger", "callAppenders"
    )
    .allowBlockingCallsInside(
        "eu.inqudium.tabellarium.KafkaAppender", "append"
    )
    .install()

This declares that logging calls are an accepted block point in the service's contract — which they have to be, regardless of which appender is used.

MDC propagation for the partitioning key

The default partitioning-key extractor reads traceId from the MDC at the moment of the log.info(...) call. Logback freezes the MDC into the ILoggingEvent at that moment, so the appender always sees a consistent snapshot — there is no risk of reading "the wrong thread's MDC" inside the appender.

The key is length-bounded at 128 characters; a longer value is treated as no key at all. Applications commonly bridge an inbound request header into the MDC, so the value can be attacker-influenced, and an unbounded key would inflate every record past max.request.size — the resulting RecordTooLargeException is deliberately ignored by the circuit breaker, so those events would flood the fallback appender indefinitely. Note that the bound limits record size, not distribution control: an application that bridges unvalidated inbound values into the MDC can still influence which partition its records land on.

The reactive concern is upstream: in Reactor code, the trace context typically lives in the Reactor Context, not in the MDC of the event-loop thread. If the service does not bridge the Reactor Context into the MDC, MDC.get("traceId") returns null at the log.info() call, and the appender consequently routes records without a partitioning key (round-robin distribution by Kafka's default partitioner, instead of partition-locality per trace).

Verify with a quick check in any reactive handler:

@GetMapping("/test")
fun test(): Mono<String> = Mono.fromCallable {
    val traceId = MDC.get("traceId")
    log.info("traceId in MDC: {}", traceId)
    "ok"
}

If traceId is null in the log line, the MDC bridge is missing. Common bridges:

• Micrometer Tracing with reactor.MicrometerTracingObservationHandler
• Reactor Core 3.5+ with ContextSnapshotFactory (Hooks.enableAutomaticContextPropagation())
• Spring Boot 3.2+ with spring.reactor.context-propagation=auto

Fixing this is a service-side concern, not an appender concern. A service without trace-id-in-MDC still functions correctly; it just loses partition-locality for related events.

Metrics

The appender publishes hot-path counters, send-duration timers and queue-depth gauges to a Micrometer MeterRegistry. Metrics are opt-in: the appender runs without Micrometer on the classpath and emits no metrics until bindMeterRegistry() is called.

Metric inventory

Metric	Type	Tags	Meaning
kafka.appender.events.accepted	Counter	topic.class	Events entering KafkaAppender.append
kafka.appender.events.dispatched	Counter	topic.class	Events handed to producer.send (callback outcome unknown)
kafka.appender.events.fallback	Counter	topic.class, reason	Events diverted from Kafka (to the fallback if configured, otherwise dropped)
kafka.appender.send.duration	Timer	topic.class, outcome	Wall-clock send duration from invocation to callback
kafka.appender.fallback.dropped	Counter	—	Events lost because the fallback dispatcher queue was full
kafka.appender.fallback.queue.size	Gauge	—	Current depth of the fallback dispatcher queue
kafka.appender.fallback.queue.capacity	Gauge	—	Maximum depth of the fallback dispatcher queue
kafka.appender.send.queue.size	Gauge	topic.class	Current depth of the class's send dispatcher queue
kafka.appender.send.queue.capacity	Gauge	topic.class	Maximum depth of the class's send dispatcher queue

reason values: breaker.open, throttle, send.error, encoder.error, queue.full, shutdown. outcome values: success, error. topic.class values: audit, functional, technical, performance.

Cardinality budget per appender instance: ~51 time series. At 100 microservices in a shared Prometheus this is ~5 100 series — well within the default cardinality budget.

Additional bindings

When bindMeterRegistry() is called, two additional metric sources are bound:

• Circuit-breaker metrics (resilience4j.circuitbreaker.*) are published by the appender's own binder — no resilience4j-micrometer needed. Metric names and tags mirror TaggedCircuitBreakerMetrics, plus the same appender tag the appender's own meters carry, so multiple appender instances on one registry never collide.
• Micrometer Kafka binder (part of micrometer-core for older versions, micrometer-binders-kafka for newer) publishes the underlying Kafka producer's internal metrics (kafka.producer.*) if it is on the classpath. One binding per active topic class, tagged with topic.class and appender.

A missing Kafka binder is silently skipped — that binding is best-effort, not fail-fast.

Wiring up: Spring applications

For Spring Boot applications, the library ships a small @Configuration helper class, [KafkaAppenderMetricsBinding]. Add it as a bean in any @Configuration class:

@Configuration
class LoggingConfig {
    @Bean
    fun kafkaAppenderMetricsBinding(registry: MeterRegistry) =
        KafkaAppenderMetricsBinding(registry)
}

Or import it directly:

@Configuration
@Import(KafkaAppenderMetricsBinding::class)
class LoggingConfig

The binding listens for ContextRefreshedEvent, walks the Logback LoggerContext, finds every KafkaAppender, and calls bindMeterRegistry() on each. No further application code required.

To add application-specific common tags (e.g. service name, environment), pass them to the constructor:

@Bean
fun kafkaAppenderMetricsBinding(
    registry: MeterRegistry,
    @Value("\${spring.application.name}") app: String,
) = KafkaAppenderMetricsBinding(
    registry,
    Tags.of("application", app),
)

The Spring integration is deliberately not Spring Boot auto-configuration — operators import it explicitly. This keeps the appender's dependency tree honest (no transitive Spring pull) and makes it obvious in application code where the binding happens.

Wiring up: non-Spring applications

Call bindMeterRegistry directly from any lifecycle point that happens after the MeterRegistry is available:

val loggerContext = LoggerFactory.getILoggerFactory() as LoggerContext
loggerContext.loggerList.asSequence()
    .flatMap { logger ->
        generateSequence({ logger.iteratorForAppenders() }) { null }
            .first().asSequence()
    }
    .filterIsInstance<KafkaAppender>()
    .distinct()
    .forEach { it.bindMeterRegistry(meterRegistry, Tags.empty()) }

Pre-Spring log events (Logback initialization, Spring bootstrap logging) are not counted in either setup — this is a deliberate trade-off, since capturing them would require a static MeterRegistry reference that conflicts with Spring's lifecycle.

Grafana / dashboard pointers

A minimal dashboard typically shows:

• Throughput per topic class — rate(kafka_appender_events_accepted[1m]), stacked by topic_class.
• Loss rate — rate(kafka_appender_events_fallback[1m]), stacked by reason. A sudden spike in breaker.open means the cluster failed; in throttle means a sustained recovery probe; in send.error means individual send rejections (e.g. RecordTooLargeException after the deliberate exclusion).
• Send latency — histogram_quantile(0.99, kafka_appender_send_duration_seconds_bucket), faceted by outcome. p99 latency under 100 ms is the healthy baseline.
• Fallback queue saturation — kafka_appender_fallback_queue_size / kafka_appender_fallback_queue_capacity. Sustained values > 0.5 mean the fallback appender (typically a FileAppender) is slower than the event arrival rate — operator action needed.
• Dropped event total — kafka_appender_fallback_dropped_total. Any non-zero rate is a data-loss signal; if the operator configured a fallback, this number should stay at zero.

Architecture

┌────────────────────────────────────────────────────────────────────┐
│                      KafkaAppender (orchestrator)                  │
│                                                                    │
│  start() ─→ validateConfiguration ─→ buildPipeline                 │
│      ┌─────────────┬──────────────┬─────────────────┐              │
│      ▼             ▼              ▼                 ▼              │
│  TopicRouter   TopicTable   MessageEnricher   ProducerRegistry     │
│  (marker →     (topic →     (meta.* headers,  (one producer per    │
│   topic)        class)       traceId key)      active class)       │
│                                                                    │
│  append(event) — caller thread, CPU-bound only, never blocks:      │
│      encode ─→ route ─→ classify ─→ enrich ─→ dispatch O(1)        │
│                                                  │                 │
│         one per active topic class               │ queue full /    │
│      ┌───────────────────────────┐               │ stopped         │
│      │ SendDispatcher            │◀──────────────┤                 │
│      │ bounded queue + worker    │               │                 │
│      └─────────────┬─────────────┘               │                 │
│                    ▼ producer.send on the worker │                 │
│      ┌───────────────────────────┐               │                 │
│      │ ResilientMessageSender    │  breaker open /                 │
│      │ half-open throttle +      │  throttled / send               │
│      │ circuit breaker per class │  failed ─────────┐              │
│      └─────────────┬─────────────┘               │  │              │
│                    ▼ async callback              ▼  ▼              │
│               Kafka broker            ┌───────────────────────┐    │
│                                       │ FallbackDispatcher    │    │
│                                       │ bounded queue + worker│    │
│                                       └───────────┬───────────┘    │
│                                                   ▼                │
│                                        fallback appender           │
│                                        (<appender-ref>, optional;  │
│                                         overflow drops, counted)   │
└────────────────────────────────────────────────────────────────────┘

Every component below KafkaAppender is internal implementation with its own dedicated unit test. The supported public API is the operator surface only — the appender with its XML elements (KafkaAppender, TopicMappingConfig/TopicMappingEntry), the TopicClass enum, and the optional KafkaAppenderMetricsBinding; see ADR-0002. Substitution seams (producer factory, breaker registry, injected clocks) exist module-internally and carry the test suite; they are deliberately not a consumer contract.

Extension points

Custom partitioning key

The partitioning key is read from MDC traceId by default. There is currently no configuration surface for a different key (session id, user id, account id) — most deployments use the trace-id default. Per ADR-0002, such an override would be added as an XML-bindable KafkaAppender property (e.g. a partitioning-key MDC name), not by exposing the internal enricher — open an issue if your deployment needs it.

Future work

Items deferred from the current revision, in roughly decreasing priority order:

Producer-registry consolidation

The current design instantiates one Kafka producer per active TopicClass. For Kubernetes deployments where the broker enforces per-IP producer-connection limits, or for memory-constrained environments where 4 × 32 MB of producer buffer is a noticeable share of the pod's memory budget, the registry could be extended to share a single producer across classes whose configurations are compatible (same acks, same idempotence setting, etc.).

This is a non-trivial change because it interacts with the per-class circuit-breaker isolation: if AUDIT and FUNCTIONAL share a producer, a fault that affects the shared producer trips both circuit breakers together, partially defeating the isolation guarantee. Probably worth doing only if a concrete deployment hits the producer-count ceiling.

How it is tested

The suite is layered so the fast loop stays offline and the expensive guarantees still get proven:

• Offline unit/component base — the default mvn verify run needs no broker and no Docker: MockProducer, hand-built fakes, injected clocks, and latch-pinned concurrency scenarios cover routing, property composition, breaker/throttle behavior, the asynchronous dispatch and shutdown accounting, and the metrics lifecycle. Appender-level tests exercise the real asynchronous worker path (there is no synchronous test mode in production code).
• Declarative-contract layer — Joran round-trip tests feed real XML through JoranConfigurator and bind every documented element, so the operator-facing configuration surface is executable, not asserted.
• Real-broker stage — mvn -Pintegration test (Testcontainers, needs Docker) proves a successful TECHNICAL and AUDIT record against an Apache Kafka container: real serializers, LZ4, headers, partitioning key, and the AUDIT acks/idempotence handshake.
• External-contract stage — mvn -Pexternal-contract test runs characterization tests of third-party behavior that are excluded from the default loop on purpose.

The full inventory — every test sentence plus the rationale block explaining what it pins and why — is generated from each build and published as Test evidence; the coverage report and the badge above come from the same run. Both are generated artifacts: no number in them is maintained by hand.

Build

Standard Maven module:

mvn verify                                      # build + run all offline tests
mvn -Dtest=KafkaAppenderTest test               # run a single test class
mvn -Dtest='*MessageEnricher*' test             # pattern-match
mvn -Pintegration test                          # + real-broker tests (needs Docker)

The default test run is offline and fast (MockProducer, no Docker). The integration profile adds the Testcontainers-based real-broker tests (@Tag("integration")), which verify a successful TECHNICAL and AUDIT record end-to-end against an Apache Kafka container — real serializers, compression, headers, and the AUDIT acks/idempotence handshake.

The module has no Maven plugins beyond the Kotlin compiler and Surefire. Java 21 and Kotlin 2.4.10.

Contributing

Contributions are welcome — see CONTRIBUTING.md for the build setup, code-style expectations, and pull-request process. Security vulnerabilities should be reported privately as described in SECURITY.md.

License

Licensed under the Apache License, Version 2.0.