Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,10 @@ JAZZER_FUZZ=1 mvn -Dtest=TopicRouterFuzzTest test
2. Keep the change focused — one logical change per pull request.
3. Make sure `mvn verify` passes.
4. Update the README / `docs/` when the configuration surface or the
metrics inventory changes.
metrics inventory changes. `DocumentationContractTest` pins the
configuration guide's defaults tables and the metrics overview's
inventory to the constants, so a changed constant fails `mvn verify`
until the tables follow.
5. Open the pull request with a description of *what* changed and *why*.

## Reporting bugs and requesting features
Expand Down
52 changes: 31 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,9 +193,11 @@ Reference for every supported element:
| `<includeCallerData>` | No | boolean | Captures caller data on the logging thread before the asynchronous hand-off (default false); only relevant when a fallback layout uses `%caller`. |
| `<appender-ref ref="..."/>` | No | ref | Single fallback appender — see [Resilience](#resilience). |

Missing or blank values for the five required elements cause the
appender to refuse startup with an explicit `addError` on Logback's
status manager. The error message identifies which element is missing.
A missing `<encoder>` or a blank `<environment>`, `<component>` or
`<cmdbId>` makes the appender refuse startup with an explicit `addError`
on Logback's status manager naming the element; a missing
`<defaultTopic>` or unusable producer properties fail the pipeline
construction the same way.

## Delivery guarantees

Expand Down Expand Up @@ -302,10 +304,12 @@ Three resilience mechanisms run independently per topic class:
source of truth for delivery outcome, so delivery failures are
never invisible.

3. **Fallback appender.** When the circuit is open or a send fails
synchronously, the original `ILoggingEvent` is routed to the
configured fallback appender. Standard Logback `<appender-ref>`
syntax is supported:
3. **Fallback appender.** Whenever an event cannot reach Kafka — an
open breaker, a throttled probe, a failed send (synchronous or via
the callback), a full send queue, a hot-path error, or the remainder
at shutdown — the original `ILoggingEvent` is routed to the
configured fallback appender, tagged with the reason in the metrics.
Standard Logback `<appender-ref>` syntax is supported:

```xml
<appender name="KAFKA_FALLBACK_FILE" class="ch.qos.logback.core.FileAppender">
Expand Down Expand Up @@ -443,7 +447,7 @@ hazard — `synchronized` blocks in the appender hot path, which cause
carrier-thread pinning on virtual threads and Reactor-Netty
event-loop stalls — does not arise: the appender extends
`UnsynchronizedAppenderBase`. There are no locks in the hot path;
only atomics and volatiles.
only atomics, volatiles and a per-thread reentry flag.

Two reactive-specific concerns remain that are worth tuning per
service.
Expand Down Expand Up @@ -505,12 +509,12 @@ buffered record.
Services that run BlockHound (`io.projectreactor.tools:blockhound`)
in their integration tests will see the appender's internal
operations flagged as blocking — most notably the
`LinkedBlockingQueue.offer()` in the [FallbackDispatcher] and the
internals of `KafkaProducer.send()`. These are not true blocks in
the harmful sense (the queue offer is non-blocking on a non-full
queue; the producer send is the operator's accepted
`max.block.ms` budget), but BlockHound's heuristics don't know
that.
`LinkedBlockingQueue.offer()` of the per-class send queue (every
event) and of the fallback dispatcher. These are not true blocks in
the harmful sense (the offer is non-blocking, a full queue rejects
instead of waiting; `KafkaProducer.send()` itself runs on the
appender's own worker threads, never on the caller), but BlockHound's
heuristics don't know that.

Add an allow-list entry in the test setup:

Expand Down Expand Up @@ -594,7 +598,7 @@ Micrometer on the classpath and emits no metrics until
| `kafka.appender.events.dispatched` | Counter | `topic.class` | Events handed to `producer.send` without a synchronous failure (callback outcome unknown) |
| `kafka.appender.events.fallback` | Counter | `topic.class`, `reason` | Events diverted from Kafka (to the fallback if configured, otherwise dropped) |
| `kafka.appender.send.duration` | Timer | `topic.class`, `outcome` | Wall-clock send duration from invocation to callback |
| `kafka.appender.fallback.dropped` | Counter | — | Events lost because the fallback dispatcher queue was full |
| `kafka.appender.fallback.dropped` | Counter | — | Events lost by the fallback dispatcher (queue full, `doAppend` threw, worker died, shutdown remainder) |
| `kafka.appender.fallback.queue.size` | Gauge | — | Current depth of the fallback dispatcher queue |
| `kafka.appender.fallback.queue.capacity` | Gauge | — | Maximum depth of the fallback dispatcher queue |
| `kafka.appender.send.queue.size` | Gauge | `topic.class` | Current depth of the class's send dispatcher queue |
Expand Down Expand Up @@ -692,15 +696,17 @@ happens after the `MeterRegistry` is available:
```kotlin
val loggerContext = LoggerFactory.getILoggerFactory() as LoggerContext
loggerContext.loggerList.asSequence()
.flatMap { logger ->
generateSequence({ logger.iteratorForAppenders() }) { null }
.first().asSequence()
}
.flatMap { logger -> logger.iteratorForAppenders().asSequence() }
.filterIsInstance<KafkaAppender>()
.distinct()
.forEach { it.bindMeterRegistry(meterRegistry, Tags.empty()) }
```

(Descend into `AsyncAppender` wrappers yourself if you use them; the
Spring binding does.) A repeated `bindMeterRegistry` call replaces the
previous binding, and a call on a stopped appender is ignored with a
status warning.

Pre-Spring log events (Logback initialization, Spring bootstrap
logging) are not counted in either setup — this is a deliberate
trade-off, since capturing them would require a static
Expand Down Expand Up @@ -861,8 +867,12 @@ AUDIT record end-to-end against an Apache Kafka container — real
serializers, compression, headers, and the AUDIT acks/idempotence
handshake.

The module has no Maven plugins beyond the Kotlin compiler and Surefire.
Java 21 and Kotlin 2.4.10.
The artifact targets Java 21 (Kotlin 2.4.10); building needs JDK 24+
because of the JVM flags in `.mvn/jvm.config`. The quality gates that
run with `mvn verify` (ktlint, JaCoCo, the documentation-contract test
that pins the guide's tables to the constants, Jazzer regression
inputs) and the CI-only scans (OSV via CycloneDX SBOM, CodeQL, nightly
fuzzing) are described in [CONTRIBUTING.md](CONTRIBUTING.md).

## Contributing

Expand Down
9 changes: 5 additions & 4 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,16 @@ java -jar benchmarks/target/benchmarks.jar AppendPipelineBenchmark -bm sample -t
```

All benchmarks use 3 forks, 5 warmup and 5 measurement iterations by
default (annotation-driven). Raw outputs of the 2026-08-29 verification
session live under `results/2026-08-29/`.
default (annotation-driven). Raw outputs live under `results/<date>/`:
the 2026-08-29 verification session, the 2026-08-30 re-run after the
header fix, and the 2026-09-07 sender-path re-measurement.

## Inventory

| Benchmark | Verifies | What it measures |
|---|---|---|
| `RecordHeadersBenchmark` | PERF_ANALYSIS-2026-08-29T11-01-08 finding 2 | Per-record header construction: production shape (5 × `headers().add`) vs. one shared pre-built header list. Primary metric: `gc.alloc.rate.norm`. |
| `SenderPathBenchmark` | finding 3 | The delivered-path worker side (`ResilientMessageSender.send` incl. breaker, record build, instant-success callback) with `metricsBound=false/true`; the param delta is the observability envelope per delivered event. |
| `RecordHeadersBenchmark` | PERF_ANALYSIS-2026-08-29T11-01-08 finding 2 | Per-record header construction: the pre-fix shape (5 × `headers().add` per record) vs. the shared pre-built header list production uses since the fix (`EnrichedRecord.headers`). Historical evidence, not a current-code regression guard. Primary metric: `gc.alloc.rate.norm`. |
| `SenderPathBenchmark` | finding 3 | The delivered-path worker side (`ResilientMessageSender.send` incl. breaker, record build, instant-success callback object) with `metricsBound=false/true`; the param delta is the observability envelope per delivered event. Also the guard for the callback object's scalar replaceability: 112 B/op unbound since the `@Volatile` removal (`results/2026-09-07/`). |
| `HandoffBenchmark` | finding 1 | The caller-side hand-off primitive: N producers `offer` into a bounded `LinkedBlockingQueue` while one batch-draining consumer keeps it near-empty (worst-case put-lock contention; `rejected` aux counter proves the regime). Thread split via `-tg 1,<producers>`. |
| `AppendPipelineBenchmark` | secondary evidence (findings 1/3), caller-side allocation | The full production `doAppend` path open-loop against a `DiscardingProducer`. Under open load it saturates the worker and measures the shedding regime — the teardown prints the achieved path mix (delivered vs. diverted), which is part of the evidence, not a hidden variable. |

Expand Down
17 changes: 10 additions & 7 deletions docs/config/example-logback-spring.xml
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@
<!-- Multi-line "key=value" Kafka producer configuration. Helm /
Spring placeholder substitution happens before the appender
parses the text. Lines starting with '#' are treated as
comments. Whitespace around keys and values is trimmed. -->
comments. Leading whitespace (XML indentation) is ignored,
trailing whitespace on values is trimmed. -->
<kafkaProducerProperties>
bootstrap.servers=${KAFKA_BOOTSTRAP_SERVERS:-kafka.example.com:9092}
security.protocol=SSL
Expand Down Expand Up @@ -155,19 +156,21 @@
Recommended: leave unset or set to false. -->
<!-- <debug>true</debug> -->

<!-- Fallback appender. When the Kafka circuit breaker is open or
a send fails, the event is delivered through this appender
instead. Without it, failed events are silently dropped. -->
<!-- Fallback appender. When an event cannot reach Kafka (open
circuit breaker, failed send, full send queue, shutdown
remainder), it is delivered through this appender instead.
Without it, such events are silently dropped. -->
<appender-ref ref="KAFKA_FALLBACK_FILE"/>

</appender>

<!-- =================================================================
Root logger configuration. Reference the KafkaAppender DIRECTLY:
wrapping it in an AsyncAppender is NOT recommended - the appender
already bounds caller-thread blocking internally (max.block.ms,
circuit breaker, async fallback dispatcher), and default
AsyncAppender settings weaken the loss semantics. See README,
already performs producer.send on its own per-class worker
threads (the logging thread only encodes and enqueues in O(1)),
and default AsyncAppender settings weaken the loss semantics.
See README,
"Should I wrap this in a Logback AsyncAppender?".

Only for sub-100 ms hard-latency SLAs is a wrapper justified,
Expand Down
Loading
Loading