Under a worst-case 100%-recording load test against a single shared Kafka broker, request latency climbed from ~12 ms to ~518 ms p50 and throughput dropped ~9x, while router CPU stayed low — the request threads were blocked, not computing.
The record enqueue itself is non-blocking (bounded channel, try-send, fail-open drop). But the per-request flush control message uses a blocking channel send, and each flush message makes the writer run a synchronous multi-second producer drain. Under a saturated broker the writer stops draining, the control channel fills, and request threads block on the send.
Work items
-
Drop path. Make the per-request flush advisory — a non-blocking try-send, or remove the per-request flush entirely (durability is already covered by the count/timer flush policy) — and reserve the long blocking producer drain for shutdown / end-of-stream only. Recording stays fail-open throughout: on saturation the sink drops and marks loss; it must never stall a live payment request. The exact choice is gated on the profiling attribution (separate issue).
-
Broker-provisioning reuse. The recording sink should inherit the application's provisioned Kafka brokers (empty broker list ⇒ inherit, non-empty ⇒ explicit override) while keeping its own producer client configured for the durability recording needs: acks=all, idempotence, bounded buffer.
Sharing the client instance is rejected: the analytics producer carries none of those guarantees, flush is client-wide (recording's flush would drain the analytics buffer), the failure domains would entangle, and recording's ~10x volume would leak into the analytics queue and could broker-throttle it. Sharing brokers + a common base-client constructor gets the provisioning benefit without those couplings.
Acceptance
A 100%-record load test against provisioned brokers no longer stalls request threads; on deliberate sink saturation, requests continue (fail-open) and dropped records are counted, not silently lost.
Under a worst-case 100%-recording load test against a single shared Kafka broker, request latency climbed from ~12 ms to ~518 ms p50 and throughput dropped ~9x, while router CPU stayed low — the request threads were blocked, not computing.
The record enqueue itself is non-blocking (bounded channel, try-send, fail-open drop). But the per-request flush control message uses a blocking channel send, and each flush message makes the writer run a synchronous multi-second producer drain. Under a saturated broker the writer stops draining, the control channel fills, and request threads block on the send.
Work items
Drop path. Make the per-request flush advisory — a non-blocking try-send, or remove the per-request flush entirely (durability is already covered by the count/timer flush policy) — and reserve the long blocking producer drain for shutdown / end-of-stream only. Recording stays fail-open throughout: on saturation the sink drops and marks loss; it must never stall a live payment request. The exact choice is gated on the profiling attribution (separate issue).
Broker-provisioning reuse. The recording sink should inherit the application's provisioned Kafka brokers (empty broker list ⇒ inherit, non-empty ⇒ explicit override) while keeping its own producer client configured for the durability recording needs: acks=all, idempotence, bounded buffer.
Sharing the client instance is rejected: the analytics producer carries none of those guarantees, flush is client-wide (recording's flush would drain the analytics buffer), the failure domains would entangle, and recording's ~10x volume would leak into the analytics queue and could broker-throttle it. Sharing brokers + a common base-client constructor gets the provisioning benefit without those couplings.
Acceptance
A 100%-record load test against provisioned brokers no longer stalls request threads; on deliberate sink saturation, requests continue (fail-open) and dropped records are counted, not silently lost.