Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ This is the second of the runs published on 2026-09-23:
5. [Shard-10 ramp](../2026-09-23-adhoc-shard-10-ramp-33c95bf/analysis.md):
ramp arrival with 1000 members x concurrency 10; 20,000 RPS is the highest
rate near the Redis reference, limited by slot capacity.
6. [Shard-20 ramp](../2026-09-23-adhoc-shard-20-ramp-33c95bf/analysis.md):
concurrency 20 per member raised the ceiling to 34,000 RPS at 16
partitions with queue time below the Redis reference.

## Purpose

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ This is the third of the runs published on 2026-09-23:
5. [Shard-10 ramp](../2026-09-23-adhoc-shard-10-ramp-33c95bf/analysis.md):
ramp arrival with 1000 members x concurrency 10; 20,000 RPS is the highest
rate near the Redis reference, limited by slot capacity.
6. [Shard-20 ramp](../2026-09-23-adhoc-shard-20-ramp-33c95bf/analysis.md):
concurrency 20 per member raised the ceiling to 34,000 RPS at 16
partitions with queue time below the Redis reference.

## Purpose

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ This is the fifth of the runs published on 2026-09-23:
5. **This run.** Hold the run 4 topology fixed and ramp arrival at 8, 16, 32,
and 64 partitions to find the highest rate that keeps queue time near the
Redis reference.
6. [Shard-20 ramp](../2026-09-23-adhoc-shard-20-ramp-33c95bf/analysis.md):
concurrency 20 per member raised the ceiling to 34,000 RPS at 16
partitions with queue time below the Redis reference.

## Purpose

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Run Analysis

Run: [run.json](run.json)

Results: [result.json](result.json)

Kafka: [compose.yaml](../2026-09-23-adhoc-share-group-1000-members-33c95bf/compose.yaml)
(`group.share.max.size=1000`, `group.share.partition.max.record.locks=4000`)

This document is generated by AI from the run metadata, machine results, and
retained diagnostics.

## Sequence

This is the sixth and last of the runs published on 2026-09-23:

1. [Harness capacity](../2026-09-23-north-star-steady-v1-harness-capacity-33c95bf/analysis.md):
`kq-bench` saturated the host near 13,000 tasks/s.
2. [Lean-probe capacity](../2026-09-23-adhoc-lean-probe-capacity-33c95bf/analysis.md):
kq reached 150,000 tasks/s, but queue p50 stayed near 465 ms.
3. [Queue-latency diagnosis](../2026-09-23-adhoc-queue-latency-diagnosis-33c95bf/analysis.md):
queue time comes from the worker generation wait.
4. [Share group with 1000 members](../2026-09-23-adhoc-share-group-1000-members-33c95bf/analysis.md):
1000 members x concurrency 10 sustained 20,000 RPS with low queue time at
8 to 32 partitions and collapsed at 2 and 4.
5. [Shard-10 ramp](../2026-09-23-adhoc-shard-10-ramp-33c95bf/analysis.md):
20,000 RPS was the highest rate near the Redis reference, limited by slot
capacity.
6. **This run.** Repeat the run 5 ramp with concurrency 20 per member to test
whether more slots per member raise that ceiling.

## Purpose

Find the highest arrival rate at which 1000 share-group members x concurrency
20 keep north-star queue time near the Redis reference (36 / 77 / 100 ms
p50 / p95 / p99), ramping arrival at 8, 16, 32, and 64 ready partitions with
the record-lock window fixed at 4000.

The acceptance limits match run 5: every task completes exactly once, enqueue
reaches at least 99% of the requested rate, completions end within 3 s of
production, and queue p50 / p95 / p99 are at most 50 / 150 / 300 ms.

## Observations

| Partitions | Requested RPS | Median completions/s | Queue p50 / p95 / p99 / max | Outcome |
| ---: | ---: | ---: | ---: | --- |
| 8 | 20,000 | 19,991 | 4 / 13 / 22 / 577 ms | meets target |
| 8 | 24,000 | 23,994 | 5 / 19 / 141 / 826 ms | meets target |
| 8 | 26,000 | 26,018 | 5 / 49 / 273 / 767 ms | meets target |
| 8 | 28,000 | 28,120 | 7 / 192 / 417 / 1,147 ms | sustained; over limit |
| 16 | 24,000 | 24,000 | 8 / 28 / 45 / 768 ms | meets target |
| 16 | 28,000 | 28,008 | 10 / 35 / 53 / 678 ms | meets target |
| 16 | 32,000 | 32,000 | 16 / 51 / 73 / 483 ms | meets target |
| 16 | 34,000 | 33,984 | 22 / 66 / 95 / 802 ms | meets target |
| 16 | 36,000 | 36,022 | 49 / 155 / 217 / 558 ms | sustained; p95 over limit |
| 32 | 32,000 | 31,987 | 27 / 83 / 115 / 837 ms | meets target |
| 32 | 34,000 | 34,013 | 38 / 108 / 181 / 718 ms | meets target |
| 32 | 36,000 | 36,067 | 94 / 351 / 481 / 714 ms | sustained; over limit |
| 64 | 24,000 | 24,002 | 25 / 87 / 121 / 504 ms | meets target |
| 64 | 28,000 | 28,000 | 32 / 109 / 164 / 815 ms | meets target |
| 64 | 32,000 | 32,010 | 47 / 135 / 184 / 606 ms | meets target |
| 64 | 34,000 | 34,001 | 65 / 182 / 262 / 579 ms | sustained; over limit |
| 64 | 36,000 | 35,849 | 127 / 1,244 / 2,522 / 3,593 ms | sustained; tail 3 s |

- Highest rate meeting the target: 26,000 RPS at 8 partitions, 34,000 at 16
and 32, and 32,000 at 64.
- The best point was 34,000 RPS at 16 partitions with 22 / 66 / 95 ms queue
p50 / p95 / p99, below the Redis reference at every percentile.
- At 20,000 RPS and 8 partitions, concurrency 20 measured 4 / 13 / 22 ms
against 14 / 48 / 99 ms for concurrency 10 in run 5.
- No tested point overloaded outright. Completions tracked arrival up to the
highest tested rate of 36,000 RPS, where 64 partitions had the only
multi-second queue tail.
- At the same arrival rate, queue time rose with partition count above 16.
- All 30,480,000 tasks across all points completed with zero duplicates and
zero missing IDs.
- Process CPU at 34,000 to 36,000 RPS: Kafka 5.7 to 6.4 cores, producers 1.6
to 1.9, workers 1.7 to 1.9, about 9.5 to 10.2 cores of the 12-core host
before other system processes.

## Interpretation

- Doubling per-member concurrency raised the latency-bounded ceiling from
20,000 to 34,000 RPS and lowered queue time at equal load. Queue time is
governed mainly by spare slot capacity. Arrivals find an idle member sooner
when more slots are free, which outweighs the longer generation wait of 20
records versus 10. This supersedes the earlier suggestion that smaller
generations are better.
- At 8 partitions, the limit near 26,000 to 28,000 RPS is consistent with the
in-flight window hypothesis from run 4:
`8 partitions x 4000 locks / about 1.1 s acquire-to-ack`. Not confirmed
from broker metrics.
- Above 16 partitions, latency again worsened with more partitions, as in
run 5.
- The throughput ceiling was not located; 36,000 RPS was the highest rate
tested. The host was near CPU saturation there, so this laptop cannot
separate a kq limit from a host limit above about 36,000 RPS.

## Suggestions

1. Continue on dedicated multi-core or cloud hardware, where Kafka, producers,
and workers do not share one host.
2. Test concurrency 40 and higher on that hardware to find where the longer
generation wait begins to outweigh added capacity.
3. Size partitions using the in-flight window estimate with headroom, and
avoid large surpluses of partitions. For this topology, 16 was the best
tested value.
4. Repeat the best points several times to bound run-to-run variance in p95
and p99.

## Caveats

- Single Dockerized Kafka broker on localhost, RF=1; one run per point with a
60 s arrival window.
- Same probe as runs 2 to 5: ready success path only, sleep-only handlers,
200-byte payloads, queue time measured from just before `Enqueue`.
- The ramps for 16, 32, and 64 partitions started at 24,000 to 32,000 RPS, not
20,000.
- The acceptance limits are an engineering judgment of "close to" the Redis
reference, not a production SLO. The Redis figures were not measured with
this workload or harness.
Loading
Loading