feat(relay): group-commit batching for plain event inserts (F1)

[tlongwell-block]

What

Group-commit batching for plain (non-replaceable, non-reaction) event inserts. At high ingest rates each plain event pays a full synchronous COMMIT (WAL flush; storage-quorum ack on Aurora). A single writer task per relay coalesces concurrent inserts into one transaction so one commit amortizes over the batch, while each caller keeps its own per-event result.

Per discussion with Tyler/Mari: one worker per relay + horizontal relay scaling is the deliberate shape — no internal worker pool until measurements force it.

Design (buzz_db::batch)

• Opportunistic, not timed. The worker takes one request, then drains whatever else is already queued (up to BUZZ_WRITE_BATCH_MAX, default 0 = off; see Perf validation). A lone request under light load flushes immediately through the direct path — zero added latency at low QPS; batches only form while a previous transaction is committing.
• ACK-after-commit. A caller's future resolves only after the transaction containing its event durably committed (or definitively failed).
• Poison isolation, statement time. Savepoint per event: an FK violation or rejected kind fails only that event; batch-mates commit.
• Poison isolation, commit time. Deferred constraint triggers (migrations 0021/0022) fire during COMMIT, after savepoints are released, so one bad row aborts the whole batch. A server-reported COMMIT error (SQLSTATE present → provable abort) replays every request individually through the unbatched path so the poison fails alone.
• Indeterminate commits are not replayed. A transport failure awaiting the COMMIT response may have committed server-side; a blind replay would misreport durable inserts as duplicates and skip their mentions. Callers get DbError::CommitOutcomeUnknown; client retries are idempotent (ON CONFLICT DO NOTHING → duplicate:). Semantics-preserving vs. today: a single-event commit whose response is lost also errors to the client today.
• Mentions stay post-commit, on the pool, log-and-succeed, only for was_inserted == true, each caller answered after its own attempt — exactly matching the unbatched wrapper. The batch path uses the existing private insert_event_with_thread_metadata_tx (now pub(crate)), which is mentions-free by construction; the fallback path uses the public wrapper which owns its mentions. Disjoint paths, no double pass.
• Degradation, never hangs. Dead worker → the batcher handle returns None → ingest falls through to today's direct path. Worker panic mid-batch drops the oneshots → callers get CommitOutcomeUnknown.

Relay wiring

• BUZZ_WRITE_BATCH_MAX env knob; 0 disables (state holds Option<EventWriteBatcher>, ingest uses the direct path — pre-batching behavior exactly).
• Only the plain persistent-event branch routes through the batcher. Kind 9007 (channel create, carries the channel-compensation flow), replaceables, and reactions stay direct.
• Coalescing counters exported as buzz_write_batch_* Prometheus counters from the existing pool-metrics poller: commits_total, events_committed_total, single_flushes_total, fallback_events_total, indeterminate_total. Committed vs. fallback vs. indeterminate are distinguishable by construction (batch_* count only successful multi-event commits).

Tests

PG-backed (#[ignore], run with --include-ignored against local PG):

• mixed_batch_maps_results_per_event — one batch of [good, dup, dup-again, FK-poison, rejected AUTH kind, threaded reply ×2]: per-event result pinning, poison fails alone, in-batch dedup, exact thread counters on the root, durability of every reported insert.
• deferred_trigger_abort_replays_per_event — pool armed with the migration-0021 created_at floor; a stale event aborts the batch at COMMIT, replay commits the good events and fails the stale one alone.
• commit_abort_classification — server-reported SQLSTATE → aborted; IO/pool-shaped errors → not proven aborted.
• dead_worker_returns_none_for_fallback — no DB; dead batcher signals fallback instead of hanging.
• concurrent_inserts_coalesce_and_all_commit — 64 concurrent inserts through the public API; committed-before-ACK asserted by reading each row from a second connection at the instant its future resolves; counters prove real multi-event commits happened.

All 5 green against local PG; clippy --all-targets -D warnings and fmt --check clean on buzz-db + buzz-relay; unfiltered cargo test -p buzz-db --lib and -p buzz-relay --lib pass (two pre-existing failures on clean main unrelated to this change: usage_metrics_lock under --include-ignored parallelism, mesh_demo 504 — both fail identically with this diff stashed).

Perf validation (done — default is OFF)

Mari's gate on the repaired-T1a base (merge tip bdbf2f0c1, release binary, alternated 0/16 restarts against the same migrated-0024 PG, 64 synchronous clients, hot permanent channel, matched accepted counts):

offered QPS	max=0 p50/p95/p99	max=16 p50/p95/p99	DB commits/msg 0 → 16
200	17.1 / 25.8 / 27.7 ms	17.5 / 31.2 / 51.5 ms	3.07 → 2.28 (-26%)
500	12.5 / 18.7 / 22.8 ms	19.8 / 30.3 / 34.6 ms	2.93 → 2.19 (-25%)
1000	11.5 / 16.6 / 20.4 ms	16.4 / 25.4 / 35.5 ms	2.91 → 2.17 (-25%)

Batching cuts DB commits/msg ~25% but regresses p50 43–58% and p99 52–74% at 500–1000 QPS: the single global batch lane serializes all channels behind one transaction at a time, which outweighs commit amortization. BUZZ_WRITE_BATCH_MAX therefore defaults to 0 (batching off, exact pre-batching behavior); the machinery ships opt-in. Flipping the default requires fixing the lane-serialization cost and remeasuring.

[tlongwell-block]

Parking this without merge, per Tyler's call after the performance gate.

Outcome summary:

• Machinery is correctness-cleared (Mari's independent review + PG-backed suite at 602e27fad), but on the repaired-T1a base the single global batch lane regresses p50 43–58% / p99 52–74% at 500–1000 QPS while saving ~25% DB commits/msg — see the gate table in the PR body.
• Root cause of the miss: Postgres already group-commits at the WAL level (concurrent transactions share fsyncs), so the direct path was getting commit amortization for free without serialization. An application-level single lane removes that concurrency to rebuild a worse version of it. (strfry-style batching wins on LMDB precisely because LMDB forces a single writer anyway.)
• The remaining open question — whether Aurora's quorum-ack commit overhead flips the math — was judged not worth carrying dormant machinery for. If commit throughput ever shows up as the production bottleneck, revive this branch and run the matched Aurora-shaped matrix Mari and Eva specified in the thread first.

Branch dawn/f1-write-batching stays at 602e27fad (correctness fixes, classifier semantics, and PG test suite intact) for revival.