Benchmarking and Optimization Phase with Fair Multi-Framework Comparison

[leynos]

Summary

This branch establishes femtologging's benchmarking and performance optimisation programme as a strategic roadmap phase with comprehensive design documentation grounded in research on competing frameworks.

Three documents are added:

1. Roadmap (docs/roadmap.md) — four-phase strategic plan (foundation, expansion, internals lab, optimisation loop) following the GIST model, covering architecture-aware performance validation and evidence-driven optimisation work.

2. Technical Design (docs/tech/benchmarking-and-optimization.md) — complete specifications for the three-leaderboard methodology, fair comparison targets across six framework variants, benchmark dimensions, measurement protocols, concrete benchmark groups A–G, and optimisation playbook.

3. Execution Plan (docs/execplans/optimization-phase-design.md) — living plan for implementing the roadmap and design, with progress tracking and decision log.

The design establishes architecture-aware fairness: queue-based and synchronous logging handlers are compared on separate leaderboards, preventing the misleading conflation of asynchronous caller latency with synchronous end-to-end work.

Research findings

Prior-art investigation into logging framework architectures confirms:

• Loguru's enqueue=True: Uses queue.Queue with single daemon worker; achieves caller latency <10 µs and throughput 500k–1000k msg/sec (vs. 100–200k synchronous).
• Stdlib QueueHandler/QueueListener: Achieves caller latency 1–5 µs and throughput 250–500k msg/sec; explicitly supports async producer/consumer separation.
• Picologging: C extension variant achieves throughput 2–3M msg/sec (5–50× stdlib); trade-off is reduced portability.
• Fair benchmarking practices: Established best practice separates caller latency (time until log() returns) from end-to-end latency (time until message persisted). Single-metric benchmarks produce misleading rankings.

Design principles

1. Never compare asynchronous caller latency with synchronous end-to-end work without labelling loudly.
2. Architectural parity: Queue-based and synchronous handlers are benchmarked separately on different leaderboards.
3. Three-leaderboard model: Caller-visible cost (producer thread time), end-to-end completion (wall-clock to flush), diagnostic internals (granular costs).
4. pyperf and Criterion: Automatic calibration, statistical significance, multiple-worker processes, regression detection.
5. Fairness rules: Disabled logs, structured fields, idiomatic-vs.-parity modes explicitly handled.
6. Evidence-driven optimisation: Every optimisation PR must cite benchmark improvements, trade-offs, and significance.

Key deliverables

• Roadmap: Four phases (foundation, expansion, internals lab, optimisation loop) with 18+ GIST-aligned milestones.
• Technical Design: Complete specifications, seven benchmark groups (disabled, null-handler, direct I/O vs. queued, file batching, socket handler, saturation, configuration), optimisation playbook by failing benchmark.
• Repository Structure: benchmarks/ with adapters (stdlib sync/queue, picologging sync/queue, loguru sync/enqueue, femtologging), cases (workload scenarios), sinks (null, counting, slow, socket), runner, schema, compare, report modules.
• Makefile targets: bench-smoke, bench-python, bench-rust, bench-compare for CI and manual runs.
• Regression policy: Per-category thresholds (disabled >3%, enabled null >5%, caller latency >5%, memory >10%, drops always fail).
• Execution Plan: Living document with constraints, tolerances, progress checkboxes, decision log.

Review walkthrough

• Start with docs/roadmap.md for strategic structure and phase sequencing.
• Review docs/tech/benchmarking-and-optimization.md §1–4 for philosophy, three-leaderboard model, and comparison targets (research-backed framework selection).
• Review §5–8 for repository structure, adapter layer, fairness rules, and measurement protocols (pyperf configuration, correctness checks).
• Review §9 for concrete benchmark groups (A–G) and §10 for optimisation playbook (by bottleneck type).
• Review docs/execplans/optimization-phase-design.md for implementation sequencing, decision log, and progress tracking.

Test plan

The benchmarking programme itself serves as the validation mechanism:

• Smoke suite: Disabled logs, enabled null handler, single-threaded file I/O, configuration speed. Pass criteria: femtologging caller latency <10 µs, stdlib sync shows expected higher end-to-end cost, no regressions.
• Concurrency matrix: 1, 2, 4, 8, 16, 32, 2×CPU producer threads. Pass criteria: femtologging maintains low caller latency; synchronous handlers degrade under contention.
• File handler variants: Tmpfs and SSD, batch and queue capacity sweeps. Pass criteria: batching trade-offs are measurable; no silent performance cliffs.
• Socket handler: Loopback TCP/Unix socket, frame validation. Pass criteria: MessagePack serialisation and framing are correct; no dropped records.
• Saturation: Bounded queue, slow/stalled consumer, producer spike during rotation. Pass criteria: bounded and predictable behaviour; drops only where specified by policy.
• Internal lab (Criterion, perf stat): Hot-path microbenchmarks, formatter profiling, worker drain-loop analysis. Pass criteria: flamegraphs identify top 3 bottlenecks; syscall and context-switch counts guide next optimisation.
• Regression gate: Manual regression introduced to smoke suite blocks merge. Pass criteria: CI catches and reports violation.

Execution plan includes running smoke suite on CI (every commit) and full suite on pinned hardware after major work.

Validation

Documentation reviewed for:

• British English and Oxford dictionary conventions (–ize, –our, –re, –yse).
• Consistency across three documents and with femtologging architecture decisions (ADRs 001–004).
• Complete coverage of user's requirements (three leaderboards, comparison matrix, benchmark groups A–G, optimisation playbook, regression policy, reporting format, strategic success criteria).
• Clarity and actionability for both maintainers (implementation) and users (interpreting results).
• Research grounding in competitor architectures (loguru, picologging, stdlib queue modes).

Execution plan includes:

• Clear constraints and tolerances.
• Progress tracking with checkboxes.
• Decision log and retrospective sections.
• Concrete acceptance criteria per phase.

References

Lody session: https://lody.ai/leynos/sessions/optimization-phase-design-2026-06-17

[sourcery-ai]

Sorry @leynos, your pull request is larger than the review limit of 150000 diff characters

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8b75a935-ce24-4033-b69d-5b85f0d298b7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch optimization-phase-design

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}