feat(bench): warm up before the measured window

[mattsse]

Summary

bench send now warms the network up before the measured window instead of measuring from the first transaction.

Multi-region Tempo runs (10 validators, 4 regions, ~50k target TPS, 90 s windows) showed three cold-start effects inside the measured window: the first multi-megabyte block body over each proposer→peer connection arrived 5–10x slower than later ones (block times of 1–2 s for the first ~15 blocks), execution caches and the node-side build estimators needed ~15 loaded blocks to converge, and an instant jump to the target rate filled every pool within two seconds and phase-locked transaction expiry to that burst. Together they pulled the headline TPS from a ~15k steady state down to 11.6k.

How it works

1. Ramp the send rate from 10% of the warm-up rate to the warm-up rate over --warmup-ramp (the token bucket's rate is now adjustable while running). The warm-up rate is --tps capped at --warmup-tps (default 2000), so a cold network never sees a 50k–100k target during startup.
2. Hold at the warm-up rate until readiness holds, evaluated from data bench send already has (block polling via the query RPC, txpool_status):
   • every expected proposer (block beneficiary, which Tempo resolves per validator) produced --warmup-proposals blocks with at least half the running median transaction count,
   • the median transaction count of the last --warmup-stable-blocks blocks agrees with the previous window within --warmup-stable-tolerance,
   • txpool_status pending counts are stable over 10 s (only when the warm-up runs at the target rate; skipped when the endpoint lacks the method),
   • at least --warmup-min elapsed. --warmup-max ends the warm-up regardless and flags the run timeout.
3. Hand off: when the warm-up rate was capped, ramp from the cap to --tps over --warmup-ramp, still inside the warm-up, so the measured window starts at full rate rather than with a step.
4. Move the origin at the boundary: start_block, the report's started_at, and metric and sample offsets all move to that moment. Load never pauses. Metrics are excluded through a checkpoint rather than a counter reset, so exported counter series stay monotonic.
5. --duration ends the measured window a fixed time after the boundary: the source is stopped and the queued backlog dropped (requests already on the wire still complete). This lets harnesses over-provision transactions for a warm-up of unknown length.

The warm-up keeps ticking while the sender drains its buffered backlog after the source ends, so the boundary can still be reached late in a run. If the source runs dry before the warm-up ends, the whole run is reported as before and the warm-up is flagged source-exhausted.

Flags and defaults

Flag	Default	Derivation
--warmup <auto|off|DURATION>	auto	off restores the previous behaviour; a duration gives a fixed phase
--warmup-tps	2000	non-empty blocks from every proposer within the minimum warm-up (~1.4k tx per 0.7 s block) while staying >10x below the 50k–100k targets that overwhelmed a cold network
--warmup-ramp	10s	pools filled in <2 s at 50k TPS; 10 s spreads admissions over ~40% of the 25 s validity window and gives ~14 blocks before full pressure; also the hand-off ramp length
--duration	none	send this long after the measured window starts, then stop
--warmup-min	30s	~29 blocks (20 s) until ten random leaders have all proposed once, rounded up to one validity window
--warmup-max	90s	two full blocks per proposer take 45–50 blocks (~35 s); 2.5x that
--warmup-proposals	2	first body per connection slow, second fast; two confirms and tolerates a small first block
--warmup-proposers	distinct --rpc-url count	override when submitting through a load balancer or to a subset of validators
--warmup-stable-blocks / --warmup-stable-tolerance	10 / 0.10	fill rate plateaued after ~15 blocks; steady windows differ ~5%, ramp windows 30–50%
--warmup-pool-check	true	compares readings 10 s apart; skipped when txpool_status is unsupported

The derivations are repeated on the flags (bench send --help) and on the constants in bench-core/src/warmup.rs.

Reports

• Console: a line at the boundary plus a warm-up section in the summary.
• JSON: a warmup object (mode, outcome, duration, blocks, warm-up rate, hand-off ramp, per-proposer full-block counts, conditions at the decision, transactions in flight at the boundary) and started_unix_ms, the measured window's start, for harnesses.
• ClickHouse: the run's started_at is the measured window's start; warmup_* keys are added to the run metadata (no schema change).

Verification

• cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, unit tests for the tracker (ramp, readiness, timeout, fixed, pool check, summary), metrics checkpointing, sample rebasing, rate-limiter rate changes, and CLI parsing.
• Smoke-tested against anvil --block-time 1 with txgen-ethereum generate at 20 TPS: fixed warm-up, auto reaching readiness (pool check pending → stable → ready at 11.6 s), readiness reached while draining the backlog after the source ended, --warmup off, the source-exhausted fallback, a capped warm-up (--warmup-tps 8 → hand-off ramp to 20 TPS → boundary) with --duration 8s ending the window on time (382 queued transactions dropped), and an unlimited --tps 0 target warming up at the cap. Block ranges, rebased time series and the JSON warmup object came out as expected.
• tests/send_auth.rs now passes --warmup off because it pins the exact query-RPC traffic of a plain send.
• Pre-existing failures unrelated to this change on this machine: tests/call_replay.rs (anvil-version dependent) and txgen-ethereum/tests/scenario_two_chain.rs; both fail identically on main.

Harness changes (separate PRs, depend on this one)

• tempoxyz/tempo-multi-region-benchmark: scripts/txgen_workload.sh generates duration + warmup max + ramp seconds of workload, passes the warm-up flags and --duration, and takes started_at/start_block from the txgen report.
• tempoxyz/tempo: contrib/bench/txgen/helpers.nu does the same for the single-runner bench (vault and zones presets keep --warmup off).
• Dashboards can use warmup_* run metadata to show warm-up blocks as pre-run context.

🤖 Generated with Claude Code