Skip to content

fix(bench): compress metric samples while recording - #218

Draft
decofe wants to merge 1 commit into
mainfrom
centaur/compress-metric-spool-1789750336
Draft

decofe wants to merge 1 commit into
mainfrom
centaur/compress-metric-spool-1789750336

Conversation

@decofe

@decofe decofe commented Sep 18, 2026

Copy link
Copy Markdown
Member

Summary

Stream temporary metric samples through fast gzip instead of writing uncompressed NDJSON during the load. Preserve the configured scrape cadence, sample values, labels, timestamps, lazy cutoffs, and decoded reporter output; no new dependency is required.

In the paired 200 ms experiment, root-disk writes fell from 74.07 / 74.77 GiB to 6.14 / 6.51 GiB per phase, a 91.5% reduction in the two-phase averages. Each phase still records approximately 66 million samples.

Draft: this is not a demonstrated TPS fix. The official verdict is Mixed Results: TPS 7,749 → 9,170 (+18.34%, reported bootstrap half-width 24.53%, neutral); builder P50/P90 are +5.97%/+5.09% and classified bad. Validator gas throughput improves 15.91%; block-time P99 is neutral. Additional unprofiled repetitions are needed before claiming a node-performance improvement.

Profile

  • The baseline's longest Node A persistence batch takes 59.9 seconds for 37 blocks, with 6.8 seconds of recorded thread CPU time. 44.8% of its on-CPU samples include page-fault handling. CPU samples do not measure elapsed I/O wait.
  • Evidence is in the workflow's tempo-bench-results artifact: profile-baseline-1-a.json.gz, node logs, metric archives, cgroup snapshots, and summary.json. Compression reduces the recorder's large unrelated write stream but adds compression/decompression CPU work; persistence stalls are not eliminated.
  • A same-settings control reports a -14.7% TPS regression despite identical binaries/settings, demonstrating that two-pair comparisons can give misleading attributions.

Verification

  • cargo test --workspace — passed; one pre-existing sizing test remains ignored.
  • cargo +nightly fmt --all -- --check — passed.
  • cargo +nightly clippy --workspace --all-targets -- -D warnings — passed.
  • Added gzip multi-batch byte-equivalence, empty-archive, and truncated-footer tests; existing label precedence, non-finite filtering, and lazy-cutoff coverage passes. A separate real-data check produced identical decoded bytes in three trials.
  • Paired benchmark: Tempo 33c72deb3774a1fa3eb397fdf8baf1f11acbb587, generator/baseline recorder 41bddc328d92b0a9f49966208129c7b8805e7e65, candidate recorder 8f98268140265b6ef4409b78e526985c701f5d39; public-mix, 600 seconds, 100,000 MiB state bloat, 50,000 target TPS, two pairs, Samply on both sides, unchanged 200 ms scraping and node arguments.

Prompted by: @mediocregopher

Stream temporary sample archives through fast gzip so metric recording does not write tens of gigabytes of uncompressed NDJSON during a load test. Preserve all samples, lazy cutoffs, and decoded reporter output.

Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants