Conversation
Stream temporary sample archives through fast gzip so metric recording does not write tens of gigabytes of uncompressed NDJSON during a load test. Preserve all samples, lazy cutoffs, and decoded reporter output. Co-authored-by: Derek Cofausper <256792747+decofe@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stream temporary metric samples through fast gzip instead of writing uncompressed NDJSON during the load. Preserve the configured scrape cadence, sample values, labels, timestamps, lazy cutoffs, and decoded reporter output; no new dependency is required.
In the paired 200 ms experiment, root-disk writes fell from 74.07 / 74.77 GiB to 6.14 / 6.51 GiB per phase, a 91.5% reduction in the two-phase averages. Each phase still records approximately 66 million samples.
Draft: this is not a demonstrated TPS fix. The official verdict is Mixed Results: TPS 7,749 → 9,170 (+18.34%, reported bootstrap half-width 24.53%, neutral); builder P50/P90 are +5.97%/+5.09% and classified bad. Validator gas throughput improves 15.91%; block-time P99 is neutral. Additional unprofiled repetitions are needed before claiming a node-performance improvement.
Profile
tempo-bench-resultsartifact:profile-baseline-1-a.json.gz, node logs, metric archives, cgroup snapshots, andsummary.json. Compression reduces the recorder's large unrelated write stream but adds compression/decompression CPU work; persistence stalls are not eliminated.Verification
cargo test --workspace— passed; one pre-existing sizing test remains ignored.cargo +nightly fmt --all -- --check— passed.cargo +nightly clippy --workspace --all-targets -- -D warnings— passed.33c72deb3774a1fa3eb397fdf8baf1f11acbb587, generator/baseline recorder41bddc328d92b0a9f49966208129c7b8805e7e65, candidate recorder8f98268140265b6ef4409b78e526985c701f5d39; public-mix, 600 seconds, 100,000 MiB state bloat, 50,000 target TPS, two pairs, Samply on both sides, unchanged 200 ms scraping and node arguments.Prompted by: @mediocregopher