Skip to content

perf: add -snapshots in pbench run for collecting time series metrics - #97

Open
yabinma wants to merge 2 commits into
mainfrom
feat/phase1-json-snapshots
Open

perf: add -snapshots in pbench run for collecting time series metrics#97
yabinma wants to merge 2 commits into
mainfrom
feat/phase1-json-snapshots

Conversation

@yabinma

@yabinma yabinma commented Jul 18, 2026

Copy link
Copy Markdown
Member

Add --snapshots: periodic query-JSON time-series capture

Summary

Today pbench saves a single query JSON after a query finishes — a terminal snapshot.
That's a photo; it can't answer time-based questions (when did throughput collapse,
which stage stalled, is a query in a runaway state). This PR adds an opt-in flag that
periodically captures /v1/query/{id} while the query runs, producing a time-series
of query-JSON frames — a movie instead of a photo.

What's new

  • New pbench run flags (all opt-in; default off):
    • --snapshots — master switch
    • --snapshot-interval (default 30s, 5s floor)
    • --snapshot-max (default 20; 0 = unlimited)
    • --snapshot-fetch-timeout (default = interval)
  • Equivalent inheritable per-stage JSON options (save_json_snapshots, …).
    Precedence: per-stage JSON > CLI flag > default.
  • Frames are written verbatim (raw Presto JSON, no schema change) to a sibling
    <query>.snapshots/snapshot_<seq>_<epochMillis>.json directory — one per query.

Behavior & safety

  • No regression: with --snapshots off, behavior/output is byte-identical to today
    (no .snapshots/ dir, no extra polling).
  • Diagnostic mode, not for timed baseline runs (polling adds coordinator load).
  • Non-overlapping polls; on stop the in-flight fetch is aborted (never delays query
    completion) and partial files are cleaned up.
  • Errors never fail the query; consecutive failures back off exponentially and the
    poller is never permanently disabled. Retention keeps the earliest frame + most recent.

Testing

  • Unit + function tests (mock coordinator), 93.8% coverage on the new poller, -race clean.
  • Cluster-tested on a real ~10-min EXCEEDED_TIME_LIMIT query: regression pass (snapshots
    off → unchanged) and feature pass (20 frames captured, clean stop; the series shows CPU
    climbing while progress flatlines — the runaway-query signature the single terminal file misses).

Also includes a Go toolchain bump 1.26.4 → 1.26.5 to clear govulncheck GO-2026-5856
(crypto/tls); verified 0 code-affecting vulnerabilities under 1.26.5.

@yabinma
yabinma requested a review from ethanyzhang as a code owner July 18, 2026 14:29
@yabinma
yabinma requested a review from wanglinsong July 20, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant