Add opt-in 7-day CPU/heap trend summary from monitoring data - #1049
Open
ash9146 wants to merge 1 commit into
Open
Add opt-in 7-day CPU/heap trend summary from monitoring data#1049ash9146 wants to merge 1 commit into
ash9146 wants to merge 1 commit into
Conversation
Point-in-time snapshots of CPU/heap usage don't show whether a cluster has been trending toward trouble, which limits their usefulness for after-the-fact investigations. This adds an opt-in --includeTrends flag that queries .monitoring-es-* (when present) for a 7-day percentile summary (p50/p95/p99) of CPU and heap usage, broken down both as an overall summary and daily buckets, written to monitoring-trends.json. Filters on type: node_stats to avoid scanning unrelated monitoring doc types sharing the same source_node.name field - verified this brought scanned docs from ~10M/node down to ~55k/node over 7 days on a test cluster, in line with the expected count at a 10s collection interval. Scoped to legacy self-monitoring indices for now; 8.x Metricbeat-based monitoring data streams use a different schema and aren't covered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1048
Changes
CollectMonitoringTrendscommand, gated behind--includeTrends(opt-in, default off)
.monitoring-es-*for a 7-day CPU/heap percentile summary(p50/p95/p99), broken down by node and by day
type: node_statsto avoid scanning unrelated monitoringdoc types — verified this reduced scanned docs from ~10M/node to
~55k/node over 7 days
indices aren't present, or if the query fails/times out (10s timeout)
monitoring data streams use a different schema and aren't covered yet
Testing
Manually verified against a running cluster:
monitoring-trends.jsonis produced without--includeTrendsby_node→by_day→cpu_pct/heap_pct) with--includeTrendstype: node_statsfilter reduced scanned docs asexpected (see [Feature Request] Add opt-in CPU/heap trend summary from monitoring data (7-day avg/p95/p99, daily breakdown) #1048 for numbers)
See #1048 for full design discussion.