[Metricbeat][statsd] Reset timer and histogram metrics on every flush

[TomoZG]

Proposed commit message

[Metricbeat][statsd] Reset timer and histogram metrics on every flush

registry.getMetric resets counters and sets after every flush, but timers (ms) and histograms (h) were never reset. Their count grew monotonically and min, max, mean, stddev and the percentiles aggregated every measurement ever recorded, dropping only once the metric went stale and was evicted by ttl. The resulting series is uninterpretable: a consumer cannot tell a new window from an update from a stale re-report.

• metrics.Histogram: Clear() after snapshotting, which resets the count and the ExpDecaySample reservoir.
• *samplingTimer: the count came from the meter, which is monotonic and has no Clear(), so the per-interval count is now tracked in the timer and reset with the sample. Sample rate extrapolation (@0.1) is preserved. The moving average rates (1m_rate, 5m_rate, 15m_rate, mean_rate) are deliberately left alone: they are defined over fixed time windows, not the flush interval.
• A timer or histogram with no measurements in an interval is not reported for that interval, so clearing the sample does not emit zeroed out min/max/mean. It stays in the registry and is reported again as soon as new measurements arrive, or is dropped at ttl.

ttl controls how long a metric is reported after it was last recorded, not what a single document aggregates, and every other metric type in the module already aggregates one flush interval.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (no configuration change)
• I have added tests that prove my fix is effective or that my feature works (run under -race)
• I have added an entry in ./changelog/fragments

Disruptive User Impact

ms and h values change, so dashboards built on the old behaviour need adjusting:

• count now describes a single flush interval. Charts that used differences to undo the monotonic count should use sum/avg over count instead; that workaround was already unreliable, giving a negative step whenever a metric expired and no value at all for an isolated data point.
• Idle timers and histograms no longer emit a document per period until their ttl expires. Anything counting documents sees fewer of them; anything averaging min/mean stops being dragged down by stale repeats.

Counters, gauges and sets are unchanged.

How to test this PR locally

go test -race -run 'Timer|Histogram|Counter|SetReset|TagsCleanup' ./x-pack/metricbeat/module/statsd/server/...

TestTimerReset, TestTimerSampledReset and TestHistogramReset fail on main with the symptom from #39987.

End to end: build metricbeat from this branch and from main, run the configuration from #39987 (period: 10s, default ttl), send processing_item:3000|h every 3s twenty times, then stay idle for 40s. statsd.processing_item.count per emitted document:

build	1	2	3	4	5	6	7	8
main	4	7	10	14	17	19	19	19
this branch	4	3	3	4	3	3	—	—

4, 7, 10 are the values quoted in #39987, and documents 7 and 8 are stale repeats emitted after the traffic stopped. On this branch each document covers one period, the counts sum to 20 — exactly the packets sent, so the reset drops no measurements — and nothing is emitted during the idle window. (main totals 19 because its first packet raced the UDP socket bind at startup; a harness artifact.)

Related issues

• Closes Metricbeat – Statsd Module issue with count value when using timer/histogram metric type #39987
• Closes Handling Timer and Histogram Metrics Reset During Flush with TTL Considerations #41002

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @TomoZG? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[TomoZG]

This is a follow-up to #41002, the backlog issue opened to analyse timer/histogram handling in the statsd module.

Could a maintainer please add:

• Team:Obs-InfraObs (the label on Metricbeat – Statsd Module issue with count value when using timer/histogram metric type #39987)
• backport-9.5

On the backport scope: this is a bug fix, but I've described the effect under "Disruptive User Impact" above.

On the conclusion in #39987 that this is expected behaviour: the period/ttl mechanics described there are accurate, but they govern when a metric is reported, not what it aggregates. Counters and sets are already reset on every flush in registry.go; timers and histograms were the only types that weren't. This PR makes them consistent with the types that were already correct.

[vishaangelova]

Because our docs are cumulative for all 9.x version, we need to declare the earliest version in which the new behavior applies.

Suggested change

	: It defines how long a metric will be reported after it was last recorded. Irrespective of the given ttl, metrics will be reported at least once. A ttl of zero means metrics will never expire. It controls how long a metric is kept, not what it aggregates: every flush reports what was recorded since the previous one, and a timer or histogram that received no measurements is not reported for that flush.
	: It defines how long a metric will be reported after it was last recorded. Irrespective of the given ttl, metrics will be reported at least once. A ttl of zero means metrics will never expire.
	{applies_to}`stack: ga 9.6+` It controls how long a metric is kept, not what it aggregates: every flush reports what was recorded since the previous one, and a timer or histogram that received no measurements is not reported for that flush.

So if this PR is backported to all active 9.x branches, then the tag would need to be:

{applies_to}`stack: ga 9.4+`

[TomoZG]

Thanks for the clarification! I left the ga 9.6+ version you suggested because there still is a small but "Disruptive User Impact". If you think ga 9.4+ is more suitable, let me know. Also, fragment defines this PR as a "bug-fix", I can switch it to "breaking-change" if you'd prefer.

[vishaangelova]

Thanks, that will do for now. I'll leave it to the developer team to decide about the backport and release note labels.

[botelastic]

This pull request doesn't have a Team:<team> label.