grpcutil: bound stream metrics peer IP label lifecycle

[JmPotato]

What problem does this PR solve?

Issue Number: Close #11040

gRPC stream duration metrics retain a histogram child for every historical (request, peer IP) label pair until the process restarts, causing stale series to accumulate in /metrics after clients disconnect.

What is changed and how does it work?

Replace the raw HistogramVec with a lifecycle-aware collector that tracks active streams by request and peer IP.

Acquire the histogram child and increment its reference count under one lock. Register context.AfterFunc on the server stream context to decrement the count and delete the child after the last matching stream ends.

Keep the HistogramVec private so child creation and deletion cannot bypass the lifecycle lock. Preserve the existing metric names, labels, and Send hot path.

Performance

These are package-level microbenchmarks of the metrics wrapper, not end-to-end gRPC benchmarks.

Environment: Apple M4 (10 cores, 24 GiB), macOS 27.0 (darwin/arm64), Go 1.26.5, GOMAXPROCS=10. The base revision is f7db42521 and the PR revision is 033149448. Each benchmark used a fixed iteration count and 10 independent runs, summarized with benchstat. The common benchmark body was identical between revisions; revision-specific adapters only account for the base revision having no release state to await.

Operation	Iterations per run	Base	PR	Difference
Steady-state Send	5,000,000	64.51 ns/op ±2%, 0 B/op, 0 allocs/op	65.00 ns/op ±2%, 0 B/op, 0 allocs/op	No significant difference (p=0.190)
Shared-label NewMetricsStream	100,000	126.6 ns/op ±5%, 80 B/op, 4 allocs/op	373.5 ns/op ±4%, 624 B/op, 10 allocs/op	+246.9 ns/op, +544 B/op, +6 allocs/op
Shared-label cancel -> asynchronous refcount release	100,000	31.01 ns/op ±5%, 0 B/op, 0 allocs/op	571.05 ns/op ±1%, 0 B/op, 0 allocs/op	+540.0 ns/op
Single-owner create -> cancel -> delete/recreate	10,000	138.7 ns/op ±7%, 80 B/op, 4 allocs/op	2,125.5 ns/op ±2%, 2,184 B/op, 26 allocs/op	+1.987 µs/op, +2,104 B/op, +22 allocs/op

The steady-state Send path has no measurable regression, which is consistent with its source code being unchanged. The shared-label setup and release benchmarks isolate lifecycle bookkeeping: all streams use the same (request, target) key, and only the final cancellation deletes the histogram child. Their combined incremental cost is approximately 0.79 µs and 544 B with 6 allocations per stream lifecycle.

The single-owner benchmark represents short-lived stream churn: every PR iteration transitions the refcount from 1 to 0, deletes the child, and recreates it on the next iteration. Its additional cost is approximately 1.99 µs, 2.1 KiB, and 22 allocations per complete stream lifecycle. This remains a one-time lifecycle cost rather than a per-message cost.

For the PR revision, cancellation benchmarks wait for the asynchronous context.AfterFunc release to complete, so the reported values include goroutine scheduling and cleanup completion. The gRPC teardown path itself does not synchronously wait for this cleanup.

Limitations: These microbenchmarks do not measure complete RPC latency or throughput, concurrent reconnect storms, Prometheus scrape contention, or the expected memory and scrape-size reduction from removing stale series.

Benchmark semantics and commands

• Send: the wrapper and observer are initialized before timing; the timed region only calls Send.
• Shared-label setup: cancellable contexts and stream stubs are prepared before timing; the timed region only calls NewMetricsStream.
• Shared-label cancellation: 100,000 streams share one label key; the PR benchmark waits for each refcount decrement, and only the final iteration calls DeleteLabelValues.
• Single-owner lifecycle: each iteration creates the only stream for one label key, cancels it, and waits for refcount 1 -> 0; the PR deletes and recreates the child on every iteration, while the base retains and reuses it.
• The temporary benchmark harness was applied to detached worktrees for both revisions and removed afterward.

GOMAXPROCS=10 go test ./pkg/utils/grpcutil -run '^$' -bench '^BenchmarkMetricsStreamSend$' -benchmem -benchtime=5000000x -count=10
GOMAXPROCS=10 go test ./pkg/utils/grpcutil -run '^$' -bench '^BenchmarkNewMetricsStream$' -benchmem -benchtime=100000x -count=10
GOMAXPROCS=10 go test ./pkg/utils/grpcutil -run '^$' -bench '^BenchmarkMetricsStreamCancel$' -benchmem -benchtime=100000x -count=10
GOMAXPROCS=10 go test ./pkg/utils/grpcutil -run '^$' -bench '^BenchmarkMetricsStreamSingleOwnerLifecycle$' -benchmem -benchtime=10000x -count=10

Results were compared with golang.org/x/perf/cmd/benchstat at v0.0.0-20260709024250-82a0b07e230d.

Check List

Tests

• Unit test

Side effects

• Increased code complexity
• Breaking backward compatibility: NewGRPCStreamSendDuration now returns *StreamSendDurationCollector, and NewMetricsStream accepts that collector instead of *prometheus.HistogramVec.

Related changes

• Need to cherry-pick to release-nextgen-202603

Release note

Prevent disconnected gRPC clients from leaving stale peer IP label series in PD and microservice metrics endpoints.

Summary by CodeRabbit

• Enhancements

  • Upgraded gRPC stream send-duration metrics to use a dedicated collector that tracks active label sets and automatically removes stale series.
  • Metrics are now tied to stream context lifecycle for reliable cleanup across cancellation, reconnection, and concurrent streams.
  • Nil streams no longer create stream metrics.

• Tests

  • Expanded coverage for stream lifecycle, concurrent initialization/reference counting, metrics cleanup on cancellation, label isolation across concurrent streams, reconnect behavior, already-canceled contexts, and concurrent stress scenarios.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cabinfeverb for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 535cd6a3-0472-4f75-bcfc-8e26cbd4e8d9

📥 Commits

Reviewing files that changed from the base of the PR and between 0331494 and 8ea77ff.

📒 Files selected for processing (2)

• pkg/utils/grpcutil/stream.go
• pkg/utils/grpcutil/stream_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

• pkg/utils/grpcutil/stream.go
• pkg/utils/grpcutil/stream_test.go

---

📝 Walkthrough

Walkthrough

gRPC stream send-duration metrics now use a reference-counted collector that removes request/target histogram series after the last matching stream ends. NewMetricsStream ties observer release to stream context cancellation, with expanded tests for lifecycle, labels, reconnection, nil streams, and concurrency.

Changes

Stream metrics lifecycle

Layer / File(s)	Summary
Reference-counted metric collector pkg/utils/grpcutil/stream.go	Introduces StreamSendDurationCollector, which owns the histogram, tracks (request, target) references, and deletes label values after the final release.
Stream lifecycle integration and validation pkg/utils/grpcutil/stream.go, pkg/utils/grpcutil/stream_test.go	Updates NewMetricsStream to acquire and context-release observers, and adds tests for cancellation, label isolation, reconnection, nil streams, and concurrent lifecycles.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant StreamHandler
  participant NewMetricsStream
  participant StreamSendDurationCollector
  participant PrometheusHistogram
  StreamHandler->>NewMetricsStream: create metrics stream
  NewMetricsStream->>StreamSendDurationCollector: acquire(request, target)
  StreamSendDurationCollector->>PrometheusHistogram: bind labeled observer
  StreamHandler->>NewMetricsStream: send messages
  NewMetricsStream->>StreamSendDurationCollector: release on context cancellation
  StreamSendDurationCollector->>PrometheusHistogram: delete labels when last stream releases

Loading

Possibly related PRs

• tikv/pd#11045: Adds related cleanup plumbing for stale gRPC stream target metric series.
• tikv/pd#10254: Introduced the gRPC stream send-duration instrumentation APIs updated here.
• tikv/pd#10339: Updated the stream metrics target label handling used by this collector.

Suggested reviewers: rleungx

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: lifecycle management for gRPC stream metrics peer IP labels.
Description check	✅ Passed	The PR description follows the template with issue number, change summary, tests, side effects, related changes, and release note.
Linked Issues check	✅ Passed	The changes implement bounded histogram-child lifecycle with refcounted cleanup on stream end, matching issue #11040's requirements.
Out of Scope Changes check	✅ Passed	The modified code stays focused on stream metric lifecycle handling and related tests, with no clear unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[JmPotato]

/retest

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.22%. Comparing base (39b6220) to head (0331494).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11042      +/-   ##
==========================================
- Coverage   79.25%   79.22%   -0.04%     
==========================================
  Files         541      542       +1     
  Lines       76037    76124      +87     
==========================================
+ Hits        60262    60306      +44     
- Misses      11534    11555      +21     
- Partials     4241     4263      +22     

Flag	Coverage Δ
unittests	79.22% <100.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.