metrics: clean up inactive gRPC stream target series

[HunDunDM]

What problem does this PR solve?

gRPC stream metrics retain target label series after clients disconnect, causing /metrics to grow continuously.

Issue Number: Close #11044

What is changed and how does it work?

Track the number of active streams for each (request, target) label pair and delete the corresponding Histogram series when the last stream exits.

Reference counting prevents one connection from deleting metrics still used by another connection from the same target.

Track active gRPC streams and clean up metrics for disconnected targets.

Check List

Tests

• Unit test

Release note

Fix unbounded PD metrics growth caused by stale gRPC stream target series.

Summary by CodeRabbit

• Bug Fixes

  • Improved cleanup of metrics for streaming operations across services.
  • Prevented stale monitoring data and unnecessary resource retention after streams close.
  • Ensured shared stream metrics remain available until all related streams are finished.
  • Added safer handling for streams without metrics instrumentation.

• Monitoring

  • Improved accuracy and lifecycle management of gRPC stream performance metrics.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hundundm for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 092c9f53-d404-4f29-99a5-bc85ba9667e5

📥 Commits

Reviewing files that changed from the base of the PR and between f7db425 and 078f99b.

📒 Files selected for processing (14)

• pkg/mcs/metastorage/server/grpc_service.go
• pkg/mcs/metastorage/server/metrics.go
• pkg/mcs/resourcemanager/server/grpc_service.go
• pkg/mcs/resourcemanager/server/metrics.go
• pkg/mcs/router/server/grpc_service.go
• pkg/mcs/router/server/metrics.go
• pkg/mcs/scheduling/server/grpc_service.go
• pkg/mcs/scheduling/server/metrics.go
• pkg/mcs/tso/server/grpc_service.go
• pkg/mcs/tso/server/metrics.go
• pkg/utils/grpcutil/stream.go
• pkg/utils/grpcutil/stream_test.go
• server/grpc_service.go
• server/metrics.go

---

📝 Walkthrough

Walkthrough

Streaming gRPC metrics now track active label sets and remove histogram labels when the last cleanup-enabled stream closes. RPC handlers defer cleanup callbacks, and tests cover cleanup, shared lifecycles, repeated closes, and persistent legacy metrics.

Changes

Stream metrics lifecycle

Layer / File(s)	Summary
Metrics stream lifecycle API pkg/utils/grpcutil/stream.go	Adds shared active-stream tracking, NewMetricsStreamWithCleanup, and idempotent MetricsStream.Close() cleanup.
Lifecycle validation pkg/utils/grpcutil/stream_test.go	Tests metric removal, shared-target reference counting, repeated closes, and legacy persistent metrics.
Streaming RPC cleanup adoption pkg/mcs/*/server/metrics.go, pkg/mcs/*/server/grpc_service.go, server/metrics.go, server/grpc_service.go	Updates stream helpers to return cleanup callbacks and defers them across streaming RPC handlers, including nil-stream handling for watch global config.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• grpcutil: stream metrics retain peer IP label series after disconnect #11040 — Adds lifecycle tracking and cleanup for histogram label series created by streaming RPC metrics.

Possibly related PRs

• tikv/pd#10254 — Modifies the same gRPC stream metrics instrumentation that this change extends with deferred cleanup.
• tikv/pd#11042 — Directly relates to lifecycle-aware histogram handling in pkg/utils/grpcutil/stream.go.
• tikv/pd#10883 — Modifies the same metastorage Service.Watch implementation.

Suggested labels: ok-to-test

Suggested reviewers: jmpotato, rleungx

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 15.79% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: cleaning up inactive gRPC stream metric target series.
Description check	✅ Passed	It includes the problem, linked issue, change summary, tests, commit message, and release note, with only non-critical template sections omitted.
Linked Issues check	✅ Passed	The PR removes stale target-series metrics for disconnected gRPC streams and covers the Resource Manager and Meta Storage cases from #11044.
Out of Scope Changes check	✅ Passed	The added stream cleanup plumbing and service call-site updates stay aligned with the metrics-series cleanup objective.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.20%. Comparing base (39b6220) to head (078f99b).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11045      +/-   ##
==========================================
- Coverage   79.25%   79.20%   -0.05%     
==========================================
  Files         541      542       +1     
  Lines       76037    76148     +111     
==========================================
+ Hits        60262    60315      +53     
- Misses      11534    11561      +27     
- Partials     4241     4272      +31     

Flag	Coverage Δ
unittests	79.20% <100.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ti-chi-bot]

@HunDunDM: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-unit-test-next-gen-3	078f99b	link	true	/test pull-unit-test-next-gen-3

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[JmPotato]

Hi @HunDunDM — just a heads-up that #11042 seems to be fixing the same issue (stale target / peer-IP label series from stream metrics after disconnect; #11040 / #11044).

Both PRs take a similar approach with refcounting on (request, target) and deleting the series when the last stream goes away. #11042 mainly folds the lifecycle into a collector and hooks cleanup via context.AfterFunc on the stream context, so call sites don't need an explicit defer Close; it also has a bit more test coverage (shared target, reconnect, already-canceled context, concurrency, etc.).

If you get a chance, mind taking a quick look at that one too? Happy to cross-check both sides — this PR is still on our radar as well 🙏