Skip to content

*: revert gRPC stream operation duration metrics#11048

Merged
ti-chi-bot[bot] merged 2 commits into
tikv:masterfrom
JmPotato:revert/fe8e20-stream-metrics
Jul 24, 2026
Merged

*: revert gRPC stream operation duration metrics#11048
ti-chi-bot[bot] merged 2 commits into
tikv:masterfrom
JmPotato:revert/fe8e20-stream-metrics

Conversation

@JmPotato

@JmPotato JmPotato commented Jul 24, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #11040

grpcutil.NewMetricsStream registers histogram children labeled by peer IP with no lifecycle cleanup. Long-running PD/MCS processes accumulate unbounded label series (observed ~840k series from ~18k historical IPs), inflating memory and /metrics scrape cost until scrapes fail.

What is changed and how does it work?

Revert the gRPC stream operation duration metrics introduced in #10254 and the dependent Grafana stream-send panels from #10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.

Check List

Tests

  • No code

Related changes

  • Need to cherry-pick to the release branch

Release note

Revert gRPC stream operation duration metrics to fix unbounded peer-IP label cardinality.

Summary by CodeRabbit

  • Bug Fixes
    • Removed gRPC stream send-duration metrics from server endpoints and Grafana dashboards.
    • Simplified streaming operations by eliminating the associated instrumentation layer.
    • Removed the unsupported gRPC receive error and related utility behavior.
    • Existing request handling and service functionality remain unchanged.

JmPotato added 2 commits July 24, 2026 17:03
tikv#10339)"

This reverts commit 0180647.

Signed-off-by: JmPotato <github@ipotato.me>
This reverts commit fe8e20c.

Signed-off-by: JmPotato <github@ipotato.me>
@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the dco. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The PR removes gRPC stream send-duration instrumentation and wrappers across PD and MCS services, deletes the shared grpcutil support and related error, unregisters the metric, and removes its Grafana panels.

Changes

gRPC stream metrics removal

Layer / File(s) Summary
Remove shared stream instrumentation
pkg/utils/grpcutil/stream.go, pkg/mcs/*/server/metrics.go, server/metrics.go, pkg/errs/errno.go, metrics/grafana/pd.json
The shared stream metrics implementation, registrations, obsolete receive error, and gRPC stream-send dashboard panels are removed.
Remove endpoint stream wrappers
pkg/mcs/*/server/grpc_service.go, server/grpc_service.go
Streaming RPC handlers now operate on the original gRPC streams without metrics wrappers.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

  • tikv/pd#11042: Refactors the same gRPC stream metrics code toward lifecycle-aware cleanup.
  • tikv/pd#11045: Adds lifecycle cleanup to the stream-send Prometheus instrumentation removed here.
  • tikv/pd#10254: Introduced the shared gRPC stream metrics wrappers removed by this PR.

Suggested reviewers: rleungx, okjiang

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The PR removes the peer-IP-labeled stream metrics and wrappers across the affected PD and MCS RPCs, which addresses #11040.
Out of Scope Changes check ✅ Passed The Grafana panel removal, test deletions, and metrics cleanup all support the stated revert and leak fix, with no clear unrelated changes.
Title check ✅ Passed The title is concise and accurately summarizes the main change: reverting gRPC stream operation duration metrics.
Description check ✅ Passed The description covers the problem, change, checklist, and release note, matching the template well enough.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [bufferflies,rleungx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-24 09:10:49.876399722 +0000 UTC m=+1568835.912494778: ☑️ agreed by rleungx.
  • 2026-07-24 09:11:12.985950496 +0000 UTC m=+1568859.022045553: ☑️ agreed by bufferflies.

@JmPotato

Copy link
Copy Markdown
Member Author

/cherry-pick release-nextgen-202603

@ti-chi-bot

Copy link
Copy Markdown
Member

@JmPotato: once the present PR merges, I will cherry-pick it on top of release-nextgen-202603 in the new PR and assign it to you.

Details

In response to this:

/cherry-pick release-nextgen-202603

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@JmPotato

Copy link
Copy Markdown
Member Author

/retest

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.29%. Comparing base (39b6220) to head (62f4676).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11048      +/-   ##
==========================================
+ Coverage   79.25%   79.29%   +0.03%     
==========================================
  Files         541      540       -1     
  Lines       76037    76010      -27     
==========================================
+ Hits        60262    60269       +7     
+ Misses      11534    11502      -32     
+ Partials     4241     4239       -2     
Flag Coverage Δ
unittests 79.29% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ti-chi-bot
ti-chi-bot Bot merged commit 070828c into tikv:master Jul 24, 2026
40 of 43 checks passed
@ti-chi-bot

Copy link
Copy Markdown
Member

@JmPotato: new pull request created to branch release-nextgen-202603: #11049.

Details

In response to this:

/cherry-pick release-nextgen-202603

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@JmPotato
JmPotato deleted the revert/fe8e20-stream-metrics branch July 24, 2026 09:58
JmPotato added a commit to ti-chi-bot/pd that referenced this pull request Jul 24, 2026
close tikv#11040\n\nRevert the gRPC stream operation duration metrics introduced in tikv#10254 and the dependent Grafana stream-send panels from tikv#10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.\n\nSigned-off-by: JmPotato <github@ipotato.me>

(cherry picked from commit 070828c)
Signed-off-by: JmPotato <github@ipotato.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

grpcutil: stream metrics retain peer IP label series after disconnect

4 participants