Skip to content

*: revert gRPC stream operation duration metrics (#11048)#11049

Open
ti-chi-bot wants to merge 1 commit into
tikv:release-nextgen-202603from
ti-chi-bot:cherry-pick-11048-to-release-nextgen-202603
Open

*: revert gRPC stream operation duration metrics (#11048)#11049
ti-chi-bot wants to merge 1 commit into
tikv:release-nextgen-202603from
ti-chi-bot:cherry-pick-11048-to-release-nextgen-202603

Conversation

@ti-chi-bot

Copy link
Copy Markdown
Member

This is an automated cherry-pick of #11048

What problem does this PR solve?

Issue Number: close #11040

grpcutil.NewMetricsStream registers histogram children labeled by peer IP with no lifecycle cleanup. Long-running PD/MCS processes accumulate unbounded label series (observed ~840k series from ~18k historical IPs), inflating memory and /metrics scrape cost until scrapes fail.

What is changed and how does it work?

Revert the gRPC stream operation duration metrics introduced in #10254 and the dependent Grafana stream-send panels from #10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.

Check List

Tests

  • No code

Related changes

  • Need to cherry-pick to the release branch

Release note

Revert gRPC stream operation duration metrics to fix unbounded peer-IP label cardinality.

Summary by CodeRabbit

  • Bug Fixes
    • Removed gRPC stream send-duration metrics from server endpoints and Grafana dashboards.
    • Simplified streaming operations by eliminating the associated instrumentation layer.
    • Removed the unsupported gRPC receive error and related utility behavior.
    • Existing request handling and service functionality remain unchanged.

@ti-chi-bot ti-chi-bot added the dco-signoff: yes Indicates the PR's author has signed the dco. label Jul 24, 2026
@ti-chi-bot ti-chi-bot Bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jul 24, 2026
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603 labels Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f4548b44-2228-4ec8-8921-3e578a24db70

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 24, 2026
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 24, 2026
close tikv#11040\n\nRevert the gRPC stream operation duration metrics introduced in tikv#10254 and the dependent Grafana stream-send panels from tikv#10339. This removes `grpcutil.MetricsStream` and the peer-IP-labeled `grpc_stream_operation_duration_seconds` histograms from PD and MCS services, eliminating the unbounded cardinality leak until a safer replacement lands.\n\nSigned-off-by: JmPotato <github@ipotato.me>

(cherry picked from commit 070828c)
Signed-off-by: JmPotato <github@ipotato.me>
@JmPotato
JmPotato force-pushed the cherry-pick-11048-to-release-nextgen-202603 branch from 3b76282 to e32c1cc Compare July 24, 2026 10:06
@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JmPotato, lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-24 09:58:37.51829552 +0000 UTC m=+1571703.554390576: ☑️ agreed by JmPotato.
  • 2026-07-24 10:13:48.198628681 +0000 UTC m=+1572614.234723727: ☑️ agreed by lhy1024.

@JmPotato

Copy link
Copy Markdown
Member

/retest

1 similar comment
@JmPotato

Copy link
Copy Markdown
Member

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. type/cherry-pick-for-release-nextgen-202603

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants