release-8.5: backport #66290 | tidb-test=release-8.5-20251125-v8.5.4

[yibin87]

This is a backport of #66290 to release-8.5-20251125-v8.5.4.

What problem does this PR solve?

Issue Number: ref #62920

Problem Summary:
Add client network in/out stats in TopSQL. And record topN network usage SQLs alongside current topN CPU usage SQLs. Note one SQL will be added at most once.

What changed and how does it work?

• Add two new session variables InPacketBytes and OutPacketBytes to record the network traffic from/to clients.
• When a query finishes, record the network traffic info to TopSQL stmtstats.
• Pick topN queries with highest network traffic.
• Bump github.com/pingcap/tipb to v0.0.0-20260522023117-6ceadabb61de (merge commit of pingcap/tipb#411) to include the TopSQL network fields protocol.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Collects client network data for topsql to pick topN sql with highest network traffic.

Summary by CodeRabbit

Release Notes

• New Features

  • TopSQL now tracks and reports incoming and outgoing network bytes per statement, providing enhanced visibility into network resource consumption alongside existing CPU metrics.
  • Improved per-session packet byte monitoring for better statement resource consumption analysis.

• Chores

  • Updated dependencies and test infrastructure configurations.

[tiprow]

Hi @yibin87. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR instruments per-session packet byte accounting and integrates it into TopSQL statement profiling. Packet bytes are tracked at the network level, threaded through the TopSQL observer API, and used by the reporter to prioritize high-network-usage statements. Supporting changes include audit test concurrency safety improvements and a tipb dependency upgrade.

Changes

TopSQL Network Byte Profiling

Layer / File(s)	Summary
Session packet byte tracking infrastructure pkg/sessionctx/variable/session.go, pkg/server/conn.go	SessionVars gains atomic InPacketBytes and OutPacketBytes counters; clientConn.readPacket increments InPacketBytes, writePacket increments OutPacketBytes, and dispatch resets both per request.
TopSQL observer API and stats implementation pkg/util/topsql/stmtstats/stmtstats.go	StatementObserver callbacks extended with inNetworkBytes and outNetworkBytes parameters; StatementStatsItem adds NetworkInBytes/NetworkOutBytes counters and merge logic; StatementStats accumulates network bytes alongside execution count and duration.
Executor integration with TopSQL observer pkg/executor/adapter.go	ExecStmt passes session InPacketBytes to observeStmtBeginForTopSQL and OutPacketBytes to observeStmtFinishedForTopSQL so per-statement network bytes are recorded.
TopSQL reporter network-byte filtering pkg/util/topsql/reporter/reporter.go	processStmtStatsData computes k-th largest combined network bytes via QuickSelect; keeps top-k statements by network usage and aggregates others into a summary record.
TopSQL data model and proto mapping pkg/util/topsql/reporter/datamodel.go	TopSQLRecordItem proto now includes StmtNetworkInBytes and StmtNetworkOutBytes; in-code diagrams updated; "others" record appended whenever non-nil.
TopSQL observer and reporter tests pkg/util/topsql/reporter/reporter_test.go, pkg/util/topsql/stmtstats/stmtstats_test.go, pkg/util/topsql/stmtstats/aggregator_test.go, pkg/util/topsql/reporter/BUILD.bazel, pkg/util/topsql/stmtstats/BUILD.bazel	Add/update tests for network-byte accumulation, merging, reporter filtering and end-to-end behavior; update test shard counts.

Audit Plugin Retry Test Concurrency Safety

Layer / File(s)	Summary
Audit plugin retry test concurrency refactor pkg/server/tests/commontest/tidb_test.go	Mutex-guard shared result collection with appendResult/resetResults/getResults helpers; refactor auto-commit and explicit-transaction retry paths to use deterministic failpoints and concurrent workloads; update assertions to use result snapshots.

tipb Dependency Upgrade

Layer / File(s)	Summary
tipb dependency version update go.mod, DEPS.bzl	Update tipb from v0.0.0-20241022082558-0607513e7fa4 to v0.0.0-20260522023117-6ceadabb61de.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• pingcap/tidb#66750: Modifies TestAuditPluginRetrying timeout behavior related to retry observation and assertion.
• pingcap/tidb#66290: Introduces overlapping session packet byte accounting and TopSQL network-metrics integration across the same core files.
• pingcap/tidb#67253: Changes StatementObserver callback signatures and implementations in stmtstats and adapter, directly connected to this PR's API adjustments.

Suggested labels

cherry-pick-approved, type/cherry-pick-for-release-8.5, approved, lgtm

Suggested reviewers

• nolouch
• zimulala
• yudongusa

Poem

🐇 Bytes now flow through sessions bright,
TopSQL catches network's height—
QuickSelect picks the chattiest friend,
While audit tests their races mend.
With tipb upgraded, all is right! 📊

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title refers to a backport of PR #66290 but lacks specifics about the main change (TopSQL network statistics). It's partially related but not sufficiently descriptive of the primary feature being added.	Consider making the title more specific, such as 'Add TopSQL network statistics tracking' or 'TopSQL network in/out bytes collection' to better convey the main change.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description check	✅ Passed	The PR description is comprehensive and follows the template structure with all required sections filled in: issue reference, problem summary, detailed changes, test checklist marked, and release notes provided.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/server/conn.go`:
- Around line 1285-1286: The counters InPacketBytes and OutPacketBytes are only
reset once in dispatch, so multi-statement COM_QUERYs will have later statements
see bytes from earlier ones; modify the dispatch/COM_QUERY handling so
per-statement metrics use a per-statement baseline or are reset before each
statement execution: either move the
cc.ctx.GetSessionVars().InPacketBytes.Store(0) and OutPacketBytes.Store(0) into
the loop that executes each parsed statement, or record a baseline via
cc.ctx.GetSessionVars().InPacketBytes.Load()/OutPacketBytes.Load() before each
statement and compute deltas when you record digest/network metrics (refer to
dispatch and the COM_QUERY execution loop and the symbols InPacketBytes,
OutPacketBytes, and any per-statement metric recording code).
- Around line 502-504: The outbound byte counter is incremented before the
packet write, so failed writes still increase OutPacketBytes; change the logic
in conn.go to call WritePacket (or the method that performs the socket write)
first, check its returned error, and only if err == nil and cc.getCtx() != nil
call cc.ctx.GetSessionVars().OutPacketBytes.Add(uint64(len(data))) to update the
counter; ensure the nil check uses cc.getCtx() and that the increment is moved
after a successful write.

In `@pkg/util/topsql/reporter/reporter_test.go`:
- Line 691: The test mixes fields from different records: currently it sums
data.DataRecords[1].Items[0].StmtNetworkInBytes with
data.DataRecords[0].Items[0].StmtNetworkOutBytes; change the assertion to sum
both StmtNetworkInBytes and StmtNetworkOutBytes from the same record (e.g.,
data.DataRecords[1].Items[0].StmtNetworkInBytes +
data.DataRecords[1].Items[0].StmtNetworkOutBytes) so the comparison uses values
from the same DataRecords index.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: bb55d828-8e10-4827-9b75-0669e286ae39

📥 Commits

Reviewing files that changed from the base of the PR and between 2389a2f and 414e2c1.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (16)

• go.mod
• pkg/executor/adapter.go
• pkg/planner/core/find_best_task.go
• pkg/planner/core/plan_to_pb.go
• pkg/server/conn.go
• pkg/server/tests/commontest/tidb_test.go
• pkg/sessionctx/variable/session.go
• pkg/util/execdetails/execdetails.go
• pkg/util/topsql/reporter/BUILD.bazel
• pkg/util/topsql/reporter/datamodel.go
• pkg/util/topsql/reporter/reporter.go
• pkg/util/topsql/reporter/reporter_test.go
• pkg/util/topsql/stmtstats/BUILD.bazel
• pkg/util/topsql/stmtstats/aggregator_test.go
• pkg/util/topsql/stmtstats/stmtstats.go
• pkg/util/topsql/stmtstats/stmtstats_test.go

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 15 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-8.5-20251125-v8.5.4@2389a2f). Learn more about missing BASE report.

Additional details and impacted files

@@                       Coverage Diff                        @@
##             release-8.5-20251125-v8.5.4     #68590   +/-   ##
================================================================
  Coverage                               ?   57.1321%           
================================================================
  Files                                  ?       1784           
  Lines                                  ?     637041           
  Branches                               ?          0           
================================================================
  Hits                                   ?     363955           
  Misses                                 ?     248624           
  Partials                               ?      24462           

Flag	Coverage Δ
integration	37.0987% <15.0000%> (?)
unit	72.5666% <87.5000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9278% <0.0000%> (?)
parser	∅ <0.0000%> (?)
br	52.9240% <0.0000%> (?)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yibin87]

/retest

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yibin87]

/test mysql-test

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yibin87]

/test mysql-test

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-05-22 10:07:03.300516113 +0000 UTC m=+1093.270681176: ☑️ agreed by zimulala.
• 2026-05-22 10:09:13.932803063 +0000 UTC m=+1223.902968116: ☑️ agreed by XuHuaiyu.

[yibin87]

/test mysql-test

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[yibin87]

/test mysql-test

[yibin87]

/test unit-test

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/test mysql-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tiprow]

@yibin87: PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test.

Details

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XuHuaiyu, yudongusa, zimulala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuHuaiyu,zimulala]
• pkg/sessionctx/variable/OWNERS [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment