Skip to content

executor, store/copr: budget merged Analyze batches - #70003

Closed
0xTars wants to merge 6 commits into
pingcap:masterfrom
0xTars:tars/analyze-two-phase
Closed

executor, store/copr: budget merged Analyze batches#70003
0xTars wants to merge 6 commits into
pingcap:masterfrom
0xTars:tars/analyze-two-phase

Conversation

@0xTars

@0xTars 0xTars commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: ref #67449

Problem Summary:

Full-sampling Analyze batching could multiply physical scan and retained
collector concurrency by the batch width. Merged child execution details could
also be consumed twice: once from child acknowledgements and again from the
outer response.

This draft follows up on #69686 with the bounded-fanout and single-owner
accounting changes found during coordinated review.

What changed and how does it work?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Validation:

  • ./tools/check/failpoint-go-test.sh pkg/executor -run '^TestAnalyzeBatchScanBudget$'
  • ./tools/check/failpoint-go-test.sh pkg/store/copr -run '^(TestHandleBatchCopResponseMergedAndUnansweredTasks|TestRUEMA)$'
  • ./tools/check/failpoint-go-test.sh pkg/store/copr -race -run '^TestRUEMA$'
  • make bazel_prepare
  • make lint
  • git diff --check

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Support resource-bounded batching for full-sampling Analyze requests.

0xPoe and others added 5 commits July 23, 2026 10:42
Capture the two boundaries that later commits must change or preserve: analyze requests do not yet join store batching, while batched tasks without a per-task response remain eligible for retry.

Signed-off-by: 0xPoe <poe.liu@pm.me>
Forward tidb_store_batch_size to V2 full-sampling Analyze and admit those internal requests to store batching without row-count hints. Leave KeepOrder unset because the consumers derive correlation and other statistics from the collected samples rather than scan order.

Signed-off-by: 0xPoe <poe.liu@pm.me>
Opt Analyze into kvproto batch-task data merging and consume explicit per-task acknowledgements without emitting empty responses. A task without an acknowledgement remains on the existing retry path, so mixed-version stores cannot cause silent statistics loss.

Temporarily replace kvproto with the revision from pingcap/kvproto#1497 until that protocol change merges.

Signed-off-by: 0xPoe <poe.liu@pm.me>
Fit full-sampling analyze requests into the configured scan concurrency:
a batched group costs its physical width (children + 1), so shrink the
outer iterator concurrency and cap children at TiKV's supported maximum
(4) instead of exempting analyze from admission entirely. concurrency=1
degenerates to unbatched requests.

Adopt the single-owner execution-details contract: the store's finalizer
aggregates a merged child's details into the main response, which TiDB
collects exactly once in handleCopResponse; stop re-collecting the
per-task ACK copy, which would double count scan/time totals in runtime
stats and runaway tracking. Depends on the TiKV finalizer aggregation
landing in the same change set.

Consolidate the RUEMA tests into subtests to keep pkg/store/copr within
the toomanytests nogo limit (47/50).

Signed-off-by: 0xTars <1196089730@qq.com>
Signed-off-by: 0xTars <1196089730@qq.com>
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/needs-triage-completed labels Jul 23, 2026
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8522bb67-d3bf-4e53-96d3-c6fe7dd017ba

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jul 23, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

Hi @0xTars. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot

ti-chi-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cfzjywxk, mjonss for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pingcap-cla-assistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ 0xPoe
❌ 0xTars
You have signed the CLA already but the status is still pending? Let us recheck it.

Signed-off-by: 0xTars <1196089730@qq.com>
@ti-chi-bot

ti-chi-bot Bot commented Jul 23, 2026

Copy link
Copy Markdown

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-tests-checked label, please finished the tests then check the finished items in description.

For example:

Tests <!-- At least one of them must be included. -->

- [x] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code

‼️ Must keep the HTML comments <!-- At least one of them must be included. -->

📖 For more info, you can check the "Contribute Code" section in the development guide.

@0xPoe 0xPoe closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contribution This PR is from a community contributor. do-not-merge/needs-tests-checked do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants