Skip to content

planner,expression,util: scatter hotspot index in certain scenes by expression index#30659

Merged
ti-chi-bot merged 37 commits into
pingcap:masterfrom
TonsnakeLin:dev-qps300k
Feb 18, 2022
Merged

planner,expression,util: scatter hotspot index in certain scenes by expression index#30659
ti-chi-bot merged 37 commits into
pingcap:masterfrom
TonsnakeLin:dev-qps300k

Conversation

@TonsnakeLin

@TonsnakeLin TonsnakeLin commented Dec 13, 2021 •
edited by youjiali1995
Loading

Copy link
Copy Markdown
Contributor

Signed-off-by: TonsnakeLin lpbgytong@163.com

What problem does this PR solve?

Issue Number: close #31040

It propose a solution to solve the problem that hot indexes hinder the scalability of TiDB in certain case.
RFC: 84

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains experimental features

Release note

None

@ti-chi-bot

ti-chi-bot commented Dec 13, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • winoros
  • you06

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 13, 2021
@TonsnakeLin TonsnakeLin mentioned this pull request Dec 13, 2021
Closed
7 tasks
cfzjywxk
cfzjywxk reviewed Dec 13, 2021
View reviewed changes
Comment thread planner/core/logical_plan_builder.go Outdated
Comment thread planner/core/rule_predicate_push_down.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
cfzjywxk
cfzjywxk reviewed Dec 13, 2021
View reviewed changes
Comment thread planner/core/rule_predicate_push_down.go Outdated
TonsnakeLin
TonsnakeLin commented Dec 16, 2021
View reviewed changes
Comment thread planner/core/rule_predicate_push_down.go Outdated
TonsnakeLin
TonsnakeLin commented Dec 16, 2021
View reviewed changes
Comment thread util/ranger/detacher.go Outdated
@TonsnakeLin
TonsnakeLin marked this pull request as ready for review December 16, 2021 11:22
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 16, 2021
@zhangjinpeng87

zhangjinpeng87 commented Dec 18, 2021

Copy link
Copy Markdown
Contributor

@TonsnakeLin Is it possible to split this PR into 2 or 3 smaller PRs?

you06
you06 reviewed Dec 20, 2021
View reviewed changes
Comment thread planner/core/rule_column_pruning.go Outdated
@TonsnakeLin

TonsnakeLin commented Dec 21, 2021

Copy link
Copy Markdown
Contributor Author

@TonsnakeLin Is it possible to split this PR into 2 or 3 smaller PRs?
All code of this PR has 1157 lines code, and there are about 550 lines test code. I can split it into a PR that supports integer shard index and a PR testing integer shard index.
@zhangjinpeng1987 @cfzjywxk

cfzjywxk
cfzjywxk reviewed Dec 21, 2021
View reviewed changes
Comment thread expression/builtin_miscellaneous.go Outdated
Comment thread expression/builtin_miscellaneous_test.go Outdated
Comment thread expression/column.go Outdated
Comment thread planner/core/logical_plan_builder.go Outdated
Comment thread planner/core/rule_predicate_push_down.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
you06
you06 reviewed Dec 22, 2021
View reviewed changes

@you06 you06 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also noticed that there are many typos in the comments.

Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
Comment thread util/ranger/detacher.go Outdated
you06
you06 reviewed Dec 22, 2021
View reviewed changes
Comment thread expression/builtin_miscellaneous_test.go Outdated
@TonsnakeLin

TonsnakeLin commented Dec 23, 2021

Copy link
Copy Markdown
Contributor Author

/run unit-test

@you06

you06 commented Dec 23, 2021

Copy link
Copy Markdown
Contributor

/run-unit-test

1 similar comment
@TonsnakeLin

TonsnakeLin commented Dec 23, 2021

Copy link
Copy Markdown
Contributor Author

/run-unit-test

@cfzjywxk

cfzjywxk commented Dec 23, 2021

Copy link
Copy Markdown
Contributor

@TonsnakeLin
If the code change is finished you could trigger the "re-request" review button in the "reviwers" column, or add some a comment to ping the related reviewers.

@TonsnakeLin

TonsnakeLin commented Feb 17, 2022

Copy link
Copy Markdown
Contributor Author

/run-unit-test

1 similar comment
@TonsnakeLin

TonsnakeLin commented Feb 17, 2022

Copy link
Copy Markdown
Contributor Author

/run-unit-test

@youjiali1995

youjiali1995 commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/merge

@TonsnakeLin

TonsnakeLin commented Feb 18, 2022

Copy link
Copy Markdown
Contributor Author

/run-unit-test

1 similar comment
@TonsnakeLin

TonsnakeLin commented Feb 18, 2022

Copy link
Copy Markdown
Contributor Author

/run-unit-test

@cfzjywxk

cfzjywxk commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/merge

@youjiali1995

youjiali1995 commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/run-unit-test

1 similar comment
@TonsnakeLin

TonsnakeLin commented Feb 18, 2022

Copy link
Copy Markdown
Contributor Author

/run-unit-test

@youjiali1995

youjiali1995 commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

Need one approval from @pingcap/tidb-configuration-reviewer
image

@cfzjywxk

cfzjywxk commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/merge

@cfzjywxk

cfzjywxk commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/run-unit-test

@ti-chi-bot

ti-chi-bot commented Feb 18, 2022

Copy link
Copy Markdown
Member

@TonsnakeLin: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@TonsnakeLin

TonsnakeLin commented Feb 18, 2022

Copy link
Copy Markdown
Contributor Author

re-run-mysql-test

@cfzjywxk

cfzjywxk commented Feb 18, 2022

Copy link
Copy Markdown
Contributor

/run-mysql-test

@ti-chi-bot
ti-chi-bot merged commit 6517f3b into pingcap:master Feb 18, 2022
@LittleFall LittleFall mentioned this pull request Feb 25, 2022
Closed
@TonsnakeLin TonsnakeLin mentioned this pull request Mar 28, 2022
Merged
16 tasks
@ti-chi-bot ti-chi-bot mentioned this pull request May 25, 2022
Merged
16 tasks
@chrysan chrysan mentioned this pull request Aug 3, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cfzjywxk cfzjywxk cfzjywxk approved these changes
@winoros winoros winoros approved these changes
@tiancaiamao tiancaiamao Awaiting requested review from tiancaiamao
@tangenta tangenta Awaiting requested review from tangenta
@qw4990 qw4990 Awaiting requested review from qw4990
@bb7133 bb7133 Awaiting requested review from bb7133
@you06 you06 Awaiting requested review from you06
@wjhuang2016 wjhuang2016 Awaiting requested review from wjhuang2016
+1 more reviewer
@youjiali1995 youjiali1995 youjiali1995 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@youjiali1995 youjiali1995

Labels

release-note-none Denotes a PR that doesn't merit a release note. sig/execution SIG execution sig/transaction SIG:Transaction size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/new-feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Using shard index to scatter integer hot index

10 participants

@TonsnakeLin @ti-chi-bot @zhangjinpeng87 @you06 @cfzjywxk @sre-bot @dianqihanwangzi @wjhuang2016 @youjiali1995 @winoros