Skip to content

branch-4.1: [perf](load) Optimize write performance with improved backpressure control - #67728

Open
sollhui wants to merge 1 commit into
apache:branch-4.1from
sollhui:pick-66847-branch-4.1
Open

branch-4.1: [perf](load) Optimize write performance with improved backpressure control#67728
sollhui wants to merge 1 commit into
apache:branch-4.1from
sollhui:pick-66847-branch-4.1

Conversation

@sollhui

@sollhui sollhui commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Backport #66847 to branch-4.1.

For S3-backed cloud loads, a memtable flush can finish after submitting asynchronous uploads, so flush-running count does not represent downstream upload pressure. This change detects S3 storage and uses the S3 upload queue for backpressure, falling back to the existing flush-running limit when the upload pool is unavailable. HDFS retains flush-count backpressure.

Branch-4.1 adaptation

branch-4.1 does not contain WriteRequestType::GROUP or CloudGroupRowsetBuilder. The row-binlog GROUP-specific changes from the original PR are therefore omitted. CloudRowsetBuilder is final in this branch, so is_s3_storage is non-virtual. Local-storage flush limits remain unchanged.

Release note

Improve load backpressure for S3-backed cloud loads. Higher concurrency can increase memory use.

Check List (For Author)

  • Test: Compilation and tests skipped as requested. clang-format 16 and git diff --check passed.
  • Behavior changed: Yes; S3-backed loads use upload queue pressure. HDFS and local-storage limits remain unchanged.
  • Does this need documentation: No.

@sollhui
sollhui requested a review from yiguolei as a code owner September 9, 2026 09:33
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui

sollhui commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants