Skip to content

dm: keep S3 external ID in import-into source dir#12681

Open
fgksgf wants to merge 1 commit into
pingcap:release-nextgen-202603from
fgksgf:codex/importinto-keep-external-id-source-dir-20260605
Open

dm: keep S3 external ID in import-into source dir#12681
fgksgf wants to merge 1 commit into
pingcap:release-nextgen-202603from
fgksgf:codex/importinto-keep-external-id-source-dir-20260605

Conversation

@fgksgf

@fgksgf fgksgf commented Jun 5, 2026

Copy link
Copy Markdown
Member

Summary

Why

DM import-into uses the same S3 location across dumpling, Lightning local metadata/file reads, and the final TiDB IMPORT INTO SQL. All of these paths need the keyspace role parameters to remain available.

Live dev validation on cluster 10492432391076049734 showed dumpling completed after the dataflow-service fix, but Lightning failed in the Load phase before submitting IMPORT INTO because Mydumper.SourceDir had already been stripped and the local import SDK could not assume the keyspace role.

TiDB IMPORT INTO now allows an explicit S3 external-id, so the correct fix is to keep the original S3 URI end to end instead of stripping it in Tiflow.

How

  • Revert MakeGlobalConfig to always set lightningCfg.Mydumper.SourceDir = cfg.Dir.
  • Remove the S3 external ID stripping helper from DM loader.
  • Keep the backend selection behavior unchanged: LoadModeImportInto still maps to BackendImportInto.

Testing

  • Ran gofmt on changed files.
  • Not run: full go test; this PR was prepared from GitHub contents without a full Tiflow checkout in the current workspace.
  • Runtime validation remains required after the DM/Tiflow image consumes this PR.

Risks and Reviewer Focus

  • Please verify the current TiDB version used by this release branch accepts explicit S3 external-id in IMPORT INTO resource parameters.
  • This change intentionally keeps the S3 URI unchanged for all consumers: dumpling, Lightning local source reads, and TiDB IMPORT INTO.

@ti-chi-bot

ti-chi-bot Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. area/dm Issues or PRs related to DM. labels Jun 5, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lance6716 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pingcap-cla-assistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ti-chi-bot ti-chi-bot Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 5, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the logic that strips the S3 external ID from the source directory when using the BackendImportInto backend in the DM loader. The helper function stripS3ExternalIDFromSourceDir has been removed, and the corresponding unit tests have been updated to verify that the S3 external ID is preserved. No review comments were provided, and there is no additional feedback to offer.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@ti-chi-bot

ti-chi-bot Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

@fgksgf: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-dm-integration-test-next-gen 6afff10 link true /test pull-dm-integration-test-next-gen

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/dm Issues or PRs related to DM. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant