fix(file-based): fire the duplicate-file guard in file-transfer mode

[devin-ai-integration]

Summary

DefaultFileBasedStream's duplicate-file guard is supposed to stop two source files from resolving to one output path. It never fires on the configuration where that actually happens (file transfer with preserve_directory_structure on — the default), because all three of its inputs are wrong:

-def _duplicated_files_names(self, slices):            # 3. cursor-filtered input
-    for file_slice in slices:
-        for file_found in file_slice[self.FILES_KEY]:
-            file_name = path.basename(file_found.uri)  # 1. key the code never uses
+def _duplicated_files_names(self, files: List[RemoteFile]):
+    for file_found in files:
+        # mirrors AbstractFileBasedStreamReader._get_file_transfer_paths
+        file_name = (
+            file_found.source_file_relative_path.lstrip("/")
+            if self.preserve_directory_structure
+            else path.basename(file_found.source_file_relative_path)
+        )
 ...
-if slices and not self.preserve_directory_structure:   # 2. never true in the default mode
+if all_files and self.use_file_transfer:

Three things to call out:

• use_file_transfer is the required gate, not preserve_directory_structure. validate_config_transfer_modes.preserve_directory_structure() returns True whenever file transfer is off, which is the only reason the old condition also excluded records mode. Dropping the gate entirely would start failing records-mode connections that have same-named files, and records mode has no output path to collide on.
• The key is source_file_relative_path, not uri. That is the value _get_file_transfer_paths() turns into the staging path and destination object key. source_file_relative_path moves up from UploadableRemoteFile to RemoteFile (same body, return self.uri) so the guard needs no getattr fallback; UploadableRemoteFile inherits it unchanged.
• The guard now evaluates all_files, not files_to_read. With the cursor-filtered list, a duplicate whose partner was synced in an earlier incremental run is invisible and the file is silently overwritten. list_files() is @cached, so this costs no extra listing calls.

DuplicatedFilesError's message hardcoded "Duplicate file names are not allowed if the Preserve Sub-Directories in File Paths option is disabled", which stops being true once the guard runs in both modes; it now names the resolved output path and still lists the conflicting source URIs. FailureType is unchanged (config_error — the user must rename or remove a file).

⚠️ Release risk — this converts silent data loss into a hard sync failure

Connections that are green today while silently overwriting one of two duplicate files will start failing. That is the intended outcome, but it is support-visible: please pair this with a release note and a heads-up to support rather than shipping it quietly.

Resolves https://github.com/airbytehq/oncall/issues/13269:

• https://github.com/airbytehq/oncall/issues/13269

Coverage

With preserve_directory_structure=true and file transfer on:

Case	Before	After
Same basename, different folders — correctly allowed	pass ✅	pass ✅
Same name, same folder (oncall#12872)	missed ❌	fires ✅
Duplicate folder names each holding a same-named file	missed ❌	fires ✅
Filename containing /	missed ❌	fires ✅
Cross-sync collision (preserve=false, incremental)	missed ❌	fires ✅
Google Doc Report vs file Report.docx	missed ❌	covered only once airbytehq/oncall#13268 lands (source-google-drive resolves the Google-native export extension before computing file-transfer paths)
File data + folder data in one parent	❌	not a name collision — a filesystem shape conflict, addressed by #1078 instead

This is change 3 of 3 from airbytehq/oncall#12872. It is independent of #1078 (unique per-file staging directory, still open): that one stops the FileNotFoundException crash, this one makes the remaining destination-key overwrite loud instead of silent.

Test Coverage

Added to unit_tests/sources/file_based/stream/test_default_file_based_stream.py:

• test_duplicate_uris_raise_when_preserving_directory_structure — identical URIs, preserve=True, file transfer on.
• test_duplicate_detected_across_incremental_syncs — list_files() returns A and B, cursor returns only B; must still raise.
• test_guard_compares_source_file_relative_path — distinct uri, colliding source_file_relative_path.
• test_same_basename_different_folders_allowed_when_preserving — regression guard for the correct-behaviour case.
• test_records_mode_never_raises_on_duplicate_uris — use_file_transfer=False must never raise.
• test_when_compute_slices_with_duplicates updated for the new message text.

The first three fail on main with the source changes reverted and the tests retained. Full run: poetry run pytest unit_tests/sources/file_based -q → 799 passed, 1 skipped; ruff check, ruff format, and mypy --config-file mypy.ini airbyte_cdk all clean.

Declarative-First Evaluation

Not applicable — the change is in the Python file-based CDK's stream/reader layer, not in a declarative connector manifest. No declarative component influences how the CDK resolves file-transfer output paths.

Versioning

No connector version bump or changelog entry: this repo publishes from release notes and pyproject.toml's version is dynamic. Not a breaking change under the connector breaking-change definition (no schema, spec, stream, or state change) — but see the release-risk note above, since previously-passing syncs will now fail loudly.

Link to Devin session: https://app.devin.ai/sessions/f5033729977945f5b915e68051d199c2

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1786420148-file-based-duplicate-guard#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1786420148-file-based-duplicate-guard

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 270 tests  +5   4 258 ✅ +5   8m 27s ⏱️ + 2m 24s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit ca42268. ± Comparison against base commit 893632c.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

4 273 tests  +5   4 261 ✅ +5   12m 22s ⏱️ ±0s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit ca42268. ± Comparison against base commit 893632c.

♻️ This comment has been updated with latest results.