feat(sync): clone recreates the target's storage buckets by default

[soustruh]

Summary

sync clone copies configs, not storage. A cloned config's input/output mappings point at buckets that a fresh target project does not have. Clone now recreates the missing buckets by default. --no-create-buckets disables it.

Why

• A customer who uses only sync to manage dozens of projects reported cloned configs that reference missing buckets.
• A clone must be complete, so bucket creation is opt-out, not opt-in.
• The Go CLI (kbc) did not write storage into the tree at all, so this is new behavior, not a regression fix.

What changed

• Pull records the backend of each bucket in storage/buckets.json. For a linked (shared) bucket, it also records the source.
• Clone reads this export, maps each id through --bucket-map, and creates the missing buckets on the recorded backend.
• Clone links a linked bucket to the same source, under the same id. An empty bucket in its place would stay empty, because nothing in the project writes into a linked bucket. The sharing settings of the source project decide if the target can link it.
• Clone skips existing buckets. It records each API failure, including a refused link, in bucket_errors and continues.
• An export from an older pull does not record linked buckets. Clone creates no bucket from it and records one error in bucket_errors.
• CloneResult has new fields: buckets_created, buckets_skipped, bucket_errors and linked_buckets.
• Clone creates only buckets, at production level, before the config push. The export does not contain tables or data.

Testing

• Unit and CLI tests cover create, skip, --bucket-map, backend, link, refused link, list failure, an export from an older pull, the opt-out, and --branch.
• Live, full clone of a reference into an empty target project: it creates the buckets, links the bucket whose source is shared with the target, and records the bucket whose source is not shared in bucket_errors. A re-run reports no_changes, and --no-create-buckets creates nothing.

Docs

CLAUDE.md, commands/context.py, keboola-expert.md, commands-reference.md, gotchas.md, sync-workflow.md.

[keboola-pr-reviewer-bot]

Verdict: auto_approve (risk 2/5) · profile keboola-mcp-server

Opt-in --create-buckets flag on sync clone, defaults off, ships with unit tests and full doc-drift updates — auto-approve.

Impact flags: possible rollback re-introduction — see Check Run summary.

Concerns:

• tests/test_sync_clone.py: No E2E test added; convention #16 targets new commands, this is a flag (borderline)
• src/keboola_agent_cli/services/_sync_clone.py: Buckets created at production level even when --branch targets a dev branch

Dismissing prior approval — a new commit was pushed and this review was for an earlier SHA. Run @keboola-pr-reviewer-bot review to get a fresh verdict.

[keboola-pr-reviewer-bot]

New commit on 8fb3b78 — dismissed 1 stale bot approval. Comment @keboola-pr-reviewer-bot review when you want a fresh review.

[keboola-pr-reviewer-bot]

Verdict: auto_approve (risk 2/5) · profile _default

Additive, well-tested, backward-compatible feature for sync clone — safe to auto-approve.

Concerns:

• src/keboola_agent_cli/commands/sync.py: Default flips to opt-out bucket creation; existing sync clone callers get new side effects.

[zajca]

Actionable findings from the automated review.

[zajca]

Reviewed by Opus.

source = record.get("source_bucket") cannot tell "not linked" (source_bucket: null, written by this version) apart from "unknown" (the key is missing because an older kbagent pulled the tree). With a legacy export, every linked (shared) bucket in the reference falls into the create_bucket branch. The target gets an empty regular bucket under the linked bucket's id, and the clone reports it as successfully created.

The mistake cannot be fixed by re-running. Once the empty bucket exists, target_id in existing skips it on every later clone, even after the user re-pulls the reference. The configs that read from the bucket then read from an empty bucket that nothing ever writes into. This is exactly the failure the linking logic exists to prevent.

The gotchas.md and sync-workflow.md notes ask users to re-pull first. But bucket creation is now ON by default, and existing customers have many trees pulled by older versions, so relying on documentation is not enough. The code can detect the case cheaply: "source_bucket" not in record (or "backend" not in record) marks a pre-feature export.

Suggested fix: when the export is legacy, create no buckets. Instead, add a bucket_errors entry or warning telling the user to re-pull the reference with the current version (or pass --no-create-buckets). Add a test with a record that has no source_bucket key.

[soustruh]

Commit 69355ad9 fixes this. When a record in storage/buckets.json has no source_bucket key, clone creates no bucket. It records one bucket_errors entry that tells the user to pull the reference again or to use --no-create-buckets. The test test_legacy_export_creates_nothing covers this case.

[zajca]

Reviewed by Opus.

overrides is the file-derived mapping of override data (bucket_map, variable_values, instance_rename), and the docstring and override_counts treat it as such. create_buckets is a boolean behaviour switch placed in the same dict. As a result, the CLI defaults it to True while clone_project reads bool(overrides.get("create_buckets")) and defaults it to False (_sync_clone.py:105).

Any future caller of SyncService.clone_project, such as a server router or MCP tool, silently gets the non-default behaviour. That contradicts the PR's statement that a clone is complete by default and that bucket creation is opt-out.

Make it an explicit keyword parameter on SyncService.clone_project / _sync_clone.clone_project (for example create_buckets: bool = True), so every entry point shares the same default and the overrides dict stays pure data.

[soustruh]

Commit 69355ad9 fixes part of this. The service now uses True as the default for create_buckets, and the CLI always sends the value. So every caller gets a complete clone by default. The test test_service_default_creates_buckets covers this.

The flag stays in overrides. A keyword parameter on SyncService.clone_project would add two code lines to sync_service.py. The file has 1654 code lines, and its frozen limit is 1655. So loc-check would fail. To add the parameter, we must first split sync_service.py. That work is out of scope for this PR.

[zajca]

Reviewed by Sonnet.

write_storage_metadata writes "source_bucket": _linked_source(b) (line 107) into storage/buckets.json, and _linked_source (lines 66-74) reads bucket.get("sourceBucket"). But the only caller that feeds real API data into write_storage_metadata is SyncService._pull at src/keboola_agent_cli/services/sync_service.py:545, which fetches buckets via client.list_buckets_with_metadata() -> self.list_buckets(include="metadata") (src/keboola_agent_cli/client/storage_tables.py:52-58). Everywhere else in this codebase that needs sourceBucket, the code explicitly passes include="linkedBuckets" instead (storage_service.py:2584, lineage_service.py:39, with storage_service.py:413's docstring stating outright that sourceBucket/sourceProject require that include value). Because sync pull only ever requests include="metadata", the Storage API response for every bucket -- including genuinely linked/shared ones -- omits sourceBucket, so _linked_source always returns None and storage/buckets.json.source_bucket is always null for real projects.

Consequently, create_buckets_from_export (same file, ~line 479 source = record.get("source_bucket")) never takes the link_bucket branch for a real pull's export: every linked bucket is treated as an ordinary bucket and created empty via create_bucket, which is exactly the outcome the PR describes as the motivating bug ("An empty bucket in its place would stay empty, because nothing in the project writes into a linked bucket"). The unit tests for this path (test_linked_bucket_records_its_source, test_linked_bucket_is_linked_to_its_source) pass only because they hand-construct a bucket dict with sourceBucket/source_bucket already present, bypassing the real fetch call, so this gap has no test coverage. The shipped docs (gotchas.md, sync-workflow.md) even promise "re-pull the reference before cloning" fixes stale exports -- but a fresh pull on this exact version still won't capture it, since the fetch call was never changed to request linkedBuckets.

Fix: have the pull path fetch buckets with include covering both metadata and linkedBuckets (the client already comma-joins multiple include values elsewhere, e.g. list_tables(include="columns,metadata,buckets")), or add a second fetch, so sourceBucket is actually present when write_storage_metadata runs.

Failure scenario: a user runs sync pull on a reference project with a linked/shared bucket (e.g. in.c-shared sourced from project 42's out.c-origin), then sync clone --source ./golden --target new-project. Because the pulled storage/buckets.json never got source_bucket populated, create_buckets_from_export creates in.c-shared as a brand-new, empty, disconnected bucket via create_bucket instead of linking it via link_bucket. The clone reports it in buckets_created, not linked_buckets, giving no indication anything is wrong, while any config that reads from that bucket in the target project silently gets empty results because nothing in the project ever writes into it.

[soustruh]

This does not match the live API. GET /v2/storage/buckets?include=metadata returns sourceBucket on a linked bucket. A request with no include also returns it. include=linkedBuckets adds the linkedBy field to the source (shared) bucket. This field lists the projects that link the bucket. It does not control sourceBucket.

I verified this on the live API. A full sync pull with this branch wrote source_bucket for both linked buckets of the reference. sync clone then linked the bucket that the source project shares with the target.

Dismissing prior approval — a new commit was pushed and this review was for an earlier SHA. Run @keboola-pr-reviewer-bot review to get a fresh verdict.

[keboola-pr-reviewer-bot]

New commit on 69355ad — dismissed 1 stale bot approval. Comment @keboola-pr-reviewer-bot review when you want a fresh review.

[keboola-pr-reviewer-bot]

Verdict: auto_approve (risk 2/5) · profile _default

Additive, opt-out, well-tested bucket-creation step for sync clone on a production CLI; safe to auto-approve.

[zajca]

No actionable findings were found by the automated review.