Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ src/keboola_agent_cli/
http_base.py # BaseHttpClient - shared retry/backoff + common HTTP infra
client/ # Storage API + Queue API package (X-StorageApi-Token);
# split by endpoint family (storage_tables/storage_files/configs/
# queue/tokens/branches/stream/query/workspaces/misc + _core/_transfer),
# queue/tokens/branches/stream/query/workspaces/editor/misc + _core/_transfer),
# composed into one KeboolaClient via mixins (#520)
manage_client.py # Manage API (X-KBC-ManageApiToken)
ai_client.py # AI Service API (component schemas, Kai)
Expand Down Expand Up @@ -954,7 +954,8 @@ kbagent config new --component-id ID [--name NAME] [--project NAME] [--output-di
# mirrors the pushed encrypted body -- placeholders would overwrite the remote on next push.

# sync: GitOps -- configs as local files. init/pull/push/diff are filesystem-local (no serve REST surface).
kbagent sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing]
kbagent sync init --project ALIAS [--directory DIR] [--git-branching] [--adopt-existing] [--with-workspaces]
# `sync init --with-workspaces` (CLI-25) sets the manifest key `syncWorkspaces`: pull/diff/push/clone then also sync shared SQL workspaces (keboola.sandboxes with no parameters.id and runtime.shared true; Python/R and legacy SQL sandboxes stay skipped), config only (services/_sync_workspace.py). A `push --force` delete removes the workspace's SQL editor sessions (Editor service, client/editor.py) before the config, `push --dry-run --force` lists them, and a plain push holds the delete back under skipped_deletions; clone warns about workspace input tables missing in the target. With --adopt-existing it turns the key on in an existing manifest. Version gate for this entry lives in gotchas.md.
kbagent sync pull --project ALIAS [--all-projects] [--force] [--theirs] [--dry-run] [--with-samples] [--no-storage] [--no-jobs] [--job-limit N] [--branch ID]
# `sync pull` auto-inits: if the target directory has no `.keboola/manifest.json`, pull runs `init` first, so a separate `sync init` is NOT needed for a first checkout. `sync pull --project X -d ./dir` on an empty dir writes the manifest and fetches the configs in one step.
# `sync pull --force` is conflict-aware (since 0.53.0): locally-modified config whose remote is UNCHANGED is preserved (delta stays pushable, never silently re-stamped); a true merge conflict (local AND remote both changed since last pull) aborts (exit 1, SYNC_CONFLICT, --json lists details.conflicts); local-untouched + remote-changed takes remote.
Expand All @@ -966,6 +967,7 @@ kbagent sync push --project ALIAS [--all-projects] [--dry-run] [--force] [--allo
# under skipped_deletions (+ skipped_deletions_reason), also in --dry-run, whose summary.deleted counts only
# what push would delete. A config/row deleted on the remote since the last pull diffs as remote_deleted and
# is never re-created (it lands in skipped). Version gate in gotchas.md.
# sync push workspace delete (CLI-25): in a `syncWorkspaces` tree, a `push --force` that deletes a shared SQL workspace also deletes its SQL editor sessions (every user's, push branch) and their backend workspaces, which `config restore` does not bring back; `push --dry-run --force` lists them (warnings[] workspace_sessions), a plain push lists the workspace under skipped_deletions and touches no session. `--force` is destructive-class (FLAG_ESCALATIONS `sync.push --force`), so `--deny-destructive` / a cli:destructive deny blocks it while a plain push stays write-class. Version gate for this entry lives in gotchas.md.
# sync push (since 0.91.0, #686): the manifest baseline `pull_config_hash` is stamped from the API
# response (or a read-back), never from disk -- push-deployed multi-statement SQL transformations
# (and anything disabled in the UI whose local YAML lacks `is_disabled`) no longer show permanent
Expand Down
2 changes: 2 additions & 0 deletions docs/TUTORIAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,6 +512,8 @@ git add -A && git commit -m "initial sync"
`manifest.json`'s `ignoredComponents` field (since 0.91.0) lets you exclude
project-specific components from every sync operation, on top of the
always-ignored `keboola.sandboxes` and `keboola.mcp-server-tool`.
`sync init --with-workspaces` *(since vNEXT)* opts a tree in to syncing its
shared SQL workspaces (`keboola.sandboxes`), config only.

What you end up with on disk:

Expand Down
5 changes: 4 additions & 1 deletion plugins/kbagent/agents/keboola-expert.md
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,10 @@ its absence is NOT a promise the entry is version-independent (see §1 Rule 6).
action `"ignored"`, distinct from `"removed"`); a stale local dir for an
already-ignored component can never classify as `DELETED` -- so
delete-dir-then-push is safe for those, but on <= 0.90.1 it still deletes
the config in production.
the config in production. Exception (vNEXT+): a manifest with
`"syncWorkspaces": true` (`sync init --with-workspaces`) syncs shared SQL
workspaces, config only; a `push --force` delete of one also deletes its SQL
editor sessions (every user's); run `push --dry-run --force` first, it lists them.
- **Native types**: `--column amount:NUMBER(18,2)` passes through; `BOOLEAN`
defaults must be lowercase; `INTEGER(10)` is invalid (use `NUMBER(3,0)`);
`--not-null` / `--default` must name a defined `--column`. In a dev branch
Expand Down

Large diffs are not rendered by default.

39 changes: 39 additions & 0 deletions plugins/kbagent/skills/kbagent/references/gotchas.md
Original file line number Diff line number Diff line change
Expand Up @@ -5523,3 +5523,42 @@ drops manifest entries whose config is gone from the remote) are closed:
whose first clone run failed part-way can report the configs it never
created as `remote_deleted`. Delete that target directory and run the clone
again.

## `sync` can sync shared SQL workspaces, opt-in per tree (CLI-25)

*(since vNEXT)* `keboola.sandboxes` is no longer skipped when the manifest sets
`"syncWorkspaces": true` (`sync init --with-workspaces`, or
`sync init --adopt-existing --with-workspaces` for an existing tree). Without
the key nothing changes. Full rules: `sync-workflow.md` > "Shared SQL
workspaces".

- **Only shared SQL workspaces.** A `keboola.sandboxes` config with no
`parameters.id` and `runtime.shared: true`. Python/R workspaces and legacy
SQL sandboxes carry `parameters.id` and stay skipped. `keboola.mcp-server-tool`
stays ignored. An `ignoredComponents` entry for `keboola.sandboxes` wins.
- **Config only.** Push writes the Storage configuration, never a job, a SQL
editor session or a table load. A `parameters.backendSize` change gets a
`workspace_backend_size` warning: an open session keeps its old size.
- **A delete is not only a config delete.** `push --force` deletes the
workspace's SQL editor sessions (every user's, in the push branch) before the
config, and that drops their backend workspaces, which `config restore` does
not bring back. Check `push --dry-run --force` first: its `workspace_sessions`
warnings list the session ids. If the sessions cannot be listed or deleted,
the config stays. A plain push deletes neither: the workspace is listed under
`skipped_deletions`, like any other deletion, and `skipped_deletions_reason`
says that `--force` also deletes the sessions.
- **`sync clone`** adds a `workspace_input_tables_missing` warning per cloned
workspace whose input tables do not exist in the target (clone creates
buckets, never tables).
- **`sync push --force` is destructive-class** (operation `sync.push --force`
in `permissions list`): a policy denying `cli:destructive`, or
`--deny-destructive`, blocks it, while a plain `sync push` stays write-class.
An allow-list that names only `sync.push` now blocks `sync push --force`
too (also in trees without `syncWorkspaces`): add
`--allow "sync.push --force"` or a glob such as `sync.*`. The same applies to
a default-allow policy that denies `cli:write` and allows `sync.push`.
- **Removing the key** makes the next `sync pull` drop the workspace entries
with action `ignored`, except a workspace edited locally and not pushed: pull
(also `--force`) keeps it and reports it as `skipped`; only `--theirs`
deletes it. A `kbc` manifest save removes the key too (`kbc` writes back only
the keys it knows).
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ The agent can still pull configs and view diffs, but cannot push changes back. N
kbagent permissions set --mode allow --deny "cli:destructive"
```
Blocks `branch.delete`, `workspace.delete`, `config.delete`. The agent can still create and modify resources.
*(since vNEXT)* It also blocks `sync push --force` (operation `sync.push --force`, a flag escalation like `auth.logout --remove-projects`): a forced push of a tree that syncs SQL workspaces deletes their SQL editor sessions and workspaces. A plain `sync push` stays write-class and allowed.

### Allow only specific commands (strict allowlist)
```bash
Expand All @@ -87,6 +88,8 @@ kbagent permissions set --mode deny \
```
Everything else is blocked. This is the most restrictive approach.

*(since vNEXT)* `sync push --force` is checked as its own operation, `sync.push --force`. An allow-list that names only `sync.push` allows a plain push and blocks a forced push (exit 6). To allow a forced push, add `--allow "sync.push --force"`, or use a glob such as `sync.*`. The same is true for a default-allow policy that denies `cli:write` and allows `sync.push`.

## Checking permissions before acting

```bash
Expand Down
76 changes: 75 additions & 1 deletion plugins/kbagent/skills/kbagent/references/sync-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,11 @@ internal state:
- **Always ignored** -- `keboola.sandboxes` (Workspaces API) and
`keboola.mcp-server-tool` (the Keboola MCP server auto-creates one empty
workspace-record config per project it touches, `configuration: {}`, name
like `mcp-workspace-<hex>`). This is a hardcoded floor; no flag disables it.
like `mcp-workspace-<hex>`). This is a hardcoded floor. The one exception
*(since vNEXT)*: the manifest key `syncWorkspaces` takes `keboola.sandboxes`
off the list for its shared SQL workspaces, see
[Shared SQL workspaces](#shared-sql-workspaces). `keboola.mcp-server-tool`
stays ignored.
- **Project-configurable** -- the manifest field `ignoredComponents` in
`.keboola/manifest.json` adds project-specific exclusions on top of the
hardcoded list, without waiting for an upstream kbagent release:
Expand All @@ -484,6 +488,76 @@ internal state:
- **Un-ignoring** a component: remove it from `ignoredComponents` and run
`sync pull` again -- it re-materializes like any newly-tracked config.

## Shared SQL workspaces

*(since vNEXT)* A tree can sync the shared SQL workspaces (Snowflake,
BigQuery) of a project. It is opt-in per tree, with the manifest key
`syncWorkspaces`:

```bash
kbagent sync init --project prod --with-workspaces # new tree
kbagent sync init --project prod --adopt-existing --with-workspaces # existing tree
```

or set `"syncWorkspaces": true` in `.keboola/manifest.json`. Without the key,
sync skips `keboola.sandboxes` exactly as before.

- **Scope.** Only `keboola.sandboxes` configs WITHOUT `parameters.id` and with
`runtime.shared: true`. A config with `parameters.id` is a Python/R
(container) workspace, or a legacy SQL sandbox from before the SQL editor;
both stay skipped. The official CLI tells SQL from Python/R by the same key.
A non-shared workspace is visible only to its creator in the UI, so pull
does not fetch it. A workspace already tracked stays tracked when someone
turns `runtime.shared` off: diff reports the change as `remote_modified`.
- **Pull / diff.** A normal `_config.yml`: `parameters.blocks` (the SQL
scripts), `parameters.backendSize`, `input` / `output` (including
`read_only_storage_access` when it is off), and under
`_configuration_extra` the `runtime.shared` flag and the
`shared_code_id` / `shared_code_row_ids` / `variables_id` /
`variables_values_id` links of a workspace created from a transformation.
The config holds no credentials.
- **Push create / update** writes the Storage configuration only: no Queue
job, no SQL editor session, no table load. The UI creates the session when
a user opens the workspace. When `parameters.backendSize` changes, push adds
a `workspace_backend_size` warning: an existing session keeps its size, the
new size applies only to a session created later. Dev branches work the
same way (config only).
- **Push delete** needs `--force`, like every delete: a plain push lists the
workspace under `skipped_deletions` and touches neither its sessions nor its
configuration; `skipped_deletions_reason` then says what `--force` also
deletes. `push --force` first deletes the workspace's SQL editor
sessions in the push branch, of every user, then the configuration, like
`kbc remote workspace delete`. Deleting a session also drops its backend
workspace, which a config restore does not bring back. When the sessions
cannot be listed, or one cannot be deleted (for example it is still
initializing), push keeps the configuration and reports the error. The
deleted session ids are in `pushed_details[].deleted_session_ids`.
`push --dry-run --force` adds one `workspace_sessions` warning per deleted
workspace with `session_count` and `session_ids`; a plain `push --dry-run`
previews no session delete. A tracked workspace whose
config now has `parameters.id` (backed by a Data Science app) is not
deleted: push reports a `VALIDATION_ERROR`, and `push --dry-run --force`
reports `workspace_delete_refused` for it instead of a session list. When
the config delete fails after the sessions were deleted, the error names
those sessions. `sync push --force` needs the
destructive permission class (`--deny-destructive` blocks it).
- **Clone** creates the workspace configs like other configs; `bucket_map`
rewrites their input mapping. Clone creates buckets, never tables, so the
clone result carries one `workspace_input_tables_missing` warning per
workspace whose input tables do not exist in the target.
- **Turning it off.** Remove the key: the next `sync pull` drops the workspace
entries and their directories with action `"ignored"`. A workspace edited
locally and not pushed is kept instead (entry and directory), reported with
action `"skipped"` and the reason; plain pull and `--force` both keep it,
only `--theirs` deletes it. Diff and push ignore the kept entry, so set the
key again and push to apply the edit. An `ignoredComponents` entry for
`keboola.sandboxes` wins over the key.
- **Mixed trees.** `kbc` reads the manifest with unknown keys ignored, so the
key does not break it. A manifest save by `kbc` writes only the keys `kbc`
knows, so it removes `syncWorkspaces`. `kbc` itself always ignores
`keboola.sandboxes`: it logs a warning for each workspace entry in the
manifest and skips it.

## `sync pull --force` is conflict-aware (since 0.53.0)

`--force` no longer blindly overwrites locally-modified configs. It branches on
Expand Down
4 changes: 3 additions & 1 deletion src/keboola_agent_cli/client/_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

``KeboolaClient`` is assembled here from the per-family mixins (storage tables,
storage files, configs, queue, tokens, branches, merge requests, stream,
query, workspaces, billing, notifications, misc) over the shared
query, workspaces, billing, notifications, editor, misc) over the shared
``_CoreClient`` plumbing base. It stays a single class exposing every
Storage/Queue method at its original signature, so ``keboola_agent_cli.Client`` and its ``.raw`` accessor
are unaffected by the split of the former single-file ``client.py`` into a
Expand All @@ -17,6 +17,7 @@
from .billing import _BillingMixin
from .branches import _BranchesMixin
from .configs import _ConfigsMixin
from .editor import _EditorMixin
from .merge_requests import _MergeRequestsMixin
from .misc import _MiscMixin
from .notifications import _NotificationsMixin
Expand All @@ -43,6 +44,7 @@ class KeboolaClient(
_WorkspacesMixin,
_BillingMixin,
_NotificationsMixin,
_EditorMixin,
_TriggersMixin,
_MiscMixin,
_CoreClient,
Expand Down
21 changes: 21 additions & 0 deletions src/keboola_agent_cli/client/_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ def __init__(self, stack_url: str, token: str, *, http_auth: httpx.Auth | None =
self._sync_actions_client: httpx.Client | None = None
self._billing_client: httpx.Client | None = None
self._notification_client: httpx.Client | None = None
self._editor_client: httpx.Client | None = None
# Lazily built on first Data Streams call (per-device OTLP sources); the
# Stream control plane is a sibling host reachable from this stack+token.
self._stream_client: StreamClient | None = None
Expand Down Expand Up @@ -107,6 +108,10 @@ def _billing_base_url(self) -> str:
def _notification_base_url(self) -> str:
return self._derive_service_url(self._stack_url, "notification")

@property
def _editor_base_url(self) -> str:
return self._derive_service_url(self._stack_url, "editor")

def close(self) -> None:
"""Close the underlying HTTP clients."""
super().close()
Expand All @@ -122,6 +127,8 @@ def close(self) -> None:
self._billing_client.close()
if self._notification_client is not None:
self._notification_client.close()
if self._editor_client is not None:
self._editor_client.close()
if self._stream_client is not None:
self._stream_client.close()

Expand Down Expand Up @@ -229,6 +236,20 @@ def _notification_request(self, method: str, path: str, **kwargs: Any) -> httpx.
method, path, client=client, base_url=self._notification_base_url, **kwargs
)

def _editor_request(self, method: str, path: str, **kwargs: Any) -> httpx.Response:
"""Execute an Editor Service (SQL editor sessions) request with retry.

The editor service is a sibling host derived from the stack URL
(``editor.{stack-suffix}``, the ``editor`` entry of ``GET /v2/storage``);
the sub-client inherits the main client's headers, so the
``X-StorageApi-Token`` auth carries over. The service also accepts a
bearer token, which ``_get_or_create_sub_client`` passes on.
"""
client = self._get_or_create_sub_client("_editor_client", self._editor_base_url)
return self._do_request(
method, path, client=client, base_url=self._editor_base_url, **kwargs
)

def _billing_get(self, path: str, **kwargs: Any) -> httpx.Response:
"""Execute a read-only Billing API request with retry.

Expand Down
Loading