diff --git a/_data/navigation.yml b/_data/navigation.yml index ffb6bb55..3f8bee18 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -51,130 +51,4 @@ items: title: Set Schedule - url: /cli/ - title: CLI - items: - - url: /cli/installation/ - title: Installation - - url: /cli/getting-started/ - title: Getting Started - - url: /cli/structure/ - title: Structure - - url: /cli/commands/ - title: Commands - items: - - url: /cli/commands/help/ - title: help - - url: /cli/commands/status/ - title: status - - url: /cli/commands/sync/ - title: sync - items: - - url: /cli/commands/sync/init/ - title: init - - url: /cli/commands/sync/pull/ - title: pull - - url: /cli/commands/sync/push/ - title: push - - url: /cli/commands/sync/diff/ - title: diff - - url: /cli/commands/ci/ - title: ci - items: - - url: /cli/commands/ci/workflows/ - title: workflows - - url: /cli/commands/local/ - title: local - items: - - url: /cli/commands/local/create/ - title: create - items: - - url: /cli/commands/local/create/config/ - title: config - - url: /cli/commands/local/create/row/ - title: row - - url: /cli/commands/local/persist/ - title: persist - - url: /cli/commands/local/encrypt/ - title: encrypt - - url: /cli/commands/local/validate/ - title: validate - items: - - url: /cli/commands/local/validate/config/ - title: config - - url: /cli/commands/local/validate/row/ - title: row - - url: /cli/commands/local/validate/schema/ - title: schema - - url: /cli/commands/local/fix-paths/ - title: fix-paths - - url: /cli/commands/remote/ - title: remote - items: - - url: /cli/commands/remote/create/ - title: create - items: - - url: /cli/commands/remote/create/branch/ - title: branch - - url: /cli/commands/remote/create/bucket/ - title: bucket - - url: /cli/commands/remote/file/ - title: file - items: - - url: /cli/commands/remote/file/download/ - title: download - - url: /cli/commands/remote/file/upload/ - title: upload - - url: /cli/commands/remote/job/ - title: job - items: - - url: /cli/commands/remote/job/run/ - title: run - - url: /cli/commands/remote/table/ - title: table - items: - - url: /cli/commands/remote/table/create/ - title: create - - url: /cli/commands/remote/table/upload/ - title: upload - - url: /cli/commands/remote/table/download/ - title: download - - url: /cli/commands/remote/table/preview/ - title: preview - - url: /cli/commands/remote/table/detail/ - title: detail - - url: /cli/commands/remote/table/import/ - title: import - - url: /cli/commands/remote/table/unload/ - title: unload - - url: /cli/commands/remote/workspace/ - title: workspace - items: - - url: /cli/commands/remote/workspace/create/ - title: create - - url: /cli/commands/remote/workspace/delete/ - title: delete - - url: /cli/commands/remote/workspace/detail/ - title: detail - - url: /cli/commands/remote/workspace/list/ - title: list - - url: /cli/commands/dbt/ - title: dbt - items: - - url: /cli/commands/dbt/init/ - title: init - - url: /cli/commands/dbt/generate/ - title: generate - items: - - url: /cli/commands/dbt/generate/profile/ - title: profile - - url: /cli/commands/dbt/generate/sources/ - title: sources - - url: /cli/commands/dbt/generate/env/ - title: env - - url: /cli/github-integration/ - title: GitHub Integration - - url: /cli/devops-use-cases/ - title: DevOps Use Cases - - url: /cli/dbt/ - title: dbt - + title: CLI (moved) diff --git a/cli/commands/ci/index.md b/cli/commands/ci/index.md index a6bff8af..15293dad 100644 --- a/cli/commands/ci/index.md +++ b/cli/commands/ci/index.md @@ -1,18 +1,7 @@ --- title: CI Command permalink: /cli/commands/ci/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Commands to manage the CI/CD pipeline. - -``` -kbc ci [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc ci workflows](/cli/commands/ci/workflows/) | Generate workflows for [GitHub Actions integration](/cli/github-integration/). | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/ci/workflows/index.md b/cli/commands/ci/workflows/index.md index 735fb2ad..ace3bcda 100644 --- a/cli/commands/ci/workflows/index.md +++ b/cli/commands/ci/workflows/index.md @@ -1,71 +1,7 @@ --- title: Workflows permalink: /cli/commands/ci/workflows/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Generate workflows for [GitHub Actions integration](/cli/github-integration/).** - -``` -kbc ci workflows [flags] -``` - -You will be prompted to choose which workflows you want to generate: -- `validate` - validates all branches on change -- `push` - pushes each change in the main branch to the project -- `pull` - pulls the main branch from the project every five minutes - -## Options - -`--ci-main-branch ` -: Name of the main branch for push/pull workflows (default "main") - -`--ci-pull ` -: Create a workflow to sync the main branch from the project every five minutes (default true) - -`--ci-push ` -: Create a workflow to push changes in the main branch to the project (default true) - -`--ci-validate ` -: Create a workflow to validate all branches on change to a GitHub branch (default true) - -[Global Options](/cli/commands/#global-options) - -## Example - -``` -➜ kbc workflows - -Please confirm the GitHub Actions you want to generate. - -? Generate "validate" workflow? -All GitHub branches will be validated on change. Yes - -? Generate "push" workflow? -Each change in the main GitHub branch will be pushed to the project. Yes - -? Generate "pull" workflow? -The main GitHub branch will be synchronized every five minutes. -If a change is found, a new commit is created and pushed. Yes - -? Please select the main GitHub branch name: main - -Generating CI workflows ... -Created file ".github/actions/install/action.yml". -Created file ".github/workflows/validate.yml". -Created file ".github/workflows/push.yml". -Created file ".github/workflows/pull.yml". - -CI workflows have been generated. -Feel free to modify them. - -Please set the secret KBC_STORAGE_API_TOKEN in the GitHub settings. -See: https://docs.github.com/en/actions/reference/encrypted-secrets -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [GitHub Integration](/cli/github-integration/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/generate/env/index.md b/cli/commands/dbt/generate/env/index.md index eb29c5f1..2377c6d3 100644 --- a/cli/commands/dbt/generate/env/index.md +++ b/cli/commands/dbt/generate/env/index.md @@ -1,67 +1,7 @@ --- title: Generate Env Command permalink: /cli/commands/dbt/generate/env/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Generates sources in the dbt project directory.** - -``` -kbc dbt generate sources [flags] -``` - -The command must be run in a directory with a dbt project (i.e., containing `dbt_project.yml`) or its subdirectory. - -The command outputs commands to create environment variables from a selected existing Snowflake workspace. - -See the [introduction to dbt support](/cli/dbt/) for more information. - -## Options - -`-H, --storage-api-host ` -: Storage API host, e.g., "connection.keboola.com" - -`-T, --target-name ` -: Target name of the profile - -`-W, --workspace-id ` -: ID of the workspace to use - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc dbt generate env - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host: connection.north-europe.azure.keboola.com - - -Please enter the Keboola Storage API token. The value will be hidden. -? API token: ************************************************** - - -Please enter the target name. -Allowed characters: a-z, A-Z, 0-9, "_". -? Target Name: target1 - - -? Workspace: dbt_workspace (12345678) - -Commands to set the environment for the dbt target: - export DBT_KBC_TARGET1_TYPE=snowflake - export DBT_KBC_TARGET1_SCHEMA=WORKSPACE_123456 - export DBT_KBC_TARGET1_WAREHOUSE=KEBOOLA_PROD_SMALL - export DBT_KBC_TARGET1_DATABASE=KEBOOLA_1234 - export DBT_KBC_TARGET1_ACCOUNT=keboola.west-europe.azure - export DBT_KBC_TARGET1_USER=KEBOOLA_WORKSPACE_123456 - export DBT_KBC_TARGET1_PASSWORD=abcd123456 -``` - -## Next Steps - -- [dbt generate](/cli/commands/dbt/generate/) -- [Introduction to dbt support](/cli/dbt/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/generate/index.md b/cli/commands/dbt/generate/index.md index 262b09b0..9e66bc62 100644 --- a/cli/commands/dbt/generate/index.md +++ b/cli/commands/dbt/generate/index.md @@ -1,22 +1,7 @@ --- title: Generate Command permalink: /cli/commands/dbt/generate/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Work with dbt inside your repository.** - -The commands must be run in a directory with a dbt project (i.e., containing `dbt_project.yml`) or its subdirectory. - -``` -kbc dbt generate [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc dbt generate profile](/cli/commands/dbt/generate/profile/) | Generate profiles for use with dbt. | -| [kbc dbt generate sources](/cli/commands/dbt/generate/sources/) | Generate sources for use with dbt. | -| [kbc dbt generate env](/cli/commands/dbt/generate/env/) | Generate environment variables for use with dbt. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/generate/profile/index.md b/cli/commands/dbt/generate/profile/index.md index 2d438558..48b878b2 100644 --- a/cli/commands/dbt/generate/profile/index.md +++ b/cli/commands/dbt/generate/profile/index.md @@ -1,58 +1,7 @@ --- title: Generate Profile Command permalink: /cli/commands/dbt/generate/profile/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Generates a profile file in the dbt project directory.** - -``` -kbc dbt generate profile [flags] -``` - -The command must be run in a directory with a dbt project (i.e., containing `dbt_project.yml`) or its subdirectory. - -The command creates a `profiles.yml` file if it does not exist yet and prepares outputs for the selected target. - -See the [introduction to dbt support](/cli/dbt/) for more information. - -## Options - -`-T, --target-name ` -: Target name of the profile - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc dbt generate profile -T target1 - -Profile stored in "profiles.yml". -``` - -The generated `profiles.yml`: - -{% raw %} -```yaml -TestProject: - target: target1 - outputs: - target1: - account: '{{ env_var("DBT_KBC_TARGET1_ACCOUNT") }}' - database: '{{ env_var("DBT_KBC_TARGET1_DATABASE") }}' - password: '{{ env_var("DBT_KBC_TARGET1_PASSWORD") }}' - schema: '{{ env_var("DBT_KBC_TARGET1_SCHEMA") }}' - type: '{{ env_var("DBT_KBC_TARGET1_TYPE") }}' - user: '{{ env_var("DBT_KBC_TARGET1_USER") }}' - warehouse: '{{ env_var("DBT_KBC_TARGET1_WAREHOUSE") }}' -send_anonymous_usage_stats: false -``` -{% endraw %} - -## Next Steps - -- [dbt generate](/cli/commands/dbt/generate/) -- [Introduction to dbt support](/cli/dbt/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/generate/sources/index.md b/cli/commands/dbt/generate/sources/index.md index 5efa9fd3..418785fe 100644 --- a/cli/commands/dbt/generate/sources/index.md +++ b/cli/commands/dbt/generate/sources/index.md @@ -1,78 +1,7 @@ --- title: Generate Sources Command permalink: /cli/commands/dbt/generate/sources/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Generates sources in the dbt project directory.** - -``` -kbc dbt generate sources [flags] -``` - -The command must be run in a directory with a dbt project (i.e., containing `dbt_project.yml`) or its subdirectory. - -The command creates a file for each Storage bucket in the `models/_sources` directory containing a dbt source for every table in the bucket. - -See the [introduction to dbt support](/cli/dbt/) for more information. - -## Options - -`-H, --storage-api-host ` -: Storage API host, e.g., "connection.keboola.com" - -`-T, --target-name ` -: Target name of the profile - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc dbt generate sources - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host: connection.north-europe.azure.keboola.com - - -Please enter the Keboola Storage API token. The value will be hidden. -? API token: ************************************************** - - -Please enter the target name. -Allowed characters: a-z, A-Z, 0-9, "_". -? Target Name: target1 - -Sources stored in the "models/_sources" directory. -``` - -A generated source file `models/_sources/in.c-test.yml`: - -{% raw %} -```yaml -version: 2 -sources: - - name: in.c-test - freshness: - warn_after: - count: 1 - period: day - database: '{{ env_var("DBT_KBC_TARGET1_DATABASE") }}' - schema: in.c-test - loaded_at_field: '"_timestamp"' - tables: - - name: products - quoting: - database: true - schema: true - identifier: true - columns: [] -``` -{% endraw %} - -## Next Steps - -- [dbt generate](/cli/commands/dbt/generate/) -- [Introduction to dbt support](/cli/dbt/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/index.md b/cli/commands/dbt/index.md index d76eb904..bfaa6e14 100644 --- a/cli/commands/dbt/index.md +++ b/cli/commands/dbt/index.md @@ -1,26 +1,7 @@ --- title: dbt Command permalink: /cli/commands/dbt/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Work with dbt inside your repository.** - -The commands must be run in a directory with a dbt project (i.e. containing `dbt_project.yml`) or its subdirectory. - -See the [introduction to dbt support](/cli/dbt/) for more information. - -``` -kbc dbt [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc dbt init](/cli/commands/dbt/init/) | Initialize profiles, sources, and environment variables for use with dbt. | -| [kbc dbt generate](/cli/commands/dbt/generate/) | Generate profiles, sources, or environment variables for use with dbt. | -| [kbc dbt generate profile](/cli/commands/dbt/generate/profile/) | Generate profiles for use with dbt. | -| [kbc dbt generate sources](/cli/commands/dbt/generate/sources/) | Generate sources for use with dbt. | -| [kbc dbt generate env](/cli/commands/dbt/generate/env/) | Generate environment variables for use with dbt. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/dbt/init/index.md b/cli/commands/dbt/init/index.md index 9a2aeafc..3b76f2c2 100644 --- a/cli/commands/dbt/init/index.md +++ b/cli/commands/dbt/init/index.md @@ -1,69 +1,7 @@ --- title: dbt Init Command permalink: /cli/commands/dbt/init/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Initialize a new Snowflake workspace, generate profiles, sources, and environment variables to use in your dbt project.** - -``` -kbc dbt init [flags] -``` - -The command must be run in a directory with a dbt project (i.e., containing `dbt_project.yml`) or its subdirectory. - -See the [introduction to dbt support](/cli/dbt/) for more information. - -## Options - -`-H, --storage-api-host ` -: Storage API host, e.g., "connection.keboola.com" - -`-T, --target-name ` -: Target name of the profile - -`-W, --workspace-name ` -: Name of the workspace to be created - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc dbt init - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host: connection.north-europe.azure.keboola.com - - -Please enter the Keboola Storage API token. The value will be hidden. -? API token: ************************************************** - - -Please enter the target name. -Allowed characters: a-z, A-Z, 0-9, "_". -? Target Name: TARGET1 - - -? Enter a name for the workspace to be created: dbt_workspace - -Creating a new workspace, please wait. -Created new workspace "dbt_workspace". -Profile stored in "profiles.yml". -Sources stored in "models/_sources" directory. -Commands to set environment for the dbt target: - export DBT_KBC_TARGET1_TYPE=snowflake - export DBT_KBC_TARGET1_SCHEMA=WORKSPACE_12345 - export DBT_KBC_TARGET1_WAREHOUSE=KEBOOLA_PROD_SMALL - export DBT_KBC_TARGET1_DATABASE=KEBOOLA_1234 - export DBT_KBC_TARGET1_ACCOUNT=keboola.west-europe.azure - export DBT_KBC_TARGET1_USER=KEBOOLA_WORKSPACE_12345 - export DBT_KBC_TARGET1_PASSWORD=abcd1234 -``` - -## Next Steps - -- [dbt generate](/cli/commands/dbt/generate/) -- [Introduction to dbt support](/cli/dbt/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/help/index.md b/cli/commands/help/index.md index 55a32dc5..41b92cac 100644 --- a/cli/commands/help/index.md +++ b/cli/commands/help/index.md @@ -1,28 +1,7 @@ --- title: Help Command permalink: /cli/commands/help/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Shows help for any command.** - -``` -kbc help [command] [flags] -``` - -## Options - -[Global Options](/cli/commands/#global-options) - -## Example - -``` -kbc help local create config -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Init](/cli/commands/sync/init/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/index.md b/cli/commands/index.md index 1aa41e27..9494a859 100644 --- a/cli/commands/index.md +++ b/cli/commands/index.md @@ -1,172 +1,7 @@ --- title: Commands permalink: /cli/commands/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - -Run `help` to list all available commands. -``` -kbc help -``` - -You can also get details of any command. -``` -kbc help -kbc help local create row -``` - -## Available Commands - -|--- -| Command | Description -|-|-|- -| [kbc help](/cli/commands/help/) | Show help for any command. | -| [kbc status](/cli/commands/status/) | Show information about a working directory. | -| | | -| **[kbc sync](/cli/commands/sync/)** | **Synchronization between a [local directory](/cli/structure/) and a [project](/cli/#subsystems).** | -| [kbc sync init](/cli/commands/sync/init/) | Initialize a new local directory and run `kbc sync pull`. | -| [kbc sync pull](/cli/commands/sync/pull/) | Sync a project to the local directory. | -| [kbc sync push](/cli/commands/sync/push/) | Sync a local directory to the project. | -| [kbc sync diff](/cli/commands/sync/diff/) | Show differences between a local directory and a project. | -| | | -| **[kbc ci](/cli/commands/ci/)** | **Manage the CI/CD pipeline.** | -| [kbc ci workflows](/cli/commands/ci/workflows/) | Generate workflows for [GitHub Actions integration](/cli/github-integration/). | -| | | -| **[kbc local](/cli/commands/local/)** | **Operations in the [local directory](/cli/structure/) don't affect the project.** | -| [kbc local create](/cli/commands/local/create/) | Create an object in the local directory. | -| [kbc local create config](/cli/commands/local/create/config/) | Create an empty [configuration](https://help.keboola.com/components/). | -| [kbc local create row](/cli/commands/local/create/row/) | Create an empty [configuration row](https://help.keboola.com/components/#configuration-rows). | -| [kbc local persist](/cli/commands/local/persist/) | Detect new directories with a [configuration](https://help.keboola.com/components/) or a [configuration row](https://help.keboola.com/components/#configuration-rows). | -| [kbc local encrypt](/cli/commands/local/encrypt/) | Encrypt all [unencrypted secrets](/overview/encryption/#encrypting-data-with-api). | -| [kbc local validate](/cli/commands/local/validate/) | Validate the local directory. | -| [kbc local validate config](/cli/commands/local/validate/config/) | Validate a configuration JSON file. | -| [kbc local validate row](/cli/commands/local/validate/row/) | Validate a configuration row JSON file. | -| [kbc local validate schema](/cli/commands/local/validate/schema/) | Validate a configuration/row JSON file by a JSON schema file. | -| [kbc local fix-paths](/cli/commands/local/fix-paths/) | Ensure that all local paths match [configured naming](/cli/structure/#naming). | -| | | -| **[kbc remote](/cli/commands/remote/)** | **Operations directly in the [project](/cli/#subsystems).** | -| [kbc remote create](/cli/commands/remote/create/) | Create an object in the project. | -| [kbc remote create branch](/cli/commands/remote/create/branch/) | Create a new [branch](https://help.keboola.com/components/branches/) from the `main` branch. | -| [kbc remote create bucket](/cli/commands/remote/create/bucket/) | Create a new [bucket](https://help.keboola.com/storage/buckets/). | -| [kbc remote file](/cli/commands/remote/file/) | Manage [files](https://help.keboola.com/storage/files/) in Storage. | -| [kbc remote file download](/cli/commands/remote/file/download/) | Download a [file](https://help.keboola.com/storage/files/) from Storage. | -| [kbc remote file upload](/cli/commands/remote/file/upload/) | Upload a [file](https://help.keboola.com/storage/files/) to Storage. | -| [kbc remote job](/cli/commands/remote/job/) | Manage [jobs](https://help.keboola.com/management/jobs/) in the project. | -| [kbc remote job run](/cli/commands/remote/job/run/) | Run one or more [jobs](https://help.keboola.com/management/jobs/). | -| [kbc remote table](/cli/commands/remote/table/) | Manage [tables](https://help.keboola.com/storage/tables/) in the project. | -| [kbc remote table create](/cli/commands/remote/table/create/) | Create a new [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table upload](/cli/commands/remote/table/upload/) | Upload a CSV file to a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table download](/cli/commands/remote/table/download/) | Download data from a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table preview](/cli/commands/remote/table/preview/) | Preview up to 1000 rows from a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table detail](/cli/commands/remote/table/detail/) | Print [table](https://help.keboola.com/storage/tables/) details. | -| [kbc remote table import](/cli/commands/remote/table/import/) | Import data to a [table](https://help.keboola.com/storage/tables/) from a [file](https://help.keboola.com/storage/files/). | -| [kbc remote table unload](/cli/commands/remote/table/unload/) | Unload a [table](https://help.keboola.com/storage/tables/) into a [file](https://help.keboola.com/storage/files/). | -| [kbc remote workspace](/cli/commands/remote/create/) | Manage workspaces in the project. | -| [kbc remote workspace create](/cli/commands/remote/workspace/create/) | Create a workspace in the project. | -| [kbc remote workspace delete](/cli/commands/remote/workspace/delete/) | Delete a workspace in the project. | -| [kbc remote workspace detail](/cli/commands/remote/workspace/detail/) | Print workspace details and credentials. | -| [kbc remote workspace list](/cli/commands/remote/workspace/list/) | List workspaces in the project. | -| | | -| **[kbc dbt](/cli/commands/dbt/)** | **Work with dbt inside your repository.** | -| [kbc dbt init](/cli/commands/dbt/init/) | Initialize profiles, sources, and environment variables for use with dbt. | -| [kbc dbt generate](/cli/commands/dbt/generate/) | Generate profiles, sources, and environment variables for use with dbt. | -| [kbc dbt generate profile](/cli/commands/dbt/generate/profile/) | Generate profiles for use with dbt. | -| [kbc dbt generate sources](/cli/commands/dbt/generate/sources/) | Generate sources for use with dbt. | -| [kbc dbt generate env](/cli/commands/dbt/generate/env/) | Generate environment variables for use with dbt. | -| | | -| **[kbc llm](/cli/commands/llm/) (BETA)** | **Export project data to AI-optimized format.** | -| [kbc llm init](/cli/commands/llm/init/) | Initialize a new local directory for LLM export. | -| [kbc llm export](/cli/commands/llm/export/) | Export project data to AI-optimized twin format. | - -## Aliases - -The most used commands have their shorter aliases. - -For example, you can use `kbc c` instead of `kbc local create`. - -|--- -| Full Command | Aliases -|-|-|- -| `kbc sync init` | `kbc init`, `kbc i` -| `kbc sync diff` | `kbc diff`, `kbc d` -| `kbc sync pull` | `kbc pull`, `kbc pl` -| `kbc sync push` | `kbc push`, `kbc ph` -| `kbc local validate` | `kbc validate`, `kbc v` -| `kbc local persist` | `kbc persist`, `kbc pt` -| `kbc local create` | `kbc create`, `kbc c` -| `kbc local encrypt` | `kbc encrypt`, `kbc e` - -## Options - -Options are a way to modify the behavior of a command, they can be: -- **[Global](#global-options)**, for all commands, see below. -- **Local**, only for a specific command, see the command help. - -#### Command-line flags - -- Entered as part of the CLI command. -- One-letter flags start with `-`, for example `-v`. -- Longer flags start with `--`, for example `--verbose`. -- **Flags take precedence over environment variables.** - - -#### Environment variables - -- Each flag can be defined via an environment variable. -- Variable name is based on the flag name, and starts with `KBC_`. -- All letters are changed to uppercase and dashes to underscores. -- For example, flag `--log-file` can be defined by the `KBC_LOG_FILE` environment variable. -- Sources and priority of the environment variables: - 1. From the OS environment. - 2. From environment files in the working directory. - 3. From environment files in the project directory. - -All found environment files are automatically loaded. -Variables are merged together according to the following priority. - -|--- -| Environment File | Environment | Priority -|-|-|- -| `.env.development.local` | Development | The highest | -| `.env.test.local` | Test | | -| `.env.production.local` | Production | | -| `.env.local` | Wherever the file is | | -| `.env.development` | Development| | -| `.env.test` | Test| | -| `.env.production` | Production| | -| `.env` | All | The lowest | - -*Note: All `.*local` environment files should be part of the `.gitignore` file, if used.* - -### Global Options - -`-h, --help` -: Show help for the command - -`-l, --log-file ` -: Path to a log file to store the details in - -`-t, --storage-api-token ` -: Storage API token to the project - -`-v, --verbose` -: Increase output verbosity - -`--verbose-api` -: Log each API request and its response - -`-V, --version` -: Show the version - -`-d, --working-dir ` -: Use another working directory - -## Next Steps - -- [Installation](/cli/installation/) -- [Getting Started](/cli/getting-started/) -- [Directory Structure](/cli/structure/) -- [GitHub Integration](/cli/github-integration/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/llm/export/index.md b/cli/commands/llm/export/index.md index e341a841..1b560516 100644 --- a/cli/commands/llm/export/index.md +++ b/cli/commands/llm/export/index.md @@ -1,122 +1,7 @@ --- title: LLM Export Command permalink: /cli/commands/llm/export/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - - -**Export project data to AI-optimized twin format directory structure.** - -``` -kbc llm export [flags] -``` - -The command must be run in a directory initialized with [kbc llm init](/cli/commands/llm/init/). - -## Description - -The twin format is designed for AI assistants to understand and work with Keboola projects directly from Git repositories. The export includes: - -- **Bucket and table metadata** with schema information -- **Transformation configurations** with platform detection -- **Component configurations** organized by type -- **Job execution history** and statistics -- **Lineage graph** showing data flow dependencies -- **Optional data samples** (controlled by flags) - -The export creates output files containing JSON with inline documentation (`_comment`, `_purpose`, `_update_frequency` fields) to help AI assistants understand the data structure. - -### Security Features - -- **Public repository detection** - Automatically detects if the directory is a public Git repository -- **Sample export disabled by default** - Data samples must be explicitly enabled with `--with-samples` -- **Encrypted secrets** - Fields starting with `#` are encrypted in the output - -## Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -`-t, --storage-api-token ` -: Storage API token from your project - -`-f, --force` -: Skip confirmation when directory contains existing files - -`--with-samples` -: Include table data samples in the export - -`--sample-limit ` -: Maximum number of rows per table sample (default: 100, max: 1000) - -`--max-samples ` -: Maximum number of tables to sample (default: 50, max: 100) - -[Global Options](/cli/commands/#global-options) - -## Output Structure - -The export creates the following directory structure: - -``` -. -├── buckets/ # Bucket and table metadata -│ └── index.json -├── transformations/ # Transformation configurations -├── components/ # Component configurations by type -├── jobs/ # Job execution history -│ ├── recent/ -│ └── by-component/ -├── indices/ # Query indices and lookups -│ └── queries/ -├── ai/ # AI assistant guides -├── samples/ # Table data samples (if --with-samples) -├── lineage.json # Data flow dependencies -└── metadata.json # Project metadata -``` - -## Examples - -### Basic Export - -``` -➜ kbc llm export - -[1/5] Getting default branch... -Using branch: Main (ID: 1234) -[2/5] Fetching project data from APIs... -Fetched: 5 buckets, 23 tables, 150 jobs -[3/5] Processing data (lineage, platforms, sources)... -Processed: 5 buckets, 23 tables, 8 transformations, 45 lineage edges -[4/5] Generating twin format output... -[5/5] Skipping samples (not requested) -Twin format exported to: /path/to/project -Export completed successfully. -``` - -### Export with Data Samples - -``` -➜ kbc llm export --with-samples --sample-limit 50 --max-samples 20 - -[1/5] Getting default branch... -Using branch: Main (ID: 1234) -[2/5] Fetching project data from APIs... -Fetched: 5 buckets, 23 tables, 150 jobs -[3/5] Processing data (lineage, platforms, sources)... -Processed: 5 buckets, 23 tables, 8 transformations, 45 lineage edges -[4/5] Generating twin format output... -[5/5] Fetching and generating table samples... -Twin format exported to: /path/to/project -Export completed successfully. -``` - -## Next Steps - -- [LLM Init](/cli/commands/llm/init/) -- [All Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/llm/index.md b/cli/commands/llm/index.md index 6be085ee..7661ee4c 100644 --- a/cli/commands/llm/index.md +++ b/cli/commands/llm/index.md @@ -1,39 +1,7 @@ --- title: LLM Commands (BETA) permalink: /cli/commands/llm/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - - -**Export project data to AI-optimized format for use with AI assistants and LLMs.** - -The `kbc llm` commands create a "twin format" representation of your Keboola project, -designed for AI assistants to understand and work with your data pipelines. - -``` -kbc llm [command] -``` - -## Workflow - -1. **Initialize** - Run `kbc llm init` to set up the local directory -2. **Export** - Run `kbc llm export` to generate AI-optimized project data - -## Available Commands - -|--- -| Command | Description -|-|-|- -| [kbc llm init](/cli/commands/llm/init/) | Initialize a new local directory for LLM export. | -| [kbc llm export](/cli/commands/llm/export/) | Export project data to AI-optimized twin format. | - -## Next Steps - -- [LLM Init](/cli/commands/llm/init/) -- [LLM Export](/cli/commands/llm/export/) -- [All Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/llm/init/index.md b/cli/commands/llm/init/index.md index fdc65541..756f28dd 100644 --- a/cli/commands/llm/init/index.md +++ b/cli/commands/llm/init/index.md @@ -1,69 +1,7 @@ --- title: LLM Init Command permalink: /cli/commands/llm/init/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - - -**Initialize a new local directory for LLM export.** - -``` -kbc llm init [flags] -``` - -The command must be run in an empty directory. - -This command creates the local manifest and metadata directory (`.keboola/`) without pulling any data from Keboola Connection. -Use [kbc llm export](/cli/commands/llm/export/) after initialization to generate the AI-optimized project data. - -If the command is run without options, it will start an interactive dialog asking for: -- URL of the [stack](https://help.keboola.com/overview/#stacks), for example, `connection.keboola.com`. -- [Storage API token](https://help.keboola.com/management/project/tokens/) to your project. -- Allowed [branches](https://help.keboola.com/tutorial/branches/) to work with. - -## Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -`-t, --storage-api-token ` -: Storage API token from your project - -`-b, --branches ` -: Comma-separated list of branch IDs or name globs (use "*" for all) - -`--allow-target-env` -: Allow usage of `KBC_PROJECT_ID` and `KBC_BRANCH_ID` environment variables for future operations - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc llm init - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host: connection.north-europe.azure.keboola.com - -Please enter the Keboola Storage API token. Its value will be hidden. -? API token: *************************************************** - -Please select which project's branches you want to use with this CLI. -? Allowed project's branches: only main branch - -Created metadata directory ".keboola". -Created manifest file ".keboola/manifest.json". -Created file ".env.local" - it contains the API token, keep it local and secret. -Created file ".env.dist" - an ".env.local" template. -Created file ".gitignore" - to keep ".env.local" local. -``` - -## Next Steps - -- [LLM Export](/cli/commands/llm/export/) -- [All Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/create/config/index.md b/cli/commands/local/create/config/index.md index 00737034..c5fab549 100644 --- a/cli/commands/local/create/config/index.md +++ b/cli/commands/local/create/config/index.md @@ -1,71 +1,7 @@ --- title: Create Configuration permalink: /cli/commands/local/create/config/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create an empty [configuration](https://help.keboola.com/components/).** - -``` -kbc local create config [flags] -``` - -Or shorter: - -``` -kbc create config [flags] -``` - -``` -kbc c config [flags] -``` - -Create an empty configuration in your [local directory](/cli/structure/) and assign it a unique ID (i.e., the [persist](/cli/commands/local/persist/) -command is called automatically). To save it to the project, run the [kbc sync push](/cli/commands/sync/push/) command afterwards. You will -be prompted for a name, a branch, and a component ID. - -Some components have a default content that will be used (if specified by the component author). -For others, `config.json` will only contain an empty JSON document `{}`. - - -*Tip: You can create a new configuration by copying an existing one and running the [persist](/cli/commands/local/persist/) -command.* - -### Options - -`-b, --branch string ` -: Id or name of the branch - -`-c, --component-id ` -: Id of the component - -`-n, --name ` -: Name of the new configuration - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc create config - -? Enter a name for the new config invoices - -? Select the target branch Main (4908) - -? Select the target component MySQL extractor (keboola.ex-db-mysql) -Created new config "main/extractor/keboola.ex-db-mysql/invoices" -``` - -``` -➜ kbc create config -n invoices -b main -c keboola.ex-db-mysql -Created new config "main/extractor/keboola.ex-db-mysql/invoices" -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Create Configuration Row](/cli/commands/local/create/row/) -- [Create Branch](/cli/commands/remote/create/brabch/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/create/index.md b/cli/commands/local/create/index.md index 74d9fcf2..7ae9d6ea 100644 --- a/cli/commands/local/create/index.md +++ b/cli/commands/local/create/index.md @@ -1,27 +1,7 @@ --- title: Create Local Object permalink: /cli/commands/local/create/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create an object in the [local directory](/cli/structure/).** - -``` -kbc local create [config/row] [flags] -``` - -Or shorter: -``` -kbc create [config/row] [flags] -kbc c [config/row] [flags] -``` - -Shows an interactive dialog if you do not enter a sub-command. - -|--- -| Command | Description -|-|-|- -| [kbc local create config](/cli/commands/local/create/config/) | Create an empty [configuration](https://help.keboola.com/components/). | -| [kbc local create row](/cli/commands/local/create/row/) | Create an empty [configuration row](https://help.keboola.com/components/#configuration-rows). | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/create/row/index.md b/cli/commands/local/create/row/index.md index e49ab498..69b0a411 100644 --- a/cli/commands/local/create/row/index.md +++ b/cli/commands/local/create/row/index.md @@ -1,65 +1,7 @@ --- title: Create Configuration Row permalink: /cli/commands/local/create/row/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create an empty [configuration row](https://help.keboola.com/components/#configuration-rows).** - -``` -kbc local create row [flags] -``` - -Or shorter: -``` -kbc create row [flags] -kbc c row [flags] -``` - -Create a new configuration row in your [local directory](/cli/structure/) and assign it a unique ID (i.e., the [persist](/cli/commands/local/persist/) -command is called automatically). To save it to the project, run the [kbc sync push](/cli/commands/sync/push/) command afterwards. You will -be prompted for a name, a branch, and a component ID. - -Some components have a default content that will be used (if specified by the component author). -For others, `config.json` will only contain an empty JSON document `{}`. - -*Tip: You can create a new configuration row by copying an existing one and running the [persist](/cli/commands/local/persist/) command.* - -### Options - -`-b, --branch string ` -: Id or name of the branch - -`-c, --config ` -: Id or name of the configuration - -`-n, --name ` -: Name of the new configuration row - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc create row - -? Enter a name for the new config row customer - -? Select the target branch Main (4908) - -? Select the target config invoices (7475544) -Created new config row "main/extractor/keboola.ex-db-mysql/invoices/rows/customer" -``` - -``` -➜ kbc create config -n customer -b main -c invoices -Created new config row "main/extractor/keboola.ex-db-mysql/invoices/rows/customer" -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Create Configuration](/cli/commands/local/create/config/) -- [Create Branch](/cli/commands/remote/create/brabch/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/encrypt/index.md b/cli/commands/local/encrypt/index.md index 6c056371..1f048b7b 100644 --- a/cli/commands/local/encrypt/index.md +++ b/cli/commands/local/encrypt/index.md @@ -1,83 +1,7 @@ --- title: Encrypt Command permalink: /cli/commands/local/encrypt/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Encrypt all [unencrypted secrets](/overview/encryption/#encrypting-data-with-api) in the [local directory](/cli/structure/). - -``` -kbc local encrypt [flags] -``` - -Or shorter: -``` -kbc e [flags] -``` - -[Unencrypted secrets](/overview/encryption/#encrypting-data-with-api) are values of properties prefixed by `#` that have not been encrypted -yet. - -For example, `{"#someSecretProperty": "secret value"}` -will be transformed into `{"#someSecretProperty": "KBC::ProjectSecure::"}`. - -## Options - -`--dry-run` -: Preview all values that would be affected - -[Global Options](/cli/commands/#global-options) - -## Examples - -Let's say you create a configuration for the MySQL extractor: - -```json -{ - "parameters": { - "host": "our.mysql.server.dev", - "user": "keboola", - "#password": "super-secret" - } -} -``` - -The preview will look like this: - -``` -➜ kbc encrypt --dry-run -Plan for "encrypt" operation: - C main/extractor/keboola.ex-db-mysql/invoices - parameters.#password -Dry run, nothing changed. -``` - -The actual encrypt command: - -``` -➜ kbc encrypt -Plan for "encrypt" operation: - C main/extractor/keboola.ex-db-mysql/invoices - parameters.#password -Encrypt done. -``` - -And the configuration now looks like this: - -```json -{ - "parameters": { - "host": "our.mysql.server.dev", - "user": "keboola", - "#password": "KBC::ProjectSecureKV::eJxLtDK2qs60MrIutrI0s1K695WJQWmhYOI9j2l/twSJl0/nsf6auv/Fs7n5VWvj+tbwvtyz/PSh30Jz8/Y2B0QyPDwteXK/3d7GN55b/y3rK+BbXLF1ne5sg6/Lja/vfzlT4TbvXfkFIuHL9DU0knh8yvedF0lXss60MgbaZQS0Kz01Tzc1L7mosqAkv8jM0NIgzdTU0NQw1cACpMoEqMrYyEop1cjM0DjZzNzE0tLYxNTQMtEw0dLYKCnN0sDS1DjV3EzJuhYAzUBL0A==" - } -} -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Diff](/cli/commands/sync/diff/) -- [Push](/cli/commands/sync/push/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/fix-paths/index.md b/cli/commands/local/fix-paths/index.md index 391e035e..5fa53e97 100644 --- a/cli/commands/local/fix-paths/index.md +++ b/cli/commands/local/fix-paths/index.md @@ -1,41 +1,7 @@ --- title: Fix Paths Command permalink: /cli/commands/local/fix-paths/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Ensure that all local paths match [configured naming](/cli/structure/#naming).** - -``` -kbc local fix-paths [flags] -``` - -The command unifies names of configurations, rows, and other directories based on [configured naming](/cli/structure/#naming). -For example, if the configuration name in `meta.json` changes, this command renames the directory by that name. -It is run automatically after [pull](/cli/commands/sync/pull/). - -## Options - -`--dry-run` -: Preview all paths that would be affected - -[Global Options](/cli/commands/#global-options) - -## Examples - -When you have a config and rename it in its `meta.json`, run the command afterwards. It will rename the directory: - -``` -➜ kbc fix-paths --dry-run -Plan for "rename" operation: - - main/extractor/ex-generic-v2/{wiki-001 -> wiki-2} -Dry run, nothing changed. -Fix paths done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Persist](/cli/commands/local/persist) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/index.md b/cli/commands/local/index.md index f3f70dbf..72498069 100644 --- a/cli/commands/local/index.md +++ b/cli/commands/local/index.md @@ -1,25 +1,7 @@ --- title: Local Command permalink: /cli/commands/local/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Operations in the [local directory](/cli/structure/) don't affect the project.** - -``` -kbc local [command] -``` - -|--- -| Command | Description -|-|-|- -| **[kbc local create](/cli/commands/local/create/)** | **Create an object in the local directory.** | -| [kbc local create config](/cli/commands/local/create/config/) | Create an empty [configuration](https://help.keboola.com/components/). | -| [kbc local create row](/cli/commands/local/create/row/) | Create an empty [configuration row](https://help.keboola.com/components/#configuration-rows). | -| | | -| [kbc local persist](/cli/commands/local/persist/) | Detect new directories with a [configuration](https://help.keboola.com/components/) or a [configuration row](https://help.keboola.com/components/#configuration-rows). | -| [kbc local encrypt](/cli/commands/local/encrypt/) | Encrypt all [unencrypted secrets](/overview/encryption/#encrypting-data-with-api). | -| [kbc local validate](/cli/commands/local/validate/) | Validate the local directory. | -| [kbc local fix-paths](/cli/commands/local/fix-paths/) | Ensure that all local paths match [configured naming](/cli/structure/#naming). | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/persist/index.md b/cli/commands/local/persist/index.md index 02c4aad0..76231d89 100644 --- a/cli/commands/local/persist/index.md +++ b/cli/commands/local/persist/index.md @@ -1,50 +1,7 @@ --- title: Persist Command permalink: /cli/commands/local/persist/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Detect new directories with a [configuration](https://help.keboola.com/components/) or a [configuration row](https://help.keboola.com/components/#configuration-rows) in the [local directory](/cli/structure/).** - -``` -kbc local persist [flags] -``` - -Or shorter: -``` -kbc p [flags] -``` - -Propagate changes in the [local directory](/cli/structure/) to the manifest. When you manually create a configuration or a row (e.g., by -copy & paste of another existing configuration), the command will add its record to the [manifest](/cli/structure/#manifest) and generate a new ID. -When you delete a configuration/row directory, the command will remove its record from the [manifest](/cli/structure/#manifest). If you want -to propagate the changes to the project, call the [push](/cli/commands/sync/push/) command afterwards. - -## Options - -`--dry-run` -: Preview all changes - -[Global Options](/cli/commands/#global-options) - -## Examples - -When you copy & paste a directory of a MySQL extractor configuration, the command will look like this: - -``` -➜ kbc persist --dry-run -Plan for "persist" operation: - + C main/extractor/keboola.ex-db-mysql/invoices 2 - + R main/extractor/keboola.ex-db-mysql/invoices 2/rows/customer -Dry run, nothing changed. -Persist done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Diff](/cli/commands/sync/diff/) -- [Push](/cli/commands/sync/push/) -- [Fix Paths](/cli/commands/local/fix-paths/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/validate/config/index.md b/cli/commands/local/validate/config/index.md index 88c482a4..1f1f1a30 100644 --- a/cli/commands/local/validate/config/index.md +++ b/cli/commands/local/validate/config/index.md @@ -1,49 +1,7 @@ --- title: Validate Config Command permalink: /cli/commands/local/validate/config/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - -**Validate a [configuration JSON file](https://help.keboola.com/extend/common-interface/config-file/).** - -``` -kbc local validate config component.id config.json [flags] -``` - -Each [component](https://help.keboola.com/extend/component/) definition optionally contains a **schema of the configuration `parameters` key**. - -The command validates the content of the specified JSON file against the schema. -It can be used both in a project [local directory](/cli/structure/) and also separately. - -## Options - -[Global Options](/cli/commands/#global-options) - -## Example - -A successful run, the configuration is valid: -``` -➜ kbc local validate config keboola.ex-azure-cost-management config.json -Validation done. -``` - -A validation error: -``` -➜ kbc local validate config keboola.ex-azure-cost-management config.json -Error: missing properties: "subscriptionId" -``` - -If there is no schema in the component definition, a warning is printed: -``` -➜ kbc local validate config ex-generic-v2 config.json -Component "ex-generic-v2" has no configuration JSON schema. -Validation done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Validate Local Project](/cli/commands/local/validate/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/validate/index.md b/cli/commands/local/validate/index.md index 84b834a0..15de03b8 100644 --- a/cli/commands/local/validate/index.md +++ b/cli/commands/local/validate/index.md @@ -1,50 +1,7 @@ --- title: Validate Local Project Command permalink: /cli/commands/local/validate/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - -**Validate the [local project directory](/cli/structure/).** - -``` -kbc local validate [flags] -``` - -Or shorter: -``` -kbc v [flags] -``` - -Validate the directory structure and file contents of the local directory. Configurations of components having a JSON schema -will be validated against the schema. - -## Options - -[Global Options](/cli/commands/#global-options) - -## Example - -``` -➜ kbc validate -Everything is good. -``` - -## Sub Commands - -|--- -| Command | Description -|-|-|- -| [kbc local validate config](/cli/commands/local/validate/config/) | Validate a configuration JSON file. | -| [kbc local validate row](/cli/commands/local/validate/row/) | Validate a configuration row JSON file. | -| [kbc local validate schema](/cli/commands/local/validate/schema/) | Validate a configuration/row JSON file by a JSON schema file. | - - -## Next Steps - -- [All Commands](/cli/commands/) -- [Diff](/cli/commands/sync/diff/) -- [Push](/cli/commands/sync/push/) -- [Fix Paths](/cli/commands/local/fix-paths/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/validate/row/index.md b/cli/commands/local/validate/row/index.md index a4b9a6c2..689f8022 100644 --- a/cli/commands/local/validate/row/index.md +++ b/cli/commands/local/validate/row/index.md @@ -1,51 +1,7 @@ --- title: Validate Row Command permalink: /cli/commands/local/validate/row/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - -**Validate a [configuration row JSON file](https://help.keboola.com/components/#configuration-rows).** - -``` -kbc local validate row component.id row.json [flags] -``` - -Each [component](https://help.keboola.com/extend/component/) definition optionally contains a **schema of the configuration row `parameters` key**. - -The command validates the content of the specified JSON file against the schema. -It can be used both in a project [local directory](/cli/structure/) and also separately. - -## Options - -[Global Options](/cli/commands/#global-options) - -## Example - -A successful run, the configuration row is valid: -``` -➜ kbc local validate row keboola.ex-azure-cost-management row.json -Validation done. -``` - -A validation error: -``` -➜ kbc local validate row keboola.ex-azure-cost-management row.json -Error: -- "export": missing properties: "aggregation" -- "export.groupingDimensions": expected array, but got string -``` - -If there is no schema in the component definition, a warning is printed: -``` -➜ kbc local validate row ex-generic-v2 row.json -Component "ex-generic-v2" has no configuration row JSON schema. -Validation done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Validate Local Project](/cli/commands/local/validate/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/local/validate/schema/index.md b/cli/commands/local/validate/schema/index.md index e8e9e41b..7baafc4c 100644 --- a/cli/commands/local/validate/schema/index.md +++ b/cli/commands/local/validate/schema/index.md @@ -1,47 +1,7 @@ --- title: Validate Schema Command permalink: /cli/commands/local/validate/schema/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - - -**Validate a [configuration](https://help.keboola.com/extend/common-interface/config-file/)/[row](https://help.keboola.com/components/#configuration-rows) JSON file by a JSON schema file.** - -``` -kbc local validate schema schema.json config.json [flags] -``` - -Validate the content of the specified JSON file -against the specified JSON schema file. - -The JSON schema should contain a schema for the `parameters` key, -just like the configuration/row schema in a [component](https://help.keboola.com/extend/component/) definition. - -The main purpose of this command is to **test -a new JSON schema before it is changed in a component definition**. -It can be used both in a project [local directory](/cli/structure/) and also separately. - -## Options - -[Global Options](/cli/commands/#global-options) - -## Example - -A successful run, the JSON file is valid: -``` -➜ kbc local validate schema schema.json config.json -Validation done. -``` - -A validation error: -``` -➜ kbc local validate schema schema.json config.json -Error: missing properties: "subscriptionId" -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Validate Local Project](/cli/commands/local/validate/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/create/branch/index.md b/cli/commands/remote/create/branch/index.md index 82474faa..6cb8f242 100644 --- a/cli/commands/remote/create/branch/index.md +++ b/cli/commands/remote/create/branch/index.md @@ -1,50 +1,7 @@ --- title: Create Branch permalink: /cli/commands/remote/create/branch/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create a new [branch](https://help.keboola.com/components/branches/) from the `main` branch.** - -``` -kbc remote create branch [flags] -``` - -Create a new dev branch as a copy of the main branch in the project and pull its state back to the local directory. -If you have some local changes of the main branch, push them to the project first. - -**Limitation:** -A branch cannot be created locally, it must be created directly in the project from the `main` branch. - -### Options - -`-n, --name ` -: Name of the branch to be created - -`--output-json ` -: Output as a JSON file - - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote create branch -n try1 - -The branch was successfully created. -Pulling objects to the local directory. -Plan for "pull" operation: - * C main/extractor/keboola.ex-google-drive/my-google-drive-data-source | changed: configuration -Pull done. -Created new branch "try1". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Create Configuration](/cli/commands/local/create/config/) -- [Create Configuration Row](/cli/commands/local/create/row/) - +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/create/bucket/index.md b/cli/commands/remote/create/bucket/index.md index c2f1ab03..b652770e 100644 --- a/cli/commands/remote/create/bucket/index.md +++ b/cli/commands/remote/create/bucket/index.md @@ -1,54 +1,7 @@ --- title: Create Bucket permalink: /cli/commands/remote/create/bucket/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create a new [bucket](https://help.keboola.com/storage/buckets/) in Keboola Storage.** - -``` -kbc remote create bucket [flags] -``` - -### Options - -`--stage ` -: Stage of the bucket, allowed values: `in`, `out` - -`--display-name ` -: Display name of the bucket for the UI - -`--name ` -: Name of the bucket - -`--description ` -: Description of the bucket - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote create bucket - -? Select a stage for the bucket: [Use arrows to move, type to filter] - in -> out -? Select a stage for the bucket: out - -Enter a display name for the bucket: Bucket1 - -Enter a name for the bucket: bucket1 - -? Enter a description for the bucket: Test description - -Created bucket "out.c-bucket1". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Create Configuration](/cli/commands/local/create/config/) -- [Create Configuration Row](/cli/commands/local/create/row/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/create/index.md b/cli/commands/remote/create/index.md index 9215c01c..a1cda0ec 100644 --- a/cli/commands/remote/create/index.md +++ b/cli/commands/remote/create/index.md @@ -1,21 +1,7 @@ --- title: Create Remote Object permalink: /cli/commands/remote/create/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create an object directly in the [project](/cli/#subsystems).** - -``` -kbc remote create [command] -``` - -Shows an interactive dialog if you do not enter a sub-command. - -|--- -| Command | Description -|-|-|- -| [kbc remote create branch](/cli/commands/remote/create/branch/) | Create a new [branch](https://help.keboola.com/components/branches/) from the `main` branch. | -| [kbc remote create bucket](/cli/commands/remote/create/bucket/) | Create a new [bucket](https://help.keboola.com/storage/buckets/). | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/file/download/index.md b/cli/commands/remote/file/download/index.md index c7f949c2..a6c055b1 100644 --- a/cli/commands/remote/file/download/index.md +++ b/cli/commands/remote/file/download/index.md @@ -1,58 +1,7 @@ --- title: File Download permalink: /cli/commands/remote/file/download/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Download a [file](https://help.keboola.com/storage/files/) from [Storage](https://help.keboola.com/storage/).** - -``` -kbc remote file download [id] [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`-o, --output ` -: Path and/or name of the destination file (if the file is not sliced) or directory (if the file is sliced). If `-`, output goes to `stdout` without any extra text, so the command is pipeable. - -`--allow-sliced` -: Allow sliced files to appear sliced locally. (default false) - - By default, sliced files are stitched together to form a single file. - If this flag is set when downloading a sliced file, the resulting file will instead be stored as a directory, and each slice will be stored as a separate file in that directory. - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ $ kbc remote file download 1234567 -o name -File "1234567" downloaded to "name.csv". -``` - -If you don't specify the file ID, the command will let you select a file by name. -``` -➜ $ kbc remote file download -? File: -? Enter a name for the destination: - -File "1234567" downloaded to "name.csv" -``` - -If you specify `-` as output, the file will be printed to standard output. -``` -➜ kbc remote file download 1234567 -o - -col1,col2,col3 -val1,val2,val3 -... -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Files Storage](https://help.keboola.com/storage/files/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/file/index.md b/cli/commands/remote/file/index.md index 93464e2d..531fcfd6 100644 --- a/cli/commands/remote/file/index.md +++ b/cli/commands/remote/file/index.md @@ -1,19 +1,7 @@ --- title: File Commands permalink: /cli/commands/remote/file/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC - {:toc} - -**Manage [files](https://help.keboola.com/storage/files/) directly in the [project](/cli/#subsystems).** - -``` -kbc remote file [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc remote file download](/cli/commands/remote/file/download/) | Download a [file](https://help.keboola.com/storage/files/) from Storage. | -| [kbc remote file upload](/cli/commands/remote/file/upload/) | Upload a [file](https://help.keboola.com/storage/files/) to Storage. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/file/upload/index.md b/cli/commands/remote/file/upload/index.md index 9522d979..0e36ded7 100644 --- a/cli/commands/remote/file/upload/index.md +++ b/cli/commands/remote/file/upload/index.md @@ -1,61 +1,7 @@ --- title: File Upload permalink: /cli/commands/remote/file/upload/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Upload a [file](https://help.keboola.com/storage/files/) to [Storage](https://help.keboola.com/storage/).** - -``` -kbc remote file upload [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--data ` -: Path and/or name of the source file. If `-`, input is expected from standard input, so the command is pipeable. - -`--file-name ` -: Name of the file to be created - -`--file-tags ` -: Comma-separated list of tags - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ $ kbc remote file upload --name file1 --data ./name.csv --tags tag1,tag2 -File "file1" uploaded with file id "1234567". -``` - -If you don't specify the options, the command will ask you for them. -``` -➜ $ kbc remote file upload -? Enter a name for the file: file1 - -Enter a path for the file input or - to read from standard input. -? File: name.csv - -Enter a comma-separated list of tags. -? Tags: tag1,tag2 - -File "file1" uploaded with file id "1234567". -``` - -If you specify `-` as input, the file will be read from standard input. -``` -➜ cat ./name.csv | kbc remote file upload --file-name file1 --data - -File "file1" uploaded with file id "1234567". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Files Storage](https://help.keboola.com/storage/files/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/index.md b/cli/commands/remote/index.md index e6f259e1..e463e8a6 100644 --- a/cli/commands/remote/index.md +++ b/cli/commands/remote/index.md @@ -1,38 +1,7 @@ --- title: Remote Commands permalink: /cli/commands/remote/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Operations directly in the [project](/cli/#subsystems).** - -``` -kbc remote [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc remote create](/cli/commands/remote/create/) | Create an object in the project. | -| [kbc remote create branch](/cli/commands/remote/create/branch/) | Create a new [branch](https://help.keboola.com/components/branches/) from the `main` branch. | -| [kbc remote create bucket](/cli/commands/remote/create/bucket/) | Create a new [bucket](https://help.keboola.com/storage/buckets/). | -| [kbc remote file](/cli/commands/remote/file/) | Manage [files](https://help.keboola.com/storage/files/) in Storage. | -| [kbc remote file download](/cli/commands/remote/file/download/) | Download a [file](https://help.keboola.com/storage/files/) from Storage. | -| [kbc remote file upload](/cli/commands/remote/file/upload/) | Upload a [file](https://help.keboola.com/storage/files/) to Storage. | -| [kbc remote job](/cli/commands/remote/job/) | Manage [jobs](https://help.keboola.com/management/jobs/) in the project. | -| [kbc remote job run](/cli/commands/remote/job/run/) | Run one or more [jobs](https://help.keboola.com/management/jobs/). | -| [kbc remote table](/cli/commands/remote/table/) | Manage [tables](https://help.keboola.com/storage/tables/) in the project. | -| [kbc remote table create](/cli/commands/remote/table/create/) | Create a new [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table upload](/cli/commands/remote/table/upload/) | Upload a CSV file to a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table download](/cli/commands/remote/table/download/) | Download data from a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table preview](/cli/commands/remote/table/preview/) | Preview up to 1000 rows from a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table detail](/cli/commands/remote/table/detail/) | Print [table](https://help.keboola.com/storage/tables/) details. | -| [kbc remote table import](/cli/commands/remote/table/import/) | Import data to a [table](https://help.keboola.com/storage/tables/) from a [file](https://help.keboola.com/storage/files/). | -| [kbc remote table unload](/cli/commands/remote/table/unload/) | Unload a [table](https://help.keboola.com/storage/tables/) into a [file](https://help.keboola.com/storage/files/). | -| [kbc remote workspace](/cli/commands/remote/create/) | Manage workspaces in the project. | -| [kbc remote workspace create](/cli/commands/remote/workspace/create/) | Create a workspace in the project. | -| [kbc remote workspace delete](/cli/commands/remote/workspace/delete/) | Delete a workspace in the project. | -| [kbc remote workspace detail](/cli/commands/remote/workspace/detail/) | Print workspace details and credentials. | -| [kbc remote workspace list](/cli/commands/remote/workspace/list/) | List workspaces in the project. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/job/index.md b/cli/commands/remote/job/index.md index 87c7e921..b65f2465 100644 --- a/cli/commands/remote/job/index.md +++ b/cli/commands/remote/job/index.md @@ -1,18 +1,7 @@ --- title: Job Commands permalink: /cli/commands/remote/job/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Manage [jobs](https://help.keboola.com/management/jobs/) directly in the [project](/cli/#subsystems).** - -``` -kbc remote job [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc remote job run](/cli/commands/remote/job/run/) | Run a job. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/job/run/index.md b/cli/commands/remote/job/run/index.md index b6cab3e5..f34f8f41 100644 --- a/cli/commands/remote/job/run/index.md +++ b/cli/commands/remote/job/run/index.md @@ -1,74 +1,7 @@ --- title: Job Run permalink: /cli/commands/remote/job/run/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Run one or more [jobs](https://help.keboola.com/management/jobs/).** - -``` -kbc remote job run [branch1/]component1/config1[@tag] [branch2/]component2/config2[@tag] ... [flags] -``` - -If no `branch` is specified, the `main` branch is used. - -If no `@tag` is specified, the default version of the component is used. - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--timeout ` -: How long to wait for the job to finish (default `2m`) - - Specified as a sequence of decimal numbers with unit suffixes, e.g., `5m10s` or `1.5h`. - Available units are `ms`, `s`, `m`, and `h`. - -`--async` -: Do not wait for jobs to finish (default false) - -[Global Options](/cli/commands/#global-options) - -### Examples - -Run one configuration and wait: -``` -➜ kbc remote job run ex-db-snowflake/978904392 -Starting job. -Started job "328904392" using config "ex-db-snowflake/978904392" -Waiting for "328904392" -Waiting for "328904392" -Finished job "328904392" -Finished all jobs. -``` - -Run multiple configurations and wait; the `branch` and the component version `@tag` are specified: -``` -➜ kbc remote job run keboola.ex-db-snowflake/978904392 12345/keboola.ex-db-oracle/947204232@v2.3.4 -Starting 2 jobs. -Started job "328904393" using config "keboola.ex-db-snowflake/978904392" -Started job "328904394" using config "12345/keboola.ex-db-oracle/947204232@v2.3.4" -Waiting for "328904393", "328904394" -Finished job "328904393" -Waiting for "328904394" -Waiting for "328904394" -Finished job "328904394" -Finished all jobs. -``` - -Run and don't wait; the `--async` flag is used: -``` -➜ kbc remote job run keboola.ex-db-snowflake/978904392 keboola.ex-db-oracle/947204232 --async -Starting 2 jobs. -Started job "328904393" using config "keboola.ex-db-snowflake/978904392" -Started job "328904394" using config "keboola.ex-db-oracle/947204232" -Started all jobs. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Jobs](https://help.keboola.com/management/jobs/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/create/index.md b/cli/commands/remote/table/create/index.md index 34fd426f..ce56007e 100644 --- a/cli/commands/remote/table/create/index.md +++ b/cli/commands/remote/table/create/index.md @@ -1,169 +1,7 @@ --- title: Create Table permalink: /cli/commands/remote/table/create/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -To create a [table](https://help.keboola.com/storage/tables/) in Keboola Storage directly from the command line interface, use the following command: - -``` -kbc remote create table [flags] -``` - -### Options - -`--bucket ` -: Specifies the bucket ID where the table will be created. - -`--columns ` -: Defines a comma-separated list of column names for the table. - -`--columns-from ` -: Indicates the path to the column definition file in json. - -`--name ` -: Sets the name of the new table. - -`--primary-key ` -: Determines a comma-separated list of columns to be used as the primary key. - -`--options-from ` -: The path to the table definition file with backend-specific options. -: This flag is enabled only for projects with the BigQuery backend and must be combined with a `--columns-from` flag because these settings must have specific column types. - -[Global Options](/cli/commands/#global-options) - -### Usage Examples - -**Creating a table without defining column types:** - -``` -➜ kbc remote create table - -? Select a bucket: [Use arrows to move, type to filter] - bucket1 (in.c-bucket1) -> bucket2 (in.c-bucket2) - -Enter the table name. -? Table name: my-table - -Want to define column types? -? Columns Types Definition: [? for help] (Y/n) -``` -If you want to skip defining column types, select `n/N` when prompted and enter the names of the columns. -``` -Want to define column types? -? Columns Types Definition: No - -Enter a comma-separated list of column names. -? Columns: id,name,age - -? Select columns for the primary key: [Use arrows to move, space to select] -> [x] id - [ ] name - [ ] age - -Created table "in.c-bucket2.my-table". -``` -**Defining column types:** - -To define column types, select `y/Y`. Then, start an editor. - -``` -Want to define column types? -? Columns Types Definition: Yes - -Columns definition from file -? Columns definition from file: [Enter to launch editor] -``` -**Edit the YAML file in the editor:** - -Edit or replace this part of the text with your definition. Keep the same format. Then save your changes and close the editor. - -``` -- name: id - definition: - type: VARCHAR - nullable: false - length: 500 - basetype: STRING -- name: name - definition: - type: VARCHAR - nullable: true - basetype: STRING -``` -``` -Columns definition from file -? Columns definition from file: - -? Select columns for the primary key: [Use arrows to move, space to select] -> [x] id - -Created table "in.c-bucket2.my-table". -``` -**Defining column types using a JSON file:** - -``` -kbc remote create table --columns-from [flags] -``` -Example JSON file: -```json -[ - { - "name": "id", - "definition": { - "type": "VARCHAR", - "nullable": false, - "length": "500" - }, - "basetype": "STRING" - }, - { - "name": "name", - "definition": { - "type": "VARCHAR", - "nullable": true - }, - "basetype": "STRING" - } -] -``` -**Writing a JSON file that defines Bigquery settings:** - -``` -kbc remote create table --columns-from --options-from [flags] -``` -Example JSON file: -```json -{ - "timePartitioning": { - "type": "DAY", - "expirationMs": 864000000, - "field": "time" - }, - "clustering": { - "fields": [ - "id" - ] - }, - "rangePartitioning": { - "field": "id", - "range": { - "start": 0, - "end": 10, - "interval": 1 - } - } -} -``` - - - -## Next Steps - -- [All Commands](/cli/commands/) -- [Create a Bucket](/cli/commands/remote/create/bucket/) -- [Table Upload](/cli/commands/remote/table/upload/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/detail/index.md b/cli/commands/remote/table/detail/index.md index 689931f8..c58dd260 100644 --- a/cli/commands/remote/table/detail/index.md +++ b/cli/commands/remote/table/detail/index.md @@ -1,76 +1,7 @@ --- title: Table Detail permalink: /cli/commands/remote/table/detail/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Print [table](https://help.keboola.com/storage/tables/) details.** - -``` -kbc remote table detail [table] [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -[Global Options](/cli/commands/#global-options) - -### Examples - -Print the details of a table: -``` -➜ kbc remote table detail in.c-demo-keboola-ex-google-drive-1234567.account - -Table "in.c-demo-keboola-ex-google-drive-1234567.account": - Name: issues - Primary key: Id, Name - Columns: Id, Name, Region, First_Order - Rows: 7801 - Size: 92 MB - Created at: 2023-02-01T11:22:05.000Z - Last import at: 2023-02-01T13:09:19.000Z - Last changed at: 2023-02-01T13:09:19.000Z -``` - -Print the details of a table without knowing its ID: -``` -➜ kbc remote table detail -? Table: [Use arrows to move, type to filter] -> in.c-my-bucket.data - in.c-demo-keboola-ex-google-drive-1234567.account - in.c-facebook-extractor.uses - ... - -(down arrow pressed) - -➜ kbc remote table preview -? Table: [Use arrows to move, type to filter] - in.c-my-bucket.data -> in.c-demo-keboola-ex-google-drive-1234567.account - in.c-facebook-extractor.uses - ... - -(enter pressed) - -➜ kbc remote table detail -? Table: in.c-demo-keboola-ex-google-drive-1234567.account - -Table "in.c-demo-keboola-ex-google-drive-1234567.account": - Name: issues - Primary key: Id, Name - Columns: Id, Name, Region, First_Order - Rows: 7801 - Size: 92 MB - Created at: 2023-02-01T11:22:05.000Z - Last import at: 2023-02-01T13:09:19.000Z - Last changed at: 2023-02-01T13:09:19.000Z -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/download/index.md b/cli/commands/remote/table/download/index.md index 1c8791ac..57d2941b 100644 --- a/cli/commands/remote/table/download/index.md +++ b/cli/commands/remote/table/download/index.md @@ -1,86 +1,7 @@ --- title: Table Download permalink: /cli/commands/remote/table/download/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Download data from a [table](https://help.keboola.com/storage/tables/).** - -``` -kbc remote table download [table] [flags] -``` - -`file` -: Path and/or name of the source file. If `-`, input is expected from standard input, so the command is pipeable. - -`table` -: ID of the destination table. - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`-o, --output ` -: Path and/or name of the destination file (if the file is not sliced) or directory (if the file is sliced). If `-`, output goes to `stdout` without any extra text, so the command is pipeable. - -`--changed-since ` -: Only export rows imported after this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--changed-until ` -: Only export rows imported before this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--columns ` -: Comma-separated list of columns to export. - -`--format ` -: Output format. Supported formats are `json` and `csv`. - - The `json` format is only supported in projects with the Snowflake backend. - -`--header` -: First line of the CSV file contains column names. - -`--limit ` -: Limit the number of exported rows. A value of 0 means no limit. (default 0) - -`--order ` -: Order the data by one or more columns. - - Accepts a comma-separated list of column+order pairs, such as `First_Name,Last_Name=desc`. - If the order for a column is not specified, it defaults to ascending, such as `First_Name` in the example above. - -`--where ` -: Filter the data. - - Accepts a semicolon-separated list of expressions, each of which specifies a column and a comparison to one or more values, such as `First_Name=Ivan,Pavel;Birth_Date>=1990-01-01` - -`--allow-sliced` -: Allow sliced files to appear sliced locally. (default false) - - By default, sliced files are stitched together to form a single file. - If this flag is set when downloading a sliced file, the resulting file will instead be stored as a directory, and each slice will be stored as a separate file in that directory. - - -[Global Options](/cli/commands/#global-options) - -### Examples - -Download 2000 rows from a table: -``` -➜ kbc remote table download in.c-gdrive.account -o account.csv --limit 2000 -Unloading table, please wait. -Table "in.c-gdrive.account" unloaded to file "734370450". -File "734370450" downloaded to "account.csv". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/import/index.md b/cli/commands/remote/table/import/index.md index 1797331c..c89b99e7 100644 --- a/cli/commands/remote/table/import/index.md +++ b/cli/commands/remote/table/import/index.md @@ -1,60 +1,7 @@ --- title: Table Import permalink: /cli/commands/remote/table/import/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Import data to a table from a Storage file. - -``` -kbc remote table import [table] [file] [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--columns ` -: Comma-separated list of column names. If present, the first row in the CSV file is not treated as a header. - -`--incremental-load ` -: Data are either added to existing data in the table or replace the existing data. - -`--file-delimiter ` -: Delimiter of the CSV file. Default is `,`. - -`--file-enclosure ` -: Enclosure of the CSV file. Default is `"`. - -`--file-escaped-by ` -: Escape character of the CSV file. By default, no escaping is used. (Note: You can specify either the `enclosure` or `escapedBy` parameter, but not both.) - -`--file-without-headers` -: States if the CSV file contains headers on the first row or not. - -[Global Options](/cli/commands/#global-options) - -### Examples - -Preview a table in the terminal: -``` -➜ $ cat my.csv | kbc remote table import in.c-main.products 1234567 -File with id "1234567" imported to table "in.c-main.products" -``` - -``` -➜ kbc remote table import -? Table: -? File: - -File with id "1234567" imported to table "in.c-main.products" -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Upload files to Storage](/cli/commands/remote/file/upload/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/index.md b/cli/commands/remote/table/index.md index 5848f146..b81aca62 100644 --- a/cli/commands/remote/table/index.md +++ b/cli/commands/remote/table/index.md @@ -1,24 +1,7 @@ --- title: Table Commands permalink: /cli/commands/remote/table/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Manage [tables](https://help.keboola.com/storage/tables/) directly in the [project](/cli/#subsystems).** - -``` -kbc remote table [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc remote table create](/cli/commands/remote/table/create/) | Create a new [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table upload](/cli/commands/remote/table/upload/) | Upload a CSV file to a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table download](/cli/commands/remote/table/download/) | Download data from a [table](https://help.keboola.com/storage/tables/). | -| [kbc remote table preview](/cli/commands/remote/table/preview/) | Retrieve up to 1000 rows from a table. | -| [kbc remote table detail](/cli/commands/remote/table/detail/) | Print [table](https://help.keboola.com/storage/tables/) details. | -| [kbc remote table import](/cli/commands/remote/table/import/) | Import data to a [table](https://help.keboola.com/storage/tables/) from a [file](https://help.keboola.com/storage/files/). | -| [kbc remote table unload](/cli/commands/remote/table/unload/) | Unload a [table](https://help.keboola.com/storage/tables/) into a [file](https://help.keboola.com/storage/files/). | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/preview/index.md b/cli/commands/remote/table/preview/index.md index 9969accb..1a37e4d8 100644 --- a/cli/commands/remote/table/preview/index.md +++ b/cli/commands/remote/table/preview/index.md @@ -1,138 +1,7 @@ --- title: Table Preview permalink: /cli/commands/remote/table/preview/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Preview up to 1000 rows from a [table](https://help.keboola.com/storage/tables/).** - -``` -kbc remote table preview [table] [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--changed-since ` -: Only export rows imported after this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--changed-until ` -: Only export rows imported before this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--columns ` -: Comma-separated list of columns to export. - -`--format ` -: Output format. Supported formats are `json`, `csv`, and `pretty`. (default `pretty`) - - `csv` is formatted according to [RFC 4180](https://www.ietf.org/rfc/rfc4180.txt). - - `json` is formatted as follows: - ```json - { - "columns": ["Id", "Name", "Region"], - "rows": [ - ["Id0", "Name0", "Region0"], - ["Id1", "Name1", "Region1"], - ["Id2", "Name2", "Region2"] - ] - } - ``` - -`-o, --out ` -: Write the data to a file. Fails if the file already exists. - -`--force` -: When combined with `--out`, the file will be overwritten if it already exists. - -`--limit ` -: Limit the number of exported rows (maximum 1000, default 100). - -`--order ` -: Order the data by one or more columns. - - Accepts a comma-separated list of column+order pairs, such as `First_Name,Last_Name=desc`. - If the order for a column is not specified, it defaults to ascending, such as `First_Name` in the example above. - -`--where ` -: Filter the data. - - Accepts a semicolon-separated list of expressions, each of which specifies a column and a comparison to one or more values, such as `First_Name=Ivan,Pavel;Birth_Date>=1990-01-01` - -[Global Options](/cli/commands/#global-options) - -### Examples - -Preview a table in the terminal: -``` -➜ kbc remote table preview in.c-demo-keboola-ex-google-drive-1234567.account -┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓ -┃ Id ┃ Name ┃ Region ┃ First_Order ┃ -┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━╋━━━━━━━━━━━━━━┫ -┃ f030ed64cbc8babbe50901a26675a2ee ┃ CSK Auto ┃ US West ┃ 2015-01-23 ┃ -┃ 06c0b954b0d2088e3da2132d1ba96f31 ┃ AM/PM Camp ┃ Global ┃ 2015-02-04 ┃ -┃ fffe0e30b4a34f01063330a4b908fde5 ┃ Super Saver Foods ┃ Global ┃ 2015-02-06 ┃ -┃ 33025ad4a425b6ee832e76beb250ae1c ┃ Netcore ┃ Global ┃ 2015-03-02 ┃ -┃ ... ┃ ... ┃ ... ┃ ... ┃ -┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━┻━━━━━━━━━━━━━━┛ -``` - -Preview a table with where filters and ordering, and output the result to a CSV -file: -``` -➜ kbc remote table preview in.c-demo-keboola-ex-google-drive-1234567.account \ - --where Age>23 \ - --order Age=asc \ - --limit 1000 \ - --format csv \ - --out accounts-preview.csv -Fetching the data, please wait. -Table "in.c-gdrive.account" preview successfully written to "accounts-preview.csv". -``` - -Preview a table in the terminal without knowing its ID: -``` -➜ kbc remote table preview -? Table: [Use arrows to move, type to filter] -> in.c-my-bucket.data - in.c-demo-keboola-ex-google-drive-1234567.account - in.c-facebook-extractor.uses - ... - -(down arrow pressed) - -➜ kbc remote table preview -? Table: [Use arrows to move, type to filter] - in.c-my-bucket.data -> in.c-demo-keboola-ex-google-drive-1234567.account - in.c-facebook-extractor.uses - ... - -(enter pressed) - -➜ kbc remote table preview -? Table: in.c-demo-keboola-ex-google-drive-1234567.account -Fetching the data, please wait. -┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┓ -┃ Id ┃ Name ┃ Region ┃ First_Order ┃ -┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━╋━━━━━━━━━━━━━━┫ -┃ f030ed64cbc8babbe50901a26675a2ee ┃ CSK Auto ┃ US West ┃ 2015-01-23 ┃ -┃ 06c0b954b0d2088e3da2132d1ba96f31 ┃ AM/PM Camp ┃ Global ┃ 2015-02-04 ┃ -┃ fffe0e30b4a34f01063330a4b908fde5 ┃ Super Saver Foods ┃ Global ┃ 2015-02-06 ┃ -┃ 33025ad4a425b6ee832e76beb250ae1c ┃ Netcore ┃ Global ┃ 2015-03-02 ┃ -┃ ... ┃ ... ┃ ... ┃ ... ┃ -┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━┻━━━━━━━━━━━━━━┛ -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/unload/index.md b/cli/commands/remote/table/unload/index.md index 5a06b191..7465823e 100644 --- a/cli/commands/remote/table/unload/index.md +++ b/cli/commands/remote/table/unload/index.md @@ -1,112 +1,7 @@ --- title: Table Unload permalink: /cli/commands/remote/table/unload/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Unload a [table](https://help.keboola.com/storage/tables/) into a [file](https://help.keboola.com/storage/files/).** - -``` -kbc remote table unload [table] [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--changed-since ` -: Only export rows imported after this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--changed-until ` -: Only export rows imported before this date. - - Date may be written in any format compatible with [strtotime](https://www.php.net/manual/en/function.strtotime.php). - -`--columns ` -: Comma-separated list of columns to export. - -`--format ` -: Output format. Supported formats are `json` and `csv`. - - The `json` format is only supported in projects with the Snowflake backend. - -`--limit ` -: Limit the number of exported rows. A value of 0 means no limit. (default 0) - -`--order ` -: Order the data by one or more columns. - - Accepts a comma-separated list of column+order pairs, such as `First_Name,Last_Name=desc`. - If the order for a column is not specified, it defaults to ascending, such as `First_Name` in the example above. - -`--where ` -: Filter the data. - - Accepts a semicolon-separated list of expressions, each of which specifies a column and a comparison to one or more values, such as `First_Name=Ivan,Pavel;Birth_Date>=1990-01-01` - -`--timeout ` -: How long to wait for the storage job to finish (default `2m`) - - Specified as a sequence of decimal numbers with unit suffixes, e.g., `5m10s` or `1.5h`. - Available units are `ms`, `s`, `m`, and `h`. - -`--async` -: Do not wait for the storage job to finish (default false) - -[Global Options](/cli/commands/#global-options) - -### Examples - -Unload a table: -``` -➜ kbc remote table unload in.c-gdrive.account -Unloading table, please wait. -Table "in.c-gdrive.account" unloaded to file "734370450". -``` - -Unload a table with where filters and ordering: -``` -➜ kbc remote table unload in.c-gdrive.account \ - --where Age>23 \ - --order Age=asc \ - --limit 1000 \ - --format csv \ -Unloading table, please wait. -Table "in.c-gdrive.account" unloaded to file "734370450". -``` - -Unload a table in the terminal without knowing its ID: -``` -➜ kbc remote table unload -? Table: [Use arrows to move, type to filter] -> in.c-my-bucket.data - in.c-gdrive.account - in.c-facebook-extractor.uses - ... - -(down arrow pressed) - -➜ kbc remote table unload -? Table: [Use arrows to move, type to filter] - in.c-my-bucket.data -> in.c-gdrive.account - in.c-facebook-extractor.uses - ... - -(enter pressed) - -➜ kbc remote table unload -? Table: in.c-gdrive.account -Unloading table, please wait. -Table "in.c-gdrive.account" unloaded to file "734370450". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/table/upload/index.md b/cli/commands/remote/table/upload/index.md index dbdc08a1..dfa988fb 100644 --- a/cli/commands/remote/table/upload/index.md +++ b/cli/commands/remote/table/upload/index.md @@ -1,74 +1,7 @@ --- title: Table Upload permalink: /cli/commands/remote/table/upload/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Upload a CSV file to a [table](https://help.keboola.com/storage/tables/).** - -``` -kbc remote table upload [table] [file] [flags] -``` - -`table` -: ID of the destination table. - -`file` -: Path and/or name of the source file. If `-`, input is expected from standard input, so the command is pipeable. - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., `connection.keboola.com` - -`--columns ` -: Comma-separated list of column names. If present, the first row in the CSV file is not treated as a header. - -`--incremental-load` -: Data are either added to existing data in the table or replace the existing data. - -`--primary-key ` -: Comma-separated list of columns representing the primary key for the newly created table if the table doesn't exist. - -`--file-name ` -: Name of the file to be created - -`--file-tags ` -: Comma-separated list of tags for the uploaded file - -`--file-delimiter ` -: Delimiter of the CSV file. Default is `,`. - -`--file-enclosure ` -: Enclosure of the CSV file. Default is `"`. - -`--file-escaped-by ` -: Escape character of the CSV file. By default, no escaping is used. (***Note:** you can specify either the `enclosure` or `escapedBy` parameter, not both.*) - -`--file-without-headers` -: States if the CSV file contains headers on the first row or not. - - - -[Global Options](/cli/commands/#global-options) - -### Examples - -Create a table from a CSV file: -``` -➜ kbc remote table upload in.c-users.accounts accounts.csv \ - --file-name accounts.csv - --file-tags local-file - --primary-key Id -File "accounts.csv" uploaded with file id "734370450". -Table "in.c-users.accounts" does not exist, creating it. -Bucket "in.c-users" does not exist, creating it. -Created new table "in.c-users.accounts" from file with id "734370450". -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Tables](https://help.keboola.com/storage/tables/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/workspace/create/index.md b/cli/commands/remote/workspace/create/index.md index f060113d..be5fc573 100644 --- a/cli/commands/remote/workspace/create/index.md +++ b/cli/commands/remote/workspace/create/index.md @@ -1,50 +1,7 @@ --- title: Create Workspace permalink: /cli/commands/remote/workspace/create/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Create a new [workspace](https://help.keboola.com/transformations/workspace/).** - -``` -kbc remote workspace create [flags] -``` - -### Options - -`--name ` -: Name of the workspace to be created - -`--type ` -: Type of the workspace to be created - -`--size ` -: Size of the workspace to be created. It is ignored for database workspaces. - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote workspace create --name foo --type snowflake - -Creating a new workspace, please wait. -Created the new workspace "foo" (). -Credentials: - Host: - User: - Password: - Database: - Schema: - Warehouse: -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Workspaces](https://help.keboola.com/transformations/workspace/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/workspace/delete/index.md b/cli/commands/remote/workspace/delete/index.md index a1f3807c..a2383b1b 100644 --- a/cli/commands/remote/workspace/delete/index.md +++ b/cli/commands/remote/workspace/delete/index.md @@ -1,37 +1,7 @@ --- title: Delete Workspace permalink: /cli/commands/remote/workspace/delete/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Delete a [workspace](https://help.keboola.com/transformations/workspace/).** - -``` -kbc remote workspace delete [flags] -``` - -### Options - -`-W, --workspace-id string` -: ID of the workspace to be deleted. You can find it using the [List Workspaces](/cli/commands/remote/workspace/list/) command. - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote workspace delete -W - -Deleting the workspace "foo" (), please wait. -Delete done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Workspaces](https://help.keboola.com/transformations/workspace/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/workspace/detail/index.md b/cli/commands/remote/workspace/detail/index.md index 8f8c678f..55fe6491 100644 --- a/cli/commands/remote/workspace/detail/index.md +++ b/cli/commands/remote/workspace/detail/index.md @@ -1,45 +1,7 @@ --- title: Workspace Detail permalink: /cli/commands/remote/workspace/detail/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Print the credentials and details of a [workspace](https://help.keboola.com/transformations/workspace/)** - -``` -kbc remote workspace detail [flags] -``` - -### Options - -`-W, --workspace-id string` -: ID of the workspace to be detailed. You can find it using the [List Workspaces](/cli/commands/remote/workspace/list/) command. - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote workspace detail -W - -Workspace "foo" -ID: -Type: snowflake -Credentials: - Host: - User: - Password: - Database: - Schema: - Warehouse: -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Workspaces](https://help.keboola.com/transformations/workspace/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/workspace/index.md b/cli/commands/remote/workspace/index.md index 40b271f6..97459870 100644 --- a/cli/commands/remote/workspace/index.md +++ b/cli/commands/remote/workspace/index.md @@ -1,21 +1,7 @@ --- title: Workspace Commands permalink: /cli/commands/remote/workspace/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Manage workspaces directly in the [project](/cli/#subsystems).** - -``` -kbc remote workspace [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc remote workspace create](/cli/commands/remote/workspace/create/) | Create a workspace in the project. | -| [kbc remote workspace delete](/cli/commands/remote/workspace/delete/) | Delete a workspace in the project. | -| [kbc remote workspace detail](/cli/commands/remote/workspace/detail/) | Print workspace details and credentials. | -| [kbc remote workspace list](/cli/commands/remote/workspace/list/) | List workspaces in the project. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/remote/workspace/list/index.md b/cli/commands/remote/workspace/list/index.md index 854b3857..daa2b684 100644 --- a/cli/commands/remote/workspace/list/index.md +++ b/cli/commands/remote/workspace/list/index.md @@ -1,37 +1,7 @@ --- title: List Workspaces permalink: /cli/commands/remote/workspace/list/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Print a list of [workspaces](https://help.keboola.com/transformations/workspace/).** - -``` -kbc remote workspace list [flags] -``` - -### Options - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -[Global Options](/cli/commands/#global-options) - -### Examples - -``` -➜ kbc remote workspace list - -Loading workspaces, please wait. -Found workspaces: - foo (ID: , Type: snowflake) - bar (ID: , Type: snowflake) - baz (ID: , Type: python, Size: small) -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Learn more about Workspaces](https://help.keboola.com/transformations/workspace/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/status/index.md b/cli/commands/status/index.md index 56603ce2..9510d915 100644 --- a/cli/commands/status/index.md +++ b/cli/commands/status/index.md @@ -1,54 +1,7 @@ --- title: Status Command permalink: /cli/commands/status/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Show information about the current working directory.** - -``` -kbc status [flags] -``` - -## Options - -[Global Options](/cli/commands/#global-options) - -## Examples - -Status of a project directory: -``` -➜ kbc status -Project directory: /home/kbc-project -Working directory: . -Manifest path: .keboola/manifest.json -``` - -Status of a template repository directory: -``` -➜ kbc status -Repository directory: /home/kbc-repository -Working directory: . -Manifest path: .keboola/repository.json -``` - -Status of a template directory: -``` -➜ kbc status -Template directory: /home/kbc-repository/my-template/v1 -Working directory: . -Manifest path: src/manifest.jsonnet -``` - -Status of an unknown directory: -``` -Directory "/home/kbc-test" is not a project or template repository. -``` - - -## Next Steps - -- [All Commands](/cli/commands/) -- [Init](/cli/commands/sync/init/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/sync/diff/index.md b/cli/commands/sync/diff/index.md index 9a771f0f..5a70abd4 100644 --- a/cli/commands/sync/diff/index.md +++ b/cli/commands/sync/diff/index.md @@ -1,70 +1,7 @@ --- title: Diff Command permalink: /cli/commands/sync/diff/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Show differences between a [local directory](/cli/structure/) and a [project](/cli/#subsystems).** - -``` -kbc sync diff [flags] -``` - -Or shorter: -``` -kbc diff [flags] -kbc d [flags] -``` - -## Options - -`--details` -: Show changed fields - -[Global Options](/cli/commands/#global-options) - -## Examples - -When you change a configuration option of one component (e.g., an output table for a sheet -in the [Google Drive extractor](/components/extractors/storage/google-drive/)), the output will look like this: - -``` -➜ kbc diff -* changed -- remote state -+ local state - -Diff: -* R main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source/rows/share-cities-2 | changed: configuration -+ C main/extractor/keboola.ex-db-mysql/invoices -+ R main/extractor/keboola.ex-db-mysql/invoices/rows/customer - -Use --details flag to list the changed fields. -``` - -If you want more details: - -``` -➜ kbc diff --details -* changed -- remote state -+ local state - -Diff: -* R main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source/rows/jakubm-share-cities-2 - configuration: - parameters.key: - - cities2.csv - + cities.csv -+ C main/extractor/keboola.ex-db-mysql/invoices -+ R main/extractor/keboola.ex-db-mysql/invoices/rows/customer -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Init](/cli/commands/sync/init/) -- [Pull](/cli/commands/sync/pull/) -- [Push](/cli/commands/sync/push/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/sync/index.md b/cli/commands/sync/index.md index 6da7fed3..7fea9850 100644 --- a/cli/commands/sync/index.md +++ b/cli/commands/sync/index.md @@ -1,21 +1,7 @@ --- title: Sync Command permalink: /cli/commands/sync/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Synchronization between a [local directory](/cli/structure/) and a [project](/cli/#subsystems).** - -``` -kbc sync [command] -``` - -|--- -| Command | Description -|-|-|- -| [kbc sync init](/cli/commands/sync/init/) | Initialize a new local directory and run `kbc sync pull`. | -| [kbc sync pull](/cli/commands/sync/pull/) | Sync a project to the local directory. | -| [kbc sync push](/cli/commands/sync/push/) | Sync the local directory to the project. | -| [kbc sync diff](/cli/commands/sync/diff/) | Show differences between the local directory and the project. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/sync/init/index.md b/cli/commands/sync/init/index.md index 914464bb..2f1dba03 100644 --- a/cli/commands/sync/init/index.md +++ b/cli/commands/sync/init/index.md @@ -1,104 +1,7 @@ --- title: Init Command permalink: /cli/commands/sync/init/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Initialize a new [local directory](/cli/structure/) and run the first [pull](/cli/commands/sync/pull/).** - -``` -kbc sync init [flags] -``` - -Or shorter: -``` -kbc init [flags] -kbc i [flags] -``` - -The command must be run in an empty directory. - -If the command is run without options, it will start an interactive dialog asking for: -- URL of the [stack](https://help.keboola.com/overview/#stacks), for example, `connection.keboola.com`. -- [Master token](https://help.keboola.com/management/project/tokens/#master-tokens) to your project. -- Allowed [branches](https://help.keboola.com/tutorial/branches/) to work with. - -It will allow you to create GitHub Actions workflows in the directory. - -## Options - -`--allow-target-env` -: Allow usage of `KBC_PROJECT_ID` and `KBC_BRANCH_ID` environment variables for future operations. -Sets `true` to the `allowTargetEnv` field in the [manifest.json](/cli/structure/#manifest). - -`-b, --branches ` -: Comma-separated list of branch IDs or name globs (use "*" for all) for branches you want to work with locally (default "main"); other branches in the project will be ignored. - -`--ci-main-branch ` -: Name of the main branch for push/pull workflows (default "main") - -`--ci-pull ` -: Create a workflow to sync the main branch from the project every 5 minutes (default true) - -`--ci-push ` -: Create a workflow to push changes in the main branch to the project (default true) - -`--ci-validate ` -: Create a workflow to validate all branches on change (default true) - -`--skip-workflows` -: Skip the interactive GitHub workflow setup - -`-H, --storage-api-host ` -: Keboola instance URL, e.g., "connection.keboola.com" - -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc init - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host connection.north-europe.azure.keboola.com - -Please enter the Keboola Storage API token. Its value will be hidden. -? API token *************************************************** - -Please select which project's branches you want to use with this CLI. -The other branches will still exist, but they will be invisible in the CLI. -? Allowed project's branches: only main branch - -Created metadata directory ".keboola". -Created manifest file ".keboola/manifest.json". -Created file ".env.local" - it contains the API token, keep it local and secret. -Created file ".env.dist" - an ".env.local" template. -Created file ".gitignore" - to keep ".env.local" local. - -? Generate workflows files for GitHub Actions? No - -Init done. Running pull. -Plan for "pull" operation: -+ B main -+ C main/extractor/ex-generic-v2/empty -+ C main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source -+ R main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source/rows/share-cities -+ C main/extractor/keboola.ex-google-drive/my-google-drive-data-source -+ C main/extractor/keboola.ex-google-drive/my-google-drive-data-source/schedules/scheduler-for-7243915 -+ C main/other/keboola.orchestrator/daily -+ C main/other/keboola.orchestrator/daily/schedules/scheduler-for-7243915 -+ C main/other/keboola.sandboxes/address -+ C main/transformation/keboola.snowflake-transformation/address -+ C main/transformation/keboola.snowflake-transformation/address/variables -+ R main/transformation/keboola.snowflake-transformation/address/variables/values/default -Pull done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Pull](/cli/commands/sync/pull/) -- [Push](/cli/commands/sync/push/) -- [Diff](/cli/commands/sync/diff/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/sync/pull/index.md b/cli/commands/sync/pull/index.md index ed460502..dd877513 100644 --- a/cli/commands/sync/pull/index.md +++ b/cli/commands/sync/pull/index.md @@ -1,59 +1,7 @@ --- title: Pull Command permalink: /cli/commands/sync/pull/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Sync a [project](/cli/#subsystems) to the [local directory](/cli/structure/).** - -``` -kbc sync pull [flags] -``` - -Or shorter: -``` -kbc pull [flags] -kbc pl [flags] -``` - -Local changes will be overwritten to match the state of the project. - -If your local state is invalid, the command will fail unless you use the `--force` flag. - -## Options - -`--dry-run` -: Preview all changes - -`--force` -: Ignore invalid local state - -`--cleanup-rename-conflicts` -: Enable cleanup mode for handling rename conflicts. When configurations are renamed in the UI (e.g., in a chain like A→B, C→A), this option removes conflicting destinations to allow the rename to proceed. **Only use this for UI-only workflows** where changes are made exclusively in the Keboola UI and synced down. This option is not needed for normal Git-based development workflows. - -
- -[Global Options](/cli/commands/#global-options) - -## Examples - -``` -➜ kbc pull --dry-run -Pulling objects to the local directory. -Plan for "pull" operation: - × C main/extractor/keboola.ex-db-mysql/7511990/invoices - × R main/extractor/keboola.ex-db-mysql/7511990/invoices/rows/customer -Pull done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Init](/cli/commands/sync/init/) -- [Push](/cli/commands/sync/push/) -- [Diff](/cli/commands/sync/diff/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/commands/sync/push/index.md b/cli/commands/sync/push/index.md index adcbd2d5..90687321 100644 --- a/cli/commands/sync/push/index.md +++ b/cli/commands/sync/push/index.md @@ -1,55 +1,7 @@ --- title: Push Command permalink: /cli/commands/sync/push/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -**Sync a [local directory](/cli/structure/) to the [project](/cli/#subsystems).** - -``` -kbc sync push [flags] -``` - -Or shorter: -``` -kbc push [flags] -kbc ph [flags] -``` - -The project state will be overwritten to match the local state. - -## Options - -`--dry-run` -: Preview all changes - -`--encrypt` -: Encrypt unencrypted values before the push - -`--force` -: Delete configurations missing in the local directory - -[Global Options](/cli/commands/#global-options) - -## Example - -When you [create a configuration](/cli/commands/local/create/config/) of the MySQL extractor, the command will look like this: - -``` -➜ kbc push --dry-run - -Plan for "push" operation: - + C main/extractor/keboola.ex-db-mysql/7511990/invoices - + R main/extractor/keboola.ex-db-mysql/7511990/invoices/rows/customer -Dry run, nothing changed. -Push done. -``` - -## Next Steps - -- [All Commands](/cli/commands/) -- [Init](/cli/commands/sync/init/) -- [Pull](/cli/commands/sync/pull/) -- [Diff](/cli/commands/sync/diff/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/dbt/index.md b/cli/dbt/index.md index 643eef80..ba96675d 100644 --- a/cli/dbt/index.md +++ b/cli/dbt/index.md @@ -1,139 +1,7 @@ --- title: dbt permalink: /cli/dbt/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Keboola CLI allows you to integrate with your dbt project. The commands must be run in a directory with a dbt project -(i.e., containing `dbt_project.yml`) or its subdirectory. - -[kbc dbt init](/cli/commands/dbt/init/) command creates a Snowflake [workspace](https://help.keboola.com/transformations/workspace/) -in Keboola, configures a dbt target with it, generates source files for every table in Keboola Storage and outputs -commands to create environmental variables so that you don't store Snowflake credentials directly in the dbt configuration files. - -The command output will look like this: - -``` -➜ kbc dbt init - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host: connection.north-europe.azure.keboola.com - - -Please enter the Keboola Storage API token. The value will be hidden. -? API token: ************************************************** - - -Please enter the target name. -Allowed characters: a-z, A-Z, 0-9, "_". -? Target Name: target1 - - -? Enter a name for the workspace to be created: dbt_workspace - -Creating a new workspace, please wait. -Created the new workspace "dbt_workspace". -Profile stored in "profiles.yml". -Sources stored in the "models/_sources" directory. -Commands to set the environment for the dbt target: - export DBT_KBC_TARGET1_TYPE=snowflake - export DBT_KBC_TARGET1_SCHEMA=WORKSPACE_12345 - export DBT_KBC_TARGET1_WAREHOUSE=KEBOOLA_PROD_SMALL - export DBT_KBC_TARGET1_DATABASE=KEBOOLA_1234 - export DBT_KBC_TARGET1_ACCOUNT=keboola.west-europe.azure - export DBT_KBC_TARGET1_USER=KEBOOLA_WORKSPACE_12345 - export DBT_KBC_TARGET1_PASSWORD=abcd1234 -``` - -## Profile - -The target name is used for the configuration in the dbt's `profiles.yml` file. See the official documentation for more information: [https://docs.getdbt.com/reference/profiles.yml](https://docs.getdbt.com/reference/profiles.yml). - -The created target in `profiles.yml` does not contain any sensitive information, it just references environmental variables. - -{% raw %} -```yaml -TestProject: - target: target1 - outputs: - target1: - account: '{{ env_var("DBT_KBC_TARGET1_ACCOUNT") }}' - database: '{{ env_var("DBT_KBC_TARGET1_DATABASE") }}' - password: '{{ env_var("DBT_KBC_TARGET1_PASSWORD") }}' - schema: '{{ env_var("DBT_KBC_TARGET1_SCHEMA") }}' - type: '{{ env_var("DBT_KBC_TARGET1_TYPE") }}' - user: '{{ env_var("DBT_KBC_TARGET1_USER") }}' - warehouse: '{{ env_var("DBT_KBC_TARGET1_WAREHOUSE") }}' -send_anonymous_usage_stats: false -``` -{% endraw %} - -## Sources - -The sources are stored in the `models/_sources` directory and the command generates a separate file for every bucket. The directory -can look like this: - -``` -➜ ls -1 models/_sources - -in.c-keboola-ex-facebook-8103426.yml -in.c-keboola-ex-facebook-ads-15044494.yml -in.c-keboola-ex-gcalendar-1279777.yml -in.c-test.yml -in.c-test1647518938917259000.yml -``` - -Each source file contains the definition for all tables in the specific bucket, like this: - -{% raw %} -```yaml -version: 2 -sources: - - name: in.c-test - freshness: - warn_after: - count: 1 - period: day - database: '{{ env_var("DBT_KBC_TARGET1_DATABASE") }}' - schema: in.c-test - loaded_at_field: '"_timestamp"' - tables: - - name: products - quoting: - database: true - schema: true - identifier: true - columns: [] -``` -{% endraw %} - -See the official documentation for more information: [https://docs.getdbt.com/docs/build/sources](https://docs.getdbt.com/docs/build/sources). - -## Env - -The command in the end outputs commands for setting all environment variables you are going to need for the dbt project. - -``` - export DBT_KBC_TARGET1_TYPE=snowflake - export DBT_KBC_TARGET1_SCHEMA=WORKSPACE_12345 - export DBT_KBC_TARGET1_WAREHOUSE=KEBOOLA_PROD_SMALL - export DBT_KBC_TARGET1_DATABASE=KEBOOLA_1234 - export DBT_KBC_TARGET1_ACCOUNT=keboola.west-europe.azure - export DBT_KBC_TARGET1_USER=KEBOOLA_WORKSPACE_12345 - export DBT_KBC_TARGET1_PASSWORD=abcd1234 - ``` - -Single steps in this command can be run separately; see [kbc dbt generate](/cli/commands/dbt/generate/). - -## Available Commands - -|--- -| Command | Description -|-|-|- -| [kbc dbt init](/cli/commands/dbt/init/) | Initialize profiles, sources, and environment variables for use with dbt. | -| [kbc dbt generate](/cli/commands/dbt/generate/) | Generate profiles, sources, or environment variables for use with dbt. | -| [kbc dbt generate profile](/cli/commands/dbt/generate/profile/) | Generate profiles for use with dbt. | -| [kbc dbt generate sources](/cli/commands/dbt/generate/sources/) | Generate sources for use with dbt. | -| [kbc dbt generate env](/cli/commands/dbt/generate/env/) | Generate environment variables for use with dbt. | +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/devops-use-cases/branch_management.png b/cli/devops-use-cases/branch_management.png deleted file mode 100644 index 0bca0869..00000000 Binary files a/cli/devops-use-cases/branch_management.png and /dev/null differ diff --git a/cli/devops-use-cases/dev_prod_flow.png b/cli/devops-use-cases/dev_prod_flow.png deleted file mode 100644 index 1182aba6..00000000 Binary files a/cli/devops-use-cases/dev_prod_flow.png and /dev/null differ diff --git a/cli/devops-use-cases/devprod.png b/cli/devops-use-cases/devprod.png deleted file mode 100644 index b620a2b1..00000000 Binary files a/cli/devops-use-cases/devprod.png and /dev/null differ diff --git a/cli/devops-use-cases/devtools.png b/cli/devops-use-cases/devtools.png deleted file mode 100644 index 5b23a720..00000000 Binary files a/cli/devops-use-cases/devtools.png and /dev/null differ diff --git a/cli/devops-use-cases/index.md b/cli/devops-use-cases/index.md index f774459f..5caf3114 100644 --- a/cli/devops-use-cases/index.md +++ b/cli/devops-use-cases/index.md @@ -1,203 +1,7 @@ --- title: DevOps Use Cases permalink: /cli/devops-use-cases/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Keboola CLI provides a Keboola project representation in -a [directory structure](/cli/structure/#directory-structure) with [JSON files](/cli/structure/#configurations). -The [--allow-target-env](https://developers.keboola.com/cli/commands/sync/init/#:~:text=Options-,%2D%2Dallow%2Dtarget%2Denv,-Allow%20usage%20of) -init mode -enables you to apply a GitOps management framework to all your projects. Here we list several example use cases that are possible -using this mode. - -## Git-Based Branch Management - -By overriding the destination branch via the `KBC_BRANCH_ID` environment variable, you can -map each Git branch to a particular GUI Dev branch and then use Git to perform the merge or rebase even between different -branches. - -{: .image-popup} -![branch_management.png](/cli/devops-use-cases/branch_management.png) - -### Tutorial - -#### Initialization - -1. First, let's initialize a GitHub repository with a single main branch via the `kbc init --allow-target-env --skip-workflows` command. - -- Fill in all parameters as usual. -- Select only the main branch in this case. - - {: .image-popup} - ![init.png](/cli/devops-use-cases/init.png) - -2. Initialize the GitHub repository using the `git init` command. - -Now, you can perform `kbc push` or `kbc pull` as normal, enabling you to sync the production development branch into main. - -#### Creating a new branch - -Let’s create a new branch named `new-feature`. - -```shell -git checkout -b new-feature -``` - -Now, you can link the current branch to an actual Keboola development branch: - -1. Create a new Keboola Dev Branch named `new-feature`. The following command will create a remote branch off the production - branch. - - ```shell - kbc remote create branch -n new-feature --output-json new_branch.json - ``` - -2. Obtain the newly created branch ID. - The newly created branch ID can be found in the resulting file `new_branch.json`. - ```json - { - "newBranchId": 123 - } - ``` - -3. Override the destination branch by setting the `KBC_BRANCH_ID` environment variable. - - ```shell - export KBC_BRANCH_ID=123 - ``` - -4. Run `kbc push` to synchronize the local changes. - -Any changes that you perform in the remote branch will be now synchronized back using the `kbc pull` command as long as -the `KBC_BRANCH_ID` variable is set. - -#### Merging the changes - -Once you are ready, you can commit the changes and compare to the main branch and eventually merge the new branch to -your main. - -**WARNING:** Once you switch back to the main branch, do not forget to unset the `KBC_BRANCH_ID` (using -the `unset KBC_BRANCH_ID` command) variable so the `kbc push/pull` commands run against the main branch again. - -## Deployment to Multiple Projects - -By using the `kbc init --allow-target-env` mode, you can override the destination project. This is, for instance, leveraged in -the Dev/Prod Manager example use case. You can use this, for instance, to distribute a single project (as a -“template”) into multiple ones to start from. - -{: .image-popup} -![project_deploy.png](/cli/devops-use-cases/project_deploy.png) - -### Tutorial - -1. First, let's **create a GitHub repository** with a single main branch via the `kbc init --allow-target-env --skip-workflows` command: - -- Fill in all parameters as usual. -- Select only the main branch in this case. - - {: .image-popup} - ![init.png](/cli/devops-use-cases/init.png) - -2. **Obtain the main branch ID**: - - To get the destination project main branch ID, you can use - the [List Branches API call](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches) - and search for a branch named `main`. - - Alternatively, from any component configuration, go to the Developer Tools (`F12` in Chrome) - and search for any underlying API call, e.g., `versions`. You will see the branch ID in the URL ( - *xxx/branch/{BRANCH_ID}/xx*). - - {: .image-popup} - ![devtools.png](/cli/devops-use-cases/devtools.png) - - Now you can perform `kbc push` and the project definition will be transferred into the main branch of the selected - project. - -3. **Change the destination project ID and its main branch:** - -```shell -export KBC_PROJECT_ID=1234 -export KBC_BRANCH_ID=972851 -``` - -## Multi-Stage (and Multi-Project) Environment Management - -Keboola's [native branching environment](https://help.keboola.com/components/branches/) is typically sufficient for small -to medium projects. However, in an enterprise setup, it may -be necessary to have completely separate environments where both data and data pipeline definitions (code) are isolated. -In such setups, administrators may need to define complex “branch protection” rules to closely control who can release -new features into the production environment, as well as how and when these releases occur. In the software engineering -world, this is often achieved with version control systems like Git. - -Thanks -to[ Keboola's CLI functionality](https://developers.keboola.com/cli/commands/sync/init/#:~:text=Options-,%2D%2Dallow%2Dtarget%2Denv,-Allow%20usage%20of), -it is possible to define and synchronize separate environments, including the -ones with a [multi-project architecture](https://help.keboola.com/catalog/multi-project/) setup, entirely via Git. This -gives users the freedom to establish deployment rules -according to their needs and allows for the testing of entire pipelines across multiple projects in completely isolated -environments. - -{: .image-popup} -![devprod.png](/cli/devops-use-cases/devprod.png) - -### High-Level Workflow - -To implement the above suggested setup, we need the following tools: -- **Keboola CLI**: sync project representations with an enabled overridden target environment -- **Keboola Variables Vault**: a feature that allows users to define variables and secrets on a project level and reference them in configurations -- **GitHub & Git Actions**: a versioning system to hold the project representations and define deployment rules and validations - -{: .image-popup} -![dev_prod_flow.png](/cli/devops-use-cases/dev_prod_flow.png) - -We have prepared a sample [Streamlit application](https://github.com/keboola/cli-based-sync-generator) -that can be deployed as a [Data App](https://help.keboola.com/components/data-apps/#git-repository) in the Keboola environment to assist with the initialization process. - -This application includes GitHub actions that allow you to manage this scenario. It is expected that users will modify this flow to their needs. - -To learn about the full use case, please refer to [this blog post](https://www.keboola.com/blog/keboola-dev-prod-lifecycle-via-git), where we describe the workflow in depth. - -## UI-Only Sync Workflow - -When working in a purely UI-based workflow where all changes are made in the Keboola user interface and no local development occurs in the Git repository, you may encounter rename conflicts during pull operations. - -### Rename Conflicts - -Rename conflicts occur when configurations are renamed in a chain. For example: -- Configuration A is renamed to B in the UI -- Configuration C is renamed to A in the UI - -When pulling these changes, the standard `kbc pull` command will fail because it tries to rename C to A, but A still exists (it hasn't been renamed to B yet in the local operation order). - -### Using --cleanup-rename-conflicts - -For UI-only workflows, use the `--cleanup-rename-conflicts` flag: - -```shell -kbc pull --cleanup-rename-conflicts -``` - -This flag enables cleanup mode, which: -1. Detects when a rename destination already exists -2. Removes the conflicting destination -3. Proceeds with the rename operation - -This is safe for UI-only workflows because there are no uncommitted local changes that could be lost. - -
- -### When NOT to Use This Flag - -Do **not** use `--cleanup-rename-conflicts` if: -- You are making local changes to configurations in your Git repository -- You are following a Git-based development workflow -- You have uncommitted local modifications - -For Git-based development workflows, standard `kbc pull` and `kbc push` operations handle renames correctly without requiring cleanup mode. +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/devops-use-cases/init.png b/cli/devops-use-cases/init.png deleted file mode 100644 index 7c23263a..00000000 Binary files a/cli/devops-use-cases/init.png and /dev/null differ diff --git a/cli/devops-use-cases/project_deploy.png b/cli/devops-use-cases/project_deploy.png deleted file mode 100644 index c8db9616..00000000 Binary files a/cli/devops-use-cases/project_deploy.png and /dev/null differ diff --git a/cli/getting-started/configurations-copy-1.jpg b/cli/getting-started/configurations-copy-1.jpg deleted file mode 100644 index 83d58e23..00000000 Binary files a/cli/getting-started/configurations-copy-1.jpg and /dev/null differ diff --git a/cli/getting-started/configurations-copy-2.jpg b/cli/getting-started/configurations-copy-2.jpg deleted file mode 100644 index 7cfc027f..00000000 Binary files a/cli/getting-started/configurations-copy-2.jpg and /dev/null differ diff --git a/cli/getting-started/index.md b/cli/getting-started/index.md index 6d631b8f..e8bfc2a0 100644 --- a/cli/getting-started/index.md +++ b/cli/getting-started/index.md @@ -1,197 +1,7 @@ --- title: Getting Started permalink: /cli/getting-started/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -## Init the Directory - -To manage a project using Keboola CLI, you need to initialize a directory. Create an empty directory, hop into it and run -the init command. - -``` -mkdir my-kbc-project -cd my-kbc-project -kbc init -``` - -The command runs interactively by default and asks for the URL of the Keboola instance you want to use and -a [Master token](https://help.keboola.com/management/project/tokens/#master-tokens) to your project. It pulls all -configurations from the project to the local directory. - -``` -➜ kbc init - -Please enter the Keboola Storage API host, e.g., "connection.keboola.com". -? API host connection.north-europe.azure.keboola.com - -Please enter the Keboola Storage API token. Its value will be hidden. -? API token *************************************************** - -Please select which project's branches you want to use with this CLI. -The other branches will still exist, but they will be invisible in the CLI. -? Allowed project's branches: only main branch - -Created metadata directory ".keboola". -Created manifest file ".keboola/manifest.json". -Created file ".env.local" - it contains the API token, keep it local and secret. -Created file ".env.dist" - an ".env.local" template. -Created file ".gitignore" - to keep ".env.local" local. - -? Generate workflows files for GitHub Actions? No - -Init done. Running pull. -Plan for "pull" operation: -+ B main -+ C main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source -+ R main/extractor/keboola.ex-aws-s3/my-aws-s-3-data-source/rows/share-cities -+ C main/extractor/keboola.ex-google-drive/my-config -+ C main/extractor/keboola.ex-google-drive/my-config/schedules/scheduler-for-7241051 -+ C main/other/keboola.orchestrator/daily -+ C main/other/keboola.orchestrator/daily/schedules/scheduler-for-7243915 -+ C main/other/keboola.sandboxes/address -+ C main/transformation/keboola.snowflake-transformation/address -+ C main/transformation/keboola.snowflake-transformation/address/variables -+ R main/transformation/keboola.snowflake-transformation/address/variables/values/default -Pull done. -``` - -You could want to version your project in a git repository. Feel free to call `git init` and make an initial commit. -The init command can prepare workflows for GitHub Actions to keep your directory and the project in sync. - -## Push Changes to the Project - -When you update your local directory, you can compare the changes with the project: - -``` -➜ kbc diff --details -* changed -- remote state -+ local state - -Diff: -* C main/extractor/keboola.ex-google-drive/my-config | changed: configuration -``` - -Before pushing the changes to the project, you are able to preview them first: - -``` -➜ kbc push --dry-run -Plan for "push" operation: - * C main/extractor/keboola.ex-google-drive/my-config | changed: configuration -Dry run, nothing changed. -Push done. -``` - -And finally, perform the actual changes to the project: - -``` -➜ kbc push -Plan for "push" operation: - * C main/extractor/keboola.ex-google-drive/my-config | changed: configuration -Push done. -``` - -## Create New Configurations - -The configuration can be created in two ways. You can create an empty configuration or copy an existing one. - -### Create Empty Configuration -Let's say you want to download some data from Wikipedia. You can run [`kbc create`](/cli/commands/local/create/config/) -without options and be guided by an interactive dialog, or fill all the options: - -``` -➜ kbc create config -b main -c ex-generic-v2 -n wiki -Created new config "main/extractor/ex-generic-v2/wiki" -``` - -Edit file `main/extractor/ex-generic-v2/wiki/config.json` as -a [Generic Extractor](https://help.keboola.com/components/extractors/generic-extractor/) configuration. A super basic -configuration could look like this: - -```json -{ - "api": { - "baseUrl": "https://wikipedia.org" - } -} -``` - -Now we can push it to the project: - -``` -➜ kbc push -Plan for "push" operation: - + C main/extractor/ex-generic-v2/wiki -Push done. -``` - -### Create Configurations by Copy & Paste - -Let's say you want to copy a configuration of your Generic Extractor. Duplicate its directory: - -{: .image-popup} -![Screenshot -- Duplicate a configuration directory](/cli/getting-started/configurations-copy-1.jpg) - -Run the `persist` command: - -``` -➜ kbc persist -Plan for "persist" operation: - + C main/extractor/ex-generic-v2/wiki 2 -Persist done. -Plan for "rename" operation: - - main/extractor/ex-generic-v2/{wiki 2 -> wiki-001} -Rename done. -``` - -The directory name is fixed and the configuration added to the manifest: - -{: .image-popup} -![Screenshot -- Fixed configuration directory](/cli/getting-started/configurations-copy-2.jpg) - -## Pull Changes from the Project - -When you create or change configurations in the project, you can pull them to the local directory. - -Show the changes between the project and the local directory: - -``` -➜ kbc diff --details -* changed -- remote state -+ local state - -Diff: -* C main/extractor/ex-generic-v2/wiki - configuration: - api.baseUrl: - - https://en.wikipedia.org/wiki/Git - + https://wikipedia.org -``` - -Preview the pull command without changing anything first: - -``` -➜ kbc pull --dry-run -Plan for "pull" operation: - * C main/extractor/ex-generic-v2/wiki | changed: configuration -Dry run, nothing changed. -Pull done. -``` - -And finally, pull the changes to the local directory. Note that it will override any changes to your local directory: - -``` -➜ kbc pull -Plan for "pull" operation: - * C main/extractor/ex-generic-v2/wiki | changed: configuration -Pull done. -``` - -## Next Steps - -- [Directory Structure](/cli/structure/) -- [Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/github-integration/github-actions.jpg b/cli/github-integration/github-actions.jpg deleted file mode 100644 index 70f380f6..00000000 Binary files a/cli/github-integration/github-actions.jpg and /dev/null differ diff --git a/cli/github-integration/index.md b/cli/github-integration/index.md index da14ebbb..23029006 100644 --- a/cli/github-integration/index.md +++ b/cli/github-integration/index.md @@ -1,43 +1,7 @@ --- title: GitHub Integration permalink: /cli/github-integration/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -The tool can generate workflows for GitHub Actions within commands [init](/cli/commands/sync/init/) -and [workflows](/cli/commands/ci/workflows/). For automated CI/CD scenarios, use the `--skip-workflows` flag -with the init command to bypass interactive workflow setup prompts. - -Secret `KBC_STORAGE_API_TOKEN` with your master token needs to be added to the GitHub -[secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository). - -{: .image-popup} -![Screenshot -- GitHub Actions](/cli/github-integration/github-actions.jpg) - -## Pull - -The Pull workflow is set to run automatically every hour to [pull](/cli/commands/sync/pull/) the changes from -the project in Keboola. If it finds any changes, it creates a commit to the repository. - -*Note: GitHub does not guarantee periodic running at exact times. The triggers may be delayed a few minutes -depending on the current GitHub Actions workload.* - -{: .image-popup} -![Screenshot -- A commit by Pull action](/cli/github-integration/pull-commit.jpg) - -The commit contains description of the change: - -{: .image-popup} -![Screenshot -- A change description by Pull action](/cli/github-integration/pull-description.jpg) - -## Push - -The Push workflow is triggered by a push to the GitHub repository to [push](/cli/commands/sync/push/) the changes from -the local directory to the project in Keboola. - -## Validate - -The Validate workflow is triggered by a push to a branch in the GitHub repository to validate and preview its changes by -a [push --dry-run](/cli/commands/sync/push/). +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/github-integration/pull-commit.jpg b/cli/github-integration/pull-commit.jpg deleted file mode 100644 index 7c412900..00000000 Binary files a/cli/github-integration/pull-commit.jpg and /dev/null differ diff --git a/cli/github-integration/pull-description.jpg b/cli/github-integration/pull-description.jpg deleted file mode 100644 index 228e28f6..00000000 Binary files a/cli/github-integration/pull-description.jpg and /dev/null differ diff --git a/cli/index.md b/cli/index.md index 8b440ba1..9ab448b0 100644 --- a/cli/index.md +++ b/cli/index.md @@ -1,80 +1,7 @@ --- title: CLI permalink: /cli/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -Keboola CLI (Command Line Interface), known also as "Keboola as Code", is a set of commands for operating your cloud data -pipeline. It is available to install in the Windows, macOS, and Linux environments. - -The whole Keboola project is represented by a local [directory structure](/cli/structure/#directory-structure). -[Component configurations](https://help.keboola.com/components) are represented by [JSON files](/cli/structure/#configurations). - -## Use Cases - -Keboola CLI can be used, for example, to: -- Pull your entire project to a local directory in seconds. See the [init](/cli/commands/sync/init/) and [pull](/cli/commands/sync/pull/) commands. -- Bulk edit [component configurations](https://help.keboola.com/components) in your IDE. -- Compare the local version with the current project state. See the [diff](/cli/commands/sync/diff/) command. -- Copy a [configuration](https://help.keboola.com/components) as a directory in the project and between projects. See the [persist](/cli/commands/local/persist/) command. -- Apply all changes back to the project in a moment. See the [push](/cli/commands/sync/push/) command. -- Manage project history in a git repository. -- Automate the whole process in a CI/CD pipeline. See [GitHub Integration](/cli/github-integration/). Use the `--skip-workflows` flag during initialization to avoid interactive prompts in automated environments. -- Merge and rebase Keboola Branches via Git. Learn more in the [Example Use Cases]() section. -- Distribute a single project definition into multiple projects. See the [Example Use Cases]() section. -- Multi-stage (and multi-project) environment management via Git. See the [Example Use Cases]() section. -- Locally develop and test your dbt transformation code. - -## Subsystems - -A brief overview of supported subsystems of the project. - -### Configurations - -- [Component configurations](https://help.keboola.com/components) and [configuration rows](https://help.keboola.com/components/#configuration-rows) are fully supported. -- This includes all special types of components, such as: - - [Transformations](/cli/structure/#transformations), [Variables](/cli/structure/#variables), [Shared Codes](/cli/structure/#shared-code), [Schedules](/cli/structure/#schedules) and [Orchestrations](/cli/structure/#orchestrations). - -### Development Branches - -- A [branch](https://help.keboola.com/components/branches/) can be [pulled](/cli/commands/sync/pull/) and then edited or deleted locally. -- Changes can be [pushed](/cli/commands/sync/push/) back to the project. -- There is one limitation, **a branch cannot be created locally**. - - A branch must be created directly in the project, from the `main` branch. - - See the [Create Branch](/cli/commands/remote/create/branch/) command. - -### Storage - -At the moment, all [Storage](https://help.keboola.com/storage/) related operations are sub-commands of the [kbc remote](/cli/commands/remote/) command. They operate directly on a project. This means that any changes you make using the CLI are immediately applied to your project. We have plans to add support for managing buckets and tables locally using definition files just like component configurations. - - -#### Files - -- To upload a file, use the [file upload](/cli/commands/remote/file/upload/) command. -- To download a file, use the [file download](/cli/commands/remote/file/download/) command. - -#### Buckets and tables - -These commands can be used to manage the [buckets](https://help.keboola.com/storage/buckets/) and [tables](https://help.keboola.com/storage/tables/) in your project: -- To create a new bucket, use the [create bucket](/cli/commands/remote/create/bucket/) command. -- To create a new table, use the [create table](/cli/commands/remote/table/create) command. - -The resulting [tables](https://help.keboola.com/storage/tables/) will be empty, so you may want to use: -- The [table import](/cli/commands/remote/table/import/) command to import data. -- The [table unload](/cli/commands/remote/table/unload/) command can be used to take data out of a table and store it in a file. - -For convenience, you can use combined commands: -- The [table upload](/cli/commands/remote/table/upload/) command combines the [file upload](/cli/commands/remote/file/upload/) + [table import](/cli/commands/remote/table/import/) operations. -- The [table download](/cli/commands/remote/table/download/) command combines the [table unload](/cli/commands/remote/table/unload/) + [file download](/cli/commands/remote/file/download/) operations. - -These commands may be a little heavy if you are dealing with a lot of data. -- If you just want a quick sample, use the [table preview](/cli/commands/remote/table/preview/) command. - -## Next Steps - -- [Installation](/cli/installation/) -- [Getting Started](/cli/getting-started/) -- [Directory Structure](/cli/structure/) -- [Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/installation/index.md b/cli/installation/index.md index 90946c5b..9c99d915 100644 --- a/cli/installation/index.md +++ b/cli/installation/index.md @@ -1,181 +1,7 @@ --- title: Installation permalink: /cli/installation/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -The recommended way to install Keboola CLI is with one of the package managers listed below. -This allows you to easily upgrade to a new version. - -Alternatively, you can: -- Download precompiled binaries from [cli-dist.keboola.com](https://cli-dist.keboola.com/?prefix=zip/). -- Or build binary from [source code](#build-from-source). - -Changelog can be found at [github.com/keboola/keboola-as-code/releases](https://github.com/keboola/keboola-as-code/releases). - -## macOS - -Installation on macOS is managed by Homebrew. If you don't have Homebrew available on your system, -[install it](https://docs.brew.sh/Installation.html) before continuing. - -Install: - -```bash -brew tap keboola/keboola-cli -brew install keboola-cli -kbc --version -``` - -Upgrade: - -```bash -brew upgrade keboola-cli -``` - -## Debian / Ubuntu - -Install: - -```bash -sudo wget -P /etc/apt/trusted.gpg.d https://cli-dist.keboola.com/deb/keboola.gpg -echo "deb https://cli-dist.keboola.com/deb /" | sudo tee /etc/apt/sources.list.d/keboola.list -sudo apt-get update -sudo apt-get install keboola-cli -kbc --version -``` - -Upgrade: - -```bash -sudo apt-get update -sudo apt-get install keboola-cli -``` - -## Fedora - -Install: - -```bash -sudo rpm --import https://cli-dist.keboola.com/rpm/keboola.gpg -echo "[keboola] -name=keboola -baseurl=https://cli-dist.keboola.com/rpm -enabled=1 -gpgcheck=1 -gpgkey=https://cli-dist.keboola.com/rpm/keboola.gpg -" | sudo tee /etc/yum.repos.d/keboola.repo -sudo dnf install keboola-cli -kbc --version -``` - -Upgrade: - -```bash -sudo dnf update keboola-cli -``` - -## Alpine - -Install: - -```bash -echo "https://cli-dist.keboola.com/apk" | sudo tee -a /etc/apk/repositories -sudo wget -P /etc/apk/keys/ https://cli-dist.keboola.com/apk/keboola.rsa.pub -sudo apk update -sudo apk add keboola-cli -kbc --version -``` - -Upgrade: - -```bash -apk update -apk add --upgrade keboola-cli -``` - -## Windows - -### WinGet - -If you don't have App Installer available on your system, either install it [from the Microsoft Store](https://apps.microsoft.com/detail/9NBLGGH4NNS1) or [download directly](https://winget.run/) before continuing. - -Install: - -```shell -winget install Keboola.KeboolaCLI -kbc --version -``` - -Upgrade: - -```shell -winget upgrade Keboola.KeboolaCLI -``` - - -### Chocolatey - -If you don't have Chocolatey available on your system, [install it](https://chocolatey.org/install) before continuing. - -Install: - -```shell -choco install keboola-cli -kbc --version -``` - -Upgrade: - -```shell -choco upgrade keboola-cli -``` - -### Scoop - -If you don't have Scoop available on your system, [install it](https://scoop.sh/) before continuing. - -Install: - -```shell -scoop bucket add keboola https://github.com/keboola/scoop-keboola-cli.git -scoop install keboola/keboola-cli -kbc --version -``` - -Upgrade: - -```shell -scoop update keboola/keboola-cli -``` - -### Download - -Use a [msi installer](https://cli-dist.keboola.com/?prefix=msi/) or a [precompiled binary](https://cli-dist.keboola.com/?prefix=zip/). - -## Build From Source - -1. Install the [Go environment](https://golang.org/doc/install) (if you haven't done so already). -2. Clone the source from GitHub: -``` -git clone https://github.com/keboola/keboola-as-code -cd keboola-as-code -``` - -3. Run the build: -On Linux or macOS: -``` -go build -o target/kbc ./cmd/kbc/main.go -``` -On Windows: -``` -go build -o target/kbc.exe ./cmd/kbc/main.go -``` - -4. Binary is located in `target/kbc` or `target/kbc.exe`. - -## Next Steps - -- [Getting Started](/cli/getting-started/) -- [Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/structure/directory-example.jpg b/cli/structure/directory-example.jpg deleted file mode 100644 index d692763b..00000000 Binary files a/cli/structure/directory-example.jpg and /dev/null differ diff --git a/cli/structure/directory-orchestration-example.png b/cli/structure/directory-orchestration-example.png deleted file mode 100644 index c0af546c..00000000 Binary files a/cli/structure/directory-orchestration-example.png and /dev/null differ diff --git a/cli/structure/directory-rows-example.jpg b/cli/structure/directory-rows-example.jpg deleted file mode 100644 index be2dd639..00000000 Binary files a/cli/structure/directory-rows-example.jpg and /dev/null differ diff --git a/cli/structure/directory-transformation-example.jpg b/cli/structure/directory-transformation-example.jpg deleted file mode 100644 index bf8e7ef4..00000000 Binary files a/cli/structure/directory-transformation-example.jpg and /dev/null differ diff --git a/cli/structure/index.md b/cli/structure/index.md index 5ca86233..5f41bee6 100644 --- a/cli/structure/index.md +++ b/cli/structure/index.md @@ -1,559 +1,7 @@ --- title: Project Directory Structure permalink: /cli/structure/ +redirect_to: https://github.com/keboola/keboola-as-code --- -* TOC -{:toc} - -The initial configuration of your local directory can be done using the [init command](/cli/commands/sync/init/). This command initializes -the directory and pulls configurations from the project. - -The **Storage API token** for your project is stored in the `.env.local` file under the `KBC_STORAGE_API_TOKEN` directive. -Currently, you must use a [master token](https://help.keboola.com/management/project/tokens/#master-tokens). -To maintain security, `.env.local` is automatically included in the .gitignore file to prevent it from being committed to your Git repository. - -[Manifest - Naming](#naming) defines directory names. Typically, this setting does not need to be changed. -Each object (branch, configuration, row) is guaranteed to have a unique directory, even if objects share the same name. - -Below is an example of a default project directory structure. Some files and directories are specific to the component type. -For example, transformations are represented by native files. A more detailed description can be found in the chapters below. - -
- -``` -🟫 .gitignore - excludes ".env.local" from the Git repository -🟫 .env.local - contains the Storage API token -🟫 .env.dist - template for ".env.local" -📂 .keboola - project metadata directory -┣ 🟦 manifest.json - contains object IDs, paths, naming and other configuration details -┣ 🟦 project.json - project cache for local commands, including backends and features -┗ 🟫 .kbcignore - optional file listing paths to configurations to exclude from CLI sync -🟩 description.md - project description -📂 [branch-name] - branch directory (e.g., "main") -┣ 🟦 meta.json -┣ 🟩 description.md -┣ 📂 _shared - shared code directory -┃ ┗ 📂 [target-component] - target component (e.g., "keboola.python-transfomation") -┃ ┗ 📂 codes -┃ ┗ 📂[code-name] - shared code directory -┃ ┣ 🟫 code.[ext] - native file (e.g., ".sql" or ".py") -┃ ┣ 🟦 config.json -┃ ┣ 🟦 meta.json -┃ ┗ 🟩 description.md -┗ 📂 [component-type] - e.g., extractor, app, ... - ┗ 📂 [component-id] - e.g., keboola.ex-db-oracle - ┗ 📂 [config-name] - configuration directory (e.g., "raw-data") - ┣ 🟦 config.json - ┣ 🟦 meta.json - ┣ 🟩 description.md - ┣ 📂 rows - only if the configuration has some rows - ┃ ┗ 📂 [row-name] - configuration row directory (e.g., "prod-fact-table") - ┃ ┣ 🟦 config.json - ┃ ┣ 🟦 meta.json - ┃ ┗ 🟩 description.md - ┣ 📂 blocks - only if the configuration is a transformation - ┃ ┗ 📂 001-block-1 - block directory - ┃ ┣ 🟦 meta.json - ┃ ┗ 📂 001-code-1 - code directory - ┃ ┣ 🟫 code.[ext] - native file (e.g., ".sql" or ".py") - ┃ ┗ 🟦 meta.json - ┣ 📂 phases - only if the configuration is an orchestration - ┃ ┗ 📂 001-phase - phase directory - ┃ ┣ 🟦 phase.json - ┃ ┗ 📂 001-task - task directory - ┃ ┗ 🟦 task.json - ┣ 📂 schedules - only if the configuration has some schedules - ┃ ┗ 📂 [schedule-name] - schedule directory - ┃ ┣ 🟦 config.json - ┃ ┣ 🟦 meta.json - ┃ ┗ 🟩 description.md - ┗ 📂 variables - only if the configuration has some variables defined - ┣ 🟦 config.json - variable definition, name, and type - ┣ 🟦 meta.json - ┣ 🟩 description.md - ┗ 📂 values - multiple sets of values can be defined - ┗ 📂 default - default values directory - ┣ 🟦 config.json - default values - ┣ 🟦 meta.json - ┗ 🟩 description.md -``` - -## Branches - -The tool works with [development branches](/components/branches/) by default. You can specify which branches from the project -you want to work with locally during the [init](/cli/commands/sync/init/) command. Alternatively, you can ignore the development branches concept and work exclusively -with the main branch. However, note that all configurations will then be stored in the `main` directory. - -The main branch directory is simply named `main` and does not include the branch ID. This makes it easily distinguishable from the other branches. - -Each branch directory contains: - -- `description.md`: Use this file to write a branch description formatted in [Markdown](https://www.markdownguide.org/). -- `meta.json`: Contains the name of the branch and a flag indicating whether it is the default branch. - -Example of `meta.json`: -```json -{ - "name": "Main", - "isDefault": true -} -``` - -Within the branch directory, configurations are organized into thematic directories: `extractor`, `other`, `transformation`, and `writer`. - -Example of a branch folder with components configurations: - -{: .image-popup} -![Screenshot -- A configuration directory example](/cli/structure/directory-example.jpg) - - -## Configurations - -Each configuration directory contains the following files: - -- `config.json`: Includes parameters specific to the component. -- `description.md`: A description file formatted in [Markdown](https://www.markdownguide.org/). -- `meta.json`: Contains the name of the configuration. - -Example of `config.json` for the Generic extractor: -```json -{ - "parameters": { - "api": { - "baseUrl": "https://wikipedia.org" - } - } -} -``` - -Example of `meta.json`: -```json -{ - "name": "Wikipedia" -} -``` - -Configuration directories can be copied freely within the project and between projects. Their IDs are stored -in the [manifest](/cli/structure/#manifest). After copying, run the [persist command](/cli/commands/local/persist/) to generate a new ID for the configuration and update it in the manifest. - -## Configuration Rows - -The directory structure for configuration rows is identical to that of configurations. The component configuration -includes a `rows` directory, which contains a subdirectory for each row. Each row directory includes `config.json`, -`description.md`, and `meta.json`. - -Example of `meta.json`: -```json -{ - "name": "share/cities2", - "isDisabled": false -} -``` - -Example of a Google Drive extractor configuration: - -{: .image-popup} -![Screenshot -- A configuration rows directory example](/cli/structure/directory-rows-example.jpg) - -## Transformations - -In addition to standard configurations, transformation directories include a `blocks` directory containing a list of codes. -Codes are stored as native files corresponding to the transformation type. For example, Snowflake transformations store codes -in `.sql` files. - -Example of a Snowflake transformation configuration: - -{: .image-popup} -![Screenshot -- A transformation directory example](/cli/structure/directory-transformation-example.jpg) - -## Variables - -The [variables](https://help.keboola.com/transformations/variables/#variables) directory, in addition to the standard -configuration layout, contains a `values` subdirectory. - -For example, suppose you have the following two variables in your transformation: - -{: .image-popup} -![Screenshot -- Variables in the UI](/cli/structure/variables-ui.jpg) - -When you [pull](/cli/commands/sync/pull/) them to the local directory, the structure will look like this: - -{: .image-popup} -![Screenshot -- Configuration directory with the variables](/cli/structure/variables-directory.jpg) - -Variables configuration in `variables/config.json`: - -```json -{ - "variables": [ - { - "name": "state", - "type": "string" - }, - { - "name": "city", - "type": "string" - } - ] -} -``` - -Default values configuration in `variables/values/default/config.json`: - -```json -{ - "values": [ - { - "name": "state", - "value": "NY" - }, - { - "name": "city", - "value": "Boston" - } - ] -} -``` - -## Configuration Variables Overview - -### One-to-One Relationship Variable Types - -1. **`variablesFor`**: This relation links a set of variables to a specific configuration or object that requires - them to function properly. It ensures the dependency between variables and their target is clearly defined. - -2. **`variablesFrom`**: This type indicates that variables are derived or used from another configuration, - allowing shared or inherited data between objects. It helps track the source of the variables. - -3. **`variablesValuesFor`**: This relation connects specific variable values to a configuration, often used in - scenarios like environment-specific settings. It enables precise association and management of variable values. - -4. **`variablesValuesFrom`**: It represents a connection where values are inherited from another configuration, - supporting reusability and consistent data sharing across objects. This avoids redundancy in value definitions. - -### Many-to-One Relationship Variable Types - -5. **`sharedCodeVariablesFor`**: Links shared code with associated variables, allowing multiple configurations to - refer to common logic. It promotes reuse and reduces duplication. - -6. **`sharedCodeVariablesFrom`**: This defines that variables are sourced from shared code, ensuring consistent - integration with centralized reusable logic. It maintains modularity and cohesion. - -7. **`schedulerFor`**: Connects a configuration to a scheduler, indicating tasks that need to be - timed or executed automatically. Commonly used in orchestration or automation setups. - -8. **`usedInOrchestrator`**: Marks an object as used within an orchestrator, linking configurations or components - required in workflows or pipelines. It ensures proper integration of orchestrated processes. - -9. **`usedInConfigInputMapping`**: Denotes that a configuration is used for input data mapping, linking data - sources to transformations. It is essential for aligning inputs with the appropriate configurations. - -10. **`usedInRowInputMapping`**: Similar to `usedInConfigInputMapping`, but applies to input mapping at - the row level within a configuration. It allows fine-grained control of input relationships. - -## Shared Code - -[Shared code](https://help.keboola.com/transformations/variables/#shared-code) blocks are stored in the branch directory -under the `_shared` subdirectory, enabling reuse across different configurations. - -If you create shared code from a block: - -{: .image-popup} -![Screenshot -- Shared code directory](/cli/structure/shared-code-ui.jpg) - -It will move to the `_shared` directory: - -{: .image-popup} -![Screenshot -- Shared code directory](/cli/structure/shared-code-directory.jpg) - -The code in the transformation file `blocks/block-1/join/code.sql` will then be replaced with: - -{: .image-popup} -![Screenshot -- Shared code code](/cli/structure/shared-code-code.jpg) - - -## Schedules - -The [Orchestrator](https://help.keboola.com/orchestrator/running) or any other component can have a schedule to run -automatically and periodically. The schedule configuration is stored within a specific directory. - -{: .image-popup} -![Screenshot -- Scheduler directory](/cli/structure/scheduler-directory.jpg) - -The `config.json` file for the schedule contains the schedule in [crontab](https://crontab.guru/) format, the timezone, and a flag -indiciating whether the schedule is enabled. - -For example, the following configuration runs at the 40th minute of every hour: - -```json -{ - "schedule": { - "cronTab": "40 */1 * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "mode": "run" - } -} -``` - -## Orchestrations - -Orchestrator directories include the `phases` directory, which contains a list of tasks for execution. - -Example: - -{: .image-popup} -![Screenshot -- An orchestration directory](/cli/structure/directory-orchestration-example.png) - -Example `phase.json`: - -```json -{ - "name": "Transformation", - "dependsOn": [ - "001-extraction" - ] -} -``` - -Example `task.json`: - -```json -{ - "name": "keboola.snowflake-transformation-7241628", - "task": { - "mode": "run", - "configPath": "transformation/keboola.snowflake-transformation/address-completion" - }, - "continueOnFailure": false, - "enabled": true -} -``` - -**Using `kbcdir.jsonnet` for different orchestration phases:** - -The `kbcdir.jsonnet` file can be used to specify which directories in the phases folder should be ignored for different project backends. By setting the `isIgnored` value to true in the file, you can exclude specific directories. - -Example `kbcdir.jsonnet`: -```jsonnet -{ - "isIgnored":false -} -``` - -## Manifest - -The local state of the project is stored in the `.keboola/manifest.json` file. It is not recommended to modify -this file manually. - -### Basic Manifest Structure - -- `version`: Current major version (e.g., `2`) -- `project`: Information about the project - - `id`: ID of the project - - `apiHost`: URL of the Keboola instance (e.g., `connection.keboola.com`) -- `allowTargetEnv`: Boolean (default: `false`) - - If `true`, allows environment variables `KBC_PROJECT_ID` and `KBC_BRANCH_ID` to temporary override the target project and branch without modifying the manifest. - - The mapping is bidirectional and occurs during the manifest's save and load operations. - - For more information, see the [--allow-target-env](/cli/commands/sync/init/#options) option in the [kbc sync init](/cli/commands/sync/init/) command. -- `sortBy`: Property name used for sorting configurations (default: `id`) -- `naming`: Rules for directory naming ([see details](/cli/structure/#naming)) -- `allowedBranches`: Array of branches to work with -- `ignoredComponents`: Array of components to exclude -- `templates`: - - `repositories` (*array*): - - Local repository: - - `type` = `dir` - - `name`: Repository name - - `url`: Absolute or relative path to a local directory - - Relative path must be relative to the project directory. - - Git-based repository: - - `type` = `git` - - `name`: Repository name - - `url`: URL of the Git repository - - E.g., `https://github.com/keboola/keboola-as-code-templates.git` - - `ref`: Git `branch` or `tag` (e.g., `main` or `v1.2.3`) -- `branches`: List of used branches - - `id`: Branch ID - - `path`: Directory name (e.g., `main`) -- `configurations`: List of component configurations - - `branchId`: Branch ID - - `componentId`: Component ID (e.g., `keboola.ex-aws-s3`) - - `id`: Configuration ID - - `path`: Path to the configuration in the local directory (e.g., `extractor/keboola.ex-aws-s3/7241111/my-aws-s3-data-source`) - - `rows`: List of configuration rows (if the component supports rows) - - `id`: Row ID - - `path`: Path to the row from the configuration directory (e.g., `rows/cities`) - -### Naming - -Directory names for configurations follow the rules in the [manifest](/cli/structure/#manifest) under the `naming` section. -These are the default values: - -```json -{ - "branch": "{branch_name}", - "config": "{component_type}/{component_id}/{config_name}", - "configRow": "rows/{config_row_name}", - "schedulerConfig": "schedules/{config_name}", - "sharedCodeConfig": "_shared/{target_component_id}", - "sharedCodeConfigRow": "codes/{config_row_name}", - "variablesConfig": "variables", - "variablesValuesRow": "values/{config_row_name}" - } -``` - -To include object IDs in directory names, use these values: - -```json -{ - "branch": "{branch_id}-{branch_name}", - "config": "{component_type}/{component_id}/{config_id}-{config_name}", - "configRow": "rows/{config_row_id}-{config_row_name}", - "schedulerConfig": "schedules/{config_name}", - "sharedCodeConfig": "_shared/{target_component_id}", - "sharedCodeConfigRow": "codes/{config_row_name}", - "variablesConfig": "variables", - "variablesValuesRow": "values/{config_row_name}" - } -``` - -Use the [fix-paths](/cli/commands/local/fix-paths/) command to rebuild the directory structure with updated naming rules. - -## Project Cache - -The project cache is stored in `.keboola/project.json` and is used by local commands without making authorized requests to the Storage API. - -This is its basic structure: - -- `backends`: List of project backends -- `features`: List of project features -- `defaultBranchId`: ID of the default branch - -Example: - -```json -{ - "backends": [ - "snowflake" - ], - "features": [ - "workspace-snowflake-dynamic-backend-size", - "input-mapping-read-only-storage", - "oauth-v3" - ], - "defaultBranchId": 123 -} -``` -## .kbcignore - -You can exclude specific configurations, configuration rows, or individual fields from the sync process by creating a `.kbcignore` file in the `.keboola` directory. - -It is a plain text file where each line specifies either: - -- A path to a **configuration or configuration row** to exclude entirely, in the format `{component_id}/{configuration_id}/{row_id}` (the `row_id` is optional for [row-based configurations](https://help.keboola.com/components/#configuration-rows)). -- A **field-level ignore** rule in the format `{component_id}/{configuration_id}:{field_name}`, which excludes a single field from synchronization while keeping the rest of the configuration managed by the CLI. - -Comments (lines starting with `#`) and empty lines are ignored. - -### Configuration and Row Ignore - -Example `.kbcignore` file: - -``` -keboola.python-transformation-v2/1197618481 -keboola.keboola.wr-db-snowflake/1196309603/1196309605 -``` - -This excludes: - -- The configuration of the Python transformation (`keboola.python-transformation-v2`) with the ID `1197618481`. -- Row ID `1196309605` in the configuration of the Snowflake writer (`keboola.keboola.wr-db-snowflake`) with the ID `1196309603`. - -As a result, the `kbc sync pull` and `kbc sync push` commands will not synchronize these configurations. - -### Field-Level Ignore - -In addition to ignoring entire configurations or rows, you can ignore **individual fields** within a configuration. -This is useful when you want to manage most of a configuration via the CLI but let a specific field be controlled -exclusively in the Keboola UI (or vice versa). - -The syntax is: - -``` -{component_id}/{configuration_id}:{field_name} -``` - -Where `field_name` is either: - -- A **struct-level field** of the configuration — currently `isDisabled` is supported. -- A **dot-notation content key** referring to a path inside the configuration's `config.json` content, e.g., `schedule.cronTab`. - -Example `.kbcignore` with field-level rules: - -``` -# Ignore the isDisabled flag — let the UI control whether this config is enabled -ex-generic-v2/798412456:isDisabled - -# Ignore the cron schedule — let the UI control the schedule timing -keboola.scheduler/801234567:schedule.cronTab -``` - -Field-level ignore is **bidirectional**: - -- **`kbc sync push`**: The remote value of the ignored field is kept. Any local change to that field is discarded before - the diff is computed, so the field is never pushed. -- **`kbc sync pull`**: The local value of the ignored field is kept. Any remote change to that field is discarded before - the diff is computed, so the field is never pulled. - -In both cases the rest of the configuration is synchronized normally. - -
- -### Configuration-Level Ignore - -**`kbc push` operation** - -The `kbc push` command will skip the excluded configurations and will not push them back to the project, even if they exist or have been modified in the local folder structure. -The log will display the following message: - -``` -➜ kbc push -Plan for "push" operation: - × main/transformation/keboola.python-transformation-v2/dev-l0-sample-data - IGNORED -Skipped remote objects deletion, use "--force" to delete them. -Push done. -``` - -The log clearly identifies configurations that were ignored, even if they are absent in the local folder structure. - -**`kbc pull` operation** - -The `kbc pull` command will exclude the matched configurations and not pull them from the project. - -
-

Warning:
- If the matched configuration is already present locally, it will be deleted from both the filesystem and manifest.json.

-
- - -If the configuration was already present locally, the log will indicate its deletion as shown below: - -``` -➜ kbc pull -Plan for "pull" operation: - × C main/writer/keboola.wr-db-snowflake/my-snowflake-data-destination - × R main/writer/keboola.wr-db-snowflake/my-snowflake-data-destination/rows/test-sheet1 -Pull done. -``` - -## Next Steps - -- [Commands](/cli/commands/) +The Keboola as Code CLI documentation has been retired. The tool itself lives at [keboola/keboola-as-code](https://github.com/keboola/keboola-as-code). This page's content remains in this repository's git history. diff --git a/cli/structure/scheduler-directory.jpg b/cli/structure/scheduler-directory.jpg deleted file mode 100644 index afe1ae2b..00000000 Binary files a/cli/structure/scheduler-directory.jpg and /dev/null differ diff --git a/cli/structure/shared-code-code.jpg b/cli/structure/shared-code-code.jpg deleted file mode 100644 index 633abe80..00000000 Binary files a/cli/structure/shared-code-code.jpg and /dev/null differ diff --git a/cli/structure/shared-code-directory.jpg b/cli/structure/shared-code-directory.jpg deleted file mode 100644 index 6df81929..00000000 Binary files a/cli/structure/shared-code-directory.jpg and /dev/null differ diff --git a/cli/structure/shared-code-ui.jpg b/cli/structure/shared-code-ui.jpg deleted file mode 100644 index 0b9bc7fc..00000000 Binary files a/cli/structure/shared-code-ui.jpg and /dev/null differ diff --git a/cli/structure/variables-directory.jpg b/cli/structure/variables-directory.jpg deleted file mode 100644 index 0b913295..00000000 Binary files a/cli/structure/variables-directory.jpg and /dev/null differ diff --git a/cli/structure/variables-ui.jpg b/cli/structure/variables-ui.jpg deleted file mode 100644 index bf506e17..00000000 Binary files a/cli/structure/variables-ui.jpg and /dev/null differ