diff --git a/_data/navigation.yml b/_data/navigation.yml index d448777da..546d7762c 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -4,6 +4,11 @@ items: - url: /overview/ title: Keboola Overview + items: + - url: /overview/api/ + title: Our APIs + - url: /overview/encryption/ + title: Encryption - url: /tutorial/ title: Getting Started Tutorial diff --git a/public/overview/api/apiary-console.png b/public/overview/api/apiary-console.png new file mode 100644 index 000000000..1965d06e3 Binary files /dev/null and b/public/overview/api/apiary-console.png differ diff --git a/public/overview/api/postman-import.png b/public/overview/api/postman-import.png new file mode 100644 index 000000000..12333eb04 Binary files /dev/null and b/public/overview/api/postman-import.png differ diff --git a/public/overview/encryption-1.png b/public/overview/encryption-1.png new file mode 100644 index 000000000..9d28c0707 Binary files /dev/null and b/public/overview/encryption-1.png differ diff --git a/public/overview/encryption-2.png b/public/overview/encryption-2.png new file mode 100644 index 000000000..51bb3da41 Binary files /dev/null and b/public/overview/encryption-2.png differ diff --git a/src/content/docs/components/index.md b/src/content/docs/components/index.md index 4c2469ff8..7e7ab83ca 100644 --- a/src/content/docs/components/index.md +++ b/src/content/docs/components/index.md @@ -138,7 +138,7 @@ The [developer guide](https://developers.keboola.com/integrate/storage/api/confi **Important**: Component configurations do not count towards your project quota. The version list is unlimited. Configuration versions are also created when the configurations are manipulated -programmatically using [the API](https://developers.keboola.com/overview/api/). In other words, all configuration modifications are recorded. +programmatically using [the API](/overview/api/). In other words, all configuration modifications are recorded. ### Compare Versions You can compare adjacent versions by clicking the *compare* icon: @@ -151,7 +151,7 @@ The differences between the raw JSON configurations are displayed when you compa When you roll back a configuration, a new version is created. This means you never lose any version of a configuration, and there is always an option to get back to it. Configuration versions are also created when -the configurations are manipulated programmatically via [the API](https://developers.keboola.com/overview/api/). +the configurations are manipulated programmatically via [the API](/overview/api/). ### Rollback Version If you need to return to an older version of the configuration, you can also roll back to it (the other option is to make its copy). diff --git a/src/content/docs/components/ip-addresses/index.md b/src/content/docs/components/ip-addresses/index.md index ab3f04322..758bb33fb 100644 --- a/src/content/docs/components/ip-addresses/index.md +++ b/src/content/docs/components/ip-addresses/index.md @@ -17,7 +17,7 @@ to successfully connect to your system. This applies to all components including **Important:** These IP addresses can change in the future! For your convenience, you can programmatically fetch and process the [list of existing IP addresses in JSON format](/components/ip-addresses/kbc-public-ip.json). -Below are listed the available [Keboola Stack endpoints](https://developers.keboola.com/overview/api/#regions-and-endpoints). +Below are listed the available [Keboola Stack endpoints](/overview/api/#regions-and-endpoints). For ease of identification, our outbound IP addresses on AWS stacks (except for legacy services) now have reverse DNS records. Each IP address has a unique name like `outbound-if-issue-contact-support-at-keboola-com.keboola.com`, embedding a reference for a support email. diff --git a/src/content/docs/external-integrations/index.md b/src/content/docs/external-integrations/index.md index 2bdf56546..2db0a945c 100644 --- a/src/content/docs/external-integrations/index.md +++ b/src/content/docs/external-integrations/index.md @@ -5,7 +5,7 @@ slug: 'external-integrations' -Keboola's external integration capabilities let you seamlessly extend the platform with your existing tools and workflows. By connecting to Keboola through our [REST API](https://developers.keboola.com/overview/api) or [MCP (Model Context Protocol)](/ai/mcp-server/) server, you can orchestrate data pipelines, trigger jobs, and embed Keboola into a broader automation ecosystem. +Keboola's external integration capabilities let you seamlessly extend the platform with your existing tools and workflows. By connecting to Keboola through our [REST API](/overview/api) or [MCP (Model Context Protocol)](/ai/mcp-server/) server, you can orchestrate data pipelines, trigger jobs, and embed Keboola into a broader automation ecosystem. > Keboola integrates with external tools at multiple levels, from low-level API access to ready-made workflow platforms. This flexibility allows you to choose the approach that best fits your team's needs. diff --git a/src/content/docs/external-integrations/n8n/index.md b/src/content/docs/external-integrations/n8n/index.md index c6cecb1af..40f490259 100644 --- a/src/content/docs/external-integrations/n8n/index.md +++ b/src/content/docs/external-integrations/n8n/index.md @@ -107,7 +107,7 @@ If you need functionality that isn’t covered by the node’s built-in actions, ## Resources -- [Keboola API Documentation](https://developers.keboola.com/overview/api) +- [Keboola API Documentation](/overview/api) - [n8n Documentation](https://docs.n8n.io) - [n8n Community Nodes Guide](https://docs.n8n.io/integrations/#community-nodes) - [NPM Package](https://www.npmjs.com/package/@keboola/n8n-nodes-keboola) diff --git a/src/content/docs/management/project/tokens/index.md b/src/content/docs/management/project/tokens/index.md index 4bbe743f1..dbf864ac5 100644 --- a/src/content/docs/management/project/tokens/index.md +++ b/src/content/docs/management/project/tokens/index.md @@ -21,7 +21,7 @@ Normally, when you are using the user interface, your API token is exchanged aut the server backend. Therefore you need to work with tokens only when working with Keboola programmatically (or if you need to limit a user's authorization to certain operations or data). To learn more about all the available programmatic approaches, please follow our -[developers documentation](https://developers.keboola.com/overview/api/). +[developers documentation](/overview/api/). Tokens can be managed from the **Project Settings > API Tokens** page. @@ -49,7 +49,7 @@ API tokens are created Automatically created tokens have the lowest possible permissions for their task and also set expiration if possible. These are the typical reasons to manually create a new API token: -- You want to use the [APIs](https://developers.keboola.com/overview/api/); this includes all of the [Storage clients](https://developers.keboola.com/integrate/storage/#storage-api-clients). +- You want to use the [APIs](/overview/api/); this includes all of the [Storage clients](https://developers.keboola.com/integrate/storage/#storage-api-clients). - You need to limit access to certain data (for example, share a single table) or components. Although tokens cannot be used to directly log in to the Keboola user interface, they do allow executing almost all @@ -163,7 +163,7 @@ people can send data directly to your Keboola project instead of struggling with To revoke the access, simply delete or refresh the token. The token can then be used with the [Storage API](https://developers.keboola.com/integrate/) -or [other APIs](https://developers.keboola.com/overview/api/). +or [other APIs](/overview/api/). ### Storage Console Typical usecase of sharing a token with someone is giving them a partial access to your project storage. The @@ -177,7 +177,7 @@ The Storage Console allows some basic operations with the project [Storage](/sto ![Screenshot - Storage Console](/management/project/tokens/storage-console.png) The link to the Storage API Console is available at the token retrieval page as it is different for each -[region](https://developers.keboola.com/overview/api/): +[region](/overview/api/): - [AWS US Region](https://storage-api-console.keboola.com/?endpoint=https%3A%2F%2Fconnection.keboola.com) - [AWS EU Region](https://storage-api-console.keboola.com/?endpoint=https%3A%2F%2Fconnection.eu-central-1.keboola.com) diff --git a/src/content/docs/overview/api/apiary-console.png b/src/content/docs/overview/api/apiary-console.png new file mode 100644 index 000000000..1965d06e3 Binary files /dev/null and b/src/content/docs/overview/api/apiary-console.png differ diff --git a/src/content/docs/overview/api/index.md b/src/content/docs/overview/api/index.md new file mode 100644 index 000000000..e055086ff --- /dev/null +++ b/src/content/docs/overview/api/index.md @@ -0,0 +1,250 @@ +--- +title: Our APIs +slug: 'overview/api' +--- + + +All our [Keboola services](/overview/) have a public API on [api.keboola.com](https://api.keboola.com/). We recommend using either the API Console or Postman Client for sending requests to our +API. Most of our APIs accept and return data in JSON format. +Many of these APIs require a *Storage API token*, specified in the `X-StorageApi-Token` header. + +## List of Keboola APIs + +All parts of the Keboola platform can be controlled via an API. +The main APIs for our components are: + +
+Note: The api.keboola.com links in the table below open the API documentation portal for the US Virginia AWS stack. +If you are using a different stack, navigate to your stack's API portal first — see API Documentation Portals below — and then select the service there. +Using a portal for a different stack than your token's stack will result in Invalid Token errors. +
+ +| API | Description | +|-------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Keboola Storage API](https://api.keboola.com/?service=storage) ([source](https://github.com/keboola/storage-api-php-client/blob/master/apiary.apib)) | [Storage](/storage/) is the main Keboola component storing all data. | +| [Keboola Management API](https://api.keboola.com/?service=manage) | API managing Keboola projects and users (and notifications and features). | +| [AI API](https://api.keboola.com/?service=ai) | API for supporting AI features. | +| [Billing API](https://api.keboola.com/?service=billing) | Billing API for Pay as You Go projects. | +| [Developer Portal API](https://api.keboola.com/?service=developer-portal) | Developer Portal is an application separated from Keboola for [creating components](https://developers.keboola.com/extend/component/). | +| [Editor API](https://api.keboola.com/?service=editor) | API for managing SQL editor sessions. | +| [Encryption API](https://api.keboola.com/?service=encryption) | Provides [Encryption](/overview/encryption/). | +| [Importer API](https://api.keboola.com/?service=import) | [Importer](https://developers.keboola.com/integrate/storage/api/importer/) is a helper service for easy table imports. | +| [Notifications API](https://api.keboola.com/?service=notification) | API to subscribe to events, e.g., failed orchestrations. | +| [OAuth Broker API](https://api.keboola.com/?service=oauth) | OAuth Broker is a component managing [OAuth authorizations](https://developers.keboola.com/extend/common-interface/oauth/#authorize) of other components. | +| [Query API](https://api.keboola.com/?service=query) | Query is a service for running SQL queries on Snowflake and BigQuery. | +| [Queue API](https://api.keboola.com/?service=job-queue) | Queue is a service for [running components](https://developers.keboola.com/extend/job-queue/) and managing [Jobs](https://developers.keboola.com/integrate/jobs/). | +| [Sandboxes Service API](https://api.keboola.com/?service=sandboxes-service) | API for managing Apps and Python/R workspaces. | +| [Scheduler API](https://api.keboola.com/?service=scheduler) | API to automate configurations. | +| [Stream API](https://api.keboola.com/?service=stream) | The Keboola Stream API allows you to ingest small and frequent events into your project's storage. | +| [Synchronous Actions API](https://api.keboola.com/?service=sync-actions) | API to trigger [Synchronous Actions](https://developers.keboola.com/extend/common-interface/actions/). | +| [Vault](https://api.keboola.com/?service=vault) | Service handling variables & credentials storage. | + +If you're unsure which API to use, refer to our [integration guide](https://developers.keboola.com/integrate/). It describes the roles of different APIs and contains examples of commonly +performed actions. + +## Stacks and Endpoints +Keboola is available in multiple [stacks](/overview/#stacks), which can be +either multi-tenant or single-tenant. Current multi-tenant stacks are: + +- US Virginia AWS – [connection.keboola.com](https://connection.keboola.com/) +- US Virginia GCP - [connection.us-east4.gcp.keboola.com](https://connection.us-east4.gcp.keboola.com/) +- EU Frankfurt AWS – [connection.eu-central-1.keboola.com](https://connection.eu-central-1.keboola.com/) +- EU Ireland Azure – [connection.north-europe.azure.keboola.com](https://connection.north-europe.azure.keboola.com/) +- EU Frankfurt GCP - [connection.europe-west3.gcp.keboola.com](https://connection.europe-west3.gcp.keboola.com/) + +Each stack operates as an independent instance of Keboola services with its own data, users, and tokens. +Single-tenant stacks are available for a single enterprise customer, with a domain name +in the format `connection.CUSTOMER_NAME.keboola.com`. + +### API Documentation Portals + +The API documentation portal (`api.*`) is deployed independently per stack. Always use the portal +for your own stack — tokens are not valid across stacks, and using the wrong portal will cause +`Invalid Token` errors when trying out API calls. + +| Stack | API Documentation Portal | +|---|---| +| US Virginia AWS | [api.keboola.com](https://api.keboola.com/) | +| EU Frankfurt AWS | [api.eu-central-1.keboola.com](https://api.eu-central-1.keboola.com/) | +| EU Ireland Azure | [api.north-europe.azure.keboola.com](https://api.north-europe.azure.keboola.com/) | +| EU Frankfurt GCP | [api.europe-west3.gcp.keboola.com](https://api.europe-west3.gcp.keboola.com/) | +| US Virginia GCP | [api.us-east4.gcp.keboola.com](https://api.us-east4.gcp.keboola.com/) | + +### Service Endpoints + +If you are calling the APIs directly (not through the portal), modify the hostname accordingly. +Otherwise, you may encounter `Invalid Token` or unauthorized errors. The *authoritative list* of available endpoints is provided by the [Storage API Index Call](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/components/-componentId-). The following is a sample response: + +```json +{ + ..., + "services": [ + { + "id": "import", + "url": "https://import.keboola.com" + }, + { + "id": "oauth", + "url": "https://oauth.keboola.com" + }, + { + "id": "queue", + "url": "https://queue.keboola.com" + }, + { + "id": "billing", + "url": "https://billing.keboola.com" + }, + { + "id": "encryption", + "url": "https://encryption.keboola.com" + }, + { + "id": "scheduler", + "url": "https://scheduler.keboola.com" + }, + { + "id": "sync-actions", + "url": "https://sync-actions.keboola.com" + }, + { + "id": "notification", + "url": "https://notification.keboola.com" + } + ], +} +``` + +The services listed above are: + +- `import` --- [Storage Importer Service](https://developers.keboola.com/integrate/storage/api/importer/) +- `oauth` --- [OAuth Manager Service](https://developers.keboola.com/extend/common-interface/oauth/) +- `queue` --- [Service for Running Components](https://developers.keboola.com/extend/job-queue/) +- `billing` --- Service for Computing Credits +- `encryption` --- Service for [Encryption](/overview/encryption/) +- `scheduler` --- [Service for Configuring Schedules](https://developers.keboola.com/automate/set-schedule/) +- `sync-actions` --- [Service for Running Synchronous Actions](https://developers.keboola.com/extend/common-interface/actions/) +- `notification` --- Service for Configuring Job Notifications + +For convenience, the following table lists active services and their URLs, though for an authoritative answer +and in application integrations, we strongly suggest using the above API call. + +| API | Service | Region | URL | +|------------------------|----------------|------------------|-----------------------------------------------------| +| AI | `ai` | US Virginia AWS | https://ai.keboola.com | +| AI | `ai` | US Virginia GCP | https://ai.us-east4.gcp.keboola.com | +| AI | `ai` | EU Frankfurt AWS | https://ai.eu-central-1.keboola.com | +| AI | `ai` | EU Ireland Azure | https://ai.north-europe.azure.keboola.com | +| AI | `ai` | EU Frankfurt GCP | https://ai.europe-west3.gcp.keboola.com | +| Billing | `billing` | US Virginia AWS | https://billing.keboola.com | +| Billing | `billing` | US Virginia GCP | https://billing.us-east4.gcp.keboola.com | +| Billing | `billing` | EU Frankfurt AWS | https://billing.eu-central-1.keboola.com | +| Billing | `billing` | EU Ireland Azure | https://billing.north-europe.azure.keboola.com | +| Billing | `billing` | EU Frankfurt GCP | https://billing.europe-west3.gcp.keboola.com | +| Developer Portal | `developer` | US Virginia AWS | https://developer.keboola.com | +| Developer Portal | `developer` | US Virginia GCP | https://developer.us-east4.gcp.keboola.com | +| Developer Portal | `developer` | EU Frankfurt AWS | https://developer.eu-central-1.keboola.com | +| Developer Portal | `developer` | EU Ireland Azure | https://developer.north-europe.azure.keboola.com | +| Developer Portal | `developer` | EU Frankfurt GCP | https://developer.europe-west3.gcp.keboola.com | +| Editor | `editor` | US Virginia AWS | https://editor.keboola.com | +| Editor | `editor` | US Virginia GCP | https://editor.us-east4.gcp.keboola.com | +| Editor | `editor` | EU Frankfurt AWS | https://editor.eu-central-1.keboola.com | +| Editor | `editor` | EU Ireland Azure | https://editor.north-europe.azure.keboola.com | +| Editor | `editor` | EU Frankfurt GCP | https://editor.europe-west3.gcp.keboola.com | +| Encryption | `encryption` | US Virginia AWS | https://encryption.keboola.com | +| Encryption | `encryption` | US Virginia GCP | https://encryption.us-east4.gcp.keboola.com | +| Encryption | `encryption` | EU Frankfurt AWS | https://encryption.eu-central-1.keboola.com | +| Encryption | `encryption` | EU Ireland Azure | https://encryption.north-europe.azure.keboola.com | +| Encryption | `encryption` | EU Frankfurt GCP | https://encryption.europe-west3.gcp.keboola.com | +| Importer | `import` | US Virginia AWS | https://import.keboola.com | +| Importer | `import` | US Virginia GCP | https://import.us-east4.gcp.keboola.com | +| Importer | `import` | EU Frankfurt AWS | https://import.eu-central-1.keboola.com | +| Importer | `import` | EU Ireland Azure | https://import.north-europe.azure.keboola.com | +| Importer | `import` | EU Frankfurt GCP | https://import.europe-west3.gcp.keboola.com | +| Management | `management` | US Virginia AWS | https://management.keboola.com | +| Management | `management` | US Virginia GCP | https://management.us-east4.gcp.keboola.com | +| Management | `management` | EU Frankfurt AWS | https://management.eu-central-1.keboola.com | +| Management | `management` | EU Ireland Azure | https://management.north-europe.azure.keboola.com | +| Management | `management` | EU Frankfurt GCP | https://management.europe-west3.gcp.keboola.com | +| Notification | `notification` | US Virginia AWS | https://notification.keboola.com | +| Notification | `notification` | US Virginia GCP | https://notification.us-east4.gcp.keboola.com | +| Notification | `notification` | EU Frankfurt AWS | https://notification.eu-central-1.keboola.com | +| Notification | `notification` | EU Ireland Azure | https://notification.north-europe.azure.keboola.com | +| Notification | `notification` | EU Frankfurt GCP | https://notification.europe-west3.gcp.keboola.com | +| OAuth | `oauth` | US Virginia AWS | https://oauth.keboola.com | +| OAuth | `oauth` | US Virginia GCP | https://oauth.europe-west3.gcp.keboola.com | +| OAuth | `oauth` | EU Frankfurt AWS | https://oauth.eu-central-1.keboola.com | +| OAuth | `oauth` | EU Ireland Azure | https://oauth.north-europe.azure.keboola.com | +| OAuth | `oauth` | EU Frankfurt GCP | https://oauth.europe-west3.gcp.keboola.com | +| Query | `query` | US Virginia AWS | https://query.keboola.com | +| Query | `query` | US Virginia GCP | https://query.us-east4.gcp.keboola.com | +| Query | `query` | EU Frankfurt AWS | https://query.eu-central-1.keboola.com | +| Query | `query` | EU Ireland Azure | https://query.north-europe.azure.keboola.com | +| Query | `query` | EU Frankfurt GCP | https://query.europe-west3.gcp.keboola.com | +| Queue | `queue` | US Virginia AWS | https://queue.keboola.com | +| Queue | `queue` | US Virginia GCP | https://queue.us-east4.gcp.keboola.com | +| Queue | `queue` | EU Frankfurt AWS | https://queue.eu-central-1.keboola.com | +| Queue | `queue` | EU Ireland Azure | https://queue.north-europe.azure.keboola.com | +| Queue | `queue` | EU Frankfurt GCP | https://queue.europe-west3.gcp.keboola.com | +| Scheduler | `scheduler` | US Virginia AWS | https://scheduler.keboola.com | +| Scheduler | `scheduler` | US Virginia GCP | https://scheduler.us-east4.gcp.keboola.com | +| Scheduler | `scheduler` | EU Frankfurt AWS | https://scheduler.eu-central-1.keboola.com | +| Scheduler | `scheduler` | EU Ireland Azure | https://scheduler.north-europe.azure.keboola.com | +| Scheduler | `scheduler` | EU Frankfurt GCP | https://scheduler.europe-west3.gcp.keboola.com | +| Storage | | US Virginia AWS | https://connection.keboola.com/ | +| Storage | | US Virginia GCP | https://connection.us-east4.gcp.keboola.com | +| Storage | | EU Frankfurt AWS | https://connection.eu-central-1.keboola.com/ | +| Storage | | EU Ireland Azure | https://connection.north-europe.azure.keboola.com/ | +| Storage | | EU Frankfurt GCP | https://connection.europe-west3.gcp.keboola.com/ | +| Stream | `stream` | US Virginia AWS | https://stream.keboola.com | +| Stream | `stream` | US Virginia GCP | https://stream.us-east4.gcp.keboola.com | +| Stream | `stream` | EU Frankfurt AWS | https://stream.eu-central-1.keboola.com | +| Stream | `stream` | EU Ireland Azure | https://stream.north-europe.azure.keboola.com | +| Stream | `stream` | EU Frankfurt GCP | https://stream.europe-west3.gcp.keboola.com | +| Sync Actions | `sync-actions` | US Virginia AWS | https://sync-actions.keboola.com/ | +| Sync Actions | `sync-actions` | US Virginia GCP | https://sync-actions.us-east4.gcp.keboola.com | +| Sync Actions | `sync-actions` | EU Frankfurt AWS | https://sync-actions.eu-central-1.keboola.com | +| Sync Actions | `sync-actions` | EU Ireland Azure | https://sync-actions.north-europe.azure.keboola.com | +| Sync Actions | `sync-actions` | EU Frankfurt GCP | https://sync-actions.europe-west3.gcp.keboola.com | +| Vault | `vault` | US Virginia AWS | https://vault.keboola.com | +| Vault | `vault` | US Virginia GCP | https://vault.us-east4.gcp.keboola.com | +| Vault | `vault` | EU Frankfurt AWS | https://vault.eu-central-1.keboola.com | +| Vault | `vault` | EU Ireland Azure | https://vault.north-europe.azure.keboola.com | +| Vault | `vault` | EU Frankfurt GCP | https://vault.europe-west3.gcp.keboola.com | + +***Important**: Each stack also uses its own set of [IP addresses](/components/ip-addresses/).* + +## Calling API + +There are several ways to send requests to our APIs: + +### Apiary Console +Send requests to our API directly from the Apiary console by clicking on **Switch to console** or **Try**. +Fill in the request headers and parameters, then click **Call Resource**. + +![Apiary console](/overview/api/apiary-console.png) + +The Apiary console is fine if you send API requests only occasionally. It requires no application installation; +however, it has no history and no other useful features. + +### Postman Client +[Postman](https://www.getpostman.com/) is a generic HTTP API client, suitable for more regular API work. +We also provide a collection of [useful API calls](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb) with examples. +The collection contains code examples in various languages; the requests can also be imported into the Postman application. + +![Postman Docs](/overview/api/postman-import.png) + +### cURL +[cURL](https://curl.haxx.se/) is a common library with a [command-line interface (CLI)](https://curl.haxx.se/docs/manpage.html). +You can use the cURL CLI to create simple scripts for interacting with Keboola APIs. For example, to [run a job](https://developers.keboola.com/integrate/jobs/): + +```shell +curl --location --request POST 'https://queue.keboola.com/jobs' \ +--header 'X-StorageApi-Token: YourStorageToken' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "mode": "run", + "component": "keboola.ex-db-mysql", + "config": "sampledatabase" +}' +``` diff --git a/src/content/docs/overview/api/postman-import.png b/src/content/docs/overview/api/postman-import.png new file mode 100644 index 000000000..12333eb04 Binary files /dev/null and b/src/content/docs/overview/api/postman-import.png differ diff --git a/src/content/docs/overview/encryption-1.png b/src/content/docs/overview/encryption-1.png new file mode 100644 index 000000000..9d28c0707 Binary files /dev/null and b/src/content/docs/overview/encryption-1.png differ diff --git a/src/content/docs/overview/encryption-2.png b/src/content/docs/overview/encryption-2.png new file mode 100644 index 000000000..51bb3da41 Binary files /dev/null and b/src/content/docs/overview/encryption-2.png differ diff --git a/src/content/docs/overview/encryption/index.md b/src/content/docs/overview/encryption/index.md new file mode 100644 index 000000000..23832b7b5 --- /dev/null +++ b/src/content/docs/overview/encryption/index.md @@ -0,0 +1,146 @@ +--- +title: Encryption +slug: 'overview/encryption' +--- + + +Many [Keboola components](/overview/) use the Encryption API to encrypt sensitive values +intended for secure storage. These values are then decrypted within the component itself. +This process ensures that the encrypted values are only accessible inside the components and not +by API users. Additionally, no decryption API is available, meaning end-users cannot decrypt +these values. + +Decryption occurs solely during the serialization of configuration to the Docker container's +configuration file. The decrypted data are stored on the Docker host drive and are promptly +deleted after the container's completion. The component code exclusively accesses the decrypted data. + +## UI Interaction +When saving arbitrary configuration data, if a key is prefixed with the `#` character, the associated value is automatically encrypted. +For instance, consider the following configuration: + +![Screenshot - Configuration editor - before](/overview/encryption-1.png) + +After saving, the configuration appears as follows: + +![Screenshot - Configuration editor - after](/overview/encryption-2.png) + +Once saved, the value becomes encrypted and irreversible. The component defines which values are +encrypted, indicating that not all values can be encrypted unless explicitly supported by the component. + +For example, a component requiring the following configuration: + +```json +{ + "username": "JohnDoe", + "#password": "password" +} +``` + +indicates that the password will be encrypted while the username will not. Adding a +prefix `#` to `username` is ineffective, as the component does not recognize such a key, +even though its value would be encrypted and decrypted normally. Internally, the +[Encryption API](#encrypting-data-with-api) encrypts these values before saving. + +### UI Configuration Adjustment +The UI prioritizes encrypted values over plain ones. If both `password` and `#password` are provided, only `#password` will be retained. +Consequently, this configuration: + +```json +{ + "username": "JohnDoe", + "#password": "KBC::ProjectSecure::ENCODEDSTRING", + "password": "secret", +} +``` + +will be transformed to: + +```json +{ + "username": "JohnDoe", + "#password": "KBC::ProjectSecure::ENCODEDSTRING" +} +``` + +## Encrypting Data with API +The [Encryption API](https://api.keboola.com/?service=encryption#post-/encrypt) can handle +both strings and arbitrary JSON data. For strings, the entire string is encrypted. In JSON data, +only scalar keys starting with `#` are encrypted. For example, encrypting the following: + +```json +{ + "foo": "bar", + "#encryptMe": "secret", + "#encryptMeToo": { + "another": "secret" + } +} +``` + +results in: + +```json +{ + "foo": "bar", + "#encryptMe": "KBC::ProjectSecure::ENCODEDSTRING", + "#encryptMeToo": { + "another": "secret" + } +} +``` + +To encrypt a single string, such as a password, submit the text string for encryption +(no JSON or quotation is used). For example, encrypting + + mySecretPassword + +yields + + KBC::ProjectSecure::ENCODEDSTRING + +The `Content-Type` header in the request differentiates whether the body is treated as a string (`text/plain`) or JSON (`application/json`). + +### Encryption Parameters +The Encryption API accepts the following **optional** parameters: + +- `componentId` --- ID of a [Keboola component](https://developers.keboola.com/extend/component/tutorial/#creating-a-component), +- `projectId` --- ID of a Keboola project, +- `configId` --- ID of a component configuration, and +- `branchType` --- Branch type --- either `default` (meaning the default production branch) or `dev` (meaning any development branch other than the production). + +The cipher created depends on the provided parameters: + +- With only `componentId`, the cipher starts with `KBC::ComponentSecure::` and is decryptable +across all configurations of that component. This is recommended for **component-specific secrets** +applicable across all customers (e.g., as a master authorization token). + +- Adding `projectId` to the `componentId` changes the prefix to `KBC::ProjectSecure::`, making the cipher decryptable within +the project's component configurations. This is recommended for **all secrets** used within a typical Keboola project. + +- Providing all three IDs (`componentId`, `projectId`, `configId`) generates a cipher starting with +`KBC::ConfigSecure::`, limiting decryption to a specific configuration. This is useful for preventing the copying of configurations. + +- Using only `projectId` yields a cipher that begins with `KBC::ProjectWideSecure::`, decryptable across the project's configurations. +This cipher type helps encrypt information shared across multiple components, e.g., SSH tunnel settings. + +- Adding `branchType` restricts the encryption to the default production branch or to development branches. This means an encrypted value with this setting cannot be moved between production and development branches or vice versa. It is not possible to encrypt a value for just one development branch. + + - Using `branchType` with `componentId` and `projectId` results in a cipher beginning with `KBC::BranchTypeSecure::`. This allows decryption either in the production or in the development configuration of the specified component in the project. + + - Using `branchType` with all three IDs (`componentId`, `projectId`, `configId`) creates a cipher that starts with `KBC::BranchTypeConfigSecure::`. It can only be decrypted within a specific production or development component configuration in a specific project. + + - Using `branchType` with `projectId` creates a cipher beginning with `KBC::ProjectWideBranchTypeSecure::`. This cipher allows decryption either in the production or in the development configurations in the project. + +The following rules apply to all ciphers: + +- Providing only a `configId` without a `projectId` is not allowed. Similarly, providing only `branchType` without `projectId` is also not allowed. +- Cipher decryption is only possible in the [region](/overview/api/#regions-and-endpoints) where the cipher was created. For example, ciphers with prefixes `KBC::ProjectSecureKV::` (Azure) or `KBC::ProjectSecureGKMS::` (GCP), instead of `KBC::ProjectSecure::` (AWS), use the same business logic but are specific to their region and technology and are not interchangeable. +- There is no decryption API; the cipher is decrypted internally before a component is run. +- Ciphering a value that is already encrypted does not change its encryption. +- There is no way to retrieve the component, project, configuration ID, or branch type from the cipher. +- The IDs referenced during cipher creation do not need to exist then. For example, you can create a cipher for a component not yet registered, which will start working as soon as the component is registered. Similarly, ciphers can be created for projects and configurations without access to them. + +By default, values encrypted in component configurations are encrypted using the `KBC::ProjectSecure::` cipher, meaning +the cipher is not transferable between regions, components, or projects. It is transferable between +different configurations of the same component within the project where it was created. If you create a configuration containing `KBC::ConfigSecure::` ciphers, +note that the configuration will not work when copied. diff --git a/src/content/docs/transformations/python-plain/index.md b/src/content/docs/transformations/python-plain/index.md index 61e32894a..156073b10 100644 --- a/src/content/docs/transformations/python-plain/index.md +++ b/src/content/docs/transformations/python-plain/index.md @@ -11,7 +11,7 @@ redirect_from: other operations are too difficult. Common data operations like joining, sorting, or grouping are still easier and faster to do in [SQL Transformations](/transformations/#backends). -***Warning:** Python transformations have **no facility for encrypting secrets**. Any credential you place in transformation code — API keys, passwords, tokens, connection strings — is stored as **plaintext** in the configuration. It is not encrypted at rest, it is readable by anyone with access to the project's configuration, and it is included when the configuration is processed by features such as the AI **Generate description**. **Do not put credentials in transformation code.** Instead, store them in the [Custom Python](/components/applications/custom-python/) application, where any parameter whose key starts with `#` is [encrypted](https://developers.keboola.com/overview/encryption/) and made available to your code as an environment variable at runtime.* +***Warning:** Python transformations have **no facility for encrypting secrets**. Any credential you place in transformation code — API keys, passwords, tokens, connection strings — is stored as **plaintext** in the configuration. It is not encrypted at rest, it is readable by anyone with access to the project's configuration, and it is included when the configuration is processed by features such as the AI **Generate description**. **Do not put credentials in transformation code.** Instead, store them in the [Custom Python](/components/applications/custom-python/) application, where any parameter whose key starts with `#` is [encrypted](/overview/encryption/) and made available to your code as an environment variable at runtime.* ## Environment The Python script is running in an isolated [environment](https://developers.keboola.com/extend/#component). diff --git a/src/content/docs/transformations/snowflake-plain/index.md b/src/content/docs/transformations/snowflake-plain/index.md index 90733d0df..6281fb5a0 100644 --- a/src/content/docs/transformations/snowflake-plain/index.md +++ b/src/content/docs/transformations/snowflake-plain/index.md @@ -219,7 +219,7 @@ CREATE TABLE "out" AS Do not use `ALTER SESSION` queries to modify the default timestamp format, as the loading and unloading sessions are separate from your transformation/sandbox session and the format may change unexpectedly. -**Important:** In the AWS US Keboola [region](https://developers.keboola.com/overview/api/#regions-and-endpoints) +**Important:** In the AWS US Keboola [region](/overview/api/#regions-and-endpoints) (connection.keboola.com), the following [Snowflake default](https://docs.snowflake.com/en/sql-reference/parameters) parameters are overridden: diff --git a/src/sidebar.mjs b/src/sidebar.mjs index c4caa8432..8599acadc 100644 --- a/src/sidebar.mjs +++ b/src/sidebar.mjs @@ -3,7 +3,15 @@ export const sidebar = [ { label: "Home", slug: "index" }, - { slug: "overview" }, + { + label: "Keboola Overview", + collapsed: true, + items: [ + { label: "Overview", slug: "overview" }, + { slug: "overview/api" }, + { slug: "overview/encryption" }, + ], + }, { label: "Getting Started Tutorial", collapsed: true,