diff --git a/_data/navigation.yml b/_data/navigation.yml index 3f8bee18..d466f737 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -4,51 +4,18 @@ items: - url: /overview/ title: Keboola Overview - items: - - url: /overview/api/ - title: Our APIs - - - url: /overview/encryption/ - title: Encryption - url: /extend/ - title: Extending Keboola + title: Extending Keboola (moved) - url: /integrate/ - title: Integration + title: Integration (moved) items: - - url: /integrate/jobs/ - title: Component Jobs - - - url: /integrate/artifacts/ - title: Artifacts - items: - - url: /integrate/artifacts/tutorial/ - title: Tutorial - - url: /integrate/database/ title: SSH Tunnel for Database Extractors - - url: /integrate/data-streams/ - title: Data Streams - items: - - url: /integrate/data-streams/tutorial/ - title: Tutorial - - - url: /integrate/data-streams/overview/ - title: Overview - - url: /automate/ - title: Automation/Common Tasks - items: - - url: /automate/run-job/ - title: Run Job - - - url: /automate/run-orchestration/ - title: Run Orchestration - - - url: /automate/set-schedule/ - title: Set Schedule + title: Automation (moved) - url: /cli/ title: CLI (moved) diff --git a/automate/index.md b/automate/index.md index 6536f3f2..43915a54 100644 --- a/automate/index.md +++ b/automate/index.md @@ -1,47 +1,5 @@ --- title: Automation/Common Tasks permalink: /automate/ +redirect_to: https://help.keboola.com/flows/ --- - -## Automation - -Use the Keboola Orchestrator component to specify what tasks should be executed in what order and -configure their automatic execution (specified intervals, specified times of the day, etc.). - -The set of [Keboola APIs](/overview/api/) provides full automation of the data warehouse cycle. -The end-to-end serverless solution automatically enables you to connect data sources, automatically store data -in the correct format, check for format inconsistencies, and choose different metadata providers based on the -operation you wish to perform on the data. The platform scales the needed resources automatically across various -types of data (structured, semi-structured, and non-structured) and processes. - -The whole environment tracks all the [operational metadata](https://api.keboola.com/?service=storage#tag--Events) -and can be accessed without needing a server via APIs. This is useful when automating development, testing and -production run of data jobs with automatic controls of [pipelines](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches/-id-). - -As Storage API is part of the wider Keboola platform, it is an essential element in providing coherent data -fabric across clouds, users, services, and on premise. - - -## CI/CD - -No matter whether you use Jenkins, CircleCI, AWS CodeBuilder, or Azure DevOps, you can utilise -the Keboola API within your existing CI/CD pipeline to deploy and manage new versions of your data -pipeline and data process automation tasks. - -## Documentation - -You can learn about how to set up our Orchestrator on [help.keboola.com/tutorial/automate/](https://help.keboola.com/tutorial/automate/). - -{% comment %} - - Load data from your system - - Trigger orchestrations - - Send data - - Copy buckets from different projects - - -Orchestrator - -spusteni jobu -API pro konfiguraci, nepouzivat normalni api -custom joby -{% endcomment %} diff --git a/automate/job-parameters.png b/automate/job-parameters.png deleted file mode 100644 index 2e813553..00000000 Binary files a/automate/job-parameters.png and /dev/null differ diff --git a/automate/job-row-parameters.png b/automate/job-row-parameters.png deleted file mode 100644 index d48d7ec3..00000000 Binary files a/automate/job-row-parameters.png and /dev/null differ diff --git a/automate/orchestration-parameters.png b/automate/orchestration-parameters.png deleted file mode 100644 index 83fafe0a..00000000 Binary files a/automate/orchestration-parameters.png and /dev/null differ diff --git a/automate/run-job.md b/automate/run-job.md index 66337d64..b7a07093 100644 --- a/automate/run-job.md +++ b/automate/run-job.md @@ -1,47 +1,5 @@ ---- -title: Run a Job -permalink: /automate/run-job/ ---- - -A [job](https://help.keboola.com/management/jobs/) represents a work being done in Keboola. -You can create (run) a job from the UI or via scheduled [Orchestrations](https://help.keboola.com/orchestrator/) or Flows. -A job can also be created manually via the API. The easiest way to get started is to create -a [configuration](https://help.keboola.com/components/) of the component you want to run and run it manually in the UI. -Once you're satisfied with the result, look at the successful job: - -{: .image-popup} -![Screenshot -- Job Parameters](/automate/job-parameters.png) - -In the job detail, you can see the parameters required to run the configuration, in this case: - -``` -mode: run -component: keboola.ex-db-snowflake -config: 493493 -``` - -Then create a [Storage API token](https://help.keboola.com/management/project/tokens/) which you will use to -run the API requests (if you don't have one already). We recommend to create -as restricted token as possible -- in this case limit it to to the component with ID `keboola.ex-db-snowflake`: - -{: .image-popup} -![Screenshot -- Token Settings](/automate/token-settings.png) - -Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to -create a job with the same parameters -(see [example](https://documenter.getpostman.com/view/3086797/77h845D#fd60aa15-485c-4922-8536-c2ba2f27e8ea)): - -```bash -curl --location --request POST 'https://queue.keboola.com/jobs' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "mode": "run", - "component": "keboola.ex-db-snowflake", - "config": "493493" -}' -``` - -Take care to use the right endpoint depending on which [Stack](https://help.keboola.com/overview/#stacks) are you using. -You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about -the concept of [Jobs](/integrate/jobs/). +--- +title: Run a Job +permalink: /automate/run-job/ +redirect_to: https://help.keboola.com/management/jobs/api/ +--- diff --git a/automate/run-orchestration.md b/automate/run-orchestration.md index f5b1f35c..29ce8e96 100644 --- a/automate/run-orchestration.md +++ b/automate/run-orchestration.md @@ -1,49 +1,5 @@ ---- -title: Run Orchestration -permalink: /automate/run-orchestration/ ---- - -Running an [Orchestration](https://help.keboola.com/orchestrator/) or Flow is in principle same as running -a [job](/automate/run-job/). -You can create (run) an orchestration job from the UI. A job can also be created manually via the API. -The easiest way to get started is to create a configuration of orchestration or flow and run it manually in the UI. -Once you're satisfied with the result, look at the successful job: - -{: .image-popup} -![Screenshot -- Orchestration Parameters](/automate/orchestration-parameters.png) - -In the job detail, you can see the parameters required to run the configuration, in this case: - -``` -mode: run -component: keboola.orchestrator -config: 1496488 -``` - -The difference between running an arbitrary [component job](/automate/run-job/) and an orchestrator job is only in -that the `component` value is always `keboola.orchestrator`. The same component is used for both Orchestrator -and Flow jobs. - -Create a [Storage API token](https://help.keboola.com/management/project/tokens/) which you will use to -run the API requests (if you don't have one already). We recommend to create a token with **Full Access** to -components. Though you can list the components used in the orchestration, this leads to a fragile setup when -modifying the orchestration may need the modification of the the token too. - -Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to -create a job with the same parameters -(see [example](https://documenter.getpostman.com/view/3086797/77h845D#3e71b131-afd4-44be-9831-6534e581f2e0)): - -```bash -curl --location --request POST 'https://queue.keboola.com/jobs' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "mode": "run", - "component": "keboola.orchestrator", - "config": "1496488" -}' -``` - -Take care to use the right endpoint depending on which [Stack](https://help.keboola.com/overview/#stacks) are you using. -You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about -the concept of [Jobs](/integrate/jobs/). +--- +title: Run Orchestration +permalink: /automate/run-orchestration/ +redirect_to: https://help.keboola.com/management/jobs/api/ +--- diff --git a/automate/set-schedule.md b/automate/set-schedule.md index a77621d6..ad64304f 100644 --- a/automate/set-schedule.md +++ b/automate/set-schedule.md @@ -1,302 +1,5 @@ ---- -title: Set Schedule -permalink: /automate/set-schedule/ ---- - -In the UI, you can set a time schedule for [orchestration](https://help.keboola.com/orchestrator/running/#automation). -Via the API you can set a time schedule for any [configuration](https://help.keboola.com/components/) or even multiple -schedules for a single configuration. - -Assuming, you have already setup of any [configuration](https://help.keboola.com/components/) -(including Orchestrator and Flow) and you can successfully run it through the UI or through the -[API](/automate/run-job/). Look at the successful job: - -{: .image-popup} -![Screenshot -- Job Parameters](/automate/job-parameters.png) - -In the job detail, you can see the parameters required to run the configuration, in this case: - -``` -mode: run -component: keboola.ex-db-snowflake -config: 493493 -``` - -To create a schedule, you have to create the schedule configuration first using the -[Create Configuration API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/components/-componentId-/configs). -With the contents similar to this: - -```json -{ - "schedule": { - "cronTab": "0 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "mode": "run" - } -} -``` - -The `cronTab` field defines the schedule in [cronTab expression](https://crontab.guru/#0_*_*_*_*) format. The -`target` defines which configuration should be run. In the above example we use the same configuration from the -job. To create the configuration itself, use the `keboola.scheduler` component -(see an [example](https://documenter.getpostman.com/view/3086797/77h845D#b721ecd8-159c-4895-9d0b-8b735880b714)): - -```bash -curl --location --request POST 'https://connection.keboola.com/v2/storage/components/keboola.scheduler/configs/' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---form 'name="Example Schedule"' \ ---form 'configuration="{ - \"schedule\": { - \"cronTab\": \"0 * * * *\", - \"timezone\": \"UTC\", - \"state\": \"enabled\" - }, - \"target\": { - \"componentId\": \"keboola.ex-db-snowflake\", - \"configurationId\": \"493493\", - \"mode\": \"run\" - } -}"' -``` - -Take care to use the right endpoint depending on which [Stack](https://help.keboola.com/overview/#stacks) are you using. -You'll see `Invalid access token` error message if you are using the wrong endpoint or token. -An example request response will contain: - -```json -{ - "id": "10850624", - "name": "Example Schedule", - "description": "", - "created": "2022-01-31T00:00:59+0100", - "creatorToken": { - "id": 25144, - "description": "Run Job" - }, - "version": 1, - "changeDescription": "Configuration created", - "isDisabled": false, - "isDeleted": false, - "configuration": { - "schedule": { - "cronTab": "0 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "mode": "run" - } - }, - "state": {}, - "currentVersion": { - "created": "2022-01-31T00:00:59+0100", - "creatorToken": { - "id": 25144, - "description": "Run Job" - }, - "changeDescription": "Configuration created" - } -} -``` - -The important field is `id` (with value `10850624` in the above example). In the second step, you need to activate -the schedule via the [Activate Schedule API call](https://api.keboola.com/?service=scheduler#scheduler/tag/schedules/POST/schedules) -with he following body: - -```json -{ - "configurationId": "10850624" -} -``` - -See an [example](https://documenter.getpostman.com/view/3086797/77h845D#130cbfc4-4dd5-4f6d-99c0-2444c48ee551)): - -```bash -curl --location --request POST 'https://scheduler.azure.keboola.com/schedules' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "configurationId": "10850624" -} -' -``` - -Note: You have to use a [Master Token](https://help.keboola.com/management/project/tokens/#master-tokens) to execute the above API call. - -## Configure Schedule for Row -If the configuration you're scheduling uses [Configuration rows](https://help.keboola.com/components/#configuration-rows), you can -also schedule individual rows. Assuming you have a job running a single row: - -{: .image-popup} -![Screenshot -- Row Parameters](/automate/job-row-parameters.png) - -You will see the the following in job parameters: - -``` -mode: run -component: keboola.ex-db-snowflake -config: 493493 -row: 48094 -``` - -Create a new scheduler configuration with the following configuration contents: - -```json -{ - "schedule": { - "cronTab": "10,20,30,40,50 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "configurationRowIds": ["48094"], - "mode": "run" - } -} -``` - -See [example](https://documenter.getpostman.com/view/3086797/77h845D#0fadefb7-9352-45b0-8c4e-6f3742feea2a): - -```bash -curl --location --request POST 'https://connection.keboola.com/v2/storage/components/keboola.scheduler/configs/' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---form 'name="Example Row Schedule"' \ ---form 'configuration="{ - \"schedule\": { - \"cronTab\": \"10,20,30,40,50 * * * *\", - \"timezone\": \"UTC\", - \"state\": \"enabled\" - }, - \"target\": { - \"componentId\": \"keboola.ex-db-snowflake\", - \"configurationId\": \"493493\", - \"configurationRowIds\": [\"48094\"], - \"mode\": \"run\" - } -}"' -``` - -You'll obtain the following example: - -```json -{ - "id": "10852379", - "name": "Example Row Schedule", - "description": "", - "created": "2022-01-31T00:42:15+0100", - "creatorToken": { - "id": 322, - "description": "ondrej.popelka@keboola.com" - }, - "version": 1, - "changeDescription": "Configuration created", - "isDisabled": false, - "isDeleted": false, - "configuration": { - "schedule": { - "cronTab": "10,20,30,40,50 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "configurationRowIds": [ - "48094" - ], - "mode": "run" - } - }, - "state": {}, - "currentVersion": { - "created": "2022-01-31T00:42:15+0100", - "creatorToken": { - "id": 322, - "description": "ondrej.popelka@keboola.com" - }, - "changeDescription": "Configuration created" - } -} -``` - -The created configuration has id `10852379`. You can now [Activate the Schedule](https://api.keboola.com/?service=scheduler#scheduler/tag/schedules/POST/schedules) -with he following body: - -```json -{ - "configurationId": "10852379" -} -``` - -See an [example](https://documenter.getpostman.com/view/3086797/77h845D#130cbfc4-4dd5-4f6d-99c0-2444c48ee551)): - -```bash -curl --location --request POST 'https://scheduler.keboola.com/schedules' \ ---header 'X-StorageApi-Token: YOUR_TOKEN' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "configurationId": "10852379" -} -' -``` - -This also demonstrates that you can set multiple schedules for a single configuration. In this case, the configuration -`10852379` of the `keboola.ex-db-snowflake` will be executed at the beginning of every hour. Then the row `48094` of -this configuration will also be executed every 10 minutes. To list the schedules use the -[Get Schedules API call](https://api.keboola.com/?service=scheduler#scheduler/tag/schedules/GET/schedules) -- -(see [example](https://documenter.getpostman.com/view/3086797/77h845D#6e0f31b1-9d50-4b51-9570-5334936d569c)). You'll get -a response similar to this: - -```json -[ - { - "id": "743", - "tokenId": "25147", - "configurationId": "10850624", - "configurationVersionId": "1", - "schedule": { - "cronTab": "0 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "configurationRowIds": [], - "mode": "run", - "tag": "" - }, - "executions": [] - }, - { - "id": "744", - "tokenId": "25148", - "configurationId": "10852379", - "configurationVersionId": "1", - "schedule": { - "cronTab": "10,20,30,40,50 * * * *", - "timezone": "UTC", - "state": "enabled" - }, - "target": { - "componentId": "keboola.ex-db-snowflake", - "configurationId": "493493", - "configurationRowIds": [ - "48094" - ], - "mode": "run", - "tag": "" - }, - "executions": [] - } -] -``` +--- +title: Set Schedule +permalink: /automate/set-schedule/ +redirect_to: https://help.keboola.com/flows/schedule-api/ +--- diff --git a/automate/token-settings.png b/automate/token-settings.png deleted file mode 100644 index 0266b27f..00000000 Binary files a/automate/token-settings.png and /dev/null differ diff --git a/extend/index.md b/extend/index.md index 8bc85aa0..736745db 100644 --- a/extend/index.md +++ b/extend/index.md @@ -1,67 +1,5 @@ --- title: Extending Keboola permalink: /extend/ +redirect_to: https://help.keboola.com/extend/ --- - -As an open system consisting of many built-in, interoperating components, -such as Storage or Extractors, [Keboola](/overview/) can be easily extended. -We encourage you to [**build your own components**](https://help.keboola.com/extend/component/tutorial/), whether for -your own use or to be offered to other Keboola users and customers. - -
-Note: Component development documentation — components, the common interface, Job Queue and publishing — -now lives at help.keboola.com/extend/. Generic Extractor and -Generic Writer now live under help.keboola.com/components/. -The old URLs here redirect there. -
- -* TOC -{:toc} - -There are two main options for extending Keboola: (a) creating your own **component** and (b) using **Generic -Extractor** to build an extractor for a RESTful API. - -## Advantages of Extending Keboola - -Depending on your role, extending Keboola offers various advantages: - -- If you already are a **Keboola customer**: - - Create your own component to convert your business problem into cloud. We will take care of the technical arrangements around running it. - - Create extractors or writers for communicating with your legacy systems, even if they are completely non-standard. - - Create components to experiment with new business solutions. No need to ask your IT to allocate resources to you. [Fail fast](https://en.wikipedia.org/wiki/Fail-fast#Business). - - Easily access data from many different sources. -- If you are an **external company**: - - Create connectors (Extractors/Writers) so that Keboola users can easily connect to your service and broaden your customer base. - - Create applications containing or using your algorithms and easily "deploy" them to Keboola customers. They won't be exposed to end-users, neither will be the end-user data exposed to you. - - Easily deliver the data back to your customers. -- If you are a **data scientist**: - - Create applications for delivering your work to your customer. We will take care of the technical arrangements. No need to rent servers and feed data to them. - - Make your application or algorithm available to all existing Keboola subscribers and implementation partners. - - Focus only on areas of your product where you are adding value. - - Let Keboola be in charge of the billing. - -## Component -A [component](https://help.keboola.com/extend/component/) can be used as: - -- **Extractor**, allowing customers to get data from new sources. It only processes input tables from external sources (usually API). -- **Application**, further enriching the data or adding value in new ways. It processes input tables stored as CSV files or database tables and generates result tables as CSV files or database tables. -- **Transformation**, allowing customers to modify their data. It is a constrained form of an application. -- **Writer**, pushing data into new systems and consumption methods. It does not generate any data in Keboola projects. -- **Processor**, adjusting the inputs or outputs of other components. It has to be run together with one of the above components. - -All components are run using [Job Queue](https://help.keboola.com/extend/job-queue/), a service that takes -care of their authentication, starting, stopping, isolation, reading data from and writing it to Keboola Storage. They must adhere to the -[common interface](https://help.keboola.com/extend/common-interface/). Creating components requires an elementary knowledge of [Docker](https://www.docker.com/why-docker). -They can be implemented in virtually any programming language and be fully customized and tailored to anyone's needs. -They also support OAuth authorization. To get started with building a component, see our [**tutorial**](https://help.keboola.com/extend/component/tutorial/). - -## Generic Extractor -[Generic Extractor](https://help.keboola.com/components/extractors/generic-extractor/) is a Keboola component acting like a -customizable [HTTP REST client](https://help.keboola.com/components/extractors/generic-extractor/tutorial/rest/). It can be configured to extract data -from virtually any API and offers a vast amount of configuration options. With Generic Extractor, you can build an -entirely new extractor for Keboola in less than an hour. - -Components based on Generic Extractor are built using [JSON configuration](https://help.keboola.com/components/extractors/generic-extractor/tutorial/) and a -[published template](https://help.keboola.com/components/extractors/generic-extractor/publish/). They have a predefined UI, require no knowledge of Docker or -other tools, and they use a Keboola owned [repository](https://github.com/keboola/kbc-ui-templates/). To get -started with Generic Extractor, see our [**tutorial**](https://help.keboola.com/components/extractors/generic-extractor/tutorial/). diff --git a/integrate/artifacts/artifacts-tutorial-1.png b/integrate/artifacts/artifacts-tutorial-1.png deleted file mode 100644 index b35de9b1..00000000 Binary files a/integrate/artifacts/artifacts-tutorial-1.png and /dev/null differ diff --git a/integrate/artifacts/artifacts-tutorial-2.png b/integrate/artifacts/artifacts-tutorial-2.png deleted file mode 100644 index 3b78098b..00000000 Binary files a/integrate/artifacts/artifacts-tutorial-2.png and /dev/null differ diff --git a/integrate/artifacts/artifacts-tutorial-3.png b/integrate/artifacts/artifacts-tutorial-3.png deleted file mode 100644 index d9ba4784..00000000 Binary files a/integrate/artifacts/artifacts-tutorial-3.png and /dev/null differ diff --git a/integrate/artifacts/artifacts-tutorial-4.png b/integrate/artifacts/artifacts-tutorial-4.png deleted file mode 100644 index ae120cc9..00000000 Binary files a/integrate/artifacts/artifacts-tutorial-4.png and /dev/null differ diff --git a/integrate/artifacts/index.md b/integrate/artifacts/index.md index dee1b5b4..ab851557 100644 --- a/integrate/artifacts/index.md +++ b/integrate/artifacts/index.md @@ -1,129 +1,5 @@ --- title: Artifacts permalink: /integrate/artifacts/ +redirect_to: https://help.keboola.com/extend/common-interface/artifacts/ --- - -* TOC -{:toc} - -*Note: This is a preview feature and as such may change considerably in the future. The project must have an `artifacts` feature enabled.* - -**Artifacts** are additional files that can be produced or consumed by a [component](https://help.keboola.com/extend/component/). - -See [Tutorial](/integrate/artifacts/tutorial) for step-by-step example. - -## Introduction -In some cases it's useful if a component not only extracts, transforms or uploads data, but also generate some other output, metadata or other runtime-discovered data. -These could be for example: -- AI models -- performance graphs of such models -- status updates from long-running tasks -- documentation -- data quality checks from in-progress tasks - -These additional information can be stored in artifacts and processed by another component or 3rd party tool. - -## Storage -Artifacts are stored in Keboola File Storage. - -## Types of artifacts -There are three types of artifacts for now `runs`, `custom` and `shared`. -The type specifies which components will have access to the artifact or which artifacts to download for the component to process. -Types are used in a configuration of a consumer component to specify which artifacts to download. - -- **runs** - artifacts from previous runs of the same configuration - -- **custom** - artifacts from previous runs of a different configuration. The configuration which produced the artifacts will be defined in the consumer configuration (configurationId, componentId, branchId) - -- **shared** - artifacts shared within an orchestration - -`runs` and `custom` types are the same from the producer point of view. To produce a `shared` artifact, it has to be written into a `shared` folder. Read more in [File structure](#file-structure) section. - -## File structure -Artifact is a unique set of files associated with a successful job, component and configuration. -A component can either produce or consume artifacts or both. - -### Produce -To produce an artifact, store one or more files in the following `output` directories. Subdirectories are also supported. -- `/data/artifacts/out/current` to create an artifact of type `runs` / `custom`. -- `/data/artifacts/out/shared` to create an artifact of type `shared`, which can be accessed by any component within the same orchestration. - -After the component job is finished all files and directories inside `current` and `shared` folders will be compressed into an archive and uploaded to File Storage with corresponding tags as a `artifact`. - -### Consume -To consume created artifacts you have to specify, in the configuration of a component, which artifacts (type) to download. - - `runs` to download artifacts produced by the same configuration and component. These will be stored in `/data/artifacts/in/runs/jobs/job-%job_id%` directory. - - `custom` to download artifacts produced by another configuration or component. These will be stored in `/data/artifacts/in/custom/jobs/job-%job_id%` directory. - - `shared` to download artifacts created within the same orchestration by any artifact producing component that has already finished. These will be stored in `/data/artifacts/in/shared/jobs/job-%job_id%` directory. - -## Configuration -Each type of artifact has a separate node in configuration. All the types can be used simultaneously. -Each type node has an attribute "enabled", which enables or disables download of the corresponding artifact type. - -### Runs - - **enabled** [true|false] - enable or disable download of this artifact type - - **filter** - - **date_since** - only artifacts from jobs younger than this will be downloaded - - **limit** - maximum number of the latest jobs from which to download artifacts - -### Custom -- **enabled** [true|false] - enable or disable download of this artifact type -- **filter** - - **branch_id**, **component_id**, **config_id** - specify the configuration to download artifacts from - - **date_since** - only artifacts from jobs younger than this will be downloaded - - **limit** - maximum number of the latest jobs from which to download artifacts - -### Shared -- **enabled** [true|false] - enable or disable download of this artifact type - -Full configuration example with all artifact types: - -```json -{ - "parameters": {}, - "artifacts": { - "runs": { - "enabled": true, - "filter": { - "date_since": "-7 days", - "limit": 5 - } - }, - "custom": { - "enabled": true, - "filter": { - "component_id": "keboola.python-transformation", - "config_id": "12345", - "branch_id": "default", - "date_since": "-7 days", - "limit": 5 - } - }, - "orchestration": { - "enabled": true - } - } -} -``` - -## Artifacts life-cycle in a job -Job runner checks if the project has enabled `artifacts` feature. -Job runner checks the configuration of the component. -If artifacts are enabled, it downloads artifacts to corresponding folders as configured (i.e. `runs`, `custom`, `shared`) and unzips them. - -Component process start and the component can: - -- access and process the downloaded artifacts in shared or custom directory - -- write artifacts to `current` or `shared` directory - -Component finishes and job runner does: - -- gzip the content of runs/current - -- tag the gzipped file with jobId, componentId, configId, runId, branchId and other tags if needed - -- upload the file to File Storage - -## File size limit -All the artifacts produced by a job shouldn’t be bigger than 1 GB. diff --git a/integrate/artifacts/tutorial.md b/integrate/artifacts/tutorial.md index 628df13a..7d2df123 100644 --- a/integrate/artifacts/tutorial.md +++ b/integrate/artifacts/tutorial.md @@ -1,219 +1,5 @@ --- title: Artifacts Tutorial permalink: /integrate/artifacts/tutorial/ +redirect_to: https://help.keboola.com/extend/common-interface/artifacts/tutorial/ --- - -* TOC -{:toc} - -This tutorial will show you how to work with artifacts. -In the following example we will use Python Transformation component to produce and consume artifacts. -But these principles would work inside any component. - -In the examples, we use the `curl` console tool to interact with our APIs. - -*Note: `artifacts` feature needs to be enabled in your project. Please contact [support@keboola.com](mailto:support@keboola.com) to enable the feature in your project* -*Note 2: `artifacts` configuration can be created or edited only via [Configuration API](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/components/-componentId-/configs) for now* - -## Examples - -For each example we will need [Storage API Token](https://help.keboola.com/management/project/tokens/) to make the API call. - -1. Obtain a Storage API token from the user interface of your project, see this [Guide](https://help.keboola.com/management/project/tokens). -2. Store the token and url to the environment variable. - - ```shell - export STORAGE_API_HOST="https://connection.keboola.com" - export TOKEN="..." - ``` - -### 1. Produce artifact - -This is very simple example. We will just create a Python Transformation, which will write a file to the artifacts "upload" folder. -This file will be then uploaded as "artifact" to File Storage. - -1. In your Keboola project, create a new Python transformation, and paste this code into it: - ``` - import os - with open("/data/artifacts/out/current/myartifact1", "w") as file: - file.write("this is my artifact file content") - ``` - - {: .image-popup} - ![Artifacts - transformation](/integrate/artifacts/artifacts-tutorial-1.png) - -2. Run the transformation - it should upload the file to File Storage as "artifact" - - {: .image-popup} - ![Artifacts - Job](/integrate/artifacts/artifacts-tutorial-2.png) - -3. The file is now visible in File Storage with appropriate tags - - {: .image-popup} - ![Artifacts - File Storage](/integrate/artifacts/artifacts-tutorial-2.png) - - -### 2. Produce & consume artifacts - -To consume (download) artifacts for component to work with, we need to enable and configure artifacts download in the configuration of a component. - -We will create another configuration of the Python transformation via API. - -The artifacts part of the configuration will look like this. -It will enable download of artifacts of type `runs` with limit 5, which means this will download artifacts created by the last 5 runs of the same component configuration - - ```json - { - "artifacts":{ - "runs":{ - "enabled":true, - "filter":{ - "limit":5 - } - } - } - } - ``` - -The script of the transformation will look like following. -Files read from `/data/artifacts/in/runs/*/*` will be displayed at output - these are the artifact files downloaded. -The script will also generate a new artifact and write it to `/data/artifacts/out/current/myartifact1` as in previous example. - - ```python - import os - import glob - - # Download - print(glob.glob("/data/artifacts/in/runs/*/*")) - - # Upload - with open("/data/artifacts/out/current/myartifact1", "w") as file: - file.write("value1") - ``` -1. Run this curl command to create the configuration: - - ```shell - curl -X POST "$STORAGE_API_HOST/v2/storage/branch/default/components/keboola.python-transformation-v2/configs" \ - -H "X-StorageApi-Token: $TOKEN" \ - -H 'Content-Type: application/x-www-form-urlencoded' \ - --data-urlencode 'configuration={"parameters":{"blocks":[{"name":"Block 1","codes":[{"name":"artifacts","script":["import os\nimport glob\n\n# Download\nprint(glob.glob(\"/data/artifacts/in/runs/*/*\")) \n\n# Upload\nwith open(\"/data/artifacts/out/current/myartifact1\", \"w\") as file:\n file.write(\"value1\")"]}]}]},"artifacts":{"runs":{"enabled":true,"filter":{"limit":5}}}}' \ - --data-urlencode 'name=Artifacts upload & download' \ - --data-urlencode 'description=Test Artifacts upload & download' - ``` - - -### 3. Consume artifacts from different component -Similar to previous example we will create a configuration of Python Transformation component. -But this time we will download artifacts produced by the configuration from `Example 2`. - -1. Export the id of the previously created configuration into an environment variable: - ```shell - export CONFIG_ID="..." - ``` - -2. Run curl command - ```shell - curl -X POST "$STORAGE_API_HOST/v2/storage/branch/default/components/keboola.python-transformation-v2/configs" \ - -H "X-StorageApi-Token: $TOKEN" \ - -H 'Content-Type: application/x-www-form-urlencoded' \ - --data-urlencode 'configuration={"parameters":{"blocks":[{"name":"Block 1","codes":[{"name":"artifacts","script":["import os\nimport glob\n\n# Download\nprint(glob.glob(\"/data/artifacts/in/custom/*/*\"))"]}]}]},"artifacts":{"custom":{"enabled":true,"component_id":"keboola.python-transformation","config_id":"$CONFIG_ID","branch_id":"default","filter":{"limit":5}}}}' \ - --data-urlencode 'name=Artifacts upload & download' \ - --data-urlencode 'description=Test Artifacts upload & download' - ``` - -The whole configuration now looks like this: - - ```json - { - "parameters": { - "blocks": [ - { - "name": "Block 1", - "codes": [ - { - "name": "artifacts", - "script": [ - "import os\nimport glob\n\n# Download\nprint(glob.glob(\"/data/artifacts/in/custom/*/*\"))" - ] - } - ] - } - ] - }, - "artifacts": { - "custom": { - "enabled": true, - "component_id": "keboola.python-transformation", - "config_id": "$CONFIG_ID", - "branch_id": "default", - "filter": { - "limit": 5 - } - } - } - } - ``` - -### 4. Shared artifacts -This example will show how to share artifacts within an orchestration -We will create two configurations of Python Transformation component. -One will produce a shared artifact and the other will consume it. -Both configurations needs to be in the same orchestration. -The configuration producing artifact needs to be in a phase that precedes the consuming one. - -1. Create "Producer" configuration - The Python code will write a file into a shared folder: - - ```python - import os - with open(path+\"/myartifact-shared\", \"w\") as file: - file.write(\"value1\")" - ``` - - Run curl command to create the configuration: - - ```shell - curl -X POST "$STORAGE_API_HOST/v2/storage/branch/default/components/keboola.python-transformation-v2/configs" \ - -H "X-StorageApi-Token: $TOKEN" \ - -H 'Content-Type: application/x-www-form-urlencoded' \ - --data-urlencode 'configuration={"parameters":{"blocks":[{"name":"Block 1","codes":[{"name":"Upload shared","script":["import os\npath = \"/data/artifacts/out/shared\"\nwith open(path+\"/myartifact3\", \"w\") as file:\n file.write(\"value1\")"]}]}]},"artifacts":{"runs":{"enabled":true,"filter":{"limit":5}}}}' \ - --data-urlencode 'name=Artifacts shared Producer' \ - --data-urlencode 'description=Artifacts upload shared' - ``` - -2. Create "Consumer" configuration - - The artifacts configuration: - ```json - { - "artifacts": { - "shared": { - "enabled": true - } - } - } - ``` - - The Python script: - - ```python - import os - import glob - print(glob.glob("/data/artifacts/in/shared/*/*")) - ``` - - Run curl command to create the configurtion: - - ```shell - curl -X POST "$STORAGE_API_HOST/v2/storage/branch/default/components/keboola.python-transformation-v2/configs" \ - -H "X-StorageApi-Token: $TOKEN" \ - -H 'Content-Type: application/x-www-form-urlencoded' \ - --data-urlencode 'configuration={"parameters":{"blocks":[{"name":"Block 1","codes":[{"name":"Download shared","script":["import os\nimport glob\n\nprint(glob.glob(\"/data/artifacts/in/shared/*/*\")) "]}]}]},"artifacts":{"shared":{"enabled":true}}}' \ - --data-urlencode 'name=Artifacts shared Consumer' \ - --data-urlencode 'description=Artifacts download shared' - ``` - -3. Now put each of the configurations into an Orchestration. "Artifacts shared Producer" into phase 1 and "Artifacts shared Consumer" into phase 2. - - {: .image-popup} - ![Artifacts orchestration](/integrate/artifacts/artifacts-tutorial-4.png) diff --git a/integrate/data-streams/index.md b/integrate/data-streams/index.md index 022ce6df..6f1fc78f 100644 --- a/integrate/data-streams/index.md +++ b/integrate/data-streams/index.md @@ -1,17 +1,7 @@ --- title: Data Streams permalink: /integrate/data-streams/ -redirect_from: /integrate/push-data/ +redirect_from: + - /integrate/push-data/ +redirect_to: https://help.keboola.com/storage/data-streams/ --- - -
- -The Keboola Stream API allows you to ingest small, frequent events into your project's storage. - -![data streams diagram](./push_data.drawio.png) - -See the [tutorial](/integrate/data-streams/tutorial/) to learn how to create and manage event sources. - -## Next Steps -- [Data Streams Tutorial](/integrate/data-streams/tutorial/) -- [Data Streams Overview](/integrate/data-streams/overview/) diff --git a/integrate/data-streams/overview/index.md b/integrate/data-streams/overview/index.md index 7aee0c4c..860b6b9b 100644 --- a/integrate/data-streams/overview/index.md +++ b/integrate/data-streams/overview/index.md @@ -1,165 +1,7 @@ --- title: Data Streams Overview permalink: /integrate/data-streams/overview/ -redirect_from: /integrate/push-data/overview/ +redirect_from: + - /integrate/push-data/overview/ +redirect_to: https://help.keboola.com/storage/data-streams/reference/ --- - -* TOC -{:toc} - -![Data Streams diagram](../push_data.drawio.png) - -A source represents an endpoint for receiving events. - -Sources are managed using the Stream API. The full API reference is available at https://stream.keboola.com/v1/documentation/, and the OpenAPI specification is available at https://stream.keboola.com/v1/documentation/openapi3.json. - -Events are received via HTTP. Each source can be associated with up to 20 `sinks`, which represent `mappings` from event data to `columns` in a destination `table`. Data may be mapped using pre-defined mappings or a custom `template`. - -## Columns - -| Field | Type | Description | -|---|---|---| -| `name` | string | Name of the column. Names must be unique. | -| `type` | string | The type of the column. Available types and their descriptions are listed below. | - -***Note:** Data streams do not support primary keys. Every incoming event is appended to the -destination table and no deduplication happens on import. If you need unique rows (e.g., one row -per event ID), handle deduplication downstream: create a deduplication transformation that reads -the stream table and writes the result to a new table whose output mapping primary key performs -the deduplication on load, and schedule it to run regularly with a conditional flow.* - -The available column types are: - -| Type | Description | -|---|---| -| `id`| Event ID | -| `datetime` | Time of the event | -| `ip` | IP of the event sender | -| `body` | The unaltered event body | -| `headers` | The unaltered request headers | -| `path` | A field from the JSON object | -| `template` | A custom mapping using a template language | - -### Path - -The `path` column type can be used to fetch a single field from a `JSON` object. Optionally, you can use `rawString` option to remove the quotes around a JSON string or the `defaultValue` option to define a value when the field doesn't exist. - -```json -{ - "type": "json", - "name": "id", - "path": "issue.id", - "defaultValue": "undefined", - "rawString": true -} -``` - -### Template (Jsonnet) - -***Note:** It is recommended to use the faster `path` type instead of the `jsonnet` function `Body(string)` when possible.* - -The `template` column type currently supports the `jsonnet` templating language. The following `jsonnet` globals are available: - -|Name|Description|Usage example|Example value| -|:-|:-|:-|:-| -| `Ip()` | IP address of the client | `Ip()` | `127.0.0.1` | -| `Body()` | Get the entire request body as an object. | `Body()` | `{ "a": "b" }` | -| `Body(string)` | Get a field from the request body by path. Fails if the field does not exist; in that case, the record will not be saved. | `Body("deeply.nested.path")` | `1000` | -| `Body(string, any)` | Get a field from the request body by path, or a default value. | `Body("deeply.nested.path", 2000)` | `1000` | -| `BodyStr()` | Get the entire request body as a string. | `BodyStr()` | `"{\"a\":\"b\"}"` | -| `Header()` | Get all request headers. | `Header()` | `{ "Content-Type": "application/json" }` | -| `Header(string)` | Get the value of a single request header. Fails if the header does not exist; in that case, the record will not be saved. | `Header("Content-Type")` | `"application/json"` | -| `Header(string, string)` | Get the value of a single request header or a default value. | `Header("Content-Type")` | `"application/json"` | -| `HeaderStr()` | Get the request headers as a string, each line containing one "header: value" pair. The lines are sorted alphabetically. | `HeaderStr()` | `Content-Type: application/json` | -| `Now()` | Get the current UTC datetime as a string formatted using the default format. | `Now()` | `"2023-01-14T08:04:05.123Z"` | -| `Now(string)` | Get the current UTC datetime as a string with the custom [`strftime`](https://man7.org/linux/man-pages/man3/strftime.3.html)-compatible format. | `Now("%Y-%m-%d")` | `2023-01-14` | - -### Conditions - -Incoming events are mapped to the schema defined in each sink, and each new row is appended to a CSV file on the local hard disk (local storage). - -When the local storage accumulates enough records or a short time passes, the records from local storage are appended to a CSV file stored in your Keboola project (staging storage). - -Once certain conditions are met, the data from the file is imported into the destination table (target storage). These `conditions` are defined by the sink: - -| Condition | Minimum | Maximum | Default | -|:-|:-:|:-:|:-:| -| `time` | 30 seconds | 24 hours | 1 minute | -| `size` | 100 B | 500 MB | 50 MB | -| `count` | 1 | 10 million | 50 thousand | - -Changing these conditions will trigger an immediate import of waiting files, after which the stream will follow the updated conditions. - -## Create Sources and Sinks - -Sources can be created using the [`POST /v1/branches/{branchId}/sources`](https://stream.keboola.com/v1/documentation/#/configuration/CreateSource) endpoint. - -If a source or sink `id` is omitted, it will be generated from the corresponding `name` field. - -A source may be created without any sinks. The sinks can then be created separately using the [`POST /v1/branches/{branchId}/sources/{sourceId}/sinks`](https://stream.keboola.com/v1/documentation/#/configuration/CreateSink) endpoint. - -***Warning**: Events sent to a source without any sinks will be permanently lost. This is because data is buffered per sink, not per source.* - -The requests are asynchronous and create a task that must be completed before the source or sink is ready to use. The task status can be checked using the [`GET /v1/branches/{branchId}/sources/{sourceId}/tasks/{taskId}`](https://stream.keboola.com/v1/documentation/#/configuration/GetTask) endpoint. - -Sink tables are created if they do not exist. If they already exist, the schema defined by `sink.columns` must match the existing schema. If the table schema is manually altered and it no longer matches, the import from staging storage to the table will fail. The data is kept in the staging storage for up to 7 days during which you can recover any failures. - -## Delete Sources and Sinks - -Sources may be deleted using the [`DELETE /v1/branches/{branchId}/sources/{sourceId}`](https://stream.keboola.com/v1/documentation/#/configuration/DeleteSource) endpoint. Sinks may be deleted using the [`DELETE /v1/branches/{branchId}/sources/{sourceId}/sinks/{sinkId}`](https://stream.keboola.com/v1/documentation/#/configuration/DeleteSink) endpoint. - -## Update Sources and Sinks - -A source may be updated using the [`PATCH /v1/branches/{branchId}/sources/{sourceId}`](https://stream.keboola.com/v1/documentation/#/configuration/UpdateSource) endpoint. Sinks maybe updated using the [`PATCH /v1/branches/{branchId}/sources/{sourceId}/sinks/{sinkId}`](https://stream.keboola.com/v1/documentation/#/configuration/UpdateSink) endpoint. - -The `UpdateSource` endpoint may only update the source's name. Sinks may only be updated separately. - -If a sink's `mapping.tableId` is updated, it is handled the same way as in the create operation. If the table exists, `mapping.columns` must match the existing table's schema. If the table does not exist, it is created. - -## Source and Sink Settings - -The import conditions mentioned above can be accessed using the [`GET /v1/branches/{branchId}/sources/{sourceId}/settings`](https://stream.keboola.com/v1/documentation/#/configuration/GetSourceSettings) endpoint and changed using the [`PATCH /v1/branches/{branchId}/sources/{sourceId}/settings`](https://stream.keboola.com/v1/documentation/#/configuration/PatchSourceSettings) endpoint. - -Same settings also exist for a sink. Use the [`GET /v1/branches/{branchId}/sources/{sourceId}/sinks/{sinkId}/settings`](https://stream.keboola.com/v1/documentation/#/configuration/GetSinkSettings) endpoint and the [`PATCH /v1/branches/{branchId}/sources/{sourceId}/sinks/{sinkId}/settings`](https://stream.keboola.com/v1/documentation/#/configuration/PatchSinkSettings) endpoint in that case. - -## Delivery Guarantees - -Depending on your use case, you may require different delivery guarantees for your stream. Follow the guidelines below to ensure the desired outcome. - -### At Most Once - -To ensure that no record is delivered twice, make sure the client doesn't retry when sending the records. In this case, it's beneficial to use the setting endpoints to set `"storage.level.local.encoding.sync.wait"` to `false` to increase throughput. - -### At Least Once - -To ensure that every record is delivered at least once, the client needs to implement retries when sending the records. Also, use the setting endpoints to confirm that `"storage.level.local.encoding.sync.wait"` is set to `true` (default behavior). Note that this setting guarantees that the record is written to the local disk. - -## Tokens - -A token is generated for each source sink. These tokens have the minimum possible scope with `write` permission for the bucket in which the destination table is -stored. You can view these tokens at `https://connection.keboola.com/admin/projects//tokens-settings`. Their description follows the format -`[_internal] Stream Sink /`. - -These tokens should not be deleted or refreshed manually. To refresh a token, you can disable and re-enable the sink. - -## Kafka Integration -To connect Keboola with [Apache Kafka®](https://kafka.apache.org/) and ingest data from Kafka topics via data streams, use the Kafka Connect HTTP Sink Connector -to establish a communication channel between Kafka and Keboola. - -The Kafka Connect HTTP Sink Connector acts as a bridge, seamlessly integrating Kafka with Keboola's Data Stream HTTP API. Here's a breakdown of the process: - -- Data Consumption: The connector continuously reads data records from one or more Kafka topics. -- Batching: Events can be efficiently grouped based on a predefined maximum size (batch.max.size). -- API Interaction: Data is sent as a POST request in JSON format to Keboola's Data Stream API URL. - -**Key Points to Remember:** - -- This integration relies on the Kafka Connect HTTP Sink Connector, which requires configuration on the Kafka side. -- Data records from Kafka topics are transformed into strings before being sent to Keboola. -- The target Keboola API URL corresponds to the data stream created in Keboola. -- Only POST HTTP methods are supported for data ingestion. - -## Next Steps - -- [Data Streams Tutorial](/integrate/data-streams/tutorial/) -- [Stream API Reference](https://stream.keboola.com/v1/documentation/) diff --git a/integrate/data-streams/tutorial/index.md b/integrate/data-streams/tutorial/index.md index 9aebad58..59b8f52b 100644 --- a/integrate/data-streams/tutorial/index.md +++ b/integrate/data-streams/tutorial/index.md @@ -1,195 +1,7 @@ --- title: Data Streams Tutorial permalink: /integrate/data-streams/tutorial/ -redirect_from: /integrate/push-data/tutorial/ +redirect_from: + - /integrate/push-data/tutorial/ +redirect_to: https://help.keboola.com/storage/data-streams/tutorial/ --- - -* TOC -{:toc} - - -In this tutorial, we will set up a source for the [`issues`](https://docs.github.com/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#issues) event from GitHub Webhooks. This will allow you to monitor and analyze activity related to issues in any of your GitHub repositories. - -You will need your project's master token, and a GitHub repository where you have the `Admin` role. - -## Creating a Source - -To start ingesting events, you must first create a source. Send the following payload to the `https://stream.keboola.com/v1/branches/{branchId}/sources` endpoint: -``` -{ - "name": "Github Issues", - "exports": [ - { - "name": "Events", - "conditions": { "count": 1 }, - "mapping": { - "tableId": "in.c-github.issues", - "columns": [ - { - "type": "id", - "name": "id" - }, - { "type": "datetime", "name": "datetime" }, - { "type": "ip", "name": "ip" }, - { "type": "body", "name": "body" }, - { "type": "headers", "name": "headers" }, - { - "type": "path", - "name": "id", - "path": "issue.id", - "defaultValue": "undefined", - "rawString": true - }, - { - "type": "template", - "name": "template", - "template": { - "language": "jsonnet", - "content": "'#' + Body('issue.id') + ': ' + Body('issue.body', 'n/a')" - } - } - ] - } - } - ] -} -``` - -You can do this using `curl`, or anything else that allows you to send an HTTP request: -``` -$ curl --header 'Content-Type: application/json' \ - --header 'X-StorageApi-Token: ' \ - --data '{ ...the payload above... }' \ - https://stream.keboola.com/v1/branches/{branchId}/sources -``` - -The response will contain the task that has been created: -``` -{ - "id": "2023-02-16T16:04:39.570Z_Pg7U4", - "sourceId": "github-issues", - "url": "https://stream.keboola.com/v1/branches/{branchId}/sources/github-issues/tasks/source.create/2023-02-16T16:04:39.570Z_Pg7U4", - "type": "source.create", - "createdAt": "2023-02-17T11:20:57.406Z", - "isFinished": false, - "result": "" -} -``` - -You can query the task's status by querying the `url` field and wait until the `isFinished` field is set to `true`: -``` -{ - "id": "2023-02-16T16:04:39.570Z_Pg7U4", - "sourceId": "github-issues", - "url": "https://stream.keboola.com/v1/branches/{branchId}/sources/github-issues/tasks/source.create/2023-02-16T16:04:39.570Z_Pg7U4", - "type": "source.create", - "createdAt": "2023-02-17T11:20:57.406Z", - "finishedAt": "2023-02-17T11:20:57.753Z", - "isFinished": true, - "duration": 343, - "result": "source created" -} -``` - -Upon success, query the source URL `https://stream.keboola.com/v1/branches/{branchId}/sources/github-issues`, and the response will contain the source you've just created: -``` -{ - "id": "github-issues", - "url": "https://stream.keboola.com/stream//github-issues/" - "name": "Github Issues", - "exports": [ - { - "id": "events", - "name": "Events", - "conditions": { - "count": 1, - "size": "5MB", - "time": "5m" - }, - "mapping": { - "tableId": "in.c-github.issues", - "columns": [ - { - "type": "id", - "name": "id" - }, - { "type": "datetime", "name": "datetime" }, - { "type": "ip", "name": "ip" }, - { "type": "body", "name": "body" }, - { "type": "headers", "name": "headers" }, - { - "type": "path", - "name": "id", - "path": "issue.id", - "defaultValue": "undefined", - "rawString": true - }, - { - "type": "template", - "name": "template", - "template": { - "language": "jsonnet", - "content": "'#' + Body('issue.id') + ': ' + Body('issue.body', 'n/a')", - } - } - ] - } - } - ] -} -``` - -The most important part of the response is the `url` field. This is the endpoint to which you will point your GitHub webhook. Once you've created the source and obtained its `url` field, you are ready to configure the GitHub webhook. - -Normally, the URL only returns a short response to reduce traffic. You can add `?verbose=true` to the URL to receive more information about what happened with the request. Note that this makes the response slower, so we recommend using this parameter for testing purposes only. - -## Configuring the Github Webhook - -Go to the `Settings` tab of your repository. - -![Github repository tabs](./gh-tabs.png) - -Open the `Webhooks` page. - -![Github settings pages](./gh-settings-webhook.png) - -Click `Add webhook`. - -![Github add webhook](./gh-settings-webhook-add.png) - -Enter the source `url` into the `Payload URL` field, and set the `Content Type` to `application/json`. - -For `Which events would you like to trigger this webhook?`, click `Let me select individual events`, then find `Issues` and tick it: - -![Github webhook let me select individual events selected](./gh-settings-webhook-individual-events.png) -![Github webhook issues checkbox selected](./gh-settings-webhook-issues.png) - -Click `Add webhook` at the bottom of the page. - -Any events related to issues in your repository will now be buffered by the source, and uploaded to your table every minute. - -To see your integration at work, head over to your repository and [open a few issues](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-an-issue). - -## Results - -The following token was generated. - -![Keboola token settings screenshot showing the generated token](./token.png) - -This token only has the minimal set of permissions, which, in this case, is access to a single bucket and the ability to manipulate files. Currently, files are used as staging storage to prevent data loss. You can see these files in your project's Storage. - -![Keboola storage file](./github_webhook_export_file.png) - -Since the table `in.c-github-issues` did not exist, it was created. - -![Keboola storage table](./github_webhook_export_table.png) - -Finally, you can take a look at the destination table's data sample to find your data, ready for further processing. - -![Keboola storage table sample data](./github_webhook_export_table_data.png) - - -## Next Steps - -- [Data Streams Overview](/integrate/data-streams/overview/) -- [Stream API Reference](https://stream.keboola.com/v1/documentation/) diff --git a/integrate/index.md b/integrate/index.md index 689c8c9e..c4adbdcd 100644 --- a/integrate/index.md +++ b/integrate/index.md @@ -1,34 +1,5 @@ --- title: Integration permalink: /integrate/ +redirect_to: https://help.keboola.com/overview/ --- - -You can look at Keboola as a system of independent and loosely coupled microservices (components). - -Each microservice has its own code base, and a publicly accessible API and configuration. -We do not cheat or have any advantage over other developers; our UI and other components use only these public APIs. - -As a result, it is very easy to, for example, write custom scripts to bootstrap a project, or do something that our UI does not offer. -Let's have a look into this! - -One of the very important components is [Storage](https://help.keboola.com/storage/api/), which not only stores all data in a -project, but also provides additional functions such as managing other components and their configurations. -When you are integrating your systems with Keboola, **chances are that you want to start with [Storage](https://help.keboola.com/storage/api/)**. - - -{% comment %} - - - gdwriter - - - Storage - - API - - Curl - - Commandline - - Transformation - - API - - IO Mapping - - Sandbox - - Extractors - - ... - - Writers -{% endcomment %} \ No newline at end of file diff --git a/integrate/jobs/index.md b/integrate/jobs/index.md index beee92b0..89503052 100644 --- a/integrate/jobs/index.md +++ b/integrate/jobs/index.md @@ -1,498 +1,7 @@ ---- -title: Component Jobs -permalink: /integrate/jobs/ -redirect_from: /overview/jobs/ ---- - -* TOC -{:toc} - -Most operations, such as extracting data or running an application are executed in Keboola as -background, asynchronous [jobs](https://help.keboola.com/management/jobs/). When an operation is triggered, for example, you run an extractor, a -*job* is created. The job starts executing or waits in the queue until it can start executing. -The job execution and queuing are fully automatic. The job execution is asynchronous, so you need to - -- *create* (run) a job, and -- *wait* for it to finish. - -The core API for working with jobs is the [Queue API](https://api.keboola.com/?service=job-queue#job-queue). It provides operations for -running/creating, terminating and listing jobs. -[Components](/overview/) differ in their upper limits on how long a job can be executing and how much memory it is allowed to consume. -These limits are set by the component developer and act primarily as a safeguard. - -## Job Properties -When you create a job it automatically transitions through states until it reaches some of the final states. -When you create or retrieve a job, you'll obtain a JSON with Job object, whose properties are described below in more detail. - -
- Click to expand the response. - -{% highlight json %} -{ - "id": "10440535", - "runId": "10440530.10440533.10440534.10440535", - "parentRunId": "10440530.10440533.10440534", - "project": { - "id": "66", - "name": "Sandbox" - }, - "token": { - "id": "7455", - "description": "[_internal] Scheduler" - }, - "status": "success", - "desiredStatus": "processing", - "mode": "run", - "component": "keboola.ex-db-snowflake", - "config": "493493", - "configData": [], - "configRowIds": [ - "41510" - ], - "tag": "5.5.0", - "createdTime": "2022-01-24T22:41:10+00:00", - "startTime": "2022-01-24T22:41:13+00:00", - "endTime": "2022-01-24T22:41:48+00:00", - "durationSeconds": 35, - "result": { - "input": { - "tables": [] - }, - "images": [ - [ - { - "id": "developer-portal-v2/keboola.ex-db-snowflake:5.5.0", - "digests": [ - "developer-portal-v2/keboola.ex-db-snowflake@sha256:0f9428c52afea457ec3865cab7cfe457f4f875f3cf45d36f1876c709211da9cf" - ] - } - ] - ], - "output": { - "tables": [ - { - "id": "in.c-keboola-ex-db-snowflake-493493.opportunity", - "name": "opportunity", - "columns": [ - { - "name": "Id" - }, - { - "name": "Name" - }, - { - "name": "AccountId" - }, - { - "name": "OwnerId" - }, - { - "name": "Amount" - }, - { - "name": "StageName" - }, - { - "name": "CreatedDate" - }, - { - "name": "CloseDate" - }, - { - "name": "Probability" - }, - { - "name": "Start_Date" - }, - { - "name": "End_Date" - }, - { - "name": "Record_Type_Name" - }, - { - "name": "AdvertiserName" - }, - { - "name": "Advertiser_Vertical" - }, - { - "name": "Type" - } - ], - "displayName": "opportunity" - } - ] - }, - "message": "Component processing finished.", - "configVersion": "16" - }, - "usageData": [], - "isFinished": true, - "url": "https://queue.north-europe.azure.keboola.com/jobs/10440535", - "branchId": null, - "variableValuesId": null, - "variableValuesData": { - "values": [] - }, - "backend": [], - "metrics": { - "backend": { - "size": null - }, - "storage": { - "inputTablesBytesSum": 0 - } - }, - "behavior": { - "onError": null - }, - "parallelism": "2", - "type": "standard" -} -{% endhighlight %} -
- -### Job Status -A job can have different values for `status`: -- `created` (the job is created, but has not started executing yet) -- `waiting` (the job is waiting for other jobs to finish) -- `processing` (job stuff is being done) -- `success` (the job is finished) -- `error` (the job is finished) -- `warning` (the job is finished, but one of its child jobs failed) -- `terminating` (the user has requested to abort the job) -- `cancelled` (the job was created, but it was aborted before its execution actually began) -- `terminated` (the job was created and it was aborted in the middle of its execution) - -{: .image-popup} -![Job State transitions](/integrate/jobs/states.png) - -When you create a job it is in the `created` state. In a success scenario it will transition to a `processing` state and when the actual work is done, to the - `success` state. If you change your mind and terminate a job, it will enter `terminating` state and then ends with either `terminated` (execution terminated) - or `cancelled` (execution did not actually start). The difference is that you can be sure that a `cancelled` job did absolutely no operations, - whereas a terminated job, could've done even all of the work it was supposed to do. - - If a job cannot be executed, it will enter the `waiting` state. The waiting state means that the job cannot be executed due to reasons on the Keboola - project side. This means that the reasons for waiting jobs lie solely in what jobs are already running in the given project. There are three core reasons for waiting jobs: - - If you run two jobs of the same configuration, the second one will wait until the first one is finished. This behavior is - called "configuration lock" and protects your project from [race conditions](https://en.wikipedia.org/wiki/Race_condition). - - Orchestration [phases](https://help.keboola.com/orchestrator/tasks/#organize-tasks). When you run an orchestration, the jobs for all phases are created. Phases that depend on other phases enter the `waiting` state. - - Setting parallel limits. If you run a configuration with 10 tables and set parallelism to 2, then 10 jobs will be created, 2 will enter `processing` - state and 8 of them will immediately enter the `waiting` state. - - If a job cannot be run due to platform reasons (e.g. insufficient resources, platform outage), it will remain in the `created` state. In rare situations - (e.g. hardware failure), the job may return back to created state. Moving the job out of the `created` state is out of the control of the end-user. - - Of all the states a job can be in, only the state `processing` is considered to be job runtime (see `durationSeconds` field) and therefore billable. - That means `waiting` or `created` jobs do not have any costs associated with them, they represent a plan of what is going to happen. - - The states `terminated`, `cancelled`, `success` and `error` are final and end the job transitions. When a job is in final state, the `isFinished` flag is true. - The job object is both immutable and eventually consistent. Once you create a job, you cannot change any of it's properties. Any changing properties are - self-modifying and they will stop modifying once the job reaches one of the final states. - -Apart from the `status` field, the job also has `desiredStatus` field. This is either `processing` or `terminating`. The desired status is -processing until a job termination is requested. This changes the desired status to `terminating`. Other changes are not permitted. - -### Job ID -When a job is created, an `id` and `runId` and optionally `parentRunId` are assigned to it. The `runId` and `parentRunId` represent -parent-child relationship between jobs. Parent-child hierarchical relationship can be defined via the `X-KBC-RunId` header, when used the -newly created job will become child of the job with the provided `RunId`. - -The `runId` field contains job `id`s with representing the job hierarchy. If there is no hierarchy, then `runId` is equal to `id`. If there is -hierarchy then `runId` is `parentId` concatenated with `id`. The hierarchy delimiter is dot `.`. Examples: - -- `id=123`, `runId=123`, `parentRunId=null` -- Job has no parent -- `id=345`, `runId=123.345`, `parentRunId=123` -- Job is a child of job `123`. -- `id=678`, `runId=123.345.678`, `parentRunId=123.345` -- Job is a child of job `345` which in turn is a child of job `123`. - -Jobs may be nested without limits. The parent-child relationship itself is a weak relationship. By itself it does not mean anything -special outside of UI grouping and the function that terminating a parent job issues a termination request to all its children. -Running a job as a child of another job does not by itself cause the parent to wait for child -completion or any other added functionality. -Such functionality is implemented in specific components (e.g. Orchestrator) or for specific [job types](todo). - -### Job Configuration -To create a job, you must provide the [configuration](https://help.keboola.com/components/) to run. A configuration is always tied to a specific -[component](https://help.keboola.com/extend/component/). - -A configuration can be provided in multiple ways. The easiest is to provide a reference to -a stored configuration ID using the `config` field as shown above. Configurations can be stored and listed using the -[Component Configurations API endpoint](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/components/-componentId-/configs). -When using a configuration which contains [Configuration Rows](https://help.keboola.com/components/#configuration-rows), the job can optionally execute -only certain rows. Use the `configRowIds` field to list row IDs to execute. Note that if you do not list any rows, then all rows will be executed except -for disabled rows. When you enumerate rows to execute, then the enumerated rows will be executed even if they are disabled. To run a job of -a configuration in a branch, provide the [branch ID](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches) -in the `branchId` field. If you do not provide `branchId`, then the default branch is used. -Take care that only the **combination of component ID, configuration ID and branch ID is unique**. It is possible for two configurations with the -same ID to exist (either for different component or for a different branch). - -Another option is to provide the entire configuration in the `configData` field. In that case the whole configuration data -has to be provided in the request. If you are retrieving a -[stored configuration](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/components/-componentId-/configs/-configurationId-), take -note that the configuration data is the contents of the `configuration` node and not the entire -response. When using the `configData` field, the `configRowIds` and `branchId` values are ignored. When using the `configData` field the `config` field -is ignored for the purpose of reading the configuration, but may still be required in case the component is using -[Default Bucket](https://help.keboola.com/extend/component/tutorial/output-mapping/#configuring-default-bucket). In that case, the -configuration referenced in `config` is used to generate the name of the output bucket. It still holds that configuration data is not read from it. -That means that `configData` always fully overrides the `config` field. - -### Job Mode -When creating a job, you need to provide `mode`. This can be one of `run`, `forceRun` and `debug`. The basic `mode` choice is `run`. -Use the `forceRun` mode to run a configuration that is disabled. The `debug` can be used during [Component Development & Debugging](https://help.keboola.com/extend/component/tutorial/debugging/). - -### Job Runtime configuration -You may provide runtime settings for a job. Runtime settings do not affect what the job does, they affect how the job does it. The available runtime settings are: - -- `backend.type` --- for Snowflake transformations this is the size of the [Snowflake warehouse](https://help.keboola.com/transformations/snowflake-plain/#dynamic-backends) used for the job; otherwise it affects the [container size](https://help.keboola.com/transformations/python-plain/#dynamic-backends). Available values for backend type are `small`, `medium`, `large`. -- `parallelism` --- runs [Configuration Rows](https://help.keboola.com/components/#configuration-rows) (if present in the configuration) in parallel. Allowed values are integer values and `infinity`, which runs all rows in parallel. When not specified, the rows are run sequentially. -- `tag` --- runs the component with a specific version of code. This is mostly used during component development, testing and debugging. - -Runtime parameters can be specified on various levels. The values can be specified in the component configuration. They can also be specified -when creating a job, in which case it overrides the configuration. It may also be specified for an orchestration, in which case it overrides what is specified -in individual jobs of that orchestration. - -When stored in the component configuration, the runtime settings live in a top-level `runtime` node of the configuration JSON --- a sibling of `parameters`, -not inside it. For example, to pin all jobs of a configuration to a specific image tag: - -{% highlight json %} -{ - "parameters": { - "...": "..." - }, - "runtime": { - "tag": "my-branch-3" - } -} -{% endhighlight %} - -Jobs of this configuration then run the given image tag (the job detail shows the resolved value in its `tag` field) until the `runtime.tag` key -is removed from the configuration. This is the usual way of testing a development build of a component in a single project without affecting -other projects --- the tag in the [Developer Portal](https://help.keboola.com/extend/publish/) stays untouched. Do not forget to remove the key when done; a pinned -configuration keeps running the old image even after new versions of the component are released. - -### Job Type -Job can be of one of the four types `standard`, `container`, `phaseContainer` and `orchestrationContainer`. The `standard` is something which does actual work. -Only standard jobs consume billable time and are counted towards consumption of any resources. Other job types are virtual containers encapsulating standard jobs. - -The `container` job represent a job containing [parallel executions](/integrate/jobs/#job-runtime-configuration) -of configuration rows. `phaseContainer` type contains standard jobs in a single -phase of an orchestration. `orchestrationContainer` job type represents an [orchestration](https://help.keboola.com/orchestrator/) and -contains phase jobs of that orchestration. What these job types have in common is a strong -[parent-child relationship](/integrate/jobs/#job-id). This means for example that when a child job fails, the container fails too. The -behavior can be further controlled by the `onError` setting. You cannot specify job type when creating a job, it is selected automatically as needed. - -## Working with the Jobs API -The main API to run the jobs is [Job Queue API](https://api.keboola.com/?service=job-queue#job-queue). There are some API calls from other services -which might be useful when working with jobs: - -- [Create configurations](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/components/-componentId-/configs) -- [List Job Events](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/events) -- [Encrypt values](https://api.keboola.com/?service=encryption#post-/encrypt) -- [Run Synchronous Actions](https://api.keboola.com/?service=sync-actions#sync-actions/POST/actions) -- [Subscribe to Job Events](https://api.keboola.com/?service=notification#notification/tag/project-subscriptions/POST/project-subscriptions) -- [Schedule jobs](https://api.keboola.com/?service=scheduler#scheduler/tag/schedules/POST/schedules) - -The component jobs are asynchronous operations, this means that you create it and then you have to actively wait for the result. Note that there -are other *unrelated* cases of asynchronous operations in Keboola Platform which are in principle the same, but may differ in little details. -The most common one is: -[Storage Jobs](https://api.keboola.com/?service=storage#get-/v2/storage/jobs/-jobId-), triggered, for instance, by -[asynchronous imports](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/buckets/-id-/tables-async) -or [exports](https://keboola.docs.apiary.io/#reference/tables/unload-data-asynchronously/asynchronous-export) - -### Run a Job -You need to know the *component Id* and *configuration Id* to create a job. You can get these from the UI links. To use the API to obtain a -list of all components available in the project, and their configuration, you can use the -[Get components](https://api.keboola.com/?service=storage#get-/v2/storage). -See an [example](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb). -A snippet of the response is below: - -{% highlight json %} -[ - { - "id": "keboola.ex-db-snowflake", - "type": "extractor", - "name": "Snowflake", - "description": "Cloud-Native Elastic Data Warehouse Service", - "documentationUrl": "https://github.com/keboola/db-extractor-snowflake/blob/master/README.md", - "configurations": [ - { - "id": "554424643", - "name": "Sample database", - "description": "", - "created": "2019-12-03T11:18:28+0100", - "creatorToken": { - "id": 199182, - "description": "ondrej.popelka@keboola.com" - }, - "version": 3, - "changeDescription": "Quickstart config creation", - "isDisabled": false, - "isDeleted": false, - "currentVersion": { - "created": "2019-12-03T11:19:50+0100", - "creatorToken": { - "id": 199182, - "description": "ondrej.popelka@keboola.com" - }, - "changeDescription": "Quickstart config creation" - } - } - ] - } -] -{% endhighlight %} - -From there, the important part is the `id` field and `configurations.id` field. For instance, in the -above, there is a database extractor with the `id` `keboola.ex-db-snowflake` and a -configuration with the id `554424643`. - -Then use the [create a job](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) -API call and pass the configuration ID and component ID in request body: - -```json -{ - "component": "keboola.ex-db-snowflake", - "config": "554424643", - "mode": "run" -} -``` - -See an [example](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb). -When a job is created, you will obtain a response similar to this: - -```json -{ - "id": "807932655", - "runId": "807932655", - "parentRunId": "", - "project": { - "id": "7150", - "name": "Sandbox" - }, - "token": { - "id": "199182", - "description": "ondrej.popelka@keboola.com" - }, - "status": "created", - "desiredStatus": "processing", - "mode": "run", - "component": "keboola.ex-db-snowflake", - "config": "554424643", - "configData": [], - "configRowIds": [], - "tag": "5.5.0", - "createdTime": "2022-01-25T16:34:40+00:00", - "startTime": null, - "endTime": null, - "durationSeconds": 0, - "result": [], - "usageData": [], - "isFinished": false, - "url": "https://queue.keboola.com/jobs/807932655", - "branchId": null, - "variableValuesId": null, - "variableValuesData": { - "values": [] - }, - "backend": [], - "metrics": [], - "behavior": { - "onError": null - }, - "parallelism": null, - "type": "standard" -} -``` - -This means that the job was `created` and will automatically start executing. -From the above response, the most important part is `url`, which gives you the URL of the resource for -[Job status polling](https://en.wikipedia.org/wiki/Polling_(computer_science)). - -### Job Polling -If you want to get the actual job result, poll the [Job API](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/GET/jobs/{jobId}) -for the current state of the job. See an [example](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb). - -You will receive a response in the same format as when you crated the job: - -```json -{ - "id": "807933826", - "runId": "807933826", - "parentRunId": "", - "project": { - "id": "7150", - "name": "7150" - }, - "token": { - "id": "199182", - "description": "ondrej.popelka@keboola.com" - }, - "status": "processing", - "desiredStatus": "processing", - "mode": "run", - "component": "keboola.ex-db-snowflake", - "config": "554424643", - "configData": [], - "configRowIds": [], - "tag": "5.5.0", - "createdTime": "2022-01-25T16:41:12+00:00", - "startTime": "2022-01-25T16:41:22+00:00", - "endTime": null, - "durationSeconds": 0, - "result": [], - "usageData": [], - "isFinished": false, - "url": "https://queue.keboola.com/jobs/807933826", - "branchId": null, - "variableValuesId": null, - "variableValuesData": { - "values": [] - }, - "backend": [], - "metrics": [], - "behavior": { - "onError": null - }, - "parallelism": null, - "type": "standard" -} -``` - -From the above response, the most important part is the `status` field (`processing`, in this case). -To obtain the Job result, periodically send the above API call until the job status changes -to one of the finished states or until `isFinished` is true. - -### Run a Debug job -To run a debug job, use `debug` for the mode. Optionally you can provide the component version which should run -to [live test](https://help.keboola.com/extend/component/deployment/#test-live-configurations) an image. - -```json -{ - "component": "keboola.ex-db-snowflake", - "config": "554424643", - "mode": "debug", - "tag": "5.5.0" -} -``` - -The debug mode creates a job that prepares the data folder including the serialized configuration files. Then it compresses the -[data folder](https://help.keboola.com/extend/component/running/#preparing-data-folder) and uploads it to your project's Files in Storage. This way you will get a snapshot -of what the data folder looked like before the component started. If processors are used, a snapshot of the data folder is created before each processor. After the entire component finishes, another snapshot is made. For example, if you run component A with processor B and C in the after section, you will receive: - -- `stage_0` file with contents of the data folder before component A was run -- `stage_1` file with contents of the data folder before processor B was run -- `stage_2` file with contents of the data folder before processor C was run -- `stage_output` file with contents of the data folder before output mapping was about to be performed (after C finished). - -If configuration rows are used, then the above is repeated for each configuration row. If the job finishes with and error, only the stages before the error are uploaded. - -This API call does not upload any tables or files to Storage. I.e. when the component finishes, its output is discarded and the output mapping to storage -is not performed. This makes this API call generally very safe to call, because it cannot break the Keboola project in any way. However, keep -in mind, that if the component has any outside side effects, these will get executed. This applies typically to writers which will write the data -into the external system even when running in debug mode. - -Note that the snapshot archive will contain all files in the data folder including any temporary files produced be the component. The snapshot will not -contain the output state.json file. This is because the snapshot is made before a component is run where the out state of the previous component is -not available any more. Also note that all encrypted values are removed from the configuration file and there is no way to retrieve them. It is -also advisable to run this command with limited input mapping so that you don't end up with gigabyte size archives. +--- +title: Component Jobs +permalink: /integrate/jobs/ +redirect_from: + - /overview/jobs/ +redirect_to: https://help.keboola.com/management/jobs/api/ +--- diff --git a/integrate/jobs/states.png b/integrate/jobs/states.png deleted file mode 100644 index b3e0e6e9..00000000 Binary files a/integrate/jobs/states.png and /dev/null differ diff --git a/integrate/orchestrator/index.md b/integrate/orchestrator/index.md index ea18f2e0..e920a627 100644 --- a/integrate/orchestrator/index.md +++ b/integrate/orchestrator/index.md @@ -1,12 +1,5 @@ ---- -title: Orchestrator -permalink: /integrate/orchestrator/ ---- - -{% comment %} -spusteni jobu -API pro konfiguraci, nepouzivat normalni api -custom joby - -sloucit s /automate/ -{% endcomment %} \ No newline at end of file +--- +title: Orchestrator +permalink: /integrate/orchestrator/ +redirect_to: https://help.keboola.com/flows/ +--- diff --git a/overview/api/apiary-console.png b/overview/api/apiary-console.png deleted file mode 100644 index 1965d06e..00000000 Binary files a/overview/api/apiary-console.png and /dev/null differ diff --git a/overview/api/index.md b/overview/api/index.md index eb65e91b..863e6ee0 100644 --- a/overview/api/index.md +++ b/overview/api/index.md @@ -1,281 +1,5 @@ ---- -title: Our APIs -permalink: /overview/api/ ---- - -* TOC -{:toc} - -All our [Keboola services](/overview/) have a public API on [api.keboola.com](https://api.keboola.com/). We recommend using either the API Console or Postman Client for sending requests to our -API. Most of our APIs accept and return data in JSON format. -Many of these APIs require a *Storage API token*, specified in the `X-StorageApi-Token` header. - -## List of Keboola APIs - -All parts of the Keboola platform can be controlled via an API. -The main APIs for our components are: - -
-Note: The api.keboola.com links in the table below open the API documentation portal for the US Virginia AWS stack. -If you are using a different stack, navigate to your stack's API portal first — see API Documentation Portals below — and then select the service there. -Using a portal for a different stack than your token's stack will result in Invalid Token errors. -
- -| API | Description | -|-------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Keboola Storage API](https://api.keboola.com/?service=storage) ([source](https://github.com/keboola/storage-api-php-client/blob/master/apiary.apib)) | [Storage](https://help.keboola.com/storage/api/) is the main Keboola component storing all data. | -| [Keboola Management API](https://api.keboola.com/?service=manage) | API managing Keboola projects and users (and notifications and features). | -| [AI API](https://api.keboola.com/?service=ai) | API for supporting AI features. | -| [Billing API](https://api.keboola.com/?service=billing) | Billing API for Pay as You Go projects. | -| [Developer Portal API](https://api.keboola.com/?service=developer-portal) | Developer Portal is an application separated from Keboola for [creating components](https://help.keboola.com/extend/component/). | -| [Editor API](https://api.keboola.com/?service=editor) | API for managing SQL editor sessions. | -| [Encryption API](https://api.keboola.com/?service=encryption) | Provides [Encryption](/overview/encryption/). | -| [Importer API](https://api.keboola.com/?service=import) | [Importer](https://help.keboola.com/storage/api/importer/) is a helper service for easy table imports. | -| [Notifications API](https://api.keboola.com/?service=notification) | API to subscribe to events, e.g., failed orchestrations. | -| [OAuth Broker API](https://api.keboola.com/?service=oauth) | OAuth Broker is a component managing [OAuth authorizations](https://help.keboola.com/extend/common-interface/oauth/#authorize) of other components. | -| [Query API](https://api.keboola.com/?service=query) | Query is a service for running SQL queries on Snowflake and BigQuery. | -| [Queue API](https://api.keboola.com/?service=job-queue) | Queue is a service for [running components](https://help.keboola.com/extend/job-queue/) and managing [Jobs](/integrate/jobs/). | -| [Sandboxes Service API](https://api.keboola.com/?service=sandboxes-service) | API for managing Apps and Python/R workspaces. | -| [Scheduler API](https://api.keboola.com/?service=scheduler) | API to automate configurations. | -| [Stream API](https://api.keboola.com/?service=stream) | The Keboola Stream API allows you to ingest small and frequent events into your project's storage. | -| [Synchronous Actions API](https://api.keboola.com/?service=sync-actions) | API to trigger [Synchronous Actions](https://help.keboola.com/extend/common-interface/actions/). | -| [Vault](https://api.keboola.com/?service=vault) | Service handling variables & credentials storage. | - -If you're unsure which API to use, refer to our [integration guide](/integrate/). It describes the roles of different APIs and contains examples of commonly -performed actions. - -## Stacks and Endpoints -Keboola is available in multiple [stacks](https://help.keboola.com/overview/#stacks), which can be -either multi-tenant or single-tenant. Current multi-tenant stacks are: - -- US Virginia AWS – [connection.keboola.com](https://connection.keboola.com/) -- US Virginia GCP - [connection.us-east4.gcp.keboola.com](https://connection.us-east4.gcp.keboola.com/) -- EU Frankfurt AWS – [connection.eu-central-1.keboola.com](https://connection.eu-central-1.keboola.com/) -- EU Ireland Azure – [connection.north-europe.azure.keboola.com](https://connection.north-europe.azure.keboola.com/) -- EU Frankfurt GCP - [connection.europe-west3.gcp.keboola.com](https://connection.europe-west3.gcp.keboola.com/) - -Each stack operates as an independent instance of Keboola services with its own data, users, and tokens. -Single-tenant stacks are available for a single enterprise customer, with a domain name -in the format `connection.CUSTOMER_NAME.keboola.com`. - -### API Documentation Portals - -The API documentation portal (`api.*`) is deployed independently per stack. Always use the portal -for your own stack — tokens are not valid across stacks, and using the wrong portal will cause -`Invalid Token` errors when trying out API calls. - -| Stack | API Documentation Portal | -|---|---| -| US Virginia AWS | [api.keboola.com](https://api.keboola.com/) | -| EU Frankfurt AWS | [api.eu-central-1.keboola.com](https://api.eu-central-1.keboola.com/) | -| EU Ireland Azure | [api.north-europe.azure.keboola.com](https://api.north-europe.azure.keboola.com/) | -| EU Frankfurt GCP | [api.europe-west3.gcp.keboola.com](https://api.europe-west3.gcp.keboola.com/) | -| US Virginia GCP | [api.us-east4.gcp.keboola.com](https://api.us-east4.gcp.keboola.com/) | - -### Machine-Readable API Index - -For agentic usage and tooling (AI agents, MCP servers, CI), each stack's API portal also publishes a -machine-readable index of its APIs at `https://api./apis.json` — for example, -[api.keboola.com/apis.json](https://api.keboola.com/apis.json). The index lists each available service with -its base `apiUrl` and a link to its OpenAPI specification (`openApiSpecUrl`), so tools can discover and load -the specs programmatically: - -{% highlight json %} -{ - "stack": "keboola.com", - "services": [ - { - "id": "storage", - "name": "Storage API", - "apiUrl": "https://connection.keboola.com", - "openApiSpecUrl": "https://api.keboola.com/specs/storage.json" - } - ] -} -{% endhighlight %} - -The index is stack-specific (excluded services are omitted). The raw specs under `/specs/` keep their original -`servers`, so consumers should use the `apiUrl` from the index as the base URL. The `openApiSpecUrl` extension -mirrors the source document (`.json` or `.yaml`) — use the exact URL from the index rather than assuming one. - -### Service Endpoints - -If you are calling the APIs directly (not through the portal), modify the hostname accordingly. -Otherwise, you may encounter `Invalid Token` or unauthorized errors. The *authoritative list* of available endpoints is provided by the [Storage API Index Call](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/components/-componentId-). The following is a sample response: - -{% highlight json %} -{ - ..., - "services": [ - { - "id": "import", - "url": "https://import.keboola.com" - }, - { - "id": "oauth", - "url": "https://oauth.keboola.com" - }, - { - "id": "queue", - "url": "https://queue.keboola.com" - }, - { - "id": "billing", - "url": "https://billing.keboola.com" - }, - { - "id": "encryption", - "url": "https://encryption.keboola.com" - }, - { - "id": "scheduler", - "url": "https://scheduler.keboola.com" - }, - { - "id": "sync-actions", - "url": "https://sync-actions.keboola.com" - }, - { - "id": "notification", - "url": "https://notification.keboola.com" - } - ], -} -{% endhighlight %} - -The services listed above are: - -- `import` --- [Storage Importer Service](https://help.keboola.com/storage/api/importer/) -- `oauth` --- [OAuth Manager Service](https://help.keboola.com/extend/common-interface/oauth/) -- `queue` --- [Service for Running Components](https://help.keboola.com/extend/job-queue/) -- `billing` --- Service for Computing Credits -- `encryption` --- Service for [Encryption](https://developers.keboola.com/overview/encryption/) -- `scheduler` --- [Service for Configuring Schedules](https://developers.keboola.com/automate/set-schedule/) -- `sync-actions` --- [Service for Running Synchronous Actions](https://help.keboola.com/extend/common-interface/actions/) -- `notification` --- Service for Configuring Job Notifications - -For convenience, the following table lists active services and their URLs, though for an authoritative answer -and in application integrations, we strongly suggest using the above API call. - -| API | Service | Region | URL | -|------------------------|----------------|------------------|-----------------------------------------------------| -| AI | `ai` | US Virginia AWS | https://ai.keboola.com | -| AI | `ai` | US Virginia GCP | https://ai.us-east4.gcp.keboola.com | -| AI | `ai` | EU Frankfurt AWS | https://ai.eu-central-1.keboola.com | -| AI | `ai` | EU Ireland Azure | https://ai.north-europe.azure.keboola.com | -| AI | `ai` | EU Frankfurt GCP | https://ai.europe-west3.gcp.keboola.com | -| Billing | `billing` | US Virginia AWS | https://billing.keboola.com | -| Billing | `billing` | US Virginia GCP | https://billing.us-east4.gcp.keboola.com | -| Billing | `billing` | EU Frankfurt AWS | https://billing.eu-central-1.keboola.com | -| Billing | `billing` | EU Ireland Azure | https://billing.north-europe.azure.keboola.com | -| Billing | `billing` | EU Frankfurt GCP | https://billing.europe-west3.gcp.keboola.com | -| Developer Portal | `developer` | US Virginia AWS | https://developer.keboola.com | -| Developer Portal | `developer` | US Virginia GCP | https://developer.us-east4.gcp.keboola.com | -| Developer Portal | `developer` | EU Frankfurt AWS | https://developer.eu-central-1.keboola.com | -| Developer Portal | `developer` | EU Ireland Azure | https://developer.north-europe.azure.keboola.com | -| Developer Portal | `developer` | EU Frankfurt GCP | https://developer.europe-west3.gcp.keboola.com | -| Editor | `editor` | US Virginia AWS | https://editor.keboola.com | -| Editor | `editor` | US Virginia GCP | https://editor.us-east4.gcp.keboola.com | -| Editor | `editor` | EU Frankfurt AWS | https://editor.eu-central-1.keboola.com | -| Editor | `editor` | EU Ireland Azure | https://editor.north-europe.azure.keboola.com | -| Editor | `editor` | EU Frankfurt GCP | https://editor.europe-west3.gcp.keboola.com | -| Encryption | `encryption` | US Virginia AWS | https://encryption.keboola.com | -| Encryption | `encryption` | US Virginia GCP | https://encryption.us-east4.gcp.keboola.com | -| Encryption | `encryption` | EU Frankfurt AWS | https://encryption.eu-central-1.keboola.com | -| Encryption | `encryption` | EU Ireland Azure | https://encryption.north-europe.azure.keboola.com | -| Encryption | `encryption` | EU Frankfurt GCP | https://encryption.europe-west3.gcp.keboola.com | -| Importer | `import` | US Virginia AWS | https://import.keboola.com | -| Importer | `import` | US Virginia GCP | https://import.us-east4.gcp.keboola.com | -| Importer | `import` | EU Frankfurt AWS | https://import.eu-central-1.keboola.com | -| Importer | `import` | EU Ireland Azure | https://import.north-europe.azure.keboola.com | -| Importer | `import` | EU Frankfurt GCP | https://import.europe-west3.gcp.keboola.com | -| Management | `management` | US Virginia AWS | https://management.keboola.com | -| Management | `management` | US Virginia GCP | https://management.us-east4.gcp.keboola.com | -| Management | `management` | EU Frankfurt AWS | https://management.eu-central-1.keboola.com | -| Management | `management` | EU Ireland Azure | https://management.north-europe.azure.keboola.com | -| Management | `management` | EU Frankfurt GCP | https://management.europe-west3.gcp.keboola.com | -| Notification | `notification` | US Virginia AWS | https://notification.keboola.com | -| Notification | `notification` | US Virginia GCP | https://notification.us-east4.gcp.keboola.com | -| Notification | `notification` | EU Frankfurt AWS | https://notification.eu-central-1.keboola.com | -| Notification | `notification` | EU Ireland Azure | https://notification.north-europe.azure.keboola.com | -| Notification | `notification` | EU Frankfurt GCP | https://notification.europe-west3.gcp.keboola.com | -| OAuth | `oauth` | US Virginia AWS | https://oauth.keboola.com | -| OAuth | `oauth` | US Virginia GCP | https://oauth.europe-west3.gcp.keboola.com | -| OAuth | `oauth` | EU Frankfurt AWS | https://oauth.eu-central-1.keboola.com | -| OAuth | `oauth` | EU Ireland Azure | https://oauth.north-europe.azure.keboola.com | -| OAuth | `oauth` | EU Frankfurt GCP | https://oauth.europe-west3.gcp.keboola.com | -| Query | `query` | US Virginia AWS | https://query.keboola.com | -| Query | `query` | US Virginia GCP | https://query.us-east4.gcp.keboola.com | -| Query | `query` | EU Frankfurt AWS | https://query.eu-central-1.keboola.com | -| Query | `query` | EU Ireland Azure | https://query.north-europe.azure.keboola.com | -| Query | `query` | EU Frankfurt GCP | https://query.europe-west3.gcp.keboola.com | -| Queue | `queue` | US Virginia AWS | https://queue.keboola.com | -| Queue | `queue` | US Virginia GCP | https://queue.us-east4.gcp.keboola.com | -| Queue | `queue` | EU Frankfurt AWS | https://queue.eu-central-1.keboola.com | -| Queue | `queue` | EU Ireland Azure | https://queue.north-europe.azure.keboola.com | -| Queue | `queue` | EU Frankfurt GCP | https://queue.europe-west3.gcp.keboola.com | -| Scheduler | `scheduler` | US Virginia AWS | https://scheduler.keboola.com | -| Scheduler | `scheduler` | US Virginia GCP | https://scheduler.us-east4.gcp.keboola.com | -| Scheduler | `scheduler` | EU Frankfurt AWS | https://scheduler.eu-central-1.keboola.com | -| Scheduler | `scheduler` | EU Ireland Azure | https://scheduler.north-europe.azure.keboola.com | -| Scheduler | `scheduler` | EU Frankfurt GCP | https://scheduler.europe-west3.gcp.keboola.com | -| Storage | | US Virginia AWS | https://connection.keboola.com/ | -| Storage | | US Virginia GCP | https://connection.us-east4.gcp.keboola.com | -| Storage | | EU Frankfurt AWS | https://connection.eu-central-1.keboola.com/ | -| Storage | | EU Ireland Azure | https://connection.north-europe.azure.keboola.com/ | -| Storage | | EU Frankfurt GCP | https://connection.europe-west3.gcp.keboola.com/ | -| Stream | `stream` | US Virginia AWS | https://stream.keboola.com | -| Stream | `stream` | US Virginia GCP | https://stream.us-east4.gcp.keboola.com | -| Stream | `stream` | EU Frankfurt AWS | https://stream.eu-central-1.keboola.com | -| Stream | `stream` | EU Ireland Azure | https://stream.north-europe.azure.keboola.com | -| Stream | `stream` | EU Frankfurt GCP | https://stream.europe-west3.gcp.keboola.com | -| Sync Actions | `sync-actions` | US Virginia AWS | https://sync-actions.keboola.com/ | -| Sync Actions | `sync-actions` | US Virginia GCP | https://sync-actions.us-east4.gcp.keboola.com | -| Sync Actions | `sync-actions` | EU Frankfurt AWS | https://sync-actions.eu-central-1.keboola.com | -| Sync Actions | `sync-actions` | EU Ireland Azure | https://sync-actions.north-europe.azure.keboola.com | -| Sync Actions | `sync-actions` | EU Frankfurt GCP | https://sync-actions.europe-west3.gcp.keboola.com | -| Vault | `vault` | US Virginia AWS | https://vault.keboola.com | -| Vault | `vault` | US Virginia GCP | https://vault.us-east4.gcp.keboola.com | -| Vault | `vault` | EU Frankfurt AWS | https://vault.eu-central-1.keboola.com | -| Vault | `vault` | EU Ireland Azure | https://vault.north-europe.azure.keboola.com | -| Vault | `vault` | EU Frankfurt GCP | https://vault.europe-west3.gcp.keboola.com | - - -***Important**: Each stack also uses its own set of [IP addresses](https://help.keboola.com/extractors/ip-addresses/).* - -## Calling API - -There are several ways to send requests to our APIs: - -### Apiary Console -Send requests to our API directly from the Apiary console by clicking on **Switch to console** or **Try**. -Fill in the request headers and parameters, then click **Call Resource**. - -{: .image-popup} -![Apiary console](/overview/api/apiary-console.png) - -The Apiary console is fine if you send API requests only occasionally. It requires no application installation; -however, it has no history and no other useful features. - -### Postman Client -[Postman](https://www.getpostman.com/) is a generic HTTP API client, suitable for more regular API work. -We also provide a collection of [useful API calls](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D?version=latest#9b9f3e7b-de3b-4c90-bad6-a8760e3852eb) with examples. -The collection contains code examples in various languages; the requests can also be imported into the Postman application. - -{: .image-popup} -![Postman Docs](/overview/api/postman-import.png) - -### cURL -[cURL](https://curl.haxx.se/) is a common library with a [command-line interface (CLI)](https://curl.haxx.se/docs/manpage.html). -You can use the cURL CLI to create simple scripts for interacting with Keboola APIs. For example, to [run a job](/integrate/jobs/): - -{% highlight shell %} -curl --location --request POST 'https://queue.keboola.com/jobs' \ ---header 'X-StorageApi-Token: YourStorageToken' \ ---header 'Content-Type: application/json' \ ---data-raw '{ - "mode": "run", - "component": "keboola.ex-db-mysql", - "config": "sampledatabase" -}' -{% endhighlight %} +--- +title: Our APIs +permalink: /overview/api/ +redirect_to: https://help.keboola.com/overview/#apis-and-service-endpoints +--- diff --git a/overview/api/postman-import.png b/overview/api/postman-import.png deleted file mode 100644 index 12333eb0..00000000 Binary files a/overview/api/postman-import.png and /dev/null differ diff --git a/overview/encryption-1.png b/overview/encryption-1.png deleted file mode 100644 index 9d28c070..00000000 Binary files a/overview/encryption-1.png and /dev/null differ diff --git a/overview/encryption-2.png b/overview/encryption-2.png deleted file mode 100644 index 51bb3da4..00000000 Binary files a/overview/encryption-2.png and /dev/null differ diff --git a/overview/encryption.md b/overview/encryption.md index 1c66764d..aec01c2b 100644 --- a/overview/encryption.md +++ b/overview/encryption.md @@ -1,150 +1,5 @@ --- title: Encryption permalink: /overview/encryption/ +redirect_to: https://help.keboola.com/extend/common-interface/encryption/ --- - -* TOC -{:toc} - -Many [Keboola components](/overview/) use the Encryption API to encrypt sensitive values -intended for secure storage. These values are then decrypted within the component itself. -This process ensures that the encrypted values are only accessible inside the components and not -by API users. Additionally, no decryption API is available, meaning end-users cannot decrypt -these values. - -Decryption occurs solely during the serialization of configuration to the Docker container's -configuration file. The decrypted data are stored on the Docker host drive and are promptly -deleted after the container's completion. The component code exclusively accesses the decrypted data. - -## UI Interaction -When saving arbitrary configuration data, if a key is prefixed with the `#` character, the associated value is automatically encrypted. -For instance, consider the following configuration: - -{: .image-popup} -![Screenshot - Configuration editor - before](/overview/encryption-1.png) - -After saving, the configuration appears as follows: - -{: .image-popup} -![Screenshot - Configuration editor - after](/overview/encryption-2.png) - -Once saved, the value becomes encrypted and irreversible. The component defines which values are -encrypted, indicating that not all values can be encrypted unless explicitly supported by the component. - -For example, a component requiring the following configuration: - -{% highlight json %} -{ - "username": "JohnDoe", - "#password": "password" -} -{% endhighlight %} - -indicates that the password will be encrypted while the username will not. Adding a -prefix `#` to `username` is ineffective, as the component does not recognize such a key, -even though its value would be encrypted and decrypted normally. Internally, the -[Encryption API](#encrypting-data-with-api) encrypts these values before saving. - -### UI Configuration Adjustment -The UI prioritizes encrypted values over plain ones. If both `password` and `#password` are provided, only `#password` will be retained. -Consequently, this configuration: - -{% highlight json %} -{ - "username": "JohnDoe", - "#password": "KBC::ProjectSecure::ENCODEDSTRING", - "password": "secret", -} -{% endhighlight %} - -will be transformed to: - -{% highlight json %} -{ - "username": "JohnDoe", - "#password": "KBC::ProjectSecure::ENCODEDSTRING" -} -{% endhighlight %} - -## Encrypting Data with API -The [Encryption API](https://api.keboola.com/?service=encryption#post-/encrypt) can handle -both strings and arbitrary JSON data. For strings, the entire string is encrypted. In JSON data, -only scalar keys starting with `#` are encrypted. For example, encrypting the following: - -{% highlight json %} -{ - "foo": "bar", - "#encryptMe": "secret", - "#encryptMeToo": { - "another": "secret" - } -} -{% endhighlight %} - -results in: - -{% highlight json %} -{ - "foo": "bar", - "#encryptMe": "KBC::ProjectSecure::ENCODEDSTRING", - "#encryptMeToo": { - "another": "secret" - } -} -{% endhighlight %} - -To encrypt a single string, such as a password, submit the text string for encryption -(no JSON or quotation is used). For example, encrypting - - mySecretPassword - -yields - - KBC::ProjectSecure::ENCODEDSTRING - -The `Content-Type` header in the request differentiates whether the body is treated as a string (`text/plain`) or JSON (`application/json`). - -### Encryption Parameters -The Encryption API accepts the following **optional** parameters: - -- `componentId` --- ID of a [Keboola component](https://help.keboola.com/extend/component/tutorial/#creating-component), -- `projectId` --- ID of a Keboola project, -- `configId` --- ID of a component configuration, and -- `branchType` --- Branch type --- either `default` (meaning the default production branch) or `dev` (meaning any development branch other than the production). - -The cipher created depends on the provided parameters: - -- With only `componentId`, the cipher starts with `KBC::ComponentSecure::` and is decryptable -across all configurations of that component. This is recommended for **component-specific secrets** -applicable across all customers (e.g., as a master authorization token). - -- Adding `projectId` to the `componentId` changes the prefix to `KBC::ProjectSecure::`, making the cipher decryptable within -the project's component configurations. This is recommended for **all secrets** used within a typical Keboola project. - -- Providing all three IDs (`componentId`, `projectId`, `configId`) generates a cipher starting with -`KBC::ConfigSecure::`, limiting decryption to a specific configuration. This is useful for preventing the copying of configurations. - -- Using only `projectId` yields a cipher that begins with `KBC::ProjectWideSecure::`, decryptable across the project's configurations. -This cipher type helps encrypt information shared across multiple components, e.g., SSH tunnel settings. - -- Adding `branchType` restricts the encryption to the default production branch or to development branches. This means an encrypted value with this setting cannot be moved between production and development branches or vice versa. It is not possible to encrypt a value for just one development branch. - - - Using `branchType` with `componentId` and `projectId` results in a cipher beginning with `KBC::BranchTypeSecure::`. This allows decryption either in the production or in the development configuration of the specified component in the project. - - - Using `branchType` with all three IDs (`componentId`, `projectId`, `configId`) creates a cipher that starts with `KBC::BranchTypeConfigSecure::`. It can only be decrypted within a specific production or development component configuration in a specific project. - - - Using `branchType` with `projectId` creates a cipher beginning with `KBC::ProjectWideBranchTypeSecure::`. This cipher allows decryption either in the production or in the development configurations in the project. - -The following rules apply to all ciphers: - -- Providing only a `configId` without a `projectId` is not allowed. Similarly, providing only `branchType` without `projectId` is also not allowed. -- Cipher decryption is only possible in the [region](/overview/api/#regions-and-endpoints) where the cipher was created. For example, ciphers with prefixes `KBC::ProjectSecureKV::` (Azure) or `KBC::ProjectSecureGKMS::` (GCP), instead of `KBC::ProjectSecure::` (AWS), use the same business logic but are specific to their region and technology and are not interchangeable. -- There is no decryption API; the cipher is decrypted internally before a component is run. -- Ciphering a value that is already encrypted does not change its encryption. -- There is no way to retrieve the component, project, configuration ID, or branch type from the cipher. -- The IDs referenced during cipher creation do not need to exist then. For example, you can create a cipher for a component not yet registered, which will start working as soon as the component is registered. Similarly, ciphers can be created for projects and configurations without access to them. - -By default, values encrypted in component configurations are encrypted using the `KBC::ProjectSecure::` cipher, meaning -the cipher is not transferable between regions, components, or projects. It is transferable between -different configurations of the same component within the project where it was created. If you create a configuration containing `KBC::ConfigSecure::` ciphers, -note that the configuration will not work when copied.