Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 3 additions & 36 deletions _data/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,51 +4,18 @@ items:

- url: /overview/
title: Keboola Overview
items:
- url: /overview/api/
title: Our APIs

- url: /overview/encryption/
title: Encryption

- url: /extend/
title: Extending Keboola
title: Extending Keboola (moved)

- url: /integrate/
title: Integration
title: Integration (moved)
items:
- url: /integrate/jobs/
title: Component Jobs

- url: /integrate/artifacts/
title: Artifacts
items:
- url: /integrate/artifacts/tutorial/
title: Tutorial

- url: /integrate/database/
title: SSH Tunnel for Database Extractors

- url: /integrate/data-streams/
title: Data Streams
items:
- url: /integrate/data-streams/tutorial/
title: Tutorial

- url: /integrate/data-streams/overview/
title: Overview

- url: /automate/
title: Automation/Common Tasks
items:
- url: /automate/run-job/
title: Run Job

- url: /automate/run-orchestration/
title: Run Orchestration

- url: /automate/set-schedule/
title: Set Schedule
title: Automation (moved)

- url: /cli/
title: CLI (moved)
44 changes: 1 addition & 43 deletions automate/index.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,5 @@
---
title: Automation/Common Tasks
permalink: /automate/
redirect_to: https://help.keboola.com/flows/
---

## Automation

Use the Keboola Orchestrator component to specify what tasks should be executed in what order and
configure their automatic execution (specified intervals, specified times of the day, etc.).

The set of [Keboola APIs](/overview/api/) provides full automation of the data warehouse cycle.
The end-to-end serverless solution automatically enables you to connect data sources, automatically store data
in the correct format, check for format inconsistencies, and choose different metadata providers based on the
operation you wish to perform on the data. The platform scales the needed resources automatically across various
types of data (structured, semi-structured, and non-structured) and processes.

The whole environment tracks all the [operational metadata](https://api.keboola.com/?service=storage#tag--Events)
and can be accessed without needing a server via APIs. This is useful when automating development, testing and
production run of data jobs with automatic controls of [pipelines](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches/-id-).

As Storage API is part of the wider Keboola platform, it is an essential element in providing coherent data
fabric across clouds, users, services, and on premise.


## CI/CD

No matter whether you use Jenkins, CircleCI, AWS CodeBuilder, or Azure DevOps, you can utilise
the Keboola API within your existing CI/CD pipeline to deploy and manage new versions of your data
pipeline and data process automation tasks.

## Documentation

You can learn about how to set up our Orchestrator on [help.keboola.com/tutorial/automate/](https://help.keboola.com/tutorial/automate/).

{% comment %}
- Load data from your system
- Trigger orchestrations
- Send data
- Copy buckets from different projects


Orchestrator

spusteni jobu
API pro konfiguraci, nepouzivat normalni api
custom joby
{% endcomment %}
Binary file removed automate/job-parameters.png
Binary file not shown.
Binary file removed automate/job-row-parameters.png
Binary file not shown.
Binary file removed automate/orchestration-parameters.png
Binary file not shown.
52 changes: 5 additions & 47 deletions automate/run-job.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,5 @@
---
title: Run a Job
permalink: /automate/run-job/
---

A [job](https://help.keboola.com/management/jobs/) represents a work being done in Keboola.
You can create (run) a job from the UI or via scheduled [Orchestrations](https://help.keboola.com/orchestrator/) or Flows.
A job can also be created manually via the API. The easiest way to get started is to create
a [configuration](https://help.keboola.com/components/) of the component you want to run and run it manually in the UI.
Once you're satisfied with the result, look at the successful job:

{: .image-popup}
![Screenshot -- Job Parameters](/automate/job-parameters.png)

In the job detail, you can see the parameters required to run the configuration, in this case:

```
mode: run
component: keboola.ex-db-snowflake
config: 493493
```

Then create a [Storage API token](https://help.keboola.com/management/project/tokens/) which you will use to
run the API requests (if you don't have one already). We recommend to create
as restricted token as possible -- in this case limit it to to the component with ID `keboola.ex-db-snowflake`:

{: .image-popup}
![Screenshot -- Token Settings](/automate/token-settings.png)

Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to
create a job with the same parameters
(see [example](https://documenter.getpostman.com/view/3086797/77h845D#fd60aa15-485c-4922-8536-c2ba2f27e8ea)):

```bash
curl --location --request POST 'https://queue.keboola.com/jobs' \
--header 'X-StorageApi-Token: YOUR_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"mode": "run",
"component": "keboola.ex-db-snowflake",
"config": "493493"
}'
```

Take care to use the right endpoint depending on which [Stack](https://help.keboola.com/overview/#stacks) are you using.
You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about
the concept of [Jobs](/integrate/jobs/).
---
title: Run a Job
permalink: /automate/run-job/
redirect_to: https://help.keboola.com/management/jobs/api/
---
54 changes: 5 additions & 49 deletions automate/run-orchestration.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,5 @@
---
title: Run Orchestration
permalink: /automate/run-orchestration/
---

Running an [Orchestration](https://help.keboola.com/orchestrator/) or Flow is in principle same as running
a [job](/automate/run-job/).
You can create (run) an orchestration job from the UI. A job can also be created manually via the API.
The easiest way to get started is to create a configuration of orchestration or flow and run it manually in the UI.
Once you're satisfied with the result, look at the successful job:

{: .image-popup}
![Screenshot -- Orchestration Parameters](/automate/orchestration-parameters.png)

In the job detail, you can see the parameters required to run the configuration, in this case:

```
mode: run
component: keboola.orchestrator
config: 1496488
```

The difference between running an arbitrary [component job](/automate/run-job/) and an orchestrator job is only in
that the `component` value is always `keboola.orchestrator`. The same component is used for both Orchestrator
and Flow jobs.

Create a [Storage API token](https://help.keboola.com/management/project/tokens/) which you will use to
run the API requests (if you don't have one already). We recommend to create a token with **Full Access** to
components. Though you can list the components used in the orchestration, this leads to a fragile setup when
modifying the orchestration may need the modification of the the token too.

Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to
create a job with the same parameters
(see [example](https://documenter.getpostman.com/view/3086797/77h845D#3e71b131-afd4-44be-9831-6534e581f2e0)):

```bash
curl --location --request POST 'https://queue.keboola.com/jobs' \
--header 'X-StorageApi-Token: YOUR_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"mode": "run",
"component": "keboola.orchestrator",
"config": "1496488"
}'
```

Take care to use the right endpoint depending on which [Stack](https://help.keboola.com/overview/#stacks) are you using.
You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about
the concept of [Jobs](/integrate/jobs/).
---
title: Run Orchestration
permalink: /automate/run-orchestration/
redirect_to: https://help.keboola.com/management/jobs/api/
---
Loading
Loading