Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
216 changes: 216 additions & 0 deletions MIGRATION-REPORT.md

Large diffs are not rendered by default.

369 changes: 369 additions & 0 deletions _data/navigation.yml

Large diffs are not rendered by default.

Binary file added public/automate/job-parameters.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/automate/job-row-parameters.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/automate/orchestration-parameters.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/automate/token-settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added public/extend/common-interface/sandbox-output.png
Binary file added public/extend/component/deployment/gitlab-1.png
Binary file added public/extend/component/deployment/gitlab-2.png
Binary file added public/extend/component/dynamic-mapping.png
Binary file added public/extend/component/running/sandbox-data.png
Binary file added public/extend/component/tutorial/debug-1.png
Binary file added public/extend/component/tutorial/debug-2.png
Binary file added public/extend/component/tutorial/debug-3.png
Binary file added public/extend/component/tutorial/debug-4.png
Binary file added public/extend/component/tutorial/gh-build-1.png
Binary file added public/extend/component/tutorial/gh-build-2.png
Binary file added public/extend/component/tutorial/hello-world.png
Binary file added public/extend/component/tutorial/join-vendor.png
Binary file added public/extend/component/tutorial/processors-1.png
Binary file added public/extend/component/ui-options/auth-0.png
Binary file added public/extend/component/ui-options/auth-1.png
Binary file added public/extend/component/ui-options/form.png
Binary file added public/extend/data.zip
Binary file not shown.
Binary file added public/extend/generic-extractor/events.png
Binary file added public/extend/generic-extractor/functions.png
Binary file added public/extend/generic-extractor/generic-intro.png
Binary file added public/extend/generic-extractor/schema-test.png
Binary file added public/extend/generic-extractor/template-1.png

Large diffs are not rendered by default.

Binary file added public/extend/generic-extractor/tutorial/img.png
Binary file added public/extend/generic-extractor/ui.png
2 changes: 2 additions & 0 deletions public/extend/job-queue/docker-runner.svg

Large diffs are not rendered by default.

Binary file added public/extend/publish/approve.png
Binary file added public/extend/publish/schema-bad.png
Binary file added public/extend/publish/schema-good.png
Binary file added public/integrate/data-streams/tutorial/table.png
Binary file added public/integrate/data-streams/tutorial/token.png
Binary file added public/integrate/database/ssh-tunnel.jpg
Binary file added public/integrate/jobs/states.png
1 change: 1 addition & 0 deletions public/integrate/storage/api/async-import-handling.svg

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions public/integrate/storage/new-table.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
"id","secondCol"
"1","a"
"2","b"
"3","c"
"4","d"
2 changes: 2 additions & 0 deletions public/integrate/storage/sys.c-table-importer.test-config.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
"table","primaryKey","incremental","enclosure","delimiter","escapedBy","tag","rowId"
"in.c-main.new-table","","0","""",",","","new-data","1"
21 changes: 21 additions & 0 deletions public/integrate/variables/countries.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
"COUNTRY","CARS"
"Belgium","6293781"
"Finland","3358232"
"Italy","41393877"
"Romania","6541260"
"Turkey","20193915"
"Bulgaria","2823705"
"France","38720798"
"Netherlands","8977994"
"Russia","42201083"
"Ukraine","8655700"
"Czech Republic","5116750"
"Germany","47418800"
"Poland","20671278"
"Spain","27528877"
"United Kingdom","33792233"
"Azerbaijan","1080912"
"Denmark","2723040"
"Hungary","3393075"
"Portugal","5650428"
"Sweden","5126572"
Binary file added public/integrate/variables/tutorial-1.png
Binary file added public/integrate/variables/tutorial-2.png
3 changes: 3 additions & 0 deletions public/integrate/variables/variables.svg

Large diffs are not rendered by default.

Binary file added public/kbc_structure.png
Binary file added public/overview/api/apiary-console.png
Binary file added public/overview/api/postman-import.png
Binary file added public/overview/encryption-1.png
Binary file added public/overview/encryption-2.png
494 changes: 494 additions & 0 deletions scripts/migrate-devdocs.mjs

Large diffs are not rendered by default.

7 changes: 4 additions & 3 deletions src/content/docs/ai/mcp-server/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
title: Keboola Model Context Protocol (MCP) Server
slug: 'ai/mcp-server'
redirect_from:
- /integrate/mcp/
- /external-integrations/mcp-server/
---

Expand Down Expand Up @@ -61,7 +62,7 @@ In case your AI assistant supports remote connection, you can connect to Keboola
2. Copy the server URL and paste it into your AI assistant's settings.
3. Once you save the settings and refresh your AI assistant, you will be prompted to authenticate with your Keboola account and select the project you want to connect to.

For other options of local deployments see the [Developers Documentation](https://developers.keboola.com/integrate/mcp/#running-keboola-mcp-server-locally-using-uv-command).
For other options of local deployments see the [Developers Documentation](/ai/mcp-server/).


### Using with Claude Desktop
Expand All @@ -81,7 +82,7 @@ These steps must be done by a Claude organization owner or primary owner, or on

If you don't have a paid version you can still use the [`mcp-remote`](https://github.com/geelen/mcp-remote) adapter to connect Claude Desktop to Keboola's MCP Server.

> NOTE: This method requires you to have Node.js installed on your computer. For more information refer to the [Developers Documentation](https://developers.keboola.com/integrate/mcp)
> NOTE: This method requires you to have Node.js installed on your computer. For more information refer to the [Developers Documentation](/ai/mcp-server/)

1. Open the Claude menu on your computer and select **"Settings…"**
2. Click on **"Developer"** in the left-hand bar of the Settings pane, and then click on **"Edit Config"**
Expand Down Expand Up @@ -286,7 +287,7 @@ These headers are set by the client (e.g., your AI agent integration or custom M

For example, setting `X-Read-Only-Mode: true` allows agents to query and explore data but prevents them from creating or modifying configurations.

For detailed technical documentation including the full list of read-only tools and header combination behavior, see the [Developer Documentation](https://developers.keboola.com/integrate/mcp/#tool-authorization-and-access-control).
For detailed technical documentation including the full list of read-only tools and header combination behavior, see the [Developer Documentation](/ai/mcp-server/).

## Advanced Setup Options
These methods are for developers or specific use cases (e.g., testing, contributing to the MCP server).
Expand Down
45 changes: 45 additions & 0 deletions src/content/docs/automate/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: Automation/Common Tasks
slug: 'automate'
---

## Automation

Use the Keboola Orchestrator component to specify what tasks should be executed in what order and
configure their automatic execution (specified intervals, specified times of the day, etc.).

The set of [Keboola APIs](/overview/api/) provides full automation of the data warehouse cycle.
The end-to-end serverless solution automatically enables you to connect data sources, automatically store data
in the correct format, check for format inconsistencies, and choose different metadata providers based on the
operation you wish to perform on the data. The platform scales the needed resources automatically across various
types of data (structured, semi-structured, and non-structured) and processes.

The whole environment tracks all the [operational metadata](https://api.keboola.com/?service=storage#tag--Events)
and can be accessed without needing a server via APIs. This is useful when automating development, testing and
production run of data jobs with automatic controls of [pipelines](https://api.keboola.com/?service=storage#get-/v2/storage/dev-branches/-id-).

As Storage API is part of the wider Keboola platform, it is an essential element in providing coherent data
fabric across clouds, users, services, and on premise.

## CI/CD

No matter whether you use Jenkins, CircleCI, AWS CodeBuilder, or Azure DevOps, you can utilise
the Keboola API within your existing CI/CD pipeline to deploy and manage new versions of your data
pipeline and data process automation tasks.

## Documentation

You can learn about how to set up our Orchestrator on [help.keboola.com/tutorial/automate/](/tutorial/automate/).

<!--
- Load data from your system
- Trigger orchestrations
- Send data
- Copy buckets from different projects

Orchestrator

spusteni jobu
API pro konfiguraci, nepouzivat normalni api
custom joby
-->
Binary file added src/content/docs/automate/job-parameters.png
Binary file added src/content/docs/automate/job-row-parameters.png
45 changes: 45 additions & 0 deletions src/content/docs/automate/run-job/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: Run a Job
slug: 'automate/run-job'
---

A [job](/management/jobs/) represents a work being done in Keboola.
You can create (run) a job from the UI or via scheduled [Orchestrations](/orchestrator/) or Flows.
A job can also be created manually via the API. The easiest way to get started is to create
a [configuration](/components/) of the component you want to run and run it manually in the UI.
Once you're satisfied with the result, look at the successful job:

![Screenshot -- Job Parameters](/automate/job-parameters.png)

In the job detail, you can see the parameters required to run the configuration, in this case:

```
mode: run
component: keboola.ex-db-snowflake
config: 493493
```

Then create a [Storage API token](/management/project/tokens/) which you will use to
run the API requests (if you don't have one already). We recommend to create
as restricted token as possible -- in this case limit it to to the component with ID `keboola.ex-db-snowflake`:

![Screenshot -- Token Settings](/automate/token-settings.png)

Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to
create a job with the same parameters
(see [example](https://documenter.getpostman.com/view/3086797/77h845D#fd60aa15-485c-4922-8536-c2ba2f27e8ea)):

```bash
curl --location --request POST 'https://queue.keboola.com/jobs' \
--header 'X-StorageApi-Token: YOUR_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"mode": "run",
"component": "keboola.ex-db-snowflake",
"config": "493493"
}'
```

Take care to use the right endpoint depending on which [Stack](/overview/#stacks) are you using.
You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about
the concept of [Jobs](/integrate/jobs/).
48 changes: 48 additions & 0 deletions src/content/docs/automate/run-orchestration/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
title: Run Orchestration
slug: 'automate/run-orchestration'
---

Running an [Orchestration](/orchestrator/) or Flow is in principle same as running
a [job](/automate/run-job/).
You can create (run) an orchestration job from the UI. A job can also be created manually via the API.
The easiest way to get started is to create a configuration of orchestration or flow and run it manually in the UI.
Once you're satisfied with the result, look at the successful job:

![Screenshot -- Orchestration Parameters](/automate/orchestration-parameters.png)

In the job detail, you can see the parameters required to run the configuration, in this case:

```
mode: run
component: keboola.orchestrator
config: 1496488
```

The difference between running an arbitrary [component job](/automate/run-job/) and an orchestrator job is only in
that the `component` value is always `keboola.orchestrator`. The same component is used for both Orchestrator
and Flow jobs.

Create a [Storage API token](/management/project/tokens/) which you will use to
run the API requests (if you don't have one already). We recommend to create a token with **Full Access** to
components. Though you can list the components used in the orchestration, this leads to a fragile setup when
modifying the orchestration may need the modification of the the token too.

Then use the [Create Job API call](https://api.keboola.com/?service=job-queue#job-queue/tag/jobs/POST/jobs) to
create a job with the same parameters
(see [example](https://documenter.getpostman.com/view/3086797/77h845D#3e71b131-afd4-44be-9831-6534e581f2e0)):

```bash
curl --location --request POST 'https://queue.keboola.com/jobs' \
--header 'X-StorageApi-Token: YOUR_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"mode": "run",
"component": "keboola.orchestrator",
"config": "1496488"
}'
```

Take care to use the right endpoint depending on which [Stack](/overview/#stacks) are you using.
You'll see `Invalid access token` error message if you are using the wrong endpoint or token. Read more about
the concept of [Jobs](/integrate/jobs/).
Loading
Loading