Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/content/docs/ai/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Allows users to get AI suggestions when searching for a component. To activate t
### AI Rules

The AI Rules section allows users to define specific instructions for AI functionality within Keboola platform.
[Learn more about AI Rules →](/management/project/ai-rules/#main-header)
[Learn more about AI Rules →](/management/project/ai-rules/)

### AI-Generated Descriptions
Automatically generates a description using AI.
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/catalog/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Sharing data via the data catalog is useful in numerous ways -- for example:
instead of distributing updates throughout your company, you can create a project with the product table shared to
the rest of the company. Regardless of how and how often the table is updated, this ensures that everyone always looks at the
same data.
- You might want to use [**Multi-project architecture**](#multi-project-architecture).
- You might want to use [**Multi-project architecture**](/catalog/multi-project/).

The following terminology is used:

Expand Down Expand Up @@ -118,7 +118,7 @@ Select the tables and/or buckets you want to share:

![Screenshot -- Select Tables](/catalog/catalog-7.png)

Enter the bucket name and a [sharing type](#sharing-type). Assign a **Datashare Owner**, and optionally customize the bucket color and enter a description:
Enter the bucket name and a [sharing type](#sharing-types). Assign a **Datashare Owner**, and optionally customize the bucket color and enter a description:

![Screenshot -- Select Tables](/catalog/catalog-8.png)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Running the connector creates a background job that
- exports the results from Google Cloud Storage and stores them in specified tables in Keboola Storage.
- removes the results from Google Cloud Storage.

***Note:** Using the Google BigQuery connector is also described in our [Getting Started Tutorial](/tutorial/ad-hoc/#using-bigquery-extractor).*
***Note:** Using the Google BigQuery connector is also described in our [Getting Started Tutorial](/tutorial/ad-hoc/#using-bigquery-connector).*

## Initial Setup

Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/components/extractors/database/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Several variants of connectors may exist for each database type, depending on th
These connectors work on a relational level, performing queries against the source database to synchronize data.
This straightforward approach suits most use cases and supports Timestamp-based CDC replication.

All are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).
All are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/#advanced-mode).

Their basic configuration is also part of the [Tutorial - Loading Data from Database](/tutorial/load/database/).

Expand All @@ -51,7 +51,7 @@ Typically, these connectors are useful in the following scenarios:

Unlike the connectors for SQL databases, connectors for **NoSQL databases** require a different configuration (except
the [BigQuery data source connector](/components/extractors/database/bigquery/) for the [BigQuery](https://cloud.google.com/bigquery/)
database, which is quite similar to SQL databases and also supports the [advanced mode](/components/extractors/database/sqldb/)):
database, which is quite similar to SQL databases and also supports the [advanced mode](/components/extractors/database/sqldb/#advanced-mode)):

- [MongoDB connector](/components/extractors/database/mongodb/) for the [MongoDB](https://www.mongodb.com/) database and the [CosmosDB for MongoDB API](https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction).
- [CosmosDB connector](/components/extractors/database/cosmosdb/) for the [CosmosDB SQL API](https://docs.microsoft.com/en-us/azure/cosmos-db/tutorial-query-sql-api).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This connector supports the most recent versions of both SQL Server and Azure SQ
This [standard SQL database connector](/components/extractors/database/sqldb) performs queries against the source database to synchronize data.
It offers a straightforward approach suitable for most use cases, enabling [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

All SQL database connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).
All SQL database connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/#advanced-mode).

For guidance on basic configuration, please refer to our tutorial: [Loading Data with Database data source connector](/tutorial/load/database/).

Expand Down
10 changes: 5 additions & 5 deletions src/content/docs/components/extractors/database/mysql/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ slug: 'components/extractors/database/mysql'

Our connectors support the latest versions of MySQL and AWS Aurora. You can choose among different strategies to synchronize your data:

- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration)
- [Log-based CDC](/components/extractors/database/mysql#log-based-binlog-cdc)
- [Query-based connector](/components/extractors/database/sqldb/#initial-setup)
- [Log-based CDC](/components/extractors/database/mysql/#mysql-log-based-cdc)

## Query-Based Connector

This [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) performs queries against the source database to sync data.
It is a straightforward approach suitable for most use cases, allowing for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

All connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and
offer an [advanced mode](/components/extractors/database/sqldb/).
All connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and
offer an [advanced mode](/components/extractors/database/sqldb/#advanced-mode).

Basic configuration is covered in the [Tutorial - Loading Data from Database](/tutorial/load/database/).

Expand Down Expand Up @@ -336,7 +336,7 @@ Descriptions of MySQL binlog configuration properties:
| `log_bin` | The value of `log_bin` is the base name of the sequence of binlog files. |
| `binlog_format` | The `binlog-format` must be set to `ROW` or `row`. |
| `binlog_row_image` | The `binlog_row_image` must be set to `FULL` or `full`. |
| `binlog_expire_logs_seconds` | The `binlog_expire_logs_seconds` corresponds to the deprecated system variable `expire_logs_days`. This is the number of seconds for automatic binlog file removal. The default is `2592000`, which equals 30 days. Set the value to match the needs of your environment. See [MySQL purges binlog files](#mysql-purges-binlog-files-used-by-debezium). |
| `binlog_expire_logs_seconds` | The `binlog_expire_logs_seconds` corresponds to the deprecated system variable `expire_logs_days`. This is the number of seconds for automatic binlog file removal. The default is `2592000`, which equals 30 days. Set the value to match the needs of your environment. See [MySQL purges binlog files](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-purges-binlog-files-used-by-debezium). |

#### Enabling GTIDs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ slug: 'components/extractors/database/oracle'
This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data.
It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/).
They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/#advanced-mode).

Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,17 @@ slug: 'components/extractors/database/postgresql'

Our connectors support the most recent versions of PostgreSQL. You may choose different strategies to sync your data:

- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration)
- [Log-based CDC](/components/extractors/database/postgresql/#log-based-cdc)
- [Query-based connector](/components/extractors/database/sqldb/#initial-setup)
- [Log-based CDC](/components/extractors/database/postgresql/#postgresql-log-based-cdc)


## Query-Based Connector

This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data.
It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/).
They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/#advanced-mode).

Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Every job passes through predictable states:
**waiting** → **processing** → **success** / **error**

**How billing relates to job state:**
- Jobs in the **waiting** state are not billed at the job level. A job only consumes [credits](/management/project/limits/#project-power) once it starts **processing**.
- Jobs in the **waiting** state are not billed at the job level. A job only consumes [credits](/management/project/limits/#project-power--time-credits) once it starts **processing**.
- Jobs in the **processing** state are billed based on compute resources consumed.

**Important — container runtime billing:** Some components run inside a container that orchestrates multiple child jobs. In these cases, the parent container may continue running and accumulating runtime costs even while individual child jobs are in the waiting state. Setting very high parallelism in a container-based component does not pause the container while jobs queue — the container remains active throughout.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ files and optionally uploads them to a destination (Tableau Server,
to be used together with [Tableau Desktop](https://www.tableau.com/products/desktop) or
[Tableau Server](https://www.tableau.com/products/server). An alternative approach is to send data using the
**[Snowflake data destination connector](/components/writers/database/snowflake/)** through a
[Keboola-provisioned database](/components/writers/database/snowflake/#using-keboola-provisioned-database). That is more suitable to be used
[Keboola-provisioned database](/components/writers/database/snowflake/#using-keboola-snowflake-database). That is more suitable to be used
together with [Tableau Online](https://www.tableau.com/products/cloud-bi) and also for larger data sets.
Both approaches are interchangeable though.

Expand Down
6 changes: 3 additions & 3 deletions src/content/docs/extend/common-interface/config-file/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ way you wish. Your component should validate the contents of this section. For p
data, use [encryption](/overview/encryption/). This section is not available in Transformations.
- `image_parameters`: See [below](#image-parameters).
- `authorization`: Contains Oauth2 [authorization contents](/extend/common-interface/oauth/) or
[Workspace credentials](/extend/common-interface/folders/#exchanging-data-via-workspace) .
[Workspace credentials](/extend/common-interface/folders/#exchanging-data-via-database-workspace) .
- `action`: Name of the [action](/extend/common-interface/actions/) to execute; defaults to `run`. All
actions except `run` have a strict execution time limit of 30 seconds.
See [actions](/extend/common-interface/actions/) for more details.
Expand Down Expand Up @@ -213,7 +213,7 @@ write the usage file regularly** during the component run, not only at the end.
validated and a wrong format will cause a component failure.*

## Examples
To create an example configuration, use the [Run Job API call in debug mode](/extend/component/running/#preparing-the-data-folder). You will get a
To create an example configuration, use the [Run Job API call in debug mode](/extend/component/running/#preparing-data-folder). You will get a
`stage_0.zip` archive in your **Storage** > **File Uploads**, which will contain the `config.json` file.
You can also use these configuration structure to create an API request for
actually [running a component](https://api.keboola.com/?service=job-queue#post-/jobs).
Expand Down Expand Up @@ -387,7 +387,7 @@ Download 2 days of data from the `in.c-storage.StoredData` table to `/data/table
```

#### Input mapping — column types
This is applicable only to [workspace mapping](/extend/common-interface/folders/#exchanging-data-via-workspace), for CSV files this setting has no effect. The `column_types` setting maps to [Storage API load options](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/workspaces/-workspaceId-/load). It also acts the same way as `columns` setting allowing you to limit the table columns.
This is applicable only to [workspace mapping](/extend/common-interface/folders/#exchanging-data-via-database-workspace), for CSV files this setting has no effect. The `column_types` setting maps to [Storage API load options](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/workspaces/-workspaceId-/load). It also acts the same way as `columns` setting allowing you to limit the table columns.
If both `column_types` and `columns` setting are used, then the listed columns must match. If you omit `columns` and use only `column_types` (recommended) then `columns` will be propagated automatically from `column_types`.

```json
Expand Down
10 changes: 5 additions & 5 deletions src/content/docs/extend/common-interface/folders/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ depends fully on your component code (or Dockerfile). If you want to use a diffe
**use the [`KBC_DATADIR` environment variable](/extend/common-interface/environment/#environment-variables)**. In production,
this variable will always be set to `/data/`. During development, you can set it to your liking.

To create a data folder sample, use the [Debug mode](/extend/component/running/#preparing-the-data-folder) on the
To create a data folder sample, use the [Debug mode](/extend/component/running/#preparing-data-folder) on the
[Create Job API](https://api.keboola.com/?service=job-queue#post-/jobs).
All the resources you need in your component will be provided in a ZIP archive.

Expand Down Expand Up @@ -114,7 +114,7 @@ The following is an example of specifying columns in the manifest file `/data/ou
}

All files from the folder are uploaded irrespective of their name or extension. They are uploaded
to Storage in parallel and in an undefined order. Use sliced tables in case you want to upload tables [larger than 5GB](/storage/file-uploads/#limits). The slices may be compressed by gzip.
to Storage in parallel and in an undefined order. Use sliced tables in case you want to upload tables [larger than 5GB](/storage/files/#limits). The slices may be compressed by gzip.
A rule of thumb is that slices are [best around 10-100 MB](https://docs.snowflake.net/manuals/user-guide/data-load-considerations-prepare.html#splitting-large-data-files-before-loading) in size **compressed**.

### Folder /data/in/files/
Expand All @@ -135,7 +135,7 @@ In this case, the data folders contain only [manifest files](/extend/common-inte
not the actual data. This mode of operation can be enabled by setting the **Staging storage input** option to **AWS S3** in
[component settings](https://components.keboola.com/). If this option is enabled, all the data folders
will contain only manifest files, extended with an additional
[`s3` section](/extend/common-interface/manifest-files/#s3-staging).
[`s3` section](/extend/common-interface/manifest-files/in-files-s3-staging/).

**Note**: Exchanging data via S3 is currently only available for input mapping.

Expand All @@ -145,7 +145,7 @@ In this case, the data folders contain only [manifest files](/extend/common-inte
not the actual data. This mode of operation can be enabled by setting the **Staging storage input** option to **ABS** in
[component settings](https://components.keboola.com/). If this option is enabled, all the data folders
will contain only manifest files, extended with an additional
[`abs` section](/extend/common-interface/manifest-files/#abs-staging).
[`abs` section](/extend/common-interface/manifest-files/in-files-abs-staging/).

**Note**: Exchanging data via ABS is currently only available for input mapping.

Expand Down Expand Up @@ -198,7 +198,7 @@ When exchanging data via workspace, there are couple of differences to loading d
are always saved to the directory structure.
- The `days` attribute is not supported for filtering table, use `changed_since` instead.
- [Automatic Incremental Processing](/storage/tables/#automatic-incremental-processing) (also known as Adaptive Input Mapping) is not supported.
- When used for output mapping, the `columns` of the output table **must be** specified, this can be done either in the [output manifest](/extend/common-interface/manifest-files/#dataouttables-manifests) or in the [output mapping](/extend/common-interface/config-file/#output-mapping--headless-csv).
- When used for output mapping, the `columns` of the output table **must be** specified, this can be done either in the [output manifest](/extend/common-interface/manifest-files/out-tables-manifests/) or in the [output mapping](/extend/common-interface/config-file/#output-mapping--headless-csv).

**Note**: Currently only some combinations of input/output staging storage settings are supported:
`local<->local`, `local<->s3`, `workspace-snowflake<->workspace-snowflake`, `workspace-redshift<->workspace-redshift`.
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/extend/component/code-patterns/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ slug: 'extend/component/code-patterns'

Code Patterns are a special type of [component](/extend/component/). They

- **generate code** for [transformations](/transformations/#new-transformations),
- **generate code** for [transformations](/transformations/),
- implement the [generate action](/extend/component/code-patterns/interface#generate-action), and
- use the [configuration format](/extend/component/code-patterns/interface#configuration-format).
- use the [configuration format](/extend/component/code-patterns/interface/#configuration).

The generated code is written in a specific [output format](/extend/component/code-patterns/interface#output-format).

Expand Down
Loading