Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions en/toc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -168,8 +168,12 @@ items:
- name: Localization
href: tools/docs/translate.md
items:
- name: Machine translation
href: tools/docs/translate-yandex.md
- name: AI translation
href: tools/docs/translate-ai.md
- name: XLIFF exchange with CAT tools
href: tools/docs/translate-xliff.md
- name: Deploying to S3
href: tools/docs/publish-s3.md
- name: Portable CLI build
Expand Down
6 changes: 3 additions & 3 deletions en/tools/docs/translate-ai.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ keywords: ['translate', 'ai', 'llm', 'yandexgpt', 'openai', 'openrouter', 'anthr

The command `{{PROGRAM}} translate` can translate documentation using large language models (LLMs). Supported providers are `yandexgpt`, `openai`, `openrouter`, and `anthropic`.

The pipeline is the same as for [other translation providers](translate.md): text is extracted from the markup, translated, and assembled back. Markdown markup, HTML tags, code, and Liquid constructs do not reach the model — only text segments are translated.
The pipeline is the same as for [other translation methods](translate.md#pipeline): text is extracted from the markup, translated, and assembled back. Markdown markup, HTML tags, code, and Liquid constructs do not reach the model — only text segments are translated.

Here, a provider describes an API protocol, not a specific vendor: any compatible installation (self-hosted model, internal gateway) can be connected with the same provider by [replacing the API address](#custom-api).

Expand Down Expand Up @@ -90,11 +90,11 @@ The request path for each provider is fixed: if the gateway uses a non-standard

## Options reference {#options}

Common command options (`--source`, `--target`, `--files`, `--include`, `--exclude`, `--include-vcs-diff`, `--dry-run`, and others) are described on the [Localization](translate.md) page. The `--target` option can be passed multiple times - translation will be performed into each language. Below are the AI provider options.
Common command options (`--source`, `--target`, `--files`, `--include`, `--exclude`, `--include-vcs-diff`, `--dry-run`, and others) are described on the [Localization](translate.md#options) page. The `--target` option can be passed multiple times - translation will be performed into each language. Below are the AI provider options.

#|
|| **Option** | **Default** | **Description** ||
|| `--provider` | `yandex` | Translation provider. For AI translation: `yandexgpt`, `openai`, `openrouter`, or `anthropic`. The default value `yandex` is machine translation via [Yandex Translate](translate.md#auto), not an LLM ||
|| `--provider` | `yandex` | Translation provider. For AI translation: `yandexgpt`, `openai`, `openrouter`, or `anthropic`. The default value `yandex` is machine translation via [Yandex Translate](translate-yandex.md), not an LLM ||
|| `--auth` | from the environment variable | Token or path to a file with the token. Cannot be placed in the configuration file ||
|| `--model` | depends on the provider | Model identifier ||
|| `--fallback-model` | - | Fallback model in the same format as `--model`. See [Fallback model](#fallback) ||
Expand Down
105 changes: 105 additions & 0 deletions en/tools/docs/translate-xliff.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
---
keywords: ['translate', 'xliff', 'cat', 'extract', 'compose', 'trados', 'smartcat', 'crowdin']
---
# XLIFF exchange with CAT tools

When translation is done by people - in-house translators or an agency - they usually work in a Computer Assisted Translation (CAT) tool: Trados, Phrase, Smartcat, Crowdin, and the like. The standard exchange format for such tools is [XLIFF](https://en.wikipedia.org/wiki/XLIFF).

The `extract` and `compose` subcommands of the `{{PROGRAM}} translate` command implement the full cycle of such translation:

1. `extract` exports the translatable project text into `*.xliff` files.
2. The files are translated in a CAT tool.
3. `compose` assembles the translated `*.xliff` back into documentation files.

## How it works {#how-it-works}

`extract` splits each documentation file into two parts:

* `<file>.xliff` - translatable segments: sentences, headings, table cells;
* `<file>.skl` - the skeleton: the source file with markers in place of the segments.

Markup, code, and Liquid constructs stay in the skeleton and never reach the CAT tool - see [How translation works](translate.md#pipeline) for details.

Both files are saved under the target language path. For example, when translating from `ru` into `en`, the file `ru/guide/index.md` produces `en/guide/index.md.xliff` and `en/guide/index.md.skl`.

`compose` performs the reverse operation: it finds `.xliff` + `.skl` pairs in a directory and assembles a translated file from each - `en/guide/index.md`. Files without a pair are skipped with a warning.

## Full cycle example {#example}

```bash
# Export segments: en/**/*.xliff and en/**/*.skl appear in ./xliff
{{PROGRAM}} translate extract -i ./docs -o ./xliff --source ru --target en

# ...translate *.xliff in a CAT tool...

# Assemble translated files into ./docs/en
{{PROGRAM}} translate compose -i ./xliff -o ./docs
```

Only the `*.xliff` files are handed over to the CAT tool, but during assembly the translated `*.xliff` must sit next to their `*.skl` - don't delete the skeletons between steps.

After `compose`, the translated version is built with a regular `{{PROGRAM}} build`.

## XLIFF format {#format}

`extract` produces XLIFF version 1.2. Each segment is a `<trans-unit>` element with the source text in `<source>`. The translation must go into the `<target>` element - CAT tools add it themselves:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<xliff xmlns="urn:oasis:names:tc:xliff:document:1.2" version="1.2">
<file original="file.ext" source-language="ru-RU" target-language="en-US" datatype="markdown">
<header>
<skeleton>
<external-file href="file.skl"></external-file>
</skeleton>
</header>
<body>
<trans-unit id="1">
<source xml:space="preserve" xml:lang="ru-RU">Document title</source>
</trans-unit>
</body>
</file>
</xliff>
```

Inline markup inside a segment - emphasis, links, code - is encoded with the auxiliary `<g>` and `<x/>` tags. They must be preserved during translation: `compose` uses them to restore the original markup.

## extract parameters {#extract}

#|
|| **Parameter** | **Description** ||
|| `--source`, `-sl` |
Source language in ISO 639-1 format: `ru` or `ru-RU`. Required
||
|| `--target`, `-tl` |
Target language: `en` or `en-US`. Can be passed multiple times - the export is performed for each language
||
|| `--filter` |
Export only files reachable from `toc.yaml`. By default, all project files are exported
||
|| `--schema` |
Paths to files with custom [translation schemas](translate.md#json-schemas) for YAML and JSON. Several paths can be specified
||
|| `--no-ref-resolve` |
Do not resolve `$ref` in OpenAPI specifications during export
||
|#

The common parameters `--input`, `--output`, `--files`, `--include`, and `--exclude` are also supported - see [Localization](translate.md#options).

## compose parameters {#compose}

#|
|| **Parameter** | **Description** ||
|| `--input`, `-i` |
Directory with `*.xliff` + `*.skl` pairs. Defaults to the directory the command is run from
||
|| `--output`, `-o` |
Path to the project **root** where the assembled files should be saved. Defaults to `input`
||
|| `--use-source` |
Assemble files from the source text (`<source>`) instead of the translation. Useful for debugging the export
||
|#

The `--include` and `--exclude` parameters filter file pairs the same way as during translation - see [Localization](translate.md#options).
49 changes: 49 additions & 0 deletions en/tools/docs/translate-yandex.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
keywords: ['translate', 'yandex translate', 'machine translation', 'i18n', 'l10n']
---
# Machine translation

Without the `--provider` option, the `{{PROGRAM}} translate` command translates documentation via [Yandex Translate](https://yandex.cloud/en/services/translate). This is the fastest translation method: it suits draft versions and regular language synchronization, but the result usually needs proofreading. For higher quality, use [AI translation](translate-ai.md) or [CAT tools](translate-xliff.md).

## Usage {#usage}

1. Get an authorization token: an [OAuth token](https://yandex.cloud/en/docs/iam/concepts/authorization/oauth-token), an [IAM token](https://yandex.cloud/en/docs/iam/concepts/authorization/iam-token), or a service account [API key](https://yandex.cloud/en/docs/iam/operations/api-key/create).
2. Find out the [folder ID](https://yandex.cloud/en/docs/resource-manager/operations/folder/get-id) for which your account has the `ai.translate.user` role or higher.
3. Estimate the translation volume without API requests:

```bash
{{PROGRAM}} translate -i ./docs --source ru --target en --auth <token> --folder <folder-id> --dry-run
```

4. Run the translation:

```bash
{{PROGRAM}} translate -i ./docs --source ru --target en --auth <token> --folder <folder-id>
```

Translated files appear in the target language folder - `docs/en` in the example above.

## Parameters {#options}

Common parameters (`--source`, `--target`, `--files`, `--include`, `--exclude`, `--dry-run`, and others) are described on the [Localization](translate.md#options) page. Below are the `yandex` provider parameters.

#|
|| **Parameter** | **Description** ||
|| `--auth` |
Authorization token: a value or a path to a file with the token. The type is detected by prefix: `y0_` - OAuth token, `t1.` - IAM token, `AQVN` - service account API key. Required
||
|| `--folder` |
[Folder ID](https://yandex.cloud/en/docs/resource-manager/operations/folder/get-id) for which your account has the `ai.translate.user` role or higher. Required
||
|| `--glossary` |
Path to a YAML file with a [glossary](https://yandex.cloud/en/docs/translate/concepts/glossary) - pairs of terms that must be translated in a fixed way
||
|#

## Limits {#limits}

Yandex Translate has [limits](https://yandex.cloud/en/docs/translate/concepts/limits) on the amount of translated text. The CLI reduces the volume on its own: documents are split into segments, and repeated segments are translated once.

If a limit is still exceeded, the command fails with the `TRANSLATE_LIMIT_EXCEED` error. In that case, retry later or narrow the file set with [filters](translate.md#options) - already translated files can be excluded.

The `--dry-run` option helps estimate the text volume before running.
Loading