Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 12 additions & 0 deletions content/de/developer/integration/ai/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
title: "AI"
description: "Verbinden Sie KI-Plattformen über S3-kompatible Objektspeicher-Schnittstellen mit RustFS."
---

Nutzen Sie **RustFS** als Objektspeicher-Layer für KI- und Machine-Learning-Plattformen, die einen S3-kompatiblen Endpunkt unterstützen.

## Plattformen

- [Ray](./ray.md)

Speichern Sie Trainingsdaten und Checkpoints in dedizierten Buckets und beschränken Sie die Anmeldeinformationen auf die erforderlichen Bucket-Operationen.
6 changes: 6 additions & 0 deletions content/de/developer/integration/ai/meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"title": "AI",
"pages": [
"ray"
]
}
103 changes: 103 additions & 0 deletions content/de/developer/integration/ai/ray.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
title: "Ray"
description: "Use Ray Data with RustFS as S3-compatible storage for dataset writes and reads."
---

This guide connects [Ray](https://github.com/ray-project/ray) — the distributed AI and Python compute framework — to **RustFS** through Ray Data's S3 filesystem support. You will run a Ray job inside the official image, write a dataset as Parquet to a RustFS bucket, read it back, and verify the objects. The workflow was verified with `rayproject/ray:2.44.0-py311` (Ray 2.44, pyarrow filesystem) and `rustfs/rustfs-x86-musl:v2.3.1`.

You need Docker. This deployment is intended for local integration testing, not production.

## Architecture

```mermaid
flowchart LR
Job["Ray job"] -->|"ray.data"| DS["Dataset"]
DS -->|"Parquet files"| RustFS["RustFS :9000"]
```

Ray Data reads and writes datasets through pyarrow's `S3FileSystem`. Passing an `S3FileSystem` configured for RustFS redirects every dataset operation — Parquet, CSV, JSON — to the bucket.

## 1. Create the job file

Create the script, replacing all connection placeholders:

```python title="ray_s3.py"
import ray
ray.init(ignore_reinit_error=True)

import pandas as pd
from pyarrow.fs import S3FileSystem

fs = S3FileSystem(
endpoint_override="http://<your-rustfs-endpoint>:9000",
access_key="<your-access-key>",
secret_key="<your-secret-key>",
region="us-east-1",
)

df = pd.DataFrame({"id": range(5), "value": [x * 1.5 for x in range(5)]})
ds = ray.data.from_pandas(df)
ds.write_parquet("my-bucket/ray-demo/events/", filesystem=fs)

back = ray.data.read_parquet("my-bucket/ray-demo/events/", filesystem=fs).take_all()
print("rows:", len(back))
print("sample:", back[0])
ray.shutdown()
```

`endpoint_override` takes the full endpoint URL including the scheme. pyarrow's `S3FileSystem` uses path-style requests for custom endpoints, so no extra flag is needed. The same filesystem object works for `write_csv`, `read_json`, and the other Ray Data methods.

## 2. Run the job

Run the script in the Ray image on the same Docker network as RustFS:

```bash
docker run --rm --network oo-rustfs_default \
-v "$PWD/ray_s3.py":/tmp/ray_s3.py \
rayproject/ray:2.44.0-py311 python /tmp/ray_s3.py
```

```text
rows: 5
sample: {'id': 0, 'value': 0.0}
```

## 3. Verify objects in RustFS

List the dataset prefix:

```bash
rc ls rustfs/my-bucket/ray-demo/ -r
```

Ray Data wrote the dataset as a Parquet block:

```text
ray-demo/events/0_000000_000000.parquet
```

![Ray dataset files stored in the RustFS Console](./images/rustfs-ray-data.png)

## 4. Stop or reset

Ray Data holds no state of its own. To delete the demo dataset:

```bash
rc rm rustfs/my-bucket/ray-demo/ --recursive --force
```

## Troubleshooting

### `Unable to connect to endpoint` or timeouts

Confirm `endpoint_override` includes the scheme and is reachable from the Ray container. Inside a Compose network the hostname is `rustfs`; from the host use `http://localhost:9000`.

### `Access Denied` on write

Confirm the access key and secret key are passed to `S3FileSystem` itself — Ray does not read the container's AWS environment variables through this code path.

## Next steps

- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Ray operations.
- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token).
- Follow the [Ray Data documentation](https://docs.ray.io/en/latest/data/data.html) to chain transformations, training ingestion, and checkpointing on the same bucket.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions content/de/developer/integration/backup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ Verwenden Sie **RustFS** als Objektspeicher-Backend für Backup-Tools, die Repos
## Systeme

- [Restic](./restic.md)
- [Velero](./velero.md)
- [Kopia](./kopia.md)
- [Longhorn](./longhorn.md)

Halten Sie Backup-Jobs in einem eigenen Bucket und Präfix und verwenden Sie Anmeldedaten, die auf die erforderlichen Bucket-Operationen beschränkt sind.
125 changes: 125 additions & 0 deletions content/de/developer/integration/backup/kopia.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
---
title: "Kopia"
description: "Back up files to RustFS with Kopia's S3 repository backend."
---

This guide connects [Kopia](https://github.com/kopia/kopia) — the open-source backup and restore tool — to **RustFS** as an S3 repository. You will create a repository in a RustFS bucket, take a snapshot of a directory, restore it into an empty directory, and compare checksums. The workflow was verified with `kopia/kopia:0.18.1` and `rustfs/rustfs-x86-musl:v2.3.1`.

You need Docker. This deployment is intended for local integration testing, not production.

## Architecture

```mermaid
flowchart LR
Source["Source files"] -->|snapshot| Kopia["Kopia"]
Kopia -->|"encrypted blocks"| RustFS["RustFS :9000"]
Kopia -->|restore| Restore["Restored files"]
```

Kopia stores the repository format files and deduplicated, encrypted content blocks in the bucket. Restores read the blocks back and reassemble the original files, so the checksum of every restored file must match the source.

## 1. Create the repository

Create the bucket first, then initialize the Kopia repository inside it. The endpoint is a bare `host:port` (no scheme); `--disable-tls` switches the client to plain HTTP:

```bash
rc alias set rustfs http://<your-rustfs-endpoint>:9000 <your-access-key> <your-secret-key>
rc mb rustfs/kopia-backups

docker run --rm --network oo-rustfs_default kopia/kopia:0.18.1 repository create s3 \
--bucket kopia-backups \
--access-key <your-access-key> \
--secret-access-key <your-secret-key> \
--endpoint <your-rustfs-endpoint>:9000 \
--region us-east-1 \
--disable-tls \
--password <your-kopia-password> \
--override-username demo --override-hostname workstation
```

Kopia validates the provider by reading and writing through the S3 API before it reports success.

## 2. Connect, snapshot, and restore

Run the following commands from the directory holding your `repository.config` (created by the previous step). Kopia reads the connection settings from that file, so the S3 flags are only needed once:

```bash
export KOPIA_PASSWORD=<your-kopia-password>
export KOPIA_CONFIG_PATH=/config/repository.config

alias kopia='docker run --rm --network oo-rustfs_default \
-e KOPIA_PASSWORD -e KOPIA_CONFIG_PATH \
-v "$PWD/config:/config" \
-v "$PWD/source:/source:ro" \
-v "$PWD/restore:/restore" kopia/kopia:0.18.1'

kopia repository connect s3 \
--bucket kopia-backups \
--access-key <your-access-key> \
--secret-access-key <your-secret-key> \
--endpoint <your-rustfs-endpoint>:9000 \
--region us-east-1 --disable-tls \
--override-username demo --override-hostname workstation

kopia snapshot create /source
kopia snapshot list
```

Restore the snapshot into an empty directory and compare checksums with the source. The snapshot ID is the `ka...` identifier printed by `snapshot list`:

```bash
kopia restore <snapshot-id> /restore

sha256sum source/blob.bin restore/blob.bin
```

```text
bec4530e2798465b... source/blob.bin
bec4530e2798465b... restore/blob.bin
```

## 3. Verify objects in RustFS

List the bucket:

```bash
rc ls rustfs/kopia-backups/ -r
```

The output shows the repository format files plus the packed content blocks written by the snapshot:

```text
kopia.blobcfg
kopia.repository
p0000.../...
```

![Kopia repository blocks stored in the RustFS Console](./images/rustfs-kopia-repo.png)

## 4. Stop or reset

Kopia is a client-side tool and holds no running state. To delete the repository and all snapshots, remove the bucket:

```bash
rc rb rustfs/kopia-backups --force
```

## Troubleshooting

### `Endpoint url cannot have fully qualified paths`

The endpoint must be a bare `host:port` value without a scheme or path — Kopia builds the object URLs itself.

### `server gave HTTP response to HTTPS client`

Without `--disable-tls`, Kopia speaks HTTPS. RustFS without TLS needs the `--disable-tls` flag on both `repository create s3` and `repository connect s3`.

### `can't connect to storage` with a DNS error

The S3 client is using virtual-hosted addressing. Keep the endpoint as a bare `host:port` value; Kopia uses path-style requests for endpoints given in that form.

## Next steps

- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Kopia operations.
- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token).
- Follow the [Kopia repository documentation](https://kopia.io/docs/repositories/) to add policies, retention, and scheduled snapshots.
4 changes: 3 additions & 1 deletion content/de/developer/integration/backup/meta.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{
"title": "Backup",
"pages": [
"kopia",
"longhorn",
"restic",
"longhorn"
"velero"
]
}
Loading
Loading