diff --git a/content/de/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/de/developer/integration/big-data/images/rustfs-vitess-backups.png new file mode 100644 index 00000000..5a939c8d Binary files /dev/null and b/content/de/developer/integration/big-data/images/rustfs-vitess-backups.png differ diff --git a/content/de/developer/integration/big-data/index.md b/content/de/developer/integration/big-data/index.md index b5850b9f..5931816f 100644 --- a/content/de/developer/integration/big-data/index.md +++ b/content/de/developer/integration/big-data/index.md @@ -21,5 +21,6 @@ Use **RustFS** as the object storage layer for data analytics systems that suppo - [Spark](./spark.md) - [Flink](./flink.md) - [Trino](./trino.md) +- [Vitess](./vitess.md) Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file diff --git a/content/de/developer/integration/big-data/meta.json b/content/de/developer/integration/big-data/meta.json index 249e219d..29684232 100644 --- a/content/de/developer/integration/big-data/meta.json +++ b/content/de/developer/integration/big-data/meta.json @@ -15,6 +15,7 @@ "pyiceberg", "spark", "trino", + "vitess", "zeppelin" ] } diff --git a/content/de/developer/integration/big-data/vitess.md b/content/de/developer/integration/big-data/vitess.md new file mode 100644 index 00000000..2260b744 --- /dev/null +++ b/content/de/developer/integration/big-data/vitess.md @@ -0,0 +1,189 @@ +--- +title: "Vitess" +description: "Back up Vitess to RustFS through the S3 backup storage implementation." +--- + +This guide connects [Vitess](https://github.com/vitessio/vitess) — the database clustering system for horizontal scaling of MySQL — to **RustFS** through Vitess's S3 backup storage implementation. You will bring up a minimal Vitess topology on a single host, run `Backup` on a tablet, and confirm the backup chunks and `MANIFEST` in the bucket, then read the backup list back from RustFS. The workflow was verified with `vitess/lite` (Vitess v25.0.0-SNAPSHOT, built 2026-09-22), MySQL 8.4, and `rustfs/rustfs-x86-musl:v2.3.1`; stable Vitess 20+ releases use the same flags. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["vtctldclient"] -->|"Backup"| Tablet["vttablet"] + Tablet -->|"mysqld dump"| Chunks["chunks + MANIFEST"] + Tablet -->|"S3 API"| RustFS["RustFS :9000"] +``` + +`vtctld` triggers the backup, but the `vttablet` owning the tablet performs it: it snapshots MySQL, compresses the data into numbered chunks, and uploads them together with a `MANIFEST` through the S3-compatible endpoint. + +## 1. Run etcd and the Vitess image + +Vitess stores its topology in etcd, and the `vitess/lite` image ships all Vitess binaries plus MySQL: + +```bash +docker run -d --name etcd --hostname etcd --network oo-rustfs_default \ + quay.io/coreos/etcd:v3.5.17 etcd \ + --advertise-client-urls http://etcd:2379 \ + --listen-client-urls http://0.0.0.0:2379 + +docker run -d --name vtess --hostname vtess --network oo-rustfs_default \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + vitess/lite:latest sleep infinity +``` + +The AWS environment variables supply the credentials for the S3 backup storage in both `vtctld` and `vttablet`. + +## 2. Initialize MySQL + +Initialize a MySQL instance in the standard Vitess tablet directory so `vttablet` can load its `my.cnf` later: + +```bash +docker exec vtess sh -c \ + "/vt/bin/mysqlctl --log_dir /tmp/vtlogs init --tablet-dir vt_0000000100" +``` + +The instance is ready when the socket file exists: + +```text +/vt/vtdataroot/vt_0000000100/mysql.sock +``` + +## 3. Start vtctld and vttablet + +Start `vtctld` with the S3 backup flags, register the cell, and start the tablet. Replace all connection placeholders: + +```bash +docker exec vtess sh -c "nohup /vt/bin/vtctld \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --service-map grpc-vtctl,grpc-vtctld \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --port 15999 --grpc-port 15998 \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vtctld.out 2>&1 &" + +docker exec vtess sh -c \ + "/vt/bin/vtctldclient --server localhost:15998 \ + AddCellInfo --root /vitess/zone1 --server-address etcd:2379 zone1" + +docker exec vtess sh -c "nohup /vt/bin/vttablet \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --tablet-path zone1-0000000100 \ + --init-keyspace commerce --init-shard 0 --init-tablet-type replica \ + --port 15100 --grpc-port 15101 \ + --service-map grpc-queryservice,grpc-tabletmanager,grpc-throttler \ + --mycnf-file /vt/vtdataroot/vt_0000000100/my.cnf \ + --db-dba-user root --db-allprivs-user root --db-app-user root --db-repl-user root \ + --db-dba-use-ssl=false --db-allprivs-use-ssl=false \ + --db-app-use-ssl=false --db-repl-use-ssl=false \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vttablet.out 2>&1 &" +``` + +Both processes carry the S3 flags because `vttablet` performs the backup while `vtctld` lists and removes backups. `--s3-backup-force-path-style` is mandatory for non-AWS endpoints. Check that the tablet registered: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetTablets +``` + +```text +zone1-0000000100 commerce 0 replica vtess:15100 vtess:3306 [] +``` + +## 4. Run the backup + +Create the bucket and trigger the backup: + +```bash +rc mb rustfs/vitess-backups + +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 \ + Backup zone1-0000000100 +``` + +The stream ends with the engine writing the manifest: + +```text +commerce/0 (zone1-0000000100): ... value:"Completed backing up MANIFEST (attempt 1/2)" +``` + +## 5. Verify objects in RustFS + +List the backup prefix: + +```bash +rc ls rustfs/vitess-backups/ -r | head -4 +``` + +The tablet uploaded compressed data chunks plus the metadata files: + +```text +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/0 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/1 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/MANIFEST +``` + +![Vitess backup files stored in the RustFS Console](./images/rustfs-vitess-backups.png) + +Read the backup list back from RustFS through `vtctld`: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetBackups commerce/0 +``` + +```text +2026-09-23.011225.zone1-0000000100 +``` + +## 6. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f vtess etcd +``` + +To delete the stored backups: + +```bash +rc rm rustfs/vitess-backups/ --recursive --force +``` + +## Troubleshooting + +### `cannot perform backup without my.cnf` + +Passing connection parameters such as `--db-socket` or `--db-host` makes `vttablet` skip loading `my.cnf`, and backups refuse to run. Start `vttablet` with `--mycnf-file` pointing at the instance config and let it discover the socket from there — this is why the MySQL instance lives in the `vt_0000000100` directory from step 2. + +### `unknown service vtctlservice.Vtctld` + +`vtctld` exposes the API used by `vtctldclient` only when the service map includes it: `--service-map grpc-vtctl,grpc-vtctld`. Also make sure `vtctldclient --server` points at the gRPC port (`15998` here), not the web UI port. + +### `node doesn't exist: /vitess/global/cells/zone1/CellInfo` + +The cell must exist before tablets register. Run `vtctldclient AddCellInfo` as shown in step 3 before starting `vttablet`. + +### Flag parse errors such as `unknown shorthand flag` + +Vitess 20+ normalizes underscores to dashes, so write flags in dash style (`--tablet-path`). A single-dash long flag like `-tablet_dir` is parsed as shorthand options and fails. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Vitess storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Vitess backup and restore documentation](https://vitess.io/docs/user-guides/configuration-basic/#backups) to schedule backups and restore tablets from the bucket. diff --git a/content/de/developer/integration/cloud-native/cortex.md b/content/de/developer/integration/cloud-native/cortex.md new file mode 100644 index 00000000..3db19b47 --- /dev/null +++ b/content/de/developer/integration/cloud-native/cortex.md @@ -0,0 +1,218 @@ +--- +title: "Cortex" +description: "Run Cortex with RustFS as the S3-compatible blocks, ruler, and alertmanager storage backend." +--- + +This guide connects [Cortex](https://github.com/cortexproject/cortex) — the horizontally scalable Prometheus-compatible metrics backend — to **RustFS** through Cortex's native S3 bucket storage. You will run Cortex in single-binary mode with blocks, ruler, and alertmanager storage pointed at a RustFS bucket, push metrics with remote write, store a rule group, and verify the objects. The workflow was verified with `cortexproject/cortex:v1.21.1` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Prom["Prometheus"] -->|"remote write"| Cortex["Cortex :9009"] + Cortex -->|"blocks, rules, configs"| RustFS["RustFS :9000"] +``` + +Cortex stores every TSDB block, ruler rule group, and alertmanager configuration in object storage. Pointing the `s3` backend at RustFS makes the bucket the single source of truth for all tenant data. + +## 1. Create the Cortex config + +Create the config file, replacing all connection placeholders: + +```yaml title="cortex.yaml" +target: all +auth_enabled: false + +server: + http_listen_port: 9009 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester: + lifecycler: + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +blocks_storage: + backend: s3 + s3: &s3 + endpoint: :9000 + region: us-east-1 + bucket_name: cortex + access_key_id: + secret_access_key: + insecure: true + bucket_lookup_type: path + tsdb: + dir: /data/tsdb + block_ranges_period: [15m] + ship_interval: 30s + bucket_store: + sync_dir: /data/tsdb-sync + bucket_index: + enabled: true + +compactor: + data_dir: /data/compactor + sharding_ring: + kvstore: + store: inmemory + +ruler: + enable_api: true + rule_path: /data/ruler + +ruler_storage: + backend: s3 + s3: + <<: *s3 + +alertmanager: + external_url: /alertmanager + enable_api: true + data_dir: /data/alertmanager + +alertmanager_storage: + backend: s3 + s3: + <<: *s3 +``` + +Cortex names the bucket field `bucket_name` and controls addressing with `bucket_lookup_type: path`. `insecure: true` allows the plain-HTTP endpoint. The `block_ranges_period` and `ship_interval` values shorten the block cycle so the test produces objects quickly; keep the defaults in production. + +## 2. Run Cortex + +Create the bucket and start Cortex on the same Docker network as RustFS: + +```bash +rc mb rustfs/cortex + +docker run -d --name cortex --network oo-rustfs_default -p 9009:9009 \ + -v "$PWD/cortex.yaml":/etc/cortex/cortex.yaml:ro \ + -v /opt/cortex/data:/data \ + cortexproject/cortex:v1.21.1 -config.file=/etc/cortex/cortex.yaml +``` + +```text +ts=... caller=cortex.go:469 level=info msg="Cortex started" +``` + +## 3. Push metrics and rules + +Start a Prometheus that scrapes itself and remote-writes into Cortex: + +```yaml title="prometheus.yml" +global: + scrape_interval: 5s +scrape_configs: + - job_name: self + static_configs: + - targets: ["localhost:9090"] +remote_write: + - url: http://cortex:9009/api/v1/push +``` + +```bash +docker run -d --name prom-writer --network oo-rustfs_default \ + -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml:ro \ + prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml +``` + +Store a rule group through the ruler API: + +```bash +cat > rules.yaml << "EOF" +name: rustfs-demo +rules: + - alert: RustFSAlwaysFiring + expr: vector(1) + labels: + severity: demo + annotations: + summary: "demo alert stored in RustFS" +EOF + +curl -s -o /dev/null -w "%{http_code}\n" \ + -X POST http://localhost:9009/api/v1/rules/rustfs-demo \ + --data-binary @rules.yaml -H "Content-Type: application/yaml" +``` + +```text +202 +``` + +Query the ingested series back: + +```bash +curl -s "http://localhost:9009/prometheus/api/v1/query?query=up" +``` + +```text +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up",...},"value":[...,"1"]}]}} +``` + +## 4. Verify objects in RustFS + +List the bucket prefixes: + +```bash +rc ls rustfs/cortex/ -r +``` + +The ruler config landed immediately, and the ingester shipped two TSDB blocks (each with `chunks`, `index`, and `meta.json`): + +```text +fake/01M35WE4B1G6RRKHRV0HSGADY1/chunks/000001 +fake/01M35WE4B1G6RRKHRV0HSGADY1/index +fake/01M35WE4B1G6RRKHRV0HSGADY1/meta.json +fake/01M35WWS2YCC7FBK1JSBKFQGRP/chunks/000001 +fake/01M35WWS2YCC7FBK1JSBKFQGRP/index +fake/01M35WWS2YCC7FBK1JSBKFQGRP/meta.json +rules/fake/cnVzdGZzLWRlbW8=/cnVzdGZzLWRlbW8= +``` + +![Cortex blocks and rules stored in the RustFS Console](./images/rustfs-cortex-blocks.png) + +## 5. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f cortex prom-writer +``` + +To delete the stored data: + +```bash +rc rm rustfs/cortex/ --recursive --force +``` + +## Troubleshooting + +### `field bucket not found in type s3.Config` + +Cortex uses `bucket_name`, not `bucket`. Path-style addressing is set with `bucket_lookup_type: path` — the `force_path_style` key used by Thanos does not exist in Cortex. + +### Blocks never appear in the bucket + +The ingester only ships a block after the head is compacted at a `block_ranges_period` boundary (default 2h). For tests, set a small range such as `[15m]` together with `ship_interval: 30s` and wait for one period. + +### `Unauthorized` or empty bucket listings + +Confirm `access_key_id` and `secret_access_key` are present in every `s3` block that uses the bucket, and that `insecure: true` matches the plain-HTTP endpoint. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Cortex storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Combine Cortex with [Thanos](/developer/integration/observability/thanos) when you need block storage for long-term Prometheus data as well. diff --git a/content/de/developer/integration/cloud-native/flux.md b/content/de/developer/integration/cloud-native/flux.md new file mode 100644 index 00000000..0319cc85 --- /dev/null +++ b/content/de/developer/integration/cloud-native/flux.md @@ -0,0 +1,157 @@ +--- +title: "Flux" +description: "Deliver Kubernetes manifests to Flux CD from RustFS through the Bucket source API." +--- + +This guide connects [Flux CD](https://github.com/fluxcd/flux2) — the GitOps toolkit for Kubernetes — to **RustFS** through the source-controller Bucket API. You will seed a RustFS bucket with Kubernetes manifests, register the bucket as a Flux `Bucket` source with the `generic` provider, and let a Kustomization apply everything it contains. The workflow was verified with `flux v2.9.5` (source-controller) on `k3s v1.36.4+k3s1` against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a Kubernetes cluster with Flux installed (`flux install`) and the `rc` client. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Manifests["Kustomize manifests"] -->|"rc cp"| Bucket["RustFS :9000"] + source-controller -->|"S3 API"| Bucket + source-controller -->|"artifact"| kustomize-controller + kustomize-controller -->|"apply"| Cluster["Kubernetes cluster"] +``` + +The source-controller lists and downloads objects from the bucket over the S3 API, packs them into an artifact, and the kustomize-controller applies the manifests from that artifact. RustFS acts as the pull-based source of truth for the cluster. + +## 1. Seed the bucket + +Store a Kustomize overlay in the bucket, replacing all connection placeholders: + +```yaml title="clusters/staging/kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - rustfs-demo-configmap.yaml +``` + +```yaml title="clusters/staging/rustfs-demo-configmap.yaml" +apiVersion: v1 +kind: ConfigMap +metadata: + name: rustfs-flux-demo + namespace: default +data: + storage: rustfs + source: s3-bucket +``` + +Upload the files: + +```bash +rc mb rustfs/flux-src +rc cp --recursive ./clusters rustfs/flux-src/ +rc ls rustfs/flux-src/ -r +``` + +```text +staging/kustomization.yaml +staging/rustfs-demo-configmap.yaml +``` + +## 2. Create the credentials secret + +Flux reads the credentials from a secret in the same namespace as the source, using lowercase keys: + +```bash +kubectl -n flux-system create secret generic rustfs-creds \ + --from-literal=accesskey= \ + --from-literal=secretkey= +``` + +The field names must be `accesskey` and `secretkey` — capitalization such as `accessKey` fails with an `AuthenticationFailed` status. + +## 3. Register the Bucket source + +Create the Bucket source with the `generic` provider: + +```bash +flux create source bucket rustfs-demo \ + --bucket-name=flux-src \ + --endpoint=:9000 \ + --insecure \ + --secret-ref=rustfs-creds \ + --provider=generic \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ Bucket source reconciliation completed +``` + +If reconciliation is still running, check the status with `flux get sources bucket`: + +```text +NAME REVISION SUSPENDED READY MESSAGE +rustfs-demo sha256:481816d1 False True stored artifact: revision 'sha256:481816d1' +``` + +The `generic` provider uses path-style requests and plain HTTP with `--insecure`, so the endpoint is the bare `host:port`. + +## 4. Apply the manifests + +Create a Kustomization that consumes the artifact: + +```bash +flux create kustomization rustfs-demo \ + --source=Bucket/rustfs-demo \ + --path="./staging" \ + --prune=true \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ applied revision sha256:481816d1 +``` + +Confirm the manifest landed on the cluster: + +```bash +kubectl -n default get configmap rustfs-flux-demo -o jsonpath="{.data}" +``` + +```text +{"source":"s3-bucket","storage":"rustfs"} +``` + +## 5. Stop or reset + +Suspend or delete the Flux resources without touching the bucket: + +```bash +flux suspend source bucket rustfs-demo --namespace=flux-system +flux delete kustomization rustfs-demo --namespace=flux-system +``` + +To delete the bucket contents: + +```bash +rc rm rustfs/flux-src/ --recursive --force +``` + +## Troubleshooting + +### `invalid 'rustfs-creds' secret data: required fields 'accesskey' and 'secretkey'` + +The secret keys are lowercase. Recreate the secret with `accesskey` and `secretkey` as literal key names. + +### Reconciliation never becomes ready + +Confirm the endpoint is reachable from inside the cluster — pods cannot use `localhost`, so point the source at the host IP or the Docker bridge gateway (commonly `172.17.0.1`) that publishes the RustFS port. `--insecure` is required for plain HTTP. + +### `AuthenticationFailed` despite valid credentials + +Check that the secret lives in the same namespace as the Bucket source and that the keys have no trailing whitespace or quoting. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional source types. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Flux Bucket source documentation](https://fluxcd.io/flux/components/source/buckets/) to combine bucket sources with Helm releases and image automation. diff --git a/content/de/developer/integration/cloud-native/images/rustfs-cortex-blocks.png b/content/de/developer/integration/cloud-native/images/rustfs-cortex-blocks.png new file mode 100644 index 00000000..70ae5240 Binary files /dev/null and b/content/de/developer/integration/cloud-native/images/rustfs-cortex-blocks.png differ diff --git a/content/de/developer/integration/cloud-native/images/rustfs-flux-bucket.png b/content/de/developer/integration/cloud-native/images/rustfs-flux-bucket.png new file mode 100644 index 00000000..9336ea08 Binary files /dev/null and b/content/de/developer/integration/cloud-native/images/rustfs-flux-bucket.png differ diff --git a/content/de/developer/integration/cloud-native/index.md b/content/de/developer/integration/cloud-native/index.md new file mode 100644 index 00000000..1b514aea --- /dev/null +++ b/content/de/developer/integration/cloud-native/index.md @@ -0,0 +1,13 @@ +--- +title: "Cloud Native" +description: "Connect cloud native platforms to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for cloud native platforms that support an S3-compatible endpoint. + +## Platforms + +- [Cortex](./cortex.md) +- [Flux](./flux.md) + +Keep platform state in a dedicated bucket, and use credentials scoped to the required bucket operations. diff --git a/content/de/developer/integration/cloud-native/meta.json b/content/de/developer/integration/cloud-native/meta.json new file mode 100644 index 00000000..30b84775 --- /dev/null +++ b/content/de/developer/integration/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Cloud Native", + "pages": [ + "cortex", + "flux" + ] +} diff --git a/content/de/developer/integration/index.md b/content/de/developer/integration/index.md index a7e68a04..b4d9e222 100644 --- a/content/de/developer/integration/index.md +++ b/content/de/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Integrieren Sie RustFS mit Reverse Proxies, Backup-Tools, Datenanalyse-Systemen, Observability-Plattformen, Container-Registries und DevOps-Tooling." +description: "Integrieren Sie RustFS mit Reverse Proxies, Backup-Tools, Datenanalyse-Systemen, KI- und Cloud-Native-Plattformen, Observability-Plattformen, Container-Registries und DevOps-Tooling." --- Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. @@ -10,7 +10,8 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Reverse Proxy](./reverse-proxy/index.md) covers Nginx, Traefik, Caddy, and HAProxy. - [Backup](./backup/index.md) covers Kopia, Longhorn, Restic, and Velero. - [AI](./ai/index.md) covers AI platforms including Ray. -- [Datenanalyse](./big-data/index.md) covers analytics systems including ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL, and Zeppelin. +- [Datenanalyse](./big-data/index.md) covers analytics systems including ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL, Vitess, and Zeppelin. +- [Cloud Native](./cloud-native/index.md) covers Cortex and Flux. - [Observability](./observability/index.md) covers telemetry systems including Fluentd, OpenObserve, OpenTelemetry, Thanos, and Tempo. - [Others](./others/index.md) covers the community-driven capo SDK for Python. - [Registry](./registry/index.md) covers Harbor. diff --git a/content/de/developer/integration/meta.json b/content/de/developer/integration/meta.json index a3c989a5..f5a8d87c 100644 --- a/content/de/developer/integration/meta.json +++ b/content/de/developer/integration/meta.json @@ -5,6 +5,7 @@ "backup", "big-data", "ai", + "cloud-native", "observability", "others", "registry", diff --git a/content/en/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/en/developer/integration/big-data/images/rustfs-vitess-backups.png new file mode 100644 index 00000000..5a939c8d Binary files /dev/null and b/content/en/developer/integration/big-data/images/rustfs-vitess-backups.png differ diff --git a/content/en/developer/integration/big-data/index.md b/content/en/developer/integration/big-data/index.md index ba66ba0d..78651eea 100644 --- a/content/en/developer/integration/big-data/index.md +++ b/content/en/developer/integration/big-data/index.md @@ -21,5 +21,6 @@ Use **RustFS** as the object storage layer for data analytics systems that suppo - [Spark](./spark.md) - [Flink](./flink.md) - [Trino](./trino.md) +- [Vitess](./vitess.md) Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file diff --git a/content/en/developer/integration/big-data/meta.json b/content/en/developer/integration/big-data/meta.json index 249e219d..29684232 100644 --- a/content/en/developer/integration/big-data/meta.json +++ b/content/en/developer/integration/big-data/meta.json @@ -15,6 +15,7 @@ "pyiceberg", "spark", "trino", + "vitess", "zeppelin" ] } diff --git a/content/en/developer/integration/big-data/vitess.md b/content/en/developer/integration/big-data/vitess.md new file mode 100644 index 00000000..2260b744 --- /dev/null +++ b/content/en/developer/integration/big-data/vitess.md @@ -0,0 +1,189 @@ +--- +title: "Vitess" +description: "Back up Vitess to RustFS through the S3 backup storage implementation." +--- + +This guide connects [Vitess](https://github.com/vitessio/vitess) — the database clustering system for horizontal scaling of MySQL — to **RustFS** through Vitess's S3 backup storage implementation. You will bring up a minimal Vitess topology on a single host, run `Backup` on a tablet, and confirm the backup chunks and `MANIFEST` in the bucket, then read the backup list back from RustFS. The workflow was verified with `vitess/lite` (Vitess v25.0.0-SNAPSHOT, built 2026-09-22), MySQL 8.4, and `rustfs/rustfs-x86-musl:v2.3.1`; stable Vitess 20+ releases use the same flags. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["vtctldclient"] -->|"Backup"| Tablet["vttablet"] + Tablet -->|"mysqld dump"| Chunks["chunks + MANIFEST"] + Tablet -->|"S3 API"| RustFS["RustFS :9000"] +``` + +`vtctld` triggers the backup, but the `vttablet` owning the tablet performs it: it snapshots MySQL, compresses the data into numbered chunks, and uploads them together with a `MANIFEST` through the S3-compatible endpoint. + +## 1. Run etcd and the Vitess image + +Vitess stores its topology in etcd, and the `vitess/lite` image ships all Vitess binaries plus MySQL: + +```bash +docker run -d --name etcd --hostname etcd --network oo-rustfs_default \ + quay.io/coreos/etcd:v3.5.17 etcd \ + --advertise-client-urls http://etcd:2379 \ + --listen-client-urls http://0.0.0.0:2379 + +docker run -d --name vtess --hostname vtess --network oo-rustfs_default \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + vitess/lite:latest sleep infinity +``` + +The AWS environment variables supply the credentials for the S3 backup storage in both `vtctld` and `vttablet`. + +## 2. Initialize MySQL + +Initialize a MySQL instance in the standard Vitess tablet directory so `vttablet` can load its `my.cnf` later: + +```bash +docker exec vtess sh -c \ + "/vt/bin/mysqlctl --log_dir /tmp/vtlogs init --tablet-dir vt_0000000100" +``` + +The instance is ready when the socket file exists: + +```text +/vt/vtdataroot/vt_0000000100/mysql.sock +``` + +## 3. Start vtctld and vttablet + +Start `vtctld` with the S3 backup flags, register the cell, and start the tablet. Replace all connection placeholders: + +```bash +docker exec vtess sh -c "nohup /vt/bin/vtctld \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --service-map grpc-vtctl,grpc-vtctld \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --port 15999 --grpc-port 15998 \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vtctld.out 2>&1 &" + +docker exec vtess sh -c \ + "/vt/bin/vtctldclient --server localhost:15998 \ + AddCellInfo --root /vitess/zone1 --server-address etcd:2379 zone1" + +docker exec vtess sh -c "nohup /vt/bin/vttablet \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --tablet-path zone1-0000000100 \ + --init-keyspace commerce --init-shard 0 --init-tablet-type replica \ + --port 15100 --grpc-port 15101 \ + --service-map grpc-queryservice,grpc-tabletmanager,grpc-throttler \ + --mycnf-file /vt/vtdataroot/vt_0000000100/my.cnf \ + --db-dba-user root --db-allprivs-user root --db-app-user root --db-repl-user root \ + --db-dba-use-ssl=false --db-allprivs-use-ssl=false \ + --db-app-use-ssl=false --db-repl-use-ssl=false \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vttablet.out 2>&1 &" +``` + +Both processes carry the S3 flags because `vttablet` performs the backup while `vtctld` lists and removes backups. `--s3-backup-force-path-style` is mandatory for non-AWS endpoints. Check that the tablet registered: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetTablets +``` + +```text +zone1-0000000100 commerce 0 replica vtess:15100 vtess:3306 [] +``` + +## 4. Run the backup + +Create the bucket and trigger the backup: + +```bash +rc mb rustfs/vitess-backups + +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 \ + Backup zone1-0000000100 +``` + +The stream ends with the engine writing the manifest: + +```text +commerce/0 (zone1-0000000100): ... value:"Completed backing up MANIFEST (attempt 1/2)" +``` + +## 5. Verify objects in RustFS + +List the backup prefix: + +```bash +rc ls rustfs/vitess-backups/ -r | head -4 +``` + +The tablet uploaded compressed data chunks plus the metadata files: + +```text +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/0 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/1 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/MANIFEST +``` + +![Vitess backup files stored in the RustFS Console](./images/rustfs-vitess-backups.png) + +Read the backup list back from RustFS through `vtctld`: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetBackups commerce/0 +``` + +```text +2026-09-23.011225.zone1-0000000100 +``` + +## 6. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f vtess etcd +``` + +To delete the stored backups: + +```bash +rc rm rustfs/vitess-backups/ --recursive --force +``` + +## Troubleshooting + +### `cannot perform backup without my.cnf` + +Passing connection parameters such as `--db-socket` or `--db-host` makes `vttablet` skip loading `my.cnf`, and backups refuse to run. Start `vttablet` with `--mycnf-file` pointing at the instance config and let it discover the socket from there — this is why the MySQL instance lives in the `vt_0000000100` directory from step 2. + +### `unknown service vtctlservice.Vtctld` + +`vtctld` exposes the API used by `vtctldclient` only when the service map includes it: `--service-map grpc-vtctl,grpc-vtctld`. Also make sure `vtctldclient --server` points at the gRPC port (`15998` here), not the web UI port. + +### `node doesn't exist: /vitess/global/cells/zone1/CellInfo` + +The cell must exist before tablets register. Run `vtctldclient AddCellInfo` as shown in step 3 before starting `vttablet`. + +### Flag parse errors such as `unknown shorthand flag` + +Vitess 20+ normalizes underscores to dashes, so write flags in dash style (`--tablet-path`). A single-dash long flag like `-tablet_dir` is parsed as shorthand options and fails. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Vitess storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Vitess backup and restore documentation](https://vitess.io/docs/user-guides/configuration-basic/#backups) to schedule backups and restore tablets from the bucket. diff --git a/content/en/developer/integration/cloud-native/cortex.md b/content/en/developer/integration/cloud-native/cortex.md new file mode 100644 index 00000000..3db19b47 --- /dev/null +++ b/content/en/developer/integration/cloud-native/cortex.md @@ -0,0 +1,218 @@ +--- +title: "Cortex" +description: "Run Cortex with RustFS as the S3-compatible blocks, ruler, and alertmanager storage backend." +--- + +This guide connects [Cortex](https://github.com/cortexproject/cortex) — the horizontally scalable Prometheus-compatible metrics backend — to **RustFS** through Cortex's native S3 bucket storage. You will run Cortex in single-binary mode with blocks, ruler, and alertmanager storage pointed at a RustFS bucket, push metrics with remote write, store a rule group, and verify the objects. The workflow was verified with `cortexproject/cortex:v1.21.1` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Prom["Prometheus"] -->|"remote write"| Cortex["Cortex :9009"] + Cortex -->|"blocks, rules, configs"| RustFS["RustFS :9000"] +``` + +Cortex stores every TSDB block, ruler rule group, and alertmanager configuration in object storage. Pointing the `s3` backend at RustFS makes the bucket the single source of truth for all tenant data. + +## 1. Create the Cortex config + +Create the config file, replacing all connection placeholders: + +```yaml title="cortex.yaml" +target: all +auth_enabled: false + +server: + http_listen_port: 9009 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester: + lifecycler: + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +blocks_storage: + backend: s3 + s3: &s3 + endpoint: :9000 + region: us-east-1 + bucket_name: cortex + access_key_id: + secret_access_key: + insecure: true + bucket_lookup_type: path + tsdb: + dir: /data/tsdb + block_ranges_period: [15m] + ship_interval: 30s + bucket_store: + sync_dir: /data/tsdb-sync + bucket_index: + enabled: true + +compactor: + data_dir: /data/compactor + sharding_ring: + kvstore: + store: inmemory + +ruler: + enable_api: true + rule_path: /data/ruler + +ruler_storage: + backend: s3 + s3: + <<: *s3 + +alertmanager: + external_url: /alertmanager + enable_api: true + data_dir: /data/alertmanager + +alertmanager_storage: + backend: s3 + s3: + <<: *s3 +``` + +Cortex names the bucket field `bucket_name` and controls addressing with `bucket_lookup_type: path`. `insecure: true` allows the plain-HTTP endpoint. The `block_ranges_period` and `ship_interval` values shorten the block cycle so the test produces objects quickly; keep the defaults in production. + +## 2. Run Cortex + +Create the bucket and start Cortex on the same Docker network as RustFS: + +```bash +rc mb rustfs/cortex + +docker run -d --name cortex --network oo-rustfs_default -p 9009:9009 \ + -v "$PWD/cortex.yaml":/etc/cortex/cortex.yaml:ro \ + -v /opt/cortex/data:/data \ + cortexproject/cortex:v1.21.1 -config.file=/etc/cortex/cortex.yaml +``` + +```text +ts=... caller=cortex.go:469 level=info msg="Cortex started" +``` + +## 3. Push metrics and rules + +Start a Prometheus that scrapes itself and remote-writes into Cortex: + +```yaml title="prometheus.yml" +global: + scrape_interval: 5s +scrape_configs: + - job_name: self + static_configs: + - targets: ["localhost:9090"] +remote_write: + - url: http://cortex:9009/api/v1/push +``` + +```bash +docker run -d --name prom-writer --network oo-rustfs_default \ + -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml:ro \ + prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml +``` + +Store a rule group through the ruler API: + +```bash +cat > rules.yaml << "EOF" +name: rustfs-demo +rules: + - alert: RustFSAlwaysFiring + expr: vector(1) + labels: + severity: demo + annotations: + summary: "demo alert stored in RustFS" +EOF + +curl -s -o /dev/null -w "%{http_code}\n" \ + -X POST http://localhost:9009/api/v1/rules/rustfs-demo \ + --data-binary @rules.yaml -H "Content-Type: application/yaml" +``` + +```text +202 +``` + +Query the ingested series back: + +```bash +curl -s "http://localhost:9009/prometheus/api/v1/query?query=up" +``` + +```text +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up",...},"value":[...,"1"]}]}} +``` + +## 4. Verify objects in RustFS + +List the bucket prefixes: + +```bash +rc ls rustfs/cortex/ -r +``` + +The ruler config landed immediately, and the ingester shipped two TSDB blocks (each with `chunks`, `index`, and `meta.json`): + +```text +fake/01M35WE4B1G6RRKHRV0HSGADY1/chunks/000001 +fake/01M35WE4B1G6RRKHRV0HSGADY1/index +fake/01M35WE4B1G6RRKHRV0HSGADY1/meta.json +fake/01M35WWS2YCC7FBK1JSBKFQGRP/chunks/000001 +fake/01M35WWS2YCC7FBK1JSBKFQGRP/index +fake/01M35WWS2YCC7FBK1JSBKFQGRP/meta.json +rules/fake/cnVzdGZzLWRlbW8=/cnVzdGZzLWRlbW8= +``` + +![Cortex blocks and rules stored in the RustFS Console](./images/rustfs-cortex-blocks.png) + +## 5. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f cortex prom-writer +``` + +To delete the stored data: + +```bash +rc rm rustfs/cortex/ --recursive --force +``` + +## Troubleshooting + +### `field bucket not found in type s3.Config` + +Cortex uses `bucket_name`, not `bucket`. Path-style addressing is set with `bucket_lookup_type: path` — the `force_path_style` key used by Thanos does not exist in Cortex. + +### Blocks never appear in the bucket + +The ingester only ships a block after the head is compacted at a `block_ranges_period` boundary (default 2h). For tests, set a small range such as `[15m]` together with `ship_interval: 30s` and wait for one period. + +### `Unauthorized` or empty bucket listings + +Confirm `access_key_id` and `secret_access_key` are present in every `s3` block that uses the bucket, and that `insecure: true` matches the plain-HTTP endpoint. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Cortex storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Combine Cortex with [Thanos](/developer/integration/observability/thanos) when you need block storage for long-term Prometheus data as well. diff --git a/content/en/developer/integration/cloud-native/flux.md b/content/en/developer/integration/cloud-native/flux.md new file mode 100644 index 00000000..0319cc85 --- /dev/null +++ b/content/en/developer/integration/cloud-native/flux.md @@ -0,0 +1,157 @@ +--- +title: "Flux" +description: "Deliver Kubernetes manifests to Flux CD from RustFS through the Bucket source API." +--- + +This guide connects [Flux CD](https://github.com/fluxcd/flux2) — the GitOps toolkit for Kubernetes — to **RustFS** through the source-controller Bucket API. You will seed a RustFS bucket with Kubernetes manifests, register the bucket as a Flux `Bucket` source with the `generic` provider, and let a Kustomization apply everything it contains. The workflow was verified with `flux v2.9.5` (source-controller) on `k3s v1.36.4+k3s1` against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a Kubernetes cluster with Flux installed (`flux install`) and the `rc` client. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Manifests["Kustomize manifests"] -->|"rc cp"| Bucket["RustFS :9000"] + source-controller -->|"S3 API"| Bucket + source-controller -->|"artifact"| kustomize-controller + kustomize-controller -->|"apply"| Cluster["Kubernetes cluster"] +``` + +The source-controller lists and downloads objects from the bucket over the S3 API, packs them into an artifact, and the kustomize-controller applies the manifests from that artifact. RustFS acts as the pull-based source of truth for the cluster. + +## 1. Seed the bucket + +Store a Kustomize overlay in the bucket, replacing all connection placeholders: + +```yaml title="clusters/staging/kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - rustfs-demo-configmap.yaml +``` + +```yaml title="clusters/staging/rustfs-demo-configmap.yaml" +apiVersion: v1 +kind: ConfigMap +metadata: + name: rustfs-flux-demo + namespace: default +data: + storage: rustfs + source: s3-bucket +``` + +Upload the files: + +```bash +rc mb rustfs/flux-src +rc cp --recursive ./clusters rustfs/flux-src/ +rc ls rustfs/flux-src/ -r +``` + +```text +staging/kustomization.yaml +staging/rustfs-demo-configmap.yaml +``` + +## 2. Create the credentials secret + +Flux reads the credentials from a secret in the same namespace as the source, using lowercase keys: + +```bash +kubectl -n flux-system create secret generic rustfs-creds \ + --from-literal=accesskey= \ + --from-literal=secretkey= +``` + +The field names must be `accesskey` and `secretkey` — capitalization such as `accessKey` fails with an `AuthenticationFailed` status. + +## 3. Register the Bucket source + +Create the Bucket source with the `generic` provider: + +```bash +flux create source bucket rustfs-demo \ + --bucket-name=flux-src \ + --endpoint=:9000 \ + --insecure \ + --secret-ref=rustfs-creds \ + --provider=generic \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ Bucket source reconciliation completed +``` + +If reconciliation is still running, check the status with `flux get sources bucket`: + +```text +NAME REVISION SUSPENDED READY MESSAGE +rustfs-demo sha256:481816d1 False True stored artifact: revision 'sha256:481816d1' +``` + +The `generic` provider uses path-style requests and plain HTTP with `--insecure`, so the endpoint is the bare `host:port`. + +## 4. Apply the manifests + +Create a Kustomization that consumes the artifact: + +```bash +flux create kustomization rustfs-demo \ + --source=Bucket/rustfs-demo \ + --path="./staging" \ + --prune=true \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ applied revision sha256:481816d1 +``` + +Confirm the manifest landed on the cluster: + +```bash +kubectl -n default get configmap rustfs-flux-demo -o jsonpath="{.data}" +``` + +```text +{"source":"s3-bucket","storage":"rustfs"} +``` + +## 5. Stop or reset + +Suspend or delete the Flux resources without touching the bucket: + +```bash +flux suspend source bucket rustfs-demo --namespace=flux-system +flux delete kustomization rustfs-demo --namespace=flux-system +``` + +To delete the bucket contents: + +```bash +rc rm rustfs/flux-src/ --recursive --force +``` + +## Troubleshooting + +### `invalid 'rustfs-creds' secret data: required fields 'accesskey' and 'secretkey'` + +The secret keys are lowercase. Recreate the secret with `accesskey` and `secretkey` as literal key names. + +### Reconciliation never becomes ready + +Confirm the endpoint is reachable from inside the cluster — pods cannot use `localhost`, so point the source at the host IP or the Docker bridge gateway (commonly `172.17.0.1`) that publishes the RustFS port. `--insecure` is required for plain HTTP. + +### `AuthenticationFailed` despite valid credentials + +Check that the secret lives in the same namespace as the Bucket source and that the keys have no trailing whitespace or quoting. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional source types. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Flux Bucket source documentation](https://fluxcd.io/flux/components/source/buckets/) to combine bucket sources with Helm releases and image automation. diff --git a/content/en/developer/integration/cloud-native/images/rustfs-cortex-blocks.png b/content/en/developer/integration/cloud-native/images/rustfs-cortex-blocks.png new file mode 100644 index 00000000..70ae5240 Binary files /dev/null and b/content/en/developer/integration/cloud-native/images/rustfs-cortex-blocks.png differ diff --git a/content/en/developer/integration/cloud-native/images/rustfs-flux-bucket.png b/content/en/developer/integration/cloud-native/images/rustfs-flux-bucket.png new file mode 100644 index 00000000..9336ea08 Binary files /dev/null and b/content/en/developer/integration/cloud-native/images/rustfs-flux-bucket.png differ diff --git a/content/en/developer/integration/cloud-native/index.md b/content/en/developer/integration/cloud-native/index.md new file mode 100644 index 00000000..1b514aea --- /dev/null +++ b/content/en/developer/integration/cloud-native/index.md @@ -0,0 +1,13 @@ +--- +title: "Cloud Native" +description: "Connect cloud native platforms to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for cloud native platforms that support an S3-compatible endpoint. + +## Platforms + +- [Cortex](./cortex.md) +- [Flux](./flux.md) + +Keep platform state in a dedicated bucket, and use credentials scoped to the required bucket operations. diff --git a/content/en/developer/integration/cloud-native/meta.json b/content/en/developer/integration/cloud-native/meta.json new file mode 100644 index 00000000..30b84775 --- /dev/null +++ b/content/en/developer/integration/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Cloud Native", + "pages": [ + "cortex", + "flux" + ] +} diff --git a/content/en/developer/integration/index.md b/content/en/developer/integration/index.md index 92980c37..12798bd1 100644 --- a/content/en/developer/integration/index.md +++ b/content/en/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Integrate RustFS with reverse proxies, backup tools, data analytics systems, observability platforms, container registries, and DevOps tooling." +description: "Integrate RustFS with reverse proxies, backup tools, data analytics systems, AI and cloud native platforms, observability platforms, container registries, and DevOps tooling." --- Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. @@ -10,7 +10,8 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Reverse Proxy](./reverse-proxy/index.md) covers Nginx, Traefik, Caddy, and HAProxy. - [Backup](./backup/index.md) covers Kopia, Longhorn, Restic, and Velero. - [AI](./ai/index.md) covers AI platforms including Ray. -- [Data Analytics](./big-data/index.md) covers analytics systems including ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL, and Zeppelin. +- [Data Analytics](./big-data/index.md) covers analytics systems including ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL, Vitess, and Zeppelin. +- [Cloud Native](./cloud-native/index.md) covers Cortex and Flux. - [Observability](./observability/index.md) covers telemetry systems including Fluentd, OpenObserve, OpenTelemetry, Thanos, and Tempo. - [Others](./others/index.md) covers the community-driven capo SDK for Python. - [Registry](./registry/index.md) covers Harbor. diff --git a/content/en/developer/integration/meta.json b/content/en/developer/integration/meta.json index a3c989a5..f5a8d87c 100644 --- a/content/en/developer/integration/meta.json +++ b/content/en/developer/integration/meta.json @@ -5,6 +5,7 @@ "backup", "big-data", "ai", + "cloud-native", "observability", "others", "registry", diff --git a/content/fr/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/fr/developer/integration/big-data/images/rustfs-vitess-backups.png new file mode 100644 index 00000000..5a939c8d Binary files /dev/null and b/content/fr/developer/integration/big-data/images/rustfs-vitess-backups.png differ diff --git a/content/fr/developer/integration/big-data/index.md b/content/fr/developer/integration/big-data/index.md index add692d8..701a3710 100644 --- a/content/fr/developer/integration/big-data/index.md +++ b/content/fr/developer/integration/big-data/index.md @@ -21,5 +21,6 @@ Use **RustFS** as the object storage layer for data analytics systems that suppo - [Spark](./spark.md) - [Flink](./flink.md) - [Trino](./trino.md) +- [Vitess](./vitess.md) Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file diff --git a/content/fr/developer/integration/big-data/meta.json b/content/fr/developer/integration/big-data/meta.json index 249e219d..29684232 100644 --- a/content/fr/developer/integration/big-data/meta.json +++ b/content/fr/developer/integration/big-data/meta.json @@ -15,6 +15,7 @@ "pyiceberg", "spark", "trino", + "vitess", "zeppelin" ] } diff --git a/content/fr/developer/integration/big-data/vitess.md b/content/fr/developer/integration/big-data/vitess.md new file mode 100644 index 00000000..2260b744 --- /dev/null +++ b/content/fr/developer/integration/big-data/vitess.md @@ -0,0 +1,189 @@ +--- +title: "Vitess" +description: "Back up Vitess to RustFS through the S3 backup storage implementation." +--- + +This guide connects [Vitess](https://github.com/vitessio/vitess) — the database clustering system for horizontal scaling of MySQL — to **RustFS** through Vitess's S3 backup storage implementation. You will bring up a minimal Vitess topology on a single host, run `Backup` on a tablet, and confirm the backup chunks and `MANIFEST` in the bucket, then read the backup list back from RustFS. The workflow was verified with `vitess/lite` (Vitess v25.0.0-SNAPSHOT, built 2026-09-22), MySQL 8.4, and `rustfs/rustfs-x86-musl:v2.3.1`; stable Vitess 20+ releases use the same flags. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["vtctldclient"] -->|"Backup"| Tablet["vttablet"] + Tablet -->|"mysqld dump"| Chunks["chunks + MANIFEST"] + Tablet -->|"S3 API"| RustFS["RustFS :9000"] +``` + +`vtctld` triggers the backup, but the `vttablet` owning the tablet performs it: it snapshots MySQL, compresses the data into numbered chunks, and uploads them together with a `MANIFEST` through the S3-compatible endpoint. + +## 1. Run etcd and the Vitess image + +Vitess stores its topology in etcd, and the `vitess/lite` image ships all Vitess binaries plus MySQL: + +```bash +docker run -d --name etcd --hostname etcd --network oo-rustfs_default \ + quay.io/coreos/etcd:v3.5.17 etcd \ + --advertise-client-urls http://etcd:2379 \ + --listen-client-urls http://0.0.0.0:2379 + +docker run -d --name vtess --hostname vtess --network oo-rustfs_default \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + vitess/lite:latest sleep infinity +``` + +The AWS environment variables supply the credentials for the S3 backup storage in both `vtctld` and `vttablet`. + +## 2. Initialize MySQL + +Initialize a MySQL instance in the standard Vitess tablet directory so `vttablet` can load its `my.cnf` later: + +```bash +docker exec vtess sh -c \ + "/vt/bin/mysqlctl --log_dir /tmp/vtlogs init --tablet-dir vt_0000000100" +``` + +The instance is ready when the socket file exists: + +```text +/vt/vtdataroot/vt_0000000100/mysql.sock +``` + +## 3. Start vtctld and vttablet + +Start `vtctld` with the S3 backup flags, register the cell, and start the tablet. Replace all connection placeholders: + +```bash +docker exec vtess sh -c "nohup /vt/bin/vtctld \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --service-map grpc-vtctl,grpc-vtctld \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --port 15999 --grpc-port 15998 \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vtctld.out 2>&1 &" + +docker exec vtess sh -c \ + "/vt/bin/vtctldclient --server localhost:15998 \ + AddCellInfo --root /vitess/zone1 --server-address etcd:2379 zone1" + +docker exec vtess sh -c "nohup /vt/bin/vttablet \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --tablet-path zone1-0000000100 \ + --init-keyspace commerce --init-shard 0 --init-tablet-type replica \ + --port 15100 --grpc-port 15101 \ + --service-map grpc-queryservice,grpc-tabletmanager,grpc-throttler \ + --mycnf-file /vt/vtdataroot/vt_0000000100/my.cnf \ + --db-dba-user root --db-allprivs-user root --db-app-user root --db-repl-user root \ + --db-dba-use-ssl=false --db-allprivs-use-ssl=false \ + --db-app-use-ssl=false --db-repl-use-ssl=false \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vttablet.out 2>&1 &" +``` + +Both processes carry the S3 flags because `vttablet` performs the backup while `vtctld` lists and removes backups. `--s3-backup-force-path-style` is mandatory for non-AWS endpoints. Check that the tablet registered: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetTablets +``` + +```text +zone1-0000000100 commerce 0 replica vtess:15100 vtess:3306 [] +``` + +## 4. Run the backup + +Create the bucket and trigger the backup: + +```bash +rc mb rustfs/vitess-backups + +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 \ + Backup zone1-0000000100 +``` + +The stream ends with the engine writing the manifest: + +```text +commerce/0 (zone1-0000000100): ... value:"Completed backing up MANIFEST (attempt 1/2)" +``` + +## 5. Verify objects in RustFS + +List the backup prefix: + +```bash +rc ls rustfs/vitess-backups/ -r | head -4 +``` + +The tablet uploaded compressed data chunks plus the metadata files: + +```text +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/0 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/1 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/MANIFEST +``` + +![Vitess backup files stored in the RustFS Console](./images/rustfs-vitess-backups.png) + +Read the backup list back from RustFS through `vtctld`: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetBackups commerce/0 +``` + +```text +2026-09-23.011225.zone1-0000000100 +``` + +## 6. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f vtess etcd +``` + +To delete the stored backups: + +```bash +rc rm rustfs/vitess-backups/ --recursive --force +``` + +## Troubleshooting + +### `cannot perform backup without my.cnf` + +Passing connection parameters such as `--db-socket` or `--db-host` makes `vttablet` skip loading `my.cnf`, and backups refuse to run. Start `vttablet` with `--mycnf-file` pointing at the instance config and let it discover the socket from there — this is why the MySQL instance lives in the `vt_0000000100` directory from step 2. + +### `unknown service vtctlservice.Vtctld` + +`vtctld` exposes the API used by `vtctldclient` only when the service map includes it: `--service-map grpc-vtctl,grpc-vtctld`. Also make sure `vtctldclient --server` points at the gRPC port (`15998` here), not the web UI port. + +### `node doesn't exist: /vitess/global/cells/zone1/CellInfo` + +The cell must exist before tablets register. Run `vtctldclient AddCellInfo` as shown in step 3 before starting `vttablet`. + +### Flag parse errors such as `unknown shorthand flag` + +Vitess 20+ normalizes underscores to dashes, so write flags in dash style (`--tablet-path`). A single-dash long flag like `-tablet_dir` is parsed as shorthand options and fails. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Vitess storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Vitess backup and restore documentation](https://vitess.io/docs/user-guides/configuration-basic/#backups) to schedule backups and restore tablets from the bucket. diff --git a/content/fr/developer/integration/cloud-native/cortex.md b/content/fr/developer/integration/cloud-native/cortex.md new file mode 100644 index 00000000..3db19b47 --- /dev/null +++ b/content/fr/developer/integration/cloud-native/cortex.md @@ -0,0 +1,218 @@ +--- +title: "Cortex" +description: "Run Cortex with RustFS as the S3-compatible blocks, ruler, and alertmanager storage backend." +--- + +This guide connects [Cortex](https://github.com/cortexproject/cortex) — the horizontally scalable Prometheus-compatible metrics backend — to **RustFS** through Cortex's native S3 bucket storage. You will run Cortex in single-binary mode with blocks, ruler, and alertmanager storage pointed at a RustFS bucket, push metrics with remote write, store a rule group, and verify the objects. The workflow was verified with `cortexproject/cortex:v1.21.1` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Prom["Prometheus"] -->|"remote write"| Cortex["Cortex :9009"] + Cortex -->|"blocks, rules, configs"| RustFS["RustFS :9000"] +``` + +Cortex stores every TSDB block, ruler rule group, and alertmanager configuration in object storage. Pointing the `s3` backend at RustFS makes the bucket the single source of truth for all tenant data. + +## 1. Create the Cortex config + +Create the config file, replacing all connection placeholders: + +```yaml title="cortex.yaml" +target: all +auth_enabled: false + +server: + http_listen_port: 9009 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester: + lifecycler: + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +blocks_storage: + backend: s3 + s3: &s3 + endpoint: :9000 + region: us-east-1 + bucket_name: cortex + access_key_id: + secret_access_key: + insecure: true + bucket_lookup_type: path + tsdb: + dir: /data/tsdb + block_ranges_period: [15m] + ship_interval: 30s + bucket_store: + sync_dir: /data/tsdb-sync + bucket_index: + enabled: true + +compactor: + data_dir: /data/compactor + sharding_ring: + kvstore: + store: inmemory + +ruler: + enable_api: true + rule_path: /data/ruler + +ruler_storage: + backend: s3 + s3: + <<: *s3 + +alertmanager: + external_url: /alertmanager + enable_api: true + data_dir: /data/alertmanager + +alertmanager_storage: + backend: s3 + s3: + <<: *s3 +``` + +Cortex names the bucket field `bucket_name` and controls addressing with `bucket_lookup_type: path`. `insecure: true` allows the plain-HTTP endpoint. The `block_ranges_period` and `ship_interval` values shorten the block cycle so the test produces objects quickly; keep the defaults in production. + +## 2. Run Cortex + +Create the bucket and start Cortex on the same Docker network as RustFS: + +```bash +rc mb rustfs/cortex + +docker run -d --name cortex --network oo-rustfs_default -p 9009:9009 \ + -v "$PWD/cortex.yaml":/etc/cortex/cortex.yaml:ro \ + -v /opt/cortex/data:/data \ + cortexproject/cortex:v1.21.1 -config.file=/etc/cortex/cortex.yaml +``` + +```text +ts=... caller=cortex.go:469 level=info msg="Cortex started" +``` + +## 3. Push metrics and rules + +Start a Prometheus that scrapes itself and remote-writes into Cortex: + +```yaml title="prometheus.yml" +global: + scrape_interval: 5s +scrape_configs: + - job_name: self + static_configs: + - targets: ["localhost:9090"] +remote_write: + - url: http://cortex:9009/api/v1/push +``` + +```bash +docker run -d --name prom-writer --network oo-rustfs_default \ + -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml:ro \ + prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml +``` + +Store a rule group through the ruler API: + +```bash +cat > rules.yaml << "EOF" +name: rustfs-demo +rules: + - alert: RustFSAlwaysFiring + expr: vector(1) + labels: + severity: demo + annotations: + summary: "demo alert stored in RustFS" +EOF + +curl -s -o /dev/null -w "%{http_code}\n" \ + -X POST http://localhost:9009/api/v1/rules/rustfs-demo \ + --data-binary @rules.yaml -H "Content-Type: application/yaml" +``` + +```text +202 +``` + +Query the ingested series back: + +```bash +curl -s "http://localhost:9009/prometheus/api/v1/query?query=up" +``` + +```text +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up",...},"value":[...,"1"]}]}} +``` + +## 4. Verify objects in RustFS + +List the bucket prefixes: + +```bash +rc ls rustfs/cortex/ -r +``` + +The ruler config landed immediately, and the ingester shipped two TSDB blocks (each with `chunks`, `index`, and `meta.json`): + +```text +fake/01M35WE4B1G6RRKHRV0HSGADY1/chunks/000001 +fake/01M35WE4B1G6RRKHRV0HSGADY1/index +fake/01M35WE4B1G6RRKHRV0HSGADY1/meta.json +fake/01M35WWS2YCC7FBK1JSBKFQGRP/chunks/000001 +fake/01M35WWS2YCC7FBK1JSBKFQGRP/index +fake/01M35WWS2YCC7FBK1JSBKFQGRP/meta.json +rules/fake/cnVzdGZzLWRlbW8=/cnVzdGZzLWRlbW8= +``` + +![Cortex blocks and rules stored in the RustFS Console](./images/rustfs-cortex-blocks.png) + +## 5. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f cortex prom-writer +``` + +To delete the stored data: + +```bash +rc rm rustfs/cortex/ --recursive --force +``` + +## Troubleshooting + +### `field bucket not found in type s3.Config` + +Cortex uses `bucket_name`, not `bucket`. Path-style addressing is set with `bucket_lookup_type: path` — the `force_path_style` key used by Thanos does not exist in Cortex. + +### Blocks never appear in the bucket + +The ingester only ships a block after the head is compacted at a `block_ranges_period` boundary (default 2h). For tests, set a small range such as `[15m]` together with `ship_interval: 30s` and wait for one period. + +### `Unauthorized` or empty bucket listings + +Confirm `access_key_id` and `secret_access_key` are present in every `s3` block that uses the bucket, and that `insecure: true` matches the plain-HTTP endpoint. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Cortex storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Combine Cortex with [Thanos](/developer/integration/observability/thanos) when you need block storage for long-term Prometheus data as well. diff --git a/content/fr/developer/integration/cloud-native/flux.md b/content/fr/developer/integration/cloud-native/flux.md new file mode 100644 index 00000000..0319cc85 --- /dev/null +++ b/content/fr/developer/integration/cloud-native/flux.md @@ -0,0 +1,157 @@ +--- +title: "Flux" +description: "Deliver Kubernetes manifests to Flux CD from RustFS through the Bucket source API." +--- + +This guide connects [Flux CD](https://github.com/fluxcd/flux2) — the GitOps toolkit for Kubernetes — to **RustFS** through the source-controller Bucket API. You will seed a RustFS bucket with Kubernetes manifests, register the bucket as a Flux `Bucket` source with the `generic` provider, and let a Kustomization apply everything it contains. The workflow was verified with `flux v2.9.5` (source-controller) on `k3s v1.36.4+k3s1` against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a Kubernetes cluster with Flux installed (`flux install`) and the `rc` client. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Manifests["Kustomize manifests"] -->|"rc cp"| Bucket["RustFS :9000"] + source-controller -->|"S3 API"| Bucket + source-controller -->|"artifact"| kustomize-controller + kustomize-controller -->|"apply"| Cluster["Kubernetes cluster"] +``` + +The source-controller lists and downloads objects from the bucket over the S3 API, packs them into an artifact, and the kustomize-controller applies the manifests from that artifact. RustFS acts as the pull-based source of truth for the cluster. + +## 1. Seed the bucket + +Store a Kustomize overlay in the bucket, replacing all connection placeholders: + +```yaml title="clusters/staging/kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - rustfs-demo-configmap.yaml +``` + +```yaml title="clusters/staging/rustfs-demo-configmap.yaml" +apiVersion: v1 +kind: ConfigMap +metadata: + name: rustfs-flux-demo + namespace: default +data: + storage: rustfs + source: s3-bucket +``` + +Upload the files: + +```bash +rc mb rustfs/flux-src +rc cp --recursive ./clusters rustfs/flux-src/ +rc ls rustfs/flux-src/ -r +``` + +```text +staging/kustomization.yaml +staging/rustfs-demo-configmap.yaml +``` + +## 2. Create the credentials secret + +Flux reads the credentials from a secret in the same namespace as the source, using lowercase keys: + +```bash +kubectl -n flux-system create secret generic rustfs-creds \ + --from-literal=accesskey= \ + --from-literal=secretkey= +``` + +The field names must be `accesskey` and `secretkey` — capitalization such as `accessKey` fails with an `AuthenticationFailed` status. + +## 3. Register the Bucket source + +Create the Bucket source with the `generic` provider: + +```bash +flux create source bucket rustfs-demo \ + --bucket-name=flux-src \ + --endpoint=:9000 \ + --insecure \ + --secret-ref=rustfs-creds \ + --provider=generic \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ Bucket source reconciliation completed +``` + +If reconciliation is still running, check the status with `flux get sources bucket`: + +```text +NAME REVISION SUSPENDED READY MESSAGE +rustfs-demo sha256:481816d1 False True stored artifact: revision 'sha256:481816d1' +``` + +The `generic` provider uses path-style requests and plain HTTP with `--insecure`, so the endpoint is the bare `host:port`. + +## 4. Apply the manifests + +Create a Kustomization that consumes the artifact: + +```bash +flux create kustomization rustfs-demo \ + --source=Bucket/rustfs-demo \ + --path="./staging" \ + --prune=true \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ applied revision sha256:481816d1 +``` + +Confirm the manifest landed on the cluster: + +```bash +kubectl -n default get configmap rustfs-flux-demo -o jsonpath="{.data}" +``` + +```text +{"source":"s3-bucket","storage":"rustfs"} +``` + +## 5. Stop or reset + +Suspend or delete the Flux resources without touching the bucket: + +```bash +flux suspend source bucket rustfs-demo --namespace=flux-system +flux delete kustomization rustfs-demo --namespace=flux-system +``` + +To delete the bucket contents: + +```bash +rc rm rustfs/flux-src/ --recursive --force +``` + +## Troubleshooting + +### `invalid 'rustfs-creds' secret data: required fields 'accesskey' and 'secretkey'` + +The secret keys are lowercase. Recreate the secret with `accesskey` and `secretkey` as literal key names. + +### Reconciliation never becomes ready + +Confirm the endpoint is reachable from inside the cluster — pods cannot use `localhost`, so point the source at the host IP or the Docker bridge gateway (commonly `172.17.0.1`) that publishes the RustFS port. `--insecure` is required for plain HTTP. + +### `AuthenticationFailed` despite valid credentials + +Check that the secret lives in the same namespace as the Bucket source and that the keys have no trailing whitespace or quoting. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional source types. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Flux Bucket source documentation](https://fluxcd.io/flux/components/source/buckets/) to combine bucket sources with Helm releases and image automation. diff --git a/content/fr/developer/integration/cloud-native/images/rustfs-cortex-blocks.png b/content/fr/developer/integration/cloud-native/images/rustfs-cortex-blocks.png new file mode 100644 index 00000000..70ae5240 Binary files /dev/null and b/content/fr/developer/integration/cloud-native/images/rustfs-cortex-blocks.png differ diff --git a/content/fr/developer/integration/cloud-native/images/rustfs-flux-bucket.png b/content/fr/developer/integration/cloud-native/images/rustfs-flux-bucket.png new file mode 100644 index 00000000..9336ea08 Binary files /dev/null and b/content/fr/developer/integration/cloud-native/images/rustfs-flux-bucket.png differ diff --git a/content/fr/developer/integration/cloud-native/index.md b/content/fr/developer/integration/cloud-native/index.md new file mode 100644 index 00000000..1b514aea --- /dev/null +++ b/content/fr/developer/integration/cloud-native/index.md @@ -0,0 +1,13 @@ +--- +title: "Cloud Native" +description: "Connect cloud native platforms to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for cloud native platforms that support an S3-compatible endpoint. + +## Platforms + +- [Cortex](./cortex.md) +- [Flux](./flux.md) + +Keep platform state in a dedicated bucket, and use credentials scoped to the required bucket operations. diff --git a/content/fr/developer/integration/cloud-native/meta.json b/content/fr/developer/integration/cloud-native/meta.json new file mode 100644 index 00000000..30b84775 --- /dev/null +++ b/content/fr/developer/integration/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "Cloud Native", + "pages": [ + "cortex", + "flux" + ] +} diff --git a/content/fr/developer/integration/index.md b/content/fr/developer/integration/index.md index 19dfbdf8..af4a9c16 100644 --- a/content/fr/developer/integration/index.md +++ b/content/fr/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Intégrez RustFS avec des reverse proxies, des outils de sauvegarde, des systèmes d'analyse de données, des plateformes d'observabilité, des registries de conteneurs et des outils DevOps." +description: "Intégrez RustFS avec des reverse proxies, des outils de sauvegarde, des systèmes d'analyse de données, des plateformes d'IA et cloud native, des plateformes d'observabilité, des registries de conteneurs et des outils DevOps." --- Utilisez cette section pour connecter **RustFS** à des plateformes d'infrastructure et d'applications via son API compatible S3. @@ -10,7 +10,8 @@ Utilisez cette section pour connecter **RustFS** à des plateformes d'infrastruc - [Reverse Proxy](./reverse-proxy/index.md) couvre Nginx, Traefik, Caddy et HAProxy. - [Backup](./backup/index.md) couvre Kopia, Longhorn, Restic et Velero. - [IA](./ai/index.md) couvre les plateformes d'IA incluant Ray. -- [Analyse de données](./big-data/index.md) couvre les systèmes d'analyse incluant ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL et Zeppelin. +- [Analyse de données](./big-data/index.md) couvre les systèmes d'analyse incluant ClickHouse, Doris, Hudi, Iceberg, lakeFS, Milvus, OpenDAL, Vitess et Zeppelin. +- [Cloud Native](./cloud-native/index.md) couvre Cortex et Flux. - [Observabilité](./observability/index.md) couvre les systèmes de télémétrie incluant Fluentd, OpenObserve, OpenTelemetry, Thanos et Tempo. - [Autres](./others/index.md) couvre le SDK communautaire capo pour Python. - [Registre](./registry/index.md) couvre Harbor. diff --git a/content/fr/developer/integration/meta.json b/content/fr/developer/integration/meta.json index a3c989a5..f5a8d87c 100644 --- a/content/fr/developer/integration/meta.json +++ b/content/fr/developer/integration/meta.json @@ -5,6 +5,7 @@ "backup", "big-data", "ai", + "cloud-native", "observability", "others", "registry", diff --git a/content/ja/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/ja/developer/integration/big-data/images/rustfs-vitess-backups.png new file mode 100644 index 00000000..5a939c8d Binary files /dev/null and b/content/ja/developer/integration/big-data/images/rustfs-vitess-backups.png differ diff --git a/content/ja/developer/integration/big-data/index.md b/content/ja/developer/integration/big-data/index.md index cd8c7ce7..d610500d 100644 --- a/content/ja/developer/integration/big-data/index.md +++ b/content/ja/developer/integration/big-data/index.md @@ -21,5 +21,6 @@ Use **RustFS** as the object storage layer for data analytics systems that suppo - [Spark](./spark.md) - [Flink](./flink.md) - [Trino](./trino.md) +- [Vitess](./vitess.md) Keep application data in a dedicated bucket and prefix, and use credentials scoped to the required bucket operations. \ No newline at end of file diff --git a/content/ja/developer/integration/big-data/meta.json b/content/ja/developer/integration/big-data/meta.json index 6ebd60a9..22b9c24b 100644 --- a/content/ja/developer/integration/big-data/meta.json +++ b/content/ja/developer/integration/big-data/meta.json @@ -15,6 +15,7 @@ "pyiceberg", "spark", "trino", + "vitess", "zeppelin" ] } diff --git a/content/ja/developer/integration/big-data/vitess.md b/content/ja/developer/integration/big-data/vitess.md new file mode 100644 index 00000000..2260b744 --- /dev/null +++ b/content/ja/developer/integration/big-data/vitess.md @@ -0,0 +1,189 @@ +--- +title: "Vitess" +description: "Back up Vitess to RustFS through the S3 backup storage implementation." +--- + +This guide connects [Vitess](https://github.com/vitessio/vitess) — the database clustering system for horizontal scaling of MySQL — to **RustFS** through Vitess's S3 backup storage implementation. You will bring up a minimal Vitess topology on a single host, run `Backup` on a tablet, and confirm the backup chunks and `MANIFEST` in the bucket, then read the backup list back from RustFS. The workflow was verified with `vitess/lite` (Vitess v25.0.0-SNAPSHOT, built 2026-09-22), MySQL 8.4, and `rustfs/rustfs-x86-musl:v2.3.1`; stable Vitess 20+ releases use the same flags. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["vtctldclient"] -->|"Backup"| Tablet["vttablet"] + Tablet -->|"mysqld dump"| Chunks["chunks + MANIFEST"] + Tablet -->|"S3 API"| RustFS["RustFS :9000"] +``` + +`vtctld` triggers the backup, but the `vttablet` owning the tablet performs it: it snapshots MySQL, compresses the data into numbered chunks, and uploads them together with a `MANIFEST` through the S3-compatible endpoint. + +## 1. Run etcd and the Vitess image + +Vitess stores its topology in etcd, and the `vitess/lite` image ships all Vitess binaries plus MySQL: + +```bash +docker run -d --name etcd --hostname etcd --network oo-rustfs_default \ + quay.io/coreos/etcd:v3.5.17 etcd \ + --advertise-client-urls http://etcd:2379 \ + --listen-client-urls http://0.0.0.0:2379 + +docker run -d --name vtess --hostname vtess --network oo-rustfs_default \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + vitess/lite:latest sleep infinity +``` + +The AWS environment variables supply the credentials for the S3 backup storage in both `vtctld` and `vttablet`. + +## 2. Initialize MySQL + +Initialize a MySQL instance in the standard Vitess tablet directory so `vttablet` can load its `my.cnf` later: + +```bash +docker exec vtess sh -c \ + "/vt/bin/mysqlctl --log_dir /tmp/vtlogs init --tablet-dir vt_0000000100" +``` + +The instance is ready when the socket file exists: + +```text +/vt/vtdataroot/vt_0000000100/mysql.sock +``` + +## 3. Start vtctld and vttablet + +Start `vtctld` with the S3 backup flags, register the cell, and start the tablet. Replace all connection placeholders: + +```bash +docker exec vtess sh -c "nohup /vt/bin/vtctld \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --service-map grpc-vtctl,grpc-vtctld \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --port 15999 --grpc-port 15998 \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vtctld.out 2>&1 &" + +docker exec vtess sh -c \ + "/vt/bin/vtctldclient --server localhost:15998 \ + AddCellInfo --root /vitess/zone1 --server-address etcd:2379 zone1" + +docker exec vtess sh -c "nohup /vt/bin/vttablet \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --tablet-path zone1-0000000100 \ + --init-keyspace commerce --init-shard 0 --init-tablet-type replica \ + --port 15100 --grpc-port 15101 \ + --service-map grpc-queryservice,grpc-tabletmanager,grpc-throttler \ + --mycnf-file /vt/vtdataroot/vt_0000000100/my.cnf \ + --db-dba-user root --db-allprivs-user root --db-app-user root --db-repl-user root \ + --db-dba-use-ssl=false --db-allprivs-use-ssl=false \ + --db-app-use-ssl=false --db-repl-use-ssl=false \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vttablet.out 2>&1 &" +``` + +Both processes carry the S3 flags because `vttablet` performs the backup while `vtctld` lists and removes backups. `--s3-backup-force-path-style` is mandatory for non-AWS endpoints. Check that the tablet registered: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetTablets +``` + +```text +zone1-0000000100 commerce 0 replica vtess:15100 vtess:3306 [] +``` + +## 4. Run the backup + +Create the bucket and trigger the backup: + +```bash +rc mb rustfs/vitess-backups + +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 \ + Backup zone1-0000000100 +``` + +The stream ends with the engine writing the manifest: + +```text +commerce/0 (zone1-0000000100): ... value:"Completed backing up MANIFEST (attempt 1/2)" +``` + +## 5. Verify objects in RustFS + +List the backup prefix: + +```bash +rc ls rustfs/vitess-backups/ -r | head -4 +``` + +The tablet uploaded compressed data chunks plus the metadata files: + +```text +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/0 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/1 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/MANIFEST +``` + +![Vitess backup files stored in the RustFS Console](./images/rustfs-vitess-backups.png) + +Read the backup list back from RustFS through `vtctld`: + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetBackups commerce/0 +``` + +```text +2026-09-23.011225.zone1-0000000100 +``` + +## 6. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f vtess etcd +``` + +To delete the stored backups: + +```bash +rc rm rustfs/vitess-backups/ --recursive --force +``` + +## Troubleshooting + +### `cannot perform backup without my.cnf` + +Passing connection parameters such as `--db-socket` or `--db-host` makes `vttablet` skip loading `my.cnf`, and backups refuse to run. Start `vttablet` with `--mycnf-file` pointing at the instance config and let it discover the socket from there — this is why the MySQL instance lives in the `vt_0000000100` directory from step 2. + +### `unknown service vtctlservice.Vtctld` + +`vtctld` exposes the API used by `vtctldclient` only when the service map includes it: `--service-map grpc-vtctl,grpc-vtctld`. Also make sure `vtctldclient --server` points at the gRPC port (`15998` here), not the web UI port. + +### `node doesn't exist: /vitess/global/cells/zone1/CellInfo` + +The cell must exist before tablets register. Run `vtctldclient AddCellInfo` as shown in step 3 before starting `vttablet`. + +### Flag parse errors such as `unknown shorthand flag` + +Vitess 20+ normalizes underscores to dashes, so write flags in dash style (`--tablet-path`). A single-dash long flag like `-tablet_dir` is parsed as shorthand options and fails. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Vitess storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Vitess backup and restore documentation](https://vitess.io/docs/user-guides/configuration-basic/#backups) to schedule backups and restore tablets from the bucket. diff --git a/content/ja/developer/integration/cloud-native/cortex.md b/content/ja/developer/integration/cloud-native/cortex.md new file mode 100644 index 00000000..3db19b47 --- /dev/null +++ b/content/ja/developer/integration/cloud-native/cortex.md @@ -0,0 +1,218 @@ +--- +title: "Cortex" +description: "Run Cortex with RustFS as the S3-compatible blocks, ruler, and alertmanager storage backend." +--- + +This guide connects [Cortex](https://github.com/cortexproject/cortex) — the horizontally scalable Prometheus-compatible metrics backend — to **RustFS** through Cortex's native S3 bucket storage. You will run Cortex in single-binary mode with blocks, ruler, and alertmanager storage pointed at a RustFS bucket, push metrics with remote write, store a rule group, and verify the objects. The workflow was verified with `cortexproject/cortex:v1.21.1` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Prom["Prometheus"] -->|"remote write"| Cortex["Cortex :9009"] + Cortex -->|"blocks, rules, configs"| RustFS["RustFS :9000"] +``` + +Cortex stores every TSDB block, ruler rule group, and alertmanager configuration in object storage. Pointing the `s3` backend at RustFS makes the bucket the single source of truth for all tenant data. + +## 1. Create the Cortex config + +Create the config file, replacing all connection placeholders: + +```yaml title="cortex.yaml" +target: all +auth_enabled: false + +server: + http_listen_port: 9009 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester: + lifecycler: + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +blocks_storage: + backend: s3 + s3: &s3 + endpoint: :9000 + region: us-east-1 + bucket_name: cortex + access_key_id: + secret_access_key: + insecure: true + bucket_lookup_type: path + tsdb: + dir: /data/tsdb + block_ranges_period: [15m] + ship_interval: 30s + bucket_store: + sync_dir: /data/tsdb-sync + bucket_index: + enabled: true + +compactor: + data_dir: /data/compactor + sharding_ring: + kvstore: + store: inmemory + +ruler: + enable_api: true + rule_path: /data/ruler + +ruler_storage: + backend: s3 + s3: + <<: *s3 + +alertmanager: + external_url: /alertmanager + enable_api: true + data_dir: /data/alertmanager + +alertmanager_storage: + backend: s3 + s3: + <<: *s3 +``` + +Cortex names the bucket field `bucket_name` and controls addressing with `bucket_lookup_type: path`. `insecure: true` allows the plain-HTTP endpoint. The `block_ranges_period` and `ship_interval` values shorten the block cycle so the test produces objects quickly; keep the defaults in production. + +## 2. Run Cortex + +Create the bucket and start Cortex on the same Docker network as RustFS: + +```bash +rc mb rustfs/cortex + +docker run -d --name cortex --network oo-rustfs_default -p 9009:9009 \ + -v "$PWD/cortex.yaml":/etc/cortex/cortex.yaml:ro \ + -v /opt/cortex/data:/data \ + cortexproject/cortex:v1.21.1 -config.file=/etc/cortex/cortex.yaml +``` + +```text +ts=... caller=cortex.go:469 level=info msg="Cortex started" +``` + +## 3. Push metrics and rules + +Start a Prometheus that scrapes itself and remote-writes into Cortex: + +```yaml title="prometheus.yml" +global: + scrape_interval: 5s +scrape_configs: + - job_name: self + static_configs: + - targets: ["localhost:9090"] +remote_write: + - url: http://cortex:9009/api/v1/push +``` + +```bash +docker run -d --name prom-writer --network oo-rustfs_default \ + -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml:ro \ + prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml +``` + +Store a rule group through the ruler API: + +```bash +cat > rules.yaml << "EOF" +name: rustfs-demo +rules: + - alert: RustFSAlwaysFiring + expr: vector(1) + labels: + severity: demo + annotations: + summary: "demo alert stored in RustFS" +EOF + +curl -s -o /dev/null -w "%{http_code}\n" \ + -X POST http://localhost:9009/api/v1/rules/rustfs-demo \ + --data-binary @rules.yaml -H "Content-Type: application/yaml" +``` + +```text +202 +``` + +Query the ingested series back: + +```bash +curl -s "http://localhost:9009/prometheus/api/v1/query?query=up" +``` + +```text +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up",...},"value":[...,"1"]}]}} +``` + +## 4. Verify objects in RustFS + +List the bucket prefixes: + +```bash +rc ls rustfs/cortex/ -r +``` + +The ruler config landed immediately, and the ingester shipped two TSDB blocks (each with `chunks`, `index`, and `meta.json`): + +```text +fake/01M35WE4B1G6RRKHRV0HSGADY1/chunks/000001 +fake/01M35WE4B1G6RRKHRV0HSGADY1/index +fake/01M35WE4B1G6RRKHRV0HSGADY1/meta.json +fake/01M35WWS2YCC7FBK1JSBKFQGRP/chunks/000001 +fake/01M35WWS2YCC7FBK1JSBKFQGRP/index +fake/01M35WWS2YCC7FBK1JSBKFQGRP/meta.json +rules/fake/cnVzdGZzLWRlbW8=/cnVzdGZzLWRlbW8= +``` + +![Cortex blocks and rules stored in the RustFS Console](./images/rustfs-cortex-blocks.png) + +## 5. Stop or reset + +To tear down the demo while keeping the bucket objects: + +```bash +docker rm -f cortex prom-writer +``` + +To delete the stored data: + +```bash +rc rm rustfs/cortex/ --recursive --force +``` + +## Troubleshooting + +### `field bucket not found in type s3.Config` + +Cortex uses `bucket_name`, not `bucket`. Path-style addressing is set with `bucket_lookup_type: path` — the `force_path_style` key used by Thanos does not exist in Cortex. + +### Blocks never appear in the bucket + +The ingester only ships a block after the head is compacted at a `block_ranges_period` boundary (default 2h). For tests, set a small range such as `[15m]` together with `ship_interval: 30s` and wait for one period. + +### `Unauthorized` or empty bucket listings + +Confirm `access_key_id` and `secret_access_key` are present in every `s3` block that uses the bucket, and that `insecure: true` matches the plain-HTTP endpoint. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional Cortex storage options. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Combine Cortex with [Thanos](/developer/integration/observability/thanos) when you need block storage for long-term Prometheus data as well. diff --git a/content/ja/developer/integration/cloud-native/flux.md b/content/ja/developer/integration/cloud-native/flux.md new file mode 100644 index 00000000..0319cc85 --- /dev/null +++ b/content/ja/developer/integration/cloud-native/flux.md @@ -0,0 +1,157 @@ +--- +title: "Flux" +description: "Deliver Kubernetes manifests to Flux CD from RustFS through the Bucket source API." +--- + +This guide connects [Flux CD](https://github.com/fluxcd/flux2) — the GitOps toolkit for Kubernetes — to **RustFS** through the source-controller Bucket API. You will seed a RustFS bucket with Kubernetes manifests, register the bucket as a Flux `Bucket` source with the `generic` provider, and let a Kustomization apply everything it contains. The workflow was verified with `flux v2.9.5` (source-controller) on `k3s v1.36.4+k3s1` against `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a Kubernetes cluster with Flux installed (`flux install`) and the `rc` client. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Manifests["Kustomize manifests"] -->|"rc cp"| Bucket["RustFS :9000"] + source-controller -->|"S3 API"| Bucket + source-controller -->|"artifact"| kustomize-controller + kustomize-controller -->|"apply"| Cluster["Kubernetes cluster"] +``` + +The source-controller lists and downloads objects from the bucket over the S3 API, packs them into an artifact, and the kustomize-controller applies the manifests from that artifact. RustFS acts as the pull-based source of truth for the cluster. + +## 1. Seed the bucket + +Store a Kustomize overlay in the bucket, replacing all connection placeholders: + +```yaml title="clusters/staging/kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - rustfs-demo-configmap.yaml +``` + +```yaml title="clusters/staging/rustfs-demo-configmap.yaml" +apiVersion: v1 +kind: ConfigMap +metadata: + name: rustfs-flux-demo + namespace: default +data: + storage: rustfs + source: s3-bucket +``` + +Upload the files: + +```bash +rc mb rustfs/flux-src +rc cp --recursive ./clusters rustfs/flux-src/ +rc ls rustfs/flux-src/ -r +``` + +```text +staging/kustomization.yaml +staging/rustfs-demo-configmap.yaml +``` + +## 2. Create the credentials secret + +Flux reads the credentials from a secret in the same namespace as the source, using lowercase keys: + +```bash +kubectl -n flux-system create secret generic rustfs-creds \ + --from-literal=accesskey= \ + --from-literal=secretkey= +``` + +The field names must be `accesskey` and `secretkey` — capitalization such as `accessKey` fails with an `AuthenticationFailed` status. + +## 3. Register the Bucket source + +Create the Bucket source with the `generic` provider: + +```bash +flux create source bucket rustfs-demo \ + --bucket-name=flux-src \ + --endpoint=:9000 \ + --insecure \ + --secret-ref=rustfs-creds \ + --provider=generic \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ Bucket source reconciliation completed +``` + +If reconciliation is still running, check the status with `flux get sources bucket`: + +```text +NAME REVISION SUSPENDED READY MESSAGE +rustfs-demo sha256:481816d1 False True stored artifact: revision 'sha256:481816d1' +``` + +The `generic` provider uses path-style requests and plain HTTP with `--insecure`, so the endpoint is the bare `host:port`. + +## 4. Apply the manifests + +Create a Kustomization that consumes the artifact: + +```bash +flux create kustomization rustfs-demo \ + --source=Bucket/rustfs-demo \ + --path="./staging" \ + --prune=true \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ applied revision sha256:481816d1 +``` + +Confirm the manifest landed on the cluster: + +```bash +kubectl -n default get configmap rustfs-flux-demo -o jsonpath="{.data}" +``` + +```text +{"source":"s3-bucket","storage":"rustfs"} +``` + +## 5. Stop or reset + +Suspend or delete the Flux resources without touching the bucket: + +```bash +flux suspend source bucket rustfs-demo --namespace=flux-system +flux delete kustomization rustfs-demo --namespace=flux-system +``` + +To delete the bucket contents: + +```bash +rc rm rustfs/flux-src/ --recursive --force +``` + +## Troubleshooting + +### `invalid 'rustfs-creds' secret data: required fields 'accesskey' and 'secretkey'` + +The secret keys are lowercase. Recreate the secret with `accesskey` and `secretkey` as literal key names. + +### Reconciliation never becomes ready + +Confirm the endpoint is reachable from inside the cluster — pods cannot use `localhost`, so point the source at the host IP or the Docker bridge gateway (commonly `172.17.0.1`) that publishes the RustFS port. `--insecure` is required for plain HTTP. + +### `AuthenticationFailed` despite valid credentials + +Check that the secret lives in the same namespace as the Bucket source and that the keys have no trailing whitespace or quoting. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional source types. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Flux Bucket source documentation](https://fluxcd.io/flux/components/source/buckets/) to combine bucket sources with Helm releases and image automation. diff --git a/content/ja/developer/integration/cloud-native/images/rustfs-cortex-blocks.png b/content/ja/developer/integration/cloud-native/images/rustfs-cortex-blocks.png new file mode 100644 index 00000000..70ae5240 Binary files /dev/null and b/content/ja/developer/integration/cloud-native/images/rustfs-cortex-blocks.png differ diff --git a/content/ja/developer/integration/cloud-native/images/rustfs-flux-bucket.png b/content/ja/developer/integration/cloud-native/images/rustfs-flux-bucket.png new file mode 100644 index 00000000..9336ea08 Binary files /dev/null and b/content/ja/developer/integration/cloud-native/images/rustfs-flux-bucket.png differ diff --git a/content/ja/developer/integration/cloud-native/index.md b/content/ja/developer/integration/cloud-native/index.md new file mode 100644 index 00000000..4c079efe --- /dev/null +++ b/content/ja/developer/integration/cloud-native/index.md @@ -0,0 +1,13 @@ +--- +title: "クラウドネイティブ" +description: "Connect cloud native platforms to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for cloud native platforms that support an S3-compatible endpoint. + +## Platforms + +- [Cortex](./cortex.md) +- [Flux](./flux.md) + +Keep platform state in a dedicated bucket, and use credentials scoped to the required bucket operations. diff --git a/content/ja/developer/integration/cloud-native/meta.json b/content/ja/developer/integration/cloud-native/meta.json new file mode 100644 index 00000000..92c564ac --- /dev/null +++ b/content/ja/developer/integration/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "クラウドネイティブ", + "pages": [ + "cortex", + "flux" + ] +} diff --git a/content/ja/developer/integration/index.md b/content/ja/developer/integration/index.md index 23dd692e..1746d641 100644 --- a/content/ja/developer/integration/index.md +++ b/content/ja/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "RustFS をリバースプロキシ、バックアップツール、データ分析システム、オブザーバビリティプラットフォーム、コンテナレジストリ、DevOps ツールと連携させます。" +description: "RustFS をリバースプロキシ、バックアップツール、データ分析システム、AI とクラウドネイティブプラットフォーム、オブザーバビリティプラットフォーム、コンテナレジストリ、DevOps ツールと連携させます。" --- このセクションでは、**RustFS** を S3 互換 API 経由でインフラストラクチャとアプリケーションプラットフォームに接続します。 @@ -10,7 +10,8 @@ description: "RustFS をリバースプロキシ、バックアップツール - [Reverse Proxy](./reverse-proxy/index.md) は Nginx、Traefik、Caddy、HAProxy を扱います。 - [Backup](./backup/index.md) は Kopia、Longhorn、Restic、Velero を扱います。 - [AI](./ai/index.md) は Ray などの AI プラットフォームを扱います。 -- [データ分析](./big-data/index.md) は ClickHouse、Doris、Hudi、Iceberg、lakeFS、Milvus、OpenDAL、Zeppelin などの分析システムを扱います。 +- [データ分析](./big-data/index.md) は ClickHouse、Doris、Hudi、Iceberg、lakeFS、Milvus、OpenDAL、Vitess、Zeppelin などの分析システムを扱います。 +- [クラウドネイティブ](./cloud-native/index.md) は Cortex、Flux を扱います。 - [オブザーバビリティ](./observability/index.md) は Fluentd、OpenObserve、OpenTelemetry、Thanos、Tempo などのテレメトリシステムを扱います。 - [その他](./others/index.md) はコミュニティ主導の Python 用 capo SDK を扱います。 - [コンテナレジストリ](./registry/index.md) は Harbor を扱います。 diff --git a/content/ja/developer/integration/meta.json b/content/ja/developer/integration/meta.json index a3c989a5..f5a8d87c 100644 --- a/content/ja/developer/integration/meta.json +++ b/content/ja/developer/integration/meta.json @@ -5,6 +5,7 @@ "backup", "big-data", "ai", + "cloud-native", "observability", "others", "registry", diff --git a/content/zh/developer/integration/big-data/images/rustfs-vitess-backups.png b/content/zh/developer/integration/big-data/images/rustfs-vitess-backups.png new file mode 100644 index 00000000..d19b3239 Binary files /dev/null and b/content/zh/developer/integration/big-data/images/rustfs-vitess-backups.png differ diff --git a/content/zh/developer/integration/big-data/index.md b/content/zh/developer/integration/big-data/index.md index fb0d70cf..a8a56f91 100644 --- a/content/zh/developer/integration/big-data/index.md +++ b/content/zh/developer/integration/big-data/index.md @@ -21,5 +21,6 @@ description: "通过 S3 兼容的对象存储接口将数据分析系统连接 - [Spark](./spark.md) - [Flink](./flink.md) - [Trino](./trino.md) +- [Vitess](./vitess.md) 将应用程序数据保存在专用存储桶和前缀中，并使用作用域限定为所需存储桶操作的凭证。 \ No newline at end of file diff --git a/content/zh/developer/integration/big-data/meta.json b/content/zh/developer/integration/big-data/meta.json index f56c8131..f18ae39d 100644 --- a/content/zh/developer/integration/big-data/meta.json +++ b/content/zh/developer/integration/big-data/meta.json @@ -15,6 +15,7 @@ "pyiceberg", "spark", "trino", + "vitess", "zeppelin" ] } diff --git a/content/zh/developer/integration/big-data/vitess.md b/content/zh/developer/integration/big-data/vitess.md new file mode 100644 index 00000000..16a5af99 --- /dev/null +++ b/content/zh/developer/integration/big-data/vitess.md @@ -0,0 +1,189 @@ +--- +title: "Vitess" +description: "通过 S3 备份存储实现，将 Vitess 备份保存到 RustFS。" +--- + +本指南将 MySQL 水平扩展数据库集群系统 [Vitess](https://github.com/vitessio/vitess) 通过其 S3 备份存储实现连接到 **RustFS**。你将在单机上架起一个最小 Vitess 拓扑，对 tablet 执行 `Backup`，确认桶内的备份分块与 `MANIFEST`，并从 RustFS 读回备份列表。整个流程使用 `vitess/lite`（Vitess v25.0.0-SNAPSHOT，2026-09-22 构建）、MySQL 8.4 对 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过；Vitess 20+ 稳定版本使用相同的 flag。 + +你需要安装 Docker。本部署用于本地集成测试，不适用于生产环境。 + +## 架构 + +```mermaid +flowchart LR + Client["vtctldclient"] -->|"Backup"| Tablet["vttablet"] + Tablet -->|"mysqld dump"| Chunks["chunks + MANIFEST"] + Tablet -->|"S3 API"| RustFS["RustFS :9000"] +``` + +`vtctld` 负责触发备份，但实际执行由持有该 tablet 的 `vttablet` 完成：它对 MySQL 做快照，把数据压缩成编号分块，并通过 S3 兼容端点连同 `MANIFEST` 一起上传。 + +## 1. 运行 etcd 与 Vitess 镜像 + +Vitess 将拓扑存储在 etcd 中，`vitess/lite` 镜像自带全部 Vitess 二进制和 MySQL： + +```bash +docker run -d --name etcd --hostname etcd --network oo-rustfs_default \ + quay.io/coreos/etcd:v3.5.17 etcd \ + --advertise-client-urls http://etcd:2379 \ + --listen-client-urls http://0.0.0.0:2379 + +docker run -d --name vtess --hostname vtess --network oo-rustfs_default \ + -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + vitess/lite:latest sleep infinity +``` + +AWS 环境变量为 `vtctld` 和 `vttablet` 中的 S3 备份存储提供凭证。 + +## 2. 初始化 MySQL + +在 Vitess 标准的 tablet 目录中初始化 MySQL 实例，便于后续让 `vttablet` 加载其 `my.cnf`： + +```bash +docker exec vtess sh -c \ + "/vt/bin/mysqlctl --log_dir /tmp/vtlogs init --tablet-dir vt_0000000100" +``` + +socket 文件出现即表示实例就绪： + +```text +/vt/vtdataroot/vt_0000000100/mysql.sock +``` + +## 3. 启动 vtctld 与 vttablet + +带 S3 备份 flag 启动 `vtctld`，注册 cell，再启动 tablet。替换全部连接占位符： + +```bash +docker exec vtess sh -c "nohup /vt/bin/vtctld \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --service-map grpc-vtctl,grpc-vtctld \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --port 15999 --grpc-port 15998 \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vtctld.out 2>&1 &" + +docker exec vtess sh -c \ + "/vt/bin/vtctldclient --server localhost:15998 \ + AddCellInfo --root /vitess/zone1 --server-address etcd:2379 zone1" + +docker exec vtess sh -c "nohup /vt/bin/vttablet \ + --topo-implementation etcd2 \ + --topo-global-server-address etcd:2379 \ + --topo-global-root /vitess/global \ + --tablet-path zone1-0000000100 \ + --init-keyspace commerce --init-shard 0 --init-tablet-type replica \ + --port 15100 --grpc-port 15101 \ + --service-map grpc-queryservice,grpc-tabletmanager,grpc-throttler \ + --mycnf-file /vt/vtdataroot/vt_0000000100/my.cnf \ + --db-dba-user root --db-allprivs-user root --db-app-user root --db-repl-user root \ + --db-dba-use-ssl=false --db-allprivs-use-ssl=false \ + --db-app-use-ssl=false --db-repl-use-ssl=false \ + --backup-storage-implementation s3 \ + --s3-backup-aws-endpoint http://:9000 \ + --s3-backup-aws-region us-east-1 \ + --s3-backup-force-path-style \ + --s3-backup-storage-bucket vitess-backups \ + --s3-backup-storage-root commerce \ + --log_dir /tmp/vtlogs > /tmp/vtlogs/vttablet.out 2>&1 &" +``` + +两个进程都要携带 S3 flag：`vttablet` 执行备份，`vtctld` 负责列举和删除备份。对非 AWS 端点必须加 `--s3-backup-force-path-style`。确认 tablet 已注册： + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetTablets +``` + +```text +zone1-0000000100 commerce 0 replica vtess:15100 vtess:3306 [] +``` + +## 4. 执行备份 + +创建存储桶并触发备份： + +```bash +rc mb rustfs/vitess-backups + +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 \ + Backup zone1-0000000100 +``` + +输出末尾会显示引擎写入清单： + +```text +commerce/0 (zone1-0000000100): ... value:"Completed backing up MANIFEST (attempt 1/2)" +``` + +## 5. 验证 RustFS 中的对象 + +列出备份前缀： + +```bash +rc ls rustfs/vitess-backups/ -r | head -4 +``` + +tablet 上传了压缩数据分块与元数据文件： + +```text +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/0 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/1 +commerce/commerce/0/2026-09-23.011225.zone1-0000000100/MANIFEST +``` + +![存储在 RustFS 控制台中的 Vitess 备份文件](./images/rustfs-vitess-backups.png) + +通过 `vtctld` 从 RustFS 读回备份列表： + +```bash +docker exec vtess /vt/bin/vtctldclient --server localhost:15998 GetBackups commerce/0 +``` + +```text +2026-09-23.011225.zone1-0000000100 +``` + +## 6. 停止或重置 + +保留桶内对象、仅拆除演示环境： + +```bash +docker rm -f vtess etcd +``` + +删除已存储的备份： + +```bash +rc rm rustfs/vitess-backups/ --recursive --force +``` + +## 故障排查 + +### `cannot perform backup without my.cnf` + +传入 `--db-socket` 或 `--db-host` 等连接参数会让 `vttablet` 跳过加载 `my.cnf`，备份随之拒绝执行。应使用 `--mycnf-file` 指向实例配置并让它从那里发现 socket——这正是第 2 步把 MySQL 实例放在 `vt_0000000100` 目录的原因。 + +### `unknown service vtctlservice.Vtctld` + +只有当 service map 包含该服务时，`vtctld` 才会暴露 `vtctldclient` 使用的 API：`--service-map grpc-vtctl,grpc-vtctld`。同时确保 `vtctldclient --server` 指向 gRPC 端口（此处为 `15998`），而不是 Web UI 端口。 + +### `node doesn't exist: /vitess/global/cells/zone1/CellInfo` + +cell 必须先存在，tablet 才能注册。按第 3 步在启动 `vttablet` 之前执行 `vtctldclient AddCellInfo`。 + +### `unknown shorthand flag` 之类的 flag 解析错误 + +Vitess 20+ 会把下划线规范化为连字符，flag 请写成连字符风格（`--tablet-path`）。单连字符的长 flag（如 `-tablet_dir`）会被当作短选项解析并报错。 + +## 下一步 + +- 在启用更多 Vitess 存储选项前，先阅读 [S3 兼容性说明](/administration/protocols/s3)。 +- 使用[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 按照 [Vitess 备份与恢复文档](https://vitess.io/docs/user-guides/configuration-basic/#backups)安排备份计划，并从存储桶恢复 tablet。 diff --git a/content/zh/developer/integration/cloud-native/cortex.md b/content/zh/developer/integration/cloud-native/cortex.md new file mode 100644 index 00000000..660e094b --- /dev/null +++ b/content/zh/developer/integration/cloud-native/cortex.md @@ -0,0 +1,218 @@ +--- +title: "Cortex" +description: "以 RustFS 作为 Cortex 的 S3 兼容 blocks、ruler 与 alertmanager 存储后端。" +--- + +本指南将 [Cortex](https://github.com/cortexproject/cortex)——可水平扩展的 Prometheus 兼容指标后端——通过其原生 S3 存储连接到 **RustFS**。你将以单二进制模式运行 Cortex，把 blocks、ruler 与 alertmanager 存储全部指向一个 RustFS 存储桶，通过 remote write 推送指标，保存一个规则组，并验证桶内对象。整个流程使用 `cortexproject/cortex:v1.21.1` 和 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过。 + +你需要安装 Docker。本部署用于本地集成测试，不适用于生产环境。 + +## 架构 + +```mermaid +flowchart LR + Prom["Prometheus"] -->|"remote write"| Cortex["Cortex :9009"] + Cortex -->|"blocks, rules, configs"| RustFS["RustFS :9000"] +``` + +Cortex 将所有 TSDB 块、ruler 规则组和 alertmanager 配置存放在对象存储中。把 `s3` 后端指向 RustFS 后，该存储桶就是所有租户数据的唯一事实来源。 + +## 1. 创建 Cortex 配置 + +创建配置文件，并替换全部连接占位符： + +```yaml title="cortex.yaml" +target: all +auth_enabled: false + +server: + http_listen_port: 9009 + +distributor: + shard_by_all_labels: true + pool: + health_check_ingesters: true + +ingester: + lifecycler: + min_ready_duration: 0s + final_sleep: 0s + num_tokens: 512 + ring: + kvstore: + store: inmemory + replication_factor: 1 + +blocks_storage: + backend: s3 + s3: &s3 + endpoint: :9000 + region: us-east-1 + bucket_name: cortex + access_key_id: + secret_access_key: + insecure: true + bucket_lookup_type: path + tsdb: + dir: /data/tsdb + block_ranges_period: [15m] + ship_interval: 30s + bucket_store: + sync_dir: /data/tsdb-sync + bucket_index: + enabled: true + +compactor: + data_dir: /data/compactor + sharding_ring: + kvstore: + store: inmemory + +ruler: + enable_api: true + rule_path: /data/ruler + +ruler_storage: + backend: s3 + s3: + <<: *s3 + +alertmanager: + external_url: /alertmanager + enable_api: true + data_dir: /data/alertmanager + +alertmanager_storage: + backend: s3 + s3: + <<: *s3 +``` + +Cortex 的桶字段名为 `bucket_name`，路径风格寻址通过 `bucket_lookup_type: path` 控制。`insecure: true` 允许使用纯 HTTP 端点。`block_ranges_period` 与 `ship_interval` 的取值缩短了切块周期，便于测试快速产出对象；生产环境请保持默认值。 + +## 2. 运行 Cortex + +创建存储桶，并在与 RustFS 同一 Docker 网络中启动 Cortex： + +```bash +rc mb rustfs/cortex + +docker run -d --name cortex --network oo-rustfs_default -p 9009:9009 \ + -v "$PWD/cortex.yaml":/etc/cortex/cortex.yaml:ro \ + -v /opt/cortex/data:/data \ + cortexproject/cortex:v1.21.1 -config.file=/etc/cortex/cortex.yaml +``` + +```text +ts=... caller=cortex.go:469 level=info msg="Cortex started" +``` + +## 3. 推送指标与规则 + +启动一个抓取自身并向 Cortex remote write 的 Prometheus： + +```yaml title="prometheus.yml" +global: + scrape_interval: 5s +scrape_configs: + - job_name: self + static_configs: + - targets: ["localhost:9090"] +remote_write: + - url: http://cortex:9009/api/v1/push +``` + +```bash +docker run -d --name prom-writer --network oo-rustfs_default \ + -v "$PWD/prometheus.yml":/etc/prometheus/prometheus.yml:ro \ + prom/prometheus:latest --config.file=/etc/prometheus/prometheus.yml +``` + +通过 ruler API 保存一个规则组： + +```bash +cat > rules.yaml << "EOF" +name: rustfs-demo +rules: + - alert: RustFSAlwaysFiring + expr: vector(1) + labels: + severity: demo + annotations: + summary: "demo alert stored in RustFS" +EOF + +curl -s -o /dev/null -w "%{http_code}\n" \ + -X POST http://localhost:9009/api/v1/rules/rustfs-demo \ + --data-binary @rules.yaml -H "Content-Type: application/yaml" +``` + +```text +202 +``` + +查询已写入的序列： + +```bash +curl -s "http://localhost:9009/prometheus/api/v1/query?query=up" +``` + +```text +{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up",...},"value":[...,"1"]}]}} +``` + +## 4. 验证 RustFS 中的对象 + +列出桶内前缀： + +```bash +rc ls rustfs/cortex/ -r +``` + +ruler 配置立即落盘，ingester 上传了两个 TSDB 块（每个包含 `chunks`、`index` 与 `meta.json`）： + +```text +fake/01M35WE4B1G6RRKHRV0HSGADY1/chunks/000001 +fake/01M35WE4B1G6RRKHRV0HSGADY1/index +fake/01M35WE4B1G6RRKHRV0HSGADY1/meta.json +fake/01M35WWS2YCC7FBK1JSBKFQGRP/chunks/000001 +fake/01M35WWS2YCC7FBK1JSBKFQGRP/index +fake/01M35WWS2YCC7FBK1JSBKFQGRP/meta.json +rules/fake/cnVzdGZzLWRlbW8=/cnVzdGZzLWRlbW8= +``` + +![存储在 RustFS 控制台中的 Cortex blocks 与规则](./images/rustfs-cortex-blocks.png) + +## 5. 停止或重置 + +保留桶内对象、仅拆除演示环境： + +```bash +docker rm -f cortex prom-writer +``` + +删除已存储的数据： + +```bash +rc rm rustfs/cortex/ --recursive --force +``` + +## 故障排查 + +### `field bucket not found in type s3.Config` + +Cortex 使用 `bucket_name` 而不是 `bucket`。路径风格寻址通过 `bucket_lookup_type: path` 设置——Thanos 使用的 `force_path_style` 键在 Cortex 中不存在。 + +### 块一直没有出现在桶里 + +ingester 只会在 head 达到 `block_ranges_period` 边界（默认 2h）完成压缩后才上传块。测试时可设置较小周期（如 `[15m]`），并配合 `ship_interval: 30s`，等待一个周期即可。 + +### `Unauthorized` 或桶列表为空 + +确认每一个使用该桶的 `s3` 配置块都带有 `access_key_id` 与 `secret_access_key`，且 `insecure: true` 与纯 HTTP 端点相匹配。 + +## 下一步 + +- 在启用更多 Cortex 存储选项前，先阅读 [S3 兼容性说明](/administration/protocols/s3)。 +- 使用[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 如果还需要为 Prometheus 长期数据提供块存储，可以结合 [Thanos](/developer/integration/observability/thanos) 一起使用。 diff --git a/content/zh/developer/integration/cloud-native/flux.md b/content/zh/developer/integration/cloud-native/flux.md new file mode 100644 index 00000000..89fd80cb --- /dev/null +++ b/content/zh/developer/integration/cloud-native/flux.md @@ -0,0 +1,157 @@ +--- +title: "Flux" +description: "通过 Bucket source API，将 RustFS 作为 Flux CD 的 Kubernetes 清单来源。" +--- + +本指南将 Kubernetes 的 GitOps 工具包 [Flux CD](https://github.com/fluxcd/flux2) 通过 source-controller 的 Bucket API 连接到 **RustFS**。你将向 RustFS 存储桶播种 Kubernetes 清单，把该桶注册为 `generic` provider 的 Flux `Bucket` source，并由 Kustomization 应用其中的全部内容。整个流程使用 `flux v2.9.5`（source-controller）在 `k3s v1.36.4+k3s1` 上对 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过。 + +你需要一个已安装 Flux（`flux install`）的 Kubernetes 集群以及 `rc` 客户端。本部署用于本地集成测试，不适用于生产环境。 + +## 架构 + +```mermaid +flowchart LR + Manifests["Kustomize manifests"] -->|"rc cp"| Bucket["RustFS :9000"] + source-controller -->|"S3 API"| Bucket + source-controller -->|"artifact"| kustomize-controller + kustomize-controller -->|"apply"| Cluster["Kubernetes cluster"] +``` + +source-controller 通过 S3 API 列举并下载桶内对象，打包为工件（artifact），kustomize-controller 再应用工件中的清单。RustFS 充当集群拉取式的事实来源。 + +## 1. 播种存储桶 + +向桶内存放一个 Kustomize overlay，并替换全部连接占位符： + +```yaml title="clusters/staging/kustomization.yaml" +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - rustfs-demo-configmap.yaml +``` + +```yaml title="clusters/staging/rustfs-demo-configmap.yaml" +apiVersion: v1 +kind: ConfigMap +metadata: + name: rustfs-flux-demo + namespace: default +data: + storage: rustfs + source: s3-bucket +``` + +上传文件： + +```bash +rc mb rustfs/flux-src +rc cp --recursive ./clusters rustfs/flux-src/ +rc ls rustfs/flux-src/ -r +``` + +```text +staging/kustomization.yaml +staging/rustfs-demo-configmap.yaml +``` + +## 2. 创建凭证 secret + +Flux 从与 source 同命名空间的 secret 中读取凭证，键名必须为小写： + +```bash +kubectl -n flux-system create secret generic rustfs-creds \ + --from-literal=accesskey= \ + --from-literal=secretkey= +``` + +字段名必须是 `accesskey` 和 `secretkey`——写成 `accessKey` 之类的大写形式会以 `AuthenticationFailed` 状态失败。 + +## 3. 注册 Bucket source + +使用 `generic` provider 创建 Bucket source： + +```bash +flux create source bucket rustfs-demo \ + --bucket-name=flux-src \ + --endpoint=:9000 \ + --insecure \ + --secret-ref=rustfs-creds \ + --provider=generic \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ Bucket source reconciliation completed +``` + +如果调和仍在进行，可用 `flux get sources bucket` 查看状态： + +```text +NAME REVISION SUSPENDED READY MESSAGE +rustfs-demo sha256:481816d1 False True stored artifact: revision 'sha256:481816d1' +``` + +`generic` provider 使用路径风格请求，配合 `--insecure` 走纯 HTTP，因此 endpoint 为裸 `host:port`。 + +## 4. 应用清单 + +创建消费该工件的 Kustomization： + +```bash +flux create kustomization rustfs-demo \ + --source=Bucket/rustfs-demo \ + --path="./staging" \ + --prune=true \ + --interval=30s \ + --namespace=flux-system +``` + +```text +✔ applied revision sha256:481816d1 +``` + +确认清单已落到集群： + +```bash +kubectl -n default get configmap rustfs-flux-demo -o jsonpath="{.data}" +``` + +```text +{"source":"s3-bucket","storage":"rustfs"} +``` + +## 5. 停止或重置 + +保留桶内对象、挂起或删除 Flux 资源： + +```bash +flux suspend source bucket rustfs-demo --namespace=flux-system +flux delete kustomization rustfs-demo --namespace=flux-system +``` + +删除桶内内容： + +```bash +rc rm rustfs/flux-src/ --recursive --force +``` + +## 故障排查 + +### `invalid 'rustfs-creds' secret data: required fields 'accesskey' and 'secretkey'` + +secret 的键为小写。用 `accesskey` 和 `secretkey` 作为字面键名重建 secret。 + +### 调和一直无法就绪 + +确认端点可以从集群内部访问——Pod 无法使用 `localhost`，应将 source 指向发布 RustFS 端口的主机 IP 或 Docker 网桥网关（通常为 `172.17.0.1`）。纯 HTTP 必须 `--insecure`。 + +### 凭证有效仍报 `AuthenticationFailed` + +检查 secret 是否与 Bucket source 位于同一命名空间，且键值没有多余空白或引号。 + +## 下一步 + +- 在启用更多 source 类型前，先阅读 [S3 兼容性说明](/administration/protocols/s3)。 +- 使用[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 按照 [Flux Bucket source 文档](https://fluxcd.io/flux/components/source/buckets/)将桶 source 与 Helm release、镜像自动化组合使用。 diff --git a/content/zh/developer/integration/cloud-native/images/rustfs-cortex-blocks.png b/content/zh/developer/integration/cloud-native/images/rustfs-cortex-blocks.png new file mode 100644 index 00000000..f0c73990 Binary files /dev/null and b/content/zh/developer/integration/cloud-native/images/rustfs-cortex-blocks.png differ diff --git a/content/zh/developer/integration/cloud-native/images/rustfs-flux-bucket.png b/content/zh/developer/integration/cloud-native/images/rustfs-flux-bucket.png new file mode 100644 index 00000000..35454e99 Binary files /dev/null and b/content/zh/developer/integration/cloud-native/images/rustfs-flux-bucket.png differ diff --git a/content/zh/developer/integration/cloud-native/index.md b/content/zh/developer/integration/cloud-native/index.md new file mode 100644 index 00000000..3cb7947b --- /dev/null +++ b/content/zh/developer/integration/cloud-native/index.md @@ -0,0 +1,13 @@ +--- +title: "云原生" +description: "通过 S3 兼容的对象存储接口，将云原生平台连接到 RustFS。" +--- + +将 **RustFS** 用作支持 S3 兼容端点的云原生平台的对象存储层。 + +## 平台 + +- [Cortex](./cortex.md) +- [Flux](./flux.md) + +请将平台状态保存在专用的存储桶中，并为凭证仅授予所需桶操作的权限。 diff --git a/content/zh/developer/integration/cloud-native/meta.json b/content/zh/developer/integration/cloud-native/meta.json new file mode 100644 index 00000000..4fd91163 --- /dev/null +++ b/content/zh/developer/integration/cloud-native/meta.json @@ -0,0 +1,7 @@ +{ + "title": "云原生", + "pages": [ + "cortex", + "flux" + ] +} diff --git a/content/zh/developer/integration/index.md b/content/zh/developer/integration/index.md index 24b32f8d..ff68281e 100644 --- a/content/zh/developer/integration/index.md +++ b/content/zh/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "集成" -description: "将 RustFS 与反向代理、备份工具、数据分析系统、可观测性平台、容器镜像仓库和 DevOps 工具集成。" +description: "将 RustFS 与反向代理、备份工具、数据分析系统、AI 与云原生平台、可观测性平台、容器镜像仓库和 DevOps 工具集成。" --- 通过 S3 兼容 API 将 **RustFS** 连接到基础设施和应用平台。 @@ -10,7 +10,8 @@ description: "将 RustFS 与反向代理、备份工具、数据分析系统、 - [反向代理](./reverse-proxy/index.md)涵盖 Nginx、Traefik、Caddy 和 HAProxy。 - [备份](./backup/index.md)涵盖 Kopia、Longhorn、Restic 和 Velero。 - [AI](./ai/index.md)涵盖 Ray 等 AI 平台。 -- [数据分析](./big-data/index.md)涵盖 ClickHouse、Doris、Hudi、Iceberg、lakeFS、Milvus、OpenDAL 和 Zeppelin 等数据分析系统。 +- [数据分析](./big-data/index.md)涵盖 ClickHouse、Doris、Hudi、Iceberg、lakeFS、Milvus、OpenDAL、Vitess 和 Zeppelin 等数据分析系统。 +- [云原生](./cloud-native/index.md)涵盖 Cortex 与 Flux。 - [可观测性](./observability/index.md)涵盖 Fluentd、OpenObserve、OpenTelemetry、Thanos 和 Tempo 等遥测系统。 - [其他](./others/index.md)涵盖社区驱动的 Python capo SDK。 - [镜像仓库](./registry/index.md)涵盖 Harbor。 diff --git a/content/zh/developer/integration/meta.json b/content/zh/developer/integration/meta.json index a3c989a5..f5a8d87c 100644 --- a/content/zh/developer/integration/meta.json +++ b/content/zh/developer/integration/meta.json @@ -5,6 +5,7 @@ "backup", "big-data", "ai", + "cloud-native", "observability", "others", "registry",