diff --git a/content/de/developer/integration/devops/elasticsearch.md b/content/de/developer/integration/devops/elasticsearch.md new file mode 100644 index 00000000..148211db --- /dev/null +++ b/content/de/developer/integration/devops/elasticsearch.md @@ -0,0 +1,262 @@ +--- +title: "Elasticsearch" +description: "Use RustFS as the S3 snapshot repository for Elasticsearch indices, deployed with Docker Compose." +--- + +This guide connects [Elasticsearch](https://github.com/elastic/elasticsearch) — the distributed search and analytics engine — to **RustFS** as the S3 snapshot repository for its indices. You will start Elasticsearch with the `repository-s3` plugin, register a snapshot repository backed by RustFS, index documents, take a snapshot, and prove the full cycle by deleting the index and restoring it from RustFS. The workflow was verified with `docker.elastic.co/elasticsearch/elasticsearch:8.18.0` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker with the Compose plugin. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["Indexing client"] -->|"index documents"| ES["Elasticsearch :9200"] + ES -->|"snapshot blobs"| RustFS["RustFS :9000"] + Init["init-bucket job"] -->|"create my-bucket"| RustFS +``` + +Elasticsearch stores its primary data on local disks and offloads index backups to a snapshot repository. The `repository-s3` plugin writes snapshot metadata and shard data blobs to RustFS through the S3 API with path-style addressing over plain HTTP. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-elasticsearch +cd rustfs-elasticsearch +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the bucket. Do not commit `.env` to source control. + +Elasticsearch splits its S3 settings between `elasticsearch.yml` (non-secret values) and the keystore (credentials). Create the configuration file: + +```yaml title="elasticsearch.yml" +discovery.type: single-node +xpack.security.enabled: false +s3.client.default.endpoint: "rustfs:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +s3.client.default.region: "us-east-1" +``` + +Create the Compose file: + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs-x86-musl:v2.3.1 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: ":9000" + RUSTFS_CONSOLE_ADDRESS: ":9001" + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-sf", "http://127.0.0.1:9000/health"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 10s + networks: + - es + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - es + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.18.0 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + volumes: + - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro + entrypoint: > + bash -c ' + bin/elasticsearch-plugin install --batch repository-s3 && + echo "$${RUSTFS_ACCESS_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.access_key && + echo "$${RUSTFS_SECRET_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.secret_key && + exec bin/elasticsearch' + ports: + - "9200:9200" + depends_on: + create-bucket: + condition: service_completed_successfully + networks: + - es + +networks: + es: + +volumes: + rustfs-data: +``` + +`repository-s3` is a bundled plugin and installs without network access. Only the access key and secret key belong in the keystore — non-secure settings such as the endpoint must live in `elasticsearch.yml`, otherwise the node refuses to start. + +## 2. Start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services and wait for Elasticsearch to finish booting (the first start installs the plugin and creates the keystore entries, which takes a minute or two): + +```bash +docker compose up -d +curl -s http://localhost:9200 +``` + +## 3. Register the snapshot repository and index documents + +Register RustFS as the snapshot repository, create an index, and index five documents: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo" \ + -H "Content-Type: application/json" \ + -d '{"type":"s3","settings":{"bucket":"my-bucket"}}' + +curl -s -X PUT "http://localhost:9200/rustfs_index" \ + -H "Content-Type: application/json" \ + -d '{"mappings":{"properties":{"label":{"type":"keyword"}}}}' + +for v in 1 2 3 4 5; do + curl -s -X POST "http://localhost:9200/rustfs_index/_doc" \ + -H "Content-Type: application/json" \ + -d "{\"label\":\"rustfs-es-doc-$v\",\"value\":$v}" > /dev/null +done +curl -s -X POST "http://localhost:9200/rustfs_index/_refresh" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +## 4. Take a snapshot + +Create a snapshot with `wait_for_completion` so the result is known immediately: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo/snap1?wait_for_completion=true" +``` + +```text +{"snapshot":{"snapshot":"snap1",...,"indices":["rustfs_index"],"shards":{"total":1,"failed":0,"successful":1}}} +``` + +## 5. Verify objects in RustFS + +List the snapshot objects through the bucket-initializer image: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc ls rustfs/my-bucket/indices --recursive' +``` + +Snapshot metadata and shard data blobs live under `indices/` in the bucket: + +```text +[2026-09-21 02:58:01] 3.56 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__LbLsSgUUTpqji_EXvqHYCg +[2026-09-21 02:58:01] 3.21 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__VIuGNJn3RB-RBWvr4tMn7Q +[2026-09-21 02:58:01] 1.00 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/index-e9P8q6E9QhitpkiOdgjVDQ +``` + +You can also browse the prefix in the RustFS Console: + +![Elasticsearch snapshot blobs stored in the RustFS Console](./images/rustfs-es-snapshots.png) + +## 6. Restore the index from RustFS + +Delete the index, then restore it from the snapshot: + +```bash +curl -s -X DELETE "http://localhost:9200/rustfs_index" > /dev/null +curl -s -X POST "http://localhost:9200/_snapshot/rustfs_repo/snap1/_restore?wait_for_completion=true" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +The documents come back because the snapshot blobs were read from RustFS. + +## 7. Stop or reset the stack + +Stop the containers while keeping the RustFS data volume: + +```bash +docker compose down +``` + +To delete the snapshots and start from an empty RustFS volume, explicitly include `--volumes`: + +```bash +docker compose down --volumes +``` + +## Troubleshooting + +### The node refuses to start with "non-secure setting ... must be stored inside elasticsearch.yml" + +Only `s3.client.default.access_key` and `s3.client.default.secret_key` belong in the keystore. Endpoint, protocol, path-style access, and region are non-secure settings and must be defined in `elasticsearch.yml`. + +### Restored or new shards stay unassigned + +Elasticsearch stops allocating shards when disk usage passes the low watermark (85 percent by default). Free disk space, or disable the check for a local test: + +```bash +curl -s -X PUT "http://localhost:9200/_cluster/settings" \ + -H "Content-Type: application/json" \ + -d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +### Restore fails because the index already exists + +A previous failed restore leaves the index behind. Delete it with `DELETE /rustfs_index` and run the restore again. + +### AccessDenied or 403 responses + +Confirm that the credentials in the keystore match the RustFS credentials and that the `create-bucket` service completed successfully: + +```bash +docker compose logs create-bucket +``` + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Elasticsearch snapshot documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) for snapshot lifecycle management (SLM). diff --git a/content/de/developer/integration/devops/images/rustfs-es-snapshots.png b/content/de/developer/integration/devops/images/rustfs-es-snapshots.png new file mode 100644 index 00000000..0cdfe8ca Binary files /dev/null and b/content/de/developer/integration/devops/images/rustfs-es-snapshots.png differ diff --git a/content/de/developer/integration/devops/images/rustfs-terraform-state.png b/content/de/developer/integration/devops/images/rustfs-terraform-state.png new file mode 100644 index 00000000..cce7d4a1 Binary files /dev/null and b/content/de/developer/integration/devops/images/rustfs-terraform-state.png differ diff --git a/content/de/developer/integration/devops/index.md b/content/de/developer/integration/devops/index.md new file mode 100644 index 00000000..78391d8f --- /dev/null +++ b/content/de/developer/integration/devops/index.md @@ -0,0 +1,13 @@ +--- +title: "DevOps" +description: "Verbinden Sie DevOps-Plattformen und Infrastructure-Tooling über S3-kompatible Objektspeicher-Schnittstellen mit RustFS." +--- + +Nutzen Sie **RustFS** als Objektspeicher-Layer für DevOps-Plattformen und Infrastructure-Tooling, die einen S3-kompatiblen Endpunkt unterstützen. + +## Plattformen und Tools + +- [Elasticsearch](./elasticsearch.md) +- [Terraform](./terraform.md) + +Speichern Sie Artefakte, State und Telemetriedaten in dedizierten Buckets und beschränken Sie die Anmeldeinformationen auf die erforderlichen Bucket-Operationen. diff --git a/content/de/developer/integration/devops/meta.json b/content/de/developer/integration/devops/meta.json new file mode 100644 index 00000000..a032acbd --- /dev/null +++ b/content/de/developer/integration/devops/meta.json @@ -0,0 +1,7 @@ +{ + "title": "DevOps", + "pages": [ + "elasticsearch", + "terraform" + ] +} diff --git a/content/de/developer/integration/devops/terraform.md b/content/de/developer/integration/devops/terraform.md new file mode 100644 index 00000000..8744833a --- /dev/null +++ b/content/de/developer/integration/devops/terraform.md @@ -0,0 +1,144 @@ +--- +title: "Terraform" +description: "Store Terraform state in RustFS object storage with the S3 backend, including state locking." +--- + +This guide connects [Terraform](https://github.com/hashicorp/terraform) — the infrastructure as code tool from HashiCorp — to **RustFS** through the S3 backend: the state file, and the lock file that prevents concurrent runs, are stored as objects in RustFS. You will initialize the backend, apply a small configuration, and verify the state object in RustFS. The workflow was verified with `terraform 1.12.2` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a RustFS deployment reachable from your workstation (see the guides under [Installation](/installation)) and Terraform 1.10 or later, which supports S3-native locking via `use_lockfile`. + +## Architecture + +```mermaid +flowchart LR + Client["terraform init / apply"] -->|"state + lock objects"| RustFS["RustFS :9000"] +``` + +The S3 backend stores the state file under the `key` prefix in the bucket. With `use_lockfile = true`, Terraform writes a `.tflock` object while a run holds the lock, replacing the former DynamoDB-based locking for this setup. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-terraform +cd rustfs-terraform +``` + +Create the configuration. Replace the credential placeholders and point the endpoint at your RustFS server — `localhost:9000` when Terraform runs on the same host: + +```hcl title="main.tf" +terraform { + backend "s3" { + bucket = "my-bucket" + key = "rustfs-demo/terraform.tfstate" + region = "us-east-1" + endpoint = "http://localhost:9000" + access_key = "" + secret_key = "" + skip_credentials_validation = true + skip_region_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + force_path_style = true + insecure = true + use_lockfile = true + } +} + +provider "local" {} + +resource "local_file" "demo" { + content = "provisioned with terraform, state stored in RustFS" + filename = "${path.module}/demo.txt" +} +``` + +`force_path_style` and `insecure` select path-style addressing over plain HTTP, which is what RustFS expects for the local endpoint. `use_lockfile` enables S3-native state locking so concurrent runs cannot corrupt the state. + +## 2. Initialize the backend + +Download the provider and configure the S3 backend: + +```bash +terraform init -input=false +``` + +```text +Successfully configured the backend "s3"! Terraform will automatically +use this backend unless the backend configuration changes. +``` + +## 3. Apply the configuration + +Create the resource and write the state to RustFS: + +```bash +terraform apply -auto-approve -input=false +``` + +```text +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +## 4. Verify the state in RustFS + +List the state objects through the `rc` CLI: + +```bash +docker run --rm --network -v "$PWD:/m" \ + --entrypoint /bin/sh rustfs/rc:latest \ + -c 'rc alias set rustfs http://rustfs:9000 >/dev/null && rc ls rustfs/my-bucket/rustfs-demo --recursive' +``` + +```text +[2026-09-21 03:05:03] 1.62 KiB rustfs-demo/terraform.tfstate +``` + +You can also browse the prefix in the RustFS Console: + +![The Terraform state file stored in the RustFS Console](./images/rustfs-terraform-state.png) + +## 5. Confirm the state is reloaded from RustFS + +Run a plan — Terraform reads the state from RustFS and compares it against the configuration: + +```bash +terraform plan -input=false +``` + +```text +No changes. Your infrastructure matches the configuration. +``` + +The plan is clean because the state was read back from RustFS, not from any local file. + +## 6. Destroy and reset + +Tear the resource down; the state update is written to RustFS the same way: + +```bash +terraform destroy -auto-approve -input=false +``` + +Delete the state objects in the RustFS Console (or with `rc rm`) to start from scratch. + +## Troubleshooting + +### "dial tcp: lookup rustfs ... server misbehaving" or connection failures + +The `endpoint` must be reachable from the machine where Terraform runs. Inside a Compose network use `http://rustfs:9000`; from the host use `http://localhost:9000`. + +### "Bucket cannot have ACLs set" or signature errors + +`skip_credentials_validation`, `skip_region_validation`, `skip_metadata_api_check`, `skip_requesting_account_id`, `force_path_style`, and `insecure` are all required for a non-AWS endpoint; missing any of them makes the backend talk to AWS defaults instead of RustFS. + +### The state does not appear in the bucket + +Confirm that the bucket exists and that the credentials match the RustFS credentials. The state object appears under the `key` path (`rustfs-demo/terraform.tfstate` in this guide) after the first `init` or `apply`. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Terraform S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) for options such as workspace prefixes and role assumption. diff --git a/content/de/developer/integration/index.md b/content/de/developer/integration/index.md index d363aff6..31beb5a1 100644 --- a/content/de/developer/integration/index.md +++ b/content/de/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Integrieren Sie RustFS mit Reverse Proxies, Backup-Tools, Datenanalyse-Systemen, Observability-Plattformen und Container-Registries." +description: "Integrieren Sie RustFS mit Reverse Proxies, Backup-Tools, Datenanalyse-Systemen, Observability-Plattformen, Container-Registries und DevOps-Tooling." --- Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. @@ -13,6 +13,6 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Observability](./observability/index.md) covers OpenObserve. - [Others](./others/index.md) covers the community-driven capo SDK for Python. - [Registry](./registry/index.md) covers Harbor. -- [Registry](./registry/index.md) covers Harbor. +- [DevOps](./devops/index.md) covers Elasticsearch and Terraform. Each guide identifies the RustFS endpoint and addressing requirements to use when configuring the integrating system. \ No newline at end of file diff --git a/content/de/developer/integration/meta.json b/content/de/developer/integration/meta.json index 88d4e064..b4b29480 100644 --- a/content/de/developer/integration/meta.json +++ b/content/de/developer/integration/meta.json @@ -6,6 +6,7 @@ "big-data", "observability", "others", - "registry" + "registry", + "devops" ] } diff --git a/content/en/developer/integration/devops/elasticsearch.md b/content/en/developer/integration/devops/elasticsearch.md new file mode 100644 index 00000000..148211db --- /dev/null +++ b/content/en/developer/integration/devops/elasticsearch.md @@ -0,0 +1,262 @@ +--- +title: "Elasticsearch" +description: "Use RustFS as the S3 snapshot repository for Elasticsearch indices, deployed with Docker Compose." +--- + +This guide connects [Elasticsearch](https://github.com/elastic/elasticsearch) — the distributed search and analytics engine — to **RustFS** as the S3 snapshot repository for its indices. You will start Elasticsearch with the `repository-s3` plugin, register a snapshot repository backed by RustFS, index documents, take a snapshot, and prove the full cycle by deleting the index and restoring it from RustFS. The workflow was verified with `docker.elastic.co/elasticsearch/elasticsearch:8.18.0` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker with the Compose plugin. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["Indexing client"] -->|"index documents"| ES["Elasticsearch :9200"] + ES -->|"snapshot blobs"| RustFS["RustFS :9000"] + Init["init-bucket job"] -->|"create my-bucket"| RustFS +``` + +Elasticsearch stores its primary data on local disks and offloads index backups to a snapshot repository. The `repository-s3` plugin writes snapshot metadata and shard data blobs to RustFS through the S3 API with path-style addressing over plain HTTP. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-elasticsearch +cd rustfs-elasticsearch +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the bucket. Do not commit `.env` to source control. + +Elasticsearch splits its S3 settings between `elasticsearch.yml` (non-secret values) and the keystore (credentials). Create the configuration file: + +```yaml title="elasticsearch.yml" +discovery.type: single-node +xpack.security.enabled: false +s3.client.default.endpoint: "rustfs:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +s3.client.default.region: "us-east-1" +``` + +Create the Compose file: + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs-x86-musl:v2.3.1 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: ":9000" + RUSTFS_CONSOLE_ADDRESS: ":9001" + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-sf", "http://127.0.0.1:9000/health"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 10s + networks: + - es + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - es + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.18.0 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + volumes: + - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro + entrypoint: > + bash -c ' + bin/elasticsearch-plugin install --batch repository-s3 && + echo "$${RUSTFS_ACCESS_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.access_key && + echo "$${RUSTFS_SECRET_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.secret_key && + exec bin/elasticsearch' + ports: + - "9200:9200" + depends_on: + create-bucket: + condition: service_completed_successfully + networks: + - es + +networks: + es: + +volumes: + rustfs-data: +``` + +`repository-s3` is a bundled plugin and installs without network access. Only the access key and secret key belong in the keystore — non-secure settings such as the endpoint must live in `elasticsearch.yml`, otherwise the node refuses to start. + +## 2. Start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services and wait for Elasticsearch to finish booting (the first start installs the plugin and creates the keystore entries, which takes a minute or two): + +```bash +docker compose up -d +curl -s http://localhost:9200 +``` + +## 3. Register the snapshot repository and index documents + +Register RustFS as the snapshot repository, create an index, and index five documents: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo" \ + -H "Content-Type: application/json" \ + -d '{"type":"s3","settings":{"bucket":"my-bucket"}}' + +curl -s -X PUT "http://localhost:9200/rustfs_index" \ + -H "Content-Type: application/json" \ + -d '{"mappings":{"properties":{"label":{"type":"keyword"}}}}' + +for v in 1 2 3 4 5; do + curl -s -X POST "http://localhost:9200/rustfs_index/_doc" \ + -H "Content-Type: application/json" \ + -d "{\"label\":\"rustfs-es-doc-$v\",\"value\":$v}" > /dev/null +done +curl -s -X POST "http://localhost:9200/rustfs_index/_refresh" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +## 4. Take a snapshot + +Create a snapshot with `wait_for_completion` so the result is known immediately: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo/snap1?wait_for_completion=true" +``` + +```text +{"snapshot":{"snapshot":"snap1",...,"indices":["rustfs_index"],"shards":{"total":1,"failed":0,"successful":1}}} +``` + +## 5. Verify objects in RustFS + +List the snapshot objects through the bucket-initializer image: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc ls rustfs/my-bucket/indices --recursive' +``` + +Snapshot metadata and shard data blobs live under `indices/` in the bucket: + +```text +[2026-09-21 02:58:01] 3.56 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__LbLsSgUUTpqji_EXvqHYCg +[2026-09-21 02:58:01] 3.21 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__VIuGNJn3RB-RBWvr4tMn7Q +[2026-09-21 02:58:01] 1.00 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/index-e9P8q6E9QhitpkiOdgjVDQ +``` + +You can also browse the prefix in the RustFS Console: + +![Elasticsearch snapshot blobs stored in the RustFS Console](./images/rustfs-es-snapshots.png) + +## 6. Restore the index from RustFS + +Delete the index, then restore it from the snapshot: + +```bash +curl -s -X DELETE "http://localhost:9200/rustfs_index" > /dev/null +curl -s -X POST "http://localhost:9200/_snapshot/rustfs_repo/snap1/_restore?wait_for_completion=true" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +The documents come back because the snapshot blobs were read from RustFS. + +## 7. Stop or reset the stack + +Stop the containers while keeping the RustFS data volume: + +```bash +docker compose down +``` + +To delete the snapshots and start from an empty RustFS volume, explicitly include `--volumes`: + +```bash +docker compose down --volumes +``` + +## Troubleshooting + +### The node refuses to start with "non-secure setting ... must be stored inside elasticsearch.yml" + +Only `s3.client.default.access_key` and `s3.client.default.secret_key` belong in the keystore. Endpoint, protocol, path-style access, and region are non-secure settings and must be defined in `elasticsearch.yml`. + +### Restored or new shards stay unassigned + +Elasticsearch stops allocating shards when disk usage passes the low watermark (85 percent by default). Free disk space, or disable the check for a local test: + +```bash +curl -s -X PUT "http://localhost:9200/_cluster/settings" \ + -H "Content-Type: application/json" \ + -d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +### Restore fails because the index already exists + +A previous failed restore leaves the index behind. Delete it with `DELETE /rustfs_index` and run the restore again. + +### AccessDenied or 403 responses + +Confirm that the credentials in the keystore match the RustFS credentials and that the `create-bucket` service completed successfully: + +```bash +docker compose logs create-bucket +``` + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Elasticsearch snapshot documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) for snapshot lifecycle management (SLM). diff --git a/content/en/developer/integration/devops/images/rustfs-es-snapshots.png b/content/en/developer/integration/devops/images/rustfs-es-snapshots.png new file mode 100644 index 00000000..0cdfe8ca Binary files /dev/null and b/content/en/developer/integration/devops/images/rustfs-es-snapshots.png differ diff --git a/content/en/developer/integration/devops/images/rustfs-terraform-state.png b/content/en/developer/integration/devops/images/rustfs-terraform-state.png new file mode 100644 index 00000000..cce7d4a1 Binary files /dev/null and b/content/en/developer/integration/devops/images/rustfs-terraform-state.png differ diff --git a/content/en/developer/integration/devops/index.md b/content/en/developer/integration/devops/index.md new file mode 100644 index 00000000..a4a85574 --- /dev/null +++ b/content/en/developer/integration/devops/index.md @@ -0,0 +1,13 @@ +--- +title: "DevOps" +description: "Connect DevOps platforms and infrastructure tooling to RustFS through S3-compatible object storage interfaces." +--- + +Use **RustFS** as the object storage layer for DevOps platforms and infrastructure tooling. + +## Platforms + +- [Elasticsearch](./elasticsearch.md) +- [Terraform](./terraform.md) + +Keep artifacts, state, and telemetry in dedicated buckets, and use credentials scoped to the required bucket operations. diff --git a/content/en/developer/integration/devops/meta.json b/content/en/developer/integration/devops/meta.json new file mode 100644 index 00000000..a032acbd --- /dev/null +++ b/content/en/developer/integration/devops/meta.json @@ -0,0 +1,7 @@ +{ + "title": "DevOps", + "pages": [ + "elasticsearch", + "terraform" + ] +} diff --git a/content/en/developer/integration/devops/terraform.md b/content/en/developer/integration/devops/terraform.md new file mode 100644 index 00000000..8744833a --- /dev/null +++ b/content/en/developer/integration/devops/terraform.md @@ -0,0 +1,144 @@ +--- +title: "Terraform" +description: "Store Terraform state in RustFS object storage with the S3 backend, including state locking." +--- + +This guide connects [Terraform](https://github.com/hashicorp/terraform) — the infrastructure as code tool from HashiCorp — to **RustFS** through the S3 backend: the state file, and the lock file that prevents concurrent runs, are stored as objects in RustFS. You will initialize the backend, apply a small configuration, and verify the state object in RustFS. The workflow was verified with `terraform 1.12.2` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a RustFS deployment reachable from your workstation (see the guides under [Installation](/installation)) and Terraform 1.10 or later, which supports S3-native locking via `use_lockfile`. + +## Architecture + +```mermaid +flowchart LR + Client["terraform init / apply"] -->|"state + lock objects"| RustFS["RustFS :9000"] +``` + +The S3 backend stores the state file under the `key` prefix in the bucket. With `use_lockfile = true`, Terraform writes a `.tflock` object while a run holds the lock, replacing the former DynamoDB-based locking for this setup. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-terraform +cd rustfs-terraform +``` + +Create the configuration. Replace the credential placeholders and point the endpoint at your RustFS server — `localhost:9000` when Terraform runs on the same host: + +```hcl title="main.tf" +terraform { + backend "s3" { + bucket = "my-bucket" + key = "rustfs-demo/terraform.tfstate" + region = "us-east-1" + endpoint = "http://localhost:9000" + access_key = "" + secret_key = "" + skip_credentials_validation = true + skip_region_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + force_path_style = true + insecure = true + use_lockfile = true + } +} + +provider "local" {} + +resource "local_file" "demo" { + content = "provisioned with terraform, state stored in RustFS" + filename = "${path.module}/demo.txt" +} +``` + +`force_path_style` and `insecure` select path-style addressing over plain HTTP, which is what RustFS expects for the local endpoint. `use_lockfile` enables S3-native state locking so concurrent runs cannot corrupt the state. + +## 2. Initialize the backend + +Download the provider and configure the S3 backend: + +```bash +terraform init -input=false +``` + +```text +Successfully configured the backend "s3"! Terraform will automatically +use this backend unless the backend configuration changes. +``` + +## 3. Apply the configuration + +Create the resource and write the state to RustFS: + +```bash +terraform apply -auto-approve -input=false +``` + +```text +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +## 4. Verify the state in RustFS + +List the state objects through the `rc` CLI: + +```bash +docker run --rm --network -v "$PWD:/m" \ + --entrypoint /bin/sh rustfs/rc:latest \ + -c 'rc alias set rustfs http://rustfs:9000 >/dev/null && rc ls rustfs/my-bucket/rustfs-demo --recursive' +``` + +```text +[2026-09-21 03:05:03] 1.62 KiB rustfs-demo/terraform.tfstate +``` + +You can also browse the prefix in the RustFS Console: + +![The Terraform state file stored in the RustFS Console](./images/rustfs-terraform-state.png) + +## 5. Confirm the state is reloaded from RustFS + +Run a plan — Terraform reads the state from RustFS and compares it against the configuration: + +```bash +terraform plan -input=false +``` + +```text +No changes. Your infrastructure matches the configuration. +``` + +The plan is clean because the state was read back from RustFS, not from any local file. + +## 6. Destroy and reset + +Tear the resource down; the state update is written to RustFS the same way: + +```bash +terraform destroy -auto-approve -input=false +``` + +Delete the state objects in the RustFS Console (or with `rc rm`) to start from scratch. + +## Troubleshooting + +### "dial tcp: lookup rustfs ... server misbehaving" or connection failures + +The `endpoint` must be reachable from the machine where Terraform runs. Inside a Compose network use `http://rustfs:9000`; from the host use `http://localhost:9000`. + +### "Bucket cannot have ACLs set" or signature errors + +`skip_credentials_validation`, `skip_region_validation`, `skip_metadata_api_check`, `skip_requesting_account_id`, `force_path_style`, and `insecure` are all required for a non-AWS endpoint; missing any of them makes the backend talk to AWS defaults instead of RustFS. + +### The state does not appear in the bucket + +Confirm that the bucket exists and that the credentials match the RustFS credentials. The state object appears under the `key` path (`rustfs-demo/terraform.tfstate` in this guide) after the first `init` or `apply`. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Terraform S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) for options such as workspace prefixes and role assumption. diff --git a/content/en/developer/integration/index.md b/content/en/developer/integration/index.md index d15bad08..dadfe77b 100644 --- a/content/en/developer/integration/index.md +++ b/content/en/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Integrate RustFS with reverse proxies, backup tools, data analytics systems, observability platforms, and container registries." +description: "Integrate RustFS with reverse proxies, backup tools, data analytics systems, observability platforms, container registries, and DevOps tooling." --- Use this section to connect **RustFS** to infrastructure and application platforms through its S3-compatible API. @@ -13,6 +13,6 @@ Use this section to connect **RustFS** to infrastructure and application platfor - [Observability](./observability/index.md) covers OpenObserve. - [Others](./others/index.md) covers the community-driven capo SDK for Python. - [Registry](./registry/index.md) covers Harbor. -- [Registry](./registry/index.md) covers Harbor. +- [DevOps](./devops/index.md) covers Elasticsearch and Terraform. Each guide identifies the RustFS endpoint and addressing requirements to use when configuring the integrating system. \ No newline at end of file diff --git a/content/en/developer/integration/meta.json b/content/en/developer/integration/meta.json index 88d4e064..b4b29480 100644 --- a/content/en/developer/integration/meta.json +++ b/content/en/developer/integration/meta.json @@ -6,6 +6,7 @@ "big-data", "observability", "others", - "registry" + "registry", + "devops" ] } diff --git a/content/fr/developer/integration/devops/elasticsearch.md b/content/fr/developer/integration/devops/elasticsearch.md new file mode 100644 index 00000000..148211db --- /dev/null +++ b/content/fr/developer/integration/devops/elasticsearch.md @@ -0,0 +1,262 @@ +--- +title: "Elasticsearch" +description: "Use RustFS as the S3 snapshot repository for Elasticsearch indices, deployed with Docker Compose." +--- + +This guide connects [Elasticsearch](https://github.com/elastic/elasticsearch) — the distributed search and analytics engine — to **RustFS** as the S3 snapshot repository for its indices. You will start Elasticsearch with the `repository-s3` plugin, register a snapshot repository backed by RustFS, index documents, take a snapshot, and prove the full cycle by deleting the index and restoring it from RustFS. The workflow was verified with `docker.elastic.co/elasticsearch/elasticsearch:8.18.0` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker with the Compose plugin. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["Indexing client"] -->|"index documents"| ES["Elasticsearch :9200"] + ES -->|"snapshot blobs"| RustFS["RustFS :9000"] + Init["init-bucket job"] -->|"create my-bucket"| RustFS +``` + +Elasticsearch stores its primary data on local disks and offloads index backups to a snapshot repository. The `repository-s3` plugin writes snapshot metadata and shard data blobs to RustFS through the S3 API with path-style addressing over plain HTTP. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-elasticsearch +cd rustfs-elasticsearch +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the bucket. Do not commit `.env` to source control. + +Elasticsearch splits its S3 settings between `elasticsearch.yml` (non-secret values) and the keystore (credentials). Create the configuration file: + +```yaml title="elasticsearch.yml" +discovery.type: single-node +xpack.security.enabled: false +s3.client.default.endpoint: "rustfs:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +s3.client.default.region: "us-east-1" +``` + +Create the Compose file: + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs-x86-musl:v2.3.1 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: ":9000" + RUSTFS_CONSOLE_ADDRESS: ":9001" + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-sf", "http://127.0.0.1:9000/health"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 10s + networks: + - es + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - es + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.18.0 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + volumes: + - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro + entrypoint: > + bash -c ' + bin/elasticsearch-plugin install --batch repository-s3 && + echo "$${RUSTFS_ACCESS_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.access_key && + echo "$${RUSTFS_SECRET_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.secret_key && + exec bin/elasticsearch' + ports: + - "9200:9200" + depends_on: + create-bucket: + condition: service_completed_successfully + networks: + - es + +networks: + es: + +volumes: + rustfs-data: +``` + +`repository-s3` is a bundled plugin and installs without network access. Only the access key and secret key belong in the keystore — non-secure settings such as the endpoint must live in `elasticsearch.yml`, otherwise the node refuses to start. + +## 2. Start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services and wait for Elasticsearch to finish booting (the first start installs the plugin and creates the keystore entries, which takes a minute or two): + +```bash +docker compose up -d +curl -s http://localhost:9200 +``` + +## 3. Register the snapshot repository and index documents + +Register RustFS as the snapshot repository, create an index, and index five documents: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo" \ + -H "Content-Type: application/json" \ + -d '{"type":"s3","settings":{"bucket":"my-bucket"}}' + +curl -s -X PUT "http://localhost:9200/rustfs_index" \ + -H "Content-Type: application/json" \ + -d '{"mappings":{"properties":{"label":{"type":"keyword"}}}}' + +for v in 1 2 3 4 5; do + curl -s -X POST "http://localhost:9200/rustfs_index/_doc" \ + -H "Content-Type: application/json" \ + -d "{\"label\":\"rustfs-es-doc-$v\",\"value\":$v}" > /dev/null +done +curl -s -X POST "http://localhost:9200/rustfs_index/_refresh" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +## 4. Take a snapshot + +Create a snapshot with `wait_for_completion` so the result is known immediately: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo/snap1?wait_for_completion=true" +``` + +```text +{"snapshot":{"snapshot":"snap1",...,"indices":["rustfs_index"],"shards":{"total":1,"failed":0,"successful":1}}} +``` + +## 5. Verify objects in RustFS + +List the snapshot objects through the bucket-initializer image: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc ls rustfs/my-bucket/indices --recursive' +``` + +Snapshot metadata and shard data blobs live under `indices/` in the bucket: + +```text +[2026-09-21 02:58:01] 3.56 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__LbLsSgUUTpqji_EXvqHYCg +[2026-09-21 02:58:01] 3.21 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__VIuGNJn3RB-RBWvr4tMn7Q +[2026-09-21 02:58:01] 1.00 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/index-e9P8q6E9QhitpkiOdgjVDQ +``` + +You can also browse the prefix in the RustFS Console: + +![Elasticsearch snapshot blobs stored in the RustFS Console](./images/rustfs-es-snapshots.png) + +## 6. Restore the index from RustFS + +Delete the index, then restore it from the snapshot: + +```bash +curl -s -X DELETE "http://localhost:9200/rustfs_index" > /dev/null +curl -s -X POST "http://localhost:9200/_snapshot/rustfs_repo/snap1/_restore?wait_for_completion=true" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +The documents come back because the snapshot blobs were read from RustFS. + +## 7. Stop or reset the stack + +Stop the containers while keeping the RustFS data volume: + +```bash +docker compose down +``` + +To delete the snapshots and start from an empty RustFS volume, explicitly include `--volumes`: + +```bash +docker compose down --volumes +``` + +## Troubleshooting + +### The node refuses to start with "non-secure setting ... must be stored inside elasticsearch.yml" + +Only `s3.client.default.access_key` and `s3.client.default.secret_key` belong in the keystore. Endpoint, protocol, path-style access, and region are non-secure settings and must be defined in `elasticsearch.yml`. + +### Restored or new shards stay unassigned + +Elasticsearch stops allocating shards when disk usage passes the low watermark (85 percent by default). Free disk space, or disable the check for a local test: + +```bash +curl -s -X PUT "http://localhost:9200/_cluster/settings" \ + -H "Content-Type: application/json" \ + -d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +### Restore fails because the index already exists + +A previous failed restore leaves the index behind. Delete it with `DELETE /rustfs_index` and run the restore again. + +### AccessDenied or 403 responses + +Confirm that the credentials in the keystore match the RustFS credentials and that the `create-bucket` service completed successfully: + +```bash +docker compose logs create-bucket +``` + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Elasticsearch snapshot documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) for snapshot lifecycle management (SLM). diff --git a/content/fr/developer/integration/devops/images/rustfs-es-snapshots.png b/content/fr/developer/integration/devops/images/rustfs-es-snapshots.png new file mode 100644 index 00000000..0cdfe8ca Binary files /dev/null and b/content/fr/developer/integration/devops/images/rustfs-es-snapshots.png differ diff --git a/content/fr/developer/integration/devops/images/rustfs-terraform-state.png b/content/fr/developer/integration/devops/images/rustfs-terraform-state.png new file mode 100644 index 00000000..cce7d4a1 Binary files /dev/null and b/content/fr/developer/integration/devops/images/rustfs-terraform-state.png differ diff --git a/content/fr/developer/integration/devops/index.md b/content/fr/developer/integration/devops/index.md new file mode 100644 index 00000000..1b4d0f17 --- /dev/null +++ b/content/fr/developer/integration/devops/index.md @@ -0,0 +1,13 @@ +--- +title: "DevOps" +description: "Connectez les plateformes DevOps et l'outillage d'infrastructure à RustFS via des interfaces de stockage objet compatibles S3." +--- + +Utilisez **RustFS** comme couche de stockage objet pour les plateformes DevOps et l'outillage d'infrastructure qui prennent en charge un point de terminaison compatible S3. + +## Plateformes et outils + +- [Elasticsearch](./elasticsearch.md) +- [Terraform](./terraform.md) + +Conservez les artefacts, l'état et les données de télémétrie dans des buckets dédiés et limitez les identifiants aux opérations de bucket requises. diff --git a/content/fr/developer/integration/devops/meta.json b/content/fr/developer/integration/devops/meta.json new file mode 100644 index 00000000..a032acbd --- /dev/null +++ b/content/fr/developer/integration/devops/meta.json @@ -0,0 +1,7 @@ +{ + "title": "DevOps", + "pages": [ + "elasticsearch", + "terraform" + ] +} diff --git a/content/fr/developer/integration/devops/terraform.md b/content/fr/developer/integration/devops/terraform.md new file mode 100644 index 00000000..8744833a --- /dev/null +++ b/content/fr/developer/integration/devops/terraform.md @@ -0,0 +1,144 @@ +--- +title: "Terraform" +description: "Store Terraform state in RustFS object storage with the S3 backend, including state locking." +--- + +This guide connects [Terraform](https://github.com/hashicorp/terraform) — the infrastructure as code tool from HashiCorp — to **RustFS** through the S3 backend: the state file, and the lock file that prevents concurrent runs, are stored as objects in RustFS. You will initialize the backend, apply a small configuration, and verify the state object in RustFS. The workflow was verified with `terraform 1.12.2` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a RustFS deployment reachable from your workstation (see the guides under [Installation](/installation)) and Terraform 1.10 or later, which supports S3-native locking via `use_lockfile`. + +## Architecture + +```mermaid +flowchart LR + Client["terraform init / apply"] -->|"state + lock objects"| RustFS["RustFS :9000"] +``` + +The S3 backend stores the state file under the `key` prefix in the bucket. With `use_lockfile = true`, Terraform writes a `.tflock` object while a run holds the lock, replacing the former DynamoDB-based locking for this setup. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-terraform +cd rustfs-terraform +``` + +Create the configuration. Replace the credential placeholders and point the endpoint at your RustFS server — `localhost:9000` when Terraform runs on the same host: + +```hcl title="main.tf" +terraform { + backend "s3" { + bucket = "my-bucket" + key = "rustfs-demo/terraform.tfstate" + region = "us-east-1" + endpoint = "http://localhost:9000" + access_key = "" + secret_key = "" + skip_credentials_validation = true + skip_region_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + force_path_style = true + insecure = true + use_lockfile = true + } +} + +provider "local" {} + +resource "local_file" "demo" { + content = "provisioned with terraform, state stored in RustFS" + filename = "${path.module}/demo.txt" +} +``` + +`force_path_style` and `insecure` select path-style addressing over plain HTTP, which is what RustFS expects for the local endpoint. `use_lockfile` enables S3-native state locking so concurrent runs cannot corrupt the state. + +## 2. Initialize the backend + +Download the provider and configure the S3 backend: + +```bash +terraform init -input=false +``` + +```text +Successfully configured the backend "s3"! Terraform will automatically +use this backend unless the backend configuration changes. +``` + +## 3. Apply the configuration + +Create the resource and write the state to RustFS: + +```bash +terraform apply -auto-approve -input=false +``` + +```text +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +## 4. Verify the state in RustFS + +List the state objects through the `rc` CLI: + +```bash +docker run --rm --network -v "$PWD:/m" \ + --entrypoint /bin/sh rustfs/rc:latest \ + -c 'rc alias set rustfs http://rustfs:9000 >/dev/null && rc ls rustfs/my-bucket/rustfs-demo --recursive' +``` + +```text +[2026-09-21 03:05:03] 1.62 KiB rustfs-demo/terraform.tfstate +``` + +You can also browse the prefix in the RustFS Console: + +![The Terraform state file stored in the RustFS Console](./images/rustfs-terraform-state.png) + +## 5. Confirm the state is reloaded from RustFS + +Run a plan — Terraform reads the state from RustFS and compares it against the configuration: + +```bash +terraform plan -input=false +``` + +```text +No changes. Your infrastructure matches the configuration. +``` + +The plan is clean because the state was read back from RustFS, not from any local file. + +## 6. Destroy and reset + +Tear the resource down; the state update is written to RustFS the same way: + +```bash +terraform destroy -auto-approve -input=false +``` + +Delete the state objects in the RustFS Console (or with `rc rm`) to start from scratch. + +## Troubleshooting + +### "dial tcp: lookup rustfs ... server misbehaving" or connection failures + +The `endpoint` must be reachable from the machine where Terraform runs. Inside a Compose network use `http://rustfs:9000`; from the host use `http://localhost:9000`. + +### "Bucket cannot have ACLs set" or signature errors + +`skip_credentials_validation`, `skip_region_validation`, `skip_metadata_api_check`, `skip_requesting_account_id`, `force_path_style`, and `insecure` are all required for a non-AWS endpoint; missing any of them makes the backend talk to AWS defaults instead of RustFS. + +### The state does not appear in the bucket + +Confirm that the bucket exists and that the credentials match the RustFS credentials. The state object appears under the `key` path (`rustfs-demo/terraform.tfstate` in this guide) after the first `init` or `apply`. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Terraform S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) for options such as workspace prefixes and role assumption. diff --git a/content/fr/developer/integration/index.md b/content/fr/developer/integration/index.md index 03cb4e9e..986f76de 100644 --- a/content/fr/developer/integration/index.md +++ b/content/fr/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "Intégrez RustFS avec des reverse proxies, des outils de sauvegarde, des systèmes d'analyse de données, des plateformes d'observabilité et des registries de conteneurs." +description: "Intégrez RustFS avec des reverse proxies, des outils de sauvegarde, des systèmes d'analyse de données, des plateformes d'observabilité, des registries de conteneurs et des outils DevOps." --- Utilisez cette section pour connecter **RustFS** à des plateformes d'infrastructure et d'applications via son API compatible S3. @@ -13,6 +13,6 @@ Utilisez cette section pour connecter **RustFS** à des plateformes d'infrastruc - [Observabilité](./observability/index.md) couvre OpenObserve. - [Autres](./others/index.md) couvre le SDK communautaire capo pour Python. - [Registre](./registry/index.md) couvre Harbor. -- [Registre](./registry/index.md) couvre Harbor. +- [DevOps](./devops/index.md) couvre Elasticsearch et Terraform. Chaque guide indique le point de terminaison RustFS et les exigences d'adressage à utiliser lors de la configuration du système intégré. \ No newline at end of file diff --git a/content/fr/developer/integration/meta.json b/content/fr/developer/integration/meta.json index 88d4e064..b4b29480 100644 --- a/content/fr/developer/integration/meta.json +++ b/content/fr/developer/integration/meta.json @@ -6,6 +6,7 @@ "big-data", "observability", "others", - "registry" + "registry", + "devops" ] } diff --git a/content/ja/developer/integration/devops/elasticsearch.md b/content/ja/developer/integration/devops/elasticsearch.md new file mode 100644 index 00000000..148211db --- /dev/null +++ b/content/ja/developer/integration/devops/elasticsearch.md @@ -0,0 +1,262 @@ +--- +title: "Elasticsearch" +description: "Use RustFS as the S3 snapshot repository for Elasticsearch indices, deployed with Docker Compose." +--- + +This guide connects [Elasticsearch](https://github.com/elastic/elasticsearch) — the distributed search and analytics engine — to **RustFS** as the S3 snapshot repository for its indices. You will start Elasticsearch with the `repository-s3` plugin, register a snapshot repository backed by RustFS, index documents, take a snapshot, and prove the full cycle by deleting the index and restoring it from RustFS. The workflow was verified with `docker.elastic.co/elasticsearch/elasticsearch:8.18.0` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need Docker with the Compose plugin. This deployment is intended for local integration testing, not production. + +## Architecture + +```mermaid +flowchart LR + Client["Indexing client"] -->|"index documents"| ES["Elasticsearch :9200"] + ES -->|"snapshot blobs"| RustFS["RustFS :9000"] + Init["init-bucket job"] -->|"create my-bucket"| RustFS +``` + +Elasticsearch stores its primary data on local disks and offloads index backups to a snapshot repository. The `repository-s3` plugin writes snapshot metadata and shard data blobs to RustFS through the S3 API with path-style addressing over plain HTTP. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-elasticsearch +cd rustfs-elasticsearch +``` + +Create an environment file and replace both credential placeholders: + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +Use dedicated credentials for the bucket. Do not commit `.env` to source control. + +Elasticsearch splits its S3 settings between `elasticsearch.yml` (non-secret values) and the keystore (credentials). Create the configuration file: + +```yaml title="elasticsearch.yml" +discovery.type: single-node +xpack.security.enabled: false +s3.client.default.endpoint: "rustfs:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +s3.client.default.region: "us-east-1" +``` + +Create the Compose file: + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs-x86-musl:v2.3.1 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: ":9000" + RUSTFS_CONSOLE_ADDRESS: ":9001" + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-sf", "http://127.0.0.1:9000/health"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 10s + networks: + - es + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - es + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.18.0 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + volumes: + - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro + entrypoint: > + bash -c ' + bin/elasticsearch-plugin install --batch repository-s3 && + echo "$${RUSTFS_ACCESS_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.access_key && + echo "$${RUSTFS_SECRET_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.secret_key && + exec bin/elasticsearch' + ports: + - "9200:9200" + depends_on: + create-bucket: + condition: service_completed_successfully + networks: + - es + +networks: + es: + +volumes: + rustfs-data: +``` + +`repository-s3` is a bundled plugin and installs without network access. Only the access key and secret key belong in the keystore — non-secure settings such as the endpoint must live in `elasticsearch.yml`, otherwise the node refuses to start. + +## 2. Start the deployment + +Resolve the Compose file before starting containers: + +```bash +docker compose config +``` + +Start the services and wait for Elasticsearch to finish booting (the first start installs the plugin and creates the keystore entries, which takes a minute or two): + +```bash +docker compose up -d +curl -s http://localhost:9200 +``` + +## 3. Register the snapshot repository and index documents + +Register RustFS as the snapshot repository, create an index, and index five documents: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo" \ + -H "Content-Type: application/json" \ + -d '{"type":"s3","settings":{"bucket":"my-bucket"}}' + +curl -s -X PUT "http://localhost:9200/rustfs_index" \ + -H "Content-Type: application/json" \ + -d '{"mappings":{"properties":{"label":{"type":"keyword"}}}}' + +for v in 1 2 3 4 5; do + curl -s -X POST "http://localhost:9200/rustfs_index/_doc" \ + -H "Content-Type: application/json" \ + -d "{\"label\":\"rustfs-es-doc-$v\",\"value\":$v}" > /dev/null +done +curl -s -X POST "http://localhost:9200/rustfs_index/_refresh" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +## 4. Take a snapshot + +Create a snapshot with `wait_for_completion` so the result is known immediately: + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo/snap1?wait_for_completion=true" +``` + +```text +{"snapshot":{"snapshot":"snap1",...,"indices":["rustfs_index"],"shards":{"total":1,"failed":0,"successful":1}}} +``` + +## 5. Verify objects in RustFS + +List the snapshot objects through the bucket-initializer image: + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc ls rustfs/my-bucket/indices --recursive' +``` + +Snapshot metadata and shard data blobs live under `indices/` in the bucket: + +```text +[2026-09-21 02:58:01] 3.56 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__LbLsSgUUTpqji_EXvqHYCg +[2026-09-21 02:58:01] 3.21 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__VIuGNJn3RB-RBWvr4tMn7Q +[2026-09-21 02:58:01] 1.00 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/index-e9P8q6E9QhitpkiOdgjVDQ +``` + +You can also browse the prefix in the RustFS Console: + +![Elasticsearch snapshot blobs stored in the RustFS Console](./images/rustfs-es-snapshots.png) + +## 6. Restore the index from RustFS + +Delete the index, then restore it from the snapshot: + +```bash +curl -s -X DELETE "http://localhost:9200/rustfs_index" > /dev/null +curl -s -X POST "http://localhost:9200/_snapshot/rustfs_repo/snap1/_restore?wait_for_completion=true" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +The documents come back because the snapshot blobs were read from RustFS. + +## 7. Stop or reset the stack + +Stop the containers while keeping the RustFS data volume: + +```bash +docker compose down +``` + +To delete the snapshots and start from an empty RustFS volume, explicitly include `--volumes`: + +```bash +docker compose down --volumes +``` + +## Troubleshooting + +### The node refuses to start with "non-secure setting ... must be stored inside elasticsearch.yml" + +Only `s3.client.default.access_key` and `s3.client.default.secret_key` belong in the keystore. Endpoint, protocol, path-style access, and region are non-secure settings and must be defined in `elasticsearch.yml`. + +### Restored or new shards stay unassigned + +Elasticsearch stops allocating shards when disk usage passes the low watermark (85 percent by default). Free disk space, or disable the check for a local test: + +```bash +curl -s -X PUT "http://localhost:9200/_cluster/settings" \ + -H "Content-Type: application/json" \ + -d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +### Restore fails because the index already exists + +A previous failed restore leaves the index behind. Delete it with `DELETE /rustfs_index` and run the restore again. + +### AccessDenied or 403 responses + +Confirm that the credentials in the keystore match the RustFS credentials and that the `create-bucket` service completed successfully: + +```bash +docker compose logs create-bucket +``` + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Elasticsearch snapshot documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html) for snapshot lifecycle management (SLM). diff --git a/content/ja/developer/integration/devops/images/rustfs-es-snapshots.png b/content/ja/developer/integration/devops/images/rustfs-es-snapshots.png new file mode 100644 index 00000000..0cdfe8ca Binary files /dev/null and b/content/ja/developer/integration/devops/images/rustfs-es-snapshots.png differ diff --git a/content/ja/developer/integration/devops/images/rustfs-terraform-state.png b/content/ja/developer/integration/devops/images/rustfs-terraform-state.png new file mode 100644 index 00000000..cce7d4a1 Binary files /dev/null and b/content/ja/developer/integration/devops/images/rustfs-terraform-state.png differ diff --git a/content/ja/developer/integration/devops/index.md b/content/ja/developer/integration/devops/index.md new file mode 100644 index 00000000..c77f6f04 --- /dev/null +++ b/content/ja/developer/integration/devops/index.md @@ -0,0 +1,13 @@ +--- +title: "DevOps" +description: "S3 互換オブジェクトストレージインターフェースを経由して、DevOps プラットフォームとインフラストラクチャツールを RustFS に接続します。" +--- + +S3 互換エンドポイントをサポートする DevOps プラットフォームおよびインフラストラクチャツールのオブジェクトストレージ層として **RustFS** を使用します。 + +## プラットフォームとツール + +- [Elasticsearch](./elasticsearch.md) +- [Terraform](./terraform.md) + +アーティファクト、ステート、テレメトリデータは専用バケットに保存し、必要なバケット操作のみに権限が絞られた認証情報を使用してください。 diff --git a/content/ja/developer/integration/devops/meta.json b/content/ja/developer/integration/devops/meta.json new file mode 100644 index 00000000..a032acbd --- /dev/null +++ b/content/ja/developer/integration/devops/meta.json @@ -0,0 +1,7 @@ +{ + "title": "DevOps", + "pages": [ + "elasticsearch", + "terraform" + ] +} diff --git a/content/ja/developer/integration/devops/terraform.md b/content/ja/developer/integration/devops/terraform.md new file mode 100644 index 00000000..8744833a --- /dev/null +++ b/content/ja/developer/integration/devops/terraform.md @@ -0,0 +1,144 @@ +--- +title: "Terraform" +description: "Store Terraform state in RustFS object storage with the S3 backend, including state locking." +--- + +This guide connects [Terraform](https://github.com/hashicorp/terraform) — the infrastructure as code tool from HashiCorp — to **RustFS** through the S3 backend: the state file, and the lock file that prevents concurrent runs, are stored as objects in RustFS. You will initialize the backend, apply a small configuration, and verify the state object in RustFS. The workflow was verified with `terraform 1.12.2` and `rustfs/rustfs-x86-musl:v2.3.1`. + +You need a RustFS deployment reachable from your workstation (see the guides under [Installation](/installation)) and Terraform 1.10 or later, which supports S3-native locking via `use_lockfile`. + +## Architecture + +```mermaid +flowchart LR + Client["terraform init / apply"] -->|"state + lock objects"| RustFS["RustFS :9000"] +``` + +The S3 backend stores the state file under the `key` prefix in the bucket. With `use_lockfile = true`, Terraform writes a `.tflock` object while a run holds the lock, replacing the former DynamoDB-based locking for this setup. + +## 1. Create the project files + +Create a working directory: + +```bash +mkdir rustfs-terraform +cd rustfs-terraform +``` + +Create the configuration. Replace the credential placeholders and point the endpoint at your RustFS server — `localhost:9000` when Terraform runs on the same host: + +```hcl title="main.tf" +terraform { + backend "s3" { + bucket = "my-bucket" + key = "rustfs-demo/terraform.tfstate" + region = "us-east-1" + endpoint = "http://localhost:9000" + access_key = "" + secret_key = "" + skip_credentials_validation = true + skip_region_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + force_path_style = true + insecure = true + use_lockfile = true + } +} + +provider "local" {} + +resource "local_file" "demo" { + content = "provisioned with terraform, state stored in RustFS" + filename = "${path.module}/demo.txt" +} +``` + +`force_path_style` and `insecure` select path-style addressing over plain HTTP, which is what RustFS expects for the local endpoint. `use_lockfile` enables S3-native state locking so concurrent runs cannot corrupt the state. + +## 2. Initialize the backend + +Download the provider and configure the S3 backend: + +```bash +terraform init -input=false +``` + +```text +Successfully configured the backend "s3"! Terraform will automatically +use this backend unless the backend configuration changes. +``` + +## 3. Apply the configuration + +Create the resource and write the state to RustFS: + +```bash +terraform apply -auto-approve -input=false +``` + +```text +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +## 4. Verify the state in RustFS + +List the state objects through the `rc` CLI: + +```bash +docker run --rm --network -v "$PWD:/m" \ + --entrypoint /bin/sh rustfs/rc:latest \ + -c 'rc alias set rustfs http://rustfs:9000 >/dev/null && rc ls rustfs/my-bucket/rustfs-demo --recursive' +``` + +```text +[2026-09-21 03:05:03] 1.62 KiB rustfs-demo/terraform.tfstate +``` + +You can also browse the prefix in the RustFS Console: + +![The Terraform state file stored in the RustFS Console](./images/rustfs-terraform-state.png) + +## 5. Confirm the state is reloaded from RustFS + +Run a plan — Terraform reads the state from RustFS and compares it against the configuration: + +```bash +terraform plan -input=false +``` + +```text +No changes. Your infrastructure matches the configuration. +``` + +The plan is clean because the state was read back from RustFS, not from any local file. + +## 6. Destroy and reset + +Tear the resource down; the state update is written to RustFS the same way: + +```bash +terraform destroy -auto-approve -input=false +``` + +Delete the state objects in the RustFS Console (or with `rc rm`) to start from scratch. + +## Troubleshooting + +### "dial tcp: lookup rustfs ... server misbehaving" or connection failures + +The `endpoint` must be reachable from the machine where Terraform runs. Inside a Compose network use `http://rustfs:9000`; from the host use `http://localhost:9000`. + +### "Bucket cannot have ACLs set" or signature errors + +`skip_credentials_validation`, `skip_region_validation`, `skip_metadata_api_check`, `skip_requesting_account_id`, `force_path_style`, and `insecure` are all required for a non-AWS endpoint; missing any of them makes the backend talk to AWS defaults instead of RustFS. + +### The state does not appear in the bucket + +Confirm that the bucket exists and that the credentials match the RustFS credentials. The state object appears under the `key` path (`rustfs-demo/terraform.tfstate` in this guide) after the first `init` or `apply`. + +## Next steps + +- Review [S3 compatibility notes](/administration/protocols/s3) before adopting additional S3 operations. +- Create dedicated production credentials with [Access Key Management](/security-compliance/iam/access-token). +- Follow the [Terraform S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) for options such as workspace prefixes and role assumption. diff --git a/content/ja/developer/integration/index.md b/content/ja/developer/integration/index.md index 7fc4bf29..075c44b3 100644 --- a/content/ja/developer/integration/index.md +++ b/content/ja/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "Integration" -description: "RustFS をリバースプロキシ、バックアップツール、データ分析システム、オブザーバビリティプラットフォーム、コンテナレジストリと連携させます。" +description: "RustFS をリバースプロキシ、バックアップツール、データ分析システム、オブザーバビリティプラットフォーム、コンテナレジストリ、DevOps ツールと連携させます。" --- このセクションでは、**RustFS** を S3 互換 API 経由でインフラストラクチャとアプリケーションプラットフォームに接続します。 @@ -13,6 +13,6 @@ description: "RustFS をリバースプロキシ、バックアップツール - [オブザーバビリティ](./observability/index.md) は OpenObserve を扱います。 - [その他](./others/index.md) はコミュニティ主導の Python 用 capo SDK を扱います。 - [コンテナレジストリ](./registry/index.md) は Harbor を扱います。 -- [コンテナレジストリ](./registry/index.md) は Harbor を扱います。 +- [DevOps](./devops/index.md) は Elasticsearch と Terraform を扱います。 各ガイドでは、連携先システムを設定する際に使用する RustFS のエンドポイントとアドレス指定の要件を示します。 \ No newline at end of file diff --git a/content/ja/developer/integration/meta.json b/content/ja/developer/integration/meta.json index 88d4e064..b4b29480 100644 --- a/content/ja/developer/integration/meta.json +++ b/content/ja/developer/integration/meta.json @@ -6,6 +6,7 @@ "big-data", "observability", "others", - "registry" + "registry", + "devops" ] } diff --git a/content/zh/developer/integration/devops/elasticsearch.md b/content/zh/developer/integration/devops/elasticsearch.md new file mode 100644 index 00000000..551bf915 --- /dev/null +++ b/content/zh/developer/integration/devops/elasticsearch.md @@ -0,0 +1,262 @@ +--- +title: "Elasticsearch" +description: "使用 Docker Compose 部署 Elasticsearch，以 RustFS 作为其索引的 S3 快照仓库。" +--- + +本指南将 [Elasticsearch](https://github.com/elastic/elasticsearch)——分布式搜索与分析引擎——连接到 **RustFS**，作为其索引的 S3 快照仓库。你将启动带 `repository-s3` 插件的 Elasticsearch，注册以 RustFS 为后端的快照仓库，写入文档，打快照，并通过删除索引后从 RustFS 恢复来验证完整闭环。整个流程使用 `docker.elastic.co/elasticsearch/elasticsearch:8.18.0` 和 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过。 + +你需要安装带有 Compose 插件的 Docker。本部署用于本地集成测试，不适用于生产环境。 + +## 架构 + +```mermaid +flowchart LR + Client["Indexing client"] -->|"index documents"| ES["Elasticsearch :9200"] + ES -->|"snapshot blobs"| RustFS["RustFS :9000"] + Init["init-bucket job"] -->|"create my-bucket"| RustFS +``` + +Elasticsearch 的主数据存放在本地磁盘，索引备份则卸载到快照仓库。`repository-s3` 插件通过 S3 API、以纯 HTTP 上的 path-style 寻址，把快照元数据和分片数据 blob 写入 RustFS。 + +## 1. 创建项目文件 + +创建工作目录： + +```bash +mkdir rustfs-elasticsearch +cd rustfs-elasticsearch +``` + +创建环境变量文件，并替换两个凭证占位符： + +```ini title=".env" +RUSTFS_ACCESS_KEY= +RUSTFS_SECRET_KEY= +``` + +请为桶使用专用的凭证，不要将 `.env` 提交到版本控制。 + +Elasticsearch 把 S3 设置分成两部分：非机密值放在 `elasticsearch.yml`，凭证放在 keystore。创建配置文件： + +```yaml title="elasticsearch.yml" +discovery.type: single-node +xpack.security.enabled: false +s3.client.default.endpoint: "rustfs:9000" +s3.client.default.protocol: "http" +s3.client.default.path_style_access: "true" +s3.client.default.region: "us-east-1" +``` + +创建 Compose 文件： + +```yaml title="compose.yaml" +services: + rustfs: + image: rustfs/rustfs-x86-musl:v2.3.1 + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + RUSTFS_VOLUMES: /data + RUSTFS_ADDRESS: ":9000" + RUSTFS_CONSOLE_ADDRESS: ":9001" + RUSTFS_CONSOLE_ENABLE: "true" + volumes: + - rustfs-data:/data + ports: + - "9000:9000" + - "9001:9001" + healthcheck: + test: ["CMD", "curl", "-sf", "http://127.0.0.1:9000/health"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 10s + networks: + - es + + create-bucket: + image: rustfs/rc:latest + depends_on: + rustfs: + condition: service_healthy + environment: + RUSTFS_ACCESS_KEY: ${RUSTFS_ACCESS_KEY} + RUSTFS_SECRET_KEY: ${RUSTFS_SECRET_KEY} + entrypoint: + - /bin/sh + - -c + - | + until /usr/bin/rc alias set rustfs http://rustfs:9000 "$${RUSTFS_ACCESS_KEY}" "$${RUSTFS_SECRET_KEY}"; do + echo "Waiting for RustFS..." + sleep 2 + done + /usr/bin/rc ls rustfs/my-bucket >/dev/null 2>&1 || /usr/bin/rc mb rustfs/my-bucket + networks: + - es + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:8.18.0 + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + volumes: + - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro + entrypoint: > + bash -c ' + bin/elasticsearch-plugin install --batch repository-s3 && + echo "$${RUSTFS_ACCESS_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.access_key && + echo "$${RUSTFS_SECRET_KEY}" | bin/elasticsearch-keystore add -f -x s3.client.default.secret_key && + exec bin/elasticsearch' + ports: + - "9200:9200" + depends_on: + create-bucket: + condition: service_completed_successfully + networks: + - es + +networks: + es: + +volumes: + rustfs-data: +``` + +`repository-s3` 是内置插件，无需联网即可安装。keystore 中只放访问密钥和私有密钥——端点等非机密设置必须写在 `elasticsearch.yml` 中，否则节点会拒绝启动。 + +## 2. 启动部署 + +启动容器前先解析 Compose 文件： + +```bash +docker compose config +``` + +启动服务并等待 Elasticsearch 完成启动（首次启动会安装插件并创建 keystore 条目，需要一两分钟）： + +```bash +docker compose up -d +curl -s http://localhost:9200 +``` + +## 3. 注册快照仓库并写入文档 + +注册 RustFS 快照仓库，创建索引并写入五个文档： + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo" \ + -H "Content-Type: application/json" \ + -d '{"type":"s3","settings":{"bucket":"my-bucket"}}' + +curl -s -X PUT "http://localhost:9200/rustfs_index" \ + -H "Content-Type: application/json" \ + -d '{"mappings":{"properties":{"label":{"type":"keyword"}}}}' + +for v in 1 2 3 4 5; do + curl -s -X POST "http://localhost:9200/rustfs_index/_doc" \ + -H "Content-Type: application/json" \ + -d "{\"label\":\"rustfs-es-doc-$v\",\"value\":$v}" > /dev/null +done +curl -s -X POST "http://localhost:9200/rustfs_index/_refresh" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +## 4. 打快照 + +使用 `wait_for_completion` 创建快照，立即得到结果： + +```bash +curl -s -X PUT "http://localhost:9200/_snapshot/rustfs_repo/snap1?wait_for_completion=true" +``` + +```text +{"snapshot":{"snapshot":"snap1",...,"indices":["rustfs_index"],"shards":{"total":1,"failed":0,"successful":1}}} +``` + +## 5. 在 RustFS 中验证对象 + +通过桶初始化镜像列出快照对象： + +```bash +docker compose run --rm --entrypoint /bin/sh create-bucket -c \ + '/usr/bin/rc alias set rustfs http://rustfs:9000 "$RUSTFS_ACCESS_KEY" "$RUSTFS_SECRET_KEY" >/dev/null && /usr/bin/rc ls rustfs/my-bucket/indices --recursive' +``` + +快照元数据和分片数据 blob 位于桶内 `indices/` 之下： + +```text +[2026-09-21 02:58:01] 3.56 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__LbLsSgUUTpqji_EXvqHYCg +[2026-09-21 02:58:01] 3.21 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/__VIuGNJn3RB-RBWvr4tMn7Q +[2026-09-21 02:58:01] 1.00 KiB indices/mvyat2-dSEOG3uiCCiFzRA/0/index-e9P8q6E9QhitpkiOdgjVDQ +``` + +你也可以在 RustFS 控制台中浏览该前缀： + +![RustFS 控制台中存储的 Elasticsearch 快照 blob](./images/rustfs-es-snapshots.png) + +## 6. 从 RustFS 恢复索引 + +删除索引，然后从快照恢复： + +```bash +curl -s -X DELETE "http://localhost:9200/rustfs_index" > /dev/null +curl -s -X POST "http://localhost:9200/_snapshot/rustfs_repo/snap1/_restore?wait_for_completion=true" > /dev/null +curl -s "http://localhost:9200/rustfs_index/_count" +``` + +```text +{"count":5,...} +``` + +文档全部恢复，因为快照 blob 是从 RustFS 读取的。 + +## 7. 停止或重置环境 + +停止容器并保留 RustFS 数据卷： + +```bash +docker compose down +``` + +如需删除快照并从空的 RustFS 数据卷开始，请显式加上 `--volumes`： + +```bash +docker compose down --volumes +``` + +## 故障排除 + +### 节点拒绝启动并提示 "non-secure setting ... must be stored inside elasticsearch.yml" + +keystore 中只允许存放 `s3.client.default.access_key` 和 `s3.client.default.secret_key`。端点、协议、path-style 和区域属于非机密设置，必须定义在 `elasticsearch.yml` 中。 + +### 恢复或新建的分片一直未分配 + +磁盘使用率超过低水位线（默认 85%）时，Elasticsearch 会停止分配分片。请释放磁盘空间，或在本地测试中关闭该检查： + +```bash +curl -s -X PUT "http://localhost:9200/_cluster/settings" \ + -H "Content-Type: application/json" \ + -d '{"transient":{"cluster.routing.allocation.disk.threshold_enabled":false}}' +``` + +### 恢复失败并提示索引已存在 + +之前失败的恢复会留下索引。先执行 `DELETE /rustfs_index` 删除，再重新恢复。 + +### 返回 AccessDenied 或 403 响应 + +确认 keystore 中的凭证与 RustFS 凭证一致，并确认 `create-bucket` 任务已成功完成： + +```bash +docker compose logs create-bucket +``` + +## 后续步骤 + +- 在采用其他 S3 操作前，请查看 [S3 兼容性说明](/administration/protocols/s3)。 +- 通过[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 按照 [Elasticsearch 快照文档](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html)了解快照生命周期管理（SLM）。 diff --git a/content/zh/developer/integration/devops/images/rustfs-es-snapshots.png b/content/zh/developer/integration/devops/images/rustfs-es-snapshots.png new file mode 100644 index 00000000..be78e0c1 Binary files /dev/null and b/content/zh/developer/integration/devops/images/rustfs-es-snapshots.png differ diff --git a/content/zh/developer/integration/devops/images/rustfs-terraform-state.png b/content/zh/developer/integration/devops/images/rustfs-terraform-state.png new file mode 100644 index 00000000..6757ffcd Binary files /dev/null and b/content/zh/developer/integration/devops/images/rustfs-terraform-state.png differ diff --git a/content/zh/developer/integration/devops/index.md b/content/zh/developer/integration/devops/index.md new file mode 100644 index 00000000..396b1f28 --- /dev/null +++ b/content/zh/developer/integration/devops/index.md @@ -0,0 +1,13 @@ +--- +title: "DevOps" +description: "通过 S3 兼容的对象存储接口，将 DevOps 平台与基础设施工具连接到 RustFS。" +--- + +将 **RustFS** 用作支持 S3 兼容端点的 DevOps 平台与基础设施工具的对象存储层。 + +## 平台与工具 + +- [Elasticsearch](./elasticsearch.md) +- [Terraform](./terraform.md) + +请使用专用的存储桶保存制品、状态与遥测数据，并为凭证仅授予所需桶操作的权限。 diff --git a/content/zh/developer/integration/devops/meta.json b/content/zh/developer/integration/devops/meta.json new file mode 100644 index 00000000..a032acbd --- /dev/null +++ b/content/zh/developer/integration/devops/meta.json @@ -0,0 +1,7 @@ +{ + "title": "DevOps", + "pages": [ + "elasticsearch", + "terraform" + ] +} diff --git a/content/zh/developer/integration/devops/terraform.md b/content/zh/developer/integration/devops/terraform.md new file mode 100644 index 00000000..7b0ca970 --- /dev/null +++ b/content/zh/developer/integration/devops/terraform.md @@ -0,0 +1,144 @@ +--- +title: "Terraform" +description: "使用 S3 后端把 Terraform 状态（含状态锁）存储在 RustFS 对象存储中。" +--- + +本指南将 HashiCorp 的基础设施即代码工具 [Terraform](https://github.com/hashicorp/terraform) 通过 S3 backend 连接到 **RustFS**：状态文件以及防止并发运行的状态锁都以对象形式存储在 RustFS 中。你将初始化后端、应用一个小配置，并在 RustFS 中验证状态对象。整个流程使用 `terraform 1.12.2` 和 `rustfs/rustfs-x86-musl:v2.3.1` 验证通过。 + +你需要一个可从工作站访问的 RustFS 部署（参见[安装](/installation)章节），以及支持通过 `use_lockfile` 实现 S3 原生锁的 Terraform 1.10 或更高版本。 + +## 架构 + +```mermaid +flowchart LR + Client["terraform init / apply"] -->|"state + lock objects"| RustFS["RustFS :9000"] +``` + +S3 backend 把状态文件存储在桶内 `key` 前缀之下。启用 `use_lockfile = true` 后，运行期间 Terraform 会写入一个 `.tflock` 对象作为锁，取代本场景中基于 DynamoDB 的锁。 + +## 1. 创建项目文件 + +创建工作目录： + +```bash +mkdir rustfs-terraform +cd rustfs-terraform +``` + +创建配置。请替换凭证占位符，并把端点指向你的 RustFS 服务器——Terraform 与其同机时使用 `localhost:9000`： + +```hcl title="main.tf" +terraform { + backend "s3" { + bucket = "my-bucket" + key = "rustfs-demo/terraform.tfstate" + region = "us-east-1" + endpoint = "http://localhost:9000" + access_key = "" + secret_key = "" + skip_credentials_validation = true + skip_region_validation = true + skip_metadata_api_check = true + skip_requesting_account_id = true + force_path_style = true + insecure = true + use_lockfile = true + } +} + +provider "local" {} + +resource "local_file" "demo" { + content = "provisioned with terraform, state stored in RustFS" + filename = "${path.module}/demo.txt" +} +``` + +`force_path_style` 和 `insecure` 表示对本地端点使用纯 HTTP 上的 path-style 寻址，这正是 RustFS 所期望的。`use_lockfile` 启用 S3 原生状态锁，防止并发运行损坏状态。 + +## 2. 初始化后端 + +下载 provider 并配置 S3 后端： + +```bash +terraform init -input=false +``` + +```text +Successfully configured the backend "s3"! Terraform will automatically +use this backend unless the backend configuration changes. +``` + +## 3. 应用配置 + +创建资源并把状态写入 RustFS： + +```bash +terraform apply -auto-approve -input=false +``` + +```text +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +## 4. 在 RustFS 中验证状态 + +通过 `rc` CLI 列出状态对象： + +```bash +docker run --rm --network -v "$PWD:/m" \ + --entrypoint /bin/sh rustfs/rc:latest \ + -c 'rc alias set rustfs http://rustfs:9000 >/dev/null && rc ls rustfs/my-bucket/rustfs-demo --recursive' +``` + +```text +[2026-09-21 03:05:03] 1.62 KiB rustfs-demo/terraform.tfstate +``` + +你也可以在 RustFS 控制台中浏览该前缀： + +![RustFS 控制台中存储的 Terraform 状态文件](./images/rustfs-terraform-state.png) + +## 5. 确认状态是从 RustFS 重新读取的 + +执行 plan——Terraform 会从 RustFS 读取状态并与配置比对： + +```bash +terraform plan -input=false +``` + +```text +No changes. Your infrastructure matches the configuration. +``` + +plan 结果干净，说明状态是从 RustFS 读回的，而不是来自任何本地文件。 + +## 6. 销毁并重置 + +销毁资源；状态更新同样写回 RustFS： + +```bash +terraform destroy -auto-approve -input=false +``` + +在 RustFS 控制台（或用 `rc rm`）删除状态对象即可从零开始。 + +## 故障排除 + +### "dial tcp: lookup rustfs ... server misbehaving" 或连接失败 + +`endpoint` 必须能被运行 Terraform 的机器访问。Compose 网络内使用 (`http://rustfs:9000`)，宿主机上使用 (`http://localhost:9000`)。 + +### "Bucket cannot have ACLs set" 或签名错误 + +对于非 AWS 端点，`skip_credentials_validation`、`skip_region_validation`、`skip_metadata_api_check`、`skip_requesting_account_id`、`force_path_style` 和 `insecure` 都必须设置；缺任何一个都会导致 backend 与 AWS 默认端点通信而不是 RustFS。 + +### 状态没有出现在桶里 + +确认桶已存在且凭证与 RustFS 凭证一致。状态对象会在第一次 `init` 或 `apply` 之后出现在 `key` 路径下（本指南为 `rustfs-demo/terraform.tfstate`）。 + +## 后续步骤 + +- 在采用其他 S3 操作前，请查看 [S3 兼容性说明](/administration/protocols/s3)。 +- 通过[访问密钥管理](/security-compliance/iam/access-token)创建专用的生产凭证。 +- 阅读 [Terraform S3 backend 文档](https://developer.hashicorp.com/terraform/language/backend/s3)了解 workspace 前缀、角色扮演等选项。 diff --git a/content/zh/developer/integration/index.md b/content/zh/developer/integration/index.md index 1bfbf253..d5c4a62f 100644 --- a/content/zh/developer/integration/index.md +++ b/content/zh/developer/integration/index.md @@ -1,6 +1,6 @@ --- title: "集成" -description: "将 RustFS 与反向代理、备份工具、数据分析系统、可观测性平台和容器镜像仓库集成。" +description: "将 RustFS 与反向代理、备份工具、数据分析系统、可观测性平台、容器镜像仓库和 DevOps 工具集成。" --- 通过 S3 兼容 API 将 **RustFS** 连接到基础设施和应用平台。 @@ -13,6 +13,6 @@ description: "将 RustFS 与反向代理、备份工具、数据分析系统、 - [可观测性](./observability/index.md)涵盖 OpenObserve。 - [其他](./others/index.md)涵盖社区驱动的 Python capo SDK。 - [镜像仓库](./registry/index.md)涵盖 Harbor。 -- [镜像仓库](./registry/index.md)涵盖 Harbor。 +- [DevOps](./devops/index.md)涵盖 Elasticsearch 和 Terraform。 每篇指南都会说明配置集成系统时需要使用的 RustFS 端点和寻址要求。 \ No newline at end of file diff --git a/content/zh/developer/integration/meta.json b/content/zh/developer/integration/meta.json index 88d4e064..b4b29480 100644 --- a/content/zh/developer/integration/meta.json +++ b/content/zh/developer/integration/meta.json @@ -6,6 +6,7 @@ "big-data", "observability", "others", - "registry" + "registry", + "devops" ] }